Commit graph

2476 commits

Author SHA1 Message Date
Eelco Dolstra ad716378dc Fix download-via-ssh
http://hydra.nixos.org/build/13286020
2014-08-17 14:28:17 +02:00
Eelco Dolstra fde819f2e2 nix-log2xml: Handle newlines 2014-08-13 19:06:20 +02:00
Eelco Dolstra cb921f67c3 Remove log2html.xsl and friends
It's part of Hydra now.
2014-08-13 19:01:14 +02:00
Eelco Dolstra e9b609bf9a nix-log2xml: Handle UTF-8 characters
C++ chars can be negative...
2014-08-13 19:00:07 +02:00
Eelco Dolstra 9367046fef Use regular file GC roots if possible
This makes hydra-eval-jobs create roots as regular files. See
1c208f2b7e.
2014-08-13 17:44:41 +02:00
Eelco Dolstra 5f05197df4 Handle compound single dash options properly
So now

  nix-instantiate --eval -E '{x}: x' --argstr x -xyzzy

correctly prints "-xyzzy", rather than giving an error.

Issue NixOS/hydra#176.
2014-08-13 04:08:33 +02:00
Eelco Dolstra 47e185847e Refactor option handling 2014-08-13 03:50:44 +02:00
Eelco Dolstra 5bed74d1b0 Fix warning about non-existant -I directories 2014-08-13 02:57:59 +02:00
Eelco Dolstra 401ab50566 Remove pointless NIX_LOG_TYPE environment variable 2014-08-13 01:19:02 +02:00
Eelco Dolstra 98873ff7de Remove unnecessary call to addTempRoot() 2014-08-05 16:41:42 +02:00
Eelco Dolstra f02c52e3d4 Doh 2014-08-05 10:19:57 +02:00
Eelco Dolstra 988bf59421 Move some options out of globals 2014-08-04 18:13:14 +02:00
Eelco Dolstra daccd68999 Refactor 2014-08-04 18:02:29 +02:00
Eelco Dolstra d5a076c36f Add option ‘build-extra-chroot-dirs’
This is useful for extending (rather than overriding) the default set
of chroot paths.
2014-08-04 18:00:00 +02:00
Eelco Dolstra 4d73e2e893 Get rid of "killing <pid>" message for unused build hooks 2014-08-04 17:27:45 +02:00
Eelco Dolstra bb45092f72 Make chroot builds easier to set up
By default, we now include /bin/sh as a bind-mount of bash.
2014-08-04 17:09:26 +02:00
Eelco Dolstra 51485dcda2 Remove ugly hack for detecting build environment setup errors 2014-08-01 19:38:21 +02:00
Eelco Dolstra 4c4b219c07 Call commonChildInit() before doing chroot init
This ensures that daemon clients see error messages from the chroot
setup.
2014-08-01 19:29:03 +02:00
Eelco Dolstra eea0401d7a Eliminate redundant copy 2014-08-01 17:30:51 +02:00
Eelco Dolstra e0d7d0e45c findRoots(): Prevent a call to lstat()
This means that getting the roots from /nix/var/nix/.../hydra-roots
doesn't need any I/O other than reading the directory.
2014-08-01 17:20:25 +02:00
Eelco Dolstra daf3f2c11f Make readDirectory() return inode / file type 2014-08-01 17:14:28 +02:00
Eelco Dolstra 1c208f2b7e Allow regular files as GC roots
If a root is a regular file, then its name must denote a store
path. For instance, the existence of the file

  /nix/var/nix/gcroots/per-user/eelco/hydra-roots/wzc3cy1wwwd6d0dgxpa77ijr1yp50s6v-libxml2-2.7.7

would cause

  /nix/store/wzc3cy1wwwd6d0dgxpa77ijr1yp50s6v-libxml2-2.7.7

to be a root.

This is useful because it involves less I/O (no need for a readlink()
call) and takes up less disk space (the symlink target typically takes
up a full disk block, while directory entries are packed more
efficiently). This is particularly important for hydra.nixos.org,
which has hundreds of thousands of roots, and where reading the roots
can take 25 minutes.
2014-08-01 17:14:28 +02:00
Eelco Dolstra 50dc1f5b71 Restore default SIGPIPE handler before invoking ‘man’
Fixes NixOS/nixpkgs#3410.
2014-07-31 10:31:17 +02:00
Eelco Dolstra 3d221a7bb1 Rename nixPath to __nixPath
The name ‘nixPath’ breaks existing code.
2014-07-30 11:28:39 +02:00
Eelco Dolstra 67937907ca nix-daemon: Pass on the user's $SSH_AUTH_SOCK to the SSH substituter 2014-07-25 18:02:48 +02:00
Eelco Dolstra 71a20d4d95 Change the default for use-ssh-substituter to ‘true’
Now you only have to pass ‘--option ssh-substituter-hosts
nix-ssh@bla’ to enable SSH substitution.
2014-07-25 12:57:12 +02:00
Eelco Dolstra 1a44f1cb65 nix-copy-closure: Drop --bzip2, --xz, --show-progress
These are too difficult to implement via nix-store --serve.

‘--show-progress’ could be re-implemented fairly easily via a
sink/source wrapper class.
2014-07-24 17:11:54 +02:00
Eelco Dolstra 03103c0a36 Implement nix-copy-closure --from via nix-store --serve 2014-07-24 16:19:41 +02:00
Eelco Dolstra 62309a2c56 build-remote.pl: Be less verbose on failing builds 2014-07-24 12:43:59 +02:00
Eelco Dolstra d7a92ed4a9 nix-store --serve: Only monitor stdin during builds
Other operations cannot hang indefinitely (except when we're reading
from stdin, in which case we'll notice a client disconnect). But
monitoring works badly during compressed imports, since there the
client can close the connection before we've sent an ack.

http://hydra.nixos.org/build/12711638
2014-07-24 11:59:35 +02:00
Eelco Dolstra 0fae20c362 Use pthread_cancel instead of a signal
Signal handlers are process-wide, so sending SIGINT to the monitor
thread will cause the normal SIGINT handler to run. This sets the
isInterrupted flag, which is not what we want. So use pthread_cancel
instead.
2014-07-24 11:47:51 +02:00
Eelco Dolstra aa1560ca07 Fix bogus pass by reference
http://hydra.nixos.org/build/12711659
2014-07-24 09:58:07 +02:00
Eelco Dolstra 24c6d992c6 More debugging 2014-07-24 01:21:56 +02:00
Eelco Dolstra 1eb0af7ed5 Add some assertions 2014-07-24 00:16:06 +02:00
Eelco Dolstra d3c61d83be Remove some dead code 2014-07-24 00:00:53 +02:00
Eelco Dolstra 62ad3dfc43 Remove some obsolete files 2014-07-23 23:56:58 +02:00
Eelco Dolstra bd91453bb1 Pass -pthread only for programs that need it 2014-07-23 19:48:18 +02:00
Eelco Dolstra ece531d105 nix-daemon: Less verbosity 2014-07-23 19:43:46 +02:00
Eelco Dolstra d0fad1779f nix-daemon: Simplify stderr handling 2014-07-23 19:37:40 +02:00
Eelco Dolstra 78224cd871 nix-store --serve: Monitor for client disconnects
This is necessary because build-remote.pl now builds via ‘nix-store
--serve’. So if a build hangs without writing to stdout/stderr, and
the client disconnects, then we need to detect that.
2014-07-23 19:26:34 +02:00
Eelco Dolstra 49fe9592a4 nix-daemon: Use a thread instead of SIGPOLL to catch client disconnects
The thread calls poll() to wait until a HUP (or other error event)
happens on the client connection. If so, it sends SIGINT to the main
thread, which is then cleaned up normally. This is much nicer than
messing around with SIGPOLL.
2014-07-23 19:21:00 +02:00
Eelco Dolstra fdee1ced43 startProcess: Make writing error messages from the child more robust 2014-07-23 19:11:26 +02:00
Eelco Dolstra 5989966ed3 Remove dead code 2014-07-23 14:46:28 +02:00
Eelco Dolstra ee3c5d7916 Revert old useBuildHook behaviour 2014-07-19 02:25:47 +02:00
Eelco Dolstra 2e77bd70fa Better fix for strcasecmp on Darwin 2014-07-18 12:54:30 +02:00
Eelco Dolstra 8ddffe7aac Ugly hack to fix building on old Darwin
http://hydra.nixos.org/build/12580878
2014-07-17 23:57:17 +02:00
Eelco Dolstra 049c0eb49c nix-daemon: Add trusted-users and allowed-users options
‘trusted-users’ is a list of users and groups that have elevated
rights, such as the ability to specify binary caches. It defaults to
‘root’. A typical value would be ‘@wheel’ to specify all users in the
wheel group.

‘allowed-users’ is a list of users and groups that are allowed to
connect to the daemon. It defaults to ‘*’. A typical value would be
‘@users’ to specify the ‘users’ group.
2014-07-17 16:57:07 +02:00
Eelco Dolstra 0c730887c4 nix-daemon: Show name of connecting user 2014-07-17 15:49:33 +02:00
Eelco Dolstra 77c972c898 nix-daemon: Only print connection info if we have SO_PEERCRED 2014-07-17 15:44:51 +02:00
Eelco Dolstra 8f72e702a1 nix-daemon: Fix compat with older clients 2014-07-17 15:23:31 +02:00
Eelco Dolstra 2304a7dd21 Get rid of a compiler warning 2014-07-16 16:32:26 +02:00
Eelco Dolstra 985f1595fe Be more strict about file names in NARs 2014-07-16 16:30:50 +02:00
Eelco Dolstra 276a40b31f Handle case collisions on case-insensitive systems
When running NixOps under Mac OS X, we need to be able to import store
paths built on Linux into the local Nix store. However, HFS+ is
usually case-insensitive, so if there are directories with file names
that differ only in case, then importing will fail.

The solution is to add a suffix ("~nix~case~hack~<integer>") to
colliding files. For instance, if we have a directory containing
xt_CONNMARK.h and xt_connmark.h, then the latter will be renamed to
"xt_connmark.h~nix~case~hack~1". If a store path is dumped as a NAR,
the suffixes are removed. Thus, importing and exporting via a
case-insensitive Nix store is round-tripping. So when NixOps calls
nix-copy-closure to copy the path to a Linux machine, you get the
original file names back.

Closes #119.
2014-07-16 16:02:05 +02:00
Eelco Dolstra fa13d3f4f3 build-remote.pl: Fix building multiple output derivations
We were importing paths without sorting them topologically, leading to
"path is not valid" errors.

See e.g. http://hydra.nixos.org/build/12451761
2014-07-14 12:19:27 +02:00
Eelco Dolstra a5c6347ff0 build-remote.pl: Use ‘nix-store --serve’ on the remote side
This makes things more efficient (we don't need to use an SSH master
connection, and we only start a single remote process) and gets rid of
locking issues (the remote nix-store process will keep inputs and
outputs locked as long as they're needed).

It also makes it more or less secure to connect directly to the root
account on the build machine, using a forced command
(e.g. ‘command="nix-store --serve --write"’). This bypasses the Nix
daemon and is therefore more efficient.

Also, don't call nix-store to import the output paths.
2014-07-11 16:22:24 +02:00
Eelco Dolstra e196eecbe6 Allow $NIX_BUILD_HOOK to be relative to Nix libexec directory 2014-07-11 13:55:06 +02:00
Eelco Dolstra d0eb970fb4 Fix broken Pid constructor 2014-07-10 21:48:21 +02:00
Eelco Dolstra edbfe2232e Replace message "importing path <...>" with "exporting path <...>"
This causes nix-copy-closure to show what it's doing before rather
than after.
2014-07-10 21:30:22 +02:00
Eelco Dolstra 7bb632b024 nix-copy-closure -s: Do substitutions via ‘nix-store --serve’
This means we no longer need an SSH master connection, since we only
execute a single command on the remote host.
2014-07-10 20:43:04 +02:00
Eelco Dolstra 43b64f5038 Remove tabs 2014-07-10 17:32:21 +02:00
Eelco Dolstra 8e9140cfde Refactoring: Move all fork handling into a higher-order function
C++11 lambdas ftw.
2014-07-10 16:58:09 +02:00
Eelco Dolstra 1114c7bd57 nix-copy-closure: Restore compression and the progress viewer 2014-07-10 14:15:12 +02:00
Eelco Dolstra 7911e4c27a Remove maybeVfork 2014-07-10 13:35:44 +02:00
Eelco Dolstra 04170d06bf nix-copy-closure: Fix race condition
There is a long-standing race condition when copying a closure to a
remote machine, particularly affecting build-remote.pl: the client
first asks the remote machine which paths it already has, then copies
over the missing paths. If the garbage collector kicks in on the
remote machine between the first and second step, the already-present
paths may be deleted. The missing paths may then refer to deleted
paths, causing nix-copy-closure to fail. The client now performs both
steps using a single remote Nix call (using ‘nix-store --serve’),
locking all paths in the closure while querying.

I changed the --serve protocol a bit (getting rid of QueryCommand), so
this breaks the SSH substituter from older versions. But it was marked
experimental anyway.

Fixes #141.
2014-07-10 11:58:59 +02:00
Eelco Dolstra 2c3a8f787b Fix security hole in ‘nix-store --serve’
Since it didn't check that the path received from the client is a
store path, the client could dump any path in the file system.
2014-07-10 11:46:01 +02:00
Eelco Dolstra 0e5d0c1543 Fix compilation error on some versions of GCC
src/libexpr/primops.cc:42:8: error: looser throw specifier for 'virtual nix::InvalidPathError::~InvalidPathError()'
src/libexpr/nixexpr.hh:12:1: error:   overriding 'virtual nix::EvalError::~EvalError() noexcept (true)'

http://hydra.nixos.org/build/12385750
2014-07-09 12:14:40 +02:00
Eelco Dolstra beaf3e90af Add builtin function ‘fromJSON’
Fixes #294.
2014-07-04 13:34:15 +02:00
Eelco Dolstra e477f0e938 Style fix 2014-06-27 11:36:23 +02:00
Paul Colomiets 858b8f9760 Add --json argument to nix-instantiate 2014-06-27 11:23:40 +02:00
Eelco Dolstra 8504e7d604 allow-arbitrary-code-during-evaluation -> allow-unsafe-native-code-during-evaluation 2014-06-27 11:20:16 +02:00
Shea Levy d62f46e500 Only add the importNative primop if the allow-arbitrary-code-during-evaluation option is true (default false) 2014-06-24 10:50:03 -04:00
Shea Levy 5cd022d6c0 Add importNative primop
This can be used to import a dynamic shared object and return an
arbitrary value, including new primops. This can be used both to test
new primops without having to recompile nix every time, and to build
specialized primops that probably don't belong upstream (e.g. a function
that calls out to gpg to decrypt a nixops secret as-needed).

The imported function should initialize the Value & as needed. A single
import can define multiple values by creating an attrset or list, of
course.

An example initialization function might look like:

extern "C" void initialize(nix::EvalState & state, nix::Value & v)
{
    v.type = nix::tPrimOp;
    v.primOp = NEW nix::PrimOp(myFun, 1, state.symbols.create("myFun"));
}

Then `builtins.importNative ./example.so "initialize"` will evaluate to
the primop defined in the myFun function.
2014-06-17 12:08:01 -04:00
Eelco Dolstra 9d0709e8c4 Don't use member initialisers
They're a little bit too recent (only supported since GCC 4.7).

http://hydra.nixos.org/build/11851475
2014-06-12 17:30:37 +02:00
Eelco Dolstra 48495f67ed Fix bogus warnings about dumping large paths
Also, yay for C++11 non-static initialisers.
2014-06-12 13:15:35 +02:00
Eelco Dolstra 0960d674d4 Drop ImportError and FindError
We're not catching these anywhere.
2014-06-12 13:00:54 +02:00
Shea Levy 718f20da6d findFile: Realise the context of the path attributes 2014-06-12 12:57:14 +02:00
Shea Levy a8fb575c98 Share code between scopedImport and import
In addition to reducing duplication, this fixes both import from
derivation and import of derivation for scopedImport
2014-06-12 12:52:39 +02:00
Eelco Dolstra ee7fe64c0a == operator: Ignore string context
There really is no case I can think of where taking the context into
account is useful. Mostly it's just very inconvenient.
2014-06-10 14:02:56 +02:00
Eelco Dolstra b1beed97a0 Report daemon OOM better
When copying a large path causes the daemon to run out of memory, you
now get:

  error: Nix daemon out of memory

instead of:

  error: writing to file: Broken pipe
2014-06-10 13:45:50 +02:00
Eelco Dolstra 829af22759 Print a warning when loading a large path into memory
I.e. if you have a derivation with

  src = ./huge-directory;

you'll get a warning that this is not a good idea.
2014-06-10 13:30:09 +02:00
Eelco Dolstra 3c6b8a5215 nix-env -qa --json: Generate valid JSON even if there are invalid meta attrs 2014-06-02 17:58:43 +02:00
Eelco Dolstra becc2b0167 Sort nixPath attributes 2014-05-29 19:02:14 +02:00
Eelco Dolstra 54a34119f3 Use std::unordered_set 2014-05-26 17:53:17 +02:00
Eelco Dolstra d8c061e044 Remove ExprBuiltin
It's slower than ExprVar since it doesn't compute a static
displacement. Since we're not using the throw primop in the
implementation of <...> anymore, it's also not really needed.
2014-05-26 17:14:28 +02:00
Eelco Dolstra 62a6eeb1f3 Make the Nix search path declarative
Nix search path lookups like <nixpkgs> are now desugared to ‘findFile
nixPath <nixpkgs>’, where ‘findFile’ is a new primop. Thus you can
override the search path simply by saying

  let
    nixPath = [ { prefix = "nixpkgs"; path = "/my-nixpkgs"; } ];
  in ... <nixpkgs> ...

In conjunction with ‘scopedImport’ (commit
c273c15cb1), the Nix search path can be
propagated across imports, e.g.

  let

    overrides = {
      nixPath = [ ... ] ++ builtins.nixPath;
      import = fn: scopedImport overrides fn;
      scopedImport = attrs: fn: scopedImport (overrides // attrs) fn;
      builtins = builtins // overrides;
    };

  in scopedImport overrides ./nixos
2014-05-26 17:02:22 +02:00
Eelco Dolstra 39d72640c2 Ensure that -I flags get included in nixPath
Also fixes #261.
2014-05-26 16:52:31 +02:00
Eelco Dolstra a8edf185a9 Add constant ‘nixPath’
It contains the Nix expression search path as a list of { prefix, path
} sets, e.g.

  [ { path = "/nix/var/nix/profiles/per-user/root/channels/nixos"; prefix = ""; }
    { path = "/etc/nixos/configuration.nix"; prefix = "nixos-config"; }
    { path = "/home/eelco/Dev/nix/inst/share/nix/corepkgs"; prefix = "nix"; }
  ]
2014-05-26 14:55:47 +02:00
Eelco Dolstra c273c15cb1 Add primop ‘scopedImport’
‘scopedImport’ works like ‘import’, except that it takes a set of
attributes to be added to the lexical scope of the expression,
essentially extending or overriding the builtin variables.  For
instance, the expression

  scopedImport { x = 1; } ./foo.nix

where foo.nix contains ‘x’, will evaluate to 1.

This has a few applications:

* It allows getting rid of function argument specifications in package
  expressions. For instance, a package expression like:

    { stdenv, fetchurl, libfoo }:

    stdenv.mkDerivation { ... buildInputs = [ libfoo ]; }

  can now we written as just

    stdenv.mkDerivation { ... buildInputs = [ libfoo ]; }

  and imported in all-packages.nix as:

    bar = scopedImport pkgs ./bar.nix;

  So whereas we once had dependencies listed in three places
  (buildInputs, the function, and the call site), they now only need
  to appear in one place.

* It allows overriding builtin functions. For instance, to trace all
  calls to ‘map’:

  let
    overrides = {
      map = f: xs: builtins.trace "map called!" (map f xs);

      # Ensure that our override gets propagated by calls to
      # import/scopedImport.
      import = fn: scopedImport overrides fn;

      scopedImport = attrs: fn: scopedImport (overrides // attrs) fn;

      # Also update ‘builtins’.
      builtins = builtins // overrides;
    };
  in scopedImport overrides ./bla.nix

* Similarly, it allows extending the set of builtin functions. For
  instance, during Nixpkgs/NixOS evaluation, the Nixpkgs library
  functions could be added to the default scope.

There is a downside: calls to scopedImport are not memoized, unlike
import. So importing a file multiple times leads to multiple parsings
/ evaluations. It would be possible to construct the AST only once,
but that would require careful handling of variables/environments.
2014-05-26 14:26:29 +02:00
Eelco Dolstra f0fdbd0897 Shut up some signedness warnings 2014-05-26 12:34:15 +02:00
Eelco Dolstra 0321ef9bb2 Ugly hack to allow --argstr values starting with a dash
Fixes #265.
2014-05-23 14:43:58 +02:00
Eelco Dolstra 9f9080e2c0 nix-store -l: Fetch build logs from the Internet
If a build log is not available locally, then ‘nix-store -l’ will now
try to download it from the servers listed in the ‘log-servers’ option
in nix.conf. For instance, if you have:

  log-servers = http://hydra.nixos.org/log

then it will try to get logs from http://hydra.nixos.org/log/<base
name of the store path>. So you can do things like:

  $ nix-store -l $(which xterm)

and get a log even if xterm wasn't built locally.
2014-05-21 17:19:36 +02:00
Shea Levy eac5841970 Provide a more useful error message when a dynamic attr lookup fails 2014-05-15 17:56:24 +02:00
Eelco Dolstra 8d5f472f2c lvlInfo -> lvlTalkative 2014-05-15 11:37:44 +02:00
Eelco Dolstra 84813af5b9 nix-store --optimise: Remove bogus statistics 2014-05-15 11:33:46 +02:00
Eelco Dolstra 690adeb03d Remove tab 2014-05-15 11:19:16 +02:00
Eelco Dolstra a1b66f316e Merge branch 'master' of github.com:wmertens/nix 2014-05-15 11:18:29 +02:00
Wout Mertens 3b9ea8452f Shortcut store files before lstat
readdir() already returns the inode numbers, so we don't need to call
lstat to know if a file was already linked or not.
2014-05-15 09:02:22 +02:00
Wout Mertens d73ffc552f Use the inodes given by readdir directly 2014-05-14 22:52:10 +02:00
Wout Mertens e974f20c98 Preload linked hashes to speed up lookups
By preloading all inodes in the /nix/store/.links directory, we can
quickly determine of a hardlinked file was already linked to the hashed
links.
This is tolerant of removing the .links directory, it will simply
recalculate all hashes in the store.
2014-05-13 23:10:06 +02:00
Eelco Dolstra 95501c4dee nix-instantiate --eval: Apply auto-arguments if the result is a function
Fixes #254.
2014-05-13 12:56:48 +02:00
wmertens a84f503d86 Shortcut already-hardlinked files
If an inode in the Nix store has more than 1 link, it probably means that it was linked into .links/ by us. If so, skip.

There's a possibility that something else hardlinked the file, so it would be nice to be able to override this.

Also, by looking at the number of hardlinks for each of the files in .links/, you can get deduplication numbers and space savings.
2014-05-10 15:53:01 +02:00
Eelco Dolstra 696f960dee Set up directories and permissions for multi-user install automatically
This automatically creates /nix/var/nix/profiles/per-user and sets the
permissions/ownership on /nix/store to 1775 and root:nixbld.
2014-05-02 14:31:15 +02:00
Eelco Dolstra de4cdd0d47 Set build-max-jobs to the number of available cores by default
More zero configuration.
2014-05-02 12:51:43 +02:00
Eelco Dolstra ada3e3fa15 When running as root, use build users by default
This removes the need to have a nix.conf, and prevents people from
accidentally running Nix builds as root.
2014-05-02 12:46:03 +02:00
Ricardo M. Correia 700c678c2e nix-env: Minor change to '--delete-generations Nd' semantics
The option '--delete-generations Nd' deletes all generations older than N
days. However, most likely the user does not want to delete the
generation that was active N days ago.

For example, say that you have these 3 generations:

1: <30 days ago>
2: <15 days ago>
3: <1 hour ago>

If you do --delete-generations 7d (say, as part of a cron job), most
likely you still want to keep generation 2, i.e. the generation that was
active 7 days ago (and for most of the past 7 days, in fact).

This patch fixes this issue. Note that this also affects
'nix-collect-garbage --delete-older-than Nd'.

Thanks to @roconnor for noticing the issue!
2014-04-15 15:34:58 +02:00
Eelco Dolstra dfa2f77d2e If a .drv cannot be parsed, show its path
Otherwise you just get ‘expected string `Derive(['’ which isn't very helpful.
2014-04-08 19:24:29 +02:00
Eelco Dolstra 8e5fbf4d73 Show position info in attribute selection errors 2014-04-04 22:52:14 +02:00
Eelco Dolstra 4c5faad994 Show position info in Boolean operations 2014-04-04 22:43:52 +02:00
Eelco Dolstra bd9b1d97b4 Show position info in string concatenation / addition errors 2014-04-04 22:19:33 +02:00
Eelco Dolstra a5fe730940 forceString: Show position info 2014-04-04 21:14:11 +02:00
Eelco Dolstra 27b44b8cf7 forceAttrs: Show position info 2014-04-04 19:11:40 +02:00
Eelco Dolstra 96b695ccab forceList: Show position info 2014-04-04 19:05:36 +02:00
Eelco Dolstra b62d36963c forceInt: Show position info 2014-04-04 18:59:29 +02:00
Eelco Dolstra c28de6d96e Pass position information to primop calls
For example:

  error: `tail' called on an empty list, at
    /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/ex-2/default.nix:13:7
2014-04-04 18:59:29 +02:00
Eelco Dolstra 8b31ffd10d Remove unnecessary quotes around file names 2014-04-04 18:59:29 +02:00
Eelco Dolstra b72c8d2e5b Include position info in function application
This allows error messages like:

  error: the anonymous function at `/etc/nixos/configuration.nix:1:1'
    called without required argument `foo', at
    `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:77:59'
2014-04-04 18:59:29 +02:00
Danny Wilson ae6b631dc4 Fix compile errors on Illumos 2014-04-03 17:39:57 +02:00
Eelco Dolstra f0de86357c Tweak error message 2014-04-03 15:24:02 +02:00
Ludovic Courtès e7720aa10a Make sure /dev/pts/ptmx is world-writable
While running Python 3’s test suite, we noticed that on some systems
/dev/pts/ptmx is created with permissions 0 (that’s the case with my
Nixpkgs-originating 3.0.43 kernel, but someone with a Debian-originating
3.10-3 reported not having this problem.)

There’s still the problem that people without
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y are screwed (as noted in build.cc),
but I don’t see how we could work around it.
2014-04-03 13:42:45 +02:00
Eelco Dolstra ac6ceea764 Fix potential segfault
The newEnv variable was accessed (via the dynamicEnv) pointer after it
had gone out of scope.

Fixes #234.
2014-04-01 17:04:38 +02:00
Ricardo M. Correia 7ef7597f71 nix-env: Add support for --delete-generations 15d
It will delete all generations older than the specified number of days.
2014-03-30 00:54:16 +01:00
Eelco Dolstra 1c2550a2ae boost::shared_ptr -> std::shared_ptr 2014-03-30 00:49:23 +01:00
Eelco Dolstra 9becaa041f Drop pointless #include 2014-03-29 22:20:33 +01:00
Eelco Dolstra acb8facbbc Fix potential segfault in waitForInput()
Since the addition of build-max-log-size, a call to
handleChildOutput() can result in cancellation of a goal.  This
invalidated the "j" iterator in the waitForInput() loop, even though
it was still used afterwards.  Likewise for the maxSilentTime
handling.

Probably fixes #231.  At least it gets rid of the valgrind warnings.
2014-03-29 22:14:11 +01:00
Eelco Dolstra 90dc50b07c restoreSIGPIPE(): Fill in sa_mask
Issue #231.
2014-03-29 20:20:14 +01:00
Eelco Dolstra 49009573bc Don't interpret strings as format strings
Ludo reported this error:

  unexpected Nix daemon error: boost::too_few_args: format-string refered to more arguments than were passed

coming from this line:

  printMsg(lvlError, run.program + ": " + string(err, 0, p));

The problem here is that the string ends up implicitly converted to a
Boost format() object, so % characters are treated specially.  I
always assumed (wrongly) that strings are converted to a format object
that outputs the string as-is.

Since this assumption appears in several places that may be hard to
grep for, I've added some C++ type hackery to ensures that the right
thing happens.  So you don't have to worry about % in statements like

  printMsg(lvlError, "foo: " + s);

or

  throw Error("foo: " + s);
2014-03-28 16:59:26 +01:00
Ludovic Courtès 24cb65efc3 Make /dev/kvm optional
The daemon now creates /dev deterministically (thanks!).  However, it
expects /dev/kvm to be present.

The patch below restricts that requirement (1) to Linux-based systems,
and (2) to systems where /dev/kvm already exists.

I’m not sure about the way to handle (2).  We could special-case
/dev/kvm and create it (instead of bind-mounting it) in the chroot, so
it’s always available; however, it wouldn’t help much since most likely,
if /dev/kvm missing, then KVM support is missing.
2014-03-21 17:27:19 +01:00
Eelco Dolstra 3fc056927c Fix tabs 2014-03-18 23:23:55 +01:00
Ludovic Courtès 51800e06de Allow recovery from isValidPath RPCs with an invalid path
Currently, clients cannot recover from an isValidPath RPC with an
invalid path parameter because the daemon closes the connection when
that happens.

More precisely:

  1. in performOp, wopIsValidPath case, ‘readStorePath’ raises an
     ‘Error’ exception;

  2. that exception is caught by the handler in ‘processConnection’;

  3. the handler determines errorAllowed == false, and thus exits after
     sending the message.

This last part is fixed by calling ‘startWork’ early on, as in the patch
below.

The same reasoning could be applied to all the RPCs that take one or
more store paths as inputs, but isValidPath is, by definition, likely to
be passed invalid paths in the first place, so it’s important for this
one to allow recovery.
2014-03-18 23:22:49 +01:00
Eelco Dolstra f93e97517e Fix -j and other flags when using the daemon 2014-03-17 17:35:11 +01:00
Eelco Dolstra fb8d8f5428 Remove unnecessary null pointer checks
Fixes #225.
2014-03-12 14:42:25 +01:00
Eelco Dolstra 25386e5edc Fix passing meta attribute to buildenv.nix
Since the meta attributes were not sorted, attribute lookup could
fail, leading to package priorities and active flags not working
correctly.

Broken since 0f24400d90.
2014-03-11 17:34:02 +01:00
Shea Levy 049a379ec6 The expr of AttrNames/DynamicAttrDefs is always an ExprConcatStrings 2014-03-10 10:14:50 +01:00
Shea Levy 908e9ce259 If a dynamic attribute name evaluates to null, remove it from the set 2014-03-10 10:14:50 +01:00
Eelco Dolstra 2caab81660 Revert "Make ifs and asserts tail-recursive"
This reverts commit 273322c773.
2014-03-05 16:18:13 +01:00
Eelco Dolstra 1017bd68ea Set up a private /dev/pts in the chroot 2014-02-27 23:35:23 +01:00
Eelco Dolstra 3fd01b171a Set up a minimal /dev in chroots
Not bind-mounting the /dev from the host also solves the problem with
/dev/shm being a symlink to something not in the chroot.
2014-02-27 23:17:53 +01:00
Eelco Dolstra c9f6232304 Correctly detect infinite recursion in function application
If we're evaluating some application ‘v = f x’, we can't store ‘f’
temporarily in ‘v’, because if ‘f x’ refers to ‘v’, it will get ‘f’
rather than an infinite recursion error.

Unfortunately, this breaks the tail call optimisation introduced in
c897bac549.

Fixes #217.
2014-02-27 21:47:59 +01:00
Eelco Dolstra 29cde917fe Fix deadlock in SubstitutionGoal
We were relying on SubstitutionGoal's destructor releasing the lock,
but if a goal is a top-level goal, the destructor won't run in a
timely manner since its reference count won't drop to zero.  So
release it explicitly.

Fixes #178.
2014-02-27 13:34:13 +01:00
Eelco Dolstra 91f25f0510 And another one 2014-02-26 18:55:18 +01:00
Eelco Dolstra 432328cc55 Remove another unused function 2014-02-26 18:49:36 +01:00
Eelco Dolstra 509993e598 Remove unused function 2014-02-26 18:48:32 +01:00
Eelco Dolstra 6b9cd59a41 nix-store -r: Respect --add-root for non-derivations
Fixes #68.
Fixes #117.
2014-02-26 16:32:46 +01:00
Eelco Dolstra f14ef84a51 Warn about missing -I paths
Fixes #121.  Note that we don't warn about missing $NIX_PATH entries
because it's intended that some may be missing (cf. the default
$NIX_PATH on NixOS, which includes paths like /etc/nixos/nixpkgs for
backward compatibility).
2014-02-26 15:21:56 +01:00
Eelco Dolstra 42eb4afd7a Simplify getting use-ssh-substituter from untrusted users 2014-02-26 13:58:46 +01:00
Eelco Dolstra dcaea042fc Only start download-via-ssh if it's enabled 2014-02-26 13:41:03 +01:00
Shea Levy df5de9dfd7 Add use-ssh-substituter setting.
It defaults to false and can be overridden by RemoteStore.

Untested currently, just quickly put this together
2014-02-26 13:32:47 +01:00
Eelco Dolstra a897b58373 nix-instantiate: Allow --dry-run as a synonym for --readonly-mode
--dry-run is more consistent with nix-env and nix-store.
2014-02-19 16:46:33 +01:00
Eelco Dolstra e1cf40fa95 nix-instantiate: Rename --eval-only to --eval, --parse-only to --parse 2014-02-19 16:34:24 +01:00
Eelco Dolstra c31836008e nix-instantiate: Add a flag --expr / -E to read expressions from the command line
This is basically a shortcut for ‘echo 'expr...' | nix-instantiate -’.
Also supported by nix-build and nix-shell.
2014-02-19 16:30:19 +01:00
Shea Levy a7e70518b8 lexer-tab.o and parser-tab.o require each other's headers 2014-02-18 17:31:30 +01:00
Eelco Dolstra 8129cf33d9 Slight simplification 2014-02-18 10:46:30 +01:00
Eelco Dolstra 1aa19b24b2 Add a flag ‘--check’ to verify build determinism
The flag ‘--check’ to ‘nix-store -r’ or ‘nix-build’ will cause Nix to
redo the build of a derivation whose output paths are already valid.
If the new output differs from the original output, an error is
printed.  This makes it easier to test if a build is deterministic.
(Obviously this cannot catch all sources of non-determinism, but it
catches the most common one, namely the current time.)

For example:

  $ nix-build '<nixpkgs>' -A patchelf
  ...
  $ nix-build '<nixpkgs>' -A patchelf --check
  error: derivation `/nix/store/1ipvxsdnbhl1rw6siz6x92s7sc8nwkkb-patchelf-0.6' may not be deterministic: hash mismatch in output `/nix/store/4pc1dmw5xkwmc6q3gdc9i5nbjl4dkjpp-patchelf-0.6.drv'

The --check build fails if not all outputs are valid.  Thus the first
call to nix-build is necessary to ensure that all outputs are valid.

The current outputs are left untouched: the new outputs are either put
in a chroot or diverted to a different location in the store using
hash rewriting.
2014-02-18 01:01:14 +01:00
Eelco Dolstra b6def5b542 Make --repair work on Darwin
Mac OS X doesn't allow renaming a read-only directory.

http://hydra.nixos.org/build/9113895
2014-02-17 23:09:48 +01:00
Eelco Dolstra dfbcb7c403 Refactoring 2014-02-17 23:04:52 +01:00
Eelco Dolstra 71adb090f0 When using a build hook, only copy missing paths 2014-02-17 22:58:21 +01:00
Eelco Dolstra 69fe6c58fa Move some code around
In particular, do replacing of valid paths during repair later.  This
prevents us from replacing a valid path after the build fails.
2014-02-17 22:25:15 +01:00
Eelco Dolstra 1da6ae4f99 nix-store --gc --max-freed: Support a unit specifier
E.g. "--max-freed 10G" means "free ten gigabytes".
2014-02-17 14:48:50 +01:00
Eelco Dolstra 00d30496ca Heuristically detect if a build may have failed due to a full disk
This will allow Hydra to detect that a build should not be marked as
"permanently failed", allowing it to be retried later.
2014-02-17 14:15:56 +01:00
Eelco Dolstra a9d99ab55f download-via-ssh: Use readStorePath 2014-02-14 12:31:10 +01:00
Eelco Dolstra 4db572062c download-via-ssh: Show where we're downloading from 2014-02-14 12:20:12 +01:00
Eelco Dolstra dba33d4018 Minor style fixes 2014-02-14 11:48:42 +01:00
Shea Levy f67f52751f Indendation fix
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-12 07:33:07 -05:00
Shea Levy 62eb9eb76d Remove relic of old code
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-12 07:27:45 -05:00
Shea Levy 7438f0bc2b error messages start in lowercase
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-12 07:26:35 -05:00
Shea Levy 2246aa77d2 Remove using declarations from download-via-ssh
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-12 07:22:36 -05:00
Shea Levy c89d6b9b63 nix-store --serve: Use a versioned protocol
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 07:43:13 -05:00
Shea Levy 38c3beac1a Move StoreApi::serve into opServe
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:52:48 -05:00
Shea Levy 1614603165 Pass in params by const ref
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:49:37 -05:00
Shea Levy 78d979567f Clarify comment
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:43:29 -05:00
Shea Levy 64e23d0a38 Add download-via-ssh substituter
This substituter connects to a remote host, runs nix-store --serve
there, and then forwards substituter commands on to the remote host and
sends their results to the calling program. The ssh-substituter-hosts
option can be specified as a list of hosts to try.

This is an initial implementation and, while it works, it has some
limitations:

* Only the first host is used
* There is no caching of query results (all queries are sent to the
  remote machine)
* There is no informative output (such as progress bars)
* Some failure modes may cause unhelpful error messages
* There is no concept of trusted-ssh-substituter-hosts

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy 5671188eb2 nix-store --serve: Flush out after every loop
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy 73874629ef nix-store --serve: Use dump instead of export
Also remove signing support

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy 188f96500b nix-store --serve: Don't fail if asked for info about non-valid path
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy 9488447594 nix-store --serve: Don't loop forever
nix-store --export takes a tmproot, which can only release by exiting.
Substituters don't currently work in a way that could take advantage of
the looping, anyway.

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:32 -05:00
Shea Levy 3a38d0f356 Add the nix-store --serve command
This is essentially the substituter API operating on the local store,
which will be used by the ssh substituter. It runs in a loop rather than
just taking one command so that in the future nix will be able to keep
one connection open for multiple instances of the substituter.

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:32 -05:00
Shea Levy 84a8b5e9af nix-instantiate --eval-only --read-write-mode: Don't depend on ordering
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-07 18:03:38 +01:00
Shea Levy e4058fab64 Rename --no-readonly-mode --read-write-mode
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-07 18:03:38 +01:00
Shea Levy 0c3e8a616e nix-instantiate: Add a --no-readonly-mode flag
This allows running nix-instantiate --eval-only without performing the
evaluation in readonly mode, letting features like import from
derivation and automatic substitution of builtins.storePath paths work.

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-07 18:03:38 +01:00
Eelco Dolstra d210cdc435 Fix assertion failure in ‘nix-store --load-db’
Namely:

  nix-store: derivations.cc:242: nix::Hash nix::hashDerivationModulo(nix::StoreAPI&, nix::Derivation): Assertion `store.isValidPath(i->first)' failed.

This happened because of the derivation output correctness check being
applied before the references of a derivation are valid.
2014-02-03 22:36:07 +01:00
Eelco Dolstra d6582c04c1 Give a friendly error message if the DB directory is not writable
Previously we would say "error: setting synchronous mode: unable to
open database file" which isn't very helpful.
2014-02-01 16:57:38 +01:00
Eelco Dolstra 844d83352c More "make dist" fixes 2014-02-01 15:18:48 +01:00
Eelco Dolstra 6ef32bddc1 Fix "make dist" 2014-02-01 14:38:12 +01:00
Eelco Dolstra 0c6d62cf27 Remove Automakefiles 2014-02-01 13:54:38 +01:00
Eelco Dolstra 16e7d69209 Update Makefile variable names 2014-02-01 13:54:38 +01:00
Eelco Dolstra e0234dfddc Rename Makefile -> local.mk 2014-01-30 12:11:06 +01:00
Eelco Dolstra 4a2ec9c659 Install nix-worker symlink 2014-01-30 12:08:26 +01:00
Eelco Dolstra 94f9c14d52 Fix some clang warnings 2014-01-21 18:29:55 +01:00
Eelco Dolstra 68cde53c47 Fix building against Bison 3.0.2 2014-01-21 15:34:04 +01:00
Eelco Dolstra 81628a6ccc Merge branch 'master' into make
Conflicts:
	src/libexpr/eval.cc
2014-01-21 15:30:01 +01:00
Eelco Dolstra 5ef8508a92 Remove unused type 2014-01-21 15:11:57 +01:00
Shea Levy f5e5793cd2 Bare dynamic attrs: Match interpolation semantics
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-01-14 14:00:15 +01:00
Shea Levy f9913f4422 Allow "bare" dynamic attrs
Now, in addition to a."${b}".c, you can write a.${b}.c (applicable
wherever dynamic attributes are valid).

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-01-14 14:00:15 +01:00
Eelco Dolstra b1db599dd0 Generate schema.sql.hh 2014-01-09 22:10:35 +01:00
Eelco Dolstra b4c684e0f9 Update Makefiles 2014-01-09 16:53:47 +01:00
Eelco Dolstra 11cb4bfb25 Fix checking of NAR hashes
*headdesk*
*headdesk*
*headdesk*

So since commit 22144afa8d, Nix hasn't
actually checked whether the content of a downloaded NAR matches the
hash specified in the manifest / NAR info file.  Urghhh...
2014-01-08 17:35:49 +01:00
Eelco Dolstra 7a61c88dbb Merge branch 'dynamic-attrs-no-sugar' of github.com:shlevy/nix 2014-01-06 15:46:18 +01:00
Domen Kožar 485f4740ee wording 2014-01-06 11:38:24 +01:00
Shea Levy cd49fe4f9b Don't use any syntactic sugar for dynamic attrs
This doesn't change any functionality but moves some behavior out of the
parser and into the evaluator in order to simplify the code.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-31 23:56:26 +00:00
Shea Levy 6f3a51809a Fold dynamic binds handling into addAttr
Since addAttr has to iterate through the AttrPath we pass it, it makes
more sense to just iterate through the AttrNames in addAttr instead. As
an added bonus, this allows attrsets where two dynamic attribute paths
have the same static leading part (see added test case for an example
that failed previously).

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-31 17:57:10 -05:00
Shea Levy 18fefacf7d Dynamic attrs
This adds new syntax for attribute names:

* attrs."${name}" => getAttr name attrs
* attrs ? "${name}" => isAttrs attrs && hasAttr attrs name
* attrs."${name}" or def => if attrs ? "${name}" then attrs."${name}" else def
* { "${name}" = value; } => listToAttrs [{ inherit name value; }]

Of course, it's a bit more complicated than that. The attribute chains
can be arbitrarily long and contain combinations of static and dynamic
parts (e.g. attrs."${foo}".bar."${baz}" or qux), which is relatively
straightforward for the getAttrs/hasAttrs cases but is more complex for
the listToAttrs case due to rules about duplicate attribute definitions.

For attribute sets with dynamic attribute names, duplicate static
attributes are detected at parse time while duplicate dynamic attributes
are detected when the attribute set is forced. So, for example, { a =
null; a.b = null; "${"c"}" = true; } will be a parse-time error, while
{ a = {}; "${"a"}".b = null; c = true; } will be an eval-time error
(technically that case could theoretically be detected at parse time,
but the general case would require full evaluation). Moreover, duplicate
dynamic attributes are not allowed even in cases where they would be
with static attributes ({ a.b.d = true; a.b.c = false; } is legal, but {
a."${"b"}".d = true; a."${"b"}".c = false; } is not). This restriction
might be relaxed in the future in cases where the static variant would
not be an error, but it is not obvious that that is desirable.

Finally, recursive attribute sets with dynamic attributes have the
static attributes in scope but not the dynamic ones. So rec { a = true;
"${"b"}" = a; } is equivalent to { a = true; b = true; } but rec {
"${"a"}" = true; b = a; } would be an error or use a from the
surrounding scope if it exists.

Note that the getAttr, getAttr or default, and hasAttr are all
implemented purely in the parser as syntactic sugar, while attribute
sets with dynamic attribute names required changes to the AST to be
implemented cleanly.

This is an alternative solution to and closes #167

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-31 20:59:49 +00:00
Shea Levy 136f2f7046 Add the ExprBuiltin Expr type to the AST
Certain desugaring schemes may require the parser to use some builtin
function to do some of the work (e.g. currently `throw` is used to
lazily cause an error if a `<>`-style path is not in the search path)
Unfortunately, these names are not reserved keywords, so an expression
that uses such a syntactic sugar will not see the expected behavior
(see tests/lang/eval-okay-redefine-builtin.nix for an example).

This adds the ExprBuiltin AST type, which when evaluated uses the value
from the rootmost variable scope (which of course is initialized
internally and can't shadow any of the builtins).

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-31 17:45:47 +00:00
Eelco Dolstra 8931bf7168 Doh 2013-12-20 13:09:12 +00:00
Eelco Dolstra 7d203faff6 nix-env --set-flag: Barf if a selector doesn't match any installed package
Fixes #184.
2013-12-20 13:56:42 +01:00
Eelco Dolstra 769f662165 nix-shell: Don't warn about the lack of a GC root 2013-12-20 12:19:10 +00:00
Eelco Dolstra 65a6452240 nix-shell: Handle --option correctly
Fixes #181.
2013-12-20 13:10:14 +01:00
Shea Levy e36229d27f Bump language version for new storePath feature
This will allow e.g. channel expressions to use builtins.storePath IFF
it is safe to do so without knowing if the path is valid yet.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-10 15:07:04 +01:00
Shea Levy 22d665019a builtins.storePath: Try to substitute the path if it is not yet valid
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-10 15:07:04 +01:00
Eelco Dolstra a6add93d73 Garbage collector: Release locks on temporary root files
This allows processes waiting for such locks to proceed during the
trash deletion phase of the garbage collector.
2013-12-10 13:13:59 +01:00
Eelco Dolstra c5b8fe3151 Print a trace message if a build fails due to the platform being unknown 2013-12-05 14:31:57 -05:00
Eelco Dolstra f3cf0436b5 Install bsdiff and bspatch in $(libexecdir)/nix 2013-12-02 20:21:31 +00:00
Eelco Dolstra 0202ce6b94 Add support for ‘make installcheck’ 2013-11-25 18:47:03 +01:00
Eelco Dolstra 7ce0e05ad8 Rename Makefile.new -> Makefile 2013-11-25 15:25:13 +00:00
Eelco Dolstra 1a1d8b073a Add a Makefile for bsdiff 2013-11-25 11:39:21 +00:00
Eelco Dolstra 2bd0fcc966 Use libnix as a prefix for all Nix libraries
In particular "libutil" was always a problem because it collides with
Glibc's libutil.  Even if we install into $(libdir)/nix, the linker
sometimes got confused (e.g. if a program links against libstore but
not libutil, then ld would report undefined symbols in libstore
because it was looking at Glibc's libutil).
2013-11-23 23:53:41 +00:00
Eelco Dolstra abb5bd66de Add Makefile for nix-log2xml 2013-11-23 23:42:50 +00:00
Eelco Dolstra c1f3a1a89b Disallow undefined symbols in dynamic libraries by default
This encourages that each library declares its own dependencies
properly.
2013-11-24 00:10:00 +01:00
Eelco Dolstra 5a1114ecdb Drop the dependency on libgc in libmain
Instead, libexpr now depends on libgc.  This means commands like
nix-store that don't do any evaluation no longer require libgc.
2013-11-23 20:19:36 +00:00
Eelco Dolstra 06a8ac96e7 Initialise Boehm GC only once 2013-11-23 20:15:57 +00:00
Eelco Dolstra 90dfb37f14 Allow (dynamic) libraries to depend on other libraries 2013-11-23 20:11:02 +00:00
Eelco Dolstra 611868a909 Implement basic ‘make install’ 2013-11-23 17:04:27 +00:00
Eelco Dolstra d1b3ca0b4a Improve building dynamic libraries
They now get a correct RPATH.
2013-11-23 17:26:43 +01:00
Eelco Dolstra 6dd74b48f8 Support building dynamic libraries 2013-11-22 20:56:58 +00:00
Eelco Dolstra 9b11a8bfbc Fix building without Boehm GC 2013-11-22 20:05:00 +00:00
Eelco Dolstra 6b5f89f2cf Drop the dependency on Automake 2013-11-22 19:30:24 +00:00
Eelco Dolstra 754c05ed6c Rename $(here) to $(d) for brevity, and remove trailing slash 2013-11-22 16:45:52 +00:00
Eelco Dolstra 62e35cc3a8 Add ‘make dist’ support 2013-11-22 16:42:25 +01:00
Eelco Dolstra eff6c4b791 Add missing #include 2013-11-22 15:41:48 +00:00
Eelco Dolstra 1474ecfe42 Generate the parser and the lexer 2013-11-22 16:22:31 +01:00
Eelco Dolstra b8e9efc476 New non-recursive, plain Make-based build system 2013-11-22 15:54:18 +01:00
Eelco Dolstra 709cbe4e76 Include <cstring> for memset
This should fix building on Illumos.
2013-11-22 10:00:43 +00:00
Eelco Dolstra 30b986908e Check meta values and warn about bad ones 2013-11-19 14:29:39 +01:00
Eelco Dolstra 0f24400d90 Generalise meta attributes 2013-11-19 14:09:14 +01:00
Eelco Dolstra 990126cde0 Shorter error message 2013-11-19 14:09:14 +01:00
Eelco Dolstra af94a70ba6 Drop support for user environment manifests in ATerm format 2013-11-19 14:09:14 +01:00
Eelco Dolstra 245e26408f nix-env -q: Add a --json flag 2013-11-19 00:41:45 +01:00
Eelco Dolstra 5fea98111b Refactor JSON output 2013-11-19 00:33:06 +01:00
Eelco Dolstra 77c13cdf56 Add a toJSON primop 2013-11-19 00:04:11 +01:00
Eelco Dolstra 285df765b9 Add a primop unsafeGetAttrPos to return the position of an attribute 2013-11-18 22:22:35 +01:00
Eelco Dolstra fc33fd86b7 Add a symbol __curPos that expands to the current source location
I.e. an attribute set { file = <string>; line = <int>; column = <int>; }.
2013-11-18 20:16:02 +01:00
Eelco Dolstra 90b5e69284 Support quoted attribute names in -A
This is requires if you have attribute names with dots in them.  So
you can now say:

  $ nix-instantiate '<nixos>' -A 'config.systemd.units."postgresql.service".text' --eval-only

Fixes #151.
2013-11-18 10:21:12 +00:00
Eelco Dolstra a478e8a7bb Remove nix-setuid-helper
AFAIK, nobody uses it, it's not maintained, and it has no tests.
2013-11-14 11:57:37 +01:00
Eelco Dolstra 89e6781cc5 Make function calls show up in stack traces again
Note that adding --show-trace prevents functions calls from being
tail-recursive, so an expression that evaluates without --show-trace
may fail with a stack overflow if --show-trace is given.
2013-11-12 12:51:59 +01:00
Eelco Dolstra c897bac549 Make function calls tail-recursive 2013-11-12 11:32:23 +00:00
Eelco Dolstra 273322c773 Make ifs and asserts tail-recursive
The local Value object prevented g++ from making a tail call.  Not
clear why.  In any case, not using a temporary makes g++ do the tail
call.
2013-11-12 11:32:23 +00:00
Eelco Dolstra 4badd7ed17 Get rid of an intermediary on the stack 2013-11-12 11:32:23 +00:00
Eelco Dolstra 8d6418d46e Fix building without a garbage collector
http://hydra.nixos.org/build/6695350
2013-10-28 22:51:12 +01:00
Eelco Dolstra dec2f19502 Fix a segfault in genericClosure
It kept temporary data in STL containers that were not scanned by
Boehm GC, so Nix programs using genericClosure could randomly crash if
the garbage collector kicked in at a bad time.

Also make it a bit more efficient by copying points to values rather
than values.
2013-10-28 18:52:26 +01:00
Eelco Dolstra ea6bf0c21f Slightly optimize listToAttrs 2013-10-28 07:34:44 +01:00
Eelco Dolstra 5bc41d78ff Rename "attribute sets" to "sets"
We don't have any other kind of sets so calling them attribute sets is
unnecessarily verbose.
2013-10-24 16:41:04 +02:00
Eelco Dolstra 69befd33a9 Remove unnecessary call to forceStringNoCtx 2013-10-24 03:08:34 +02:00
Eelco Dolstra 05d02f798f Add a typeOf primop
We already have some primops for determining the type of a value, such
as isString, but they're incomplete: for instance, there is no isPath.
Rather than adding more isBla functions, the generic typeOf function
returns a string representing the type of the argument (e.g. "int").
2013-10-24 02:49:13 +02:00
Eelco Dolstra 543d8a5942 Don't require NIX_SHOW_STATS for NIX_COUNT_CALLS 2013-10-24 02:20:54 +02:00
Eelco Dolstra fe95650487 Memoize evalFile() lookups under both the original and resolved name
Previously we only used the resolved name, causing repeated resolution
(e.g. /dir to /dir/default.nix).
2013-10-23 11:19:01 +00:00
Eelco Dolstra c086183843 For auto roots, show the intermediate link
I.e. "nix-store -q --roots" will now show (for example)

  /home/eelco/Dev/nixpkgs/result

rather than

  /nix/var/nix/gcroots/auto/53222qsppi12s2hkap8dm2lg8xhhyk6v
2013-10-22 11:39:10 +02:00
Eelco Dolstra 792fd51f41 Fold two stack trace messages in derivations
Combined with the previous changes, stack traces involving derivations
are now much less verbose, since something like

  while evaluating the builtin function `getAttr':
  while evaluating the builtin function `derivationStrict':
  while instantiating the derivation named `gtk+-2.24.20' at `/home/eelco/Dev/nixpkgs/pkgs/development/libraries/gtk+/2.x.nix:11:3':
  while evaluating the derivation attribute `propagatedNativeBuildInputs' at `/home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/default.nix:78:17':
  while evaluating the attribute `outPath' at `/nix/store/212ngf4ph63mp6p1np2bapkfikpakfv7-nix-1.6/share/nix/corepkgs/derivation.nix:18:9':
  ...

now reads

  while evaluating the attribute `propagatedNativeBuildInputs' of the derivation `gtk+-2.24.20' at `/home/eelco/Dev/nixpkgs/pkgs/development/libraries/gtk+/2.x.nix:11:3':
  ...
2013-10-17 11:57:25 +02:00
Eelco Dolstra f440558acc Don't show <nix/derivation.nix> in stack traces
Messages like

  while evaluating the attribute `outPath' at `/nix/store/212ngf4ph63mp6p1np2bapkfikpakfv7-nix-1.6/share/nix/corepkgs/derivation.nix:18:9':

are redundant, because Nix already shows that it's evaluating a derivation:

  while instantiating the derivation named `firefox-24.0' at `/home/eelco/Dev/nixpkgs/pkgs/applications/networking/browsers/firefox/default.nix:131:5':
  while evaluating the derivation attribute `nativeBuildInputs' at `/home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/default.nix:76:17':
2013-10-17 11:47:38 +02:00
Eelco Dolstra b8034e5581 Ensure proper type checking/coercion of "${expr}"
Now we only rewrite "${expr}" to expr if expr is a string literal.
2013-10-17 00:57:24 +02:00
Eelco Dolstra d6a7aa8f48 Revert the behaviour of antiquoted paths to pre-Nix 1.6
Commit 159e621d1a accidentally changed
the behaviour of antiquoted paths, e.g.

  "${/foo}/bar"

used to evaluate to "/nix/store/<hash>-foo/bar" (where /foo gets
copied to the store), but in Nix 1.6 it evaluates to "/foo/bar".  This
is inconsistent, since

  " ${/foo}/bar"

evaluates to " /nix/store/<hash>-foo/bar".  So revert to the old
behaviour.
2013-10-17 00:39:59 +02:00
Eelco Dolstra a737f51fd9 Retry all SQLite operations
To deal with SQLITE_PROTOCOL, we also need to retry read-only
operations.
2013-10-16 15:58:20 +02:00
Eelco Dolstra ff02f5336c Fix a race in registerFailedPath()
Registering the path as failed can fail if another process does the
same thing after the call to hasPathFailed().  This is extremely
unlikely though.
2013-10-16 14:55:53 +02:00
Eelco Dolstra 4bd5282573 Convenience macros for retrying a SQLite transaction 2013-10-16 14:46:35 +02:00
Eelco Dolstra bce14d0f61 Don't wrap read-only queries in a transaction
There is no risk of getting an inconsistent result here: if the ID
returned by queryValidPathId() is deleted from the database
concurrently, subsequent queries involving that ID will simply fail
(since IDs are never reused).
2013-10-16 14:36:53 +02:00
Eelco Dolstra 7cdefdbe73 Print a distinct warning for SQLITE_PROTOCOL 2013-10-16 14:27:36 +02:00
Eelco Dolstra d05bf04444 Treat SQLITE_PROTOCOL as SQLITE_BUSY
In the Hydra build farm we fairly regularly get SQLITE_PROTOCOL errors
(e.g., "querying path in database: locking protocol").  The docs for
this error code say that it "is returned if some other process is
messing with file locks and has violated the file locking protocol
that SQLite uses on its rollback journal files."  However, the SQLite
source code reveals that this error can also occur under high load:

  if( cnt>5 ){
    int nDelay = 1;                      /* Pause time in microseconds */
    if( cnt>100 ){
      VVA_ONLY( pWal->lockError = 1; )
      return SQLITE_PROTOCOL;
    }
    if( cnt>=10 ) nDelay = (cnt-9)*238;  /* Max delay 21ms. Total delay 996ms */
    sqlite3OsSleep(pWal->pVfs, nDelay);
  }

i.e. if certain locks cannot be not acquired, SQLite will retry a
number of times before giving up and returing SQLITE_PROTOCOL.  The
comments say:

  Circumstances that cause a RETRY should only last for the briefest
  instances of time.  No I/O or other system calls are done while the
  locks are held, so the locks should not be held for very long. But
  if we are unlucky, another process that is holding a lock might get
  paged out or take a page-fault that is time-consuming to resolve,
  during the few nanoseconds that it is holding the lock.  In that case,
  it might take longer than normal for the lock to free.
  ...
  The total delay time before giving up is less than 1 second.

On a heavily loaded machine like lucifer (the main Hydra server),
which often has dozens of processes waiting for I/O, it seems to me
that a page fault could easily take more than a second to resolve.
So, let's treat SQLITE_PROTOCOL as SQLITE_BUSY and retry the
transaction.

Issue NixOS/hydra#14.
2013-10-16 14:19:59 +02:00
Eelco Dolstra 7bdb85453d printStats(): Print the size of the symbol table in bytes 2013-10-08 15:37:08 +02:00
Eelco Dolstra 9deb822180 Deduplicate filenames in Pos
This saves ~4 MiB of RAM for NixOS system instantiation, and ~18 MiB
for "nix-env -qa".
2013-10-08 15:36:10 +02:00
Eelco Dolstra b1e3b1a4ac Treat undefined variable errors consistently
Previously, a undefined variable inside a "with" caused an EvalError
(which can be caught), while outside, it caused a ParseError (which
cannot be caught).  Now both cause an UndefinedVarError (which cannot
be caught).
2013-10-08 14:45:36 +02:00
Eelco Dolstra 6b47de580f Show the exact position of undefined variables
In particular, undefined variable errors in a "with" previously didn't
show *any* position information, so this should help a lot in those
cases.
2013-10-08 14:40:51 +02:00
Eelco Dolstra a5e0f64db3 Remove some unused functions 2013-10-08 12:30:23 +00:00
Eelco Dolstra 221a2daf34 Merge VarRef into ExprVar 2013-10-08 14:24:53 +02:00
Eelco Dolstra 176c666f36 Don't show calls to primops in stack traces
Since they don't have location information, they just give you crap
like:

  while evaluating the builtin function `getAttr':
  while evaluating the builtin function `derivationStrict':
  ...
2013-10-07 18:02:32 +02:00
Eelco Dolstra c945f015de Fix segfault in nix-repl / hydra-eval-jobs
If a "with" attribute set fails to evaluate, we have to make sure its
Env record remains unchanged.  Otherwise, repeated evaluation gives a
segfault:

  nix-repl> :a with 0; { a = x; b = x; }
  Added 2 variables.

  nix-repl> a
  error: value is an integer while an attribute set was expected

  nix-repl> b
  Segmentation fault
2013-10-02 15:24:45 +02:00
Eelco Dolstra 28e0742966 Report OOM errors better 2013-10-02 14:34:36 +02:00
Eelco Dolstra a5fb4b5b7c Fix typo 2013-10-02 14:22:49 +02:00
Eelco Dolstra f53574ebd6 RestoreSink: Slightly reduce the number of concurrent FDs 2013-09-17 12:06:59 +00:00
Eelco Dolstra 0220da3e10 Remove stray debug line 2013-09-06 17:20:19 +02:00
Eelco Dolstra 936f9d45ba Don't apply the CPU affinity hack to nix-shell (and other Perl programs)
As discovered by Todd Veldhuizen, the shell started by nix-shell has
its affinity set to a single CPU.  This is because nix-shell connects
to the Nix daemon, which causes the affinity hack to be applied.  So
we turn this off for Perl programs.
2013-09-06 16:36:56 +02:00
Eelco Dolstra 2c1ecf8e81 nix-env -i: Add a flag ‘--remove-all’ / ‘-r’
This is equivalent to running ‘nix-env -e '*'’ first, except that it
happens in a single transaction.  Thus, ‘nix-env -i pkgs...’ replaces
the profile with the specified set of packages.

The main motivation is to support declarative package management
(similar to environment.systemPackages in NixOS).  That is, if you
have a specification ‘profile.nix’ like this:

  with import <nixpkgs> {};
  [ thunderbird
    geeqie
    ...
  ]

then after any change to ‘profile.nix’, you can run:

  $ nix-env -f profile.nix -ir

to update the profile to match the specification.  (Without the ‘-r’
flag, if you remove a package from ‘profile.nix’, it won't be removed
from the actual profile.)

Suggested by @zefhemel.
2013-09-03 21:21:14 +02:00
Eelco Dolstra 88c07341a6 nix-env: Use wildcard match by default
That is, you don't need to pass '*' anymore, so

  nix-env -qa

is equivalent to

  nix-env -qa '*'
2013-09-03 16:35:49 +02:00
Eelco Dolstra 07a08bddf0 nix-env: Load files in ~/.nix-defexpr on demand
So if you do "nix-env -qa -A nixos", then other channels won't be
parsed/evaluated at all.
2013-09-03 15:45:32 +02:00
Eelco Dolstra c57ed84e28 Check for name collisions in the input Nix expressions 2013-09-03 15:25:51 +02:00
Eelco Dolstra ef4f5ba85e Work on Values instead of Exprs
This prevents some duplicate evaluation in nix-env and
nix-instantiate.

Also, when traversing ~/.nix-defexpr, only read regular files with the
extension .nix.  Previously it was reading files like
.../channels/binary-caches/<name>.  The only reason this didn't cause
problems is pure luck (namely, <name> shadows an actual Nix
expression, the binary-caches files happen to be syntactically valid
Nix expressions, and we iterate over the directory contents in just
the right order).
2013-09-03 13:17:51 +00:00
Eelco Dolstra 06bb2d95b4 Reformat 2013-09-03 11:04:21 +00:00
Eelco Dolstra 6f809194d7 Get rid of the parse tree cache
Since we already cache files in normal form (fileEvalCache), caching
parse trees is redundant.

Note that getting rid of this cache doesn't actually save much memory
at the moment, because parse trees are currently not freed / GC'ed.
2013-09-03 13:01:42 +02:00
Eelco Dolstra 57d18df7d0 Add some support code for nix-repl 2013-09-02 18:34:04 +02:00
Eelco Dolstra 92077b4547 Get rid of a signedness warning 2013-09-02 16:39:17 +02:00
Eelco Dolstra 33972629d7 Fix whitespace 2013-09-02 16:29:15 +02:00
Eelco Dolstra b29d3f4aee Only show trace messages when tracing is enabled 2013-09-02 12:01:04 +02:00
Eelco Dolstra efe4289464 Add an option to limit the log output of builders
This is mostly useful for Hydra to deal with builders that get stuck
in an infinite loop writing data to stdout/stderr.
2013-09-02 11:58:18 +02:00
Shea Levy afc6c1bad6 Simplify inherited attribute handling
This reduces the difference between inherited and non-inherited
attribute handling to the choice of which env to use (in recs and lets)
by setting the AttrDef::e to a new ExprVar in the parser rather than
carrying a separate AttrDef::v VarRef member.

As an added bonus, this allows inherited attributes that inherit from a
with to delay forcing evaluation of the with's attributes.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-08-26 11:31:56 +02:00
Ivan Kozik 34bb806f74 Fix typos, especially those that end up in the Nix manual 2013-08-26 11:15:22 +02:00
Gergely Risko c6c024ca6f Fix personality switching from x86_64 to i686
On Linux, Nix can build i686 packages even on x86_64 systems.  It's not
enough to recognize this situation by settings.thisSystem, we also have
to consult uname().  E.g. we can be running on a i686 Debian with an
amd64 kernel.  In that situation settings.thisSystem is i686-linux, but
we still need to change personality to i686 to make builds consistent.
2013-08-26 11:12:35 +02:00
Eelco Dolstra d308aeaf53 Store Nix integers as longs
So on 64-bit systems, integers are now 64-bit.

Fixes #158.
2013-08-19 12:35:03 +02:00
Eelco Dolstra 46222bbc43 Typo 2013-08-14 22:32:49 +02:00
Eelco Dolstra 3fb7ae0586 Respect MINSIGSTKSZ when allocating an alternative stack
http://hydra.nixos.org/build/5663577
2013-08-07 15:44:19 +00:00
Eelco Dolstra 161a2ccf7a Fix build on non-Linux
http://hydra.nixos.org/build/5662914
2013-08-07 17:26:41 +02:00
Eelco Dolstra a583a2bc59 Run the daemon worker on the same CPU as the client
On a system with multiple CPUs, running Nix operations through the
daemon is significantly slower than "direct" mode:

$ NIX_REMOTE= nix-instantiate '<nixos>' -A system
real    0m0.974s
user    0m0.875s
sys     0m0.088s

$ NIX_REMOTE=daemon nix-instantiate '<nixos>' -A system
real    0m2.118s
user    0m1.463s
sys     0m0.218s

The main reason seems to be that the client and the worker get moved
to a different CPU after every call to the worker.  This patch adds a
hack to lock them to the same CPU.  With this, the overhead of going
through the daemon is very small:

$ NIX_REMOTE=daemon nix-instantiate '<nixos>' -A system
real    0m1.074s
user    0m0.809s
sys     0m0.098s
2013-08-07 14:02:04 +02:00
Eelco Dolstra fd7d979c79 Remove obsolete reference to ATerms 2013-08-06 14:15:11 +02:00
Eelco Dolstra 46ffcce0c3 In the profiler output, show function names (if available) 2013-08-02 18:29:23 +00:00
Eelco Dolstra 8e74c0bfd1 Let the ordering operators also work on strings
E.g. ‘"foo" < "bar"’ now works.
2013-08-02 18:53:02 +02:00
Eelco Dolstra 3d77b28eac Add comparison operators ‘<’, ‘<=’, ‘>’ and ‘>=’ 2013-08-02 18:39:40 +02:00
Eelco Dolstra 47701677e8 Add integer ‘-’, ‘*’ and ‘/’ operators 2013-08-02 16:03:02 +00:00
Eelco Dolstra 5d147e125c Add a unary integer negation operator
This allows saying "-1" instead of "builtins.sub 0 1".
2013-08-02 15:43:31 +00:00
Eelco Dolstra 159e621d1a Overload the ‘+’ operator to support integer addition 2013-08-02 15:21:17 +00:00
Eelco Dolstra 0a470fc345 Make Env smaller
Commit 20866a7031 added a ‘withAttrs’
field to Env, which is annoying because it makes every Env structure
bigger and we allocate millions of them.  E.g. NixOS evaluation took
18 MiB more.  So this commit squeezes ‘withAttrs’ into values[0].
Probably should use a union...
2013-07-31 12:44:21 +02:00
Eelco Dolstra 8ae6d55db1 Don't use NULL 2013-07-31 12:11:14 +02:00
Shea Levy e068f49f7d Avoid thunks when a fromWith var can be looked up without evaluation
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-07-31 11:48:39 +02:00
Shea Levy 20866a7031 Delay evaulation of with attrs until a variable lookup needs them
Evaluation of attribute sets is strict in the attribute names, which
means immediate evaluation of `with` attribute sets rules out some
potentially interesting use cases (e.g. where the attribute names of one
set depend in some way on another but we want to bring those names into
scope for some values in the second set).

The major example of this is overridable self-referential package sets
(e.g. all-packages.nix). With immediate `with` evaluation, the only
options for such sets are to either make them non-recursive and
explicitly use the name of the overridden set in non-overridden one
every time you want to reference another package, or make the set
recursive and use the `__overrides` hack. As shown in the test case that
comes with this commit, though, delayed `with` evaluation allows a nicer
third alternative.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-07-31 11:48:39 +02:00
Eelco Dolstra 70e68e0ec6 Detect stack overflows
Previously, if the Nix evaluator gets a stack overflow due to a deep
or infinite recursion in the Nix expression, the user gets an
unhelpful message ("Segmentation fault") that doesn't indicate that
the problem is in the user's code rather than Nix itself.  Now it
prints:

  error: stack overflow (possible infinite recursion)

This only works on x86_64-linux and i686-linux.

Fixes #35.
2013-07-30 23:37:10 +02:00
Shea Levy e87d1a63bd killUser: Don't let the child kill itself on Apple
The kill(2) in Apple's libc follows POSIX semantics, which means that
kill(-1, SIGKILL) will kill the calling process too. Since nix has no
way to distinguish between the process successfully killing everything
and the process being killed by a rogue builder in that case, it can't
safely conclude that killUser was successful.

Luckily, the actual kill syscall takes a parameter that determines
whether POSIX semantics are followed, so we can call that syscall
directly and avoid the issue on Apple.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-07-30 21:25:16 +02:00
Eelco Dolstra a4921b8ceb Revert "build-remote.pl: Enforce timeouts locally"
This reverts commit 69b8f9980f.

The timeout should be enforced remotely.  Otherwise, if the garbage
collector is running either locally or remotely, if will block the
build or closure copying for some time.  If the garbage collector
takes too long, the build may time out, which is not what we want.
Also, on heavily loaded systems, copying large paths to and from the
remote machine can take a long time, also potentially resulting in a
timeout.
2013-07-18 12:52:29 +02:00
Shea Levy 16591eb3cc Allow bind-mounting regular files into the chroot
mount(2) with MS_BIND allows mounting a regular file on top of a regular
file, so there's no reason to only bind directories. This allows finer
control over just which files are and aren't included in the chroot
without having to build symlink trees or the like.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-07-15 16:01:33 +02:00
Eelco Dolstra aeb810b01e Garbage collector: Don't follow symlinks arbitrarily
Only indirect roots (symlinks to symlinks to the Nix store) are now
supported.
2013-07-12 14:03:36 +02:00
Ludovic Courtès 212e96f39c Leave `HAVE_HUP_NOTIFICATION' undefined on GNU/Hurd. 2013-07-07 13:21:06 +02:00
Eelco Dolstra 7ccd946407 Don't set $preferLocalBuild and $requiredSystemFeatures in builders
With C++ std::map, doing a comparison like ‘map["foo"] == ...’ has the
side-effect of adding a mapping from "foo" to the empty string if
"foo" doesn't exist in the map.  So we ended up setting some
environment variables by accident.
2013-06-20 18:07:27 +00:00
Eelco Dolstra 5558652709 Don't substitute derivations that have preferLocalBuild set
In particular this means that "trivial" derivations such as writeText
are not substituted, reducing the number of GET requests to the binary
cache by about 200 on a typical NixOS configuration.
2013-06-20 19:26:31 +02:00
Eelco Dolstra 1906cce6fc Increase SQLite's auto-checkpoint interval
Common operations like instantiating a NixOS system config no longer
fitted in 8192 pages, leading to more fsyncs.  So increase this limit.
2013-06-20 14:01:33 +00:00
Eelco Dolstra 9b11165aec Disable the copy-from-other-stores substituter
This substituter basically cannot work reliably since we switched to
SQLite, since SQLite databases may need write access to open them even
just for reading (and in WAL mode they always do).
2013-06-20 12:01:33 +02:00
Eelco Dolstra 22144afa8d Don't keep "disabled" substituters running
For instance, it's pointless to keep copy-from-other-stores running if
there are no other stores, or download-using-manifests if there are no
manifests.  This also speeds things up because we don't send queries
to those substituters.
2013-06-20 11:55:15 +02:00
Eelco Dolstra 1b6ee8f4c7 Allow hard links between the outputs of a derivation 2013-06-13 17:29:56 +02:00
Eelco Dolstra cd49ee0897 Fix a security bug in hash rewriting
Before calling dumpPath(), we have to make sure the files are owned by
the build user.  Otherwise, the build could contain a hard link to
(say) /etc/shadow, which would then be read by the daemon and
rewritten as a world-readable file.

This only affects systems that don't have hard link restrictions
enabled.
2013-06-13 17:12:24 +02:00
Eelco Dolstra 1e2c7c04b1 Fix assertion failure in canonicalisePathMetaData() after hash rewriting
The assertion in canonicalisePathMetaData() failed because the
ownership of the path already changed due to the hash rewriting.  The
solution is not to check the ownership of rewritten paths.

Issue #122.
2013-06-13 17:12:06 +02:00
Eelco Dolstra 6cc2a8f8ed computeFSClosure: Only process the missing/corrupt paths
Issue #122.
2013-06-13 16:43:20 +02:00
Eelco Dolstra f9ff67e948 In repair mode, update the hash of rebuilt paths
Otherwise subsequent invocations of "--repair" will keep rebuilding
the path.  This only happens if the path content differs between
builds (e.g. due to timestamps).
2013-06-13 14:46:07 +02:00
Eelco Dolstra 6b05f688ee nix-daemon: Trust options like binary-caches when the client is root
Fixes #127.
2013-06-12 12:10:26 +02:00
Eelco Dolstra ca70fba0bf Remove obsolete EOF checks 2013-06-07 15:10:23 +02:00
Eelco Dolstra 5959c591a0 Process stderr from substituters while doing have/info queries 2013-06-07 15:02:14 +02:00
Eelco Dolstra c5f9d0d080 Buffer reads from the substituter
This greatly reduces the number of system calls.
2013-06-07 14:00:23 +02:00
Eelco Dolstra b09b87321c nix-store --export: Export paths in topologically sorted order
Fixes #118.
2013-05-23 14:55:36 -04:00
Eelco Dolstra 18a48d80a0 Show function names in error messages
Functions in Nix are anonymous, but if they're assigned to a
variable/attribute, we can use the variable/attribute name in error
messages, e.g.

while evaluating `concatMapStrings' at `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/strings.nix:18:25':
...
2013-05-16 19:08:02 +02:00
Eelco Dolstra 1b3a03f161 Show which function argument was unexpected
Fixes #116.
2013-05-16 17:56:14 +02:00
Eelco Dolstra 229567293c Shut up a compiler warning 2013-05-16 17:48:19 +02:00
Eelco Dolstra 2ee9da9e22 In trace messages, don't print the output path
This doesn't work if there is no output named "out".  Hydra didn't use
it anyway.
2013-05-10 00:24:33 +02:00
Eelco Dolstra 6eba05613a Communicate build timeouts to Hydra 2013-05-09 18:39:04 +02:00
Eelco Dolstra 69b8f9980f build-remote.pl: Enforce timeouts locally
Don't pass --timeout / --max-silent-time to the remote builder.
Instead, let the local Nix process terminate the build if it exceeds a
timeout.  The remote builder will be killed as a side-effect.  This
gives better error reporting (since the timeout message from the
remote side wasn't properly propagated) and handles non-Nix problems
like SSH hangs.
2013-05-09 17:17:17 +02:00
Eelco Dolstra 470553bd05 Don't let stderr writes in substituters cause a deadlock 2013-05-01 13:21:39 +02:00
Eelco Dolstra 0374d94437 addAdditionalRoots(): Check each path only once 2013-04-26 12:07:25 +02:00
Eelco Dolstra 772b70952f Fix --timeout
I'm not sure if it has ever worked correctly.  The line "lastWait =
after;" seems to mean that the timer was reset every time a build
produced log output.

Note that the timeout is now per build, as documented ("the maximum
number of seconds that a builder can run").
2013-04-23 18:04:59 +02:00
Eelco Dolstra 934cf2d1f4 Nix daemon: respect build timeout from the client 2013-04-23 16:59:06 +02:00
Eelco Dolstra 258897c265 Complain if /homeless-shelter exists 2013-04-04 11:16:26 +02:00
Shea Levy cc63db1dd5 makeStoreWritable: Ask forgiveness, not permission
It is surprisingly impossible to check if a mountpoint is a bind mount
on Linux, and in my previous commit I forgot to check if /nix/store was
even a mountpoint at all. statvfs.f_flag is not populated with MS_BIND
(and even if it were, my check was wrong in the previous commit).

Luckily, the semantics of mount with MS_REMOUNT | MS_BIND make both
checks unnecessary: if /nix/store is not a mountpoint, then mount will
fail with EINVAL, and if /nix/store is not a bind-mount, then it will
not be made writable. Thus, if /nix/store is not a mountpoint, we fail
immediately (since we don't know how to make it writable), and if
/nix/store IS a mountpoint but not a bind-mount, we fail at first write
(see below for why we can't check and fail immediately).

Note that, due to what is IMO buggy behavior in Linux, calling mount
with MS_REMOUNT | MS_BIND on a non-bind readonly mount makes the
mountpoint appear writable in two places: In the sixth (but not the
10th!) column of mountinfo, and in the f_flags member of struct statfs.
All other syscalls behave as if the mount point were still readonly (at
least for Linux 3.9-rc1, but I don't think this has changed recently or
is expected to soon). My preferred semantics would be for MS_REMOUNT |
MS_BIND to fail on a non-bind mount, as it doesn't make sense to remount
a non bind-mount as a bind mount.
2013-03-25 19:00:16 +01:00
Shea Levy 2c9cf50746 makeStoreWritable: Use statvfs instead of /proc/self/mountinfo to find out if /nix/store is a read-only bind mount
/nix/store could be a read-only bind mount even if it is / in its own filesystem, so checking the 4th field in mountinfo is insufficient.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-03-25 19:00:16 +01:00
Eelco Dolstra 804709706c Fix building against Bison 2.6 2013-03-14 18:31:08 +01:00
Eelco Dolstra c56bc3d81c Make sure that thunks are restored properly if an exception occurs
Fixes Hydra bug #67.
2013-03-14 17:21:13 +01:00
Eelco Dolstra bdd4646338 Revert "Prevent config.h from being clobbered"
This reverts commit 28bba8c44f.
2013-03-08 01:24:59 +01:00
Eelco Dolstra 28bba8c44f Prevent config.h from being clobbered 2013-03-07 23:55:55 +01:00
Eelco Dolstra 8057a192e3 Handle systems without lutimes() or lchown() 2013-02-28 19:55:09 +01:00
Eelco Dolstra f45c731cd7 Handle symlinks properly
Now it's really brown paper bag time...
2013-02-28 14:51:08 +01:00
Eelco Dolstra 0111ba98ea Handle hard links to other files in the output 2013-02-27 17:18:41 +01:00
Eelco Dolstra b008674e46 Refactoring: Split off the non-recursive canonicalisePathMetaData()
Also, change the file mode before changing the owner.  This prevents a
slight time window in which a setuid binary would be setuid root.
2013-02-27 16:42:19 +01:00
Eelco Dolstra 5526a282b5 Security: Don't allow builders to change permissions on files they don't own
It turns out that in multi-user Nix, a builder may be able to do

  ln /etc/shadow $out/foo

Afterwards, canonicalisePathMetaData() will be applied to $out/foo,
causing /etc/shadow's mode to be set to 444 (readable by everybody but
writable by nobody).  That's obviously Very Bad.

Fortunately, this fails in NixOS's default configuration because
/nix/store is a bind mount, so "ln" will fail with "Invalid
cross-device link".  It also fails if hard-link restrictions are
enabled, so a workaround is:

  echo 1 > /proc/sys/fs/protected_hardlinks

The solution is to check that all files in $out are owned by the build
user.  This means that innocuous operations like "ln
${pkgs.foo}/some-file $out/" are now rejected, but that already failed
in chroot builds anyway.
2013-02-26 02:30:19 +01:00
Eelco Dolstra dadf7a5b46 build-remote: Use the --quiet flag
‘--option verbosity 0’ doesn't actually do anything.
2013-02-19 16:56:31 +01:00
Ludovic Courtès 3e067ac11c Add `Settings::nixDaemonSocketFile'. 2013-02-19 10:19:18 +01:00
Ludovic Courtès 5ea138dc4b Enable chroot support on old glibc versions. 2013-02-19 10:19:11 +01:00
Eelco Dolstra 5f18cd2e84 Make "${./path} ..." evaluate to a string, not a path
Wacky string coercion semantics caused expressions like

  exec = "${./my-script} params...";

to evaluate to a path (‘/path/my-script params’), because
anti-quotations are desuged to string concatenation:

  exec = ./my-script + " params...";

By constrast, adding a space at the start would yield a string as
expected:

  exec = " ${./my-script} params...";

Now the first example also evaluates to a string.
2013-02-08 20:04:04 +01:00
Eelco Dolstra 52172607cf Rename "hash" to "hashString" and handle SHA-1 2013-02-08 19:36:23 +01:00
Marc Weber 01a5ea9914 experimental/hash
adding primop function calculating hash of a string

Signed-off-by: Marc Weber <marco-oweber@gmx.de>
2013-02-08 19:26:34 +01:00
Eelco Dolstra d6143801fd Support the colonies 2013-01-30 18:50:58 +01:00
Eelco Dolstra 5e9c3da412 Only warn about SQLite being busy once
No need to get annoying.
2013-01-23 16:45:10 +01:00
Eelco Dolstra 99ed558a93 Correctly handle missing logs 2013-01-22 22:07:25 +01:00
Eelco Dolstra 536c85ea49 Store build logs in /nix/var/log/nix/drvs/<XX>
...where <XX> is the first two characters of the derivation.
Otherwise /nix/var/log/nix/drvs may become so large that we run into
all sorts of weird filesystem limits/inefficiences.  For instance,
ext3/ext4 filesystems will barf with "ext4_dx_add_entry:1551:
Directory index full!" once you hit a few million files.
2013-01-17 15:47:26 +01:00
Eelco Dolstra e42df686f3 Delete a left-over trash directory before doing a GC 2013-01-05 00:13:29 +01:00
Eelco Dolstra 92926be2fe Fix "0 store paths deleted" message 2013-01-04 15:17:19 +01:00
Eelco Dolstra b424d29d1b Open the database after removing immutable bits 2013-01-03 13:29:17 +01:00
Eelco Dolstra 3007f57377 Remove tabs 2013-01-03 13:00:46 +01:00
Eelco Dolstra def5160b61 Clear any immutable bits in the Nix store
Doing this once makes subsequent operations like garbage collecting
more efficient since we don't have to call makeMutable() first.
2013-01-03 12:59:23 +01:00
Eelco Dolstra 299141ecbd If a substitute closure is incomplete, build dependencies, then retry the substituter
Issue #77.
2013-01-02 12:38:28 +01:00
Eelco Dolstra 1b3a78a459 Automatically fall back if the references of a substitute are not substitutable
Fixes #77.
2013-01-02 12:00:26 +01:00
Eelco Dolstra 12f9129f60 nix-build: Support talking to old daemons
Fixes #76.
2012-12-29 23:21:46 +01:00
Eelco Dolstra b7629778ef Allow mounting a path in a different location in the chroot
Fixes #24.
2012-12-29 23:04:02 +01:00
Eelco Dolstra 2754a07ead nix-store -q --roots: Respect the gc-keep-outputs/gc-keep-derivations settings
So if a path is not garbage solely because it's reachable from a root
due to the gc-keep-outputs or gc-keep-derivations settings, ‘nix-store
-q --roots’ now shows that root.
2012-12-20 18:41:44 +01:00
Eelco Dolstra 06f62defe6 Yet another rewrite of the garbage collector
But this time it's *obviously* correct!  No more segfaults due to
infinite recursions for sure, etc.

Also, move directories to /nix/store/trash instead of renaming them to
/nix/store/bla-gc-<pid>.  Then we can just delete /nix/store/trash at
the end.
2012-12-20 17:32:15 +01:00
Eelco Dolstra e775d4d84f If gc-keep-derivations is set, only keep the actual deriver
This prevents zillions of derivations from being kept, and fixes an
infinite recursion in the garbage collector (due to an obscure cycle
that can occur with fixed-output derivations).
2012-12-19 15:33:09 +01:00
Eelco Dolstra fbf0b2fa45 Kill the build hook rather than shutting it down cleanly
Waiting for the hook to shut down cleanly sometimes seems to lead to
hangs.
2012-12-19 11:47:42 +01:00
Eelco Dolstra 228ea7c2f9 Revert brain fart
This reverts commit cc511fd65b.
2012-12-18 20:43:47 +01:00
Eelco Dolstra cc511fd65b Check for potential infinite select() loops when building 2012-12-18 18:50:05 +01:00
Stuart Pernsteiner a9045c727f fix use-after-free bug in mkString(Value&, Symbol&) 2012-12-13 11:13:20 +01:00
Eelco Dolstra 9fa12fc201 Allow setting the profile location using $NIX_PROFILE
Fixes #69.
2012-12-12 16:01:46 +01:00
Eelco Dolstra 772778c0ec On SQLITE_BUSY, wait a random amount of time
If all contending processes wait a fixed amount of time (100 ms),
there is a good probability that they'll just collide again.
2012-12-11 11:49:42 +01:00
Eelco Dolstra 3100b29fc5 Tiny optimisation in the filter primop 2012-12-04 17:22:20 +01:00
Eelco Dolstra 5ad89398d1 nix-env: Install all outputs of a derivation
If you explicitly install a package, presumably you want all of it.
So symlink all outputs in the user environment.
2012-12-04 14:20:36 +01:00
Eelco Dolstra d62fc71b85 Fix the ‘--prebuilt-only’ flag 2012-12-03 21:01:41 +01:00
Eelco Dolstra 4bb4d5479a Whitespace 2012-12-03 18:19:49 +01:00
Eelco Dolstra 8eed07cda4 nix-env -q --out-path: Support multiple outputs
We now print all output paths of a package, e.g.

  openssl-1.0.0i  bin=/nix/store/gq2mvh0wb9l90djvsagln3aqywqmr6vl-openssl-1.0.0i-bin;man=/nix/store/7zwf5r5hsdarl3n86dasvb4chm2xzw9n-openssl-1.0.0i-man;/nix/store/cj7xvk7fjp9q887359j75pw3pzjfmqf1-openssl-1.0.0i

or (in XML mode)

  <item attrPath="openssl" name="openssl-1.0.0i" system="x86_64-linux">
    <output name="bin" path="/nix/store/gq2mvh0wb9l90djvsagln3aqywqmr6vl-openssl-1.0.0i-bin" />
    <output name="man" path="/nix/store/7zwf5r5hsdarl3n86dasvb4chm2xzw9n-openssl-1.0.0i-man" />
    <output name="out" path="/nix/store/cj7xvk7fjp9q887359j75pw3pzjfmqf1-openssl-1.0.0i" />
  </item>
2012-11-28 13:49:44 +01:00
Eelco Dolstra 6c98e6a5de Optionally ignore null-valued derivation attributes
This allows adding attributes like

  attr = if stdenv.system == "bla" then something else null;

without changing the resulting derivation on non-<bla> platforms.

We once considered adding a special "ignore" value for this purpose,
but using null seems more elegant.
2012-11-27 15:01:32 +01:00
Eelco Dolstra 8b8ee53bc7 Add builtin constants ‘langVersion’ and ‘nixVersion’
The integer constant ‘langVersion’ denotes the current language
version.  It gets increased every time a language feature is
added/changed/removed.  It's currently 1.

The string constant ‘nixVersion’ contains the current Nix version,
e.g. "1.2pre2980_9de6bc5".
2012-11-27 13:29:55 +01:00
Eelco Dolstra 5943f41b8b queryMissing(): Handle partially valid derivations 2012-11-26 18:25:12 +01:00
Eelco Dolstra 408a7bfac1 nix-instantiate: Fix read-only evaluation 2012-11-26 17:41:10 +01:00
Eelco Dolstra 8d8d47abd2 Only substitute wanted outputs of a derivation
If a derivation has multiple outputs, then we only want to download
those outputs that are actuallty needed.  So if we do "nix-build -A
openssl.man", then only the "man" output should be downloaded.
Likewise if another package depends on ${openssl.man}.

The tricky part is that different derivations can depend on different
outputs of a given derivation, so we may need to restart the
corresponding derivation goal if that happens.
2012-11-26 17:15:09 +01:00
Eelco Dolstra 46a369ad95 Make "nix-build -A <derivation>.<output>" do the right thing
For example, given a derivation with outputs "out", "man" and "bin":

  $ nix-build -A pkg

produces ./result pointing to the "out" output;

  $ nix-build -A pkg.man

produces ./result-man pointing to the "man" output;

  $ nix-build -A pkg.all

produces ./result, ./result-man and ./result-bin;

  $ nix-build -A pkg.all -A pkg2

produces ./result, ./result-man, ./result-bin and ./result-2.
2012-11-26 15:39:10 +01:00
Eelco Dolstra 9de6bc5d05 nix-store -r: Add ‘--ignore-unknown’ flag
This flag causes paths that do not have a known substitute to be
quietly ignored.  This is mostly useful for Charon, allowing it to
speed up deployment by letting a machine use substitutes for all
substitutable paths, instead of uploading them.  The latter is
frequently faster, e.g. if the target machine has a fast Internet
connection while the source machine is on a slow ADSL line.
2012-11-20 00:27:25 +01:00
Eelco Dolstra bf3725da2a nix-store -r: Don't quietly ignore missing paths 2012-11-19 23:51:56 +01:00
Eelco Dolstra 17dc306aa3 Revert "prim_toPath: Actually make the string a path"
This reverts commit 2980d1fba9.  It
causes a regression in NixOS evaluation:

string `/nix/store/ya3s5gmj3b28170fpbjhgsk8wzymkpa1-pommed-1.39/etc/pommed.conf' cannot refer to other paths
2012-11-15 23:04:34 +01:00
Eelco Dolstra f794465ca8 Disable use of vfork()
vfork() is just too weird.  For instance, in this build:

  http://hydra.nixos.org/build/3330487

the value fromHook.writeSide becomes corrupted in the parent, even
though the child only reads from it.  At -O0 the problem goes away.
Probably the child is overriding some spilled temporary variable.

If I get bored I may implement using posix_spawn() instead.
2012-11-15 19:35:18 +01:00
Eelco Dolstra 8541d27fce Don't use std::cerr in a few places
Slightly scared of using std::cerr in a vforked process...
2012-11-15 15:01:02 +01:00
Eelco Dolstra 3acc8adcad Add some debug code 2012-11-15 13:55:02 +01:00
Eelco Dolstra ea89df2b76 Use vfork() instead of fork() if available
Hopefully this reduces the chance of hitting ‘unable to fork: Cannot
allocate memory’ errors.  vfork() is used for everything except
starting builders.
2012-11-09 18:00:33 +01:00
Eelco Dolstra 48c19c4633 Remove definition of non-existant function 2012-11-09 17:59:41 +01:00
Eelco Dolstra 198dbe7fa1 Remove some redundant close() calls
They are unnecessary because we set the close-on-exec flag.
2012-11-09 16:58:51 +01:00
Eelco Dolstra 10dcee99ed Remove the quickExit function 2012-11-09 16:42:10 +01:00
Eelco Dolstra 4c9e3fa641 Remove a Darwin hack that should no longer be needed 2012-11-09 16:35:42 +01:00
Eelco Dolstra 88164325fa Fix a segfault when auto-calling a "a@{...}" function
Since the called function can return its argument attribute set
(e.g. "a"), the latter should not be allocated on the stack.

Reported by Shea.
2012-11-09 15:09:31 +01:00
Eelco Dolstra f581ce0b0c Don't start copy-from-other-stores if $NIX_OTHER_STORES is unset
Slight optimisation.
2012-11-09 14:58:10 +01:00
Eelco Dolstra 91ef4d9a81 Remove unnecessary call to closeMostFDs()
We have close-on-exec on all FDs now, and there is no security risk in
passing open FDs to substituters anyway.
2012-11-09 14:43:47 +01:00
Shea Levy d0fc615af6 canonicalizePathMetaData: Fall-back to utimes if lutimes fails due to ENOSYS 2012-11-06 11:29:59 +01:00
Eelco Dolstra 4c34d384e6 If hashes do not match, print them in base-32 for SHA-1/SHA-256
Fixes #57.
2012-10-23 18:05:50 +02:00
Shea Levy 2980d1fba9 prim_toPath: Actually make the string a path 2012-10-23 16:39:35 +02:00
Eelco Dolstra e345182055 Shorten the names of temporary build directories 2012-10-11 14:03:06 -04:00
Eelco Dolstra 70f75be199 getDerivation(): Don't always quietly ignore assertion failure
Ignoring assertion failures makes some sense for nix-env -qa, but not
for nix-instantiate/nix-build or hydra-eval-jobs.
2012-10-04 15:22:25 -04:00
Eelco Dolstra ad328bea15 XML writer: flush after newlines
This is useful for hydra-eval-jobs.
2012-10-04 14:07:36 -04:00
Eelco Dolstra 904f50412c nix-store --verify: Continue on errors 2012-10-04 10:20:23 -04:00
Eelco Dolstra 90b8a34f82 Fix regular expression
http://hydra.nixos.org/build/3123177
2012-10-04 09:46:10 -04:00
Eelco Dolstra e35d6f78dc Rename nix-worker to nix-daemon 2012-10-03 17:59:23 -04:00
Eelco Dolstra 522ecab9b8 Drop support for running nix-worker in "slave" mode
AFAIK nobody uses this, setuid binaries are evil, and there is no good
reason why people can't just run the daemon.
2012-10-03 17:30:45 -04:00
Eelco Dolstra 7586095504 Remove bin2c 2012-10-03 16:59:28 -04:00
Eelco Dolstra a562d544d8 When ‘--help’ is given, just run ‘man’ to show the manual page
I.e. do what git does.  I'm too lazy to keep the builtin help text up
to date :-)

Also add ‘--help’ to various commands that lacked it
(e.g. nix-collect-garbage).
2012-10-03 16:40:09 -04:00
Eelco Dolstra 2bbc4a214e nix-env: Support ‘--repair’ flag 2012-10-03 15:35:42 -04:00
Eelco Dolstra 0a7084567f Add a ‘--repair’ flag to nix-instantiate
This allows repairing corrupted derivations and other source files.
2012-10-03 15:09:18 -04:00
Eelco Dolstra a807edfae8 Handle repairing paths that are in build-chroot-dirs 2012-10-03 11:20:16 -04:00
Eelco Dolstra a3f205b249 When repairing a derivation, check and repair the entire output closure
If we find a corrupted path in the output closure, we rebuild the
derivation that produced that particular path.
2012-10-03 10:38:09 -04:00
Eelco Dolstra 2001895f3d Add a --repair flag to ‘nix-store -r’ to repair derivation outputs
With this flag, if any valid derivation output is missing or corrupt,
it will be recreated by using a substitute if available, or by
rebuilding the derivation.  The latter may use hash rewriting if
chroots are not available.
2012-10-02 17:13:46 -04:00
Eelco Dolstra cf46f19444 nix-store -r: Get rid of an unnecessary call to buildPaths/ensurePaths 2012-10-02 16:00:09 -04:00
Eelco Dolstra 8e3a7bd712 nix-store --verify: Add an option ‘--repair’ to repair all missing/corrupt paths
Also, return a non-zero exit code if errors remain after
verifying/repairing.
2012-10-02 15:12:56 -04:00
Eelco Dolstra 9958bd6992 Add operation ‘nix-store --repair-path’
This operation allows fixing corrupted or accidentally deleted store
paths by redownloading them using substituters, if available.

Since the corrupted path cannot be replaced atomically, there is a
very small time window (one system call) during which neither the old
(corrupted) nor the new (repaired) contents are available.  So
repairing should be used with some care on critical packages like
Glibc.
2012-10-02 14:08:59 -04:00
Eelco Dolstra e666e1156f Handle octal escapes in /proc/self/mountinfo 2012-09-28 21:39:30 -04:00
Eelco Dolstra f406288cc7 Print a more descriptive error message if setting up the build environment fails 2012-09-28 21:37:02 -04:00
Eelco Dolstra 95c74eae26 Allow dashes in identifiers
In Nixpkgs, the attribute in all-packages.nix corresponding to a
package is usually equal to the package name.  However, this doesn't
work if the package contains a dash, which is fairly common.  The
convention is to replace the dash with an underscore (e.g. "dbus-lib"
becomes "dbus_glib"), but that's annoying.  So now dashes are valid in
variable / attribute names, allowing you to write:

  dbus-glib = callPackage ../development/libraries/dbus-glib { };

and

  buildInputs = [ dbus-glib ];

Since we don't have a negation or subtraction operation in Nix, this
is unambiguous.
2012-09-27 15:49:20 -04:00
Eelco Dolstra d534f137f0 Make the store writable before creating /nix/store/.links 2012-09-25 16:30:08 -04:00
Eelco Dolstra e464b0247d Merge branch 'readonly-store' 2012-09-25 15:38:00 -04:00
Eelco Dolstra 28bf183d2d Include <sys/types.h> for off_t
Reported by "gio" on IRC.
2012-09-25 13:00:19 -04:00
Eelco Dolstra b9c2b4d5b4 Remove setting of the immutable bit
Using the immutable bit is problematic, especially in conjunction with
store optimisation.  For instance, if the garbage collector deletes a
file, it has to clear its immutable bit, but if the file has
additional hard links, we can't set the bit afterwards because we
don't know the remaining paths.

So now that we support having the entire Nix store as a read-only
mount, we may as well drop the immutable bit.  Unfortunately, we have
to keep the code to clear the immutable bit for backwards
compatibility.
2012-09-19 16:17:54 -04:00
Eelco Dolstra b9124a5c33 Support having /nix/store as a read-only bind mount
It turns out that the immutable bit doesn't work all that well.  A
better way is to make the entire Nix store a read-only bind mount,
i.e. by doing

  $ mount --bind /nix/store /nix/store
  $ mount -o remount,ro,bind /nix/store

(This would typically done in an early boot script, before anything
from /nix/store is used.)

Since Nix needs to be able to write to the Nix store, it now detects
if /nix/store is a read-only bind mount and then makes it writable in
a private mount namespace.
2012-09-19 15:45:29 -04:00
Eelco Dolstra 76e88871b2 Templatise tokenizeString() 2012-09-19 15:43:23 -04:00
Eelco Dolstra 00092b2d35 Keep build directory if not all expected outputs were produced
Fixes issue #123 in Nixpkgs.
2012-09-18 10:11:42 -04:00
Eelco Dolstra 9fd9dedf12 nix-env --delete-generations: Support --dry-run flag
Fixes #43.
2012-09-13 18:05:04 -04:00
Eelco Dolstra 8fbe96cb31 RemoteStore::connectToDaemon(): Set close-on-exec flag
This ensures that "nix-build --run-env" doesn't keep a connection to
the worker open, preventing it from exiting.
2012-09-13 14:46:37 -04:00
Eelco Dolstra e6e495649c Vacuum the SQLite DB after running the garbage collector 2012-09-13 14:33:41 -04:00
Eelco Dolstra 2923b55f9d Delete paths in a component in topologically sorted order
The outputs of a derivation can refer to each other (even though they
cannot have cycles), so they have to be deleted in the right order.

http://hydra.nixos.org/build/3026118
2012-09-13 13:08:27 -04:00
Eelco Dolstra 6c4ac29917 Disable auto store optimisation for now
I've seen operations like "nix-store --import" take much longer on one
system.  So default to off until I've investigated this a bit further.
2012-09-13 10:28:20 -04:00
Eelco Dolstra e56f71edaf In startBuilder(), only print the new paths we're building 2012-09-13 10:06:24 -04:00
Eelco Dolstra 4fca02077c Handle gc-keep-outputs and gc-keep-derivations both enabled
If the options gc-keep-outputs and gc-keep-derivations are both
enabled, you can get a cycle in the liveness graph.  There was a hack
to handle this, but it didn't work with multiple-output derivations,
causing the garbage collector to fail with errors like ‘error: cannot
delete path `...' because it is in use by `...'’.  The garbage
collector now handles strongly connected components in the liveness
graph as a unit and decides whether to delete all or none of the paths
in an SCC.
2012-09-12 18:49:35 -04:00
Eelco Dolstra 479e9172b3 Build hook: Pass the timeout to the remote builder
Note that this will only work if the client has a very recent Nix
version (post 15e1b2c223), otherwise the
--option flag will just be ignored.

Fixes #50.
2012-09-12 12:18:14 -04:00
Eelco Dolstra 5e2ffd0b8a Fix "non-zero padding" error
Probably it's not a good idea to pass a temporary object to
StringSource.
2012-09-11 19:09:01 -04:00
Eelco Dolstra 8cf672091e Support building a derivation if some outputs are already valid (non-chroot case)
This uses scary hash rewriting.

Fixes #21.
2012-09-11 18:39:22 -04:00
Eelco Dolstra 9dbda2b3fe Remove debug line 2012-09-11 17:05:35 -04:00
Eelco Dolstra a2785b7391 Support building a derivation if some outputs are already valid
This handles the chroot and build hook cases, which are easy.
Supporting the non-chroot-build case will require more work (hash
rewriting!).

Issue #21.
2012-09-11 16:59:59 -04:00
Eelco Dolstra 295027f533 Include config.h before any other header
"config.h" must be included first, because otherwise the compiler
might not see the right value of _FILE_OFFSET_BITS.  We've had this
before; see 705868a8a9.  In this case,
GCC would compute a different address for ‘settings.useSubstitutes’ in
misc.cc because of the off_t in ‘settings’.

Reverts 3854fc9b42.

http://hydra.nixos.org/build/3016700
2012-09-11 14:45:42 -04:00
Eelco Dolstra f7b650d234 Make "nix-instantiate -" interruptible 2012-08-28 11:47:26 -04:00
Eelco Dolstra cfd968dd94 Fix stupid type error in calling std::max 2012-08-27 14:17:13 -04:00