Commit graph

2125 commits

Author SHA1 Message Date
Eelco Dolstra 985f1595fe Be more strict about file names in NARs 2014-07-16 16:30:50 +02:00
Eelco Dolstra 276a40b31f Handle case collisions on case-insensitive systems
When running NixOps under Mac OS X, we need to be able to import store
paths built on Linux into the local Nix store. However, HFS+ is
usually case-insensitive, so if there are directories with file names
that differ only in case, then importing will fail.

The solution is to add a suffix ("~nix~case~hack~<integer>") to
colliding files. For instance, if we have a directory containing
xt_CONNMARK.h and xt_connmark.h, then the latter will be renamed to
"xt_connmark.h~nix~case~hack~1". If a store path is dumped as a NAR,
the suffixes are removed. Thus, importing and exporting via a
case-insensitive Nix store is round-tripping. So when NixOps calls
nix-copy-closure to copy the path to a Linux machine, you get the
original file names back.

Closes #119.
2014-07-16 16:02:05 +02:00
Eelco Dolstra fa13d3f4f3 build-remote.pl: Fix building multiple output derivations
We were importing paths without sorting them topologically, leading to
"path is not valid" errors.

See e.g. http://hydra.nixos.org/build/12451761
2014-07-14 12:19:27 +02:00
Eelco Dolstra a5c6347ff0 build-remote.pl: Use ‘nix-store --serve’ on the remote side
This makes things more efficient (we don't need to use an SSH master
connection, and we only start a single remote process) and gets rid of
locking issues (the remote nix-store process will keep inputs and
outputs locked as long as they're needed).

It also makes it more or less secure to connect directly to the root
account on the build machine, using a forced command
(e.g. ‘command="nix-store --serve --write"’). This bypasses the Nix
daemon and is therefore more efficient.

Also, don't call nix-store to import the output paths.
2014-07-11 16:22:24 +02:00
Eelco Dolstra e196eecbe6 Allow $NIX_BUILD_HOOK to be relative to Nix libexec directory 2014-07-11 13:55:06 +02:00
Eelco Dolstra d0eb970fb4 Fix broken Pid constructor 2014-07-10 21:48:21 +02:00
Eelco Dolstra edbfe2232e Replace message "importing path <...>" with "exporting path <...>"
This causes nix-copy-closure to show what it's doing before rather
than after.
2014-07-10 21:30:22 +02:00
Eelco Dolstra 7bb632b024 nix-copy-closure -s: Do substitutions via ‘nix-store --serve’
This means we no longer need an SSH master connection, since we only
execute a single command on the remote host.
2014-07-10 20:43:04 +02:00
Eelco Dolstra 43b64f5038 Remove tabs 2014-07-10 17:32:21 +02:00
Eelco Dolstra 8e9140cfde Refactoring: Move all fork handling into a higher-order function
C++11 lambdas ftw.
2014-07-10 16:58:09 +02:00
Eelco Dolstra 1114c7bd57 nix-copy-closure: Restore compression and the progress viewer 2014-07-10 14:15:12 +02:00
Eelco Dolstra 7911e4c27a Remove maybeVfork 2014-07-10 13:35:44 +02:00
Eelco Dolstra 04170d06bf nix-copy-closure: Fix race condition
There is a long-standing race condition when copying a closure to a
remote machine, particularly affecting build-remote.pl: the client
first asks the remote machine which paths it already has, then copies
over the missing paths. If the garbage collector kicks in on the
remote machine between the first and second step, the already-present
paths may be deleted. The missing paths may then refer to deleted
paths, causing nix-copy-closure to fail. The client now performs both
steps using a single remote Nix call (using ‘nix-store --serve’),
locking all paths in the closure while querying.

I changed the --serve protocol a bit (getting rid of QueryCommand), so
this breaks the SSH substituter from older versions. But it was marked
experimental anyway.

Fixes #141.
2014-07-10 11:58:59 +02:00
Eelco Dolstra 2c3a8f787b Fix security hole in ‘nix-store --serve’
Since it didn't check that the path received from the client is a
store path, the client could dump any path in the file system.
2014-07-10 11:46:01 +02:00
Eelco Dolstra 0e5d0c1543 Fix compilation error on some versions of GCC
src/libexpr/primops.cc:42:8: error: looser throw specifier for 'virtual nix::InvalidPathError::~InvalidPathError()'
src/libexpr/nixexpr.hh:12:1: error:   overriding 'virtual nix::EvalError::~EvalError() noexcept (true)'

http://hydra.nixos.org/build/12385750
2014-07-09 12:14:40 +02:00
Eelco Dolstra beaf3e90af Add builtin function ‘fromJSON’
Fixes #294.
2014-07-04 13:34:15 +02:00
Eelco Dolstra e477f0e938 Style fix 2014-06-27 11:36:23 +02:00
Paul Colomiets 858b8f9760 Add --json argument to nix-instantiate 2014-06-27 11:23:40 +02:00
Eelco Dolstra 8504e7d604 allow-arbitrary-code-during-evaluation -> allow-unsafe-native-code-during-evaluation 2014-06-27 11:20:16 +02:00
Shea Levy d62f46e500 Only add the importNative primop if the allow-arbitrary-code-during-evaluation option is true (default false) 2014-06-24 10:50:03 -04:00
Shea Levy 5cd022d6c0 Add importNative primop
This can be used to import a dynamic shared object and return an
arbitrary value, including new primops. This can be used both to test
new primops without having to recompile nix every time, and to build
specialized primops that probably don't belong upstream (e.g. a function
that calls out to gpg to decrypt a nixops secret as-needed).

The imported function should initialize the Value & as needed. A single
import can define multiple values by creating an attrset or list, of
course.

An example initialization function might look like:

extern "C" void initialize(nix::EvalState & state, nix::Value & v)
{
    v.type = nix::tPrimOp;
    v.primOp = NEW nix::PrimOp(myFun, 1, state.symbols.create("myFun"));
}

Then `builtins.importNative ./example.so "initialize"` will evaluate to
the primop defined in the myFun function.
2014-06-17 12:08:01 -04:00
Eelco Dolstra 9d0709e8c4 Don't use member initialisers
They're a little bit too recent (only supported since GCC 4.7).

http://hydra.nixos.org/build/11851475
2014-06-12 17:30:37 +02:00
Eelco Dolstra 48495f67ed Fix bogus warnings about dumping large paths
Also, yay for C++11 non-static initialisers.
2014-06-12 13:15:35 +02:00
Eelco Dolstra 0960d674d4 Drop ImportError and FindError
We're not catching these anywhere.
2014-06-12 13:00:54 +02:00
Shea Levy 718f20da6d findFile: Realise the context of the path attributes 2014-06-12 12:57:14 +02:00
Shea Levy a8fb575c98 Share code between scopedImport and import
In addition to reducing duplication, this fixes both import from
derivation and import of derivation for scopedImport
2014-06-12 12:52:39 +02:00
Eelco Dolstra ee7fe64c0a == operator: Ignore string context
There really is no case I can think of where taking the context into
account is useful. Mostly it's just very inconvenient.
2014-06-10 14:02:56 +02:00
Eelco Dolstra b1beed97a0 Report daemon OOM better
When copying a large path causes the daemon to run out of memory, you
now get:

  error: Nix daemon out of memory

instead of:

  error: writing to file: Broken pipe
2014-06-10 13:45:50 +02:00
Eelco Dolstra 829af22759 Print a warning when loading a large path into memory
I.e. if you have a derivation with

  src = ./huge-directory;

you'll get a warning that this is not a good idea.
2014-06-10 13:30:09 +02:00
Eelco Dolstra 3c6b8a5215 nix-env -qa --json: Generate valid JSON even if there are invalid meta attrs 2014-06-02 17:58:43 +02:00
Eelco Dolstra becc2b0167 Sort nixPath attributes 2014-05-29 19:02:14 +02:00
Eelco Dolstra 54a34119f3 Use std::unordered_set 2014-05-26 17:53:17 +02:00
Eelco Dolstra d8c061e044 Remove ExprBuiltin
It's slower than ExprVar since it doesn't compute a static
displacement. Since we're not using the throw primop in the
implementation of <...> anymore, it's also not really needed.
2014-05-26 17:14:28 +02:00
Eelco Dolstra 62a6eeb1f3 Make the Nix search path declarative
Nix search path lookups like <nixpkgs> are now desugared to ‘findFile
nixPath <nixpkgs>’, where ‘findFile’ is a new primop. Thus you can
override the search path simply by saying

  let
    nixPath = [ { prefix = "nixpkgs"; path = "/my-nixpkgs"; } ];
  in ... <nixpkgs> ...

In conjunction with ‘scopedImport’ (commit
c273c15cb1), the Nix search path can be
propagated across imports, e.g.

  let

    overrides = {
      nixPath = [ ... ] ++ builtins.nixPath;
      import = fn: scopedImport overrides fn;
      scopedImport = attrs: fn: scopedImport (overrides // attrs) fn;
      builtins = builtins // overrides;
    };

  in scopedImport overrides ./nixos
2014-05-26 17:02:22 +02:00
Eelco Dolstra 39d72640c2 Ensure that -I flags get included in nixPath
Also fixes #261.
2014-05-26 16:52:31 +02:00
Eelco Dolstra a8edf185a9 Add constant ‘nixPath’
It contains the Nix expression search path as a list of { prefix, path
} sets, e.g.

  [ { path = "/nix/var/nix/profiles/per-user/root/channels/nixos"; prefix = ""; }
    { path = "/etc/nixos/configuration.nix"; prefix = "nixos-config"; }
    { path = "/home/eelco/Dev/nix/inst/share/nix/corepkgs"; prefix = "nix"; }
  ]
2014-05-26 14:55:47 +02:00
Eelco Dolstra c273c15cb1 Add primop ‘scopedImport’
‘scopedImport’ works like ‘import’, except that it takes a set of
attributes to be added to the lexical scope of the expression,
essentially extending or overriding the builtin variables.  For
instance, the expression

  scopedImport { x = 1; } ./foo.nix

where foo.nix contains ‘x’, will evaluate to 1.

This has a few applications:

* It allows getting rid of function argument specifications in package
  expressions. For instance, a package expression like:

    { stdenv, fetchurl, libfoo }:

    stdenv.mkDerivation { ... buildInputs = [ libfoo ]; }

  can now we written as just

    stdenv.mkDerivation { ... buildInputs = [ libfoo ]; }

  and imported in all-packages.nix as:

    bar = scopedImport pkgs ./bar.nix;

  So whereas we once had dependencies listed in three places
  (buildInputs, the function, and the call site), they now only need
  to appear in one place.

* It allows overriding builtin functions. For instance, to trace all
  calls to ‘map’:

  let
    overrides = {
      map = f: xs: builtins.trace "map called!" (map f xs);

      # Ensure that our override gets propagated by calls to
      # import/scopedImport.
      import = fn: scopedImport overrides fn;

      scopedImport = attrs: fn: scopedImport (overrides // attrs) fn;

      # Also update ‘builtins’.
      builtins = builtins // overrides;
    };
  in scopedImport overrides ./bla.nix

* Similarly, it allows extending the set of builtin functions. For
  instance, during Nixpkgs/NixOS evaluation, the Nixpkgs library
  functions could be added to the default scope.

There is a downside: calls to scopedImport are not memoized, unlike
import. So importing a file multiple times leads to multiple parsings
/ evaluations. It would be possible to construct the AST only once,
but that would require careful handling of variables/environments.
2014-05-26 14:26:29 +02:00
Eelco Dolstra f0fdbd0897 Shut up some signedness warnings 2014-05-26 12:34:15 +02:00
Eelco Dolstra 0321ef9bb2 Ugly hack to allow --argstr values starting with a dash
Fixes #265.
2014-05-23 14:43:58 +02:00
Eelco Dolstra 9f9080e2c0 nix-store -l: Fetch build logs from the Internet
If a build log is not available locally, then ‘nix-store -l’ will now
try to download it from the servers listed in the ‘log-servers’ option
in nix.conf. For instance, if you have:

  log-servers = http://hydra.nixos.org/log

then it will try to get logs from http://hydra.nixos.org/log/<base
name of the store path>. So you can do things like:

  $ nix-store -l $(which xterm)

and get a log even if xterm wasn't built locally.
2014-05-21 17:19:36 +02:00
Shea Levy eac5841970 Provide a more useful error message when a dynamic attr lookup fails 2014-05-15 17:56:24 +02:00
Eelco Dolstra 8d5f472f2c lvlInfo -> lvlTalkative 2014-05-15 11:37:44 +02:00
Eelco Dolstra 84813af5b9 nix-store --optimise: Remove bogus statistics 2014-05-15 11:33:46 +02:00
Eelco Dolstra 690adeb03d Remove tab 2014-05-15 11:19:16 +02:00
Eelco Dolstra a1b66f316e Merge branch 'master' of github.com:wmertens/nix 2014-05-15 11:18:29 +02:00
Wout Mertens 3b9ea8452f Shortcut store files before lstat
readdir() already returns the inode numbers, so we don't need to call
lstat to know if a file was already linked or not.
2014-05-15 09:02:22 +02:00
Wout Mertens d73ffc552f Use the inodes given by readdir directly 2014-05-14 22:52:10 +02:00
Wout Mertens e974f20c98 Preload linked hashes to speed up lookups
By preloading all inodes in the /nix/store/.links directory, we can
quickly determine of a hardlinked file was already linked to the hashed
links.
This is tolerant of removing the .links directory, it will simply
recalculate all hashes in the store.
2014-05-13 23:10:06 +02:00
Eelco Dolstra 95501c4dee nix-instantiate --eval: Apply auto-arguments if the result is a function
Fixes #254.
2014-05-13 12:56:48 +02:00
wmertens a84f503d86 Shortcut already-hardlinked files
If an inode in the Nix store has more than 1 link, it probably means that it was linked into .links/ by us. If so, skip.

There's a possibility that something else hardlinked the file, so it would be nice to be able to override this.

Also, by looking at the number of hardlinks for each of the files in .links/, you can get deduplication numbers and space savings.
2014-05-10 15:53:01 +02:00
Eelco Dolstra 696f960dee Set up directories and permissions for multi-user install automatically
This automatically creates /nix/var/nix/profiles/per-user and sets the
permissions/ownership on /nix/store to 1775 and root:nixbld.
2014-05-02 14:31:15 +02:00
Eelco Dolstra de4cdd0d47 Set build-max-jobs to the number of available cores by default
More zero configuration.
2014-05-02 12:51:43 +02:00
Eelco Dolstra ada3e3fa15 When running as root, use build users by default
This removes the need to have a nix.conf, and prevents people from
accidentally running Nix builds as root.
2014-05-02 12:46:03 +02:00
Ricardo M. Correia 700c678c2e nix-env: Minor change to '--delete-generations Nd' semantics
The option '--delete-generations Nd' deletes all generations older than N
days. However, most likely the user does not want to delete the
generation that was active N days ago.

For example, say that you have these 3 generations:

1: <30 days ago>
2: <15 days ago>
3: <1 hour ago>

If you do --delete-generations 7d (say, as part of a cron job), most
likely you still want to keep generation 2, i.e. the generation that was
active 7 days ago (and for most of the past 7 days, in fact).

This patch fixes this issue. Note that this also affects
'nix-collect-garbage --delete-older-than Nd'.

Thanks to @roconnor for noticing the issue!
2014-04-15 15:34:58 +02:00
Eelco Dolstra dfa2f77d2e If a .drv cannot be parsed, show its path
Otherwise you just get ‘expected string `Derive(['’ which isn't very helpful.
2014-04-08 19:24:29 +02:00
Eelco Dolstra 8e5fbf4d73 Show position info in attribute selection errors 2014-04-04 22:52:14 +02:00
Eelco Dolstra 4c5faad994 Show position info in Boolean operations 2014-04-04 22:43:52 +02:00
Eelco Dolstra bd9b1d97b4 Show position info in string concatenation / addition errors 2014-04-04 22:19:33 +02:00
Eelco Dolstra a5fe730940 forceString: Show position info 2014-04-04 21:14:11 +02:00
Eelco Dolstra 27b44b8cf7 forceAttrs: Show position info 2014-04-04 19:11:40 +02:00
Eelco Dolstra 96b695ccab forceList: Show position info 2014-04-04 19:05:36 +02:00
Eelco Dolstra b62d36963c forceInt: Show position info 2014-04-04 18:59:29 +02:00
Eelco Dolstra c28de6d96e Pass position information to primop calls
For example:

  error: `tail' called on an empty list, at
    /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/ex-2/default.nix:13:7
2014-04-04 18:59:29 +02:00
Eelco Dolstra 8b31ffd10d Remove unnecessary quotes around file names 2014-04-04 18:59:29 +02:00
Eelco Dolstra b72c8d2e5b Include position info in function application
This allows error messages like:

  error: the anonymous function at `/etc/nixos/configuration.nix:1:1'
    called without required argument `foo', at
    `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:77:59'
2014-04-04 18:59:29 +02:00
Danny Wilson ae6b631dc4 Fix compile errors on Illumos 2014-04-03 17:39:57 +02:00
Eelco Dolstra f0de86357c Tweak error message 2014-04-03 15:24:02 +02:00
Ludovic Courtès e7720aa10a Make sure /dev/pts/ptmx is world-writable
While running Python 3’s test suite, we noticed that on some systems
/dev/pts/ptmx is created with permissions 0 (that’s the case with my
Nixpkgs-originating 3.0.43 kernel, but someone with a Debian-originating
3.10-3 reported not having this problem.)

There’s still the problem that people without
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y are screwed (as noted in build.cc),
but I don’t see how we could work around it.
2014-04-03 13:42:45 +02:00
Eelco Dolstra ac6ceea764 Fix potential segfault
The newEnv variable was accessed (via the dynamicEnv) pointer after it
had gone out of scope.

Fixes #234.
2014-04-01 17:04:38 +02:00
Ricardo M. Correia 7ef7597f71 nix-env: Add support for --delete-generations 15d
It will delete all generations older than the specified number of days.
2014-03-30 00:54:16 +01:00
Eelco Dolstra 1c2550a2ae boost::shared_ptr -> std::shared_ptr 2014-03-30 00:49:23 +01:00
Eelco Dolstra 9becaa041f Drop pointless #include 2014-03-29 22:20:33 +01:00
Eelco Dolstra acb8facbbc Fix potential segfault in waitForInput()
Since the addition of build-max-log-size, a call to
handleChildOutput() can result in cancellation of a goal.  This
invalidated the "j" iterator in the waitForInput() loop, even though
it was still used afterwards.  Likewise for the maxSilentTime
handling.

Probably fixes #231.  At least it gets rid of the valgrind warnings.
2014-03-29 22:14:11 +01:00
Eelco Dolstra 90dc50b07c restoreSIGPIPE(): Fill in sa_mask
Issue #231.
2014-03-29 20:20:14 +01:00
Eelco Dolstra 49009573bc Don't interpret strings as format strings
Ludo reported this error:

  unexpected Nix daemon error: boost::too_few_args: format-string refered to more arguments than were passed

coming from this line:

  printMsg(lvlError, run.program + ": " + string(err, 0, p));

The problem here is that the string ends up implicitly converted to a
Boost format() object, so % characters are treated specially.  I
always assumed (wrongly) that strings are converted to a format object
that outputs the string as-is.

Since this assumption appears in several places that may be hard to
grep for, I've added some C++ type hackery to ensures that the right
thing happens.  So you don't have to worry about % in statements like

  printMsg(lvlError, "foo: " + s);

or

  throw Error("foo: " + s);
2014-03-28 16:59:26 +01:00
Ludovic Courtès 24cb65efc3 Make /dev/kvm optional
The daemon now creates /dev deterministically (thanks!).  However, it
expects /dev/kvm to be present.

The patch below restricts that requirement (1) to Linux-based systems,
and (2) to systems where /dev/kvm already exists.

I’m not sure about the way to handle (2).  We could special-case
/dev/kvm and create it (instead of bind-mounting it) in the chroot, so
it’s always available; however, it wouldn’t help much since most likely,
if /dev/kvm missing, then KVM support is missing.
2014-03-21 17:27:19 +01:00
Eelco Dolstra 3fc056927c Fix tabs 2014-03-18 23:23:55 +01:00
Ludovic Courtès 51800e06de Allow recovery from isValidPath RPCs with an invalid path
Currently, clients cannot recover from an isValidPath RPC with an
invalid path parameter because the daemon closes the connection when
that happens.

More precisely:

  1. in performOp, wopIsValidPath case, ‘readStorePath’ raises an
     ‘Error’ exception;

  2. that exception is caught by the handler in ‘processConnection’;

  3. the handler determines errorAllowed == false, and thus exits after
     sending the message.

This last part is fixed by calling ‘startWork’ early on, as in the patch
below.

The same reasoning could be applied to all the RPCs that take one or
more store paths as inputs, but isValidPath is, by definition, likely to
be passed invalid paths in the first place, so it’s important for this
one to allow recovery.
2014-03-18 23:22:49 +01:00
Eelco Dolstra f93e97517e Fix -j and other flags when using the daemon 2014-03-17 17:35:11 +01:00
Eelco Dolstra fb8d8f5428 Remove unnecessary null pointer checks
Fixes #225.
2014-03-12 14:42:25 +01:00
Eelco Dolstra 25386e5edc Fix passing meta attribute to buildenv.nix
Since the meta attributes were not sorted, attribute lookup could
fail, leading to package priorities and active flags not working
correctly.

Broken since 0f24400d90.
2014-03-11 17:34:02 +01:00
Shea Levy 049a379ec6 The expr of AttrNames/DynamicAttrDefs is always an ExprConcatStrings 2014-03-10 10:14:50 +01:00
Shea Levy 908e9ce259 If a dynamic attribute name evaluates to null, remove it from the set 2014-03-10 10:14:50 +01:00
Eelco Dolstra 2caab81660 Revert "Make ifs and asserts tail-recursive"
This reverts commit 273322c773.
2014-03-05 16:18:13 +01:00
Eelco Dolstra 1017bd68ea Set up a private /dev/pts in the chroot 2014-02-27 23:35:23 +01:00
Eelco Dolstra 3fd01b171a Set up a minimal /dev in chroots
Not bind-mounting the /dev from the host also solves the problem with
/dev/shm being a symlink to something not in the chroot.
2014-02-27 23:17:53 +01:00
Eelco Dolstra c9f6232304 Correctly detect infinite recursion in function application
If we're evaluating some application ‘v = f x’, we can't store ‘f’
temporarily in ‘v’, because if ‘f x’ refers to ‘v’, it will get ‘f’
rather than an infinite recursion error.

Unfortunately, this breaks the tail call optimisation introduced in
c897bac549.

Fixes #217.
2014-02-27 21:47:59 +01:00
Eelco Dolstra 29cde917fe Fix deadlock in SubstitutionGoal
We were relying on SubstitutionGoal's destructor releasing the lock,
but if a goal is a top-level goal, the destructor won't run in a
timely manner since its reference count won't drop to zero.  So
release it explicitly.

Fixes #178.
2014-02-27 13:34:13 +01:00
Eelco Dolstra 91f25f0510 And another one 2014-02-26 18:55:18 +01:00
Eelco Dolstra 432328cc55 Remove another unused function 2014-02-26 18:49:36 +01:00
Eelco Dolstra 509993e598 Remove unused function 2014-02-26 18:48:32 +01:00
Eelco Dolstra 6b9cd59a41 nix-store -r: Respect --add-root for non-derivations
Fixes #68.
Fixes #117.
2014-02-26 16:32:46 +01:00
Eelco Dolstra f14ef84a51 Warn about missing -I paths
Fixes #121.  Note that we don't warn about missing $NIX_PATH entries
because it's intended that some may be missing (cf. the default
$NIX_PATH on NixOS, which includes paths like /etc/nixos/nixpkgs for
backward compatibility).
2014-02-26 15:21:56 +01:00
Eelco Dolstra 42eb4afd7a Simplify getting use-ssh-substituter from untrusted users 2014-02-26 13:58:46 +01:00
Eelco Dolstra dcaea042fc Only start download-via-ssh if it's enabled 2014-02-26 13:41:03 +01:00
Shea Levy df5de9dfd7 Add use-ssh-substituter setting.
It defaults to false and can be overridden by RemoteStore.

Untested currently, just quickly put this together
2014-02-26 13:32:47 +01:00
Eelco Dolstra a897b58373 nix-instantiate: Allow --dry-run as a synonym for --readonly-mode
--dry-run is more consistent with nix-env and nix-store.
2014-02-19 16:46:33 +01:00
Eelco Dolstra e1cf40fa95 nix-instantiate: Rename --eval-only to --eval, --parse-only to --parse 2014-02-19 16:34:24 +01:00
Eelco Dolstra c31836008e nix-instantiate: Add a flag --expr / -E to read expressions from the command line
This is basically a shortcut for ‘echo 'expr...' | nix-instantiate -’.
Also supported by nix-build and nix-shell.
2014-02-19 16:30:19 +01:00
Shea Levy a7e70518b8 lexer-tab.o and parser-tab.o require each other's headers 2014-02-18 17:31:30 +01:00
Eelco Dolstra 8129cf33d9 Slight simplification 2014-02-18 10:46:30 +01:00
Eelco Dolstra 1aa19b24b2 Add a flag ‘--check’ to verify build determinism
The flag ‘--check’ to ‘nix-store -r’ or ‘nix-build’ will cause Nix to
redo the build of a derivation whose output paths are already valid.
If the new output differs from the original output, an error is
printed.  This makes it easier to test if a build is deterministic.
(Obviously this cannot catch all sources of non-determinism, but it
catches the most common one, namely the current time.)

For example:

  $ nix-build '<nixpkgs>' -A patchelf
  ...
  $ nix-build '<nixpkgs>' -A patchelf --check
  error: derivation `/nix/store/1ipvxsdnbhl1rw6siz6x92s7sc8nwkkb-patchelf-0.6' may not be deterministic: hash mismatch in output `/nix/store/4pc1dmw5xkwmc6q3gdc9i5nbjl4dkjpp-patchelf-0.6.drv'

The --check build fails if not all outputs are valid.  Thus the first
call to nix-build is necessary to ensure that all outputs are valid.

The current outputs are left untouched: the new outputs are either put
in a chroot or diverted to a different location in the store using
hash rewriting.
2014-02-18 01:01:14 +01:00
Eelco Dolstra b6def5b542 Make --repair work on Darwin
Mac OS X doesn't allow renaming a read-only directory.

http://hydra.nixos.org/build/9113895
2014-02-17 23:09:48 +01:00
Eelco Dolstra dfbcb7c403 Refactoring 2014-02-17 23:04:52 +01:00
Eelco Dolstra 71adb090f0 When using a build hook, only copy missing paths 2014-02-17 22:58:21 +01:00
Eelco Dolstra 69fe6c58fa Move some code around
In particular, do replacing of valid paths during repair later.  This
prevents us from replacing a valid path after the build fails.
2014-02-17 22:25:15 +01:00
Eelco Dolstra 1da6ae4f99 nix-store --gc --max-freed: Support a unit specifier
E.g. "--max-freed 10G" means "free ten gigabytes".
2014-02-17 14:48:50 +01:00
Eelco Dolstra 00d30496ca Heuristically detect if a build may have failed due to a full disk
This will allow Hydra to detect that a build should not be marked as
"permanently failed", allowing it to be retried later.
2014-02-17 14:15:56 +01:00
Eelco Dolstra a9d99ab55f download-via-ssh: Use readStorePath 2014-02-14 12:31:10 +01:00
Eelco Dolstra 4db572062c download-via-ssh: Show where we're downloading from 2014-02-14 12:20:12 +01:00
Eelco Dolstra dba33d4018 Minor style fixes 2014-02-14 11:48:42 +01:00
Shea Levy f67f52751f Indendation fix
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-12 07:33:07 -05:00
Shea Levy 62eb9eb76d Remove relic of old code
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-12 07:27:45 -05:00
Shea Levy 7438f0bc2b error messages start in lowercase
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-12 07:26:35 -05:00
Shea Levy 2246aa77d2 Remove using declarations from download-via-ssh
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-12 07:22:36 -05:00
Shea Levy c89d6b9b63 nix-store --serve: Use a versioned protocol
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 07:43:13 -05:00
Shea Levy 38c3beac1a Move StoreApi::serve into opServe
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:52:48 -05:00
Shea Levy 1614603165 Pass in params by const ref
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:49:37 -05:00
Shea Levy 78d979567f Clarify comment
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:43:29 -05:00
Shea Levy 64e23d0a38 Add download-via-ssh substituter
This substituter connects to a remote host, runs nix-store --serve
there, and then forwards substituter commands on to the remote host and
sends their results to the calling program. The ssh-substituter-hosts
option can be specified as a list of hosts to try.

This is an initial implementation and, while it works, it has some
limitations:

* Only the first host is used
* There is no caching of query results (all queries are sent to the
  remote machine)
* There is no informative output (such as progress bars)
* Some failure modes may cause unhelpful error messages
* There is no concept of trusted-ssh-substituter-hosts

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy 5671188eb2 nix-store --serve: Flush out after every loop
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy 73874629ef nix-store --serve: Use dump instead of export
Also remove signing support

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy 188f96500b nix-store --serve: Don't fail if asked for info about non-valid path
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy 9488447594 nix-store --serve: Don't loop forever
nix-store --export takes a tmproot, which can only release by exiting.
Substituters don't currently work in a way that could take advantage of
the looping, anyway.

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:32 -05:00
Shea Levy 3a38d0f356 Add the nix-store --serve command
This is essentially the substituter API operating on the local store,
which will be used by the ssh substituter. It runs in a loop rather than
just taking one command so that in the future nix will be able to keep
one connection open for multiple instances of the substituter.

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:32 -05:00
Shea Levy 84a8b5e9af nix-instantiate --eval-only --read-write-mode: Don't depend on ordering
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-07 18:03:38 +01:00
Shea Levy e4058fab64 Rename --no-readonly-mode --read-write-mode
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-07 18:03:38 +01:00
Shea Levy 0c3e8a616e nix-instantiate: Add a --no-readonly-mode flag
This allows running nix-instantiate --eval-only without performing the
evaluation in readonly mode, letting features like import from
derivation and automatic substitution of builtins.storePath paths work.

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-07 18:03:38 +01:00
Eelco Dolstra d210cdc435 Fix assertion failure in ‘nix-store --load-db’
Namely:

  nix-store: derivations.cc:242: nix::Hash nix::hashDerivationModulo(nix::StoreAPI&, nix::Derivation): Assertion `store.isValidPath(i->first)' failed.

This happened because of the derivation output correctness check being
applied before the references of a derivation are valid.
2014-02-03 22:36:07 +01:00
Eelco Dolstra d6582c04c1 Give a friendly error message if the DB directory is not writable
Previously we would say "error: setting synchronous mode: unable to
open database file" which isn't very helpful.
2014-02-01 16:57:38 +01:00
Eelco Dolstra 844d83352c More "make dist" fixes 2014-02-01 15:18:48 +01:00
Eelco Dolstra 6ef32bddc1 Fix "make dist" 2014-02-01 14:38:12 +01:00
Eelco Dolstra 0c6d62cf27 Remove Automakefiles 2014-02-01 13:54:38 +01:00
Eelco Dolstra 16e7d69209 Update Makefile variable names 2014-02-01 13:54:38 +01:00
Eelco Dolstra e0234dfddc Rename Makefile -> local.mk 2014-01-30 12:11:06 +01:00
Eelco Dolstra 4a2ec9c659 Install nix-worker symlink 2014-01-30 12:08:26 +01:00
Eelco Dolstra 94f9c14d52 Fix some clang warnings 2014-01-21 18:29:55 +01:00
Eelco Dolstra 68cde53c47 Fix building against Bison 3.0.2 2014-01-21 15:34:04 +01:00
Eelco Dolstra 81628a6ccc Merge branch 'master' into make
Conflicts:
	src/libexpr/eval.cc
2014-01-21 15:30:01 +01:00
Eelco Dolstra 5ef8508a92 Remove unused type 2014-01-21 15:11:57 +01:00
Shea Levy f5e5793cd2 Bare dynamic attrs: Match interpolation semantics
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-01-14 14:00:15 +01:00
Shea Levy f9913f4422 Allow "bare" dynamic attrs
Now, in addition to a."${b}".c, you can write a.${b}.c (applicable
wherever dynamic attributes are valid).

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-01-14 14:00:15 +01:00
Eelco Dolstra b1db599dd0 Generate schema.sql.hh 2014-01-09 22:10:35 +01:00
Eelco Dolstra b4c684e0f9 Update Makefiles 2014-01-09 16:53:47 +01:00
Eelco Dolstra 11cb4bfb25 Fix checking of NAR hashes
*headdesk*
*headdesk*
*headdesk*

So since commit 22144afa8d, Nix hasn't
actually checked whether the content of a downloaded NAR matches the
hash specified in the manifest / NAR info file.  Urghhh...
2014-01-08 17:35:49 +01:00
Eelco Dolstra 7a61c88dbb Merge branch 'dynamic-attrs-no-sugar' of github.com:shlevy/nix 2014-01-06 15:46:18 +01:00
Domen Kožar 485f4740ee wording 2014-01-06 11:38:24 +01:00
Shea Levy cd49fe4f9b Don't use any syntactic sugar for dynamic attrs
This doesn't change any functionality but moves some behavior out of the
parser and into the evaluator in order to simplify the code.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-31 23:56:26 +00:00
Shea Levy 6f3a51809a Fold dynamic binds handling into addAttr
Since addAttr has to iterate through the AttrPath we pass it, it makes
more sense to just iterate through the AttrNames in addAttr instead. As
an added bonus, this allows attrsets where two dynamic attribute paths
have the same static leading part (see added test case for an example
that failed previously).

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-31 17:57:10 -05:00
Shea Levy 18fefacf7d Dynamic attrs
This adds new syntax for attribute names:

* attrs."${name}" => getAttr name attrs
* attrs ? "${name}" => isAttrs attrs && hasAttr attrs name
* attrs."${name}" or def => if attrs ? "${name}" then attrs."${name}" else def
* { "${name}" = value; } => listToAttrs [{ inherit name value; }]

Of course, it's a bit more complicated than that. The attribute chains
can be arbitrarily long and contain combinations of static and dynamic
parts (e.g. attrs."${foo}".bar."${baz}" or qux), which is relatively
straightforward for the getAttrs/hasAttrs cases but is more complex for
the listToAttrs case due to rules about duplicate attribute definitions.

For attribute sets with dynamic attribute names, duplicate static
attributes are detected at parse time while duplicate dynamic attributes
are detected when the attribute set is forced. So, for example, { a =
null; a.b = null; "${"c"}" = true; } will be a parse-time error, while
{ a = {}; "${"a"}".b = null; c = true; } will be an eval-time error
(technically that case could theoretically be detected at parse time,
but the general case would require full evaluation). Moreover, duplicate
dynamic attributes are not allowed even in cases where they would be
with static attributes ({ a.b.d = true; a.b.c = false; } is legal, but {
a."${"b"}".d = true; a."${"b"}".c = false; } is not). This restriction
might be relaxed in the future in cases where the static variant would
not be an error, but it is not obvious that that is desirable.

Finally, recursive attribute sets with dynamic attributes have the
static attributes in scope but not the dynamic ones. So rec { a = true;
"${"b"}" = a; } is equivalent to { a = true; b = true; } but rec {
"${"a"}" = true; b = a; } would be an error or use a from the
surrounding scope if it exists.

Note that the getAttr, getAttr or default, and hasAttr are all
implemented purely in the parser as syntactic sugar, while attribute
sets with dynamic attribute names required changes to the AST to be
implemented cleanly.

This is an alternative solution to and closes #167

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-31 20:59:49 +00:00