Commit graph

670 commits

Author SHA1 Message Date
Eelco Dolstra bf4a577a58 Fix broken patch 2014-02-26 13:48:23 +01:00
Ian-Woo Kim 8a02fdc38e use USER environmental variable if getting user id by getpwuid is failed in perl scripts: download-from-binary-cache.pl and nix-channel 2014-02-26 13:42:11 +01:00
Eelco Dolstra 36b90e72d7 nix-shell: Add --packages flag
This allows you to easily set up a build environment containing the
specified packages from Nixpkgs.  For example:

  $ nix-shell -p sqlite xorg.libX11 hello

will start a shell in which the given packages are present.
2014-02-19 17:08:01 +01:00
Eelco Dolstra c31836008e nix-instantiate: Add a flag --expr / -E to read expressions from the command line
This is basically a shortcut for ‘echo 'expr...' | nix-instantiate -’.
Also supported by nix-build and nix-shell.
2014-02-19 16:30:19 +01:00
Eelco Dolstra 73f74ebba0 nix-shell: Don't leave a temporary directory in /tmp behind 2014-02-19 15:01:04 +01:00
Eelco Dolstra 1aa19b24b2 Add a flag ‘--check’ to verify build determinism
The flag ‘--check’ to ‘nix-store -r’ or ‘nix-build’ will cause Nix to
redo the build of a derivation whose output paths are already valid.
If the new output differs from the original output, an error is
printed.  This makes it easier to test if a build is deterministic.
(Obviously this cannot catch all sources of non-determinism, but it
catches the most common one, namely the current time.)

For example:

  $ nix-build '<nixpkgs>' -A patchelf
  ...
  $ nix-build '<nixpkgs>' -A patchelf --check
  error: derivation `/nix/store/1ipvxsdnbhl1rw6siz6x92s7sc8nwkkb-patchelf-0.6' may not be deterministic: hash mismatch in output `/nix/store/4pc1dmw5xkwmc6q3gdc9i5nbjl4dkjpp-patchelf-0.6.drv'

The --check build fails if not all outputs are valid.  Thus the first
call to nix-build is necessary to ensure that all outputs are valid.

The current outputs are left untouched: the new outputs are either put
in a chroot or diverted to a different location in the store using
hash rewriting.
2014-02-18 01:01:14 +01:00
Eelco Dolstra e81d38c02b nix-shell: Execute shellHook if it exists
Since normal builds don't execute shellHook, this allows nix-shell
specific customisation.  Suggested by Domen.
2014-02-17 13:34:24 +01:00
Eelco Dolstra 1f841c9d50 Force use of Bash
"echo -n" doesn't work with /bin/sh on Darwin.
2014-02-10 17:42:36 +01:00
Eelco Dolstra 57386c9bae Binary tarball: Automatically create /nix
The tarball can now be unpacked anywhere.  The installation script
uses "sudo" to create /nix if it doesn't exist.  It also fetches the
nixpkgs-unstable channel.
2014-02-10 16:35:59 +01:00
Eelco Dolstra c5839752b9 Binary tarball: Automatically fetch the Nixpkgs channel 2014-02-10 10:50:29 +01:00
Eelco Dolstra b632153ebd nix-shell: Use shell.nix as the default expression if it exists 2014-02-10 10:25:13 +01:00
Eelco Dolstra 2f9bb5c7e7 Merge branch 'make' 2014-02-01 16:41:52 +01:00
Eelco Dolstra 6ef32bddc1 Fix "make dist" 2014-02-01 14:38:12 +01:00
Eelco Dolstra 0c6d62cf27 Remove Automakefiles 2014-02-01 13:54:38 +01:00
Eelco Dolstra 16e7d69209 Update Makefile variable names 2014-02-01 13:54:38 +01:00
Eelco Dolstra e0234dfddc Rename Makefile -> local.mk 2014-01-30 12:11:06 +01:00
Eelco Dolstra c5ba081333 nix-shell: Add --impure flag
This is currently the default, but I might change that to --pure in
the future.
2014-01-23 13:34:04 +01:00
Eelco Dolstra 79dee4283d nix-shell: Preserve the TZ variable of the user 2014-01-23 13:34:04 +01:00
Eelco Dolstra 7fdee6e136 nix-build: Refactor 2014-01-23 13:34:04 +01:00
Eelco Dolstra 81628a6ccc Merge branch 'master' into make
Conflicts:
	src/libexpr/eval.cc
2014-01-21 15:30:01 +01:00
Eelco Dolstra bf0ad8aabc nix-profile.sh: Add the Nixpkgs channel to $NIX_PATH 2014-01-15 14:34:49 +01:00
Eelco Dolstra f1357059a4 nix-shell: Don't set NIX_INDENT_MAKE
It generally is not useful in interactive environments (and messes up
some non-ANSI-compliant terminals).
2014-01-13 13:46:44 +01:00
Eelco Dolstra ea59f39326 nix-shell: Set $IN_NIX_SHELL before evaluating 2014-01-13 13:43:31 +01:00
Eelco Dolstra 0fdf4da0e9 Support cryptographically signed binary caches
NAR info files in binary caches can now have a cryptographic signature
that Nix will verify before using the corresponding NAR file.

To create a private/public key pair for signing and verifying a binary
cache, do:

  $ openssl genrsa -out ./cache-key.sec 2048
  $ openssl rsa -in ./cache-key.sec -pubout > ./cache-key.pub

You should also come up with a symbolic name for the key, such as
"cache.example.org-1".  This will be used by clients to look up the
public key.  (It's a good idea to number keys, in case you ever need
to revoke/replace one.)

To create a binary cache signed with the private key:

  $ nix-push --dest /path/to/binary-cache --key ./cache-key.sec --key-name cache.example.org-1

The public key (cache-key.pub) should be distributed to the clients.
They should have a nix.conf should contain something like:

  signed-binary-caches = *
  binary-cache-public-key-cache.example.org-1 = /path/to/cache-key.pub

If all works well, then if Nix fetches something from the signed
binary cache, you will see a message like:

  *** Downloading ‘http://cache.example.org/nar/7dppcj5sc1nda7l54rjc0g5l1hamj09j-subversion-1.7.11’ (signed by ‘cache.example.org-1’) to ‘/nix/store/7dppcj5sc1nda7l54rjc0g5l1hamj09j-subversion-1.7.11’...

On the other hand, if the signature is wrong, you get a message like

  NAR info file `http://cache.example.org/7dppcj5sc1nda7l54rjc0g5l1hamj09j.narinfo' has an invalid signature; ignoring

Signatures are implemented as a single line appended to the NAR info
file, which looks like this:

  Signature: 1;cache.example.org-1;HQ9Xzyanq9iV...muQ==

Thus the signature has 3 fields: a version (currently "1"), the ID of
key, and the base64-encoded signature of the SHA-256 hash of the
contents of the NAR info file up to but not including the Signature
line.

Issue #75.
2014-01-08 15:42:53 +01:00
Eelco Dolstra 405434e084 Revert "nix-shell: Set $IN_NIX_SHELL before evaluation"
This reverts commit 0c1198cf08.
2014-01-06 17:38:04 +01:00
Shea Levy 5ba5993470 nix-shell --pure: Don't clear IN_NIX_SHELL
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-30 16:59:24 +01:00
Eelco Dolstra b352fe2775 Revert "Scan /proc/<pid>/cmdline for GC roots"
This reverts commit 194e3374b8.
Checking the command line for GC roots means that

  $ nix-store --delete $path

will fail because $path is now a root because it's mentioned on the
command line.
2013-12-20 14:18:24 +01:00
Eelco Dolstra 194e3374b8 Scan /proc/<pid>/cmdline for GC roots 2013-12-20 13:31:31 +01:00
Eelco Dolstra 769f662165 nix-shell: Don't warn about the lack of a GC root 2013-12-20 12:19:10 +00:00
Eelco Dolstra 0c1198cf08 nix-shell: Set $IN_NIX_SHELL before evaluation
This has some hacky applications.
2013-12-20 13:11:41 +01:00
Eelco Dolstra 65a6452240 nix-shell: Handle --option correctly
Fixes #181.
2013-12-20 13:10:14 +01:00
Eelco Dolstra 7b0d8fb23d nix-shell --pure: Keep $TERM 2013-12-17 18:16:04 +01:00
Eelco Dolstra f3cf0436b5 Install bsdiff and bspatch in $(libexecdir)/nix 2013-12-02 20:21:31 +00:00
Eelco Dolstra 0202ce6b94 Add support for ‘make installcheck’ 2013-11-25 18:47:03 +01:00
Eelco Dolstra 7ce0e05ad8 Rename Makefile.new -> Makefile 2013-11-25 15:25:13 +00:00
Eelco Dolstra 962551a071 Add a Makefile for the scripts directory 2013-11-25 15:17:16 +00:00
Eelco Dolstra 2d9bb56e55 Fix segfault on Darwin
Ever since SQLite in Nixpkgs was updated to 3.8.0.2, Nix has randomly
segfaulted on Darwin:

  http://hydra.nixos.org/build/6175515
  http://hydra.nixos.org/build/6611038

It turns out that this is because the binary cache substituter somehow
ends up loading two versions of SQLite: the one in Nixpkgs and the
other from /usr/lib/libsqlite3.dylib.  It's not exactly clear why the
latter is loaded, but it appears to be because WWW::Curl indirectly loads
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation,
which in turn seems to load /usr/lib/libsqlite3.dylib.  This leads to
a segfault when Perl exits:

  #0  0x00000001010375f4 in sqlite3_finalize ()
  #1  0x000000010125806e in sqlite_st_destroy ()
  #2  0x000000010124bc30 in XS_DBD__SQLite__st_DESTROY ()
  #3  0x00000001001c8155 in XS_DBI_dispatch ()
  ...
  #14 0x0000000100023224 in perl_destruct ()
  #15 0x0000000100000d6a in main ()
  ...

The workaround is to explicitly load DBD::SQLite before WWW::Curl.
2013-10-24 19:15:52 +02:00
Eelco Dolstra 4f4a14453a Don't set $PS1 in non-interactive shells
Shouldn't really matter, but you never know.
2013-10-18 14:51:25 +02:00
Shea Levy 4ea034a5c5 nix-shell: Play nicely with non-interactive shells
nix-shell with the --command flag might be used non-interactively, but
if bash starts non-interactively (i.e. with stdin or stderr not a
terminal), it won't source the script given in --rcfile. However, in
that case it *will* source the script found in $BASH_ENV, so we can use
that instead.

Also, don't source ~/.bashrc in a non-interactive shell (detectable by
checking the PS1 env var)

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-18 14:34:46 +02:00
Eelco Dolstra c1994fecf9 nix-shell: Fix bash completion
Nixpkgs's stdenv setup script sets the "nullglob" option, but doing so
breaks Bash completion on NixOS (when ‘programs.bash.enableCompletion’
is set) and on Ubuntu.  So clear that flag afterwards.  Of course,
this may break stdenv functions in subtle ways...
2013-10-14 15:28:43 +02:00
Eelco Dolstra faaae44f2e build-remote.pl: Don't use substituters on the remote
It's kinda pointless to check substituters on the remote side, since
we just checked them locally.
2013-09-18 14:04:03 +02:00
Eelco Dolstra 5904262640 nix-shell: Support a .drv as argument
Fixes #161.
2013-09-06 14:58:05 +02:00
Eelco Dolstra 511455965e nix-shell: Don't disable Automake dependency tracking
Nixpkgs' stdenv disables dependency tracking by default.  That makes
sense for one-time builds, but in an interactive environment we expect
repeated "make" invocations to do the right thing.
2013-07-31 13:17:50 +02:00
Eelco Dolstra 15e5ac8039 nix-shell: Set $IN_NIX_SHELL
This allows scripts to distinguish between a real build and a Nix
shell.
2013-07-19 14:32:05 +02:00
Eelco Dolstra 48858ad5ca nix-shell: Set some environment variables also set by build.cc
Setting $NIX_STORE causes the purity checks in gcc/ld-wrapper to kick
in, so that's why we unset $NIX_ENFORCE_PURITY.
2013-07-19 14:06:58 +02:00
Eelco Dolstra 2bc5de8635 Rename ‘nix-build --run-env’ to ‘nix-shell’ 2013-07-19 12:02:44 +02:00
Eelco Dolstra dc5f2e7da6 nix-build --run-env: Add a ‘--pure’ flag
This causes the environment to be (almost) cleared, thus giving a
shell that more closely resembled the actual Nix derivation.
2013-07-19 11:23:32 +02:00
Eelco Dolstra a4921b8ceb Revert "build-remote.pl: Enforce timeouts locally"
This reverts commit 69b8f9980f.

The timeout should be enforced remotely.  Otherwise, if the garbage
collector is running either locally or remotely, if will block the
build or closure copying for some time.  If the garbage collector
takes too long, the build may time out, which is not what we want.
Also, on heavily loaded systems, copying large paths to and from the
remote machine can take a long time, also potentially resulting in a
timeout.
2013-07-18 12:52:29 +02:00
Eelco Dolstra 6fcc290619 build-remote.pl: Move "building ..." message to a better place 2013-07-11 17:16:45 +02:00
Eelco Dolstra ed09d0c0a4 nix-build --run-env: Always use Bash
Fixes #113.
Fixes #131.
2013-07-11 14:40:29 +02:00
Eelco Dolstra 656390062a nix-build --run-env: Source $stdenv/setup in the interactive shell
This ensures that not just environment variables are set, but also
shell functions such as unpackPhase, configurePhase and so on.
2013-07-11 14:32:22 +02:00
Eelco Dolstra 5116214343 Add support for uncompressed NARs in binary caches
Issue NixOS/hydra#102.
2013-07-01 21:03:14 +02:00
Eelco Dolstra 7986711632 copy-from-other-stores.pl: Respect $NIX_BIN_DIR 2013-07-01 13:30:28 +02:00
Eelco Dolstra 1917d750a0 copy-from-other-stores.pl: Report downloaded size as 0 2013-07-01 13:29:59 +02:00
Eelco Dolstra 22144afa8d Don't keep "disabled" substituters running
For instance, it's pointless to keep copy-from-other-stores running if
there are no other stores, or download-using-manifests if there are no
manifests.  This also speeds things up because we don't send queries
to those substituters.
2013-06-20 11:55:15 +02:00
Eelco Dolstra 5c06e5297d download-from-binary-cache.pl: Respect $NIX_CONNECT_TIMEOUT 2013-06-07 16:25:12 +02:00
Eelco Dolstra 24a356bf71 Replace $NIX_DEBUG_SUBST with an option ‘debug-subst’
Thus passing ‘--option debug-subst 1’ allows daemon users to turn on
debug info and see what the substituter is doing.
2013-06-07 15:36:36 +02:00
Eelco Dolstra 24e063efdc download-from-binary-cache.pl: Show if we're waiting for a URL
Previously, if a binary cache is hanging/unreachable/slow,
download-from-binary-cache.pl would also hang without any indication
to the user.  Now, if fetching a URL takes more than 5 seconds, it
will print a message to that effect.
2013-06-07 15:33:44 +02:00
Eelco Dolstra 75e12b8e66 download-from-binary-cache.pl: Fix race condition
Fixes the error "DBD::SQLite::db do failed: column url is not unique".
2013-06-05 16:25:07 +02:00
Eelco Dolstra f0576d6775 Update the default binary cache URL to cache.nixos.org 2013-06-05 13:36:43 +02:00
Eelco Dolstra ff08306746 download-from-binary-cache.pl: Treat a 403 error as a 404
Amazon S3 returns HTTP status code 403 if a file doesn't exist and the
user has no permission to list the contents of the bucket.  So treat
it as 404 (meaning it's cached in the NARExistence table).
2013-06-04 15:20:37 +02:00
Eelco Dolstra bc2e43f3c8 build-remote.pl: Remove meaningless signing when importing the output paths
The "$UID != 0" makes no sense: if the local side has write access to
the Nix store (which is always the case) then it doesn't matter if
we're root - we can import unsigned paths either way.
2013-05-29 18:18:59 +02:00
Eelco Dolstra 107505e13a build-remote.pl: Copy all outputs in one operation 2013-05-23 14:39:58 -04:00
Eelco Dolstra 72d8209548 build-remote.pl: Indicate if remote machine is refusing builds
Fixes #120.
2013-05-23 14:04:46 -04:00
Eelco Dolstra a4cb62ac25 download-from-binary-cache.pl: Get rid of an uninitialized value warning
Reported by Pablo Costa.
2013-05-15 15:47:05 +02:00
Eelco Dolstra 3a0cc43ac8 build-remote.pl: Properly close the SSH connection between attempts 2013-05-10 02:38:50 +02:00
Eelco Dolstra be0b9dda31 build-remote.pl: Pass /dev/null as SSH's stdin
Otherwise it will set the parent's stdin to non-blocking mode, causing
the subsequent read of the set of inputs/outputs to fail randomly.
That's insane.
2013-05-10 02:32:13 +02:00
Eelco Dolstra 78206f06ec build-remote.pl: Allow a machine to refuse a build
Before selecting a machine, build-remote.pl will try to run the
command "nix-builds-inhibited" on the machine.  If this command exists
and returns a 0 exit code, then the machine won't be used.  It's up to
the user to provide this command, but it would typically be a script
that checks whether there is enough disk space and whether the load is
not too high.
2013-05-10 01:09:46 +02:00
Eelco Dolstra 7a03cbf09d build-remote.pl: Create one process fewer on the remote side 2013-05-09 17:30:07 +02:00
Eelco Dolstra 69b8f9980f build-remote.pl: Enforce timeouts locally
Don't pass --timeout / --max-silent-time to the remote builder.
Instead, let the local Nix process terminate the build if it exceeds a
timeout.  The remote builder will be killed as a side-effect.  This
gives better error reporting (since the timeout message from the
remote side wasn't properly propagated) and handles non-Nix problems
like SSH hangs.
2013-05-09 17:17:17 +02:00
Eelco Dolstra ea019e9a26 Add option ‘extra-binary-caches’
This allows providing additional binary caches, useful in scripts like
Hydra's build reproduction scripts, in particular because untrusted
caches are ignored.
2013-05-07 15:37:28 +02:00
Eelco Dolstra c51b6a893c nix-copy-closure: Show a proper error message if no host name is given 2013-05-03 11:12:11 +02:00
Lluís Batlle i Rossell e6c44d166a Fixing the pv position regarding compression
Problem noticed by niksnut.
2013-05-03 11:08:51 +02:00
Lluís Batlle i Rossell 7391533ea5 Fixing the pv reference; I didn't mean to change it 2013-05-03 11:08:51 +02:00
Lluís Batlle i Rossell 5cc2fc46ec Adding ETA support to the --show-progress in nix-copy-closure
Based on https://github.com/NixOS/nix/pull/6 from shlevy
2013-05-03 11:08:51 +02:00
Eelco Dolstra 4ddd077bfa find-runtime-roots.pl: Don't hardcode /nix/store 2013-04-26 12:15:54 +02:00
Eelco Dolstra 00f698eb8b find-runtime-roots.pl: Search process environments for roots
For instance, this prevents paths from being deleted that are in use
by a "nix-build --run-env" session.
2013-04-26 12:06:39 +02:00
Eelco Dolstra 938092a213 find-runtime-roots.pl: Use Nix::Utils::readFile 2013-04-26 11:44:19 +02:00
Eelco Dolstra 6955d41f2b nix-build: Respect --timeout 2013-04-23 17:16:01 +02:00
Eelco Dolstra 08d96ffad0 Fix --fallback with the binary cache substituter
Reported by Peter Simons.
2013-04-23 12:45:01 +02:00
Eelco Dolstra c642441beb Test NAR info caching 2013-04-23 12:43:28 +02:00
Eelco Dolstra bdd4646338 Revert "Prevent config.h from being clobbered"
This reverts commit 28bba8c44f.
2013-03-08 01:24:59 +01:00
Eelco Dolstra 28bba8c44f Prevent config.h from being clobbered 2013-03-07 23:55:55 +01:00
Eelco Dolstra dadf7a5b46 build-remote: Use the --quiet flag
‘--option verbosity 0’ doesn't actually do anything.
2013-02-19 16:56:31 +01:00
Eelco Dolstra 96fbbbde55 build-remote.pl: Don't keep build logs on the build slave 2013-01-21 22:26:25 +01:00
Eelco Dolstra 0a4e90395c Urgggh
http://hydra.nixos.org/build/3661100
2013-01-02 23:52:15 +01:00
Eelco Dolstra aebea2e489 Reinstate the http://nixos.org/binary-cache default for the binary-caches setting 2013-01-02 22:16:37 +01:00
Eelco Dolstra 68dcbb187e download-from-binary-cache: Use the channels of the calling user rather than root
This should make live easier for single-user (non-daemon)
installations.  Note that when the daemon is used, the "calling user"
is root so we're not using any untrusted caches.
2012-12-21 15:00:07 +01:00
Eelco Dolstra 8cc19ed089 Set a long SQLite timeout in the binary cache substituter 2012-12-06 11:43:34 +01:00
Eelco Dolstra 566afa1e9c Support xz compression of nixexprs.tar in channels 2012-12-05 11:02:44 +01:00
Eelco Dolstra 08964d7328 Undo accidental debug change 2012-11-26 17:57:14 +01:00
Eelco Dolstra 46a369ad95 Make "nix-build -A <derivation>.<output>" do the right thing
For example, given a derivation with outputs "out", "man" and "bin":

  $ nix-build -A pkg

produces ./result pointing to the "out" output;

  $ nix-build -A pkg.man

produces ./result-man pointing to the "man" output;

  $ nix-build -A pkg.all

produces ./result, ./result-man and ./result-bin;

  $ nix-build -A pkg.all -A pkg2

produces ./result, ./result-man, ./result-bin and ./result-2.
2012-11-26 15:39:10 +01:00
Eelco Dolstra a3d6585c5a nix-copy-closure: Add flag ‘--use-substitutes’ 2012-11-23 16:20:16 +01:00
Eelco Dolstra a9a8baaccb Use a shorter Nixpkgs channel URL 2012-11-09 13:33:35 +01:00
Eelco Dolstra e28b683324 download-from-binary-cache: Try next cache if downloading a NAR fails 2012-11-06 17:55:53 +01:00
Eelco Dolstra bbc107ef1e Process binary caches in order of priority
Binary caches can now specify a priority in their nix-cache-info file.
The binary cache substituter checks caches in order of priority.  This
is to ensure that fast, static caches like nixos.org/binary-cache are
processed before slow, dynamic caches like hydra.nixos.org.
2012-11-06 17:45:20 +01:00
Eelco Dolstra 82951e5582 nix-push: Handle pushing a symlink 2012-11-06 13:53:53 +01:00
Eelco Dolstra 620e92e880 Add an option ‘use-binary-caches’
This allows disabling the use of binary caches, e.g.

  $ nix-build ... --option use-binary-caches false

Note that

  $ nix-build ... --option binary-caches ''

does not disable all binary caches, since the caches defined by
channels will still be used.
2012-11-06 13:44:59 +01:00
Eelco Dolstra df27db712d Fix "Not an ARRAY reference" error
It's a mystery why this error is not triggered in the build farm
(e.g. http://hydra.nixos.org/build/3265602).  Ah well.
2012-11-06 13:43:46 +01:00
Eelco Dolstra a28b4445a4 nix-prefetch-url: Improve option handling 2012-10-23 18:04:11 +02:00