Commit graph

933 commits

Author SHA1 Message Date
Oliver Dunkl de91a42c6e Moves runHook to a later execution position
It moves runHook to a later position in the rcfile. After that we are
able to set the PS1 environment-variable for a nix-shell environment
e.g.:

  # turn the color of the prompt to blue
  shellHook = ''
    export PS1="\n\[\033[1;34m\][\u@\h:\w]$\[\033[0m\] ";
  '';
2015-01-28 13:39:48 +01:00
Eelco Dolstra f6716e95bb Shut up "Wide character in print" warning in copy-from-other-stores.pl 2015-01-15 17:56:56 +01:00
Eelco Dolstra a5e2c8e560 Set correct user agent for NAR downloads from binary caches 2015-01-15 12:05:27 +01:00
Eelco Dolstra 128538ef06 nix-shell: Add --run flag
‘--run’ is like ‘--command’, except that it runs the command in a
non-interactive shell. This is important if you do things like:

  $ nix-shell --command make

Hitting Ctrl-C while make is running drops you into the interactive
Nix shell, which is probably not what you want. So you can now do

  $ nix-shell --run make

instead.
2015-01-08 15:14:38 +01:00
Eelco Dolstra b76589206a nix-shell: Interpret filenames relative to the #!-script
So you can have a script like:

  #! /usr/bin/env nix-shell
  #! nix-shell script.nix -i python

  import prettytable

  x = prettytable.PrettyTable(["Foo", "Bar"])
  for i in range(1, 10): x.add_row([i, i**2])
  print x

with a ‘script.nix’ in the same directory:

  with import <nixpkgs> {};

  runCommand "dummy" { buildInputs = [ python pythonPackages.prettytable ]; } ""

(Of course, in this particular case, using the ‘-p’ flag is more
convenient.)
2015-01-08 14:56:14 +01:00
Eelco Dolstra a957893b26 Allow nix-shell to be used as a #! interpreter
This allows scripts to fetch their own dependencies via nix-shell. For
instance, here is a Haskell script that, when executed, pulls in GHC
and the HTTP package:

  #! /usr/bin/env nix-shell
  #! nix-shell -i runghc -p haskellPackages.ghc haskellPackages.HTTP

  import Network.HTTP

  main = do
    resp <- Network.HTTP.simpleHTTP (getRequest "http://nixos.org/")
    body <- getResponseBody resp
    print (take 100 body)

Or a Perl script that pulls in Perl and some CPAN packages:

  #! /usr/bin/env nix-shell
  #! nix-shell -i perl -p perl perlPackages.HTMLTokeParserSimple perlPackages.LWP

  use HTML::TokeParser::Simple;

  my $p = HTML::TokeParser::Simple->new(url => 'http://nixos.org/');

  while (my $token = $p->get_tag("a")) {
      my $href = $token->get_attr("href");
      print "$href\n" if $href;
  }

Note that the options to nix-shell must be given on a separate line
that starts with the magic string ‘#! nix-shell’. This is because
‘env’ does not allow passing arguments to an interpreter directly.
2015-01-08 14:32:45 +01:00
Eelco Dolstra 7ba0e9cb48 nix-shell --command: Remove bogus argument to "exit"
Fixes "exit: Inappropriate: numeric argument required" errors.
2015-01-07 16:10:20 +01:00
Eelco Dolstra c2a552b075 Install cacert before running nix-channel
Also, make it more robust against incorrent SSL_CERT_FILE values.
2014-12-13 16:53:21 +01:00
Eelco Dolstra 20cf0127f5 Include cacert in the binary tarball
This prevents having to fetch Nixpkgs or cacert over http.
2014-12-10 16:05:08 +01:00
Eelco Dolstra 2f16946064 Always use https to fetch the Nixpkgs channel 2014-12-10 11:35:56 +01:00
Eelco Dolstra 39fe52f7ac Fix bad comment 2014-12-10 11:35:05 +01:00
Eelco Dolstra af555d7694 Doh 2014-12-09 20:43:12 +01:00
Eelco Dolstra d44d923be9 Add option to disable binary cache certificate checking 2014-12-09 13:16:02 +01:00
Eelco Dolstra 5510d21193 Provide some fallback defaults for the CA bundle 2014-12-09 13:16:02 +01:00
Eelco Dolstra e5891f2ea8 Use https://cache.nixos.org instead of http://cache.nixos.org 2014-12-09 13:16:02 +01:00
Eelco Dolstra bf78a27ac9 Fix another operator precedence issue found by Perl 5.20 2014-12-05 19:25:13 +01:00
Eelco Dolstra 5d064e2698 Add a test for the binary tarball installer 2014-11-18 14:50:05 +01:00
Eelco Dolstra 5ef2453139 build-remote.pl.in: Add some more trace messages
This allows hydra-build to keep track of the actual build time (so
excluding time required to copy closures around).
2014-11-12 13:56:01 +01:00
Eelco Dolstra 8979562ed7 download-from-binary-cache.pl: Fix flushing of stderr 2014-11-04 14:37:58 +01:00
Shea Levy bca6d35636 Fix nix-copy-closure --from
http://hydra.nixos.org/build/15885652
2014-10-15 15:05:13 -04:00
Eelco Dolstra 138c257dcd Revert "binary download: Use $NIX_CURL_FLAGS"
This reverts commit bc4795919a. It
breaks the build:

  http://hydra.nixos.org/build/15860847
2014-10-15 10:22:17 +02:00
wmertens bc4795919a binary download: Use $NIX_CURL_FLAGS
As in 5c0816567d/pkgs/build-support/fetchurl/builder.sh (L17)
2014-10-14 15:36:14 +02:00
Eelco Dolstra 2c1711ae33 nix-channel: Add --rollback flag
Fixes #368.
2014-10-14 12:08:04 +02:00
Eelco Dolstra a8925a510c nix-channel --add: Validate URL / channel ID
Fixes #369.
2014-10-14 11:28:13 +02:00
Eelco Dolstra f1adf4c998 Remove unused @sshOpts flag
Closes #300.
2014-10-14 11:01:18 +02:00
Eelco Dolstra a79e56de68 nix-copy-closure: Use strict 2014-10-14 10:56:33 +02:00
Shell Turner 9dd54bc7bb Pass through --set from nix-install-package command line to nix-env 2014-09-23 15:11:24 +02:00
Eelco Dolstra d2d5543625 build-remote.pl: UTF-8-decode errors 2014-09-05 11:53:11 +02:00
Eelco Dolstra e74113a305 Fix build-remote.pl
Apparently, turning on utf8 encoding on stderr changes its flushing
behaviour, causing sendReply to not send anything.

http://hydra.nixos.org/build/13944384
2014-09-05 11:43:14 +02:00
Eelco Dolstra 27a01d92c2 Shut up "Wide character" warnings in Perl scripts 2014-08-29 17:48:25 +02:00
Eelco Dolstra 163fdf292e Set a curl timeout on binary cache lookups 2014-08-21 15:15:50 +02:00
Eelco Dolstra 11849a320e Use proper quotes everywhere 2014-08-20 18:03:48 +02:00
Eelco Dolstra 42e9ad8fd1 Propagate remote timeouts properly 2014-08-17 19:09:03 +02:00
Eelco Dolstra c6e85ee474 nix-build: Propagate exit status from nix-store -r 2014-08-17 18:37:09 +02:00
Eelco Dolstra ccd4fe5c49 build-remote.pl: Provide defaults for $NIX_CURRENT_LOAD and $NIX_REMOTE_SYSTEMS 2014-08-17 18:05:24 +02:00
Eelco Dolstra 3e5dbb2433 nix-shell: Use $XDG_RUNTIME_DIR
This prevents collisions with other users.

Fixes #262.
2014-08-13 23:16:08 +02:00
Eelco Dolstra 3bea429ae8 Use $XDG_RUNTIME_DIR for temporary files 2014-08-13 23:12:57 +02:00
Eelco Dolstra 57695d8f76 Warn about untrusted binary caches in extra-binary-caches 2014-08-07 22:46:35 +02:00
Eelco Dolstra 6e1f4c1488 nix-install-package: Use extra-binary-caches 2014-08-07 22:41:44 +02:00
Eelco Dolstra 393a9b92b1 download-from-binary-cache.pl: Respect $SSL_CERT_FILE 2014-08-07 22:41:15 +02:00
Eelco Dolstra 4a198dae74 install-nix-from-closure.sh: Use https channel if possible 2014-08-07 15:37:21 +02:00
Eelco Dolstra 2ea2703fe9 install-nix-from-closure.sh: Install cacert 2014-07-29 17:12:15 +02:00
Eelco Dolstra 50ed345fad nix-profile.sh: Set $SSL_CERT_FILE 2014-07-29 17:11:54 +02:00
Eelco Dolstra 1a44f1cb65 nix-copy-closure: Drop --bzip2, --xz, --show-progress
These are too difficult to implement via nix-store --serve.

‘--show-progress’ could be re-implemented fairly easily via a
sink/source wrapper class.
2014-07-24 17:11:54 +02:00
Eelco Dolstra 7f7d4ab686 nix-copy-closure: Implement --gzip via ssh's -C flag 2014-07-24 16:32:55 +02:00
Eelco Dolstra 7b351da680 Remove obsolete SSH master connection code 2014-07-24 16:19:42 +02:00
Eelco Dolstra 03103c0a36 Implement nix-copy-closure --from via nix-store --serve 2014-07-24 16:19:41 +02:00
Eelco Dolstra 62309a2c56 build-remote.pl: Be less verbose on failing builds 2014-07-24 12:43:59 +02:00
Eelco Dolstra 5bcb982711 Remove cruft 2014-07-14 12:39:53 +02:00
Eelco Dolstra a00a98548e build-remote.pl: Fix build log 2014-07-12 00:09:43 +02:00
Eelco Dolstra a5c6347ff0 build-remote.pl: Use ‘nix-store --serve’ on the remote side
This makes things more efficient (we don't need to use an SSH master
connection, and we only start a single remote process) and gets rid of
locking issues (the remote nix-store process will keep inputs and
outputs locked as long as they're needed).

It also makes it more or less secure to connect directly to the root
account on the build machine, using a forced command
(e.g. ‘command="nix-store --serve --write"’). This bypasses the Nix
daemon and is therefore more efficient.

Also, don't call nix-store to import the output paths.
2014-07-11 16:22:24 +02:00
Eelco Dolstra 7bb632b024 nix-copy-closure -s: Do substitutions via ‘nix-store --serve’
This means we no longer need an SSH master connection, since we only
execute a single command on the remote host.
2014-07-10 20:43:04 +02:00
Eelco Dolstra 1114c7bd57 nix-copy-closure: Restore compression and the progress viewer 2014-07-10 14:15:12 +02:00
Aristid Breitkreuz a457d5ad4d nix-build: --add-root also takes 1 parameter 2014-05-26 17:24:43 +02:00
Eelco Dolstra 924e19341a Don't barf when installing as root 2014-04-10 23:42:48 +02:00
Eelco Dolstra 2b6c8ef401 nix-shell --pure: Keep the user's $PAGER 2014-04-08 14:08:57 +02:00
Ricardo M. Correia 034b6f6062 nix-collect-garbage: Add --delete-older-than option 2014-03-30 00:54:16 +01:00
Maxim Ivanov 59c9019685 Fix nix-shell for derivation with multiple outputs
If derivation declares multiple outputs and first (default) output
if not "out", then "nix-instantiate" calls return path with output
names appended after "!". Than suffix must be stripped before
ant path checks are done.
2014-03-30 00:52:47 +01:00
Eelco Dolstra 77e2cc6c8e nix-build: Fix --cores flag 2014-03-17 17:33:13 +01:00
Eelco Dolstra 506d86394d Installer: Handle Darwin
"cp -r" doesn't copy symlinks properly on Darwin, but "cp -R" does.

Fixes #215.
2014-02-26 17:23:55 +01:00
Eelco Dolstra 7f74513b4e Also provide an option for setting the curl connection timeout 2014-02-26 16:07:43 +01:00
Eelco Dolstra 00d761016a Respect $NIX_CONNECT_TIMEOUT properly
We were 1) using CURLOPT_TIMEOUT instead of CURLOPT_CONNECTTIMEOUT; 2)
not passing it to the curl child process.

Issue #93.
2014-02-26 15:58:37 +01:00
Eelco Dolstra d761009e3c Add ~/.nix-profile/sbin to $PATH
Fixes #112.
2014-02-26 15:24:48 +01:00
Eelco Dolstra bf4a577a58 Fix broken patch 2014-02-26 13:48:23 +01:00
Ian-Woo Kim 8a02fdc38e use USER environmental variable if getting user id by getpwuid is failed in perl scripts: download-from-binary-cache.pl and nix-channel 2014-02-26 13:42:11 +01:00
Eelco Dolstra 36b90e72d7 nix-shell: Add --packages flag
This allows you to easily set up a build environment containing the
specified packages from Nixpkgs.  For example:

  $ nix-shell -p sqlite xorg.libX11 hello

will start a shell in which the given packages are present.
2014-02-19 17:08:01 +01:00
Eelco Dolstra c31836008e nix-instantiate: Add a flag --expr / -E to read expressions from the command line
This is basically a shortcut for ‘echo 'expr...' | nix-instantiate -’.
Also supported by nix-build and nix-shell.
2014-02-19 16:30:19 +01:00
Eelco Dolstra 73f74ebba0 nix-shell: Don't leave a temporary directory in /tmp behind 2014-02-19 15:01:04 +01:00
Eelco Dolstra 1aa19b24b2 Add a flag ‘--check’ to verify build determinism
The flag ‘--check’ to ‘nix-store -r’ or ‘nix-build’ will cause Nix to
redo the build of a derivation whose output paths are already valid.
If the new output differs from the original output, an error is
printed.  This makes it easier to test if a build is deterministic.
(Obviously this cannot catch all sources of non-determinism, but it
catches the most common one, namely the current time.)

For example:

  $ nix-build '<nixpkgs>' -A patchelf
  ...
  $ nix-build '<nixpkgs>' -A patchelf --check
  error: derivation `/nix/store/1ipvxsdnbhl1rw6siz6x92s7sc8nwkkb-patchelf-0.6' may not be deterministic: hash mismatch in output `/nix/store/4pc1dmw5xkwmc6q3gdc9i5nbjl4dkjpp-patchelf-0.6.drv'

The --check build fails if not all outputs are valid.  Thus the first
call to nix-build is necessary to ensure that all outputs are valid.

The current outputs are left untouched: the new outputs are either put
in a chroot or diverted to a different location in the store using
hash rewriting.
2014-02-18 01:01:14 +01:00
Eelco Dolstra e81d38c02b nix-shell: Execute shellHook if it exists
Since normal builds don't execute shellHook, this allows nix-shell
specific customisation.  Suggested by Domen.
2014-02-17 13:34:24 +01:00
Eelco Dolstra 1f841c9d50 Force use of Bash
"echo -n" doesn't work with /bin/sh on Darwin.
2014-02-10 17:42:36 +01:00
Eelco Dolstra 57386c9bae Binary tarball: Automatically create /nix
The tarball can now be unpacked anywhere.  The installation script
uses "sudo" to create /nix if it doesn't exist.  It also fetches the
nixpkgs-unstable channel.
2014-02-10 16:35:59 +01:00
Eelco Dolstra c5839752b9 Binary tarball: Automatically fetch the Nixpkgs channel 2014-02-10 10:50:29 +01:00
Eelco Dolstra b632153ebd nix-shell: Use shell.nix as the default expression if it exists 2014-02-10 10:25:13 +01:00
Eelco Dolstra 2f9bb5c7e7 Merge branch 'make' 2014-02-01 16:41:52 +01:00
Eelco Dolstra 6ef32bddc1 Fix "make dist" 2014-02-01 14:38:12 +01:00
Eelco Dolstra 0c6d62cf27 Remove Automakefiles 2014-02-01 13:54:38 +01:00
Eelco Dolstra 16e7d69209 Update Makefile variable names 2014-02-01 13:54:38 +01:00
Eelco Dolstra e0234dfddc Rename Makefile -> local.mk 2014-01-30 12:11:06 +01:00
Eelco Dolstra c5ba081333 nix-shell: Add --impure flag
This is currently the default, but I might change that to --pure in
the future.
2014-01-23 13:34:04 +01:00
Eelco Dolstra 79dee4283d nix-shell: Preserve the TZ variable of the user 2014-01-23 13:34:04 +01:00
Eelco Dolstra 7fdee6e136 nix-build: Refactor 2014-01-23 13:34:04 +01:00
Eelco Dolstra 81628a6ccc Merge branch 'master' into make
Conflicts:
	src/libexpr/eval.cc
2014-01-21 15:30:01 +01:00
Eelco Dolstra bf0ad8aabc nix-profile.sh: Add the Nixpkgs channel to $NIX_PATH 2014-01-15 14:34:49 +01:00
Eelco Dolstra f1357059a4 nix-shell: Don't set NIX_INDENT_MAKE
It generally is not useful in interactive environments (and messes up
some non-ANSI-compliant terminals).
2014-01-13 13:46:44 +01:00
Eelco Dolstra ea59f39326 nix-shell: Set $IN_NIX_SHELL before evaluating 2014-01-13 13:43:31 +01:00
Eelco Dolstra 0fdf4da0e9 Support cryptographically signed binary caches
NAR info files in binary caches can now have a cryptographic signature
that Nix will verify before using the corresponding NAR file.

To create a private/public key pair for signing and verifying a binary
cache, do:

  $ openssl genrsa -out ./cache-key.sec 2048
  $ openssl rsa -in ./cache-key.sec -pubout > ./cache-key.pub

You should also come up with a symbolic name for the key, such as
"cache.example.org-1".  This will be used by clients to look up the
public key.  (It's a good idea to number keys, in case you ever need
to revoke/replace one.)

To create a binary cache signed with the private key:

  $ nix-push --dest /path/to/binary-cache --key ./cache-key.sec --key-name cache.example.org-1

The public key (cache-key.pub) should be distributed to the clients.
They should have a nix.conf should contain something like:

  signed-binary-caches = *
  binary-cache-public-key-cache.example.org-1 = /path/to/cache-key.pub

If all works well, then if Nix fetches something from the signed
binary cache, you will see a message like:

  *** Downloading ‘http://cache.example.org/nar/7dppcj5sc1nda7l54rjc0g5l1hamj09j-subversion-1.7.11’ (signed by ‘cache.example.org-1’) to ‘/nix/store/7dppcj5sc1nda7l54rjc0g5l1hamj09j-subversion-1.7.11’...

On the other hand, if the signature is wrong, you get a message like

  NAR info file `http://cache.example.org/7dppcj5sc1nda7l54rjc0g5l1hamj09j.narinfo' has an invalid signature; ignoring

Signatures are implemented as a single line appended to the NAR info
file, which looks like this:

  Signature: 1;cache.example.org-1;HQ9Xzyanq9iV...muQ==

Thus the signature has 3 fields: a version (currently "1"), the ID of
key, and the base64-encoded signature of the SHA-256 hash of the
contents of the NAR info file up to but not including the Signature
line.

Issue #75.
2014-01-08 15:42:53 +01:00
Eelco Dolstra 405434e084 Revert "nix-shell: Set $IN_NIX_SHELL before evaluation"
This reverts commit 0c1198cf08.
2014-01-06 17:38:04 +01:00
Shea Levy 5ba5993470 nix-shell --pure: Don't clear IN_NIX_SHELL
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-30 16:59:24 +01:00
Eelco Dolstra b352fe2775 Revert "Scan /proc/<pid>/cmdline for GC roots"
This reverts commit 194e3374b8.
Checking the command line for GC roots means that

  $ nix-store --delete $path

will fail because $path is now a root because it's mentioned on the
command line.
2013-12-20 14:18:24 +01:00
Eelco Dolstra 194e3374b8 Scan /proc/<pid>/cmdline for GC roots 2013-12-20 13:31:31 +01:00
Eelco Dolstra 769f662165 nix-shell: Don't warn about the lack of a GC root 2013-12-20 12:19:10 +00:00
Eelco Dolstra 0c1198cf08 nix-shell: Set $IN_NIX_SHELL before evaluation
This has some hacky applications.
2013-12-20 13:11:41 +01:00
Eelco Dolstra 65a6452240 nix-shell: Handle --option correctly
Fixes #181.
2013-12-20 13:10:14 +01:00
Eelco Dolstra 7b0d8fb23d nix-shell --pure: Keep $TERM 2013-12-17 18:16:04 +01:00
Eelco Dolstra f3cf0436b5 Install bsdiff and bspatch in $(libexecdir)/nix 2013-12-02 20:21:31 +00:00
Eelco Dolstra 0202ce6b94 Add support for ‘make installcheck’ 2013-11-25 18:47:03 +01:00
Eelco Dolstra 7ce0e05ad8 Rename Makefile.new -> Makefile 2013-11-25 15:25:13 +00:00
Eelco Dolstra 962551a071 Add a Makefile for the scripts directory 2013-11-25 15:17:16 +00:00
Eelco Dolstra 2d9bb56e55 Fix segfault on Darwin
Ever since SQLite in Nixpkgs was updated to 3.8.0.2, Nix has randomly
segfaulted on Darwin:

  http://hydra.nixos.org/build/6175515
  http://hydra.nixos.org/build/6611038

It turns out that this is because the binary cache substituter somehow
ends up loading two versions of SQLite: the one in Nixpkgs and the
other from /usr/lib/libsqlite3.dylib.  It's not exactly clear why the
latter is loaded, but it appears to be because WWW::Curl indirectly loads
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation,
which in turn seems to load /usr/lib/libsqlite3.dylib.  This leads to
a segfault when Perl exits:

  #0  0x00000001010375f4 in sqlite3_finalize ()
  #1  0x000000010125806e in sqlite_st_destroy ()
  #2  0x000000010124bc30 in XS_DBD__SQLite__st_DESTROY ()
  #3  0x00000001001c8155 in XS_DBI_dispatch ()
  ...
  #14 0x0000000100023224 in perl_destruct ()
  #15 0x0000000100000d6a in main ()
  ...

The workaround is to explicitly load DBD::SQLite before WWW::Curl.
2013-10-24 19:15:52 +02:00
Eelco Dolstra 4f4a14453a Don't set $PS1 in non-interactive shells
Shouldn't really matter, but you never know.
2013-10-18 14:51:25 +02:00
Shea Levy 4ea034a5c5 nix-shell: Play nicely with non-interactive shells
nix-shell with the --command flag might be used non-interactively, but
if bash starts non-interactively (i.e. with stdin or stderr not a
terminal), it won't source the script given in --rcfile. However, in
that case it *will* source the script found in $BASH_ENV, so we can use
that instead.

Also, don't source ~/.bashrc in a non-interactive shell (detectable by
checking the PS1 env var)

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-18 14:34:46 +02:00
Eelco Dolstra c1994fecf9 nix-shell: Fix bash completion
Nixpkgs's stdenv setup script sets the "nullglob" option, but doing so
breaks Bash completion on NixOS (when ‘programs.bash.enableCompletion’
is set) and on Ubuntu.  So clear that flag afterwards.  Of course,
this may break stdenv functions in subtle ways...
2013-10-14 15:28:43 +02:00
Eelco Dolstra faaae44f2e build-remote.pl: Don't use substituters on the remote
It's kinda pointless to check substituters on the remote side, since
we just checked them locally.
2013-09-18 14:04:03 +02:00
Eelco Dolstra 5904262640 nix-shell: Support a .drv as argument
Fixes #161.
2013-09-06 14:58:05 +02:00
Eelco Dolstra 511455965e nix-shell: Don't disable Automake dependency tracking
Nixpkgs' stdenv disables dependency tracking by default.  That makes
sense for one-time builds, but in an interactive environment we expect
repeated "make" invocations to do the right thing.
2013-07-31 13:17:50 +02:00
Eelco Dolstra 15e5ac8039 nix-shell: Set $IN_NIX_SHELL
This allows scripts to distinguish between a real build and a Nix
shell.
2013-07-19 14:32:05 +02:00
Eelco Dolstra 48858ad5ca nix-shell: Set some environment variables also set by build.cc
Setting $NIX_STORE causes the purity checks in gcc/ld-wrapper to kick
in, so that's why we unset $NIX_ENFORCE_PURITY.
2013-07-19 14:06:58 +02:00
Eelco Dolstra 2bc5de8635 Rename ‘nix-build --run-env’ to ‘nix-shell’ 2013-07-19 12:02:44 +02:00
Eelco Dolstra dc5f2e7da6 nix-build --run-env: Add a ‘--pure’ flag
This causes the environment to be (almost) cleared, thus giving a
shell that more closely resembled the actual Nix derivation.
2013-07-19 11:23:32 +02:00
Eelco Dolstra a4921b8ceb Revert "build-remote.pl: Enforce timeouts locally"
This reverts commit 69b8f9980f.

The timeout should be enforced remotely.  Otherwise, if the garbage
collector is running either locally or remotely, if will block the
build or closure copying for some time.  If the garbage collector
takes too long, the build may time out, which is not what we want.
Also, on heavily loaded systems, copying large paths to and from the
remote machine can take a long time, also potentially resulting in a
timeout.
2013-07-18 12:52:29 +02:00
Eelco Dolstra 6fcc290619 build-remote.pl: Move "building ..." message to a better place 2013-07-11 17:16:45 +02:00
Eelco Dolstra ed09d0c0a4 nix-build --run-env: Always use Bash
Fixes #113.
Fixes #131.
2013-07-11 14:40:29 +02:00
Eelco Dolstra 656390062a nix-build --run-env: Source $stdenv/setup in the interactive shell
This ensures that not just environment variables are set, but also
shell functions such as unpackPhase, configurePhase and so on.
2013-07-11 14:32:22 +02:00
Eelco Dolstra 5116214343 Add support for uncompressed NARs in binary caches
Issue NixOS/hydra#102.
2013-07-01 21:03:14 +02:00
Eelco Dolstra 7986711632 copy-from-other-stores.pl: Respect $NIX_BIN_DIR 2013-07-01 13:30:28 +02:00
Eelco Dolstra 1917d750a0 copy-from-other-stores.pl: Report downloaded size as 0 2013-07-01 13:29:59 +02:00
Eelco Dolstra 22144afa8d Don't keep "disabled" substituters running
For instance, it's pointless to keep copy-from-other-stores running if
there are no other stores, or download-using-manifests if there are no
manifests.  This also speeds things up because we don't send queries
to those substituters.
2013-06-20 11:55:15 +02:00
Eelco Dolstra 5c06e5297d download-from-binary-cache.pl: Respect $NIX_CONNECT_TIMEOUT 2013-06-07 16:25:12 +02:00
Eelco Dolstra 24a356bf71 Replace $NIX_DEBUG_SUBST with an option ‘debug-subst’
Thus passing ‘--option debug-subst 1’ allows daemon users to turn on
debug info and see what the substituter is doing.
2013-06-07 15:36:36 +02:00
Eelco Dolstra 24e063efdc download-from-binary-cache.pl: Show if we're waiting for a URL
Previously, if a binary cache is hanging/unreachable/slow,
download-from-binary-cache.pl would also hang without any indication
to the user.  Now, if fetching a URL takes more than 5 seconds, it
will print a message to that effect.
2013-06-07 15:33:44 +02:00
Eelco Dolstra 75e12b8e66 download-from-binary-cache.pl: Fix race condition
Fixes the error "DBD::SQLite::db do failed: column url is not unique".
2013-06-05 16:25:07 +02:00
Eelco Dolstra f0576d6775 Update the default binary cache URL to cache.nixos.org 2013-06-05 13:36:43 +02:00
Eelco Dolstra ff08306746 download-from-binary-cache.pl: Treat a 403 error as a 404
Amazon S3 returns HTTP status code 403 if a file doesn't exist and the
user has no permission to list the contents of the bucket.  So treat
it as 404 (meaning it's cached in the NARExistence table).
2013-06-04 15:20:37 +02:00
Eelco Dolstra bc2e43f3c8 build-remote.pl: Remove meaningless signing when importing the output paths
The "$UID != 0" makes no sense: if the local side has write access to
the Nix store (which is always the case) then it doesn't matter if
we're root - we can import unsigned paths either way.
2013-05-29 18:18:59 +02:00
Eelco Dolstra 107505e13a build-remote.pl: Copy all outputs in one operation 2013-05-23 14:39:58 -04:00
Eelco Dolstra 72d8209548 build-remote.pl: Indicate if remote machine is refusing builds
Fixes #120.
2013-05-23 14:04:46 -04:00
Eelco Dolstra a4cb62ac25 download-from-binary-cache.pl: Get rid of an uninitialized value warning
Reported by Pablo Costa.
2013-05-15 15:47:05 +02:00
Eelco Dolstra 3a0cc43ac8 build-remote.pl: Properly close the SSH connection between attempts 2013-05-10 02:38:50 +02:00
Eelco Dolstra be0b9dda31 build-remote.pl: Pass /dev/null as SSH's stdin
Otherwise it will set the parent's stdin to non-blocking mode, causing
the subsequent read of the set of inputs/outputs to fail randomly.
That's insane.
2013-05-10 02:32:13 +02:00
Eelco Dolstra 78206f06ec build-remote.pl: Allow a machine to refuse a build
Before selecting a machine, build-remote.pl will try to run the
command "nix-builds-inhibited" on the machine.  If this command exists
and returns a 0 exit code, then the machine won't be used.  It's up to
the user to provide this command, but it would typically be a script
that checks whether there is enough disk space and whether the load is
not too high.
2013-05-10 01:09:46 +02:00
Eelco Dolstra 7a03cbf09d build-remote.pl: Create one process fewer on the remote side 2013-05-09 17:30:07 +02:00
Eelco Dolstra 69b8f9980f build-remote.pl: Enforce timeouts locally
Don't pass --timeout / --max-silent-time to the remote builder.
Instead, let the local Nix process terminate the build if it exceeds a
timeout.  The remote builder will be killed as a side-effect.  This
gives better error reporting (since the timeout message from the
remote side wasn't properly propagated) and handles non-Nix problems
like SSH hangs.
2013-05-09 17:17:17 +02:00
Eelco Dolstra ea019e9a26 Add option ‘extra-binary-caches’
This allows providing additional binary caches, useful in scripts like
Hydra's build reproduction scripts, in particular because untrusted
caches are ignored.
2013-05-07 15:37:28 +02:00
Eelco Dolstra c51b6a893c nix-copy-closure: Show a proper error message if no host name is given 2013-05-03 11:12:11 +02:00
Lluís Batlle i Rossell e6c44d166a Fixing the pv position regarding compression
Problem noticed by niksnut.
2013-05-03 11:08:51 +02:00
Lluís Batlle i Rossell 7391533ea5 Fixing the pv reference; I didn't mean to change it 2013-05-03 11:08:51 +02:00
Lluís Batlle i Rossell 5cc2fc46ec Adding ETA support to the --show-progress in nix-copy-closure
Based on https://github.com/NixOS/nix/pull/6 from shlevy
2013-05-03 11:08:51 +02:00
Eelco Dolstra 4ddd077bfa find-runtime-roots.pl: Don't hardcode /nix/store 2013-04-26 12:15:54 +02:00
Eelco Dolstra 00f698eb8b find-runtime-roots.pl: Search process environments for roots
For instance, this prevents paths from being deleted that are in use
by a "nix-build --run-env" session.
2013-04-26 12:06:39 +02:00
Eelco Dolstra 938092a213 find-runtime-roots.pl: Use Nix::Utils::readFile 2013-04-26 11:44:19 +02:00
Eelco Dolstra 6955d41f2b nix-build: Respect --timeout 2013-04-23 17:16:01 +02:00
Eelco Dolstra 08d96ffad0 Fix --fallback with the binary cache substituter
Reported by Peter Simons.
2013-04-23 12:45:01 +02:00
Eelco Dolstra c642441beb Test NAR info caching 2013-04-23 12:43:28 +02:00
Eelco Dolstra bdd4646338 Revert "Prevent config.h from being clobbered"
This reverts commit 28bba8c44f.
2013-03-08 01:24:59 +01:00
Eelco Dolstra 28bba8c44f Prevent config.h from being clobbered 2013-03-07 23:55:55 +01:00
Eelco Dolstra dadf7a5b46 build-remote: Use the --quiet flag
‘--option verbosity 0’ doesn't actually do anything.
2013-02-19 16:56:31 +01:00
Eelco Dolstra 96fbbbde55 build-remote.pl: Don't keep build logs on the build slave 2013-01-21 22:26:25 +01:00
Eelco Dolstra 0a4e90395c Urgggh
http://hydra.nixos.org/build/3661100
2013-01-02 23:52:15 +01:00
Eelco Dolstra aebea2e489 Reinstate the http://nixos.org/binary-cache default for the binary-caches setting 2013-01-02 22:16:37 +01:00
Eelco Dolstra 68dcbb187e download-from-binary-cache: Use the channels of the calling user rather than root
This should make live easier for single-user (non-daemon)
installations.  Note that when the daemon is used, the "calling user"
is root so we're not using any untrusted caches.
2012-12-21 15:00:07 +01:00
Eelco Dolstra 8cc19ed089 Set a long SQLite timeout in the binary cache substituter 2012-12-06 11:43:34 +01:00
Eelco Dolstra 566afa1e9c Support xz compression of nixexprs.tar in channels 2012-12-05 11:02:44 +01:00
Eelco Dolstra 08964d7328 Undo accidental debug change 2012-11-26 17:57:14 +01:00
Eelco Dolstra 46a369ad95 Make "nix-build -A <derivation>.<output>" do the right thing
For example, given a derivation with outputs "out", "man" and "bin":

  $ nix-build -A pkg

produces ./result pointing to the "out" output;

  $ nix-build -A pkg.man

produces ./result-man pointing to the "man" output;

  $ nix-build -A pkg.all

produces ./result, ./result-man and ./result-bin;

  $ nix-build -A pkg.all -A pkg2

produces ./result, ./result-man, ./result-bin and ./result-2.
2012-11-26 15:39:10 +01:00
Eelco Dolstra a3d6585c5a nix-copy-closure: Add flag ‘--use-substitutes’ 2012-11-23 16:20:16 +01:00
Eelco Dolstra a9a8baaccb Use a shorter Nixpkgs channel URL 2012-11-09 13:33:35 +01:00
Eelco Dolstra e28b683324 download-from-binary-cache: Try next cache if downloading a NAR fails 2012-11-06 17:55:53 +01:00
Eelco Dolstra bbc107ef1e Process binary caches in order of priority
Binary caches can now specify a priority in their nix-cache-info file.
The binary cache substituter checks caches in order of priority.  This
is to ensure that fast, static caches like nixos.org/binary-cache are
processed before slow, dynamic caches like hydra.nixos.org.
2012-11-06 17:45:20 +01:00
Eelco Dolstra 82951e5582 nix-push: Handle pushing a symlink 2012-11-06 13:53:53 +01:00
Eelco Dolstra 620e92e880 Add an option ‘use-binary-caches’
This allows disabling the use of binary caches, e.g.

  $ nix-build ... --option use-binary-caches false

Note that

  $ nix-build ... --option binary-caches ''

does not disable all binary caches, since the caches defined by
channels will still be used.
2012-11-06 13:44:59 +01:00
Eelco Dolstra df27db712d Fix "Not an ARRAY reference" error
It's a mystery why this error is not triggered in the build farm
(e.g. http://hydra.nixos.org/build/3265602).  Ah well.
2012-11-06 13:43:46 +01:00
Eelco Dolstra a28b4445a4 nix-prefetch-url: Improve option handling 2012-10-23 18:04:11 +02:00
Eelco Dolstra dde6486eab nix-push: Add a flag ‘--manifest-path’ to write the manifest to another directory 2012-10-18 10:36:16 -04:00
Eelco Dolstra c8daeba303 nix-push: Add ‘--link’ flag
If ‘--link’ is given, nix-push will create hard links to the NAR files
in the store, rather than copying them.  This is faster and requires
less disk space.  However, it doesn't work if the store is on a
different file system.
2012-10-17 17:01:23 -04:00
Eelco Dolstra 167e36a5c3 nix-push: Only generate and copy a NAR if it doesn't already exist
This prevents unnecessary and slow rebuilds of NARs that already exist
in the binary cache.
2012-10-17 16:58:05 -04:00
Eelco Dolstra 600daf972f download-from-binary-cache: Remove duplicate entries in trustedURLs 2012-10-11 14:21:22 -04:00
Eelco Dolstra a562d544d8 When ‘--help’ is given, just run ‘man’ to show the manual page
I.e. do what git does.  I'm too lazy to keep the builtin help text up
to date :-)

Also add ‘--help’ to various commands that lacked it
(e.g. nix-collect-garbage).
2012-10-03 16:40:09 -04:00
Eelco Dolstra 2e90a5a2a7 nix-build: Support ‘--repair’ flag 2012-10-03 15:14:02 -04:00
Eelco Dolstra 9958bd6992 Add operation ‘nix-store --repair-path’
This operation allows fixing corrupted or accidentally deleted store
paths by redownloading them using substituters, if available.

Since the corrupted path cannot be replaced atomically, there is a
very small time window (one system call) during which neither the old
(corrupted) nor the new (repaired) contents are available.  So
repairing should be used with some care on critical packages like
Glibc.
2012-10-02 14:08:59 -04:00
Eelco Dolstra c1f91570b3 Style fix 2012-09-21 15:02:33 -04:00
Eelco Dolstra 20582e9ae3 Support xz compression in the download-using-manifests substituter 2012-09-19 17:33:42 -04:00
Eelco Dolstra 983220bcd4 nix-collect-garbage: Support --dry-run 2012-09-13 18:09:20 -04:00
Eelco Dolstra aac14222f5 nix-profile.sh: Revert to single-user version
Commit 6a214f3e06 copied most of the Nix
shell initialisation code from NixOS to nix-profile.sh; however, that
code assumes a multi-user install and is Linux-specific (e.g. it calls
the "stat" command).  So go back to the simple single-user version.

Fixes #49.
2012-09-13 17:48:19 -04:00
Eelco Dolstra 47ae3ce2ca Put a time-to-live on negative binary cache lookups
Negative lookups are purged from the DB after a day, at most once per
day.  However, for non-"have" lookups (e.g. all except "nix-env
-qas"), negative lookups are ignored after one hour.  This is to
ensure that you don't have to wait a day for an operation like
"nix-env -i" to start using new binaries in the cache.

Should probably make this configurable.
2012-09-13 16:39:16 -04:00
Eelco Dolstra e03a8a1c92 Doh 2012-09-13 14:58:56 -04:00
Eelco Dolstra c845c0ccad nix-channel: Add option to force fetching of manifests 2012-09-13 12:11:40 -04:00
Eelco Dolstra b14717ab90 Delete manifests in "nix-channel --remove" or when a binary cache is available 2012-09-13 11:35:46 -04:00
Eelco Dolstra 479e9172b3 Build hook: Pass the timeout to the remote builder
Note that this will only work if the client has a very recent Nix
version (post 15e1b2c223), otherwise the
--option flag will just be ignored.

Fixes #50.
2012-09-12 12:18:14 -04:00
Eelco Dolstra 6762424e24 Compatibility fix for WWW::Curl < 4.14
Older versions of WWW::Curl don't support scalar references for
CURLOPT_WRITEDATA directly.

http://hydra.nixos.org/build/3017188
2012-09-12 10:41:17 -04:00
Eelco Dolstra a2785b7391 Support building a derivation if some outputs are already valid
This handles the chroot and build hook cases, which are easy.
Supporting the non-chroot-build case will require more work (hash
rewriting!).

Issue #21.
2012-09-11 16:59:59 -04:00
Eelco Dolstra 9e2fc6951c Disable the binary cache substituter by default for now 2012-08-27 14:20:14 -04:00
Eelco Dolstra 8b8fe6139e Drop dependency on List::MoreUtils 2012-08-27 11:28:34 -04:00
Eelco Dolstra e94806d030 Merge branch 'master' into no-manifests 2012-08-27 11:09:07 -04:00
Eelco Dolstra a9e6752bbd Include the output name in the GC root link
Output names are now appended to resulting GC symlinks, e.g. by
nix-build.  For backwards compatibility, if the output is named "out",
nothing is appended.  E.g. doing "nix-build -A foo" on a derivation
that produces outputs "out", "bin" and "dev" will produce symlinks
"./result", "./result-bin" and "./result-dev", respectively.
2012-08-24 16:58:11 -04:00
Eelco Dolstra 5170c5691a nix-channel: Use binary caches advertised by channels
Channels can now advertise a binary cache by creating a file
<channel-url>/binary-cache-url.  The channel unpacker puts these in
its "binary-caches" subdirectory.  Thus, the URLS of the binary caches
for the channels added by root appear in
/nix/var/nix/profiles/per-user/eelco/channels/binary-caches/*.  The
binary cache substituter reads these and adds them to the list of
binary caches.
2012-08-01 17:56:11 -04:00
Eelco Dolstra 8a25d787d7 download-from-binary-cache: Remove duplicate URLs 2012-08-01 11:33:15 -04:00
Eelco Dolstra c770a2422a Report substituter errors to clients of the Nix daemon 2012-08-01 11:19:24 -04:00
Eelco Dolstra 4d1b64f118 Allow daemon users to override ‘binary-caches’
For security reasons, daemon users can only specify caches that appear
in the ‘binary-caches’ and ‘trusted-binary-caches’ options in
nix.conf.
2012-07-31 18:56:22 -04:00
Eelco Dolstra 9cd63d2244 Do some validation of URLs 2012-07-30 17:09:36 -04:00
Eelco Dolstra 9de6d10d11 Get rid of $NIX_BINARY_CACHES
You can use ‘--option binary-caches URLs’ instead.
2012-07-30 16:39:31 -04:00
Eelco Dolstra ab42bf1dab nix-install-package: Support binary caches
The .nixpkg file format is extended to optionally include the URL of a
binary cache, which will be used in preference to the manifest URL
(which can be set to a non-existent value).
2012-07-30 16:11:02 -04:00
Eelco Dolstra 66a3ac6a56 Allow a binary cache to declare that it doesn't support "nix-env -qas"
Querying all substitutable paths via "nix-env -qas" is potentially
hard on a server, since it involves sending thousands of HEAD
requests.  So a binary cache must now have a meta-info file named
"nix-cache-info" that specifies whether the server wants this.  It
also specifies the store prefix so that we don't send useless queries
to a binary cache for a different store prefix.
2012-07-27 18:16:05 -04:00
Eelco Dolstra 73acb8b836 Let build.cc verify the expected hash of a substituter's output
Since SubstitutionGoal::finished() in build.cc computes the hash
anyway, we can prevent the inefficiency of computing the hash twice by
letting the substituter tell Nix about the expected hash, which can
then verify it.
2012-07-27 12:16:02 -04:00
Eelco Dolstra 3a8f841612 download-using-manifests: Don't use nix-prefetch-url
Instead call curl directly and pipe it into ‘nix-store --restore’.
This saves I/O and prevents creating garbage in the Nix store.
2012-07-27 10:47:36 -04:00
Eelco Dolstra b4ea83249b Remove trailing whitespace / tabs 2012-07-27 09:59:18 -04:00
Eelco Dolstra 7f8e805c8e download-from-binary-cache: Only use the default cache for /nix/store 2012-07-27 09:57:42 -04:00
Eelco Dolstra 67c6f3eded nix-push: Support generating a manifest again
This makes all the tests succeed.  Woohoo!
2012-07-26 18:28:12 -04:00
Eelco Dolstra e3ce54ccee nix-push: Remove the upload feature 2012-07-26 17:13:38 -04:00
Eelco Dolstra 7892ad15ab download-from-binary-cache: Support file://
The file:// URI schema requires checking for errors in a more general
way.  Also, don't cache file:// lookups.
2012-07-26 17:11:11 -04:00
Eelco Dolstra 8c79100839 Merge branch 'master' into no-manifests 2012-07-26 15:14:33 -04:00
Eelco Dolstra 2605f4f4e6 nix-profile.sh: Don't set NIX_REMOTE on single user installations
Commit 6a214f3e06 reused the NixOS
environment initialisation for nix-profile.sh, but this is
inappropriate on systems that don't have multi-user support enabled.
2012-07-25 17:06:09 -04:00
Eelco Dolstra 477b0fbeca Subscribe to the Nixpkgs rather than NixOS channel 2012-07-25 16:56:56 -04:00
Eelco Dolstra ed59bf7a18 nix-build: Support the ‘-’ argument to build an expression from stdin 2012-07-23 17:11:12 -04:00
Eelco Dolstra aa115e22df download-from-binary-cache: Print correct URL 2012-07-18 11:01:17 -04:00
Eelco Dolstra a6f348599c Print some debug output 2012-07-17 16:19:40 -04:00
Eelco Dolstra f2bdc87595 Update the other substituters 2012-07-11 18:52:09 -04:00
Eelco Dolstra 15c15da482 Add some missing --version switches 2012-07-11 18:07:41 -04:00
Eelco Dolstra d287b62b64 Set the User-Agent header to "Nix/<version>" 2012-07-11 18:05:30 -04:00
Eelco Dolstra b74d92755d download-from-binary-cache: Use HEAD requests if possible
In "nix-env -qas", we don't need the substitute info, we just need to
know if it exists.  This can be done using a HTTP HEAD request, which
saves bandwidth.

Note however that curl currently has a bug that prevents it from
reusing HTTP connections if HEAD requests return a 404:

https://sourceforge.net/tracker/?func=detail&aid=3542731&group_id=976&atid=100976

Without the patch attached to the issue, using HEAD is actually quite
a bit slower than GET.
2012-07-11 17:53:20 -04:00
Eelco Dolstra 5ee8944155 Cleanup 2012-07-11 10:13:16 -04:00
Eelco Dolstra 099125435f download-from-binary-cache: add nix.conf options 2012-07-09 10:57:28 -04:00
Eelco Dolstra 400e556b34 Cleanup 2012-07-08 18:39:07 -04:00
Eelco Dolstra 11800e6198 download-from-binary-cache: parallelise fetching of NAR info files
Getting substitute information using the binary cache substituter has
non-trivial latency overhead.  A package or NixOS system configuration
can have hundreds of dependencies, and in the worst case (when the
local info cache is empty) we have to do a separate HTTP request for
each of these.  If the ping time to the server is t, getting N info
files will take tN seconds; e.g., with a ping time of 0.1s to
nixos.org, sequentially downloading 1000 info files (a typical NixOS
config) will take at least 100 seconds.

To fix this problem, the binary cache substituter can now perform
requests in parallel.  This required changing the substituter
interface to support a function querySubstitutablePathInfos() that
queries multiple paths at the same time, and rewriting queryMissing()
to take advantage of parallelism.  (Due to local caching,
parallelising queryMissing() is sufficient for most use cases, since
it's almost always called before building a derivation and thus fills
the local info cache.)

For example, parallelism speeds up querying all 1056 paths in a
particular NixOS system configuration from 116s to 2.6s.  It works so
well because the eccentricity of the top-level derivation in the
dependency graph is only 9.  So we only need 10 round-trips (when
using an unlimited number of parallel connections) to get everything.

Currently we do a maximum of 150 parallel connections to the server.
Thus it's important that the binary cache server (e.g. nixos.org) has
a high connection limit.  Alternatively we could use HTTP pipelining,
but WWW::Curl doesn't support it and libcurl has a hard-coded limit of
5 requests per pipeline.
2012-07-06 19:08:20 -04:00
Eelco Dolstra cd94665f38 download-from-binary-cache: use WWW::Curl
Using WWW::Curl rather than running an external curl process for every
NAR info file halves the time it takes to get info thanks to libcurl's
support for persistent HTTP connections.  (We save a roundtrip per
file.)  But the real gain will come from using parallel and/or
pipelined requests.
2012-07-06 00:30:40 -04:00
Eelco Dolstra ae60643c15 download-from-binary-cache: do negative NAR info caching
I.e. if a NAR info file does *not* exist, we record it in the cache DB
so that we don't retry it later.
2012-07-03 18:54:46 -04:00
Eelco Dolstra 89380c03e9 download-from-binary-cache: in queries, preferred cached info 2012-07-03 18:35:39 -04:00
Eelco Dolstra 2a8e5c8b11 download-from-binary-cache: strip trailing / from URLs 2012-07-03 17:47:01 -04:00
Eelco Dolstra d694c599e2 download-from-binary-cache: cache binary cache info in a SQLite DB 2012-07-03 17:29:33 -04:00
Eelco Dolstra 8319b1ab9f download-from-binary-cache: Verify NAR hashes 2012-07-02 18:53:04 -04:00
Eelco Dolstra cf49472d60 nix-push: Always generate base-32 hashes 2012-07-02 18:05:57 -04:00
Eelco Dolstra 6ec7460af1 Binary caches: use a better key
Use the hash part of the store path as a key rather than a hash of the
store path.  This is enough to get the desired privacy property.
2012-07-02 12:42:58 -04:00
Eelco Dolstra 440adfbdd1 Add an environment variable $NIX_BINARY_CACHES specifying URLs of binary caches 2012-07-01 21:55:36 -04:00
Eelco Dolstra f4190c38ba Allow both bzip2 and xz compression 2012-07-01 18:46:38 -04:00
Eelco Dolstra 000132cbd1 nix-push: Don't pollute the current directory with result symlink 2012-06-29 18:30:28 -04:00
Eelco Dolstra 37f7098464 First attempt at the manifest-less substituter 2012-06-29 18:28:52 -04:00
Eelco Dolstra 91b8814f0d Doh 2012-06-29 17:16:00 -04:00
Eelco Dolstra 4911a10a4e Use XZ compression in binary caches
XZ compresses significantly better than bzip2.  Here are the
compression ratios and execution times (using 4 cores in parallel) on
my /var/run/current-system (3.1 GiB):

  bzip2: total compressed size 849.56 MiB, 30.8% [2m08]
  xz -6: total compressed size 641.84 MiB, 23.4% [6m53]
  xz -7: total compressed size 621.82 MiB, 22.6% [7m19]
  xz -8: total compressed size 599.33 MiB, 21.8% [7m18]
  xz -9: total compressed size 588.18 MiB, 21.4% [7m40]

Note that compression takes much longer.  More importantly, however,
decompression is much faster:

  bzip2: 1m47.274s
  xz -6: 0m55.446s
  xz -7: 0m54.119s
  xz -8: 0m52.388s
  xz -9: 0m51.842s

The only downside to using -9 is that decompression takes a fair
amount (~65 MB) of memory.
2012-06-29 15:24:52 -04:00
Eelco Dolstra 49cd7387ad nix-push: create a manifest-less binary cache
Manifests are a huge pain, since users need to run nix-pull directly
or indirectly to obtain them.  They tend to be large and lag behind
the available binaries; also, the downloaded manifests in
/nix/var/nix/manifest need to be in sync with the Nixpkgs sources.  So
we want to get rid of them.

The idea of manifest-free operation works as follows.  Nix is
configured with a set of URIs of binary caches, e.g.

  http://nixos.org/binary-cache

Whenever Nix needs a store path X, it checks each binary cache for the
existence of a file <CACHE-URI>/<SHA-256 hash of X>.narinfo, e.g.

  http://nixos.org/binary-cache/bi1gh9...ia17.narinfo

The .narinfo file contains the necessary information about the store
path that was formerly kept in the manifest, i.e., (relative) URI of
the compressed NAR, references, size, hash, etc.  For example:

  StorePath: /nix/store/xqp4l88cr9bxv01jinkz861mnc9p7qfi-neon-0.29.6
  URL: 1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2
  CompressedHash: sha256:1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd
  CompressedSize: 202542
  NarHash: sha256:1af26536781e6134ab84201b33408759fc59b36cc5530f57c0663f67b588e15f
  NarSize: 700440
  References: 043zrsanirjh8nbc5vqpjn93hhrf107f-bash-4.2-p24 cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 ...
  Deriver: 4idz1bgi58h3pazxr3akrw4fsr6zrf3r-neon-0.29.6.drv
  System: x86_64-linux

Nix then knows that it needs to download

  http://nixos.org/binary-cache/1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2

to substitute the store path.

Note that the store directory is omitted from the References and
Deriver fields to save space, and that the URL can be relative to the
binary cache prefix.

This patch just makes nix-push create binary caches in this format.
The next step is to make a substituter that supports them.
2012-06-28 17:19:32 -04:00
Michel Alexandre Salim 6a214f3e06 Update nix profile: - incorporate NixOS's configuration so that nix is usable by normal users - install as a data file, not a program file 2012-05-31 08:59:36 -04:00
Eelco Dolstra 6814b1dfa1 Generate binary tarballs for installing Nix
For several platforms we don't currently have "native" Nix packages
(e.g. Mac OS X and FreeBSD).  This provides the next best thing: a
tarball containing the closure of Nix, plus a simple script
"nix-finish-install" that initialises the Nix database, registers the
paths in the closure as valid, and runs "nix-env -i /path/to/nix" to
initialise the user profile.

The tarball must be unpacked in the root directory.  It creates
/nix/store/... and /usr/bin/nix-finish-install.  Typical installation
is as follows:

  $ cd /
  $ tar xvf /path/to/nix-1.1pre1234_abcdef-x86_64-linux.tar.bz2
  $ nix-finish-install
  (if necessary add ~/.nix-profile/etc/profile.d/nix.sh to the shell
  login scripts)

After this, /usr/bin/nix-finish-install can be deleted, if desired.

The downside to the binary tarball is that it's pretty big (~55 MiB
for x86_64-linux).
2012-05-22 18:36:54 -04:00
Eelco Dolstra 591aab7e21 Remove $FONTCONFIG_FILE hack from nix-profile.sh
It's no longer needed because Nixpkgs' fontconfig uses
/etc/fonts/fonts.conf as a default, just like other distributions.
2012-05-22 14:00:08 -04:00
Eelco Dolstra 0c4c8f7a9d Remove obsolete files (moved to release tree) 2012-05-08 15:43:54 -04:00
Eelco Dolstra afa7b8a479 nix-channel --update: allow updating only the specified channels 2012-05-07 17:55:56 -04:00
Eelco Dolstra 147f10157f Now *really* prevent accumulation of old manifests 2012-05-07 17:23:26 -04:00
Eelco Dolstra e060c99447 Use mkpath instead of make_path
Perl <= 5.10 doesn't have make_path.  See
e.g. http://hydra.nixos.org/build/2493981.
2012-05-01 16:35:18 -04:00
Eelco Dolstra 59a26360c7 Support mandatory system features in the build hook
Mandatory features are features that MUST be present in a derivation's
requiredSystemFeatures attribute.  One application is performance
testing, where we have a dedicated machine to run performance tests
(and nothing else).  Then we would add the label "perf" to the
machine's mandatory features and to the performance testing
derivations.
2012-04-30 17:22:45 -04:00
Eelco Dolstra 6de5d53416 Fix a warning in the build hook about $progressViewer 2012-04-24 12:56:30 +02:00
Eelco Dolstra 259c6d3fbb Fix URL set by nix-channel 2012-04-18 13:40:18 +02:00
Eelco Dolstra 8cf1719e3e Hack to prevent accumulation of old manifests 2012-04-16 19:52:31 +02:00
Eelco Dolstra 0b5107c016 If the (redirected) channel URL contains a version number, use it 2012-04-14 21:05:28 +02:00
Eelco Dolstra 1d0bafb74d Follow redirects early to ensure consistency between the manifest and the tarball
Without this change, if the URL is a redirect that changes in between,
we can get a manifest that doesn't match the tarball.
2012-04-14 21:04:22 +02:00
Eelco Dolstra ef902274fd Remove unnecessary "system" argument 2012-04-14 18:48:11 +02:00
Eelco Dolstra e855c7e2c9 nix-channel improvements
"nix-channel --add" now accepts a second argument: the channel name.
This allows channels to have a nicer name than (say) nixpkgs_unstable.
If no name is given, it defaults to the last component of the URL
(with "-unstable" or "-stable" removed).

Also, channels are now stored in a profile
(/nix/var/nix/profiles/per-user/$USER/channels).  One advantage of
this is that it allows rollbacks (e.g. if "nix-channel --update" gives
an undesirable update).
2012-04-14 18:38:52 +02:00
Eelco Dolstra 969a14599d If "pv" is available at compile time, hard-code its path 2012-04-13 14:41:33 +02:00
Shea Levy 34a85c5405 nix-copy-closure: Move the progressViewer directly adjacent to the ssh call so that network progress is what's measured 2012-04-13 14:29:12 +02:00
Shea Levy 055e803851 Add the '--show-progress' flag to nix-copy-closure 2012-04-13 14:29:12 +02:00
Eelco Dolstra 1fb30e6d50 Handle the case where $SHELL is not set 2012-04-10 13:52:37 +02:00
Hydra mirror 3b859ead37 nix-generate-patches: Optionally write new patches to $NIX_ALL_PATCHES 2012-04-04 20:46:18 +00:00
Eelco Dolstra 0837348049 Follow redirects when checking for the existence of MANIFEST.bz2
If we don't follow redirects, we might think that MANIFEST.bz2 exists
just because the HEAD request succeeds on the redirector URI.
2012-04-04 15:41:35 +02:00
Eelco Dolstra b8fb0ce563 Update "nix-build --help" 2012-03-27 11:59:36 +02:00
Eelco Dolstra 5144abe5b6 Add an option "--exclude" to filter dependencies in "nix-build --run-env"
Sometimes when doing "nix-build --run-env" you don't want all
dependencies to be built.  For instance, if we want to do "--run-env"
on the "build" attribute in Hydra's release.nix (to get Hydra's build
environment), we don't want its "tarball" dependency to be built.  So
we can do:

  $ nix-build --run-env release.nix -A build --exclude 'hydra-tarball'

This will skip the dependency whose name matches the "hydra-tarball"
regular expression.  The "--exclude" option can be repeated any number
of times.
2012-03-27 11:40:47 +02:00
Eelco Dolstra 3e94ffffd6 Allow the command executed by "nix-build --run-env" to be overriden with --command 2012-03-27 11:16:43 +02:00
Eelco Dolstra 7f38087f35 Add a command "nix-build --run-env" to reproduce the environment of a derivation
This command builds or fetches all dependencies of the given
derivation, then starts a shell with the environment variables from
the derivation.  This shell also sources $stdenv/setup to initialise
the environment further.

The current directory is not changed.  Thus this is a convenient way
to reproduce a build environment in an existing working tree.

Existing environment variables are left untouched (unless the
derivation overrides them).  As a special hack, the original value of
$PATH is appended to the $PATH produced by $stdenv/setup.

Example session:

$ nix-build --run-env '<nixpkgs>' -A xterm
(the dependencies of xterm are built/fetched...)
$ tar xf $src
$ ./configure
$ make
$ emacs
(... hack source ...)
$ make
$ ./xterm
2012-03-19 04:14:21 +01:00
Eelco Dolstra d645633643 nix-copy-closure: don't print copied path on stdout
We're already printing progress on stderr, so printing them on stdout
afterwards is kind of useless.
2012-03-05 20:52:24 +01:00
Eelco Dolstra 2b4964f319 Restore progress indication during nix-copy-closure 2012-03-05 18:13:44 +01:00
Eelco Dolstra 56042a120a build-remote.pl: don't wait forever for the upload lock
In the build hook, don't wait forever to get the upload lock.  This
ensures progress if another process gets stuck while holding the
upload lock.
2012-03-05 17:58:09 +01:00
Eelco Dolstra ce86a9a0ae * nix-channel: fail if we don't have write permission to the manifests
directory.  Previously in this situation we did add the Nix
  expressions from the channel to allow installation from source, but
  this doesn't work for binary-only channels and leads to confusing
  error messages.
2012-02-02 12:25:56 +00:00
Eelco Dolstra 2a3f4110c5 * nix-build: put the temporary derivation symlink in a temporary
directory rather than the current directory.
* nix-build: --drv-link now implies --add-drv-link.
2012-01-13 23:35:07 +00:00
Eelco Dolstra d4e6b9f2d6 * Doh! 2012-01-05 21:07:12 +00:00
Eelco Dolstra a7366a764a * Add a -I flag to the Perl bindings to nix-build and some other
scripts.
* Include the version and architecture in the -I flag so that there is
  at least a chance that a Nix binary built for one Perl version will
  run on another version.
2012-01-05 20:33:46 +00:00
Eelco Dolstra 39d45a6b09 * Add a test for nix-channel.
* Refactor the nix-channel unpacker a bit.
2012-01-03 01:51:38 +00:00
Eelco Dolstra dadbb51d96 * Use Nix::Config. 2012-01-03 00:47:27 +00:00
Eelco Dolstra 48cea0d01e * Refactoring: Get rid of a few subdirectories in corepkgs/, and some
other simplifications.
* Use <nix/...> to locate the corepkgs.  This allows them to be
  overriden through $NIX_PATH.
* Use bash's pipefail option in the NAR builder so that we don't need
  to create a temporary file.
2012-01-03 00:16:29 +00:00
Eelco Dolstra 4d728bc3e6 * Security: make sure the lock files used by build-remote.pl are not
readable to other users.  Otherwise, any user can open the lock file
  for reading and lock it, thus DoSing the remote build mechanism.
2011-12-21 19:11:58 +00:00
Eelco Dolstra 69d6f0936a * Use a lock to ensure that only one build-remote instance can copy a
closure to a given machine at the same time.  This prevents the case
  where multiple instances try to copy the same missing store path to
  the target machine, which is very wasteful.
2011-12-21 18:59:25 +00:00
Eelco Dolstra a67b8ae224 * Typo. 2011-12-15 14:04:35 +00:00
Eelco Dolstra 3964d95abf * nix-prefetch-url: rewritten in Perl. 2011-12-02 12:09:50 +00:00
Eelco Dolstra 1749a7b0ae * download-using-manifests: use the Perl bindings. 2011-11-29 13:01:24 +00:00
Eelco Dolstra 216440b3ff * For consistency with "nix-store -q --hash", produce hashes in
base-32.  (This affects Hydra manifests.)
2011-11-29 12:32:55 +00:00
Eelco Dolstra 1df120cb05 * Get rid of the shell in ssh calls. 2011-11-29 10:51:22 +00:00
Eelco Dolstra d5ac78e0d6 * Add bzip2 and xz support to nix-copy-closure. 2011-11-23 15:29:58 +00:00
Eelco Dolstra 5bbd693cae * Add an API function exportPaths() that provides the functionality of
‘nix-store --export’.
* Add a Perl module that provides the functionality of
  ‘nix-copy-closure --to’.  This is used by build-remote.pl so it no
  longer needs to start a separate nix-copy-closure process.  Also, it
  uses the Perl API to do the export, so it doesn't need to start a
  separate nix-store process either.  As a result, nix-copy-closure
  and build-remote.pl should no longer fail on very large closures due
  to an "Argument list too long" error.  (Note that having very many
  dependencies in a single derivation can still fail because the
  environment can become too large.  Can't be helped though.)
2011-11-23 15:13:37 +00:00
Eelco Dolstra ab20af3e6f * build-remote.pl: drop a hard-coded reference to /nix/etc/nix. 2011-11-23 12:21:35 +00:00
Eelco Dolstra f8e609c341 * nix-pull: update the Nix manifest cache if necessary. Also, don't
read the manifest just to check the version and print the number of
  paths.  This makes nix-pull very fast for the cached cache (speeding
  up nixos-rebuild without the ‘--no-pull’ or ‘--fast’ options).
2011-11-16 16:41:48 +00:00
Eelco Dolstra d7d7910ba4 * Don't decompress the manifests in /nix/var/nix/manifest. This saves
disk space, and, since they're typically only decompressed once (to
  fill the manifest cache), doesn't make things slower.
2011-11-16 16:25:38 +00:00
Eelco Dolstra 00b41e46ed * Print a consistent message. 2011-10-19 21:34:13 +00:00
Eelco Dolstra f186a9141e * nix-copy-closure: support ‘--dry-run’ and ‘--include-outputs’. 2011-10-18 21:21:22 +00:00
Eelco Dolstra 6761757428 * Use the Store API bindings in nix-copy-closure. 2011-10-11 15:41:13 +00:00
Eelco Dolstra c362e4d718 * Move SSH.pm. 2011-10-11 11:45:36 +00:00
Eelco Dolstra 5090c34ee1 * Set the executable bit on scripts. 2011-10-10 22:40:17 +00:00
Eelco Dolstra cd6d02c366 2011-10-10 21:30:59 +00:00
Eelco Dolstra 6fcdbcac20 * Install NixManifest.pm, NixConfig.pm and GeneratePatches.pm under
the Nix:: namespace.
2011-10-10 21:11:08 +00:00
Eelco Dolstra 5bcdc7e351 * Update the cleanup script. 2011-08-27 16:59:32 +00:00
Eelco Dolstra a95ba4cdd9 * Use last_insert_id instead of sqlite_last_insert_rowid, which you're
not really supposed to use according to the DBD::SQLite docs, and
  fails on some systems (e.g. http://hydra.nixos.org/build/1246662).
2011-08-17 14:17:19 +00:00
Eelco Dolstra 1ecc97b6bd * Add a Nix expression search path feature. Paths between angle
brackets, e.g.

    import <nixpkgs/pkgs/lib>

  are resolved by looking them up relative to the elements listed in
  the search path.  This allows us to get rid of hacks like

    import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib"

  The search path can be specified through the ‘-I’ command-line flag
  and through the colon-separated ‘NIX_PATH’ environment variable,
  e.g.,

    $ nix-build -I /etc/nixos ...

  If a file is not found in the search path, an error message is
  lazily thrown.
2011-08-06 16:05:24 +00:00
Eelco Dolstra 0243eea4b9 * Create a symlink to /nix/var/nix/manifests in /nix/var/nix/gcroots
if it doesn't exist.
2011-07-20 11:47:00 +00:00
Eelco Dolstra e649f3168b * Fix concurrency issues in download-using-manifests' handling of the
SQLite manifest cache.  The DBI AutoCommit feature caused every
  process to have an active transaction at all times, which could
  indefinitely block processes wanting to update the manifest cache.

* Disable fsync() in the manifest cache because we don't need
  integrity (the cache can always be recreated if it gets corrupted).
2011-07-13 14:05:54 +00:00
Eelco Dolstra d94cb02bfe * nix-install-package: unset NIX_REMOTE because $NIX_MANIFESTS_DIR
doesn't work when building through the Nix daemon.  This also
  ensures an error message when the user doesn't have sufficient
  privileges to do nix-pull.
2011-04-19 10:52:35 +00:00
Eelco Dolstra 412914d004 * Read manifests directly into the database, rather than first reading
them into memory.  This brings memory use down to (more or less)
  O(1).  For instance, on my test case, the maximum resident size of
  download-using-manifests while filling the DB went from 142 MiB to
  11 MiB.
2011-04-11 13:16:54 +00:00
Eelco Dolstra 08c8971498 * Lock the database during updates. 2011-04-11 12:40:13 +00:00
Eelco Dolstra 3cbf680f5d * configure: detect whether DBD::SQLite is present. If necessary the
location to DBI and DBD::SQLite can be passed with --with-dbi and
  --with-dbd-sqlite.
2011-04-11 10:13:53 +00:00
Eelco Dolstra 83252b4ca9 * Subtle bug: if you import File::stat in one module, it affects other
modules as well.  So use symbolic field names everywhere (which is
  nicer anyway).
2011-04-11 08:21:30 +00:00
Eelco Dolstra b1882c3ef7 * Create $manifestDir if it doesn't exist. 2011-04-11 08:06:14 +00:00
Eelco Dolstra 5591fcc529 * Cache the manifests in /nix/var/nix/manifests in a SQLite database.
This significantly speeds up the download-using-manifests
  substituter, especially if manifests are very large.  For instance,
  one "nix-build -A geeqie" operation that updated four packages using
  binary patches went from 18.5s to 1.6s.  It also significantly
  reduces memory use.

  The cache is kept in /nix/var/nix/manifests/cache.sqlite.  It's
  updated automatically when manifests are added to or removed from
  /nix/var/nix/manifests.  It might be interesting to have nix-pull
  store manifests directly in the DB, rather than storing them as
  separate flat files, but then we would need a command line interface
  to delete manifests from the DB.
2011-04-10 23:22:46 +00:00
Eelco Dolstra 1e7e4f21ba * Remove the localPaths feature in manifests since it's no longer used
and redundant anyway.
2011-04-06 09:16:22 +00:00
Eelco Dolstra 0423d0692a * Print a better error message. 2011-03-16 15:55:57 +00:00
Eelco Dolstra 802c2651c8 2011-02-17 15:08:33 +00:00
Eelco Dolstra e945b52a31 * nix-push: handle the case where the hash is not set in the DB. 2011-02-17 13:26:33 +00:00
Eelco Dolstra d6c8b995c5 * In Hydra manifests the Size field is missing, so don't rely on it.
This caused a lot of "Use of uninitialized value" warnings from
  Perl.
2010-12-22 10:32:32 +00:00