Commit graph

151 commits

Author SHA1 Message Date
Eelco Dolstra 7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra c2d27d30cf nix-copy-closure / build-remote.pl: Disable signature checking
This restores the Nix 1.11 behaviour.
2016-05-31 11:48:05 +02:00
Eelco Dolstra b5ac83f3a3 nix-copy-closure: Drop Nix <= 1.7 compat 2016-05-31 11:48:05 +02:00
Eelco Dolstra 4e17e7a4da Fix segfault in nix-copy-closure 2016-05-04 12:05:46 +02:00
Eelco Dolstra f435f82475 Remove OpenSSL-based signing 2016-05-04 11:01:48 +02:00
Eelco Dolstra e0204f8d46 Move path info caching from BinaryCacheStore to Store
Caching path info is generally useful. For instance, it speeds up "nix
path-info -rS /run/current-system" (i.e. showing the closure sizes of
all paths in the closure of the current system) from 5.6s to 0.15s.

This also eliminates some APIs like Store::queryDeriver() and
Store::queryReferences().
2016-04-19 18:52:53 +02:00
Eelco Dolstra 867967265b Remove manifest support
Manifests have been superseded by binary caches for years. This also
gets rid of nix-pull, nix-generate-patches and bsdiff/bspatch.
2016-04-11 16:20:15 +02:00
Eelco Dolstra bcc9943cee importPaths(): Optionally add NARs to binary cache accessor
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
2016-02-26 15:20:10 +01:00
Eelco Dolstra c4d22997f3 Add C++ functions for .narinfo processing / signing
This is currently only used by the Hydra queue runner rework, but like
eff5021eaa it presumably will be useful
for the C++ rewrite of nix-push and
download-from-binary-cache. (@shlevy)
2016-02-16 16:38:44 +01:00
Eelco Dolstra 5ac27053e9 Rename ValidPathInfo::hash -> narHash for consistency 2016-02-16 11:49:12 +01:00
Eelco Dolstra 03109e9580 Fix nix-copy-closure
http://hydra.nixos.org/build/32005971
2016-02-15 20:09:30 +01:00
Eelco Dolstra fa7cd5369b StoreAPI -> Store
Calling a class an API is a bit redundant...
2016-02-04 14:48:42 +01:00
Eelco Dolstra c10c61449f Eliminate the "store" global variable
Also, move a few free-standing functions into StoreAPI and Derivation.

Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
2016-02-04 14:28:26 +01:00
Eelco Dolstra 94595f42eb Fix bad use of croak()
Detected by -Werror=format-security.

http://hydra.nixos.org/build/29973207
2016-01-07 14:33:13 +01:00
Eelco Dolstra 133a421bb4 Provide addTempRoot in the Perl API
Needed by Hydra.
2015-10-09 12:49:47 +02:00
Eelco Dolstra 94378910fb Handle base-16 NarHash fields in signed .narinfo files 2015-06-03 15:33:17 +02:00
Eelco Dolstra a64da5915d Be more robust wrt broken .narinfo files 2015-06-03 15:19:32 +02:00
Eelco Dolstra b8b571cfc1 Include the cache.nixos.org signing key by default 2015-06-01 17:14:03 +02:00
Eelco Dolstra 7ea6ecf855 addToStore(): Take explicit name argument 2015-03-25 17:06:12 +01:00
Eelco Dolstra 9f3eb56b46 Reduce verbosity in build-remote.pl 2015-03-04 16:27:42 +01:00
Eelco Dolstra f19b4abfb2 Include NAR size in fingerprint computation
This is not strictly needed for integrity (since we already include
the NAR hash in the fingerprint) but it helps against endless data
attacks [1]. (However, this will also require
download-from-binary-cache.pl to bail out if it receives more than the
specified number of bytes.)

[1] https://isis.poly.edu/~jcappos/papers/cappos_mirror_ccs_08.pdf
2015-02-17 13:16:58 +01:00
Eelco Dolstra 1c972cba14 Make libsodium an optional dependency 2015-02-10 11:54:06 +01:00
Eelco Dolstra f3a5930488 Sign a subset of the .narinfo
We only need to sign the store path, NAR hash and references (the
"fingerprint"). Everything else is irrelevant to security. For
instance, the compression algorithm or the hash of the compressed NAR
don't matter as long as the contents of the uncompressed NAR are
correct.

(Maybe we should include derivers in the fingerprint, but they're
broken and nobody cares about them. Also, it might be nice in the
future if .narinfos contained signatures from multiple independent
signers. But that's impossible if the deriver is included in the
fingerprint, since everybody will tend to have a different deriver for
the same store path.)

Also renamed the "Signature" field to "Sig" since the format changed
in an incompatible way.
2015-02-04 17:59:31 +01:00
Eelco Dolstra e0def5bc4b Use libsodium instead of OpenSSL for binary cache signing
Sodium's Ed25519 signatures are much shorter than OpenSSL's RSA
signatures. Public keys are also much shorter, so they're now
specified directly in the nix.conf option ‘binary-cache-public-keys’.

The new command ‘nix-store --generate-binary-cache-key’ generates and
prints a public and secret key.
2015-02-04 17:10:31 +01:00
Eelco Dolstra 048ec3d3f2 Fix bad operator
Spotted by Perl 5.20:

  Possible precedence issue with control flow operator at /usr/lib/perl5/site_perl/5.20.1/x86_64-linux-gnu-thread-multi/Nix/Utils.pm line 46.
2014-11-20 11:45:01 +01:00
Eelco Dolstra d436e44ae3 SSH.pm: Print a friendlier message if connecting fails
"got EOF while expecting 8 bytes from remote side" is not very
helpful.
2014-11-10 16:03:51 +01:00
Eelco Dolstra f1adf4c998 Remove unused @sshOpts flag
Closes #300.
2014-10-14 11:01:18 +02:00
Eelco Dolstra 726abdecfb Remove debug statement 2014-09-18 20:04:49 +02:00
Eelco Dolstra 11849a320e Use proper quotes everywhere 2014-08-20 18:03:48 +02:00
Eelco Dolstra 3bea429ae8 Use $XDG_RUNTIME_DIR for temporary files 2014-08-13 23:12:57 +02:00
Eelco Dolstra 1a44f1cb65 nix-copy-closure: Drop --bzip2, --xz, --show-progress
These are too difficult to implement via nix-store --serve.

‘--show-progress’ could be re-implemented fairly easily via a
sink/source wrapper class.
2014-07-24 17:11:54 +02:00
Eelco Dolstra 17ef234672 Fix NIX_SSHOPTS
Closes #302.
2014-07-24 16:30:37 +02:00
Eelco Dolstra 7b351da680 Remove obsolete SSH master connection code 2014-07-24 16:19:42 +02:00
Eelco Dolstra 56131a2709 Refactor 2014-07-24 12:24:25 +02:00
Eelco Dolstra b2e0293f02 build-remote.pl: Don't keep a shell process around 2014-07-12 00:43:28 +02:00
Eelco Dolstra a00a98548e build-remote.pl: Fix build log 2014-07-12 00:09:43 +02:00
Eelco Dolstra a5c6347ff0 build-remote.pl: Use ‘nix-store --serve’ on the remote side
This makes things more efficient (we don't need to use an SSH master
connection, and we only start a single remote process) and gets rid of
locking issues (the remote nix-store process will keep inputs and
outputs locked as long as they're needed).

It also makes it more or less secure to connect directly to the root
account on the build machine, using a forced command
(e.g. ‘command="nix-store --serve --write"’). This bypasses the Nix
daemon and is therefore more efficient.

Also, don't call nix-store to import the output paths.
2014-07-11 16:22:24 +02:00
Eelco Dolstra b8f24f2535 Fix closure size display 2014-07-11 14:27:17 +02:00
Eelco Dolstra 42d91b079c Fix use of sysread 2014-07-10 21:17:21 +02:00
Eelco Dolstra 7bb632b024 nix-copy-closure -s: Do substitutions via ‘nix-store --serve’
This means we no longer need an SSH master connection, since we only
execute a single command on the remote host.
2014-07-10 20:43:04 +02:00
Eelco Dolstra 7c3a5090bf nix-copy-closure: Fix --dry-run 2014-07-10 17:44:18 +02:00
Eelco Dolstra 1114c7bd57 nix-copy-closure: Restore compression and the progress viewer 2014-07-10 14:15:12 +02:00
Eelco Dolstra 04170d06bf nix-copy-closure: Fix race condition
There is a long-standing race condition when copying a closure to a
remote machine, particularly affecting build-remote.pl: the client
first asks the remote machine which paths it already has, then copies
over the missing paths. If the garbage collector kicks in on the
remote machine between the first and second step, the already-present
paths may be deleted. The missing paths may then refer to deleted
paths, causing nix-copy-closure to fail. The client now performs both
steps using a single remote Nix call (using ‘nix-store --serve’),
locking all paths in the closure while querying.

I changed the --serve protocol a bit (getting rid of QueryCommand), so
this breaks the SSH substituter from older versions. But it was marked
experimental anyway.

Fixes #141.
2014-07-10 11:58:59 +02:00
Eelco Dolstra 625ffd441d Ugly hack to fix building with clang 2014-01-21 16:38:03 +01:00
Eelco Dolstra 81628a6ccc Merge branch 'master' into make
Conflicts:
	src/libexpr/eval.cc
2014-01-21 15:30:01 +01:00
Eelco Dolstra 0fdf4da0e9 Support cryptographically signed binary caches
NAR info files in binary caches can now have a cryptographic signature
that Nix will verify before using the corresponding NAR file.

To create a private/public key pair for signing and verifying a binary
cache, do:

  $ openssl genrsa -out ./cache-key.sec 2048
  $ openssl rsa -in ./cache-key.sec -pubout > ./cache-key.pub

You should also come up with a symbolic name for the key, such as
"cache.example.org-1".  This will be used by clients to look up the
public key.  (It's a good idea to number keys, in case you ever need
to revoke/replace one.)

To create a binary cache signed with the private key:

  $ nix-push --dest /path/to/binary-cache --key ./cache-key.sec --key-name cache.example.org-1

The public key (cache-key.pub) should be distributed to the clients.
They should have a nix.conf should contain something like:

  signed-binary-caches = *
  binary-cache-public-key-cache.example.org-1 = /path/to/cache-key.pub

If all works well, then if Nix fetches something from the signed
binary cache, you will see a message like:

  *** Downloading ‘http://cache.example.org/nar/7dppcj5sc1nda7l54rjc0g5l1hamj09j-subversion-1.7.11’ (signed by ‘cache.example.org-1’) to ‘/nix/store/7dppcj5sc1nda7l54rjc0g5l1hamj09j-subversion-1.7.11’...

On the other hand, if the signature is wrong, you get a message like

  NAR info file `http://cache.example.org/7dppcj5sc1nda7l54rjc0g5l1hamj09j.narinfo' has an invalid signature; ignoring

Signatures are implemented as a single line appended to the NAR info
file, which looks like this:

  Signature: 1;cache.example.org-1;HQ9Xzyanq9iV...muQ==

Thus the signature has 3 fields: a version (currently "1"), the ID of
key, and the base64-encoded signature of the SHA-256 hash of the
contents of the NAR info file up to but not including the Signature
line.

Issue #75.
2014-01-08 15:42:53 +01:00
Petr Rockai f1e5dedb61 perl: Call loadConfFile() in doInit to avoid screwing sqlite journal mode.
If the database is opened through perl bindings (and even though nix.conf has
use-sqlite-wal set to false), the database is automatically converted into WAL
mode. This makes the next nix process to access the database convert it back to
"truncate". If the database is still open at the time in wal mode by the perl
program, this fails and crashes the nix doing the wal -> truncate conversion.
2013-12-20 13:59:52 +01:00
Eelco Dolstra f3cf0436b5 Install bsdiff and bspatch in $(libexecdir)/nix 2013-12-02 20:21:31 +00:00
Eelco Dolstra 9285f0aa2b Add a Makefile for the Perl stuff 2013-11-25 16:38:33 +00:00
Eelco Dolstra 2d9bb56e55 Fix segfault on Darwin
Ever since SQLite in Nixpkgs was updated to 3.8.0.2, Nix has randomly
segfaulted on Darwin:

  http://hydra.nixos.org/build/6175515
  http://hydra.nixos.org/build/6611038

It turns out that this is because the binary cache substituter somehow
ends up loading two versions of SQLite: the one in Nixpkgs and the
other from /usr/lib/libsqlite3.dylib.  It's not exactly clear why the
latter is loaded, but it appears to be because WWW::Curl indirectly loads
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation,
which in turn seems to load /usr/lib/libsqlite3.dylib.  This leads to
a segfault when Perl exits:

  #0  0x00000001010375f4 in sqlite3_finalize ()
  #1  0x000000010125806e in sqlite_st_destroy ()
  #2  0x000000010124bc30 in XS_DBD__SQLite__st_DESTROY ()
  #3  0x00000001001c8155 in XS_DBI_dispatch ()
  ...
  #14 0x0000000100023224 in perl_destruct ()
  #15 0x0000000100000d6a in main ()
  ...

The workaround is to explicitly load DBD::SQLite before WWW::Curl.
2013-10-24 19:15:52 +02:00
Eelco Dolstra 936f9d45ba Don't apply the CPU affinity hack to nix-shell (and other Perl programs)
As discovered by Todd Veldhuizen, the shell started by nix-shell has
its affinity set to a single CPU.  This is because nix-shell connects
to the Nix daemon, which causes the affinity hack to be applied.  So
we turn this off for Perl programs.
2013-09-06 16:36:56 +02:00
Eelco Dolstra 22144afa8d Don't keep "disabled" substituters running
For instance, it's pointless to keep copy-from-other-stores running if
there are no other stores, or download-using-manifests if there are no
manifests.  This also speeds things up because we don't send queries
to those substituters.
2013-06-20 11:55:15 +02:00
Eelco Dolstra 3a0cc43ac8 build-remote.pl: Properly close the SSH connection between attempts 2013-05-10 02:38:50 +02:00
Lluís Batlle i Rossell e6c44d166a Fixing the pv position regarding compression
Problem noticed by niksnut.
2013-05-03 11:08:51 +02:00
Lluís Batlle i Rossell 5cc2fc46ec Adding ETA support to the --show-progress in nix-copy-closure
Based on https://github.com/NixOS/nix/pull/6 from shlevy
2013-05-03 11:08:51 +02:00
Eelco Dolstra bdd4646338 Revert "Prevent config.h from being clobbered"
This reverts commit 28bba8c44f.
2013-03-08 01:24:59 +01:00
Eelco Dolstra e73d9e9488 Fix annoying Perl 5.16 warnings
I.e.

Subroutine Nix::Store::isValidPath redefined at /nix/store/clfzsf6gi7qh5i9c0vks1ifjam47rijn-perl-5.16.2/lib/perl5/5.16.2/XSLoader.pm line 92.

and so on.
2013-03-08 00:27:32 +01:00
Eelco Dolstra 28bba8c44f Prevent config.h from being clobbered 2013-03-07 23:55:55 +01:00
Eelco Dolstra 8add116acd Nix::Store::derivationFromPath: Return derivation outputs 2013-02-05 16:02:57 +01:00
Eelco Dolstra 9c29a2ed35 Give a better error message if writeFile fails due to permission issues 2012-12-20 12:22:13 +01:00
Eelco Dolstra a3d6585c5a nix-copy-closure: Add flag ‘--use-substitutes’ 2012-11-23 16:20:16 +01:00
Eelco Dolstra 167e36a5c3 nix-push: Only generate and copy a NAR if it doesn't already exist
This prevents unnecessary and slow rebuilds of NARs that already exist
in the binary cache.
2012-10-17 16:58:05 -04:00
Eelco Dolstra 20582e9ae3 Support xz compression in the download-using-manifests substituter 2012-09-19 17:33:42 -04:00
Eelco Dolstra b14717ab90 Delete manifests in "nix-channel --remove" or when a binary cache is available 2012-09-13 11:35:46 -04:00
Eelco Dolstra 8b8fe6139e Drop dependency on List::MoreUtils 2012-08-27 11:28:34 -04:00
Eelco Dolstra babe54bf97 Add missing file 2012-08-27 11:11:30 -04:00
Eelco Dolstra 97421eb5ec Refactor settings processing
Put all Nix configuration flags in a Settings object.
2012-07-30 19:55:41 -04:00
Eelco Dolstra d059bf48e4 Pass configuration settings to the substituters
Previously substituters could read nix.conf themselves, but this
didn't take --option flags into account.
2012-07-30 16:09:54 -04:00
Eelco Dolstra fe241ece29 Merge branch 'master' into no-manifests 2012-07-18 10:47:59 -04:00
Eelco Dolstra ccc52adfb2 Add function queryPathFromHashPart()
To implement binary caches efficiently, Hydra needs to be able to map
the hash part of a store path (e.g. "gbg...zr7") to the full store
path (e.g. "/nix/store/gbg...kzr7-subversion-1.7.5").  (The binary
cache mechanism uses hash parts as a key for looking up store paths to
ensure privacy.)  However, doing a search in the Nix store for
/nix/store/<hash>* is expensive since it requires reading the entire
directory.  queryPathFromHashPart() prevents this by doing a cheap
database lookup.
2012-07-17 18:55:39 -04:00
Eelco Dolstra 220818f758 queryPathInfo(): return hash in base-32 if desired
Cherry-picked from the no-manifests branch.
2012-07-17 16:55:45 -04:00
Eelco Dolstra d287b62b64 Set the User-Agent header to "Nix/<version>" 2012-07-11 18:05:30 -04:00
Eelco Dolstra 099125435f download-from-binary-cache: add nix.conf options 2012-07-09 10:57:28 -04:00
Eelco Dolstra cf49472d60 nix-push: Always generate base-32 hashes 2012-07-02 18:05:57 -04:00
Eelco Dolstra 37f7098464 First attempt at the manifest-less substituter 2012-06-29 18:28:52 -04:00
Eelco Dolstra 4911a10a4e Use XZ compression in binary caches
XZ compresses significantly better than bzip2.  Here are the
compression ratios and execution times (using 4 cores in parallel) on
my /var/run/current-system (3.1 GiB):

  bzip2: total compressed size 849.56 MiB, 30.8% [2m08]
  xz -6: total compressed size 641.84 MiB, 23.4% [6m53]
  xz -7: total compressed size 621.82 MiB, 22.6% [7m19]
  xz -8: total compressed size 599.33 MiB, 21.8% [7m18]
  xz -9: total compressed size 588.18 MiB, 21.4% [7m40]

Note that compression takes much longer.  More importantly, however,
decompression is much faster:

  bzip2: 1m47.274s
  xz -6: 0m55.446s
  xz -7: 0m54.119s
  xz -8: 0m52.388s
  xz -9: 0m51.842s

The only downside to using -9 is that decompression takes a fair
amount (~65 MB) of memory.
2012-06-29 15:24:52 -04:00
Eelco Dolstra 4f7bab7db1 Support building with the Perl XS bindings disabled
Since the Perl bindings require shared libraries, this is required on
platforms such as Cygwin where we do a static build.
2012-05-10 19:03:23 -04:00
Shea Levy 34a85c5405 nix-copy-closure: Move the progressViewer directly adjacent to the ssh call so that network progress is what's measured 2012-04-13 14:29:12 +02:00
Shea Levy 055e803851 Add the '--show-progress' flag to nix-copy-closure 2012-04-13 14:29:12 +02:00
Eelco Dolstra 209927bb27 Unconfuse Rob 2012-04-10 16:40:51 +02:00
Eelco Dolstra 7f38087f35 Add a command "nix-build --run-env" to reproduce the environment of a derivation
This command builds or fetches all dependencies of the given
derivation, then starts a shell with the environment variables from
the derivation.  This shell also sources $stdenv/setup to initialise
the environment further.

The current directory is not changed.  Thus this is a convenient way
to reproduce a build environment in an existing working tree.

Existing environment variables are left untouched (unless the
derivation overrides them).  As a special hack, the original value of
$PATH is appended to the $PATH produced by $stdenv/setup.

Example session:

$ nix-build --run-env '<nixpkgs>' -A xterm
(the dependencies of xterm are built/fetched...)
$ tar xf $src
$ ./configure
$ make
$ emacs
(... hack source ...)
$ make
$ ./xterm
2012-03-19 04:14:21 +01:00
Eelco Dolstra d645633643 nix-copy-closure: don't print copied path on stdout
We're already printing progress on stderr, so printing them on stdout
afterwards is kind of useless.
2012-03-05 20:52:24 +01:00
Eelco Dolstra 2b4964f319 Restore progress indication during nix-copy-closure 2012-03-05 18:13:44 +01:00
Eelco Dolstra 6c31232e14 * Sync with the trunk. 2012-01-03 12:59:31 +00:00
Eelco Dolstra 502d94048a * Ignore missing manifest symlinks. 2012-01-03 12:59:00 +00:00
Eelco Dolstra 39d45a6b09 * Add a test for nix-channel.
* Refactor the nix-channel unpacker a bit.
2012-01-03 01:51:38 +00:00
Eelco Dolstra 92d6a5ed73 * Add some more functions to the Perl bindings. 2011-12-02 12:09:24 +00:00
Eelco Dolstra f35c4351e5 * Don't require a specific Perl version. 2011-11-29 22:15:07 +00:00
Eelco Dolstra 1749a7b0ae * download-using-manifests: use the Perl bindings. 2011-11-29 13:01:24 +00:00
Eelco Dolstra 216440b3ff * For consistency with "nix-store -q --hash", produce hashes in
base-32.  (This affects Hydra manifests.)
2011-11-29 12:32:55 +00:00
Eelco Dolstra 5bbd693cae * Add an API function exportPaths() that provides the functionality of
‘nix-store --export’.
* Add a Perl module that provides the functionality of
  ‘nix-copy-closure --to’.  This is used by build-remote.pl so it no
  longer needs to start a separate nix-copy-closure process.  Also, it
  uses the Perl API to do the export, so it doesn't need to start a
  separate nix-store process either.  As a result, nix-copy-closure
  and build-remote.pl should no longer fail on very large closures due
  to an "Argument list too long" error.  (Note that having very many
  dependencies in a single derivation can still fail because the
  environment can become too large.  Can't be helped though.)
2011-11-23 15:13:37 +00:00
Eelco Dolstra ab20af3e6f * build-remote.pl: drop a hard-coded reference to /nix/etc/nix. 2011-11-23 12:21:35 +00:00
Eelco Dolstra 993fa94fb4 * Move initialisation of variables like nixConfDir from libmain to
libstore so that the Perl bindings can use it as well.  It's vital
  that the Perl bindings use the configuration file, because otherwise
  nix-copy-closure will fail with a ‘database locked’ message if the
  value of ‘use-sqlite-wal’ is changed from the default.
2011-11-22 17:28:41 +00:00
Eelco Dolstra f8e609c341 * nix-pull: update the Nix manifest cache if necessary. Also, don't
read the manifest just to check the version and print the number of
  paths.  This makes nix-pull very fast for the cached cache (speeding
  up nixos-rebuild without the ‘--no-pull’ or ‘--fast’ options).
2011-11-16 16:41:48 +00:00
Eelco Dolstra d7d7910ba4 * Don't decompress the manifests in /nix/var/nix/manifest. This saves
disk space, and, since they're typically only decompressed once (to
  fill the manifest cache), doesn't make things slower.
2011-11-16 16:25:38 +00:00
Eelco Dolstra 63ee5e4d2a * Remove obsolete line. 2011-11-16 11:56:19 +00:00
Eelco Dolstra a5952405d2 * Re-use prepared statements across insertions into the manifest cache
DB.  This speeds up creating the cache from 16.1s to 7.9s on my
  system.
2011-11-16 11:37:40 +00:00
Eelco Dolstra 6761757428 * Use the Store API bindings in nix-copy-closure. 2011-10-11 15:41:13 +00:00
Eelco Dolstra c362e4d718 * Move SSH.pm. 2011-10-11 11:45:36 +00:00
Eelco Dolstra 6fcdbcac20 * Install NixManifest.pm, NixConfig.pm and GeneratePatches.pm under
the Nix:: namespace.
2011-10-10 21:11:08 +00:00