Commit graph

91 commits

Author SHA1 Message Date
Eelco Dolstra f2b67fbf2a nix-push: Support -j
Fixes #548.
2015-06-08 14:16:06 +02:00
Eelco Dolstra f19b4abfb2 Include NAR size in fingerprint computation
This is not strictly needed for integrity (since we already include
the NAR hash in the fingerprint) but it helps against endless data
attacks [1]. (However, this will also require
download-from-binary-cache.pl to bail out if it receives more than the
specified number of bytes.)

[1] https://isis.poly.edu/~jcappos/papers/cappos_mirror_ccs_08.pdf
2015-02-17 13:16:58 +01:00
Eelco Dolstra f3a5930488 Sign a subset of the .narinfo
We only need to sign the store path, NAR hash and references (the
"fingerprint"). Everything else is irrelevant to security. For
instance, the compression algorithm or the hash of the compressed NAR
don't matter as long as the contents of the uncompressed NAR are
correct.

(Maybe we should include derivers in the fingerprint, but they're
broken and nobody cares about them. Also, it might be nice in the
future if .narinfos contained signatures from multiple independent
signers. But that's impossible if the deriver is included in the
fingerprint, since everybody will tend to have a different deriver for
the same store path.)

Also renamed the "Signature" field to "Sig" since the format changed
in an incompatible way.
2015-02-04 17:59:31 +01:00
Eelco Dolstra e0def5bc4b Use libsodium instead of OpenSSL for binary cache signing
Sodium's Ed25519 signatures are much shorter than OpenSSL's RSA
signatures. Public keys are also much shorter, so they're now
specified directly in the nix.conf option ‘binary-cache-public-keys’.

The new command ‘nix-store --generate-binary-cache-key’ generates and
prints a public and secret key.
2015-02-04 17:10:31 +01:00
Eelco Dolstra 27a01d92c2 Shut up "Wide character" warnings in Perl scripts 2014-08-29 17:48:25 +02:00
Eelco Dolstra 11849a320e Use proper quotes everywhere 2014-08-20 18:03:48 +02:00
Eelco Dolstra 3bea429ae8 Use $XDG_RUNTIME_DIR for temporary files 2014-08-13 23:12:57 +02:00
Eelco Dolstra 0fdf4da0e9 Support cryptographically signed binary caches
NAR info files in binary caches can now have a cryptographic signature
that Nix will verify before using the corresponding NAR file.

To create a private/public key pair for signing and verifying a binary
cache, do:

  $ openssl genrsa -out ./cache-key.sec 2048
  $ openssl rsa -in ./cache-key.sec -pubout > ./cache-key.pub

You should also come up with a symbolic name for the key, such as
"cache.example.org-1".  This will be used by clients to look up the
public key.  (It's a good idea to number keys, in case you ever need
to revoke/replace one.)

To create a binary cache signed with the private key:

  $ nix-push --dest /path/to/binary-cache --key ./cache-key.sec --key-name cache.example.org-1

The public key (cache-key.pub) should be distributed to the clients.
They should have a nix.conf should contain something like:

  signed-binary-caches = *
  binary-cache-public-key-cache.example.org-1 = /path/to/cache-key.pub

If all works well, then if Nix fetches something from the signed
binary cache, you will see a message like:

  *** Downloading ‘http://cache.example.org/nar/7dppcj5sc1nda7l54rjc0g5l1hamj09j-subversion-1.7.11’ (signed by ‘cache.example.org-1’) to ‘/nix/store/7dppcj5sc1nda7l54rjc0g5l1hamj09j-subversion-1.7.11’...

On the other hand, if the signature is wrong, you get a message like

  NAR info file `http://cache.example.org/7dppcj5sc1nda7l54rjc0g5l1hamj09j.narinfo' has an invalid signature; ignoring

Signatures are implemented as a single line appended to the NAR info
file, which looks like this:

  Signature: 1;cache.example.org-1;HQ9Xzyanq9iV...muQ==

Thus the signature has 3 fields: a version (currently "1"), the ID of
key, and the base64-encoded signature of the SHA-256 hash of the
contents of the NAR info file up to but not including the Signature
line.

Issue #75.
2014-01-08 15:42:53 +01:00
Eelco Dolstra 5116214343 Add support for uncompressed NARs in binary caches
Issue NixOS/hydra#102.
2013-07-01 21:03:14 +02:00
Eelco Dolstra 82951e5582 nix-push: Handle pushing a symlink 2012-11-06 13:53:53 +01:00
Eelco Dolstra dde6486eab nix-push: Add a flag ‘--manifest-path’ to write the manifest to another directory 2012-10-18 10:36:16 -04:00
Eelco Dolstra c8daeba303 nix-push: Add ‘--link’ flag
If ‘--link’ is given, nix-push will create hard links to the NAR files
in the store, rather than copying them.  This is faster and requires
less disk space.  However, it doesn't work if the store is on a
different file system.
2012-10-17 17:01:23 -04:00
Eelco Dolstra 167e36a5c3 nix-push: Only generate and copy a NAR if it doesn't already exist
This prevents unnecessary and slow rebuilds of NARs that already exist
in the binary cache.
2012-10-17 16:58:05 -04:00
Eelco Dolstra a562d544d8 When ‘--help’ is given, just run ‘man’ to show the manual page
I.e. do what git does.  I'm too lazy to keep the builtin help text up
to date :-)

Also add ‘--help’ to various commands that lacked it
(e.g. nix-collect-garbage).
2012-10-03 16:40:09 -04:00
Eelco Dolstra 20582e9ae3 Support xz compression in the download-using-manifests substituter 2012-09-19 17:33:42 -04:00
Eelco Dolstra 66a3ac6a56 Allow a binary cache to declare that it doesn't support "nix-env -qas"
Querying all substitutable paths via "nix-env -qas" is potentially
hard on a server, since it involves sending thousands of HEAD
requests.  So a binary cache must now have a meta-info file named
"nix-cache-info" that specifies whether the server wants this.  It
also specifies the store prefix so that we don't send useless queries
to a binary cache for a different store prefix.
2012-07-27 18:16:05 -04:00
Eelco Dolstra 67c6f3eded nix-push: Support generating a manifest again
This makes all the tests succeed.  Woohoo!
2012-07-26 18:28:12 -04:00
Eelco Dolstra e3ce54ccee nix-push: Remove the upload feature 2012-07-26 17:13:38 -04:00
Eelco Dolstra cf49472d60 nix-push: Always generate base-32 hashes 2012-07-02 18:05:57 -04:00
Eelco Dolstra 6ec7460af1 Binary caches: use a better key
Use the hash part of the store path as a key rather than a hash of the
store path.  This is enough to get the desired privacy property.
2012-07-02 12:42:58 -04:00
Eelco Dolstra f4190c38ba Allow both bzip2 and xz compression 2012-07-01 18:46:38 -04:00
Eelco Dolstra 000132cbd1 nix-push: Don't pollute the current directory with result symlink 2012-06-29 18:30:28 -04:00
Eelco Dolstra 91b8814f0d Doh 2012-06-29 17:16:00 -04:00
Eelco Dolstra 4911a10a4e Use XZ compression in binary caches
XZ compresses significantly better than bzip2.  Here are the
compression ratios and execution times (using 4 cores in parallel) on
my /var/run/current-system (3.1 GiB):

  bzip2: total compressed size 849.56 MiB, 30.8% [2m08]
  xz -6: total compressed size 641.84 MiB, 23.4% [6m53]
  xz -7: total compressed size 621.82 MiB, 22.6% [7m19]
  xz -8: total compressed size 599.33 MiB, 21.8% [7m18]
  xz -9: total compressed size 588.18 MiB, 21.4% [7m40]

Note that compression takes much longer.  More importantly, however,
decompression is much faster:

  bzip2: 1m47.274s
  xz -6: 0m55.446s
  xz -7: 0m54.119s
  xz -8: 0m52.388s
  xz -9: 0m51.842s

The only downside to using -9 is that decompression takes a fair
amount (~65 MB) of memory.
2012-06-29 15:24:52 -04:00
Eelco Dolstra 49cd7387ad nix-push: create a manifest-less binary cache
Manifests are a huge pain, since users need to run nix-pull directly
or indirectly to obtain them.  They tend to be large and lag behind
the available binaries; also, the downloaded manifests in
/nix/var/nix/manifest need to be in sync with the Nixpkgs sources.  So
we want to get rid of them.

The idea of manifest-free operation works as follows.  Nix is
configured with a set of URIs of binary caches, e.g.

  http://nixos.org/binary-cache

Whenever Nix needs a store path X, it checks each binary cache for the
existence of a file <CACHE-URI>/<SHA-256 hash of X>.narinfo, e.g.

  http://nixos.org/binary-cache/bi1gh9...ia17.narinfo

The .narinfo file contains the necessary information about the store
path that was formerly kept in the manifest, i.e., (relative) URI of
the compressed NAR, references, size, hash, etc.  For example:

  StorePath: /nix/store/xqp4l88cr9bxv01jinkz861mnc9p7qfi-neon-0.29.6
  URL: 1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2
  CompressedHash: sha256:1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd
  CompressedSize: 202542
  NarHash: sha256:1af26536781e6134ab84201b33408759fc59b36cc5530f57c0663f67b588e15f
  NarSize: 700440
  References: 043zrsanirjh8nbc5vqpjn93hhrf107f-bash-4.2-p24 cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 ...
  Deriver: 4idz1bgi58h3pazxr3akrw4fsr6zrf3r-neon-0.29.6.drv
  System: x86_64-linux

Nix then knows that it needs to download

  http://nixos.org/binary-cache/1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2

to substitute the store path.

Note that the store directory is omitted from the References and
Deriver fields to save space, and that the URL can be relative to the
binary cache prefix.

This patch just makes nix-push create binary caches in this format.
The next step is to make a substituter that supports them.
2012-06-28 17:19:32 -04:00
Eelco Dolstra ef902274fd Remove unnecessary "system" argument 2012-04-14 18:48:11 +02:00
Eelco Dolstra dadbb51d96 * Use Nix::Config. 2012-01-03 00:47:27 +00:00
Eelco Dolstra 48cea0d01e * Refactoring: Get rid of a few subdirectories in corepkgs/, and some
other simplifications.
* Use <nix/...> to locate the corepkgs.  This allows them to be
  overriden through $NIX_PATH.
* Use bash's pipefail option in the NAR builder so that we don't need
  to create a temporary file.
2012-01-03 00:16:29 +00:00
Eelco Dolstra 216440b3ff * For consistency with "nix-store -q --hash", produce hashes in
base-32.  (This affects Hydra manifests.)
2011-11-29 12:32:55 +00:00
Eelco Dolstra 5090c34ee1 * Set the executable bit on scripts. 2011-10-10 22:40:17 +00:00
Eelco Dolstra 6fcdbcac20 * Install NixManifest.pm, NixConfig.pm and GeneratePatches.pm under
the Nix:: namespace.
2011-10-10 21:11:08 +00:00
Eelco Dolstra 3cbf680f5d * configure: detect whether DBD::SQLite is present. If necessary the
location to DBI and DBD::SQLite can be passed with --with-dbi and
  --with-dbd-sqlite.
2011-04-11 10:13:53 +00:00
Eelco Dolstra 83252b4ca9 * Subtle bug: if you import File::stat in one module, it affects other
modules as well.  So use symbolic field names everywhere (which is
  nicer anyway).
2011-04-11 08:21:30 +00:00
Eelco Dolstra 802c2651c8 2011-02-17 15:08:33 +00:00
Eelco Dolstra e945b52a31 * nix-push: handle the case where the hash is not set in the DB. 2011-02-17 13:26:33 +00:00
Eelco Dolstra 365f3028dd * Use CamelCase for the Perl modules. 2010-12-05 17:50:29 +00:00
Eelco Dolstra 06699d4219 * Store the NAR size in the manifest. 2010-11-17 12:57:52 +00:00
Eelco Dolstra 5693b8a7e2 * nix-push: no need to compute the NAR hash, since the Nix database
already has it (`nix-store -q --hash').
2010-11-17 12:51:54 +00:00
Eelco Dolstra 2897286487 * Unify exportReferencesGraph and exportBuildReferencesGraph, and make
sure that it works as expected when you pass it a derivation.  That
  is, we have to make sure that all build-time dependencies are built,
  and that they are all in the input closure (otherwise remote builds
  might fail, for example).  This is ensured at instantiation time by
  adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 17:36:42 +00:00
Eelco Dolstra f8713e1287 * Dirty hack to make nix-push work properly on derivations: the
derivation should be a source rather than a derivation dependency of
  the call to the NAR derivation.  Otherwise the derivation (and all
  its dependencies) will be built as a side-effect, which may not even
  succeed.
2008-12-04 10:40:41 +00:00
Eelco Dolstra 4213b8d8ec * Urgh. 2008-11-20 15:44:59 +00:00
Eelco Dolstra aab530e971 * Primop builtins.storePath for declaring a store path as a
dependency.  `storePath /nix/store/bla' gives exactly the same
  result as `toPath /nix/store/bla', except that the former includes
  /nix/store/bla in the dependency context of the string.

  Useful in some generated Nix expressions like nix-push, which now
  finally does the right thing wrt distributed builds.  (Previously
  the path to be packed wasn't an explicit dependency, so it wouldn't
  be copied to the remote machine.)
2008-11-19 23:26:19 +00:00
Eelco Dolstra f106868110 * Cleanup. 2008-03-20 10:16:36 +00:00
Eelco Dolstra de012e76e2 * Don't use "store expression", it's obsolete. 2007-12-30 23:35:45 +00:00
Eelco Dolstra c370755583 * Flag `--no-build-hook' to disable distributed builds.
* queryDeriver in daemon mode: don't barf if the other side returns an
  empty string (which means there is no deriver).
2007-11-16 16:15:26 +00:00
Eelco Dolstra 9441550acb * nix-push / generate-patches: bzip the manifest. 2007-09-04 15:38:09 +00:00
Eelco Dolstra dbc0170ed9 * Show errors in nix-prefetch-url. 2007-08-15 09:24:06 +00:00
Eelco Dolstra d98f750fd8 * tmpnam() -> File::Temp::tempdir(). 2006-10-04 18:58:11 +00:00
Eelco Dolstra 68ae953d8a * Clean up calls to system(). 2006-09-25 10:44:27 +00:00
Eelco Dolstra 76c9710091 * Use builtins.toPath. 2006-09-25 10:29:25 +00:00