Commit graph

82 commits

Author SHA1 Message Date
Philip Potter 4f3cf06c97 Verify TLS certificate before downloading binaries
The --insecure flag to curl tells curl not to bother checking if the TLS
certificate presented by the server actually matches the hostname
requested, and actually is issued by a trusted CA chain.  This almost
entirely negates any benefit from using TLS in the first place.

This removes the --insecure flag to ensure we actually have a secure
connection to the intended hostname before downloading binaries.

Manually tested locally within a dev-shell; was able to download
binaries from https://cache.nixos.org without issue.

[Note: --insecure was only used for fetching NARs, whose integrity is
verified by Nix anyway using the hash from the .narinfo. But if we can
fetch the .narinfo without --insecure, we can also fetch the .nar, so
there is not much point to using --insecure. --Eelco]
2016-01-05 14:19:46 +01:00
Eelco Dolstra c0d4173263 Set default binary-caches-parallel-connections to 25
Some benchmarking suggested this as a good value. Running

  $ benchmark -f ... -t 25 -- sh -c 'rm -f /nix/var/nix/binary-cache*; nix-store -r /nix/store/x5z8a2yvz8h6ccmhwrwrp9igg03575jg-nixos-15.09.git.5fd87e1M.drv --dry-run --option binary-caches-parallel-connections <N>'

gave the following mean elapsed times for these values of N:

N=10:  3.3541
N=20:  2.9320
N=25:  2.6690
N=30:  2.9417
N=50:  3.2021
N=100: 3.5718
N=150: 4.2079

Memory usage is also reduced (N=150 used 186 MB, N=25 only 68 MB).

Closes #708.
2015-11-25 17:13:11 +01:00
Utku Demir a49514a2e2 Also set CURLOPT_SSL_VERIFYHOST=0 when "verify-https-binary-caches" is false
This makes that option even more insecure, by also not checking the SSL host.

But without this parameter, one can still get SSL errors even when
"verify-https-binary-caches" is false, which is unexpected IMO.
2015-09-25 22:34:08 +03:00
Eelco Dolstra 175935e053 FIXMEs 2015-02-19 14:10:33 +01:00
Eelco Dolstra a5e2c8e560 Set correct user agent for NAR downloads from binary caches 2015-01-15 12:05:27 +01:00
Eelco Dolstra af555d7694 Doh 2014-12-09 20:43:12 +01:00
Eelco Dolstra d44d923be9 Add option to disable binary cache certificate checking 2014-12-09 13:16:02 +01:00
Eelco Dolstra 5510d21193 Provide some fallback defaults for the CA bundle 2014-12-09 13:16:02 +01:00
Eelco Dolstra e5891f2ea8 Use https://cache.nixos.org instead of http://cache.nixos.org 2014-12-09 13:16:02 +01:00
Eelco Dolstra 8979562ed7 download-from-binary-cache.pl: Fix flushing of stderr 2014-11-04 14:37:58 +01:00
Eelco Dolstra 138c257dcd Revert "binary download: Use $NIX_CURL_FLAGS"
This reverts commit bc4795919a. It
breaks the build:

  http://hydra.nixos.org/build/15860847
2014-10-15 10:22:17 +02:00
wmertens bc4795919a binary download: Use $NIX_CURL_FLAGS
As in 5c0816567d/pkgs/build-support/fetchurl/builder.sh (L17)
2014-10-14 15:36:14 +02:00
Eelco Dolstra 27a01d92c2 Shut up "Wide character" warnings in Perl scripts 2014-08-29 17:48:25 +02:00
Eelco Dolstra 163fdf292e Set a curl timeout on binary cache lookups 2014-08-21 15:15:50 +02:00
Eelco Dolstra 11849a320e Use proper quotes everywhere 2014-08-20 18:03:48 +02:00
Eelco Dolstra 57695d8f76 Warn about untrusted binary caches in extra-binary-caches 2014-08-07 22:46:35 +02:00
Eelco Dolstra 393a9b92b1 download-from-binary-cache.pl: Respect $SSL_CERT_FILE 2014-08-07 22:41:15 +02:00
Eelco Dolstra 7f74513b4e Also provide an option for setting the curl connection timeout 2014-02-26 16:07:43 +01:00
Eelco Dolstra 00d761016a Respect $NIX_CONNECT_TIMEOUT properly
We were 1) using CURLOPT_TIMEOUT instead of CURLOPT_CONNECTTIMEOUT; 2)
not passing it to the curl child process.

Issue #93.
2014-02-26 15:58:37 +01:00
Eelco Dolstra bf4a577a58 Fix broken patch 2014-02-26 13:48:23 +01:00
Ian-Woo Kim 8a02fdc38e use USER environmental variable if getting user id by getpwuid is failed in perl scripts: download-from-binary-cache.pl and nix-channel 2014-02-26 13:42:11 +01:00
Eelco Dolstra 0fdf4da0e9 Support cryptographically signed binary caches
NAR info files in binary caches can now have a cryptographic signature
that Nix will verify before using the corresponding NAR file.

To create a private/public key pair for signing and verifying a binary
cache, do:

  $ openssl genrsa -out ./cache-key.sec 2048
  $ openssl rsa -in ./cache-key.sec -pubout > ./cache-key.pub

You should also come up with a symbolic name for the key, such as
"cache.example.org-1".  This will be used by clients to look up the
public key.  (It's a good idea to number keys, in case you ever need
to revoke/replace one.)

To create a binary cache signed with the private key:

  $ nix-push --dest /path/to/binary-cache --key ./cache-key.sec --key-name cache.example.org-1

The public key (cache-key.pub) should be distributed to the clients.
They should have a nix.conf should contain something like:

  signed-binary-caches = *
  binary-cache-public-key-cache.example.org-1 = /path/to/cache-key.pub

If all works well, then if Nix fetches something from the signed
binary cache, you will see a message like:

  *** Downloading ‘http://cache.example.org/nar/7dppcj5sc1nda7l54rjc0g5l1hamj09j-subversion-1.7.11’ (signed by ‘cache.example.org-1’) to ‘/nix/store/7dppcj5sc1nda7l54rjc0g5l1hamj09j-subversion-1.7.11’...

On the other hand, if the signature is wrong, you get a message like

  NAR info file `http://cache.example.org/7dppcj5sc1nda7l54rjc0g5l1hamj09j.narinfo' has an invalid signature; ignoring

Signatures are implemented as a single line appended to the NAR info
file, which looks like this:

  Signature: 1;cache.example.org-1;HQ9Xzyanq9iV...muQ==

Thus the signature has 3 fields: a version (currently "1"), the ID of
key, and the base64-encoded signature of the SHA-256 hash of the
contents of the NAR info file up to but not including the Signature
line.

Issue #75.
2014-01-08 15:42:53 +01:00
Eelco Dolstra 2d9bb56e55 Fix segfault on Darwin
Ever since SQLite in Nixpkgs was updated to 3.8.0.2, Nix has randomly
segfaulted on Darwin:

  http://hydra.nixos.org/build/6175515
  http://hydra.nixos.org/build/6611038

It turns out that this is because the binary cache substituter somehow
ends up loading two versions of SQLite: the one in Nixpkgs and the
other from /usr/lib/libsqlite3.dylib.  It's not exactly clear why the
latter is loaded, but it appears to be because WWW::Curl indirectly loads
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation,
which in turn seems to load /usr/lib/libsqlite3.dylib.  This leads to
a segfault when Perl exits:

  #0  0x00000001010375f4 in sqlite3_finalize ()
  #1  0x000000010125806e in sqlite_st_destroy ()
  #2  0x000000010124bc30 in XS_DBD__SQLite__st_DESTROY ()
  #3  0x00000001001c8155 in XS_DBI_dispatch ()
  ...
  #14 0x0000000100023224 in perl_destruct ()
  #15 0x0000000100000d6a in main ()
  ...

The workaround is to explicitly load DBD::SQLite before WWW::Curl.
2013-10-24 19:15:52 +02:00
Eelco Dolstra 5116214343 Add support for uncompressed NARs in binary caches
Issue NixOS/hydra#102.
2013-07-01 21:03:14 +02:00
Eelco Dolstra 22144afa8d Don't keep "disabled" substituters running
For instance, it's pointless to keep copy-from-other-stores running if
there are no other stores, or download-using-manifests if there are no
manifests.  This also speeds things up because we don't send queries
to those substituters.
2013-06-20 11:55:15 +02:00
Eelco Dolstra 5c06e5297d download-from-binary-cache.pl: Respect $NIX_CONNECT_TIMEOUT 2013-06-07 16:25:12 +02:00
Eelco Dolstra 24a356bf71 Replace $NIX_DEBUG_SUBST with an option ‘debug-subst’
Thus passing ‘--option debug-subst 1’ allows daemon users to turn on
debug info and see what the substituter is doing.
2013-06-07 15:36:36 +02:00
Eelco Dolstra 24e063efdc download-from-binary-cache.pl: Show if we're waiting for a URL
Previously, if a binary cache is hanging/unreachable/slow,
download-from-binary-cache.pl would also hang without any indication
to the user.  Now, if fetching a URL takes more than 5 seconds, it
will print a message to that effect.
2013-06-07 15:33:44 +02:00
Eelco Dolstra 75e12b8e66 download-from-binary-cache.pl: Fix race condition
Fixes the error "DBD::SQLite::db do failed: column url is not unique".
2013-06-05 16:25:07 +02:00
Eelco Dolstra f0576d6775 Update the default binary cache URL to cache.nixos.org 2013-06-05 13:36:43 +02:00
Eelco Dolstra ff08306746 download-from-binary-cache.pl: Treat a 403 error as a 404
Amazon S3 returns HTTP status code 403 if a file doesn't exist and the
user has no permission to list the contents of the bucket.  So treat
it as 404 (meaning it's cached in the NARExistence table).
2013-06-04 15:20:37 +02:00
Eelco Dolstra a4cb62ac25 download-from-binary-cache.pl: Get rid of an uninitialized value warning
Reported by Pablo Costa.
2013-05-15 15:47:05 +02:00
Eelco Dolstra ea019e9a26 Add option ‘extra-binary-caches’
This allows providing additional binary caches, useful in scripts like
Hydra's build reproduction scripts, in particular because untrusted
caches are ignored.
2013-05-07 15:37:28 +02:00
Eelco Dolstra 08d96ffad0 Fix --fallback with the binary cache substituter
Reported by Peter Simons.
2013-04-23 12:45:01 +02:00
Eelco Dolstra c642441beb Test NAR info caching 2013-04-23 12:43:28 +02:00
Eelco Dolstra aebea2e489 Reinstate the http://nixos.org/binary-cache default for the binary-caches setting 2013-01-02 22:16:37 +01:00
Eelco Dolstra 68dcbb187e download-from-binary-cache: Use the channels of the calling user rather than root
This should make live easier for single-user (non-daemon)
installations.  Note that when the daemon is used, the "calling user"
is root so we're not using any untrusted caches.
2012-12-21 15:00:07 +01:00
Eelco Dolstra 8cc19ed089 Set a long SQLite timeout in the binary cache substituter 2012-12-06 11:43:34 +01:00
Eelco Dolstra e28b683324 download-from-binary-cache: Try next cache if downloading a NAR fails 2012-11-06 17:55:53 +01:00
Eelco Dolstra bbc107ef1e Process binary caches in order of priority
Binary caches can now specify a priority in their nix-cache-info file.
The binary cache substituter checks caches in order of priority.  This
is to ensure that fast, static caches like nixos.org/binary-cache are
processed before slow, dynamic caches like hydra.nixos.org.
2012-11-06 17:45:20 +01:00
Eelco Dolstra 620e92e880 Add an option ‘use-binary-caches’
This allows disabling the use of binary caches, e.g.

  $ nix-build ... --option use-binary-caches false

Note that

  $ nix-build ... --option binary-caches ''

does not disable all binary caches, since the caches defined by
channels will still be used.
2012-11-06 13:44:59 +01:00
Eelco Dolstra df27db712d Fix "Not an ARRAY reference" error
It's a mystery why this error is not triggered in the build farm
(e.g. http://hydra.nixos.org/build/3265602).  Ah well.
2012-11-06 13:43:46 +01:00
Eelco Dolstra 167e36a5c3 nix-push: Only generate and copy a NAR if it doesn't already exist
This prevents unnecessary and slow rebuilds of NARs that already exist
in the binary cache.
2012-10-17 16:58:05 -04:00
Eelco Dolstra 600daf972f download-from-binary-cache: Remove duplicate entries in trustedURLs 2012-10-11 14:21:22 -04:00
Eelco Dolstra 9958bd6992 Add operation ‘nix-store --repair-path’
This operation allows fixing corrupted or accidentally deleted store
paths by redownloading them using substituters, if available.

Since the corrupted path cannot be replaced atomically, there is a
very small time window (one system call) during which neither the old
(corrupted) nor the new (repaired) contents are available.  So
repairing should be used with some care on critical packages like
Glibc.
2012-10-02 14:08:59 -04:00
Eelco Dolstra c1f91570b3 Style fix 2012-09-21 15:02:33 -04:00
Eelco Dolstra 47ae3ce2ca Put a time-to-live on negative binary cache lookups
Negative lookups are purged from the DB after a day, at most once per
day.  However, for non-"have" lookups (e.g. all except "nix-env
-qas"), negative lookups are ignored after one hour.  This is to
ensure that you don't have to wait a day for an operation like
"nix-env -i" to start using new binaries in the cache.

Should probably make this configurable.
2012-09-13 16:39:16 -04:00
Eelco Dolstra e03a8a1c92 Doh 2012-09-13 14:58:56 -04:00
Eelco Dolstra b14717ab90 Delete manifests in "nix-channel --remove" or when a binary cache is available 2012-09-13 11:35:46 -04:00
Eelco Dolstra 6762424e24 Compatibility fix for WWW::Curl < 4.14
Older versions of WWW::Curl don't support scalar references for
CURLOPT_WRITEDATA directly.

http://hydra.nixos.org/build/3017188
2012-09-12 10:41:17 -04:00