Commit graph

53 commits

Author SHA1 Message Date
Eelco Dolstra f0576d6775 Update the default binary cache URL to cache.nixos.org 2013-06-05 13:36:43 +02:00
Eelco Dolstra ff08306746 download-from-binary-cache.pl: Treat a 403 error as a 404
Amazon S3 returns HTTP status code 403 if a file doesn't exist and the
user has no permission to list the contents of the bucket.  So treat
it as 404 (meaning it's cached in the NARExistence table).
2013-06-04 15:20:37 +02:00
Eelco Dolstra a4cb62ac25 download-from-binary-cache.pl: Get rid of an uninitialized value warning
Reported by Pablo Costa.
2013-05-15 15:47:05 +02:00
Eelco Dolstra ea019e9a26 Add option ‘extra-binary-caches’
This allows providing additional binary caches, useful in scripts like
Hydra's build reproduction scripts, in particular because untrusted
caches are ignored.
2013-05-07 15:37:28 +02:00
Eelco Dolstra 08d96ffad0 Fix --fallback with the binary cache substituter
Reported by Peter Simons.
2013-04-23 12:45:01 +02:00
Eelco Dolstra c642441beb Test NAR info caching 2013-04-23 12:43:28 +02:00
Eelco Dolstra aebea2e489 Reinstate the http://nixos.org/binary-cache default for the binary-caches setting 2013-01-02 22:16:37 +01:00
Eelco Dolstra 68dcbb187e download-from-binary-cache: Use the channels of the calling user rather than root
This should make live easier for single-user (non-daemon)
installations.  Note that when the daemon is used, the "calling user"
is root so we're not using any untrusted caches.
2012-12-21 15:00:07 +01:00
Eelco Dolstra 8cc19ed089 Set a long SQLite timeout in the binary cache substituter 2012-12-06 11:43:34 +01:00
Eelco Dolstra e28b683324 download-from-binary-cache: Try next cache if downloading a NAR fails 2012-11-06 17:55:53 +01:00
Eelco Dolstra bbc107ef1e Process binary caches in order of priority
Binary caches can now specify a priority in their nix-cache-info file.
The binary cache substituter checks caches in order of priority.  This
is to ensure that fast, static caches like nixos.org/binary-cache are
processed before slow, dynamic caches like hydra.nixos.org.
2012-11-06 17:45:20 +01:00
Eelco Dolstra 620e92e880 Add an option ‘use-binary-caches’
This allows disabling the use of binary caches, e.g.

  $ nix-build ... --option use-binary-caches false

Note that

  $ nix-build ... --option binary-caches ''

does not disable all binary caches, since the caches defined by
channels will still be used.
2012-11-06 13:44:59 +01:00
Eelco Dolstra df27db712d Fix "Not an ARRAY reference" error
It's a mystery why this error is not triggered in the build farm
(e.g. http://hydra.nixos.org/build/3265602).  Ah well.
2012-11-06 13:43:46 +01:00
Eelco Dolstra 167e36a5c3 nix-push: Only generate and copy a NAR if it doesn't already exist
This prevents unnecessary and slow rebuilds of NARs that already exist
in the binary cache.
2012-10-17 16:58:05 -04:00
Eelco Dolstra 600daf972f download-from-binary-cache: Remove duplicate entries in trustedURLs 2012-10-11 14:21:22 -04:00
Eelco Dolstra 9958bd6992 Add operation ‘nix-store --repair-path’
This operation allows fixing corrupted or accidentally deleted store
paths by redownloading them using substituters, if available.

Since the corrupted path cannot be replaced atomically, there is a
very small time window (one system call) during which neither the old
(corrupted) nor the new (repaired) contents are available.  So
repairing should be used with some care on critical packages like
Glibc.
2012-10-02 14:08:59 -04:00
Eelco Dolstra c1f91570b3 Style fix 2012-09-21 15:02:33 -04:00
Eelco Dolstra 47ae3ce2ca Put a time-to-live on negative binary cache lookups
Negative lookups are purged from the DB after a day, at most once per
day.  However, for non-"have" lookups (e.g. all except "nix-env
-qas"), negative lookups are ignored after one hour.  This is to
ensure that you don't have to wait a day for an operation like
"nix-env -i" to start using new binaries in the cache.

Should probably make this configurable.
2012-09-13 16:39:16 -04:00
Eelco Dolstra e03a8a1c92 Doh 2012-09-13 14:58:56 -04:00
Eelco Dolstra b14717ab90 Delete manifests in "nix-channel --remove" or when a binary cache is available 2012-09-13 11:35:46 -04:00
Eelco Dolstra 6762424e24 Compatibility fix for WWW::Curl < 4.14
Older versions of WWW::Curl don't support scalar references for
CURLOPT_WRITEDATA directly.

http://hydra.nixos.org/build/3017188
2012-09-12 10:41:17 -04:00
Eelco Dolstra 9e2fc6951c Disable the binary cache substituter by default for now 2012-08-27 14:20:14 -04:00
Eelco Dolstra 8b8fe6139e Drop dependency on List::MoreUtils 2012-08-27 11:28:34 -04:00
Eelco Dolstra 5170c5691a nix-channel: Use binary caches advertised by channels
Channels can now advertise a binary cache by creating a file
<channel-url>/binary-cache-url.  The channel unpacker puts these in
its "binary-caches" subdirectory.  Thus, the URLS of the binary caches
for the channels added by root appear in
/nix/var/nix/profiles/per-user/eelco/channels/binary-caches/*.  The
binary cache substituter reads these and adds them to the list of
binary caches.
2012-08-01 17:56:11 -04:00
Eelco Dolstra 8a25d787d7 download-from-binary-cache: Remove duplicate URLs 2012-08-01 11:33:15 -04:00
Eelco Dolstra c770a2422a Report substituter errors to clients of the Nix daemon 2012-08-01 11:19:24 -04:00
Eelco Dolstra 4d1b64f118 Allow daemon users to override ‘binary-caches’
For security reasons, daemon users can only specify caches that appear
in the ‘binary-caches’ and ‘trusted-binary-caches’ options in
nix.conf.
2012-07-31 18:56:22 -04:00
Eelco Dolstra 9cd63d2244 Do some validation of URLs 2012-07-30 17:09:36 -04:00
Eelco Dolstra 9de6d10d11 Get rid of $NIX_BINARY_CACHES
You can use ‘--option binary-caches URLs’ instead.
2012-07-30 16:39:31 -04:00
Eelco Dolstra 66a3ac6a56 Allow a binary cache to declare that it doesn't support "nix-env -qas"
Querying all substitutable paths via "nix-env -qas" is potentially
hard on a server, since it involves sending thousands of HEAD
requests.  So a binary cache must now have a meta-info file named
"nix-cache-info" that specifies whether the server wants this.  It
also specifies the store prefix so that we don't send useless queries
to a binary cache for a different store prefix.
2012-07-27 18:16:05 -04:00
Eelco Dolstra 73acb8b836 Let build.cc verify the expected hash of a substituter's output
Since SubstitutionGoal::finished() in build.cc computes the hash
anyway, we can prevent the inefficiency of computing the hash twice by
letting the substituter tell Nix about the expected hash, which can
then verify it.
2012-07-27 12:16:02 -04:00
Eelco Dolstra 3a8f841612 download-using-manifests: Don't use nix-prefetch-url
Instead call curl directly and pipe it into ‘nix-store --restore’.
This saves I/O and prevents creating garbage in the Nix store.
2012-07-27 10:47:36 -04:00
Eelco Dolstra b4ea83249b Remove trailing whitespace / tabs 2012-07-27 09:59:18 -04:00
Eelco Dolstra 7f8e805c8e download-from-binary-cache: Only use the default cache for /nix/store 2012-07-27 09:57:42 -04:00
Eelco Dolstra 7892ad15ab download-from-binary-cache: Support file://
The file:// URI schema requires checking for errors in a more general
way.  Also, don't cache file:// lookups.
2012-07-26 17:11:11 -04:00
Eelco Dolstra aa115e22df download-from-binary-cache: Print correct URL 2012-07-18 11:01:17 -04:00
Eelco Dolstra a6f348599c Print some debug output 2012-07-17 16:19:40 -04:00
Eelco Dolstra d287b62b64 Set the User-Agent header to "Nix/<version>" 2012-07-11 18:05:30 -04:00
Eelco Dolstra b74d92755d download-from-binary-cache: Use HEAD requests if possible
In "nix-env -qas", we don't need the substitute info, we just need to
know if it exists.  This can be done using a HTTP HEAD request, which
saves bandwidth.

Note however that curl currently has a bug that prevents it from
reusing HTTP connections if HEAD requests return a 404:

https://sourceforge.net/tracker/?func=detail&aid=3542731&group_id=976&atid=100976

Without the patch attached to the issue, using HEAD is actually quite
a bit slower than GET.
2012-07-11 17:53:20 -04:00
Eelco Dolstra 5ee8944155 Cleanup 2012-07-11 10:13:16 -04:00
Eelco Dolstra 099125435f download-from-binary-cache: add nix.conf options 2012-07-09 10:57:28 -04:00
Eelco Dolstra 400e556b34 Cleanup 2012-07-08 18:39:07 -04:00
Eelco Dolstra 11800e6198 download-from-binary-cache: parallelise fetching of NAR info files
Getting substitute information using the binary cache substituter has
non-trivial latency overhead.  A package or NixOS system configuration
can have hundreds of dependencies, and in the worst case (when the
local info cache is empty) we have to do a separate HTTP request for
each of these.  If the ping time to the server is t, getting N info
files will take tN seconds; e.g., with a ping time of 0.1s to
nixos.org, sequentially downloading 1000 info files (a typical NixOS
config) will take at least 100 seconds.

To fix this problem, the binary cache substituter can now perform
requests in parallel.  This required changing the substituter
interface to support a function querySubstitutablePathInfos() that
queries multiple paths at the same time, and rewriting queryMissing()
to take advantage of parallelism.  (Due to local caching,
parallelising queryMissing() is sufficient for most use cases, since
it's almost always called before building a derivation and thus fills
the local info cache.)

For example, parallelism speeds up querying all 1056 paths in a
particular NixOS system configuration from 116s to 2.6s.  It works so
well because the eccentricity of the top-level derivation in the
dependency graph is only 9.  So we only need 10 round-trips (when
using an unlimited number of parallel connections) to get everything.

Currently we do a maximum of 150 parallel connections to the server.
Thus it's important that the binary cache server (e.g. nixos.org) has
a high connection limit.  Alternatively we could use HTTP pipelining,
but WWW::Curl doesn't support it and libcurl has a hard-coded limit of
5 requests per pipeline.
2012-07-06 19:08:20 -04:00
Eelco Dolstra cd94665f38 download-from-binary-cache: use WWW::Curl
Using WWW::Curl rather than running an external curl process for every
NAR info file halves the time it takes to get info thanks to libcurl's
support for persistent HTTP connections.  (We save a roundtrip per
file.)  But the real gain will come from using parallel and/or
pipelined requests.
2012-07-06 00:30:40 -04:00
Eelco Dolstra ae60643c15 download-from-binary-cache: do negative NAR info caching
I.e. if a NAR info file does *not* exist, we record it in the cache DB
so that we don't retry it later.
2012-07-03 18:54:46 -04:00
Eelco Dolstra 89380c03e9 download-from-binary-cache: in queries, preferred cached info 2012-07-03 18:35:39 -04:00
Eelco Dolstra 2a8e5c8b11 download-from-binary-cache: strip trailing / from URLs 2012-07-03 17:47:01 -04:00
Eelco Dolstra d694c599e2 download-from-binary-cache: cache binary cache info in a SQLite DB 2012-07-03 17:29:33 -04:00
Eelco Dolstra 8319b1ab9f download-from-binary-cache: Verify NAR hashes 2012-07-02 18:53:04 -04:00
Eelco Dolstra 6ec7460af1 Binary caches: use a better key
Use the hash part of the store path as a key rather than a hash of the
store path.  This is enough to get the desired privacy property.
2012-07-02 12:42:58 -04:00