Commit graph

3035 commits

Author SHA1 Message Date
Eelco Dolstra fe241ece29 Merge branch 'master' into no-manifests 2012-07-18 10:47:59 -04:00
Eelco Dolstra ccc52adfb2 Add function queryPathFromHashPart()
To implement binary caches efficiently, Hydra needs to be able to map
the hash part of a store path (e.g. "gbg...zr7") to the full store
path (e.g. "/nix/store/gbg...kzr7-subversion-1.7.5").  (The binary
cache mechanism uses hash parts as a key for looking up store paths to
ensure privacy.)  However, doing a search in the Nix store for
/nix/store/<hash>* is expensive since it requires reading the entire
directory.  queryPathFromHashPart() prevents this by doing a cheap
database lookup.
2012-07-17 18:55:39 -04:00
Eelco Dolstra 220818f758 queryPathInfo(): return hash in base-32 if desired
Cherry-picked from the no-manifests branch.
2012-07-17 16:55:45 -04:00
Eelco Dolstra a6f348599c Print some debug output 2012-07-17 16:19:40 -04:00
Eelco Dolstra 3a9fdf2747 Return an exit code of 100 for cached failed builds
Exit code 100 should be returned for all permanent failures.  This
includes cached failures.

Fixes #34.
2012-07-17 15:55:30 -04:00
Eelco Dolstra 1217204c81 Remove dead code 2012-07-17 14:07:52 -04:00
Eelco Dolstra 51d71ad3d7 Manual: Don't claim we support Cygwin 2012-07-17 11:49:47 -04:00
Eelco Dolstra 6c01fb4d68 Update Nix 1.1 release notes 2012-07-17 10:06:20 -04:00
Eelco Dolstra 53b24f3518 Allow disabling log compression 2012-07-17 09:40:12 -04:00
Eelco Dolstra a7a43adb79 builtins.storePath: resolve symlinks
Needed for Charon/Hydra interaction.
2012-07-12 18:25:01 -04:00
Eelco Dolstra 04559a0d45 Merge branch 'master' of github.com:NixOS/nix into no-manifests 2012-07-11 18:53:27 -04:00
Eelco Dolstra e4d6bcb6cd Update release notes 2012-07-11 18:52:23 -04:00
Eelco Dolstra f2bdc87595 Update the other substituters 2012-07-11 18:52:09 -04:00
Eelco Dolstra 15c15da482 Add some missing --version switches 2012-07-11 18:07:41 -04:00
Eelco Dolstra d287b62b64 Set the User-Agent header to "Nix/<version>" 2012-07-11 18:05:30 -04:00
Eelco Dolstra b74d92755d download-from-binary-cache: Use HEAD requests if possible
In "nix-env -qas", we don't need the substitute info, we just need to
know if it exists.  This can be done using a HTTP HEAD request, which
saves bandwidth.

Note however that curl currently has a bug that prevents it from
reusing HTTP connections if HEAD requests return a 404:

https://sourceforge.net/tracker/?func=detail&aid=3542731&group_id=976&atid=100976

Without the patch attached to the issue, using HEAD is actually quite
a bit slower than GET.
2012-07-11 17:53:20 -04:00
Eelco Dolstra 09a6321aeb Replace hasSubstitutes() with querySubstitutablePaths()
querySubstitutablePaths() takes a set of paths, so this greatly
reduces daemon <-> client latency.
2012-07-11 17:52:18 -04:00
Eelco Dolstra 58ef4d9a95 Add a function queryValidPaths()
queryValidPaths() combines multiple calls to isValidPath() in one.
This matters when using the Nix daemon because it reduces latency.
For instance, on "nix-env -qas \*" it reduces execution time from 5.7s
to 4.7s (which is indistinguishable from the non-daemon case).
2012-07-11 11:08:47 -04:00
Eelco Dolstra 667d5f1936 Rename queryValidPaths() to queryAllValidPaths() 2012-07-11 10:49:04 -04:00
Eelco Dolstra eb3036da87 Implement querySubstitutablePathInfos() in the daemon
Also removed querySubstitutablePathInfo().
2012-07-11 10:43:24 -04:00
Eelco Dolstra 6586414bc7 nix-env: Determine which paths have substitutes in parallel 2012-07-11 10:14:06 -04:00
Eelco Dolstra 5ee8944155 Cleanup 2012-07-11 10:13:16 -04:00
Eelco Dolstra eae802459d Pass --insecure to curl so that https works 2012-07-09 15:49:20 -04:00
Eelco Dolstra 2dd3117c27 Inline fetchurl.sh 2012-07-09 15:48:55 -04:00
Eelco Dolstra 51f9f9924b Add a test for the fetchurl function 2012-07-09 15:41:43 -04:00
Shea Levy 035aa11403 Remove obsolete comment 2012-07-09 15:29:49 -04:00
Shea Levy a2865f6b3d corepkgs/fetchurl: Build locally and outside of the chroot 2012-07-09 15:29:49 -04:00
Shea Levy 53f52c2111 corepkgs/fetchurl: the 'system' argument can be optional 2012-07-09 15:29:49 -04:00
Shea Levy 543bf742c9 corepkgs: distribute fetchurl files 2012-07-09 15:29:49 -04:00
Shea Levy f863673a90 corepkgs/fetchurl: Call the shell directly instead of using the shebang 2012-07-09 15:29:49 -04:00
Shea Levy a994eb92a4 corepkgs/fetchurl.sh: Use config.nix's curl 2012-07-09 15:29:49 -04:00
Shea Levy 9d94a28bed The fetchurl builder is now fetchurl.sh 2012-07-09 15:29:49 -04:00
Shea Levy fd2630e1f7 Remove old fetchurl makefile 2012-07-09 15:29:49 -04:00
Shea Levy 6450f5699f Move fetchurl files out of their subdirectory 2012-07-09 15:29:49 -04:00
Shea Levy 40c01ec467 corepkgs/config.nix.in: We'll need curl 2012-07-09 15:29:49 -04:00
Shea Levy c4df747267 Resurrect old corepkgs fetchurl 2012-07-09 15:29:49 -04:00
Eelco Dolstra 27f0c34390 Really fix RPM builds 2012-07-09 13:16:09 -04:00
Eelco Dolstra ae72be1b8b Add WWW::Curl as a dependency 2012-07-09 13:11:37 -04:00
Eelco Dolstra a560124cdf Fix RPM builds
http://hydra.nixos.org/build/2784908
2012-07-09 11:58:12 -04:00
Eelco Dolstra 087dee6e1b Get rid of nix.conf.example
No need to duplicate the nix.conf manpage.
2012-07-09 11:56:55 -04:00
Eelco Dolstra 5755a5c354 Install a nix.conf manpage 2012-07-09 11:33:38 -04:00
Eelco Dolstra 099125435f download-from-binary-cache: add nix.conf options 2012-07-09 10:57:28 -04:00
Eelco Dolstra 98a423b75a prim_import(): prefetch substitute info in parallel using queryMissing() 2012-07-09 09:59:34 -04:00
Eelco Dolstra 425cc612ad build.cc: Don't use hasSubstitute()
Instead make a single call to querySubstitutablePathInfo() per
derivation output.  This is faster and prevents having to implement
the "have" function in the binary cache substituter.
2012-07-08 18:39:24 -04:00
Eelco Dolstra 400e556b34 Cleanup 2012-07-08 18:39:07 -04:00
Eelco Dolstra 11800e6198 download-from-binary-cache: parallelise fetching of NAR info files
Getting substitute information using the binary cache substituter has
non-trivial latency overhead.  A package or NixOS system configuration
can have hundreds of dependencies, and in the worst case (when the
local info cache is empty) we have to do a separate HTTP request for
each of these.  If the ping time to the server is t, getting N info
files will take tN seconds; e.g., with a ping time of 0.1s to
nixos.org, sequentially downloading 1000 info files (a typical NixOS
config) will take at least 100 seconds.

To fix this problem, the binary cache substituter can now perform
requests in parallel.  This required changing the substituter
interface to support a function querySubstitutablePathInfos() that
queries multiple paths at the same time, and rewriting queryMissing()
to take advantage of parallelism.  (Due to local caching,
parallelising queryMissing() is sufficient for most use cases, since
it's almost always called before building a derivation and thus fills
the local info cache.)

For example, parallelism speeds up querying all 1056 paths in a
particular NixOS system configuration from 116s to 2.6s.  It works so
well because the eccentricity of the top-level derivation in the
dependency graph is only 9.  So we only need 10 round-trips (when
using an unlimited number of parallel connections) to get everything.

Currently we do a maximum of 150 parallel connections to the server.
Thus it's important that the binary cache server (e.g. nixos.org) has
a high connection limit.  Alternatively we could use HTTP pipelining,
but WWW::Curl doesn't support it and libcurl has a hard-coded limit of
5 requests per pipeline.
2012-07-06 19:08:20 -04:00
Eelco Dolstra cd94665f38 download-from-binary-cache: use WWW::Curl
Using WWW::Curl rather than running an external curl process for every
NAR info file halves the time it takes to get info thanks to libcurl's
support for persistent HTTP connections.  (We save a roundtrip per
file.)  But the real gain will come from using parallel and/or
pipelined requests.
2012-07-06 00:30:40 -04:00
Eelco Dolstra ae60643c15 download-from-binary-cache: do negative NAR info caching
I.e. if a NAR info file does *not* exist, we record it in the cache DB
so that we don't retry it later.
2012-07-03 18:54:46 -04:00
Eelco Dolstra 89380c03e9 download-from-binary-cache: in queries, preferred cached info 2012-07-03 18:35:39 -04:00
Eelco Dolstra 2a8e5c8b11 download-from-binary-cache: strip trailing / from URLs 2012-07-03 17:47:01 -04:00