Commit graph

612 commits

Author SHA1 Message Date
Eelco Dolstra 75e12b8e66 download-from-binary-cache.pl: Fix race condition
Fixes the error "DBD::SQLite::db do failed: column url is not unique".
2013-06-05 16:25:07 +02:00
Eelco Dolstra f0576d6775 Update the default binary cache URL to cache.nixos.org 2013-06-05 13:36:43 +02:00
Eelco Dolstra ff08306746 download-from-binary-cache.pl: Treat a 403 error as a 404
Amazon S3 returns HTTP status code 403 if a file doesn't exist and the
user has no permission to list the contents of the bucket.  So treat
it as 404 (meaning it's cached in the NARExistence table).
2013-06-04 15:20:37 +02:00
Eelco Dolstra bc2e43f3c8 build-remote.pl: Remove meaningless signing when importing the output paths
The "$UID != 0" makes no sense: if the local side has write access to
the Nix store (which is always the case) then it doesn't matter if
we're root - we can import unsigned paths either way.
2013-05-29 18:18:59 +02:00
Eelco Dolstra 107505e13a build-remote.pl: Copy all outputs in one operation 2013-05-23 14:39:58 -04:00
Eelco Dolstra 72d8209548 build-remote.pl: Indicate if remote machine is refusing builds
Fixes #120.
2013-05-23 14:04:46 -04:00
Eelco Dolstra a4cb62ac25 download-from-binary-cache.pl: Get rid of an uninitialized value warning
Reported by Pablo Costa.
2013-05-15 15:47:05 +02:00
Eelco Dolstra 3a0cc43ac8 build-remote.pl: Properly close the SSH connection between attempts 2013-05-10 02:38:50 +02:00
Eelco Dolstra be0b9dda31 build-remote.pl: Pass /dev/null as SSH's stdin
Otherwise it will set the parent's stdin to non-blocking mode, causing
the subsequent read of the set of inputs/outputs to fail randomly.
That's insane.
2013-05-10 02:32:13 +02:00
Eelco Dolstra 78206f06ec build-remote.pl: Allow a machine to refuse a build
Before selecting a machine, build-remote.pl will try to run the
command "nix-builds-inhibited" on the machine.  If this command exists
and returns a 0 exit code, then the machine won't be used.  It's up to
the user to provide this command, but it would typically be a script
that checks whether there is enough disk space and whether the load is
not too high.
2013-05-10 01:09:46 +02:00
Eelco Dolstra 7a03cbf09d build-remote.pl: Create one process fewer on the remote side 2013-05-09 17:30:07 +02:00
Eelco Dolstra 69b8f9980f build-remote.pl: Enforce timeouts locally
Don't pass --timeout / --max-silent-time to the remote builder.
Instead, let the local Nix process terminate the build if it exceeds a
timeout.  The remote builder will be killed as a side-effect.  This
gives better error reporting (since the timeout message from the
remote side wasn't properly propagated) and handles non-Nix problems
like SSH hangs.
2013-05-09 17:17:17 +02:00
Eelco Dolstra ea019e9a26 Add option ‘extra-binary-caches’
This allows providing additional binary caches, useful in scripts like
Hydra's build reproduction scripts, in particular because untrusted
caches are ignored.
2013-05-07 15:37:28 +02:00
Eelco Dolstra c51b6a893c nix-copy-closure: Show a proper error message if no host name is given 2013-05-03 11:12:11 +02:00
Lluís Batlle i Rossell e6c44d166a Fixing the pv position regarding compression
Problem noticed by niksnut.
2013-05-03 11:08:51 +02:00
Lluís Batlle i Rossell 7391533ea5 Fixing the pv reference; I didn't mean to change it 2013-05-03 11:08:51 +02:00
Lluís Batlle i Rossell 5cc2fc46ec Adding ETA support to the --show-progress in nix-copy-closure
Based on https://github.com/NixOS/nix/pull/6 from shlevy
2013-05-03 11:08:51 +02:00
Eelco Dolstra 4ddd077bfa find-runtime-roots.pl: Don't hardcode /nix/store 2013-04-26 12:15:54 +02:00
Eelco Dolstra 00f698eb8b find-runtime-roots.pl: Search process environments for roots
For instance, this prevents paths from being deleted that are in use
by a "nix-build --run-env" session.
2013-04-26 12:06:39 +02:00
Eelco Dolstra 938092a213 find-runtime-roots.pl: Use Nix::Utils::readFile 2013-04-26 11:44:19 +02:00
Eelco Dolstra 6955d41f2b nix-build: Respect --timeout 2013-04-23 17:16:01 +02:00
Eelco Dolstra 08d96ffad0 Fix --fallback with the binary cache substituter
Reported by Peter Simons.
2013-04-23 12:45:01 +02:00
Eelco Dolstra c642441beb Test NAR info caching 2013-04-23 12:43:28 +02:00
Eelco Dolstra bdd4646338 Revert "Prevent config.h from being clobbered"
This reverts commit 28bba8c44f.
2013-03-08 01:24:59 +01:00
Eelco Dolstra 28bba8c44f Prevent config.h from being clobbered 2013-03-07 23:55:55 +01:00
Eelco Dolstra dadf7a5b46 build-remote: Use the --quiet flag
‘--option verbosity 0’ doesn't actually do anything.
2013-02-19 16:56:31 +01:00
Eelco Dolstra 96fbbbde55 build-remote.pl: Don't keep build logs on the build slave 2013-01-21 22:26:25 +01:00
Eelco Dolstra 0a4e90395c Urgggh
http://hydra.nixos.org/build/3661100
2013-01-02 23:52:15 +01:00
Eelco Dolstra aebea2e489 Reinstate the http://nixos.org/binary-cache default for the binary-caches setting 2013-01-02 22:16:37 +01:00
Eelco Dolstra 68dcbb187e download-from-binary-cache: Use the channels of the calling user rather than root
This should make live easier for single-user (non-daemon)
installations.  Note that when the daemon is used, the "calling user"
is root so we're not using any untrusted caches.
2012-12-21 15:00:07 +01:00
Eelco Dolstra 8cc19ed089 Set a long SQLite timeout in the binary cache substituter 2012-12-06 11:43:34 +01:00
Eelco Dolstra 566afa1e9c Support xz compression of nixexprs.tar in channels 2012-12-05 11:02:44 +01:00
Eelco Dolstra 08964d7328 Undo accidental debug change 2012-11-26 17:57:14 +01:00
Eelco Dolstra 46a369ad95 Make "nix-build -A <derivation>.<output>" do the right thing
For example, given a derivation with outputs "out", "man" and "bin":

  $ nix-build -A pkg

produces ./result pointing to the "out" output;

  $ nix-build -A pkg.man

produces ./result-man pointing to the "man" output;

  $ nix-build -A pkg.all

produces ./result, ./result-man and ./result-bin;

  $ nix-build -A pkg.all -A pkg2

produces ./result, ./result-man, ./result-bin and ./result-2.
2012-11-26 15:39:10 +01:00
Eelco Dolstra a3d6585c5a nix-copy-closure: Add flag ‘--use-substitutes’ 2012-11-23 16:20:16 +01:00
Eelco Dolstra a9a8baaccb Use a shorter Nixpkgs channel URL 2012-11-09 13:33:35 +01:00
Eelco Dolstra e28b683324 download-from-binary-cache: Try next cache if downloading a NAR fails 2012-11-06 17:55:53 +01:00
Eelco Dolstra bbc107ef1e Process binary caches in order of priority
Binary caches can now specify a priority in their nix-cache-info file.
The binary cache substituter checks caches in order of priority.  This
is to ensure that fast, static caches like nixos.org/binary-cache are
processed before slow, dynamic caches like hydra.nixos.org.
2012-11-06 17:45:20 +01:00
Eelco Dolstra 82951e5582 nix-push: Handle pushing a symlink 2012-11-06 13:53:53 +01:00
Eelco Dolstra 620e92e880 Add an option ‘use-binary-caches’
This allows disabling the use of binary caches, e.g.

  $ nix-build ... --option use-binary-caches false

Note that

  $ nix-build ... --option binary-caches ''

does not disable all binary caches, since the caches defined by
channels will still be used.
2012-11-06 13:44:59 +01:00
Eelco Dolstra df27db712d Fix "Not an ARRAY reference" error
It's a mystery why this error is not triggered in the build farm
(e.g. http://hydra.nixos.org/build/3265602).  Ah well.
2012-11-06 13:43:46 +01:00
Eelco Dolstra a28b4445a4 nix-prefetch-url: Improve option handling 2012-10-23 18:04:11 +02:00
Eelco Dolstra dde6486eab nix-push: Add a flag ‘--manifest-path’ to write the manifest to another directory 2012-10-18 10:36:16 -04:00
Eelco Dolstra c8daeba303 nix-push: Add ‘--link’ flag
If ‘--link’ is given, nix-push will create hard links to the NAR files
in the store, rather than copying them.  This is faster and requires
less disk space.  However, it doesn't work if the store is on a
different file system.
2012-10-17 17:01:23 -04:00
Eelco Dolstra 167e36a5c3 nix-push: Only generate and copy a NAR if it doesn't already exist
This prevents unnecessary and slow rebuilds of NARs that already exist
in the binary cache.
2012-10-17 16:58:05 -04:00
Eelco Dolstra 600daf972f download-from-binary-cache: Remove duplicate entries in trustedURLs 2012-10-11 14:21:22 -04:00
Eelco Dolstra a562d544d8 When ‘--help’ is given, just run ‘man’ to show the manual page
I.e. do what git does.  I'm too lazy to keep the builtin help text up
to date :-)

Also add ‘--help’ to various commands that lacked it
(e.g. nix-collect-garbage).
2012-10-03 16:40:09 -04:00
Eelco Dolstra 2e90a5a2a7 nix-build: Support ‘--repair’ flag 2012-10-03 15:14:02 -04:00
Eelco Dolstra 9958bd6992 Add operation ‘nix-store --repair-path’
This operation allows fixing corrupted or accidentally deleted store
paths by redownloading them using substituters, if available.

Since the corrupted path cannot be replaced atomically, there is a
very small time window (one system call) during which neither the old
(corrupted) nor the new (repaired) contents are available.  So
repairing should be used with some care on critical packages like
Glibc.
2012-10-02 14:08:59 -04:00
Eelco Dolstra c1f91570b3 Style fix 2012-09-21 15:02:33 -04:00
Eelco Dolstra 20582e9ae3 Support xz compression in the download-using-manifests substituter 2012-09-19 17:33:42 -04:00
Eelco Dolstra 983220bcd4 nix-collect-garbage: Support --dry-run 2012-09-13 18:09:20 -04:00
Eelco Dolstra aac14222f5 nix-profile.sh: Revert to single-user version
Commit 6a214f3e06 copied most of the Nix
shell initialisation code from NixOS to nix-profile.sh; however, that
code assumes a multi-user install and is Linux-specific (e.g. it calls
the "stat" command).  So go back to the simple single-user version.

Fixes #49.
2012-09-13 17:48:19 -04:00
Eelco Dolstra 47ae3ce2ca Put a time-to-live on negative binary cache lookups
Negative lookups are purged from the DB after a day, at most once per
day.  However, for non-"have" lookups (e.g. all except "nix-env
-qas"), negative lookups are ignored after one hour.  This is to
ensure that you don't have to wait a day for an operation like
"nix-env -i" to start using new binaries in the cache.

Should probably make this configurable.
2012-09-13 16:39:16 -04:00
Eelco Dolstra e03a8a1c92 Doh 2012-09-13 14:58:56 -04:00
Eelco Dolstra c845c0ccad nix-channel: Add option to force fetching of manifests 2012-09-13 12:11:40 -04:00
Eelco Dolstra b14717ab90 Delete manifests in "nix-channel --remove" or when a binary cache is available 2012-09-13 11:35:46 -04:00
Eelco Dolstra 479e9172b3 Build hook: Pass the timeout to the remote builder
Note that this will only work if the client has a very recent Nix
version (post 15e1b2c223), otherwise the
--option flag will just be ignored.

Fixes #50.
2012-09-12 12:18:14 -04:00
Eelco Dolstra 6762424e24 Compatibility fix for WWW::Curl < 4.14
Older versions of WWW::Curl don't support scalar references for
CURLOPT_WRITEDATA directly.

http://hydra.nixos.org/build/3017188
2012-09-12 10:41:17 -04:00
Eelco Dolstra a2785b7391 Support building a derivation if some outputs are already valid
This handles the chroot and build hook cases, which are easy.
Supporting the non-chroot-build case will require more work (hash
rewriting!).

Issue #21.
2012-09-11 16:59:59 -04:00
Eelco Dolstra 9e2fc6951c Disable the binary cache substituter by default for now 2012-08-27 14:20:14 -04:00
Eelco Dolstra 8b8fe6139e Drop dependency on List::MoreUtils 2012-08-27 11:28:34 -04:00
Eelco Dolstra e94806d030 Merge branch 'master' into no-manifests 2012-08-27 11:09:07 -04:00
Eelco Dolstra a9e6752bbd Include the output name in the GC root link
Output names are now appended to resulting GC symlinks, e.g. by
nix-build.  For backwards compatibility, if the output is named "out",
nothing is appended.  E.g. doing "nix-build -A foo" on a derivation
that produces outputs "out", "bin" and "dev" will produce symlinks
"./result", "./result-bin" and "./result-dev", respectively.
2012-08-24 16:58:11 -04:00
Eelco Dolstra 5170c5691a nix-channel: Use binary caches advertised by channels
Channels can now advertise a binary cache by creating a file
<channel-url>/binary-cache-url.  The channel unpacker puts these in
its "binary-caches" subdirectory.  Thus, the URLS of the binary caches
for the channels added by root appear in
/nix/var/nix/profiles/per-user/eelco/channels/binary-caches/*.  The
binary cache substituter reads these and adds them to the list of
binary caches.
2012-08-01 17:56:11 -04:00
Eelco Dolstra 8a25d787d7 download-from-binary-cache: Remove duplicate URLs 2012-08-01 11:33:15 -04:00
Eelco Dolstra c770a2422a Report substituter errors to clients of the Nix daemon 2012-08-01 11:19:24 -04:00
Eelco Dolstra 4d1b64f118 Allow daemon users to override ‘binary-caches’
For security reasons, daemon users can only specify caches that appear
in the ‘binary-caches’ and ‘trusted-binary-caches’ options in
nix.conf.
2012-07-31 18:56:22 -04:00
Eelco Dolstra 9cd63d2244 Do some validation of URLs 2012-07-30 17:09:36 -04:00
Eelco Dolstra 9de6d10d11 Get rid of $NIX_BINARY_CACHES
You can use ‘--option binary-caches URLs’ instead.
2012-07-30 16:39:31 -04:00
Eelco Dolstra ab42bf1dab nix-install-package: Support binary caches
The .nixpkg file format is extended to optionally include the URL of a
binary cache, which will be used in preference to the manifest URL
(which can be set to a non-existent value).
2012-07-30 16:11:02 -04:00
Eelco Dolstra 66a3ac6a56 Allow a binary cache to declare that it doesn't support "nix-env -qas"
Querying all substitutable paths via "nix-env -qas" is potentially
hard on a server, since it involves sending thousands of HEAD
requests.  So a binary cache must now have a meta-info file named
"nix-cache-info" that specifies whether the server wants this.  It
also specifies the store prefix so that we don't send useless queries
to a binary cache for a different store prefix.
2012-07-27 18:16:05 -04:00
Eelco Dolstra 73acb8b836 Let build.cc verify the expected hash of a substituter's output
Since SubstitutionGoal::finished() in build.cc computes the hash
anyway, we can prevent the inefficiency of computing the hash twice by
letting the substituter tell Nix about the expected hash, which can
then verify it.
2012-07-27 12:16:02 -04:00
Eelco Dolstra 3a8f841612 download-using-manifests: Don't use nix-prefetch-url
Instead call curl directly and pipe it into ‘nix-store --restore’.
This saves I/O and prevents creating garbage in the Nix store.
2012-07-27 10:47:36 -04:00
Eelco Dolstra b4ea83249b Remove trailing whitespace / tabs 2012-07-27 09:59:18 -04:00
Eelco Dolstra 7f8e805c8e download-from-binary-cache: Only use the default cache for /nix/store 2012-07-27 09:57:42 -04:00
Eelco Dolstra 67c6f3eded nix-push: Support generating a manifest again
This makes all the tests succeed.  Woohoo!
2012-07-26 18:28:12 -04:00
Eelco Dolstra e3ce54ccee nix-push: Remove the upload feature 2012-07-26 17:13:38 -04:00
Eelco Dolstra 7892ad15ab download-from-binary-cache: Support file://
The file:// URI schema requires checking for errors in a more general
way.  Also, don't cache file:// lookups.
2012-07-26 17:11:11 -04:00
Eelco Dolstra 8c79100839 Merge branch 'master' into no-manifests 2012-07-26 15:14:33 -04:00
Eelco Dolstra 2605f4f4e6 nix-profile.sh: Don't set NIX_REMOTE on single user installations
Commit 6a214f3e06 reused the NixOS
environment initialisation for nix-profile.sh, but this is
inappropriate on systems that don't have multi-user support enabled.
2012-07-25 17:06:09 -04:00
Eelco Dolstra 477b0fbeca Subscribe to the Nixpkgs rather than NixOS channel 2012-07-25 16:56:56 -04:00
Eelco Dolstra ed59bf7a18 nix-build: Support the ‘-’ argument to build an expression from stdin 2012-07-23 17:11:12 -04:00
Eelco Dolstra aa115e22df download-from-binary-cache: Print correct URL 2012-07-18 11:01:17 -04:00
Eelco Dolstra a6f348599c Print some debug output 2012-07-17 16:19:40 -04:00
Eelco Dolstra f2bdc87595 Update the other substituters 2012-07-11 18:52:09 -04:00
Eelco Dolstra 15c15da482 Add some missing --version switches 2012-07-11 18:07:41 -04:00
Eelco Dolstra d287b62b64 Set the User-Agent header to "Nix/<version>" 2012-07-11 18:05:30 -04:00
Eelco Dolstra b74d92755d download-from-binary-cache: Use HEAD requests if possible
In "nix-env -qas", we don't need the substitute info, we just need to
know if it exists.  This can be done using a HTTP HEAD request, which
saves bandwidth.

Note however that curl currently has a bug that prevents it from
reusing HTTP connections if HEAD requests return a 404:

https://sourceforge.net/tracker/?func=detail&aid=3542731&group_id=976&atid=100976

Without the patch attached to the issue, using HEAD is actually quite
a bit slower than GET.
2012-07-11 17:53:20 -04:00
Eelco Dolstra 5ee8944155 Cleanup 2012-07-11 10:13:16 -04:00
Eelco Dolstra 099125435f download-from-binary-cache: add nix.conf options 2012-07-09 10:57:28 -04:00
Eelco Dolstra 400e556b34 Cleanup 2012-07-08 18:39:07 -04:00
Eelco Dolstra 11800e6198 download-from-binary-cache: parallelise fetching of NAR info files
Getting substitute information using the binary cache substituter has
non-trivial latency overhead.  A package or NixOS system configuration
can have hundreds of dependencies, and in the worst case (when the
local info cache is empty) we have to do a separate HTTP request for
each of these.  If the ping time to the server is t, getting N info
files will take tN seconds; e.g., with a ping time of 0.1s to
nixos.org, sequentially downloading 1000 info files (a typical NixOS
config) will take at least 100 seconds.

To fix this problem, the binary cache substituter can now perform
requests in parallel.  This required changing the substituter
interface to support a function querySubstitutablePathInfos() that
queries multiple paths at the same time, and rewriting queryMissing()
to take advantage of parallelism.  (Due to local caching,
parallelising queryMissing() is sufficient for most use cases, since
it's almost always called before building a derivation and thus fills
the local info cache.)

For example, parallelism speeds up querying all 1056 paths in a
particular NixOS system configuration from 116s to 2.6s.  It works so
well because the eccentricity of the top-level derivation in the
dependency graph is only 9.  So we only need 10 round-trips (when
using an unlimited number of parallel connections) to get everything.

Currently we do a maximum of 150 parallel connections to the server.
Thus it's important that the binary cache server (e.g. nixos.org) has
a high connection limit.  Alternatively we could use HTTP pipelining,
but WWW::Curl doesn't support it and libcurl has a hard-coded limit of
5 requests per pipeline.
2012-07-06 19:08:20 -04:00
Eelco Dolstra cd94665f38 download-from-binary-cache: use WWW::Curl
Using WWW::Curl rather than running an external curl process for every
NAR info file halves the time it takes to get info thanks to libcurl's
support for persistent HTTP connections.  (We save a roundtrip per
file.)  But the real gain will come from using parallel and/or
pipelined requests.
2012-07-06 00:30:40 -04:00
Eelco Dolstra ae60643c15 download-from-binary-cache: do negative NAR info caching
I.e. if a NAR info file does *not* exist, we record it in the cache DB
so that we don't retry it later.
2012-07-03 18:54:46 -04:00
Eelco Dolstra 89380c03e9 download-from-binary-cache: in queries, preferred cached info 2012-07-03 18:35:39 -04:00
Eelco Dolstra 2a8e5c8b11 download-from-binary-cache: strip trailing / from URLs 2012-07-03 17:47:01 -04:00
Eelco Dolstra d694c599e2 download-from-binary-cache: cache binary cache info in a SQLite DB 2012-07-03 17:29:33 -04:00
Eelco Dolstra 8319b1ab9f download-from-binary-cache: Verify NAR hashes 2012-07-02 18:53:04 -04:00
Eelco Dolstra cf49472d60 nix-push: Always generate base-32 hashes 2012-07-02 18:05:57 -04:00