Commit graph

628 commits

Author SHA1 Message Date
Eelco Dolstra fe241ece29 Merge branch 'master' into no-manifests 2012-07-18 10:47:59 -04:00
Eelco Dolstra ccc52adfb2 Add function queryPathFromHashPart()
To implement binary caches efficiently, Hydra needs to be able to map
the hash part of a store path (e.g. "gbg...zr7") to the full store
path (e.g. "/nix/store/gbg...kzr7-subversion-1.7.5").  (The binary
cache mechanism uses hash parts as a key for looking up store paths to
ensure privacy.)  However, doing a search in the Nix store for
/nix/store/<hash>* is expensive since it requires reading the entire
directory.  queryPathFromHashPart() prevents this by doing a cheap
database lookup.
2012-07-17 18:55:39 -04:00
Eelco Dolstra 3a9fdf2747 Return an exit code of 100 for cached failed builds
Exit code 100 should be returned for all permanent failures.  This
includes cached failures.

Fixes #34.
2012-07-17 15:55:30 -04:00
Eelco Dolstra 6c01fb4d68 Update Nix 1.1 release notes 2012-07-17 10:06:20 -04:00
Eelco Dolstra 53b24f3518 Allow disabling log compression 2012-07-17 09:40:12 -04:00
Eelco Dolstra f2bdc87595 Update the other substituters 2012-07-11 18:52:09 -04:00
Eelco Dolstra 09a6321aeb Replace hasSubstitutes() with querySubstitutablePaths()
querySubstitutablePaths() takes a set of paths, so this greatly
reduces daemon <-> client latency.
2012-07-11 17:52:18 -04:00
Eelco Dolstra 58ef4d9a95 Add a function queryValidPaths()
queryValidPaths() combines multiple calls to isValidPath() in one.
This matters when using the Nix daemon because it reduces latency.
For instance, on "nix-env -qas \*" it reduces execution time from 5.7s
to 4.7s (which is indistinguishable from the non-daemon case).
2012-07-11 11:08:47 -04:00
Eelco Dolstra 667d5f1936 Rename queryValidPaths() to queryAllValidPaths() 2012-07-11 10:49:04 -04:00
Eelco Dolstra eb3036da87 Implement querySubstitutablePathInfos() in the daemon
Also removed querySubstitutablePathInfo().
2012-07-11 10:43:24 -04:00
Eelco Dolstra 425cc612ad build.cc: Don't use hasSubstitute()
Instead make a single call to querySubstitutablePathInfo() per
derivation output.  This is faster and prevents having to implement
the "have" function in the binary cache substituter.
2012-07-08 18:39:24 -04:00
Eelco Dolstra 11800e6198 download-from-binary-cache: parallelise fetching of NAR info files
Getting substitute information using the binary cache substituter has
non-trivial latency overhead.  A package or NixOS system configuration
can have hundreds of dependencies, and in the worst case (when the
local info cache is empty) we have to do a separate HTTP request for
each of these.  If the ping time to the server is t, getting N info
files will take tN seconds; e.g., with a ping time of 0.1s to
nixos.org, sequentially downloading 1000 info files (a typical NixOS
config) will take at least 100 seconds.

To fix this problem, the binary cache substituter can now perform
requests in parallel.  This required changing the substituter
interface to support a function querySubstitutablePathInfos() that
queries multiple paths at the same time, and rewriting queryMissing()
to take advantage of parallelism.  (Due to local caching,
parallelising queryMissing() is sufficient for most use cases, since
it's almost always called before building a derivation and thus fills
the local info cache.)

For example, parallelism speeds up querying all 1056 paths in a
particular NixOS system configuration from 116s to 2.6s.  It works so
well because the eccentricity of the top-level derivation in the
dependency graph is only 9.  So we only need 10 round-trips (when
using an unlimited number of parallel connections) to get everything.

Currently we do a maximum of 150 parallel connections to the server.
Thus it's important that the binary cache server (e.g. nixos.org) has
a high connection limit.  Alternatively we could use HTTP pipelining,
but WWW::Curl doesn't support it and libcurl has a hard-coded limit of
5 requests per pipeline.
2012-07-06 19:08:20 -04:00
Eelco Dolstra 1aba0bf0fa nix-store -r: do substitutions in parallel
I.e. when multiple non-derivation arguments are passed to ‘nix-store
-r’ to be substituted, do them in parallel.
2012-06-27 16:58:15 -04:00
Eelco Dolstra 42f5a2fc29 Mount an empty /dev/shm tmpfs in the chroot
This ensures that whatever the builder writes in /dev/shm is
automatically cleaned up.
2012-06-27 09:52:27 -04:00
Eelco Dolstra 3ee208516f Check the return code of the clone() call 2012-06-27 09:52:06 -04:00
Eelco Dolstra 1db38ae81b When using chroots, use a private PID namespace
In a private PID namespace, processes have PIDs that are separate from
the rest of the system.  The initial child gets PID 1.  Processes in
the chroot cannot see processes outside of the chroot.  This improves
isolation between builds.  However, processes on the outside can see
processes in the chroot and send signals to them (if they have
appropriate rights).

Since the builder gets PID 1, it serves as the reaper for zombies in
the chroot.  This might turn out to be a problem.  In that case we'll
need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 15:45:16 -04:00
Eelco Dolstra 5489086456 Use a private UTS namespace to provide a deterministic host/domain name to builders
In chroot builds, set the host name to "localhost" and the domain name
to "(none)" (the latter being the kernel's default).  This improves
determinism a bit further.

P.S. I have to idea what UTS stands for.
2012-06-25 14:12:17 -04:00
Eelco Dolstra 723a68c826 Improve error message 2012-06-23 00:57:14 -04:00
Eelco Dolstra 7ffa523fd1 In chroot builds, use a private SysV IPC namespace
This improves isolation a bit further, and it's just one extra flag in
the unshare() call.

P.S. It would be very cool to use CLONE_NEWPID (to put the builder in
a private PID namespace) as well, but that's slightly more risky since
having a builder start as PID 1 may cause problems.
2012-06-23 00:51:40 -04:00
Eelco Dolstra df716c98d2 In chroot builds, use a private network namespace
On Linux it's possible to run a process in its own network namespace,
meaning that it gets its own set of network interfaces, disjunct from
the rest of the system.  We use this to completely remove network
access to chroot builds, except that they get a private loopback
interface.  This means that:

- Builders cannot connect to the outside network or to other processes
  on the same machine, except processes within the same build.

- Vice versa, other processes cannot connect to processes in a chroot
  build, and open ports/connections do not show up in "netstat".

- If two concurrent builders try to listen on the same port (e.g. as
  part of a test), they no longer conflict with each other.

This was inspired by the "PrivateNetwork" flag in systemd.
2012-06-23 00:28:35 -04:00
Eelco Dolstra f5398d374b Compress build logs on the fly using bzip2 2012-05-30 10:12:29 -04:00
Eelco Dolstra 4bc4da331a Reserve some disk space for the garbage collector
We can't open a SQLite database if the disk is full.  Since this
prevents the garbage collector from running when it's most needed, we
reserve some dummy space that we can free just before doing a garbage
collection.  This actually revives some old code from the Berkeley DB
days.

Fixes #27.
2012-05-29 22:59:12 -04:00
Eelco Dolstra 2c26985835 Add option ‘build-keep-log’ to enable/disable writing of build logs
Fixes #26.
2012-05-29 16:42:05 -04:00
Eelco Dolstra db5b86ef13 * Add an option ‘build-use-substitutes’, which can be set to ‘false’
to disable use of substitutes; i.e., force building from source.
  Fixes Nix/221.
2012-04-30 19:15:34 -04:00
Eelco Dolstra 46cdc6ad51 Handle EPERM when creating a hard link for the chroot
There is a race condition when doing parallel builds with chroots and
the immutable bit enabled.  One process may call makeImmutable()
before the other has called link(), in which case link() will fail
with EPERM.  We could retry or wrap the operation in a lock, but since
this condition is rare and I'm lazy, we just use the existing copy
fallback.

Fixes #9.
2012-04-30 10:58:04 -04:00
Eelco Dolstra 6387830c3b Merge pull request #2 from viric/master
Again, adding the sync option
2012-04-23 03:29:03 -07:00
Eelco Dolstra 1132dd27ea Fix obscure race condition in GC root creation
This should fix rare Hydra errors of the form:

error: symlinking `/nix/var/nix/gcroots/per-user/hydra/hydra-roots/7sfhs5fdmjxm8sqgcpd0pgcsmz1kq0l0-nixos-iso-0.1pre33785-33795' to `/nix/store/7sfhs5fdmjxm8sqgcpd0pgcsmz1kq0l0-nixos-iso-0.1pre33785-33795': File exists
2012-04-16 18:47:01 +02:00
Eelco Dolstra 31e34fcf93 Close almost all file descriptors in the builder
This regression was accidentally introduced in
35355fc1fc.
2012-04-15 00:20:32 +02:00
Eelco Dolstra f2467eb6a0 On Linux, pretend we're building on Linux 2.6
Setting the UNAME26 personality causes "uname" to return "2.6.x",
regardless of the kernel version.  This improves determinism in
a few misbehaved packages.
2012-04-05 13:03:19 +02:00
Eelco Dolstra 446b827bae Mac OS X fix 2012-03-26 23:18:36 +02:00
Eelco Dolstra 480dda0e42 Delete non-directory valid paths right away
It's unlikely that rename() is faster than unlink() on a regular file
or symlink, so don't bother.
2012-03-26 20:56:30 +02:00
Eelco Dolstra 117670be57 Make the garbage collector more concurrent
Make the garbage collector more concurrent by deleting valid paths
outside the region where we're holding the global GC lock.  This
should greatly reduce the time during which new builds are blocked,
since the deletion accounts for the vast majority of the time spent in
the GC.

To ensure that this is safe, the valid paths are invalidated and
renamed to some arbitrary path while we're holding the lock.  This
ensures that we when we finally delete the path, it's not a (newly)
valid or locked path.
2012-03-26 20:43:33 +02:00
Eelco Dolstra 8be1979f1a Remove the --max-links GC option
We don't need this anymore now that current filesystems support more
than 32,000 files in a directory.
2012-03-26 20:00:02 +02:00
Lluís Batlle i Rossell 85799bf89e Fixing the default of sync-before-registering
Setting 'false' as default, as suggested by Eelco.

I also added a comment about the setting in the code.
2012-03-22 20:05:54 +01:00
Lluís Batlle i Rossell 25de80e2b6 Adding a nix option to sync before registering a path, for non-ext*
filesystems.
2012-03-22 19:57:42 +01:00
Eelco Dolstra 19d9762ad5 nix-store --clear-failed-paths: Clear derivation outputs
If the argument to ‘nix-store --clear-failed-paths’ is a derivation,
then clear the failed state of its outputs.
2012-03-20 18:23:26 +01:00
Eelco Dolstra b006a31d52 Drop the externals directory
Nix now requires SQLite and bzip2 to be pre-installed.  SQLite is
detected using pkg-config.  We required DBD::SQLite anyway, so
depending on SQLite is not a big problem.

The --with-bzip2, --with-openssl and --with-sqlite flags are gone.
2012-03-18 23:54:57 +01:00
Eelco Dolstra bd50c01972 Ensure that Perl processes delete their entry in the temproots directory
By moving the destructor object to libstore.so, it's also run when
download-using-manifests and nix-prefetch-url exit.  This prevents
them from cluttering /nix/var/nix/temproots with stale files.
2012-03-13 17:07:49 +01:00
Eelco Dolstra 35355fc1fc Set the close-on-exec flag on file descriptors 2012-03-05 20:29:00 +01:00
Eelco Dolstra 7b22bec252 Don't leak a file descriptor in commonChildInit() 2012-03-05 20:28:09 +01:00
Eelco Dolstra 2b4964f319 Restore progress indication during nix-copy-closure 2012-03-05 18:13:44 +01:00
Eelco Dolstra 8afd28a922 Remove dependency on sqlite3_table_column_metadata
Not all SQLite builds have the function sqlite3_table_column_metadata.
We were only using it in a schema upgrade check for compatibility with
databases that were probably never seen in the wild.  So remove it.
2012-03-01 16:06:49 +01:00
Eelco Dolstra 1c94524458 Fix an uninitialised variable
The variable ‘useChroot’ was not initialised properly.  This caused
random failures if using the build hook.  Seen on Mac OS X 10.7 with Clang.
Thanks to KolibriFX for finding this :-)
2012-03-01 16:06:49 +01:00
Eelco Dolstra 6fe13e6aba Fix chroots builds
Chroots are initialised by hard-linking inputs from the Nix store to
the chroot.  This doesn't work if the input has its immutable bit set,
because it's forbidden to create hard links to immutable files.  So
temporarily clear the immutable bit when creating and destroying the
chroot.

Note that making regular files in the Nix store immutable isn't very
reliable, since the bit can easily become cleared: for instance, if we
run the garbage collector after running ‘nix-store --optimise’.  So
maybe we should only make directories immutable.
2012-02-18 01:23:52 +01:00
Eelco Dolstra bd013b6f98 On Linux, make the Nix store really read-only by using the immutable bit
I was bitten one time too many by Python modifying the Nix store by
creating *.pyc files when run as root.  On Linux, we can prevent this
by setting the immutable bit on files and directories (as in ‘chattr
+i’).  This isn't supported by all filesystems, so it's not an error
if setting the bit fails.  The immutable bit is cleared by the garbage
collector before deleting a path.  The only tricky aspect is in
optimiseStore(), since it's forbidden to create hard links to an
immutable file.  Thus optimiseStore() temporarily clears the immutable
bit before creating the link.
2012-02-15 01:31:56 +01:00
Eelco Dolstra 5e57047d87 Fix a broken guard around utime()
Because of an outdated check for a timestamp of 0, we were calling
utime() even when it wasn't necessary.
2012-02-15 01:00:30 +01:00
Eelco Dolstra 58ac7a17a4 Don't use deletePath() to delete a single file 2012-02-15 00:28:01 +01:00
Eelco Dolstra d5a5a83ad4 Use data() instead of c_str() where appropriate 2012-02-09 18:27:45 +01:00
Eelco Dolstra a71d02440b * Oops. 2011-12-30 17:13:25 +00:00
Eelco Dolstra 6f5e3326ce * Move topoSortPaths() out of gc.cc. 2011-12-30 15:02:50 +00:00