Commit graph

2725 commits

Author SHA1 Message Date
Eelco Dolstra 4911a10a4e Use XZ compression in binary caches
XZ compresses significantly better than bzip2.  Here are the
compression ratios and execution times (using 4 cores in parallel) on
my /var/run/current-system (3.1 GiB):

  bzip2: total compressed size 849.56 MiB, 30.8% [2m08]
  xz -6: total compressed size 641.84 MiB, 23.4% [6m53]
  xz -7: total compressed size 621.82 MiB, 22.6% [7m19]
  xz -8: total compressed size 599.33 MiB, 21.8% [7m18]
  xz -9: total compressed size 588.18 MiB, 21.4% [7m40]

Note that compression takes much longer.  More importantly, however,
decompression is much faster:

  bzip2: 1m47.274s
  xz -6: 0m55.446s
  xz -7: 0m54.119s
  xz -8: 0m52.388s
  xz -9: 0m51.842s

The only downside to using -9 is that decompression takes a fair
amount (~65 MB) of memory.
2012-06-29 15:24:52 -04:00
Eelco Dolstra 49cd7387ad nix-push: create a manifest-less binary cache
Manifests are a huge pain, since users need to run nix-pull directly
or indirectly to obtain them.  They tend to be large and lag behind
the available binaries; also, the downloaded manifests in
/nix/var/nix/manifest need to be in sync with the Nixpkgs sources.  So
we want to get rid of them.

The idea of manifest-free operation works as follows.  Nix is
configured with a set of URIs of binary caches, e.g.

  http://nixos.org/binary-cache

Whenever Nix needs a store path X, it checks each binary cache for the
existence of a file <CACHE-URI>/<SHA-256 hash of X>.narinfo, e.g.

  http://nixos.org/binary-cache/bi1gh9...ia17.narinfo

The .narinfo file contains the necessary information about the store
path that was formerly kept in the manifest, i.e., (relative) URI of
the compressed NAR, references, size, hash, etc.  For example:

  StorePath: /nix/store/xqp4l88cr9bxv01jinkz861mnc9p7qfi-neon-0.29.6
  URL: 1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2
  CompressedHash: sha256:1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd
  CompressedSize: 202542
  NarHash: sha256:1af26536781e6134ab84201b33408759fc59b36cc5530f57c0663f67b588e15f
  NarSize: 700440
  References: 043zrsanirjh8nbc5vqpjn93hhrf107f-bash-4.2-p24 cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 ...
  Deriver: 4idz1bgi58h3pazxr3akrw4fsr6zrf3r-neon-0.29.6.drv
  System: x86_64-linux

Nix then knows that it needs to download

  http://nixos.org/binary-cache/1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2

to substitute the store path.

Note that the store directory is omitted from the References and
Deriver fields to save space, and that the URL can be relative to the
binary cache prefix.

This patch just makes nix-push create binary caches in this format.
The next step is to make a substituter that supports them.
2012-06-28 17:19:32 -04:00
Eelco Dolstra 1aba0bf0fa nix-store -r: do substitutions in parallel
I.e. when multiple non-derivation arguments are passed to ‘nix-store
-r’ to be substituted, do them in parallel.
2012-06-27 16:58:15 -04:00
Eelco Dolstra 42f5a2fc29 Mount an empty /dev/shm tmpfs in the chroot
This ensures that whatever the builder writes in /dev/shm is
automatically cleaned up.
2012-06-27 09:52:27 -04:00
Eelco Dolstra 3ee208516f Check the return code of the clone() call 2012-06-27 09:52:06 -04:00
Eelco Dolstra 1db38ae81b When using chroots, use a private PID namespace
In a private PID namespace, processes have PIDs that are separate from
the rest of the system.  The initial child gets PID 1.  Processes in
the chroot cannot see processes outside of the chroot.  This improves
isolation between builds.  However, processes on the outside can see
processes in the chroot and send signals to them (if they have
appropriate rights).

Since the builder gets PID 1, it serves as the reaper for zombies in
the chroot.  This might turn out to be a problem.  In that case we'll
need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 15:45:16 -04:00
Eelco Dolstra 5489086456 Use a private UTS namespace to provide a deterministic host/domain name to builders
In chroot builds, set the host name to "localhost" and the domain name
to "(none)" (the latter being the kernel's default).  This improves
determinism a bit further.

P.S. I have to idea what UTS stands for.
2012-06-25 14:12:17 -04:00
Eelco Dolstra 8da6772ed4 Update release notes 2012-06-23 14:59:13 -04:00
Eelco Dolstra 723a68c826 Improve error message 2012-06-23 00:57:14 -04:00
Eelco Dolstra 7ffa523fd1 In chroot builds, use a private SysV IPC namespace
This improves isolation a bit further, and it's just one extra flag in
the unshare() call.

P.S. It would be very cool to use CLONE_NEWPID (to put the builder in
a private PID namespace) as well, but that's slightly more risky since
having a builder start as PID 1 may cause problems.
2012-06-23 00:51:40 -04:00
Eelco Dolstra df716c98d2 In chroot builds, use a private network namespace
On Linux it's possible to run a process in its own network namespace,
meaning that it gets its own set of network interfaces, disjunct from
the rest of the system.  We use this to completely remove network
access to chroot builds, except that they get a private loopback
interface.  This means that:

- Builders cannot connect to the outside network or to other processes
  on the same machine, except processes within the same build.

- Vice versa, other processes cannot connect to processes in a chroot
  build, and open ports/connections do not show up in "netstat".

- If two concurrent builders try to listen on the same port (e.g. as
  part of a test), they no longer conflict with each other.

This was inspired by the "PrivateNetwork" flag in systemd.
2012-06-23 00:28:35 -04:00
Eelco Dolstra 2f3f413e91 Support socket-based, on-demand activation of the Nix daemon with systemd
Systemd can start the Nix daemon on demand when the Nix daemon socket
is first accessed.  This is signalled through the LISTEN_FDS
environment variable, so all we need to do is check for that and then
use file descriptor 3 as the listen socket instead of creating one
ourselves.
2012-06-18 23:01:46 -04:00
Eelco Dolstra 02fb6323e0 Add Emacs to the disk image 2012-05-31 09:50:58 -04:00
Michel Alexandre Salim 221626e715 fixes to nix-worker systemd service descriptor: - remove commented-out lines - register the file for distribution in Makefile.am 2012-05-31 08:59:36 -04:00
Michel Alexandre Salim a7ed1f67ee On systems with SystemD, install the service descriptor for nix-worker, and enable and start it 2012-05-31 08:59:36 -04:00
Michel Alexandre Salim 8922346305 Major spec update: - Fix license field - Split into subpackages - Update build dependencies - Configure users and groups for multi-user mode - Fix installation location of Perl modules 2012-05-31 08:59:36 -04:00
Michel Alexandre Salim 6a214f3e06 Update nix profile: - incorporate NixOS's configuration so that nix is usable by normal users - install as a data file, not a program file 2012-05-31 08:59:36 -04:00
Michel Alexandre Salim d0308073c3 - only enable deprecated spec sections when building on systems with older RPM versions - move tests to dedicated %check section - use standard build macros 2012-05-31 08:59:36 -04:00
Michel Alexandre Salim e545a7f9a8 - replace %define with %global 2012-05-31 08:59:36 -04:00
Eelco Dolstra f5398d374b Compress build logs on the fly using bzip2 2012-05-30 10:12:29 -04:00
Eelco Dolstra 881beb170d "nix-store -l": support compressed logs 2012-05-30 00:00:02 -04:00
Eelco Dolstra 4bc4da331a Reserve some disk space for the garbage collector
We can't open a SQLite database if the disk is full.  Since this
prevents the garbage collector from running when it's most needed, we
reserve some dummy space that we can free just before doing a garbage
collection.  This actually revives some old code from the Berkeley DB
days.

Fixes #27.
2012-05-29 22:59:12 -04:00
Eelco Dolstra 2c26985835 Add option ‘build-keep-log’ to enable/disable writing of build logs
Fixes #26.
2012-05-29 16:42:05 -04:00
Eelco Dolstra 8058dab26e Clean up the installation section; document the generic binary tarballs 2012-05-24 12:04:07 -04:00
Eelco Dolstra 0301525e6c Fix owner/group in tar invocation 2012-05-22 19:40:40 -04:00
Eelco Dolstra 6814b1dfa1 Generate binary tarballs for installing Nix
For several platforms we don't currently have "native" Nix packages
(e.g. Mac OS X and FreeBSD).  This provides the next best thing: a
tarball containing the closure of Nix, plus a simple script
"nix-finish-install" that initialises the Nix database, registers the
paths in the closure as valid, and runs "nix-env -i /path/to/nix" to
initialise the user profile.

The tarball must be unpacked in the root directory.  It creates
/nix/store/... and /usr/bin/nix-finish-install.  Typical installation
is as follows:

  $ cd /
  $ tar xvf /path/to/nix-1.1pre1234_abcdef-x86_64-linux.tar.bz2
  $ nix-finish-install
  (if necessary add ~/.nix-profile/etc/profile.d/nix.sh to the shell
  login scripts)

After this, /usr/bin/nix-finish-install can be deleted, if desired.

The downside to the binary tarball is that it's pretty big (~55 MiB
for x86_64-linux).
2012-05-22 18:36:54 -04:00
Eelco Dolstra 591aab7e21 Remove $FONTCONFIG_FILE hack from nix-profile.sh
It's no longer needed because Nixpkgs' fontconfig uses
/etc/fonts/fonts.conf as a default, just like other distributions.
2012-05-22 14:00:08 -04:00
Eelco Dolstra e071f87dc5 Add an experimental nix-make file
To use it, just do (e.g.) "nix-build build.nix -A nix_env".
2012-05-21 09:43:01 -04:00
Eelco Dolstra cac9eb39fe Bump version number 2012-05-12 00:07:08 -04:00
Eelco Dolstra dfc6a43b72 Fix the install check 2012-05-11 23:30:47 -04:00
Eelco Dolstra 587b408210 Set release date 2012-05-11 17:40:58 -04:00
Eelco Dolstra 8a08813d6c Manual updates 2012-05-11 17:39:06 -04:00
Eelco Dolstra 2b00e6990c CSS tweaks 2012-05-11 16:21:21 -04:00
Eelco Dolstra 58d1de08d9 Use perl.libPrefix to (hopefully) fix the Cygwin build
http://hydra.nixos.org/build/2602599
2012-05-11 09:41:39 -04:00
Eelco Dolstra 37fa47908b Build Ubuntu 12.04 packages 2012-05-10 22:12:20 -04:00
Eelco Dolstra 4d383f57f4 Document "nix-build --run-env" 2012-05-10 19:29:36 -04:00
Eelco Dolstra 4f7bab7db1 Support building with the Perl XS bindings disabled
Since the Perl bindings require shared libraries, this is required on
platforms such as Cygwin where we do a static build.
2012-05-10 19:03:23 -04:00
Eelco Dolstra 6a7b24a3f2 Document "nix-store --add" 2012-05-10 18:09:45 -04:00
Eelco Dolstra cda1fd8ec8 Remove an obsolete hack 2012-05-10 16:56:36 -04:00
Eelco Dolstra 663c06e8cd Disable building in chroot for Nix's corepkgs
The dependencies of the corepkgs are not necessarily in the chroot (or
in the Nix store), so don't build them in a chroot.
2012-05-09 22:14:36 -04:00
Eelco Dolstra cb1248d208 Document some nix-store subcommands 2012-05-09 19:06:39 -04:00
Eelco Dolstra 7a213ffc69 Document $NIX_PATH / -I 2012-05-09 19:06:13 -04:00
Eelco Dolstra a58efdb69b Update the release notes 2012-05-09 19:05:30 -04:00
Eelco Dolstra 0c4c8f7a9d Remove obsolete files (moved to release tree) 2012-05-08 15:43:54 -04:00
Eelco Dolstra afa7b8a479 nix-channel --update: allow updating only the specified channels 2012-05-07 17:55:56 -04:00
Eelco Dolstra 147f10157f Now *really* prevent accumulation of old manifests 2012-05-07 17:23:26 -04:00
Eelco Dolstra 464089365e Fix some 32-bit builds
Perl on some 32-bit systems needs -D_FILE_OFFSET_BITS=64.  See also commit
02f1363e19.
2012-05-04 21:40:56 -04:00
Eelco Dolstra c6acb219f9 Drop the Perl-specific CFLAGS
This fixes the Darwin build (http://hydra.nixos.org/build/2517380).
Hopefully it doesn't break other builds.
2012-05-04 18:50:34 -04:00
Eelco Dolstra 56c82f3d9d Don't build for old Debian/Ubuntu releases that don't have a sufficiently new SQLite 2012-05-04 17:45:21 -04:00
Eelco Dolstra 373e875ac2 Don't build for old Fedora releases that don't have a sufficiently new SQLite 2012-05-04 17:25:25 -04:00