Commit graph

518 commits

Author SHA1 Message Date
Eelco Dolstra ae60643c15 download-from-binary-cache: do negative NAR info caching
I.e. if a NAR info file does *not* exist, we record it in the cache DB
so that we don't retry it later.
2012-07-03 18:54:46 -04:00
Eelco Dolstra 89380c03e9 download-from-binary-cache: in queries, preferred cached info 2012-07-03 18:35:39 -04:00
Eelco Dolstra 2a8e5c8b11 download-from-binary-cache: strip trailing / from URLs 2012-07-03 17:47:01 -04:00
Eelco Dolstra d694c599e2 download-from-binary-cache: cache binary cache info in a SQLite DB 2012-07-03 17:29:33 -04:00
Eelco Dolstra 8319b1ab9f download-from-binary-cache: Verify NAR hashes 2012-07-02 18:53:04 -04:00
Eelco Dolstra cf49472d60 nix-push: Always generate base-32 hashes 2012-07-02 18:05:57 -04:00
Eelco Dolstra 6ec7460af1 Binary caches: use a better key
Use the hash part of the store path as a key rather than a hash of the
store path.  This is enough to get the desired privacy property.
2012-07-02 12:42:58 -04:00
Eelco Dolstra 440adfbdd1 Add an environment variable $NIX_BINARY_CACHES specifying URLs of binary caches 2012-07-01 21:55:36 -04:00
Eelco Dolstra f4190c38ba Allow both bzip2 and xz compression 2012-07-01 18:46:38 -04:00
Eelco Dolstra 000132cbd1 nix-push: Don't pollute the current directory with result symlink 2012-06-29 18:30:28 -04:00
Eelco Dolstra 37f7098464 First attempt at the manifest-less substituter 2012-06-29 18:28:52 -04:00
Eelco Dolstra 91b8814f0d Doh 2012-06-29 17:16:00 -04:00
Eelco Dolstra 4911a10a4e Use XZ compression in binary caches
XZ compresses significantly better than bzip2.  Here are the
compression ratios and execution times (using 4 cores in parallel) on
my /var/run/current-system (3.1 GiB):

  bzip2: total compressed size 849.56 MiB, 30.8% [2m08]
  xz -6: total compressed size 641.84 MiB, 23.4% [6m53]
  xz -7: total compressed size 621.82 MiB, 22.6% [7m19]
  xz -8: total compressed size 599.33 MiB, 21.8% [7m18]
  xz -9: total compressed size 588.18 MiB, 21.4% [7m40]

Note that compression takes much longer.  More importantly, however,
decompression is much faster:

  bzip2: 1m47.274s
  xz -6: 0m55.446s
  xz -7: 0m54.119s
  xz -8: 0m52.388s
  xz -9: 0m51.842s

The only downside to using -9 is that decompression takes a fair
amount (~65 MB) of memory.
2012-06-29 15:24:52 -04:00
Eelco Dolstra 49cd7387ad nix-push: create a manifest-less binary cache
Manifests are a huge pain, since users need to run nix-pull directly
or indirectly to obtain them.  They tend to be large and lag behind
the available binaries; also, the downloaded manifests in
/nix/var/nix/manifest need to be in sync with the Nixpkgs sources.  So
we want to get rid of them.

The idea of manifest-free operation works as follows.  Nix is
configured with a set of URIs of binary caches, e.g.

  http://nixos.org/binary-cache

Whenever Nix needs a store path X, it checks each binary cache for the
existence of a file <CACHE-URI>/<SHA-256 hash of X>.narinfo, e.g.

  http://nixos.org/binary-cache/bi1gh9...ia17.narinfo

The .narinfo file contains the necessary information about the store
path that was formerly kept in the manifest, i.e., (relative) URI of
the compressed NAR, references, size, hash, etc.  For example:

  StorePath: /nix/store/xqp4l88cr9bxv01jinkz861mnc9p7qfi-neon-0.29.6
  URL: 1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2
  CompressedHash: sha256:1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd
  CompressedSize: 202542
  NarHash: sha256:1af26536781e6134ab84201b33408759fc59b36cc5530f57c0663f67b588e15f
  NarSize: 700440
  References: 043zrsanirjh8nbc5vqpjn93hhrf107f-bash-4.2-p24 cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 ...
  Deriver: 4idz1bgi58h3pazxr3akrw4fsr6zrf3r-neon-0.29.6.drv
  System: x86_64-linux

Nix then knows that it needs to download

  http://nixos.org/binary-cache/1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2

to substitute the store path.

Note that the store directory is omitted from the References and
Deriver fields to save space, and that the URL can be relative to the
binary cache prefix.

This patch just makes nix-push create binary caches in this format.
The next step is to make a substituter that supports them.
2012-06-28 17:19:32 -04:00
Michel Alexandre Salim 6a214f3e06 Update nix profile: - incorporate NixOS's configuration so that nix is usable by normal users - install as a data file, not a program file 2012-05-31 08:59:36 -04:00
Eelco Dolstra 6814b1dfa1 Generate binary tarballs for installing Nix
For several platforms we don't currently have "native" Nix packages
(e.g. Mac OS X and FreeBSD).  This provides the next best thing: a
tarball containing the closure of Nix, plus a simple script
"nix-finish-install" that initialises the Nix database, registers the
paths in the closure as valid, and runs "nix-env -i /path/to/nix" to
initialise the user profile.

The tarball must be unpacked in the root directory.  It creates
/nix/store/... and /usr/bin/nix-finish-install.  Typical installation
is as follows:

  $ cd /
  $ tar xvf /path/to/nix-1.1pre1234_abcdef-x86_64-linux.tar.bz2
  $ nix-finish-install
  (if necessary add ~/.nix-profile/etc/profile.d/nix.sh to the shell
  login scripts)

After this, /usr/bin/nix-finish-install can be deleted, if desired.

The downside to the binary tarball is that it's pretty big (~55 MiB
for x86_64-linux).
2012-05-22 18:36:54 -04:00
Eelco Dolstra 591aab7e21 Remove $FONTCONFIG_FILE hack from nix-profile.sh
It's no longer needed because Nixpkgs' fontconfig uses
/etc/fonts/fonts.conf as a default, just like other distributions.
2012-05-22 14:00:08 -04:00
Eelco Dolstra 0c4c8f7a9d Remove obsolete files (moved to release tree) 2012-05-08 15:43:54 -04:00
Eelco Dolstra afa7b8a479 nix-channel --update: allow updating only the specified channels 2012-05-07 17:55:56 -04:00
Eelco Dolstra 147f10157f Now *really* prevent accumulation of old manifests 2012-05-07 17:23:26 -04:00
Eelco Dolstra e060c99447 Use mkpath instead of make_path
Perl <= 5.10 doesn't have make_path.  See
e.g. http://hydra.nixos.org/build/2493981.
2012-05-01 16:35:18 -04:00
Eelco Dolstra 59a26360c7 Support mandatory system features in the build hook
Mandatory features are features that MUST be present in a derivation's
requiredSystemFeatures attribute.  One application is performance
testing, where we have a dedicated machine to run performance tests
(and nothing else).  Then we would add the label "perf" to the
machine's mandatory features and to the performance testing
derivations.
2012-04-30 17:22:45 -04:00
Eelco Dolstra 6de5d53416 Fix a warning in the build hook about $progressViewer 2012-04-24 12:56:30 +02:00
Eelco Dolstra 259c6d3fbb Fix URL set by nix-channel 2012-04-18 13:40:18 +02:00
Eelco Dolstra 8cf1719e3e Hack to prevent accumulation of old manifests 2012-04-16 19:52:31 +02:00
Eelco Dolstra 0b5107c016 If the (redirected) channel URL contains a version number, use it 2012-04-14 21:05:28 +02:00
Eelco Dolstra 1d0bafb74d Follow redirects early to ensure consistency between the manifest and the tarball
Without this change, if the URL is a redirect that changes in between,
we can get a manifest that doesn't match the tarball.
2012-04-14 21:04:22 +02:00
Eelco Dolstra ef902274fd Remove unnecessary "system" argument 2012-04-14 18:48:11 +02:00
Eelco Dolstra e855c7e2c9 nix-channel improvements
"nix-channel --add" now accepts a second argument: the channel name.
This allows channels to have a nicer name than (say) nixpkgs_unstable.
If no name is given, it defaults to the last component of the URL
(with "-unstable" or "-stable" removed).

Also, channels are now stored in a profile
(/nix/var/nix/profiles/per-user/$USER/channels).  One advantage of
this is that it allows rollbacks (e.g. if "nix-channel --update" gives
an undesirable update).
2012-04-14 18:38:52 +02:00
Eelco Dolstra 969a14599d If "pv" is available at compile time, hard-code its path 2012-04-13 14:41:33 +02:00
Shea Levy 34a85c5405 nix-copy-closure: Move the progressViewer directly adjacent to the ssh call so that network progress is what's measured 2012-04-13 14:29:12 +02:00
Shea Levy 055e803851 Add the '--show-progress' flag to nix-copy-closure 2012-04-13 14:29:12 +02:00
Eelco Dolstra 1fb30e6d50 Handle the case where $SHELL is not set 2012-04-10 13:52:37 +02:00
Hydra mirror 3b859ead37 nix-generate-patches: Optionally write new patches to $NIX_ALL_PATCHES 2012-04-04 20:46:18 +00:00
Eelco Dolstra 0837348049 Follow redirects when checking for the existence of MANIFEST.bz2
If we don't follow redirects, we might think that MANIFEST.bz2 exists
just because the HEAD request succeeds on the redirector URI.
2012-04-04 15:41:35 +02:00
Eelco Dolstra b8fb0ce563 Update "nix-build --help" 2012-03-27 11:59:36 +02:00
Eelco Dolstra 5144abe5b6 Add an option "--exclude" to filter dependencies in "nix-build --run-env"
Sometimes when doing "nix-build --run-env" you don't want all
dependencies to be built.  For instance, if we want to do "--run-env"
on the "build" attribute in Hydra's release.nix (to get Hydra's build
environment), we don't want its "tarball" dependency to be built.  So
we can do:

  $ nix-build --run-env release.nix -A build --exclude 'hydra-tarball'

This will skip the dependency whose name matches the "hydra-tarball"
regular expression.  The "--exclude" option can be repeated any number
of times.
2012-03-27 11:40:47 +02:00
Eelco Dolstra 3e94ffffd6 Allow the command executed by "nix-build --run-env" to be overriden with --command 2012-03-27 11:16:43 +02:00
Eelco Dolstra 7f38087f35 Add a command "nix-build --run-env" to reproduce the environment of a derivation
This command builds or fetches all dependencies of the given
derivation, then starts a shell with the environment variables from
the derivation.  This shell also sources $stdenv/setup to initialise
the environment further.

The current directory is not changed.  Thus this is a convenient way
to reproduce a build environment in an existing working tree.

Existing environment variables are left untouched (unless the
derivation overrides them).  As a special hack, the original value of
$PATH is appended to the $PATH produced by $stdenv/setup.

Example session:

$ nix-build --run-env '<nixpkgs>' -A xterm
(the dependencies of xterm are built/fetched...)
$ tar xf $src
$ ./configure
$ make
$ emacs
(... hack source ...)
$ make
$ ./xterm
2012-03-19 04:14:21 +01:00
Eelco Dolstra d645633643 nix-copy-closure: don't print copied path on stdout
We're already printing progress on stderr, so printing them on stdout
afterwards is kind of useless.
2012-03-05 20:52:24 +01:00
Eelco Dolstra 2b4964f319 Restore progress indication during nix-copy-closure 2012-03-05 18:13:44 +01:00
Eelco Dolstra 56042a120a build-remote.pl: don't wait forever for the upload lock
In the build hook, don't wait forever to get the upload lock.  This
ensures progress if another process gets stuck while holding the
upload lock.
2012-03-05 17:58:09 +01:00
Eelco Dolstra ce86a9a0ae * nix-channel: fail if we don't have write permission to the manifests
directory.  Previously in this situation we did add the Nix
  expressions from the channel to allow installation from source, but
  this doesn't work for binary-only channels and leads to confusing
  error messages.
2012-02-02 12:25:56 +00:00
Eelco Dolstra 2a3f4110c5 * nix-build: put the temporary derivation symlink in a temporary
directory rather than the current directory.
* nix-build: --drv-link now implies --add-drv-link.
2012-01-13 23:35:07 +00:00
Eelco Dolstra d4e6b9f2d6 * Doh! 2012-01-05 21:07:12 +00:00
Eelco Dolstra a7366a764a * Add a -I flag to the Perl bindings to nix-build and some other
scripts.
* Include the version and architecture in the -I flag so that there is
  at least a chance that a Nix binary built for one Perl version will
  run on another version.
2012-01-05 20:33:46 +00:00
Eelco Dolstra 39d45a6b09 * Add a test for nix-channel.
* Refactor the nix-channel unpacker a bit.
2012-01-03 01:51:38 +00:00
Eelco Dolstra dadbb51d96 * Use Nix::Config. 2012-01-03 00:47:27 +00:00
Eelco Dolstra 48cea0d01e * Refactoring: Get rid of a few subdirectories in corepkgs/, and some
other simplifications.
* Use <nix/...> to locate the corepkgs.  This allows them to be
  overriden through $NIX_PATH.
* Use bash's pipefail option in the NAR builder so that we don't need
  to create a temporary file.
2012-01-03 00:16:29 +00:00
Eelco Dolstra 4d728bc3e6 * Security: make sure the lock files used by build-remote.pl are not
readable to other users.  Otherwise, any user can open the lock file
  for reading and lock it, thus DoSing the remote build mechanism.
2011-12-21 19:11:58 +00:00
Eelco Dolstra 69d6f0936a * Use a lock to ensure that only one build-remote instance can copy a
closure to a given machine at the same time.  This prevents the case
  where multiple instances try to copy the same missing store path to
  the target machine, which is very wasteful.
2011-12-21 18:59:25 +00:00
Eelco Dolstra a67b8ae224 * Typo. 2011-12-15 14:04:35 +00:00
Eelco Dolstra 3964d95abf * nix-prefetch-url: rewritten in Perl. 2011-12-02 12:09:50 +00:00
Eelco Dolstra 1749a7b0ae * download-using-manifests: use the Perl bindings. 2011-11-29 13:01:24 +00:00
Eelco Dolstra 216440b3ff * For consistency with "nix-store -q --hash", produce hashes in
base-32.  (This affects Hydra manifests.)
2011-11-29 12:32:55 +00:00
Eelco Dolstra 1df120cb05 * Get rid of the shell in ssh calls. 2011-11-29 10:51:22 +00:00
Eelco Dolstra d5ac78e0d6 * Add bzip2 and xz support to nix-copy-closure. 2011-11-23 15:29:58 +00:00
Eelco Dolstra 5bbd693cae * Add an API function exportPaths() that provides the functionality of
‘nix-store --export’.
* Add a Perl module that provides the functionality of
  ‘nix-copy-closure --to’.  This is used by build-remote.pl so it no
  longer needs to start a separate nix-copy-closure process.  Also, it
  uses the Perl API to do the export, so it doesn't need to start a
  separate nix-store process either.  As a result, nix-copy-closure
  and build-remote.pl should no longer fail on very large closures due
  to an "Argument list too long" error.  (Note that having very many
  dependencies in a single derivation can still fail because the
  environment can become too large.  Can't be helped though.)
2011-11-23 15:13:37 +00:00
Eelco Dolstra ab20af3e6f * build-remote.pl: drop a hard-coded reference to /nix/etc/nix. 2011-11-23 12:21:35 +00:00
Eelco Dolstra f8e609c341 * nix-pull: update the Nix manifest cache if necessary. Also, don't
read the manifest just to check the version and print the number of
  paths.  This makes nix-pull very fast for the cached cache (speeding
  up nixos-rebuild without the ‘--no-pull’ or ‘--fast’ options).
2011-11-16 16:41:48 +00:00
Eelco Dolstra d7d7910ba4 * Don't decompress the manifests in /nix/var/nix/manifest. This saves
disk space, and, since they're typically only decompressed once (to
  fill the manifest cache), doesn't make things slower.
2011-11-16 16:25:38 +00:00
Eelco Dolstra 00b41e46ed * Print a consistent message. 2011-10-19 21:34:13 +00:00
Eelco Dolstra f186a9141e * nix-copy-closure: support ‘--dry-run’ and ‘--include-outputs’. 2011-10-18 21:21:22 +00:00
Eelco Dolstra 6761757428 * Use the Store API bindings in nix-copy-closure. 2011-10-11 15:41:13 +00:00
Eelco Dolstra c362e4d718 * Move SSH.pm. 2011-10-11 11:45:36 +00:00
Eelco Dolstra 5090c34ee1 * Set the executable bit on scripts. 2011-10-10 22:40:17 +00:00
Eelco Dolstra cd6d02c366 2011-10-10 21:30:59 +00:00
Eelco Dolstra 6fcdbcac20 * Install NixManifest.pm, NixConfig.pm and GeneratePatches.pm under
the Nix:: namespace.
2011-10-10 21:11:08 +00:00
Eelco Dolstra 5bcdc7e351 * Update the cleanup script. 2011-08-27 16:59:32 +00:00
Eelco Dolstra a95ba4cdd9 * Use last_insert_id instead of sqlite_last_insert_rowid, which you're
not really supposed to use according to the DBD::SQLite docs, and
  fails on some systems (e.g. http://hydra.nixos.org/build/1246662).
2011-08-17 14:17:19 +00:00
Eelco Dolstra 1ecc97b6bd * Add a Nix expression search path feature. Paths between angle
brackets, e.g.

    import <nixpkgs/pkgs/lib>

  are resolved by looking them up relative to the elements listed in
  the search path.  This allows us to get rid of hacks like

    import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib"

  The search path can be specified through the ‘-I’ command-line flag
  and through the colon-separated ‘NIX_PATH’ environment variable,
  e.g.,

    $ nix-build -I /etc/nixos ...

  If a file is not found in the search path, an error message is
  lazily thrown.
2011-08-06 16:05:24 +00:00
Eelco Dolstra 0243eea4b9 * Create a symlink to /nix/var/nix/manifests in /nix/var/nix/gcroots
if it doesn't exist.
2011-07-20 11:47:00 +00:00
Eelco Dolstra e649f3168b * Fix concurrency issues in download-using-manifests' handling of the
SQLite manifest cache.  The DBI AutoCommit feature caused every
  process to have an active transaction at all times, which could
  indefinitely block processes wanting to update the manifest cache.

* Disable fsync() in the manifest cache because we don't need
  integrity (the cache can always be recreated if it gets corrupted).
2011-07-13 14:05:54 +00:00
Eelco Dolstra d94cb02bfe * nix-install-package: unset NIX_REMOTE because $NIX_MANIFESTS_DIR
doesn't work when building through the Nix daemon.  This also
  ensures an error message when the user doesn't have sufficient
  privileges to do nix-pull.
2011-04-19 10:52:35 +00:00
Eelco Dolstra 412914d004 * Read manifests directly into the database, rather than first reading
them into memory.  This brings memory use down to (more or less)
  O(1).  For instance, on my test case, the maximum resident size of
  download-using-manifests while filling the DB went from 142 MiB to
  11 MiB.
2011-04-11 13:16:54 +00:00
Eelco Dolstra 08c8971498 * Lock the database during updates. 2011-04-11 12:40:13 +00:00
Eelco Dolstra 3cbf680f5d * configure: detect whether DBD::SQLite is present. If necessary the
location to DBI and DBD::SQLite can be passed with --with-dbi and
  --with-dbd-sqlite.
2011-04-11 10:13:53 +00:00
Eelco Dolstra 83252b4ca9 * Subtle bug: if you import File::stat in one module, it affects other
modules as well.  So use symbolic field names everywhere (which is
  nicer anyway).
2011-04-11 08:21:30 +00:00
Eelco Dolstra b1882c3ef7 * Create $manifestDir if it doesn't exist. 2011-04-11 08:06:14 +00:00
Eelco Dolstra 5591fcc529 * Cache the manifests in /nix/var/nix/manifests in a SQLite database.
This significantly speeds up the download-using-manifests
  substituter, especially if manifests are very large.  For instance,
  one "nix-build -A geeqie" operation that updated four packages using
  binary patches went from 18.5s to 1.6s.  It also significantly
  reduces memory use.

  The cache is kept in /nix/var/nix/manifests/cache.sqlite.  It's
  updated automatically when manifests are added to or removed from
  /nix/var/nix/manifests.  It might be interesting to have nix-pull
  store manifests directly in the DB, rather than storing them as
  separate flat files, but then we would need a command line interface
  to delete manifests from the DB.
2011-04-10 23:22:46 +00:00
Eelco Dolstra 1e7e4f21ba * Remove the localPaths feature in manifests since it's no longer used
and redundant anyway.
2011-04-06 09:16:22 +00:00
Eelco Dolstra 0423d0692a * Print a better error message. 2011-03-16 15:55:57 +00:00
Eelco Dolstra 802c2651c8 2011-02-17 15:08:33 +00:00
Eelco Dolstra e945b52a31 * nix-push: handle the case where the hash is not set in the DB. 2011-02-17 13:26:33 +00:00
Eelco Dolstra d6c8b995c5 * In Hydra manifests the Size field is missing, so don't rely on it.
This caused a lot of "Use of uninitialized value" warnings from
  Perl.
2010-12-22 10:32:32 +00:00
Eelco Dolstra a0be433fec * Disable X11 forwarding, it's not needed. 2010-12-15 14:25:54 +00:00
Eelco Dolstra f1a6b97639 * nix-copy-closure: make sure that the shell doesn't do globbing of
`+' and `?' in filenames.  This is very slow if /nix/store is very
  large.  (This is a quick hack - a cleaner solution would be to
  bypass the shell entirely.)
2010-12-15 08:39:37 +00:00
Eelco Dolstra d787285af9 * nix-instantiate: return exit status 100 to denote a permanent build
failure.  The build hook can use this to distinguish between
  transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
Eelco Dolstra 542fc69062 * When doing a query (e.g. `nix-store -r --dry-run'), don't make a lot
of expensive calls to `nix-store --check-validity'.
2010-12-13 08:39:10 +00:00
Eelco Dolstra 7d0444e244 * Bad things happen when a filehandle has the same name as a module. 2010-12-07 12:33:42 +00:00
Eelco Dolstra 365f3028dd * Use CamelCase for the Perl modules. 2010-12-05 17:50:29 +00:00
Eelco Dolstra f42a505ab7 * Add a script `nix-generate-patches'.
* Fix the binary patching test.
2010-12-05 17:36:02 +00:00
Eelco Dolstra d3bba0c2d8 * Move the patch generator into a module. 2010-11-29 15:26:28 +00:00
Eelco Dolstra 9737a7eba0 * Don't generate patches if the system type differs. 2010-11-29 09:26:05 +00:00
Eelco Dolstra bd48fd97f6 * Impose a configurable time limit on patch generation. This is
necessary because bsdiff has some pathological worst-case behaviour,
  e.g. O(n^2) on files consisting only of 0s.
2010-11-29 09:15:45 +00:00
Eelco Dolstra 1a211d812f * Oops. 2010-11-17 17:54:49 +00:00
Eelco Dolstra a4f0365b2d * When checking whether a patch is applicable, for efficiency, use
`nix-store -q --hash' to get the hash of the base path rather than
  `nix-hash'.  However, only do this for estimating the size of a
  download, not for the actual substitution, because sometimes the
  contents of store paths are modified (which they shouldn't, of
  course).
2010-11-17 17:41:59 +00:00
Eelco Dolstra 3d38a49840 * In the download size indication, take binary patches into account.
Hopefully this doesn't slow things down too much.
2010-11-17 17:32:25 +00:00
Eelco Dolstra bdf089f463 * Before a build, show the disk space that the downloaded store paths
will approximately require.
2010-11-17 14:31:42 +00:00
Eelco Dolstra 06699d4219 * Store the NAR size in the manifest. 2010-11-17 12:57:52 +00:00