Commit graph

610 commits

Author SHA1 Message Date
Eelco Dolstra f4190c38ba Allow both bzip2 and xz compression 2012-07-01 18:46:38 -04:00
Eelco Dolstra 000132cbd1 nix-push: Don't pollute the current directory with result symlink 2012-06-29 18:30:28 -04:00
Eelco Dolstra 37f7098464 First attempt at the manifest-less substituter 2012-06-29 18:28:52 -04:00
Eelco Dolstra 91b8814f0d Doh 2012-06-29 17:16:00 -04:00
Eelco Dolstra 4911a10a4e Use XZ compression in binary caches
XZ compresses significantly better than bzip2.  Here are the
compression ratios and execution times (using 4 cores in parallel) on
my /var/run/current-system (3.1 GiB):

  bzip2: total compressed size 849.56 MiB, 30.8% [2m08]
  xz -6: total compressed size 641.84 MiB, 23.4% [6m53]
  xz -7: total compressed size 621.82 MiB, 22.6% [7m19]
  xz -8: total compressed size 599.33 MiB, 21.8% [7m18]
  xz -9: total compressed size 588.18 MiB, 21.4% [7m40]

Note that compression takes much longer.  More importantly, however,
decompression is much faster:

  bzip2: 1m47.274s
  xz -6: 0m55.446s
  xz -7: 0m54.119s
  xz -8: 0m52.388s
  xz -9: 0m51.842s

The only downside to using -9 is that decompression takes a fair
amount (~65 MB) of memory.
2012-06-29 15:24:52 -04:00
Eelco Dolstra 49cd7387ad nix-push: create a manifest-less binary cache
Manifests are a huge pain, since users need to run nix-pull directly
or indirectly to obtain them.  They tend to be large and lag behind
the available binaries; also, the downloaded manifests in
/nix/var/nix/manifest need to be in sync with the Nixpkgs sources.  So
we want to get rid of them.

The idea of manifest-free operation works as follows.  Nix is
configured with a set of URIs of binary caches, e.g.

  http://nixos.org/binary-cache

Whenever Nix needs a store path X, it checks each binary cache for the
existence of a file <CACHE-URI>/<SHA-256 hash of X>.narinfo, e.g.

  http://nixos.org/binary-cache/bi1gh9...ia17.narinfo

The .narinfo file contains the necessary information about the store
path that was formerly kept in the manifest, i.e., (relative) URI of
the compressed NAR, references, size, hash, etc.  For example:

  StorePath: /nix/store/xqp4l88cr9bxv01jinkz861mnc9p7qfi-neon-0.29.6
  URL: 1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2
  CompressedHash: sha256:1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd
  CompressedSize: 202542
  NarHash: sha256:1af26536781e6134ab84201b33408759fc59b36cc5530f57c0663f67b588e15f
  NarSize: 700440
  References: 043zrsanirjh8nbc5vqpjn93hhrf107f-bash-4.2-p24 cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 ...
  Deriver: 4idz1bgi58h3pazxr3akrw4fsr6zrf3r-neon-0.29.6.drv
  System: x86_64-linux

Nix then knows that it needs to download

  http://nixos.org/binary-cache/1bjxbg52l32wj8ww47sw9f4qz0r8n5vs71l93lcbgk2506v3cpfd.nar.bz2

to substitute the store path.

Note that the store directory is omitted from the References and
Deriver fields to save space, and that the URL can be relative to the
binary cache prefix.

This patch just makes nix-push create binary caches in this format.
The next step is to make a substituter that supports them.
2012-06-28 17:19:32 -04:00
Michel Alexandre Salim 6a214f3e06 Update nix profile: - incorporate NixOS's configuration so that nix is usable by normal users - install as a data file, not a program file 2012-05-31 08:59:36 -04:00
Eelco Dolstra 6814b1dfa1 Generate binary tarballs for installing Nix
For several platforms we don't currently have "native" Nix packages
(e.g. Mac OS X and FreeBSD).  This provides the next best thing: a
tarball containing the closure of Nix, plus a simple script
"nix-finish-install" that initialises the Nix database, registers the
paths in the closure as valid, and runs "nix-env -i /path/to/nix" to
initialise the user profile.

The tarball must be unpacked in the root directory.  It creates
/nix/store/... and /usr/bin/nix-finish-install.  Typical installation
is as follows:

  $ cd /
  $ tar xvf /path/to/nix-1.1pre1234_abcdef-x86_64-linux.tar.bz2
  $ nix-finish-install
  (if necessary add ~/.nix-profile/etc/profile.d/nix.sh to the shell
  login scripts)

After this, /usr/bin/nix-finish-install can be deleted, if desired.

The downside to the binary tarball is that it's pretty big (~55 MiB
for x86_64-linux).
2012-05-22 18:36:54 -04:00
Eelco Dolstra 591aab7e21 Remove $FONTCONFIG_FILE hack from nix-profile.sh
It's no longer needed because Nixpkgs' fontconfig uses
/etc/fonts/fonts.conf as a default, just like other distributions.
2012-05-22 14:00:08 -04:00
Eelco Dolstra 0c4c8f7a9d Remove obsolete files (moved to release tree) 2012-05-08 15:43:54 -04:00
Eelco Dolstra afa7b8a479 nix-channel --update: allow updating only the specified channels 2012-05-07 17:55:56 -04:00
Eelco Dolstra 147f10157f Now *really* prevent accumulation of old manifests 2012-05-07 17:23:26 -04:00
Eelco Dolstra e060c99447 Use mkpath instead of make_path
Perl <= 5.10 doesn't have make_path.  See
e.g. http://hydra.nixos.org/build/2493981.
2012-05-01 16:35:18 -04:00
Eelco Dolstra 59a26360c7 Support mandatory system features in the build hook
Mandatory features are features that MUST be present in a derivation's
requiredSystemFeatures attribute.  One application is performance
testing, where we have a dedicated machine to run performance tests
(and nothing else).  Then we would add the label "perf" to the
machine's mandatory features and to the performance testing
derivations.
2012-04-30 17:22:45 -04:00
Eelco Dolstra 6de5d53416 Fix a warning in the build hook about $progressViewer 2012-04-24 12:56:30 +02:00
Eelco Dolstra 259c6d3fbb Fix URL set by nix-channel 2012-04-18 13:40:18 +02:00
Eelco Dolstra 8cf1719e3e Hack to prevent accumulation of old manifests 2012-04-16 19:52:31 +02:00
Eelco Dolstra 0b5107c016 If the (redirected) channel URL contains a version number, use it 2012-04-14 21:05:28 +02:00
Eelco Dolstra 1d0bafb74d Follow redirects early to ensure consistency between the manifest and the tarball
Without this change, if the URL is a redirect that changes in between,
we can get a manifest that doesn't match the tarball.
2012-04-14 21:04:22 +02:00
Eelco Dolstra ef902274fd Remove unnecessary "system" argument 2012-04-14 18:48:11 +02:00
Eelco Dolstra e855c7e2c9 nix-channel improvements
"nix-channel --add" now accepts a second argument: the channel name.
This allows channels to have a nicer name than (say) nixpkgs_unstable.
If no name is given, it defaults to the last component of the URL
(with "-unstable" or "-stable" removed).

Also, channels are now stored in a profile
(/nix/var/nix/profiles/per-user/$USER/channels).  One advantage of
this is that it allows rollbacks (e.g. if "nix-channel --update" gives
an undesirable update).
2012-04-14 18:38:52 +02:00
Eelco Dolstra 969a14599d If "pv" is available at compile time, hard-code its path 2012-04-13 14:41:33 +02:00
Shea Levy 34a85c5405 nix-copy-closure: Move the progressViewer directly adjacent to the ssh call so that network progress is what's measured 2012-04-13 14:29:12 +02:00
Shea Levy 055e803851 Add the '--show-progress' flag to nix-copy-closure 2012-04-13 14:29:12 +02:00
Eelco Dolstra 1fb30e6d50 Handle the case where $SHELL is not set 2012-04-10 13:52:37 +02:00
Hydra mirror 3b859ead37 nix-generate-patches: Optionally write new patches to $NIX_ALL_PATCHES 2012-04-04 20:46:18 +00:00
Eelco Dolstra 0837348049 Follow redirects when checking for the existence of MANIFEST.bz2
If we don't follow redirects, we might think that MANIFEST.bz2 exists
just because the HEAD request succeeds on the redirector URI.
2012-04-04 15:41:35 +02:00
Eelco Dolstra b8fb0ce563 Update "nix-build --help" 2012-03-27 11:59:36 +02:00
Eelco Dolstra 5144abe5b6 Add an option "--exclude" to filter dependencies in "nix-build --run-env"
Sometimes when doing "nix-build --run-env" you don't want all
dependencies to be built.  For instance, if we want to do "--run-env"
on the "build" attribute in Hydra's release.nix (to get Hydra's build
environment), we don't want its "tarball" dependency to be built.  So
we can do:

  $ nix-build --run-env release.nix -A build --exclude 'hydra-tarball'

This will skip the dependency whose name matches the "hydra-tarball"
regular expression.  The "--exclude" option can be repeated any number
of times.
2012-03-27 11:40:47 +02:00
Eelco Dolstra 3e94ffffd6 Allow the command executed by "nix-build --run-env" to be overriden with --command 2012-03-27 11:16:43 +02:00
Eelco Dolstra 7f38087f35 Add a command "nix-build --run-env" to reproduce the environment of a derivation
This command builds or fetches all dependencies of the given
derivation, then starts a shell with the environment variables from
the derivation.  This shell also sources $stdenv/setup to initialise
the environment further.

The current directory is not changed.  Thus this is a convenient way
to reproduce a build environment in an existing working tree.

Existing environment variables are left untouched (unless the
derivation overrides them).  As a special hack, the original value of
$PATH is appended to the $PATH produced by $stdenv/setup.

Example session:

$ nix-build --run-env '<nixpkgs>' -A xterm
(the dependencies of xterm are built/fetched...)
$ tar xf $src
$ ./configure
$ make
$ emacs
(... hack source ...)
$ make
$ ./xterm
2012-03-19 04:14:21 +01:00
Eelco Dolstra d645633643 nix-copy-closure: don't print copied path on stdout
We're already printing progress on stderr, so printing them on stdout
afterwards is kind of useless.
2012-03-05 20:52:24 +01:00
Eelco Dolstra 2b4964f319 Restore progress indication during nix-copy-closure 2012-03-05 18:13:44 +01:00
Eelco Dolstra 56042a120a build-remote.pl: don't wait forever for the upload lock
In the build hook, don't wait forever to get the upload lock.  This
ensures progress if another process gets stuck while holding the
upload lock.
2012-03-05 17:58:09 +01:00
Eelco Dolstra ce86a9a0ae * nix-channel: fail if we don't have write permission to the manifests
directory.  Previously in this situation we did add the Nix
  expressions from the channel to allow installation from source, but
  this doesn't work for binary-only channels and leads to confusing
  error messages.
2012-02-02 12:25:56 +00:00
Eelco Dolstra 2a3f4110c5 * nix-build: put the temporary derivation symlink in a temporary
directory rather than the current directory.
* nix-build: --drv-link now implies --add-drv-link.
2012-01-13 23:35:07 +00:00
Eelco Dolstra d4e6b9f2d6 * Doh! 2012-01-05 21:07:12 +00:00
Eelco Dolstra a7366a764a * Add a -I flag to the Perl bindings to nix-build and some other
scripts.
* Include the version and architecture in the -I flag so that there is
  at least a chance that a Nix binary built for one Perl version will
  run on another version.
2012-01-05 20:33:46 +00:00
Eelco Dolstra 39d45a6b09 * Add a test for nix-channel.
* Refactor the nix-channel unpacker a bit.
2012-01-03 01:51:38 +00:00
Eelco Dolstra dadbb51d96 * Use Nix::Config. 2012-01-03 00:47:27 +00:00
Eelco Dolstra 48cea0d01e * Refactoring: Get rid of a few subdirectories in corepkgs/, and some
other simplifications.
* Use <nix/...> to locate the corepkgs.  This allows them to be
  overriden through $NIX_PATH.
* Use bash's pipefail option in the NAR builder so that we don't need
  to create a temporary file.
2012-01-03 00:16:29 +00:00
Eelco Dolstra 4d728bc3e6 * Security: make sure the lock files used by build-remote.pl are not
readable to other users.  Otherwise, any user can open the lock file
  for reading and lock it, thus DoSing the remote build mechanism.
2011-12-21 19:11:58 +00:00
Eelco Dolstra 69d6f0936a * Use a lock to ensure that only one build-remote instance can copy a
closure to a given machine at the same time.  This prevents the case
  where multiple instances try to copy the same missing store path to
  the target machine, which is very wasteful.
2011-12-21 18:59:25 +00:00
Eelco Dolstra a67b8ae224 * Typo. 2011-12-15 14:04:35 +00:00
Eelco Dolstra 3964d95abf * nix-prefetch-url: rewritten in Perl. 2011-12-02 12:09:50 +00:00
Eelco Dolstra 1749a7b0ae * download-using-manifests: use the Perl bindings. 2011-11-29 13:01:24 +00:00
Eelco Dolstra 216440b3ff * For consistency with "nix-store -q --hash", produce hashes in
base-32.  (This affects Hydra manifests.)
2011-11-29 12:32:55 +00:00
Eelco Dolstra 1df120cb05 * Get rid of the shell in ssh calls. 2011-11-29 10:51:22 +00:00
Eelco Dolstra d5ac78e0d6 * Add bzip2 and xz support to nix-copy-closure. 2011-11-23 15:29:58 +00:00
Eelco Dolstra 5bbd693cae * Add an API function exportPaths() that provides the functionality of
‘nix-store --export’.
* Add a Perl module that provides the functionality of
  ‘nix-copy-closure --to’.  This is used by build-remote.pl so it no
  longer needs to start a separate nix-copy-closure process.  Also, it
  uses the Perl API to do the export, so it doesn't need to start a
  separate nix-store process either.  As a result, nix-copy-closure
  and build-remote.pl should no longer fail on very large closures due
  to an "Argument list too long" error.  (Note that having very many
  dependencies in a single derivation can still fail because the
  environment can become too large.  Can't be helped though.)
2011-11-23 15:13:37 +00:00
Eelco Dolstra ab20af3e6f * build-remote.pl: drop a hard-coded reference to /nix/etc/nix. 2011-11-23 12:21:35 +00:00
Eelco Dolstra f8e609c341 * nix-pull: update the Nix manifest cache if necessary. Also, don't
read the manifest just to check the version and print the number of
  paths.  This makes nix-pull very fast for the cached cache (speeding
  up nixos-rebuild without the ‘--no-pull’ or ‘--fast’ options).
2011-11-16 16:41:48 +00:00
Eelco Dolstra d7d7910ba4 * Don't decompress the manifests in /nix/var/nix/manifest. This saves
disk space, and, since they're typically only decompressed once (to
  fill the manifest cache), doesn't make things slower.
2011-11-16 16:25:38 +00:00
Eelco Dolstra 00b41e46ed * Print a consistent message. 2011-10-19 21:34:13 +00:00
Eelco Dolstra f186a9141e * nix-copy-closure: support ‘--dry-run’ and ‘--include-outputs’. 2011-10-18 21:21:22 +00:00
Eelco Dolstra 6761757428 * Use the Store API bindings in nix-copy-closure. 2011-10-11 15:41:13 +00:00
Eelco Dolstra c362e4d718 * Move SSH.pm. 2011-10-11 11:45:36 +00:00
Eelco Dolstra 5090c34ee1 * Set the executable bit on scripts. 2011-10-10 22:40:17 +00:00
Eelco Dolstra cd6d02c366 2011-10-10 21:30:59 +00:00
Eelco Dolstra 6fcdbcac20 * Install NixManifest.pm, NixConfig.pm and GeneratePatches.pm under
the Nix:: namespace.
2011-10-10 21:11:08 +00:00
Eelco Dolstra 5bcdc7e351 * Update the cleanup script. 2011-08-27 16:59:32 +00:00
Eelco Dolstra a95ba4cdd9 * Use last_insert_id instead of sqlite_last_insert_rowid, which you're
not really supposed to use according to the DBD::SQLite docs, and
  fails on some systems (e.g. http://hydra.nixos.org/build/1246662).
2011-08-17 14:17:19 +00:00
Eelco Dolstra 1ecc97b6bd * Add a Nix expression search path feature. Paths between angle
brackets, e.g.

    import <nixpkgs/pkgs/lib>

  are resolved by looking them up relative to the elements listed in
  the search path.  This allows us to get rid of hacks like

    import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib"

  The search path can be specified through the ‘-I’ command-line flag
  and through the colon-separated ‘NIX_PATH’ environment variable,
  e.g.,

    $ nix-build -I /etc/nixos ...

  If a file is not found in the search path, an error message is
  lazily thrown.
2011-08-06 16:05:24 +00:00
Eelco Dolstra 0243eea4b9 * Create a symlink to /nix/var/nix/manifests in /nix/var/nix/gcroots
if it doesn't exist.
2011-07-20 11:47:00 +00:00
Eelco Dolstra e649f3168b * Fix concurrency issues in download-using-manifests' handling of the
SQLite manifest cache.  The DBI AutoCommit feature caused every
  process to have an active transaction at all times, which could
  indefinitely block processes wanting to update the manifest cache.

* Disable fsync() in the manifest cache because we don't need
  integrity (the cache can always be recreated if it gets corrupted).
2011-07-13 14:05:54 +00:00
Eelco Dolstra d94cb02bfe * nix-install-package: unset NIX_REMOTE because $NIX_MANIFESTS_DIR
doesn't work when building through the Nix daemon.  This also
  ensures an error message when the user doesn't have sufficient
  privileges to do nix-pull.
2011-04-19 10:52:35 +00:00
Eelco Dolstra 412914d004 * Read manifests directly into the database, rather than first reading
them into memory.  This brings memory use down to (more or less)
  O(1).  For instance, on my test case, the maximum resident size of
  download-using-manifests while filling the DB went from 142 MiB to
  11 MiB.
2011-04-11 13:16:54 +00:00
Eelco Dolstra 08c8971498 * Lock the database during updates. 2011-04-11 12:40:13 +00:00
Eelco Dolstra 3cbf680f5d * configure: detect whether DBD::SQLite is present. If necessary the
location to DBI and DBD::SQLite can be passed with --with-dbi and
  --with-dbd-sqlite.
2011-04-11 10:13:53 +00:00
Eelco Dolstra 83252b4ca9 * Subtle bug: if you import File::stat in one module, it affects other
modules as well.  So use symbolic field names everywhere (which is
  nicer anyway).
2011-04-11 08:21:30 +00:00
Eelco Dolstra b1882c3ef7 * Create $manifestDir if it doesn't exist. 2011-04-11 08:06:14 +00:00
Eelco Dolstra 5591fcc529 * Cache the manifests in /nix/var/nix/manifests in a SQLite database.
This significantly speeds up the download-using-manifests
  substituter, especially if manifests are very large.  For instance,
  one "nix-build -A geeqie" operation that updated four packages using
  binary patches went from 18.5s to 1.6s.  It also significantly
  reduces memory use.

  The cache is kept in /nix/var/nix/manifests/cache.sqlite.  It's
  updated automatically when manifests are added to or removed from
  /nix/var/nix/manifests.  It might be interesting to have nix-pull
  store manifests directly in the DB, rather than storing them as
  separate flat files, but then we would need a command line interface
  to delete manifests from the DB.
2011-04-10 23:22:46 +00:00
Eelco Dolstra 1e7e4f21ba * Remove the localPaths feature in manifests since it's no longer used
and redundant anyway.
2011-04-06 09:16:22 +00:00
Eelco Dolstra 0423d0692a * Print a better error message. 2011-03-16 15:55:57 +00:00
Eelco Dolstra 802c2651c8 2011-02-17 15:08:33 +00:00
Eelco Dolstra e945b52a31 * nix-push: handle the case where the hash is not set in the DB. 2011-02-17 13:26:33 +00:00
Eelco Dolstra d6c8b995c5 * In Hydra manifests the Size field is missing, so don't rely on it.
This caused a lot of "Use of uninitialized value" warnings from
  Perl.
2010-12-22 10:32:32 +00:00
Eelco Dolstra a0be433fec * Disable X11 forwarding, it's not needed. 2010-12-15 14:25:54 +00:00
Eelco Dolstra f1a6b97639 * nix-copy-closure: make sure that the shell doesn't do globbing of
`+' and `?' in filenames.  This is very slow if /nix/store is very
  large.  (This is a quick hack - a cleaner solution would be to
  bypass the shell entirely.)
2010-12-15 08:39:37 +00:00
Eelco Dolstra d787285af9 * nix-instantiate: return exit status 100 to denote a permanent build
failure.  The build hook can use this to distinguish between
  transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
Eelco Dolstra 542fc69062 * When doing a query (e.g. `nix-store -r --dry-run'), don't make a lot
of expensive calls to `nix-store --check-validity'.
2010-12-13 08:39:10 +00:00
Eelco Dolstra 7d0444e244 * Bad things happen when a filehandle has the same name as a module. 2010-12-07 12:33:42 +00:00
Eelco Dolstra 365f3028dd * Use CamelCase for the Perl modules. 2010-12-05 17:50:29 +00:00
Eelco Dolstra f42a505ab7 * Add a script `nix-generate-patches'.
* Fix the binary patching test.
2010-12-05 17:36:02 +00:00
Eelco Dolstra d3bba0c2d8 * Move the patch generator into a module. 2010-11-29 15:26:28 +00:00
Eelco Dolstra 9737a7eba0 * Don't generate patches if the system type differs. 2010-11-29 09:26:05 +00:00
Eelco Dolstra bd48fd97f6 * Impose a configurable time limit on patch generation. This is
necessary because bsdiff has some pathological worst-case behaviour,
  e.g. O(n^2) on files consisting only of 0s.
2010-11-29 09:15:45 +00:00
Eelco Dolstra 1a211d812f * Oops. 2010-11-17 17:54:49 +00:00
Eelco Dolstra a4f0365b2d * When checking whether a patch is applicable, for efficiency, use
`nix-store -q --hash' to get the hash of the base path rather than
  `nix-hash'.  However, only do this for estimating the size of a
  download, not for the actual substitution, because sometimes the
  contents of store paths are modified (which they shouldn't, of
  course).
2010-11-17 17:41:59 +00:00
Eelco Dolstra 3d38a49840 * In the download size indication, take binary patches into account.
Hopefully this doesn't slow things down too much.
2010-11-17 17:32:25 +00:00
Eelco Dolstra bdf089f463 * Before a build, show the disk space that the downloaded store paths
will approximately require.
2010-11-17 14:31:42 +00:00
Eelco Dolstra 06699d4219 * Store the NAR size in the manifest. 2010-11-17 12:57:52 +00:00
Eelco Dolstra 5693b8a7e2 * nix-push: no need to compute the NAR hash, since the Nix database
already has it (`nix-store -q --hash').
2010-11-17 12:51:54 +00:00
Eelco Dolstra bfa6ee7d91 * Don't use SSH's `-tt' flag because it doesn't seem to work
on OpenSolaris when using connection sharing.  Instead have
  the remote side check for disconnection and kill the process 
  group when that happens.
2010-10-04 12:30:46 +00:00
Eelco Dolstra bf0dde9597 * Always print hook output on stderr, even if --no-build-output is
set.
* In the build hook, print a trace message to allow Hydra to pick up
  the name of the remote machine used for the build.
2010-08-31 12:36:24 +00:00
Eelco Dolstra 80e722278c * When using the build hook, distinguish between the stderr of the
hook script proper, and the stdout/stderr of the builder.  Only the
  latter should be saved in /nix/var/log/nix/drvs.
* Allow the verbosity to be set through an option.
* Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
Eelco Dolstra 766f708418 * Experimental feature: allow a derivation to tell the build hook that
it requires a certain feature on the build machine, e.g.

    requiredSystemFeatures = [ "kvm" ];

  We need this in Hydra to make sure that builds that require KVM
  support are forwarded to machines that have KVM support.  Probably
  this should also be enforced for local builds.
2010-08-27 13:18:13 +00:00
Eelco Dolstra e437b08250 * Made the build hook mechanism more efficient. Rather than starting
the hook every time we want to ask whether we can run a remote build
  (which can be very often), we now reuse a hook process for answering
  those queries until it accepts a build.  So if there are N
  derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
Eelco Dolstra 1a396f3789 * Don't call "cat". 2010-08-25 12:19:30 +00:00
Eelco Dolstra 95deba581d * In the build hook, temporarily register the derivation and its
output as GC roots.  This prevents a race if the garbage collector
  is running during the build.
2010-08-25 11:54:11 +00:00
Eelco Dolstra 1e5f5ea2e9 * Correctly handle SSH failing to establish a connection. Because
this didn't work right, the build hook wouldn't fall back to using
  other machines of the desired type.
2010-08-24 14:27:07 +00:00
Eelco Dolstra 5f9aad44ca * nix-build: recognise --cores. 2010-08-11 15:28:02 +00:00
Eelco Dolstra 587dc8aa00 * Sync with the trunk. 2010-08-04 17:48:29 +00:00
Eelco Dolstra 532d766c27 * Don't barf if the source NAR for a patch has disappeared. 2010-07-21 11:30:23 +00:00
Eelco Dolstra 560ab22f7d * Ignore packages that don't have a version.
* Work on a manifest instead of a channel directory.
2010-06-23 21:11:33 +00:00
Eelco Dolstra 87ef5907e9 * Sync. 2010-06-22 14:42:34 +00:00
Eelco Dolstra 819548d92f * Pass `--fallback' to the remote build to ignore failing
substituters.
2010-06-22 14:41:22 +00:00
Eelco Dolstra c778ed1768 * Fix the copy-from-other-stores substituter. 2010-04-26 13:39:55 +00:00
Eelco Dolstra 05fbf61f0e * nix-prefetch-url depends on sed (see
http://hydra.nixos.org/build/311170).
2010-03-05 18:26:47 +00:00
Eelco Dolstra 68e55cd9da * Since Hydra now escapes +', =' and `?' to %NN in URIs,
nix-prefetch-url should unescape them, because `%' is not a valid
  character in store path names.
2010-02-16 00:10:39 +00:00
Eelco Dolstra 719cebcac2 * "Fix" incorrect help message. 2010-02-04 09:38:09 +00:00
Eelco Dolstra 443673620d * Don't use ssh's -f flag since it leads to lots of lingering ssh
processes.
2010-02-04 02:38:40 +00:00
Eelco Dolstra 7ec5a65925 * Doh! The scope of $slotLock should extend to the end of the script,
because otherwise the lock will be released at the end of the while
  loop.
2010-02-04 02:18:29 +00:00
Eelco Dolstra c9b2d80bcd * Typo. 2010-02-04 02:05:34 +00:00
Eelco Dolstra 3cfe65e516 * Doh! Calling `system' in an END block causes the exit status in $?
to be changed to 0.
2010-02-04 02:05:22 +00:00
Eelco Dolstra e51a276907 * Remove the `cat' calls when not using --gzip. 2010-02-04 01:39:23 +00:00
Eelco Dolstra 408913bbaf * Revert r19796 for now. 2010-02-04 00:12:57 +00:00
Eelco Dolstra c45de33c67 * Respect @sysconfdir@. 2010-02-03 21:13:37 +00:00
Eelco Dolstra d0c32dc135 * In the build hook, if connecting to a machine fails, try the other
machines of the right type (if available).  This makes the build
  farm more robust to failures.
2010-02-03 20:35:37 +00:00
Eelco Dolstra f56a039775 * Use SSH connection sharing in the remote build script.
* Removed the Cygwin password hack since the problem is apparently
  fixed in Visual Studio.
2010-02-03 20:12:18 +00:00
Eelco Dolstra bc1e478db1 * nix-copy-closure: start only one SSH connection to the server, or
recycle an already existing connection (using  OpenSSH's connection
  sharing feature).
2010-02-03 15:34:52 +00:00
Eelco Dolstra e839802720 * Don't do a chdir to $tmpDir. It's not necessary, and Windows doesn't
support deleting the current directory.
2010-02-02 15:29:18 +00:00
Eelco Dolstra 50e34891f0 * Disable gzip compression in build-remote.pl because it puts too much
load on the Hydra build farm (where it's unnecessary anyway because
  it has a fast connection to the build machines).  In any case,
  compression can be enabled by using the `-C' option to ssh.
2010-01-25 16:14:45 +00:00
Eelco Dolstra 7ca9972636 * When doing a nix-pull, remove old manifests downloaded from the same
URL.  This prevents lots of old cruft accumulating in
  /nix/var/nix/manifests.
2009-12-09 19:36:54 +00:00
Eelco Dolstra c4c84d1edb * nix-build: be less verbose. 2009-12-09 18:08:28 +00:00
Eelco Dolstra 997db91e07 * Don't pass -K. It should really inherit the setting of the calling
Nix though.
2009-11-17 16:22:39 +00:00
Eelco Dolstra 3392d32e8b * In nix-pull/nix-channel, create the manifests directory if it
doesn't exist.  The Debian packages don't include the manifests
  directory, so nix-channel would silently skip doing a nix-pull,
  resulting in everything being built from source.  Thanks to Juan
  Pedro Bolívar Puente.
2009-11-13 10:08:31 +00:00
Eelco Dolstra e1df4ef73c 2009-09-17 17:02:14 +00:00
Eelco Dolstra 86408b3f47 * build-remote.pl: Pick machines in a round-robin order, rather than
giving jobs to the first machine until it hits its job limit, then
  the second machine and so on.  This should improve utilisation of
  the Hydra build farm a lot.  Also take an optional speed factor
  into account to cause fast machines to be preferred over slower
  machines with a similar load.
2009-09-17 15:48:17 +00:00
Eelco Dolstra 57e0d73c77 * build-remote.pl: allow the system type to be a comma-separated list
of system types.  Don't treat the x86_64-linux system type
  specially.
2009-09-17 13:51:04 +00:00
Eelco Dolstra 20b6f94b65 * nix-build: pass the --show-trace flag. 2009-07-15 09:10:38 +00:00
Eelco Dolstra f24cf5d303 * nix-build: if nix-instantiate or nix-store fails due to a signal
(e.g. out of memory or a segfault), print an error message.
  Otherwise the user doesn't see anything.
2009-06-10 11:30:34 +00:00
Eelco Dolstra c710fe540e * Machines of type x86_64-linux can do i686-linux builds. 2009-04-21 12:42:47 +00:00
Eelco Dolstra 0460ea4c39 * Cleanup. 2009-04-16 09:27:33 +00:00
Eelco Dolstra 096affb55b * Update the mtime on the lock file to make it easy to see when a slot
was last used.
2009-03-29 18:40:44 +00:00
Eelco Dolstra 737423a89c * Use polling to wait for a remote build slot when using a build hook
(that is, call the build hook with a certain interval until it
  accepts the build).
* build-remote.pl was totally broken: for all system types other than
  the local system type, it would send all builds to the *first*
  machine of the appropriate type.
2009-03-29 18:00:45 +00:00
Eelco Dolstra 47706e3924 * nix-copy-closure: compute the closure in one call to nix-store,
which is much faster.
2009-03-28 21:10:29 +00:00
Eelco Dolstra 3a2bbe7f8a * Simplify communication with the hook a bit (don't use file
descriptors 3/4, just use stdin/stderr).
2009-03-28 19:29:55 +00:00
Eelco Dolstra 7fb548aa26 * Clean up the output a bit. 2009-03-28 16:12:46 +00:00
Eelco Dolstra 7e05b8b75e * Future proofing: assume we can read manifests up to version 10
(which should therefore be backwards compatible).
2009-03-19 10:02:02 +00:00
Eelco Dolstra b88460bcbc * Disregard the Hash field in manifests. 2009-03-19 09:47:34 +00:00
Eelco Dolstra 2897286487 * Unify exportReferencesGraph and exportBuildReferencesGraph, and make
sure that it works as expected when you pass it a derivation.  That
  is, we have to make sure that all build-time dependencies are built,
  and that they are all in the input closure (otherwise remote builds
  might fail, for example).  This is ensured at instantiation time by
  adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 17:36:42 +00:00
Eelco Dolstra 1dcf208f56 * Clean up some tests (use nix-build where appropriate). 2009-03-18 13:15:55 +00:00
Eelco Dolstra 1273d355ac * nix-install-package: don't pollute /nix/var/nix/manifests. 2009-02-27 14:06:38 +00:00
Eelco Dolstra d4ca5c3952 * Use the regular progress bar; the hash bar isn't very useful when
the size of the download isn't known in advance.
2009-02-27 13:55:11 +00:00
Eelco Dolstra fd2e14b3c8 * nix-build: support --option. 2009-02-27 12:09:30 +00:00
Eelco Dolstra 8ab6bc5a49 * nix-channel: use nix-build. 2009-02-27 11:01:03 +00:00
Eelco Dolstra f052c10eed * Check the manifest version. 2009-02-27 09:53:58 +00:00
Eelco Dolstra a7cee528c5 * Handle base-16 hashes in manifests. 2009-02-26 21:12:35 +00:00
Eelco Dolstra 041717eda3 * download-using-manifests: don't check the cryptographic hash of
downloaded files; rather, check the hash of the unpacked store
  path.

  When the server produces bzipped NAR archives on demand (like Hydra
  does), the hash of the file is not known in advance; it's streamed
  from the server.  Thus the manifest doesn't contain a hash for the
  bzipped NAR archive.  However, the server does know the hash of the
  *uncompressed* NAR archive (the "NarHash" field), since it's stored
  in the Nix database (nix-store -q --hash /nix/store/bla).  So we use
  that instead for checking the integrity of the download.
2009-02-19 23:46:37 +00:00
Michael Raskin 621093cb1c Replace wrong (w.r.t. PATH) sed call with in-shell substitution 2009-02-19 20:46:45 +00:00
Eelco Dolstra 1bb0f1e84b * Build hook: compress the transferred data. 2009-02-03 10:34:15 +00:00
Eelco Dolstra b682fae9d9 * Build hooks: use nix-store --import. This prevents a redundant
scan for runtime dependencies (i.e. the local machine shouldn't do a
  scan that the remote machine has already done).  Also pipe directly
  into `nix-store --import': don't use a temporary file.
2009-02-02 17:24:10 +00:00
Eelco Dolstra 019176137f * When using a build hook, distinguish between transient failures
(e.g. an SSH connection problem) and permanent failures (i.e. the
  builder failed).  This matters to Hydra (it wants to know whether it
  makes sense to retry a build).
2009-01-13 11:39:09 +00:00
Eelco Dolstra 4ce692df88 2009-01-13 10:42:13 +00:00
Eelco Dolstra d91dc086bb 2008-12-04 16:51:36 +00:00
Eelco Dolstra 9ac3f5df9c * Propagate --max-silent-time to remote machines. 2008-12-04 16:51:16 +00:00
Eelco Dolstra bcfe98acff * Prefer building on a remote machine over a local machine. This
makes more sense for the build farm, otherwise every nix-store
  invocation will lead to at least one local build.  Will come up with
  a better solution later...
2008-12-04 16:35:47 +00:00
Eelco Dolstra 909fbb9de1 * When using build hooks, for any nix-store -r build operation, it is
necessary that at least one build hook doesn't return "postpone",
  otherwise nix-store will barf ("waiting for a build slot, yet there
  are no running children").  So inform the build hook when this is
  the case, so that it can start a build even when that would exceed
  the maximum load on a machine.
2008-12-04 14:29:41 +00:00
Eelco Dolstra 5dfba0b4db * Force allocation of a pseudo-terminal to clean up the remote
nix-store process when the connection is interrupted.
2008-12-04 13:36:52 +00:00
Eelco Dolstra 9fd9c4c635 * Support multiple system types per remote machine, e.g. a machine
list like

    root@example.org x86_64-linux /root/.ssh/id_buildfarm 1
    root@example.org i686-darwin  /root/.ssh/id_buildfarm 1

  This is possible when the Nix installation on example.org itself has
  remote builds enabled.
2008-12-04 12:20:06 +00:00
Eelco Dolstra f8713e1287 * Dirty hack to make nix-push work properly on derivations: the
derivation should be a source rather than a derivation dependency of
  the call to the NAR derivation.  Otherwise the derivation (and all
  its dependencies) will be built as a side-effect, which may not even
  succeed.
2008-12-04 10:40:41 +00:00
Eelco Dolstra 6bbff48079 2008-11-20 17:22:42 +00:00
Eelco Dolstra 4213b8d8ec * Urgh. 2008-11-20 15:44:59 +00:00
Eelco Dolstra aab530e971 * Primop builtins.storePath for declaring a store path as a
dependency.  `storePath /nix/store/bla' gives exactly the same
  result as `toPath /nix/store/bla', except that the former includes
  /nix/store/bla in the dependency context of the string.

  Useful in some generated Nix expressions like nix-push, which now
  finally does the right thing wrt distributed builds.  (Previously
  the path to be packed wasn't an explicit dependency, so it wouldn't
  be copied to the remote machine.)
2008-11-19 23:26:19 +00:00
Eelco Dolstra fa791116a3 * Get rid of nix-pack-closure / nix-unpack-closure, they're redundant. 2008-11-18 14:43:40 +00:00
Eelco Dolstra 2b2aa8a820 * Doh. 2008-09-08 11:02:15 +00:00
Michael Raskin 2a01d06da6 Added nix-http-export.cgi to extra distributed scripts; so it can be installed from releases, not only from SVN. 2008-08-29 08:34:38 +00:00
Michael Raskin b455c4c45c Updates to nix-reduce-build
Common code in local build package sources refactored out in a function; before building the real set of derivations needed is found (slightly slower for only one build strategy, but less garbage on output and better performance for multiple build strategies).

Now you have full choice of best-effort build regardless of method (substituters or actual build), using substituters, building only fixed derivations (should get you all the downloads) and local build without even trying substituters. 

Some minor fix in the help text about behavior with no package sources.
2008-08-06 19:43:53 +00:00
Eelco Dolstra 7592f48c83 * nix-build: `--dry-run' flag. 2008-08-04 13:46:01 +00:00
Eelco Dolstra 42043953c3 * Doh. 2008-08-04 13:15:47 +00:00
Michael Raskin b3c26180e3 Updates to nix-reduce-build: only realize fixed derivations if user asks so, or only use substituters. Oh, and add possibility to use : for things like /etc/nixos/nixpkgs:-A:gnused 2008-08-02 16:43:25 +00:00
Eelco Dolstra 3c92ea399d * Make nix-env --dry-run print the paths to be substituted correctly
again.  (After the previous substituter mechanism refactoring I
  didn't update the code that obtains the references of substitutable
  paths.)  This required some refactoring: the substituter programs
  are now kept running and receive/respond to info requests via
  stdin/stdout.
2008-08-02 12:54:35 +00:00
Eelco Dolstra 096198d11f * A quick hack to make nix-prefetch-url support mirror:// URLs. It
requires that $NIXPKGS_ALL points at a Nixpkgs tree.
2008-07-23 16:02:58 +00:00
Eelco Dolstra 660244f65f * Make sure that copy-from-other-stores.pl is built. 2008-07-23 09:38:38 +00:00
Eelco Dolstra 989176c56e * Allow read-only access to the store (e.g., non-root users on NixOS
can do operations like "nix-store -qR <path>" even without the Nix
  daemon).
2008-07-18 15:34:46 +00:00
Eelco Dolstra 8bc591a6f0 * Use the copy-from-other-stores substituter by default. Of course,
it only does something if $NIX_OTHER_STORES (not really a good
  name...) is set.
* Do globbing on the elements of $NIX_OTHER_STORES.  E.g. you could
  set it to /mnts/*/nix or something.
* Install substituters in libexec/nix/substituters.
2008-07-18 13:05:10 +00:00
Eelco Dolstra 15f39aba8c * Quick prototype of a substituter that copies paths from other Nix
stores (typically remote Nix stores mounted via e.g. NFS, or the Nix
  store on the NixOS installation CD).  Example use:

  $ sshfs foo@example.org:/ /mnt
  $ NIX_OTHER_STORES=/mnt/nix \
    NIX_SUBSTITUTERS=.../copy-from-other-stores.pl \
    nix-env -i foo

  This will be especially useful for the installation CD since it
  doesn't require a manifest for the CD contents.
2008-07-12 18:58:24 +00:00
Michael Raskin 955b8841cd Also trying to build derivers in case we cannot get substituters 2008-06-14 08:48:40 +00:00
Michael Raskin 18e27629d3 Added local best-effort builds (i.e. one failure does not ruin all packages you would like to see built) 2008-06-14 08:30:35 +00:00
Michael Raskin 194c66eeeb Stupid error in script 2008-06-13 14:34:19 +00:00
Michael Raskin f903d86740 OK, I will believe that fix does no worse.. 2008-06-13 13:53:14 +00:00
Michael Raskin ce85b55cf0 Updated help text 2008-06-12 17:45:38 +00:00
Michael Raskin 4532e4b90d Added verbosity for nix-reduce-build 2008-06-12 16:26:53 +00:00
Michael Raskin 9819bb20da Added support for file:// archive (.nar.gz) repositories to nix-reduce-build. /tmp/nix-export created by nix-http-export.cgi is OK. 2008-05-11 15:54:30 +00:00
Michael Raskin b4bc8b7616 --proxy=proxy:3128 2008-05-07 14:18:28 +00:00
Michael Raskin b1e321d6ce Added http alternative transport for nix-reduce-build 2008-04-29 04:03:54 +00:00
Eelco Dolstra f8985d195e * Fix for NIX-101 (should use an absolute path for call to nix-hash). 2008-03-28 17:52:33 +00:00
Eelco Dolstra f106868110 * Cleanup. 2008-03-20 10:16:36 +00:00
Eelco Dolstra d7caac3e81 * Don't install nix-reduce-build by default yet please, I first want
to understand better what it does...
2008-02-28 21:27:47 +00:00
Michael Raskin 11d512e7a8 Added nix-reduce-build. You point it to some path you want to build and it fetches whatever it can from specified computers via nix-copy-closure. NOTE: You do want to set up RSA keys or ssh-agent or something... You really do want it. It will run separate ssh instances insane number of times. 2008-02-27 21:26:47 +00:00
Eelco Dolstra de012e76e2 * Don't use "store expression", it's obsolete. 2007-12-30 23:35:45 +00:00
Eelco Dolstra c370755583 * Flag `--no-build-hook' to disable distributed builds.
* queryDeriver in daemon mode: don't barf if the other side returns an
  empty string (which means there is no deriver).
2007-11-16 16:15:26 +00:00
Eelco Dolstra 6784b14241 * nix-build: pass --argstr to nix-instantiate. 2007-11-15 16:52:40 +00:00
Eelco Dolstra ca210d2a58 * Doh! 2007-11-15 15:33:30 +00:00
Eelco Dolstra bfec245efa * Add build-remote.pl to the Nix distribution. 2007-11-15 14:28:08 +00:00
Eelco Dolstra 5b74a59570 * nix-prefetch-url: don't fail if /tmp/nix-prefetch-url-<pid> exists,
instead use a counter just like we do for temporary build
  directories.
2007-11-05 18:12:29 +00:00
Eelco Dolstra f8ed2e64a5 2007-10-22 12:05:18 +00:00
Eelco Dolstra a8629de827 * New command `nix-store --optimise' to reduce Nix store disk space
usage by finding identical files in the store and hard-linking them
  to each other.  It typically reduces the size of the store by
  something like 25-35%.  This is what the optimise-store.pl script
  did, but the new command is faster and more correct (it's safe wrt
  garbage collection and concurrent builds).
2007-10-09 22:14:27 +00:00
Eelco Dolstra 26f981c2e5 * Remove garbage. 2007-09-18 08:26:55 +00:00