Commit graph

258 commits

Author SHA1 Message Date
Eelco Dolstra 947e645789 * Hide warnings about a missing "lsof" (NIX-54). 2006-09-20 15:14:19 +00:00
Eelco Dolstra a060adf165 * Use paths, not strings, when calling the function that generates
NARs.  Fixes the impurity of nix-push (NIX-21).
* Better help.
2006-09-20 15:04:04 +00:00
Eelco Dolstra 7dd342e482 * Doh. 2006-09-19 16:40:22 +00:00
Eelco Dolstra ee6cf99660 * Doh! Of course we have to take execute permission into account.
* Restore the mtime on modified directories.
2006-09-19 16:14:15 +00:00
Eelco Dolstra 6dbed1bf38 * `optimise-store.pl' reduces disk space consumption by hard-linking
all identitical files in the Nix store to each other.  (Previously
  it only computed the size that would be saved by doing so.)
2006-09-19 14:58:35 +00:00
Eelco Dolstra 07cec27848 * Cleanups. 2006-09-19 14:27:28 +00:00
Eelco Dolstra 9488ae7357 * `show-duplication.pl', a small utility that shows the amount of
package duplication present in (e.g.) a profile.  It shows the
  number of instances of each package in a closure, along with the
  size in bytes of each instance as well as the "waste" (the
  difference between the sum of the sizes of all instances and the
  average size).

  $ ./show-duplication.pl /nix/var/nix/profiles/default
  gcc 11
    3.3.6 19293318
    3.4.4 21425257
    ...
    average 14942970, waste 149429707
  coreutils 6
  ...
  average package duplication 1.87628865979381, total size 3486330471, total waste 1335324237, 38.3017114443825% wasted

  This utility is useful for measuring the cost in terms of disk space
  of the Nix approach.
2006-09-19 13:53:35 +00:00
Eelco Dolstra 2e210b2387 * Convenience option `nix-collect-garbage -d' (--delete-old): removes
old generations of *all* profiles in /nix/var/nix/profiles, then
  runs the garbage collector.  Quick way to get rid of all old stuff.
  Of course, one cannot roll back to earlier points in time after
  this.
2006-09-06 14:23:04 +00:00
Eelco Dolstra 3e239a37ff * file:/ -> file:// 2006-08-09 19:37:23 +00:00
Armijn Hemel c0bfcbdd45 rework the --target flag. If this flag is used, a URI should be given. Default
values are not changed.
2006-08-08 15:42:33 +00:00
Eelco Dolstra 5a6b45e252 * In nar.nix, path -> storePath, otherwise we get a collision between
environment variable names on Cygwin (where they are case
  insensitive).
2006-08-07 18:22:57 +00:00
Armijn Hemel a61129c48c add coreutils to the default PATH for this scripts, so we know for sure we have tools like rm, mkdir, and so on 2006-08-05 00:33:52 +00:00
Armijn Hemel f1947cce93 prevent doing recursive chroots, by unsetting NIX_ROOT in the scripts. 2006-08-05 00:31:04 +00:00
Eelco Dolstra f4a502a09a * Use old-school pipe opens; the new style is Perl >= 5.8.0. 2006-08-04 11:51:46 +00:00
Eelco Dolstra 981eff065b * Remove the dependency on `date', use strftime instead. 2006-08-04 11:45:50 +00:00
Armijn Hemel fcb784051f urgh...for some weird reason this one-liner was not in svn. PEBKAC! 2006-08-03 11:48:09 +00:00
Eelco Dolstra a29b64a231 * Better error checking. 2006-08-01 13:50:10 +00:00
Armijn Hemel 4fde308ec0 add a flag --target, so we can override the URL in the MANIFEST file. This
is only for local copies (so file:///)
2006-08-01 13:15:55 +00:00
Eelco Dolstra 4661282fde * `nix-instantiate ... --arg NAME VALUE': allow arguments to be passed
to functions from the command line.
* nix-build: started removing backticks.
2006-07-28 16:03:28 +00:00
Eelco Dolstra dcded7da47 * Don't try to do DNS lookups. 2006-07-21 12:28:25 +00:00
Eelco Dolstra ebcccbd358 * Added a tool to find additional roots for the garbage collector,
such as open files, current directories, mmaped files, etc.  This is
  inherently unportable, but it's easy to adapt this script to other
  platforms.  Currently we call `lsof' and try to read various bits in
  /proc/NNN.

  The goal is to prevent the garbage collector from removing store
  paths that are no longer reachable from a permanent root but that
  are still in use (for instance, after the user has done "nix-env -e"
  on a running program).
2006-07-19 16:49:59 +00:00
Eelco Dolstra 04cf72287b * This may be useful in the future. 2006-05-31 09:24:54 +00:00
Eelco Dolstra e3c07782d1 * Remove old manifests in `nix-channel --update'. 2006-05-08 20:00:28 +00:00
Eelco Dolstra 05bb644890 * Allow the resulting symlink of nix-build to be named, e.g.,
$ nix-build .../i686-linux.nix -A apacheHttpd -o apache
2006-03-14 16:35:01 +00:00
Eelco Dolstra 9811815429 * Write messages to stderr, not stdout. 2006-03-10 09:41:28 +00:00
Eelco Dolstra 120f00c04f * More online help. 2006-03-03 14:25:07 +00:00
Eelco Dolstra 2d54312f87 * Rewrote nix-build in Perl, since sh is just too limited (turns out
that arrays are a bash extension, so it didn't work on FreeBSD).
  Also fixes NIX-8 (readlink(1) dependency).
2006-03-03 14:15:02 +00:00
Eelco Dolstra 85793fa438 * Remove debug statement. 2006-02-24 17:10:38 +00:00
Eelco Dolstra e2f16b9cae * A script to remove from a manifest those patches whose base or
target no longer applies to any available release.  This is a
  partial fix for NIX-34 (when producing linear patch sequences
  between releases, the number of patches grows without bound).
2006-02-24 16:05:55 +00:00
Eelco Dolstra 5168f9bb00 * Canonicalise manifests a bit by sorting them. 2006-02-24 16:02:36 +00:00
Eelco Dolstra 43fa1c20e7 * Separate the cache reading code. 2006-02-22 14:35:19 +00:00
Eelco Dolstra 928cce5922 * Directory for nix.cs.uu.nl manifest / cache maintenance scripts. 2006-02-22 14:13:02 +00:00
Eelco Dolstra fcec32a2c5 * Use right directories for NAR files and patches.
* Print errors to STDERR.
2006-02-22 14:10:02 +00:00
Eelco Dolstra 982399bb14 * Enable the --attr in nix-build as well (and add -A as an alias).
Example:

    $ nix-build ./all-packages.nix -A xlibs.libX11

  So finally it's easy to perform a test build of a Nix expression!
2006-02-10 17:37:35 +00:00
Eelco Dolstra 151f10dbc7 * Add @bindir@. 2006-02-01 12:41:54 +00:00
Eelco Dolstra 83424eb71b * Don't force a build of derivations. 2006-01-26 23:18:46 +00:00
Eelco Dolstra e4d4969ae9 * New tools nix-pack-closure and nix-unpack-closure. These provide a
useful way to transfer the closure of a store path to another
  machine.

  These commands provide functionality previously possible through
  `nix-push --copy'.  However, they are much more convenient in many
  situations (though possibly less efficient).
  
  Example:
  $ nix-pack-closure /nix/store/hj232g1r...-subversion-1.3.0 > svn.closure
  (on another machine:)
  $ nix-unpack-closure < svn.closure

  Note that Subversion is added to the store, but not installed into a
  user environment.  One should do `nix-env -i
  /nix/store/hj232g1r...-subversion-1.3.0' for that.

  Another example: copy the application Azureus to the machine
  `scratchy' through ssh:
  
  $ nix-pack-closure $(which azureus) | ssh scratchy nix-unpack-closure
2006-01-12 15:17:51 +00:00
Eelco Dolstra ab5c6bb3a3 * Change referer' to referrer' throughout. In particular, the
nix-store query options `--referer' and `--referer-closure' have
  been changed to `--referrer' and `--referrer-closure' (but the old
  ones are still accepted for compatibility).
2005-12-13 21:04:48 +00:00
Eelco Dolstra dc528128cc * FreeBSD compatibility fix. 2005-11-17 13:58:23 +00:00
Eelco Dolstra 221c79013f * Turn off build hooks in nix-push because of an impurity (NIX-21). 2005-10-29 18:17:45 +00:00
Armijn Hemel 15ff877438 add @coreutils@ to correctly use coreutils to create a profile. This is needed
for NixOS, where we might not know our PATH in advance.
2005-10-11 17:30:57 +00:00
Eelco Dolstra 607a2f01e6 * Remove other uses of IPC::Open2. 2005-09-21 17:14:52 +00:00
Eelco Dolstra 95304172a5 * Don't use IPC::Open2, it has a subtle race bug on Mac OS X 10.4. If
the parent runs before the child, it closes some pipe file
  descriptors which causes the child to fail due to a bad file
  descriptor.  So we just use the normal open() function instead.
  
  This fixes NIX-14 (intermittent nix-pull failures).
2005-09-21 17:06:06 +00:00
Eelco Dolstra d34fa9a6cc * Remove dead file. 2005-09-16 09:05:18 +00:00
Eelco Dolstra 846b53bde4 * Set the current directory to something well-defined. Might help in
setuid installations.
2005-09-16 09:03:24 +00:00
Eelco Dolstra 7f384d9c1b * Use a proper temporary directory. 2005-09-15 15:21:35 +00:00
Eelco Dolstra a5ceb5bc0b * nix-build: default to `./default.nix' if no paths are specified.
So when using Nix as a build tool, you can just say `nix-build' and
  it will build the top-level derivation defined in `default.nix'.
2005-07-13 17:39:10 +00:00
Eelco Dolstra 928a7c06dc * Don't create patches for archives >= 150 MB because bsdiff can't
handle it.  It crashed on the 234 MB tetex archive.  Probably we
  will never be able to handle archives of that size on 32-bit
  machines (because bsdiff does everything in memory requiring
  max(17*n,9*n+m)+O(1) bytes, so the address space simply isn't
  there).
2005-06-18 14:20:24 +00:00
Eelco Dolstra cbc8d083ac * Make unpacking of patch sequences much faster by not doing redundant
unpacking and repacking of intermediate paths.
2005-05-10 14:22:36 +00:00
Eelco Dolstra 9f3601a36c * Argh! The patch downloader was broken due to the renaming of the
`--isvalid' flag in nix-store.
2005-04-12 10:51:38 +00:00
Eelco Dolstra d5219a351a * Damn. Disable the USE heuristic for now, since the deriver in the
database isn't always in the manifest (so the reference graph cannot
  be reconstructed fully).
2005-04-12 10:07:02 +00:00
Eelco Dolstra c702dfca3f * nix-store: --substitute' -> --register-substitutes'. 2005-04-08 13:48:41 +00:00
Eelco Dolstra 7d876f8fa7 * Get rid of fetchurl, we don't need it anymore. 2005-04-07 14:35:44 +00:00
Eelco Dolstra c815aff21b * `nix-store --add-fixed' to preload the outputs of fixed-output
derivations.  This is mostly to simplify the implementation of
  nix-prefetch-{url, svn}, which now work properly in setuid
  installations.

* Enforce valid store names in `nix-store --add / --add-fixed'.
2005-04-07 14:01:51 +00:00
Eelco Dolstra 31e140d70b * I said it couldn't be done. I was wrong. 2005-04-04 15:18:19 +00:00
Eelco Dolstra ebe342c9c1 * Better error checking. 2005-03-25 14:30:01 +00:00
Eelco Dolstra 7272c3f817 * Ignore hash conflicts in gc-releases.pl. 2005-03-18 09:43:25 +00:00
Eelco Dolstra e52ae1c0ff * Use SHA-256 for nix-push. 2005-03-15 11:12:48 +00:00
Eelco Dolstra 5675d5f488 * Idem. 2005-03-14 18:55:46 +00:00
Eelco Dolstra 6fb5f7e532 * Pass `--base32' unless using MD5. 2005-03-14 18:55:29 +00:00
Eelco Dolstra bb2e53699f * Parse new hash format properly. 2005-03-14 17:05:42 +00:00
Eelco Dolstra 5863f24722 * Print SHA-1 hashes in base-32 by default. 2005-03-14 17:05:20 +00:00
Eelco Dolstra bd333b939c * Prefix hash algorithm in patch generator too. 2005-03-14 16:46:19 +00:00
Eelco Dolstra 8eff18cd43 * Set NAR name to content hash; previous nix-push names were not
unique.
* Drop `hashAlgo' attribute in manifests; prefix hashes with the hash
  algorithm instead.
2005-03-14 15:09:53 +00:00
Eelco Dolstra 1562dfe9ba * Script to garbage collect nix-push directories. It prints out all
file names in the directory not included in any of the manifests
  specified on the command line.
2005-03-14 14:03:41 +00:00
Eelco Dolstra 536f324177 * nix-install-package: install outPath, not drvPath, for now.
* nix-prefecth-url: print out in base-16.
2005-03-11 15:27:37 +00:00
Eelco Dolstra 9e6bca8765 * Channel fix. 2005-03-03 13:10:52 +00:00
Eelco Dolstra 9e50e648a4 * Doh! 2005-03-01 11:27:38 +00:00
Eelco Dolstra 8d364e5baa * Add missing file to dist. 2005-03-01 11:27:22 +00:00
Eelco Dolstra db322a47ff * Use a weighted use heuristic to disambiguate between multiple
occurances of a component.  If the shortest path distance between a
  component P and Q in the referers graph is D, then the contribution
  of Q to the use of P is 1 / R^D, where R >= 1, typically 2.  This
  expresses that distant indirect uses are less important than nearby
  uses.

  For instance, this can disambiguate between the bootstrap GCC in
  Nixpkgs and the GCC of the final stdenv (the former has more uses,
  but they are further away),  and between the GCC of the final stdenv
  and the GCC+G77 build (the latter has very few uses).
2005-03-01 10:33:55 +00:00
Eelco Dolstra 2c4302dd7a * Added a disambiguation heuristic: if two components have the same
name but differ to much in sice (by more than a factor of 3), then
  never generate a patch.
2005-02-28 14:12:06 +00:00
Eelco Dolstra 8376fff151 * Add a version number to manifests. 2005-02-25 16:12:52 +00:00
Eelco Dolstra 8d3c346559 * Pause if errors occur. 2005-02-25 15:58:00 +00:00
Eelco Dolstra 6bafeafb88 * nix-install-package: Use the new (trivial) package format generated
by the build farm.  See e.g.,
  http://catamaran.labs.cs.uu.nl/dist/nixpkgs-0.8/nixpkgs-0.7pre2302/;
  the user can click on packages, and they will be installed (assuming
  the `application/nix-package' MIME type has been associated with
  `nix-install-package').

  Nix expressions are no longer involved: a "package" is just a
  pointer to a manifest, and the top-level store derivation to be
  added to the user environment.  This makes these packages
  independent from Nix expression evolution.

  Note that we install the store derivation ($drvPath), not the
  resulting output path ($outPath).  This is equivalent, except that
  installing the derivation maintains the back-link from the output
  path to the derivation that built it.  This is useful for
  maintenance.

* Automatically re-exec in an xterm so that the user sees something
  when `nix-install-package' is run from a browser.
2005-02-25 15:42:52 +00:00
Eelco Dolstra 3259ae5811 * Properly specify the hash algorithm in the manifests, and read it
too.
* Change the default hash for nix-prefetch-url back to md5, since
  that's what we use in Nixpkgs (for now; a birthday attack is rather
  unlikely there).
2005-02-24 17:36:42 +00:00
Eelco Dolstra 95e870a113 * (Unnecessary) refactoring. 2005-02-24 14:06:18 +00:00
Eelco Dolstra 3a2c3f0cf2 * Support for fixed-output hashes over directory trees (i.e., over the
NAR dump of the path).
2005-02-22 21:14:41 +00:00
Eelco Dolstra fb5dae8694 * Fix nix-channel.
* Add `--help' flag; fixes NIX-5.
* Add `--remove' flag; fixes NIX-6.
* Add `--list' flag.
2005-02-17 10:06:12 +00:00
Eelco Dolstra 98df735b51 * Propagate the deriver of a path through the substitute mechanism.
* Removed some dead code (successor stuff) from nix-push.
* Updated terminology in the tests (store expr -> drv path).
* Check that the deriver is set properly in the tests.
2005-02-09 12:57:13 +00:00
Eelco Dolstra 60feff82cf * Set umask to prevent permission problems. 2005-02-08 13:00:39 +00:00
Eelco Dolstra 48ebe4527e * Better error reporting in readmanifest.
* Use force flag in `mv' to prevent silly interactive questions (this
  happens with shared Nix stores).
2005-02-08 11:40:19 +00:00
Eelco Dolstra 9f6835c282 * Remove debug code. 2005-02-01 17:52:11 +00:00
Eelco Dolstra c3981d81f6 * Make check fixes. 2005-02-01 17:50:48 +00:00
Eelco Dolstra 65b6c8ab4c * Move root finding from nix-collect-garbage' to nix-store --gc'.
This was necessary becase root finding must be done after
  acquisition of the global GC lock.

  This makes `nix-collect-garbage' obsolete; it is now just a wrapper
  around `nix-store --gc'.

* Automatically remove stale GC roots (i.e., indirect GC roots that
  point to non-existent paths).
2005-02-01 15:05:32 +00:00
Eelco Dolstra 630ae0c9d7 * nix-build: use an indirection scheme to make it easier for users to
get rid of GC roots.  Nix-build places a symlink `result' in the
  current directory.  Previously, removing that symlink would not
  remove the store path being linked to as a GC root.  Now, the GC
  root created by nix-build is actually a symlink in
  `/nix/var/nix/gcroots/auto' to `result'.  So if that symlink is
  removed the GC root automatically becomes invalid (since it can no
  longer be resolved).  The root itself is not automatically removed -
  the garbage collector should delete dangling roots.
2005-02-01 13:48:46 +00:00
Eelco Dolstra dcc37c236c * nix-store, nix-instantiate: added an option `--add-root' to
immediately add the result as a permanent GC root.  This is the only
  way to prevent a race with the garbage collector.  For instance, the
  old style

    ln -s $(nix-store -r $(nix-instantiate foo.nix)) \
      /nix/var/nix/gcroots/result

  has two time windows in which the garbage collector can interfere
  (by GC'ing the derivation and the output, respectively).  On the
  other hand,

    nix-store --add-root /nix/var/nix/gcroots/result -r \
      $(nix-instantiate --add-root /nix/var/nix/gcroots/drv \
        foo.nix)

  is safe.

* nix-build: use `--add-root' to prevent GC races.
2005-02-01 12:36:25 +00:00
Eelco Dolstra a6b65fd5e1 * Get rid of hardcoded paths. 2005-02-01 09:54:56 +00:00
Eelco Dolstra 1328aa3307 * Start of concurrent garbage collection. Processes write temporary
roots to a per-process temporary file in /nix/var/nix/temproots
  while holding a write lock on that file.  The garbage collector
  acquires read locks on all those files, thus blocking further
  progress in other Nix processes, and reads the sets of temporary
  roots.
2005-01-31 10:27:25 +00:00
Eelco Dolstra c505702265 * Fix and simplify the garbage collector (it's still not concurrent,
though).  In particular it's now much easier to register a GC root.
  Just place a symlink to whatever store path it is that you want to
  keep in /nix/var/nix/gcroots.
2005-01-27 15:21:29 +00:00
Eelco Dolstra 066da4ab85 * Really fix the substitute mechanism, i.e., ensure the closure
invariant by registering references through the manifest.
* Added a test for nix-pull.
2005-01-25 17:08:52 +00:00
Eelco Dolstra 6ff48e77f6 * Set the Perl search path properly (reported by Roy van den Broek). 2005-01-19 21:55:02 +00:00
Eelco Dolstra f3dc231250 * Removed the `id' attribute hack.
* Formalise the notion of fixed-output derivations, i.e., derivations
  for which a cryptographic hash of the output is known in advance.
  Changes to such derivations should not propagate upwards through the
  dependency graph.  Previously this was done by specifying the hash
  component of the output path through the `id' attribute, but this is
  insecure since you can lie about it (i.e., you can specify any hash
  and then produce a completely different output).  Now the
  responsibility for checking the output is moved from the builder to
  Nix itself.

  A fixed-output derivation can be created by specifying the
  `outputHash' and `outputHashAlgo' attributes, the latter taking
  values `md5', `sha1', and `sha256', and the former specifying the
  actual hash in hexadecimal or in base-32 (auto-detected by looking
  at the length of the attribute value).  MD5 is included for
  compatibility but should be considered deprecated.

* Removed the `drvPath' pseudo-attribute in derivation results.  It's
  no longer necessary.

* Cleaned up the support for multiple output paths in derivation store
  expressions.  Each output now has a unique identifier (e.g., `out',
  `devel', `docs').  Previously there was no way to tell output paths
  apart at the store expression level.

* `nix-hash' now has a flag `--base32' to specify that the hash should
  be printed in base-32 notation.

* `fetchurl' accepts parameters `sha256' and `sha1' in addition to
  `md5'.

* `nix-prefetch-url' now prints out a SHA-1 hash in base-32.  (TODO: a
  flag to specify the hash.)
2005-01-17 16:55:19 +00:00
Eelco Dolstra 9ee88bb2f2 * Use absolute paths. 2005-01-14 13:50:00 +00:00
Eelco Dolstra b17e7cf979 * Script to remove patches from manifests. 2005-01-12 10:40:59 +00:00
Eelco Dolstra 0bc41f632b * Print out less garbage. 2005-01-12 10:37:18 +00:00
Eelco Dolstra 6af4a5a71f * Prototype store optimiser. It searched the Nix store for identical
files and hard-links them to each other to save disk space.

  Currently it doesn't actually do the hard-linking, it just reports
  the amount of space saved if it did.
2005-01-05 09:58:12 +00:00
Eelco Dolstra c53898cb65 * If a patch already exists, it must still be included in the manifest. 2004-12-31 11:07:12 +00:00
Eelco Dolstra 35b76a81c4 * More instrumentation (statistics go to /nix/var/log/nix/downloads). 2004-12-30 17:19:47 +00:00
Eelco Dolstra 3745cecc6a * Fix handling of chained patches: don't skip patches if intermediate
paths are missing, etc.
2004-12-30 17:09:57 +00:00
Eelco Dolstra 581bcb986f * Some logging for evaluation. 2004-12-30 16:34:54 +00:00