Commit graph

217 commits

Author SHA1 Message Date
Eelco Dolstra 607a2f01e6 * Remove other uses of IPC::Open2. 2005-09-21 17:14:52 +00:00
Eelco Dolstra 95304172a5 * Don't use IPC::Open2, it has a subtle race bug on Mac OS X 10.4. If
the parent runs before the child, it closes some pipe file
  descriptors which causes the child to fail due to a bad file
  descriptor.  So we just use the normal open() function instead.
  
  This fixes NIX-14 (intermittent nix-pull failures).
2005-09-21 17:06:06 +00:00
Eelco Dolstra d34fa9a6cc * Remove dead file. 2005-09-16 09:05:18 +00:00
Eelco Dolstra 846b53bde4 * Set the current directory to something well-defined. Might help in
setuid installations.
2005-09-16 09:03:24 +00:00
Eelco Dolstra 7f384d9c1b * Use a proper temporary directory. 2005-09-15 15:21:35 +00:00
Eelco Dolstra a5ceb5bc0b * nix-build: default to `./default.nix' if no paths are specified.
So when using Nix as a build tool, you can just say `nix-build' and
  it will build the top-level derivation defined in `default.nix'.
2005-07-13 17:39:10 +00:00
Eelco Dolstra 928a7c06dc * Don't create patches for archives >= 150 MB because bsdiff can't
handle it.  It crashed on the 234 MB tetex archive.  Probably we
  will never be able to handle archives of that size on 32-bit
  machines (because bsdiff does everything in memory requiring
  max(17*n,9*n+m)+O(1) bytes, so the address space simply isn't
  there).
2005-06-18 14:20:24 +00:00
Eelco Dolstra cbc8d083ac * Make unpacking of patch sequences much faster by not doing redundant
unpacking and repacking of intermediate paths.
2005-05-10 14:22:36 +00:00
Eelco Dolstra 9f3601a36c * Argh! The patch downloader was broken due to the renaming of the
`--isvalid' flag in nix-store.
2005-04-12 10:51:38 +00:00
Eelco Dolstra d5219a351a * Damn. Disable the USE heuristic for now, since the deriver in the
database isn't always in the manifest (so the reference graph cannot
  be reconstructed fully).
2005-04-12 10:07:02 +00:00
Eelco Dolstra c702dfca3f * nix-store: --substitute' -> --register-substitutes'. 2005-04-08 13:48:41 +00:00
Eelco Dolstra 7d876f8fa7 * Get rid of fetchurl, we don't need it anymore. 2005-04-07 14:35:44 +00:00
Eelco Dolstra c815aff21b * `nix-store --add-fixed' to preload the outputs of fixed-output
derivations.  This is mostly to simplify the implementation of
  nix-prefetch-{url, svn}, which now work properly in setuid
  installations.

* Enforce valid store names in `nix-store --add / --add-fixed'.
2005-04-07 14:01:51 +00:00
Eelco Dolstra 31e140d70b * I said it couldn't be done. I was wrong. 2005-04-04 15:18:19 +00:00
Eelco Dolstra ebe342c9c1 * Better error checking. 2005-03-25 14:30:01 +00:00
Eelco Dolstra 7272c3f817 * Ignore hash conflicts in gc-releases.pl. 2005-03-18 09:43:25 +00:00
Eelco Dolstra e52ae1c0ff * Use SHA-256 for nix-push. 2005-03-15 11:12:48 +00:00
Eelco Dolstra 5675d5f488 * Idem. 2005-03-14 18:55:46 +00:00
Eelco Dolstra 6fb5f7e532 * Pass `--base32' unless using MD5. 2005-03-14 18:55:29 +00:00
Eelco Dolstra bb2e53699f * Parse new hash format properly. 2005-03-14 17:05:42 +00:00
Eelco Dolstra 5863f24722 * Print SHA-1 hashes in base-32 by default. 2005-03-14 17:05:20 +00:00
Eelco Dolstra bd333b939c * Prefix hash algorithm in patch generator too. 2005-03-14 16:46:19 +00:00
Eelco Dolstra 8eff18cd43 * Set NAR name to content hash; previous nix-push names were not
unique.
* Drop `hashAlgo' attribute in manifests; prefix hashes with the hash
  algorithm instead.
2005-03-14 15:09:53 +00:00
Eelco Dolstra 1562dfe9ba * Script to garbage collect nix-push directories. It prints out all
file names in the directory not included in any of the manifests
  specified on the command line.
2005-03-14 14:03:41 +00:00
Eelco Dolstra 536f324177 * nix-install-package: install outPath, not drvPath, for now.
* nix-prefecth-url: print out in base-16.
2005-03-11 15:27:37 +00:00
Eelco Dolstra 9e6bca8765 * Channel fix. 2005-03-03 13:10:52 +00:00
Eelco Dolstra 9e50e648a4 * Doh! 2005-03-01 11:27:38 +00:00
Eelco Dolstra 8d364e5baa * Add missing file to dist. 2005-03-01 11:27:22 +00:00
Eelco Dolstra db322a47ff * Use a weighted use heuristic to disambiguate between multiple
occurances of a component.  If the shortest path distance between a
  component P and Q in the referers graph is D, then the contribution
  of Q to the use of P is 1 / R^D, where R >= 1, typically 2.  This
  expresses that distant indirect uses are less important than nearby
  uses.

  For instance, this can disambiguate between the bootstrap GCC in
  Nixpkgs and the GCC of the final stdenv (the former has more uses,
  but they are further away),  and between the GCC of the final stdenv
  and the GCC+G77 build (the latter has very few uses).
2005-03-01 10:33:55 +00:00
Eelco Dolstra 2c4302dd7a * Added a disambiguation heuristic: if two components have the same
name but differ to much in sice (by more than a factor of 3), then
  never generate a patch.
2005-02-28 14:12:06 +00:00
Eelco Dolstra 8376fff151 * Add a version number to manifests. 2005-02-25 16:12:52 +00:00
Eelco Dolstra 8d3c346559 * Pause if errors occur. 2005-02-25 15:58:00 +00:00
Eelco Dolstra 6bafeafb88 * nix-install-package: Use the new (trivial) package format generated
by the build farm.  See e.g.,
  http://catamaran.labs.cs.uu.nl/dist/nixpkgs-0.8/nixpkgs-0.7pre2302/;
  the user can click on packages, and they will be installed (assuming
  the `application/nix-package' MIME type has been associated with
  `nix-install-package').

  Nix expressions are no longer involved: a "package" is just a
  pointer to a manifest, and the top-level store derivation to be
  added to the user environment.  This makes these packages
  independent from Nix expression evolution.

  Note that we install the store derivation ($drvPath), not the
  resulting output path ($outPath).  This is equivalent, except that
  installing the derivation maintains the back-link from the output
  path to the derivation that built it.  This is useful for
  maintenance.

* Automatically re-exec in an xterm so that the user sees something
  when `nix-install-package' is run from a browser.
2005-02-25 15:42:52 +00:00
Eelco Dolstra 3259ae5811 * Properly specify the hash algorithm in the manifests, and read it
too.
* Change the default hash for nix-prefetch-url back to md5, since
  that's what we use in Nixpkgs (for now; a birthday attack is rather
  unlikely there).
2005-02-24 17:36:42 +00:00
Eelco Dolstra 95e870a113 * (Unnecessary) refactoring. 2005-02-24 14:06:18 +00:00
Eelco Dolstra 3a2c3f0cf2 * Support for fixed-output hashes over directory trees (i.e., over the
NAR dump of the path).
2005-02-22 21:14:41 +00:00
Eelco Dolstra fb5dae8694 * Fix nix-channel.
* Add `--help' flag; fixes NIX-5.
* Add `--remove' flag; fixes NIX-6.
* Add `--list' flag.
2005-02-17 10:06:12 +00:00
Eelco Dolstra 98df735b51 * Propagate the deriver of a path through the substitute mechanism.
* Removed some dead code (successor stuff) from nix-push.
* Updated terminology in the tests (store expr -> drv path).
* Check that the deriver is set properly in the tests.
2005-02-09 12:57:13 +00:00
Eelco Dolstra 60feff82cf * Set umask to prevent permission problems. 2005-02-08 13:00:39 +00:00
Eelco Dolstra 48ebe4527e * Better error reporting in readmanifest.
* Use force flag in `mv' to prevent silly interactive questions (this
  happens with shared Nix stores).
2005-02-08 11:40:19 +00:00
Eelco Dolstra 9f6835c282 * Remove debug code. 2005-02-01 17:52:11 +00:00
Eelco Dolstra c3981d81f6 * Make check fixes. 2005-02-01 17:50:48 +00:00
Eelco Dolstra 65b6c8ab4c * Move root finding from nix-collect-garbage' to nix-store --gc'.
This was necessary becase root finding must be done after
  acquisition of the global GC lock.

  This makes `nix-collect-garbage' obsolete; it is now just a wrapper
  around `nix-store --gc'.

* Automatically remove stale GC roots (i.e., indirect GC roots that
  point to non-existent paths).
2005-02-01 15:05:32 +00:00
Eelco Dolstra 630ae0c9d7 * nix-build: use an indirection scheme to make it easier for users to
get rid of GC roots.  Nix-build places a symlink `result' in the
  current directory.  Previously, removing that symlink would not
  remove the store path being linked to as a GC root.  Now, the GC
  root created by nix-build is actually a symlink in
  `/nix/var/nix/gcroots/auto' to `result'.  So if that symlink is
  removed the GC root automatically becomes invalid (since it can no
  longer be resolved).  The root itself is not automatically removed -
  the garbage collector should delete dangling roots.
2005-02-01 13:48:46 +00:00
Eelco Dolstra dcc37c236c * nix-store, nix-instantiate: added an option `--add-root' to
immediately add the result as a permanent GC root.  This is the only
  way to prevent a race with the garbage collector.  For instance, the
  old style

    ln -s $(nix-store -r $(nix-instantiate foo.nix)) \
      /nix/var/nix/gcroots/result

  has two time windows in which the garbage collector can interfere
  (by GC'ing the derivation and the output, respectively).  On the
  other hand,

    nix-store --add-root /nix/var/nix/gcroots/result -r \
      $(nix-instantiate --add-root /nix/var/nix/gcroots/drv \
        foo.nix)

  is safe.

* nix-build: use `--add-root' to prevent GC races.
2005-02-01 12:36:25 +00:00
Eelco Dolstra a6b65fd5e1 * Get rid of hardcoded paths. 2005-02-01 09:54:56 +00:00
Eelco Dolstra 1328aa3307 * Start of concurrent garbage collection. Processes write temporary
roots to a per-process temporary file in /nix/var/nix/temproots
  while holding a write lock on that file.  The garbage collector
  acquires read locks on all those files, thus blocking further
  progress in other Nix processes, and reads the sets of temporary
  roots.
2005-01-31 10:27:25 +00:00
Eelco Dolstra c505702265 * Fix and simplify the garbage collector (it's still not concurrent,
though).  In particular it's now much easier to register a GC root.
  Just place a symlink to whatever store path it is that you want to
  keep in /nix/var/nix/gcroots.
2005-01-27 15:21:29 +00:00
Eelco Dolstra 066da4ab85 * Really fix the substitute mechanism, i.e., ensure the closure
invariant by registering references through the manifest.
* Added a test for nix-pull.
2005-01-25 17:08:52 +00:00
Eelco Dolstra 6ff48e77f6 * Set the Perl search path properly (reported by Roy van den Broek). 2005-01-19 21:55:02 +00:00
Eelco Dolstra f3dc231250 * Removed the `id' attribute hack.
* Formalise the notion of fixed-output derivations, i.e., derivations
  for which a cryptographic hash of the output is known in advance.
  Changes to such derivations should not propagate upwards through the
  dependency graph.  Previously this was done by specifying the hash
  component of the output path through the `id' attribute, but this is
  insecure since you can lie about it (i.e., you can specify any hash
  and then produce a completely different output).  Now the
  responsibility for checking the output is moved from the builder to
  Nix itself.

  A fixed-output derivation can be created by specifying the
  `outputHash' and `outputHashAlgo' attributes, the latter taking
  values `md5', `sha1', and `sha256', and the former specifying the
  actual hash in hexadecimal or in base-32 (auto-detected by looking
  at the length of the attribute value).  MD5 is included for
  compatibility but should be considered deprecated.

* Removed the `drvPath' pseudo-attribute in derivation results.  It's
  no longer necessary.

* Cleaned up the support for multiple output paths in derivation store
  expressions.  Each output now has a unique identifier (e.g., `out',
  `devel', `docs').  Previously there was no way to tell output paths
  apart at the store expression level.

* `nix-hash' now has a flag `--base32' to specify that the hash should
  be printed in base-32 notation.

* `fetchurl' accepts parameters `sha256' and `sha1' in addition to
  `md5'.

* `nix-prefetch-url' now prints out a SHA-1 hash in base-32.  (TODO: a
  flag to specify the hash.)
2005-01-17 16:55:19 +00:00
Eelco Dolstra 9ee88bb2f2 * Use absolute paths. 2005-01-14 13:50:00 +00:00
Eelco Dolstra b17e7cf979 * Script to remove patches from manifests. 2005-01-12 10:40:59 +00:00
Eelco Dolstra 0bc41f632b * Print out less garbage. 2005-01-12 10:37:18 +00:00
Eelco Dolstra 6af4a5a71f * Prototype store optimiser. It searched the Nix store for identical
files and hard-links them to each other to save disk space.

  Currently it doesn't actually do the hard-linking, it just reports
  the amount of space saved if it did.
2005-01-05 09:58:12 +00:00
Eelco Dolstra c53898cb65 * If a patch already exists, it must still be included in the manifest. 2004-12-31 11:07:12 +00:00
Eelco Dolstra 35b76a81c4 * More instrumentation (statistics go to /nix/var/log/nix/downloads). 2004-12-30 17:19:47 +00:00
Eelco Dolstra 3745cecc6a * Fix handling of chained patches: don't skip patches if intermediate
paths are missing, etc.
2004-12-30 17:09:57 +00:00
Eelco Dolstra 581bcb986f * Some logging for evaluation. 2004-12-30 16:34:54 +00:00
Eelco Dolstra 6270aa727d * Propagate patches from the source distribution to the destination
distribution insofar they are applicable.
2004-12-29 22:17:26 +00:00
Eelco Dolstra 54d8f08588 * Reject patches larger than the full archives they produce. 2004-12-29 19:32:55 +00:00
Eelco Dolstra 2fdb27e7f2 * Atomic file replacement is good. 2004-12-29 19:04:21 +00:00
Eelco Dolstra e1e9c036f9 * A utility to generate patches between releases based on their
manifests.
2004-12-29 18:58:15 +00:00
Eelco Dolstra 9022cf9adf * A small utility to add the Size and NarHash fields to old manifests. 2004-12-28 21:12:00 +00:00
Eelco Dolstra 4bf58d5379 * Added a function to write manifests. 2004-12-28 21:11:28 +00:00
Eelco Dolstra 3d1b2101cc * Place manifests in /nix/var/nix/manifests.
* Use the new patch downloader.
2004-12-20 16:38:50 +00:00
Eelco Dolstra 7eed57e784 * Sync with changed substitute mechanism.
* Accept the NarHash line.
* Clear substitutes in `nix-channel --update'.
2004-12-20 14:57:03 +00:00
Eelco Dolstra 4d25b0b0bb * Fix nix-pull. 2004-12-16 15:31:50 +00:00
Eelco Dolstra 77970f8daf * Remove `prebuilts.conf' file, it's not like anybody was using it.
* Add /nix/var/nix/manifests directory.
2004-12-16 14:31:49 +00:00
Eelco Dolstra e3b051aeeb * Include the size of the bzipped archive (necessary for computing the
cheapest download path), as well as the hash of the contents of the
  path (necessary for checking patch applicability).
2004-12-13 16:56:18 +00:00
Eelco Dolstra 862f4c154e * Patch deployment. `download.pl' (intended to be used in the
substitute mechanism) creates a store path by downloading full NAR
  archives and/or patches specified in the available manifests.

  Any combination of present paths, full downloads, and patches can be
  used to construct the target path.  In particular, patches can be
  chained in sequence; and full NAR archives of the target path can be
  omitted (i.e., patch-only deployment is possible).  A shortest path
  algorithm is used to find the smallest set of files to be downloaded
  (the edge weights are currently file sizes, but one can imagine
  taking the network speed to the various source into account).

  Patches are binary deltas between two store paths.  To be precise,
  they are the output of the `bsdiff' program applied to the NAR
  archives obtained by dumping (`nix-store --dump') the two store
  paths.  The advantage of diff'ing NAR archives (and not, say, doing
  file-by-file diffs) is that file renames/moves are handled
  automatically.  The disadvantage is that we cannot optimise creation
  of unchanged files (by hard-linking).
2004-12-13 13:47:38 +00:00
Eelco Dolstra dca48aed34 * Allow an optional hash to be provided. This prevents redundant
fetches.
2004-12-13 13:35:36 +00:00
Eelco Dolstra feb3ceaee0 * Better error messages. 2004-11-04 20:20:39 +00:00
Eelco Dolstra 2aa1f4717b * Fix File exists' errors if the result' symlink exists but is
dangling.
2004-10-31 12:01:16 +00:00
Eelco Dolstra 2155c0a673 * Register channels as roots of the garbage collector (in
$(localstatedir)/nix/gcroots/channels). 
* In setuid installations, create gcroots/tmp and gcroots/channels
  group-writable.
2004-10-20 14:42:38 +00:00
Eelco Dolstra 88888160d2 * Fix nix-prefetch-url in setuid Nix installations. 2004-10-20 14:40:54 +00:00
Eelco Dolstra 99da51d4de * Show error messages from curl. 2004-10-20 14:05:48 +00:00
Eelco Dolstra eb233e728f * `--min-age' flag in nix-store and nix-collect-garbage to only delete
unreachable paths that haven't been used for N hours.  For instance,
  `nix-collect-garbage --min-age 168' only deletes paths that haven't
  been accessed in the last week.

  This is useful for instance in the build farm where many derivations
  can be shared between consecutive builds, and we wouldn't want a
  garbage collect to throw them all away.  We could of course register
  them as roots, but then we'd to unregister them at some point, which
  would be a pain to manage.  The `--min-age' flag gives us a sort of
  MRU caching scheme.

  BUG: this really shouldn't be in gc.cc since that violates
  mechanism/policy separation.
2004-08-25 16:54:08 +00:00
Eelco Dolstra fdec72c6cc * `nix-collect-garbage' now actually performs a garbage collection, it
doesn't just print the set of paths that should be deleted.  So
  there is no more need to pipe the result into `nix-store --delete'
  (which doesn't even exist anymore).
2004-08-25 15:39:13 +00:00
Eelco Dolstra 2d35116c13 * Setuid support for sharing a Nix installation between multiple
users.

  If the configure flag `--enable-setuid' is used, the Nix programs
  nix-env, nix-store, etc. are installed with the setuid bit turned on
  so that they are executed as the user and group specified by
  `--with-nix-user=USER' and `--with-nix-group=GROUP', respectively
  (with defaults `nix' and `nix').

  The setuid programs drop all special privileges if they are executed
  by a user who is not a member of the Nix group.

  The setuid feature is a quick hack to enable sharing of a Nix
  installation between users who trust each other.  It is not
  generally secure, since any user in the Nix group can modify (by
  building an appropriate derivation) any object in the store, and for
  instance inject trojans into binaries used by other users.

  The setuid programs are owned by root, not the Nix user.  This is
  because on Unix normal users cannot change the real uid, only the
  effective uid.  Many programs don't work properly when the real uid
  differs from the effective uid.  For instance, Perl will turn on
  taint mode.  However, the setuid programs drop all root privileges
  immediately, changing all uids and gids to the Nix user and group.
2004-08-20 14:49:05 +00:00
Eelco Dolstra 937ce0cd21 * Flag `--no-link' suppresses symlinking to the output path.
* Handle multiple derivations correctly.
2004-08-18 12:11:31 +00:00
Eelco Dolstra e8a95108c0 * Nix-build places a symlink `result' in the current directory to the
store object just built.
2004-07-28 13:32:45 +00:00
Eelco Dolstra d051cd40e1 * Nix-instantiate can return multiple store expressions. 2004-06-22 15:17:04 +00:00
Eelco Dolstra 3093af58a7 * A utility script `nix-build' that builds Nix expressions and prints
their output paths (and only that) on standard output.
2004-06-22 15:12:34 +00:00
Eelco Dolstra 37ee6cef99 * Adapted nix-pull to use the new substitute mechanism. 2004-06-21 09:51:23 +00:00
Eelco Dolstra 759c953196 * Look for GC roots in @localstatedir@/nix/gcroots. 2004-04-22 14:17:57 +00:00
Eelco Dolstra 21655a70f5 * Channels. These allow you to stay current with an evolving set of
Nix expressions.

  To subscribe to a channel (needs to be done only once):

    nix-channel --add \
      http://catamaran.labs.cs.uu.nl/dist/nix/channels/nixpkgs-unstable

  This just adds the given URL to ~/.nix-channels (which can also be
  edited manually).

  To update from all channels:

    nix-channel --update

  This fetches the latest expressions and pulls cache manifests.  The
  default Nix expression (~/.nix-defexpr) is made to point to the
  conjunction of the expressions downloaded from all channels.

  So to update all installed derivations in the current user
  environment:

    nix-channel --update
    nix-env --upgrade '*'

  If you are really courageous, you can put this in a cronjob or
  something.

  You can subscribe to multiple channels.  It is not entirely clear
  what happens when there are name clashes between derivations from
  different channels.  From nix-env/main.cc it appears that the one
  with the lowest (highest?) hash will be used, which is pretty
  meaningless.
2004-04-21 14:54:05 +00:00
Eelco Dolstra 7cce0c34e1 * Allow extra parameters to be passed to Curl through the `CURL_FLAGS'
environment variable.  This is useful for passing authentication
  information (it won't show up in `ps').  Hacky - nix-push should
  abstract over the use of Curl.
2004-04-15 15:14:16 +00:00
Eelco Dolstra b275f2ed3b * `*.gcroot' files can now containing multiple roots. 2004-04-14 08:09:27 +00:00
Eelco Dolstra 153429520a * Distributed the wrong file. 2004-04-06 15:55:27 +00:00
Eelco Dolstra 03f1d1ecb5 * Switched from wget to curl.
* Made the dependencies on bzip2 and the shell explicit.
2004-04-06 08:18:51 +00:00
Eelco Dolstra beda10f5a2 * Make perl a dependency of Nix. 2004-03-15 15:23:53 +00:00
Eelco Dolstra 7f0ed370da * Use $(storedir) instead of $(prefix)/store. 2004-02-20 11:32:30 +00:00
Eelco Dolstra 1ad9d11247 * Only include predecessors that are themselves being pushed.
Otherwise the substitute mechanism can break in subtle ways.
2004-02-13 10:43:31 +00:00
Eelco Dolstra 92e832348d * Lots of manual stuff. Reference pages for most Nix commands.
* nix-pull now requires the full url to the manifest, i.e.,
  `/MANIFEST/' is no longer automatically appended.
* nix-prefetch-url works again.
2004-02-10 16:14:47 +00:00
Eelco Dolstra 66e94d3275 * Improvements to profiles. Generations are now per-profile, e.g.,
default -> default-94-link
  default-82-link -> /nix/store/cc4480...
  default-83-link -> /nix/store/caeec8...
  ...
  default-94-link -> /nix/store/2896ca...
  experimental -> experimental-2-link
  experimental-1-link -> /nix/store/cc4480...
  experimental-2-link -> /nix/store/a3148f...

* `--profile' / `-p' -> `--switch-profile' / `-S'
* `--link' / `-l' -> `--profile' / `-p'
* The default profile is stored in $prefix/var/nix/profiles.
  $prefix/var/nix/links is gone.  Profiles can be stored anywhere.
* The current profile is now referenced from ~/.nix-profile, not
  ~/.nix-userenv.
* The roots to the garbage collector now have extension `.gcroot', not
  `.id'.
2004-02-06 10:30:20 +00:00
Eelco Dolstra f899e8ce4d * Test whether the symlink, not its target, exists. 2004-01-16 15:17:36 +00:00
Eelco Dolstra b1c5f3c10d * Doh! Edited readmanifest.pm' instead of readmanifest.pm.in'. 2004-01-16 14:54:39 +00:00
Eelco Dolstra 9a404e45c9 * Synchronous nix-pull' with nix-push'.
* Use curl instead of wget.
2004-01-14 14:20:33 +00:00
Eelco Dolstra 16f9b133ec * Improved `nix-push': it now uses HTTP PUT (instead of rsync) to copy
files.  Target location is no longer hard-coded; it accepts a number
  of URLs on the command line.

* `nix-install-package': compatibility fixes.
2004-01-14 11:13:08 +00:00