Commit graph

230 commits

Author SHA1 Message Date
Eelco Dolstra e39999ed48 Sign locally-built paths
Locally-built paths are now signed automatically using the secret keys
specified by the ‘secret-key-files’ option.
2016-04-07 15:16:57 +02:00
Eelco Dolstra d0f5719c2a Add "nix copy-sigs" command
This imports signatures from one store into another. E.g.

  $ nix copy-sigs -r /run/current-system -s https://cache.nixos.org/
  imported 595 signatures
2016-04-05 16:39:23 +02:00
Eelco Dolstra 9cee600c88 LocalStore: Keep track of ultimately trusted paths
These are content-addressed paths or outputs of locally performed
builds. They are trusted even if they don't have signatures, so "nix
verify-paths" won't complain about them.
2016-03-30 17:35:48 +02:00
Eelco Dolstra 3d119f0a3b Improve the SQLite wrapper API
In particular, this eliminates a bunch of boilerplate code.
2016-03-30 15:50:45 +02:00
Eelco Dolstra d9c5e3bbf0 Factour out SQLite handling 2016-03-30 13:27:25 +02:00
Eelco Dolstra bcc9943cee importPaths(): Optionally add NARs to binary cache accessor
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
2016-02-26 15:20:10 +01:00
Eelco Dolstra 1042c10fd0 Add NAR / Store accessor abstraction
This is primary to allow hydra-queue-runner to extract files like
"nix-support/hydra-build-products" from NARs in binary caches.
2016-02-25 17:43:19 +01:00
Eelco Dolstra 28e7e29abd Eliminate reserveSpace flag 2016-02-24 17:44:55 +01:00
Eelco Dolstra c8f4d89a34 Expose the export magic value and move LocalStore::queryReferences to Store 2016-02-15 15:01:26 +01:00
Eelco Dolstra fa7cd5369b StoreAPI -> Store
Calling a class an API is a bit redundant...
2016-02-04 14:48:42 +01:00
Eelco Dolstra c10c61449f Eliminate the "store" global variable
Also, move a few free-standing functions into StoreAPI and Derivation.

Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
2016-02-04 14:28:26 +01:00
Eelco Dolstra 9e7c1a4bbd Use the daemon when we don't have write access to the Nix database 2016-01-31 10:28:48 +01:00
Eelco Dolstra 4ba6bc184c Shut up clang warnings 2015-09-18 01:22:35 +02:00
Eelco Dolstra 1511aa9f48 Allow remote builds without sending the derivation closure
Previously, to build a derivation remotely, we had to copy the entire
closure of the .drv file to the remote machine, even though we only
need the top-level derivation. This is very wasteful: the closure can
contain thousands of store paths, and in some Hydra use cases, include
source paths that are very large (e.g. Git/Mercurial checkouts).

So now there is a new operation, StoreAPI::buildDerivation(), that
performs a build from an in-memory representation of a derivation
(BasicDerivation) rather than from a on-disk .drv file. The only files
that need to be in the Nix store are the sources of the derivation
(drv.inputSrcs), and the needed output paths of the dependencies (as
described by drv.inputDrvs). "nix-store --serve" exposes this
interface.

Note that this is a privileged operation, because you can construct a
derivation that builds any store path whatsoever. Fixing this will
require changing the hashing scheme (i.e., the output paths should be
computed from the other fields in BasicDerivation, allowing them to be
verified without access to other derivations). However, this would be
quite nice because it would allow .drv-free building (e.g. "nix-env
-i" wouldn't have to write any .drv files to disk).

Fixes #173.
2015-07-17 17:57:40 +02:00
Eelco Dolstra 7ea6ecf855 addToStore(): Take explicit name argument 2015-03-25 17:06:12 +01:00
Eelco Dolstra 1256ab3b44 Clean up temp roots in a more C++ way 2014-11-19 17:07:29 +01:00
Ludovic Courtès 8fb8c26b6d Add an 'optimiseStore' remote procedure call. 2014-09-01 23:53:26 +02:00
Eelco Dolstra 54a34119f3 Use std::unordered_set 2014-05-26 17:53:17 +02:00
Eelco Dolstra 84813af5b9 nix-store --optimise: Remove bogus statistics 2014-05-15 11:33:46 +02:00
Wout Mertens 3b9ea8452f Shortcut store files before lstat
readdir() already returns the inode numbers, so we don't need to call
lstat to know if a file was already linked or not.
2014-05-15 09:02:22 +02:00
Wout Mertens d73ffc552f Use the inodes given by readdir directly 2014-05-14 22:52:10 +02:00
Wout Mertens e974f20c98 Preload linked hashes to speed up lookups
By preloading all inodes in the /nix/store/.links directory, we can
quickly determine of a hardlinked file was already linked to the hashed
links.
This is tolerant of removing the .links directory, it will simply
recalculate all hashes in the store.
2014-05-13 23:10:06 +02:00
Eelco Dolstra 1aa19b24b2 Add a flag ‘--check’ to verify build determinism
The flag ‘--check’ to ‘nix-store -r’ or ‘nix-build’ will cause Nix to
redo the build of a derivation whose output paths are already valid.
If the new output differs from the original output, an error is
printed.  This makes it easier to test if a build is deterministic.
(Obviously this cannot catch all sources of non-determinism, but it
catches the most common one, namely the current time.)

For example:

  $ nix-build '<nixpkgs>' -A patchelf
  ...
  $ nix-build '<nixpkgs>' -A patchelf --check
  error: derivation `/nix/store/1ipvxsdnbhl1rw6siz6x92s7sc8nwkkb-patchelf-0.6' may not be deterministic: hash mismatch in output `/nix/store/4pc1dmw5xkwmc6q3gdc9i5nbjl4dkjpp-patchelf-0.6.drv'

The --check build fails if not all outputs are valid.  Thus the first
call to nix-build is necessary to ensure that all outputs are valid.

The current outputs are left untouched: the new outputs are either put
in a chroot or diverted to a different location in the store using
hash rewriting.
2014-02-18 01:01:14 +01:00
Eelco Dolstra a478e8a7bb Remove nix-setuid-helper
AFAIK, nobody uses it, it's not maintained, and it has no tests.
2013-11-14 11:57:37 +01:00
Eelco Dolstra a737f51fd9 Retry all SQLite operations
To deal with SQLITE_PROTOCOL, we also need to retry read-only
operations.
2013-10-16 15:58:20 +02:00
Ivan Kozik 34bb806f74 Fix typos, especially those that end up in the Nix manual 2013-08-26 11:15:22 +02:00
Eelco Dolstra 22144afa8d Don't keep "disabled" substituters running
For instance, it's pointless to keep copy-from-other-stores running if
there are no other stores, or download-using-manifests if there are no
manifests.  This also speeds things up because we don't send queries
to those substituters.
2013-06-20 11:55:15 +02:00
Eelco Dolstra 1b6ee8f4c7 Allow hard links between the outputs of a derivation 2013-06-13 17:29:56 +02:00
Eelco Dolstra 5959c591a0 Process stderr from substituters while doing have/info queries 2013-06-07 15:02:14 +02:00
Eelco Dolstra c5f9d0d080 Buffer reads from the substituter
This greatly reduces the number of system calls.
2013-06-07 14:00:23 +02:00
Eelco Dolstra bdd4646338 Revert "Prevent config.h from being clobbered"
This reverts commit 28bba8c44f.
2013-03-08 01:24:59 +01:00
Eelco Dolstra 28bba8c44f Prevent config.h from being clobbered 2013-03-07 23:55:55 +01:00
Eelco Dolstra b008674e46 Refactoring: Split off the non-recursive canonicalisePathMetaData()
Also, change the file mode before changing the owner.  This prevents a
slight time window in which a setuid binary would be setuid root.
2013-02-27 16:42:19 +01:00
Eelco Dolstra 5526a282b5 Security: Don't allow builders to change permissions on files they don't own
It turns out that in multi-user Nix, a builder may be able to do

  ln /etc/shadow $out/foo

Afterwards, canonicalisePathMetaData() will be applied to $out/foo,
causing /etc/shadow's mode to be set to 444 (readable by everybody but
writable by nobody).  That's obviously Very Bad.

Fortunately, this fails in NixOS's default configuration because
/nix/store is a bind mount, so "ln" will fail with "Invalid
cross-device link".  It also fails if hard-link restrictions are
enabled, so a workaround is:

  echo 1 > /proc/sys/fs/protected_hardlinks

The solution is to check that all files in $out are owned by the build
user.  This means that innocuous operations like "ln
${pkgs.foo}/some-file $out/" are now rejected, but that already failed
in chroot builds anyway.
2013-02-26 02:30:19 +01:00
Eelco Dolstra def5160b61 Clear any immutable bits in the Nix store
Doing this once makes subsequent operations like garbage collecting
more efficient since we don't have to call makeMutable() first.
2013-01-03 12:59:23 +01:00
Eelco Dolstra 2754a07ead nix-store -q --roots: Respect the gc-keep-outputs/gc-keep-derivations settings
So if a path is not garbage solely because it's reachable from a root
due to the gc-keep-outputs or gc-keep-derivations settings, ‘nix-store
-q --roots’ now shows that root.
2012-12-20 18:41:44 +01:00
Eelco Dolstra 06f62defe6 Yet another rewrite of the garbage collector
But this time it's *obviously* correct!  No more segfaults due to
infinite recursions for sure, etc.

Also, move directories to /nix/store/trash instead of renaming them to
/nix/store/bla-gc-<pid>.  Then we can just delete /nix/store/trash at
the end.
2012-12-20 17:32:15 +01:00
Eelco Dolstra ea89df2b76 Use vfork() instead of fork() if available
Hopefully this reduces the chance of hitting ‘unable to fork: Cannot
allocate memory’ errors.  vfork() is used for everything except
starting builders.
2012-11-09 18:00:33 +01:00
Eelco Dolstra 0a7084567f Add a ‘--repair’ flag to nix-instantiate
This allows repairing corrupted derivations and other source files.
2012-10-03 15:09:18 -04:00
Eelco Dolstra a3f205b249 When repairing a derivation, check and repair the entire output closure
If we find a corrupted path in the output closure, we rebuild the
derivation that produced that particular path.
2012-10-03 10:38:09 -04:00
Eelco Dolstra 2001895f3d Add a --repair flag to ‘nix-store -r’ to repair derivation outputs
With this flag, if any valid derivation output is missing or corrupt,
it will be recreated by using a substitute if available, or by
rebuilding the derivation.  The latter may use hash rewriting if
chroots are not available.
2012-10-02 17:13:46 -04:00
Eelco Dolstra 8e3a7bd712 nix-store --verify: Add an option ‘--repair’ to repair all missing/corrupt paths
Also, return a non-zero exit code if errors remain after
verifying/repairing.
2012-10-02 15:12:56 -04:00
Eelco Dolstra 9958bd6992 Add operation ‘nix-store --repair-path’
This operation allows fixing corrupted or accidentally deleted store
paths by redownloading them using substituters, if available.

Since the corrupted path cannot be replaced atomically, there is a
very small time window (one system call) during which neither the old
(corrupted) nor the new (repaired) contents are available.  So
repairing should be used with some care on critical packages like
Glibc.
2012-10-02 14:08:59 -04:00
Eelco Dolstra b9124a5c33 Support having /nix/store as a read-only bind mount
It turns out that the immutable bit doesn't work all that well.  A
better way is to make the entire Nix store a read-only bind mount,
i.e. by doing

  $ mount --bind /nix/store /nix/store
  $ mount -o remount,ro,bind /nix/store

(This would typically done in an early boot script, before anything
from /nix/store is used.)

Since Nix needs to be able to write to the Nix store, it now detects
if /nix/store is a read-only bind mount and then makes it writable in
a private mount namespace.
2012-09-19 15:45:29 -04:00
Eelco Dolstra e6e495649c Vacuum the SQLite DB after running the garbage collector 2012-09-13 14:33:41 -04:00
Eelco Dolstra e94806d030 Merge branch 'master' into no-manifests 2012-08-27 11:09:07 -04:00
Eelco Dolstra 6763084ae5 Count bytes freed deleting unused links 2012-08-01 22:43:03 -04:00
Eelco Dolstra 01d56c1eec Drop the block count in the garbage collector 2012-08-01 22:34:46 -04:00
Eelco Dolstra c770a2422a Report substituter errors to clients of the Nix daemon 2012-08-01 11:19:24 -04:00
Eelco Dolstra 97421eb5ec Refactor settings processing
Put all Nix configuration flags in a Settings object.
2012-07-30 19:55:41 -04:00
Eelco Dolstra f9613da180 Remove unused variables 2012-07-30 15:43:37 -04:00
Eelco Dolstra 8c79100839 Merge branch 'master' into no-manifests 2012-07-26 15:14:33 -04:00
Eelco Dolstra 680ab6f83d Garbage collect unused links in /nix/store/.links
Incremental optimisation requires creating links in /nix/store/.links
to all files in the store.  However, this means that if we delete a
store path, no files are actually deleted because links in
/nix/store/.links still exists.  So we need to check /nix/store/.links
for files with a link count of 1 and delete them.
2012-07-23 17:14:16 -04:00
Eelco Dolstra 6193105710 Automatically optimise the Nix store when a new path is added
Auto-optimisation is enabled by default.  It can be turned off by
setting auto-optimise-store to false in nix.conf.
2012-07-23 17:14:15 -04:00
Eelco Dolstra 564fb7d9fa optimiseStore(): Use a content-addressed file store in /nix/store/.links
optimiseStore() now creates persistent, content-addressed hard links
in /nix/store/.links.  For instance, if it encounters a file P with
hash H, it will create a hard link

  P' = /nix/store/.link/<H>

to P if P' doesn't already exist; if P' exist, then P is replaced by a
hard link to P'.  This is better than the previous in-memory map,
because it had the tendency to unnecessarily replace hard links with a
hard link to whatever happened to be the first file with a given hash
it encountered.  It also allows on-the-fly, incremental optimisation.
2012-07-23 17:14:15 -04:00
Eelco Dolstra b7fd2c2822 Use "#pragma once" to prevent repeated header file inclusion 2012-07-18 14:59:03 -04:00
Eelco Dolstra fe241ece29 Merge branch 'master' into no-manifests 2012-07-18 10:47:59 -04:00
Eelco Dolstra ccc52adfb2 Add function queryPathFromHashPart()
To implement binary caches efficiently, Hydra needs to be able to map
the hash part of a store path (e.g. "gbg...zr7") to the full store
path (e.g. "/nix/store/gbg...kzr7-subversion-1.7.5").  (The binary
cache mechanism uses hash parts as a key for looking up store paths to
ensure privacy.)  However, doing a search in the Nix store for
/nix/store/<hash>* is expensive since it requires reading the entire
directory.  queryPathFromHashPart() prevents this by doing a cheap
database lookup.
2012-07-17 18:55:39 -04:00
Eelco Dolstra 09a6321aeb Replace hasSubstitutes() with querySubstitutablePaths()
querySubstitutablePaths() takes a set of paths, so this greatly
reduces daemon <-> client latency.
2012-07-11 17:52:18 -04:00
Eelco Dolstra 58ef4d9a95 Add a function queryValidPaths()
queryValidPaths() combines multiple calls to isValidPath() in one.
This matters when using the Nix daemon because it reduces latency.
For instance, on "nix-env -qas \*" it reduces execution time from 5.7s
to 4.7s (which is indistinguishable from the non-daemon case).
2012-07-11 11:08:47 -04:00
Eelco Dolstra 667d5f1936 Rename queryValidPaths() to queryAllValidPaths() 2012-07-11 10:49:04 -04:00
Eelco Dolstra eb3036da87 Implement querySubstitutablePathInfos() in the daemon
Also removed querySubstitutablePathInfo().
2012-07-11 10:43:24 -04:00
Eelco Dolstra 11800e6198 download-from-binary-cache: parallelise fetching of NAR info files
Getting substitute information using the binary cache substituter has
non-trivial latency overhead.  A package or NixOS system configuration
can have hundreds of dependencies, and in the worst case (when the
local info cache is empty) we have to do a separate HTTP request for
each of these.  If the ping time to the server is t, getting N info
files will take tN seconds; e.g., with a ping time of 0.1s to
nixos.org, sequentially downloading 1000 info files (a typical NixOS
config) will take at least 100 seconds.

To fix this problem, the binary cache substituter can now perform
requests in parallel.  This required changing the substituter
interface to support a function querySubstitutablePathInfos() that
queries multiple paths at the same time, and rewriting queryMissing()
to take advantage of parallelism.  (Due to local caching,
parallelising queryMissing() is sufficient for most use cases, since
it's almost always called before building a derivation and thus fills
the local info cache.)

For example, parallelism speeds up querying all 1056 paths in a
particular NixOS system configuration from 116s to 2.6s.  It works so
well because the eccentricity of the top-level derivation in the
dependency graph is only 9.  So we only need 10 round-trips (when
using an unlimited number of parallel connections) to get everything.

Currently we do a maximum of 150 parallel connections to the server.
Thus it's important that the binary cache server (e.g. nixos.org) has
a high connection limit.  Alternatively we could use HTTP pipelining,
but WWW::Curl doesn't support it and libcurl has a hard-coded limit of
5 requests per pipeline.
2012-07-06 19:08:20 -04:00
Eelco Dolstra 1aba0bf0fa nix-store -r: do substitutions in parallel
I.e. when multiple non-derivation arguments are passed to ‘nix-store
-r’ to be substituted, do them in parallel.
2012-06-27 16:58:15 -04:00
Eelco Dolstra 4bc4da331a Reserve some disk space for the garbage collector
We can't open a SQLite database if the disk is full.  Since this
prevents the garbage collector from running when it's most needed, we
reserve some dummy space that we can free just before doing a garbage
collection.  This actually revives some old code from the Berkeley DB
days.

Fixes #27.
2012-05-29 22:59:12 -04:00
Eelco Dolstra 117670be57 Make the garbage collector more concurrent
Make the garbage collector more concurrent by deleting valid paths
outside the region where we're holding the global GC lock.  This
should greatly reduce the time during which new builds are blocked,
since the deletion accounts for the vast majority of the time spent in
the GC.

To ensure that this is safe, the valid paths are invalidated and
renamed to some arbitrary path while we're holding the lock.  This
ensures that we when we finally delete the path, it's not a (newly)
valid or locked path.
2012-03-26 20:43:33 +02:00
Eelco Dolstra 194d21f9f6 * Sync with the trunk. 2011-12-16 23:33:01 +00:00
Eelco Dolstra 273b288a7e * importPath() -> importPaths(). Because of buffering of the input
stream it's now necessary for the daemon to process the entire
  sequence of exported paths, rather than letting the client do it.
2011-12-16 22:31:25 +00:00
Shea Levy af2e53fd48 Include all outputs of derivations in the closure of explicitly-passed derivation paths
This required adding a queryOutputDerivationNames function in the store API
2011-11-06 06:28:20 +00:00
Eelco Dolstra 13114daa3e * Ouch. A store upgrade could cause a substituter to be triggered,
causing a deadlock.
2011-09-12 09:07:43 +00:00
Eelco Dolstra b2027f70d9 * Fix a huuuuge security hole in the Nix daemon. It didn't check that
derivations added to the store by clients have "correct" output
  paths (meaning that the output paths are computed by hashing the
  derivation according to a certain algorithm).  This means that a
  malicious user could craft a special .drv file to build *any*
  desired path in the store with any desired contents (so long as the
  path doesn't already exist).  Then the attacker just needs to wait
  for a victim to come along and install the compromised path.

  For instance, if Alice (the attacker) knows that the latest Firefox
  derivation in Nixpkgs produces the path

    /nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1

  then (provided this path doesn't already exist) she can craft a .drv
  file that creates that path (i.e., has it as one of its outputs),
  add it to the store using "nix-store --add", and build it with
  "nix-store -r".  So the fake .drv could write a Trojan to the
  Firefox path.  Then, if user Bob (the victim) comes along and does

    $ nix-env -i firefox
    $ firefox

  he executes the Trojan injected by Alice.

  The fix is to have the Nix daemon verify that derivation outputs are
  correct (in addValidPath()).  This required some refactoring to move
  the hash computation code to libstore.
2011-07-20 18:10:47 +00:00
Eelco Dolstra 8062d3af30 * `nix-store --verify --check-contents': don't hold the global GC lock
while checking the contents, since this operation can take a very
  long time to finish.  Also, fill in missing narSize fields in the DB
  while doing this.
2010-12-06 15:29:38 +00:00
Eelco Dolstra e60c962fb8 * Add an operation `nix-store -q --size'. 2010-11-17 12:40:52 +00:00
Eelco Dolstra a3883cbd28 * Store the size of a store path in the database (to be precise, the
size of the NAR serialisation of the path, i.e., `nix-store --dump
  PATH').  This is useful for Hydra.
2010-11-16 17:11:46 +00:00
Eelco Dolstra e2e168f7c2 `nix-store --verify' improvements:
* If a path has disappeared, check its referrers first, and don't try
  to invalidate paths that have valid referrers.  Otherwise we get a
  foreign key constraint violation.
* Read the whole Nix store directory instead of statting each valid
  path, which is slower.
* Acquire the global GC lock.
2010-08-31 11:47:31 +00:00
Eelco Dolstra 3e5e0faf9c * Okay, putting a lock on the temporary directory used by importPath()
doesn't work because the garbage collector doesn't actually look at
  locks.  So r22253 was stupid.  Use addTempRoot() instead.  Also,
  locking the temporary directory in exportPath() was silly because it
  isn't even in the store.
2010-06-21 11:08:09 +00:00
Eelco Dolstra f92c9a0ac5 * Allow unprivileged users to do `nix-store --clear-failed-paths' and
`nix-store --query-failed-paths'.
2010-05-04 10:45:10 +00:00
Eelco Dolstra 6199f9b93e * Added a command `nix-store --clear-failed-paths <PATHS>' to clear
the "failed" status of the given store paths.  The special value `*'
  clears all failed paths.
2010-04-26 12:56:42 +00:00
Eelco Dolstra 2398af13c5 * Add an command `nix-store --query-failed-paths' to list the cached
failed paths (when using the `build-cache-failure' option).
2010-04-26 12:43:42 +00:00
Eelco Dolstra e33f67ff0b * Refactor the upgrade / database initialisation logic a bit. 2010-02-24 16:30:20 +00:00
Eelco Dolstra cfe742cfc5 * A function to query just the database id of a valid path. 2010-02-24 15:07:23 +00:00
Eelco Dolstra a3c63d0d6c * Disable fsync() in SQLite if the fsync-metadata option is set to
false.
* Change the default for `fsync-metadata' to true.
* Disable `fsync-metadata' in `make check'.
2010-02-24 13:12:57 +00:00
Eelco Dolstra c4d388add4 * Get derivation outputs from the database instead of the .drv file,
which requires more I/O.
2010-02-22 12:44:36 +00:00
Eelco Dolstra 103cfee056 * Revert r19650 (implement gc-keep-outputs by looking for derivations
with the same name as the output) and instead use the
  DerivationOutputs table in the database, which is the correct way to
  to do things.
2010-02-22 11:44:17 +00:00
Eelco Dolstra 299ff64812 * Put the derivation outputs in the database. This is useful for the
garbage collector.
2010-02-22 11:15:50 +00:00
Eelco Dolstra 1930570ad9 * Foreign key support in SQLite is not a persistent setting, so enable
it at startup.
* Implement negative caching.  Now `make check' passes.
2010-02-19 17:15:22 +00:00
Eelco Dolstra 9c9a88e9e2 * Implement more stuff. 2010-02-19 16:43:25 +00:00
Eelco Dolstra 762cee72cc * Implement registerValidPath(). 2010-02-19 16:04:51 +00:00
Eelco Dolstra 836e5b6f57 * Implemented queryReferrers(). 2010-02-18 16:21:59 +00:00
Eelco Dolstra 77cb9e3fb1 * Implement queryPathInfo(). 2010-02-18 15:52:57 +00:00
Eelco Dolstra 885e22b16e * Implement isValidPath(). 2010-02-18 15:11:08 +00:00
Eelco Dolstra e0305bb7a8 * Some wrapper objects to ensure that SQLite objects are properly
destroyed.
2010-02-18 14:30:42 +00:00
Eelco Dolstra c1a07f9445 * Convert the Nix database to SQLite. 2010-02-18 13:16:59 +00:00
Eelco Dolstra 07ffdc2862 * Added an option "fsync-metadata" to fsync() changes to
/nix/var/nix/db.
* Removed the function writeStringToFile since it does (almost) the
  same thing as writeFile.
2010-01-29 12:22:58 +00:00
Eelco Dolstra 5388944e8d * Make the garbage collector do the right thing when `gc-keep-outputs'
is enabled by not depending on the deriver.
2010-01-25 16:04:32 +00:00
Eelco Dolstra c364d5d1e3 * Made the garbage collector a lot faster. It no longer computes the
complete set of live and dead paths before starting the actual
  deletion, but determines liveness on demand.  I.e. for any path in
  the store, it first tries to delete all the referrers, and then the
  path itself.  This means that the collector can start deleting paths
  almost immediately.
2009-11-23 16:34:24 +00:00
Eelco Dolstra 6b9f6b0222 * Remove a prototype for a function that no longer exists. 2009-10-22 08:12:38 +00:00
Eelco Dolstra 14bc3ce3d6 * Canonicalise timestamps in the Nix store to 1 (1970-01-01 00:00:01
UTC) rather than 0 (00:00:00).  1 is a better choice because some
  programs use 0 as a special value.  For instance, the Template
  Toolkit uses a timestamp of 0 to denote the non-existence of a file,
  so it barfs on files in the Nix store (see
  template-toolkit-nix-store.patch in Nixpkgs).  Similarly, Maya 2008
  fails to load script directories with a timestamp of 0 and can't be
  patched because it's closed source.

  This will also shut up those "implausibly old time stamp" GNU tar
  warnings.
2009-06-13 16:30:58 +00:00
Eelco Dolstra c7152c8f97 * Don't use the non-standard __gnu_cxx::stdio_filebuf class. 2009-03-28 19:41:53 +00:00
Eelco Dolstra 92f525ecf4 * Negative caching, i.e. caching of build failures. Disabled by
default.  This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
Eelco Dolstra 8fce03e0ad * nix-store --verify: repair bad hash fields in the metadata file. 2008-12-16 13:28:18 +00:00
Eelco Dolstra 82ae85de27 * addToStore() in nix-worker: don't write the NAR dump received from
the client to a temporary directory, as that is highly inefficient.
2008-12-03 18:05:14 +00:00
Eelco Dolstra ff762fb499 * Pass HashType values instead of strings. 2008-12-03 16:10:17 +00:00
Eelco Dolstra 64519cfd65 * Unify the treatment of sources copied to the store, and recursive
SHA-256 outputs of fixed-output derivations.  I.e. they now produce
  the same store path:

  $ nix-store --add x
  /nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x

  $ nix-store --add-fixed --recursive sha256 x
  /nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x

  the latter being the same as the path that a derivation

    derivation {
      name = "x";
      outputHashAlgo = "sha256";
      outputHashMode = "recursive";
      outputHash = "...";
      ...
    };

  produces.

  This does change the output path for such fixed-output derivations.
  Fortunately they are quite rare.  The most common use is fetchsvn
  calls with SHA-256 hashes.  (There are a handful of those is
  Nixpkgs, mostly unstable development packages.)
  
* Documented the computation of store paths (in store-api.cc).
2008-12-03 15:06:30 +00:00
Eelco Dolstra c987061aa4 * Some refactoring. Better output with `-v' for --use-atime. 2008-09-17 12:54:07 +00:00
Eelco Dolstra 7ab68961e4 * Garbage collector: added an option `--use-atime' to delete paths in
order of ascending last access time.  This is useful in conjunction
  with --max-freed or --max-links to prefer deleting non-recently used
  garbage, which is good (especially in the build farm) since garbage
  may become live again.

  The code could easily be modified to accept other criteria for
  ordering garbage by changing the comparison operator used by the
  priority queue in collectGarbage().
2008-09-17 10:02:55 +00:00
Eelco Dolstra 5adbb0aabe * build.cc: only use a substituter if it returns info for a path. 2008-08-04 13:15:35 +00:00
Eelco Dolstra 3c92ea399d * Make nix-env --dry-run print the paths to be substituted correctly
again.  (After the previous substituter mechanism refactoring I
  didn't update the code that obtains the references of substitutable
  paths.)  This required some refactoring: the substituter programs
  are now kept running and receive/respond to info requests via
  stdin/stdout.
2008-08-02 12:54:35 +00:00
Eelco Dolstra a72709afd8 * Some refactoring: put the GC options / results in separate structs.
* The garbage collector now also prints the number of blocks freed.
2008-06-18 09:34:17 +00:00
Eelco Dolstra 826b271d9a * Garbage collector: don't do a complete topological sort of the Nix
store under the reference relation, since that means that the
  garbage collector will need a long time to start deleting paths.
  Instead just delete the referrers of a path first.
2008-06-13 18:25:24 +00:00
Eelco Dolstra b0e92f6d47 * Merged the no-bdb branch (-r10900:HEAD
https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
Eelco Dolstra 66c51dc215 * nix-store --dump-db / --load-db to dump/load the Nix DB.
* nix-store --register-validity: option to supply the content hash of
  each path.
* Removed compatibility with Nix <= 0.7 stores.
2008-01-29 18:17:36 +00:00
Eelco Dolstra a8629de827 * New command `nix-store --optimise' to reduce Nix store disk space
usage by finding identical files in the store and hard-linking them
  to each other.  It typically reduces the size of the store by
  something like 25-35%.  This is what the optimise-store.pl script
  did, but the new command is faster and more correct (it's safe wrt
  garbage collection and concurrent builds).
2007-10-09 22:14:27 +00:00
Eelco Dolstra 3757ee589f * Bump the Nix database schema version number; delete the substitutes
table.
2007-08-13 11:37:39 +00:00
Eelco Dolstra 9e975458b4 * Get rid of the substitutes database table (NIX-47). Instead, if we
need any info on substitutable paths, we just call the substituters
  (such as download-using-manifests.pl) directly.  This means that
  it's no longer necessary for nix-pull to register substitutes or for
  nix-channel to clear them, which makes those operations much faster
  (NIX-95).  Also, we don't have to worry about keeping nix-pull
  manifests (in /nix/var/nix/manifests) and the database in sync with
  each other.

  The downside is that there is some overhead in calling an external
  program to get the substitutes info.  For instance, "nix-env -qas"
  takes a bit longer.

  Abolishing the substitutes table also makes the logic in
  local-store.cc simpler, as we don't need to store info for invalid
  paths.  On the downside, you cannot do things like "nix-store -qR"
  on a substitutable but invalid path (but nobody did that anyway).

* Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
Eelco Dolstra 6d1a1191b0 * Support queryDeriver() in multi-user installations. 2007-06-12 16:53:44 +00:00
Eelco Dolstra 7046c35843 * Typo (reported by Marc Weber). 2007-05-14 12:16:41 +00:00
Eelco Dolstra 43c4d18c6a * nix-store --import': import an archive created by nix-store
--export' into the Nix store, and optionally check the cryptographic
  signatures against /nix/etc/nix/signing-key.pub.  (TODO: verify
  against a set of public keys.)
2007-02-21 15:45:32 +00:00
Eelco Dolstra b824a1daee * Start of `nix-store --export' operation for serialising a store
path.  This is like `nix-store --dump', only it also dumps the
  meta-information of the store path (references, deriver).  Will add
  a `--sign' flag later to add a cryptographic signature, which we
  will use for exchanging store paths between build farm machines in a
  secure manner.
2007-02-20 23:17:20 +00:00
Eelco Dolstra 8f67b35886 * Make the garbage collector more resilient to certain consistency
errors: in-use paths now cause a warning, not a fatal error.
2007-01-14 16:24:49 +00:00
Eelco Dolstra a3e6415ba8 * New primop builtins.filterSource, which can be used to filter files
from a source directory.  All files for which a predicate function
  returns true are copied to the store.  Typical example is to leave
  out the .svn directory:

    stdenv.mkDerivation {
      ...
      src = builtins.filterSource
        (path: baseNameOf (toString path) != ".svn")
        ./source-dir;
      # as opposed to
      #   src = ./source-dir;
    }

  This is important because the .svn directory influences the hash in
  a rather unpredictable and variable way.
2006-12-12 23:05:01 +00:00
Eelco Dolstra 5f681988f2 * Use deletePathWrapped() in more places. 2006-12-09 00:26:24 +00:00
Eelco Dolstra ec23ecc64d * In the garbage collector, if deleting a path fails, try to fix its
ownership, then try again.
2006-12-07 15:54:52 +00:00
Eelco Dolstra a9c4f66cfb * Allow unprivileged users to run the garbage collector and to do
`nix-store --delete'.  But unprivileged users are not allowed to
  ignore liveness.
* `nix-store --delete --ignore-liveness': ignore the runtime roots as
  well.
2006-12-05 02:18:46 +00:00
Eelco Dolstra 29cf434a35 * The determination of the root set should be made by the privileged
process, so forward the operation.
* Spam the user about GC misconfigurations (NIX-71).
* findRoots: skip all roots that are unreadable - the warnings with
  which we spam the user should be enough.
2006-12-05 01:31:45 +00:00
Eelco Dolstra 74033a844f * Add indirect root registration to the protocol so that unprivileged
processes can register indirect roots.  Of course, there is still
  the problem that the garbage collector can only read the targets of
  the indirect roots when it's running as root...
2006-12-04 23:29:16 +00:00
Eelco Dolstra e25fad691a * Move addTempRoot() to the store API, and add another function
syncWithGC() to allow clients to register GC roots without needing
  write access to the global roots directory or the GC lock.
2006-12-02 16:41:36 +00:00
Eelco Dolstra a824d58b56 * Merge addToStore and addToStoreFixed.
* addToStore now adds unconditionally, it doesn't use readOnlyMode.
  Read-only operation is up to the caller (who can call
  computeStorePathForPath).
2006-12-01 20:51:18 +00:00
Eelco Dolstra 6ecb840fd1 * Put building in the store API. 2006-11-30 18:02:04 +00:00
Eelco Dolstra e2ef5e07fd * Refactoring. There is now an abstract interface class StoreAPI
containing functions that operate on the Nix store.  One
  implementation is LocalStore, which operates on the Nix store
  directly.  The next step, to enable secure multi-user Nix, is to
  create a different implementation RemoteStore that talks to a
  privileged daemon process that uses LocalStore to perform the actual
  operations.
2006-11-30 17:43:04 +00:00