optimiseStore() now creates persistent, content-addressed hard links
in /nix/store/.links. For instance, if it encounters a file P with
hash H, it will create a hard link
P' = /nix/store/.link/<H>
to P if P' doesn't already exist; if P' exist, then P is replaced by a
hard link to P'. This is better than the previous in-memory map,
because it had the tendency to unnecessarily replace hard links with a
hard link to whatever happened to be the first file with a given hash
it encountered. It also allows on-the-fly, incremental optimisation.
We can't open a SQLite database if the disk is full. Since this
prevents the garbage collector from running when it's most needed, we
reserve some dummy space that we can free just before doing a garbage
collection. This actually revives some old code from the Berkeley DB
days.
Fixes#27.
environment of the given derivation in a format that can be sourced
by the shell, e.g.
$ eval "$(nix-store --print-env $(nix-instantiate /etc/nixos/nixpkgs -A pkg))"
$ NIX_BUILD_TOP=/tmp
$ source $stdenv/setup
This is especially useful to reproduce the environment used to build
a package outside of its builder for development purposes.
TODO: add a nix-build option to do the above and fetch the
dependencies of the derivation as well.
‘nix-store --export’.
* Add a Perl module that provides the functionality of
‘nix-copy-closure --to’. This is used by build-remote.pl so it no
longer needs to start a separate nix-copy-closure process. Also, it
uses the Perl API to do the export, so it doesn't need to start a
separate nix-store process either. As a result, nix-copy-closure
and build-remote.pl should no longer fail on very large closures due
to an "Argument list too long" error. (Note that having very many
dependencies in a single derivation can still fail because the
environment can become too large. Can't be helped though.)
the contents of any of the given store paths have been modified.
E.g.
$ nix-store --verify-path $(nix-store -qR /var/run/current-system)
path `/nix/store/m2smyiwbxidlprfxfz4rjlvz2c3mg58y-etc' was modified! expected hash `fc87e271c5fdf179b47939b08ad13440493805584b35e3014109d04d8436e7b8', got `20f1a47281b3c0cbe299ce47ad5ca7340b20ab34246426915fce0ee9116483aa'
All paths are checked; the exit code is 1 if any path has been
modified, 0 otherwise.
This should also fix:
nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed.
which was caused by hashDerivationModulo() calling the ‘store’
object (during store upgrades) before openStore() assigned it.
because it defines _FILE_OFFSET_BITS. Without this, on
OpenSolaris the system headers define it to be 32, and then
the 32-bit stat() ends up being called with a 64-bit "struct
stat", or vice versa.
This also ensures that we get 64-bit file sizes everywhere.
* Remove the redundant call to stat() in parseExprFromFile().
The file cannot be a symlink because that's the exit condition
of the loop before.
(Linux) machines no longer maintain the atime because it's too
expensive, and on the machines where --use-atime is useful (like the
buildfarm), reading the atimes on the entire Nix store takes way too
much time to make it practical.
would just silently store only (fileSize % 2^32) bytes.
* Use posix_fallocate if available when unpacking archives.
* Provide a better error message when trying to unpack something that
isn't a NAR archive.
SHA-256 outputs of fixed-output derivations. I.e. they now produce
the same store path:
$ nix-store --add x
/nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x
$ nix-store --add-fixed --recursive sha256 x
/nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x
the latter being the same as the path that a derivation
derivation {
name = "x";
outputHashAlgo = "sha256";
outputHashMode = "recursive";
outputHash = "...";
...
};
produces.
This does change the output path for such fixed-output derivations.
Fortunately they are quite rare. The most common use is fetchsvn
calls with SHA-256 hashes. (There are a handful of those is
Nixpkgs, mostly unstable development packages.)
* Documented the computation of store paths (in store-api.cc).
accessed time of paths that may be deleted. Anything more recently
used won't be deleted. The time is specified in time_t,
e.g. seconds since 1970-01-01 00:00:00 UTC; use `date +%s' to
convert to time_t from the command line.
Example: to delete everything that hasn't been used in the last two
months:
$ nix-store --gc -v --max-atime $(date +%s -d "2 months ago")
order of ascending last access time. This is useful in conjunction
with --max-freed or --max-links to prefer deleting non-recently used
garbage, which is good (especially in the build farm) since garbage
may become live again.
The code could easily be modified to accept other criteria for
ordering garbage by changing the comparison operator used by the
priority queue in collectGarbage().
bytes have been freed, `--max-links' to stop when the Nix store
directory has fewer than N hard links (the latter being important
for very large Nix stores on filesystems with a 32000 subdirectories
limit).
$ nix-env -e $(which firefox)
or
$ nix-env -e /nix/store/nywzlygrkfcgz7dfmhm5xixlx1l0m60v-pan-0.132
* nix-env -i: if an argument contains a slash anywhere, treat it as a
path and follow it through symlinks into the Nix store. This allows
things like
$ nix-build -A firefox
$ nix-env -i ./result
* nix-env -q/-i/-e: don't complain when the `*' selector doesn't match
anything. In particular, `nix-env -q \*' doesn't fail anymore on an
empty profile.
usage by finding identical files in the store and hard-linking them
to each other. It typically reduces the size of the store by
something like 25-35%. This is what the optimise-store.pl script
did, but the new command is faster and more correct (it's safe wrt
garbage collection and concurrent builds).
need any info on substitutable paths, we just call the substituters
(such as download-using-manifests.pl) directly. This means that
it's no longer necessary for nix-pull to register substitutes or for
nix-channel to clear them, which makes those operations much faster
(NIX-95). Also, we don't have to worry about keeping nix-pull
manifests (in /nix/var/nix/manifests) and the database in sync with
each other.
The downside is that there is some overhead in calling an external
program to get the substitutes info. For instance, "nix-env -qas"
takes a bit longer.
Abolishing the substitutes table also makes the logic in
local-store.cc simpler, as we don't need to store info for invalid
paths. On the downside, you cannot do things like "nix-store -qR"
on a substitutable but invalid path (but nobody did that anyway).
* Never catch interrupts (the Interrupted exception).
which paths specified on the command line are invalid (i.e., don't
barf when encountering an invalid path, just print it). This is
useful for build-remote.pl to figure out which paths need to be
copied to a remote machine. (Currently we use rsync, but that's
rather inefficient.)
--export' into the Nix store, and optionally check the cryptographic
signatures against /nix/etc/nix/signing-key.pub. (TODO: verify
against a set of public keys.)
path. This is like `nix-store --dump', only it also dumps the
meta-information of the store path (references, deriver). Will add
a `--sign' flag later to add a cryptographic signature, which we
will use for exchanging store paths between build farm machines in a
secure manner.
* nix-unpack-closure: extract the top-level paths from the closure and
print them on stdout. This allows them to be installed, e.g.,
"nix-env -i $(nix-unpack-closure)". (NIX-64)
`nix-store --delete'. But unprivileged users are not allowed to
ignore liveness.
* `nix-store --delete --ignore-liveness': ignore the runtime roots as
well.
* Some refactoring: put the NAR archive integer/string serialisation
code in a separate file so it can be reused by the worker protocol
implementation.
containing functions that operate on the Nix store. One
implementation is LocalStore, which operates on the Nix store
directly. The next step, to enable secure multi-user Nix, is to
create a different implementation RemoteStore that talks to a
privileged daemon process that uses LocalStore to perform the actual
operations.
gives a huge speedup in operations that read or write from standard
input/output. (So libstdc++'s I/O isn't that bad, you just have to
call std::ios::sync_with_stdio(false).) For instance, `nix-store
--register-substitutes' went from 1.4 seconds to 0.1 seconds on a
certain input. Another victory for Valgrind.
available. For instance,
$ nix-store -l $(which svn) | less
lets you read the build log of the Subversion instance in your
profile.
* `nix-store -qb': if applied to a non-derivation, take the deriver.
the disk is full (because to delete something from the Nix store, we
need a Berkeley DB transaction, which takes up disk space). Under
normal operation, we make sure that there exists a file
/nix/var/nix/db/reserved of 1 MB. When running the garbage
collector, we delete that file before we open the Berkeley DB
environment.
deletes a path even if it is reachable from a root. However, it
won't delete a path that still has referrers (since that would
violate store invariants).
Don't try this at home. It's a useful hack for recovering from
certain situations in a somewhat clean way (e.g., holes in closures
due to disk corruption).
