evaluator. This was important because the NixOS expressions started
to hit 2 MB default stack size on Linux.
GCC is really dumb about stack space: it just adds up all the local
variables and temporaries of every scope into one huge stack frame.
This is really bad for deeply recursive functions. For instance,
every `throw Error(format("error message"))' causes a format object
of a few hundred bytes to be allocated on the stack. As a result,
every recursive call to evalExpr2() consumed 4680 bytes. By
splitting evalExpr2() and by moving the exception-throwing code out
of the main functions, evalExpr2() now only consumes 40 bytes.
Similar for evalExpr().
which paths specified on the command line are invalid (i.e., don't
barf when encountering an invalid path, just print it). This is
useful for build-remote.pl to figure out which paths need to be
copied to a remote machine. (Currently we use rsync, but that's
rather inefficient.)
--export' into the Nix store, and optionally check the cryptographic
signatures against /nix/etc/nix/signing-key.pub. (TODO: verify
against a set of public keys.)
path. This is like `nix-store --dump', only it also dumps the
meta-information of the store path (references, deriver). Will add
a `--sign' flag later to add a cryptographic signature, which we
will use for exchanging store paths between build farm machines in a
secure manner.
attribute) about installed packages in user environments. Thus, an
operation like `nix-env -q --description' shows useful information
not only on available packages but also on installed packages.
* nix-env now passes the entire manifest as an argument to the Nix
expression of the user environment builder (not just a list of
paths), so that in particular the user environment builder has
access to the meta attributes.
* New operation `--set-flag' in nix-env to change meta info of
installed packages. This will be useful to pass per-package
policies to the user environment builder (e.g., how to resolve
collision or whether to disable a package (NIX-80)) or upgrade
policies in nix-env (e.g., that a package should be "masked", that
is, left untouched by upgrade actions). Example:
$ nix-env --set-flag enabled false ghc-6.4
computing the store path (NIX-77). This is an important security
property in multi-user Nix stores.
Note that this changes the store paths of derivations (since the
derivation aterms are added using addTextToStore), but not most
outputs (unless they use builtins.toFile).
* `sub' to subtract two numbers.
* `stringLength' to get the length of a string.
* `substring' to get a substring of a string. These should be enough
to allow most string operations to be expressed.
programs, so if a builder uses TMPDIR, then it will fail when
executed through nix-setuid-helper. In fact Glibc clears a whole
bunch of variables (see sysdeps/generic/unsecvars.h in the Glibc
sources), but only TMPDIR should matter in practice. As a
workaround, we reinitialise TMPDIR from NIX_BUILD_TOP.
important to get garbage collection to work if there is any
inconsistency in the database (because the referrer table is used to
determine whether it is safe to delete a path).
* `nix-store --verify': show some progress.
* nix-unpack-closure: extract the top-level paths from the closure and
print them on stdout. This allows them to be installed, e.g.,
"nix-env -i $(nix-unpack-closure)". (NIX-64)
<derivation outPath=... drvPath=...> attrs </derivation>. Only emit
the attributes of any specific derivation only. This prevents
exponententially large XML output due to the absense of sharing.
from a source directory. All files for which a predicate function
returns true are copied to the store. Typical example is to leave
out the .svn directory:
stdenv.mkDerivation {
...
src = builtins.filterSource
(path: baseNameOf (toString path) != ".svn")
./source-dir;
# as opposed to
# src = ./source-dir;
}
This is important because the .svn directory influences the hash in
a rather unpredictable and variable way.
single derivation specified by the argument. This is useful when we
want to have a profile for a single derivation, such as a server
configuration. Then we can just say (e.g.)
$ nix-env -p /.../server-profile -f server.nix --set -A server
We can't do queries or upgrades on such a profile, but we can do
rollbacks. The advantage over -i is that we don't have to worry
about other packages having been installed in the profile
previously; --set gets rid of them.
matters when running as root, since then we don't use the setuid
helper (which already used lchown()).
* Also check for an obscure security problem on platforms that don't
have lchown. Then we can't change the ownership of symlinks, which
doesn't matter *except* when the containing directory is writable by
the owner (which is the case with the top-level Nix store directory).
* Throw more exceptions as BuildErrors instead of Errors. This
matters when --keep-going is turned on. (A BuildError is caught
and terminates the goal in question, an Error terminates the
program.)
seconds without producing output on stdout or stderr (NIX-65). This
timeout can be specified using the `--max-silent-time' option or the
`build-max-silent-time' configuration setting. The default is
infinity (0).
* Fix a tricky race condition: if we kill the build user before the
child has done its setuid() to the build user uid, then it won't be
killed, and we'll potentially lock up in pid.wait(). So also send a
conventional kill to the child.
that have to be done as root: running builders under different uids,
changing ownership of build results, and deleting paths in the store
with the wrong ownership).
`nix-store --delete'. But unprivileged users are not allowed to
ignore liveness.
* `nix-store --delete --ignore-liveness': ignore the runtime roots as
well.
process, so forward the operation.
* Spam the user about GC misconfigurations (NIX-71).
* findRoots: skip all roots that are unreadable - the warnings with
which we spam the user should be enough.
processes can register indirect roots. Of course, there is still
the problem that the garbage collector can only read the targets of
the indirect roots when it's running as root...
* SIGIO -> SIGPOLL (POSIX calls it that).
* Use sigaction instead of signal to register the SIGPOLL handler.
Sigaction is better defined, and a handler registered with signal
appears not to interrupt fcntl(..., F_SETLKW, ...), which is bad.
via the Unix domain socket in /nix/var/nix/daemon.socket. The
server forks a worker process per connection.
* readString(): use the heap, not the stack.
* Some protocol fixes.
* Added `build-users-group', the group under which builds are to be
performed.
* Check that /nix/store has 1775 permission and is owner by the
build-users-group.
The problem is that when we kill the client while the worker is
building, and the builder is not writing anything to stderr, then
the worker never notice that the socket is closed on the other side,
so it just continues indefinitely. The solution is to catch SIGIO,
which is sent when the far side of the socket closes, and simulate
an normal interruption. Of course, SIGIO is also sent every time
the client sends data over the socket, so we only enable the signal
handler when we're not expecting any data...
mode. Presumably nix-worker would be setuid to the Nix store user.
The worker performs all operations on the Nix store and database, so
the caller can be completely unprivileged.
This is already much more secure than the old setuid scheme, since
the worker doesn't need to do Nix expression evaluation and so on.
Most importantly, this means that it doesn't need to access any user
files, with all resulting security risks; it only performs pure
store operations.
Once this works, it is easy to move to a daemon model that forks off
a worker for connections established through a Unix domain socket.
That would be even more secure.
* Some refactoring: put the NAR archive integer/string serialisation
code in a separate file so it can be reused by the worker protocol
implementation.
containing functions that operate on the Nix store. One
implementation is LocalStore, which operates on the Nix store
directly. The next step, to enable secure multi-user Nix, is to
create a different implementation RemoteStore that talks to a
privileged daemon process that uses LocalStore to perform the actual
operations.
Rather, setuid support is now always compiled in (at least on
platforms that have the setresuid system call, e.g., Linux and
FreeBSD), but it must enabled by chowning/chmodding the Nix
binaries.
gives a huge speedup in operations that read or write from standard
input/output. (So libstdc++'s I/O isn't that bad, you just have to
call std::ios::sync_with_stdio(false).) For instance, `nix-store
--register-substitutes' went from 1.4 seconds to 0.1 seconds on a
certain input. Another victory for Valgrind.
graph to be passed to a builder. This attribute should be a list of
pairs [name1 path1 name2 path2 ...]. The references graph of each
`pathN' will be stored in a text file `nameN' in the temporary build
directory. The text files have the format used by `nix-store
--register-validity'. However, the deriver fields are left empty.
`exportReferencesGraph' is useful for builders that want to do
something with the closure of a store path. Examples: the builders
that make initrds and ISO images for NixOS.
`exportReferencesGraph' is entirely pure. It's necessary because
otherwise the only way for a builder to get this information would
be to call `nix-store' directly, which is not allowed (though
unfortunately possible).
available. For instance,
$ nix-store -l $(which svn) | less
lets you read the build log of the Subversion instance in your
profile.
* `nix-store -qb': if applied to a non-derivation, take the deriver.
check that the references of the output of a derivation are in the
specified set. For instance,
allowedReferences = [];
specifies that the output cannot have any references. (This is
useful, for instance, for the generation of bootstrap binaries for
stdenv-linux, which must not have any references for purity). It
could also be used to guard against undesired runtime dependencies,
e.g.,
{gcc, dynlib}: derivation {
...
allowedReferences = [dynlib];
}
says that the output can refer to the path of `dynlib' but not
`gcc'. A `forbiddedReferences' attribute would be more useful for
this, though.
concatenation and string coercion. This was a big mess (see
e.g. NIX-67). Contexts are now folded into strings, so that they
don't cause evaluation errors when they're not expected. The
semantics of paths has been clarified (see nixexpr-ast.def).
toString() and coerceToString() have been merged.
Semantic change: paths are now copied to the store when they're in a
concatenation (and in most other situations - that's the
formalisation of the meaning of a path). So
"foo " + ./bla
evaluates to "foo /nix/store/hash...-bla", not "foo
/path/to/current-dir/bla". This prevents accidental impurities, and
is more consistent with the treatment of derivation outputs, e.g.,
`"foo " + bla' where `bla' is a derivation. (Here `bla' would be
replaced by the output path of `bla'.)
side should be a path, I guess.
* Handle paths that are in the store but not direct children of the
store directory.
* Ugh, hack to prevent double context wrapping.
attribute existence and to return an attribute from an attribute
set, respectively. Example: `hasAttr "foo" {foo = 1;}'. They
differ from the `?' and `.' operators in that the attribute name is
an arbitrary expression. (NIX-61)
Nix-env failed to call addPermRoot(), which is necessary to safely
add a new root. So if nix-env started after and finished before the
garbage collector, the user environment (plus all other new stuff)
it built might be garbage collected, leading to a dangling symlink
chain in ~/.nix-profile...
* Be more explicit if we block on the GC lock ("waiting for the big
garbage collector lock...").
* Don't loop trying to create a new generation. It's not necessary
anymore since profiles are locked nowadays.
and returns its path. This can be used to (for instance) write
builders inside a Nix expression, e.g.,
stdenv.mkDerivation {
builder = "
source $stdenv/setup
...
";
...
}
derivation attributes to flatten them into strings. This is
possible since string can nowadays be wrapped in contexts that
describe the derivations/sources referenced by the evaluation of the
string.
all the primops. This allows Nix expressions to test for new
primops and take appropriate action if they're not available. For
instance, rather than calling a primop `foo' directly, they could
say `if builtins ? foo then builtins.foo ... else ...'.
writable. File permissions on Cygwin are rather complex, and in this
case this check introduced a problem with build jobs invoke from
outside of Cygwin (MSYS). It seemed almost impossible to fix the
permissions of the directory, so for now this safety check is disabled
on Cygwin.
https://svn.cs.uu.nl:12443/repos/trace/buildfarm-control/trunk/ext/nix/,
with some modifications. This allows `nix-env -qa' to show the
attribute path that can be used to unambiguously install a package
using `nix-env -i -A'. Example:
$ nix-env -f top-level/all-packages.nix -qaA subversion xorg-server
subversionWithJava subversion-1.2.3
subversion subversion-1.3.2
subversion14 subversion-1.4.0pre-rc1
xorg.xorgserver xorg-server-1.1.0
e.g.,
$ nix-env -i -A subversion xorg.xorgserver
The main advantage over using symbolic names is that using attribute
names is unambiguous and much, much faster.
argument has a valid value, i.e., is in a certain domain. E.g.,
{ foo : [true false]
, bar : ["a" "b" "c"]
}: ...
This previously could be done using assertions, but domain checks
will allow the buildfarm to automatically extract the configuration
space from functions.
"--with-freetype2-library=" + freetype + "/lib"
can now be written as
"--with-freetype2-library=${freetype}/lib"
An arbitrary expression can be enclosed within ${...}, not just
identifiers.
* Escaping in string literals: \n, \r, \t interpreted as in C, any
other character following \ is interpreted as-is.
* Newlines are now allowed in string literals.
packages (provided that they have a `meta.description' attribute).
E.g.,
$ ./src/nix-env/nix-env -qa --description gcc
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for sparc-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for mips-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for arm-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x
to be queried, e.g., `nix-env -qa firefox'. This does require the
argument '*' to be passed if one wants information about all
derivations, so the old `nix-env -qa' now is `nix-env -qa "*"'.
instantiation, e.g. "nix-env -i" and "nix-env -qas" (but not
"nix-env -qa"). It turns out that many redundant calls to
addToStore(path) were made, which reads and hashes the entire path.
For instance, the bash bootstrap binary in Nixpkgs would be read and
hashed many times. As a result nix-env would spend around 92% of
its time in the function sha256_block (according to callgrind).
Some simple memoization fixes this.
expressions that cause an assertion failure (like `assert system ==
"i686-linux"'). This allows all-packages.nix in Nixpkgs to be used
on all platforms, even if some Nix expressions don't work on all
platforms.
Not sure if this is a good idea; it's a bit hacky. In particular,
due to laziness some derivations might appear in `nix-env -qa' but
disappear in `nix-env -qas' or `nix-env -i'.
Commit 5000!
with the same name *and* version number, and pick the first one
(this means that the order in which channels appear in
~/.nix-channels matters). E.g.:
$ nix-env ii aterm
warning: there are multiple derivations named `aterm-2.4.2'; using the first one
installing `aterm-2.4.2'
the disk is full (because to delete something from the Nix store, we
need a Berkeley DB transaction, which takes up disk space). Under
normal operation, we make sure that there exists a file
/nix/var/nix/db/reserved of 1 MB. When running the garbage
collector, we delete that file before we open the Berkeley DB
environment.
implementations of MD5, SHA-1 and SHA-256. The main benefit is that
we get assembler-optimised implementations of MD5 and SHA-1 (though
not SHA-256 (at least on x86), unfortunately). OpenSSL's SHA-1
implementation on Intel is twice as fast as ours.
derivation(s) we're interested, e.g.,
$ nix-instantiate ./all-packages.nix --attr xlibs.libX11
List elements can also be selected:
$ nix-instantiate ./build-for-release.nix --attr 0.subversion
This allows a non-ambiguous specification of a derivation. Of
course, this should also be added to nix-env and nix-build.
creates a new process group but also a new session. New sessions
have no controlling tty, so child processes like ssh cannot open
/dev/tty (which is bad).
intended). This ensures that any ssh child processes to remote
machines are also killed, and thus the Nix process on the remote
machine also exits. Without this, the remote Nix process will
continue until it exists or until its stdout buffer gets full and it
locks up. (Partially fixes NIX-35.)
deletes a path even if it is reachable from a root. However, it
won't delete a path that still has referrers (since that would
violate store invariants).
Don't try this at home. It's a useful hack for recovering from
certain situations in a somewhat clean way (e.g., holes in closures
due to disk corruption).
nix-store query options `--referer' and `--referer-closure' have
been changed to `--referrer' and `--referrer-closure' (but the old
ones are still accepted for compatibility).
mapping. The referer table is replaced by a referrer table (note
spelling fix) that stores each referrer separately. That is,
instead of having
referer[P] = {Q_1, Q_2, Q_3, ...}
we store
referer[(P, Q_1)] = ""
referer[(P, Q_2)] = ""
referer[(P, Q_3)] = ""
...
To find the referrers of P, we enumerate over the keys with a value
lexicographically greater than P. This requires the referrer table
to be stored as a B-Tree rather than a hash table.
(The tuples (P, Q) are stored as P + null-byte + Q.)
Old Nix databases are upgraded automatically to the new schema.
Nix is properly shut down when it receives those signals. In
particular this ensures that killing the garbage collector doesn't
cause a subsequent database recovery.
builder. Instead, require that the Nix store has sticky permission
(S_ISVTX); everyone can created files in the Nix store, but they
cannot delete, rename or modify files created by others.
root (or setuid root), then builds will be performed under one of
the users listed in the `build-users' configuration variables. This
is to make it impossible to influence build results externally,
allowing locally built derivations to be shared safely between
users (see ASE-2005 paper).
To do: only one builder should be active per build user.
versions to available versions, or vice versa.
For example, the following compares installed versions to available
versions:
$ nix-env -qc
autoconf-2.59 = 2.59
automake-1.9.4 < 1.9.6
f-spot-0.0.10 - ?
firefox-1.0.4 < 1.0.7
...
I.e., there are newer versions available (in the current default Nix
expression) for Automake and Firefox, but not for Autoconf, and
F-Spot is missing altogether.
Conversely, the available versions can be compared to the installed
versions:
$ nix-env -qac
autoconf-2.59 = 2.59
automake-1.9.6 > 1.9.4
bash-3.0 - ?
firefox-1.0.7 > 1.0.4
...
Note that bash is available but no version of it is installed.
If multiple versions are available for comparison, then the highest
is used. E.g., if Subversion 1.2.0 is installed, and Subversion
1.1.4 and 1.2.3 are available, then `nix-env -qc' will print `<
1.2.3', not `> 1.1.4'.
If higher versions are available, the version column is printed in
red (using ANSI escape codes).
dependencyClosure { ... searchPath = [ ../foo ../bar ]; ... }
* Primop `dirOf' to return the directory part of a path (e.g., dirOf
/a/b/c == /a/b).
* Primop `relativise' (according to Webster that's a real word!) that
given paths A and B returns a string representing path B relative
path to A; e.g., relativise /a/b/c a/b/x/y => "../x/y".
determination (e.g., finding the header files dependencies of a C
file) in Nix low-level builds automatically.
For instance, in the function `compileC' in make/lib/default.nix, we
find the header file dependencies of C file `main' as follows:
localIncludes =
dependencyClosure {
scanner = file:
import (findIncludes {
inherit file;
});
startSet = [main];
};
The function works by "growing" the set of dependencies, starting
with the set `startSet', and calling the function `scanner' for each
file to get its dependencies (which should yield a list of strings
representing relative paths). For instance, when `scanner' is
called on a file `foo.c' that includes the line
#include "../bar/fnord.h"
then `scanner' should yield ["../bar/fnord.h"]. This list of
dependencies is absolutised relative to the including file and added
to the set of dependencies. The process continues until no more
dependencies are found (hence its a closure).
`dependencyClosure' yields a list that contains in alternation a
dependency, and its relative path to the directory of the start
file, e.g.,
[ /bla/bla/foo.c
"foo.c"
/bla/bar/fnord.h
"../bar/fnord.h"
]
These relative paths are necessary for the builder that compiles
foo.c to reconstruct the relative directory structure expected by
foo.c.
The advantage of `dependencyClosure' over the old approach (using
the impure `__currentTime') is that it's completely pure, and more
efficient because it only rescans for dependencies (i.e., by
building the derivations yielded by `scanner') if sources have
actually changed. The old approach rescanned every time.
(closed(closed(closed(...)))) since this reduces performance by
producing bigger terms and killing caching (which incidentally also
prevents useful infinite recursion detection).
with default values automatically. I.e., e -> e {}.
This feature makes convenience expressions such as
pkgs/system/i686-linux.nix in Nixpkgs obsolete, since we can just do
$ nix-instantiate ./pkgs/system/all-packages.nix
since all-packages.nix takes a single argument (system) that has a
default value (__thisSystem).
`removeAttrs attrs ["x", "y"]' returns the set `attrs' with the
attributes named `x' and `y' removed. It is not an error for the
named attributes to be missing from the input set.
* Make the `derivation' primitive much more lazy. The expression
`derivation attrs' now evaluates to (essentially)
attrs // {
type = "derivation";
outPath = derivation! attrs;
drvPath = derivation! attrs;
}
where `derivation!' is a primop that does the actual derivation
instantiation (i.e., it does what `derivation' used to do). The
advantage is that it allows commands such as `nix-env -qa' and
`nix-env -i' to be much faster since they no longer need to
instantiate all derivations, just the `name' attribute. (However,
`nix-env' doesn't yet take advantage of this since it still always
evaluates the `outPath' and `drvPath' attributes).
Also, this allows derivations to cyclically reference each other,
for example,
webServer = derivation {
...
hostName = "svn.cs.uu.nl";
services = [svnService];
};
svnService = derivation {
...
hostName = webServer.hostName;
};
Previously, this would yield a black hole (infinite recursion).
derivations. This is mostly to simplify the implementation of
nix-prefetch-{url, svn}, which now work properly in setuid
installations.
* Enforce valid store names in `nix-store --add / --add-fixed'.
continue building when one fails unless `--keep-going' is
specified.
* When `--keep-going' is specified, print out the set of failing
derivations at the end (otherwise it can be hard to find out which
failed).
multiple times is also a top-level goal, then the second and later
instantiations would never be created because there would be a
stable pointer to the first one that would keep it alive in the
WeakGoalMap.
* Some tracing code for debugging this kind of problem.
of the given derivation. Useful for getting a quick overview of how
something was built. E.g., to find out how the `baffle' program in
your user environment was built, you can do
$ nix-store -q --tree $(nix-store -qd $(which baffle))
Tree nesting depth is minimised (?) by topologically sorting paths
under the relation A < B iff A \in closure(B).
environment elements from one user environment to another, e.g.,
$ nix-env -i --from-profile /nix/var/nix/profiles/other-profile aterm
copies the `aterm' component installed in the `other-profile' to the
user's current profile.
user environment, e.g.,
$ nix-env -i /nix/store/z58v41v21xd3ywrqk1vmvdwlagjx7f10-aterm-2.3.1.drv
or
$ nix-env -i /nix/store/hsyj5pbn0d9iz7q0aj0fga7cpaadvp1l-aterm-2.3.1
This is useful because it allows Nix expressions to be bypassed
entirely. For instance, if only a nix-pull manifest is provided,
plus the top-level path of some component, it can be installed
without having to supply the Nix expression (e.g., for obfuscation,
or to be independent of Nix expression language changes or context
dependencies).
install derivations from a Nix expression specified on the command
line. This is particularly useful for disambiguation if there are
multiple derivations with the same name. For instance, in Nixpkgs,
to install the Firefox wrapper rather than the plain Firefox
component:
$ nix-env -f .../i686-linux.nix -i -E 'x: x.firefoxWrapper'
The Nix expressions should be functions to which the default Nix
expression (in this case, `i686-linux.nix') is passed, hence `x:
...'.
This might also be a nice way to deal with high-level (user-level)
variability, e.g.,
$ nix-env -f ./server.nix -i -E 'x: x {port = 8080; ssl = false;}'
to derivations in user environments. Nice for developers (since it
prevents build-time-only dependencies from being GC'ed, in
conjunction with `gc-keep-outputs'). Turned off by default.
* Set the references for the user environment manifest properly.
* Don't copy the manifest (this was accidental).
* Don't store derivation paths in the manifest (maybe this should be
made optional). This cleans up the semantics of nix-env, which were
weird.
* Hash on the output paths of activated components, not on derivation
paths. This is because we don't know the derivation path of already
installed components anymore, and it allows the installation of
components by store path (skipping Nix expressions entirely).
* Query options `--out-path' and `--drv-path' to show the output and
derivation paths of components, respectively (the latter replaces
the `--expr' query).
* Removed some dead code (successor stuff) from nix-push.
* Updated terminology in the tests (store expr -> drv path).
* Check that the deriver is set properly in the tests.
for finding build-time dependencies (possibly after a build). E.g.,
$ nix-store -qb aterm $(nix-store -qd $(which strc))
/nix/store/jw7c7s65n1gwhxpn35j9rgcci6ilzxym-aterm-2.3.1
* Arguments to nix-store can be files within store objects, e.g.,
/nix/store/jw7c...-aterm-2.3.1/bin/baffle.
* Idem for garbage collector roots.
This was necessary becase root finding must be done after
acquisition of the global GC lock.
This makes `nix-collect-garbage' obsolete; it is now just a wrapper
around `nix-store --gc'.
* Automatically remove stale GC roots (i.e., indirect GC roots that
point to non-existent paths).
get rid of GC roots. Nix-build places a symlink `result' in the
current directory. Previously, removing that symlink would not
remove the store path being linked to as a GC root. Now, the GC
root created by nix-build is actually a symlink in
`/nix/var/nix/gcroots/auto' to `result'. So if that symlink is
removed the GC root automatically becomes invalid (since it can no
longer be resolved). The root itself is not automatically removed -
the garbage collector should delete dangling roots.
immediately add the result as a permanent GC root. This is the only
way to prevent a race with the garbage collector. For instance, the
old style
ln -s $(nix-store -r $(nix-instantiate foo.nix)) \
/nix/var/nix/gcroots/result
has two time windows in which the garbage collector can interfere
(by GC'ing the derivation and the output, respectively). On the
other hand,
nix-store --add-root /nix/var/nix/gcroots/result -r \
$(nix-instantiate --add-root /nix/var/nix/gcroots/drv \
foo.nix)
is safe.
* nix-build: use `--add-root' to prevent GC races.
being created after the garbage collector has read the temproots
directory. This blocks the creation of new processes, but the
garbage collector could periodically release the GC lock to allow
them to run.
that they are deleted in an order that maintains the closure
invariant.
* Presence of a path in a temporary roots file does not imply that all
paths in its closure are also present, so add the closure.
roots to a per-process temporary file in /nix/var/nix/temproots
while holding a write lock on that file. The garbage collector
acquires read locks on all those files, thus blocking further
progress in other Nix processes, and reads the sets of temporary
roots.
though). In particular it's now much easier to register a GC root.
Just place a symlink to whatever store path it is that you want to
keep in /nix/var/nix/gcroots.
This simplifies garbage collection and `nix-store --query
--requisites' since we no longer need to treat derivations
specially.
* Better maintaining of the invariants, e.g., setReferences() can only
be called on a valid/substitutable path.
closure of the referers relation rather than the references
relation, i.e., the set of all paths that directly or indirectly
refer to the given path. Note that contrary to the references
closure this set is not fixed; it can change as paths are added to
or removed from the store.
promise :-) This allows derivations to specify on *what* output
paths of input derivations they are dependent. This helps to
prevent unnecessary downloads. For instance, a build might be
dependent on the `devel' and `lib' outputs of some library
component, but not the `docs' output.
graph. That is, `nix-store --query --references PATH' shows the set
of paths referenced by PATH, and `nix-store --query --referers PATH'
shows the set of paths referencing PATH.
`derivations.cc', etc.
* Store the SHA-256 content hash of store paths in the database after
they have been built/added. This is so that we can check whether
the store has been messed with (a la `rpm --verify').
* When registering path validity, verify that the closure property
holds.
representation of closures as ATerms in the Nix store. Instead, the
file system pointer graph is now stored in the Nix database. This
has many advantages:
- It greatly simplifies the implementation (we can drop the notion
of `successors', and so on).
- It makes registering roots for the garbage collector much easier.
Instead of specifying the closure expression as a root, you can
simply specify the store path that must be retained as a root.
This could not be done previously, since there was no way to find
the closure store expression containing a given store path.
- Better traceability: it is now possible to query what paths are
referenced by a path, and what paths refer to a path.
* Formalise the notion of fixed-output derivations, i.e., derivations
for which a cryptographic hash of the output is known in advance.
Changes to such derivations should not propagate upwards through the
dependency graph. Previously this was done by specifying the hash
component of the output path through the `id' attribute, but this is
insecure since you can lie about it (i.e., you can specify any hash
and then produce a completely different output). Now the
responsibility for checking the output is moved from the builder to
Nix itself.
A fixed-output derivation can be created by specifying the
`outputHash' and `outputHashAlgo' attributes, the latter taking
values `md5', `sha1', and `sha256', and the former specifying the
actual hash in hexadecimal or in base-32 (auto-detected by looking
at the length of the attribute value). MD5 is included for
compatibility but should be considered deprecated.
* Removed the `drvPath' pseudo-attribute in derivation results. It's
no longer necessary.
* Cleaned up the support for multiple output paths in derivation store
expressions. Each output now has a unique identifier (e.g., `out',
`devel', `docs'). Previously there was no way to tell output paths
apart at the store expression level.
* `nix-hash' now has a flag `--base32' to specify that the hash should
be printed in base-32 notation.
* `fetchurl' accepts parameters `sha256' and `sha1' in addition to
`md5'.
* `nix-prefetch-url' now prints out a SHA-1 hash in base-32. (TODO: a
flag to specify the hash.)
bits, then encode them in a radix-32 representation (using digits
and letters except e, o, u, and t). This produces store paths like
/nix/store/4i0zb0z7f88mwghjirkz702a71dcfivn-aterm-2.3.1. The nice
thing about this is that the hash part of the file name is still 32
characters, as before with MD5.
(Of course, shortening SHA-256 to 160 bits makes it no better than
SHA-160 in theory, but hopefully it's a bit more resistant to
attacks; it's certainly a lot slower.)
* Start cleaning up unique store path generation (they weren't always
unique; in particular the suffix ("-aterm-2.2", "-builder.sh") was
not part of the hash, therefore changes to the suffix would cause
multiple store objects with the same hash).
http://www.daemonology.net/bsdiff/bsdiff-4.2.tar.gz) into the source
tree. The license is a bit peculiar, but it does allow verbatim
copying, which is what we do here (i.e., so don't make any changes
to the sources).
- Drop the store expression. So now a substitute is just a
command-line invocation (a program name + arguments). If you
register a substitute you are responsible for registering the
expression that built it (if any) as a root of the garbage
collector.
- Drop the substitutes-rev DB table.
Instead we generate data bindings (build and match functions) for
the constructors specified in `constructors.def'. In particular
this removes the conversions between AFuns and strings, and Nix
expression evaluation now seems 3 to 4 times faster.
out the AST as an ATerm.
* Mode `--eval-only' to parse and evaluate the input, and print the
resulting normal form as an ATerm.
Neither of these modes require store/DB write permission.
The expression `with E1; E2' evaluates to E2 with all bindings in
the attribute set E1 substituted. E.g.,
with {x = 123;}; x
evaluates to 123. That is, the attribute set E1 is in scope in E2.
This is particularly useful when importing files containing lots
definitions. E.g., instead of
let {
inherit (import ./foo.nix) a b c d e f;
body = ... a ... f ...;
}
we can now say
with import ./foo.nix;
... a ... f ...
I.e., we don't have to say what variables should be brought into scope.
permission to the Nix store or database. E.g., `nix-env -qa' will
work, but `nix-env -qas' won't (the latter needs DB access). The
option `--readonly-mode' forces this mode; otherwise, it's only
activated when the database cannot be opened.
derivation, since NormalisationGoal would first run a
NormalisationGoal on the subderivation (a no-op, since in a
situation where we need fallback the successor is known), and then
runs a RealisationGoal on the normal form, which then cannot do a
fallback because it doesn't know the derivation expression for which
it is a normal form.
Tossed out the 2-phase normalisation/realisation in
NormalisationGoal and SubstitutionGoal since it's no longer needed -
a RealisationGoal will run a NormalisationGoal if necessary.
profile. Arguments are either generation number, or `old' to delete
all non-current generations. Typical use:
$ nix-env --delete-generations old
$ nix-collect-garbage
* istringstream -> string2Int.
Previously there was the problem that all files read by nix-env
etc. should be reachable and readable by the Nix user. So for
instance building a Nix expression in your home directory meant that
the home directory should have at least g+x or o+x permission so
that the Nix user could reach the Nix expression. Now we just
switch back to the original user just prior to reading sources and
the like. The places where this happens are somewhat arbitrary,
however. Any scope that has a live SwitchToOriginalUser object in
it is executed as the original user.
* Back out r1385. setreuid() sets the saved uid to the new
real/effective uid, which prevents us from switching back to the
original uid. setresuid() doesn't have this problem (although the
manpage has a bug: specifying -1 for the saved uid doesn't leave it
unchanged; an explicit value must be specified).
more common than the latter (which exists only on Linux and
FreeBSD). We don't really care about dropping the saved IDs since
there apparently is no way to quiry them in any case, so it can't
influence the build (unlike the effective IDs which are checked by
Perl for instance).
setuid installation, since the calling user may have a more fascist
umask (say, 0077), which would cause the store objects built by Nix
to be unreadable to anyone other than the Nix user.
unreachable paths that haven't been used for N hours. For instance,
`nix-collect-garbage --min-age 168' only deletes paths that haven't
been accessed in the last week.
This is useful for instance in the build farm where many derivations
can be shared between consecutive builds, and we wouldn't want a
garbage collect to throw them all away. We could of course register
them as roots, but then we'd to unregister them at some point, which
would be a pain to manage. The `--min-age' flag gives us a sort of
MRU caching scheme.
BUG: this really shouldn't be in gc.cc since that violates
mechanism/policy separation.
doesn't just print the set of paths that should be deleted. So
there is no more need to pipe the result into `nix-store --delete'
(which doesn't even exist anymore).
suboperations `--print-live', `--print-dead', and `--delete'. The
roots are not determined by nix-store; they are read from standard
input. This is to make it easy to customise what the roots are.
The collector now no longer fails when store expressions are missing
(which legally happens when using substitutes). It never tries to
fetch paths through substitutes.
TODO: acquire a global lock on the store while garbage collecting.
* Removed `nix-store --delete'.
set the real uid and gid to the effective uid and gid, the Nix
binaries can be installed as owned by the Nix user and group instead
of root, so no root involvement of any kind is necessary.
Linux and FreeBSD have these functions.
users.
If the configure flag `--enable-setuid' is used, the Nix programs
nix-env, nix-store, etc. are installed with the setuid bit turned on
so that they are executed as the user and group specified by
`--with-nix-user=USER' and `--with-nix-group=GROUP', respectively
(with defaults `nix' and `nix').
The setuid programs drop all special privileges if they are executed
by a user who is not a member of the Nix group.
The setuid feature is a quick hack to enable sharing of a Nix
installation between users who trust each other. It is not
generally secure, since any user in the Nix group can modify (by
building an appropriate derivation) any object in the store, and for
instance inject trojans into binaries used by other users.
The setuid programs are owned by root, not the Nix user. This is
because on Unix normal users cannot change the real uid, only the
effective uid. Many programs don't work properly when the real uid
differs from the effective uid. For instance, Perl will turn on
taint mode. However, the setuid programs drop all root privileges
immediately, changing all uids and gids to the Nix user and group.
* Builder output is written to standard error by default.
* The option `-B' is gone.
* The option `-Q' suppresses builder output.
The result of this is that most Nix invocations shouldn't need any
flags w.r.t. logging.
derivation disables scanning for dependencies. Use at your own
risk. This is a quick hack to speed up UML image generation (image
are very big, say 1 GB).
It would be better if the scanner were faster, and didn't read the
whole file into memory.
system types other than the current system. I.e., `nix-env -i'
won't install derivations for other system types, and `nix-env -q'
won't show them. The flag `--system-filter SYSTEM' can be used to
override the system type used for filtering (but not for
building!). The value `*' can be used not to filter anything.
Whenever Nix attempts to realise a derivation for which a closure is
already known, but this closure cannot be realised, fall back on
normalising the derivation.
The most common scenario in which this is useful is when we have
registered substitutes in order to perform binary distribution from,
say, a network repository. If the repository is down, the
realisation of the derivation will fail. When this option is
specified, Nix will build the derivation instead. Thus, binary
installation falls back on a source installation. This option is
not the default since it is generally not desirable for a transient
failure in obtaining the substitutes to lead to a full build from
source (with the related consumption of resources).
much as possible. (This is similar to GNU Make's `-k' flag.)
* Refactoring to implement this: previously we just bombed out when
a build failed, but now we have to clean up. In particular this
means that goals must be freed quickly --- they shouldn't hang
around until the worker exits. So the worker now maintains weak
pointers in order not to prevent garbage collection.
* Documented the `-k' and `-j' flags.
improve throughput.
* Don't build the `substitute-rev' table for now, since it caused
Theta(N^2) time and log file consumption when adding N substitutes.
Maybe we can do without it.
* A better substitute mechanism.
Instead of generating a store expression for each store path for
which we have a substitute, we can have a single store expression
that builds a generic program that is invoked to build the desired
store path, which is passed as an argument.
This means that operations like `nix-pull' only produce O(1) files
instead of O(N) files in the store when registering N substitutes.
(It consumes O(N) database storage, of course, but that's not a
performance problem).
* Added a test for the substitute mechanism.
* `nix-store --substitute' reads the substitutes from standard input,
instead of from the command line. This prevents us from running
into the kernel's limit on command line length.
* When a fast build wakes up a goal, try to start that goal in the
same iteration of the startBuild() loop of run(). Otherwise no job
might be started until the next job terminates.
in parallel. Hooks are more efficient: locks on output paths are
only acquired when the hook says that it is willing to accept a
build job. Hooks now work in two phases. First, they should first
tell Nix whether they are willing to accept a job. Nix guarantuees
that no two hooks will ever be in the first phase at the same time
(this simplifies the implementation of hooks, since they don't have
to perform locking (?)). Second, if they accept a job, they are
then responsible for building it (on the remote system), and copying
the result back. These can be run in parallel with other hooks and
locally executed jobs.
The implementation is a bit messy right now, though.
* The directory `distributed' shows a (hacky) example of a hook that
distributes build jobs over a set of machines listed in a
configuration file.
distributing a build action to another machine. In particular, the
paths in the input closures, the output paths, and successor mapping
for sub-derivations.
parallel as possible (similar to GNU Make's `-j' switch). This is
useful on SMP systems, but it is especially useful for doing builds
on multiple machines. The idea is that a large derivation is
initiated on one master machine, which then distributes
sub-derivations to any number of slave machines. This should not
happen synchronously or in lock-step, so the master must be capable
of dealing with multiple parallel build jobs. We now have the
infrastructure to support this.
TODO: substitutes are currently broken.
print a nice backtrace of the stack, rather than vomiting a gigantic
(and useless) aterm on the screen. Example:
error: while evaluating file `.../pkgs/system/test.nix':
while evaluating attribute `subversion' at `.../pkgs/system/all-packages-generic.nix', line 533:
while evaluating function at `.../pkgs/applications/version-management/subversion/default.nix', line 1:
assertion failed at `.../pkgs/applications/version-management/subversion/default.nix', line 13
Since the Nix expression language is lazy, the trace may be
misleading. The purpose is to provide a hint as to the location of
the problem.
instead of `derivation' triggered a huge slowdown in the Nix
expression evaluator. Total execution time of `nix-env -qa' went up
by a factor of 60 or so.
This scalability problem was caused by expressions such as
(x: y: ... x ...) a b
where `a' is a large term (say, the one in
`all-packages-generic.nix'). Then the first beta-reduction would
produce
(y: ... a ...) b
by substituting `a' for `x'. The second beta-reduction would then
substitute `b' for `y' into the body `... a ...', which is a large
term due to `a', and thus causes a large traversal to be performed
by substitute() in the second reduction. This is however entirely
redundant, since `a' cannot contain free variables (since we never
substitute below a weak head normal form).
The solution is to wrap substituted terms into a `Closed'
constructor, i.e.,
subst(subs, Var(x)) = Closed(e) iff subs[x] = e
have substitution not descent into closed terms,
subst(subs, Closed(x)) = Closed(x)
and otherwise ignore them for evaluation,
eval(Closed(x)) = eval(x).
* Fix a typo that caused incorrect substitutions to be performed in
simple lambdas, e.g., `(x: x: x) a' would reduce to `(x: a)'.
