programs, so if a builder uses TMPDIR, then it will fail when
executed through nix-setuid-helper. In fact Glibc clears a whole
bunch of variables (see sysdeps/generic/unsecvars.h in the Glibc
sources), but only TMPDIR should matter in practice. As a
workaround, we reinitialise TMPDIR from NIX_BUILD_TOP.
important to get garbage collection to work if there is any
inconsistency in the database (because the referrer table is used to
determine whether it is safe to delete a path).
* `nix-store --verify': show some progress.
* nix-unpack-closure: extract the top-level paths from the closure and
print them on stdout. This allows them to be installed, e.g.,
"nix-env -i $(nix-unpack-closure)". (NIX-64)
<derivation outPath=... drvPath=...> attrs </derivation>. Only emit
the attributes of any specific derivation only. This prevents
exponententially large XML output due to the absense of sharing.
from a source directory. All files for which a predicate function
returns true are copied to the store. Typical example is to leave
out the .svn directory:
stdenv.mkDerivation {
...
src = builtins.filterSource
(path: baseNameOf (toString path) != ".svn")
./source-dir;
# as opposed to
# src = ./source-dir;
}
This is important because the .svn directory influences the hash in
a rather unpredictable and variable way.
single derivation specified by the argument. This is useful when we
want to have a profile for a single derivation, such as a server
configuration. Then we can just say (e.g.)
$ nix-env -p /.../server-profile -f server.nix --set -A server
We can't do queries or upgrades on such a profile, but we can do
rollbacks. The advantage over -i is that we don't have to worry
about other packages having been installed in the profile
previously; --set gets rid of them.
matters when running as root, since then we don't use the setuid
helper (which already used lchown()).
* Also check for an obscure security problem on platforms that don't
have lchown. Then we can't change the ownership of symlinks, which
doesn't matter *except* when the containing directory is writable by
the owner (which is the case with the top-level Nix store directory).
* Throw more exceptions as BuildErrors instead of Errors. This
matters when --keep-going is turned on. (A BuildError is caught
and terminates the goal in question, an Error terminates the
program.)
seconds without producing output on stdout or stderr (NIX-65). This
timeout can be specified using the `--max-silent-time' option or the
`build-max-silent-time' configuration setting. The default is
infinity (0).
* Fix a tricky race condition: if we kill the build user before the
child has done its setuid() to the build user uid, then it won't be
killed, and we'll potentially lock up in pid.wait(). So also send a
conventional kill to the child.
that have to be done as root: running builders under different uids,
changing ownership of build results, and deleting paths in the store
with the wrong ownership).
`nix-store --delete'. But unprivileged users are not allowed to
ignore liveness.
* `nix-store --delete --ignore-liveness': ignore the runtime roots as
well.
process, so forward the operation.
* Spam the user about GC misconfigurations (NIX-71).
* findRoots: skip all roots that are unreadable - the warnings with
which we spam the user should be enough.
processes can register indirect roots. Of course, there is still
the problem that the garbage collector can only read the targets of
the indirect roots when it's running as root...
* SIGIO -> SIGPOLL (POSIX calls it that).
* Use sigaction instead of signal to register the SIGPOLL handler.
Sigaction is better defined, and a handler registered with signal
appears not to interrupt fcntl(..., F_SETLKW, ...), which is bad.
via the Unix domain socket in /nix/var/nix/daemon.socket. The
server forks a worker process per connection.
* readString(): use the heap, not the stack.
* Some protocol fixes.
* Added `build-users-group', the group under which builds are to be
performed.
* Check that /nix/store has 1775 permission and is owner by the
build-users-group.
The problem is that when we kill the client while the worker is
building, and the builder is not writing anything to stderr, then
the worker never notice that the socket is closed on the other side,
so it just continues indefinitely. The solution is to catch SIGIO,
which is sent when the far side of the socket closes, and simulate
an normal interruption. Of course, SIGIO is also sent every time
the client sends data over the socket, so we only enable the signal
handler when we're not expecting any data...
mode. Presumably nix-worker would be setuid to the Nix store user.
The worker performs all operations on the Nix store and database, so
the caller can be completely unprivileged.
This is already much more secure than the old setuid scheme, since
the worker doesn't need to do Nix expression evaluation and so on.
Most importantly, this means that it doesn't need to access any user
files, with all resulting security risks; it only performs pure
store operations.
Once this works, it is easy to move to a daemon model that forks off
a worker for connections established through a Unix domain socket.
That would be even more secure.
* Some refactoring: put the NAR archive integer/string serialisation
code in a separate file so it can be reused by the worker protocol
implementation.
containing functions that operate on the Nix store. One
implementation is LocalStore, which operates on the Nix store
directly. The next step, to enable secure multi-user Nix, is to
create a different implementation RemoteStore that talks to a
privileged daemon process that uses LocalStore to perform the actual
operations.
Rather, setuid support is now always compiled in (at least on
platforms that have the setresuid system call, e.g., Linux and
FreeBSD), but it must enabled by chowning/chmodding the Nix
binaries.
gives a huge speedup in operations that read or write from standard
input/output. (So libstdc++'s I/O isn't that bad, you just have to
call std::ios::sync_with_stdio(false).) For instance, `nix-store
--register-substitutes' went from 1.4 seconds to 0.1 seconds on a
certain input. Another victory for Valgrind.
graph to be passed to a builder. This attribute should be a list of
pairs [name1 path1 name2 path2 ...]. The references graph of each
`pathN' will be stored in a text file `nameN' in the temporary build
directory. The text files have the format used by `nix-store
--register-validity'. However, the deriver fields are left empty.
`exportReferencesGraph' is useful for builders that want to do
something with the closure of a store path. Examples: the builders
that make initrds and ISO images for NixOS.
`exportReferencesGraph' is entirely pure. It's necessary because
otherwise the only way for a builder to get this information would
be to call `nix-store' directly, which is not allowed (though
unfortunately possible).
available. For instance,
$ nix-store -l $(which svn) | less
lets you read the build log of the Subversion instance in your
profile.
* `nix-store -qb': if applied to a non-derivation, take the deriver.
check that the references of the output of a derivation are in the
specified set. For instance,
allowedReferences = [];
specifies that the output cannot have any references. (This is
useful, for instance, for the generation of bootstrap binaries for
stdenv-linux, which must not have any references for purity). It
could also be used to guard against undesired runtime dependencies,
e.g.,
{gcc, dynlib}: derivation {
...
allowedReferences = [dynlib];
}
says that the output can refer to the path of `dynlib' but not
`gcc'. A `forbiddedReferences' attribute would be more useful for
this, though.
concatenation and string coercion. This was a big mess (see
e.g. NIX-67). Contexts are now folded into strings, so that they
don't cause evaluation errors when they're not expected. The
semantics of paths has been clarified (see nixexpr-ast.def).
toString() and coerceToString() have been merged.
Semantic change: paths are now copied to the store when they're in a
concatenation (and in most other situations - that's the
formalisation of the meaning of a path). So
"foo " + ./bla
evaluates to "foo /nix/store/hash...-bla", not "foo
/path/to/current-dir/bla". This prevents accidental impurities, and
is more consistent with the treatment of derivation outputs, e.g.,
`"foo " + bla' where `bla' is a derivation. (Here `bla' would be
replaced by the output path of `bla'.)
side should be a path, I guess.
* Handle paths that are in the store but not direct children of the
store directory.
* Ugh, hack to prevent double context wrapping.
attribute existence and to return an attribute from an attribute
set, respectively. Example: `hasAttr "foo" {foo = 1;}'. They
differ from the `?' and `.' operators in that the attribute name is
an arbitrary expression. (NIX-61)
Nix-env failed to call addPermRoot(), which is necessary to safely
add a new root. So if nix-env started after and finished before the
garbage collector, the user environment (plus all other new stuff)
it built might be garbage collected, leading to a dangling symlink
chain in ~/.nix-profile...
* Be more explicit if we block on the GC lock ("waiting for the big
garbage collector lock...").
* Don't loop trying to create a new generation. It's not necessary
anymore since profiles are locked nowadays.
and returns its path. This can be used to (for instance) write
builders inside a Nix expression, e.g.,
stdenv.mkDerivation {
builder = "
source $stdenv/setup
...
";
...
}
derivation attributes to flatten them into strings. This is
possible since string can nowadays be wrapped in contexts that
describe the derivations/sources referenced by the evaluation of the
string.
all the primops. This allows Nix expressions to test for new
primops and take appropriate action if they're not available. For
instance, rather than calling a primop `foo' directly, they could
say `if builtins ? foo then builtins.foo ... else ...'.
writable. File permissions on Cygwin are rather complex, and in this
case this check introduced a problem with build jobs invoke from
outside of Cygwin (MSYS). It seemed almost impossible to fix the
permissions of the directory, so for now this safety check is disabled
on Cygwin.
https://svn.cs.uu.nl:12443/repos/trace/buildfarm-control/trunk/ext/nix/,
with some modifications. This allows `nix-env -qa' to show the
attribute path that can be used to unambiguously install a package
using `nix-env -i -A'. Example:
$ nix-env -f top-level/all-packages.nix -qaA subversion xorg-server
subversionWithJava subversion-1.2.3
subversion subversion-1.3.2
subversion14 subversion-1.4.0pre-rc1
xorg.xorgserver xorg-server-1.1.0
e.g.,
$ nix-env -i -A subversion xorg.xorgserver
The main advantage over using symbolic names is that using attribute
names is unambiguous and much, much faster.
argument has a valid value, i.e., is in a certain domain. E.g.,
{ foo : [true false]
, bar : ["a" "b" "c"]
}: ...
This previously could be done using assertions, but domain checks
will allow the buildfarm to automatically extract the configuration
space from functions.
"--with-freetype2-library=" + freetype + "/lib"
can now be written as
"--with-freetype2-library=${freetype}/lib"
An arbitrary expression can be enclosed within ${...}, not just
identifiers.
* Escaping in string literals: \n, \r, \t interpreted as in C, any
other character following \ is interpreted as-is.
* Newlines are now allowed in string literals.
packages (provided that they have a `meta.description' attribute).
E.g.,
$ ./src/nix-env/nix-env -qa --description gcc
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for sparc-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for mips-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for arm-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x
to be queried, e.g., `nix-env -qa firefox'. This does require the
argument '*' to be passed if one wants information about all
derivations, so the old `nix-env -qa' now is `nix-env -qa "*"'.
instantiation, e.g. "nix-env -i" and "nix-env -qas" (but not
"nix-env -qa"). It turns out that many redundant calls to
addToStore(path) were made, which reads and hashes the entire path.
For instance, the bash bootstrap binary in Nixpkgs would be read and
hashed many times. As a result nix-env would spend around 92% of
its time in the function sha256_block (according to callgrind).
Some simple memoization fixes this.
expressions that cause an assertion failure (like `assert system ==
"i686-linux"'). This allows all-packages.nix in Nixpkgs to be used
on all platforms, even if some Nix expressions don't work on all
platforms.
Not sure if this is a good idea; it's a bit hacky. In particular,
due to laziness some derivations might appear in `nix-env -qa' but
disappear in `nix-env -qas' or `nix-env -i'.
Commit 5000!
with the same name *and* version number, and pick the first one
(this means that the order in which channels appear in
~/.nix-channels matters). E.g.:
$ nix-env ii aterm
warning: there are multiple derivations named `aterm-2.4.2'; using the first one
installing `aterm-2.4.2'
the disk is full (because to delete something from the Nix store, we
need a Berkeley DB transaction, which takes up disk space). Under
normal operation, we make sure that there exists a file
/nix/var/nix/db/reserved of 1 MB. When running the garbage
collector, we delete that file before we open the Berkeley DB
environment.
implementations of MD5, SHA-1 and SHA-256. The main benefit is that
we get assembler-optimised implementations of MD5 and SHA-1 (though
not SHA-256 (at least on x86), unfortunately). OpenSSL's SHA-1
implementation on Intel is twice as fast as ours.
derivation(s) we're interested, e.g.,
$ nix-instantiate ./all-packages.nix --attr xlibs.libX11
List elements can also be selected:
$ nix-instantiate ./build-for-release.nix --attr 0.subversion
This allows a non-ambiguous specification of a derivation. Of
course, this should also be added to nix-env and nix-build.
creates a new process group but also a new session. New sessions
have no controlling tty, so child processes like ssh cannot open
/dev/tty (which is bad).
intended). This ensures that any ssh child processes to remote
machines are also killed, and thus the Nix process on the remote
machine also exits. Without this, the remote Nix process will
continue until it exists or until its stdout buffer gets full and it
locks up. (Partially fixes NIX-35.)
deletes a path even if it is reachable from a root. However, it
won't delete a path that still has referrers (since that would
violate store invariants).
Don't try this at home. It's a useful hack for recovering from
certain situations in a somewhat clean way (e.g., holes in closures
due to disk corruption).
nix-store query options `--referer' and `--referer-closure' have
been changed to `--referrer' and `--referrer-closure' (but the old
ones are still accepted for compatibility).
mapping. The referer table is replaced by a referrer table (note
spelling fix) that stores each referrer separately. That is,
instead of having
referer[P] = {Q_1, Q_2, Q_3, ...}
we store
referer[(P, Q_1)] = ""
referer[(P, Q_2)] = ""
referer[(P, Q_3)] = ""
...
To find the referrers of P, we enumerate over the keys with a value
lexicographically greater than P. This requires the referrer table
to be stored as a B-Tree rather than a hash table.
(The tuples (P, Q) are stored as P + null-byte + Q.)
Old Nix databases are upgraded automatically to the new schema.
Nix is properly shut down when it receives those signals. In
particular this ensures that killing the garbage collector doesn't
cause a subsequent database recovery.
builder. Instead, require that the Nix store has sticky permission
(S_ISVTX); everyone can created files in the Nix store, but they
cannot delete, rename or modify files created by others.
root (or setuid root), then builds will be performed under one of
the users listed in the `build-users' configuration variables. This
is to make it impossible to influence build results externally,
allowing locally built derivations to be shared safely between
users (see ASE-2005 paper).
To do: only one builder should be active per build user.
versions to available versions, or vice versa.
For example, the following compares installed versions to available
versions:
$ nix-env -qc
autoconf-2.59 = 2.59
automake-1.9.4 < 1.9.6
f-spot-0.0.10 - ?
firefox-1.0.4 < 1.0.7
...
I.e., there are newer versions available (in the current default Nix
expression) for Automake and Firefox, but not for Autoconf, and
F-Spot is missing altogether.
Conversely, the available versions can be compared to the installed
versions:
$ nix-env -qac
autoconf-2.59 = 2.59
automake-1.9.6 > 1.9.4
bash-3.0 - ?
firefox-1.0.7 > 1.0.4
...
Note that bash is available but no version of it is installed.
If multiple versions are available for comparison, then the highest
is used. E.g., if Subversion 1.2.0 is installed, and Subversion
1.1.4 and 1.2.3 are available, then `nix-env -qc' will print `<
1.2.3', not `> 1.1.4'.
If higher versions are available, the version column is printed in
red (using ANSI escape codes).
dependencyClosure { ... searchPath = [ ../foo ../bar ]; ... }
* Primop `dirOf' to return the directory part of a path (e.g., dirOf
/a/b/c == /a/b).
* Primop `relativise' (according to Webster that's a real word!) that
given paths A and B returns a string representing path B relative
path to A; e.g., relativise /a/b/c a/b/x/y => "../x/y".
determination (e.g., finding the header files dependencies of a C
file) in Nix low-level builds automatically.
For instance, in the function `compileC' in make/lib/default.nix, we
find the header file dependencies of C file `main' as follows:
localIncludes =
dependencyClosure {
scanner = file:
import (findIncludes {
inherit file;
});
startSet = [main];
};
The function works by "growing" the set of dependencies, starting
with the set `startSet', and calling the function `scanner' for each
file to get its dependencies (which should yield a list of strings
representing relative paths). For instance, when `scanner' is
called on a file `foo.c' that includes the line
#include "../bar/fnord.h"
then `scanner' should yield ["../bar/fnord.h"]. This list of
dependencies is absolutised relative to the including file and added
to the set of dependencies. The process continues until no more
dependencies are found (hence its a closure).
`dependencyClosure' yields a list that contains in alternation a
dependency, and its relative path to the directory of the start
file, e.g.,
[ /bla/bla/foo.c
"foo.c"
/bla/bar/fnord.h
"../bar/fnord.h"
]
These relative paths are necessary for the builder that compiles
foo.c to reconstruct the relative directory structure expected by
foo.c.
The advantage of `dependencyClosure' over the old approach (using
the impure `__currentTime') is that it's completely pure, and more
efficient because it only rescans for dependencies (i.e., by
building the derivations yielded by `scanner') if sources have
actually changed. The old approach rescanned every time.
(closed(closed(closed(...)))) since this reduces performance by
producing bigger terms and killing caching (which incidentally also
prevents useful infinite recursion detection).
with default values automatically. I.e., e -> e {}.
This feature makes convenience expressions such as
pkgs/system/i686-linux.nix in Nixpkgs obsolete, since we can just do
$ nix-instantiate ./pkgs/system/all-packages.nix
since all-packages.nix takes a single argument (system) that has a
default value (__thisSystem).
`removeAttrs attrs ["x", "y"]' returns the set `attrs' with the
attributes named `x' and `y' removed. It is not an error for the
named attributes to be missing from the input set.
* Make the `derivation' primitive much more lazy. The expression
`derivation attrs' now evaluates to (essentially)
attrs // {
type = "derivation";
outPath = derivation! attrs;
drvPath = derivation! attrs;
}
where `derivation!' is a primop that does the actual derivation
instantiation (i.e., it does what `derivation' used to do). The
advantage is that it allows commands such as `nix-env -qa' and
`nix-env -i' to be much faster since they no longer need to
instantiate all derivations, just the `name' attribute. (However,
`nix-env' doesn't yet take advantage of this since it still always
evaluates the `outPath' and `drvPath' attributes).
Also, this allows derivations to cyclically reference each other,
for example,
webServer = derivation {
...
hostName = "svn.cs.uu.nl";
services = [svnService];
};
svnService = derivation {
...
hostName = webServer.hostName;
};
Previously, this would yield a black hole (infinite recursion).
derivations. This is mostly to simplify the implementation of
nix-prefetch-{url, svn}, which now work properly in setuid
installations.
* Enforce valid store names in `nix-store --add / --add-fixed'.
continue building when one fails unless `--keep-going' is
specified.
* When `--keep-going' is specified, print out the set of failing
derivations at the end (otherwise it can be hard to find out which
failed).
multiple times is also a top-level goal, then the second and later
instantiations would never be created because there would be a
stable pointer to the first one that would keep it alive in the
WeakGoalMap.
* Some tracing code for debugging this kind of problem.
of the given derivation. Useful for getting a quick overview of how
something was built. E.g., to find out how the `baffle' program in
your user environment was built, you can do
$ nix-store -q --tree $(nix-store -qd $(which baffle))
Tree nesting depth is minimised (?) by topologically sorting paths
under the relation A < B iff A \in closure(B).
