fixed-output derivations or substitutions try to build the same
store path at the same time. Locking generally catches this, but
not between multiple goals in the same process. This happened
especially often (actually, only) in the build farm with fetchurl
downloads of the same file being executed on multiple machines and
then copied back to the main machine where they would clobber each
other (NIXBF-13).
Solution: if a goal notices that the output path is already locked,
then go to sleep until another goal finishes (hopefully the one
locking the path) and try again.
need any info on substitutable paths, we just call the substituters
(such as download-using-manifests.pl) directly. This means that
it's no longer necessary for nix-pull to register substitutes or for
nix-channel to clear them, which makes those operations much faster
(NIX-95). Also, we don't have to worry about keeping nix-pull
manifests (in /nix/var/nix/manifests) and the database in sync with
each other.
The downside is that there is some overhead in calling an external
program to get the substitutes info. For instance, "nix-env -qas"
takes a bit longer.
Abolishing the substitutes table also makes the logic in
local-store.cc simpler, as we don't need to store info for invalid
paths. On the downside, you cannot do things like "nix-store -qR"
on a substitutable but invalid path (but nobody did that anyway).
* Never catch interrupts (the Interrupted exception).
;-)
* Channels: fix channels that are plain lists of derivations (like
strategoxt-unstable) instead of functions (like nixpkgs-unstable).
This fixes the error message "error: the left-hand side of the
function call is neither a function nor a primop (built-in
operation) but a list".
get the basename of the channel URL (e.g., nixpkgs-unstable). The
top-level Nix expression of the channel is now an attribute set, the
attributes of which are the individual channels (e.g.,
{nixpkgs_unstable = ...; strategoxt_unstable = ...}). This makes
attribute paths ("nix-env -qaA" and "nix-env -iA") more sensible,
e.g., "nix-env -iA nixpkgs_unstable.subversion".
by priority and version install. That is, if there are multiple
packages with the same name, then pick the package with the highest
priority, and only use the version if there are multiple packages
with the same priority.
This makes it possible to mark specific versions/variant in Nixpkgs
more or less desirable than others. A typical example would be a
beta version of some package (e.g., "gcc-4.2.0rc1") which should not
be installed even though it is the highest version, except when it
is explicitly selected (e.g., "nix-env -i gcc-4.2.0rc1").
* Idem for nix-env -u, only the semantics are a bit trickier since we
also need to take into account the priority of the currently
installed package (we never upgrade to a lower priority, unless
--always is given).
a user environment by an install or upgrade action. This is
particularly useful if you have a version installed that you don't
want to upgrade (e.g., because the newer versions are broken).
Example:
$ nix-env -u zapping --dry-run
(dry run; not doing anything)
upgrading `zapping-0.9.6' to `zapping-0.10cvs6'
$ nix-env --set-flag keep true zapping
$ nix-env -u zapping --dry-run
(dry run; not doing anything)
However, "-e" will still uninstall the package. (Maybe we should
require the keep flag to be explicitly set to false before it can be
uninstalled.)
to show only those derivations whose output is already in the Nix
store or that can be substituted (i.e., downloaded from somewhere).
In other words, it shows the packages that can be installed “quickly”,
i.e., don’t need to be built from source.
evaluator. This was important because the NixOS expressions started
to hit 2 MB default stack size on Linux.
GCC is really dumb about stack space: it just adds up all the local
variables and temporaries of every scope into one huge stack frame.
This is really bad for deeply recursive functions. For instance,
every `throw Error(format("error message"))' causes a format object
of a few hundred bytes to be allocated on the stack. As a result,
every recursive call to evalExpr2() consumed 4680 bytes. By
splitting evalExpr2() and by moving the exception-throwing code out
of the main functions, evalExpr2() now only consumes 40 bytes.
Similar for evalExpr().
which paths specified on the command line are invalid (i.e., don't
barf when encountering an invalid path, just print it). This is
useful for build-remote.pl to figure out which paths need to be
copied to a remote machine. (Currently we use rsync, but that's
rather inefficient.)
--export' into the Nix store, and optionally check the cryptographic
signatures against /nix/etc/nix/signing-key.pub. (TODO: verify
against a set of public keys.)
path. This is like `nix-store --dump', only it also dumps the
meta-information of the store path (references, deriver). Will add
a `--sign' flag later to add a cryptographic signature, which we
will use for exchanging store paths between build farm machines in a
secure manner.
attribute) about installed packages in user environments. Thus, an
operation like `nix-env -q --description' shows useful information
not only on available packages but also on installed packages.
* nix-env now passes the entire manifest as an argument to the Nix
expression of the user environment builder (not just a list of
paths), so that in particular the user environment builder has
access to the meta attributes.
* New operation `--set-flag' in nix-env to change meta info of
installed packages. This will be useful to pass per-package
policies to the user environment builder (e.g., how to resolve
collision or whether to disable a package (NIX-80)) or upgrade
policies in nix-env (e.g., that a package should be "masked", that
is, left untouched by upgrade actions). Example:
$ nix-env --set-flag enabled false ghc-6.4
computing the store path (NIX-77). This is an important security
property in multi-user Nix stores.
Note that this changes the store paths of derivations (since the
derivation aterms are added using addTextToStore), but not most
outputs (unless they use builtins.toFile).
* `sub' to subtract two numbers.
* `stringLength' to get the length of a string.
* `substring' to get a substring of a string. These should be enough
to allow most string operations to be expressed.
programs, so if a builder uses TMPDIR, then it will fail when
executed through nix-setuid-helper. In fact Glibc clears a whole
bunch of variables (see sysdeps/generic/unsecvars.h in the Glibc
sources), but only TMPDIR should matter in practice. As a
workaround, we reinitialise TMPDIR from NIX_BUILD_TOP.
important to get garbage collection to work if there is any
inconsistency in the database (because the referrer table is used to
determine whether it is safe to delete a path).
* `nix-store --verify': show some progress.
* nix-unpack-closure: extract the top-level paths from the closure and
print them on stdout. This allows them to be installed, e.g.,
"nix-env -i $(nix-unpack-closure)". (NIX-64)
<derivation outPath=... drvPath=...> attrs </derivation>. Only emit
the attributes of any specific derivation only. This prevents
exponententially large XML output due to the absense of sharing.
from a source directory. All files for which a predicate function
returns true are copied to the store. Typical example is to leave
out the .svn directory:
stdenv.mkDerivation {
...
src = builtins.filterSource
(path: baseNameOf (toString path) != ".svn")
./source-dir;
# as opposed to
# src = ./source-dir;
}
This is important because the .svn directory influences the hash in
a rather unpredictable and variable way.
single derivation specified by the argument. This is useful when we
want to have a profile for a single derivation, such as a server
configuration. Then we can just say (e.g.)
$ nix-env -p /.../server-profile -f server.nix --set -A server
We can't do queries or upgrades on such a profile, but we can do
rollbacks. The advantage over -i is that we don't have to worry
about other packages having been installed in the profile
previously; --set gets rid of them.
matters when running as root, since then we don't use the setuid
helper (which already used lchown()).
* Also check for an obscure security problem on platforms that don't
have lchown. Then we can't change the ownership of symlinks, which
doesn't matter *except* when the containing directory is writable by
the owner (which is the case with the top-level Nix store directory).
* Throw more exceptions as BuildErrors instead of Errors. This
matters when --keep-going is turned on. (A BuildError is caught
and terminates the goal in question, an Error terminates the
program.)
seconds without producing output on stdout or stderr (NIX-65). This
timeout can be specified using the `--max-silent-time' option or the
`build-max-silent-time' configuration setting. The default is
infinity (0).
* Fix a tricky race condition: if we kill the build user before the
child has done its setuid() to the build user uid, then it won't be
killed, and we'll potentially lock up in pid.wait(). So also send a
conventional kill to the child.
that have to be done as root: running builders under different uids,
changing ownership of build results, and deleting paths in the store
with the wrong ownership).
`nix-store --delete'. But unprivileged users are not allowed to
ignore liveness.
* `nix-store --delete --ignore-liveness': ignore the runtime roots as
well.
process, so forward the operation.
* Spam the user about GC misconfigurations (NIX-71).
* findRoots: skip all roots that are unreadable - the warnings with
which we spam the user should be enough.
processes can register indirect roots. Of course, there is still
the problem that the garbage collector can only read the targets of
the indirect roots when it's running as root...
* SIGIO -> SIGPOLL (POSIX calls it that).
* Use sigaction instead of signal to register the SIGPOLL handler.
Sigaction is better defined, and a handler registered with signal
appears not to interrupt fcntl(..., F_SETLKW, ...), which is bad.
via the Unix domain socket in /nix/var/nix/daemon.socket. The
server forks a worker process per connection.
* readString(): use the heap, not the stack.
* Some protocol fixes.
* Added `build-users-group', the group under which builds are to be
performed.
* Check that /nix/store has 1775 permission and is owner by the
build-users-group.
The problem is that when we kill the client while the worker is
building, and the builder is not writing anything to stderr, then
the worker never notice that the socket is closed on the other side,
so it just continues indefinitely. The solution is to catch SIGIO,
which is sent when the far side of the socket closes, and simulate
an normal interruption. Of course, SIGIO is also sent every time
the client sends data over the socket, so we only enable the signal
handler when we're not expecting any data...
mode. Presumably nix-worker would be setuid to the Nix store user.
The worker performs all operations on the Nix store and database, so
the caller can be completely unprivileged.
This is already much more secure than the old setuid scheme, since
the worker doesn't need to do Nix expression evaluation and so on.
Most importantly, this means that it doesn't need to access any user
files, with all resulting security risks; it only performs pure
store operations.
Once this works, it is easy to move to a daemon model that forks off
a worker for connections established through a Unix domain socket.
That would be even more secure.
* Some refactoring: put the NAR archive integer/string serialisation
code in a separate file so it can be reused by the worker protocol
implementation.
containing functions that operate on the Nix store. One
implementation is LocalStore, which operates on the Nix store
directly. The next step, to enable secure multi-user Nix, is to
create a different implementation RemoteStore that talks to a
privileged daemon process that uses LocalStore to perform the actual
operations.
Rather, setuid support is now always compiled in (at least on
platforms that have the setresuid system call, e.g., Linux and
FreeBSD), but it must enabled by chowning/chmodding the Nix
binaries.
gives a huge speedup in operations that read or write from standard
input/output. (So libstdc++'s I/O isn't that bad, you just have to
call std::ios::sync_with_stdio(false).) For instance, `nix-store
--register-substitutes' went from 1.4 seconds to 0.1 seconds on a
certain input. Another victory for Valgrind.
graph to be passed to a builder. This attribute should be a list of
pairs [name1 path1 name2 path2 ...]. The references graph of each
`pathN' will be stored in a text file `nameN' in the temporary build
directory. The text files have the format used by `nix-store
--register-validity'. However, the deriver fields are left empty.
`exportReferencesGraph' is useful for builders that want to do
something with the closure of a store path. Examples: the builders
that make initrds and ISO images for NixOS.
`exportReferencesGraph' is entirely pure. It's necessary because
otherwise the only way for a builder to get this information would
be to call `nix-store' directly, which is not allowed (though
unfortunately possible).
available. For instance,
$ nix-store -l $(which svn) | less
lets you read the build log of the Subversion instance in your
profile.
* `nix-store -qb': if applied to a non-derivation, take the deriver.
check that the references of the output of a derivation are in the
specified set. For instance,
allowedReferences = [];
specifies that the output cannot have any references. (This is
useful, for instance, for the generation of bootstrap binaries for
stdenv-linux, which must not have any references for purity). It
could also be used to guard against undesired runtime dependencies,
e.g.,
{gcc, dynlib}: derivation {
...
allowedReferences = [dynlib];
}
says that the output can refer to the path of `dynlib' but not
`gcc'. A `forbiddedReferences' attribute would be more useful for
this, though.
concatenation and string coercion. This was a big mess (see
e.g. NIX-67). Contexts are now folded into strings, so that they
don't cause evaluation errors when they're not expected. The
semantics of paths has been clarified (see nixexpr-ast.def).
toString() and coerceToString() have been merged.
Semantic change: paths are now copied to the store when they're in a
concatenation (and in most other situations - that's the
formalisation of the meaning of a path). So
"foo " + ./bla
evaluates to "foo /nix/store/hash...-bla", not "foo
/path/to/current-dir/bla". This prevents accidental impurities, and
is more consistent with the treatment of derivation outputs, e.g.,
`"foo " + bla' where `bla' is a derivation. (Here `bla' would be
replaced by the output path of `bla'.)
side should be a path, I guess.
* Handle paths that are in the store but not direct children of the
store directory.
* Ugh, hack to prevent double context wrapping.
attribute existence and to return an attribute from an attribute
set, respectively. Example: `hasAttr "foo" {foo = 1;}'. They
differ from the `?' and `.' operators in that the attribute name is
an arbitrary expression. (NIX-61)
Nix-env failed to call addPermRoot(), which is necessary to safely
add a new root. So if nix-env started after and finished before the
garbage collector, the user environment (plus all other new stuff)
it built might be garbage collected, leading to a dangling symlink
chain in ~/.nix-profile...
* Be more explicit if we block on the GC lock ("waiting for the big
garbage collector lock...").
* Don't loop trying to create a new generation. It's not necessary
anymore since profiles are locked nowadays.
and returns its path. This can be used to (for instance) write
builders inside a Nix expression, e.g.,
stdenv.mkDerivation {
builder = "
source $stdenv/setup
...
";
...
}
derivation attributes to flatten them into strings. This is
possible since string can nowadays be wrapped in contexts that
describe the derivations/sources referenced by the evaluation of the
string.
all the primops. This allows Nix expressions to test for new
primops and take appropriate action if they're not available. For
instance, rather than calling a primop `foo' directly, they could
say `if builtins ? foo then builtins.foo ... else ...'.
writable. File permissions on Cygwin are rather complex, and in this
case this check introduced a problem with build jobs invoke from
outside of Cygwin (MSYS). It seemed almost impossible to fix the
permissions of the directory, so for now this safety check is disabled
on Cygwin.
https://svn.cs.uu.nl:12443/repos/trace/buildfarm-control/trunk/ext/nix/,
with some modifications. This allows `nix-env -qa' to show the
attribute path that can be used to unambiguously install a package
using `nix-env -i -A'. Example:
$ nix-env -f top-level/all-packages.nix -qaA subversion xorg-server
subversionWithJava subversion-1.2.3
subversion subversion-1.3.2
subversion14 subversion-1.4.0pre-rc1
xorg.xorgserver xorg-server-1.1.0
e.g.,
$ nix-env -i -A subversion xorg.xorgserver
The main advantage over using symbolic names is that using attribute
names is unambiguous and much, much faster.
argument has a valid value, i.e., is in a certain domain. E.g.,
{ foo : [true false]
, bar : ["a" "b" "c"]
}: ...
This previously could be done using assertions, but domain checks
will allow the buildfarm to automatically extract the configuration
space from functions.
