unreachable paths that haven't been used for N hours. For instance,
`nix-collect-garbage --min-age 168' only deletes paths that haven't
been accessed in the last week.
This is useful for instance in the build farm where many derivations
can be shared between consecutive builds, and we wouldn't want a
garbage collect to throw them all away. We could of course register
them as roots, but then we'd to unregister them at some point, which
would be a pain to manage. The `--min-age' flag gives us a sort of
MRU caching scheme.
BUG: this really shouldn't be in gc.cc since that violates
mechanism/policy separation.
doesn't just print the set of paths that should be deleted. So
there is no more need to pipe the result into `nix-store --delete'
(which doesn't even exist anymore).
suboperations `--print-live', `--print-dead', and `--delete'. The
roots are not determined by nix-store; they are read from standard
input. This is to make it easy to customise what the roots are.
The collector now no longer fails when store expressions are missing
(which legally happens when using substitutes). It never tries to
fetch paths through substitutes.
TODO: acquire a global lock on the store while garbage collecting.
* Removed `nix-store --delete'.
set the real uid and gid to the effective uid and gid, the Nix
binaries can be installed as owned by the Nix user and group instead
of root, so no root involvement of any kind is necessary.
Linux and FreeBSD have these functions.
users.
If the configure flag `--enable-setuid' is used, the Nix programs
nix-env, nix-store, etc. are installed with the setuid bit turned on
so that they are executed as the user and group specified by
`--with-nix-user=USER' and `--with-nix-group=GROUP', respectively
(with defaults `nix' and `nix').
The setuid programs drop all special privileges if they are executed
by a user who is not a member of the Nix group.
The setuid feature is a quick hack to enable sharing of a Nix
installation between users who trust each other. It is not
generally secure, since any user in the Nix group can modify (by
building an appropriate derivation) any object in the store, and for
instance inject trojans into binaries used by other users.
The setuid programs are owned by root, not the Nix user. This is
because on Unix normal users cannot change the real uid, only the
effective uid. Many programs don't work properly when the real uid
differs from the effective uid. For instance, Perl will turn on
taint mode. However, the setuid programs drop all root privileges
immediately, changing all uids and gids to the Nix user and group.
* Builder output is written to standard error by default.
* The option `-B' is gone.
* The option `-Q' suppresses builder output.
The result of this is that most Nix invocations shouldn't need any
flags w.r.t. logging.
derivation disables scanning for dependencies. Use at your own
risk. This is a quick hack to speed up UML image generation (image
are very big, say 1 GB).
It would be better if the scanner were faster, and didn't read the
whole file into memory.
system types other than the current system. I.e., `nix-env -i'
won't install derivations for other system types, and `nix-env -q'
won't show them. The flag `--system-filter SYSTEM' can be used to
override the system type used for filtering (but not for
building!). The value `*' can be used not to filter anything.
Whenever Nix attempts to realise a derivation for which a closure is
already known, but this closure cannot be realised, fall back on
normalising the derivation.
The most common scenario in which this is useful is when we have
registered substitutes in order to perform binary distribution from,
say, a network repository. If the repository is down, the
realisation of the derivation will fail. When this option is
specified, Nix will build the derivation instead. Thus, binary
installation falls back on a source installation. This option is
not the default since it is generally not desirable for a transient
failure in obtaining the substitutes to lead to a full build from
source (with the related consumption of resources).
much as possible. (This is similar to GNU Make's `-k' flag.)
* Refactoring to implement this: previously we just bombed out when
a build failed, but now we have to clean up. In particular this
means that goals must be freed quickly --- they shouldn't hang
around until the worker exits. So the worker now maintains weak
pointers in order not to prevent garbage collection.
* Documented the `-k' and `-j' flags.
improve throughput.
* Don't build the `substitute-rev' table for now, since it caused
Theta(N^2) time and log file consumption when adding N substitutes.
Maybe we can do without it.
* A better substitute mechanism.
Instead of generating a store expression for each store path for
which we have a substitute, we can have a single store expression
that builds a generic program that is invoked to build the desired
store path, which is passed as an argument.
This means that operations like `nix-pull' only produce O(1) files
instead of O(N) files in the store when registering N substitutes.
(It consumes O(N) database storage, of course, but that's not a
performance problem).
* Added a test for the substitute mechanism.
* `nix-store --substitute' reads the substitutes from standard input,
instead of from the command line. This prevents us from running
into the kernel's limit on command line length.
* When a fast build wakes up a goal, try to start that goal in the
same iteration of the startBuild() loop of run(). Otherwise no job
might be started until the next job terminates.
in parallel. Hooks are more efficient: locks on output paths are
only acquired when the hook says that it is willing to accept a
build job. Hooks now work in two phases. First, they should first
tell Nix whether they are willing to accept a job. Nix guarantuees
that no two hooks will ever be in the first phase at the same time
(this simplifies the implementation of hooks, since they don't have
to perform locking (?)). Second, if they accept a job, they are
then responsible for building it (on the remote system), and copying
the result back. These can be run in parallel with other hooks and
locally executed jobs.
The implementation is a bit messy right now, though.
* The directory `distributed' shows a (hacky) example of a hook that
distributes build jobs over a set of machines listed in a
configuration file.
distributing a build action to another machine. In particular, the
paths in the input closures, the output paths, and successor mapping
for sub-derivations.
parallel as possible (similar to GNU Make's `-j' switch). This is
useful on SMP systems, but it is especially useful for doing builds
on multiple machines. The idea is that a large derivation is
initiated on one master machine, which then distributes
sub-derivations to any number of slave machines. This should not
happen synchronously or in lock-step, so the master must be capable
of dealing with multiple parallel build jobs. We now have the
infrastructure to support this.
TODO: substitutes are currently broken.
print a nice backtrace of the stack, rather than vomiting a gigantic
(and useless) aterm on the screen. Example:
error: while evaluating file `.../pkgs/system/test.nix':
while evaluating attribute `subversion' at `.../pkgs/system/all-packages-generic.nix', line 533:
while evaluating function at `.../pkgs/applications/version-management/subversion/default.nix', line 1:
assertion failed at `.../pkgs/applications/version-management/subversion/default.nix', line 13
Since the Nix expression language is lazy, the trace may be
misleading. The purpose is to provide a hint as to the location of
the problem.
instead of `derivation' triggered a huge slowdown in the Nix
expression evaluator. Total execution time of `nix-env -qa' went up
by a factor of 60 or so.
This scalability problem was caused by expressions such as
(x: y: ... x ...) a b
where `a' is a large term (say, the one in
`all-packages-generic.nix'). Then the first beta-reduction would
produce
(y: ... a ...) b
by substituting `a' for `x'. The second beta-reduction would then
substitute `b' for `y' into the body `... a ...', which is a large
term due to `a', and thus causes a large traversal to be performed
by substitute() in the second reduction. This is however entirely
redundant, since `a' cannot contain free variables (since we never
substitute below a weak head normal form).
The solution is to wrap substituted terms into a `Closed'
constructor, i.e.,
subst(subs, Var(x)) = Closed(e) iff subs[x] = e
have substitution not descent into closed terms,
subst(subs, Closed(x)) = Closed(x)
and otherwise ignore them for evaluation,
eval(Closed(x)) = eval(x).
* Fix a typo that caused incorrect substitutions to be performed in
simple lambdas, e.g., `(x: x: x) a' would reduce to `(x: a)'.
`bla:' is now no longer parsed as a URL.
* Re-enabled support for the `args' attribute in derivations to
specify command line arguments to the builder, e.g.,
...
builder = /usr/bin/python;
args = ["-c" ./builder.py];
...
This is because the contents of these symlinks are not incorporated
into the hashes of derivations, and could therefore cause a mismatch
between the build system and the target system. E.g., if
`/nix/store' is a symlink to `/data/nix/store', then a builder could
expand this path and store the result. If on the target system
`/nix/store' is not a symlink, or is a symlink that points somewhere
else, we have a dangling pointer.
The trigger for this change is that gcc 3.3.3 does exactly that (it
applies realpath() to some files, such as libraries, which causes
our impurity checker to bail out.)
An annoying side-effect of this change is that it makes it harder to
move the Nix store to a different file system. On Linux, bind
mounts can be used instead of symlink for this purpose (e.g., `mount
-o bind /data/nix/store /nix/store').
writes to stderr:
- `pretty': the old nested style (default)
- `escapes': uses escape codes to indicate nesting and message
level; can be processed using `log2xml'
- `flat': just plain text, no nesting
These can be set using `--log-type TYPE' or the NIX_LOG_TYPE
environment variable.
unimportant messages, it is collapsed by the default.
* Also added an optional integer argument to the escape code for opening a nesting
level to indicate lack of importance. If set, the tree is collapsed by default.
build logs. The program `log2xml' converts a Nix build log (read
from standard input) into XML file that can then be converted to
XHTML by the `log2html.xsl' stylesheet. The CSS stylesheet
`logfile.css' is necessary to make it look good.
This is primarily useful if the log file has a *tree structure*,
i.e., that sub-tasks such as the various phases of a build (unpack,
configure, make, etc.) or recursive invocations of Make are
represented as such. While a log file is in principle an
unstructured plain text file, builders can communicate this tree
structure to `log2xml' by using escape sequences:
- "\e[p" starts a new nesting level; the first line following the
escape code is the header;
- "\e[q" ends the current nesting level.
The generic builder in nixpkgs (not yet committed) uses this. It
shouldn't be to hard to patch GNU Make to speak this protocol.
Further improvements to the generated HTML pages are to allow
collapsing/expanding of subtrees, and to abbreviate store paths (but
to show the full path by hovering the mouse over it).
builders to point to the store and the temporary build directory,
respectively. Useful for purity checking.
* Also set TEMPDIR, TMPDIR, TEMP, and TEMP to NIX_BUILD_TOP to make
sure that tools in the builder store temporary files in the right
location.
* Do not create stuff in localstatedir when doing `make install'
(since we may not have write access). In general, installation of
constant code/data should be separate from the initialisation of
mutable state.
chroot() environment.
* A operation `--validpath' to register path validity. Useful for
bootstrapping in a pure Nix environment.
* Safety checks: ensure that files involved in store operations are in
the store.
derivation (i.e., the closure store expression) a root of the
garbage collector. This ensures that running `nix-collect-garbage
--no-successors' is safe.
