improve throughput.
* Don't build the `substitute-rev' table for now, since it caused
Theta(N^2) time and log file consumption when adding N substitutes.
Maybe we can do without it.
* A better substitute mechanism.
Instead of generating a store expression for each store path for
which we have a substitute, we can have a single store expression
that builds a generic program that is invoked to build the desired
store path, which is passed as an argument.
This means that operations like `nix-pull' only produce O(1) files
instead of O(N) files in the store when registering N substitutes.
(It consumes O(N) database storage, of course, but that's not a
performance problem).
* Added a test for the substitute mechanism.
* `nix-store --substitute' reads the substitutes from standard input,
instead of from the command line. This prevents us from running
into the kernel's limit on command line length.
* When a fast build wakes up a goal, try to start that goal in the
same iteration of the startBuild() loop of run(). Otherwise no job
might be started until the next job terminates.
in parallel. Hooks are more efficient: locks on output paths are
only acquired when the hook says that it is willing to accept a
build job. Hooks now work in two phases. First, they should first
tell Nix whether they are willing to accept a job. Nix guarantuees
that no two hooks will ever be in the first phase at the same time
(this simplifies the implementation of hooks, since they don't have
to perform locking (?)). Second, if they accept a job, they are
then responsible for building it (on the remote system), and copying
the result back. These can be run in parallel with other hooks and
locally executed jobs.
The implementation is a bit messy right now, though.
* The directory `distributed' shows a (hacky) example of a hook that
distributes build jobs over a set of machines listed in a
configuration file.
distributing a build action to another machine. In particular, the
paths in the input closures, the output paths, and successor mapping
for sub-derivations.
parallel as possible (similar to GNU Make's `-j' switch). This is
useful on SMP systems, but it is especially useful for doing builds
on multiple machines. The idea is that a large derivation is
initiated on one master machine, which then distributes
sub-derivations to any number of slave machines. This should not
happen synchronously or in lock-step, so the master must be capable
of dealing with multiple parallel build jobs. We now have the
infrastructure to support this.
TODO: substitutes are currently broken.
print a nice backtrace of the stack, rather than vomiting a gigantic
(and useless) aterm on the screen. Example:
error: while evaluating file `.../pkgs/system/test.nix':
while evaluating attribute `subversion' at `.../pkgs/system/all-packages-generic.nix', line 533:
while evaluating function at `.../pkgs/applications/version-management/subversion/default.nix', line 1:
assertion failed at `.../pkgs/applications/version-management/subversion/default.nix', line 13
Since the Nix expression language is lazy, the trace may be
misleading. The purpose is to provide a hint as to the location of
the problem.
instead of `derivation' triggered a huge slowdown in the Nix
expression evaluator. Total execution time of `nix-env -qa' went up
by a factor of 60 or so.
This scalability problem was caused by expressions such as
(x: y: ... x ...) a b
where `a' is a large term (say, the one in
`all-packages-generic.nix'). Then the first beta-reduction would
produce
(y: ... a ...) b
by substituting `a' for `x'. The second beta-reduction would then
substitute `b' for `y' into the body `... a ...', which is a large
term due to `a', and thus causes a large traversal to be performed
by substitute() in the second reduction. This is however entirely
redundant, since `a' cannot contain free variables (since we never
substitute below a weak head normal form).
The solution is to wrap substituted terms into a `Closed'
constructor, i.e.,
subst(subs, Var(x)) = Closed(e) iff subs[x] = e
have substitution not descent into closed terms,
subst(subs, Closed(x)) = Closed(x)
and otherwise ignore them for evaluation,
eval(Closed(x)) = eval(x).
* Fix a typo that caused incorrect substitutions to be performed in
simple lambdas, e.g., `(x: x: x) a' would reduce to `(x: a)'.
`bla:' is now no longer parsed as a URL.
* Re-enabled support for the `args' attribute in derivations to
specify command line arguments to the builder, e.g.,
...
builder = /usr/bin/python;
args = ["-c" ./builder.py];
...
This is because the contents of these symlinks are not incorporated
into the hashes of derivations, and could therefore cause a mismatch
between the build system and the target system. E.g., if
`/nix/store' is a symlink to `/data/nix/store', then a builder could
expand this path and store the result. If on the target system
`/nix/store' is not a symlink, or is a symlink that points somewhere
else, we have a dangling pointer.
The trigger for this change is that gcc 3.3.3 does exactly that (it
applies realpath() to some files, such as libraries, which causes
our impurity checker to bail out.)
An annoying side-effect of this change is that it makes it harder to
move the Nix store to a different file system. On Linux, bind
mounts can be used instead of symlink for this purpose (e.g., `mount
-o bind /data/nix/store /nix/store').
writes to stderr:
- `pretty': the old nested style (default)
- `escapes': uses escape codes to indicate nesting and message
level; can be processed using `log2xml'
- `flat': just plain text, no nesting
These can be set using `--log-type TYPE' or the NIX_LOG_TYPE
environment variable.
unimportant messages, it is collapsed by the default.
* Also added an optional integer argument to the escape code for opening a nesting
level to indicate lack of importance. If set, the tree is collapsed by default.
build logs. The program `log2xml' converts a Nix build log (read
from standard input) into XML file that can then be converted to
XHTML by the `log2html.xsl' stylesheet. The CSS stylesheet
`logfile.css' is necessary to make it look good.
This is primarily useful if the log file has a *tree structure*,
i.e., that sub-tasks such as the various phases of a build (unpack,
configure, make, etc.) or recursive invocations of Make are
represented as such. While a log file is in principle an
unstructured plain text file, builders can communicate this tree
structure to `log2xml' by using escape sequences:
- "\e[p" starts a new nesting level; the first line following the
escape code is the header;
- "\e[q" ends the current nesting level.
The generic builder in nixpkgs (not yet committed) uses this. It
shouldn't be to hard to patch GNU Make to speak this protocol.
Further improvements to the generated HTML pages are to allow
collapsing/expanding of subtrees, and to abbreviate store paths (but
to show the full path by hovering the mouse over it).
builders to point to the store and the temporary build directory,
respectively. Useful for purity checking.
* Also set TEMPDIR, TMPDIR, TEMP, and TEMP to NIX_BUILD_TOP to make
sure that tools in the builder store temporary files in the right
location.
* Do not create stuff in localstatedir when doing `make install'
(since we may not have write access). In general, installation of
constant code/data should be separate from the initialisation of
mutable state.
chroot() environment.
* A operation `--validpath' to register path validity. Useful for
bootstrapping in a pure Nix environment.
* Safety checks: ensure that files involved in store operations are in
the store.
derivation (i.e., the closure store expression) a root of the
garbage collector. This ensures that running `nix-collect-garbage
--no-successors' is safe.
whether we want to upgrade if the current version is less than the
available version (default), when it is less or equal, or always.
* Added a flag `--dry-run' to show what would happen in `--install',
`--uninstall', and `--upgrade', without actually performing the
operation.
of the current profile, e.g.,
$ nix-env --list-generations
...
39 2004-02-02 17:53:53
40 2004-02-02 17:55:18
41 2004-02-02 17:55:41
42 2004-02-02 17:55:50 (current)
$ nix-env --switch-generation 39
$ ls -l /nix/var/nix/profiles/default
... default -> default-39-link
* Also a command `--rollback' which is just a convenience operation to
rollback to the oldest generation younger than the current one.
Note that generations properly form a tree. E.g., if after
switching to generation 39, we perform an installation action,
a generation 43 is created which is a descendant of 39, not 42. So
a rollback from 43 ought to go back to 39. This is not currently
implemented; generations form a linear sequence.
default -> default-94-link
default-82-link -> /nix/store/cc4480...
default-83-link -> /nix/store/caeec8...
...
default-94-link -> /nix/store/2896ca...
experimental -> experimental-2-link
experimental-1-link -> /nix/store/cc4480...
experimental-2-link -> /nix/store/a3148f...
* `--profile' / `-p' -> `--switch-profile' / `-S'
* `--link' / `-l' -> `--profile' / `-p'
* The default profile is stored in $prefix/var/nix/profiles.
$prefix/var/nix/links is gone. Profiles can be stored anywhere.
* The current profile is now referenced from ~/.nix-profile, not
~/.nix-userenv.
* The roots to the garbage collector now have extension `.gcroot', not
`.id'.
other attribute sets, rather than the current scope. E.g.,
{inherit (pkgs) gcc binutils;}
is equivalent to
{gcc = pkgs.gcc; binutils = pkgs.binutils;}
I am not so happy about the syntax.
parser (roughly 80x faster).
The absolutely latest version of Bison (1.875c) is required for
reentrant GLR support, as well as a recent version of Flex (say,
2.5.31). Note that most Unix distributions ship with the
prehistoric Flex 2.5.4, which doesn't support reentrancy.
Nix. This is to prevent Berkeley DB from becoming wedged.
Unfortunately it is not possible to throw C++ exceptions from a
signal handler. In fact, you can't do much of anything except
change variables of type `volatile sig_atomic_t'. So we set an
interrupt flag in the signal handler and check it at various
strategic locations in the code (by calling checkInterrupt()).
Since this is unlikely to cover all cases (e.g., (semi-)infinite
loops), sometimes SIGTERM may now be required to kill Nix.
the output path of a derivation, not the path of its store
expression. This ensures that changes that affect the path of the
store expression but not the output path, do not affect the
`installed' state of a derivation.
it automatically removes log files when they are no longer needed.
*** IMPORTANT ***
If you have an existing Nix installation, you must checkpoint the
Nix database to prevent recent transactions from being undone. Do
the following:
- optional: make a backup of $prefix/var/nix/db.
- run `db_checkpoint' from Berkeley DB 4.1:
$ db_checkpoint -h $prefix/var/nix/db -1
- optional (?): run `db_recover' from Berkeley DB 4.1:
$ db_recover -h $prefix/var/nix/db
- remove $prefix/var/nix/db/log* and $prefix/var/nix/db/__db*
path of the Nix expression to be used with the import, upgrade, and
query commands. For instance,
$ nix-env -I ~/nixpkgs/pkgs/system/i686-linux.nix
$ nix-env --query --available [aka -qa]
sylpheed-0.9.7
bison-1.875
pango-1.2.5
subversion-0.35.1
...
$ nix-env -i sylpheed
$ nix-env -u subversion
There can be only one default at a time.
* If the path to a Nix expression is a symlink, follow the symlink
prior to resolving relative path references in the expression.
the symlink ~/.nix-userenv to the given argument (which defaults to
.../links/current). /etc/profile.d/nix-profile creates this symlink
if it doesn't exist yet. Example use:
$ nix-env -l my_profile -i foo.nix subversion quake
$ nix-env -p my_profile
I don't like the term "profile". Let's deprecate it :-)
nix-env -u foo.nix strategoxt
to replace the installed `strategoxt' derivation with the one from `foo.nix', if
the latter has a higher version number. This is a no-op if `strategoxt' is not
installed. Wildcards are also accepted, so
nix-env -u foo.nix '*'
will replace any installed derivation with newer versions from `foo.nix', if
available.
The notion of "version number" is somewhat ad hoc, but should be useful in most
cases, as evidenced by the following unit tests for the version comparator:
TEST("1.0", "2.3", -1);
TEST("2.1", "2.3", -1);
TEST("2.3", "2.3", 0);
TEST("2.5", "2.3", 1);
TEST("3.1", "2.3", 1);
TEST("2.3.1", "2.3", 1);
TEST("2.3.1", "2.3a", 1);
TEST("2.3pre1", "2.3", -1);
TEST("2.3pre3", "2.3pre12", -1);
TEST("2.3a", "2.3c", -1);
TEST("2.3pre1", "2.3c", -1);
TEST("2.3pre1", "2.3q", -1);
(-1 = less, 0 = equal, 1 = greater)
* A new verbosity level `lvlInfo', between `lvlError' and `lvlTalkative'. This is
the default for `nix-env', so without any `-v' flags users should get useful
output, e.g.,
$ nix-env -u foo.nix strategoxt
upgrading `strategoxt-0.9.2' to `strategoxt-0.9.3'
turned out to be a huge performance bottleneck (the text to printed
would always be evaluated, even when it was above the verbosity
level). This reduces fix-ng execution time by over 50%.
gprof(1) is very useful. :-)
The ATerm library doesn't search the heap for pointers to ATerms
when garbage collecting. As a result, C++ containers such as
`map<ATerm, ATerm>' will cause pointer to be hidden from the garbage
collector, causing crashes. Instead, we now use ATermTables.
