improve throughput.
* Don't build the `substitute-rev' table for now, since it caused
Theta(N^2) time and log file consumption when adding N substitutes.
Maybe we can do without it.
* A better substitute mechanism.
Instead of generating a store expression for each store path for
which we have a substitute, we can have a single store expression
that builds a generic program that is invoked to build the desired
store path, which is passed as an argument.
This means that operations like `nix-pull' only produce O(1) files
instead of O(N) files in the store when registering N substitutes.
(It consumes O(N) database storage, of course, but that's not a
performance problem).
* Added a test for the substitute mechanism.
* `nix-store --substitute' reads the substitutes from standard input,
instead of from the command line. This prevents us from running
into the kernel's limit on command line length.
* When a fast build wakes up a goal, try to start that goal in the
same iteration of the startBuild() loop of run(). Otherwise no job
might be started until the next job terminates.
in parallel. Hooks are more efficient: locks on output paths are
only acquired when the hook says that it is willing to accept a
build job. Hooks now work in two phases. First, they should first
tell Nix whether they are willing to accept a job. Nix guarantuees
that no two hooks will ever be in the first phase at the same time
(this simplifies the implementation of hooks, since they don't have
to perform locking (?)). Second, if they accept a job, they are
then responsible for building it (on the remote system), and copying
the result back. These can be run in parallel with other hooks and
locally executed jobs.
The implementation is a bit messy right now, though.
* The directory `distributed' shows a (hacky) example of a hook that
distributes build jobs over a set of machines listed in a
configuration file.
distributing a build action to another machine. In particular, the
paths in the input closures, the output paths, and successor mapping
for sub-derivations.
parallel as possible (similar to GNU Make's `-j' switch). This is
useful on SMP systems, but it is especially useful for doing builds
on multiple machines. The idea is that a large derivation is
initiated on one master machine, which then distributes
sub-derivations to any number of slave machines. This should not
happen synchronously or in lock-step, so the master must be capable
of dealing with multiple parallel build jobs. We now have the
infrastructure to support this.
TODO: substitutes are currently broken.
print a nice backtrace of the stack, rather than vomiting a gigantic
(and useless) aterm on the screen. Example:
error: while evaluating file `.../pkgs/system/test.nix':
while evaluating attribute `subversion' at `.../pkgs/system/all-packages-generic.nix', line 533:
while evaluating function at `.../pkgs/applications/version-management/subversion/default.nix', line 1:
assertion failed at `.../pkgs/applications/version-management/subversion/default.nix', line 13
Since the Nix expression language is lazy, the trace may be
misleading. The purpose is to provide a hint as to the location of
the problem.
instead of `derivation' triggered a huge slowdown in the Nix
expression evaluator. Total execution time of `nix-env -qa' went up
by a factor of 60 or so.
This scalability problem was caused by expressions such as
(x: y: ... x ...) a b
where `a' is a large term (say, the one in
`all-packages-generic.nix'). Then the first beta-reduction would
produce
(y: ... a ...) b
by substituting `a' for `x'. The second beta-reduction would then
substitute `b' for `y' into the body `... a ...', which is a large
term due to `a', and thus causes a large traversal to be performed
by substitute() in the second reduction. This is however entirely
redundant, since `a' cannot contain free variables (since we never
substitute below a weak head normal form).
The solution is to wrap substituted terms into a `Closed'
constructor, i.e.,
subst(subs, Var(x)) = Closed(e) iff subs[x] = e
have substitution not descent into closed terms,
subst(subs, Closed(x)) = Closed(x)
and otherwise ignore them for evaluation,
eval(Closed(x)) = eval(x).
* Fix a typo that caused incorrect substitutions to be performed in
simple lambdas, e.g., `(x: x: x) a' would reduce to `(x: a)'.
`bla:' is now no longer parsed as a URL.
* Re-enabled support for the `args' attribute in derivations to
specify command line arguments to the builder, e.g.,
...
builder = /usr/bin/python;
args = ["-c" ./builder.py];
...
This is because the contents of these symlinks are not incorporated
into the hashes of derivations, and could therefore cause a mismatch
between the build system and the target system. E.g., if
`/nix/store' is a symlink to `/data/nix/store', then a builder could
expand this path and store the result. If on the target system
`/nix/store' is not a symlink, or is a symlink that points somewhere
else, we have a dangling pointer.
The trigger for this change is that gcc 3.3.3 does exactly that (it
applies realpath() to some files, such as libraries, which causes
our impurity checker to bail out.)
An annoying side-effect of this change is that it makes it harder to
move the Nix store to a different file system. On Linux, bind
mounts can be used instead of symlink for this purpose (e.g., `mount
-o bind /data/nix/store /nix/store').
writes to stderr:
- `pretty': the old nested style (default)
- `escapes': uses escape codes to indicate nesting and message
level; can be processed using `log2xml'
- `flat': just plain text, no nesting
These can be set using `--log-type TYPE' or the NIX_LOG_TYPE
environment variable.
unimportant messages, it is collapsed by the default.
* Also added an optional integer argument to the escape code for opening a nesting
level to indicate lack of importance. If set, the tree is collapsed by default.
build logs. The program `log2xml' converts a Nix build log (read
from standard input) into XML file that can then be converted to
XHTML by the `log2html.xsl' stylesheet. The CSS stylesheet
`logfile.css' is necessary to make it look good.
This is primarily useful if the log file has a *tree structure*,
i.e., that sub-tasks such as the various phases of a build (unpack,
configure, make, etc.) or recursive invocations of Make are
represented as such. While a log file is in principle an
unstructured plain text file, builders can communicate this tree
structure to `log2xml' by using escape sequences:
- "\e[p" starts a new nesting level; the first line following the
escape code is the header;
- "\e[q" ends the current nesting level.
The generic builder in nixpkgs (not yet committed) uses this. It
shouldn't be to hard to patch GNU Make to speak this protocol.
Further improvements to the generated HTML pages are to allow
collapsing/expanding of subtrees, and to abbreviate store paths (but
to show the full path by hovering the mouse over it).
builders to point to the store and the temporary build directory,
respectively. Useful for purity checking.
* Also set TEMPDIR, TMPDIR, TEMP, and TEMP to NIX_BUILD_TOP to make
sure that tools in the builder store temporary files in the right
location.
