The fact that queryPathInfo() is synchronous meant that we needed a
thread for every concurrent binary cache lookup, even though they end
up being handled by the same download thread. Requiring hundreds of
threads is not a good idea. So now there is an asynchronous version of
queryPathInfo() that takes a callback function to process the
result. Similarly, enqueueDownload() now takes a callback rather than
returning a future.
Thus, a command like
nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5
that returns 4941 paths now takes 1.87s using only 2 threads (the main
thread and the downloader thread). (This is with a prewarmed
CloudFront.)
E.g.
$ nix-build -I nixpkgs=git://github.com/NixOS/nixpkgs '<nixpkgs>' -A hello
This is not extremely useful yet because you can't specify a
branch/revision.
The flag remembering whether an Interrupted exception was thrown is
now thread-local. Thus, all threads will (eventually) throw
Interrupted. Previously, one thread would throw Interrupted, and then
the other threads wouldn't see that they were supposed to quit.
Unlike "nix-store --verify-path", this command verifies signatures in
addition to store path contents, is multi-threaded (especially useful
when verifying binary caches), and has a progress indicator.
Example use:
$ nix verify-paths --store https://cache.nixos.org -r $(type -p thunderbird)
...
[17/132 checked] checking ‘/nix/store/rawakphadqrqxr6zri2rmnxh03gqkrl3-autogen-5.18.6’
Also, move a few free-standing functions into StoreAPI and Derivation.
Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
Sodium's Ed25519 signatures are much shorter than OpenSSL's RSA
signatures. Public keys are also much shorter, so they're now
specified directly in the nix.conf option ‘binary-cache-public-keys’.
The new command ‘nix-store --generate-binary-cache-key’ generates and
prints a public and secret key.
This was preventing destructors from running. In particular, it was
preventing the deletion of the temproot file for each worker
process. It may also have been responsible for the excessive WAL
growth on Hydra (due to the SQLite database not being closed
properly).
Apparently broken by accident in
8e9140cfde.
If a build log is not available locally, then ‘nix-store -l’ will now
try to download it from the servers listed in the ‘log-servers’ option
in nix.conf. For instance, if you have:
log-servers = http://hydra.nixos.org/log
then it will try to get logs from http://hydra.nixos.org/log/<base
name of the store path>. So you can do things like:
$ nix-store -l $(which xterm)
and get a log even if xterm wasn't built locally.
Ludo reported this error:
unexpected Nix daemon error: boost::too_few_args: format-string refered to more arguments than were passed
coming from this line:
printMsg(lvlError, run.program + ": " + string(err, 0, p));
The problem here is that the string ends up implicitly converted to a
Boost format() object, so % characters are treated specially. I
always assumed (wrongly) that strings are converted to a format object
that outputs the string as-is.
Since this assumption appears in several places that may be hard to
grep for, I've added some C++ type hackery to ensures that the right
thing happens. So you don't have to worry about % in statements like
printMsg(lvlError, "foo: " + s);
or
throw Error("foo: " + s);
For example, given a derivation with outputs "out", "man" and "bin":
$ nix-build -A pkg
produces ./result pointing to the "out" output;
$ nix-build -A pkg.man
produces ./result-man pointing to the "man" output;
$ nix-build -A pkg.all
produces ./result, ./result-man and ./result-bin;
$ nix-build -A pkg.all -A pkg2
produces ./result, ./result-man, ./result-bin and ./result-2.
This is required on systemd, which mounts filesystems as "shared"
subtrees. Changes to shared trees in a private mount namespace are
propagated to the outside world, which is bad.
Setting the environment variable NIX_COUNT_CALLS to 1 enables some
basic profiling in the evaluator. It will count calls to functions
and primops as well as evaluations of attributes.
For example, to see where evaluation of a NixOS configuration spends
its time:
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 ./src/nix-instantiate/nix-instantiate '<nixos>' -A system --readonly-mode
...
calls to 39 primops:
239532 head
233962 tail
191252 hasAttr
...
calls to 1595 functions:
224157 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:19'
221767 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:14'
221767 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:10'
...
evaluations of 7088 attributes:
167377 undefined position
132459 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/attrsets.nix:119:41'
47322 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/attrsets.nix:13:21'
...
unreachable paths. This matters when using --max-freed etc.:
unreachable paths could become reachable again, so it's nicer to
keep them if there is "real" garbage to be deleted. Also, don't use
readDirectory() but read the Nix store and delete invalid paths in
parallel. This reduces GC latency on very large Nix stores.
it requires a certain feature on the build machine, e.g.
requiredSystemFeatures = [ "kvm" ];
We need this in Hydra to make sure that builds that require KVM
support are forwarded to machines that have KVM support. Probably
this should also be enforced for local builds.
_FILE_OFFSET_BITS=64. Without it, functions like stat() fail on
large file sizes. This happened with a Nix store on squashfs:
$ nix-store --dump /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds > /dev/null
error: getting attributes of path `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds': Value too large for defined data type
$ stat /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds
File: `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds'
Size: 0 Blocks: 36028797018963968 IO Block: 1024 regular empty file
(This is a bug in squashfs or mksquashfs, but it shouldn't cause Nix
to fail.)
order of ascending last access time. This is useful in conjunction
with --max-freed or --max-links to prefer deleting non-recently used
garbage, which is good (especially in the build farm) since garbage
may become live again.
The code could easily be modified to accept other criteria for
ordering garbage by changing the comparison operator used by the
priority queue in collectGarbage().
~/.nix-defexpr, otherwise the attribute cannot be selected with the
`-A' option. Useful if you want to stick a Nix expression directly
in ~/.nix-defexpr.
again. (After the previous substituter mechanism refactoring I
didn't update the code that obtains the references of substitutable
paths.) This required some refactoring: the substituter programs
are now kept running and receive/respond to info requests via
stdin/stdout.
/tmp/nix-<pid>-<counter> for temporary build directories. This
increases purity a bit: many packages store the temporary build path
in their output, causing (generally unimportant) binary differences.
executed in a chroot that contains just the Nix store, the temporary
build directory, and a configurable set of additional directories
(/dev and /proc by default). This allows a bit more purity
enforcement: hidden build-time dependencies on directories such as
/usr or /nix/var/nix/profiles are no longer possible. As an added
benefit, accidental network downloads (cf. NIXPKGS-52) are prevented
as well (because files such as /etc/resolv.conf are not available in
the chroot).
However the usefulness of chroots is diminished by the fact that
many builders depend on /bin/sh, so you need /bin in the list of
additional directories. (And then on non-NixOS you need /lib as
well...)
need any info on substitutable paths, we just call the substituters
(such as download-using-manifests.pl) directly. This means that
it's no longer necessary for nix-pull to register substitutes or for
nix-channel to clear them, which makes those operations much faster
(NIX-95). Also, we don't have to worry about keeping nix-pull
manifests (in /nix/var/nix/manifests) and the database in sync with
each other.
The downside is that there is some overhead in calling an external
program to get the substitutes info. For instance, "nix-env -qas"
takes a bit longer.
Abolishing the substitutes table also makes the logic in
local-store.cc simpler, as we don't need to store info for invalid
paths. On the downside, you cannot do things like "nix-store -qR"
on a substitutable but invalid path (but nobody did that anyway).
* Never catch interrupts (the Interrupted exception).
--export' into the Nix store, and optionally check the cryptographic
signatures against /nix/etc/nix/signing-key.pub. (TODO: verify
against a set of public keys.)
from a source directory. All files for which a predicate function
returns true are copied to the store. Typical example is to leave
out the .svn directory:
stdenv.mkDerivation {
...
src = builtins.filterSource
(path: baseNameOf (toString path) != ".svn")
./source-dir;
# as opposed to
# src = ./source-dir;
}
This is important because the .svn directory influences the hash in
a rather unpredictable and variable way.
via the Unix domain socket in /nix/var/nix/daemon.socket. The
server forks a worker process per connection.
* readString(): use the heap, not the stack.
* Some protocol fixes.
* Set the references for the user environment manifest properly.
* Don't copy the manifest (this was accidental).
* Don't store derivation paths in the manifest (maybe this should be
made optional). This cleans up the semantics of nix-env, which were
weird.
* Hash on the output paths of activated components, not on derivation
paths. This is because we don't know the derivation path of already
installed components anymore, and it allows the installation of
components by store path (skipping Nix expressions entirely).
* Query options `--out-path' and `--drv-path' to show the output and
derivation paths of components, respectively (the latter replaces
the `--expr' query).
get rid of GC roots. Nix-build places a symlink `result' in the
current directory. Previously, removing that symlink would not
remove the store path being linked to as a GC root. Now, the GC
root created by nix-build is actually a symlink in
`/nix/var/nix/gcroots/auto' to `result'. So if that symlink is
removed the GC root automatically becomes invalid (since it can no
longer be resolved). The root itself is not automatically removed -
the garbage collector should delete dangling roots.
roots to a per-process temporary file in /nix/var/nix/temproots
while holding a write lock on that file. The garbage collector
acquires read locks on all those files, thus blocking further
progress in other Nix processes, and reads the sets of temporary
roots.
profile. Arguments are either generation number, or `old' to delete
all non-current generations. Typical use:
$ nix-env --delete-generations old
$ nix-collect-garbage
* istringstream -> string2Int.
Previously there was the problem that all files read by nix-env
etc. should be reachable and readable by the Nix user. So for
instance building a Nix expression in your home directory meant that
the home directory should have at least g+x or o+x permission so
that the Nix user could reach the Nix expression. Now we just
switch back to the original user just prior to reading sources and
the like. The places where this happens are somewhat arbitrary,
however. Any scope that has a live SwitchToOriginalUser object in
it is executed as the original user.
* Back out r1385. setreuid() sets the saved uid to the new
real/effective uid, which prevents us from switching back to the
original uid. setresuid() doesn't have this problem (although the
manpage has a bug: specifying -1 for the saved uid doesn't leave it
unchanged; an explicit value must be specified).
This is because the contents of these symlinks are not incorporated
into the hashes of derivations, and could therefore cause a mismatch
between the build system and the target system. E.g., if
`/nix/store' is a symlink to `/data/nix/store', then a builder could
expand this path and store the result. If on the target system
`/nix/store' is not a symlink, or is a symlink that points somewhere
else, we have a dangling pointer.
The trigger for this change is that gcc 3.3.3 does exactly that (it
applies realpath() to some files, such as libraries, which causes
our impurity checker to bail out.)
An annoying side-effect of this change is that it makes it harder to
move the Nix store to a different file system. On Linux, bind
mounts can be used instead of symlink for this purpose (e.g., `mount
-o bind /data/nix/store /nix/store').
writes to stderr:
- `pretty': the old nested style (default)
- `escapes': uses escape codes to indicate nesting and message
level; can be processed using `log2xml'
- `flat': just plain text, no nesting
These can be set using `--log-type TYPE' or the NIX_LOG_TYPE
environment variable.
Nix. This is to prevent Berkeley DB from becoming wedged.
Unfortunately it is not possible to throw C++ exceptions from a
signal handler. In fact, you can't do much of anything except
change variables of type `volatile sig_atomic_t'. So we set an
interrupt flag in the signal handler and check it at various
strategic locations in the code (by calling checkInterrupt()).
Since this is unlikely to cover all cases (e.g., (semi-)infinite
loops), sometimes SIGTERM may now be required to kill Nix.
path of the Nix expression to be used with the import, upgrade, and
query commands. For instance,
$ nix-env -I ~/nixpkgs/pkgs/system/i686-linux.nix
$ nix-env --query --available [aka -qa]
sylpheed-0.9.7
bison-1.875
pango-1.2.5
subversion-0.35.1
...
$ nix-env -i sylpheed
$ nix-env -u subversion
There can be only one default at a time.
* If the path to a Nix expression is a symlink, follow the symlink
prior to resolving relative path references in the expression.
nix-env -u foo.nix strategoxt
to replace the installed `strategoxt' derivation with the one from `foo.nix', if
the latter has a higher version number. This is a no-op if `strategoxt' is not
installed. Wildcards are also accepted, so
nix-env -u foo.nix '*'
will replace any installed derivation with newer versions from `foo.nix', if
available.
The notion of "version number" is somewhat ad hoc, but should be useful in most
cases, as evidenced by the following unit tests for the version comparator:
TEST("1.0", "2.3", -1);
TEST("2.1", "2.3", -1);
TEST("2.3", "2.3", 0);
TEST("2.5", "2.3", 1);
TEST("3.1", "2.3", 1);
TEST("2.3.1", "2.3", 1);
TEST("2.3.1", "2.3a", 1);
TEST("2.3pre1", "2.3", -1);
TEST("2.3pre3", "2.3pre12", -1);
TEST("2.3a", "2.3c", -1);
TEST("2.3pre1", "2.3c", -1);
TEST("2.3pre1", "2.3q", -1);
(-1 = less, 0 = equal, 1 = greater)
* A new verbosity level `lvlInfo', between `lvlError' and `lvlTalkative'. This is
the default for `nix-env', so without any `-v' flags users should get useful
output, e.g.,
$ nix-env -u foo.nix strategoxt
upgrading `strategoxt-0.9.2' to `strategoxt-0.9.3'
