Used to determine symlink size with stat and value with readlink.
This could technically result in garbage if symlink changed between
calls. Also gets around the broken stat implementation in our
network filesystem (returns size + 1 giving a byte of garbage).
Besides being unused, this function has a bug that it will incorrectly
decode the path component Ubuntu\04016.04.2\040LTS\040amd64 as
"Ubuntu.04.2 LTS amd64" instead of "Ubuntu 16.04.2 LTS amd64".
This allows <nix/fetchurl.nix> to fetch private Git/Mercurial
repositories, e.g.
import <nix/fetchurl.nix> {
url = https://edolstra@bitbucket.org/edolstra/my-private-repo/get/80a14018daed.tar.bz2;
sha256 = "1mgqzn7biqkq3hf2697b0jc4wabkqhmzq2srdymjfa6sb9zb6qs7";
}
where /etc/nix/netrc contains:
machine bitbucket.org
login edolstra
password blabla...
This works even when sandboxing is enabled.
To do: add unpacking support (i.e. fetchzip functionality).
Because config.h can #define things like _FILE_OFFSET_BITS=64 and not
every compilation unit includes config.h, we currently compile half of
Nix with _FILE_OFFSET_BITS=64 and other half with _FILE_OFFSET_BITS
unset. This causes major havoc with the Settings class on e.g. 32-bit ARM,
where different compilation units disagree with the struct layout.
E.g.:
diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc
@@ -166,6 +166,8 @@ void Settings::update()
_get(useSubstitutes, "build-use-substitutes");
+ fprintf(stderr, "at Settings::update(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
_get(buildUsersGroup, "build-users-group");
diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc
+++ b/src/libstore/remote-store.cc
@@ -138,6 +138,8 @@ void RemoteStore::initConnection(Connection & conn)
void RemoteStore::setOptions(Connection & conn)
{
+ fprintf(stderr, "at RemoteStore::setOptions(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
conn.to << wopSetOptions
Gave me:
at Settings::update(): &useSubstitutes = 0xb6e5c5cb
at RemoteStore::setOptions(): &useSubstitutes = 0xb6e5c5c7
That was not a fun one to debug!
This closes a long-time bug that allowed builds to hang Nix
indefinitely (regardless of timeouts) simply by doing
exec > /dev/null 2>&1; while true; do true; done
Now, on EOF, we just send SIGKILL to the child to make sure it's
really gone.
This allows other threads to install callbacks that run in a regular,
non-signal context. In particular, we can use this to signal the
downloader thread to quit.
Closes#1183.
We can now write
throw Error("file '%s' not found", path);
instead of
throw Error(format("file '%s' not found") % path);
and similarly
printError("file '%s' not found", path);
instead of
printMsg(lvlError, format("file '%s' not found") % path);
We were passing "p=$PATH" rather than "p=$PATH;", resulting in some
invalid shell code.
Also, construct a separate environment for the child rather than
overwriting the parent's.
The fact that queryPathInfo() is synchronous meant that we needed a
thread for every concurrent binary cache lookup, even though they end
up being handled by the same download thread. Requiring hundreds of
threads is not a good idea. So now there is an asynchronous version of
queryPathInfo() that takes a callback function to process the
result. Similarly, enqueueDownload() now takes a callback rather than
returning a future.
Thus, a command like
nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5
that returns 4941 paths now takes 1.87s using only 2 threads (the main
thread and the downloader thread). (This is with a prewarmed
CloudFront.)
This fixes instantiation of pythonPackages.pytest that produces a
directory with less permissions during one of it's tests that leads to
a nix error like:
error: opening directory ‘/tmp/nix-build-python2.7-pytest-2.9.2.drv-0/pytest-of-user/pytest-0/testdir/test_cache_failure_warns0/.cache’: Permission denied
E.g.
$ nix-build -I nixpkgs=git://github.com/NixOS/nixpkgs '<nixpkgs>' -A hello
This is not extremely useful yet because you can't specify a
branch/revision.
The flag remembering whether an Interrupted exception was thrown is
now thread-local. Thus, all threads will (eventually) throw
Interrupted. Previously, one thread would throw Interrupted, and then
the other threads wouldn't see that they were supposed to quit.
Unlike "nix-store --verify-path", this command verifies signatures in
addition to store path contents, is multi-threaded (especially useful
when verifying binary caches), and has a progress indicator.
Example use:
$ nix verify-paths --store https://cache.nixos.org -r $(type -p thunderbird)
...
[17/132 checked] checking ‘/nix/store/rawakphadqrqxr6zri2rmnxh03gqkrl3-autogen-5.18.6’
edolstra:
“…since callers of readDirectory have to handle the possibility of
DT_UNKNOWN anyway, and we don't want to do a stat call for every
directory entry unless it's really needed.”
Sodium's Ed25519 signatures are much shorter than OpenSSL's RSA
signatures. Public keys are also much shorter, so they're now
specified directly in the nix.conf option ‘binary-cache-public-keys’.
The new command ‘nix-store --generate-binary-cache-key’ generates and
prints a public and secret key.
Let's not just improve the error message itself, but also the behaviour
to actually work around the ntfs-3g symlink bug. If the readlink() call
returns a smaller size than the stat() call, this really isn't a problem
even if the symlink target really has changed between the calls.
So if stat() reports the size for the absolute path, it's most likely
that the relative path is smaller and thus it should also work for file
system bugs as mentioned in 93002d69fc58c2b71e2dfad202139230c630c53a.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Tested-by: John Ericson <Ericson2314@Yahoo.com>
A message like "error: reading symbolic link `...' : Success" really is
quite confusing, so let's not indicate "success" but rather point out
the real issue.
We could also limit the check of this to just check for non-negative
values, but this would introduce a race condition between stat() and
readlink() if the link target changes between those two calls, thus
leading to a buffer overflow vulnerability.
Reported by @Ericson2314 on IRC. Happened due to a possible ntfs-3g bug
where a relative symlink returned the absolute path (st_)size in stat()
while readlink() returned the relative size.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Tested-by: John Ericson <Ericson2314@Yahoo.com>
In low memory environments, "nix-env -qa" failed because the fork to
run the pager hit the kernel's overcommit limits. Using posix_spawn
gets around this. (Actually, you have to use posix_spawn with the
undocumented POSIX_SPAWN_USEVFORK flag, otherwise it just uses
fork/exec...)
This was preventing destructors from running. In particular, it was
preventing the deletion of the temproot file for each worker
process. It may also have been responsible for the excessive WAL
growth on Hydra (due to the SQLite database not being closed
properly).
Apparently broken by accident in
8e9140cfde.
If a build log is not available locally, then ‘nix-store -l’ will now
try to download it from the servers listed in the ‘log-servers’ option
in nix.conf. For instance, if you have:
log-servers = http://hydra.nixos.org/log
then it will try to get logs from http://hydra.nixos.org/log/<base
name of the store path>. So you can do things like:
$ nix-store -l $(which xterm)
and get a log even if xterm wasn't built locally.
Ludo reported this error:
unexpected Nix daemon error: boost::too_few_args: format-string refered to more arguments than were passed
coming from this line:
printMsg(lvlError, run.program + ": " + string(err, 0, p));
The problem here is that the string ends up implicitly converted to a
Boost format() object, so % characters are treated specially. I
always assumed (wrongly) that strings are converted to a format object
that outputs the string as-is.
Since this assumption appears in several places that may be hard to
grep for, I've added some C++ type hackery to ensures that the right
thing happens. So you don't have to worry about % in statements like
printMsg(lvlError, "foo: " + s);
or
throw Error("foo: " + s);
The kill(2) in Apple's libc follows POSIX semantics, which means that
kill(-1, SIGKILL) will kill the calling process too. Since nix has no
way to distinguish between the process successfully killing everything
and the process being killed by a rogue builder in that case, it can't
safely conclude that killUser was successful.
Luckily, the actual kill syscall takes a parameter that determines
whether POSIX semantics are followed, so we can call that syscall
directly and avoid the issue on Apple.
Signed-off-by: Shea Levy <shea@shealevy.com>
For instance, it's pointless to keep copy-from-other-stores running if
there are no other stores, or download-using-manifests if there are no
manifests. This also speeds things up because we don't send queries
to those substituters.
For example, given a derivation with outputs "out", "man" and "bin":
$ nix-build -A pkg
produces ./result pointing to the "out" output;
$ nix-build -A pkg.man
produces ./result-man pointing to the "man" output;
$ nix-build -A pkg.all
produces ./result, ./result-man and ./result-bin;
$ nix-build -A pkg.all -A pkg2
produces ./result, ./result-man, ./result-bin and ./result-2.
This is required on systemd, which mounts filesystems as "shared"
subtrees. Changes to shared trees in a private mount namespace are
propagated to the outside world, which is bad.
In a private PID namespace, processes have PIDs that are separate from
the rest of the system. The initial child gets PID 1. Processes in
the chroot cannot see processes outside of the chroot. This improves
isolation between builds. However, processes on the outside can see
processes in the chroot and send signals to them (if they have
appropriate rights).
Since the builder gets PID 1, it serves as the reaper for zombies in
the chroot. This might turn out to be a problem. In that case we'll
need to have a small PID 1 process that sits in a loop calling wait().
I was bitten one time too many by Python modifying the Nix store by
creating *.pyc files when run as root. On Linux, we can prevent this
by setting the immutable bit on files and directories (as in ‘chattr
+i’). This isn't supported by all filesystems, so it's not an error
if setting the bit fails. The immutable bit is cleared by the garbage
collector before deleting a path. The only tricky aspect is in
optimiseStore(), since it's forbidden to create hard links to an
immutable file. Thus optimiseStore() temporarily clears the immutable
bit before creating the link.
unreachable paths. This matters when using --max-freed etc.:
unreachable paths could become reachable again, so it's nicer to
keep them if there is "real" garbage to be deleted. Also, don't use
readDirectory() but read the Nix store and delete invalid paths in
parallel. This reduces GC latency on very large Nix stores.
exception handler, otherwise throw an exception. We need to ignore
write errors in exception handlers to ensure that cleanup code runs
to completion if the other side of stderr has been closed
unexpectedly.
it requires a certain feature on the build machine, e.g.
requiredSystemFeatures = [ "kvm" ];
We need this in Hydra to make sure that builds that require KVM
support are forwarded to machines that have KVM support. Probably
this should also be enforced for local builds.
Jörg Thalheim