If a NAR is already in the store, addToStore doesn't read the source
which makes the protocol go out of sync. This happens for example when
two client try to nix-copy-closure the same derivation at the same time.
Introduce the SizeSource which allows to bound how much data is being
read from a source. It also contains a drainAll() function to discard
the rest of the source, useful to keep the nix protocol in sync.
With this patch, and this file I called `log.py`:
#!/usr/bin/env nix-shell
#!nix-shell -i python3 -p python3 --pure
import sys
from pprint import pprint
stack = []
timestack = []
for line in open(sys.argv[1]):
components = line.strip().split(" ", 2)
if components[0] != "function-trace":
continue
direction = components[1]
components = components[2].rsplit(" ", 2)
loc = components[0]
_at = components[1]
time = int(components[2])
if direction == "entered":
stack.append(loc)
timestack.append(time)
elif direction == "exited":
dur = time - timestack.pop()
vst = ";".join(stack)
print(f"{vst} {dur}")
stack.pop()
and:
nix-instantiate --trace-function-calls -vvvv ../nixpkgs/pkgs/top-level/release.nix -A unstable > log.matthewbauer 2>&1
./log.py ./log.matthewbauer > log.matthewbauer.folded
flamegraph.pl --title matthewbauer-post-pr log.matthewbauer.folded > log.matthewbauer.folded.svg
I can make flame graphs like: http://gsc.io/log.matthewbauer.folded.svg
---
Includes test cases around function call failures and tryEval. Uses
RAII so the finish is always called at the end of the function.
Make curl's low speed limit configurable via stalled-download-timeout.
Before, this limit was five minutes without receiving a single byte.
This is much too long as if the remote end may not have even
acknowledged the HTTP request.
POSIX file locks are essentially incompatible with multithreading. BSD
locks have much saner semantics. We need this now that there can be
multiple concurrent LocalStore::buildPaths() invocations.
This currently fails because we're using POSIX file locks. So when the
garbage collector opens and closes its own temproots file, it causes
the lock to be released and then deleted by another GC instance.
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:
DRV_PATH=/nix/store/drv-that-has-been-built.drv
OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev
This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).
This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:
[nix]$ ./inst/bin/nix-build ./test.nix
these derivations will be built:
/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
hello!
bye!
running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
post-build-hook: + sleep 1
post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: + sleep 1
post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
post-build-hook: + sleep 1
post-build-hook: + printf 'very important stuff'
/nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
[nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
my-example-derivation> hello!
my-example-derivation> bye!
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
my-example-derivation (post)> + sleep 1
my-example-derivation (post)> + printf 'very important stuff'
[1 built, 0.0 MiB DL]
Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
startProcess does not appear to send the exit code to the helper
correctly. Not sure why this is, but it is probably safe to just
fallback on all sandbox errors.
This ensures that stdenv / setup hooks take $IN_NIX_SHELL into
account. For example, stdenv only sets
NIX_SSL_CERT_FILE=/no-cert-file.crt if we're not in a shell.
When sandbox-fallback = true (the default), the Nix builder will fall
back to disabled sandbox mode when the kernel doesn’t allow users to
set it up. This prevents hard errors from occuring in tricky places,
especially the initial installer. To restore the previous behavior,
users can set:
sandbox-fallback = false
in their /etc/nix/nix.conf configuration.
Some kernels disable "unpriveleged user namespaces". This is
unfortunate, but we can still use mount namespaces. Anyway, since each
builder has its own nixbld user, we already have most of the benefits
of user namespaces.
This replaces 'nix-env --set'. For example:
$ nix build --profile /nix/var/nix/profiles/system \
~/Misc/eelco-configurations:nixosConfigurations.vyr.config.system.build.toplevel
updates the NixOS system profile from a flake.
This could have been a separate command (e.g. 'nix set-profile') but
1) '--profile' is pretty similar to '--out-link'; and 2) '--profile'
could be useful for other command (like 'nix dev-shell').
This is a much simpler fix to the 'error 9 while decompressing xz
file' problem than 78fa47a7f0. We just
do a ranged HTTP request starting after the data that we previously
wrote into the sink.
Fixes#2952, #379.
Our use of boost::coroutine2 depends on -lboost_context,
which in turn depends on `-lboost_thread`, which in turn depends
on `-lboost_system`.
I suspect that this builds on nix only because of low-level hacks
like NIX_LDFLAGS.
This commit passes the proper linker flags, thus fixing bootstrap
builds on non-nix distributions like Ubuntu 16.04.
With these changes, I can build Nix on Ubuntu 16.04 using:
./bootstrap.sh
./configure --prefix=$HOME/editline-prefix \
--disable-doc-gen \
CXX=g++-7 \
--with-boost=$HOME/boost-prefix \
EDITLINE_CFLAGS=-I$HOME/editline-prefix/include \
EDITLINE_LIBS=-leditline \
LDFLAGS=-L$HOME/editline-prefix/lib
make
where
* g++-7 comes from gcc-7 from
https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test,
* editline 1.14 from https://github.com/troglobit/editline/releases/tag/1.14.0
was installed into `$HOME/editline-prefix`
(because Ubuntu 16.04's `editline` is too old to have the function nix uses),
* boost 1.66 from
https://www.boost.org/doc/libs/1_66_0/more/getting_started/unix-variants.html
was installed into $HOME/boost-prefix (because Ubuntu 16.04 only has 1.58)
$ sudo ./inst/bin/nix-instantiate -E '"${./.git}"'
error: The path name '.git' is invalid: it is illegal to start the
name with a period. Path names are alphanumeric and can include the
symbols +-._?= and must not begin with a period. Note: If '.git' is a
source file and you cannot rename it on disk,
builtins.path { name = ... } can be used to give it an alternative
name.
If multiple builds with fail with different errors it will be reflected
in the status code.
eg.
103 => timeout + hash mismatch
105 => timeout + check mismatch
106 => hash mismatch + check mismatch
107 => timeout + hash mismatch + check mismatch
Setting `http2 = false` in nix config (e.g. /etc/nix/nix.conf)
had no effect, and `nix-env -vvvvv -i hello` still downloaded .nar
packages using HTTP/2.
In `src/libstore/download.cc`, the `CURL_HTTP_VERSION_2TLS` option was
being explicitly set when `downloadSettings.enableHttp2` was `true`,
but, `CURL_HTTP_VERSION_1_1` option was not being explicitly set when
`downloadSettings.enableHttp2` was `false`.
This may be because `https://curl.haxx.se/libcurl/c/libcurl-env.html` states:
"You have to set this option if you want to use libcurl's HTTP/2 support."
but, also, in the changelog, states:
"DEFAULT
Since curl 7.62.0: CURL_HTTP_VERSION_2TLS
Before that: CURL_HTTP_VERSION_1_1"
So, the default setting for `libcurl` is HTTP/2 for version >= 7.62.0.
In this commit, option `CURLOPT_HTTP_VERSION` is explicitly set to
`CURL_HTTP_VERSION_1_1` when `downloadSettings.enableHttp2` nix config
setting is `false`.
This can be tested by running `nix-env -vvvvv -i hello | grep HTTP`
The default nsswitch.conf(5) file in most distros can handle many
different things including host name, user names, groups, etc. In Nix,
we want to limit the amount of impurities that come from these things.
As a result, we should only allow nss to be used for gethostbyname(3)
and getservent(3).
/cc @Ericson2314
'updateCV.notify_one()' does nothing if the update thread is not
waiting for updateCV (in particular this happens when it is sleeping
on quitCV). So also set a variable to ensure that the update isn't
lost.
--no-net causes tarballTtl to be set to the largest 32-bit integer,
which causes comparison like 'time + tarballTtl < other_time' to
fail on 32-bit systems. So cast them to 64-bit first.
https://hydra.nixos.org/build/95076624
(cherry picked from commit 29ccb2e969)
This flag
* Disables substituters.
* Sets the tarball-ttl to infinity (ensuring e.g. that the flake
registry and any downloaded flakes are considered current).
* Disables retrying downloads and sets the connection timeout to the
minimum. (So it doesn't completely disable downloads at the moment.)
(cherry picked from commit 8ea842260b)
Once we've started writing data to a Sink, we can't restart a download
request, because then we end up writing duplicate data to the
Sink. Therefore we shouldn't handle retries in Downloader but at a
higher level (in particular, in copyStorePath()).
Fixes#2952.
(cherry picked from commit a67cf5a358)
Also, make fetchGit and fetchMercurial update allowedPaths properly.
(Maybe the evaluator, rather than the caller of the evaluator, should
apply toRealPath(), but that's a bigger change.)
(cherry picked from commit 5c34d66538)
Once we've started writing data to a Sink, we can't restart a download
request, because then we end up writing duplicate data to the
Sink. Therefore we shouldn't handle retries in Downloader but at a
higher level (in particular, in copyStorePath()).
Fixes#2952.
--no-net causes tarballTtl to be set to the largest 32-bit integer,
which causes comparison like 'time + tarballTtl < other_time' to
fail on 32-bit systems. So cast them to 64-bit first.
https://hydra.nixos.org/build/95076624
This flag
* Disables substituters.
* Sets the tarball-ttl to infinity (ensuring e.g. that the flake
registry and any downloaded flakes are considered current).
* Disables retrying downloads and sets the connection timeout to the
minimum. (So it doesn't completely disable downloads at the moment.)
This allows many programs (e.g. gcc, clang, cmake) to print colorized
log output (assuming $TERM is set to a value like "xterm").
There are other ways to get colors, in particular setting
CLICOLOR_FORCE, but they're less widely supported and can break
programs that parse tool output.
Jonas Chevalier
