Commit graph

4015 commits

Author SHA1 Message Date
Eelco Dolstra 33db1d35ae
Merge pull request #2582 from LnL7/fetchgit-refs
fetchGit: allow fetching explicit refs
2019-07-02 15:44:31 +02:00
Graham Christensen 17d3ec3405
checkStoreName: give more precise/verbose error information
$ sudo ./inst/bin/nix-instantiate -E '"${./.git}"'
error: The path name '.git' is invalid: it is illegal to start the
name with a period. Path names are alphanumeric and can include the
symbols +-._?= and must not begin with a period. Note: If '.git' is a
source file and you cannot rename it on disk,
builtins.path { name = ... } can be used to give it an alternative
name.
2019-07-02 08:41:53 -04:00
Daiderd Jordan a52c331edb
build: replace 100 offset for build exit codes 2019-07-02 00:12:38 +02:00
Daiderd Jordan cbf84bcce7
build: use binary mask for build status flags
If multiple builds with fail with different errors it will be reflected
in the status code.

eg.

	103 => timeout + hash mismatch
	105 => timeout + check mismatch
	106 => hash mismatch + check mismatch
	107 => timeout + hash mismatch + check mismatch
2019-07-02 00:12:38 +02:00
Daiderd Jordan 97baf32fbc
build: add exit code for hash and check mismatches
Makes it easier to identify the failure reason in other tooling, eg.
differentiate between a non-deterministic --check vs a failed build.

	$ nix-build '<nix/fetchurl.nix>' --argstr url http://example.org --argstr sha256 0000000000000000000000000000000000000000000000000000
	hash mismatch in fixed-output derivation '/nix/store/nzi9ck45rwlxzcwr25is7qlf3hs5xl83-example.org':
	  wanted: sha256:0000000000000000000000000000000000000000000000000000
	  got:    sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm
	$ echo $?
	102

	$ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" {} "date +%s > $out"' --check
	warning: rewriting hashes in '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo'; cross fingers
	error: derivation '/nix/store/mggc8dz13ackb49qca6m23zq4fpq132q-foo.drv' may not be deterministic: output '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo' differs
	$ echo $?
	104
2019-07-02 00:12:34 +02:00
Aniket Deshpande ec58ba38c5
Fix http2 = false having no effect. Fixes #2971.
Setting `http2 = false` in nix config (e.g. /etc/nix/nix.conf)
had no effect, and `nix-env -vvvvv -i hello` still downloaded .nar
packages using HTTP/2.

In `src/libstore/download.cc`, the `CURL_HTTP_VERSION_2TLS` option was
being explicitly set when `downloadSettings.enableHttp2` was `true`,
but, `CURL_HTTP_VERSION_1_1` option was not being explicitly set when
`downloadSettings.enableHttp2` was `false`.

This may be because `https://curl.haxx.se/libcurl/c/libcurl-env.html` states:
"You have to set this option if you want to use libcurl's HTTP/2 support."
but, also, in the changelog, states:
"DEFAULT
Since curl 7.62.0: CURL_HTTP_VERSION_2TLS
Before that: CURL_HTTP_VERSION_1_1"

So, the default setting for `libcurl` is HTTP/2 for version >= 7.62.0.

In this commit, option `CURLOPT_HTTP_VERSION` is explicitly set to
`CURL_HTTP_VERSION_1_1` when `downloadSettings.enableHttp2` nix config
setting is `false`.

This can be tested by running `nix-env -vvvvv -i hello | grep HTTP`
2019-06-28 20:44:46 +05:30
Eelco Dolstra 6847c92788
Fix macOS build failure
Issue #2976.
2019-06-28 15:38:23 +02:00
Matthew Bauer ec0087df0a Don’t use entire /etc/nsswitch.conf file
The default nsswitch.conf(5) file in most distros can handle many
different things including host name, user names, groups, etc. In Nix,
we want to limit the amount of impurities that come from these things.
As a result, we should only allow nss to be used for gethostbyname(3)
and getservent(3).

/cc @Ericson2314
2019-06-27 14:31:29 -04:00
Eelco Dolstra 324a5dc92f
ProgressBar: Fix updating
'updateCV.notify_one()' does nothing if the update thread is not
waiting for updateCV (in particular this happens when it is sleeping
on quitCV). So also set a variable to ensure that the update isn't
lost.
2019-06-25 21:59:20 +02:00
Eelco Dolstra 88571219d9
nix-channel: Don't fetch binary-cache-url
This has been ignored since the Perl->C++ rewrite.
2019-06-25 13:27:16 +02:00
Eelco Dolstra 09dde33c19
Automatically use --no-net if there are no network interfaces
(cherry picked from commit 04a5976996)
2019-06-25 12:45:59 +02:00
Eelco Dolstra 5600b070a7
Add "warning" verbosity level
This ensures that "nix" shows warnings. Previously these were hidden
because they were at "info" level.

(cherry picked from commit 615a9d031d)
2019-06-25 12:44:17 +02:00
Eelco Dolstra 64ec087f58
Fix 32-bit overflow with --no-net
--no-net causes tarballTtl to be set to the largest 32-bit integer,
which causes comparison like 'time + tarballTtl < other_time' to
fail on 32-bit systems. So cast them to 64-bit first.

https://hydra.nixos.org/build/95076624
(cherry picked from commit 29ccb2e969)
2019-06-24 22:16:43 +02:00
Eelco Dolstra f8b30338ac
Refactor downloadCached() interface
(cherry picked from commit df3f5a78d5)
2019-06-24 22:12:26 +02:00
Eelco Dolstra 7b9c68766d
Add '--no-net' convenience flag
This flag

* Disables substituters.

* Sets the tarball-ttl to infinity (ensuring e.g. that the flake
  registry and any downloaded flakes are considered current).

* Disables retrying downloads and sets the connection timeout to the
  minimum. (So it doesn't completely disable downloads at the moment.)

(cherry picked from commit 8ea842260b)
2019-06-24 22:07:29 +02:00
Eelco Dolstra 78fa47a7f0
Fix 'error 9 while decompressing xz file'
Once we've started writing data to a Sink, we can't restart a download
request, because then we end up writing duplicate data to the
Sink. Therefore we shouldn't handle retries in Downloader but at a
higher level (in particular, in copyStorePath()).

Fixes #2952.

(cherry picked from commit a67cf5a358)
2019-06-24 21:59:51 +02:00
Eelco Dolstra 2fef4dd296
Downloader: Propagate exceptions from decompressionSink->finish()
(cherry picked from commit 15fa70cd1b)
2019-06-24 21:59:31 +02:00
Eelco Dolstra b43e1e186e
CachedDownloadResult: Include store path
Also, make fetchGit and fetchMercurial update allowedPaths properly.

(Maybe the evaluator, rather than the caller of the evaluator, should
apply toRealPath(), but that's a bigger change.)

(cherry picked from commit 5c34d66538)
2019-06-24 21:59:27 +02:00
Eelco Dolstra dc29e9fb47
downloadCached: Return ETag
(cherry picked from commit 529add316c)
2019-06-24 21:58:33 +02:00
Eelco Dolstra 94f11d0a61
Fix abort in fromTOML
Fixes #2969.
2019-06-24 17:09:09 +02:00
Eelco Dolstra 3cc1125595
Another attempt at getting pseudoterminals to work on macOS 2019-06-17 08:08:04 +02:00
Eelco Dolstra 2743bf0bb1 Hopefully fix macOS tests 2019-06-16 20:02:40 +02:00
Eelco Dolstra 82ca6ef390 Set $TERM 2019-06-16 17:36:30 +02:00
Eelco Dolstra e84c265645 Run builds in a pseudo-terminal
This allows many programs (e.g. gcc, clang, cmake) to print colorized
log output (assuming $TERM is set to a value like "xterm").

There are other ways to get colors, in particular setting
CLICOLOR_FORCE, but they're less widely supported and can break
programs that parse tool output.
2019-06-16 17:36:30 +02:00
Eelco Dolstra b693029ca0
Style fix 2019-06-16 09:43:20 +02:00
Eelco Dolstra 26bc876ae6
nix: Add -L alias for --print-build-logs 2019-06-15 16:45:00 +02:00
Eelco Dolstra 34fa8ce917
nix: Support -j flag 2019-06-15 16:34:06 +02:00
Félix Baylac-Jacqué 9e0f5f803f
Daemon: warn when an untrusted user cannot override a setting
In a daemon-based Nix setup, some options cannot be overridden by a
client unless the client's user is considered trusted.

Currently, if an untrusted user tries to override one of those
options, we are silently ignoring it.

This can be pretty confusing in certain situations.

e.g. a user thinks he disabled the sandbox when in reality he did not.

We are now sending a warning message letting know the user some options
have been ignored.

Related to #1761.
2019-06-15 03:59:12 -04:00
Matthew Bauer 5011a52cf3 Just enable hack on macOS
This is not needed on linux at all! Tried to explain as much as I
understand with the problem.
2019-06-05 20:20:50 -04:00
Matthew Bauer 4a3e96281d Handle SIGWINCH in main thread
For the SIGWINCH signal to be caught, it needs to be set in sigaction
on the main thread. Previously, this was broken, and updateWindowSize
was never being called. Tested on macOS 10.14.
2019-06-05 00:40:45 -04:00
Eelco Dolstra aec545c20b Fix segfault in builtin fetchurl with hashed mirrors + SRI hashes 2019-06-01 15:27:43 +02:00
Ding Xiang Fei abdedcdb38 bump cpptoml to v0.1.1 2019-05-29 17:01:39 +08:00
Eelco Dolstra 22f2744afd
Iterate over references 2019-05-28 23:05:08 +02:00
Eelco Dolstra 9eaebbf575
Merge branch 'attrPaths' of https://github.com/NinjaTrappeur/nix 2019-05-28 22:59:05 +02:00
Eelco Dolstra 4d829916e7
Merge pull request #2800 from flokli/progress-bar-hide-unknown-expected
progress-bar: hide expected if expected is 0 (unknown)
2019-05-21 13:29:28 +02:00
Eelco Dolstra 8f6c72faee
Merge pull request #2810 from NixOS/print-build-logs
nix: Add --print-build-logs flag
2019-05-15 20:38:38 +02:00
Eelco Dolstra 66b8a62101 nix: Add --print-build-logs flag
This causes 'nix' to print build log output to stderr rather than
showing the last log line in the progress bar. Log lines are prefixed
by the name of the derivation (minus the version string), e.g.

  binutils> make[1]: Leaving directory '/build/binutils-2.31.1'
  binutils-wrapper> unpacking sources
  binutils-wrapper> patching sources
  ...
  binutils-wrapper> Using dynamic linker: '/nix/store/kr51dlsj9v5cr4n8700jliyz8v5b2q7q-bootstrap-stage0-glibc/lib/ld-linux-x86-64.so.2'
  bootstrap-stage2-gcc-wrapper> unpacking sources
  ...
  linux-headers> unpacking sources
  linux-headers> unpacking source archive /nix/store/8javli69jhj3bkql2c35gsj5vl91p382-linux-4.19.16.tar.xz
2019-05-15 17:33:56 +02:00
Eelco Dolstra 5f6840fbb4
Merge pull request #2802 from LnL7/fix-needs-hashrewrite
build: move needsHashRewrite initialization to startBuilder
2019-05-13 08:53:48 +02:00
Graham Christensen f1b8e9efe7
runProgram: Uncomment chdir support 2019-05-12 17:03:01 -04:00
Daiderd Jordan ce02fc74b2
build: make needsHashRewrite a method 2019-05-12 22:56:27 +02:00
Graham Christensen 73b797c207
handleDiffHook: stop passing allowVfork 2019-05-12 13:44:22 -04:00
Graham Christensen b4a05edbfe
runProgram: support gid, uid, chdir 2019-05-12 13:17:27 -04:00
Graham Christensen dde8eeb39a
chdir, setgroups 2019-05-12 13:17:26 -04:00
Graham Christensen 6df61db060
diff hook: execute as the build user, and pass the temp dir 2019-05-12 13:17:26 -04:00
Graham Christensen c78686e411
build: run diff-hook under --check and document diff-hook 2019-05-12 13:17:24 -04:00
Daiderd Jordan ff6867ab94
build: move needsHashRewrite initialization to startBuilder
The value of useChroot is not set yet in the constructor, resulting in
hash rewriting being enabled in certain cases where it should not be.

Fixes #2801
2019-05-12 15:53:40 +02:00
Florian Klink 6ade7ec022 progress-bar: hide expected if expected is 0 (unknown)
Sometimes, "expected" can be "0", but in fact means "unknown".

This is for example the case when downloading a file while the http
server doesn't send the `Content-Length` header, like when running `nix
build` pointing to a nixpkgs checkout streamed from GitHub:

⇒  nix build -f https://github.com/NixOS/nixpkgs/archive/master.tar.gz hello
[1.8/0.0 MiB DL] downloading 'https://github.com/NixOS/nixpkgs/archive/master.tar.gz'

In that case, don't show that weird progress bar, but only the (slowly
increasing) downloaded size ("done").

⇒  nix build -f https://github.com/NixOS/nixpkgs/archive/master.tar.gz hello
[1.8 MiB DL] downloading 'https://github.com/NixOS/nixpkgs/archive/master.tar.gz'

This commit also updates fmt calls with three numbers (when something is
currently 'running' too) - I'm not sure if this can be provoked, but
showing "0" as expected doesn't make any sense, as we're obviously doing
more than nothing.
2019-05-11 12:34:39 +02:00
Eelco Dolstra 7c6391ddc7
nix-shell: Don't absolutize '-p' expressions
This prevents spurious syscalls like

  25011 lstat("/home/eelco/with import <nixpkgs> { }; (pkgs.runCommandCC or pkgs.runCommand) \"shell\" { buildInputs = [ (hello) ]; } \"\"", 0x7ffe9c67f580) = -1 ENOENT (No such file or directory)
2019-05-08 14:29:36 +02:00
Eelco Dolstra 92caa60c49
Merge branch 'repl/ctrlc' of https://github.com/xbreak/nix 2019-05-08 10:18:45 +02:00
Eelco Dolstra 3e940bbf2d
Merge pull request #2790 from samueldr/fix/minor-comment-NIX_ROOT_FINDER
findRootsNoTemp: fixes comment about findRuntimeRoots
2019-05-08 10:13:09 +02:00
Daniel Schaefer 3f192ac80c Add builtins.hashFile
For text files it is possible to do it like so:
`builtins.hashString "sha256" (builtins.readFile /tmp/a)`
but that doesn't work for binary files.

With builtins.hashFile any kind of file can be conveniently hashed.
2019-05-03 17:23:36 +02:00
Eelco Dolstra f9a2ea4486
Fix "Bad system call" running i686-linux binaries on x86_64-linux
To determine which seccomp filters to install, we were incorrectly
using settings.thisSystem, which doesn't denote the actual system when
--system is used.

Fixes #2791.
2019-05-03 10:48:33 +02:00
Samuel Dionne-Riel cbc7d9a412 findRootsNoTemp: fixes comment about findRuntimeRoots
The NIX_ROOT_FINDER environment variable was removed in
3c46fe62b8 when porting from perl to C.
2019-04-30 22:43:24 -04:00
Daiderd Jordan b614e0e53d
build: only skip hash rewriting for sandboxing on linux
The sandbox on darwin, and possibly other future platforms, doesn't have
enough isolation to redirect outputs without hash rewriting.
2019-04-17 23:41:23 +02:00
Eelco Dolstra bb6e6923f2 Add environment variable NIX_SHOW_SYMBOLS for dumping the symbol table 2019-04-11 23:04:13 +02:00
Eelco Dolstra 41ba5135e0 primeCache(): Barf if builds are needed but not allowed
Fixes #2753.
2019-04-01 21:09:49 +02:00
Eelco Dolstra 2bc6304793 getMachines(): Cache result 2019-04-01 21:09:35 +02:00
Eelco Dolstra f32fbf952d Fix Bison 3.3 warning 2019-03-27 21:09:31 +01:00
Eelco Dolstra 5c05c238e6
Merge pull request #1828 from zimbatm/isPath
Add isPath primop
2019-03-25 19:58:11 +01:00
Will Dietz 42e2d5e7b7
store-api.hh: add missing include for unordered_map 2019-03-25 09:18:30 +01:00
Jonas Chevalier 514b3c7f83
Add isPath primop
this is added for completeness' sake since all the other possible
`builtins.typeOf` results have a corresponding `builtins.is<Type>`
2019-03-24 11:36:49 +01:00
xbreak fcd7660976 repl: Restore CTRL-C behaviour
Install signal handler during `readline` to handle SIGINT to abort
partially typed expressions.
2019-03-24 09:39:48 +00:00
Linus Heckemann 2aa89daab3 eval: improve type description for primops and applied primops
This can make type errors a little easier to understand.
2019-03-21 15:31:46 +01:00
Chaker Benhamed 81a23fa7e2 Add a 5 seconds of timeouts to connect to S3.
The default is 1000ms, but we can hit it a lot of we don't have direct
link to AWS (e.g. using VPN).
2019-03-15 13:23:58 +01:00
Eelco Dolstra 1f64f4c7c8
pkg-config files: Use c++17 2019-03-14 14:11:12 +01:00
Eelco Dolstra ef52ccf035
experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Eelco Dolstra 86f3b94c8c
nix-store --gc --print-roots: Sort output 2019-03-14 13:59:10 +01:00
Eelco Dolstra 578ed7a259
findRoots(): Don't censor for trusted users
They're pretty much root anyway.
2019-03-14 13:53:34 +01:00
Eelco Dolstra 53522cb6ac
findRoots(): Add 'censor' parameter
This is less brittle than filtering paths after the fact in
nix-daemon.
2019-03-14 13:53:34 +01:00
Eelco Dolstra a3f37d87ea
findRuntimeRoots: Simplify/fix handling of /proc files
Scanning of /proc/<pid>/{exe,cwd} was broken because '{memory:' was
prepended twice. Also, get rid of the whole '{memory:...}' thing
because it's unnecessary, we can just list the file in /proc directly.
2019-03-14 13:30:25 +01:00
Guillaume Maudoux 4f4391193c Fix warning about unused variable 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 38ee16ae9c Unify internal findRootsXxx() api 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 9d7221183a unify naming of roots as links & targets 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 5c56570726 Also obfuscate the number of memory roots 2019-03-10 00:56:09 +01:00
Guillaume Maudoux fc02b1b3ee Also print rooted path in nix-store -q --roots 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 8574b70342 fixup! Make roots a map of store paths to pinning links 2019-03-10 00:56:09 +01:00
Guillaume Maudoux ebc86550f9 Make roots a map of store paths to pinning links
This new structure makes more sense as there may be many sources rooting
the same store path. Many profiles can reference the same path but this
is even more true with /proc/<pid>/maps where distinct pids can and
often do map the same store path.
This implementation is also more efficient as the `Roots` map contains
only one entry per rooted store path.
2019-03-10 00:56:09 +01:00
Guillaume Maudoux a17f86ce3a Obfuscate memory roots for non-root users 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 43331d6344 detail memory roots 2019-03-10 00:56:09 +01:00
volth fff8db205c
canBuildLocally: check for features
It could happen that the local builder match the system but lacks some features.
Now it results a failure.
The fix gracefully excludes the local builder from the set of available builders for derivation which requires the feature, so the derivation is built on remote builders only (as though it has incompatible system, like ```aarch64-linux``` when local is x86)
2019-03-06 05:03:25 +00:00
Will Dietz 0963479741
archive.cc: ignore more posix_fallocate "not supported" error codes
Fixes w/musl.
2019-03-01 10:31:17 +08:00
Dzmitry Zaitsau 06d6335987 fix indentation 2019-02-26 11:07:37 +01:00
Dzmitry Zaitsau ac200c3678 Apply param values on S3Helper initialization 2019-02-25 18:06:19 +01:00
Dzmitry Zaitsau 56c18c67d9 Extend S3 URL parsing with parameters extraction 2019-02-25 18:00:55 +01:00
Dzmitry Zaitsau 07f992a74b Extract and expose splitUriAndParams function
which splits a URL into localtor and parameter parts
2019-02-25 17:59:26 +01:00
Eelco Dolstra d8fe447139
Merge pull request #2579 from catern/dumpdb
nix-store: make --dump-db take a list of paths to dump
2019-02-14 13:08:02 +01:00
Eelco Dolstra 7a7ec22298
Merge pull request #2628 from shlevy/context-introspection
Context introspection
2019-02-12 17:57:35 +01:00
Eelco Dolstra 01d07b1e92
Revert "Restore parent mount namespace before executing a child process"
This reverts commit a0ef21262f. This
doesn't work in 'nix run' and nix-shell because setns() fails in
multithreaded programs, and Boehm GC mark threads are uncancellable.

Fixes #2646.
2019-02-05 10:49:19 +01:00
Shea Levy b30be6b450
Add builtins.appendContext.
A partner of builtins.getContext, useful for the same reasons.
2019-01-31 08:52:23 -05:00
Spencer Baugh 5f1891b795 nix-store: make --dump-db take a list of paths to dump
Inside a derivation, exportReferencesGraph already provides a way to
dump the Nix database for a specific closure. On the command line,
--dump-db gave us the same information, but only for the entire Nix
database at once.

With this change, one can now pass a list of paths to --dump-db to get
the Nix database dumped for just those paths. (The user is responsible
for ensuring this is a closure, like for --export).

Among other things, this is useful for deploying a closure to a new
host without using --import/--export; one can use tar to transfer the
store paths, and --dump-db/--load-db to transfer the validity
information. This is useful if the new host doesn't actually have Nix
yet, and the closure that is being deployed itself contains Nix.
2019-01-23 01:24:34 +00:00
Eelco Dolstra 7cc1a2593e
unsupported(): Show the name of the unsupported operation 2019-01-18 13:34:53 +01:00
Shea Levy 1d757292d0
Add builtins.getContext.
This can be very helpful when debugging, as well as enabling complex
black magic like surgically removing a single dependency from a
string's context.
2019-01-14 11:27:10 -05:00
Shea Levy 087be7281a
Treat plain derivation paths in context as normal paths.
Previously, plain derivation paths in the string context (e.g. those
that arose from builtins.storePath on a drv file, not those that arose
from accessing .drvPath of a derivation) were treated somewhat like
derivaiton paths derived from .drvPath, except their dependencies
weren't recursively added to the input set. With this change, such
plain derivation paths are simply treated as paths and added to the
source inputs set accordingly, simplifying context handling code and
removing the inconsistency. If drvPath-like behavior is desired, the
.drv file can be imported and then .drvPath can be accessed.

This is a backwards-incompatibility, but storePath is never used on
drv files within nixpkgs and almost never used elsewhere.
2019-01-13 11:29:55 -05:00
Eelco Dolstra c7bf1cdb4e
Merge pull request #2608 from dtzWill/fix/issue-2546
EvalState::resetFileCache: clear parse cache as well as eval cache
2019-01-10 20:56:31 +01:00
Eelco Dolstra 30906122e2
Merge pull request #2617 from veprbl/pr/V547_nix_repl.cc
remove some dead code in nix/repl.cc
2019-01-10 20:55:27 +01:00
Eelco Dolstra 013dd28b15
Merge pull request #2601 from dtzWill/update/json-3.5.0
Update nlohmann-json 3.0.1 -> 3.5.0
2019-01-07 23:13:35 +01:00
John Ericson fef9f5653b Remove mentions of libformat, it no longer exists 2019-01-05 14:31:29 -05:00
John Ericson e10d6ed2a7 brotli is only used as a library now 2019-01-05 14:25:54 -05:00
Will Dietz 21ea00d3ec EvalState::resetFileCache: clear parse cache as well as eval cache
Fixes #2546.

(at least the basic reproduction I've been testing)
2018-12-31 10:18:28 -06:00
Dmitry Kalinkin 93c9ba3e78 remove some dead code in nix/repl.cc
Fixes a static analyzer warning:

Expression 'isDrv' is always false.

Fixes: 3beb6f6e7 ('Show derivations more concisely')
2018-12-25 22:38:40 +03:00