Commit graph

7288 commits

Author SHA1 Message Date
Eelco Dolstra ea9df6fe51 Shut down write side before draining the read side
This is important if the remote side *does* execute
nix-store/nix-daemon successfully, but stdout is polluted
(e.g. because the remote user's bashrc script prints something to
stdout). In that case we have to shutdown the write side to force the
remote nix process to exit.
2021-09-23 18:01:04 +02:00
Eelco Dolstra 994348e9e0 SSHStore / LegacySSHStore: Show a better error message if the remote is "nologin"
Instead of

  error: serialised integer 7161674624452356180 is too large for type 'j'

we now get

  error: 'nix-store --serve' protocol mismatch from 'sshtest@localhost', got 'This account is currently not available.'

Fixes https://github.com/NixOS/nixpkgs/issues/37287.
2021-09-23 17:50:29 +02:00
Eelco Dolstra 60642aa5e2 Remove risky char * 2021-09-23 12:07:50 +02:00
Eelco Dolstra 198fa786a1 Add some more instrumentation 2021-09-23 11:42:33 +02:00
Geoff Reedy 9d67332e4b Better eval error locations for interpolation and +
Previously, type or coercion errors for string interpolation, path
interpolation, and plus expressions were always reported at the
beginning of the outer expression. This leads to confusing evaluation
error messages making it hard to accurately diagnose and then fix the
error.

For example, errors were reported as follows.

```
cannot coerce an integer to a string
1| let foo = 7; in "bar" + foo
 |                 ^

cannot add a string to an integer
1| let foo = "bar"; in 4 + foo
 |                     ^

cannot coerce an integer to a string
1| let foo = 7; in "x${foo}"
 |                  ^
```

This commit changes the ExprConcatStrings expression vector to store a
sequence of expressions *and* their expansion locations so that error
locations can be reported accurately. For interpolation, the error is
reported at the beginning of the entire `${foo}`, not at the beginning
of `foo` because I thought this was slightly clearer. The previous
errors are now reported as:

```
cannot coerce an integer to a string
1| let foo = 7; in "bar" + foo
 |                         ^

cannot add a string to an integer
1| let foo = "bar"; in 4 + foo
 |                         ^

cannot coerce an integer to a string
1| let foo = 7; in "x${foo}"
 |                   ^
```

The error is reported at this kind of precise location even for
multi-line indented strings.

This probably helps with at least some of the cases mentioned in #561
2021-09-22 20:57:34 -06:00
Eelco Dolstra d1bf7431bb Revert "Merge pull request #4922 from nrdxp/default-submodules"
This reverts commit 6678e98411, reversing
changes made to 90b2dd570c.
2021-09-22 17:25:25 +02:00
Eelco Dolstra 8623a5b595 Disable IFD selectively
It's now disabled by default for the following:

* 'nix search' (this was already implied by read-only mode)
* 'nix flake show'
* 'nix flake check', but only on the hydraJobs output
2021-09-22 17:15:07 +02:00
Eelco Dolstra d8c10028d9 Make setDefault() typed 2021-09-22 14:15:35 +02:00
Eelco Dolstra ff28fffce2 Don't cache realiseContext() errors
Errors that depend on the configuration (such as whether
allow-import-from-derivation is set) should not be cached.
2021-09-22 14:00:56 +02:00
Eelco Dolstra bcd73ebf60 Add missing #include
Fixes #5282.
2021-09-22 12:14:50 +02:00
Eelco Dolstra 60cc975d22 Set input parent at construction time 2021-09-21 14:07:16 +02:00
Eelco Dolstra 06557299b3 Allow relative paths anywhere into the parent's store path 2021-09-21 13:45:11 +02:00
Eelco Dolstra 5cbb9c5406 path fetcher: Fix relative path check 2021-09-21 13:19:26 +02:00
Michael Bishop 374fe49ff7 set the PER_LINUX32 personality flag, when building for armv6l-linux or armv7l-linux
this prevents 32bit builds from detecting a 64bit kernel and picking the wrong target
2021-09-19 23:07:10 -03:00
Gregor Kleen fd67a0f927 Fix scheme argument to s3 cache 2021-09-17 10:45:19 +02:00
Geoff Reedy cbe9ddfd1a Include subpath in flake fingerprint
Without this, flakes within the same tree and same lock data will have
the same fingerprint and the eval cache for one flake will be
incorrectly used for another.
2021-09-16 15:58:21 -06:00
Eelco Dolstra 1ec4efa6c8
Merge pull request #5257 from edolstra/dirty-lock-file
If we can't write a lock file, pretend the top-level flake is dirty
2021-09-15 20:18:23 +02:00
Eelco Dolstra c17f3c5e69 Merge branch 'mh/fix-chroot-eval' of https://github.com/obsidiansystems/nix 2021-09-15 18:37:58 +02:00
Eelco Dolstra 027344ce7e If we can't write a lock file, pretend the top-level flake is dirty
Alternative to #4639. You can still read flake.lock, but at least in
reproducible workflows like NixOS configurations where you require a
non-dirty tree, evaluation will fail because there is no rev.
2021-09-15 18:31:42 +02:00
Eelco Dolstra 991cc53386 Revert "Disallow reading flake.lock"
This reverts commit e5596113f7.
2021-09-15 18:30:37 +02:00
Emily 56025ad3b1 sandbox: allow Rosetta 2 on Darwin
This allows sandboxed x86_64-darwin builds on aarch64-darwin.
2021-09-15 02:02:03 +01:00
Eelco Dolstra fda4efff87 Rename unnecessary git@ 2021-09-14 22:53:31 +02:00
Eelco Dolstra 5ee3ee1a6b
Merge pull request #5249 from edolstra/nix-profile
Add missing 'nix profile' subcommands
2021-09-14 22:27:19 +02:00
Eelco Dolstra e5596113f7 Disallow reading flake.lock
With --no-write-lock-file, it's possible that flake.lock is out of
sync with the actual inputs used by the evaluation. So doing fromJSON
(readFile ./flake.lock) will give wrong results.

Fixes #4639.
2021-09-14 21:09:11 +02:00
Eelco Dolstra b41968f15a nix profile history: Show profile date 2021-09-14 20:47:33 +02:00
Eelco Dolstra 4b738fc7a9 Add 'nix profile wipe-history' command 2021-09-14 20:35:41 +02:00
Eelco Dolstra f359b9981b Generations -> profile versions 2021-09-14 19:57:45 +02:00
Eelco Dolstra 229ad612b8 Fix quotes 2021-09-14 19:48:16 +02:00
Eelco Dolstra 817562e694 Add "nix profile rollback" command 2021-09-14 19:32:33 +02:00
Eelco Dolstra 1fbaf36729 nix flake show --json: Add type info
For extensibility, every leaf node is now an object that contains at
least a type field (e.g. "type": "derivation").
2021-09-14 17:18:29 +02:00
Eelco Dolstra f3259af73e Merge branch 'tomberek/show_json' of https://github.com/tomberek/nix 2021-09-14 16:58:35 +02:00
Eelco Dolstra d589a6aa8a
Merge pull request #5247 from edolstra/stablize-ca-references
Remove the 'ca-references' feature check
2021-09-14 15:42:50 +02:00
Eelco Dolstra 75837bb595 Respect NO_COLOR in Markdown output 2021-09-14 14:57:58 +02:00
Eelco Dolstra 7e4dd0e279
Merge pull request #5248 from edolstra/flake-clone
Fix 'nix flake clone' on github repos
2021-09-14 14:14:59 +02:00
Eelco Dolstra 01a4fa3b6e Advise using --extra-experimental-features instead of --experimental-features
Fixes #3737.
2021-09-14 13:56:36 +02:00
Eelco Dolstra 761ac9d584 Remove the 'ca-references' feature check
Fixes #3422.
Fixes #4425.
2021-09-14 13:53:20 +02:00
Eelco Dolstra 07996c4810 github fetcher: Don't use HEAD branch when cloning
Fixes #4394.
2021-09-14 13:38:45 +02:00
Eelco Dolstra d72d31d529 github fetcher: Use git+https for cloning
git+ssh only works if you have SSH access.
2021-09-14 13:38:05 +02:00
Eelco Dolstra a4ae601b44
Merge pull request #5245 from edolstra/warnings
Change warning messages from yellow to magenta
2021-09-14 11:46:34 +02:00
Eelco Dolstra 16d4922dd2
Merge pull request #5240 from edolstra/builtin-help
nix --help: Display help using lowdown instead of man
2021-09-14 11:31:23 +02:00
Eelco Dolstra 58b5036c54 Change warnings from yellow to magenta
This matches gcc and clang.
2021-09-14 10:42:29 +02:00
Eelco Dolstra 4ffda0af7c ANSI_YELLOW -> ANSI_WARNING 2021-09-14 10:42:29 +02:00
Eelco Dolstra 76e368a3b4 Fix macOS build 2021-09-14 10:07:31 +02:00
Eelco Dolstra 1ba993d07c Fix clang warning 2021-09-14 08:15:33 +02:00
Eelco Dolstra 4ed66735b6 RunOptions: Use designated initializers
Also get rid of _killStderr because it wasn't actually checked
anywhere.
2021-09-13 23:31:04 +02:00
Eelco Dolstra c3e9acd1c0 Remove tabs 2021-09-13 23:06:33 +02:00
kvtb c6fa7775de hashFile, hashString: realize context before calculation, and discard afterwards 2021-09-13 22:34:58 +02:00
Eelco Dolstra 49a932fb18 nix --help: Display help using lowdown instead of man
Fixes #4476.
Fixes #5231.
2021-09-13 14:45:21 +02:00
Eelco Dolstra 8796b1b5e3 Fix markdown error 2021-09-13 14:44:53 +02:00
Eelco Dolstra eadb45c4db Use Bindings::{get,need} instead of find 2021-09-13 13:53:24 +02:00
Eelco Dolstra a73be28717 renderMarkdownToTerminal(): Fix terminal width computation 2021-09-13 13:52:14 +02:00
Eelco Dolstra ff09acd1f9 Remove tabs 2021-09-10 11:00:50 +02:00
Eelco Dolstra 02ece164be Make installables const 2021-09-10 10:39:39 +02:00
Eelco Dolstra 1e43bc6dc0
Merge pull request #5224 from baloo/baloo/5089/force-nss_dns-load
preloadNSS: fixup nss_dns load
2021-09-09 10:37:41 +02:00
Tom Bereknyei dc25856d74 Ensure nix flake show produces valid json 2021-09-08 19:38:22 -04:00
Arthur Gautier 3b72741f23 preloadNSS: load NSS before threads are started
preloadNSS is not thread-safe, this commit moves it before we start the
first thread.

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-09-08 18:29:31 +00:00
Eelco Dolstra b71428c907 Merge branch 'fix-3976' of https://github.com/mkenigs/nix 2021-09-08 14:43:12 +02:00
Naïm Favier 7f0d177ce7
Add missing include in util.cc 2021-09-08 12:20:08 +02:00
Eelco Dolstra 9180239081
Merge pull request #5215 from ncfavier/patch-1
Fix use-registries logic in builtins.getFlake
2021-09-08 11:19:39 +02:00
Shea Levy 6678e98411
Merge pull request #4922 from nrdxp/default-submodules
libfetchers/git: fetch submodules by default
2021-09-07 05:48:23 -04:00
Arthur Gautier 0b42a0f781 preloadNSS: fixup nss_dns load
Before this commit, the dns lookup in preloadNSS would still go through
nscd. This did not have the effect of loading the nss_dns.so as expected
(nss_dns.so being out of reach from within the sandbox).

Should LOCALDOMAIN environment variable be defined, nss will completely
avoid nscd and will do its dns resolution on its own.

By temporarly setting LOCALDOMAIN variable before calling in NSS, we can
force NSS to load the shared libraries as expected.

Fixes #5089

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-09-06 23:13:48 +00:00
Naïm Favier b0d4190f19
Fix use-registries logic in builtins.getFlake 2021-09-05 15:28:44 +02:00
Eelco Dolstra 90b2dd570c Add FIXME 2021-09-02 15:00:52 +02:00
Georges Dubus bc6e7ca046 Don't use read-only mode for nix build --dry-run
In dry run mode, new derivations can't be create, so running the command on anything that has not been evaluated before results in an error message of the form `don't know how to build these paths (may be caused by read-only store access)`.

For comparison, the classical `nix-build --dry-run` doesn't use read-only mode.

Closes #1795

(cherry picked from commit 54525682df707742e58311c32e9c9cb18de1e31f)
2021-09-02 14:50:19 +02:00
Eelco Dolstra b2e8120d25 parseInstallables(): Parse store paths as store paths
If the store path contains a flake, this means that a command like
"nix path-info /path" will show info about /path, not about the
default output of the flake in /path. If you want the latter, you can
explicitly ask for it by doing "nix path-info path:/path".

Fixes #4568.
2021-09-02 14:18:04 +02:00
Eelco Dolstra 77ca5e951c
Merge pull request #5203 from NixOS/fix-invalid-lockfile-names
Don’t create lockfiles with an invalid path name
2021-09-02 11:03:18 +02:00
Lucas Hoffmann d948415b71
Docs: Fix syntax error in json example in man page 2021-09-02 10:47:38 +02:00
regnat 497225b07d Don’t create lockfiles with an invalid path name
Store paths are only allowed to contain a limited subset of the
alphabet, which doesn’t include `!`. So don’t create lockfiles that
contain this `!` character as that would otherwise confuse (and break)
the gc.

Fix #5176
2021-09-02 09:57:41 +02:00
Eelco Dolstra f73d911628 Style 2021-09-01 21:46:25 +02:00
Eelco Dolstra 2cb7a502a4 Merge branch 'short-circuit-subst' of https://github.com/Kha/nix 2021-09-01 21:45:04 +02:00
Eelco Dolstra 7ee639f9db
Merge pull request #5066 from Radvendii/master
add antiquotations to paths
2021-09-01 12:55:04 +02:00
Taeer Bar-Yam 1ffb9f1970 fix parse of /${foo}. was // + foo
I don't think this changes the way any program would behave, but it's a
cleaner internal representation.
2021-08-31 15:55:55 -04:00
Taeer Bar-Yam 9da8f5e25d path antiquotations: canonizePath -> canonicalizePath 2021-08-31 08:02:04 -04:00
Maximilian Bosch 50edbc4ddf
nix-store --serve: pass on settings.keepFailed from SSH store
When doing e.g.

    nix-build -A package --keep-failed --option \
      builders \
      'ssh://mfhydra?remote-store=/home/bosch/store x86_64-linux - 10 4 big-parallel'

this doesn't work properly because this build-setting is ignored.

I changed this behavior by passing the `settings.keepFailed` through the
serve-protocol to remote machines to make sure that I can introspect the
build-directory (which is particularly helpful when I have to look at a
`config.log` from a failed build for instance).
2021-08-31 13:11:46 +02:00
Eelco Dolstra f3b8b4040e
Merge pull request #5071 from NixOS/dont-send-xp-features-to-the-daemon
Don’t send the experimental-features to the daemon
2021-08-31 11:55:39 +02:00
Eelco Dolstra 50a35860ee TarArchive: Small refactoring 2021-08-30 17:02:51 +02:00
Eelco Dolstra c7a7652725 Don't segfault if archive_entry_pathname() returns null
Issues #4499.
2021-08-30 17:02:39 +02:00
Eelco Dolstra eda0fee160
Merge pull request #5175 from Pamplemousse/make
Don't overwrite user provided `lib*_LDFLAGS`
2021-08-30 12:44:29 +02:00
Eelco Dolstra 323cafcb4e
Merge pull request #5191 from hercules-ci/evalstate-lifetime-hygiene
EvalState lifetime hygiene
2021-08-30 12:23:09 +02:00
regnat b2bce915ad Don’t accept experimental features from the client
If the client sends an “experimental features” setting, just ignore it
2021-08-30 10:48:05 +02:00
Robert Hensing 92778a5f80 Tidy 2021-08-30 09:52:43 +02:00
Robert Hensing 8656b130ea Fix use after free with vImportedDrvToDerivation 2021-08-29 20:42:57 +02:00
Robert Hensing 8bc76acc7c Move vCallFlake into EvalState
This fixes a use-after-free bug:

1. s = new EvalState();
2. callFlake()
3. static vCallFlake now references s
4. delete s;
5. s2 = new EvalState();
6. callFlake()
7. static vCallFlake still references s
8. crash

Nix 2.3 did not have a problem with recreating EvalState.
2021-08-29 20:42:49 +02:00
Robert Hensing f10465774f Force all Pos* to be non-null
This fixes a class of crashes and introduces ptr<T> to make the
code robust against this failure mode going forward.

Thanks regnat for the idea of a ref<T> without overhead!

Closes #4895
Closes #4893
Closes #5127
Closes #5113
2021-08-29 18:11:58 +02:00
Pamplemousse a4c6d319a8 Don't overwrite user provided lib*_LDFLAGS
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-25 08:59:19 -07:00
Eelco Dolstra af94b54db3 Coding style 2021-08-23 13:05:42 +02:00
Eelco Dolstra c6b063c31a Merge branch 'fix/subflake-follows-fix' of https://github.com/ArctarusLimited/nix 2021-08-23 12:47:09 +02:00
Eelco Dolstra a93f72c084
Merge pull request #5160 from CertainLach/fix/invalid-characters-in-json-logger
Replace invalid characters in json logger
2021-08-23 11:10:55 +02:00
Tom Bereknyei d90582be33 Allow use of path and filterSource in flakes
As filterSource and path perform work, add paths to allowedPaths.
2021-08-22 18:45:42 -04:00
Yaroslav Bolyukin bbb3bcb165
Replace invalid characters in json logger
Fixes #5159

Signed-off-by: Yaroslav Bolyukin <iam@lach.pw>
2021-08-22 00:26:22 +03:00
Jan Tojnar 2d66a31f01 nix {bundle,run}: drop broken flake attr-path prefixes
“packages” was probably meant to be “packages.${system}.” but that
is already listed in `getDefaultFlakeAttrPathPrefixes` in `installables`,
which is probably why no one noticed it was broken.
2021-08-21 20:17:05 +02:00
Jan Tojnar d7b6c8f591 nix develop: Fix devShells lookup
It currently fails with the following error:

    error: flake 'git+file://…' does not provide attribute 'devShells.x86_64-linuxhaskell', 'packages.x86_64-linux.haskell', 'legacyPackages.x86_64-linux.haskell' or 'haskell'
2021-08-21 01:24:28 +02:00
Matthew Kenigsberg d7fe36116e
nix develop --phase: chdir to flake directory
For git+file and path flakes, chdir to flake directory so that phases
that expect to be in the flake directory can run

Fixes https://github.com/NixOS/nix/issues/3976
2021-08-19 15:42:13 -05:00
Tom Bereknyei a82de5b31b Enable JSON option to show flakes
`nix-flake show --json`
2021-08-17 23:18:19 -04:00
Alex Zero b3c424f5a6
Fix follows paths in subordinate lockfiles 2021-08-17 15:25:42 +01:00
Eelco Dolstra 2cd1a5b8f3
Merge pull request #5134 from pszubiak/fix-pkgconfig-install-path
Install pkg-config files in the correct location
2021-08-16 11:56:49 +02:00
Magic_RB 8438114399
Add ignored_acls setting
Signed-off-by: Magic_RB <magic_rb@redalder.org>
2021-08-14 21:46:09 +02:00
Jörg Thalheim 21d0334e49 fix typo in ca-error message 2021-08-14 00:18:45 +02:00
Piotr Szubiakowski 9f13cb31e8
Install pkg-config files in the correct location
Use `$(libdir)` while installing .pc files looks like a more generic
solution. For example, it will work for distributions like RHEL or
Fedora where .pc files are installed in `/usr/lib64/pkgconfig`.
2021-08-13 21:08:58 +00:00
Eelco Dolstra 6b8069b823 In flake lock file diffs, show the last-modified date of inputs if available
This is a bit more informative than just the hash.

Also, format the diffs a bit nicer.
2021-08-11 19:35:28 +02:00
Pamplemousse 0da416f820 15f4d4f follow up
* libstore: `bz2` should not be linked
  * libutil: `zlib.h` should not be included

Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-10 16:04:53 -07:00
Eelco Dolstra 8943e3176d
Merge pull request #5111 from Pamplemousse/clean
Minor maintenance cleaning
2021-08-09 20:05:03 +02:00
Pamplemousse 2de7a1fe67 libexpr: Squash similar conditions
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-09 10:10:11 -07:00
Sebastian Ullrich a0f97b1f54 Expand docstring of Store::querySubstitutablePathInfos 2021-08-09 15:58:50 +02:00
Eelco Dolstra 27444d40cf
Merge pull request #5101 from fzakaria/faridzakaria/fix-nix-shell
nix-shell --pure: Let it work for any derivation
2021-08-09 14:52:11 +02:00
Eelco Dolstra 4c5ce2a345
Merge pull request #5104 from andir/refscan-race
Fix potential race-condition in reference scanning code
2021-08-09 14:49:24 +02:00
Eelco Dolstra 873df3ab03
Merge pull request #5106 from andir/libstore-waiter-set
libstore: use set instead of list for waiter list
2021-08-09 14:46:49 +02:00
Andreas Rammhold a9cb1ca32c libstore: use set instead of list for waiter list
This replaces the O(n) search complexity in our insert code with a
lookup of O(log n). It also makes removing waitees easier as we can use
the extract method provided by the set class.
2021-08-08 14:05:38 +02:00
Sebastian Ullrich 12f50b6510 Short-circuit querying substituters on success 2021-08-08 13:24:13 +02:00
Andreas Rammhold b2d3976163 Fix potential race-condition in reference scanning code
Previously the code ensures that the isBase32 array would only be
initialised once in a single-threaded context. If two threads happen to
call the function before the initialisation was completed both of them
would have completed the initialization step. This allowed for a
race-condition where one thread might be done with the initialization
but the other thread sets all the fields to false again. For a brief
moment the base32 detection would then produce false-negatives.
2021-08-07 19:10:25 +02:00
Alexander Bantyev 54c580b71f
Whitelist nixConfig.flake-registry
flake-registry should be safe to set to an aribtrary value, since it
is identical to just setting `inputs`.
2021-08-07 14:42:59 +03:00
Farid Zakaria fbf42c55ac nix-shell --pure: Let it work for any derivation
`nix-shell --pure` when applied to a non stdenv derivation doesn't seem
to clear the PATH. It expects the stdenv/setup file to do so.

This adds an explicit `unset PATH` by nix-build.cc (nix-shell) itself so
that it's not reliant on stdenv/setup anymore.

This does not break impure nix-shell since the PATH is persisted as the
variable `p` prior in the bash rcfile

fixes #5092
2021-08-06 15:30:49 -07:00
Eelco Dolstra e277c0c479
Merge pull request #5075 from andir/libutil-base64-init
libutil: initialize the base64 decode array only once
2021-08-06 17:41:11 +02:00
Taeer Bar-Yam 8f9429dcab add antiquotations to paths 2021-08-06 06:46:05 -04:00
Madeline Haraj cb1ffb7789 Use the store path as the context of the result of fetchTree, not the real path 2021-08-05 22:08:39 -04:00
Michael Fellinger 5c99ec374c
fix git init race condition 2021-08-02 13:47:59 +02:00
ldesgoui 6eeb6f9c84 nix-shell -p: pass --arg values as nixpkgs params 2021-07-31 17:25:10 +02:00
Andreas Rammhold b9c9c25766
libutil: initialize the base64 decode array only once
Previously, despite having a boolean that tracked initialization, the
decode characters have been "calculated" every single time a base64
string was being decoded.

With this change we only initialize the decode array once in a
thread-safe manner.
2021-07-30 21:07:32 +02:00
Pamplemousse d64cb33e90 Remove unused struct in libstore
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-30 10:34:28 -07:00
regnat d9ba3385a9 Don’t send the experimental-features to the daemon
The experimental features are, well, experimental, and shouldn’t be
carelessly and transparently enabled.
Besides, some (`ca-derivations` at least) need to be enabled at startup
in order to work properly.
So it’s better to just require that daemon be started with the right
`experimental-features` option.

Fix #5017
2021-07-30 10:34:50 +02:00
Viktor Kronvall 7cdaae6b9c Support SCP-like URLs in builtins.fetchGit attrs
This extends https://github.com/NixOS/nix/pull/4978 with
supporting the SCP-like urls in expressions like

```nix
builtins.fetchGit {
  url = "git@github.com:NixOS/nix.git";
  ref = "master";
}
```
2021-07-30 08:17:46 +09:00
Eelco Dolstra 7816ef6c51
Merge pull request #5042 from alyssais/pthread
Enable pthreads for new libraries
2021-07-28 12:31:01 +02:00
Eelco Dolstra ed5ad59dc1 nix develop: Support chroot stores
Fixes #5024.
2021-07-27 14:24:03 +02:00
Eelco Dolstra c000cec27f
Merge pull request #5048 from tweag/flox-eval-store
--eval-store and faster closure copying
2021-07-27 12:20:32 +02:00
Eelco Dolstra 47002108d1 nix-instantiate: Fix --eval-store 2021-07-27 11:16:47 +02:00
Eelco Dolstra f52fa47e16
Merge pull request #5046 from ncfavier/patch-1
Fix --no-gc-warning
2021-07-27 09:55:25 +02:00
Eelco Dolstra 9deeab6337
Merge pull request #5043 from alyssais/vexing
libutil: use uniform initialization in _deletePath
2021-07-27 09:54:24 +02:00
Pamplemousse c2d7c0cdb9 libexpr: Remove unused code
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-26 09:14:34 -07:00
Eelco Dolstra da55210403
Merge pull request #5037 from Pamplemousse/less_globals
Avoid global counters
2021-07-26 14:29:03 +02:00
Eelco Dolstra 72c5bac39d Revert no longer necessary change 2021-07-26 13:50:18 +02:00
Eelco Dolstra fe1f34fa60 Low-latency closure copy
This adds a new store operation 'addMultipleToStore' that reads a
number of NARs and ValidPathInfos from a Source, allowing any number
of store paths to be copied in a single call. This is much faster on
high-latency links when copying a lot of small files, like .drv
closures.

For example, on a connection with an 50 ms delay:

Before:

  $ nix copy --to 'unix:///tmp/proxy-socket?root=/tmp/dest-chroot' \
    /nix/store/90jjw94xiyg5drj70whm9yll6xjj0ca9-hello-2.10.drv \
    --derivation --no-check-sigs
  real    0m57.868s
  user    0m0.103s
  sys     0m0.056s

After:

  real    0m0.690s
  user    0m0.017s
  sys     0m0.011s
2021-07-26 13:31:09 +02:00
Naïm Favier cefa8b673f
Fix --no-gc-warning
Broken by 8e758d402b
2021-07-25 16:06:55 +02:00
Guillaume Girol 136a613cce nix-channel: use nix-env -i --remove-all to upgrade
Fixes #4110
2021-07-25 12:00:00 +00:00
Alyssa Ross 369ed71858
libutil: use uniform initialization in _deletePath
Otherwise I get a compiler error when building for NetBSD:

src/libutil/util.cc: In function 'void nix::_deletePath(const Path&, uint64_t&)':
src/libutil/util.cc:438:17: error: base operand of '->' is not a pointer
  438 |     AutoCloseFD dirfd(open(dir.c_str(), O_RDONLY));
      |                 ^~~~~
src/libutil/util.cc:439:10: error: 'dirfd' was not declared in this scope
  439 |     if (!dirfd) {
      |          ^~~~~
src/libutil/util.cc:444:17: error: 'dirfd' was not declared in this scope
  444 |     _deletePath(dirfd.get(), path, bytesFreed);
      |                 ^~~~~
2021-07-24 09:19:48 +00:00
Alyssa Ross ae0c026fe9
Enable pthreads for new libraries
Otherwise the lack of pthread causes linking to fail for NetBSD.
2021-07-24 09:15:01 +00:00
Eelco Dolstra 9957315ce0 RemoteStore::buildPaths(): Handle evalStore 2021-07-22 22:50:48 +02:00
Eelco Dolstra a7b7fcfb16 Remove redundant RealisedPath::closure() call 2021-07-22 22:43:18 +02:00
Eelco Dolstra 96c62fb66c Fix formatting error in 'nix store' manpage 2021-07-22 10:11:04 +02:00
Eelco Dolstra eb6db4fd38 buildPaths(): Add an evalStore argument
With this, we don't have to copy the entire .drv closure to the
destination store ahead of time (or at all). Instead, buildPaths()
reads .drv files from the eval store and copies inputSrcs to the
destination store if it needs to build a derivation.

Issue #5025.
2021-07-22 09:59:51 +02:00
Eelco Dolstra 668abd3e57 copyPaths: Pass store by reference 2021-07-22 09:59:51 +02:00
Eelco Dolstra 8d9f7048cd Use eval-store in more places
In particular, this now works:

  $ nix path-info --eval-store auto --store https://cache.nixos.org nixpkgs#hello

Previously this would fail as it would try to upload the hello .drv to
cache.nixos.org. Now the .drv is instantiated in the local store, and
then we check for the existence of the outputs in cache.nixos.org.
2021-07-22 09:59:51 +02:00
Eelco Dolstra 95e915a993 DummyStore: Remove redundant method 2021-07-22 09:59:51 +02:00
Eelco Dolstra 7321657746 nix-shell: Handle --eval-store correctly 2021-07-22 09:59:51 +02:00
Eelco Dolstra e9848beca7 nix-build: Copy drv closure between eval store and build store 2021-07-22 09:59:51 +02:00
Eelco Dolstra 2ff3035cf4 Support --eval-store in nix-instantiate and nix-build 2021-07-22 09:59:51 +02:00
Eelco Dolstra 3d9de41a5b Hacky fast closure copying mechanism 2021-07-22 09:59:51 +02:00
Eelco Dolstra bef40c2949 Add --eval-store option 2021-07-22 09:59:51 +02:00
Eelco Dolstra dfda499326 Downgrade warning message
If a store doesn't support GC, then we don't need to warn about the
inability to create roots.
2021-07-22 09:58:50 +02:00
Pamplemousse c1c5dd7449 Avoid global counters
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-21 16:49:52 -07:00
Eelco Dolstra 3bb8667a17 Tweak --no/use-registries doc strings 2021-07-21 14:27:37 +02:00
Eelco Dolstra fc248cf59e Merge branch 'balsoft/no-registries' of https://github.com/serokell/nix 2021-07-21 10:11:40 +02:00
Eelco Dolstra 140ccf1368 deletePath(): Return ENFILE instead of EBADF when out of file descriptors
Also remove an erroneous comment.
2021-07-20 20:59:45 +02:00
Eelco Dolstra c5fafca5a4 Merge remote-tracking branch 'origin/repl-flake-support' 2021-07-20 20:06:56 +02:00
regnat 8707773965 Properly lock the builds of CA derivations
Make sure that we can’t build the same derivation twice at the same
time.

Fix https://github.com/NixOS/nix/issues/5029
2021-07-20 06:57:56 +02:00
Robert Hensing 3bbf5558e0 nix repl: Update :edit help text
It supports functions as well. Also change `package` to
`derivation` because it operates at the language level and does
not open the derivation (which would be useful but not nearly
as much).
2021-07-19 18:13:26 +02:00
Robert Hensing ad24921de8 Rename findDerivationFilename -> findPackageFilename
It does not operate on a derivation and does not return a
derivation path. Instead it works at the language level,
where a distinct term "package" is more appropriate to
distinguish the parent object of `meta.position`; an
attribute which doesn't even make it into the derivation.
2021-07-19 18:10:10 +02:00
regnat 765a3a20cb Add a :load-flake command to the nix repl
`:lf ref` does essentially the same thing as
`:a (builtins.getFlake "ref")` (but cannonicalising `ref` if needs be,
so that `:lf .` works)

Fix #3803
2021-07-19 17:52:41 +02:00
Eelco Dolstra 58cb411db6
Merge pull request #5020 from NixOS/make-the-repl-reuse-the-whole-nix-config
Forward the whole Nix config to the repl subprocesses
2021-07-19 16:02:02 +02:00
Alexander Bantyev 3e57e3480b
Add use-registries config option (and deprecate --no-registries flag)
Some people want to avoid using registries at all on their system; Instead
of having to add --no-registries to every command, this commit allows to
set use-registries = false in the config. --no-registries is still allowed
everywhere it was allowed previously, but is now deprecated.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-07-16 23:26:04 +03:00
regnat a4ec6cb1da Be more lenient when realisations have a conflicting dependency set
- This can legitimately happen (for example because of a non-determinism
  causing a build-time dependency to be kept or not as a runtime
  reference)
- Because of older Nix versions, it can happen that we encounter a
  realisation with an (erroneously) empty set of dependencies, in which
  case we don’t want to fail, but just warn the user and try to fix it.
2021-07-16 11:48:41 +02:00
regnat c05bdef020 Forward the whole Nix config to the repl subprocesses
Fill `NIX_CONFIG` with the value of the current Nix configuration before
calling the nix subprocesses in the repl
That way the whole configuration (including the possible
`experimental-features`, a possibly `--store` option or whatever) will
be made available.

This is required for example to make `nix repl` work with a custom
`--store`
2021-07-16 09:48:45 +02:00
regnat 9b1f3cbc13 Forward the whole Nix config to the post-build-hook
Fill `NIX_CONFIG` with the value of the current Nix configuration before
calling the post-build-hook.
That way the whole configuration (including the possible
`experimental-features`, a possibly `--store` option or whatever) will
be made available to the hook
2021-07-15 18:41:56 +02:00
Eelco Dolstra 307977963c nlohmann_json: Update to 3.9.1, fix use of internal copy 2021-07-15 12:25:53 +02:00
Eelco Dolstra 77d5b37da3
Merge pull request #5011 from Pamplemousse/fix_adjustLoc
libexpr: Fix read out-of-bound on the heap
2021-07-15 09:23:59 +02:00
Pamplemousse 99f8fc995b libexpr: Fix read out-of-bound on the heap
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-14 09:09:42 -07:00
regnat 797e260e3a Make nix flake check aware of devShells 2021-07-13 17:25:27 +02:00
regnat 43d5c5f87b Make nix flake show display the devShells 2021-07-13 17:25:27 +02:00
regnat 037c86ee04 nix develop: Search in devShells.${system} by default
Make `nix develop .#foo` search `.#devShells.${system}.foo` first
2021-07-13 17:25:27 +02:00
Eelco Dolstra eb4788954d
Merge pull request #5006 from illustris/nscd
fixed-output derivations: fix incorrect responses for getpwuid
2021-07-13 17:20:37 +02:00
illustris 7bc17a903b fixed output derivations: fix incorrect responses for getpwuid
Passing nscd socket into the build environment causes unexpected behavior in programs that make getpwuid and other related calls.

relevant threads:
- https://github.com/NixOS/nix/issues/4991
- https://discourse.nixos.org/t/haunted-nix-build-breaks-isolation/13869
2021-07-13 15:34:14 +05:30
Eelco Dolstra 099df07e1e
Merge pull request #5000 from nielsegberts/master
toString also coerces a set with an outPath attribute to a string
2021-07-12 17:35:32 +02:00
Eelco Dolstra 02dff9e529 Style 2021-07-12 17:32:23 +02:00
Eelco Dolstra e06c272c12 Merge branch 'structured-attrs-shell' of https://github.com/Ma27/nix 2021-07-12 17:13:05 +02:00
Maximilian Bosch 04cd2da84c
Merge branch 'master' into structured-attrs-shell
Conflicts:
        src/nix/develop.cc
        src/nix/get-env.sh
        tests/shell.nix
2021-07-12 15:49:39 +02:00
Eelco Dolstra 91d2e8d5ad Error -> UsageError 2021-07-12 15:04:46 +02:00
Eelco Dolstra 8d97030bfd Merge branch 'days' of https://github.com/nielsegberts/nix 2021-07-12 15:02:24 +02:00
Niels Egberts 2cf14db857 Throw on unexpected input for --delete-older-than
'--delete-older-than 10' deletes the generations older than a single day, and '--delete-older-than 12m' deletes all generations older than 12 days.

This changes makes it throw on those invalid inputs, and gives an example of a valid input.
2021-07-11 11:24:43 +01:00
Niels Egberts ae0ed53b09 toString also coerces a set with an outPath attribute to a string
nix-repl> builtins.toString { outPath = "somestring"; }
"somestring"
2021-07-09 21:50:10 +01:00
Eelco Dolstra ceda58d112 Formatting 2021-07-09 14:03:51 +02:00
Eelco Dolstra 07790fdddf ref: Add equality operators 2021-07-09 14:03:48 +02:00
Eelco Dolstra 86fb01c4be nix print-dev-env: Add --json flag 2021-07-09 12:10:48 +02:00
Eelco Dolstra 5f6375a816 nix develop: Filter some bash magic variables 2021-07-09 01:18:44 +02:00
Eelco Dolstra b1f1347ade nix develop: Don't parse bash environment with regexes
Instead have get-env.sh dump the bash environment as JSON. This should
be a lot less error-prone.

Fixes #4992.
2021-07-09 01:18:37 +02:00
Eelco Dolstra d2b8b23ae9
Merge pull request #4995 from NixOS/fetchgit-name-attribute
Fix the `name` attribute in builtins.fetchGit
2021-07-09 00:32:13 +02:00
regnat a654c1d81c Restore the possibility to use a name parameter in fetchGit
Accidentally broken by 7e5c79a2d2
2021-07-08 14:57:48 +02:00
Eelco Dolstra 8648143120
Merge pull request #4988 from NixOS/fetchgit-name-attribute
Add a name attribute to the fetchers
2021-07-08 14:33:49 +02:00
regnat 7e5c79a2d2 Forbid the name attribute for fetchTree
We need to support it for the “old” fetch* functions for backwards
compatibility, but we don’t need it for fetchTree (as it’s a new
function).
Given that changing the `name` messes-up the content hashing, we can
just forbid passing a custom `name` argument to it
2021-07-08 13:53:52 +02:00
Eelco Dolstra 2172e60f7a
Merge pull request #4935 from alyssais/host_os
Apply OS checks to host platform, not build
2021-07-08 12:40:54 +02:00
Eelco Dolstra 156666de3d
Merge pull request #4969 from serokell/balsoft/fix-nixConfig-flake-registry
flake.nixConfig: fix flake-registry config settings
2021-07-08 12:35:08 +02:00
Eelco Dolstra 2c853e2a58 Merge branch 'master' of github.com:NixOS/nix 2021-07-07 10:03:04 +02:00
Eelco Dolstra 3b3e6bb1e5 Style tweaks 2021-07-07 10:02:55 +02:00
Eelco Dolstra e37ecd1282 Merge branch 'balsoft/registry' of https://github.com/serokell/nix 2021-07-07 09:57:15 +02:00
regnat 2c8240677e allow fetchFromGitHub to take a name argument 2021-07-06 08:43:19 +02:00
regnat e4b082a52b allow fetchMercurial to take a name argument 2021-07-06 08:43:06 +02:00
regnat a487a652ed allow fetchTarball to take a name argument 2021-07-06 08:42:47 +02:00
regnat 83615fcf8f Allow fetchGit to take a name argument
Fix #3388
2021-07-06 08:41:52 +02:00
Eelco Dolstra 0a8845720e
Merge pull request #4980 from Ma27/file-attr-completion
libcmd/installables: implement completion for Nix expressions passed via `-f`
2021-07-05 21:50:15 +02:00
Maximilian Bosch 3c5f69bb60
completeInstallable: also match for already typed prefixes 2021-07-05 21:37:33 +02:00
Eelco Dolstra 1f93084149
Merge pull request #4978 from NixOS/scp-uris
Allow scp-style uris in `fetchgit`
2021-07-05 14:07:04 +02:00
Eelco Dolstra cee426cc01
Merge pull request #4938 from tomcur/store-uri-parsing
Improve machine store URI parsing
2021-07-05 14:00:35 +02:00
Eelco Dolstra 24bc935462 Merge branch 'disable_gc' of https://github.com/Pamplemousse/nix 2021-07-05 13:53:52 +02:00
Michael Fellinger 087c5f5325
Fix devShell handling of env values including @ and % 2021-07-05 12:05:06 +02:00
Maximilian Bosch 70cb2ffacc
libcmd/installables: implement completion for Nix expressions passed via -f 2021-07-03 14:19:10 +02:00
Domen Kožar f9d72855ae
Merge pull request #4967 from Pamplemousse/specific_errors
Prefer to throw specific errors
2021-07-03 10:34:56 +02:00
regnat ec2c6bd470 Allow scp-style uris in fetchgit
Fix #5303
2021-07-02 19:20:07 +02:00
Yestin L. Harrison 0fe84bef72 Add $NO_COLOR check to ANSI escape conditions 2021-07-02 09:33:54 -06:00
Yestin L. Harrison 20cce079f2 Respect TERM=dumb more consistently 2021-07-01 18:19:01 -06:00
Pamplemousse 4a7a8b87cd Prefer to throw specific errors
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-01 11:09:31 -07:00
Alexander Bantyev e756a59c72
fixup! flake.nixConfig: fix flake-registry config settings 2021-07-01 17:54:22 +03:00
Alexander Bantyev ef1e7ab840
flake.nixConfig: fix flake-registry config settings
Before this commit, nixConfig.flake-registry didn't have any real effect
on the evaluation, since config was applied after inputs were evaluated.
Change this behavior: apply the config in the beginning of flake::lockFile.
2021-07-01 00:23:47 +03:00
Alexander Bantyev 811f3e8605
nix registry pin: add a way to pin to a custom locked 2021-06-30 22:16:40 +03:00
Alexander Bantyev 093ed47636
nix registry: add --registry flag 2021-06-30 22:13:32 +03:00
Pamplemousse 5be17a4b96 Allow to compile after ./configure --enable-gc=no
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-06-30 09:45:21 -07:00
Eelco Dolstra 139f7af5ec
Merge pull request #4958 from NixOS/nix-develop-remove-override-input-warning
develop: Discard the input{Overrides,Updates} when getting bash
2021-06-30 17:57:01 +02:00
Eelco Dolstra c92fbdb654
Merge pull request #4960 from NixOS/nix-repl-experimental-features
Forward the experimental features to the nix repl subprocesses
2021-06-29 15:10:02 +02:00
regnat 7daf0c6ef1 Forward the experimental features to the nix repl subprocesses
Pass the current experimental features using `NIX_CONFIG` to the various
Nix subprocesses that `nix repl` invokes.

This is quite a hack, but having `nix repl` call Nix with a subprocess
is a hack already, so I guess that’s fine.
2021-06-29 14:52:46 +02:00
Eelco Dolstra 5a18e2a533
Merge pull request #4959 from NixOS/override-non-flake-inputs
Keep the `isFlake` attribute for overriden inputs
2021-06-29 14:23:50 +02:00
regnat 4cff8188a5 Keep the isFlake attribute for overriden inputs
When overriding an input that isn’t a flake, mark the override as also not
being a flake.

Fix #3774
2021-06-29 13:56:48 +02:00
Eelco Dolstra f14c3b6f68
Merge pull request #4944 from hercules-ci/fix-gc-crash
Fix gc crash
2021-06-29 13:52:14 +02:00
regnat bf7960a4ed develop: Discard the input{Overrides,Updates} when getting bash
`nix develop` is getting bash from an (assumed existing) `nixpkgs`
flake. However, when doing so, it reuses the `lockFlags` passed to the
current flake, including the `--input-overrides` and `--input-update`
which generally don’t make sense anymore at that point (and trigger a
warning because of that)

Clear these overrides before getting the nixpkgs flake to get rid of the
warning.
2021-06-29 10:13:42 +02:00
Florian Franzen decc14d4b7
man: fix formatting of nix3-profile-remove 2021-06-28 16:27:03 +02:00
Eelco Dolstra 6182ae6898
Merge pull request #4942 from NixOS/ca/remove-lock-files
Eventually delete the CA paths lock files
2021-06-28 16:12:19 +02:00
Eelco Dolstra f5320299dd
Merge pull request #4937 from NixOS/ca/make-the-tests-useful
Make the CA tests actually test something
2021-06-28 16:06:49 +02:00
regnat 8b6fba2b63 Eventually delete the CA paths lock files
Mark the lockfiles as having to eventually be deleted so that they don’t
stay laying around in the store at the end of the build

Fix #4936
2021-06-24 15:45:05 +02:00
Maximilian Bosch 644415d391
Perform input rewrites only in LocalDerivationGoal 2021-06-24 15:25:21 +02:00
regnat 7746cb13dc Make CA derivations compatible with recursive Nix
Add an access-control list to the realisations in recursive-nix (similar
to the already existing one for store paths), so that we can build
content-addressed derivations in the restricted store.

Fix #4353
2021-06-24 14:53:10 +02:00
Robert Hensing bec83a6f95 BoehmGCStackAllocator: ignore stack protection page
This fixes a crash that looks like:

```
Thread 1 "nix-build" received signal SIGSEGV, Segmentation fault.
0x00007ffff7ad22a0 in GC_push_all_eager () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
(gdb) bt
0  0x00007ffff7ad22a0 in GC_push_all_eager () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
1  0x00007ffff7adeefb in GC_push_all_stacks () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
2  0x00007ffff7ad5ac7 in GC_mark_some () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
3  0x00007ffff7ad77bd in GC_stopped_mark () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
4  0x00007ffff7adbe3a in GC_try_to_collect_inner.part.0 () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
5  0x00007ffff7adc2a2 in GC_collect_or_expand () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
6  0x00007ffff7adc4f8 in GC_allocobj () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
7  0x00007ffff7adc88f in GC_generic_malloc_inner () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
8  0x00007ffff7ae1a04 in GC_generic_malloc_many () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
9  0x00007ffff7ae1c72 in GC_malloc_kind () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
10 0x00007ffff7e003d6 in nix::EvalState::allocValue() () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
11 0x00007ffff7e04b9c in nix::EvalState::callPrimOp(nix::Value&, nix::Value&, nix::Value&, nix::Pos const&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
12 0x00007ffff7e0a773 in nix::EvalState::callFunction(nix::Value&, nix::Value&, nix::Value&, nix::Pos const&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
13 0x00007ffff7e0a91d in nix::ExprApp::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
14 0x00007ffff7e0a8f8 in nix::ExprApp::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
15 0x00007ffff7e0e0e8 in nix::ExprOpNEq::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
16 0x00007ffff7e0d708 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
17 0x00007ffff7e0d695 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
18 0x00007ffff7e0d695 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
19 0x00007ffff7e0d695 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
20 0x00007ffff7e0d695 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
21 0x00007ffff7e09e19 in nix::ExprOpNot::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
22 0x00007ffff7e0a792 in nix::EvalState::callFunction(nix::Value&, nix::Value&, nix::Value&, nix::Pos const&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
23 0x00007ffff7e8cba0 in nix::addPath(nix::EvalState&, nix::Pos const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Value*, nix::FileIngestionMethod, std::optional<nix::Hash>, nix::Value&)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#1}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
24 0x00007ffff752e6f9 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
25 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
26 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
27 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
28 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
29 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
30 0x00007ffff757f8c0 in void boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, nix::VirtualStackAllocator, boost::coroutines2::detail::pull_coroutine<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::control_block::control_block<nix::VirtualStackAllocator, nix::sinkToSource(std::function<void (nix::Sink&)>, std::function<void ()>)::SinkToSource::read(char*, unsigned long)::{lambda(boost::coroutines2::detail::push_coroutine<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&)#1}>(boost::context::preallocated, nix::VirtualStackAllocator&&, nix::sinkToSource(std::function<void (nix::Sink&)>, std::function<void ()>)::SinkToSource::read(char*, unsigned long)::{lambda(boost::coroutines2::detail::push_coroutine<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&)#1}&&)::{lambda(boost::context::fiber&&)#1}> >(boost::context::detail::transfer_t) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
31 0x00007ffff6f331ef in make_fcontext () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libboost_context.so.1.69.0
32 0x0000000000000000 in ?? ()
```
2021-06-24 12:20:39 +02:00
regnat be7a4a6a13 Make the post-build-hook also run for unresolved CA derivations
Fix #4837
2021-06-24 11:41:57 +02:00
Thomas Churchman e3d11f9a9c Improve machine store URI parsing 2021-06-23 22:09:22 +01:00
Alyssa Ross 4f80464645
Apply OS checks to host platform, not build
Previously, the build system used uname(1) output when it wanted to
check the operating system it was being built for, which meant that it
didn't take into-account cross-compilation when the build and host
operating systems were different.

To fix this, instead of consulting uname output, we consult the host
triple, specifically the third "kernel" part.

For "kernel"s with stable ABIs, like Linux or Cygwin, we can use a
simple ifeq to test whether we're compiling for that system, but for
other platforms, like Darwin, FreeBSD, or Solaris, we have to use a
more complicated check to take into account the version numbers at the
end of the "kernel"s.  I couldn't find a way to just strip these
version numbers in GNU Make without shelling out, which would be even
more ugly IMO.  Because these checks differ between kernels, and the
patsubst ones are quite fiddly, I've added variables for each host OS
we might want to check to make them easier to reuse.
2021-06-23 15:00:36 +00:00
Eelco Dolstra 0a535dd5ac
Merge pull request #4839 from NixOS/ca/gracefully-handle-duplicate-realisations
Gracefully handle duplicate realisations
2021-06-23 11:50:18 +02:00
Eelco Dolstra f9f773b332
Merge pull request #4908 from NixOS/ca/fix-nix-develop
Make `nix develop` work with CA derivations
2021-06-23 11:27:58 +02:00
regnat c878cee895 Assert that compatible realisations have the same dependencies
Should always hold, but that’s not necessarily obvious, so better
enforce it
2021-06-23 11:27:16 +02:00
regnat 16fb7d8d95 Display the diverging paths in case of a realisation mismatch 2021-06-23 11:27:16 +02:00
regnat 40f925b2da Fix indentation 2021-06-23 11:27:16 +02:00
regnat d32cf0c17a Gracefully ignore a substituter if it holds an incompatible realisation 2021-06-23 11:27:16 +02:00
regnat b8f7177a7b Properly fail when trying to register an incoherent realisation 2021-06-23 11:27:14 +02:00
Eelco Dolstra 7945055c63
Merge pull request #4842 from NixOS/ca/fix-nix-shell
Make `nix-shell` support content-addressed derivations
2021-06-23 11:26:22 +02:00
Eelco Dolstra 4a5aa1dbf6
Merge pull request #4838 from NixOS/ca/recursively-substitute-realisations
Recursively substitute the realisations
2021-06-23 10:33:25 +02:00
Eelco Dolstra 26d2c62225
Merge pull request #4906 from NixOS/collect-garbage-ca
Make `computeFSClosure` ca-aware
2021-06-23 10:07:08 +02:00
Eelco Dolstra d9a43d3137
Merge pull request #4905 from NixOS/ca-derivations-machine-feature
Add a ca-derivations required machine feature
2021-06-23 10:05:53 +02:00
Eelco Dolstra 3e4126b67c
Merge pull request #4926 from NixOS/ca/build-remote-signal-6
Fix the remote build of CA derivations
2021-06-23 10:04:35 +02:00
Eelco Dolstra 4f9508c3b5
Merge pull request #4836 from NixOS/ca/track-drvoutput-dependencies-2-le-retour
Track the dependencies of CA realisations
2021-06-23 10:03:31 +02:00
regnat ed0e21a88d Fix indentation 2021-06-23 08:16:34 +02:00
regnat 7c96a76dd7 Reformat the sql statements 2021-06-23 08:16:34 +02:00
Théophane Hufschmitt 8d09a4f9a0 Remove a useless string split
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-06-23 08:16:34 +02:00
Théophane Hufschmitt c13d7d0b97 Pass more values by reference
Rather than copying them around everywhere

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-06-23 08:16:34 +02:00
Maximilian Bosch 6f206549ba
Move writeStructuredAttrsShell out of ParsedDerivation class 2021-06-22 21:14:20 +02:00
Maximilian Bosch 27ce722638
Prefix env vars for attrs.* files with NIX_ 2021-06-22 19:45:08 +02:00
Maximilian Bosch a92245b110
Remove now-obsolete typedef 2021-06-22 19:15:57 +02:00
Maximilian Bosch 3504c811a5
Add testcase for nix develop with __structuredAttrs 2021-06-22 19:15:57 +02:00
Maximilian Bosch f1e281c4fe
Split shell & json creation for build environments with structured attrs 2021-06-22 19:15:57 +02:00
Maximilian Bosch 447928bdb5
Fix usage of structured attrs for nix develop 2021-06-22 19:15:57 +02:00
Maximilian Bosch 3944a120ec
Set environment variables for .attrs.json & .attrs.sh
This way no derivation has to expect that these files are in the `cwd`
during the build. This is problematic for `nix-shell` where these files
would have to be inserted into the nix-shell's `cwd` which can become
problematic with e.g. recursive `nix-shell`.

To remain backwards-compatible, the location inside the build sandbox
will be kept, however using these files directly should be deprecated
from now on.
2021-06-22 19:15:57 +02:00
Maximilian Bosch 3b5429aec1
Source complete env in nix-shell with __structuredAttrs = true;
This is needed to push the adoption of structured attrs[1] forward. It's
now checked if a `__json` exists in the environment-map of the derivation
to be openend in a `nix-shell`.

Derivations with structured attributes enabled also make use of a file
named `.attrs.json` containing every environment variable represented as
JSON which is useful for e.g. `exportReferencesGraph`[2]. To
provide an environment similar to the build sandbox, `nix-shell` now
adds a `.attrs.json` to `cwd` (which is mostly equal to the one in the
build sandbox) and removes it using an exit hook when closing the shell.

To avoid leaking internals of the build-process to the `nix-shell`, the
entire logic to generate JSON and shell code for structured attrs was
moved into the `ParsedDerivation` class.

[1] https://nixos.mayflower.consulting/blog/2020/01/20/structured-attrs/
[2] https://nixos.org/manual/nix/unstable/expressions/advanced-attributes.html#advanced-attributes
2021-06-22 19:15:57 +02:00
regnat 3b58dbb356 nix-shell: Replace resolving failure error by an assertion
This shouldn’t happen in practice, so better make it explicit
2021-06-22 11:29:55 +02:00
regnat dcabb46124 Shorten a stupidly long sql query name 2021-06-21 16:28:06 +02:00
Eelco Dolstra 610baf359a
Merge pull request #4833 from NixOS/ca/json-realisations-in-worker-protocol
Always send the realisations as JSON
2021-06-21 16:12:17 +02:00
Eelco Dolstra 24e7353232
Merge pull request #4928 from NixOS/ca/remove-existing-invalid-store-path
Remove a possible existing store path when building CA derivations
2021-06-21 16:09:21 +02:00
Théophane Hufschmitt ce674cb2cf
Properly set the output env variables
Co-authored-by: John Ericson <git@JohnEricson.me>
2021-06-21 15:52:01 +02:00
regnat 608434722b Only symlink the requested outputs in nix build
Fix #4925
2021-06-21 15:47:47 +02:00
regnat 3784c66a46 Remove a possible existing store path when building CA derivations
In case a previous interrupted build left a garbage path laying around,
remove it before trying to move the path to its final location.

Fix #4858
2021-06-21 15:29:15 +02:00
regnat 498677cbed Fix the remote build of CA derivations
Make sure that the derivation we send to the remote builder is exactly
the one that we want to build locally so that the output ids are exactly
the same

Fix #4845
2021-06-21 14:18:33 +02:00
Eelco Dolstra 4202a3bc4e UDSRemoteStore: Support the 'root' store parameter
Useful when we're using a daemon with a chroot store, e.g.

  $ NIX_DAEMON_SOCKET_PATH=/tmp/chroot/nix/var/nix/daemon-socket/socket nix-daemon --store /tmp/chroot

Then the client can now connect with

  $ nix build --store unix:///tmp/chroot/nix/var/nix/daemon-socket/socket?root=/tmp/chroot nixpkgs#hello
2021-06-18 17:04:11 +02:00
Timothy DeHerrera 769ca4e26d
libfetchers/git: fetch submodules by default 2021-06-17 12:00:26 -06:00
Eelco Dolstra e6150de90d
nix develop: Filter out NIX_REMOTE
When recursive Nix is enabled, NIX_REMOTE is set to
unix:///build/.nix-socket, which doesn't work outside of the sandbox.
2021-06-15 12:06:01 +02:00
regnat 96d7170e12 Don’t check the deriver field on computeFSClosure
That doesn’t really make sense with CA derivations (and wasn’t even
really correct before because of FO derivations, though that probably
didn’t matter much in practice)
2021-06-12 12:24:53 +02:00
regnat 2cf591a134 Make nix develop work with CA derivations
Fix #4823
2021-06-11 13:35:13 +02:00
regnat 56605b4688 Make nix-shell support content-addressed derivations
Resolve the derivation before trying to load its environment −
essentially reproducing what the build loop does − so that we can
effectively access our dependencies (and not just their placeholders).

Fix #4821
2021-06-11 13:32:49 +02:00
regnat 7ac038fa4b Make computeFSClosure ca-aware
Fix #4820 by preventing nix-collect garbage from crashing if
`keep-outputs` or `keep-derivations` is true
2021-06-11 09:26:49 +02:00
regnat 7c077d2a0f Add a ca-derivations required machine feature
Make ca-derivations require a `ca-derivations` machine feature, and
ca-aware builders expose it.

That way, a network of builders can mix ca-aware and non-ca-aware
machines, and the scheduler will send them in the right place.
2021-06-11 09:12:53 +02:00
keke 50dc88a56c
fix error: 'optional' in namespace 'std' does not name a template type 2021-06-02 18:09:03 +08:00
regnat 838f862f4f doc: Wrap at 80 characters 2021-06-02 11:26:04 +02:00
Théophane Hufschmitt 7565308d04
Fix a documentation typo
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-06-02 11:25:47 +02:00
regnat 7d651f5c3f throw_ -> reportError
This function might or might not throw depending on the value of
`keepGoing`, so naming it `throw_` was a bit confusing
2021-06-02 11:24:31 +02:00
regnat d12b12a15b Let nix flake check keep going when keep-going is set
When the `keep-going` option is set to `true`, make `nix flake check`
continue as much as it can before failing.

The UI isn’t perfect as-it-is as all the lines currently start with a
mostly useless `error (ignored): error:` prefix, but I’m not sure what
the best output would be, so I’ll leave it as-it-is for the time being

(This is a bit hijacking the `keep-going` flag as it’s supposed to be a
build-time only thing. But I think it’s faire to reuse it here).

Fix https://github.com/NixOS/nix/issues/4450
2021-06-02 11:13:12 +02:00
regnat 5985b8b527 Check the CA hash when importing stuff in the local store
When adding a path to the local store (via `LocalStore::addToStore`),
ensure that the `ca` field of the provided `ValidPathInfo` does indeed
correspond to the content of the path.
Otherwise any untrusted user (or any binary cache) can add arbitrary
content-addressed paths to the store (as content-addressed paths don’t
need a signature).
2021-06-01 15:09:24 +02:00
Eelco Dolstra 48396d940e
Merge pull request #4866 from alyssais/libdl
Only link with libdl on Linux
2021-06-01 11:50:50 +02:00
Eelco Dolstra b8fbfc80fd
Merge pull request #4864 from jeremyschlatter/fix-typo
Fix typo in documentation
2021-06-01 11:49:46 +02:00
Eelco Dolstra caef6f4314
Merge pull request #4734 from p01arst0rm/fix-s3-ifdef
unified macro style for ENABLE_S3
2021-06-01 11:30:21 +02:00
Alyssa Ross c57ab17687
Only link with libdl on Linux
Linux is (as far as I know) the only mainstream operating system that
requires linking with libdl for dlopen.  On BSD, libdl doesn't exist,
so on non-FreeBSD BSDs linking will currently fail.  On macOS, it's
apparently just a symlink to libSystem (macOS libc), presumably
present for compatibility with things that assume Linux.

So the right thing to do here is to only add -ldl on Linux, not to add
it for everything that isn't FreeBSD.
2021-06-01 08:05:21 +00:00
Jeremy Schlatter f674f7f434
Fix typo in documentation 2021-06-01 00:05:34 -07:00
Timothy Klim 4da9ec772c Add .tar.zst support for TarballInputScheme 2021-05-29 16:03:26 +07:00
Eelco Dolstra b10256af51
Merge pull request #4849 from NixOS/ca/fix-nix-store--export
Make the Nar hash non modulo
2021-05-28 10:48:36 +02:00
regnat a22755721b Recursively substitute the realisations
Make sure that whenever we substitute a realisation, we also substitute
its entire closure
2021-05-26 18:44:17 +02:00
regnat 1f3ff0d193 Aso track the output path of the realisation dependencies 2021-05-26 17:09:21 +02:00
regnat cb46d70794 Add a db migration script 2021-05-26 16:59:09 +02:00
regnat 63ebfc73c5 Make copyPaths copy the whole realisations closure
Otherwise registering the realisations on the remote side might fail as
it now expects a complete closure
2021-05-26 16:59:09 +02:00
regnat 8c30acc3e8 Properly track the drvoutput references when building 2021-05-26 16:59:09 +02:00
regnat af3afd25ea Add a method to compute the closure of a realisation
Only considers the closure in term of `Realisation`, ignores all the
opaque inputs.

Dunno whether that’s the nicest solution, need to think it through a bit
2021-05-26 16:59:09 +02:00
regnat eca6ff06d6 Store the realisation deps on the local store 2021-05-26 16:59:09 +02:00
regnat 7ce0441d80 Add a dependencies field to DrvOutputInfo
Currently never used, nor set but will be useful shortly
2021-05-26 16:59:09 +02:00
regnat 7616268812 Always send the realisations as JSON
Align all the worker protocol with `buildDerivation` which inlines the
realisations as one opaque json blob.
That way we don’t have to bother changing the remote store protocol
when the definition of `Realisation` changes, as long as we keep the
json backwards-compatible
2021-05-26 16:59:09 +02:00
regnat 129384bcf3 Remove the remaining occurenceses of a NarHash modulo 2021-05-26 09:39:29 +02:00
regnat 79ae9e4558 Make the Nar hash non modulo
It makes much more sense to have the Nar hash be a plain straight hash
rather than a hash modulo
2021-05-25 10:58:43 +02:00
Michael Adler 9e9a8456d7 fix doc: nix profile info -> nix profile list
Signed-off-by: Michael Adler <therisen06@gmail.com>
2021-05-24 12:36:08 +02:00
regnat a8416866cf Always send the realisations as JSON
Align all the worker protocol with `buildDerivation` which inlines the
realisations as one opaque json blob.
That way we don’t have to bother changing the remote store protocol
when the definition of `Realisation` changes, as long as we keep the
json backwards-compatible
2021-05-19 11:45:16 +02:00
regnat 184558834a Extract a generic computeClosure function
Move the `closure` logic of `computeFSClosure` to its own (templated) function.

This doesn’t bring much by itself (except for the ability to properly
test the “closure” functionality independently from the rest), but it
allows reusing it (in particular for the realisations which will require
a very similar closure computation)
2021-05-19 11:44:58 +02:00
Eelco Dolstra 7234cf39be
Merge pull request #4831 from matthewbauer/fix-extra-slash-in-canon-path
Fix extra slash in canonPath output
2021-05-19 11:10:43 +02:00
Matthew Bauer 3d90ab9345 Fix extra slash in canonPath output
When you have a symlink like:

  /tmp -> ./private/tmp

you need to resolve ./private/tmp relative to /tmp’s dir: ‘/’. Unlike
any other path output by dirOf, / ends with a slash. We don’t want
trailing slashes here since we will append another slash in the next
comoponent, so clear s like we would if it was a symlink to an absoute
path.

This should fix at least part of the issue in
https://github.com/NixOS/nix/issues/4822, will need confirmation that
it actually fixes the problem to close though.

Introduced in f3f228700a.
2021-05-18 16:38:55 -05:00
regnat 59d0de6ea1 Restore an accidentally suppressed negation
Accidentally removed in ca96f52194. This
caused `nix run` to systematically fail with

```
error: app program '/nix/store/…' is not in the Nix store
```
2021-05-18 13:54:05 +02:00
Eelco Dolstra de77d1b924
Merge pull request #4827 from matthewbauer/run-bashrc-first-in-nix-develop
Source bashrc first in nix develop
2021-05-18 11:31:52 +02:00
Matthew Bauer 5fd8cf7667 Source bashrc first in nix develop
~/.bashrc should be sourced first in the rc script so that PATH &
other env vars give precedence over the bashrc PATH.

Also, in my bashrc I alias rm as:

  alias rm='rm -Iv'

To avoid running this alias (which shows ‘removed '/tmp/nix-shell.*'),
we can just prefix rm with command.
2021-05-17 15:07:00 -05:00
regnat ca96f52194 Split the parsing of an App and its resolving
That way things (like `nix flake check`) can evaluate the `app` outputs
without having to build anything
2021-05-17 17:50:41 +02:00
Eelco Dolstra bd6cf25952
Merge pull request #4819 from NixOS/ca/fix-nix-run
Resolve the program path in `nix run`
2021-05-17 16:28:14 +02:00
Eelco Dolstra 6849ae82de
Merge pull request #4818 from NixOS/ca/cli-use-builtpaths
Enforce the use of properly built paths in libcmd
2021-05-17 16:15:40 +02:00
regnat f0c0052d4b Resolve the program path in nix run
Fix #4768
2021-05-17 09:48:51 +02:00
regnat 2105084645 Enfore the use of properly built paths in libcmd
Replace `DerivedPathWithHints` by a new `BuiltPath` type that serves as
a proof that the corresponding path has been built.
2021-05-17 08:45:08 +02:00
Emily 559a504da7 sandbox: allow SystemVersionCompat.plist on Darwin
For whatever reason, many programs trying to access SystemVersion.plist
also open SystemVersionCompat.plist; this includes Python code and
coreutils’ `cat(1)` (but not the native macOS `/bin/cat`). Illustratory
`dtruss(1m)` output:

    open("/System/Library/CoreServices/SystemVersion.plist\0", 0x0, 0x0)		 = 3 0
    open("/System/Library/CoreServices/SystemVersionCompat.plist\0", 0x0, 0x0)		 = 4 0

I assume this is a Big Sur change relating to the 10.16.x/11.x
version compatibility divide and that it’s something along the lines of
a hook inside libSystem.

Fixes a lot of sandboxed package builds under Big Sur.
2021-05-17 01:24:31 +01:00
Eelco Dolstra 340f831ebe
Merge pull request #4798 from matthewbauer/relock-wait-for-build-slot-goals
Relock wait for build slot goals
2021-05-13 13:07:35 +02:00
Eelco Dolstra 89a4ede92b
Merge pull request #4801 from matthewbauer/fix-tokenize-output-names
Fix tokenize output names in DerivedPath
2021-05-13 13:06:29 +02:00
Matthew Bauer 8c7e043de2 Fix tokenize output names in drv
This should fix the issue described in
https://discourse.nixos.org/t/derivation-does-not-have-wanted-outputs-dev-out/12905.

Specifically, we get an error of

  error: derivation '/nix/store/_.drv' does not have wanted outputs 'dev,out'

when a path like /nix/store/_.drv!dev,out is sent to the daemon.
2021-05-12 21:40:28 -05:00
Matthew Bauer 9a14335845 Relock wait for build slot goals
When we don’t have enough free job slots to run a goal, we put it in
the waitForBuildSlot list & unlock its output locks. This will
continue from where we left off (tryLocalBuild). However, we need the
locks to get reacquired when/if the goal ever restarts. So, we need to
send it back through tryToBuild to get reqacquire those locks.

I think this bug was introduced in
https://github.com/NixOS/nix/pull/4570. It leads to some builds
starting without proper locks.
2021-05-12 12:15:32 -05:00
regnat ec613603ba DerivedPathWithHints -> BuiltPath
Just a renaming for now
2021-05-12 16:19:51 +02:00
Lorenzo Manacorda 4029f4b05b doc: mention Priority for substituters 2021-05-12 13:27:05 +02:00
Eelco Dolstra 7f9759b18d
Merge pull request #4781 from NixOS/locally_cache_the_remote_realisations
Add a realisations disk cache
2021-05-10 20:37:57 +02:00
Eelco Dolstra 8768398d5f
Merge pull request #4787 from Ma27/builtins-ceil-floor
Implement `builtins.floor` and `builtins.ceil` using the C library functions internally
2021-05-10 20:37:27 +02:00
regnat d5d19582ef Simplify the realisations disk cache 2021-05-10 17:47:14 +02:00
regnat ab96c1ee50 Remove useless parents
I never remember the exact syntax of the `switch` statement
2021-05-10 17:36:49 +02:00
Maximilian Bosch bec33858bb
primops: math.h -> cmath
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-05-10 16:41:10 +02:00
Eelco Dolstra b68a85df0a
Merge pull request #4776 from NixOS/fix-ca-normalization
Properly normalize the content-addressed paths
2021-05-10 16:11:38 +02:00
Maximilian Bosch 7f7f99f350
Implement builtins.floor and builtins.ceil using the C library functions internally
Closes #4782

Note: even though the type is internally called `NixFloat`, it's
actually a `double`.
2021-05-10 12:19:32 +02:00
Eelco Dolstra 2bb49fb6d0
Merge pull request #4786 from matthewbauer/mkdir-local-share-nix
Create parent for trusted list path before writing
2021-05-10 11:14:45 +02:00
Matthew Bauer 2f63cc02de Create parent trusted list path before writing
This makes sure that $HOME/.local/share/nix exists before we try to
write to it.
2021-05-08 21:31:28 -05:00
regnat b66234134f Add a realisations disk cache
Similar to the nar-info disk cache (and using the same db).
This makes rebuilds muuch faster.

- This works regardless of the ca-derivations experimental feature.
  I could modify the logic to not touch the db if the flag isn’t there,
  but given that this is a trash-able local cache, it doesn’t seem to be
  really worth it.
- We could unify the `NARs` and `Realisation` tables to only have one
  generic kv table. This is left as an exercise to the reader.
- I didn’t update the cache db version number as the new schema just
  adds a new table to the previous one, so the db will be transparently
  migrated and is backwards-compatible.

Fix #4746
2021-05-06 17:38:20 +02:00
Matthew Bauer 1e3a9033b7 Use derivation output name from toDerivation
This was previously done in https://github.com/NixOS/nix/pull/4515 but
got clobbered away in https://github.com/NixOS/nix/pull/4594.

--------------------------------------------------------------------------------

This fixes an issue where derivations with a primary output that is
not "out" would fail with:

$ nix profile install nixpkgs#sqlite
error: opening directory '/nix/store/2a2ydlgyydly5czcc8lg12n6qqkfz863-sqlite-3.34.1-bin': No such file or directory

This happens because while derivations produce every output when
built, you might not have them if you didn't build the derivation
yourself (for instance, the store path was fetch from a binary cache).
This uses outputName provided from DerivationInfo which appears to
match the first output of the derivation.
2021-05-05 16:33:05 -05:00
regnat bf485dcf46 Properly normalize the content-addressed paths
Make sure that their timestamp are always normalized.
Otherwise, strange − and non-deterministic − things might happen, like
https://github.com/NixOS/nixpkgs/issues/121813

Fix #4775
2021-05-05 21:00:08 +02:00
regnat dadfbce318 Fix the double-slash in the realisations path
Make sure that we always access the realisations under
`binaryCacheUrl/realisations` and not `binaryCacheUrl//realisations`

Fix #4766
2021-05-04 10:35:34 +02:00
Eelco Dolstra ef13c9c223
Merge pull request #4750 from NinjaTrappeur/nin-build-dry-run-json
nix build: make dry-run to print a json output if --json is enabled
2021-05-03 16:14:29 +02:00
Eelco Dolstra 40378fbcba
Merge pull request #4761 from emilazy/issue-4658-mark-impure-host-deps-as-optional
Mark `__impureHostDeps` paths as optional
2021-05-03 16:13:36 +02:00
Maximilian Bosch e5951a6b2f
Bump version number for DerivedPath changes
I guess I misunderstood John's initial explanation about why wildcards
for outputs are sent to older stores[1]. My `nix-daemon` from 2021-03-26
also has version 1.29, but misses the wildcard[2]. So bumping seems to
be the right call.

[1] https://github.com/NixOS/nix/pull/4759#issuecomment-830812464
[2] 255d145ba7
2021-05-03 01:12:23 +02:00
Emily c4355a52fa Mark __impureHostDeps paths as optional
Starting in macOS 11, the on-disk dylib bundles are no longer available,
but nixpkgs needs to be able to keep compatibility with older versions
that require `/usr/lib/libSystem.B.dylib` in `__impureHostDeps`. Allow
it to keep backwards compatibility with these versions by marking these
dependencies as optional.

Fixes #4658.
2021-05-02 05:30:50 +01:00
Domen Kožar e3e78ee2a2
Merge pull request #4751 from Ma27/storepath-pos
primops/storePath: add trace to pure mode error
2021-04-27 19:05:23 +02:00
Maximilian Bosch e6a360a4d9
primops/storePath: add trace to pure mode error
As described in #4745 it's otherwise fairly hard to understand where
this is coming from. Say you have an expression which uses e.g.
`types.package`:

``` nix
{ outputs = { self, nixpkgs }: {
    packages.x86_64-linux.hello = let
      foo = nixpkgs.lib.evalModules {
        modules = [
          {
            options.foo.bar = with nixpkgs.lib; mkOption { type = types.package; };
          }
          {
            foo.bar = ./.;
          }
        ];
      };
    in builtins.trace foo.config.foo.bar.outPath nixpkgs.legacyPackages.x86_64-linux.hello;
    defaultPackage.x86_64-linux = self.packages.x86_64-linux.hello;
  };
}
```

Then you'll get an error trace like this:

```
error: 'builtins.storePath' is not allowed in pure evaluation mode

       at /nix/store/p4h2x6r80njkb0j2rc1xjhhl99yri3zb-source/lib/attrsets.nix:328:15:

          327|     let
          328|       path' = builtins.storePath path;
             |               ^
          329|       res =

       … while evaluating the attribute 'config.foo.bar.outPath'

       at /nix/store/p4h2x6r80njkb0j2rc1xjhhl99yri3zb-source/lib/attrsets.nix:332:11:

          331|           name = sanitizeDerivationName (builtins.substring 33 (-1) (baseNameOf path'));
          332|           outPath = path';
             |           ^
          333|           outputs = [ "out" ];

       … while evaluating the attribute 'packages.x86_64-linux.hello'

       at /nix/store/6c1rfsqzrhjw1235palzjmf5vihcpci7-source/flake.nix:3:5:

            2| { outputs = { self, nixpkgs }: {
            3|     packages.x86_64-linux.hello = let
             |     ^
            4|       foo = nixpkgs.lib.evalModules {
```

Fixes #4745
2021-04-27 17:24:38 +02:00
Félix Baylac-Jacqué 7dca9c24c4
nix build: make dry-run to print a json output if --json is enabled 2021-04-27 11:28:47 +02:00
Eelco Dolstra fe2bf464cf Merge branch 'remove-trailing-spaces' of github.com:NixOS/nix 2021-04-23 15:09:44 +02:00
regnat 31313d1401 Replace the trailing markdown spaces by a backslash
They are equivalent according to
<https://spec.commonmark.org/0.29/#hard-line-breaks>,
and the trailing spaces tend to be a pain (because the make git
complain, editors tend to want to remove them − the `.editorconfig`
actually specifies that − etc..).
2021-04-23 14:37:21 +02:00
Eelco Dolstra 4549d65b4f
Merge pull request #4732 from NixOS/4725-always-register-the-realisations
Aways register the realisations
2021-04-23 14:22:03 +02:00
Eelco Dolstra 293220bed5
Merge pull request #4440 from Ma27/misc-pos-fixes
Miscellaneous improvements for positioning in eval-errors
2021-04-23 11:10:59 +02:00
regnat d3df747cb6 Revert "Make nix shell fallback to static outputs when needed"
This reverts commit 8d66f5f110.

This fix isn't needed anymore now that the realisations are always
properly registered
2021-04-23 09:34:43 +02:00
p01arst0rm 45473d02c9 unified macro style for ENABLE_S3 2021-04-23 07:30:05 +01:00
regnat 9161e02039 Always register the realisations of input-addressed drvs
Fix #4725
2021-04-22 20:07:02 +02:00
Eelco Dolstra d9864be4b7 Merge branch 'master' of github.com:NixOS/nix 2021-04-22 10:33:00 +02:00
Eelco Dolstra 78a2303b3d Merge branch 'nix-derivation-query-store-path' of https://github.com/matthewbauer/nix 2021-04-22 10:29:18 +02:00
Eelco Dolstra dd83b29b90
Merge pull request #4724 from NixOS/nix-shell-missing-outputs
Make `nix shell` fallback to static outputs when needed
2021-04-22 10:25:55 +02:00
Eelco Dolstra 5385755227 Remove makeDecompressionSource()
This function doesn't support all compression methods (i.e. 'none' and
'br') so it shouldn't be exposed.

Also restore the original decompress() as a wrapper around
makeDecompressionSink().
2021-04-22 10:23:20 +02:00
Eelco Dolstra a6eebcff36 Merge branch 's3-decompress' of https://github.com/lukegb/nix 2021-04-22 10:20:40 +02:00
Luke Granger-Brown 97dde3cdd9 libutil: allow decompression with none/empty method
The S3 store relies on the ability to be able to decompress things with
an empty method, because it just passes the value of the Content-Encoding
directly to decompress.

If the file is not compressed, then this will cause the compression
routine to get confused.

This caused NixOS/nixpkgs#120120.
2021-04-22 02:33:59 +00:00
Antoine Martin ac38c4cdc2 Fix typo in flake doc 2021-04-22 03:07:57 +02:00
Matthew Bauer 428062e578 Get deriver when passing --derivation
This makes Nix look up paths derivations when they are passed as a
store paths. So:

$ nix path-info --derivation /nix/store/0pisd259nldh8yfjvw663mspm60cn5v8-hello-2.10

now gives

/nix/store/qp724w90516n4bk5r9gfb37vzmqdh3z7-hello-2.10.drv

instead of "".

If no deriver is available, Nix now errors instead of silently
ignoring that argument.
2021-04-21 15:32:33 -05:00
regnat 8d66f5f110 Make nix shell fallback to static outputs when needed
In case of pure input-addressed derivations, the build loop doesn't
guaranty that the realisations are stored in the db (if the output paths
are already there or can be substituted, the realisations won't be
registered). This caused `nix shell` to fail in some cases because it
was assuming that the realisations were always existing.

A better (but more involved) fix would probably to ensure that we always
register the realisations, but in the meantime this patches the surface
issue.

Fix #4721
2021-04-21 17:46:27 +02:00
Eelco Dolstra 8d651a1f68 Fix brotli compression of files > 128 KiB
This has been broken since faa31f4084.
2021-04-20 22:54:49 +02:00
Eelco Dolstra b60b0d62d6
Merge pull request #4720 from alyssais/wait
Include sys/wait.h everywhere WIFEXITED etc is used
2021-04-20 16:09:30 +02:00
Alyssa Ross 9ac6534f7c
Include sys/wait.h everywhere WIFEXITED etc is used
This is required on NetBSD, and I think FreeBSD too.
2021-04-19 18:31:58 +00:00
Maximilian Bosch 6905291daf
libcmd/installables: force re-evaluation of cached failures
I think that it's not very helpful to get "cached failures" in a wrong
`flake.nix`. This can be very confusing when debugging a Nix expression.
See for instance NixOS/nixpkgs#118115.

In fact, the eval cache allows a forced reevaluation which is used for
e.g. `nix eval`.

This change makes sure that this is the case for `nix build` as well. So
rather than

    λ ma27 [~/Projects/exp] → ../nix/outputs/out/bin/nix build -L --rebuild --experimental-features 'nix-command flakes'
    error: cached failure of attribute 'defaultPackage.x86_64-linux'

the evaluation of already-evaluated (and failed) attributes looks like
this now:

    λ ma27 [~/Projects/exp] → ../nix/outputs/out/bin/nix build -L --rebuild --experimental-features 'nix-command flakes'
    error: attribute 'hell' missing

           at /nix/store/mrnvi9ss8zn5wj6gpn4bcd68vbh42mfh-source/flake.nix:6:35:

                5|
                6|     packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hell;
                 |                                   ^
                7|
    (use '--show-trace' to show detailed location information)
2021-04-19 19:51:04 +02:00
Eelco Dolstra 3ee0ecdda0
Merge pull request #4679 from ony/feature/one-pass-canon-path
Optimize canonPath to resolve relative symlinks in one pass
2021-04-15 14:11:51 +02:00
Eelco Dolstra 15f4d4fd43 Drop libbz2 / zlib / lzma dependency + style fixes 2021-04-15 13:55:22 +02:00
Eelco Dolstra 6fb7582413 Merge branch 'libarchive-decompress' of https://github.com/serokell/nix 2021-04-15 13:39:04 +02:00
Eelco Dolstra f716779c1f Merge branch 'revertRemoveJSON' of https://github.com/cgohla/nix 2021-04-15 13:26:56 +02:00
Eelco Dolstra 709a60a045
Merge pull request #4707 from grahamc/defaul-path-restricted-eval
EvalSettings::getDefaultNixPath: respect {restrict,pure}Eval
2021-04-14 11:02:58 +02:00
Eelco Dolstra b47b5f4061
Merge pull request #4708 from NixOS/fix-remote-registerDrvOutput
Fix registerDrvOutput with the daemon
2021-04-14 11:01:36 +02:00
Maximilian Bosch 864ef0e93d
libexpr/primops: review 2021-04-13 23:12:39 +02:00
Maximilian Bosch ad7b47dc85
primops/libexpr: use new attr-call extractor everywhere; use function's pos if attr-set pos == noPos 2021-04-13 23:12:38 +02:00
Maximilian Bosch f60473a077
libexpr/primops: Move attr name extraction into its own function
This now takes care of providing positioning for both the faulting value
and the faulting function call in case of an error.
2021-04-13 23:12:38 +02:00
Maximilian Bosch 7c76964daa
libexpr: misc improvements for proper error position
When working on some more complex Nix code, there are sometimes rather
unhelpful or misleading error messages, especially if coerce-errors are
thrown.

This patch is a first steps towards improving that. I'm happy to file
more changes after that, but I'd like to gather some feedback first.

To summarize, this patch does the following things:

* Attrsets (a.k.a. `Bindings` in `libexpr`) now have a `Pos`. This is
  helpful e.g. to identify which attribute-set in `listToAttrs` is
  invalid.

* The `Value`-struct has a new method named `determinePos` which tries
  to guess the position of a value and falls back to a default if that's
  not possible.

  This can be used to provide better messages if a coercion fails.

* The new `determinePos`-API is used by `builtins.concatMap` now. With
  that change, Nix shows the exact position in the error where a wrong
  value was returned by the lambda.

  To make sure it's still obvious that `concatMap` is the problem,
  another stack-frame was added.

* The changes described above can be added to every other `primop`, but
  first I'd like to get some feedback about the overall approach.
2021-04-13 23:12:38 +02:00
Maximilian Bosch 3550a32b25
primops/derivation: use position of currently evaluated attribute
* The position of the `name`-attribute appears in the trace.
* If e.g. `meta` has no `outPath`-attribute, a `cannot coerce set to
  string` error will be thrown where `pos` points to `name =` which is
  highly misleading.
2021-04-13 23:12:38 +02:00
regnat ed29610cc6 Fix registerDrvOutput with the daemon
Resolve a protocol issue that caused the daemon to endlessly wait for
some information that the client doesn't ever send
2021-04-13 13:58:22 +02:00
Graham Christensen d04969ffa5 EvalSettings::getDefaultNixPath: respect {restrict,pure}Eval
Otherwise Nix may look to invalid locations for channels.
2021-04-12 12:37:15 -04:00
Domen Kožar c2dfda007e
add tests for zstd compression 2021-04-09 23:13:08 +02:00
Eelco Dolstra 42f0246698
Revert "libfetchers/tarball: Lock on effectiveUrl"
This reverts commit fc6bfb261d.

Fixes #4672.
2021-04-09 13:51:57 +02:00
Eelco Dolstra 906adadacd Restore stack size in child processes
Fixes #4673.
2021-04-07 13:40:13 +02:00
Eelco Dolstra 9b9e703df4 restoreSignals() + restoreAffinity() -> restoreProcessContext() 2021-04-07 13:10:02 +02:00
Eelco Dolstra 8a29052cb2 PathSubstitutionGoal: Clean up pipe
If there were many top-level goals (which are not destroyed until the
very end), commands like

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' \
    /run/current-system --no-check-sigs --substitute-on-destination

could fail with "Too many open files". So now we do some explicit
cleanup from amDone(). It would be cleaner to separate goals from
their temporary internal state, but that would be a bigger refactor.
2021-04-07 12:21:31 +02:00
Björn Gohla 9f28dd97ae Revert "Use upstream nlohmann_json"
This reverts commit 4145cd2da0.
2021-04-05 21:24:55 +01:00
John Ericson 125a824228 Document the derived path types. 2021-04-05 10:56:48 -04:00
John Ericson d8fa7517fa buildable.{cc,hh} -> derived-path.{cc,hh} 2021-04-05 10:33:28 -04:00
John Ericson 179582872d Make DerivedPathWithHints a newtype
This allows us to namespace its constructors under it.
2021-04-05 10:05:21 -04:00
John Ericson 9b805d36ac Rename Buildable 2021-04-05 09:52:25 -04:00
John Ericson 9dfb97c987 "newtype" BuildableReq
This makes for better types errors and allows us to give it methods.
2021-04-05 09:35:55 -04:00
John Ericson 4fe41c6db3 No templates for Buildable and BuildableReq 2021-04-05 09:15:25 -04:00
John Ericson 255d145ba7 Use BuildableReq for buildPaths and ensurePath
This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {}
}` could mean "build `brvPath`" or "substitute `drvPath`" depending on
context.

It also brings the internals closer in line to the new CLI, by
generalizing the `Buildable` type is used there and makes that
distinction already.

In doing so, relegate `StorePathWithOutputs` to being a type just for
backwards compatibility (CLI and RPC).
2021-04-05 08:33:00 -04:00
John Ericson 32f4454b9f Move StorePathWithOutput utilities out of store class
These are by no means part of the notion of a store, but rather are
things that happen to use stores. (Or put another way, there's no way
we'd make them virtual methods any time soon.) It's better to move them
out of that too-big class then.

Also, this helps us remove StorePathWithOutputs from the Store interface
altogether next commit.
2021-04-05 08:31:37 -04:00
John Ericson 7a2b566dc8 Move StorePathWithOutputs into its own header/file
In the following commits it will become less prevalent.
2021-04-05 08:31:37 -04:00
John Ericson f7d9f7c338 Pull out Buildable into its own file/header in libnixstore 2021-04-05 08:31:37 -04:00
Mykola Orliuk f3f228700a canonPath in one pass 2021-03-31 04:58:49 +02:00
Bjørn Forsman edd606ae62 fetchGit: don't prefix "refs/heads/" on ref = "HEAD"
This fixes builtins.fetchGit { url = ...; ref = "HEAD"; }, that works in
stable nix (v2.3.10), but is broken in nix master:

  $ ./result/bin/nix repl
  Welcome to Nix version 2.4pre19700101_dd77f71. Type :? for help.

  nix-repl> builtins.fetchGit { url = "https://github.com/NixOS/nix"; ref = "HEAD"; }
  fetching Git repository 'https://github.com/NixOS/nix'fatal: couldn't find remote ref refs/heads/HEAD
  error: program 'git' failed with exit code 128

The documentation for builtins.fetchGit says ref = "HEAD" is the
default, so it should also be supported to explicitly pass it.

I came across this issue because poetry2nix can use ref = "HEAD" in some
situations.

Fixes #4674.
2021-03-30 11:21:38 +02:00
Eelco Dolstra dd77f71afe LocalBinaryCacheStore::upsertFile(): Fix race
When multiple threads try to upsert the same file, this could fail.

Fixes #4667.
2021-03-26 17:10:15 +01:00
Eelco Dolstra 4638bcfb2c Fix some typos
Fixes #4671.
2021-03-26 16:14:38 +01:00
Eelco Dolstra d1cb956bf2
Merge pull request #4650 from samueldr/fix/registry-config-dir
Use the appropriate config dir for the registry
2021-03-24 13:37:19 +01:00
Eelco Dolstra 1581c3e8ef
Merge pull request #4603 from DavHau/davhau-improve-nix-conf-help-builders
improve man page for nix.conf (builders)
2021-03-23 16:13:02 +01:00
Eelco Dolstra 6749d9e057
Merge pull request #4646 from Ma27/support-scoped-v6
Fix Nix to properly work with stores using a scoped IPv6 address
2021-03-23 15:03:47 +01:00
DavHau 0f40561c78 nix.conf builders: refer to manual page 2021-03-23 10:19:00 +07:00
John Ericson f44206e719 Fix typos in the last PR #4656 2021-03-22 15:18:48 +00:00
John Ericson 9d309de0de Clean up serialization for BuildResult
A few versioning mistakes were corrected:

- In 27b5747ca7, Daemon protocol had some
  version `>= 0xc` that should have been `>= 0x1c`, or `28` since the
  other conditions used decimal.

- In a2b69660a9, legacy SSH gated new CAS
  info on version 6, but version 5 in the server. It is now 6
  everywhere.

Additionally, legacy ssh was sending over more metadata than the daemon
one was. The daemon now sends that data too.

CC @regnat

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2021-03-22 14:57:41 +00:00
Samuel Dionne-Riel 66b857244f Use the appropriate config dir for the registry 2021-03-19 15:20:47 -04:00
Eelco Dolstra 3e0e443181 ProgressBar: Respect verbosity level
This makes its behaviour consistent with SimpleLogger.
2021-03-19 17:56:39 +01:00
Eelco Dolstra ef83ced4e1 Restore 'nix flake info' as a deprecated alias 2021-03-19 17:21:37 +01:00
Eelco Dolstra 8a5203d3b8
Merge pull request #4648 from hercules-ci/remove-unimplemented-hashAlgoOpt
Remove unimplemented hashAlgoOpt
2021-03-17 20:51:09 +01:00
Robert Hensing a61112aadf Remove unimplemented hashAlgoOpt
It was in the header but never implemented.
2021-03-17 11:27:11 +01:00
Maximilian Bosch ddcef3bcbb
Fix Nix to properly work with stores using a scoped IPv6 address
According to RFC4007[1], IPv6 addresses can have a so-called zone_id
separated from the actual address with `%` as delimiter. In contrast to
Nix 2.3, the version on `master` doesn't recognize it as such:

    $ nix ping-store --store ssh://root@fe80::1%18 --experimental-features nix-command
    warning: 'ping-store' is a deprecated alias for 'store ping'
    error: --- Error ----------------------------------------------------------------- nix
    don't know how to open Nix store 'ssh://root@fe80::1%18'

I modified the IPv6 match-regex accordingly to optionally detect this
part of the address. As we don't seem to do anything special with it, I
decided to leave it as part of the URL for now.

Fixes #4490

[1] https://tools.ietf.org/html/rfc4007
2021-03-16 19:14:42 +01:00
Eelco Dolstra 66fa1c7375 Merge 'nix flake {info,list-inputs}' into 'nix flake metadata'
Fixes #4613.
2021-03-16 17:19:04 +01:00
Eelco Dolstra 77f5d171e1 --override-input: Imply --no-write-lock-file
Fixes #3779.
2021-03-16 16:53:39 +01:00
Eelco Dolstra ccb8a403ee
Merge pull request #4587 from obsidiansystems/derivation-goal-detect-invalid-output
Throw error for derivation goal with bogus wanted output
2021-03-15 16:49:44 +01:00
Eelco Dolstra c0073f6268
Merge pull request #4580 from obsidiansystems/restore-test-build-remote-ca-fixed
Restore now-working build-remote-content-addressed-fixed test
2021-03-15 16:48:28 +01:00
Eelco Dolstra 91ea9c52ee
Merge pull request #4624 from NixOS/ca/realisation-nix-command
Add a `nix realisation` command for working on realisations
2021-03-15 16:37:09 +01:00
regnat 703c98c6cb Properly sign the unresolved drvs
Don't let them inherit the signature from the parent one (because it
makes no sense to do so), but re-sign them after they have been built
2021-03-15 16:35:17 +01:00
regnat 54ced9072b Check the signatures when copying store paths around
Broken atm
2021-03-15 16:35:14 +01:00
regnat 3e6017f911 pathInfoIsTrusted -> pathInfoIsUntrusted
I guess the rationale behind the old name wath that
`pathInfoIsTrusted(info)` returns `true` iff we would need to `blindly`
trust the path (because it has no valid signature and `requireSigs` is
set), but I find it to be a really confusing footgun because it's quite
natural to give it the opposite meaning.
2021-03-15 16:34:49 +01:00
regnat 826877cabf Add some logic for signing realisations
Not exposed anywhere, but built realisations are now signed (and this
should be forwarded when copy-ing them around)
2021-03-15 16:34:49 +01:00
Eelco Dolstra 306c154632
Merge pull request #4592 from NixOS/ca/remote-cache
Substitute content-addressed derivations
2021-03-15 16:22:42 +01:00
Eelco Dolstra 3bb1becdbb
Merge pull request #4566 from orbekk/master
Add support for bare git repositories when using git+file
2021-03-11 10:38:07 +01:00
Yorick 8a0c00b856
Use libarchive for all compression 2021-03-10 22:34:29 +01:00
regnat 89013bdd7e Add a nix realisation command for working on realisations
Currently only has `nix realisation info`, more to come probably
2021-03-09 10:16:44 +01:00
Eelco Dolstra 1c0e3e453d
Merge pull request #4601 from lovesegfault/fix-4598
nix-build: check that envCommand exists
2021-03-08 10:13:39 +01:00
Eelco Dolstra c5a232dda7
Merge pull request #4615 from NixOS/nix-show-stats-with-nix-cmd
Make NIX_SHOW_STATS work with new-style commands
2021-03-08 09:56:59 +01:00
Bernardo Meurer 6e849e3b0a
nix-build: set execfail
When starting a nix-shell with `-i` it was previously possible for it to
silently fail in the scenario where the specified interpreter didn't
exist. This happened due to the `exec` call masking the issue.

With this change we enable `execfail`, which causes the script using
`nix-shell` as interpreter to correctly exit with code 127.

Fixes: #4598
2021-03-04 18:54:45 -08:00
John Ericson 6212e89bf6 Avoid some StorePath -> Path -> StorePath roundtrips
There were done when StorePath was defined in Rust and there were some
FFI issues. This is no longer an issue.
2021-03-05 00:49:46 +00:00
DavHau e16431b466 improve man page for nix.conf (builders) 2021-03-04 16:14:23 +07:00
Eelco Dolstra 665d4ec2da nix repl :doc: Don't return docs for partially applied primops
This gives misleading results for Nixpkgs functions like lib.toUpper.

Fixes #4596.
2021-03-03 17:52:57 +01:00
regnat 7331da99ab Make NIX_SHOW_STATS work with new-style commands 2021-03-02 14:59:12 +01:00
dramforever fc6bfb261d libfetchers/tarball: Lock on effectiveUrl
Basically, if a tarball URL is used as a flake input, and the URL leads
to a redirect, the final redirect destination would be recorded as the
locked URL.

This allows tarballs under https://nixos.org/channels to be used as
flake inputs. If we, as before, lock on to the original URL it would
break every time the channel updates.
2021-03-02 21:56:50 +08:00
John Ericson 7ce10924c7 Fix bad wanted output error as requested
- UsageError -> Error

- include drv path too
2021-03-01 15:07:09 +00:00
Kjetil Orbekk 9931f18c2d Add support for bare git repositories with git+file
Local git repositories are normally used directly instead of
cloning. This commit checks if a repo is bare and forces a
clone.

Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2021-03-01 09:03:25 -05:00
regnat df9d4f88d5 Allow substituting drv outputs when building 2021-03-01 14:00:17 +01:00
regnat 5d1c05b075 SubstitutionGoal -> PathSubstitutionGoal
To prepare for the upcoming DrvOutputSubstitutionGoal
2021-03-01 14:00:17 +01:00
Eelco Dolstra e64cf8e0a3
Merge pull request #4574 from grahamc/libstore-ssh-host-key
libstore: support passing a builder's public SSH host key
2021-03-01 13:12:18 +01:00
John Ericson 4bbd80c536 Throw error for derivation goal with bogus wanted output 2021-02-28 00:19:35 +00:00
John Ericson ae1441e548 Fix testing fixed-output derivations in double sandboxes
What happened was that Nix was trying to unconditionally mount these
paths in fixed-output derivations, but since the outer derivation was
pure, those paths did not exist. The solution is to only mount those
paths when they exist.
2021-02-27 05:23:14 +00:00
Puck Meerburg 7241fdc3d2 Properly propagate libseccomp linker flags 2021-02-26 23:01:16 +00:00
John Ericson 553b79f8c9 Remove unused redirectedBadOutputs 2021-02-26 16:10:54 +00:00
John Ericson d560311f76 Remove temporary #if 0...#endif from previous commit 2021-02-26 16:10:52 +00:00
John Ericson 68f4c728ec Split {,local-}derivation-goal.{cc,hh}
This separates the scheduling logic (including simple hook pathway) from
the local-store needing code.

This should be the final split for now. I'm reasonably happy with how
it's turning out, even before I'm done moving code into
`local-derivation-goal`. Benefits:

1. This will help "witness" that the hook case is indeed a lot simpler,
   and also compensate for the increased complexity that comes from
   content-addressed derivation outputs.

2. It also moves us ever so slightly towards a world where we could use
   off-the-shelf storage or sandboxing, since `local-derivation-goal`
   would be gutted in those cases, but `derivation-goal` should remain
   nearly the same.

The new `#if 0` in the new files will be deleted in the following
commit. I keep it here so if it turns out more stuff can be moved over,
it's easy to do so in a way that preserves ordering --- and thus
prevents conflicts.

N.B.
```sh
git diff HEAD^^ --color-moved --find-copies-harder --patience --stat
```
makes nicer output.
2021-02-26 16:10:26 +00:00
John Ericson 05cc5a8587 Copy {,local-}derivation-goal.{cc,h}
Doing this prior to splitting, so we get better diff with default
options (e.g. on GitHub).
2021-02-26 16:01:47 +00:00
Eelco Dolstra 94637cd7e5
Merge pull request #4477 from NixOS/ca/build-remote
Build ca derivations remotely
2021-02-26 16:54:44 +01:00
regnat f54976d77b Simplify the case where the drv is a purely input-addressed one 2021-02-26 16:35:05 +01:00
Eelco Dolstra 076d2b04da
Update src/libstore/build/derivation-goal.cc 2021-02-26 16:30:12 +01:00
Eelco Dolstra 17c98e03ea
Update src/build-remote/build-remote.cc 2021-02-26 16:29:37 +01:00
Eelco Dolstra 14f51880ba
Update src/build-remote/build-remote.cc 2021-02-26 16:29:30 +01:00
Eelco Dolstra 453c3a603f nix flake update: Recreate the lock file
This is probably what most people expect it to do. Fixes #3781.

There is a new command 'nix flake lock' that has the old behaviour of
'nix flake update', i.e. it just adds missing lock file entries unless
overriden using --update-input.
2021-02-26 14:55:54 +01:00
Eelco Dolstra 20ea1de77d Use std::make_unique 2021-02-26 12:35:29 +01:00
Eelco Dolstra 73daffb81b Merge remote-tracking branch 'origin/deduplicate-static-hashes-computation' 2021-02-26 12:30:25 +01:00
Eelco Dolstra d2803406b5
Merge pull request #4487 from NixOS/ca/copy-drv-outputs
Copy ca derivation outputs
2021-02-26 12:20:42 +01:00
Eelco Dolstra 5edab777d1
Merge pull request #4530 from alyssais/kill
libutil: EPERM from kill(-1, ...) is fine
2021-02-26 12:19:16 +01:00
regnat c43f446f4e Make nix copy work without the ca-derivations flag
The experimental feature was by mistake required for `nix copy` to work
at oll
2021-02-25 17:21:51 +01:00
Théophane Hufschmitt c182aac98a Apply @edolstra stylistic suggestions
Mostly removing useless comments and adding spaces before `&`

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-25 17:21:51 +01:00
regnat 3b76f8f252 Ensure that the ca-derivations bit is set when copying realisations
This should already hold, but better ensure it for future-proof-nees
2021-02-25 17:19:59 +01:00
regnat f67ff1f575 Don't crash when copying realisations to a non-ca remote
Rather throw a proper exception, and catch&log it on the client side
2021-02-25 17:19:59 +01:00
regnat 2e199673a5 Use RealisedPaths in copyPaths
That way we can copy the realisations too (in addition to the store
paths themselves)
2021-02-25 17:18:48 +01:00
Graham Christensen 1130b28824
distributed builds: load remote builder host key from the machines file
This is already used by Hydra, and is very useful when materializing
a remote builder list from service discovery. This allows the service
discovery tool to only sync one file instead of two.
2021-02-25 09:17:34 -05:00
Shea Levy f6c5b05488
Respect command registrations in plugins. 2021-02-24 08:25:45 -05:00
Shea Levy 98d1b64400
Initialize plugins after handling initial command line flags
This is technically a breaking change, since attempting to set plugin
files after the first non-flag argument will now throw an error. This
is acceptable given the relative lack of stability in a plugin
interface and the need to tie the knot somewhere once plugins can
actually define new subcommands.
2021-02-24 08:22:17 -05:00
Shea Levy ec3497c1d6
Bail if plugin-files is set after plugins have been loaded.
We know the flag will be ignored but the user wants it to take effect.
2021-02-24 08:20:48 -05:00
Eelco Dolstra a878c448d8
Merge pull request #4551 from danieldk/system-features-compute-level
Add x86_64 compute levels as system features
2021-02-23 14:30:10 +01:00
regnat ba1a256d08 Make DerivationGoal::drv a full Derivation
This field used to be a `BasicDerivation`, but this `BasicDerivation`
was downcasted to a `Derivation` when needed (implicitely or not), so we
might as well make it a full `Derivation` and upcast it when needed.

This also allows getting rid of a weird duplication in the way we
compute the static output hashes for the derivation. We had to
do it differently and in a different place depending on whether the
derivation was a full derivation or just a basic drv, but we can now do
it unconditionally on the full derivation.

Fix #4559
2021-02-23 14:15:45 +01:00
regnat 527da73690 Properly bypass the registering step when all outputs are present
There was already some logic for that, but it didn't handle the case of
content-addressed outputs, so extend it a bit for that
2021-02-23 08:04:03 +01:00
regnat 69666b951e build-remote: Always register the missing outputs
It's possible that all the paths are already there, but just not
associated to the current drv output
2021-02-23 08:04:03 +01:00
regnat 8c385d16ee Also send ca outputs to the build hook
Otherwise they don't get registered, triggering an assertion failure
at some point later
2021-02-23 08:04:03 +01:00
regnat 27b5747ca7 RemoteStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat a2b69660a9 LegacySSHStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat 5687564a27 LocalStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat 6fbf3fe636 Make the build-hook work with ca derivations
- Pass it the name of the outputs rather than their output paths (as
  these don't exist for ca derivations)
- Get the built output paths from the remote builder
- Register the new received realisations
2021-02-23 08:04:03 +01:00
Shea Levy 35205e2e92
Warn about instability of plugin API 2021-02-22 17:10:55 -05:00
regnat e2f3b2eb42 Make missing auto-call arguments throw an eval error
The PR #4240 changed messag of the error that was thrown when an auto-called
function was missing an argument.
However this change also changed the type of the error, from `EvalError`
to a new `MissingArgumentError`. This broke hydra which was relying on
an `EvalError` being thrown.

Make `MissingArgumentError` a subclass of `EvalError` to un-break hydra.
2021-02-22 16:13:09 +01:00
Eelco Dolstra 574eb2be81
Tweak error message 2021-02-22 15:24:14 +01:00
Daniël de Kok 2de232d2b3 Add x86_64 compute levels as additional system types
When performing distributed builds of machine learning packages, it
would be nice if builders without the required SIMD instructions can
be excluded as build nodes.

Since x86_64 has accumulated a large number of different instruction
set extensions, listing all possible extensions would be unwieldy.
AMD, Intel, Red Hat, and SUSE have recently defined four different
microarchitecture levels that are now part of the x86-64 psABI
supplement and will be used in glibc 2.33:

https://gitlab.com/x86-psABIs/x86-64-ABI
https://lwn.net/Articles/844831/

This change uses libcpuid to detect CPU features and then uses them to
add the supported x86_64 levels to the additional system types. For
example on a Ryzen 3700X:

$ ~/aps/bin/nix -vv --version | grep "Additional system"
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
2021-02-22 09:11:15 +01:00
Eelco Dolstra 548437c234
Merge pull request #4541 from obsidiansystems/simpler-store-path-command
Deeper `Command` hierarchy to remove redundancy
2021-02-19 16:18:53 +01:00
regnat f483b623e9 Remove the drv resolution caching mechanism
It isn't needed anymore now that don't need to eagerly resolve
everything like we used to do. So we can safely get rid of it
2021-02-19 15:48:31 +01:00
regnat 4bc28c44f2 Store the output hashes in the initialOutputs of the drv goal
That way we
1. Don't have to recompute them several times
2. Can compute them in a place where we know the type of the parent
  derivation, meaning that we don't need the casting dance we had before
2021-02-19 15:48:31 +01:00
Théophane Hufschmitt 0bfbd04369 Don't expose the "bang" drvoutput syntax
It's not fixed nor useful atm, so better keep it hidden

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-19 15:48:31 +01:00
Théophane Hufschmitt 93d9eb78a0 Syntactic fixes
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-19 15:48:31 +01:00
regnat 87c8d3d702 Register the realisations for unresolved drvs
Once a build is done, get back to the original derivation, and register
all the newly built outputs for this derivation.

This allows Nix to work properly with derivations that don't have all
their build inputs available − thus allowing garbage collection and
(once it's implemented) binary substitution
2021-02-19 15:48:31 +01:00
regnat 263f6dbd1c Don't crash nix-build when not all outputs are realised
Change the `nix-build` logic for linking/printing the output paths to allow for
some outputs to be missing. This might happen when the toplevel
derivation didn't have to be built, either because all the required
outputs were already there, or because they have all been substituted.
2021-02-19 15:48:31 +01:00
Eelco Dolstra cd44c0af71 Increase default stack size on Linux
Workaround for #4550.
2021-02-18 19:22:37 +01:00
Eelco Dolstra 1b57825524 Document meta.mainProgram
Issue #4498.
2021-02-17 17:58:47 +01:00
Eelco Dolstra 7bd9898d5c nix run: Allow program name to be set in meta.mainProgram
This is useful when the program name doesn't match the package name
(e.g. ripgrep vs rg).

Fixes #4498.
2021-02-17 17:55:15 +01:00
Eelco Dolstra 13897afbe6 Throw an error if --arg / --argstr is used with a flake
Fixes #3949.
2021-02-17 17:32:10 +01:00
Eelco Dolstra f33878b656 Make 'nix --version -vv' work
Fixes #3743.
2021-02-17 17:11:21 +01:00
Eelco Dolstra cced73496b nix flake show: Handle 'overlays' output
Fixes #4542.
2021-02-17 16:53:39 +01:00
Eelco Dolstra 063de66909 nix develop: Fix quoted string handling
Fixes #4540.
2021-02-17 16:42:03 +01:00
John Ericson ad337c8697 Deeper Command hierarchy to remove redundancy
Simply put, we now have `StorePathCommand : public StorePathsCommand` so
`StorePathCommand` doesn't reimplement work.
2021-02-12 17:48:09 +00:00
Graham Christensen f2245091d0
Revert "narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash" 2021-02-09 12:26:41 -05:00
Eelco Dolstra ee3846b587
Merge pull request #4464 from tweag/nar-narhash-addressed
narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash
2021-02-09 14:47:39 +01:00
Eelco Dolstra fd6eaa1766
Merge pull request #4525 from sternenseemann/lowdown-0.8.0
libcmd/markdown: handle allocation errors in lowdown_term_rndr
2021-02-07 21:24:16 +01:00
Eelco Dolstra 37352aa7e1 Support --no-net for backwards compatibility 2021-02-07 20:44:56 +01:00
Alyssa Ross 7c112351d9
libutil: EPERM from kill(-1, ...) is fine
I tested a trivial program that called kill(-1, SIGKILL), which was
run as the only process for an unpriveleged user, on Linux and
FreeBSD.  On Linux, kill reported success, while on FreeBSD it failed
with EPERM.

POSIX says:

> If pid is -1, sig shall be sent to all processes (excluding an
> unspecified set of system processes) for which the process has
> permission to send that signal.

and

> The kill() function is successful if the process has permission to
> send sig to any of the processes specified by pid.  If kill() fails,
> no signal shall be sent.

and

> [EPERM]
>     The process does not have permission to send the signal to any
>     receiving process.

My reading of this is that kill(-1, ...) may fail with EPERM when
there are no other processes to kill (since the current process is
ignored).  Since kill(-1, ...) only attempts to kill processes the
user has permission to kill, it can't mean that we tried to do
something we didn't have permission to kill, so it should be fine to
interpret EPERM the same as success here for any POSIX-compliant
system.

This fixes an issue that Mic92 encountered[1] when he tried to review a
Nixpkgs PR on FreeBSD.

[1]: https://github.com/NixOS/nixpkgs/pull/81459#issuecomment-606073668
2021-02-07 13:56:50 +00:00
sternenseemann d0e34c85f8 libcmd/markdown: handle allocation errors in lowdown_term_rndr
We upgrade to lowdown 0.8.0 [1] which contains a fix/improvement to a
behavior mentioned in this issue thread [2] where a big part of
lowdown's API would just call exit(1) on allocation errors since that
is a satisfying behavior for the lowdown binary.

Now lowdown_term_rndr returns 0 if an allocation error occurred which we
check for in libcmd/markdown.cc.

Also the extern "C" { } wrapper around lowdown.h has been removed as it
is not necessary.

[1]: 6ca7c855a0/versions.xml (L987-L1006)
[2]: https://github.com/kristapsdz/lowdown/issues/45#issuecomment-756681153
2021-02-06 13:14:57 +01:00
Eelco Dolstra 480426a364 Add more instrumentation for #4270 2021-02-05 15:57:33 +01:00
Eelco Dolstra d7c27f21ab
Merge pull request #4372 from tweag/ca/drvoutputs-commands
Add a new Cmd type working on RealisedPaths
2021-02-05 13:03:50 +01:00
Eelco Dolstra c77f4a9012
Merge pull request #4515 from matthewbauer/fix-nix-profile-install-first-output
Use derivation output name from toDerivation
2021-02-05 12:19:10 +01:00
Eelco Dolstra 0187838e2e Add a trace to readLine() failures
Hopefully this helps to diagnose 'error: unexpected EOF reading a
line' on macOS.
2021-02-05 12:18:11 +01:00
regnat e69cfdebb0 Remove the visit machinery in RealisedPath
In addition to being some ugly template trickery, it was also totally
useless as it was used in only one place where I could replace it by
just a few extra characters
2021-02-05 11:42:33 +01:00
regnat d2091af231 Move the GENERATE_CMP macro to its own file
Despite being an ugly hack, it can probably be useful in a couple extra
places
2021-02-05 11:42:33 +01:00
Théophane Hufschmitt 43d409f669
Fix a whitespace issue
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-04 14:47:56 +01:00
Théophane Hufschmitt ca8facefb6
Normalize some error messages
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-04 14:47:28 +01:00
Matthew Bauer e38cd5becb Always enter first level of attrset in nix search
This makes nix search always go through the first level of an
attribute set, even if it's not a top level attribute. For instance,
you can now list all GHC compilers with:

$ nix search nixpkgs#haskell.compiler
...

This is similar to how nix-env works when you pass in -A.
2021-02-03 21:22:11 -06:00
sternenseemann 76d8bdfe35 Include note about type of catched errors in tryEval documentation
Reference #356.
2021-02-03 17:14:40 +01:00
Matthew Bauer 3d1bbabe55 Use derivation output name from toDerivation
This fixes an issue where derivations with a primary output that is
not "out" would fail with:

$ nix profile install nixpkgs#sqlite
error: opening directory '/nix/store/2a2ydlgyydly5czcc8lg12n6qqkfz863-sqlite-3.34.1-bin': No such file or directory

This happens because while derivations produce every output when
built, you might not have them if you didn't build the derivation
yourself (for instance, the store path was fetch from a binary cache).
This uses outputName provided from DerivationInfo which appears to
match the first output of the derivation.
2021-02-02 19:53:19 -06:00
Domen Kožar d0b74e2d25
--no-net -> --offline 2021-02-01 13:11:42 +00:00
Eelco Dolstra b19aec7eeb
Merge pull request #4461 from NixOS/ca/error-logging-fixes
Fix some logging with ca derivations
2021-01-29 16:12:50 +01:00
James Ottaway 991edaace5
Shorten mktemp flag for macOS
Address `mktemp: illegal option -- -`.
2021-01-29 13:55:18 +10:00
regnat 9355ecd543 Add a new Cmd type working on RealisedPaths
Where a `RealisedPath` is a store path with its history, meaning either
an opaque path for stuff that has been directly added to the store, or a
`Realisation` for stuff that has been built by a derivation

This is a low-level refactoring that doesn't bring anything by itself
(except a few dozen extra lines of code :/ ), but raising the
abstraction level a bit is important on a number of levels:

- Commands like `nix build` have to query for the realisations after the
  build is finished which is fragile (see
  27905f12e4a7207450abe37c9ed78e31603b67e1 for example). Having them
  oprate directly at the realisation level would avoid that
- Others like `nix copy` currently operate directly on (built) store
  paths, but need a bit more information as they will need to register
  the realisations on the remote side
2021-01-28 09:38:44 +01:00
Eelco Dolstra b8f345b29a
Merge pull request #4483 from shlevy/libcmd
Move command plugin interface to libnixcmd
2021-01-27 17:17:58 +01:00
Eelco Dolstra 12de0466fe Add trace to build errors during import-from-derivation
Example:

error: builder for '/nix/store/9ysqfidhipyzfiy54mh77iqn29j6cpsb-failing.drv' failed with exit code 1;
       last 1 log lines:
       > FAIL
       For full logs, run 'nix log /nix/store/9ysqfidhipyzfiy54mh77iqn29j6cpsb-failing.drv'.

       … while importing '/nix/store/pfp4a4bjh642ylxyipncqs03z6kkgfvy-failing'

       at /nix/store/25wgzr2qrqqiqfbdb1chpiry221cjglc-source/flake.nix:58:15:

           57|
           58|         ifd = import self.hydraJobs.broken;
             |               ^
           59|
2021-01-27 14:46:10 +01:00
Eelco Dolstra 965dc6070a Drop trailing whitespace 2021-01-27 14:04:49 +01:00
Eelco Dolstra c03f41055d Add traces to errors while updating flake lock file
Example:

$ nix build --show-trace
error: unable to download 'https://api.github.com/repos/NixOS/nixpkgs/commits/no-such-branch': HTTP error 422 ('')

       response body:

       {
         "message": "No commit found for SHA: no-such-branch",
         "documentation_url": "https://docs.github.com/rest/reference/repos#get-a-commit"
       }

       … while fetching the input 'github:NixOS/nixpkgs/no-such-branch'

       … while updating the flake input 'nixpkgs'

       … while updating the lock file of flake 'git+file:///home/eelco/Dev/nix'
2021-01-27 14:02:54 +01:00
Eelco Dolstra 8e758d402b Remove mkFlag() 2021-01-27 12:06:03 +01:00
regnat 9da11bac57 Fix the error message when a dep is missing
Fix a mismatch in the errors thrown when a needed output was missing
from an input derivation that was leading to a wrong and quite misleading error
message
2021-01-26 14:49:23 +01:00
regnat d3c4284133 Make the error message for missing outputs more useful
Don't only show the name of the output, but also the derivation to which
this output belongs (as otherwise it's very hard to track back what went
wrong)
2021-01-26 14:49:23 +01:00
Shea Levy 6af6e41df0
Move command plugin interface to libnixcmd 2021-01-26 06:22:24 -05:00
Eelco Dolstra 36c4d6f592 Group common options 2021-01-25 19:03:13 +01:00
Eelco Dolstra 807d963ee8 Group subcommands by category 2021-01-25 18:19:32 +01:00
Eelco Dolstra 3ba98ba8f0 Tell user to run 'nix log' to get full build logs 2021-01-25 17:15:38 +01:00
Eelco Dolstra a32073e7e8 Add FIXME 2021-01-25 14:43:16 +01:00
Eelco Dolstra b159d23800 Make '--help' do the same as 'help' (i.e. show a manpage) 2021-01-25 14:38:15 +01:00
Eelco Dolstra 488a826842
Merge pull request #4467 from edolstra/error-formatting
Improve error formatting
2021-01-25 12:50:57 +01:00
Eelco Dolstra c5b42c5a42
Merge pull request #4470 from matthewbauer/fix-4469
Handle missing etag in 304 Not Modified response
2021-01-25 12:50:13 +01:00
Eelco Dolstra 680d8a5b86
Merge pull request #4387 from obsidiansystems/non-local-store-build
Make `nix-build --store whatever` work
2021-01-25 12:24:23 +01:00
Matthew Bauer 1ea5f0b66c Remove expectedETag assert in tarball.cc 2021-01-22 23:19:52 -06:00
Matthew Bauer a766824660 Handle missing etag in 304 Not Modified response
GitHub now omits the etag, but 304 implies it matches the one we
provided. Just use that one to avoid having an etag-less resource.

Fixes #4469
2021-01-22 14:47:45 -06:00
John Ericson 53a709535b Apply suggestions from code review
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-22 15:58:58 +00:00
John Ericson 8c07ed1dda Improve documentation and test and requested 2021-01-22 15:58:58 +00:00
Eelco Dolstra b7bfc7ee52
Add FIXME 2021-01-22 12:36:50 +01:00
Eelco Dolstra 0eb22db311 Fix macOS build 2021-01-21 12:46:22 +01:00
Eelco Dolstra 55849e153e Change error position formatting
It's now

  at /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix:7:7:

instead of

  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

The new format is more standard and clickable.
2021-01-21 11:02:09 +01:00
Eelco Dolstra 40608342cb Remove trailing whitespace 2021-01-21 11:02:09 +01:00
Eelco Dolstra 8d4268d190 Improve error formatting
Changes:

* The divider lines are gone. These were in practice a bit confusing,
  in particular with --show-trace or --keep-going, since then there
  were multiple lines, suggesting a start/end which wasn't the case.

* Instead, multi-line error messages are now indented to align with
  the prefix (e.g. "error: ").

* The 'description' field is gone since we weren't really using it.

* 'hint' is renamed to 'msg' since it really wasn't a hint.

* The error is now printed *before* the location info.

* The 'name' field is no longer printed since most of the time it
  wasn't very useful since it was just the name of the exception (like
  EvalError). Ideally in the future this would be a unique, easily
  googleable error ID (like rustc).

* "trace:" is now just "…". This assumes error contexts start with
  something like "while doing X".

Example before:

  error: --- AssertionError ---------------------------------------------------------------------------------------- nix
  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

       6|
       7|   x = assert false; 1;
        |       ^
       8|

  assertion 'false' failed
  ----------------------------------------------------- show-trace -----------------------------------------------------
  trace: while evaluating the attribute 'x' of the derivation 'hello-2.10'
  at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

     191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
     192|           name = "${attrs.pname}-${attrs.version}";
        |           ^
     193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {

Example after:

  error: assertion 'false' failed

         at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

              6|
              7|   x = assert false; 1;
               |       ^
              8|

         … while evaluating the attribute 'x' of the derivation 'hello-2.10'

         at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

            191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
            192|           name = "${attrs.pname}-${attrs.version}";
               |           ^
            193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 11:02:09 +01:00
adisbladis 144cad9069
narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash
This change is to simplify [Trustix](https://github.com/tweag/trustix) indexing and makes it possible to reconstruct this URL regardless of the compression used.

In particular this means that 7c2e9ca597/contrib/nix/nar/nar.go (L61-L71) can be removed and only the bits that are required to establish trust needs to be published in the Trustix build logs.
2021-01-21 10:32:56 +01:00
Eelco Dolstra 259100332f Fix clang build 2021-01-21 10:29:51 +01:00
Eelco Dolstra d04d8463fa
Merge pull request #4281 from lilyball/shebang
Escape filename given to nix-shell in shebang mode
2021-01-20 10:00:10 +01:00
Eelco Dolstra bc90252cec nix profile install: Support installing non-flakes
Fixes #4458.
2021-01-18 23:08:58 +01:00
Eelco Dolstra 555940f065 Use enumerate() 2021-01-18 22:50:39 +01:00
Eelco Dolstra ea756b3654 --refresh: Imply setting .narinfo disk cache TTL to 0 2021-01-18 14:38:31 +01:00
Eelco Dolstra 1bbc66f865 Merge branch 'slashes-in-github-branches' of https://github.com/Ma27/nix 2021-01-18 12:55:06 +01:00
regnat 11b63740e3 Fix content-addressed flake outputs
Prevent some `nix flake` commands to crash by trying to parse a
placeholder output as a store path
2021-01-18 11:31:59 +01:00
regnat 9432c170e7 Fix the drv output map for non ca derivations
With the `ca-derivation` experimental features, non-ca derivations used
to have their output paths returned as unknown as long as they weren't
built (because of a mistake in the code that systematically erased the
previous value)
2021-01-18 11:01:39 +01:00
Eelco Dolstra 1acbb61696 Tweak 2021-01-17 19:49:28 +01:00
ryneeverett 1e13c79a91 Document expected output of 'nix store ping'.
While interpreting the output is fairly intuitive it would be better to
explicitly specify what a good invocation looks like.

That this isn't completely obvious (or at least causes folks to
second-guess themselves) can be seen in a couple user threads:

- https://discourse.nixos.org/t/nixos-cache-fetching-issue/3575/11
- https://discourse.nixos.org/t/newbie-question-cant-get-trivial-example-of-nixops-to-work-on-my-mac/1125/8
2021-01-16 19:11:10 +00:00
Domen Kožar 00f99fdfe6
Merge pull request #4240 from bburdette/2259-error-message
2259 error message - "auto-call" error
2021-01-15 18:26:21 +00:00
John Ericson 7af743470c Make public keys and requireSigs local-store specific again
Thanks @regnat and @edolstra for catching this and comming up with the
solution.

They way I had generalized those is wrong, because local settings for
non-local stores is confusing default. And due to the nature of C++
inheritance, fixing the defaults is more annoying than it should be.
Additionally, I thought we might just drop the check in the substitution
logic since `Store::addToStore` is now streaming, but @regnat rightfully
pointed out that as it downloads dependencies first, that would still be
too late, and also waste effort on possibly unneeded/unwanted
dependencies.

The simple and correct thing to do is just make a store method for the
boolean logic, keeping all the setting and key stuff the way it was
before. That new method is both used by `LocalStore::addToStore` and the
substitution goal check. Perhaps we might eventually make it fancier,
e.g. sending the ValidPathInfo to remote stores for them to validate,
but this is good enough for now.
2021-01-15 16:37:41 +00:00
Jonathan Ringer 86a2ceeb98 Fix gcc10 build 2021-01-15 10:41:35 +01:00
John Ericson 0027b05a15 Merge remote-tracking branch 'upstream/master' into non-local-store-build 2021-01-15 02:01:24 +00:00
Eelco Dolstra 7a472a76d4 Add 'nix daemon' command 2021-01-14 00:05:04 +01:00
Eelco Dolstra 28ef6ebf91 Typo 2021-01-13 23:51:27 +01:00
Eelco Dolstra d33eca8539 Rename 'nix store sign-paths' to 'nix store sign' 2021-01-13 23:32:37 +01:00
Eelco Dolstra 61216d32e1 Add 'nix store repair' command 2021-01-13 23:27:39 +01:00
Eelco Dolstra 3da9a9241c Convert option descriptions to Markdown 2021-01-13 14:18:04 +01:00
Eelco Dolstra 4e9cec79bf
Merge pull request #4444 from matthewbauer/unset-curproc-arch-affinity
Set kern.curproc_arch_affinity=0 to escape Rosetta
2021-01-13 12:16:53 +01:00
Eelco Dolstra fbfa70dc02
Merge pull request #4443 from rickynils/prefer-local-build-respect-zero-max-jobs
Don't let 'preferLocalBuild' override 'max-jobs=0'
2021-01-13 12:15:54 +01:00
Rickard Nilsson 0ca1a50132 Remove a redundant condition in DerivationGoal::tryLocalBuild() 2021-01-13 10:13:51 +01:00
Eelco Dolstra 2f463e90ed Add 'nix profile history' command
Replaces 'nix-env --list-generations'. Similar to 'nix profile
diff-closures' but shows only the changes in top-level packages.
2021-01-12 23:53:53 +01:00