Commit graph

6910 commits

Author SHA1 Message Date
Eelco Dolstra c574ab3907
Merge pull request #5388 from yvt/fix-oahd-path
Add another path where a Rosetta 2 daemon plist file is possibly located
2021-10-15 09:02:16 +02:00
Kevin Amado 1bdeef8395
add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let
    x = builtins.fetchMercurial x;
  in
  x
  ```
- Before:
  ```bash
  $ nix-instantiate --show-trace --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  nix-instantiate --show-trace --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/test/default.nix:2:7:

            1| let
            2|   x = builtins.fetchMercurial x;
             |       ^
            3| in
  ```

Mentions: #3505
2021-10-14 23:23:05 -05:00
yvt a9d9e55551 Add another path where a Rosetta 2 configuration file is possibly located 2021-10-15 09:48:15 +09:00
Eelco Dolstra 17e6ebcc90 Speed up GC by marking entire closures as live 2021-10-14 14:13:57 +02:00
Eelco Dolstra 0154fa30cf Remove GCState 2021-10-14 13:52:49 +02:00
Eelco Dolstra 0317ffdad3 Move deleteFromStore() 2021-10-14 13:34:48 +02:00
Eelco Dolstra 0be8cc1466 pathInfoCache: Use the entire base name as the cache key
This fixes a bug in the garbage collector where if a path
/nix/store/abcd-foo is valid, but we do a
isValidPath("/nix/store/abcd-foo.lock") first, then a negative entry
for /nix/store/abcd is added to pathInfoCache, so /nix/store/abcd-foo
is subsequently considered invalid and deleted.
2021-10-14 13:28:22 +02:00
Eelco Dolstra eab934cb2a Make the canReachRoots() traversal non-recursive 2021-10-14 12:34:32 +02:00
Eelco Dolstra 09b14ea97a Cleanup 2021-10-14 10:04:13 +02:00
kvtb eae29b0385
fix build with gcc11 2021-10-13 18:03:33 +00:00
Eelco Dolstra 35c98a59c5 Fix GC when there are cycles in the referrers graph
(where "referrers" includes the reverse of derivation outputs and
derivers). Now we do a full traversal to look if we can reach any
root. If not, all paths reached can be deleted.
2021-10-13 12:12:44 +02:00
Eelco Dolstra 1785ba2980 Simplify 2021-10-13 12:12:44 +02:00
Eelco Dolstra dced45f146 strcpy -> memcpy
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
2021-10-13 12:12:44 +02:00
Eelco Dolstra 262520fcfe Use a thread per connection 2021-10-13 12:12:44 +02:00
Eelco Dolstra ff453b06f9 Fix auto-gc 2021-10-13 12:12:44 +02:00
Eelco Dolstra 8614cf1334 Non-blocking garbage collector
The garbage collector no longer blocks other processes from
adding/building store paths or adding GC roots. To prevent the
collector from deleting store paths just added by another process,
processes need to connect to the garbage collector via a Unix domain
socket to register new temporary roots.
2021-10-13 12:12:44 +02:00
Eelco Dolstra 9947f1646a Remove syncWithGC() 2021-10-13 12:12:44 +02:00
Eelco Dolstra 8eac7dfad4 Remove trash directory 2021-10-13 12:12:44 +02:00
Eelco Dolstra 624dfde3df
Merge pull request #5362 from Artturin/nixunpack
nix develop: add --unpack
2021-10-13 11:39:08 +02:00
Eelco Dolstra 0fac86fd6f Style tweaks 2021-10-13 11:00:10 +02:00
Eelco Dolstra abd685d373 Merge branch 'feature/comp-level' of https://github.com/tomberek/nix 2021-10-13 10:45:44 +02:00
Eelco Dolstra 3e0c6aac9a
Merge pull request #5375 from edolstra/repl-ctrl-c
nix repl: Fix plugin-files warning
2021-10-12 17:11:39 +02:00
Eelco Dolstra 5176b072ed Cleanup 2021-10-12 16:43:00 +02:00
Eelco Dolstra 3a778ea8a0 Merge branch 'nix-repl-download-interruption' of https://github.com/Ma27/nix 2021-10-12 16:34:38 +02:00
Eelco Dolstra f6cdae5181 nix repl: Don't write to std::cout directly
Writing to std::cout doesn't play nice with ProgressBar.
2021-10-12 15:36:45 +02:00
Eelco Dolstra 9ebe02a81e nix repl: Don't build in a child process
Fixes #5356. This is a bit risky due to interrupts, but we have to
deal with those anyway (#5353).
2021-10-12 15:27:02 +02:00
Tom Bereknyei 03bb8f84e0 Add compression level for NARs
Based off on @dtzWill's #2276
2021-10-12 02:14:36 -04:00
regnat 7466048d39 (partially) Revert "Don't copy in rethrow"
This reverts some parts of commit
8430a8f086 which was trying to rethrow
some exceptions while we weren’t in the context of a `catch` block,
causing some weird “terminate called without an active exception”
errors.

Fix #5368
2021-10-11 10:51:22 +02:00
Artturin e399c6ab7f nix develop: add --unpack 2021-10-09 01:19:50 +03:00
Sergei Trofimovich d7d6fe44d6 libstore-tests: add libutil dependency (fix static link failure)
In https://github.com/NixOS/nix/pull/5350 we noticed link failures
pkgsStatic.nixUnstable. Adding explicit dependency on libutil fixes
libstore-tests linking.
2021-10-08 07:16:12 +01:00
Maximilian Bosch 0872659002
nix repl: properly deal with interruptions
When I stop a download with Ctrl-C in a `nix repl` of a flake, the REPL
refuses to do any other downloads:

    nix-repl> builtins.getFlake "nix-serve"
    [0.0 MiB DL] downloading 'https://api.github.com/repos/edolstra/nix-serve/tarball/e9828a9e01a14297d15ca41 error: download of 'e9828a9e01' was interrupted
    [0.0 MiB DL]
    nix-repl> builtins.getFlake "nix-serve"
    error: interrupted by the user
    [0.0 MiB DL]

To fix this issue, two changes were necessary:

* Reset the global `_isInterrupted` variable: only because a single
  operation was aborted, it should still be possible to continue the
  session.
* Recreate a `fileTransfer`-instance if the current one was shut down by
  an abort.
2021-10-07 23:58:02 +02:00
Eelco Dolstra d39692e6b3 Make builtins.{path,filterSource} work with chroot stores 2021-10-07 14:22:39 +02:00
Eelco Dolstra c4dcf3cf25 Add a trace to all errors in addPath() 2021-10-07 13:47:15 +02:00
Eelco Dolstra 4806f2f6b0 Allow builtins.{path,filterSource} on paths with a context
We now build the context (so this has the side-effect of making
builtins.{path,filterSource} work on derivations outputs, if IFD is
enabled) and then check that the path has no references (which is what
we really care about).
2021-10-07 13:43:17 +02:00
Eelco Dolstra 66c4b20d8b Typo 2021-10-07 13:34:04 +02:00
Eelco Dolstra 7c50568788 Remove unnecessary call to queryMissing()
Worker::run() already does this.
2021-10-07 13:15:01 +02:00
Eelco Dolstra 972405edf5 Allow access to path copied to the store
Fixes https://github.com/NixOS/nix/pull/5163#issuecomment-931733912.
2021-10-07 12:15:22 +02:00
Eelco Dolstra cfaad7168e Refactoring: Add allowPath() method 2021-10-07 12:11:00 +02:00
Eelco Dolstra c9ee634f75
Merge pull request #5341 from andir/libexpr-formals
libexpr: remove matchAttrs boolean from ExprLambda
2021-10-07 11:58:56 +02:00
Eelco Dolstra 53e4794289
Merge pull request #5286 from ilkecan/add-a-warning-to-filterSource
Warn about the usage of filterSource with Nix store paths
2021-10-06 21:02:39 +02:00
ilkecan a4a6ef4fb2 Add a warning to filterSource
Warn about the usage of `filterSource` with Nix store paths
2021-10-06 19:25:33 +03:00
Eelco Dolstra 0dc8172458 Remove no-op call to realiseContext() 2021-10-06 18:08:37 +02:00
Eelco Dolstra c497fce011 Merge branch 'flakes_filterSource' of https://github.com/tomberek/nix 2021-10-06 18:08:18 +02:00
Eelco Dolstra 57a8eb4c01 fetchTree(): Parse type attribute first
The 'url' attribute depends on whether type == 'git', so this is needed for

  builtins.fetchTree {url = "git@github.com:NixOS/nix.git"; type = "git";}
2021-10-06 17:39:02 +02:00
Eelco Dolstra 83d86cc1b0 Cleanup 2021-10-06 17:30:10 +02:00
Andreas Rammhold cae41eebff libexpr: remove matchAttrs boolean from ExprLambda
The boolean is only used to determine if the formals are set to a
non-null pointer in all our cases. We can get rid of that allocation and
instead just compare the pointer value with NULL. Saving up to
sizeof(bool) + platform specific alignment per ExprLambda instace.
Probably not a lot of memory but perhaps a few kilobyte with nixpkgs?

This also gets rid of a potential issue with dereferencing formals based on
the value of the boolean that didn't have to be aligned with the formals
pointer but was in all our cases.
2021-10-06 17:24:06 +02:00
Eelco Dolstra 46753b5e9c Merge branch 'considerate/scp-like-urls' of https://github.com/considerate/nix 2021-10-06 17:19:27 +02:00
Eelco Dolstra c6718a9d95 Don't reset the logger in a vfork
9c766a40cb broke logging from the
daemon, because commonChildInit is called when starting the build hook
in a vfork, so it ends up resetting the parent's logger. So don't
vfork.

It might be best to get rid of vfork altogether, but that may cause
problems, e.g. when we call an external program like git from the
evaluator.
2021-10-06 13:54:59 +02:00
Eelco Dolstra 08aa7daee0 Remove links to .md files in help output
Fixes #5337.
2021-10-06 13:01:18 +02:00
Eelco Dolstra fd57e7074f
Merge pull request #5335 from edolstra/socket-paths
Support arbitrary-length socket paths
2021-10-05 17:36:07 +02:00
Eelco Dolstra b14bc06955 Don't ignore SIGCHLD in createUnixDomainSocket() 2021-10-05 13:23:16 +02:00
Eelco Dolstra e4a5d64a81 Show failing PID 2021-10-05 13:19:55 +02:00
Eelco Dolstra b299560872 Typo 2021-10-05 11:24:09 +02:00
Eelco Dolstra 3b7f4c7d9d Add FIXME about ptsname 2021-10-05 11:04:46 +02:00
Eelco Dolstra 43d4d75e22 Connect/bind Unix domain sockets in a child process
In the child process, we can do a chdir() and avoid the problem of the
path not fitting into sockaddr_un.
2021-10-05 10:44:59 +02:00
Eelco Dolstra 77ebbc9f54 Add a test for RefScanSink and clean up the code
Issue #5322.
2021-10-04 14:29:42 +02:00
Eelco Dolstra ef34fd0656 scanForReferences(): Use a StorePathSet 2021-10-04 13:47:38 +02:00
Eelco Dolstra 6fadb5df05
Merge pull request #5310 from trofi/drop-spammy-rewrite
local-derivation-goal.cc: drop spammy "warning: rewriting hashes in..…
2021-10-03 19:53:06 +02:00
kvtb 638c73776a
fix creation of NAR files >4GB on 32-bit platforms
`size_t` is 32-bit on 32-bit platforms, while the file size can be bigger
2021-10-02 21:04:01 +00:00
Alex Kladov 3e884aa002
Fix typo 2021-10-02 10:17:45 +03:00
Sergei Trofimovich 621aa65325 local-derivation-goal.cc: downgrade "warning: rewriting hashes in..." down to debug
Before the changes when building the whole system with
`contentAddressedByDefault = true;` we get many noninformative messages:

    $ nix build -f nixos system --keep-going
    ...
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0'; cross fingers
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0-dev'; cross fingers
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0-python'; cross fingers
    error: 2 dependencies of derivation '/nix/store/...-hub-2.14.2.drv' failed to build
    warning: rewriting hashes in '/nix/store/...-subversion-1.14.1.drv.chroot/nix/store/...-subversion-1.14.1-dev'; cross fingers
    warning: rewriting hashes in '/nix/store/...-subversion-1.14.1.drv.chroot/nix/store/...-subversion-1.14.1-man'; cross fingers
    ...

Let's downgrade these messages down to debug().
2021-10-01 17:41:44 +01:00
Eelco Dolstra 7cc220825d
Merge pull request #5167 from Ma27/keep-failed-on-ssh-remote-build
nix-store --serve: pass on `settings.keepFailed` from SSH store
2021-10-01 16:35:02 +02:00
John Ericson 242f9bf3dc std::visit by reference
I had started the trend of doing `std::visit` by value (because a type
error once mislead me into thinking that was the only form that
existed). While the optomizer in principle should be able to deal with
extra coppying or extra indirection once the lambdas inlined, sticking
with by reference is the conventional default. I hope this might even
improve performance.
2021-09-30 21:35:09 +00:00
Taeer Bar-Yam f14660d5e2 reset yylloc when yyless(0) is called 2021-09-29 19:47:01 -04:00
Eelco Dolstra fd01c48d34
Merge pull request #5301 from Ma27/builtins-missing-feature-error
libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
2021-09-29 12:53:29 +02:00
Maximilian Bosch 2b02ce0e48
libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
I found it somewhat confusing to have an error like

    error: attribute 'getFlake' missing

if the required experimental-feature (`flakes`) is not enabled. Instead,
I'd expect Nix to throw an error just like it's the case when using e.g. `nix
flake` without `flakes` being enabled.

With this change, the error looks like this:

    $ nix-instantiate -E 'builtins.getFlake "nixpkgs"'
    error: Cannot call 'builtins.getFlake' because experimental Nix feature 'flakes' is disabled. You can enable it via '--extra-experimental-features flakes'.

           at «string»:1:1:

                1| builtins.getFlake "nixpkgs"
                 | ^

I didn't use `settings.requireExperimentalFeature` here on purpose
because this doesn't contain a position. Also, it doesn't seem as if we
need to catch the error and check for the missing feature here since
this already happens at evaluation time.
2021-09-29 11:57:15 +02:00
Eelco Dolstra 6b5efeafd1
Merge pull request #5295 from Ma27/bmcheck-fod-mismatch
build: also throw hash-mismatch errors if `buildMode == bmCheck`
2021-09-29 10:13:45 +02:00
Maximilian Bosch bb1a851bcf
build: also throw hash-mismatch errors if buildMode == bmCheck
This actually bit me quite recently in `nixpkgs` because I assumed that
`nix-build --check` would also error out if hashes don't match anymore[1]
and so I wrongly assumed that I couldn't reproduce the mismatch error.

The fix is rather simple, during the output registration a so-called
`delayedException` is instantiated e.g. if a FOD hash-mismatch occurs.
However, in case of `nix-build --check` (or `--rebuild` in case of `nix
build`), the code-path where this exception is thrown will never be
reached.

By adding that check to the if-clause that causes an early exit in case
of `bmCheck`, the issue is gone. Also added a (previously failing)
test-case to demonstrate the problem.

[1] https://github.com/NixOS/nixpkgs/pull/139238, the underlying issue
    was that `nix-prefetch-git` returns different hashes than `fetchgit`
    because the latter one fetches submodules by default.
2021-09-27 15:44:39 +02:00
Eelco Dolstra 9c766a40cb Fix 'error: reading a line: Input/output error' in startBuilder()
With -vvvv, the ProgressBar was polluting the stderr of the child,
messing up its \2 message to the parent.
2021-09-27 14:44:21 +02:00
Eelco Dolstra 8430a8f086 Don't copy in rethrow 2021-09-27 14:38:10 +02:00
Eelco Dolstra 4b2b151131 nix path-info -r: Don't duplicate the root paths
This fixes

  $ nix path-info -r $(type -P ls)
  /nix/store/vfilzcp8a467w3p0mp54ybq6bdzb8w49-coreutils-8.32
  /nix/store/5d821pjgzb90lw4zbg6xwxs7llm335wr-libunistring-0.9.10
  ...
  /nix/store/mrv4y369nw6hg4pw8d9p9bfdxj9pjw0x-acl-2.3.0
  /nix/store/vfilzcp8a467w3p0mp54ybq6bdzb8w49-coreutils-8.32

Also, output the paths in topologically sorted order like we used to.
2021-09-27 12:47:39 +02:00
Eelco Dolstra a15e65eef0 run(): Move 2021-09-27 11:12:06 +02:00
Eelco Dolstra 283e7da470
Merge pull request #5292 from edolstra/update-lowdown
Update lowdown
2021-09-27 11:10:05 +02:00
oxalica 81b8e910a0
Explicitly set initial branch name for git 2021-09-24 22:09:49 +08:00
Eelco Dolstra aedbc7b683 Use latest lowdown
This improves list rendering
(https://github.com/kristapsdz/lowdown/issues/73).
2021-09-24 15:22:51 +02:00
Eelco Dolstra 362d8f925e
Merge pull request #5253 from edolstra/flake-ifd
Don't allow IFD in flakes by default
2021-09-24 10:48:23 +02:00
Eelco Dolstra ea9df6fe51 Shut down write side before draining the read side
This is important if the remote side *does* execute
nix-store/nix-daemon successfully, but stdout is polluted
(e.g. because the remote user's bashrc script prints something to
stdout). In that case we have to shutdown the write side to force the
remote nix process to exit.
2021-09-23 18:01:04 +02:00
Eelco Dolstra 994348e9e0 SSHStore / LegacySSHStore: Show a better error message if the remote is "nologin"
Instead of

  error: serialised integer 7161674624452356180 is too large for type 'j'

we now get

  error: 'nix-store --serve' protocol mismatch from 'sshtest@localhost', got 'This account is currently not available.'

Fixes https://github.com/NixOS/nixpkgs/issues/37287.
2021-09-23 17:50:29 +02:00
Eelco Dolstra 60642aa5e2 Remove risky char * 2021-09-23 12:07:50 +02:00
Eelco Dolstra 198fa786a1 Add some more instrumentation 2021-09-23 11:42:33 +02:00
Eelco Dolstra d1bf7431bb Revert "Merge pull request #4922 from nrdxp/default-submodules"
This reverts commit 6678e98411, reversing
changes made to 90b2dd570c.
2021-09-22 17:25:25 +02:00
Eelco Dolstra 8623a5b595 Disable IFD selectively
It's now disabled by default for the following:

* 'nix search' (this was already implied by read-only mode)
* 'nix flake show'
* 'nix flake check', but only on the hydraJobs output
2021-09-22 17:15:07 +02:00
Eelco Dolstra d8c10028d9 Make setDefault() typed 2021-09-22 14:15:35 +02:00
Eelco Dolstra ff28fffce2 Don't cache realiseContext() errors
Errors that depend on the configuration (such as whether
allow-import-from-derivation is set) should not be cached.
2021-09-22 14:00:56 +02:00
Eelco Dolstra bcd73ebf60 Add missing #include
Fixes #5282.
2021-09-22 12:14:50 +02:00
Eelco Dolstra 60cc975d22 Set input parent at construction time 2021-09-21 14:07:16 +02:00
Eelco Dolstra 06557299b3 Allow relative paths anywhere into the parent's store path 2021-09-21 13:45:11 +02:00
Eelco Dolstra 5cbb9c5406 path fetcher: Fix relative path check 2021-09-21 13:19:26 +02:00
Gregor Kleen fd67a0f927 Fix scheme argument to s3 cache 2021-09-17 10:45:19 +02:00
Geoff Reedy cbe9ddfd1a Include subpath in flake fingerprint
Without this, flakes within the same tree and same lock data will have
the same fingerprint and the eval cache for one flake will be
incorrectly used for another.
2021-09-16 15:58:21 -06:00
Eelco Dolstra 1ec4efa6c8
Merge pull request #5257 from edolstra/dirty-lock-file
If we can't write a lock file, pretend the top-level flake is dirty
2021-09-15 20:18:23 +02:00
Eelco Dolstra c17f3c5e69 Merge branch 'mh/fix-chroot-eval' of https://github.com/obsidiansystems/nix 2021-09-15 18:37:58 +02:00
Eelco Dolstra 027344ce7e If we can't write a lock file, pretend the top-level flake is dirty
Alternative to #4639. You can still read flake.lock, but at least in
reproducible workflows like NixOS configurations where you require a
non-dirty tree, evaluation will fail because there is no rev.
2021-09-15 18:31:42 +02:00
Eelco Dolstra 991cc53386 Revert "Disallow reading flake.lock"
This reverts commit e5596113f7.
2021-09-15 18:30:37 +02:00
Emily 56025ad3b1 sandbox: allow Rosetta 2 on Darwin
This allows sandboxed x86_64-darwin builds on aarch64-darwin.
2021-09-15 02:02:03 +01:00
Eelco Dolstra fda4efff87 Rename unnecessary git@ 2021-09-14 22:53:31 +02:00
Eelco Dolstra 5ee3ee1a6b
Merge pull request #5249 from edolstra/nix-profile
Add missing 'nix profile' subcommands
2021-09-14 22:27:19 +02:00
Eelco Dolstra e5596113f7 Disallow reading flake.lock
With --no-write-lock-file, it's possible that flake.lock is out of
sync with the actual inputs used by the evaluation. So doing fromJSON
(readFile ./flake.lock) will give wrong results.

Fixes #4639.
2021-09-14 21:09:11 +02:00
Eelco Dolstra b41968f15a nix profile history: Show profile date 2021-09-14 20:47:33 +02:00
Eelco Dolstra 4b738fc7a9 Add 'nix profile wipe-history' command 2021-09-14 20:35:41 +02:00