Commit graph

8319 commits

Author SHA1 Message Date
Théophane Hufschmitt 8d88c3b347
Merge pull request #7307 from hercules-ci/derivation-goal-improve-comment
libstore/derivation-goal: Elaborate a TODO for performance concern
2023-01-06 13:07:57 +01:00
Eelco Dolstra 420ccecc1e
Merge pull request #7557 from NixOS/fix-7529
On macOS with auto-uid-allocation and sandboxing, use the correct gid
2023-01-06 12:35:55 +01:00
Naïm Favier f1ee4ece80
Don't check NixOS modules
NixOS modules can be paths. Rather than dig further down into the layer
violation, don't check anything specific to NixOS modules.
2023-01-05 18:23:30 +01:00
Eelco Dolstra 3a98107170
Merge pull request #7542 from edolstra/gc-deadlock
Fix deadlock between auto-GC and addTempRoot()
2023-01-05 17:08:23 +01:00
Eelco Dolstra 0fe2b222d5
Merge pull request #7539 from tweag/fix-nix-why-depends--derivation
Fix `nix why-depends --derivation`
2023-01-05 15:32:04 +01:00
Eelco Dolstra 4e84b532ed On macOS with auto-uid-allocation and sandboxing, use the correct gid
macOS doesn't have user namespacing, so the gid of the builder needs
to be nixbld. The logic got "has sandboxing enabled" confused with
"has user namespaces".

Fixes #7529.
2023-01-05 04:58:55 -08:00
Eelco Dolstra 6991e558dd Move macOS sandbox files to sr/libstore/build 2023-01-04 04:50:45 -08:00
Eelco Dolstra 609a7dc059 Include macOS sandbox files in the Nix binary
This basically reverts 6e5165b773.
It fixes errors like

  sandbox-exec: <internal init prelude>:292:47: unable to open sandbox-minimal.sb: not found

when trying to run a development Nix installed in a user's home
directory.

Also, we're trying to minimize the number of installed files
to make it possible to deploy Nix as a single statically-linked
binary.
2023-01-04 04:36:07 -08:00
Eelco Dolstra d4d1ca8b11 nix --version: Print the data directory 2023-01-03 08:30:49 -08:00
Eelco Dolstra 28d5b5cd45 Fix deadlock between auto-GC and addTempRoot()
Previously addTempRoot() acquired the LocalStore state lock and waited
for the garbage collector to reply. If the garbage collector is in the
same process (as it the case with auto-GC), this would deadlock as
soon as the garbage collector thread needs the LocalStore state lock.

So now addTempRoot() uses separate Syncs for the state that it
needs. As long at the auto-GC thread doesn't call addTempRoot() (which
it shouldn't), it shouldn't deadlock.

Fixes #3224.
2023-01-03 15:20:21 +01:00
Eelco Dolstra 224b56f10e Move creation of the temp roots file into its own function
This also moves the file handle into its own Sync object so we're not
holding the _state while acquiring the file lock. There was no real
deadlock risk here since locking a newly created file cannot block,
but it's still a bit nicer.
2023-01-03 14:51:23 +01:00
Eelco Dolstra ae31b5f50f
Merge pull request #7497 from rski/master
src/libstore: Print the reason opening the DB failed
2023-01-03 12:44:14 +01:00
Eelco Dolstra d33d15a48b Put the --show-trace hint in the logical place 2023-01-02 20:53:58 +01:00
Eelco Dolstra c548e35498 Don't use state.positions[noPos]
This caused traces 'at «none»:0: (source not available)'.
2023-01-02 20:53:58 +01:00
Eelco Dolstra 6b69652385 Merge remote-tracking branch 'origin/master' into coerce-string 2023-01-02 20:53:39 +01:00
Théophane Hufschmitt 8cac451fce Fix why-depends for CA derivations (again)
This has the same goal as b13fd4c58e81b2b2b0d72caa5ce80de861622610,but
achieves it in a different way in order to not break
`nix why-depends --derivation`.
2023-01-02 17:42:22 +01:00
Théophane Hufschmitt 105d74eb81 Revert "Fix why-depends for CA derivations"
This reverts commit b13fd4c58e.
2023-01-02 15:44:04 +01:00
Théophane Hufschmitt 9af16c5f74
Merge pull request #5941 from hercules-ci/optimize-intersectAttrs
Optimize intersectAttrs performance
2023-01-02 15:22:38 +01:00
Valentin Gagarin e0c4a95611 antiquotation -> string interpolation
as proposed by @mkaito[1] and @tazjin[2] and discussed with @edolstra
and Nix maintainers

[1]: https://github.com/NixOS/nix.dev/pull/267#issuecomment-1270076332
[2]: https://github.com/NixOS/nix.dev/pull/267#issuecomment-1270201979

Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-01-02 14:38:57 +01:00
Théophane Hufschmitt fb8fc6fda6
Merge pull request #7478 from hercules-ci/make-sure-initNix-called
libstore: Make sure that initNix has been called
2023-01-02 14:12:49 +01:00
Valentin Gagarin 9cdf8ededb remove redundant re-definition of store derivations 2023-01-02 13:37:59 +01:00
Théophane Hufschmitt cfd6c7fc9b
Merge pull request #7485 from fricklerhandwerk/doc-store-derivation
define "store derivation"
2023-01-02 13:26:41 +01:00
Théophane Hufschmitt b3285c7722
Merge pull request #7351 from NaN-git/fix-mkString
cleanup eval.hh/eval.cc
2023-01-02 11:41:52 +01:00
Théophane Hufschmitt e8a3e58171
Merge pull request #7521 from ncfavier/migration-deadlock
Release shared lock before acquiring exclusive lock
2023-01-02 11:08:43 +01:00
Steven Shaw 84b0893725
Fix error message 2023-01-01 12:37:43 +10:00
Eelco Dolstra 8c52f8ea9d
Merge pull request #7524 from ncfavier/sandbox-paths-closure
doc: sandbox-paths computes closures
2022-12-29 19:45:51 +01:00
Eelco Dolstra 8227fe819e
Merge pull request #7504 from edolstra/nix-develop-personality
nix develop: Set personality
2022-12-29 19:42:45 +01:00
Naïm Favier d5d2f50ebb
doc: sandbox-paths computes closures 2022-12-28 17:09:20 +01:00
Naïm Favier 81c3f99b36
Release shared lock before acquiring exclusive lock
In principle, this should avoid deadlocks where two instances of Nix are
holding a shared lock on big-lock and are both waiting to get an
exclusive lock.

However, it seems like `flock(2)` is supposed to do this automatically,
so it's not clear whether this is actually where the problem comes from.
2022-12-27 15:58:14 +01:00
Robert Hensing 336908cf4c Optimize intersectAttrs performance
Always traverse the shortest set.
2022-12-24 14:51:05 +01:00
Robert Hensing aba6eb348e libstore: Make sure that initNix has been called
Prevent bugs like https://github.com/cachix/cachix/pull/477
2022-12-24 14:39:30 +01:00
Philipp Otterbein 8af839f48c remove undefined function 2022-12-24 12:19:53 +01:00
Philipp Otterbein a6e9d9cb2f remove function makeImmutableStringWithLen 2022-12-24 12:09:06 +01:00
Eelco Dolstra c164d304f3 nix develop: Set personality
This makes 'nix develop' set the Linux personality in the same way
that the actual build does, allowing a command like 'nix develop
nix#devShells.i686-linux.default' on x86_64-linux to work correctly.
2022-12-23 16:33:55 +01:00
Eelco Dolstra 14f7dae3e4
Merge pull request #7503 from edolstra/fix-dirOf
Fix CanonPath::dirOf() returning a string_view of a temporary
2022-12-23 16:17:48 +01:00
Eelco Dolstra 64c60f7241 Fix CanonPath::dirOf() returning a string_view of a temporary
https://hydra.nixos.org/build/202837872
2022-12-23 15:32:54 +01:00
rski d034ed1891 src/libstore: Print the reason opening the DB failed
Without this, the error is lost, and it makes for a hard to debug
situation. Also remove some of the busyness inside the sqlite_open_v2
args.

The errcode returned is not the extended one. The only way to make open
return an extended code, would be to add SQLITE_OPEN_EXRESCODE to the
flags. In the future it might be worth making this change,
which would also simplify the existing SQLiteError code.
2022-12-23 02:55:51 +02:00
Eelco Dolstra c9eee5a84d
Merge pull request #7493 from mupdt/primops-storeDir-test-non-standard-path
primops `storeDir` test uses `settings.nixStore`
2022-12-22 16:18:48 +01:00
mupdt a33e45b60b primops storeDir test uses settings.nixStore 2022-12-21 07:01:57 -05:00
Valentin Gagarin 7797661a70 link "store derivation" to glossary definition 2022-12-21 11:42:50 +01:00
Valentin Gagarin 3a66d82e1d update description of "store derivation" in installables section
a store derivation is not a store path itself, it has a store path.
2022-12-21 11:42:22 +01:00
mupdt bc8ab21c5a [PDT] TDE-3114: prevent a race-condition when creating the S3 cache 2022-12-21 04:50:40 -05:00
Eelco Dolstra 5c97b5a398 InstallableFlake::toDerivedPaths(): Support paths and store paths
This makes 'nix build' work on paths (which will be copied to the
store) and store paths (returned as is). E.g. the following flake
output attributes can be built using 'nix build .#foo':

  foo = ./src;
  foo = self.outPath;
  foo = builtins.fetchTarball { ... };
  foo = (builtins.fetchTree { .. }).outPath;
  foo = builtins.fetchTree { .. } + "/README.md";
  foo = builtins.storePath /nix/store/...;

Note that this is potentially risky, e.g.

  foo = /.;

will cause Nix to try to copy the entire file system to the store.

What doesn't work yet:

  foo = self;
  foo = builtins.fetchTree { .. };

because we don't handle attrsets with an outPath attribute in it yet,
and

  foo = builtins.storePath /nix/store/.../README.md;

since result symlinks have to point to a store path currently (rather
than a file inside a store path).

Fixes #7417.
2022-12-20 15:11:44 +01:00
Eelco Dolstra bda879170f EvalState::copyPathToStore(): Return a StorePath 2022-12-20 14:58:39 +01:00
Eelco Dolstra 845fc3f605 Merge toDerivations() into toDerivedPaths()
toDerivedPaths() now returns DerivedPathWithInfo, which is DerivedPath
with some attributes needed by 'nix profile' etc.

Preparation for #7417.
2022-12-20 14:24:14 +01:00
Eelco Dolstra 8332ac6a1d
Merge pull request #7451 from edolstra/abstract-pos
Introduce AbstractPos
2022-12-20 12:55:13 +01:00
Théophane Hufschmitt 000dd77d8d
Merge pull request #7473 from hercules-ci/sqlite-error
Improve sqlite error messages
2022-12-20 11:27:30 +01:00
Théophane Hufschmitt c3d522dc51
Merge pull request #7480 from nrdxp/fix-develop-drv
fix(develop): make `nix develop` drv recreatable
2022-12-20 07:41:46 +01:00
Timothy DeHerrera 94cf0da7b2
fix(develop): make nix develop drv recreatable 2022-12-19 13:16:06 -07:00
Domen Kožar f118e661e0
Merge pull request #7462 from rapenne-s/improve_messaging_reusing_trusted_substituter
Inform user instead of warning them when using a trusted substituter
2022-12-19 16:04:01 +00:00
Eelco Dolstra d00bfe4833
Merge pull request #7450 from edolstra/canon-path
Add CanonPath wrapper to represent canonicalized paths
2022-12-19 16:21:11 +01:00
Robert Hensing c965f35de7 Improve sqlite error messages
They did not include the detailed error message, losing essential
information for troubleshooting.

Example message:

    warning: creating statement 'insert or rplace into NARs(cache, hashPart, namePart, url, compression, fileHash, fileSize, narHash, narSize, refs, deriver, sigs, ca, timestamp, present) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)': at offset 10: SQL logic error, near "rplace": syntax error (in '/tmp/nix-shell.grQ6f7/nix-test/tests/binary-cache/test-home/.cache/nix/binary-cache-v6.sqlite')

It's not the best example; more important information will be in
the message for e.g. a constraint violation.

I don't see why this specific error is printed as a warning, but
that's for another commit.
2022-12-17 14:51:37 +01:00
Solène Rapenne d0660c6c0b
printMsg replacement by printInfo
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-12-16 09:34:22 +01:00
Taeer Bar-Yam e5eb05c599 getBuildLog: factor out resolving derivations 2022-12-15 15:58:54 -05:00
Eelco Dolstra 0687e16c4a Fix a crash in DerivedPath::Built::toJSON() with impure derivations
The use of 'nullptr' here didn't result in a null JSON value, but in a
nullptr being cast to a string, which aborts.
2022-12-15 16:02:27 +01:00
Solène Rapenne 98e01da0b1 warnings: switch to info level when using a saved substituter 2022-12-14 14:11:21 +01:00
Naïm Favier 1f3c0a3c1d
Allow disabling build users by unsetting build-users-group
Unsetting `build-users-group` (without `auto-allocate-uids` enabled)
gives the following error:

```
src/libstore/lock.cc:25: static std::unique_ptr<nix::UserLock> nix::SimpleUserLock::acquire(): Assertion `settings.buildUsersGroup != ""' failed.
```

Fix the logic in `useBuildUsers` and document the default value
for `build-users-group`.
2022-12-14 00:40:30 +01:00
Eelco Dolstra 46b3c026fc
Merge pull request #7455 from rapenne-s/documentation_fix
Add anchors and links to definition
2022-12-13 17:26:52 +01:00
Eelco Dolstra e2a4e7aecd
Merge pull request #4543 from obsidiansystems/indexed-store-path-outputs
Low level `<drvPath>^<outputName>` installable syntax to match existing `<highLevelInstallable>^<outputNames>` syntax
2022-12-13 17:22:29 +01:00
Eelco Dolstra c9b0a85b08 Restore display of source lines for stdin/string inputs 2022-12-13 16:00:44 +01:00
Eelco Dolstra aea97f07a3 Fix compilation 2022-12-13 15:23:12 +01:00
Eelco Dolstra 1315133b50
Improve cast safety
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-12-13 12:38:33 +01:00
Solène Rapenne 09860c16ce documentation: use sections instead of list items 2022-12-13 11:47:44 +01:00
Solène Rapenne e43b0f5b12 documentation: link flake URL term to definition 2022-12-13 11:47:38 +01:00
Solène Rapenne ae27181f16 documentation: fix link to definition 2022-12-13 11:47:22 +01:00
Théophane Hufschmitt 6e31d27cba
Merge pull request #6741 from Mindavi/nix-no-gc
support building with --enable-gc=no
2022-12-13 10:36:59 +01:00
Eelco Dolstra b3fdab28a2 Introduce AbstractPos
This makes the position object used in exceptions abstract, with a
method getSource() to get the source code of the file in which the
error originated. This is needed for lazy trees because source files
don't necessarily exist in the filesystem, and we don't want to make
libutil depend on the InputAccessor type in libfetcher.
2022-12-13 00:50:43 +01:00
John Ericson f61d575810 Merge branch 'indexed-store-path-outputs' of github.com:obsidiansystems/nix into indexed-store-path-outputs 2022-12-12 17:43:10 -05:00
John Ericson 5273cf4c97 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-12-12 17:40:49 -05:00
John Ericson 32ae715db1
Fix typos in the docs
Thanks!

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-12-12 17:37:45 -05:00
John Ericson dabb03b8d0 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-12-12 17:36:02 -05:00
John Ericson d8c1c24c78 Adjust docs 2022-12-12 17:32:24 -05:00
Rick van Schijndel 672ee88231 support building with --enable-gc=no
Some minor changes fixing the build without boehm.
Fixes NixOS#6250
2022-12-12 23:31:30 +01:00
John Ericson dc075dcdd0
Apply suggestions from code review
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-12-12 16:26:10 -05:00
Eelco Dolstra 900b854084 Add CanonPath wrapper to represent canonicalized paths 2022-12-12 19:57:32 +01:00
Florian Friesdorf 8618c6cc75 Simplify loop, feedback from @tfc and @Ericson2314 2022-12-12 18:41:00 +00:00
John Ericson 7b122d43a4 Fix stack context notes to not rely on order
Make everything be in the form "while ..." (most things were already),
and in particular *don't* use other propositions that must go after or
before specific "while ..." clauses to make sense.
2022-12-12 18:41:00 +00:00
Florian Friesdorf d269976be6 Show stack trace above error message
Save developers from scrolling by displaying the error message last,
below the stack trace.
2022-12-12 18:41:00 +00:00
Florian Friesdorf 173dcb0af9 Don't reverse stack trace when showing
When debugging nix expressions the outermost trace tends to be more useful
than the innermost. It is therefore printed last to save developers from
scrolling.
2022-12-12 18:41:00 +00:00
Eelco Dolstra e408af82ab
Merge pull request #7436 from edolstra/enable-lang-tests
Enable some language tests that were accidentally disabled
2022-12-12 17:39:02 +01:00
Eelco Dolstra 17f81d3215 Fix unused variable warning 2022-12-12 16:41:46 +01:00
Benoit de Chezelles a456630a5a Allow to disable global flake-registry with "" 2022-12-12 15:32:02 +01:00
Eelco Dolstra 7a85199f87 Add docs from the lazy-trees branch 2022-12-12 14:06:13 +01:00
Eelco Dolstra fd0ed75118 Support flake references in the old CLI
Fixes #7026.
2022-12-12 14:05:52 +01:00
Eelco Dolstra ae5f62a894 Move isUri() and resolveUri() out of filetransfer.cc
These are purely related to NIX_PATH / -I command line parsing, so put
them in libexpr.
2022-12-12 14:05:35 +01:00
Eelco Dolstra 7396844676
Merge pull request #7421 from edolstra/lazy-trees-trivial-changes
Trivial changes from the lazy-trees branch
2022-12-12 13:52:56 +01:00
John Ericson 1879c7c95e
Merge branch 'master' into indexed-store-path-outputs 2022-12-12 07:33:36 -05:00
Eelco Dolstra e558e089ba -I description: Use -I examples 2022-12-12 12:51:23 +01:00
Eelco Dolstra 877ea1dab8
Use get_ptr()
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-12-12 12:46:13 +01:00
Eelco Dolstra 037d5c4299
Manual improvements
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-12-12 12:43:44 +01:00
Eelco Dolstra 786402365e Cleanup 2022-12-12 12:40:51 +01:00
Eelco Dolstra f3d1e92856
Update URL
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-12-12 12:37:55 +01:00
Eelco Dolstra 8272cd9dec
Optimize string concatenation
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-12-12 12:36:19 +01:00
Andreas Rammhold dbc8547664 Ignore the enforceDeterminism value
We used to set enforceDeterminism to true in the settings (by default)
and thus did send a non-zero value over the wire. The value should
probably be ignored as it should only matter if nrRounds is non-zero
as well.

Having the old code here where the value is expected to be zero only
works with the same version of Nix where we are sending zero. We
should always test this against older Nix versions being client or
server as otherwise upgrade in larger networks might be a pain.

Fixes 8e0946e8df
2022-12-10 17:55:07 +01:00
Théophane Hufschmitt 2affb19c92
Merge pull request #7409 from tweag/fix-6383
check the store for input before failing (hopefully fix #6383)
2022-12-09 06:33:30 +01:00
Taeer Bar-Yam 3b27181ee5 fix missing function after rebase 2022-12-08 16:59:21 -05:00
regnat 04b113f6cb Fix nix log with CA derivations
Fix #6209

When trying to run `nix log <installable>`, try first to resolve the derivation pointed to
by `<installable>` as it is the resolved one that holds the build log.

This has a couple of shortcomings:
1. It’s expensive as it requires re-reading the derivation
2. It’s brittle because if the derivation doesn’t exist anymore or can’t
   be resolved (which is the case if any one of its build inputs is missing),
   then we can’t access the log anymore

However, I don’t think we can do better (at least not right now).
The alternatives I see are:
1. Copy the build log for the un-resolved derivation. But that means a
   lot of duplication
2. Store the results of the resolving in the db. Which might be the best
   long-term solution, but leads to a whole new class of potential
   issues.
2022-12-08 16:03:20 -05:00
Eelco Dolstra 703d863a48 Trivial changes from the lazy-trees branch 2022-12-07 14:06:34 +01:00
Linus Heckemann 8e0946e8df Remove repeat and enforce-determinism options
These only functioned if a very narrow combination of conditions held:

- The result path does not yet exist (--check did not result in
  repeated builds), AND
- The result path is not available from any configured substituters, AND
- No remote builders that can build the path are available.

If any of these do not hold, a derivation would be built 0 or 1 times
regardless of the repeat option. Thus, remove it to avoid confusion.
2022-12-07 11:36:48 +01:00
Taeer Bar-Yam 1c8de7d3d0 improve style 2022-12-06 11:25:38 -05:00