Commit graph

8341 commits

Author SHA1 Message Date
John Ericson 8a3b1b7ced Simplify and document store path installable parsing 2023-01-11 19:08:19 -05:00
John Ericson ce2f91d356 Split OutputsSpec and ExtendedOutputsSpec, use the former more
`DerivedPath::Built` and `DerivationGoal` were previously using a
regular set with the convention that the empty set means all outputs.
But it is easy to forget about this rule when processing those sets.
Using `OutputSpec` forces us to get it right.
2023-01-11 18:57:18 -05:00
John Ericson a7c0cff07f Rename OutputPath -> ExtendedOutputPath
Do this prior to making a new more limitted `OutputPath` we will use in
more places.
2023-01-11 18:55:29 -05:00
John Ericson a8f45b5e5a Improve OutputsSpec slightly
A few little changes preparing for the rest.
2023-01-11 18:54:50 -05:00
Eelco Dolstra 9fc8d00d74 MonitorFdHup: Make it work on macOS again
It appears that on current macOS versions, our use of poll() to detect
client disconnects no longer works. As a workaround, poll() for
POLLRDNORM, since this *will* wake up when the client has
disconnected. The downside is that it also wakes up when input is
available. So just sleep for a bit in that case.  This means that on
macOS, a client disconnect may take up to a second to be detected,
but that's better than not being detected at all.

Fixes #7584.
2023-01-11 10:48:40 -08:00
Eelco Dolstra 7515617ad0 Backport getLine tests from lazy-trees 2023-01-11 13:49:39 +01:00
Théophane Hufschmitt a3ba80357d
Merge pull request #7543 from obsidiansystems/typed-string-context
Parse string context elements properly
2023-01-11 07:09:37 +01:00
Robert Hensing f58c301112
Merge pull request #7541 from hercules-ci/check-manual-links
Check links in the manual
2023-01-10 23:07:38 +01:00
Robert Hensing fefa3a49ce doc/manual: Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-01-10 22:30:41 +01:00
Robert Hensing e79f935718 doc/manual: Fix broken internal links
The targets I could find.
2023-01-10 22:30:41 +01:00
Robert Hensing 34a1e0d29b doc/manual: Introduce @docroot@ as a stable base for includable snippets
This way the links are clearly within the manual (ie not absolute paths),
while allowing snippets to reference the documentation root reliably,
regardless of at which base url they're included.
2023-01-10 22:30:41 +01:00
John Ericson 5576d5e987 Parse string context elements properly
Prior to this change, we had a bunch of ad-hoc string manipulation code
scattered around. This made it hard to figure out what data model for
string contexts is.

Now, we still store string contexts most of the time as encoded strings
--- I was wary of the performance implications of changing that --- but
whenever we parse them we do so only through the
`NixStringContextElem::parse` method, which handles all cases. This
creates a data type that is very similar to `DerivedPath` but:

 - Represents the funky `=<drvpath>` case as properly distinct from the
   others.

 - Only encodes a single output, no wildcards and no set, for the
   "built" case.

(I would like to deprecate `=<path>`, after which we are in spitting
distance of `DerivedPath` and could maybe get away with fewer types, but
that is another topic for another day.)
2023-01-10 13:10:49 -05:00
John Ericson da64f026dd Make clear that StorePathWithOutputs is a deprecated type
- Add a comment

- Put `OutputsSpec` in a different header (First part of #6815)

- Make a few stray uses of it in new code use `DerivedPath` instead.
2023-01-10 11:27:19 -05:00
Eelco Dolstra 59cc920cc0 Add a FIXME 2023-01-10 15:20:30 +01:00
Eelco Dolstra 7f1af270dd Clean up toDerivedPaths() logic 2023-01-10 15:08:46 +01:00
Eelco Dolstra 1123c42f90
Apply suggestions from code review
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-01-10 14:57:35 +01:00
Eelco Dolstra b4dc68f0be Show string in error message 2023-01-10 14:56:03 +01:00
Eelco Dolstra b80e4b57da ExtraInfo -> ExtraPathInfo 2023-01-10 14:52:49 +01:00
Eelco Dolstra 8e923bf4c5 Merge remote-tracking branch 'origin/master' into fix-7417 2023-01-10 14:35:06 +01:00
Will Bush 05b13aff3d
Fix typo in example for builtin function map 2023-01-06 23:04:43 -06:00
Théophane Hufschmitt 8d88c3b347
Merge pull request #7307 from hercules-ci/derivation-goal-improve-comment
libstore/derivation-goal: Elaborate a TODO for performance concern
2023-01-06 13:07:57 +01:00
Eelco Dolstra 420ccecc1e
Merge pull request #7557 from NixOS/fix-7529
On macOS with auto-uid-allocation and sandboxing, use the correct gid
2023-01-06 12:35:55 +01:00
Naïm Favier f1ee4ece80
Don't check NixOS modules
NixOS modules can be paths. Rather than dig further down into the layer
violation, don't check anything specific to NixOS modules.
2023-01-05 18:23:30 +01:00
Eelco Dolstra 3a98107170
Merge pull request #7542 from edolstra/gc-deadlock
Fix deadlock between auto-GC and addTempRoot()
2023-01-05 17:08:23 +01:00
Eelco Dolstra 0fe2b222d5
Merge pull request #7539 from tweag/fix-nix-why-depends--derivation
Fix `nix why-depends --derivation`
2023-01-05 15:32:04 +01:00
Eelco Dolstra 4e84b532ed On macOS with auto-uid-allocation and sandboxing, use the correct gid
macOS doesn't have user namespacing, so the gid of the builder needs
to be nixbld. The logic got "has sandboxing enabled" confused with
"has user namespaces".

Fixes #7529.
2023-01-05 04:58:55 -08:00
Eelco Dolstra 6991e558dd Move macOS sandbox files to sr/libstore/build 2023-01-04 04:50:45 -08:00
Eelco Dolstra 609a7dc059 Include macOS sandbox files in the Nix binary
This basically reverts 6e5165b773.
It fixes errors like

  sandbox-exec: <internal init prelude>:292:47: unable to open sandbox-minimal.sb: not found

when trying to run a development Nix installed in a user's home
directory.

Also, we're trying to minimize the number of installed files
to make it possible to deploy Nix as a single statically-linked
binary.
2023-01-04 04:36:07 -08:00
Naïm Favier 3c968191f1
move unsafeDiscardReferences out of outputChecks
It's not a check.
2023-01-03 18:53:01 +01:00
Eelco Dolstra d4d1ca8b11 nix --version: Print the data directory 2023-01-03 08:30:49 -08:00
Naïm Favier 15f7fa59be
unsafeDiscardReferences
Adds a new boolean structured attribute
`outputChecks.<output>.unsafeDiscardReferences` which disables scanning
an output for runtime references.

    __structuredAttrs = true;
    outputChecks.out.unsafeDiscardReferences = true;

This is useful when creating filesystem images containing their own embedded Nix
store: they are self-contained blobs of data with no runtime dependencies.

Setting this attribute requires the experimental feature
`discard-references` to be enabled.
2023-01-03 17:19:16 +01:00
Eelco Dolstra 28d5b5cd45 Fix deadlock between auto-GC and addTempRoot()
Previously addTempRoot() acquired the LocalStore state lock and waited
for the garbage collector to reply. If the garbage collector is in the
same process (as it the case with auto-GC), this would deadlock as
soon as the garbage collector thread needs the LocalStore state lock.

So now addTempRoot() uses separate Syncs for the state that it
needs. As long at the auto-GC thread doesn't call addTempRoot() (which
it shouldn't), it shouldn't deadlock.

Fixes #3224.
2023-01-03 15:20:21 +01:00
Eelco Dolstra 224b56f10e Move creation of the temp roots file into its own function
This also moves the file handle into its own Sync object so we're not
holding the _state while acquiring the file lock. There was no real
deadlock risk here since locking a newly created file cannot block,
but it's still a bit nicer.
2023-01-03 14:51:23 +01:00
Eelco Dolstra ae31b5f50f
Merge pull request #7497 from rski/master
src/libstore: Print the reason opening the DB failed
2023-01-03 12:44:14 +01:00
Eelco Dolstra d33d15a48b Put the --show-trace hint in the logical place 2023-01-02 20:53:58 +01:00
Eelco Dolstra c548e35498 Don't use state.positions[noPos]
This caused traces 'at «none»:0: (source not available)'.
2023-01-02 20:53:58 +01:00
Eelco Dolstra 6b69652385 Merge remote-tracking branch 'origin/master' into coerce-string 2023-01-02 20:53:39 +01:00
Théophane Hufschmitt 8cac451fce Fix why-depends for CA derivations (again)
This has the same goal as b13fd4c58e81b2b2b0d72caa5ce80de861622610,but
achieves it in a different way in order to not break
`nix why-depends --derivation`.
2023-01-02 17:42:22 +01:00
Théophane Hufschmitt 105d74eb81 Revert "Fix why-depends for CA derivations"
This reverts commit b13fd4c58e.
2023-01-02 15:44:04 +01:00
Théophane Hufschmitt 9af16c5f74
Merge pull request #5941 from hercules-ci/optimize-intersectAttrs
Optimize intersectAttrs performance
2023-01-02 15:22:38 +01:00
Valentin Gagarin e0c4a95611 antiquotation -> string interpolation
as proposed by @mkaito[1] and @tazjin[2] and discussed with @edolstra
and Nix maintainers

[1]: https://github.com/NixOS/nix.dev/pull/267#issuecomment-1270076332
[2]: https://github.com/NixOS/nix.dev/pull/267#issuecomment-1270201979

Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-01-02 14:38:57 +01:00
Théophane Hufschmitt fb8fc6fda6
Merge pull request #7478 from hercules-ci/make-sure-initNix-called
libstore: Make sure that initNix has been called
2023-01-02 14:12:49 +01:00
Valentin Gagarin 9cdf8ededb remove redundant re-definition of store derivations 2023-01-02 13:37:59 +01:00
Théophane Hufschmitt cfd6c7fc9b
Merge pull request #7485 from fricklerhandwerk/doc-store-derivation
define "store derivation"
2023-01-02 13:26:41 +01:00
Théophane Hufschmitt b3285c7722
Merge pull request #7351 from NaN-git/fix-mkString
cleanup eval.hh/eval.cc
2023-01-02 11:41:52 +01:00
Théophane Hufschmitt e8a3e58171
Merge pull request #7521 from ncfavier/migration-deadlock
Release shared lock before acquiring exclusive lock
2023-01-02 11:08:43 +01:00
Steven Shaw 84b0893725
Fix error message 2023-01-01 12:37:43 +10:00
Eelco Dolstra 8c52f8ea9d
Merge pull request #7524 from ncfavier/sandbox-paths-closure
doc: sandbox-paths computes closures
2022-12-29 19:45:51 +01:00
Eelco Dolstra 8227fe819e
Merge pull request #7504 from edolstra/nix-develop-personality
nix develop: Set personality
2022-12-29 19:42:45 +01:00
Naïm Favier d5d2f50ebb
doc: sandbox-paths computes closures 2022-12-28 17:09:20 +01:00
Naïm Favier 81c3f99b36
Release shared lock before acquiring exclusive lock
In principle, this should avoid deadlocks where two instances of Nix are
holding a shared lock on big-lock and are both waiting to get an
exclusive lock.

However, it seems like `flock(2)` is supposed to do this automatically,
so it's not clear whether this is actually where the problem comes from.
2022-12-27 15:58:14 +01:00
Robert Hensing 336908cf4c Optimize intersectAttrs performance
Always traverse the shortest set.
2022-12-24 14:51:05 +01:00
Robert Hensing aba6eb348e libstore: Make sure that initNix has been called
Prevent bugs like https://github.com/cachix/cachix/pull/477
2022-12-24 14:39:30 +01:00
Philipp Otterbein 8af839f48c remove undefined function 2022-12-24 12:19:53 +01:00
Philipp Otterbein a6e9d9cb2f remove function makeImmutableStringWithLen 2022-12-24 12:09:06 +01:00
Eelco Dolstra c164d304f3 nix develop: Set personality
This makes 'nix develop' set the Linux personality in the same way
that the actual build does, allowing a command like 'nix develop
nix#devShells.i686-linux.default' on x86_64-linux to work correctly.
2022-12-23 16:33:55 +01:00
Eelco Dolstra 14f7dae3e4
Merge pull request #7503 from edolstra/fix-dirOf
Fix CanonPath::dirOf() returning a string_view of a temporary
2022-12-23 16:17:48 +01:00
Eelco Dolstra 64c60f7241 Fix CanonPath::dirOf() returning a string_view of a temporary
https://hydra.nixos.org/build/202837872
2022-12-23 15:32:54 +01:00
rski d034ed1891 src/libstore: Print the reason opening the DB failed
Without this, the error is lost, and it makes for a hard to debug
situation. Also remove some of the busyness inside the sqlite_open_v2
args.

The errcode returned is not the extended one. The only way to make open
return an extended code, would be to add SQLITE_OPEN_EXRESCODE to the
flags. In the future it might be worth making this change,
which would also simplify the existing SQLiteError code.
2022-12-23 02:55:51 +02:00
Eelco Dolstra c9eee5a84d
Merge pull request #7493 from mupdt/primops-storeDir-test-non-standard-path
primops `storeDir` test uses `settings.nixStore`
2022-12-22 16:18:48 +01:00
mupdt a33e45b60b primops storeDir test uses settings.nixStore 2022-12-21 07:01:57 -05:00
Valentin Gagarin 7797661a70 link "store derivation" to glossary definition 2022-12-21 11:42:50 +01:00
Valentin Gagarin 3a66d82e1d update description of "store derivation" in installables section
a store derivation is not a store path itself, it has a store path.
2022-12-21 11:42:22 +01:00
mupdt bc8ab21c5a [PDT] TDE-3114: prevent a race-condition when creating the S3 cache 2022-12-21 04:50:40 -05:00
Eelco Dolstra 5c97b5a398 InstallableFlake::toDerivedPaths(): Support paths and store paths
This makes 'nix build' work on paths (which will be copied to the
store) and store paths (returned as is). E.g. the following flake
output attributes can be built using 'nix build .#foo':

  foo = ./src;
  foo = self.outPath;
  foo = builtins.fetchTarball { ... };
  foo = (builtins.fetchTree { .. }).outPath;
  foo = builtins.fetchTree { .. } + "/README.md";
  foo = builtins.storePath /nix/store/...;

Note that this is potentially risky, e.g.

  foo = /.;

will cause Nix to try to copy the entire file system to the store.

What doesn't work yet:

  foo = self;
  foo = builtins.fetchTree { .. };

because we don't handle attrsets with an outPath attribute in it yet,
and

  foo = builtins.storePath /nix/store/.../README.md;

since result symlinks have to point to a store path currently (rather
than a file inside a store path).

Fixes #7417.
2022-12-20 15:11:44 +01:00
Eelco Dolstra bda879170f EvalState::copyPathToStore(): Return a StorePath 2022-12-20 14:58:39 +01:00
Eelco Dolstra 845fc3f605 Merge toDerivations() into toDerivedPaths()
toDerivedPaths() now returns DerivedPathWithInfo, which is DerivedPath
with some attributes needed by 'nix profile' etc.

Preparation for #7417.
2022-12-20 14:24:14 +01:00
Eelco Dolstra 8332ac6a1d
Merge pull request #7451 from edolstra/abstract-pos
Introduce AbstractPos
2022-12-20 12:55:13 +01:00
Théophane Hufschmitt 000dd77d8d
Merge pull request #7473 from hercules-ci/sqlite-error
Improve sqlite error messages
2022-12-20 11:27:30 +01:00
Théophane Hufschmitt c3d522dc51
Merge pull request #7480 from nrdxp/fix-develop-drv
fix(develop): make `nix develop` drv recreatable
2022-12-20 07:41:46 +01:00
Timothy DeHerrera 94cf0da7b2
fix(develop): make nix develop drv recreatable 2022-12-19 13:16:06 -07:00
Domen Kožar f118e661e0
Merge pull request #7462 from rapenne-s/improve_messaging_reusing_trusted_substituter
Inform user instead of warning them when using a trusted substituter
2022-12-19 16:04:01 +00:00
Eelco Dolstra d00bfe4833
Merge pull request #7450 from edolstra/canon-path
Add CanonPath wrapper to represent canonicalized paths
2022-12-19 16:21:11 +01:00
Robert Hensing c965f35de7 Improve sqlite error messages
They did not include the detailed error message, losing essential
information for troubleshooting.

Example message:

    warning: creating statement 'insert or rplace into NARs(cache, hashPart, namePart, url, compression, fileHash, fileSize, narHash, narSize, refs, deriver, sigs, ca, timestamp, present) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)': at offset 10: SQL logic error, near "rplace": syntax error (in '/tmp/nix-shell.grQ6f7/nix-test/tests/binary-cache/test-home/.cache/nix/binary-cache-v6.sqlite')

It's not the best example; more important information will be in
the message for e.g. a constraint violation.

I don't see why this specific error is printed as a warning, but
that's for another commit.
2022-12-17 14:51:37 +01:00
Solène Rapenne d0660c6c0b
printMsg replacement by printInfo
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-12-16 09:34:22 +01:00
Taeer Bar-Yam e5eb05c599 getBuildLog: factor out resolving derivations 2022-12-15 15:58:54 -05:00
Eelco Dolstra 0687e16c4a Fix a crash in DerivedPath::Built::toJSON() with impure derivations
The use of 'nullptr' here didn't result in a null JSON value, but in a
nullptr being cast to a string, which aborts.
2022-12-15 16:02:27 +01:00
Solène Rapenne 98e01da0b1 warnings: switch to info level when using a saved substituter 2022-12-14 14:11:21 +01:00
Naïm Favier 1f3c0a3c1d
Allow disabling build users by unsetting build-users-group
Unsetting `build-users-group` (without `auto-allocate-uids` enabled)
gives the following error:

```
src/libstore/lock.cc:25: static std::unique_ptr<nix::UserLock> nix::SimpleUserLock::acquire(): Assertion `settings.buildUsersGroup != ""' failed.
```

Fix the logic in `useBuildUsers` and document the default value
for `build-users-group`.
2022-12-14 00:40:30 +01:00
Eelco Dolstra 46b3c026fc
Merge pull request #7455 from rapenne-s/documentation_fix
Add anchors and links to definition
2022-12-13 17:26:52 +01:00
Eelco Dolstra e2a4e7aecd
Merge pull request #4543 from obsidiansystems/indexed-store-path-outputs
Low level `<drvPath>^<outputName>` installable syntax to match existing `<highLevelInstallable>^<outputNames>` syntax
2022-12-13 17:22:29 +01:00
Eelco Dolstra c9b0a85b08 Restore display of source lines for stdin/string inputs 2022-12-13 16:00:44 +01:00
Eelco Dolstra aea97f07a3 Fix compilation 2022-12-13 15:23:12 +01:00
Eelco Dolstra 1315133b50
Improve cast safety
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-12-13 12:38:33 +01:00
Solène Rapenne 09860c16ce documentation: use sections instead of list items 2022-12-13 11:47:44 +01:00
Solène Rapenne e43b0f5b12 documentation: link flake URL term to definition 2022-12-13 11:47:38 +01:00
Solène Rapenne ae27181f16 documentation: fix link to definition 2022-12-13 11:47:22 +01:00
Théophane Hufschmitt 6e31d27cba
Merge pull request #6741 from Mindavi/nix-no-gc
support building with --enable-gc=no
2022-12-13 10:36:59 +01:00
Eelco Dolstra b3fdab28a2 Introduce AbstractPos
This makes the position object used in exceptions abstract, with a
method getSource() to get the source code of the file in which the
error originated. This is needed for lazy trees because source files
don't necessarily exist in the filesystem, and we don't want to make
libutil depend on the InputAccessor type in libfetcher.
2022-12-13 00:50:43 +01:00
John Ericson f61d575810 Merge branch 'indexed-store-path-outputs' of github.com:obsidiansystems/nix into indexed-store-path-outputs 2022-12-12 17:43:10 -05:00
John Ericson 5273cf4c97 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-12-12 17:40:49 -05:00
John Ericson 32ae715db1
Fix typos in the docs
Thanks!

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-12-12 17:37:45 -05:00
John Ericson dabb03b8d0 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-12-12 17:36:02 -05:00
John Ericson d8c1c24c78 Adjust docs 2022-12-12 17:32:24 -05:00
Rick van Schijndel 672ee88231 support building with --enable-gc=no
Some minor changes fixing the build without boehm.
Fixes NixOS#6250
2022-12-12 23:31:30 +01:00
John Ericson dc075dcdd0
Apply suggestions from code review
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-12-12 16:26:10 -05:00
Eelco Dolstra 900b854084 Add CanonPath wrapper to represent canonicalized paths 2022-12-12 19:57:32 +01:00
Florian Friesdorf 8618c6cc75 Simplify loop, feedback from @tfc and @Ericson2314 2022-12-12 18:41:00 +00:00
John Ericson 7b122d43a4 Fix stack context notes to not rely on order
Make everything be in the form "while ..." (most things were already),
and in particular *don't* use other propositions that must go after or
before specific "while ..." clauses to make sense.
2022-12-12 18:41:00 +00:00
Florian Friesdorf d269976be6 Show stack trace above error message
Save developers from scrolling by displaying the error message last,
below the stack trace.
2022-12-12 18:41:00 +00:00