Commit graph

15647 commits

Author SHA1 Message Date
John Ericson 9f39dda66c Fix building CA derivations with and eval store
I don't love the way this code looks. There are two larger problems:

- eval, build/scratch, destination stores (#5025) should have different
  types to reflect the fact that they are used for different purposes
  and those purposes correspond to different operations. It should be
  impossible to "use the wrong store" in my cases.

- Since drvs can end up in both the eval and build/scratch store, we
  should have some sort of union/layered store (not on the file sytem
  level, just conceptual level) that allows accessing both. This would
  get rid of the ugly "check both" boilerplate in this PR.

Still, it might be better to land this now / soon after minimal cleanup,
so we have a concrete idea of what problem better abstractions are
supposed to solve.
2023-12-11 12:17:36 -05:00
John Ericson 8cddda4f89
Merge pull request #9588 from obsidiansystems/queryDerivationOutputMap-evalStore
Give `Store::queryDerivationOutputMap` and `evalStore` argument
2023-12-11 11:16:18 -05:00
John Ericson 5f30c8acc7 Give Store::queryDerivationOutputMap and evalStore argument
Picking up where https://github.com/NixOS/nix/pull/9563 left off.
2023-12-11 10:39:08 -05:00
Robert Hensing 89cf53648c
Contributing branches and reverting (#9577)
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-12-11 12:26:31 +01:00
Robert Hensing da58c00ee0
Merge pull request #9586 from obsidiansystems/legacy-ssh-store-header
Create header for `LegacySSHStore`
2023-12-11 12:21:56 +01:00
John Ericson 5c917c3204
Merge pull request #9587 from amjoseph-nixpkgs/pr/wopBuildDerivation/explain
libstore/daemon.cc: note trust model difference in readDerivation()s
2023-12-10 21:42:35 -05:00
Adam Joseph e43bb655fe libstore/daemon.cc: note trust model difference in readDerivation()s
Below the comment added by this commit is a much longer comment
followed by a trust check, both of which have confused me on at
least two occasions.  I figured it out once, forgot it, then had to
ask @Ericson2314 to explain it, at which point I understood it
again.  I think this might confuse other people too, or maybe I will
just forget it a third time.  So let's add a comment.

Farther down in the function is the following check:

```
if (!(drvType.isCA() || trusted))
  throw Error("you are not privileged to build input-addressed derivations");
```

This seems really strange at first.  A key property of Nix is that
you can compute the outpath of a derivation using the derivation
(and its references-closure) without trusting anybody!

The missing insight is that at this point in the code the builder
doesn't necessarily have the references-closure of the derivation
being built, and therefore needs to trust that the derivation's
outPath is honest.  It's incredibly easy to overlook this, because
the only difference between these two cases is which of these
identically-named functions we used:

- `readDerivation(Source,Store)`
- `Store::readDerivation()`

These functions have different trust models (except in the special
case where the first function is used on the local store).  We
should call the reader's attention to this fact.

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2023-12-10 17:47:07 -08:00
John Ericson deadb3bfe9 Create header for LegacySSHStore
In https://github.com/NixOS/nix/pull/6134#issuecomment-1079199888,
@thuffschmitt proposed exposing `LegacySSHStore` in Nix for
deduplication with Hydra, at least temporarily. I think that is a good
idea.

Note that the diff will look bad unless one ignores whitespace! Also try
this locally:

```shell-session
git diff --ignore-all-space HEAD^:src/libstore/legacy-ssh-store.cc HEAD:src/libstore/legacy-ssh-store.cc
git diff --ignore-all-space HEAD^:src/libstore/legacy-ssh-store.cc HEAD:src/libstore/legacy-ssh-store.hh
```
2023-12-10 14:29:09 -05:00
John Ericson b7e016ab24
Merge pull request #9572 from obsidiansystems/serve-proto-build-options
Create `ServeProto::BuildOptions` and a serializer for it
2023-12-10 12:16:02 -05:00
Valentin Gagarin 3c200da242
document fetchTree (#9258)
* document `fetchTree`

* display experimental feature note at the top

we have to enable the new `fetchTree` experimental feature to render it
at all. this was a bug introduced when adding that new feature flag.

Co-authored-by: tomberek <tomberek@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Silvan Mosberger <github@infinisil.com>
2023-12-10 05:16:32 +00:00
Robert Hensing b1842a4f05
Merge pull request #9575 from hercules-ci/shell-on-hydra
flake.nix: Cache shell inputs through hydra
2023-12-10 04:42:29 +01:00
Robert Hensing a63be6578f flake.nix: Cache shell inputs through hydra 2023-12-09 21:22:20 +01:00
Robert Hensing 69b7876a08
Merge pull request #9573 from hercules-ci/rl-next-md-frontmatter
rl-next: Fix and support markdown frontmatter syntax
2023-12-09 21:18:04 +01:00
Robert Hensing a856f603ed Add checks.rl-next 2023-12-09 19:57:55 +01:00
Robert Hensing 3811b334c6 rl-next: Use markdown frontmatter syntax
The old syntax is still supported, as long as you don't use a {
in the description - the reason to migrate.
2023-12-09 19:57:55 +01:00
Robert Hensing 360f3b3a9e changelog-d: Use roberth fork with markdown frontmatter support 2023-12-09 19:50:33 +01:00
John Ericson 5417990e31 Create ServeProto::BuildOptions and a serializer for it
More tests, and more serializers for Hydra reuse.
2023-12-09 11:35:13 -05:00
Robert Hensing c8458bd731
Merge pull request #9555 from 9999years/positions-in-errors
Pass positions when evaluating
2023-12-09 03:55:58 +01:00
Robert Hensing 7cdc8786d9
Merge pull request #9568 from hercules-ci/revert-9553
Revert 9553
2023-12-09 03:49:53 +01:00
John Ericson 6aba2fdb4d
Merge pull request #9306 from NixOS/accessor-no-follow-symlink
Clarify `SourceAccessor` methods should never implicitly follow symlinks
2023-12-08 21:22:37 -05:00
Robert Hensing 6e8d598314 tests/lang/eval-fail-bad-string-interpolation-4: init 2023-12-09 02:52:49 +01:00
Robert Hensing b9980b377e
Update rl-next/source-positions-in-errors for Nix 2.19+ 2023-12-09 02:36:33 +01:00
Robert Hensing 9b7b7a7561 Revert "Print the value in error: cannot coerce messages (#9553)"
This reverts commit f0ac2a35d5.

The request from the sibling PR, which also applies here, was not addressed.
https://github.com/NixOS/nix/pull/9554#issuecomment-1845095735
2023-12-09 02:13:32 +01:00
John Ericson ce4ca574d2 Clarify SourceAccessor methods should never implicitly follow symlinks
The code has already been fixed (yay!) so what is left of this commit is
just updating the API docs.

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2023-12-08 20:00:47 -05:00
Robert Hensing d4f6b1d38b
Merge pull request #9497 from edolstra/move-access-control
Move restricted/pure-eval access control out of the evaluator and into the accessor
2023-12-08 22:21:50 +01:00
John Ericson 762af72728
Merge pull request #9564 from NixOS/fix-clang-build
Avoid `std::strstream`, fix the clang build
2023-12-08 15:07:36 -05:00
Théophane Hufschmitt 36ca6adc60
Merge pull request #9563 from obsidiansystems/tryResolve-evalStore
Give `Derivation::tryResolve` an `evalStore` argument
2023-12-08 19:21:35 +01:00
John Ericson f9ee1bedcf Avoid std::strstream, fix the clang build
According https://en.cppreference.com/w/cpp/io/strstream, it has been
deprecated since C++98! The Clang + Linux build systems to not have it
at all, or at least be hiding it.

We can just use `std::stringstream` instead, I think.
2023-12-08 13:18:52 -05:00
Rebecca Turner f0ac2a35d5
Print the value in error: cannot coerce messages (#9553)
* Print the value in `error: cannot coerce` messages

This extends the `error: cannot coerce a TYPE to a string` message
to print the value that could not be coerced. This helps with debugging
by making it easier to track down where the value is being produced
from, especially in errors with deep or unhelpful stack traces.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-12-08 16:36:57 +00:00
John Ericson 139982997e
Merge pull request #9560 from obsidiansystems/serve-proto-unkeyed-valid-path-info-serializer
Factor out `ServeProto::Serialiser<UnkeyedValidPathInfo>` and test
2023-12-08 11:33:24 -05:00
John Ericson 96dd757b0c Give Derivation::tryResolve an evalStore argument
This is needed for building CA deriations with a src store / dest store
split. In particular it is needed for Hydra.

https://github.com/NixOS/hydra/issues/838 currently puts realizations,
and thus build outputs, in the local store, but it should not.
2023-12-08 10:01:05 -05:00
Rebecca Turner 0b80935c22
Pass positions when evaluating
This includes position information in more places, making debugging
easier.

Before:

```
$ nix-instantiate --show-trace --eval tests/functional/lang/eval-fail-using-set-as-attr-name.nix
error:
       … while evaluating an attribute name

         at «none»:0: (source not available)

       error: value is a set while a string was expected
```

After:

```
error:
       … while evaluating an attribute name

         at /pwd/lang/eval-fail-using-set-as-attr-name.nix:5:10:

            4| in
            5|   attr.${key}
             |          ^
            6|

       error: value is a set while a string was expected
```
2023-12-07 10:27:21 -08:00
John Ericson a5521b7d94 Factor out ServeProto::Serialiser<UnkeyedValidPathInfo> and test
In the process, partially undo e89b5bd0bf
in that the ancient < 2.4 version is now supported again by the
serializer again. `LegacySSHStore`, instead of also asserting that the
version is at least 4, just checks that `narHash` is set.

This allows us to better test the serializer in isolation for both
versions (< 4 and >= 4).
2023-12-07 11:34:18 -05:00
Eelco Dolstra c3827ff634
Merge pull request #9557 from bryanhonof/bryanhonof.fix-apple-double-shenanigans
Add option to libarchive so it behaves correctly
2023-12-07 12:33:02 +01:00
Bryan Honof bf00d5ecef
fix(libutil/tarfile): add option to libarchive so it behaves correctly with AppleDouble files
AppleDouble files were extracted differently on macOS machines than on other
UNIX's.
Setting `archive_read_set_format_option(this->archive, NULL ,"mac-ext",NULL)`
fixes this problem, since it just ignores the AppleDouble file and treats it as
a normal one.
This was a problem since it caused source archives to be different between macOS
and Linux.

Ref: nixos/nix#9290
2023-12-07 11:35:15 +01:00
tomberek 82449a455f
Merge pull request #9452 from kolloch/feature/nix-hash-convert
Add nix hash convert
2023-12-06 19:47:53 -05:00
Peter Kolloch 9a1a3c43bf Store.xs: fix references to HashFormat::Nix32
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch e9a5365db6 hash.sh: Make failure tests more tolerant of additional output
"warning: you don'\''t have Internet access; disabling some network-dependent features" ...

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch bbba2055f0 Refactor concurrently added tests to use HashAlgorithm.
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch d38ec12855 Update src/libexpr/primops.cc
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-12-06 23:43:42 +01:00
Peter Kolloch 8afeaf05c4 Add docs/rl-notes for nix hash convert / builtins.convertHash
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch 7ff876b92b Add deprecation notice for old nix hash conversion subcommands.
(But not yet nix-hash since `nix hash` is still hidden behind a feature flag.)

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch fc6f29053a Renamed HashFormat::Base32 to HashFormat::Nix32
...and also adjusted parsing accordingly.

Also added CLI completion for HashFormats.

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch 837b889c41 Further HashType renaming + using mkHashAlgoOptFlag for new conversion
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch 5334c9c792 HashType: Rename to HashAlgorithm
To be consistent with CLI, nix API
and many other references.

As part of this, we also converted it to a scoped enum.

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch 0c2d5f7673 nix hash convert: s/--type/--algo/ + more functional tests
https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch 6bbd900d4f nix hash convert: added
This deviated from the proposal! See comments on the issue.

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:34 +01:00
Peter Kolloch 156ea78d74 CmdHashBase: doc comment 2023-12-06 23:41:07 +01:00
Peter Kolloch e7abf60a0c hash.cc/hash.h: Minor C++ improvements 2023-12-06 23:41:07 +01:00
Eelco Dolstra 3dcb83409d
Merge pull request #9509 from fricklerhandwerk/add-redirect
add redirect to new store page
2023-12-06 18:03:53 +01:00