Commit graph

9153 commits

Author SHA1 Message Date
Eelco Dolstra bef68e53b9 Fix symlink handling
This restores the symlink handling behaviour prior to
94812cca98.

Fixes #9298.

(cherry picked from commit 31ebc6028b3682969d86a7b39ae87131c41cc604)
2024-01-21 20:53:30 +00:00
Shea Levy ba48ab4b95
Merge branch '2.18-maintenance' into ifd-buildStore-2.18 2024-01-11 07:21:16 -05:00
Shea Levy e7c2b35827
Build IFD in the build store when using eval-store.
Previously, IFDs would be built within the eval store, even though one
is typically using `--eval-store` precisely to *avoid* local builds.

Because the resulting Nix expression must be copied back to the eval
store in order to be imported, this requires the eval store to trust
the build store's signatures.

(cherry picked from commit c3942ef85ffbd83391410fbf012f1de366d2463c)
2024-01-11 06:34:27 -05:00
John Ericson f7f37035c8 Move tests to separate directories, and document
Today, with the tests inside a `tests` intermingled with the
corresponding library's source code, we have a few problems:

- We have to be careful that wildcards don't end up with tests being
  built as part of Nix proper, or test headers being installed as part
  of Nix proper.

- Tests in libraries but not executables is not right:

  - It means each executable runs the previous unit tests again, because
    it needs the libraries.

  - It doesn't work right on Windows, which doesn't want you to load a
    DLL just for the side global variable . It could be made to work
    with the dlopen equivalent, but that's gross!

This reorg solves these problems.

There is a remaining problem which is that sibbling headers (like
`hash.hh` the test header vs `hash.hh` the main `libnixutil` header) end
up shadowing each other. This PR doesn't solve that. That is left as
future work for a future PR.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

(cherry picked from commit 91b6833686a6a6d9eac7f3f66393ec89ef1d3b57)
(cherry picked from commit a61e42adb528b3d40ce43e07c79368d779a8b624)
2023-12-01 13:05:03 -05:00
John Ericson 30dcc19d1f Put functional tests in tests/functional
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests

 - Concepts is harder to understand, the documentation makes a good
   unit vs functional vs integration distinction, but when the
   integration tests are just two subdirs within `tests/` this is not
   clear.

 - Source filtering in the `flake.nix` is more complex. We need to
   filter out some of the dirs from `tests/`, rather than simply pick
   the dirs we want and take all of them. This is a good sign the
   structure of what we are trying to do is not matching the structure
   of the files.

With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests

functional/
installer/
nixos/
```

(cherry picked from commit 68c81c737571794f7246db53fb4774e94fcf4b7e)
2023-12-01 12:06:43 -05:00
John Ericson 89a66633b7 Add missing -lrapidcheck fixing build with shared lib
https://github.com/NixOS/nixpkgs/pull/269064 makes rapidcheck be build
as a shared lib, but that broke Nix because the `-lrapidcheck` was
missing. This fixes that (and doesn't break Nix what the library is a
static archive as today).

(cherry picked from commit 46131567da96ffac298b9ec54016b37114b0dfd5)
2023-11-22 04:48:11 +00:00
DavHau 23775b7364 fetchTree: clarify docs for shallow flag
(cherry picked from commit 796a7eb92d2b0caf75685126adc7460a4c39cfec)
2023-11-20 14:16:03 +00:00
Robert Hensing af21431140 libstore: Add apple-virt to system features when available
I'm sure that we'll adjust the implementation over time, but this
at least discerns between an apple silicon bare metal machine and
a tart VM.

(cherry picked from commit 9277eb276bf0a942e88fcf499f6a6b9c262be853)
2023-11-16 18:59:06 +01:00
Valentin Gagarin a68bf15fe6 backport fix for the --help output 2023-11-10 19:22:42 +01:00
edef 82040664e4 StorePath: reject names starting with '.'
This has been the behaviour before Nix 2.4. It was dropped in a rewrite
in 759947bf72, allowing the creation of
store paths that aren't considered valid by older Nix versions or other
Nix tooling.

Nix 2.4 didn't ship in NixOS until 22.05, and stdenv.mkDerivation in
nixpkgs drops leading periods since April 2022, so it's unlikely anyone
is relying on the current lax behaviour.

Closes #9091.

Change-Id: I4a57bd9899e1b0dba56870ae5a1b680918a18ce9
(cherry picked from commit 24bda0c7b381e1a017023c6f7cb9661fae8560bd)
2023-10-08 12:21:18 +00:00
John Ericson 72b65981f9 Revert "Adapt scheduler to work with dynamic derivations"
This reverts commit 5e3986f59c. This
un-implements RFC 92 but fixes the critical bug #9052 which many people
are hitting. This is a decent stop-gap until a minimal reproduction of
that bug is found and a proper fix can be made.

Mostly fixed #9052, but I would like to leave that issue open until we
have a regression test, so I can then properly fix the bug (unbreaking
RFC 92) later.

(cherry picked from commit 8440afbed756254784d9fea3eaab06649dffd390)
2023-10-02 15:05:23 +00:00
Robert Hensing 860f64c345 pathExists: isDir when endswith /.
(cherry picked from commit f8a3893e8d77ce4a6e23719a0b2d88464cb84b9c)
2023-10-01 13:58:20 +01:00
Eelco Dolstra 126e2645f2 Disable rapidcheck tests in the coverage run
https://hydra.nixos.org/build/233688539
2023-09-19 16:04:00 +02:00
Théophane Hufschmitt 2a52ec4e92
Merge pull request #7661 from henrik-ch/repl-doc
improved help command listing.
2023-09-14 16:43:02 +02:00
Théophane Hufschmitt b99fdcf8db
Merge pull request #8959 from maralorn/print-parent
Print parent activity field in json log
2023-09-13 11:41:47 +02:00
Emil Nikolov 21783cff16
docs: make the nix develop --command example unambiguous (#8952) 2023-09-12 17:15:36 +02:00
Matthew Kenigsberg 07545add53
Drop dead code
localPath is unused
2023-09-10 12:18:03 +02:00
maralorn 682dbcab9a
Print parent activity field in json log 2023-09-09 18:01:10 +02:00
thenbe 5473e10249
fix: nix shell multiple commands example (#8950)
The `-c` flag belongs to `sh` not `nix shell`. As it stands, the command errors with:

```
$ nix shell nixpkgs#gnumake --command sh --command "cd src && make"
sh: --command: invalid option
```

https://github.com/NixOS/nix/pull/8276 was good for readability, but it missed this since that PR used a find/replace script.
2023-09-07 22:25:32 +00:00
Robert Hensing 5f55c33917
Merge pull request #8944 from fricklerhandwerk/fix-anchor
fix invalid anchor link
2023-09-07 17:58:16 +02:00
John Ericson 80d7994f52 Special-case error message to add extra information
The Derivation parser and old ATerm unfortunately leaves few ways to get
nice errors when an old version of Nix encounters a new version of the
format. The most likely scenario for this to occur is with a new client
making a derivation that the old daemon it is communicating with cannot
understand.

The extensions we just created for dynamic derivation deps will add a
version field, solving the problem going forward, but there is still the
issue of what to do about old versions of Nix up to now.

The solution here is to carefully catch the bad error from the daemon
that is likely to indicate this problem, and add some extra context to
it.

There is another "Ugly backwards compatibility hack" in
`remote-store.cc` that also works by transforming an error.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
John Ericson 7ad66cb3ef Allow dynamic derivation deps in inputDrvs
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.

The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)

`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.

As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Apply suggestions from code review

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
Valentin Gagarin 829d4d3e03 fix invalid anchor link 2023-09-07 15:13:22 +02:00
John Ericson b7edc2099f Improve derivation parsing
- Don't assert: Derivation ATerms are not necessarily produced by Nix,
  and parsers should always throw graceful errors

- Improve error message from `static void except(..)`, shows both what
  we expected and what we actually got.

The intention is that we backport it, and then hopefully a few people
might get slightly better errors if they try out new experimental drv
files (for RFC 92) with an old version of Nix.
2023-09-06 11:44:06 -04:00
John Ericson 754528241a
Merge pull request #8927 from obsidiansystems/test-derivation-aterm
Test and begin documentation of the ATerm format for derivations
2023-09-06 09:24:52 -04:00
Christina Rust 2b3a17820f
Fix globals.hh typo 2023-09-06 04:19:40 +00:00
John Ericson 880d9cabed Test and begin documentation of the ATerm format for derivations
Wanted to do this before the last dynamic derivations PR when I
introduce a variation, to make sure I wasn't changing the old version by
mistake.
2023-09-05 11:16:39 -04:00
Eelco Dolstra 5c95b32c46 Fix warning 'catching polymorphic type by value' 2023-09-01 14:49:49 +02:00
Eelco Dolstra 919781cacc
Merge branch 'master' into valid_deriver_2 2023-09-01 13:35:05 +02:00
Eelco Dolstra b88784278f
Merge pull request #8869 from hercules-ci/fix-issue-8838-pathExists-isDir
Fix #8838, pathExists: isDir when ends with `/ `
2023-09-01 13:15:54 +02:00
Eelco Dolstra 5b5f56a9d4
Merge pull request #8859 from edolstra/tarball-last-modified
Tarball trees: Propagate lastModified
2023-08-29 17:02:06 +02:00
Bryan Honof 736b9cede7
Port the flags of nix-daemon to nix daemon (#8788)
The new `nix daemon` command didn't accept the same flags that `nix-daemon` did.

* docs(daemon): clarify the daemon trust override flags
* fix: change declaration order
* docs: add examples of nix daemon usage
* Apply suggestions from code review

---------

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: tomberek <tomberek@users.noreply.github.com>
2023-08-28 13:43:34 +00:00
Robert Hensing 1e08e12d81 pathExists: isDir when endswith /
Fixes https://github.com/NixOS/nix/issues/8838
2023-08-25 17:18:37 +02:00
John Ericson 50f40ac4c0
Merge pull request #8829 from obsidiansystems/build-dynamic-derivations
Adapt scheduler to work with dynamic derivations
2023-08-25 11:13:15 -04:00
tomberek b563ef38cc
Merge pull request #8819 from VertexA115/fix/deep-follow-paths
Fix follow path checking at depths greater than 2
2023-08-25 10:33:05 -04:00
John Ericson cbd89568ca
Merge pull request #8814 from hercules-ci/exception-self-check
initLibUtil: Add exception handling self-check
2023-08-25 10:15:54 -04:00
John Ericson 5e3986f59c Adapt scheduler to work with dynamic derivations
To avoid dealing with an optional `drvPath` (because we might not know
it yet) everywhere, make an `CreateDerivationAndRealiseGoal`. This goal
just builds/substitutes the derivation file, and then kicks of a build
for that obtained derivation; in other words it does the chaining of
goals when the drv file is missing (as can already be the case) or
computed (new case).

This also means the `getDerivation` state can be removed from
`DerivationGoal`, which makes the `BasicDerivation` / in memory case and
`Derivation` / drv file file case closer together.

The map type is factored out for clarity, and because we will soon hvae
a second use for it (`Derivation` itself).

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-25 10:01:25 -04:00
John Ericson 692074f714 Use Worker::makeDerivationGoal less
We're about to split up `DerivationGoal` a bit. At that point
`makeDerivationGoal` will mean something more specific than it does
today. (Perhaps a future rename will make this clearer.)

On the other hand, the more public `Worker::makeGoal` function will
continue to work exactly as before. So by moving some call sites to use
that instead, we preemptively avoid issues in the next step.
2023-08-25 09:55:07 -04:00
John Ericson 1c4caef14b Throw MissingRealisation not plain Error in both resolveDerivedPath
Now we are consistent with the other `resolveDerivedPath`, and other
such functions.
2023-08-25 09:55:07 -04:00
John Ericson 2f5d3da806 Introduce OutputName and OutputNameView type aliases
Hopefully they make the code easier to understand!
2023-08-25 09:55:07 -04:00
Guillaume Girol 925a444b92 add nix-store --query --valid-derivers command
notably useful when nix-store --query --deriver returns a non-existing
path.

Co-authored-by: Felix Uhl <iFreilicht@users.noreply.github.com>
2023-08-24 11:37:24 +02:00
Eelco Dolstra 81045f243f Tarball trees: Propagate lastModified
This makes them behave consistently with GitHub/GitLab flakes.
2023-08-22 21:51:26 +02:00
Eelco Dolstra 3fcdea4ca0
Merge pull request #8836 from obsidiansystems/rm-addToSearchPath
Delete `EvalState::addToSearchPath`
2023-08-21 14:58:00 +02:00
Robert Hensing 10afcf06aa
Merge pull request #8812 from tweag/fix-clang-tidy
Fix some warnings/bugs found by clang-tidy
2023-08-19 16:00:12 +02:00
John Ericson fe71faa920 Delete EvalState::addToSearchPath
This function is now trivial enough that it doesn't need to exist.

`EvalState` can still be initialized with a custom search path, but we
don't have a need to mutate the search path after it has been
constructed, and I don't see why we would need to in the future.

Fixes #8229
2023-08-18 14:04:33 -04:00
John Ericson 665ad4f7c5
Merge pull request #8839 from obsidiansystems/string-context-7479
Refactor Raw pattern, part of #7479
2023-08-18 13:47:01 -04:00
John Ericson 9121fed4b4 Fixing #7479
Types converted:

- `NixStringContextElem`
- `OutputsSpec`
- `ExtendedOutputsSpec`
- `DerivationOutput`
- `DerivationType`

Existing ones mostly conforming the pattern cleaned up:

- `ContentAddressMethod`
- `ContentAddressWithReferences`

The `DerivationGoal::derivationType` field had a bogus initialization,
now caught, so I made it `std::optional`. I think #8829 can make it
non-optional again because it will ensure we always have the derivation
when we construct a `DerivationGoal`.

See that issue (#7479) for details on the general goal.

`git grep 'Raw::Raw'` indicates the two types I didn't yet convert
`DerivedPath` and `BuiltPath` (and their `Single` variants) . This is
because @roberth and I (can't find issue right now...) plan on reworking
them somewhat, so I didn't want to churn them more just yet.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-08-18 11:44:00 -04:00
Cole Helbling 73696ec716 libutil: fix double-encoding of URLs
If you have a URL that needs to be percent-encoded, such as
`http://localhost:8181/test/+3d.tar.gz`, and try to lock that in a Nix
flake such as the following:

    {
      inputs.test = { url = "http://localhost:8181/test/+3d.tar.gz"; flake = false; };
      outputs = { test, ... }: {
        t = builtins.readFile test;
      };
    }

running `nix flake metadata` shows that the input URL has been
incorrectly double-encoded (despite the flake.lock being correctly
encoded only once):

    [...snip...]
    Inputs:
    └───test: http://localhost:8181/test/%252B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D

(Notice the `%252B`? That's just `%2B` but percent-encoded again)

With this patch, the double-encoding is gone; running `nix flake
metadata` will show the proper URL:

    [...snip...]
    Inputs:
    └───test: http://localhost:8181/test/%2B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D

---

As far as I can tell, this happens because Nix already percent-encodes
the URL and stores this as the value of `inputs.asdf.url`.

However, when Nix later tries to read this out of the eval state as a
string (via `getStrAttr`), it has to run it through `parseURL` again to
get the `ParsedURL` structure.

Now, this itself isn't a problem -- the true problem arises when using
`ParsedURL::to_string` later, which then _re-escapes the path_. It is
at this point that what would have been `%2B` (`+`) becomes `%252B`
(`%2B`).
2023-08-17 14:16:19 -07:00
Cole Helbling 1d7a57cfd9 libexpr/tests: test that parseFlakeRef doesn't percent-encode twice 2023-08-17 13:45:55 -07:00
Eelco Dolstra db3bf180a5
Merge pull request #8833 from hercules-ci/jobcategory-doc
Document jobCategory()
2023-08-16 17:11:03 +02:00