Commit graph

14945 commits

Author SHA1 Message Date
Eelco Dolstra 735558bea6
Merge pull request #8845 from cole-h/fix-double-percent-encoding
libutil: fix double-encoding of URLs
2023-08-18 17:08:03 +02:00
Robert Hensing 284c180732
Merge pull request #8653 from hercules-ci/gitignore-.cache
.gitignore: Add .cache/
2023-08-18 15:02:21 +02:00
Robert Hensing 75243c9693
test/flakes/follow-paths.sh: Quote
Co-authored-by: Alex Ameen <alex.ameen.tx@gmail.com>
2023-08-18 14:46:13 +02:00
John Ericson 0f1eb7c351
Merge pull request #8832 from hercules-ci/positive-source-filter
Add positive source filter
2023-08-18 08:37:41 -04:00
Robert Hensing ba28613043
Merge pull request #8840 from iFreilicht/tests-for-nix-repl
Add tests for repl formatting with and without :p
2023-08-18 14:17:17 +02:00
Cole Helbling 73696ec716 libutil: fix double-encoding of URLs
If you have a URL that needs to be percent-encoded, such as
`http://localhost:8181/test/+3d.tar.gz`, and try to lock that in a Nix
flake such as the following:

    {
      inputs.test = { url = "http://localhost:8181/test/+3d.tar.gz"; flake = false; };
      outputs = { test, ... }: {
        t = builtins.readFile test;
      };
    }

running `nix flake metadata` shows that the input URL has been
incorrectly double-encoded (despite the flake.lock being correctly
encoded only once):

    [...snip...]
    Inputs:
    └───test: http://localhost:8181/test/%252B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D

(Notice the `%252B`? That's just `%2B` but percent-encoded again)

With this patch, the double-encoding is gone; running `nix flake
metadata` will show the proper URL:

    [...snip...]
    Inputs:
    └───test: http://localhost:8181/test/%2B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D

---

As far as I can tell, this happens because Nix already percent-encodes
the URL and stores this as the value of `inputs.asdf.url`.

However, when Nix later tries to read this out of the eval state as a
string (via `getStrAttr`), it has to run it through `parseURL` again to
get the `ParsedURL` structure.

Now, this itself isn't a problem -- the true problem arises when using
`ParsedURL::to_string` later, which then _re-escapes the path_. It is
at this point that what would have been `%2B` (`+`) becomes `%252B`
(`%2B`).
2023-08-17 14:16:19 -07:00
Cole Helbling 1d7a57cfd9 libexpr/tests: test that parseFlakeRef doesn't percent-encode twice 2023-08-17 13:45:55 -07:00
Felix Uhl 17ceec3a91 Test repl formatting with and without :p 2023-08-17 13:03:43 +02:00
Eelco Dolstra db3bf180a5
Merge pull request #8833 from hercules-ci/jobcategory-doc
Document jobCategory()
2023-08-16 17:11:03 +02:00
Eelco Dolstra 7f8c99c70c
Merge pull request #8825 from trofi/search-path-prefix
src/libexpr/search-path.cc: avoid out-of-bounds read on string_view
2023-08-16 16:44:49 +02:00
Robert Hensing d8079ee350 Document jobCategory() 2023-08-16 16:16:58 +02:00
Robert Hensing 21a188a2b4 Add gc root for nixpkgs/lib content 2023-08-16 16:01:46 +02:00
Robert Hensing 63e0b5d081 GC root for fetched nixpkgs/lib content 2023-08-16 15:46:37 +02:00
Robert Hensing b13fc7101f Add positive source filter
Source filtering is a really cool Nix feature that lets us avoid a
lot of rebuilds, which speeds up the iteration cycle a lot in cases
where the relevant source files aren't actually modified.

We used to have a source filter that marked a few files as irrelevant,
but this is the wrong approach, as we have many more files that are
irrelevant. We may call this negative filtering.

This commit switches the source filtering to positive filtering, which
is a lot more robust. Instead of marking which files we don't need
we marked the files that we do need.

It's a superior approach because it is fail safe. Instead of allowing
build performance problems to creep in over time, we require that all
source inputs are declared.

I shouldn't have to explain that declaring inputs is a good practice,
so I'll stop over-explaining here.

I do have to acknowledge that this will cause a build failure when the
filter is incomplete. This is *good*, because it's the only realistic
way we could be reminded of these problems. These events will be
infrequent, so the small cost of extending the filter is worth it,
compared to the hidden cost of longer dev cycles for things like tests,
docker image, etc, etc.

(Also rebuilding Nix for stupid unnecessary reasons makes my blood boil)
2023-08-16 14:21:59 +02:00
Vertex 20d9c672d1
Update tests/flakes/follow-paths.sh
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-15 10:10:27 +01:00
Sergei Trofimovich b74962c92b src/libexpr/search-path.cc: avoid out-of-bounds read on string_view
Without the change build with `-D_GLIBCXX_ASSERTIONS` exposes testsuite
assertion:

    $ gdb src/libexpr/tests/libnixexpr-tests
    Reading symbols from src/libexpr/tests/libnixexpr-tests...
    (gdb) break __glibcxx_assert_fail
    (gdb) run
    (gdb) bt
    in std::__glibcxx_assert_fail(char const*, int, char const*, char const*)@plt () from /mnt/archive/big/git/nix/src/libexpr/libnixexpr.so
    in std::basic_string_view<char, std::char_traits<char> >::operator[] (this=0x7fffffff56c0, __pos=4)
        at /nix/store/r74fw2j8rx5idb0w8s1s6ynwwgs0qmh9-gcc-14.0.0/include/c++/14.0.0/string_view:258
    in nix::SearchPath::Prefix::suffixIfPotentialMatch (this=0x7fffffff5780, path=...) at src/libexpr/search-path.cc:15
    in nix::SearchPathElem_suffixIfPotentialMatch_partialPrefix_Test::TestBody (this=0x555555a17540) at src/libexpr/tests/search-path.cc:62

As string sizes are usigned types `(a - b) > 0` effectively means
`a != b`. While the intention should be `a > b`.

The change fixes test suite pass.
2023-08-14 22:07:37 +01:00
Alex Zero 37a509ca2d Add release notes for the previous commit 2023-08-14 18:56:02 +01:00
Alex Zero 1ef8008ca7 Fix follow path checking at depths greater than 2
We need to recurse into the input tree to handle follows paths that
trarverse multiple inputs that may or may not be follow paths
themselves.
2023-08-14 18:55:46 +01:00
Robert Hensing 5542c1f87e
Merge pull request #8813 from obsidiansystems/outputOf
Create (experimental) `outputOf` primop.
2023-08-14 16:53:39 +02:00
John Ericson 44c8d83831 Create outputOf primop.
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.

With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!

Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.

In 60b7121d2c we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!

Part of RFC 92: dynamic derivations (tracking issue #6316)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 09:37:37 -04:00
John Ericson e7c39ff00b Rework evaluator SingleDerivedPath infra
`EvalState::mkSingleDerivedPathString` previously contained its own
inverse (printing, rather than parsing) in order to validate what was
parsed. Now that is pulled out into its own separate function:
`EvalState::coerceToSingleDerivedPath`.

In additional that pulled out logic is deduplicated with
`EvalState::mkOutputString` via `EvalState::mkOutputStringRaw`, which is
itself deduplicated (and generalized) with
`DownstreamPlaceholder::mkOutputStringRaw`.

All these changes make the unit tests simpler.

(We would ideally write more unit tests for `mkSingleDerivedPathString`
`coerceToSingleDerivedPath` directly, but we cannot yet do that because
the IO in reading the store path won't work when the dummy store cannot
hold anything. Someday we'll have a proper in-memory store which will
work for this.)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 08:44:50 -04:00
John Ericson a04720e68c Rename optOutputPath to optStaticOutputPath
This choice of variable name makes it more clear what is going on.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 08:44:48 -04:00
Robert Hensing 584ff408a4
Merge pull request #8735 from obsidiansystems/defexpr
Factor out `nix-defexpr` path computation
2023-08-11 20:32:03 +02:00
Robert Hensing c4dbb55ba9 initLibUtil: Add exception handling self-check 2023-08-11 17:25:42 +02:00
tomberek 010dc7958e
Merge pull request #8369 from obsidiansystems/inductive-derived-path
Make the Derived Path family of types inductive for dynamic derivations
2023-08-11 08:50:22 -05:00
Yorick e78e9a6bd1
SimpleLogger::log: fix unintended fallthrough 2023-08-11 12:05:45 +02:00
Yorick 2e5096e4f0
FileTransfer::download: fix use-after-move
std::move(state->data) and data.empty() were called in a loop, and
could run with no other threads intervening. Accessing moved objects
is undefined behavior, and could cause a crash.
2023-08-11 12:00:31 +02:00
Yorick 1ffb26311b
MultiCommand::toJSON: Fix use-after-move 2023-08-11 12:00:11 +02:00
Yorick b9b51f9579
Prevent overriding virtual methods that are called in a destructor
Virtual methods are no longer valid once the derived destructor has
run. This means the compiler is free to optimize them to be
non-virtual.

Found using clang-tidy
2023-08-11 11:58:33 +02:00
Théophane Hufschmitt a1fdc68c65
Merge pull request #8622 from pwaller/issue-8615
Try to realise CA derivations during queryMissing
2023-08-10 08:04:44 +02:00
John Ericson 60b7121d2c Make the Derived Path family of types inductive for dynamic derivations
We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).

To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.

`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!

Important note: some JSON formats have changed.

We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-08-10 00:08:32 -04:00
Peter Waller 4b1bd822ac Try to realise CA derivations during queryMissing
This enables nix to correctly report what will be fetched in the case
that everything is a cache hit.

Note however that if an intermediate build of something which is not
cached could still cause products to end up being substituted if the
intermediate build results in a CA path which is in the cache.

Fixes #8615.

Signed-off-by: Peter Waller <p@pwaller.net>
2023-08-09 20:57:04 +01:00
Théophane Hufschmitt d00fe5f225
Merge pull request #8805 from tweag/fix-add-to-store-existing
[V2] Fix misread of source if path is already valid
2023-08-08 14:57:45 +02:00
Théophane Hufschmitt afac001c39 Test the parallel copy over ssh-ng
Regression test for https://github.com/NixOS/nix/issues/6253
2023-08-08 11:55:09 +02:00
Eelco Dolstra 5624777988
Merge pull request #8786 from Ma27/fix-why-depends-precise
nix/why-depends: fix output of `--precise`
2023-08-07 19:32:49 +02:00
Théophane Hufschmitt 4999f42a70
Merge pull request #8322 from tweag/stabilize-discard-references
Stabilize `discard-references`
2023-08-07 17:35:02 +02:00
Eelco Dolstra eb1302670e
Merge pull request #8769 from edolstra/generalize-tarball-urls
Don't require .tar/.zip extension for tarball flakerefs
2023-08-07 17:02:17 +02:00
Théophane Hufschmitt ad410abbe0 Stabilize discard-references
It has been there for a few releases now (landed in 2.14.0), doesn't
seem to cause any major issue and is wanted in a few places
(https://github.com/NixOS/nix/pull/7087#issuecomment-1544471346).
2023-08-07 16:53:37 +02:00
Théophane Hufschmitt 5df0f1755f
Merge pull request #8692 from obsidiansystems/add-another-xp-check
Feature gate `DownstreamPlaceholder::unknownCaOutput`
2023-08-07 13:11:44 +02:00
Simon Rainerson 31a6e10fe5 Fix misread of source if path is already valid
When receiving a stream of NARs through the ssh-ng protocol, an already
existing path would cause the NAR archive to not be read in the stream,
resulting in trying to parse the NAR as a ValidPathInfo. This results in
the error message:
    error: not an absolute path: 'nix-archive-1'

Fixes #6253

Usually this problem is avoided by running QueryValidPaths before
AddMultipleToStore, but can arise when two parallel nix processes gets
the same response from QueryValidPaths. This makes the problem more
prominent when running builds in parallel.
2023-08-07 10:27:40 +02:00
John Ericson 9113b4252b
Merge pull request #8760 from iFreilicht/fix-json-load-assertion-errors
Fix derivation load assertion errors
2023-08-06 17:07:43 -07:00
Felix Uhl 3fefc2b284 Fix derivation load assertion errors
When loading a derivation from a JSON, malformed input would trigger
cryptic "assertion failed" errors. Simply replacing calls to `operator []`
with calls to `.at()` was not enough, as this would cause json.execptions
to be printed verbatim.

Display nice error messages instead and give some indication where the
error happened.

*Before:*

```
$ echo 4 | nix derivation add
error: [json.exception.type_error.305] cannot use operator[] with a string argument with number

$ nix derivation show nixpkgs#hello | nix derivation add
Assertion failed: (it != m_value.object->end()), function operator[], file /nix/store/8h9pxgq1776ns6qi5arx08ifgnhmgl22-nlohmann_json-3.11.2/include/nlohmann/json.hpp, line 2135.

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: [json.exception.type_error.302] type must be string, but is object

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error: [json.exception.type_error.302] type must be object, but is string

```

*After:*

```
$ echo 4 | nix derivation add
error: Expected JSON of derivation to be of type 'object', but it is of type 'number'

$ nix derivation show nixpkgs#hello | nix derivation add
error: Expected JSON object to contain key 'name' but it doesn't

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: Expected JSON value to be of type 'string' but it is of type 'number'

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error:
       … while reading key 'outputs'

       error: Expected JSON value to be of type 'object' but it is of type 'string'
```
2023-08-05 01:34:30 +02:00
Maximilian Bosch 7c09104a94
nix/why-depends: fix output of --precise
I haven't checked when this was exactly introduced, but on Nix 2.16 I
realized that the additional lines inserted when using `--precise` are
completely separated from the tree:

    nix why-depends /nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2 /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0 --precise --extra-experimental-features nix-command
    /nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2
        → /nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0
            → /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0
    └───bin/evcxr: …':'}.PATH=${PATH/':''/nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0/bin'':'/':'}…
        └───bin/cpp: …k disable=SC2193.[[ "/nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0/bin/cpp" = *++ ]] &&…

This is apparently because `std::cout` is buffered and flushed in the
end whereas the rest of the output isn't. The fix is rather simple, just
use `logger->cout` as it's already the case for the rest of the code.

This way we also don't need to insert additional newlines in the `hits`
map since that's something the logger takes care of.

Also added a small test to make sure that the layout of this is somehow
tested to reduce the risk of further regressions here.
2023-08-04 23:11:08 +02:00
Théophane Hufschmitt 635df5ee95
Merge pull request #8774 from NixLayeredStore/experimental-stores
Add infra for experimental store implementations
2023-08-03 17:44:10 +02:00
John Ericson 3b592c880a Add infra for experimental store implemenations
This is analogous to that for experimental settings and flags that we
have also added as of late.
2023-08-02 15:46:38 -04:00
John Ericson 3723363697
Merge pull request #8765 from NixLayeredStore/refactor-store-verify
More cleanups for `LocalStore::verifyPath`
2023-08-02 13:39:29 -04:00
John Ericson 9b908fa70a Factor out nix-defexpr path computation
Avoid duplicated code, and also avoid "on the fly" path construction
(which makes it harder to keep track of which paths we use).

The factored out code doesn't create the Nix state dir anymore, but this
is fine because other in nix-env and nix-channel does:

- nix-channel: Line 158 in this commit

- nix-env: Line 1407 in this commit
2023-08-02 12:54:48 -04:00
John Ericson 66550878df Add comment explaining the use of readDirectory(realStoreDir) 2023-08-02 12:46:07 -04:00
John Ericson 770d50e49c local-store verifying: Rename store to something more clear
It is not a `Store` but a `StorePathSet`.
2023-08-02 12:40:04 -04:00
Eelco Dolstra d00469ebf9
Merge pull request #8762 from obsidiansystems/split-out-eval-settings
Move evaluator settings (type and global) to separate file/header
2023-08-02 16:54:02 +02:00