Commit graph

1806 commits

Author SHA1 Message Date
Yueh-Shun Li 5088e6563a primops: add builtins.convertHash
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-10-19 00:58:56 +08:00
Théophane Hufschmitt c1a1766c46
Merge pull request #9169 from vkryachko/follow_cycle
Detect cycles in flake follows.
2023-10-18 07:34:03 +02:00
John Ericson 9d1f42db52
Merge pull request #9150 from vicky1999/fix/8914
`nix store ping` -> `nix store info`
2023-10-17 22:52:28 -04:00
vicky1999 891dfb4359 updated store ping to store info in files 2023-10-18 00:14:11 +05:30
vicky1999 a0f071f1d3 store info sh renamed 2023-10-18 00:12:10 +05:30
vicky1999 dcc5f801f4 Store info command help updates 2023-10-17 09:39:59 +05:30
Vladimir Kryachko b3fd7db63f Detect cycles in flake follows.
This change results in an error thrown as opposed to segfaulting due to
stack overflow.

Fixes #9144
2023-10-16 13:00:49 -04:00
John Ericson f7a36f9812 Fix language tests a bit
- Remove some stray saved error messages that didn't correspond to any
  test, because they were renamed in
  d11faa01b5.

- Need `--eval` in test failure test in order to get in "read-only" mode
  where we don't try to write to the store. (The other tests already do
  this.)

- Need `--strict` so top-level attribute sets are still forced, like
  they are without `--eval`.
2023-10-16 12:52:59 -04:00
John Ericson c27d2f8da9 Add two more completions tests
Thanks @ncfavier for catching these regressions in my PR.

Co-Authored-By: Naïm Favier <n@monade.li>
2023-10-16 10:09:10 -04:00
Eelco Dolstra 856fe13533 fetchTree cleanup
Two changes:

* The (probably unintentional) hack to handle paths as tarballs has
  been removed. This is almost certainly not what users expect and is
  inconsistent with flakeref handling everywhere else.

* The hack to support scp-style Git URLs has been moved to the Git
  fetcher, so it's now supported not just by fetchTree but by flake
  inputs.
2023-10-13 14:34:23 +02:00
Ninlives 94e91566ed
Allow CLI to pass environment variables to FOD builder (#8830)
Add a new experimental `impure-env` setting that is a key-value list of
environment variables to inject into FOD derivations that specify the
corresponding `impureEnvVars`.

This allows clients to make use of this feature (without having to change the
environment of the daemon itself) and might eventually deprecate the current
behaviour (pick whatever is in the environment of the daemon) as it's more
principled and might prevent information leakage.
2023-10-11 11:58:42 +00:00
John Ericson 47b3508665 Use positive source filtering for the standalone functional tests job
Additionally this skipping of the building is reimplemented to be a bit
more robust and use the same idioms as the functionality for skipping
the tests. In particular, it will now work even if the source files
exist, so we can do this during development too.
2023-10-09 08:29:27 -04:00
John Ericson 644ebaab5f Define NixOS tests in tests/nixos/default.nix rather than flake.nix
I think the our `flake.nix` is currently too large and too scary looking.
I think this matters --- if Nix cannot dog-food itself in a way that is
elegant, why should other people have confidence that their own code can
be elegant and easy to maintain?

We could do this at many points in time, but I think around now, when we
are thinking about stabilizing parts of Flakes, is an especially good
time.

This is a first step to make the `flake.nix` smaller, and make
individual components responsible for their own packaging. I hope we can
do this many more follow-ups like it, until the top-level `flake.nix` is
very small and just coordinates between other things.
2023-10-06 10:58:17 -04:00
John Ericson 68c81c7375 Put functional tests in tests/functional
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests

 - Concepts is harder to understand, the documentation makes a good
   unit vs functional vs integration distinction, but when the
   integration tests are just two subdirs within `tests/` this is not
   clear.

 - Source filtering in the `flake.nix` is more complex. We need to
   filter out some of the dirs from `tests/`, rather than simply pick
   the dirs we want and take all of them. This is a good sign the
   structure of what we are trying to do is not matching the structure
   of the files.

With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests

functional/
installer/
nixos/
```
2023-10-06 09:05:56 -04:00
Eelco Dolstra 3dd4475826
Merge pull request #8905 from hercules-ci/no-unknown-location
Don't print unknown locations unless requested for dev purposes
2023-10-06 14:41:01 +02:00
Théophane Hufschmitt eb68454be6 Don't run the tests that require building if we're not building
A couple of tests require building some libraries that depend on Nix,
and assume it to be built locally.
Don't run these if we only want to run the install tests.

This prevents the CI from rebuilding several times Nix (like in
https://github.com/NixOS/nix/actions/runs/6404422275/job/17384964033#step:6:6412), thus removing a fair amount of build time.
2023-10-05 16:43:26 +02:00
Robert Hensing 3c042f3b0b
Merge pull request #9032 from Ma27/structured-attrs-env-vars
structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE
2023-10-04 11:57:26 +02:00
John Ericson 8440afbed7 Revert "Adapt scheduler to work with dynamic derivations"
This reverts commit 5e3986f59c. This
un-implements RFC 92 but fixes the critical bug #9052 which many people
are hitting. This is a decent stop-gap until a minimal reproduction of
that bug is found and a proper fix can be made.

Mostly fixed #9052, but I would like to leave that issue open until we
have a regression test, so I can then properly fix the bug (unbreaking
RFC 92) later.
2023-10-01 23:43:12 -04:00
Robert Hensing 7a0886e3cc tests/structured-attrs.sh: grep -q -> grepQuiet 2023-10-01 13:25:32 +01:00
Maximilian Bosch bfdd908f7d structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE
In #4770 I implemented proper `nix-shell(1)` support for derivations
using `__structuredAttrs = true;`. Back then we decided to introduce two
new environment variables, `NIX_ATTRS_SH_FILE` for `.attrs.sh` and
`NIX_ATTRS_JSON_FILE` for `.attrs.json`. This was to avoid having to
copy these files to `$NIX_BUILD_TOP` in a `nix-shell(1)` session which
effectively meant copying these files to the project dir without
cleaning up afterwords[1].

On last NixCon I resumed hacking on `__structuredAttrs = true;` by
default for `nixpkgs` with a few other folks and getting back to it,
I identified a few problems with the how it's used in `nixpkgs`:

* A lot of builders in `nixpkgs` don't care about the env vars and
  assume that `.attrs.sh` and `.attrs.json` are in `$NIX_BUILD_TOP`.
  The sole reason why this works is that `nix-shell(1)` sources
  the contents of `.attrs.sh` and then sources `$stdenv/setup` if it
  exists. This may not be pretty, but it mostly works. One notable
  difference when using nixpkgs' stdenv as of now is however that
  `$__structuredAttrs` is set to `1` on regular builds, but set to
  an empty string in a shell session.

  Also, `.attrs.json` cannot be used in shell sessions because
  it can only be accessed by `$NIX_ATTRS_JSON_FILE` and not by
  `$NIX_BUILD_TOP/.attrs.json`.

  I considered changing Nix to be compatible with what nixpkgs
  effectively does, but then we'd have to either move $NIX_BUILD_TOP for
  shell sessions to a temporary location (and thus breaking a lot of
  assumptions) or we'd reintroduce all the problems we solved back then
  by using these two env vars.

  This is partly because I didn't document these variables back
  then (mea culpa), so I decided to drop all mentions of
  `.attrs.{json,sh}` in the  manual and only refer to `$NIX_ATTRS_SH_FILE`
  and `$NIX_ATTRS_JSON_FILE`. The same applies to all our integration tests.
  Theoretically we could deprecated using `"$NIX_BUILD_TOP"/.attrs.sh` in
  the future now.

* `nix develop` and `nix print-dev-env` don't support this environment
  variable at all even though they're supposed to be part of the replacement
  for `nix-shell` - for the drv debugging part to be precise.

  This isn't a big deal for the vast majority of derivations, i.e.
  derivations relying on nixpkgs' `stdenv` wiring things together
  properly. This is because `nix develop` effectively "clones" the
  derivation and replaces the builder with a script that dumps all of
  the environment, shell variables, functions etc, so the state of
  structured attrs being "sourced" is transmitted into the dev shell and
  most of the time you don't need to worry about `.attrs.sh` not
  existing because the shell is correctly configured and the

      if [ -e .attrs.sh ]; then source .attrs.sh; fi

  is simply omitted.

  However, this will break when having a derivation that reads e.g. from
  `.attrs.json` like

      with import <nixpkgs> {};
      runCommand "foo" { __structuredAttrs = true; foo.bar = 23; } ''
        cat $NIX_ATTRS_JSON_FILE # doesn't work because it points to /build/.attrs.json
      ''

  To work around this I employed a similar approach as it exists for
  `nix-shell`: the `NIX_ATTRS_{JSON,SH}_FILE` vars are replaced with
  temporary locations.

  The contents of `.attrs.sh` and `.attrs.json` are now written into the
  JSON by `get-env.sh`, the builder that `nix develop` injects into the
  derivation it's debugging. So finally the exact file contents are
  present and exported by `nix develop`.

  I also made `.attrs.json` a JSON string in the JSON printed by
  `get-env.sh` on purpose because then it's not necessary to serialize
  the object structure again. `nix develop` only needs the JSON
  as string because it's only written into the temporary file.

  I'm not entirely sure if it makes sense to also use a temporary
  location for `nix print-dev-env` (rather than just skipping the
  rewrite in there), but this would probably break certain cases where
  it's relied upon `$NIX_ATTRS_SH_FILE` to exist (prime example are the
  `nix print-dev-env` test-cases I wrote in this patch using
  `tests/shell.nix`, these would fail because the env var exists, but it
  cannot read from it).

[1] https://github.com/NixOS/nix/pull/4770#issuecomment-836799719
2023-10-01 13:22:48 +01:00
Robert Hensing f8a3893e8d pathExists: isDir when endswith /. 2023-09-30 02:35:26 +01:00
tomberek 976f596579
Merge branch 'master' into tomberek.absolute.attrpath.notation 2023-09-28 10:01:57 -04:00
Ilan Joselevich 13ed5d7106 flakes: adopt repl-flake behavior as default 2023-09-27 20:47:10 -04:00
Rasmus Rendal d8cebae939 Add a test for flake paths with spaces in them 2023-09-22 10:06:43 +02:00
Cole Helbling 883092e3f7 Re-enable systemd-nspawn test
It was disabled in c6953d1ff6 because
a recent Nixpkgs bump brought in a new systemd which changed how
systemd-nspawn worked.

As far as I can tell, the issue was caused by this upstream systemd
commit:
b71a0192c0

Bind-mounting the host's `/sys` and `/proc` into the container's
`/run/host/{sys,proc}` fixes the issue and allows the test to succeed.
2023-09-20 09:09:01 -07:00
Eelco Dolstra b6b2a0aea9 Use "touch -h"
https://hydra.nixos.org/build/235888160

This is needed because Nixpkgs now contains dangling symlinks
(pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/pkgs/by-name/fo/foo/foo.nix).
2023-09-19 17:21:07 +02:00
Eelco Dolstra c6953d1ff6 Disable systemd-nspawn test
This is broken because of a change in systemd in NixOS 23.05. It fails
with

  Failed to mount proc (type proc) on /proc (MS_NOSUID|MS_NODEV|MS_NOEXEC ""): Operation not permitted
2023-09-19 17:03:21 +02:00
Eelco Dolstra c451b48993 Merge remote-tracking branch 'origin/master' into p/flake-update 2023-09-19 13:33:56 +02:00
John Ericson 80d7994f52 Special-case error message to add extra information
The Derivation parser and old ATerm unfortunately leaves few ways to get
nice errors when an old version of Nix encounters a new version of the
format. The most likely scenario for this to occur is with a new client
making a derivation that the old daemon it is communicating with cannot
understand.

The extensions we just created for dynamic derivation deps will add a
version field, solving the problem going forward, but there is still the
issue of what to do about old versions of Nix up to now.

The solution here is to carefully catch the bad error from the daemon
that is likely to indicate this problem, and add some extra context to
it.

There is another "Ugly backwards compatibility hack" in
`remote-store.cc` that also works by transforming an error.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
John Ericson 7ad66cb3ef Allow dynamic derivation deps in inputDrvs
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.

The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)

`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.

As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Apply suggestions from code review

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
Robert Hensing 4c50f5d130 traces: Do not print unknown location
Solves 1/3 of the infinite recursion at unknown location meme.
See #8879 for ensuring we always have a trace (for stack overflows)
We might want to re-add this for finding missing location info
*while hacking on that problem only*.
2023-09-03 13:44:32 +02:00
Eelco Dolstra 919781cacc
Merge branch 'master' into valid_deriver_2 2023-09-01 13:35:05 +02:00
Eelco Dolstra b88784278f
Merge pull request #8869 from hercules-ci/fix-issue-8838-pathExists-isDir
Fix #8838, pathExists: isDir when ends with `/ `
2023-09-01 13:15:54 +02:00
Robert Hensing be3362e747 Fix nix-copy test 2023-08-30 19:35:02 -04:00
Eelco Dolstra 5b5f56a9d4
Merge pull request #8859 from edolstra/tarball-last-modified
Tarball trees: Propagate lastModified
2023-08-29 17:02:06 +02:00
Tom Bereknyei 696eb79b15 test: test behavior of .-prefixed attrPaths 2023-08-27 04:42:52 -04:00
Robert Hensing 1e08e12d81 pathExists: isDir when endswith /
Fixes https://github.com/NixOS/nix/issues/8838
2023-08-25 17:18:37 +02:00
Robert Hensing d2e6cfa075 tests/lang/eval-okay-pathexists: Add cases 2023-08-25 17:17:33 +02:00
John Ericson 50f40ac4c0
Merge pull request #8829 from obsidiansystems/build-dynamic-derivations
Adapt scheduler to work with dynamic derivations
2023-08-25 11:13:15 -04:00
tomberek b563ef38cc
Merge pull request #8819 from VertexA115/fix/deep-follow-paths
Fix follow path checking at depths greater than 2
2023-08-25 10:33:05 -04:00
John Ericson dd9f816b29
Merge pull request #8661 from hercules-ci/test-reformat-error-message
tests: Reformat exit code error message
2023-08-25 10:17:15 -04:00
John Ericson 5e3986f59c Adapt scheduler to work with dynamic derivations
To avoid dealing with an optional `drvPath` (because we might not know
it yet) everywhere, make an `CreateDerivationAndRealiseGoal`. This goal
just builds/substitutes the derivation file, and then kicks of a build
for that obtained derivation; in other words it does the chaining of
goals when the drv file is missing (as can already be the case) or
computed (new case).

This also means the `getDerivation` state can be removed from
`DerivationGoal`, which makes the `BasicDerivation` / in memory case and
`Derivation` / drv file file case closer together.

The map type is factored out for clarity, and because we will soon hvae
a second use for it (`Derivation` itself).

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-25 10:01:25 -04:00
Guillaume Girol 925a444b92 add nix-store --query --valid-derivers command
notably useful when nix-store --query --deriver returns a non-existing
path.

Co-authored-by: Felix Uhl <iFreilicht@users.noreply.github.com>
2023-08-24 11:37:24 +02:00
Eelco Dolstra 81045f243f Tarball trees: Propagate lastModified
This makes them behave consistently with GitHub/GitLab flakes.
2023-08-22 21:51:26 +02:00
Robert Hensing 75243c9693
test/flakes/follow-paths.sh: Quote
Co-authored-by: Alex Ameen <alex.ameen.tx@gmail.com>
2023-08-18 14:46:13 +02:00
Felix Uhl 17ceec3a91 Test repl formatting with and without :p 2023-08-17 13:03:43 +02:00
Vertex 20d9c672d1
Update tests/flakes/follow-paths.sh
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-15 10:10:27 +01:00
Alex Zero 1ef8008ca7 Fix follow path checking at depths greater than 2
We need to recurse into the input tree to handle follows paths that
trarverse multiple inputs that may or may not be follow paths
themselves.
2023-08-14 18:55:46 +01:00
John Ericson 44c8d83831 Create outputOf primop.
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.

With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!

Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.

In 60b7121d2c we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!

Part of RFC 92: dynamic derivations (tracking issue #6316)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 09:37:37 -04:00
tomberek 010dc7958e
Merge pull request #8369 from obsidiansystems/inductive-derived-path
Make the Derived Path family of types inductive for dynamic derivations
2023-08-11 08:50:22 -05:00