Commit graph

7510 commits

Author SHA1 Message Date
Eelco Dolstra d6effddd3b
Merge pull request #6387 from Uthar/fix
assert hash types for Git and Mercurial
2022-04-14 14:55:27 +02:00
Eelco Dolstra d89840b103 Make InstallableFlake::toValue() and toDerivation() behave consistently
In particular, this means that 'nix eval` (which uses toValue()) no
longer auto-calls functions or functors (because
AttrCursor::findAlongAttrPath() doesn't).

Fixes #6152.

Also use ref<> in a few places, and don't return attrpaths from
getCursor() because cursors already have a getAttrPath() method.
2022-04-14 14:07:04 +02:00
Kasper Gałkowski 2769e43f61 assert hash types for Git and Mercurial 2022-04-12 21:13:14 +02:00
Eelco Dolstra f7276bc948
Merge pull request #6392 from danpls/fix-actualUrl-mercurial
libfetchers: Fix assertion (Mercurial)
2022-04-11 11:16:04 +02:00
Eelco Dolstra 092f6d2e7a
Merge pull request #6380 from thufschmitt/fix-double-slahsh-in-uri
Allow empty path segments in urls
2022-04-11 11:15:14 +02:00
Eelco Dolstra 2311868aaa
Merge pull request #6391 from danpls/replace-regex
libfetchers: Replace regex to clarify intent
2022-04-11 11:14:10 +02:00
Sebastian Blunt 63d9a81819 Log builder args and environment variables
Previously it only logged the builder's path, this changes it to log the
arguments at the same log level, and the environment variables at the
vomit level.

This helped me debug https://github.com/svanderburg/node2nix/issues/75
2022-04-10 21:10:37 -07:00
Daniel Pauls d6b7529579 libfetchers: Fix assertion (Mercurial)
See commit 1e1cd6e7a for more information.
2022-04-09 19:10:23 +02:00
Daniel Pauls 770f7371f3 libfetchers: Replace regex to clarify intent 2022-04-09 17:00:14 +02:00
Théophane Hufschmitt 646af7325d
Merge pull request #6376 from Uthar/master
don't assume that rev is a SHA1 hash
2022-04-08 17:56:27 +02:00
Théophane Hufschmitt f3d3587ab3 Allow empty path segments in urls
Valid per https://datatracker.ietf.org/doc/html/rfc3986#section-3.3 (and
also somewhat frequently happening for local paths)
2022-04-08 16:09:49 +02:00
Eelco Dolstra c68963eaea Remove duplicate "error:" 2022-04-08 11:48:30 +02:00
Eelco Dolstra 8bd9ebf52c Error: Remove unused sname() method 2022-04-08 11:31:51 +02:00
Eelco Dolstra 168ef9f3ab Remove unused Error.name field 2022-04-08 11:31:51 +02:00
Kasper Gałkowski 2c2fd4946f don't assume that rev is a SHA1 hash
This was a problem when writing a fetcher that uses e.g. sha256 hashes
for revisions. This doesn't actually do anything new, but allows for
creating such fetchers in the future (perhaps when support for Git's
SHA256 object format gains more popularity).
2022-04-07 19:49:47 +02:00
Eelco Dolstra 8b1e328d5d
Merge pull request #6348 from cole-h/fix-restoring-mount-namespace
libutil: Fix restoring mount namespace
2022-04-07 18:15:33 +02:00
Théophane Hufschmitt b53e0a6aa0
Merge pull request #6374 from danpls/fix-actualUrl
libfetchers: Fix assertion
2022-04-07 17:57:24 +02:00
Rehno Lindeque b9c969a866 nix flake check: Warn about deprecated nixosModule output 2022-04-06 12:20:39 -04:00
Daniel Pauls 1e1cd6e7a9 libfetchers: Fix assertion
The filter expects all paths to have a prefix of the raw `actualUrl`, but
`Store::addToStore(...)` provides absolute canonicalized paths.
To fix this create an absolute and canonicalized path from the `actualUrl` and
use it instead.

Fixes #6195.
2022-04-06 17:33:23 +02:00
Eelco Dolstra 318936366d Fix empty 'nix copy' error message
This was caused by SubstitutionGoal not setting the errorMsg field in
its BuildResult. We now get a more descriptive message than in 2.7.0, e.g.

  error: path '/nix/store/13mh...' is required, but there is no substituter that can build it

instead of the misleading (since there was no build)

  error: build of '/nix/store/13mh...' failed

Fixes #6295.
2022-04-06 12:43:53 +02:00
Eelco Dolstra 589f6f267b fetchClosure: Don't allow URL query parameters
Allowing this is a potential security hole, since it allows the user
to specify parameters like 'local-nar-cache'.
2022-04-06 11:52:51 +02:00
Eelco Dolstra c0ad86f681
Merge pull request #6366 from danpls/base64-reserve
libutil: Reserve memory when en/decoding base64
2022-04-05 23:20:33 +02:00
Daniel Pauls 513652d594 tokenizeString: Fix semantic mistake
`string_view::find_first_not_of(...)` and
`string_view::find_first_of(...)` return `string_view::npos` on error
not `string::npos`.
2022-04-05 22:33:03 +02:00
Daniel Pauls 1fa0393479 libutil: Reserve memory when en/decoding base64
The size of the output when encoding to and decoding from base64 is
(roughly) known so we can allocate it in advance to prevent
reallocation.
2022-04-05 21:30:50 +02:00
Eelco Dolstra 27b952a8a1
Merge pull request #6362 from thufschmitt/verbose-doctor
doctor: Always show the output
2022-04-05 17:33:10 +02:00
Théophane Hufschmitt 9a640afc1e doctor: Always show the output
Fix https://github.com/NixOS/nix/issues/6342
2022-04-05 14:04:01 +02:00
Théophane Hufschmitt 5abe3f4aa6 Allow welcomeText when checking a flake template
Fix https://github.com/NixOS/nix/issues/6321
2022-04-05 11:03:43 +02:00
Cole Helbling 56009b2639 libutil: don't save cwd fd, use path instead
Saving the cwd fd didn't actually work well -- prior to this commit, the
following would happen:

    : ~/w/vc/nix ; doas outputs/out/bin/nix --experimental-features 'nix-command flakes' run nixpkgs#coreutils -- --coreutils-prog=pwd
    pwd: couldn't find directory entry in ‘../../../..’ with matching i-node
    : ~/w/vc/nix ; doas outputs/out/bin/nix --experimental-features 'nix-command flakes' develop -c pwd
    pwd: couldn't find directory entry in ‘../../../..’ with matching i-node
2022-04-04 10:28:08 -07:00
Cole Helbling 10b9c1b2b2 libutil: save cwd fd in restoreMountNamespace
This doesn't work very well (maybe I'm misunderstanding the desired
implementation):

    : ~/w/vc/nix ; doas outputs/out/bin/nix --experimental-features 'nix-command flakes' develop -c pwd
    pwd: couldn't find directory entry in ‘../../../..’ with matching i-node
2022-04-04 10:28:00 -07:00
Cole Helbling f89b0f7846 libutil: try restoring the cwd from fdSavedCwd 2022-04-04 08:33:59 -07:00
Cole Helbling e135d223f6 libutil: save fd to cwd instead of cwd itself 2022-04-04 08:32:45 -07:00
Cole Helbling e5b70d47aa libutil: cleanup savedCwd logic
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-04-04 08:20:11 -07:00
Eelco Dolstra a4a1de69dc Add missing #include 2022-04-04 16:49:39 +02:00
Tom Bereknyei c1e2ce4515 fix(run): set applyNixConfig lockFlag 2022-04-01 23:41:34 -04:00
Cole Helbling 2a45cf54e4 libutil: Properly guard self-allocating getcwd on GNU
It's a GNU extension, as pointed out by pennae.
2022-04-01 12:20:34 -07:00
Cole Helbling 7f5caaa7c0 libutil: Don't use std::filesystem
Just in case making libutil depend on std::filesystem is unacceptable,
here is the non-filesystem approach.
2022-04-01 10:24:31 -07:00
aszlig 435848cef1 libutil: Fix restoring mount namespace
I regularly pass around simple scripts by using nix-shell as the script
interpreter, eg. like this:

    #!/usr/bin/env nix-shell
    #!nix-shell -p dd_rescue coreutils bash -i bash

While this works most of the time, I recently had one occasion where it
would not and the above would result in the following:

    $ sudo ./myscript.sh
    bash: ./myscript.sh: No such file or directory

Note the "sudo" here, because this error only occurs if we're root.

The reason for the latter is because running Nix as root means that we
can directly access the store, which makes sure we use a filesystem
namespace to make the store writable. XXX - REWORD!

So when stracing the process, I stumbled on the following sequence:

    openat(AT_FDCWD, "/proc/self/ns/mnt", O_RDONLY) = 3
    unshare(CLONE_NEWNS)                            = 0
    ... later ...
    getcwd("/the/real/cwd", 4096)                   = 14
    setns(3, CLONE_NEWNS)                           = 0
    getcwd("/", 4096)                               = 2

In the whole strace output there are no calls to chdir() whatsoever, so
I decided to look into the kernel source to see what else could change
directories and found this[1]:

    /* Update the pwd and root */
    set_fs_pwd(fs, &root);
    set_fs_root(fs, &root);

The set_fs_pwd() call is roughly equivalent to a chdir() syscall and
this is called when the setns() syscall is invoked[2].

[1]: b14ffae378/fs/namespace.c (L4659)
[2]: b14ffae378/kernel/nsproxy.c (L346)
2022-04-01 09:30:52 -07:00
Eelco Dolstra bf4895961d
Merge pull request #6344 from flox/profile_url_uri
profile!: consistent use of url/uri. create new version
2022-04-01 14:38:32 +02:00
Eelco Dolstra fdfe737867 Fix handling of outputHash when outputHashAlgo is not specified
https://hydra.nixos.org/build/171351131
2022-04-01 12:40:49 +02:00
Eelco Dolstra c9a29d0d92
Merge pull request #6227 from NixOS/impure-derivations-ng
Impure derivations
2022-03-31 19:58:35 +02:00
Eelco Dolstra 7537097284 Provide default values for outputHashAlgo and outputHashMode 2022-03-31 16:56:44 +02:00
Eelco Dolstra a99af85a77 Fix macOS build 2022-03-31 16:39:18 +02:00
Eelco Dolstra 6051cc954b Rename 'pure' -> 'sandboxed' for consistency 2022-03-31 16:12:25 +02:00
Eelco Dolstra e279fbb16a needsNetworkAccess() -> isSandboxed() 2022-03-31 16:06:40 +02:00
Eelco Dolstra 4e043c2f32 Document isPure() 2022-03-31 16:01:50 +02:00
Eelco Dolstra d7fc33c842 Fix macOS build 2022-03-31 15:59:14 +02:00
Eelco Dolstra 18935e8b9f Support fixed-output derivations depending on impure derivations 2022-03-31 13:43:20 +02:00
Eelco Dolstra 5cd72598fe Add support for impure derivations
Impure derivations are derivations that can produce a different result
every time they're built. Example:

  stdenv.mkDerivation {
    name = "impure";
    __impure = true; # marks this derivation as impure
    outputHashAlgo = "sha256";
    outputHashMode = "recursive";
    buildCommand = "date > $out";
  };

Some important characteristics:

* This requires the 'impure-derivations' experimental feature.

* Impure derivations are not "cached". Thus, running "nix-build" on
  the example above multiple times will cause a rebuild every time.

* They are implemented similar to CA derivations, i.e. the output is
  moved to a content-addressed path in the store. The difference is
  that we don't register a realisation in the Nix database.

* Pure derivations are not allowed to depend on impure derivations. In
  the future fixed-output derivations will be allowed to depend on
  impure derivations, thus forming an "impurity barrier" in the
  dependency graph.

* When sandboxing is enabled, impure derivations can access the
  network in the same way as fixed-output derivations. In relaxed
  sandboxing mode, they can access the local filesystem.
2022-03-31 13:43:20 +02:00
Eelco Dolstra 0fe8849914
Merge pull request #6337 from danpls/fix-to-json-repl
libexpr: Throw the correct error in toJSON
2022-03-31 11:33:16 +02:00
Eelco Dolstra 28309352d9 replaceEnv(): Pass newEnv by reference 2022-03-31 10:39:53 +02:00
Tom Bereknyei 50f9f335c9 profile!: consistent use of url/uri. create new version 2022-03-30 16:35:26 -04:00
Tom Bereknyei d77823b502 bundler: update default bundler to support new bundler API 2022-03-30 16:10:42 -04:00
Daniel Pauls 629edd43ba libutil: Change return value of addTrace to void
The return value of BaseError::addTrace(...) is never used and
error-prone as subclasses calling it will return a BaseError instead of
the subclass.
This commit changes its return value to be void.
2022-03-30 18:37:32 +02:00
Daniel Pauls fa83b865a2 libexpr: Throw the correct error in toJSON
BaseError::addTrace(...) returns a BaseError, but we want to
throw a TypeError instead.

Fixes #6336.
2022-03-30 15:50:13 +02:00
Théophane Hufschmitt 390269ed87 Simplify the handling of the hash modulo
Rather than having four different but very similar types of hashes, make
only one, with a tag indicating whether it corresponds to a regular of
deferred derivation.

This implies a slight logical change: The original Nix+multiple-outputs
model assumed only one hash-modulo per derivation. Adding
multiple-outputs CA derivations changed this as these have one
hash-modulo per output. This change is now treating each derivation as
having one hash modulo per output.
This obviously means that we internally loose the guaranty that
all the outputs of input-addressed derivations have the same hash
modulo. But it turns out that it doesn’t matter because there’s nothing
in the code taking advantage of that fact (and it probably shouldn’t
anyways).

The upside is that it is now much easier to work with these hashes, and
we can get rid of a lot of useless `std::visit{ overloaded`.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2022-03-29 18:17:35 +02:00
Théophane Hufschmitt 2d572a250f
Merge pull request #6330 from edolstra/run-remote-store
nix {run,shell}: Print a better error message if the store is not local
2022-03-28 17:57:37 +02:00
Eelco Dolstra b266fd53dd nix {run,shell}: Print a better error message if the store is not local
Closes #6317
2022-03-28 14:58:38 +02:00
Eelco Dolstra 057f9ee190 nix profile install: Don't use queryDerivationOutputMap()
Instead get the outputs from Installable::build(). This will also
allow 'nix profile install' to support impure derivations.

Fixes #6286.
2022-03-28 14:23:39 +02:00
Erik Arvstedt 16860a0328 nix eval: Add option read-only 2022-03-26 11:32:38 +01:00
Erik Arvstedt 679b3b32c9 Minor comment fix 2022-03-26 11:32:37 +01:00
Eelco Dolstra 1844172dd1
Merge pull request #6314 from edolstra/experimental-primop
Only provide builtins is the corresponding experimental feature is enabled
2022-03-25 16:16:31 +01:00
Eelco Dolstra 16cf1e6089
Merge pull request #6311 from edolstra/return-wanted-paths
Make buildPathsWithResults() only return info on wanted outputs
2022-03-25 15:44:39 +01:00
Eelco Dolstra fc35b11a7c Fix mismatched tag warning on clang 2022-03-25 15:22:22 +01:00
Eelco Dolstra 8c363eb3eb Document getFlake
Fixes #5523.
2022-03-25 14:19:55 +01:00
Eelco Dolstra 86b05ccd54 Only provide builtin.{getFlake,fetchClosure} is the corresponding experimental feature is enabled
This allows writing fallback code like

  if builtins ? fetchClosure then
    builtins.fetchClose { ... }
  else
    builtins.storePath ...
2022-03-25 14:04:18 +01:00
Eelco Dolstra 55bc524019
Merge pull request #6051 from Ma27/fix-empty-nix-log
`nix log` should also work if the log didn't provide any output
2022-03-25 10:35:10 +01:00
Eelco Dolstra 50c229ad9a
Use wantOutput
Co-authored-by: John Ericson <git@JohnEricson.me>
2022-03-25 08:02:49 +01:00
polykernel cbcb69a39c nix: allow whitespace characters before command in repl
Before this change, processLine always uses the first character
as the start of the line. This cause whitespaces to matter at the
beginning of the line whereas it does not matter anywhere else.

This commit trims leading white spaces of the string line so that
subsequent operations can be performed on the string without explicitly
tracking starting and ending indices of the string.
2022-03-24 21:33:29 -04:00
Eelco Dolstra 540d7e33d8 Retry substitution after an incomplete closure only once
This avoids an infinite loop in the final test in
tests/binary-cache.sh. I think this was only not triggered previously
by accident (because we were clearing wantedOutputs in between).
2022-03-24 23:25:12 +01:00
Eelco Dolstra fe5509df9a Only return wanted outputs 2022-03-24 23:24:48 +01:00
Eelco Dolstra 09796c0263 Random cleanup 2022-03-24 23:24:10 +01:00
Eelco Dolstra 175c78591b Random cleanup 2022-03-24 23:09:43 +01:00
Maximilian Bosch c85467a1b6
Revert "TarArchive: Small refactoring"
This reverts commit 50a35860ee.

With this change Nix fails to open bzip2 logfiles that were created from
builds with no stdout/stderr.
2022-03-24 22:30:46 +01:00
Eelco Dolstra f902f3c2cb Add experimental feature 'fetch-closure' 2022-03-24 21:33:33 +01:00
Eelco Dolstra e5f7029ba4 nix store make-content-addressed: Support --from / --to 2022-03-24 21:33:33 +01:00
Eelco Dolstra 98658ae9d2 Document fetchClosure 2022-03-24 21:33:33 +01:00
Eelco Dolstra 28186b7044 Add a test for fetchClosure and 'nix store make-content-addressed' 2022-03-24 21:33:33 +01:00
Eelco Dolstra 4120930ac1 fetchClosure: Only allow some "safe" store types 2022-03-24 21:33:33 +01:00
Eelco Dolstra 7ffda0af6e fetchClosure: Skip makeContentAddressed() if toPath is already valid 2022-03-24 21:33:33 +01:00
Eelco Dolstra 5acaf13d35 Rename 'nix store make-content-addressable' to 'nix store make-content-addressed' 2022-03-24 21:33:33 +01:00
Eelco Dolstra f18607549c Fix makeContentAddressed() on self-references
LocalStore::addToStore() since
79ae9e4558 expects a regular NAR hash,
rather than a NAR hash modulo self-references. Fixes #6300.

Also, makeContentAddressed() now rewrites the entire closure (so 'nix
store make-content-addressable' no longer needs '-r'). See #6301.
2022-03-24 21:33:33 +01:00
Eelco Dolstra 545c2d0d8c fetchClosure: Allow a path to be rewritten to CA on the fly
The advantage is that the resulting closure doesn't need to be signed,
so you don't need to configure any binary cache keys on the client.
2022-03-24 21:33:33 +01:00
Eelco Dolstra 7f6fe8ca1d Rename 2022-03-24 21:33:33 +01:00
Eelco Dolstra 41659418cf fetchClosure: Require a CA path in pure mode 2022-03-24 21:33:33 +01:00
Eelco Dolstra f4bafc412f Add builtins.fetchClosure
This allows closures to be imported at evaluation time, without
requiring the user to configure substituters. E.g.

  builtins.fetchClosure {
    storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2;
    from = "https://cache.ngi0.nixos.org";
  }
2022-03-24 21:33:33 +01:00
Eelco Dolstra d67fe90375
Merge pull request #6305 from flox/genericClosure_doc
docs: genericClosure
2022-03-24 14:02:58 +01:00
Tom Bereknyei 0736f3651d docs: genericClosure 2022-03-24 08:03:59 -04:00
Rok Garbas 4546a007a4
Fix flake profile use of originalUrl vs. originalUri
Fixes #5872
2022-03-24 12:28:38 +01:00
Sergei Trofimovich 9174d884d7 lexer: add error location to lexer errors
Before the change lexter errors did not report the location:

    $ nix build -f. mc
    error: path has a trailing slash
    (use '--show-trace' to show detailed location information)

Note that it's not clear what file generates the error.

After the change location is reported:

    $ src/nix/nix --extra-experimental-features nix-command build -f ~/nm mc
    error: path has a trailing slash

           at .../pkgs/development/libraries/glib/default.nix:54:18:

               53|   };
               54|   src = /tmp/foo/;
                 |                  ^
               55|
    (use '--show-trace' to show detailed location information)

Here we see both problematic file and the string itself.
2022-03-24 08:16:14 +00:00
Hideaki Kawai 3b776cb0a7 nix edit: support kakoune 2022-03-22 23:18:02 +09:00
Eelco Dolstra a0259a21a4 Don't hide repeated values while generating manifest.nix
Fixes #6243.
2022-03-22 13:18:56 +01:00
Eelco Dolstra 732296ddc0 printValue(): <REPEAT> -> «repeated»
This ensures that it doesn't get parsed as a valid Nix expression.
2022-03-22 13:00:27 +01:00
Eelco Dolstra 0b42afe027 buildProfile(): Ignore manifest.{nix,json}
If a package installs a file named manifest.json, it caused nix-env to
consider the profile a new-style profile created by 'nix
profile'. Fixes #6032.
2022-03-22 10:48:02 +01:00
Eelco Dolstra e4ff430866
Merge pull request #6237 from obsidiansystems/store-path-string-context
Decode string context straight to using StorePaths
2022-03-22 10:29:46 +01:00
Gabriel Fontes 9720797f69
Don't partial match sourcehut refs 2022-03-19 11:04:04 -03:00
Gabriel Fontes 345a8ee0cb
Fix sourcehut tag ref resolving 2022-03-19 10:56:13 -03:00
John Ericson 4d6a3806d2 Decode string context straight to using StorePaths
I gather decoding happens on demand, so I hope don't think this should
have any perf implications one way or the other.
2022-03-18 15:36:11 +00:00
John Ericson a544ed7684 Generalize DerivationType in preparation for impure derivations 2022-03-18 14:59:56 +00:00
John Ericson 049fae155a Avoid some pointless copying of drvs 2022-03-18 14:59:56 +00:00
John Ericson 8496be7def Use Deferred when building an input-addressed drv
Easier than using dummy path with input addressed.
2022-03-18 14:59:56 +00:00