Commit graph

7379 commits

Author SHA1 Message Date
Eelco Dolstra f902f3c2cb Add experimental feature 'fetch-closure' 2022-03-24 21:33:33 +01:00
Eelco Dolstra e5f7029ba4 nix store make-content-addressed: Support --from / --to 2022-03-24 21:33:33 +01:00
Eelco Dolstra 98658ae9d2 Document fetchClosure 2022-03-24 21:33:33 +01:00
Eelco Dolstra 28186b7044 Add a test for fetchClosure and 'nix store make-content-addressed' 2022-03-24 21:33:33 +01:00
Eelco Dolstra 4120930ac1 fetchClosure: Only allow some "safe" store types 2022-03-24 21:33:33 +01:00
Eelco Dolstra 7ffda0af6e fetchClosure: Skip makeContentAddressed() if toPath is already valid 2022-03-24 21:33:33 +01:00
Eelco Dolstra 5acaf13d35 Rename 'nix store make-content-addressable' to 'nix store make-content-addressed' 2022-03-24 21:33:33 +01:00
Eelco Dolstra f18607549c Fix makeContentAddressed() on self-references
LocalStore::addToStore() since
79ae9e4558 expects a regular NAR hash,
rather than a NAR hash modulo self-references. Fixes #6300.

Also, makeContentAddressed() now rewrites the entire closure (so 'nix
store make-content-addressable' no longer needs '-r'). See #6301.
2022-03-24 21:33:33 +01:00
Eelco Dolstra 545c2d0d8c fetchClosure: Allow a path to be rewritten to CA on the fly
The advantage is that the resulting closure doesn't need to be signed,
so you don't need to configure any binary cache keys on the client.
2022-03-24 21:33:33 +01:00
Eelco Dolstra 7f6fe8ca1d Rename 2022-03-24 21:33:33 +01:00
Eelco Dolstra 41659418cf fetchClosure: Require a CA path in pure mode 2022-03-24 21:33:33 +01:00
Eelco Dolstra f4bafc412f Add builtins.fetchClosure
This allows closures to be imported at evaluation time, without
requiring the user to configure substituters. E.g.

  builtins.fetchClosure {
    storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2;
    from = "https://cache.ngi0.nixos.org";
  }
2022-03-24 21:33:33 +01:00
Eelco Dolstra d67fe90375
Merge pull request #6305 from flox/genericClosure_doc
docs: genericClosure
2022-03-24 14:02:58 +01:00
Tom Bereknyei 0736f3651d docs: genericClosure 2022-03-24 08:03:59 -04:00
Rok Garbas 4546a007a4
Fix flake profile use of originalUrl vs. originalUri
Fixes #5872
2022-03-24 12:28:38 +01:00
Sergei Trofimovich 9174d884d7 lexer: add error location to lexer errors
Before the change lexter errors did not report the location:

    $ nix build -f. mc
    error: path has a trailing slash
    (use '--show-trace' to show detailed location information)

Note that it's not clear what file generates the error.

After the change location is reported:

    $ src/nix/nix --extra-experimental-features nix-command build -f ~/nm mc
    error: path has a trailing slash

           at .../pkgs/development/libraries/glib/default.nix:54:18:

               53|   };
               54|   src = /tmp/foo/;
                 |                  ^
               55|
    (use '--show-trace' to show detailed location information)

Here we see both problematic file and the string itself.
2022-03-24 08:16:14 +00:00
Hideaki Kawai 3b776cb0a7 nix edit: support kakoune 2022-03-22 23:18:02 +09:00
Eelco Dolstra a0259a21a4 Don't hide repeated values while generating manifest.nix
Fixes #6243.
2022-03-22 13:18:56 +01:00
Eelco Dolstra 732296ddc0 printValue(): <REPEAT> -> «repeated»
This ensures that it doesn't get parsed as a valid Nix expression.
2022-03-22 13:00:27 +01:00
Eelco Dolstra 0b42afe027 buildProfile(): Ignore manifest.{nix,json}
If a package installs a file named manifest.json, it caused nix-env to
consider the profile a new-style profile created by 'nix
profile'. Fixes #6032.
2022-03-22 10:48:02 +01:00
Eelco Dolstra e4ff430866
Merge pull request #6237 from obsidiansystems/store-path-string-context
Decode string context straight to using StorePaths
2022-03-22 10:29:46 +01:00
Gabriel Fontes 9720797f69
Don't partial match sourcehut refs 2022-03-19 11:04:04 -03:00
Gabriel Fontes 345a8ee0cb
Fix sourcehut tag ref resolving 2022-03-19 10:56:13 -03:00
John Ericson 4d6a3806d2 Decode string context straight to using StorePaths
I gather decoding happens on demand, so I hope don't think this should
have any perf implications one way or the other.
2022-03-18 15:36:11 +00:00
John Ericson a544ed7684 Generalize DerivationType in preparation for impure derivations 2022-03-18 14:59:56 +00:00
John Ericson 049fae155a Avoid some pointless copying of drvs 2022-03-18 14:59:56 +00:00
John Ericson 8496be7def Use Deferred when building an input-addressed drv
Easier than using dummy path with input addressed.
2022-03-18 14:59:56 +00:00
John Ericson 197feed51d Clean up DerivationOutput, and headers
1. `DerivationOutput` now as the `std::variant` as a base class. And the
   variants are given hierarchical names under `DerivationOutput`.

   In 8e0d0689be @matthewbauer and I
   didn't know a better idiom, and so we made it a field. But this sort
   of "newtype" is anoying for literals downstream.

   Since then we leaned the base class, inherit the constructors trick,
   e.g. used in `DerivedPath`. Switching to use that makes this more
   ergonomic, and consistent.

2. `store-api.hh` and `derivations.hh` are now independent.

   In bcde5456cc I swapped the dependency,
   but I now know it is better to just keep on using incomplete types as
   much as possible for faster compilation and good separation of
   concerns.
2022-03-17 22:35:53 +00:00
Eelco Dolstra 6afc361798
Merge pull request #6277 from thufschmitt/ca/nix-build-dry-run
Fix `nix build --dry-run` with CA derivations
2022-03-17 22:23:15 +01:00
Eelco Dolstra d8f69b9a4c Merge branch 'avoid-quadratic-gc' of https://github.com/trofi/nix 2022-03-17 21:35:42 +01:00
Sergei Trofimovich d58453f72e gc: don't visit implicit referrers on garbage collection
Before the change garbage collector was not considering
`.drv` and outputs as alive even if configuration says otherwise.
As a result `nix store gc --dry-run` could visit (and parse)
`.drv` files multiple times (worst case it's quadratic).

It happens because `alive` set was populating only runtime closure
without regard for actual configuration. The change fixes it.

Benchmark: my system has about 139MB, 40K `.drv` files.

Performance before the change:

    $ time nix store gc --dry-run
    real    4m22,148s

Performance after the change:

    $ time nix store gc --dry-run
    real    0m14,178s
2022-03-17 18:47:29 +00:00
Théophane Hufschmitt 7117053457
Merge pull request #6270 from Artturin/stdineval
nix: allow using --file - to read from stdin
2022-03-17 14:35:01 +01:00
Théophane Hufschmitt 3fc4c612fb Fix nix build --dry-run with CA derivations
Don’t try and assume that we know the output paths when we’ve just built
with `--dry-run`. Instead make `--dry-run` follow a different code path
that won’t assume the knowledge of the output paths at all.

Fix #6275
2022-03-17 11:55:19 +01:00
Théophane Hufschmitt a0b517de57
Merge pull request #6242 from ncfavier/print-output-names
nix-env: always print output names in JSON and XML
2022-03-17 10:55:22 +01:00
Naïm Favier 8dcecc0738
nix-env: print a final newline after JSON 2022-03-16 21:26:19 +01:00
Naïm Favier 5736661922
nix-env: always print output names in JSON and XML
The current `--out-path` flag has two disadvantages when one is only
concerned with querying the names of outputs:
- it requires evaluating every output's `outPath`, which takes
  significantly more resources and runs into more failures
- it destroys the information of the order of outputs so we can't tell
  which one is the main output

This patch makes the output names always present (replacing paths with
`null` in JSON if `--out-path` isn't given), and adds an `outputName`
field.
2022-03-16 21:26:19 +01:00
Artturin a5c969db49 nix: allow using --file - to read from stdin 2022-03-16 21:01:51 +02:00
Eelco Dolstra a53c1dc96d
Merge pull request #6245 from trofi/fix-nix-store-gc-limit-auto-optimise-store
nix store gc: account for auto-optimised store
2022-03-16 09:19:57 +01:00
Théophane Hufschmitt 516a7ac4de
Merge pull request #6229 from obsidiansystems/refactor-hash-modulo
Overhaul derivation hash modulo somewhat
2022-03-15 21:23:44 +01:00
Maximilian Bosch 244baff2c7
libfetchers: remove obsolete filesystem #include 2022-03-15 12:32:11 +01:00
Maximilian Bosch 34e20c164c
libfetchers/path: set lastModified to path's mtime
When importing e.g. a local `nixpkgs` in a flake to test a change like

    {
      inputs.nixpkgs.url = path:/home/ma27/Projects/nixpkgs;
      outputs = /* ... */
    }

then the input is missing a `lastModified`-field that's e.g. used in
`nixpkgs.lib.nixosSystem`. Due to the missing `lastMoified`-field, the
mtime is set to 19700101:

    result -> /nix/store/b7dg1lmmsill2rsgyv2w7b6cnmixkvc1-nixos-system-nixos-22.05.19700101.dirty

With this change, the `path`-fetcher now sets a `lastModified` attribute
to the `mtime` just like it's the case in the `tarball`-fetcher already.
When building NixOS systems with `nixpkgs` being a `path`-input and this
patch, the output-path now looks like this:

    result -> /nix/store/ld2qf9c1s98dxmiwcaq5vn9k5ylzrm1s-nixos-system-nixos-22.05.20220217.dirty
2022-03-15 12:32:11 +01:00
Eelco Dolstra a61809722f
Merge pull request #6244 from Artturin/nixenvbettermessage
nix-env: Add a suggestion for when there's a name collision in channels
2022-03-14 14:17:50 +01:00
Théophane Hufschmitt 0e86ebf461
Merge pull request #6220 from obsidiansystems/log-store
Factor out a `LogStore` interface
2022-03-14 10:51:01 +01:00
Sergei Trofimovich 6b1872312f nix store gc: account for auto-optimised store
Before the change on a system with `auto-optimise-store = true`:

    $ nix store gc --verbose --max 1

deleted all the paths instead of one path (we requested 1 byte limit).

It happens because every file in `auto-optimise-store = true` has at
least 2 links: file itself and a link in /nix/store/.links/ directory.

The change conservatively assumes that any file that has one (as before)
or two links (assume auto-potimise mode) will free space.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2022-03-13 07:24:48 +00:00
Artturin cb1a76112e nix-env: Add a suggestion for when there's a name collision in channels
help new users find a solution to their problem

./result/bin/nix-env -qa hello
warning: name collision in input Nix expressions, skipping '/home/artturin/.nix-defexpr/channels_root/master'
suggestion: remove 'master' from either the root channels or the user channels
hello-2.12
hello-2.12
2022-03-13 00:24:46 +02:00
John Ericson 91adfb8894 Create some type aliases for string Contexts 2022-03-11 22:30:10 +00:00
John Ericson 0948b8e94d Reduce variants for derivation hash modulo
This changes was taken from dynamic derivation (#4628). It` somewhat
undoes the refactors I first did for floating CA derivations, as the
benefit of hindsight + requirements of dynamic derivations made me
reconsider some things.

They aren't to consequential, but I figured they might be good to land
first, before the more profound changes @thufschmitt has in the works.
2022-03-11 21:20:37 +00:00
John Ericson 015d2ad507 Desugar StorePathWithOutputs in nix-build implementation
`DerivedPath` has replaced `StorePathWithOutputs` internally, so
shrinking the usage of `StorePathWithOutputs` to just the boundary is
good.
2022-03-11 20:50:52 +00:00
John Ericson a03b1fd7f6 Deduplicate the Store downcasting with a template 2022-03-11 13:32:33 +00:00
John Ericson 678d1c2aa0 Factor out a LogStore interface
Continue progress on #5729.

Just as I hoped, this uncovered an issue: the daemon protocol is missing
a way to query build logs. This doesn't effect `unix://`, but does
effect `ssh://`. A FIXME is left for this, so we come back to it later.
2022-03-11 13:32:16 +00:00