Commit graph

251 commits

Author SHA1 Message Date
Eelco Dolstra 01232358ff Merge remote-tracking branch 'origin/master' into source-path 2023-04-24 13:20:36 +02:00
John Ericson 85f0cdc370 Use std::set<StringContextElem> not PathSet for string contexts
Motivation

`PathSet` is not correct because string contexts have other forms
(`Built` and `DrvDeep`) that are not rendered as plain store paths.
Instead of wrongly using `PathSet`, or "stringly typed" using
`StringSet`, use `std::std<StringContextElem>`.

-----

In support of this change, `NixStringContext` is now defined as
`std::std<StringContextElem>` not `std:vector<StringContextElem>`. The
old definition was just used by a `getContext` method which was only
used by the eval cache. It can be deleted altogether since the types are
now unified and the preexisting `copyContext` function already suffices.

Summarizing the previous paragraph:

Old:

  - `value/context.hh`: `NixStringContext = std::vector<StringContextElem>`
  - `value.hh`: `NixStringContext Value::getContext(...)`
  - `value.hh`: `copyContext(...)`

New:

  - `value/context.hh`: `NixStringContext = std::set<StringContextElem>`
  - `value.hh`: `copyContext(...)`
----

The string representation of string context elements no longer contains
the store dir. The diff of `src/libexpr/tests/value/context.cc` should
make clear what the new representation is, so we recommend reviewing
that file first. This was done for two reasons:

Less API churn:

`Value::mkString` and friends did not take a `Store` before. But if
`NixStringContextElem::{parse, to_string}` *do* take a store (as they
did before), then we cannot have the `Value` functions use them (in
order to work with the fully-structured `NixStringContext`) without
adding that argument.

That would have been a lot of churn of threading the store, and this
diff is already large enough, so the easier and less invasive thing to
do was simply make the element `parse` and `to_string` functions not
take the `Store` reference, and the easiest way to do that was to simply
drop the store dir.

Space usage:

Dropping the `/nix/store/` (or similar) from the internal representation
will safe space in the heap of the Nix programming being interpreted. If
the heap contains many strings with non-trivial contexts, the saving
could add up to something significant.

----

The eval cache version is bumped.

The eval cache serialization uses `NixStringContextElem::{parse,
to_string}`, and since those functions are changed per the above, that
means the on-disk representation is also changed.

This is simply done by changing the name of the used for the eval cache
from `eval-cache-v4` to eval-cache-v5`.

----

To avoid some duplication `EvalCache::mkPathString` is added to abstract
over the simple case of turning a store path to a string with just that
string in the context.

Context

This PR picks up where #7543 left off. That one introduced the fully
structured `NixStringContextElem` data type, but kept `PathSet context`
as an awkward middle ground between internal `char[][]` interpreter heap
string contexts and `NixStringContext` fully parsed string contexts.

The infelicity of `PathSet context` was specifically called out during
Nix team group review, but it was agreeing that fixing it could be left
as future work. This is that future work.

A possible follow-up step would be to get rid of the `char[][]`
evaluator heap representation, too, but it is not yet clear how to do
that. To use `NixStringContextElem` there we would need to get the STL
containers to GC pointers in the GC build, and I am not sure how to do
that.

----

PR #7543 effectively is writing the inverse of a `mkPathString`,
`mkOutputString`, and one more such function for the `DrvDeep` case. I
would like that PR to have property tests ensuring it is actually the
inverse as expected.

This PR sets things up nicely so that reworking that PR to be in that
more elegant and better tested way is possible.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-04-21 01:05:49 -04:00
Eelco Dolstra 94812cca98 Backport SourcePath from the lazy-trees branch
This introduces the SourcePath type from lazy-trees as an abstraction
for accessing files from inputs that may not be materialized in the
real filesystem (e.g. Git repositories). Currently, however, it's just
a wrapper around CanonPath, so it shouldn't change any behaviour. (On
lazy-trees, SourcePath is a <InputAccessor, CanonPath> tuple.)
2023-04-06 13:15:50 +02:00
John Ericson 570829d67e
Merge pull request #7609 from obsidiansystems/hide-experimental-settings
Hide experimental settings
2023-03-27 09:19:29 -04:00
oxalica 2941a599fa Catch eval errors in hasContent
`legacyPackages` of nixpkgs trigger eval errors in `hasContent`, causing
the whole `legacyPackages` being skipped. We should treat it as
has-content in that case.
2023-03-26 23:05:29 +08:00
John Ericson 296831f641 Move enabled experimental feature to libutil struct
This is needed in subsequent commits to allow the settings and CLI args
infrastructure itself to read this setting.
2023-03-20 11:05:22 -04:00
John Ericson bc23a44c54 Make command infra less stateful and more regular
Already, we had classes like `BuiltPathsCommand` and `StorePathsCommand`
which provided alternative `run` virtual functions providing the
implementation with more arguments. This was a very nice and easy way to
make writing command; just fill in the virtual functions and it is
fairly clear what to do.

However, exception to this pattern were `Installable{,s}Command`. These
two classes instead just had a field where the installables would be
stored, and various side-effecting `prepare` and `load` machinery too
fill them in. Command would wish out those fields.

This isn't so clear to use.

What this commit does is make those command classes like the others,
with richer `run` functions.

Not only does this restore the pattern making commands easier to write,
it has a number of other benefits:

- `prepare` and `load` are gone entirely! One command just hands just
  hands off to the next.

- `useDefaultInstallables` because `defaultInstallables`. This takes
  over `prepare` for the one case that needs it, and provides enough
  flexiblity to handle `nix repl`'s idiosyncratic migration.

- We can use `ref` instead of `std::shared_ptr`. The former must be
  initialized (so it is like Rust's `Box` rather than `Option<Box>`,
  This expresses the invariant that the installable are in fact
  initialized much better.

  This is possible because since we just have local variables not
  fields, we can stop worrying about the not-yet-initialized case.

- Fewer lines of code! (Finally I have a large refactor that makes the
  number go down not up...)

- `nix repl` is now implemented in a clearer way.

The last item deserves further mention. `nix repl` is not like the other
installable commands because instead working from once-loaded
installables, it needs to be able to load them again and again.

To properly support this, we make a new superclass
`RawInstallablesCommand`. This class has the argument parsing and
completion logic, but does *not* hand off parsed installables but
instead just the raw string arguments.

This is exactly what `nix repl` needs, and allows us to instead of
having the logic awkwardly split between `prepare`,
`useDefaultInstallables,` and `load`, have everything right next to each
other. I think this will enable future simplifications of that argument
defaulting logic, but I am saving those for a future PR --- best to keep
code motion and more complicated boolean expression rewriting separate
steps.

The "diagnostic ignored `-Woverloaded-virtual`" pragma helps because C++
doesn't like our many `run` methods. In our case, we don't mind the
shadowing it all --- it is *intentional* that the derived class only
provides a `run` method, and doesn't call any of the overridden `run`
methods.

Helps with https://github.com/NixOS/rfcs/pull/134
2023-03-15 16:29:07 -04:00
Eelco Dolstra b69a73a230 Get rid of some unchecked calls to std::cout 2023-03-02 15:02:24 +01:00
John Ericson fa4733fce5 Split out InstallableFlake and InstallableAttrPath 2023-02-20 09:09:11 -05:00
Robert Hensing 60d48eda23 nix flake show: Ignore empty attrsets
For frameworks it's important that structures are as lazy as possible
to prevent infinite recursions, performance issues and errors that
aren't related to the thing to evaluate. As a consequence, they have
to emit more attributes than strictly (sic) necessary.
However, these attributes with empty values are not useful to the user
so we omit them.
2023-01-31 18:20:26 +01:00
Max 02e81cdf62 apply showAllSystems to legacyPackages as well 2023-01-27 23:59:48 +01:00
Théophane Hufschmitt ab424a39a9 Merge remote-tracking branch 'nixos/master' into pr-flake-show-foreign 2023-01-27 09:46:46 +01:00
Guillaume Maudoux e4726a0c79 Revert "Revert "Merge pull request #6204 from layus/coerce-string""
This reverts commit 9b33ef3879.
2023-01-19 13:23:04 +01:00
Robert Hensing 9b33ef3879 Revert "Merge pull request #6204 from layus/coerce-string"
This reverts commit a75b7ba30f, reversing
changes made to 9af16c5f74.
2023-01-18 01:34:07 +01:00
Robert Hensing 4e7592b593 flake check: Recognize well known community attributes
This avoids warning fatigue, making `nix flake check` more effective.
2023-01-16 20:16:45 +01:00
John Ericson 5ba6e5d0d9 Remove default constructor from OutputsSpec
This forces us to be explicit.

It also requires to rework how `from_json` works. A `JSON_IMPL` is added
to assist with this.
2023-01-11 19:08:19 -05:00
John Ericson 5576d5e987 Parse string context elements properly
Prior to this change, we had a bunch of ad-hoc string manipulation code
scattered around. This made it hard to figure out what data model for
string contexts is.

Now, we still store string contexts most of the time as encoded strings
--- I was wary of the performance implications of changing that --- but
whenever we parse them we do so only through the
`NixStringContextElem::parse` method, which handles all cases. This
creates a data type that is very similar to `DerivedPath` but:

 - Represents the funky `=<drvpath>` case as properly distinct from the
   others.

 - Only encodes a single output, no wildcards and no set, for the
   "built" case.

(I would like to deprecate `=<path>`, after which we are in spitting
distance of `DerivedPath` and could maybe get away with fewer types, but
that is another topic for another day.)
2023-01-10 13:10:49 -05:00
John Ericson da64f026dd Make clear that StorePathWithOutputs is a deprecated type
- Add a comment

- Put `OutputsSpec` in a different header (First part of #6815)

- Make a few stray uses of it in new code use `DerivedPath` instead.
2023-01-10 11:27:19 -05:00
Naïm Favier f1ee4ece80
Don't check NixOS modules
NixOS modules can be paths. Rather than dig further down into the layer
violation, don't check anything specific to NixOS modules.
2023-01-05 18:23:30 +01:00
Eelco Dolstra 6b69652385 Merge remote-tracking branch 'origin/master' into coerce-string 2023-01-02 20:53:39 +01:00
Eelco Dolstra 703d863a48 Trivial changes from the lazy-trees branch 2022-12-07 14:06:34 +01:00
Yorick 09f00dd4d0
Replace src/libutil/json.cc with nlohmann json generation 2022-11-16 16:50:50 +01:00
Max 6da4cc92d8 showForeign -> showAllSystems 2022-09-07 20:28:30 +02:00
Guillaume Maudoux eb460a9529 WIP: broken merge but need a git checkpoint 2022-09-07 00:34:03 +02:00
Max 02bff90e7b nix flake show: don't evaluate derivations for foreign systems by default 2022-09-01 22:04:22 +02:00
Eelco Dolstra e62160579f nix flake metadata: Don't show "Inputs" if there are no inputs 2022-08-17 11:31:34 +02:00
Théophane Hufschmitt f071eb3702
Merge pull request #6693 from ncfavier/complete-flake-inputs
Improve shell completion of flake inputs
2022-07-12 09:12:00 +02:00
Théophane Hufschmitt f8fea9075c Merge remote-tracking branch 'origin/master' into better-flake-new-error-message 2022-06-29 13:44:14 +02:00
Manu [tennox] ae4c9ef8e2 #6699 flake init: fix trying to add unchanged file
After skipping because of being of identical content it tried to git add it.
2022-06-26 21:35:21 +01:00
Manu [tennox] 4374e3ec67 #6699 flake init: Apply suggestions of @thufschmitt 2022-06-26 18:12:30 +01:00
Manuel 58cbbdc5e7
Update src/nix/flake.cc
https://github.com/NixOS/nix/pull/6699#discussion_r904097147

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-06-26 18:00:57 +01:00
Manuel 117baee1b7
Update src/nix/flake.cc
https://github.com/NixOS/nix/pull/6699#discussion_r904096906

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-06-26 18:00:34 +01:00
Manu [tennox] 3a85fd077c #6542 Apply flake templates partially on conflicts
Will still exit with non-zero exit code and clearly prompt which files to merge:
 ```
nixx flake init -t github:numtide/devshell
wrote: /home/manu/dev/stuff/gopassbridge/.envrc
refusing to overwrite existing file '/home/manu/dev/stuff/gopassbridge/.gitignore'
-> merge manually with '/nix/store/ksmwhyghjwb4d9dw6hcpbvng1msdvjim-source/template/.gitignore'
wrote: /home/manu/dev/stuff/gopassbridge/devshell.toml
wrote: /home/manu/dev/stuff/gopassbridge/flake.nix
error: Encountered 1 conflicts - please merge manually
 ```
2022-06-21 17:37:02 +01:00
Naïm Favier d6d0e781bb
Complete flake inputs for all given flakes
Allow `nix build flake1 flake2 --update-input <Tab>` to complete the
inputs of both flakes.

Also do tilde expansion so that `nix build ~/flake --update-input <Tab>`
works.
2022-06-20 04:17:13 +02:00
Théophane Hufschmitt 1e7c27cb91
Merge pull request #6560 from ncfavier/patch-1
typo: defaultApps → defaultApp
2022-05-23 17:08:06 +02:00
Naïm Camille Favier 81a9bf0ad2
typo: defaultApps → defaultApp 2022-05-21 14:41:24 +02:00
Théophane Hufschmitt 452ffe5464
Hint at the source file on conflict in flake new
Add a pointer to the source file (from the template) when `nix flake new` (or `init`) encounters an already existing file

Fix #6542
2022-05-18 16:46:13 +02:00
Tom Bereknyei 8150b93968 fix: alignment during flake show of legacyPackages
Fixes:
https://github.com/NixOS/nix/issues/6240
https://github.com/NixOS/nix/issues/6045
2022-05-13 11:12:11 -04:00
Eelco Dolstra 4a79cba511 Allow selecting derivation outputs using 'installable!outputs'
E.g. 'nixpkgs#glibc^dev,static' or 'nixpkgs#glibc^*'.
2022-05-03 13:43:52 +02:00
Guillaume Maudoux e93b59fbc5 Merge remote-tracking branch 'origin/master' into coerce-string 2022-04-29 00:12:25 +02:00
Guillaume Maudoux acf990c9ea fix errors case and wording 2022-04-28 12:54:14 +02:00
Eelco Dolstra 1ddabe1a01 nix: Respect meta.outputsToInstall, and use all outputs by default
'nix profile install' will now install all outputs listed in the
package's meta.outputsToInstall attribute, or all outputs if that
attribute doesn't exist. This makes it behave consistently with
nix-env. Fixes #6385.

Furthermore, for consistency, all other 'nix' commands do this as
well. E.g. 'nix build' will build and symlink the outputs in
meta.outputsToInstall, defaulting to all outputs. Previously, it only
built/symlinked the first output. Note that this means that selecting
a specific output using attrpath selection (e.g. 'nix build
nixpkgs#libxml2.dev') no longer works. A subsequent PR will add a way
to specify the desired outputs explicitly.
2022-04-26 17:17:27 +02:00
Eelco Dolstra 474695975d EvalCache: Revert to using symbols in getAttr() 2022-04-26 14:01:42 +02:00
pennae a385e51a08 rename SymbolIdx -> Symbol, Symbol -> SymbolStr
after #6218 `Symbol` no longer confers a uniqueness invariant on the
string it wraps, it is now possible to create multiple symbols that
compare equal but whose string contents have different addresses. this
guarantee is now only provided by `SymbolIdx`, leaving `Symbol` only as
a string wrapper that knows about the intricacies of how symbols need to
be formatted for output.

this change renames `SymbolIdx` to `Symbol` to restore the previous
semantics of `Symbol` to that name. we also keep the wrapper type and
rename it to `SymbolStr` instead of returning plain strings from lookups
into the symbol table because symbols are formatted for output in many
places. theoretically we do not need `SymbolStr`, only a function that
formats a string for output as a symbol, but having to wrap every symbol
that appears in a message into eg `formatSymbol()` is error-prone and
inconvient.
2022-04-25 15:37:01 +02:00
pennae 8775be3393 store Symbols in a table as well, like positions
this slightly increases the amount of memory used for any given symbol, but this
increase is more than made up for if the symbol is referenced more than once in
the EvalState that holds it. on average every symbol should be referenced at
least twice (once to introduce a binding, once to use it), so we expect no
increase in memory on average.

symbol tables are limited to 2³² entries like position tables, and similar
arguments apply to why overflow is not likely: 2³² symbols would require as many
string instances (at 24 bytes each) and map entries (at 24 bytes or more each,
assuming that the map holds on average at most one item per bucket as the docs
say). a full symbol table would require at least 192GB of memory just for
symbols, which is well out of reach. (an ofborg eval of nixpks today creates
less than a million symbols!)
2022-04-21 21:56:31 +02:00
pennae 6526d1676b replace most Pos objects/ptrs with indexes into a position table
Pos objects are somewhat wasteful as they duplicate the origin file name and
input type for each object. on files that produce more than one Pos when parsed
this a sizeable waste of memory (one pointer per Pos). the same goes for
ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate
a position would need at least 8GB of input and 64GB of expression memory. it's
not likely that we'll hit that any time soon, so we can use a uint32_t index to
locate positions instead.
2022-04-21 21:46:06 +02:00
Eelco Dolstra c9e58aa5ff Require formatters to be packages
Because of 9b41239d8f, a formatter can
no longer be a package *or* an app. So let's require it to be a
package for now.
2022-04-19 20:48:13 +02:00
Eelco Dolstra 48a467f2b9 Merge branch 'issue-6075' of https://github.com/kamadorueda/nix 2022-04-19 20:21:32 +02:00
Eelco Dolstra d89840b103 Make InstallableFlake::toValue() and toDerivation() behave consistently
In particular, this means that 'nix eval` (which uses toValue()) no
longer auto-calls functions or functors (because
AttrCursor::findAlongAttrPath() doesn't).

Fixes #6152.

Also use ref<> in a few places, and don't return attrpaths from
getCursor() because cursors already have a getAttrPath() method.
2022-04-14 14:07:04 +02:00
Rehno Lindeque b9c969a866 nix flake check: Warn about deprecated nixosModule output 2022-04-06 12:20:39 -04:00