Commit graph

181 commits

Author SHA1 Message Date
eldritch horrors 512c1f05c3 Unify and refactor value printing
Previously, there were two mostly-identical value printers -- one in
`libexpr/eval.cc` (which didn't force values) and one in
`libcmd/repl.cc` (which did force values and also printed ANSI color
codes).

This PR unifies both of these printers into `print.cc` and provides a
`PrintOptions` struct for controlling the output, which allows for
toggling whether values are forced, whether repeated values are tracked,
and whether ANSI color codes are displayed.

Additionally, `PrintOptions` allows tuning the maximum number of
attributes, list items, and bytes in a string that will be displayed;
this makes it ideal for contexts where printing too much output (e.g.
all of Nixpkgs) is distracting. (As requested by @roberth in
https://github.com/NixOS/nix/pull/9554#issuecomment-1845095735)

Please read the tests for example output.

Future work:
- It would be nice to provide this function as a builtin, perhaps
  `builtins.toStringDebug` -- a printing function that never fails would
  be useful when debugging Nix code.
- It would be nice to support customizing `PrintOptions` members on the
  command line, e.g. `--option to-string-max-attrs 1000`.

(cherry picked from commit 0fa08b451682fb3311fe58112ff05c4fe5bee3a4, )

===

Restore ambiguous value printer for `nix-instantiate`

The Nix team has requested that this output format remain unchanged.
I've added a warning to the man page explaining that `nix-instantiate
--eval` output will not parse correctly in many situations.

(cherry picked from commit df84dd4d8dd3fd6381ac2ca3064432ab31a16b79)

Change-Id: I7cca6b4b53cd0642f2d49af657d5676a8554c9f8
2024-03-09 03:50:06 +01:00
eldritch horrors dd180911d8 Merge pull request #9582 from pennae/misc-opts
a packet of small optimizations

(cherry picked from commit ee439734e924eb337a869ff2e48aff8b989198bc)
Change-Id: I125d870710750a32a0dece48f39a3e9132b0d023
2024-03-04 07:32:31 +01:00
eldritch horrors 6feba52008 Merge pull request #8895 from hercules-ci/gc-before-stats
eval: Run a full GC before printing stats

(cherry picked from commit aeea49609be014b1928c95b7ec28dbedeb4f032a)
Change-Id: I47a23d3a7a47ea61d9a2b5727b638f879f3aaf1e
2024-03-04 04:36:47 +01:00
Eelco Dolstra 17e6b85d05 nix: Support the --repair flag 2023-04-28 17:03:04 +02:00
Eelco Dolstra 01232358ff Merge remote-tracking branch 'origin/master' into source-path 2023-04-24 13:20:36 +02:00
John Ericson 85f0cdc370 Use std::set<StringContextElem> not PathSet for string contexts
Motivation

`PathSet` is not correct because string contexts have other forms
(`Built` and `DrvDeep`) that are not rendered as plain store paths.
Instead of wrongly using `PathSet`, or "stringly typed" using
`StringSet`, use `std::std<StringContextElem>`.

-----

In support of this change, `NixStringContext` is now defined as
`std::std<StringContextElem>` not `std:vector<StringContextElem>`. The
old definition was just used by a `getContext` method which was only
used by the eval cache. It can be deleted altogether since the types are
now unified and the preexisting `copyContext` function already suffices.

Summarizing the previous paragraph:

Old:

  - `value/context.hh`: `NixStringContext = std::vector<StringContextElem>`
  - `value.hh`: `NixStringContext Value::getContext(...)`
  - `value.hh`: `copyContext(...)`

New:

  - `value/context.hh`: `NixStringContext = std::set<StringContextElem>`
  - `value.hh`: `copyContext(...)`
----

The string representation of string context elements no longer contains
the store dir. The diff of `src/libexpr/tests/value/context.cc` should
make clear what the new representation is, so we recommend reviewing
that file first. This was done for two reasons:

Less API churn:

`Value::mkString` and friends did not take a `Store` before. But if
`NixStringContextElem::{parse, to_string}` *do* take a store (as they
did before), then we cannot have the `Value` functions use them (in
order to work with the fully-structured `NixStringContext`) without
adding that argument.

That would have been a lot of churn of threading the store, and this
diff is already large enough, so the easier and less invasive thing to
do was simply make the element `parse` and `to_string` functions not
take the `Store` reference, and the easiest way to do that was to simply
drop the store dir.

Space usage:

Dropping the `/nix/store/` (or similar) from the internal representation
will safe space in the heap of the Nix programming being interpreted. If
the heap contains many strings with non-trivial contexts, the saving
could add up to something significant.

----

The eval cache version is bumped.

The eval cache serialization uses `NixStringContextElem::{parse,
to_string}`, and since those functions are changed per the above, that
means the on-disk representation is also changed.

This is simply done by changing the name of the used for the eval cache
from `eval-cache-v4` to eval-cache-v5`.

----

To avoid some duplication `EvalCache::mkPathString` is added to abstract
over the simple case of turning a store path to a string with just that
string in the context.

Context

This PR picks up where #7543 left off. That one introduced the fully
structured `NixStringContextElem` data type, but kept `PathSet context`
as an awkward middle ground between internal `char[][]` interpreter heap
string contexts and `NixStringContext` fully parsed string contexts.

The infelicity of `PathSet context` was specifically called out during
Nix team group review, but it was agreeing that fixing it could be left
as future work. This is that future work.

A possible follow-up step would be to get rid of the `char[][]`
evaluator heap representation, too, but it is not yet clear how to do
that. To use `NixStringContextElem` there we would need to get the STL
containers to GC pointers in the GC build, and I am not sure how to do
that.

----

PR #7543 effectively is writing the inverse of a `mkPathString`,
`mkOutputString`, and one more such function for the `DrvDeep` case. I
would like that PR to have property tests ensuring it is actually the
inverse as expected.

This PR sets things up nicely so that reworking that PR to be in that
more elegant and better tested way is possible.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-04-21 01:05:49 -04:00
Eelco Dolstra 94812cca98 Backport SourcePath from the lazy-trees branch
This introduces the SourcePath type from lazy-trees as an abstraction
for accessing files from inputs that may not be materialized in the
real filesystem (e.g. Git repositories). Currently, however, it's just
a wrapper around CanonPath, so it shouldn't change any behaviour. (On
lazy-trees, SourcePath is a <InputAccessor, CanonPath> tuple.)
2023-04-06 13:15:50 +02:00
Naïm Favier ff0b5a778c
Revert to copyStore = true for nix-instantiate and nix-env 2022-08-23 14:40:27 +02:00
Naïm Favier 4c2ff4a0f4
JSON: add missing newlines
after `nix eval --json` and `nix-instantiate --eval --json`.
2022-08-22 15:07:52 +02:00
Naïm Favier 062e4fcdde
JSON: print paths as strings without copying them to the store
Makes `printValueAsJSON` not copy paths to the store for `nix eval
--json`, `nix-instantiate --eval --json` and `nix-env --json`.

Fixes https://github.com/NixOS/nix/issues/5612
2022-08-22 15:01:35 +02:00
pennae 8775be3393 store Symbols in a table as well, like positions
this slightly increases the amount of memory used for any given symbol, but this
increase is more than made up for if the symbol is referenced more than once in
the EvalState that holds it. on average every symbol should be referenced at
least twice (once to introduce a binding, once to use it), so we expect no
increase in memory on average.

symbol tables are limited to 2³² entries like position tables, and similar
arguments apply to why overflow is not likely: 2³² symbols would require as many
string instances (at 24 bytes each) and map entries (at 24 bytes or more each,
assuming that the map holds on average at most one item per bucket as the docs
say). a full symbol table would require at least 192GB of memory just for
symbols, which is well out of reach. (an ofborg eval of nixpks today creates
less than a million symbols!)
2022-04-21 21:56:31 +02:00
Eelco Dolstra b55d79728c Add EvalState::coerceToStorePath() helper
This is useful whenever we want to evaluate something to a store path
(e.g. in get-drvs.cc).

Extracted from the lazy-trees branch (where we can require that a
store path must come from a store source tree accessor).
2022-03-02 23:58:58 +01:00
Eelco Dolstra df552ff53e Remove std::string alias (for real this time)
Also use std::string_view in a few more places.
2022-02-25 16:13:02 +01:00
Eelco Dolstra bd383d1b6f Make most calls to determinePos() lazy 2022-02-04 00:33:21 +01:00
Kevin Amado 49b0bb0206
forceValue: make pos mandatory
- Make passing the position to `forceValue` mandatory,
  this way we remember people that the position is
  important for better error messages
- Add pos to all `forceValue` calls
2022-01-21 16:32:43 -05:00
Domen Kožar 164179983e
Merge pull request #5428 from kreisys/add-pos-to-json-type-error
toJSON: report error position for fancier output
2021-11-15 07:57:46 -06:00
Kevin Amado d0e9e18489
toXML: display errors position
- This change applies to builtins.toXML and inner workings
- Proof of concept:
  ```nix
  let e = builtins.toXML e; in e
  ```
- Before:
  ```
  $ nix-instantiate --eval poc.nix
  error: infinite recursion encountered
  ```
- After:
  ```
  $ nix-instantiate --eval poc.nix
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/poc.nix:1:9:

            1| let e = builtins.toXML e; in e
             |
  ```
2021-11-13 20:33:34 -05:00
Shay Bergmann 465a167c43
nix-instantiate: pass pos in the --eval --json code path 2021-10-27 19:01:32 +00:00
Shay Bergmann ba81e871b2
toJSON: report error position for fancier output
Given flake:

```nix
{ description = "nix json error provenance";
  inputs = {};
  outputs = { self }: {
    jsonFunction = _: "function";
    json = builtins.toJSON (_: "function");
  };
}

```
- Before:

```console
❯ nix eval --json .#jsonFunction
error: cannot convert a function to JSON
```

- After:

```console
❯ nix eval --json .#jsonFunction
error: cannot convert a function to JSON

       at /nix/store/b7imf1c2j4jnkg3ys7fsfbj02s5j0i4f-source/testflake/flake.nix:4:5:

            3|   outputs = { self }: {
            4|     jsonFunction = _: "function";
             |     ^
            5|     json = builtins.toJSON (_: "function");
```
2021-10-25 21:17:52 +00:00
Eelco Dolstra 47002108d1 nix-instantiate: Fix --eval-store 2021-07-27 11:16:47 +02:00
Eelco Dolstra 2ff3035cf4 Support --eval-store in nix-instantiate and nix-build 2021-07-22 09:59:51 +02:00
Shea Levy 98d1b64400
Initialize plugins after handling initial command line flags
This is technically a breaking change, since attempting to set plugin
files after the first non-flag argument will now throw an error. This
is acceptable given the relative lack of stability in a plugin
interface and the need to tie the knot somewhere once plugins can
actually define new subcommands.
2021-02-24 08:22:17 -05:00
Shea Levy 6af6e41df0
Move command plugin interface to libnixcmd 2021-01-26 06:22:24 -05:00
John Ericson 6cc1541782 Split out local-fs-store.hh
This matches the already-existing `local-fs-store.cc`.
2020-10-09 20:18:08 +00:00
Eelco Dolstra 85c8be6286 Remove static variable name clashes
This was useful for an experiment with building Nix as a single
compilation unit. It's not very useful otherwise but also doesn't
hurt...
2020-10-06 13:49:20 +02:00
Eelco Dolstra 82b77a7726
addPermRoot(): Remove indirect flag 2020-09-03 11:29:34 +02:00
Eelco Dolstra 00d25e8457
Remove the --indirect flag
All GC roots are now indirect.
2020-09-03 11:22:00 +02:00
Ben Burdette e4fb9a3849 remove 'format' from Error constructor calls 2020-04-21 17:07:07 -06:00
Eelco Dolstra e322a16523 Remove global -I flags
(cherry picked from commit 2c692a3b14)
2020-03-30 15:30:19 +02:00
Eelco Dolstra c1ca4f0acc findAlongAttrPath(): Return position
(cherry picked from commit 0b013a54dc)
2020-03-24 14:06:47 +01:00
Eelco Dolstra bbe97dff8b Make the Store API more type-safe
Most functions now take a StorePath argument rather than a Path (which
is just an alias for std::string). The StorePath constructor ensures
that the path is syntactically correct (i.e. it looks like
<store-dir>/<base32-hash>-<name>). Similarly, functions like
buildPaths() now take a StorePathWithOutputs, rather than abusing Path
by adding a '!<outputs>' suffix.

Note that the StorePath type is implemented in Rust. This involves
some hackery to allow Rust values to be used directly in C++, via a
helper type whose destructor calls the Rust type's drop()
function. The main issue is the dynamic nature of C++ move semantics:
after we have moved a Rust value, we should not call the drop function
on the original value. So when we move a value, we set the original
value to bitwise zero, and the destructor only calls drop() if the
value is not bitwise zero. This should be sufficient for most types.

Also lots of minor cleanups to the C++ API to make it more modern
(e.g. using std::optional and std::string_view in some places).
2019-12-10 22:06:05 +01:00
Eelco Dolstra f6a3dfe4e0
Merge all nix-* binaries into nix
These are all symlinks to 'nix' now, reducing the installed size by
about ~1.7 MiB.
2018-10-26 12:54:00 +02:00
Eelco Dolstra 0629601da1
Move EvalState from the stack to the heap
EvalState contains a few counters (e.g. nrValues) that increase
quickly enough that they end up being interpreted as pointers by the
garbage collector. Moving it to the heap makes them invisible to the
garbage collector.

This reduces the max RSS doing 100 evaluations of
nixos.tests.firefox.x86_64-linux.drvPath from 455 MiB to 292 MiB.

Note: ideally, allocations would be much further up in the 64-bit
address space to reduce the odds of an integer being misinterpreted as
a pointer. Maybe we can use some linker magic to move the .bss segment
to a higher address.
2018-06-12 17:49:55 +02:00
Linus Heckemann 12913ccf45 nix-shell: allow symlinks to .drvs
This makes persistent shell environments easier to use.
2018-03-16 22:15:24 +00:00
Shea Levy 88cd2d41ac
Add plugins to make Nix more extensible.
All plugins in plugin-files will be dlopened, allowing them to
statically construct instances of the various Register* types Nix
supports.
2018-02-08 12:44:37 -05:00
Eelco Dolstra d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Eelco Dolstra 0d59f1ca49
nix: Respect -I, --arg, --argstr
Also, random cleanup to argument handling.
2017-10-24 12:58:34 +02:00
Jörg Thalheim 2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
Eelco Dolstra 4c9ff89c26
nix-build/nix-shell: Eliminate call to nix-instantiate / nix-store
Note that this removes the need for a derivation symlink, so the
--drv-path and --add-drv-link flags now do nothing.
2017-07-26 17:29:09 +02:00
Eelco Dolstra fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00
Eelco Dolstra b01d62285c
Improve progress indicator 2017-05-16 16:09:57 +02:00
Eelco Dolstra 215b70f51e
Revert "Get rid of unicode quotes (#1140)"
This reverts commit f78126bfd6. There
really is no need for such a massive change...
2016-11-26 00:38:01 +01:00
Guillaume Maudoux f78126bfd6 Get rid of unicode quotes (#1140) 2016-11-25 15:48:27 +01:00
Eelco Dolstra 812c0dfbe2 Allow setting the state directory as a store parameter
E.g. "local?store=/tmp/store&state=/tmp/var".
2016-06-02 16:02:48 +02:00
Eelco Dolstra 41633f9f73 Improved logging abstraction
This also gets rid of --log-type, since the nested log type isn't
useful in a multi-threaded situation, and nobody cares about the
"pretty" log type.
2016-04-25 19:18:45 +02:00
Eelco Dolstra ae4a3cfa03 Move addPermRoot into Store 2016-02-11 16:14:42 +01:00
Eelco Dolstra fa7cd5369b StoreAPI -> Store
Calling a class an API is a bit redundant...
2016-02-04 14:48:42 +01:00
Eelco Dolstra c10c61449f Eliminate the "store" global variable
Also, move a few free-standing functions into StoreAPI and Derivation.

Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
2016-02-04 14:28:26 +01:00
Eelco Dolstra 5c28943e8f int2String() -> std::to_string() 2015-10-29 13:26:55 +01:00
Eelco Dolstra 6bd2c7bb38 OCD: foreach -> C++11 ranged for 2015-07-17 20:13:56 +02:00