Commit graph

11957 commits

Author SHA1 Message Date
Eelco Dolstra 61289ceee3 Style fixes 2022-05-02 13:37:53 +02:00
Eelco Dolstra 564faa6b4e
Merge pull request #6470 from Ma27/git-followup
libfetchers/git: fix every occasion of a permission error
2022-05-02 13:33:08 +02:00
Eelco Dolstra a26be9f3b8
Merge pull request #4638 from orbekk/read_head
Resolve reference for remote git repositories (makes fetchGit work with non-'master' branch)
2022-05-02 13:31:30 +02:00
Eelco Dolstra dde71899dd tests: Don't create tests/result
https://hydra.nixos.org/log/lns780srkka4dv7r69mn4zfy6fdij4yr-nix-2.9.0pre20220428_4bb111c.drv
2022-05-02 09:38:51 +02:00
Maximilian Bosch 1849e6a1f6
libfetchers/git: fix every occasion of a permission error
I'm afraid I missed a few problematic `git(1)`-calls while implementing
PR #6440, sorry for that! Upon investigating what went wrong, I realized
that I only tested against the "cached"-case by accident because my
git-checkout with my system's flake was apparently cached during my
debugging.

I managed to trigger the original issue again by running:

    $ git commit --allow-empty -m "tmp"
    $ sudo nixos-rebuild switch --flake .# -L --builders ''

Since `repoDir` points to the checkout that's potentially owned by
another user, I decided to add `--git-dir` to each call affecting
`repoDir`.

Since the `tmpDir` for the temporary submodule-checkout is created by
Nix itself, it doesn't seem to be an issue.

Sorry for that, it should be fine now.
2022-04-30 15:56:12 +02:00
Kjetil Orbekk 9bf296c970 Extract git reference parsing to a shared library
These utility functions can be shared between the git and github fetchers.
2022-04-29 18:46:21 -04:00
Kjetil Orbekk c21afd684c Update nix flake documentation of ref handling
Update the documentation about how `ref` is resolved if it is not
specified.

Add a note about special handling of local workdirs with `git+file`.
2022-04-29 18:46:21 -04:00
Kjetil Orbekk 1203e48926 Store cached head in cached git repo
The previous head caching implementation stored two paths in the local
cache; one for the cached git repo and another textfile containing the
resolved HEAD ref. This commit instead stores the resolved HEAD by
setting the HEAD ref in the local cache appropriately.
2022-04-29 18:46:21 -04:00
Kjetil Orbekk de54e1cd3f Refactor fetching of dirty workdir
Extract the handling of a local dirty workdir to a helper function.
2022-04-29 18:46:17 -04:00
Kjetil Orbekk 05a3fbac5a Test fetchGit with non-'master' remote repo 2022-04-29 18:42:31 -04:00
Kjetil Orbekk 401e60f289 Resolve reference for remote repository
Resolves the HEAD reference from the remote repository instead
of assuming "master".
2022-04-29 18:42:28 -04:00
Eelco Dolstra 4bb111c8d4
Merge pull request #6461 from edolstra/libcxx
Fix libcxx build
2022-04-28 15:11:49 +02:00
Eelco Dolstra 70a30dbc00 Fix libcxx build
Fixes #6458.
2022-04-28 14:37:05 +02:00
Eelco Dolstra cb5361628d Add libcxxStdenv devshell 2022-04-28 14:36:48 +02:00
Eelco Dolstra d77d813017 Shut up clang warning 2022-04-28 14:24:17 +02:00
Eelco Dolstra 660835da69
Merge pull request #6459 from edolstra/fix-out-paths
Fix passing $OUT_PATHS to the post-build hook
2022-04-28 14:05:29 +02:00
Eelco Dolstra 4a9623b129 Fix passing $OUT_PATHS to the post-build hook
Fixes #6446.
2022-04-28 13:36:01 +02:00
Eelco Dolstra 35393dc2c6
Merge pull request #6444 from midchildan/feat/zsh-runhelp
feat: add integration with zsh's run-help
2022-04-26 21:26:33 +02:00
midchildan 68d51ae012
refactor: don't hardcode nix subcommands in run-help-nix 2022-04-27 01:56:10 +09:00
Eelco Dolstra a81622c21d
Merge pull request #6447 from edolstra/eval-cache-symbols
EvalCache: Use Symbol in more places
2022-04-26 15:41:40 +02:00
Théophane Hufschmitt fb5f13fb65
Merge pull request #6440 from Ma27/fix-nix-with-latest-git
libfetchers/git: hardcode `--git-dir`
2022-04-26 14:50:47 +02:00
Eelco Dolstra b12c33510c EvalCache AttrKey: Use Symbol instead of std::string 2022-04-26 14:16:20 +02:00
Eelco Dolstra 474695975d EvalCache: Revert to using symbols in getAttr() 2022-04-26 14:01:42 +02:00
Eelco Dolstra fab731a9d4 Don't pass Symbol by reference
Since Symbol is just an integer, passing it by const reference is
never advantageous.
2022-04-26 13:25:17 +02:00
Eelco Dolstra 29e52194a8 Merge branch 'symbolidx-symbol' of https://github.com/pennae/nix 2022-04-26 13:25:10 +02:00
pennae a385e51a08 rename SymbolIdx -> Symbol, Symbol -> SymbolStr
after #6218 `Symbol` no longer confers a uniqueness invariant on the
string it wraps, it is now possible to create multiple symbols that
compare equal but whose string contents have different addresses. this
guarantee is now only provided by `SymbolIdx`, leaving `Symbol` only as
a string wrapper that knows about the intricacies of how symbols need to
be formatted for output.

this change renames `SymbolIdx` to `Symbol` to restore the previous
semantics of `Symbol` to that name. we also keep the wrapper type and
rename it to `SymbolStr` instead of returning plain strings from lookups
into the symbol table because symbols are formatted for output in many
places. theoretically we do not need `SymbolStr`, only a function that
formats a string for output as a symbol, but having to wrap every symbol
that appears in a message into eg `formatSymbol()` is error-prone and
inconvient.
2022-04-25 15:37:01 +02:00
midchildan 0f7c7ab97b
fix: typo make-content-addressable -> make-content-addressed 2022-04-25 02:34:59 +09:00
midchildan c67d8876c3
feat: add integration with zsh's run-help 2022-04-25 02:21:41 +09:00
Maximilian Bosch d1f5356311
libfetchers/git: fix for nixos-rebuild
The `--git-dir=` must be `.` in some cases (for cached repos that are
"bare" repos in `~/.cache/nix/gitv3`). With this fix we can add
`--git-dir` to each `git`-invokation needed for `nixos-rebuild`.
2022-04-24 18:14:24 +02:00
Maximilian Bosch 0256e5578e
libfetchers/git: hardcode --git-dir
To demonstrate the problem:

* You need a `git` at 2.33.3 in your $PATH
* An expression like this in a git repository:

  ``` nix
  {
    outputs = { self, nixpkgs }: {
      packages.foo.x86_64-linux = with nixpkgs.legacyPackages.x86_64-linux;
        runCommand "snens" { } ''
          echo ${(builtins.fetchGit ./.).lastModifiedDate} > $out
        '';
    };
  }
  ```

Now, when instantiating the package via `builtins.getFlake`, it fails on
Nix 2.7 like this:

    $ nix-instantiate -E '(builtins.getFlake "'"$(pwd)"'").packages.foo.x86_64-linux'
    fatal: unsafe repository ('/nix/store/a7j3125km4h8l0p71q6ssfkxamfh5d61-source' is owned by someone else)
    To add an exception for this directory, call:

    	git config --global --add safe.directory /nix/store/a7j3125km4h8l0p71q6ssfkxamfh5d61-source
    error: program 'git' failed with exit code 128
    (use '--show-trace' to show detailed location information)

This breaks e.g. `nixops`-deployments using flakes with similar
expressions as shown above.

The cause for this is that `git(1)` tries to find the highest
`.git`-directory in the directory tree and if it finds a such a
directory, but with another owning user (root vs. the user who evaluates
the expression), it fails as above. This was changed recently to fix
CVE-2022-24765[1].

By explicitly specifying `--git-dir`, Git assumes to be in the top-level
directory and doesn't attempt to look for a `.git`-directory in the
parent directories and thus the code-path leading to said error is never
reached.

[1] https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/
2022-04-23 23:20:17 +02:00
Théophane Hufschmitt 7f814d6d9a
Merge pull request #6259 from Artturin/nixbuildaddprintstorepaths
nix build: add --print-out-paths flag
2022-04-22 14:21:55 +02:00
Théophane Hufschmitt 197aa2fa96 Merge remote-tracking branch 'origin/master' into nixbuildaddprintstorepaths 2022-04-22 13:20:39 +02:00
Théophane Hufschmitt 92de62fe6f
Merge pull request #6437 from NixOS/fix-darwin-build
Fix the darwin build
2022-04-22 13:19:53 +02:00
Théophane Hufschmitt be28603dca Merge remote-tracking branch 'origin/master' into nixbuildaddprintstorepaths 2022-04-22 11:11:01 +02:00
Théophane Hufschmitt 7b889f31ea Fix the darwin build
Looks like the auto-merge is indeed quite broken and merges even when the CI fails
2022-04-22 10:56:56 +02:00
Théophane Hufschmitt c4ffc8e2f8
Merge pull request #6218 from pennae/pos-symbol-tables
reduce the size of Attr from 3 pointers to 2 on 64 bit machines
2022-04-22 10:28:06 +02:00
Théophane Hufschmitt 484badfa09 Add some tests for ChunkedVector 2022-04-22 10:03:44 +02:00
Théophane Hufschmitt 7ca6fbc8ca Move ChunkedVector to its own header 2022-04-22 10:01:02 +02:00
Théophane Hufschmitt 35ca5fdf91
Merge pull request #6436 from flox/tofile_allow
fix: builtins.toFile adds path to allowedPaths
2022-04-22 08:50:54 +02:00
Tom Bereknyei f25112d383 fix: builtins.toFile adds path to allowedPaths
The produced path is then allowed be imported or utilized elsewhere:
```
assert (43 == import (builtins.toFile "source" "43")); "good"
```

This will still fail on write-only stores.
2022-04-21 16:41:37 -04:00
pennae 8adaa6acb5 remove pos<T>
it's no longer needed now that positions aren't really pointers any
more.
2022-04-21 21:56:34 +02:00
pennae 8168a4cf4a shrink Attr by 8 bytes on 64bit machines
with position and symbol tables in place we can now shrink Attr by a full
pointer with some simple field reordering. since Attr is a very hot struct this
has substantial impact on memory use, decreasing GC allocations and heap size by
10-15% each. we also get a ~15% performance improvement due to reduced GC
loading.

pure parsing has taken a hit over the branch base because positions are now
slightly more expensive to create, but overall we get a noticeable improvement.

before (on memory-friendliness):

  Benchmark 1: nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.960 s ±  0.028 s    [User: 5.832 s, System: 0.897 s]
    Range (min … max):    6.886 s …  7.005 s    20 runs

  Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     328.1 ms ±   1.7 ms    [User: 295.8 ms, System: 32.2 ms]
    Range (min … max):   324.9 ms … 331.2 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.688 s ±  0.029 s    [User: 2.365 s, System: 0.238 s]
    Range (min … max):    2.642 s …  2.742 s    20 runs

after:

  Benchmark 1: nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.902 s ±  0.039 s    [User: 5.844 s, System: 0.783 s]
    Range (min … max):    6.820 s …  6.956 s    20 runs

  Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     330.7 ms ±   2.2 ms    [User: 300.6 ms, System: 30.0 ms]
    Range (min … max):   327.5 ms … 334.5 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.330 s ±  0.027 s    [User: 2.040 s, System: 0.234 s]
    Range (min … max):    2.272 s …  2.383 s    20 runs
2022-04-21 21:56:34 +02:00
pennae 8775be3393 store Symbols in a table as well, like positions
this slightly increases the amount of memory used for any given symbol, but this
increase is more than made up for if the symbol is referenced more than once in
the EvalState that holds it. on average every symbol should be referenced at
least twice (once to introduce a binding, once to use it), so we expect no
increase in memory on average.

symbol tables are limited to 2³² entries like position tables, and similar
arguments apply to why overflow is not likely: 2³² symbols would require as many
string instances (at 24 bytes each) and map entries (at 24 bytes or more each,
assuming that the map holds on average at most one item per bucket as the docs
say). a full symbol table would require at least 192GB of memory just for
symbols, which is well out of reach. (an ofborg eval of nixpks today creates
less than a million symbols!)
2022-04-21 21:56:31 +02:00
pennae 00a3280232 don't use Symbol in Pos to represent a path
PosTable deduplicates origin information, so using symbols for paths is no
longer necessary. moving away from path Symbols also reduces the usage of
symbols for things that are not keys in attribute sets, which will become
important in the future when we turn symbols into indices as well.
2022-04-21 21:46:10 +02:00
pennae 6526d1676b replace most Pos objects/ptrs with indexes into a position table
Pos objects are somewhat wasteful as they duplicate the origin file name and
input type for each object. on files that produce more than one Pos when parsed
this a sizeable waste of memory (one pointer per Pos). the same goes for
ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate
a position would need at least 8GB of input and 64GB of expression memory. it's
not likely that we'll hit that any time soon, so we can use a uint32_t index to
locate positions instead.
2022-04-21 21:46:06 +02:00
pennae 34b72775cf make throw*Error member functions of EvalState
when we introduce position and symbol tables we'll need to do lookups to turn
indices into those tables into actual positions/symbols. having the error
functions as members of EvalState will avoid a lot of churn for adding lookups
into the tables for each caller.
2022-04-21 21:25:18 +02:00
pennae 39df15fb8e don't use full Pos for findPackageFilename/editorFor
only file and line of the returned position were ever used, it wasn't actually
used a position. as such we may as well use a path+int pair for only those two
values and remove a use of Pos that would not work well with a position table.
2022-04-21 21:25:18 +02:00
pennae 38de79fcf7 remove Bindings::need
a future commit will remove the ability to convert the symbol type used in
bindings to strings. since we only have two users we can inline the error check.
2022-04-21 21:25:18 +02:00
pennae ff0fd91ed2 remove Symbol::empty
the only use of this function is to determine whether a lambda has a non-set
formal, but this use is arguably better served by Symbol::set and using a
non-Symbol instead of an empty symbol in the parser when no such formal is present.
2022-04-21 21:25:18 +02:00
pennae 90b5c0a1a6 turn primop names into strings
we don't *need* symbols here. the only advantage they have over strings is
making call-counting slightly faster, but that's a diagnostic feature and thus
needn't be optimized.

this also fixes a move bug that previously didn't show up: PrimOp structs were
accessed after being moved from, which technically invalidates them. previously
the names remained valid because Symbol copies on move, but strings are
invalidated. we now copy the entire primop struct instead of moving since primop
registration happen once and are not performance-sensitive.
2022-04-21 21:25:17 +02:00