Commit graph

7231 commits

Author SHA1 Message Date
regnat 2105084645 Enfore the use of properly built paths in libcmd
Replace `DerivedPathWithHints` by a new `BuiltPath` type that serves as
a proof that the corresponding path has been built.
2021-05-17 08:45:08 +02:00
Emily 559a504da7 sandbox: allow SystemVersionCompat.plist on Darwin
For whatever reason, many programs trying to access SystemVersion.plist
also open SystemVersionCompat.plist; this includes Python code and
coreutils’ `cat(1)` (but not the native macOS `/bin/cat`). Illustratory
`dtruss(1m)` output:

    open("/System/Library/CoreServices/SystemVersion.plist\0", 0x0, 0x0)		 = 3 0
    open("/System/Library/CoreServices/SystemVersionCompat.plist\0", 0x0, 0x0)		 = 4 0

I assume this is a Big Sur change relating to the 10.16.x/11.x
version compatibility divide and that it’s something along the lines of
a hook inside libSystem.

Fixes a lot of sandboxed package builds under Big Sur.
2021-05-17 01:24:31 +01:00
Eelco Dolstra 340f831ebe
Merge pull request #4798 from matthewbauer/relock-wait-for-build-slot-goals
Relock wait for build slot goals
2021-05-13 13:07:35 +02:00
Eelco Dolstra 89a4ede92b
Merge pull request #4801 from matthewbauer/fix-tokenize-output-names
Fix tokenize output names in DerivedPath
2021-05-13 13:06:29 +02:00
Matthew Bauer 8c7e043de2 Fix tokenize output names in drv
This should fix the issue described in
https://discourse.nixos.org/t/derivation-does-not-have-wanted-outputs-dev-out/12905.

Specifically, we get an error of

  error: derivation '/nix/store/_.drv' does not have wanted outputs 'dev,out'

when a path like /nix/store/_.drv!dev,out is sent to the daemon.
2021-05-12 21:40:28 -05:00
Matthew Bauer 9a14335845 Relock wait for build slot goals
When we don’t have enough free job slots to run a goal, we put it in
the waitForBuildSlot list & unlock its output locks. This will
continue from where we left off (tryLocalBuild). However, we need the
locks to get reacquired when/if the goal ever restarts. So, we need to
send it back through tryToBuild to get reqacquire those locks.

I think this bug was introduced in
https://github.com/NixOS/nix/pull/4570. It leads to some builds
starting without proper locks.
2021-05-12 12:15:32 -05:00
regnat ec613603ba DerivedPathWithHints -> BuiltPath
Just a renaming for now
2021-05-12 16:19:51 +02:00
Lorenzo Manacorda 4029f4b05b doc: mention Priority for substituters 2021-05-12 13:27:05 +02:00
Eelco Dolstra 7f9759b18d
Merge pull request #4781 from NixOS/locally_cache_the_remote_realisations
Add a realisations disk cache
2021-05-10 20:37:57 +02:00
Eelco Dolstra 8768398d5f
Merge pull request #4787 from Ma27/builtins-ceil-floor
Implement `builtins.floor` and `builtins.ceil` using the C library functions internally
2021-05-10 20:37:27 +02:00
regnat d5d19582ef Simplify the realisations disk cache 2021-05-10 17:47:14 +02:00
regnat ab96c1ee50 Remove useless parents
I never remember the exact syntax of the `switch` statement
2021-05-10 17:36:49 +02:00
Maximilian Bosch bec33858bb
primops: math.h -> cmath
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-05-10 16:41:10 +02:00
Eelco Dolstra b68a85df0a
Merge pull request #4776 from NixOS/fix-ca-normalization
Properly normalize the content-addressed paths
2021-05-10 16:11:38 +02:00
Maximilian Bosch 7f7f99f350
Implement builtins.floor and builtins.ceil using the C library functions internally
Closes #4782

Note: even though the type is internally called `NixFloat`, it's
actually a `double`.
2021-05-10 12:19:32 +02:00
Eelco Dolstra 2bb49fb6d0
Merge pull request #4786 from matthewbauer/mkdir-local-share-nix
Create parent for trusted list path before writing
2021-05-10 11:14:45 +02:00
Matthew Bauer 2f63cc02de Create parent trusted list path before writing
This makes sure that $HOME/.local/share/nix exists before we try to
write to it.
2021-05-08 21:31:28 -05:00
regnat b66234134f Add a realisations disk cache
Similar to the nar-info disk cache (and using the same db).
This makes rebuilds muuch faster.

- This works regardless of the ca-derivations experimental feature.
  I could modify the logic to not touch the db if the flag isn’t there,
  but given that this is a trash-able local cache, it doesn’t seem to be
  really worth it.
- We could unify the `NARs` and `Realisation` tables to only have one
  generic kv table. This is left as an exercise to the reader.
- I didn’t update the cache db version number as the new schema just
  adds a new table to the previous one, so the db will be transparently
  migrated and is backwards-compatible.

Fix #4746
2021-05-06 17:38:20 +02:00
Matthew Bauer 1e3a9033b7 Use derivation output name from toDerivation
This was previously done in https://github.com/NixOS/nix/pull/4515 but
got clobbered away in https://github.com/NixOS/nix/pull/4594.

--------------------------------------------------------------------------------

This fixes an issue where derivations with a primary output that is
not "out" would fail with:

$ nix profile install nixpkgs#sqlite
error: opening directory '/nix/store/2a2ydlgyydly5czcc8lg12n6qqkfz863-sqlite-3.34.1-bin': No such file or directory

This happens because while derivations produce every output when
built, you might not have them if you didn't build the derivation
yourself (for instance, the store path was fetch from a binary cache).
This uses outputName provided from DerivationInfo which appears to
match the first output of the derivation.
2021-05-05 16:33:05 -05:00
regnat bf485dcf46 Properly normalize the content-addressed paths
Make sure that their timestamp are always normalized.
Otherwise, strange − and non-deterministic − things might happen, like
https://github.com/NixOS/nixpkgs/issues/121813

Fix #4775
2021-05-05 21:00:08 +02:00
regnat dadfbce318 Fix the double-slash in the realisations path
Make sure that we always access the realisations under
`binaryCacheUrl/realisations` and not `binaryCacheUrl//realisations`

Fix #4766
2021-05-04 10:35:34 +02:00
Eelco Dolstra ef13c9c223
Merge pull request #4750 from NinjaTrappeur/nin-build-dry-run-json
nix build: make dry-run to print a json output if --json is enabled
2021-05-03 16:14:29 +02:00
Eelco Dolstra 40378fbcba
Merge pull request #4761 from emilazy/issue-4658-mark-impure-host-deps-as-optional
Mark `__impureHostDeps` paths as optional
2021-05-03 16:13:36 +02:00
Maximilian Bosch e5951a6b2f
Bump version number for DerivedPath changes
I guess I misunderstood John's initial explanation about why wildcards
for outputs are sent to older stores[1]. My `nix-daemon` from 2021-03-26
also has version 1.29, but misses the wildcard[2]. So bumping seems to
be the right call.

[1] https://github.com/NixOS/nix/pull/4759#issuecomment-830812464
[2] 255d145ba7
2021-05-03 01:12:23 +02:00
Emily c4355a52fa Mark __impureHostDeps paths as optional
Starting in macOS 11, the on-disk dylib bundles are no longer available,
but nixpkgs needs to be able to keep compatibility with older versions
that require `/usr/lib/libSystem.B.dylib` in `__impureHostDeps`. Allow
it to keep backwards compatibility with these versions by marking these
dependencies as optional.

Fixes #4658.
2021-05-02 05:30:50 +01:00
Domen Kožar e3e78ee2a2
Merge pull request #4751 from Ma27/storepath-pos
primops/storePath: add trace to pure mode error
2021-04-27 19:05:23 +02:00
Maximilian Bosch e6a360a4d9
primops/storePath: add trace to pure mode error
As described in #4745 it's otherwise fairly hard to understand where
this is coming from. Say you have an expression which uses e.g.
`types.package`:

``` nix
{ outputs = { self, nixpkgs }: {
    packages.x86_64-linux.hello = let
      foo = nixpkgs.lib.evalModules {
        modules = [
          {
            options.foo.bar = with nixpkgs.lib; mkOption { type = types.package; };
          }
          {
            foo.bar = ./.;
          }
        ];
      };
    in builtins.trace foo.config.foo.bar.outPath nixpkgs.legacyPackages.x86_64-linux.hello;
    defaultPackage.x86_64-linux = self.packages.x86_64-linux.hello;
  };
}
```

Then you'll get an error trace like this:

```
error: 'builtins.storePath' is not allowed in pure evaluation mode

       at /nix/store/p4h2x6r80njkb0j2rc1xjhhl99yri3zb-source/lib/attrsets.nix:328:15:

          327|     let
          328|       path' = builtins.storePath path;
             |               ^
          329|       res =

       … while evaluating the attribute 'config.foo.bar.outPath'

       at /nix/store/p4h2x6r80njkb0j2rc1xjhhl99yri3zb-source/lib/attrsets.nix:332:11:

          331|           name = sanitizeDerivationName (builtins.substring 33 (-1) (baseNameOf path'));
          332|           outPath = path';
             |           ^
          333|           outputs = [ "out" ];

       … while evaluating the attribute 'packages.x86_64-linux.hello'

       at /nix/store/6c1rfsqzrhjw1235palzjmf5vihcpci7-source/flake.nix:3:5:

            2| { outputs = { self, nixpkgs }: {
            3|     packages.x86_64-linux.hello = let
             |     ^
            4|       foo = nixpkgs.lib.evalModules {
```

Fixes #4745
2021-04-27 17:24:38 +02:00
Félix Baylac-Jacqué 7dca9c24c4
nix build: make dry-run to print a json output if --json is enabled 2021-04-27 11:28:47 +02:00
Eelco Dolstra fe2bf464cf Merge branch 'remove-trailing-spaces' of github.com:NixOS/nix 2021-04-23 15:09:44 +02:00
regnat 31313d1401 Replace the trailing markdown spaces by a backslash
They are equivalent according to
<https://spec.commonmark.org/0.29/#hard-line-breaks>,
and the trailing spaces tend to be a pain (because the make git
complain, editors tend to want to remove them − the `.editorconfig`
actually specifies that − etc..).
2021-04-23 14:37:21 +02:00
Eelco Dolstra 4549d65b4f
Merge pull request #4732 from NixOS/4725-always-register-the-realisations
Aways register the realisations
2021-04-23 14:22:03 +02:00
Eelco Dolstra 293220bed5
Merge pull request #4440 from Ma27/misc-pos-fixes
Miscellaneous improvements for positioning in eval-errors
2021-04-23 11:10:59 +02:00
regnat d3df747cb6 Revert "Make nix shell fallback to static outputs when needed"
This reverts commit 8d66f5f110.

This fix isn't needed anymore now that the realisations are always
properly registered
2021-04-23 09:34:43 +02:00
p01arst0rm 45473d02c9 unified macro style for ENABLE_S3 2021-04-23 07:30:05 +01:00
regnat 9161e02039 Always register the realisations of input-addressed drvs
Fix #4725
2021-04-22 20:07:02 +02:00
Eelco Dolstra d9864be4b7 Merge branch 'master' of github.com:NixOS/nix 2021-04-22 10:33:00 +02:00
Eelco Dolstra 78a2303b3d Merge branch 'nix-derivation-query-store-path' of https://github.com/matthewbauer/nix 2021-04-22 10:29:18 +02:00
Eelco Dolstra dd83b29b90
Merge pull request #4724 from NixOS/nix-shell-missing-outputs
Make `nix shell` fallback to static outputs when needed
2021-04-22 10:25:55 +02:00
Eelco Dolstra 5385755227 Remove makeDecompressionSource()
This function doesn't support all compression methods (i.e. 'none' and
'br') so it shouldn't be exposed.

Also restore the original decompress() as a wrapper around
makeDecompressionSink().
2021-04-22 10:23:20 +02:00
Eelco Dolstra a6eebcff36 Merge branch 's3-decompress' of https://github.com/lukegb/nix 2021-04-22 10:20:40 +02:00
Luke Granger-Brown 97dde3cdd9 libutil: allow decompression with none/empty method
The S3 store relies on the ability to be able to decompress things with
an empty method, because it just passes the value of the Content-Encoding
directly to decompress.

If the file is not compressed, then this will cause the compression
routine to get confused.

This caused NixOS/nixpkgs#120120.
2021-04-22 02:33:59 +00:00
Antoine Martin ac38c4cdc2 Fix typo in flake doc 2021-04-22 03:07:57 +02:00
Matthew Bauer 428062e578 Get deriver when passing --derivation
This makes Nix look up paths derivations when they are passed as a
store paths. So:

$ nix path-info --derivation /nix/store/0pisd259nldh8yfjvw663mspm60cn5v8-hello-2.10

now gives

/nix/store/qp724w90516n4bk5r9gfb37vzmqdh3z7-hello-2.10.drv

instead of "".

If no deriver is available, Nix now errors instead of silently
ignoring that argument.
2021-04-21 15:32:33 -05:00
regnat 8d66f5f110 Make nix shell fallback to static outputs when needed
In case of pure input-addressed derivations, the build loop doesn't
guaranty that the realisations are stored in the db (if the output paths
are already there or can be substituted, the realisations won't be
registered). This caused `nix shell` to fail in some cases because it
was assuming that the realisations were always existing.

A better (but more involved) fix would probably to ensure that we always
register the realisations, but in the meantime this patches the surface
issue.

Fix #4721
2021-04-21 17:46:27 +02:00
Eelco Dolstra 8d651a1f68 Fix brotli compression of files > 128 KiB
This has been broken since faa31f4084.
2021-04-20 22:54:49 +02:00
Eelco Dolstra b60b0d62d6
Merge pull request #4720 from alyssais/wait
Include sys/wait.h everywhere WIFEXITED etc is used
2021-04-20 16:09:30 +02:00
Alyssa Ross 9ac6534f7c
Include sys/wait.h everywhere WIFEXITED etc is used
This is required on NetBSD, and I think FreeBSD too.
2021-04-19 18:31:58 +00:00
Maximilian Bosch 6905291daf
libcmd/installables: force re-evaluation of cached failures
I think that it's not very helpful to get "cached failures" in a wrong
`flake.nix`. This can be very confusing when debugging a Nix expression.
See for instance NixOS/nixpkgs#118115.

In fact, the eval cache allows a forced reevaluation which is used for
e.g. `nix eval`.

This change makes sure that this is the case for `nix build` as well. So
rather than

    λ ma27 [~/Projects/exp] → ../nix/outputs/out/bin/nix build -L --rebuild --experimental-features 'nix-command flakes'
    error: cached failure of attribute 'defaultPackage.x86_64-linux'

the evaluation of already-evaluated (and failed) attributes looks like
this now:

    λ ma27 [~/Projects/exp] → ../nix/outputs/out/bin/nix build -L --rebuild --experimental-features 'nix-command flakes'
    error: attribute 'hell' missing

           at /nix/store/mrnvi9ss8zn5wj6gpn4bcd68vbh42mfh-source/flake.nix:6:35:

                5|
                6|     packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hell;
                 |                                   ^
                7|
    (use '--show-trace' to show detailed location information)
2021-04-19 19:51:04 +02:00
Eelco Dolstra 3ee0ecdda0
Merge pull request #4679 from ony/feature/one-pass-canon-path
Optimize canonPath to resolve relative symlinks in one pass
2021-04-15 14:11:51 +02:00
Eelco Dolstra 15f4d4fd43 Drop libbz2 / zlib / lzma dependency + style fixes 2021-04-15 13:55:22 +02:00
Eelco Dolstra 6fb7582413 Merge branch 'libarchive-decompress' of https://github.com/serokell/nix 2021-04-15 13:39:04 +02:00
Eelco Dolstra f716779c1f Merge branch 'revertRemoveJSON' of https://github.com/cgohla/nix 2021-04-15 13:26:56 +02:00
Eelco Dolstra 709a60a045
Merge pull request #4707 from grahamc/defaul-path-restricted-eval
EvalSettings::getDefaultNixPath: respect {restrict,pure}Eval
2021-04-14 11:02:58 +02:00
Eelco Dolstra b47b5f4061
Merge pull request #4708 from NixOS/fix-remote-registerDrvOutput
Fix registerDrvOutput with the daemon
2021-04-14 11:01:36 +02:00
Maximilian Bosch 864ef0e93d
libexpr/primops: review 2021-04-13 23:12:39 +02:00
Maximilian Bosch ad7b47dc85
primops/libexpr: use new attr-call extractor everywhere; use function's pos if attr-set pos == noPos 2021-04-13 23:12:38 +02:00
Maximilian Bosch f60473a077
libexpr/primops: Move attr name extraction into its own function
This now takes care of providing positioning for both the faulting value
and the faulting function call in case of an error.
2021-04-13 23:12:38 +02:00
Maximilian Bosch 7c76964daa
libexpr: misc improvements for proper error position
When working on some more complex Nix code, there are sometimes rather
unhelpful or misleading error messages, especially if coerce-errors are
thrown.

This patch is a first steps towards improving that. I'm happy to file
more changes after that, but I'd like to gather some feedback first.

To summarize, this patch does the following things:

* Attrsets (a.k.a. `Bindings` in `libexpr`) now have a `Pos`. This is
  helpful e.g. to identify which attribute-set in `listToAttrs` is
  invalid.

* The `Value`-struct has a new method named `determinePos` which tries
  to guess the position of a value and falls back to a default if that's
  not possible.

  This can be used to provide better messages if a coercion fails.

* The new `determinePos`-API is used by `builtins.concatMap` now. With
  that change, Nix shows the exact position in the error where a wrong
  value was returned by the lambda.

  To make sure it's still obvious that `concatMap` is the problem,
  another stack-frame was added.

* The changes described above can be added to every other `primop`, but
  first I'd like to get some feedback about the overall approach.
2021-04-13 23:12:38 +02:00
Maximilian Bosch 3550a32b25
primops/derivation: use position of currently evaluated attribute
* The position of the `name`-attribute appears in the trace.
* If e.g. `meta` has no `outPath`-attribute, a `cannot coerce set to
  string` error will be thrown where `pos` points to `name =` which is
  highly misleading.
2021-04-13 23:12:38 +02:00
regnat ed29610cc6 Fix registerDrvOutput with the daemon
Resolve a protocol issue that caused the daemon to endlessly wait for
some information that the client doesn't ever send
2021-04-13 13:58:22 +02:00
Graham Christensen d04969ffa5 EvalSettings::getDefaultNixPath: respect {restrict,pure}Eval
Otherwise Nix may look to invalid locations for channels.
2021-04-12 12:37:15 -04:00
Domen Kožar c2dfda007e
add tests for zstd compression 2021-04-09 23:13:08 +02:00
Eelco Dolstra 42f0246698
Revert "libfetchers/tarball: Lock on effectiveUrl"
This reverts commit fc6bfb261d.

Fixes #4672.
2021-04-09 13:51:57 +02:00
Eelco Dolstra 906adadacd Restore stack size in child processes
Fixes #4673.
2021-04-07 13:40:13 +02:00
Eelco Dolstra 9b9e703df4 restoreSignals() + restoreAffinity() -> restoreProcessContext() 2021-04-07 13:10:02 +02:00
Eelco Dolstra 8a29052cb2 PathSubstitutionGoal: Clean up pipe
If there were many top-level goals (which are not destroyed until the
very end), commands like

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' \
    /run/current-system --no-check-sigs --substitute-on-destination

could fail with "Too many open files". So now we do some explicit
cleanup from amDone(). It would be cleaner to separate goals from
their temporary internal state, but that would be a bigger refactor.
2021-04-07 12:21:31 +02:00
Björn Gohla 9f28dd97ae Revert "Use upstream nlohmann_json"
This reverts commit 4145cd2da0.
2021-04-05 21:24:55 +01:00
John Ericson 125a824228 Document the derived path types. 2021-04-05 10:56:48 -04:00
John Ericson d8fa7517fa buildable.{cc,hh} -> derived-path.{cc,hh} 2021-04-05 10:33:28 -04:00
John Ericson 179582872d Make DerivedPathWithHints a newtype
This allows us to namespace its constructors under it.
2021-04-05 10:05:21 -04:00
John Ericson 9b805d36ac Rename Buildable 2021-04-05 09:52:25 -04:00
John Ericson 9dfb97c987 "newtype" BuildableReq
This makes for better types errors and allows us to give it methods.
2021-04-05 09:35:55 -04:00
John Ericson 4fe41c6db3 No templates for Buildable and BuildableReq 2021-04-05 09:15:25 -04:00
John Ericson 255d145ba7 Use BuildableReq for buildPaths and ensurePath
This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {}
}` could mean "build `brvPath`" or "substitute `drvPath`" depending on
context.

It also brings the internals closer in line to the new CLI, by
generalizing the `Buildable` type is used there and makes that
distinction already.

In doing so, relegate `StorePathWithOutputs` to being a type just for
backwards compatibility (CLI and RPC).
2021-04-05 08:33:00 -04:00
John Ericson 32f4454b9f Move StorePathWithOutput utilities out of store class
These are by no means part of the notion of a store, but rather are
things that happen to use stores. (Or put another way, there's no way
we'd make them virtual methods any time soon.) It's better to move them
out of that too-big class then.

Also, this helps us remove StorePathWithOutputs from the Store interface
altogether next commit.
2021-04-05 08:31:37 -04:00
John Ericson 7a2b566dc8 Move StorePathWithOutputs into its own header/file
In the following commits it will become less prevalent.
2021-04-05 08:31:37 -04:00
John Ericson f7d9f7c338 Pull out Buildable into its own file/header in libnixstore 2021-04-05 08:31:37 -04:00
Mykola Orliuk f3f228700a canonPath in one pass 2021-03-31 04:58:49 +02:00
Bjørn Forsman edd606ae62 fetchGit: don't prefix "refs/heads/" on ref = "HEAD"
This fixes builtins.fetchGit { url = ...; ref = "HEAD"; }, that works in
stable nix (v2.3.10), but is broken in nix master:

  $ ./result/bin/nix repl
  Welcome to Nix version 2.4pre19700101_dd77f71. Type :? for help.

  nix-repl> builtins.fetchGit { url = "https://github.com/NixOS/nix"; ref = "HEAD"; }
  fetching Git repository 'https://github.com/NixOS/nix'fatal: couldn't find remote ref refs/heads/HEAD
  error: program 'git' failed with exit code 128

The documentation for builtins.fetchGit says ref = "HEAD" is the
default, so it should also be supported to explicitly pass it.

I came across this issue because poetry2nix can use ref = "HEAD" in some
situations.

Fixes #4674.
2021-03-30 11:21:38 +02:00
Eelco Dolstra dd77f71afe LocalBinaryCacheStore::upsertFile(): Fix race
When multiple threads try to upsert the same file, this could fail.

Fixes #4667.
2021-03-26 17:10:15 +01:00
Eelco Dolstra 4638bcfb2c Fix some typos
Fixes #4671.
2021-03-26 16:14:38 +01:00
Eelco Dolstra d1cb956bf2
Merge pull request #4650 from samueldr/fix/registry-config-dir
Use the appropriate config dir for the registry
2021-03-24 13:37:19 +01:00
Eelco Dolstra 1581c3e8ef
Merge pull request #4603 from DavHau/davhau-improve-nix-conf-help-builders
improve man page for nix.conf (builders)
2021-03-23 16:13:02 +01:00
Eelco Dolstra 6749d9e057
Merge pull request #4646 from Ma27/support-scoped-v6
Fix Nix to properly work with stores using a scoped IPv6 address
2021-03-23 15:03:47 +01:00
DavHau 0f40561c78 nix.conf builders: refer to manual page 2021-03-23 10:19:00 +07:00
John Ericson f44206e719 Fix typos in the last PR #4656 2021-03-22 15:18:48 +00:00
John Ericson 9d309de0de Clean up serialization for BuildResult
A few versioning mistakes were corrected:

- In 27b5747ca7, Daemon protocol had some
  version `>= 0xc` that should have been `>= 0x1c`, or `28` since the
  other conditions used decimal.

- In a2b69660a9, legacy SSH gated new CAS
  info on version 6, but version 5 in the server. It is now 6
  everywhere.

Additionally, legacy ssh was sending over more metadata than the daemon
one was. The daemon now sends that data too.

CC @regnat

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2021-03-22 14:57:41 +00:00
Samuel Dionne-Riel 66b857244f Use the appropriate config dir for the registry 2021-03-19 15:20:47 -04:00
Eelco Dolstra 3e0e443181 ProgressBar: Respect verbosity level
This makes its behaviour consistent with SimpleLogger.
2021-03-19 17:56:39 +01:00
Eelco Dolstra ef83ced4e1 Restore 'nix flake info' as a deprecated alias 2021-03-19 17:21:37 +01:00
Eelco Dolstra 8a5203d3b8
Merge pull request #4648 from hercules-ci/remove-unimplemented-hashAlgoOpt
Remove unimplemented hashAlgoOpt
2021-03-17 20:51:09 +01:00
Robert Hensing a61112aadf Remove unimplemented hashAlgoOpt
It was in the header but never implemented.
2021-03-17 11:27:11 +01:00
Maximilian Bosch ddcef3bcbb
Fix Nix to properly work with stores using a scoped IPv6 address
According to RFC4007[1], IPv6 addresses can have a so-called zone_id
separated from the actual address with `%` as delimiter. In contrast to
Nix 2.3, the version on `master` doesn't recognize it as such:

    $ nix ping-store --store ssh://root@fe80::1%18 --experimental-features nix-command
    warning: 'ping-store' is a deprecated alias for 'store ping'
    error: --- Error ----------------------------------------------------------------- nix
    don't know how to open Nix store 'ssh://root@fe80::1%18'

I modified the IPv6 match-regex accordingly to optionally detect this
part of the address. As we don't seem to do anything special with it, I
decided to leave it as part of the URL for now.

Fixes #4490

[1] https://tools.ietf.org/html/rfc4007
2021-03-16 19:14:42 +01:00
Eelco Dolstra 66fa1c7375 Merge 'nix flake {info,list-inputs}' into 'nix flake metadata'
Fixes #4613.
2021-03-16 17:19:04 +01:00
Eelco Dolstra 77f5d171e1 --override-input: Imply --no-write-lock-file
Fixes #3779.
2021-03-16 16:53:39 +01:00
Eelco Dolstra ccb8a403ee
Merge pull request #4587 from obsidiansystems/derivation-goal-detect-invalid-output
Throw error for derivation goal with bogus wanted output
2021-03-15 16:49:44 +01:00
Eelco Dolstra c0073f6268
Merge pull request #4580 from obsidiansystems/restore-test-build-remote-ca-fixed
Restore now-working build-remote-content-addressed-fixed test
2021-03-15 16:48:28 +01:00
Eelco Dolstra 91ea9c52ee
Merge pull request #4624 from NixOS/ca/realisation-nix-command
Add a `nix realisation` command for working on realisations
2021-03-15 16:37:09 +01:00
regnat 703c98c6cb Properly sign the unresolved drvs
Don't let them inherit the signature from the parent one (because it
makes no sense to do so), but re-sign them after they have been built
2021-03-15 16:35:17 +01:00
regnat 54ced9072b Check the signatures when copying store paths around
Broken atm
2021-03-15 16:35:14 +01:00
regnat 3e6017f911 pathInfoIsTrusted -> pathInfoIsUntrusted
I guess the rationale behind the old name wath that
`pathInfoIsTrusted(info)` returns `true` iff we would need to `blindly`
trust the path (because it has no valid signature and `requireSigs` is
set), but I find it to be a really confusing footgun because it's quite
natural to give it the opposite meaning.
2021-03-15 16:34:49 +01:00
regnat 826877cabf Add some logic for signing realisations
Not exposed anywhere, but built realisations are now signed (and this
should be forwarded when copy-ing them around)
2021-03-15 16:34:49 +01:00
Eelco Dolstra 306c154632
Merge pull request #4592 from NixOS/ca/remote-cache
Substitute content-addressed derivations
2021-03-15 16:22:42 +01:00
Eelco Dolstra 3bb1becdbb
Merge pull request #4566 from orbekk/master
Add support for bare git repositories when using git+file
2021-03-11 10:38:07 +01:00
Yorick 8a0c00b856
Use libarchive for all compression 2021-03-10 22:34:29 +01:00
regnat 89013bdd7e Add a nix realisation command for working on realisations
Currently only has `nix realisation info`, more to come probably
2021-03-09 10:16:44 +01:00
Eelco Dolstra 1c0e3e453d
Merge pull request #4601 from lovesegfault/fix-4598
nix-build: check that envCommand exists
2021-03-08 10:13:39 +01:00
Eelco Dolstra c5a232dda7
Merge pull request #4615 from NixOS/nix-show-stats-with-nix-cmd
Make NIX_SHOW_STATS work with new-style commands
2021-03-08 09:56:59 +01:00
Bernardo Meurer 6e849e3b0a
nix-build: set execfail
When starting a nix-shell with `-i` it was previously possible for it to
silently fail in the scenario where the specified interpreter didn't
exist. This happened due to the `exec` call masking the issue.

With this change we enable `execfail`, which causes the script using
`nix-shell` as interpreter to correctly exit with code 127.

Fixes: #4598
2021-03-04 18:54:45 -08:00
John Ericson 6212e89bf6 Avoid some StorePath -> Path -> StorePath roundtrips
There were done when StorePath was defined in Rust and there were some
FFI issues. This is no longer an issue.
2021-03-05 00:49:46 +00:00
DavHau e16431b466 improve man page for nix.conf (builders) 2021-03-04 16:14:23 +07:00
Eelco Dolstra 665d4ec2da nix repl :doc: Don't return docs for partially applied primops
This gives misleading results for Nixpkgs functions like lib.toUpper.

Fixes #4596.
2021-03-03 17:52:57 +01:00
regnat 7331da99ab Make NIX_SHOW_STATS work with new-style commands 2021-03-02 14:59:12 +01:00
dramforever fc6bfb261d libfetchers/tarball: Lock on effectiveUrl
Basically, if a tarball URL is used as a flake input, and the URL leads
to a redirect, the final redirect destination would be recorded as the
locked URL.

This allows tarballs under https://nixos.org/channels to be used as
flake inputs. If we, as before, lock on to the original URL it would
break every time the channel updates.
2021-03-02 21:56:50 +08:00
John Ericson 7ce10924c7 Fix bad wanted output error as requested
- UsageError -> Error

- include drv path too
2021-03-01 15:07:09 +00:00
Kjetil Orbekk 9931f18c2d Add support for bare git repositories with git+file
Local git repositories are normally used directly instead of
cloning. This commit checks if a repo is bare and forces a
clone.

Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2021-03-01 09:03:25 -05:00
regnat df9d4f88d5 Allow substituting drv outputs when building 2021-03-01 14:00:17 +01:00
regnat 5d1c05b075 SubstitutionGoal -> PathSubstitutionGoal
To prepare for the upcoming DrvOutputSubstitutionGoal
2021-03-01 14:00:17 +01:00
Eelco Dolstra e64cf8e0a3
Merge pull request #4574 from grahamc/libstore-ssh-host-key
libstore: support passing a builder's public SSH host key
2021-03-01 13:12:18 +01:00
John Ericson 4bbd80c536 Throw error for derivation goal with bogus wanted output 2021-02-28 00:19:35 +00:00
John Ericson ae1441e548 Fix testing fixed-output derivations in double sandboxes
What happened was that Nix was trying to unconditionally mount these
paths in fixed-output derivations, but since the outer derivation was
pure, those paths did not exist. The solution is to only mount those
paths when they exist.
2021-02-27 05:23:14 +00:00
Puck Meerburg 7241fdc3d2 Properly propagate libseccomp linker flags 2021-02-26 23:01:16 +00:00
John Ericson 553b79f8c9 Remove unused redirectedBadOutputs 2021-02-26 16:10:54 +00:00
John Ericson d560311f76 Remove temporary #if 0...#endif from previous commit 2021-02-26 16:10:52 +00:00
John Ericson 68f4c728ec Split {,local-}derivation-goal.{cc,hh}
This separates the scheduling logic (including simple hook pathway) from
the local-store needing code.

This should be the final split for now. I'm reasonably happy with how
it's turning out, even before I'm done moving code into
`local-derivation-goal`. Benefits:

1. This will help "witness" that the hook case is indeed a lot simpler,
   and also compensate for the increased complexity that comes from
   content-addressed derivation outputs.

2. It also moves us ever so slightly towards a world where we could use
   off-the-shelf storage or sandboxing, since `local-derivation-goal`
   would be gutted in those cases, but `derivation-goal` should remain
   nearly the same.

The new `#if 0` in the new files will be deleted in the following
commit. I keep it here so if it turns out more stuff can be moved over,
it's easy to do so in a way that preserves ordering --- and thus
prevents conflicts.

N.B.
```sh
git diff HEAD^^ --color-moved --find-copies-harder --patience --stat
```
makes nicer output.
2021-02-26 16:10:26 +00:00
John Ericson 05cc5a8587 Copy {,local-}derivation-goal.{cc,h}
Doing this prior to splitting, so we get better diff with default
options (e.g. on GitHub).
2021-02-26 16:01:47 +00:00
Eelco Dolstra 94637cd7e5
Merge pull request #4477 from NixOS/ca/build-remote
Build ca derivations remotely
2021-02-26 16:54:44 +01:00
regnat f54976d77b Simplify the case where the drv is a purely input-addressed one 2021-02-26 16:35:05 +01:00
Eelco Dolstra 076d2b04da
Update src/libstore/build/derivation-goal.cc 2021-02-26 16:30:12 +01:00
Eelco Dolstra 17c98e03ea
Update src/build-remote/build-remote.cc 2021-02-26 16:29:37 +01:00
Eelco Dolstra 14f51880ba
Update src/build-remote/build-remote.cc 2021-02-26 16:29:30 +01:00
Eelco Dolstra 453c3a603f nix flake update: Recreate the lock file
This is probably what most people expect it to do. Fixes #3781.

There is a new command 'nix flake lock' that has the old behaviour of
'nix flake update', i.e. it just adds missing lock file entries unless
overriden using --update-input.
2021-02-26 14:55:54 +01:00
Eelco Dolstra 20ea1de77d Use std::make_unique 2021-02-26 12:35:29 +01:00
Eelco Dolstra 73daffb81b Merge remote-tracking branch 'origin/deduplicate-static-hashes-computation' 2021-02-26 12:30:25 +01:00
Eelco Dolstra d2803406b5
Merge pull request #4487 from NixOS/ca/copy-drv-outputs
Copy ca derivation outputs
2021-02-26 12:20:42 +01:00
Eelco Dolstra 5edab777d1
Merge pull request #4530 from alyssais/kill
libutil: EPERM from kill(-1, ...) is fine
2021-02-26 12:19:16 +01:00
regnat c43f446f4e Make nix copy work without the ca-derivations flag
The experimental feature was by mistake required for `nix copy` to work
at oll
2021-02-25 17:21:51 +01:00
Théophane Hufschmitt c182aac98a Apply @edolstra stylistic suggestions
Mostly removing useless comments and adding spaces before `&`

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-25 17:21:51 +01:00
regnat 3b76f8f252 Ensure that the ca-derivations bit is set when copying realisations
This should already hold, but better ensure it for future-proof-nees
2021-02-25 17:19:59 +01:00
regnat f67ff1f575 Don't crash when copying realisations to a non-ca remote
Rather throw a proper exception, and catch&log it on the client side
2021-02-25 17:19:59 +01:00
regnat 2e199673a5 Use RealisedPaths in copyPaths
That way we can copy the realisations too (in addition to the store
paths themselves)
2021-02-25 17:18:48 +01:00
Graham Christensen 1130b28824
distributed builds: load remote builder host key from the machines file
This is already used by Hydra, and is very useful when materializing
a remote builder list from service discovery. This allows the service
discovery tool to only sync one file instead of two.
2021-02-25 09:17:34 -05:00
Shea Levy f6c5b05488
Respect command registrations in plugins. 2021-02-24 08:25:45 -05:00
Shea Levy 98d1b64400
Initialize plugins after handling initial command line flags
This is technically a breaking change, since attempting to set plugin
files after the first non-flag argument will now throw an error. This
is acceptable given the relative lack of stability in a plugin
interface and the need to tie the knot somewhere once plugins can
actually define new subcommands.
2021-02-24 08:22:17 -05:00
Shea Levy ec3497c1d6
Bail if plugin-files is set after plugins have been loaded.
We know the flag will be ignored but the user wants it to take effect.
2021-02-24 08:20:48 -05:00
Eelco Dolstra a878c448d8
Merge pull request #4551 from danieldk/system-features-compute-level
Add x86_64 compute levels as system features
2021-02-23 14:30:10 +01:00
regnat ba1a256d08 Make DerivationGoal::drv a full Derivation
This field used to be a `BasicDerivation`, but this `BasicDerivation`
was downcasted to a `Derivation` when needed (implicitely or not), so we
might as well make it a full `Derivation` and upcast it when needed.

This also allows getting rid of a weird duplication in the way we
compute the static output hashes for the derivation. We had to
do it differently and in a different place depending on whether the
derivation was a full derivation or just a basic drv, but we can now do
it unconditionally on the full derivation.

Fix #4559
2021-02-23 14:15:45 +01:00
regnat 527da73690 Properly bypass the registering step when all outputs are present
There was already some logic for that, but it didn't handle the case of
content-addressed outputs, so extend it a bit for that
2021-02-23 08:04:03 +01:00
regnat 69666b951e build-remote: Always register the missing outputs
It's possible that all the paths are already there, but just not
associated to the current drv output
2021-02-23 08:04:03 +01:00
regnat 8c385d16ee Also send ca outputs to the build hook
Otherwise they don't get registered, triggering an assertion failure
at some point later
2021-02-23 08:04:03 +01:00
regnat 27b5747ca7 RemoteStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat a2b69660a9 LegacySSHStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat 5687564a27 LocalStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat 6fbf3fe636 Make the build-hook work with ca derivations
- Pass it the name of the outputs rather than their output paths (as
  these don't exist for ca derivations)
- Get the built output paths from the remote builder
- Register the new received realisations
2021-02-23 08:04:03 +01:00
Shea Levy 35205e2e92
Warn about instability of plugin API 2021-02-22 17:10:55 -05:00
regnat e2f3b2eb42 Make missing auto-call arguments throw an eval error
The PR #4240 changed messag of the error that was thrown when an auto-called
function was missing an argument.
However this change also changed the type of the error, from `EvalError`
to a new `MissingArgumentError`. This broke hydra which was relying on
an `EvalError` being thrown.

Make `MissingArgumentError` a subclass of `EvalError` to un-break hydra.
2021-02-22 16:13:09 +01:00
Eelco Dolstra 574eb2be81
Tweak error message 2021-02-22 15:24:14 +01:00
Daniël de Kok 2de232d2b3 Add x86_64 compute levels as additional system types
When performing distributed builds of machine learning packages, it
would be nice if builders without the required SIMD instructions can
be excluded as build nodes.

Since x86_64 has accumulated a large number of different instruction
set extensions, listing all possible extensions would be unwieldy.
AMD, Intel, Red Hat, and SUSE have recently defined four different
microarchitecture levels that are now part of the x86-64 psABI
supplement and will be used in glibc 2.33:

https://gitlab.com/x86-psABIs/x86-64-ABI
https://lwn.net/Articles/844831/

This change uses libcpuid to detect CPU features and then uses them to
add the supported x86_64 levels to the additional system types. For
example on a Ryzen 3700X:

$ ~/aps/bin/nix -vv --version | grep "Additional system"
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
2021-02-22 09:11:15 +01:00
Eelco Dolstra 548437c234
Merge pull request #4541 from obsidiansystems/simpler-store-path-command
Deeper `Command` hierarchy to remove redundancy
2021-02-19 16:18:53 +01:00
regnat f483b623e9 Remove the drv resolution caching mechanism
It isn't needed anymore now that don't need to eagerly resolve
everything like we used to do. So we can safely get rid of it
2021-02-19 15:48:31 +01:00
regnat 4bc28c44f2 Store the output hashes in the initialOutputs of the drv goal
That way we
1. Don't have to recompute them several times
2. Can compute them in a place where we know the type of the parent
  derivation, meaning that we don't need the casting dance we had before
2021-02-19 15:48:31 +01:00
Théophane Hufschmitt 0bfbd04369 Don't expose the "bang" drvoutput syntax
It's not fixed nor useful atm, so better keep it hidden

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-19 15:48:31 +01:00
Théophane Hufschmitt 93d9eb78a0 Syntactic fixes
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-19 15:48:31 +01:00
regnat 87c8d3d702 Register the realisations for unresolved drvs
Once a build is done, get back to the original derivation, and register
all the newly built outputs for this derivation.

This allows Nix to work properly with derivations that don't have all
their build inputs available − thus allowing garbage collection and
(once it's implemented) binary substitution
2021-02-19 15:48:31 +01:00
regnat 263f6dbd1c Don't crash nix-build when not all outputs are realised
Change the `nix-build` logic for linking/printing the output paths to allow for
some outputs to be missing. This might happen when the toplevel
derivation didn't have to be built, either because all the required
outputs were already there, or because they have all been substituted.
2021-02-19 15:48:31 +01:00
Eelco Dolstra cd44c0af71 Increase default stack size on Linux
Workaround for #4550.
2021-02-18 19:22:37 +01:00
Eelco Dolstra 1b57825524 Document meta.mainProgram
Issue #4498.
2021-02-17 17:58:47 +01:00
Eelco Dolstra 7bd9898d5c nix run: Allow program name to be set in meta.mainProgram
This is useful when the program name doesn't match the package name
(e.g. ripgrep vs rg).

Fixes #4498.
2021-02-17 17:55:15 +01:00
Eelco Dolstra 13897afbe6 Throw an error if --arg / --argstr is used with a flake
Fixes #3949.
2021-02-17 17:32:10 +01:00
Eelco Dolstra f33878b656 Make 'nix --version -vv' work
Fixes #3743.
2021-02-17 17:11:21 +01:00
Eelco Dolstra cced73496b nix flake show: Handle 'overlays' output
Fixes #4542.
2021-02-17 16:53:39 +01:00
Eelco Dolstra 063de66909 nix develop: Fix quoted string handling
Fixes #4540.
2021-02-17 16:42:03 +01:00
John Ericson ad337c8697 Deeper Command hierarchy to remove redundancy
Simply put, we now have `StorePathCommand : public StorePathsCommand` so
`StorePathCommand` doesn't reimplement work.
2021-02-12 17:48:09 +00:00
Graham Christensen f2245091d0
Revert "narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash" 2021-02-09 12:26:41 -05:00
Eelco Dolstra ee3846b587
Merge pull request #4464 from tweag/nar-narhash-addressed
narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash
2021-02-09 14:47:39 +01:00
Eelco Dolstra fd6eaa1766
Merge pull request #4525 from sternenseemann/lowdown-0.8.0
libcmd/markdown: handle allocation errors in lowdown_term_rndr
2021-02-07 21:24:16 +01:00
Eelco Dolstra 37352aa7e1 Support --no-net for backwards compatibility 2021-02-07 20:44:56 +01:00
Alyssa Ross 7c112351d9
libutil: EPERM from kill(-1, ...) is fine
I tested a trivial program that called kill(-1, SIGKILL), which was
run as the only process for an unpriveleged user, on Linux and
FreeBSD.  On Linux, kill reported success, while on FreeBSD it failed
with EPERM.

POSIX says:

> If pid is -1, sig shall be sent to all processes (excluding an
> unspecified set of system processes) for which the process has
> permission to send that signal.

and

> The kill() function is successful if the process has permission to
> send sig to any of the processes specified by pid.  If kill() fails,
> no signal shall be sent.

and

> [EPERM]
>     The process does not have permission to send the signal to any
>     receiving process.

My reading of this is that kill(-1, ...) may fail with EPERM when
there are no other processes to kill (since the current process is
ignored).  Since kill(-1, ...) only attempts to kill processes the
user has permission to kill, it can't mean that we tried to do
something we didn't have permission to kill, so it should be fine to
interpret EPERM the same as success here for any POSIX-compliant
system.

This fixes an issue that Mic92 encountered[1] when he tried to review a
Nixpkgs PR on FreeBSD.

[1]: https://github.com/NixOS/nixpkgs/pull/81459#issuecomment-606073668
2021-02-07 13:56:50 +00:00
sternenseemann d0e34c85f8 libcmd/markdown: handle allocation errors in lowdown_term_rndr
We upgrade to lowdown 0.8.0 [1] which contains a fix/improvement to a
behavior mentioned in this issue thread [2] where a big part of
lowdown's API would just call exit(1) on allocation errors since that
is a satisfying behavior for the lowdown binary.

Now lowdown_term_rndr returns 0 if an allocation error occurred which we
check for in libcmd/markdown.cc.

Also the extern "C" { } wrapper around lowdown.h has been removed as it
is not necessary.

[1]: 6ca7c855a0/versions.xml (L987-L1006)
[2]: https://github.com/kristapsdz/lowdown/issues/45#issuecomment-756681153
2021-02-06 13:14:57 +01:00
Eelco Dolstra 480426a364 Add more instrumentation for #4270 2021-02-05 15:57:33 +01:00
Eelco Dolstra d7c27f21ab
Merge pull request #4372 from tweag/ca/drvoutputs-commands
Add a new Cmd type working on RealisedPaths
2021-02-05 13:03:50 +01:00
Eelco Dolstra c77f4a9012
Merge pull request #4515 from matthewbauer/fix-nix-profile-install-first-output
Use derivation output name from toDerivation
2021-02-05 12:19:10 +01:00
Eelco Dolstra 0187838e2e Add a trace to readLine() failures
Hopefully this helps to diagnose 'error: unexpected EOF reading a
line' on macOS.
2021-02-05 12:18:11 +01:00
regnat e69cfdebb0 Remove the visit machinery in RealisedPath
In addition to being some ugly template trickery, it was also totally
useless as it was used in only one place where I could replace it by
just a few extra characters
2021-02-05 11:42:33 +01:00
regnat d2091af231 Move the GENERATE_CMP macro to its own file
Despite being an ugly hack, it can probably be useful in a couple extra
places
2021-02-05 11:42:33 +01:00
Théophane Hufschmitt 43d409f669
Fix a whitespace issue
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-04 14:47:56 +01:00
Théophane Hufschmitt ca8facefb6
Normalize some error messages
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-02-04 14:47:28 +01:00
Matthew Bauer e38cd5becb Always enter first level of attrset in nix search
This makes nix search always go through the first level of an
attribute set, even if it's not a top level attribute. For instance,
you can now list all GHC compilers with:

$ nix search nixpkgs#haskell.compiler
...

This is similar to how nix-env works when you pass in -A.
2021-02-03 21:22:11 -06:00
sternenseemann 76d8bdfe35 Include note about type of catched errors in tryEval documentation
Reference #356.
2021-02-03 17:14:40 +01:00
Matthew Bauer 3d1bbabe55 Use derivation output name from toDerivation
This fixes an issue where derivations with a primary output that is
not "out" would fail with:

$ nix profile install nixpkgs#sqlite
error: opening directory '/nix/store/2a2ydlgyydly5czcc8lg12n6qqkfz863-sqlite-3.34.1-bin': No such file or directory

This happens because while derivations produce every output when
built, you might not have them if you didn't build the derivation
yourself (for instance, the store path was fetch from a binary cache).
This uses outputName provided from DerivationInfo which appears to
match the first output of the derivation.
2021-02-02 19:53:19 -06:00
Domen Kožar d0b74e2d25
--no-net -> --offline 2021-02-01 13:11:42 +00:00
Eelco Dolstra b19aec7eeb
Merge pull request #4461 from NixOS/ca/error-logging-fixes
Fix some logging with ca derivations
2021-01-29 16:12:50 +01:00
James Ottaway 991edaace5
Shorten mktemp flag for macOS
Address `mktemp: illegal option -- -`.
2021-01-29 13:55:18 +10:00
regnat 9355ecd543 Add a new Cmd type working on RealisedPaths
Where a `RealisedPath` is a store path with its history, meaning either
an opaque path for stuff that has been directly added to the store, or a
`Realisation` for stuff that has been built by a derivation

This is a low-level refactoring that doesn't bring anything by itself
(except a few dozen extra lines of code :/ ), but raising the
abstraction level a bit is important on a number of levels:

- Commands like `nix build` have to query for the realisations after the
  build is finished which is fragile (see
  27905f12e4a7207450abe37c9ed78e31603b67e1 for example). Having them
  oprate directly at the realisation level would avoid that
- Others like `nix copy` currently operate directly on (built) store
  paths, but need a bit more information as they will need to register
  the realisations on the remote side
2021-01-28 09:38:44 +01:00
Eelco Dolstra b8f345b29a
Merge pull request #4483 from shlevy/libcmd
Move command plugin interface to libnixcmd
2021-01-27 17:17:58 +01:00
Eelco Dolstra 12de0466fe Add trace to build errors during import-from-derivation
Example:

error: builder for '/nix/store/9ysqfidhipyzfiy54mh77iqn29j6cpsb-failing.drv' failed with exit code 1;
       last 1 log lines:
       > FAIL
       For full logs, run 'nix log /nix/store/9ysqfidhipyzfiy54mh77iqn29j6cpsb-failing.drv'.

       … while importing '/nix/store/pfp4a4bjh642ylxyipncqs03z6kkgfvy-failing'

       at /nix/store/25wgzr2qrqqiqfbdb1chpiry221cjglc-source/flake.nix:58:15:

           57|
           58|         ifd = import self.hydraJobs.broken;
             |               ^
           59|
2021-01-27 14:46:10 +01:00
Eelco Dolstra 965dc6070a Drop trailing whitespace 2021-01-27 14:04:49 +01:00
Eelco Dolstra c03f41055d Add traces to errors while updating flake lock file
Example:

$ nix build --show-trace
error: unable to download 'https://api.github.com/repos/NixOS/nixpkgs/commits/no-such-branch': HTTP error 422 ('')

       response body:

       {
         "message": "No commit found for SHA: no-such-branch",
         "documentation_url": "https://docs.github.com/rest/reference/repos#get-a-commit"
       }

       … while fetching the input 'github:NixOS/nixpkgs/no-such-branch'

       … while updating the flake input 'nixpkgs'

       … while updating the lock file of flake 'git+file:///home/eelco/Dev/nix'
2021-01-27 14:02:54 +01:00
Eelco Dolstra 8e758d402b Remove mkFlag() 2021-01-27 12:06:03 +01:00
regnat 9da11bac57 Fix the error message when a dep is missing
Fix a mismatch in the errors thrown when a needed output was missing
from an input derivation that was leading to a wrong and quite misleading error
message
2021-01-26 14:49:23 +01:00
regnat d3c4284133 Make the error message for missing outputs more useful
Don't only show the name of the output, but also the derivation to which
this output belongs (as otherwise it's very hard to track back what went
wrong)
2021-01-26 14:49:23 +01:00
Shea Levy 6af6e41df0
Move command plugin interface to libnixcmd 2021-01-26 06:22:24 -05:00
Eelco Dolstra 36c4d6f592 Group common options 2021-01-25 19:03:13 +01:00
Eelco Dolstra 807d963ee8 Group subcommands by category 2021-01-25 18:19:32 +01:00
Eelco Dolstra 3ba98ba8f0 Tell user to run 'nix log' to get full build logs 2021-01-25 17:15:38 +01:00
Eelco Dolstra a32073e7e8 Add FIXME 2021-01-25 14:43:16 +01:00
Eelco Dolstra b159d23800 Make '--help' do the same as 'help' (i.e. show a manpage) 2021-01-25 14:38:15 +01:00
Eelco Dolstra 488a826842
Merge pull request #4467 from edolstra/error-formatting
Improve error formatting
2021-01-25 12:50:57 +01:00
Eelco Dolstra c5b42c5a42
Merge pull request #4470 from matthewbauer/fix-4469
Handle missing etag in 304 Not Modified response
2021-01-25 12:50:13 +01:00
Eelco Dolstra 680d8a5b86
Merge pull request #4387 from obsidiansystems/non-local-store-build
Make `nix-build --store whatever` work
2021-01-25 12:24:23 +01:00
Matthew Bauer 1ea5f0b66c Remove expectedETag assert in tarball.cc 2021-01-22 23:19:52 -06:00
Matthew Bauer a766824660 Handle missing etag in 304 Not Modified response
GitHub now omits the etag, but 304 implies it matches the one we
provided. Just use that one to avoid having an etag-less resource.

Fixes #4469
2021-01-22 14:47:45 -06:00
John Ericson 53a709535b Apply suggestions from code review
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-22 15:58:58 +00:00
John Ericson 8c07ed1dda Improve documentation and test and requested 2021-01-22 15:58:58 +00:00
Eelco Dolstra b7bfc7ee52
Add FIXME 2021-01-22 12:36:50 +01:00
Eelco Dolstra 0eb22db311 Fix macOS build 2021-01-21 12:46:22 +01:00
Eelco Dolstra 55849e153e Change error position formatting
It's now

  at /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix:7:7:

instead of

  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

The new format is more standard and clickable.
2021-01-21 11:02:09 +01:00
Eelco Dolstra 40608342cb Remove trailing whitespace 2021-01-21 11:02:09 +01:00
Eelco Dolstra 8d4268d190 Improve error formatting
Changes:

* The divider lines are gone. These were in practice a bit confusing,
  in particular with --show-trace or --keep-going, since then there
  were multiple lines, suggesting a start/end which wasn't the case.

* Instead, multi-line error messages are now indented to align with
  the prefix (e.g. "error: ").

* The 'description' field is gone since we weren't really using it.

* 'hint' is renamed to 'msg' since it really wasn't a hint.

* The error is now printed *before* the location info.

* The 'name' field is no longer printed since most of the time it
  wasn't very useful since it was just the name of the exception (like
  EvalError). Ideally in the future this would be a unique, easily
  googleable error ID (like rustc).

* "trace:" is now just "…". This assumes error contexts start with
  something like "while doing X".

Example before:

  error: --- AssertionError ---------------------------------------------------------------------------------------- nix
  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

       6|
       7|   x = assert false; 1;
        |       ^
       8|

  assertion 'false' failed
  ----------------------------------------------------- show-trace -----------------------------------------------------
  trace: while evaluating the attribute 'x' of the derivation 'hello-2.10'
  at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

     191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
     192|           name = "${attrs.pname}-${attrs.version}";
        |           ^
     193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {

Example after:

  error: assertion 'false' failed

         at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

              6|
              7|   x = assert false; 1;
               |       ^
              8|

         … while evaluating the attribute 'x' of the derivation 'hello-2.10'

         at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

            191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
            192|           name = "${attrs.pname}-${attrs.version}";
               |           ^
            193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 11:02:09 +01:00
adisbladis 144cad9069
narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash
This change is to simplify [Trustix](https://github.com/tweag/trustix) indexing and makes it possible to reconstruct this URL regardless of the compression used.

In particular this means that 7c2e9ca597/contrib/nix/nar/nar.go (L61-L71) can be removed and only the bits that are required to establish trust needs to be published in the Trustix build logs.
2021-01-21 10:32:56 +01:00
Eelco Dolstra 259100332f Fix clang build 2021-01-21 10:29:51 +01:00
Eelco Dolstra d04d8463fa
Merge pull request #4281 from lilyball/shebang
Escape filename given to nix-shell in shebang mode
2021-01-20 10:00:10 +01:00
Eelco Dolstra bc90252cec nix profile install: Support installing non-flakes
Fixes #4458.
2021-01-18 23:08:58 +01:00
Eelco Dolstra 555940f065 Use enumerate() 2021-01-18 22:50:39 +01:00
Eelco Dolstra ea756b3654 --refresh: Imply setting .narinfo disk cache TTL to 0 2021-01-18 14:38:31 +01:00
Eelco Dolstra 1bbc66f865 Merge branch 'slashes-in-github-branches' of https://github.com/Ma27/nix 2021-01-18 12:55:06 +01:00
regnat 11b63740e3 Fix content-addressed flake outputs
Prevent some `nix flake` commands to crash by trying to parse a
placeholder output as a store path
2021-01-18 11:31:59 +01:00
regnat 9432c170e7 Fix the drv output map for non ca derivations
With the `ca-derivation` experimental features, non-ca derivations used
to have their output paths returned as unknown as long as they weren't
built (because of a mistake in the code that systematically erased the
previous value)
2021-01-18 11:01:39 +01:00
Eelco Dolstra 1acbb61696 Tweak 2021-01-17 19:49:28 +01:00
ryneeverett 1e13c79a91 Document expected output of 'nix store ping'.
While interpreting the output is fairly intuitive it would be better to
explicitly specify what a good invocation looks like.

That this isn't completely obvious (or at least causes folks to
second-guess themselves) can be seen in a couple user threads:

- https://discourse.nixos.org/t/nixos-cache-fetching-issue/3575/11
- https://discourse.nixos.org/t/newbie-question-cant-get-trivial-example-of-nixops-to-work-on-my-mac/1125/8
2021-01-16 19:11:10 +00:00
Domen Kožar 00f99fdfe6
Merge pull request #4240 from bburdette/2259-error-message
2259 error message - "auto-call" error
2021-01-15 18:26:21 +00:00
John Ericson 7af743470c Make public keys and requireSigs local-store specific again
Thanks @regnat and @edolstra for catching this and comming up with the
solution.

They way I had generalized those is wrong, because local settings for
non-local stores is confusing default. And due to the nature of C++
inheritance, fixing the defaults is more annoying than it should be.
Additionally, I thought we might just drop the check in the substitution
logic since `Store::addToStore` is now streaming, but @regnat rightfully
pointed out that as it downloads dependencies first, that would still be
too late, and also waste effort on possibly unneeded/unwanted
dependencies.

The simple and correct thing to do is just make a store method for the
boolean logic, keeping all the setting and key stuff the way it was
before. That new method is both used by `LocalStore::addToStore` and the
substitution goal check. Perhaps we might eventually make it fancier,
e.g. sending the ValidPathInfo to remote stores for them to validate,
but this is good enough for now.
2021-01-15 16:37:41 +00:00
Jonathan Ringer 86a2ceeb98 Fix gcc10 build 2021-01-15 10:41:35 +01:00
John Ericson 0027b05a15 Merge remote-tracking branch 'upstream/master' into non-local-store-build 2021-01-15 02:01:24 +00:00
Eelco Dolstra 7a472a76d4 Add 'nix daemon' command 2021-01-14 00:05:04 +01:00
Eelco Dolstra 28ef6ebf91 Typo 2021-01-13 23:51:27 +01:00
Eelco Dolstra d33eca8539 Rename 'nix store sign-paths' to 'nix store sign' 2021-01-13 23:32:37 +01:00
Eelco Dolstra 61216d32e1 Add 'nix store repair' command 2021-01-13 23:27:39 +01:00
Eelco Dolstra 3da9a9241c Convert option descriptions to Markdown 2021-01-13 14:18:04 +01:00
Eelco Dolstra 4e9cec79bf
Merge pull request #4444 from matthewbauer/unset-curproc-arch-affinity
Set kern.curproc_arch_affinity=0 to escape Rosetta
2021-01-13 12:16:53 +01:00
Eelco Dolstra fbfa70dc02
Merge pull request #4443 from rickynils/prefer-local-build-respect-zero-max-jobs
Don't let 'preferLocalBuild' override 'max-jobs=0'
2021-01-13 12:15:54 +01:00
Rickard Nilsson 0ca1a50132 Remove a redundant condition in DerivationGoal::tryLocalBuild() 2021-01-13 10:13:51 +01:00
Eelco Dolstra 2f463e90ed Add 'nix profile history' command
Replaces 'nix-env --list-generations'. Similar to 'nix profile
diff-closures' but shows only the changes in top-level packages.
2021-01-12 23:53:53 +01:00
Eelco Dolstra 29007f8bc6 nix profile info -> nix profile list 2021-01-12 19:57:05 +01:00
Matthew Bauer f69820417f Set kern.curproc_arch_affinity=0 to escape Rosetta
By default, once you enter x86_64 Rosetta 2, macOS will try to run
everything in x86_64. So an x86_64 Nix will still try to use x86_64
even when system = aarch64-darwin. To avoid this we can set
kern.curproc_arch_affinity sysctl. With kern.curproc_arch_affinity=0,
we ignore this preference.

This is based on how
https://opensource.apple.com/source/system_cmds/system_cmds-880.40.5/arch.tproj/arch.c.auto.html
works. Completely undocumented, but seems to work!

Note, you can verify this works with this impure Nix expression:

```
  {
    a = derivation {
      name = "a";
      system = "aarch64-darwin";
      builder = "/bin/sh";
      args = [ "-e" (builtins.toFile "builder" ''
        [ "$(/usr/bin/arch)" = arm64 ]
        [ "$(/usr/bin/arch -arch x86_64 /bin/sh -c /usr/bin/arch)" = i386 ]
        [ "$(/usr/bin/arch -arch arm64 /bin/sh -c /usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };

    b = derivation {
      name = "b";
      system = "x86_64-darwin";
      builder = "/bin/sh";
      args = [ "-e" (builtins.toFile "builder" ''
        [ "$(/usr/bin/arch)" = i386 ]
        [ "$(/usr/bin/arch -arch x86_64 /bin/sh -c /usr/bin/arch)" = i386 ]
        [ "$(/usr/bin/arch -arch arm64 /bin/sh -c /usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };
  }
```
2021-01-11 22:40:21 -06:00
Rickard Nilsson 44fd7a05b6 Don't let 'preferLocalBuild' override 'max-jobs=0'
This resolves #3810 by changing the behavior of `max-jobs = 0`, so
that specifying the option also avoids local building of derivations
with the attribute `preferLocalBuild = true`.
2021-01-12 01:28:00 +01:00
Eelco Dolstra 6254b1f5d2 Add 'nix store delete' command 2021-01-11 19:46:59 +01:00
Eelco Dolstra 93ad6430ed nix store prefetch-tarball -> nix flake prefetch 2021-01-11 12:36:39 +01:00
Eelco Dolstra 7480f2bf20
Merge pull request #4435 from DanilaFe/flake-input-types
Allow Flake inputs to accept boolean and integer attributes
2021-01-11 11:38:53 +01:00
Eelco Dolstra fdcd62eec5 Add 'nix store gc' command 2021-01-10 23:29:14 +01:00
Danila 1db3f84bac
Upcase "Boolean" in Flake attribute type error
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-08 16:12:21 -08:00
Eelco Dolstra e21aee58f6 Fix tests 2021-01-08 14:17:06 +01:00
Eelco Dolstra 17beae299d Support binary unit prefixes in command line arguments 2021-01-08 12:51:19 +01:00
Eelco Dolstra 6548b89cc4 string2Int(): Return std::optional 2021-01-08 12:22:21 +01:00
Eelco Dolstra 29a445840a Remove unused mkFlag1 2021-01-08 11:42:44 +01:00
Eelco Dolstra 1d4954e73e Remove mkFlag integer specialisation 2021-01-08 11:40:36 +01:00
Eelco Dolstra 48a9be2aab Remove mkIntFlag 2021-01-08 10:44:55 +01:00
Danila Fedorin ba0f841a07 Use switch statement instead of sequence of ifs 2021-01-08 03:13:42 +00:00
Danila Fedorin 93f1678ec6 Allow Flake inputs to accept boolean and integer attributes
I believe that this makes it possible to do things like
Git inputs with submodules, but it also likely applies
to other input types from libfetchers.
2021-01-08 01:53:57 +00:00
Eelco Dolstra 0813350349 Add 'nix store prefetch-{file,tarball}'
These replace nix-prefetch-url and nix-prefetch-url --unpack,
respectively.
2021-01-07 21:51:46 +01:00
Eelco Dolstra 0df69d96e0 Make sodium a required dependency 2021-01-06 17:56:53 +01:00
Eelco Dolstra 9374c2baea Add commands for generating secret/public keys 2021-01-06 17:49:31 +01:00
Eelco Dolstra 555152ffe8 crypto.cc: API cleanup and add generate() / to_string() methods 2021-01-06 17:04:46 +01:00
Eelco Dolstra 146af4ee9b Move sodium_init() call 2021-01-06 16:43:09 +01:00
Eelco Dolstra 3edcb198e5
Merge pull request #4310 from matthewbauer/rosetta2-extra-platforms
Add x86_64-darwin and aarch64 to "extra-platforms" automatically when Rosetta2 is detected
2021-01-06 11:31:13 +01:00
Eelco Dolstra 8af4f886e2 Fix deadlock in LocalStore::addSignatures()
Fixes #4367.
2021-01-05 11:47:29 +01:00
Danila Fedorin 988dd0a65f Fix conversion from JSON to fetch attributes
It appears as through the fetch attribute, which
is simply a variant with 3 elements, implicitly
converts boolean arguments to integers. One must
use Explicit<bool> to correctly populate it with
a boolean. This was missing from the implementation,
and resulted in clearly boolean JSON fields being
treated as numbers.
2021-01-05 02:06:25 +00:00
Eelco Dolstra 368a972a38
Merge pull request #4411 from corngood/env-assert-fix
Fix insufficent attribute capacity in user profile
2020-12-30 21:47:21 +01:00
David McFarland d27eb0ef57 Fix insufficent attribute capacity in user profile 2020-12-30 16:20:03 -04:00
Eelco Dolstra bff706e27c
Merge pull request #4397 from matthewbauer/support-libcxx10
Cast variants fully for libc++10
2020-12-30 21:07:09 +01:00
Matthew Bauer 64904b9d5d Fixup 2020-12-28 19:40:04 -06:00
Eelco Dolstra 6262a70363
scanForReferences: Remove misleading comment
References have always been determined only by the hash part, not the
name or the store prefix.

Fixes #4396.
2020-12-28 17:21:19 +01:00
Matthew Bauer ede534a3a1
Merge branch 'master' into support-libcxx10 2020-12-24 14:16:09 -06:00
Matthew Bauer d4870462f8 Cast variants fully for libc++10
libc++10 seems to be stricter on what it allows in variant conversion.
I'm not sure what the rules are here, but this is the minimal change
needed to get through the compilation errors.
2020-12-23 23:41:58 -06:00
John Ericson fed1237246 Test nix-build with non-local-store --store
Just a few small things needed fixing!
2020-12-23 22:42:06 +00:00
John Ericson 85f2e9e8fa Expose schedule entrypoints to all stores
Remote stores still override so the other end schedules.
2020-12-23 22:42:06 +00:00
John Ericson 450c3500f1 Crudely make worker only provide a Store, not LocalStore
We downcast in a few places, this will be refactored to be better later.
2020-12-23 22:42:06 +00:00
John Ericson 12f7a1f65b build-remote no longer requires local store be local 2020-12-23 22:42:06 +00:00
John Ericson 57062179ce Move some PKI stuff from LocalStore to Store 2020-12-23 22:42:06 +00:00
Eelco Dolstra a93916b190
Merge pull request #4336 from NixOS/manpages
Documentation for nix subcommands
2020-12-23 21:10:32 +01:00
Eelco Dolstra 5178211e96
Add 'nix' manpage 2020-12-23 18:33:42 +01:00
Eelco Dolstra 26e502ceb5
Add TODO 2020-12-23 18:26:40 +01:00
Eelco Dolstra 1047cb1e53
Command: Remove examples() 2020-12-23 18:26:40 +01:00
Eelco Dolstra c9279b831e
Add 'nix flake' manpages 2020-12-23 18:26:40 +01:00
Eelco Dolstra 8927cba62f
Merge pull request #4366 from NixOS/readInvalidDerivation-on-remote-caches
Use the fs accessor for readInvalidDerivation
2020-12-23 11:55:52 +01:00
Eelco Dolstra f4a9fb67da Merge branch 'git-rev-error' of https://github.com/Ma27/nix into master 2020-12-22 16:39:32 +01:00
Eelco Dolstra e27044216b Fix tests 2020-12-22 16:23:57 +01:00
Eelco Dolstra 75efa42134 Move <nix/fetchurl.nix> into the nix binary
This makes the statically linked nix binary just work, without needing
any additional files.
2020-12-22 14:43:20 +01:00
Eelco Dolstra 5373f4be3b chrootHelper: Handle symlinks in the root directory
This is necessary on Ubuntu where /bin and /lib* are symlinks.
2020-12-22 12:28:50 +01:00
Maximilian Bosch e54971d019
Document allRefs argument of builtins.fetchTree 2020-12-22 12:02:08 +01:00
Eelco Dolstra 724b7f4fb6 Don't log from inside the logger
This deadlocks ProgressBar, e.g.

  # nix run --impure --no-substitute --store '/tmp/nix2?store=/foo' --expr 'derivation { builder = /nix/store/zi90rxslsm4mlr46l2xws1rm94g7pk8p-busybox-1.31.1-x86_64-unknown-linux-musl/bin/busybox; }'

leads to

  Thread 1 (Thread 0x7ffff6126e80 (LWP 12250)):
  #0  0x00007ffff7215d62 in __lll_lock_wait () from /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libpthread.so.0
  #1  0x00007ffff720e721 in pthread_mutex_lock () from /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libpthread.so.0
  #2  0x00007ffff7ad17fa in __gthread_mutex_lock (__mutex=0x6c5448) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:749
  #3  std::mutex::lock (this=0x6c5448) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/std_mutex.h:100
  #4  std::unique_lock<std::mutex>::lock (this=0x7fffffff09a8, this=0x7fffffff09a8) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/unique_lock.h:141
  #5  std::unique_lock<std::mutex>::unique_lock (__m=..., this=0x7fffffff09a8) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/unique_lock.h:71
  #6  nix::Sync<nix::ProgressBar::State, std::mutex>::Lock::Lock (s=0x6c5448, this=0x7fffffff09a0) at src/libutil/sync.hh:45
  #7  nix::Sync<nix::ProgressBar::State, std::mutex>::lock (this=0x6c5448) at src/libutil/sync.hh:85
  #8  nix::ProgressBar::logEI (this=0x6c5440, ei=...) at src/libmain/progress-bar.cc:131
  #9  0x00007ffff7608cfd in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x6c5440) at src/libutil/logging.hh:88
  #10 nix::getCodeLines (errPos=...) at src/libutil/error.cc:66
  #11 0x00007ffff76073f2 in nix::showErrorInfo (out=..., einfo=..., showTrace=<optimized out>) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/optional:897
  #12 0x00007ffff7ad19e7 in nix::ProgressBar::logEI (this=0x6c5440, ei=...) at src/libmain/progress-bar.cc:134
  #13 0x00007ffff7ab9d10 in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x6c5440) at src/libutil/logging.hh:88
  #14 nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) (programName="/home/eelco/Dev/nix/outputs/out/bin/nix", fun=...) at src/libmain/shared.cc:328
  #15 0x000000000046226b in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/ext/new_allocator.h:80
2020-12-22 11:15:29 +01:00
Maximilian Bosch 2857b1baaf
Add explicit allRefs = true; argument to fetchGit
Sometimes it's necessary to fetch a git repository at a revision and
it's unknown which ref contains the revision in question. An example
would be a Cargo.lock which only provides the URL and the revision when
using a git repository as build input.

However it's considered a bad practice to perform a full checkout of a
repository since this may take a lot of time and can eat up a lot of
disk space. This patch makes a full checkout explicit by adding an
`allRefs` argument to `builtins.fetchGit` which fetches all refs if
explicitly set to true.

Closes #2409
2020-12-22 10:46:00 +01:00
Maximilian Bosch 629af83b2d
Provide a more meaningful error-message for builtins.fetchGit if a revision can't be checked out
A common pitfall when using e.g. `builtins.fetchGit` is the `fatal: not
a tree object`-error when trying to fetch a revision of a git-repository
that isn't on the `master` branch and no `ref` is specified.

In order to make clear what's the problem, I added a simple check
whether the revision in question exists and if it doesn't a more
meaningful error-message is displayed:

```
nix-repl> builtins.fetchGit { url = "https://github.com/owner/myrepo"; rev = "<commit not on master>"; }
moderror: --- Error -------------------------------------------------------------------- nix
Cannot find Git revision 'bf1cc5c648e6aed7360448a3745bb2fe4fbbf0e9' in ref 'master' of repository 'https://gitlab.com/Ma27/nvim.nix'! Please make sure that the rev exists on the ref you've specified or add allRefs = true; to fetchGit.
```

Closes #2431
2020-12-22 10:46:00 +01:00
Eelco Dolstra 16e34085e8
Add 'nix profile' manpage 2020-12-21 13:32:29 +01:00
Eelco Dolstra 0c09f63de8
Add 'nix bundle' manpage
Fixes #4375.
2020-12-21 13:32:29 +01:00
Eelco Dolstra f4e9d4fcb3
Add 'nix store diff-closures' manpage 2020-12-21 13:32:29 +01:00
Eelco Dolstra 4f3e7f4eec
Add 'nix show-derivation' manpage 2020-12-21 13:32:29 +01:00
Eelco Dolstra 3b123a6ee6
nix show-derivation: Say "system" instead of "platform"
There is really no good reason to use "platform" except that that's
what we use internally (also for no good reason).
2020-12-21 13:32:29 +01:00
Eelco Dolstra daf365b0b7
Add 'nix help' manpage 2020-12-21 13:32:29 +01:00
Eelco Dolstra e6bea9c9b1
Add 'nix store make-content-addressable' manpage 2020-12-21 13:32:29 +01:00
Eelco Dolstra cdf20e04b7
Doh 2020-12-21 13:32:29 +01:00
Eelco Dolstra 2e599dbb88
Add 'nix path-info' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra cb25a89f1c
Add 'nix store optimise' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra 8dd7d7e9db
Add 'nix store verify' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra 6b32551aba
Add 'nix upgrade-nix' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra 19540744ad
Add 'nix why-depends' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra c14ed3f8b2
Add 'nix store' NAR-related manpages 2020-12-21 13:32:28 +01:00
Eelco Dolstra 2cc02bbe76
Add 'nix nar' manpages 2020-12-21 13:32:28 +01:00
Eelco Dolstra a407d14339
Add 'nix eval' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra 53ce20eab7
Add 'nix store ping' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra f34b1801a4
Tweak 2020-12-21 13:32:28 +01:00
Eelco Dolstra 58bacc85e7
Add 'nix log' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra 6ce393392b
Add 'nix repl' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra b2262be19b
Add 'nix edit' manpage 2020-12-21 13:32:28 +01:00
Eelco Dolstra 4e065229c7
Typo 2020-12-21 13:32:27 +01:00
Eelco Dolstra e90e745232
Add 'nix registry' manpages
This also documents the registry format and matching/unification
semantics (though not quite correctly).
2020-12-21 13:32:27 +01:00
Eelco Dolstra 42cc98f8d6
Add 'nix develop' and `nix print-dev-env' manpages 2020-12-21 13:32:27 +01:00
Eelco Dolstra e9de689a6e
Add 'nix search' manpage 2020-12-21 13:32:27 +01:00
Eelco Dolstra 28ee307fd8
Add 'nix copy' manpage 2020-12-21 13:32:27 +01:00
Eelco Dolstra 09660b8557
Add 'nix run' and 'nix shell' manpages 2020-12-21 13:32:27 +01:00
Eelco Dolstra ae7351dbee
Add 'nix build' manpage 2020-12-21 13:32:27 +01:00
Eelco Dolstra 346baec783
Move doc() to Args 2020-12-21 13:32:23 +01:00
Eelco Dolstra 9fab14adbc
Merge pull request #4385 from obsidiansystems/store-subclass
Overhaul store subclassing
2020-12-21 12:51:36 +01:00
Eelco Dolstra ec4a5c5b0b
Merge pull request #4355 from Infinisil/private-value-type
Refactoring for private Value type
2020-12-21 12:38:47 +01:00
John Ericson 1a1af75338 Overhaul store subclassing
We embrace virtual the rest of the way, and get rid of the
`assert(false)` 0-param constructors.

We also list config base classes first, so the constructor order is
always:

  1. all the configs
  2. all the stores

Each in the same order
2020-12-20 15:47:14 +00:00
Silvan Mosberger b70d22baca
Replace Value type setters with mk* functions
Move clearValue inside Value

mkInt instead of setInt

mkBool instead of setBool

mkString instead of setString

mkPath instead of setPath

mkNull instead of setNull

mkAttrs instead of setAttrs

mkList instead of setList*

mkThunk instead of setThunk

mkApp instead of setApp

mkLambda instead of setLambda

mkBlackhole instead of setBlackhole

mkPrimOp instead of setPrimOp

mkPrimOpApp instead of setPrimOpApp

mkExternal instead of setExternal

mkFloat instead of setFloat

Add note that the static mk* function should be removed eventually
2020-12-18 21:48:22 +01:00
Silvan Mosberger 12e65078ef
Rename Value::normalType() -> Value::type() 2020-12-17 14:45:45 +01:00
Silvan Mosberger d67e02919c
Rename ValueType -> InternalType, NormalType -> ValueType
And Value::type to Value::internalType, such that type() can be used in
the next commit to get the new ValueType
2020-12-17 14:45:22 +01:00
regnat 4d45839499 Fix the detection of already built drv outputs
PRs #4370 and #4348 had a bad interaction in that the second broke the fist
one in a not trivial way.

The issue was that since #4348 the logic for detecting whether a
derivation output is already built requires some logic that was specific
to the `LocalStore`.

It happens though that most of this logic could be upstreamed to any `Store`,
which is what this commit does.
2020-12-17 11:35:24 +01:00
Eelco Dolstra ae3c3e3bb2
Merge pull request #4370 from NixOS/ca/more-precise-build-noop
Better detect when `buildPaths` would be a no-op
2020-12-16 14:54:04 +01:00
regnat cac8d5b742 Don't ignore an absent drv file in queryPartialDrvOutputMap
This ignore was here because `queryPartialDrvOutputMap` was used both
1. as a cache to avoid having to re-read the derivation (when gc-ing for
example), and
2. as the source of truth for ca realisations

The use-case 2. required it to be able to work even when the derivation
wasn't there anymore (see https://github.com/NixOS/nix/issues/4138).
However, this use-case is now handled by `queryRealisation`, meaning
that we can safely error out if the derivation isn't there anymore
2020-12-16 13:36:17 +01:00
Eelco Dolstra 3765174691
Merge pull request #4348 from NixOS/ca/use-hashmodulo
Use the hash modulo in the derivation outputs
2020-12-16 12:48:44 +01:00
regnat 962b82ef25 Fix BinaryCacheStore::registerDrvOutput
Was crashing because coercing a json document into a string is only
valid if the json is a string, otherwise we need to call `.dump()`
2020-12-16 10:54:36 +01:00
regnat 6e899278d3 Better detect when buildPaths would be a no-op
`buildPaths` can be called even for stores where it's not defined in case it's
bound to be a no-op.
The “no-op detection” mechanism was only detecting the case wher `buildPaths`
was called on a set of (non-drv) paths that were already present on the store.

This commit extends this mechanism to also detect the case where `buildPaths`
is called on a set of derivation outputs which are already built on the store.

This only works with the ca-derivations flag. It could be possible to
extend this to also work without it, but it would add quite a bit of
complexity, and it's not used without it anyways.
2020-12-16 10:36:16 +01:00
regnat 7080321618 Use the fs accessor for readInvalidDerivation
Extend `FSAccessor::readFile` to allow not checking that the path is a
valid one, and rewrite `readInvalidDerivation` using this extended
`readFile`.

Several places in the code use `readInvalidDerivation`, either because
they need to read a derivation that has been written in the store but
not registered yet, or more generally to prevent a deadlock because
`readDerivation` tries to lock the state, so can't be called from a
place where the lock is already held.
However, `readInvalidDerivation` implicitely assumes that the store is a
`LocalFSStore`, which isn't always the case.

The concrete motivation for this is that it's required for `nix copy
--from someBinaryCache` to work, which is tremendously useful for the
tests.
2020-12-15 20:10:46 +01:00
regnat 44c3fbc6e0 Fix addTextToStore for binary caches
Because of a too eager refactoring, `addTextToStore` used to throw an
error because the input wasn't a valid nar.

Partially revert that refactoring to wrap the text into a proper nar
(using `dumpString`) to make this method work again
2020-12-15 09:38:19 +01:00
Eelco Dolstra f2f60bf5d6
Merge pull request #4330 from NixOS/ca/properly-store-outputs
Properly store the outputs of CA derivations − take 2
2020-12-14 15:01:23 +01:00
Eelco Dolstra 27b5ff354e
Merge pull request #4351 from Ma27/json-errtrace
primops/fromJSON: add error position in case of parse error
2020-12-14 11:27:44 +01:00
Maximilian Bosch f890830b33
primops/fromJSON: add error position in case of parse error
This makes it easier to track down where invalid JSON was passed to
`builtins.fromJSON`.
2020-12-13 13:55:32 +01:00
Eelco Dolstra 92438c70d2
Merge pull request #4352 from jonringer/allow-private-caches
treat s3 permission errors as file-not-found
2020-12-13 13:39:20 +01:00
Silvan Mosberger 730b152b19
Make Value::type private
This is an implementation detail and shouldn't be used. Use normalType()
and the various is<Type> functions instead
2020-12-12 03:31:52 +01:00
Silvan Mosberger bf98903967
Add ValueType checking functions for types that have the same NormalType 2020-12-12 03:31:50 +01:00
Silvan Mosberger 22ead43a0b
Use Value::normalType on all forced values instead of Value::type 2020-12-12 03:31:48 +01:00
Silvan Mosberger 9f056f7afd
Introduce Value type setters and make use of them 2020-12-12 03:31:48 +01:00
Silvan Mosberger fa307875e9
Introduce NormalType for the normal type of a Value
This will be useful to abstract over the ValueType implementation
details

Make use of it already to replace the showType(ValueType) function
2020-12-12 03:31:46 +01:00
regnat e9b39f6004 Restrict the operations on drv outputs in recursive Nix
There's currently no way to properly filter them, so disallow them
altogether instead.
2020-12-11 21:17:25 +01:00
regnat bab1cda0e6 Use the hash modulo in the derivation outputs
Rather than storing the derivation outputs as `drvPath!outputName` internally,
store them as `drvHashModulo!outputName` (or `outputHash!outputName` for
fixed-output derivations).

This makes the storage slightly more opaque, but enables an earlier
cutoff in cases where a fixed-output dependency changes (but keeps the
same output hash) − same as what we already do for input-addressed
derivations.
2020-12-11 21:17:23 +01:00
regnat 8914e01e37 Store the realisations as JSON in the binary cache
Fix #4332
2020-12-11 21:05:09 +01:00
regnat 3ac9d74eb1 Rework the db schema for derivation outputs
Add a new table for tracking the derivation output mappings.

We used to hijack the `DerivationOutputs` table for that, but (despite its
name), it isn't a really good fit:

- Its entries depend on the drv being a valid path, making it play badly with
  garbage collection and preventing us to copy a drv output without copying
  the whole drv closure too;
- It dosen't guaranty that the output path exists;

By using a different table, we can experiment with a different schema better
suited for tracking the output mappings of CA derivations.
(incidentally, this also fixes #4138)
2020-12-11 20:41:32 +01:00
regnat 58cdab64ac Store metadata about drv outputs realisations
For each known realisation, store:
- its output
- its output path

This comes with a set of needed changes:

- New `realisations` module declaring the types needed for describing
  these mappings
- New `Store::registerDrvOutput` method registering all the needed informations
  about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
- new `Store::queryRealisation` method to retrieve the informations for a
  derivations

This introcudes some redundancy on the remote-store side between
`wopQueryDerivationOutputMap` and `wopQueryRealisation`.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where `wopQueryDerivationOutputMap` is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
2020-12-11 20:41:32 +01:00
Michael Bishop 63b3536f50
treat s3 permission errors as file-not-found
Signed-off-by: Jonathan Ringer <jonringer117@gmail.com>
2020-12-11 09:49:24 -08:00
Eelco Dolstra 9c143c411b
Merge pull request #4350 from NixOS/ca/fix-build-with-nix-command
Fix the `nix` command with CA derivations
2020-12-11 15:01:49 +01:00
regnat eb45308109 Fix the nix command with CA derivations
Prevents a crash because most `nix` subcommands assumed that derivations
know their output path, which isn't the case for CA derivations
2020-12-11 10:28:09 +01:00
Eelco Dolstra c6a1bcd0ec nix store make-content-addressable: Show rewritten path 2020-12-10 17:11:56 +01:00
Eelco Dolstra a8f533b664 Add lvlNotice log level
This is like syslog's LOG_NOTICE: "normal, but significant,
condition".
2020-12-10 16:41:24 +01:00
Eelco Dolstra 253571e4ec
Merge pull request #4342 from tweag/fix-remote-build-hook
fix remote build hook
2020-12-09 12:40:00 +01:00
Maximilian Bosch 93a8a005de
libstore/openStore: fix stores with IPv6 addresses
In `nixStable` (2.3.7 to be precise) it's possible to connect to stores
using an IPv6 address:

  nix ping-store --store ssh://root@2001:db8::1

This is also useful for `nixops(1)` where you could specify an IPv6
address in `deployment.targetHost`.

However, this behavior is broken on `nixUnstable` and fails with the
following error:

  $ nix store ping --store ssh://root@2001:db8::1
  don't know how to open Nix store 'ssh://root@2001:db8::1'

This happened because `openStore` from `libstore` uses the `parseURL`
function from `libfetchers` which expects a valid URL as defined in
RFC2732. However, this is unsupported by `ssh(1)`:

  $ nix store ping --store 'ssh://root@[2001:db8::1]'
  cannot connect to 'root@[2001:db8::1]'

This patch now allows both ways of specifying a store (`root@2001:db8::1`) and
also `root@[2001:db8::1]` since the latter one is useful to pass query
parameters to the remote store.

In order to achieve this, the following changes were made:

* The URL regex from `url-parts.hh` now allows an IPv6 address in the
  form `2001:db8::1` and also `[2001:db8::1]`.

* In `libstore`, a new function named `extractConnStr` ensures that a
  proper URL is passed to e.g. `ssh(1)`:

  * If a URL looks like either `[2001:db8::1]` or `root@[2001:db8::1]`,
    the brackets will be removed using a regex. No additional validation
    is done here as only strings parsed by `parseURL` are expected.

  * In any other case, the string will be left untouched.

* The rules above only apply for `LegacySSHStore` and `SSHStore` (a.k.a
  `ssh://` and `ssh-ng://`).

Unresolved questions:

* I'm not really sure whether we want to allow both variants of IPv6
  addresses in the URL parser. However it should be noted that both seem
  to be possible according to RFC2732:

  > This document incudes an update to the generic syntax for Uniform
  > Resource Identifiers defined in RFC 2396 [URL].  It defines a syntax
  > for IPv6 addresses and allows the use of "[" and "]" within a URI
  > explicitly for this reserved purpose.

* Currently, it's not supported to specify a port number behind the
  hostname, however it seems as this is not really supported by the URL
  parser. Hence, this is probably out of scope here.
2020-12-09 12:23:29 +01:00
regnat c87267c2a4 Store the final drv outputs in memory when building remotely
The `DerivationGoal` has a variable storing the “final” derivation
output paths that is used (amongst other things) to fill the environment
for the post build hook. However this variable wasn't set when the
build-hook is used, causing a crash when both hooks are used together.

Fix this by setting this variable (from the informations in the db) after a run
of the post build hook.
2020-12-09 10:45:12 +01:00
regnat 6758e65612 Revert "Re-query for the derivation outputs in the post-build-hook"
This reverts commit 1b1e076033.

Using `queryPartialDerivationOutputMap` assumes that the derivation
exists locally which isn't the case for remote builders.
2020-12-09 09:44:07 +01:00
Eelco Dolstra 82e5511594
Merge pull request #4325 from tweag/hide-local-store-sql-statements
Hide the sqlite statements declarations for the local store
2020-12-08 14:04:16 +01:00
regnat c0f21f08f8 Hide the sqlite statements declarations for the local store
These have no need to be in the public interface and it causes spurious
rebuilds each time one wants to add or remove a new statement.
2020-12-08 13:29:13 +01:00
regnat 1b1e076033 Re-query for the derivation outputs in the post-build-hook
We can't assume that the runtime state knows about them as they might have
been built remotely, in which case we must query the db again to get
them.
2020-12-08 11:11:02 +01:00
Matthew Bauer aa07502009 Always default to cache.nixos.org even when different nix store dir
Since 0744f7f, it is now useful to have cache.nixos.org in substituers
even if /nix/store is not the Nix Store Dir. This can always be
overridden via configuration, though.
2020-12-06 23:04:42 -06:00
Eelco Dolstra 1d1a85eb0a
Merge pull request #4318 from matthewbauer/add-slash-to-trusted-binary-cache
Canonicalize binary caches with ‘/’ when one is missing
2020-12-05 10:17:17 +01:00
Matthew Bauer b9a00fd15b Canonicalize binary caches with ‘/’ when one is missing
This checks if there is a trusted substituter with a slash, so
trusting https://cache.nixos.org also implies https://cache.nixos.org/
is trusted.
2020-12-04 22:17:19 -06:00
Matthew Bauer 3c9b7029ba Use com.apple.oahd.plist for rosetta 2 detection 2020-12-04 13:26:53 -06:00
Stéphan Kochen e20a3ec756 Fix compatibility with newer AWS SDKs
Tested against AWS SDK 1.8.99. Fixes #3201.
2020-12-04 19:36:09 +01:00
Eelco Dolstra f337aa7099 Split 'nix store add-to-store' into 'add-path' and 'add-file'
This makes it consistent with 'nix hash <path|file>'.
2020-12-04 00:59:24 +01:00
Eelco Dolstra fa8dad10ed Typo 2020-12-03 23:26:23 +01:00
Eelco Dolstra ea2062a2d9 Move most store-related commands to 'nix store' 2020-12-03 23:22:22 +01:00
Eelco Dolstra a1cd805cba Add 'nix nar dump-path'
This only differs from 'nix store dump-path' in that the path doesn't
need to be a store path.
2020-12-03 22:52:01 +01:00
Eelco Dolstra af373c2ece Add deprecated aliases for renamed commands 2020-12-03 22:45:44 +01:00
Matthew Bauer 4b9acf4e21 Use posix_spawn_setbinpref_np to advise which architecture to run
When running universal binaries like /bin/bash, Darwin XNU will choose
which architecture of the binary to use based on "binary preferences".
This change sets that to the current platform for aarch64 and x86_64
builds. In addition it now uses posix_spawn instead of the usual
execve. Note, that this does not prevent the other architecture from
being run, just advises which to use.

Unfortunately, posix_spawnattr_setbinpref_np does not appear to be
inherited by child processes in x86_64 Rosetta 2 translations, meaning
that this will not always work as expected.

For example:

  {
    arm = derivation {
      name = "test";
      system = "aarch64-darwin";
      builder = "/bin/bash";
      args = [ "-e" (builtins.toFile "test" ''
        set -x
        /usr/sbin/sysctl sysctl.proc_translated
        /usr/sbin/sysctl sysctl.proc_native
        [ "$(/usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };
    rosetta = derivation {
      name = "test";
      system = "x86_64-darwin";
      builder = "/bin/bash";
      args = [ "-e" (builtins.toFile "test" ''
        set -x
        /usr/sbin/sysctl sysctl.proc_translated
        /usr/sbin/sysctl sysctl.proc_native
        [ "$(/usr/bin/arch)" = i386 ]
        echo It works!
        /usr/bin/touch $out
      '') ];
    };
  }

`arm' fails on x86_64-compiled Nix, but `arm' and `rosetta' succeed on
aarch64-compiled Nix. I suspect there is a way to fix this since:

  $ /usr/bin/arch -arch x86_64 /bin/bash \
    -c '/usr/bin/arch -arch arm64e /bin/bash -c /usr/bin/arch'
  arm64

seems to work correctly. We may need to wait for Apple to update
system_cmds in opensource.apple.com to find out how though.
2020-12-03 15:41:59 -06:00
Matthew Bauer 9b1824ecbd Add extraPlatforms for Rosetta 2 macOS
macOS systems with ARM64 can utilize a translation layer at
/Library/Apple/usr/libexec/oah to run x86_64 binaries. This change
makes Nix recognize that and it to "extra-platforms". Note that there
are two cases here since Nix could be built for either x86_64 or
aarch64. In either case, we can switch to the other architecture.
Unfortunately there is not a good way to prevent aarch64 binaries from
being run in x86_64 contexts or vice versa - programs can always
execute programs for the other architecture.
2020-12-03 15:41:43 -06:00
Eelco Dolstra 0c15ae5d4b Add FIXME 2020-12-03 20:31:45 +01:00
Eelco Dolstra ef583303f0 Move NAR-related commands to 'nix nar' 2020-12-03 18:09:02 +01:00
Eelco Dolstra 79c1967ded Introduce 'nix store' command 2020-12-03 18:07:13 +01:00
Eelco Dolstra 5781f45c46 Allow registering subcommands of subcommands 2020-12-03 17:55:55 +01:00
Eelco Dolstra b2d6c6161e Move 'nix hash-*' and 'nix to-*' to 'nix hash'
From the 'nix' UX review.
2020-12-03 17:55:55 +01:00
Eelco Dolstra 8ad2c9c4b9 Remove 'dist' target
We're not producing source tarballs anymore so this has been
bitrotting.
2020-12-03 16:17:58 +01:00
Eelco Dolstra 1b0ca3866b nix add-to-store: Move markdown docs into a separate file 2020-12-03 16:17:58 +01:00
Eelco Dolstra c3c858ac6d Make doc() return arbitrary Markdown rather than the contents of the "Description" section
Thus we can return the examples section (and any other sections) from
doc() and don't need examples() anymore.
2020-12-03 16:17:58 +01:00
Eelco Dolstra 0bd060f23a
Merge pull request #4308 from tweag/properly-test-early-cutoff
Properly test the early cutoff for CA derivations
2020-12-03 14:45:29 +01:00
Eelco Dolstra 4f25644a13
Merge pull request #4304 from NixOS/separate-manpages
Separate manpages for 'nix' subcommands
2020-12-03 13:38:29 +01:00
regnat 0afab668fa Don't fail early when -j0 is passed
If the build closure contains some CA derivations, then we can't know
ahead-of-time that we won't build anything as early-cutoff might come-in
at a laster stage
2020-12-03 13:24:36 +01:00
Eelco Dolstra df552a2645 nix eval: Add option to write a directory
This is useful for generating the nix manpages, but it may have other
applications (like generating configuration files without a Nix store).
2020-12-02 23:23:23 +01:00
Eelco Dolstra 148608ba6d Add 'nix help' 2020-12-02 23:23:23 +01:00
Greg Hale d8fc1bb7b0 fix tokens documentation 2020-12-02 10:15:18 -05:00
regnat 0d9e1af695 Remove an unknown pragma gcc warning 2020-12-02 14:33:20 +01:00
regnat a8a96dbaf8 Add forgotten override annotation 2020-12-02 14:23:38 +01:00
Eelco Dolstra 1b79b5b983 read(): Use char * instead of unsigned char *
This gets rid of some pointless casts.
2020-12-02 14:17:27 +01:00
Eelco Dolstra faa31f4084 Sink: Use std::string_view 2020-12-02 14:17:27 +01:00
Eelco Dolstra aa68486112 writeFull/writeFile: Use std::string_view 2020-12-02 14:17:27 +01:00
Eelco Dolstra e5cf501c77
Merge pull request #4284 from tweag/fixed-output-depending-on-ca
Allow fixed-output derivations to depend on (floating) content-addressed ones
2020-12-01 20:25:41 +01:00
regnat 438977731c shut up clang warnings
- Fix some class/struct discrepancies
- Explicit the overloading of `run` in the `Cmd*` classes
- Ignore a warning in the generated lexer
2020-12-01 15:04:03 +01:00
Eelco Dolstra 88798613ee replaceStrings(): Use std::string_view 2020-12-01 13:45:43 +01:00
Eelco Dolstra c0d1354b7d Macro hygiene 2020-12-01 13:45:06 +01:00
Eelco Dolstra 3b7e00ce22 Move primeCache() to Worker::run()
We need the missing path info to communicate the worker's remaining
goals to the progress bar.
2020-12-01 13:44:48 +01:00
Eelco Dolstra 5927624473 Lower verbosity for 'Failed to find a machine' message 2020-12-01 13:43:36 +01:00
Eelco Dolstra e224c16d28 Macro hygiene 2020-12-01 13:43:33 +01:00
regnat 9bd8184f1f Allow fixed-output derivations to depend on (floating) content-addressed ones
Fix an overlook of https://github.com/NixOS/nix/pull/4056
2020-11-27 15:39:24 +01:00
Eelco Dolstra 05d9442f68 builtins.fetchGit: Fix shortRev attribute for dirty trees 2020-11-26 21:45:28 +01:00
Dominique Martinet 1fd13d67e8 archive: disable preallocate-contents by default
using fallocate() to preallocate files space does more harm than good:
 - breaks compression on btrfs
 - has been called "not the right thing to do" by xfs developers
(because delayed allocation that most filesystems implement leads to smarter
allocation than what the filesystem needs to do if we upfront fallocate files)
2020-11-26 14:26:57 +01:00
Eelco Dolstra 8252a44e96 Move to separate file 2020-11-26 13:16:36 +01:00
Eelco Dolstra 9a586e34ac Record trusted/untrusted settings in ~/.local/share/nix 2020-11-26 13:11:07 +01:00
Eelco Dolstra 0287f83057 Ask for confirmation before allowing flake Nix configuration settings 2020-11-26 12:37:23 +01:00
regnat 13c557fe82 fix the hash rewriting for ca-derivations 2020-11-25 11:33:00 +01:00
Lily Ballard 437189e446 Escape filename given to nix-shell in shebang mode
This prevents spaces or other metacharacters from causing nix-shell to
execute the wrong path.

Fixes #4229.
2020-11-24 15:08:37 -08:00
Luke Granger-Brown 226116f482 fetchMercurial: set HGPLAIN when invoking hg
Without setting HGPLAIN, the user's environment leaks into
hg invocations, which means that the output may not be in the
expected format.

HGPLAIN is the Mercurial-recommended solution for this in that
it's intended for uses by scripts and programs which are looking
to parse Mercurial's output in a consistent manner.
2020-11-23 16:12:33 +00:00
Wil Taylor 07603890d2 Removed reference check from bundler command 2020-11-23 21:19:40 +10:00
Wil Taylor c3bad73e27 Added switch 2020-11-21 14:28:49 +10:00
Eelco Dolstra 4dcb183af3 AttrCursor::getStringWithContext(): Force re-evaluation if the cached context is not valid
Fixes #4236.
2020-11-19 20:59:36 +01:00
Eelco Dolstra 0327580e54 Fix assertion failure in LockFile::LockFile()
Fixes #4241.
2020-11-19 20:31:30 +01:00
John Ericson 2113ae2d85 Make drv hash modulo memo table thread-safe
Let's get one step closer to the daemon not needing to fork.
2020-11-19 16:50:06 +00:00
Eelco Dolstra 4dbd05e933
Merge pull request #4189 from edolstra/flake-config
Allow nix.conf options to be set in flake.nix
2020-11-17 15:39:14 +01:00
Eelco Dolstra f4e790cc85
Merge pull request #4182 from mkenigs/fix-1930
Print built derivations as json for build
2020-11-17 14:59:49 +01:00
Eelco Dolstra df5c69a94e
Merge pull request #4180 from Ma27/ssh-ng-substitute
Allow substituting paths when building remotely using `ssh-ng://`
2020-11-17 14:01:04 +01:00
Eelco Dolstra e6b7c7b79c Cleanup 2020-11-17 13:58:55 +01:00
Eelco Dolstra bccff827dc Fix deadlock in IFD through the daemon
Fixes #4235.
2020-11-17 13:50:36 +01:00
regnat 7de21f6664 Make the sql debug statements more useful
Print the expanded sql query (with the variables bound to their value) rather
than the original one in case of error
2020-11-17 10:05:45 +01:00
Eelco Dolstra ef84c780bb
filterANSIEscapes(): Handle UTF-8 characters 2020-11-16 16:41:53 +01:00
Eelco Dolstra 0d6419ad87
Merge pull request #4262 from jbaum98/master
Fix deadlock in nix-store when max-connections=1
2020-11-16 10:20:36 +01:00
Jake Waksbaum 01db455733 Fix deadlock in nix-store when max-connections=1
This fixes a bug I encountered where `nix-store -qR` will deadlock when
the `--include-outputs` flag is passed and `max-connections=1`.

The deadlock occurs because `RemoteStore::queryDerivationOutputs` takes
the only connection from the connection pool and uses it to check the
daemon version. If the version is new enough, it calls
`Store::queryDerivationOutputs`, which eventually calls
`RemoteStore::queryPartialDerivationOutputMap`, where we take another
connection from the connection pool to check the version again. Because
we still haven't released the connection from the caller, this waits for
a connection to be available, causing a deadlock.

This diff solves the issue by using `getProtocol` to check the protocol
version in the caller `RemoteStore::queryDerivationOutputs`, which
immediately frees the connection back to the pool before returning the
protocol version. That way we've already freed the connection by the
time we call `RemoteStore::queryPartialDerivationOutputMap`.
2020-11-16 02:35:50 -05:00
Robert Hensing ac5081d280 nix-build: Fix #4197 output order regression 2020-11-13 17:49:27 +01:00
Christian Höppner c4c3c15c19 Fix default nix-path
The default nix-path values for nixpkgs and root channels were
incorrect.
2020-11-12 15:46:08 +00:00
Ben Burdette b327de9c2d change message 2020-11-11 11:09:59 -07:00
Ben Burdette 8895fa70a4 pare down the error message 2020-11-11 11:05:21 -07:00
Ben Burdette 3edfe6090e missing argument error 2020-11-11 09:29:32 -07:00
Matthew Kenigsberg 8abb80a478 Print built derivations as json for build
Add --json option to nix build to allow machine readable output on
stdout with all built derivations

Fixes #1930
2020-11-11 10:27:02 -06:00
Ben Burdette 7d9037035e usage example location 2020-11-11 09:21:26 -07:00
Eelco Dolstra 4badb6943f Fix use of dirty Git/Mercurial inputs with chroot stores
Fixes:

  $ nix build --store /tmp/nix /home/eelco/Dev/patchelf#hydraJobs.build.x86_64-linux
  warning: Git tree '/home/eelco/Dev/patchelf' is dirty
  error: --- RestrictedPathError ------------------------------------------------------------------------------------------- nix
  access to path '/tmp/nix/nix/store/xmkvfmffk7xfnazykb5kx999aika8an4-source/flake.nix' is forbidden in restricted mode
  (use '--show-trace' to show detailed location information)
2020-11-10 23:22:45 +01:00
Eelco Dolstra cdc840d60b
Merge pull request #4242 from wizeman/fix-stack-overflow
Fix stack overflow introduced in #4206
2020-11-10 11:51:00 +01:00
Eelco Dolstra 3f680c1dcc
Merge pull request #4233 from Kha/master
nix develop: Preserve stdin with `-c`
2020-11-10 10:51:48 +01:00
Ricardo M. Correia 108a2dab7e Fix stack overflow introduced in #4206 2020-11-10 04:25:24 +01:00
Ben Burdette d8ef423a18 error message formatting 2020-11-09 19:16:50 -07:00
Ben Burdette 9f2b25ce55 remove unused ftn; reformat line breaks 2020-11-09 17:17:47 -07:00
Ben Burdette 6c2933a8d7 add position 2020-11-09 17:04:52 -07:00
Ben Burdette 107c91f5fe auto-call error 2020-11-09 16:48:35 -07:00
Sebastian Ullrich fb7735e4cf nix develop: Preserve stdin with -c 2020-11-09 22:43:14 +01:00
Eelco Dolstra b87f84cf55
Fix appending to Setting<StringSet>
Fixes: warning: unknown setting 'extra-sandbox-paths'
2020-11-09 15:04:34 +01:00
Maximilian Bosch 3a63fc6cd5
Allow substituting paths when building remotely using ssh-ng://
Until now, it was not possible to substitute missing paths from e.g.
`https://cache.nixos.org` on a remote server when building on it using
the new `ssh-ng` protocol.

This is because every store implementation except legacy `ssh://`
ignores the substitution flag passed to `Store::queryValidPaths` while
the `legacy-ssh-store` substitutes the remote store using
`cmdQueryValidPaths` when the remote store is opened with `nix-store
--serve`.

This patch slightly modifies the daemon protocol to allow passing an
integer value suggesting whether to substitute missing paths during
`wopQueryValidPaths`. To implement this on the daemon-side, the
substitution logic from `nix-store --serve` has been moved into a
protected method named `Store::substitutePaths` which gets currently
called from `LocalStore::queryValidPaths` and `Store::queryValidPaths`
if `maybeSubstitute` is `true`.

Fixes #2770
2020-11-05 20:12:37 +01:00
Eelco Dolstra 387f824cab
Merge pull request #4206 from hercules-ci/fix-coroutine-gc
Fix memory corruption caused by GC-invisible coroutine stacks
2020-11-05 10:18:31 +01:00
Eelco Dolstra 5e6eabe155 Fix error message 'assertion failed at' 2020-11-03 14:45:24 +01:00
Eelco Dolstra e8c379555f LocalStore: Get rid of recursive_mutex 2020-11-03 14:45:24 +01:00
Eelco Dolstra 797a52e31d Add FIXME 2020-11-03 14:45:24 +01:00
Eelco Dolstra 550e11f077 nix repl: Fix handling of multi-line expressions 2020-11-02 19:07:37 +01:00
Eelco Dolstra 7cf874c17d Don't use readDerivation() in addValidPath()
readDerivation() requires a valid path.

Fixes #4210.
2020-11-02 18:46:44 +01:00
Eelco Dolstra ab2ef851b6
Merge pull request #4207 from hercules-ci/fix-RemoteStore-filterSource-deadlock
Fix RemoteStore pool deadlock in filterSource etc
2020-11-02 14:46:10 +01:00
Eelco Dolstra db5424bf09 Don't send eval-related settings to the daemon 2020-11-02 13:57:58 +01:00
Robert Hensing e8a45d07bc Restore RestrictedStore.addToStoreFromDump implementation
It was accidentally removed in commit ca30abb3fb
2020-10-31 23:56:03 +01:00
Robert Hensing b43c13a916 BoehmGCStackAllocator: increase stack size to 8MB
The default stack size was not based on the normal stack size and
was too small.
2020-10-30 23:18:26 +01:00
Robert Hensing 2192cac634 Fix RemoteStore pool deadlock in filterSource etc 2020-10-30 21:47:34 +01:00
Robert Hensing c4d903ddb0 Fix memory corruption caused by GC-invisible coroutine stacks
Crucially this introduces BoehmGCStackAllocator, but it also
adds a bunch of wiring to avoid making libutil depend on bdw-gc.

Part of the solutions for #4178, #4200
2020-10-30 21:21:59 +01:00
Eelco Dolstra dc5696b84f Fix test 2020-10-30 12:00:53 +01:00
Eelco Dolstra 7f56cf67ba Fix assertion failure in tab completion for --option 2020-10-29 18:26:35 +01:00
Eelco Dolstra ff4dea63c9 Generalize extra-* settings
This removes the extra-substituters and extra-sandbox-paths settings
and instead makes every array setting extensible by setting
"extra-<name> = <value>" in the configuration file or passing
"--<name> <value>" on the command line.
2020-10-29 18:17:39 +01:00
Eelco Dolstra bb8e837e4c
Merge pull request #4199 from stefanjaax/replaceWantedBySpecified
Alter "wanted:" to "specified:" in hash mismatch output
2020-10-29 07:38:38 +01:00
Eelco Dolstra 662e67f8de
Merge pull request #4198 from mkenigs/capitalize-JSON
Capitalize JSON for consistency
2020-10-29 07:34:34 +01:00
stev 869c0321ff Alter "wanted:" to "specified:" in hash mismatch output
This makes it even clearer which of the two hashes was specified in the
nix files. Some may think that "wanted" and "got" is obvious, but:
"got" could mean "got in nix file" and "wanted" could mean "want to see in nix file".
2020-10-29 00:33:14 +01:00
Matthew Kenigsberg 6a4bf535d8 Capitalize JSON for consistency 2020-10-28 17:54:28 -05:00
Eelco Dolstra a5019f0508 Consistency 2020-10-28 20:45:57 +01:00
Eelco Dolstra 02a1facbdc
Merge pull request #4056 from tweag/non-ca-depending-on-ca
Allow non-CA derivations to depend on CA ones
2020-10-27 17:38:29 +01:00
regnat bc081bcd81 Inline unkownHashes
See https://github.com/NixOS/nix/pull/4056#discussion_r493661632
2020-10-27 07:29:25 +01:00
regnat c092fa4702 Allow non-CA derivations to depend on CA derivations 2020-10-27 07:29:23 +01:00
Eelco Dolstra 343239fc8a Allow nix.conf options to be set in flake.nix
This makes it possible to have per-project configuration in flake.nix,
e.g. binary caches and other stuff:

  nixConfig.bash-prompt-suffix = "ngi# ";
  nixConfig.substituters = [ "https://cache.ngi0.nixos.org/" ];
2020-10-26 20:45:39 +01:00
Eelco Dolstra 731edf0d9b isTrivial(): Support trivial lists 2020-10-26 20:37:11 +01:00
Eelco Dolstra 14aecbb288 BaseSetting<StringMap>::set(): Don't append to previous value 2020-10-26 20:36:46 +01:00
Eelco Dolstra b875b8f45c Remove edition field 2020-10-26 17:59:36 +01:00
Eelco Dolstra 9d5e9ef0da Move Explicit 2020-10-26 17:01:20 +01:00
Eelco Dolstra dc7d1322ef Make the prompt used in development shells configurable 2020-10-26 14:24:25 +01:00
Eelco Dolstra 750ce500c2 Fix clang build 2020-10-22 13:40:30 +02:00
Eelco Dolstra f9438fb64a nix develop: Add --redirect flag to redirect dependencies
This is primarily useful if you're hacking simultaneously on a package
and one of its dependencies. E.g. if you're hacking on Hydra and Nix,
you would start a dev shell for Nix, and then a dev shell for Hydra as
follows:

  $ nix develop \
    --redirect .#hydraJobs.build.x86_64-linux.nix ~/Dev/nix/outputs/out \
    --redirect .#hydraJobs.build.x86_64-linux.nix.dev ~/Dev/nix/outputs/dev

(This assumes hydraJobs.build.x86_64-linux has a passthru.nix
attribute. You can also use a store path.)

This causes all references in the environment to those store paths to
be rewritten to ~/Dev/nix/outputs/{out,dev}. Note: unfortunately, you
may need to set LD_LIBRARY_PATH=~/Dev/nix/outputs/out/lib because
Nixpkgs' ld-wrapper only adds -rpath entries for -L flags that point
to the Nix store.
2020-10-22 13:40:30 +02:00
Eelco Dolstra e556a1beb7 nix develop: Handle 'declare -ax' in bash output
Fixes 'nix develop nixpkgs#qpdfview'.
2020-10-21 17:54:21 +02:00
Christian Kampka 461cf2b856
Add NIX_CONFIG env var for applying nix.conf overrides 2020-10-21 13:41:26 +02:00
Matthew Kenigsberg f6aaac2b59 Make bash non-interactive for nix develop --phase
Fix #3975: Currently if Ctrl-C is pressed during a phase, the interactive subshell
is not exited. Removing --rcfile when --phase is present makes bash
non-interactive
2020-10-20 12:00:30 -05:00
Eelco Dolstra 9635fb77bd
Merge pull request #4080 from kquick/kwq/flake-int-doc
Add some internal documentation for flake support objects.
2020-10-19 11:29:12 +02:00
Eelco Dolstra fda835b231
Merge pull request #4143 from obsidiansystems/typed-goal-maps
Properly type the derivation and substitution goal maps
2020-10-18 18:12:21 +02:00
Robert Hensing bd9eb5c743 DerivationGoal: only retry if output closure incomplete is only problem 2020-10-18 14:26:37 +02:00
Robert Hensing 94f1e4a441 Typo 2020-10-18 14:26:37 +02:00
John Ericson 7ed46c1574 Explain that upcast_goal is still a static cast 2020-10-17 21:50:12 +00:00
John Ericson 57d0432b39 Just use auto in two places. 2020-10-17 21:47:52 +00:00
John Ericson e6f8ae56d8 tab -> space 2020-10-17 21:45:31 +00:00
John Ericson 619d262c97 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-17 21:44:27 +00:00
John Ericson 64be1c15c2 Add missing include for MAX_PATH
And remove one that we didn't actually need to add
2020-10-15 19:05:17 +00:00
John Ericson fccef6a7fa Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-10-15 18:55:03 +00:00
John Ericson 0fefc2a439 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-14 20:49:01 +00:00
Eelco Dolstra 11882d7c7c Create /etc/passwd *after* figuring out the sandbox uid/gid
Fixes build failures like

  # nix log /nix/store/gjaa0psfcmqvw7ivggsncx9w364p3s8s-sshd.conf-validated.drv
  No user exists for uid 30012
2020-10-14 12:20:58 +02:00
John Ericson 55592b253f Add some more docs 2020-10-13 18:04:24 +00:00
John Ericson 13804f126e Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-13 18:02:32 +00:00
Eelco Dolstra 2653801939 Merge branch 'split_build_cc' of https://github.com/obsidiansystems/nix 2020-10-13 15:36:55 +02:00
John Ericson 1b8ebe92dc Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps 2020-10-12 20:47:22 +00:00
John Ericson a73959e6be Merge remote-tracking branch 'upstream/master' into split_build_cc 2020-10-12 17:20:46 +00:00
John Ericson d334fd4882 Merge branches 'split_build_hh-0', 'split_build_hh-1', 'split_build_hh-2', 'split_build_hh-3' and 'split_build_hh-4' into split_build_cc 2020-10-12 17:20:12 +00:00
John Ericson 542972f029 Trim worker.hh 2020-10-12 17:16:48 +00:00
John Ericson 0e2306204a Rename to hand-hold git (worker.hh) 2020-10-12 17:16:48 +00:00
John Ericson 3ffa3546bd Trim substitution-goal.hh 2020-10-12 17:16:25 +00:00
John Ericson e77a2344d5 Rename to hand-hold git (substitution-goal.hh) 2020-10-12 17:16:25 +00:00
John Ericson d585b4c54f Trim hook-instance.hh 2020-10-12 17:16:13 +00:00
John Ericson 10b749a156 Rename to hand-hold git (hook-instance.hh) 2020-10-12 17:16:13 +00:00
John Ericson 8067d32f2a Trim goal.hh 2020-10-12 17:16:00 +00:00
John Ericson 0d0e345cdc Rename to hand-hold git (goal.hh) 2020-10-12 17:16:00 +00:00
John Ericson 2ce726947a Trim derivation-goal.hh 2020-10-12 17:15:32 +00:00
John Ericson 4eb8c69853 Rename to hand-hold git (derivation-goal.hh) 2020-10-12 17:15:32 +00:00
John Ericson f7099965bf Change .cc files to use split build headers 2020-10-12 17:08:52 +00:00
John Ericson 5a97621d6d Prepare for build/*.hh headers 2020-10-12 17:07:51 +00:00
volth eee18f88dd
Handle amount of disk space saved by hard linking being negative
Fixes bogus messages like "currently hard linking saves 17592186044416.00 MiB".
2020-10-12 16:06:38 +00:00
John Ericson 15fdb7cc6b Split out uds-remote-store.{cc.hh} 2020-10-11 17:37:05 +00:00
John Ericson 38e3897162 Copy {uds-,}remote-store.{cc,hh}
This prepares for the splitting that happens in the next commit.
2020-10-11 17:18:26 +00:00
John Ericson 5c74a6147b Properly type the derivation and substitution goal maps
As a bonus, Worker::removeGoal is less inefficient.
2020-10-11 17:07:14 +00:00
Eelco Dolstra b0fbf3a653
Merge pull request #4135 from obsidiansystems/split-local-fs-store-header
Split out `local-fs-store.hh`
2020-10-11 18:49:40 +02:00
John Ericson 8cc510fb79 Merge branches 'build-1', 'build-2', 'build-3', 'build-4', 'build-5', 'build-6', 'build-7' and 'build-8' into split_build_cc 2020-10-11 16:44:34 +00:00
John Ericson bcb67e1ed8 Trim lock.cc 2020-10-11 16:44:19 +00:00
John Ericson e0be04129b Rename to hand-hold git (lock.cc) 2020-10-11 16:44:14 +00:00
John Ericson dbc588651c Trim lock.hh 2020-10-11 16:44:07 +00:00
John Ericson d0004bfcab Rename to hand-hold git (lock.hh) 2020-10-11 16:43:59 +00:00
John Ericson eed53ed87a Trim build/worker.cc 2020-10-11 16:43:51 +00:00
John Ericson 904e315dae Rename to hand-hold git (build/worker.cc) 2020-10-11 16:43:42 +00:00
John Ericson d24ffe0eb1 Trim build/substitution-goal.cc 2020-10-11 16:43:24 +00:00
John Ericson 3633b3572b Rename to hand-hold git (build/substitution-goal.cc) 2020-10-11 16:43:18 +00:00
John Ericson 4bdff7d1b0 Trim build/local-store-build.cc 2020-10-11 16:43:12 +00:00
John Ericson dc5225cde5 Rename to hand-hold git (build/local-store-build.cc) 2020-10-11 16:43:05 +00:00
John Ericson 159054f730 Trim build/hook-instance.cc 2020-10-11 16:42:35 +00:00
John Ericson f0b8987299 Rename to hand-hold git (build/hook-instance.cc) 2020-10-11 16:42:08 +00:00
John Ericson 819fe848ac Trim build/goal.cc 2020-10-11 16:41:58 +00:00
John Ericson 184bfc301e Rename to hand-hold git (build/goal.cc) 2020-10-11 16:41:18 +00:00
John Ericson 3bab1c5bb0 Trim build/derivation-goal.cc 2020-10-11 16:41:11 +00:00
John Ericson 9629290eda Rename to hand-hold git (build/derivation-goal.cc) 2020-10-11 16:40:52 +00:00
John Ericson a4f0fecb03 Trim build.hh 2020-10-11 16:40:34 +00:00
John Ericson fc72cb0760 Rename to hand-hold git (build.hh) 2020-10-11 16:40:14 +00:00
John Ericson 428536fd75 Prepare for build/* files 2020-10-11 16:39:08 +00:00
John Ericson aef44cbaa9 Split out commonChildInit 2020-10-11 16:38:46 +00:00
John Ericson 6cc1541782 Split out local-fs-store.hh
This matches the already-existing `local-fs-store.cc`.
2020-10-09 20:18:08 +00:00
Eelco Dolstra 725488b892 nix develop: Unset $HOSTNAME
This is set to "localhost" by stdenv which is probably not what you
want.
2020-10-09 22:03:18 +02:00
Eelco Dolstra 44349064f7 nix develop: Source ~/.bashrc
Fixes #4104.
2020-10-09 22:02:00 +02:00
Eelco Dolstra 59bd6e87a4 Completions::add(): Guard against newlines 2020-10-09 21:55:59 +02:00
Eelco Dolstra ea4b2b985f
Merge pull request #4128 from tweag/extended-completions
Add a zsh completion script
2020-10-09 21:53:12 +02:00
John Ericson 39de73550d Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-10-09 18:26:47 +00:00
Eelco Dolstra e845d19ae3 Remove Lazy
This fixes a crash during startup when compiling Nix as a single
compilation unit.
2020-10-09 17:54:59 +02:00
Eelco Dolstra 87157b2bd3 writeFile(): Add error context to writeFull() failure
Issue #4092.
2020-10-09 16:02:53 +02:00
Eelco Dolstra 636ec17139
Remove stray DerivationOutputsAndPaths type 2020-10-09 15:41:24 +02:00
regnat 04e5d0e704 Add a description in the completion outputs
Make nix output completions in the form `completion\tdescription`.
This can't be used by bash (afaik), but other shells like zsh or fish
can display it along the completion choices
2020-10-09 09:39:51 +02:00
Eelco Dolstra 97ffc1e013 Dynamically disable user namespaces if CLONE_NEWUSER fails
This makes builds work inside nixos-enter.

Fixes #3145.
2020-10-07 22:46:01 +02:00
Eelco Dolstra 6aa64627c8 Support user namespaces being disabled
If max_user_namespaces is set to 0, then don't run the build in a user
namespace.

Fixes #4092.
2020-10-07 22:02:36 +02:00
Eelco Dolstra f66bbd8c7b Doh 2020-10-07 21:25:06 +02:00
Eelco Dolstra e705c24294 Tweak error messages 2020-10-07 17:28:43 +02:00
Eelco Dolstra c43e882f54 Serialize exceptions from the daemon to the client 2020-10-07 17:13:54 +02:00
Eelco Dolstra be149acfda Serialize exceptions from the sandbox process to the parent
Fixes #4118.
2020-10-07 16:34:03 +02:00
Eelco Dolstra 27ca87c46a Formatting 2020-10-07 16:33:19 +02:00
John Ericson 57d960dcd1 Remove generic std::optional<T> suppport from worker proto
See comment for rational; I think it's good to leave a comment lest
anyone is tempted to add such a sum-type instance again.

Fixes #4113
2020-10-07 12:50:37 +00:00
Maximilian Bosch 59f2dd8e8d
libfetchers/github: allow slashes in refs
Refs #4061
2020-10-06 20:08:51 +02:00
Eelco Dolstra d761485010 Prevent a deadlock when user namespace setup fails
Observed on Centos 7 when user namespaces are disabled:
DerivationGoal::startBuilder() throws an exception, ~DerivationGoal()
waits for the child process to exit, but the child process hangs
forever in drainFD(userNamespaceSync.readSide.get()) in
DerivationGoal::runChild(). Not sure why the SIGKILL doesn't get
through.

Issue #4092.
2020-10-06 18:57:23 +02:00
Eelco Dolstra ad143c5b3b Shut up some clang warnings 2020-10-06 14:52:30 +02:00
Eelco Dolstra 85c8be6286 Remove static variable name clashes
This was useful for an experiment with building Nix as a single
compilation unit. It's not very useful otherwise but also doesn't
hurt...
2020-10-06 13:49:20 +02:00
Eelco Dolstra 0419cd2695 Remove unneeded -lboost_* flags 2020-10-06 13:34:58 +02:00
Eelco Dolstra 636455c471 Remove 'using namespace fetchers' 2020-10-06 11:16:32 +02:00
Eelco Dolstra 6691256e79 Factor out common showBytes() 2020-10-06 10:40:49 +02:00
Eelco Dolstra d0bb544128 Add missing #pragma once 2020-10-06 10:40:07 +02:00
Eelco Dolstra 88a667e49e
Fix s3:// store
Fixes https://github.com/NixOS/nixos-org-configurations/issues/123.
2020-10-05 17:53:30 +02:00
Eelco Dolstra 51c299213b
Merge pull request #3935 from obsidiansystems/binary-cache-addToStoreFromDump
Get rid of Hash::dummy from BinaryCacheStore
2020-10-05 14:41:28 +02:00
Eelco Dolstra f3aba88737
Merge pull request #3895 from obsidiansystems/templated-daemon-protocol
More templated STL support for the daemon protocol
2020-10-05 14:40:27 +02:00
Fabian Möller d5d196b0a1
Fix profile update in nix command 2020-10-02 12:10:31 +02:00
DavHau f3280004e2 add more examples to --help of nix run 2020-10-01 11:34:13 +07:00
Eelco Dolstra 20a1e20d91 Style 2020-09-30 13:35:56 +02:00
Eelco Dolstra 274357eb6a Simplify 2020-09-30 12:09:18 +02:00
Eelco Dolstra 002ce8449d Merge branch 'access-tokens' of https://github.com/kquick/nix 2020-09-30 11:35:15 +02:00
John Ericson 69afaeace3 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-30 00:42:28 +00:00
John Ericson b759701652 nix::worker_proto -> worker_proto 2020-09-30 00:41:18 +00:00
John Ericson 45a0ed82f0 Revert "Use template structs instead of phantoms"
This reverts commit 9ab07e99f5.
2020-09-30 00:39:06 +00:00
Kevin Quick 5e7838512e
Remove github-access-token in favor of access-token. 2020-09-29 16:26:34 -07:00
Eelco Dolstra 76f000712e Merge branch 'kwq/path-dir-query' of https://github.com/kquick/nix 2020-09-29 23:42:43 +02:00
Eelco Dolstra 64e9b3c83b nix registry list: Show 'dir' attribute
Issue #4050.
2020-09-29 23:33:16 +02:00
Kevin Quick 66c3959e8c
Merge branch 'master' into access-tokens 2020-09-29 08:32:06 -07:00
Eelco Dolstra 5999978a05 Make Headers an optional argument 2020-09-29 13:05:19 +02:00
Eelco Dolstra de86abbf3f Cleanup 2020-09-29 12:55:06 +02:00
Eelco Dolstra cebd2fc35d Merge branch 'github-api-token' of https://github.com/imalsogreg/nix 2020-09-29 12:17:00 +02:00
John Ericson 00135e13f4 Clarify comment a bit 2020-09-28 18:19:10 +00:00
Kevin Quick 887be7b6f2
Switch comment format from '// ...' to '/* ... */' for consistency. 2020-09-28 09:37:26 -07:00
Kevin Quick 128c98ab09
Clarification in the description of the FlakeInput. 2020-09-28 09:34:23 -07:00
Kevin Quick 5ae164b7cf
Update description of FlakeRef, incorporating suggestion. 2020-09-28 09:23:05 -07:00
Kevin Quick bcb3da3b6b
Fix spelling error. 2020-09-28 08:58:14 -07:00
John Ericson 80e335bb58 Use drvPath2 and give it a better name 2020-09-28 15:43:56 +00:00
John Ericson 10202bbf29 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-28 15:39:11 +00:00
John Ericson 6c31297d80
Update src/libstore/binary-cache-store.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-28 11:32:58 -04:00
Eelco Dolstra 649c465873
Merge pull request #4064 from serokell/balsoft/fix-max-jobs
Fix max-jobs option
2020-09-28 10:42:02 +02:00
Eelco Dolstra 5df687c316
Merge pull request #4077 from Ma27/left-whitespace-log
libmain/progress-bar: don't trim whitespace on the left
2020-09-28 10:40:00 +02:00
Kevin Quick 5885b0cfd8
Miscellaneous spelling fixes in comments. (#4071) 2020-09-27 20:04:06 +00:00
Kevin Quick bd5328814f
Add some internal documentation for flake support objects. 2020-09-26 14:32:58 -07:00
Maximilian Bosch a76fb07314
libmain/progress-bar: don't trim whitespace on the left
When running `nix build -L` it can be fairly hard to read the output if
the build program intentionally renders whitespace on the left. A
typical example is `g++` displaying compilation errors.

With this patch, the whitespace on the left is retained to make the log
more readable:

```
foo> no configure script, doing nothing
foo> building
foo> foobar.cc: In function 'int main()':
foo> foobar.cc:5:5: error: 'wrong_func' was not declared in this scope
foo>     5 |     wrong_func(1);
foo>       |     ^~~~~~~~~~
error: --- Error ------------------------------------------------------------------------------------- nix
error: --- Error --- nix-daemon
builder for '/nix/store/i1q76cw6cyh91raaqg5p5isd1l2x6rx2-foo-1.0.drv' failed with exit code 1
```
2020-09-26 17:38:11 +02:00
John Ericson 25fffdda86
Remove redundant nar hash and size setting
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-09-26 10:17:30 -04:00
John Ericson 1832436526 Fix up BinaryCacheStore::addToStore taking a path 2020-09-26 04:56:29 +00:00
John Ericson 5db83dd771 BinaryCacheStore::addTextToStore include CA field 2020-09-26 03:21:36 +00:00
John Ericson c40c832f19 Merge remote-tracking branch 'obsidian/master' into binary-cache-addToStoreFromDump 2020-09-26 00:21:45 +00:00
Kevin Quick cb186f1e75
Use "?dir=..." portion of "registry add" local path specification.
The registry targets generally follow a URL formatting schema with
support for a query parameter of "?dir=subpath" to specify a sub-path
location below the URL root.

Alternatively, an absolute path can be specified.  This specification
mode accepts the query parameter but ignores/drops it.  It would
probably be better to either (a) disallow the query parameter for the
path form, or (b) recognize the query parameter and add to the path.

This patch implements (b) for consistency, and to make it easier for
tooling that might switch between a remote git reference and a local
path reference.

See also issue #4050.
2020-09-25 09:36:18 -07:00
Gregory Hale faa5607f54 Merge remote-tracking branch 'origin/master' into github-api-token 2020-09-25 12:10:58 -04:00
John Ericson cfe791a638 stdout_ -> cout
Better to get creative than just sprinkle arbitrary underscores.
2020-09-25 11:30:04 -04:00
Eelco Dolstra 7d81582488
Merge pull request #3953 from obsidiansystems/basic-derivation-goal-outputs
Deduplicate basic derivation goals too
2020-09-25 17:21:19 +02:00
Kevin Quick 5a35cc29bf
Re-add support for github-access-token, but mark as deprecated. 2020-09-25 08:09:56 -07:00
Kevin Quick ef2a14be19
Fix reference to older name for access-tokens config value. 2020-09-25 08:08:27 -07:00
Eelco Dolstra cbb9862cd9
Merge pull request #3626 from W95Psp/master
Make `functionArgs` primitive accept primops (fix #3624)
2020-09-25 15:14:18 +02:00
Eelco Dolstra 7b2ae472ff
expectArg(): Respect the 'optional' flag 2020-09-25 10:27:40 +02:00
Kevin Quick 8fba2a8b54
Update to use access-tokens configuration for github/gitlab access.
This change provides support for using access tokens with other
instances of GitHub and GitLab beyond just github.com and
gitlab.com (especially company-specific or foundation-specific
instances).

This change also provides the ability to specify the type of access
token being used, where different types may have different handling,
based on the forge type.
2020-09-24 22:49:44 -07:00
Kevin Quick c2f48cfcee
Complete conversion of "url" to "host" with associated variable renaming.
Completes the change begun in commit 56f1e0d to consistently use the
"host" attribute for "github" and "gitlab" inputs instead of a "url"
attribute.
2020-09-24 22:46:03 -07:00
Kevin Quick a439e9488d
Support StringMap configuration settings.
Allows Configuration values that are space-separated key=value pairs.
2020-09-24 22:42:59 -07:00
Paul Opiyo 4d863a9fcb Remove redundant value checks
std::optional had redundant checks for whether it had a value.
An object is emplaced either way so it can be dereferenced
without repeating a value check
2020-09-24 18:32:03 -05:00
Alexander Bantyev ed218e1d6c
Fix max-jobs option
After 0ed946aa61, max-jobs setting (-j/--max-jobs)
stopped working.

The reason was that nrLocalBuilds (which compared to maxBuildJobs to figure
out whether the limit is reached or not) is not incremented yet when tryBuild
is started; So, the solution is to move the check to tryLocalBuild.

Closes https://github.com/nixos/nix/issues/3763
2020-09-25 00:07:42 +03:00
Kevin Quick bd5f3dbe11
Fixes fall-through to report correct description of hash-file command. 2020-09-24 12:30:03 -07:00
Eelco Dolstra 8d9402f411
Merge pull request #4054 from edolstra/fix-4021
registerOutputs(): Don't call canonicalisePathMetaData() twice
2020-09-23 21:57:53 +02:00
Eelco Dolstra 4ce8a3ed45 Hopefully fix EPERM on macOS 2020-09-23 21:29:10 +02:00
Eelco Dolstra 9a24ece122 Fix exception 2020-09-23 20:21:08 +02:00
Eelco Dolstra 236d9ee7f7 lstat() cleanup 2020-09-23 19:17:28 +02:00
Eelco Dolstra 688bd4fb50 After rewriting a path, make it read-only 2020-09-23 19:10:16 +02:00
Dominique Martinet 2548347bba libutil/archive: add preallocate-contents option
Make archive preallocation (fallocate) optional because some filesystems
like btrfs do not behave as expected with fallocate.

See #3550.
2020-09-23 18:49:11 +02:00
Eelco Dolstra cec9473871 DerivationGoal::registerOutputs(): Don't canonicalize twice
Fixes #4021.
2020-09-23 18:21:22 +02:00
Eelco Dolstra d4f8163d10 canonicalisePathMetaData_(): Change assertion to error message 2020-09-23 18:21:22 +02:00
Eelco Dolstra e8f0b1e996 DerivationGoal::registerOutputs(): Fix bad format string 2020-09-23 18:21:22 +02:00
John Ericson 3f226f71c1 Return more info from BinaryCacheStore::addToStoreCommon
We don't need it yet, but we could/should in the future, and it's a
cost-free change since we already have the reference. I like it.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-09-23 14:40:41 +00:00
John Ericson 412b3a54fb
Clarify FIXME in BinaryCacheStore::addToStoreCommon
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2020-09-23 10:36:55 -04:00
Eelco Dolstra 8a2e10827f Remove unused Flake::vOutputs field 2020-09-23 14:08:52 +02:00
Domen Kožar dac8a1a5e1
Merge pull request #4052 from ujjwaljainnn/fix-typo
fixed typo
2020-09-23 09:58:30 +02:00
John Ericson 9fbc31a65b Get rid of Hash::dummy from BinaryCacheStore 2020-09-23 04:56:04 +00:00
ujjwal a2f5c921d4 fixed typo 2020-09-22 23:37:06 +05:30
John Ericson 993229cdaf Deduplicate basic derivation goals too
See comments for security concerns.

Also optimize goal creation by not traversing map twice.
2020-09-22 17:13:59 +00:00
John Ericson e9fc2031f0 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 14:18:31 +00:00
regnat 67cc356bb3 Merge remote-tracking branch 'origin/master' into non-ca-depending-on-ca 2020-09-22 16:11:25 +02:00
Eelco Dolstra 980edd1f3a RemoteStore::addCAToStore(): Don't hold connection while calling queryPathInfo()
This leads to a deadlock if we're at the connection limit.
2020-09-22 15:28:20 +02:00
Eelco Dolstra 5b107f2c5f
Merge pull request #4038 from maljub01/master
Add a nix.conf option for allowing a symlinked store
2020-09-22 13:19:22 +02:00
Eelco Dolstra 35a0ac1838 Style fixes 2020-09-22 11:40:19 +02:00
Eelco Dolstra 92ac8df0ec Merge branch 'add-ca-to-store' of https://github.com/hercules-ci/nix 2020-09-22 11:31:33 +02:00
Eelco Dolstra e7f1109f06
Merge pull request #4040 from OmnipotentEntity/master
Fix compatibility with nlohmann-json 3.9.1
2020-09-22 11:09:25 +02:00
Eelco Dolstra 7dd8baafe1
Merge pull request #4041 from cole-h/enum-stringify
Serialize SandboxMode enum to string for JSON
2020-09-22 11:08:56 +02:00
regnat c1e79f870c Silence a compiler warning in serialise.hh
Explicitely cast to `uint64_t` in `readNum` to avoid a "comparison
between signed and unsigned" warning
2020-09-22 10:39:29 +02:00
John Ericson 3786a801c3 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-22 04:15:55 +00:00
John Ericson b92d3b2edd Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 00:45:55 +00:00
Michael Reilly d860295e11 Bump nlohmann-json version to 3.9.1 2020-09-21 14:24:43 -04:00
Cole Helbling ba37299a03
Serialize SandboxMode enum to string for JSON
Rather than showing an integer as the default, instead show the boolean
referenced in the description.

The nix.conf.5 manpage used to show "default: 0", which is unnecessarily
opaque and confusing (doesn't 0 mean false, even though the default is
true?); now it properly shows that the default is true.
2020-09-21 10:36:45 -07:00
Eelco Dolstra ecc8672aa0 fmt.hh: Don't include boost/algorithm/string/replace.hpp
This cuts compilation time by ~49s.

Issue #4045.
2020-09-21 19:07:55 +02:00
Eelco Dolstra 557d2427ee Random header cleanup 2020-09-21 18:59:02 +02:00
Eelco Dolstra 0716adaa8b abstractsettingtojson.hh -> abstract-setting-to-json.hh 2020-09-21 18:49:43 +02:00
Eelco Dolstra 340ca382c4 Don't include nlohmann/json.hpp in globals.hh
This reduces compilation time by 207s.

Issue #4045.
2020-09-21 18:47:18 +02:00
Eelco Dolstra d51ba43047 Move Callback into its own header
This gets rid of the inclusion of <future> in util.hh, cutting
compilation time by ~20s (CPU time).

Issue #4045.
2020-09-21 18:42:21 +02:00
Marwan Aljubeh f80ffeb8c9 Update the variable name accordingly 2020-09-21 17:29:08 +01:00
Eelco Dolstra e8e1d420f3 Don't include <regex> in header files
This reduces compilation time by ~15 seconds (CPU time).

Issue #4045.
2020-09-21 18:22:45 +02:00
Marwan Aljubeh 4e1a04733d Use a better name for the config option 2020-09-21 16:32:22 +01:00
Eelco Dolstra cbe0bb29f4
Merge pull request #4035 from Ma27/url-attr
libfetchers/github: allow `url` attribute
2020-09-21 17:26:36 +02:00
Maximilian Bosch 56f1e0df05
libfetchers/github: rename url to host 2020-09-21 16:29:08 +02:00
Robert Hensing ca30abb3fb Document addCAToStore/addToStoreFromDump source drainage
Also checked that all usages satisfy the requirement and
removed dead code.
2020-09-21 07:55:47 +02:00
Robert Hensing fa08db5c4c wopAddToStore: return ValidPathInfo
A ValidPathInfo is created anyway. By returning it we can save a
roundtrip and we have a nicer interface.
2020-09-21 07:55:47 +02:00
Robert Hensing 7c68264085 wopAddToStore: add RepairFlag 2020-09-21 07:55:47 +02:00
Robert Hensing fbf509c113 parseContentAddressMethodPrefix: use string_view
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-09-21 07:55:47 +02:00
Robert Hensing 8279178b07 Move FramedSink next to FramedSource 2020-09-21 07:55:47 +02:00
Robert Hensing ecc8088cb7 wopAddToStore: Throw to clarify unused refs
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-09-21 07:55:47 +02:00
Robert Hensing c602ebfb34 Refactor wopAddToStore to make wopAddTextToStore obsolete 2020-09-21 07:55:45 +02:00
Robert Hensing e34fe47d0c Overhaul wopAddToStore 2020-09-21 07:54:05 +02:00
Marwan Aljubeh e40772cd35 Lint issue: replacing tabs with spaces 2020-09-18 17:18:45 +01:00
Marwan Aljubeh c00e078343 Add a nix.conf option for allowing a symlinked store 2020-09-18 17:10:39 +01:00
Bryan Richter 5fe375a8f1
nix-prefetch-url: Add --executable flag
pkgs.fetchurl supports an executable argument, which is especially nice
when downloading a large executable. This patch adds the same option to
nix-prefetch-url.

I have tested this to work on the simple case of prefetching a little
executable:

1. nix-prefetch-url --executable https://my/little/script
2. Paste the hash into a pkgs.fetchurl-based package, script-pkg.nix
3. Delete the output from the store to avoid any misidentified artifacts
4. Realise the package script-pkg.nix
5. Run the executable

I repeated the above while using --name, as well.

I suspect --executable would have no meaningful effect if combined with
--unpack, but I have not tried it.
2020-09-18 19:09:45 +03:00
Maximilian Bosch 2bcf8cbe7a
libfetchers/github: allow url attribute
Since 108debef6f we allow a
`url`-attribute for the `github`-fetcher to fetch tarballs from
self-hosted `gitlab`/`github` instances.

However it's not used when defining e.g. a flake-input

    foobar = {
        type = "github";
        url = "gitlab.myserver";
        /* ... */
    }

and breaks with an evaluation-error:

    error: --- Error --------------------------------------nix
    unsupported input attribute 'url'
    (use '--show-trace' to show detailed location information)

This patch allows flake-inputs to be fetched from self-hosted instances
as well.
2020-09-18 14:10:45 +02:00
Eelco Dolstra 958bf57123 nix build: find() -> get()
find() returns an iterator so "!attr" doesn't work.
2020-09-18 13:10:42 +02:00
Robert Hensing 14b30b3f3d Move FramedSource and FramedSink, extract withFramedSink 2020-09-17 20:21:04 +02:00
Robert Hensing dfa547c6a8 Add ContentAddressMethod and parse/render it 2020-09-17 20:21:04 +02:00
Robert Hensing 29c82ccc77 Add Source.drainInto(Sink) 2020-09-17 20:21:04 +02:00
Robert Hensing 9ee3122ec7 Remove redundant import 2020-09-17 20:21:04 +02:00
Robert Hensing 3f93bc0d39 Typo 2020-09-17 20:21:04 +02:00
John Ericson b7df353f27 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-17 16:33:10 +00:00
Eelco Dolstra 649d3aaf24
Merge pull request #3829 from obsidiansystems/remove-storetype-delegate-regStore
Remove storetype delegate reg store -- contains #3736
2020-09-17 13:55:01 +02:00
Eelco Dolstra b94a35ef40
Merge pull request #4027 from tweag/fix-gc-of-ca-derivations
Fix garbage collection of CA derivations
2020-09-17 13:46:26 +02:00
regnat 520895b1da Fix garbage collection of CA derivations
Fix #4026
2020-09-17 13:36:58 +02:00
Eelco Dolstra 787469c7b6 Remove corepkgs/unpack-channel.nix 2020-09-17 09:41:02 +02:00
Eelco Dolstra 10d1865f5f Remove corepkgs/derivation.nix 2020-09-17 09:41:02 +02:00
John Ericson f60b380a7f Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-16 22:35:24 +00:00
John Ericson 2741fffa35 Ensure resolved CA derivations are written
so we can link outputs to deriver and thus properly cache.
2020-09-16 17:57:32 +00:00
John Ericson 7fdbb377ba Start to fix floating CA + remote building 2020-09-16 17:50:48 +00:00
John Ericson c5ccebae00 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-16 17:50:40 +00:00
Greg Hale a303c0b6dc Fetch commits from github/gitlab using Auth header
`nix flake info` calls the github 'commits' API, which requires
authorization when the repository is private. Currently this request
fails with a 404.

This commit adds an authorization header when calling the 'commits' API.
It also changes the way that the 'tarball' API authenticates, moving the
user's token from a query parameter into the Authorization header.

The query parameter method is recently deprecated and will be disallowed
in November 2020. Using them today triggers a warning email.
2020-09-16 13:46:48 -04:00
Eelco Dolstra 5080d4e7b2 Merge branch 'document-store-options' of https://github.com/tweag/nix 2020-09-16 17:02:30 +02:00
Eelco Dolstra 0066ef6c59 Fix doc generation 2020-09-16 16:56:28 +02:00
Eelco Dolstra 2eacc1bc00 builtins.toFile: Fix indentation 2020-09-16 14:18:46 +02:00
Théophane Hufschmitt 77a0e2c5be
Remove useless exception copy
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-16 14:00:21 +02:00
regnat d72927aa7a Fix the s3 store
Add some necessary casts in the initialisation of the store's config
2020-09-16 13:53:09 +02:00
regnat e0817cbcdc Don't include nlohmann/json.hpp in config.hh
Instead make a separate header with the template implementation of
`BaseSetting<T>::toJSONObj` that can be included where needed
2020-09-16 13:53:09 +02:00
regnat 93c0e14a30 Include the full nlohmann/json header in config.hh
It is apparently required for using `toJSONObject()`, which we do inside
the header file (because it's in a template).

This was accidentally working when building Nix itself (presumably because
`config.hh` was always included after `nlohman/json.hpp`) but caused a
(pretty dirty) build failure in the perl bindings package.
2020-09-16 13:53:09 +02:00
regnat fc2d31c423 Add (StoreConfig*) casts to work around a GCC bug
Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was
already there in the code but was accidentally removed in the last
commits
2020-09-16 13:53:09 +02:00
regnat a1e82ba450 fixup! Add a default value for the settings 2020-09-16 13:53:09 +02:00
regnat 888f7afe9f Fix build issues with gcc 2020-09-16 13:53:09 +02:00
regnat 634cb2a5ae Add a markdown output to nix describe-stores 2020-09-16 13:53:09 +02:00
regnat b73adacc1e Add a name to the stores
So that it can be printed by `nix describe-stores`
2020-09-16 13:53:09 +02:00
regnat f24f0888f9 Document the new store hierarchy 2020-09-16 13:53:09 +02:00
regnat d65962db4d Make uri schemes grammar more RFC-compliant
Allow `-` and `.` in the RFC schemes as stated by
[RFC3986](https://tools.ietf.org/html/rfc3986#section-3.1).

Practically, this is needed so that `ssh-ng` is a valid URI scheme
2020-09-16 13:53:09 +02:00
regnat 29a632386e fixup! Make the store plugins more introspectable 2020-09-16 13:53:09 +02:00
regnat 1129913c4e fixup! Correctly call all the parent contructors of the stores 2020-09-16 13:53:09 +02:00
regnat 7f103dcddd Properly filter the stores according to their declared uriSchemes
When opening a store, only try the stores whose `uriSchemes()` include
the current one
2020-09-16 13:53:09 +02:00
regnat 5895184df4 Correctly call all the parent contructors of the stores
Using virtual inheritance means that only the default constructors of
the parent classes will be called, which isn't what we want
2020-09-16 13:53:09 +02:00
regnat dae39f0a7a Make nix describe-stores functional
Using the `*Config` class hierarchy
2020-09-16 13:53:08 +02:00
regnat 22afa8fb4d Separate store configs from the implems
Rework the `Store` hierarchy so that there's now one hierarchy for the
store configs and one for the implementations (where each implementation
extends the corresponding config). So a class hierarchy like

```
StoreConfig-------->Store
    |                 |
    v                 v
SubStoreConfig----->SubStore
    |                 |
    v                 v
SubSubStoreConfig-->SubSubStore
```

(with virtual inheritance to prevent DDD).

The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-16 13:53:08 +02:00
regnat aa4eac3788 fixup! Separate the instantiation and initialisation of the stores 2020-09-16 13:53:08 +02:00
regnat 35042c9623 Add a default value for the settings
The default value is initialized when creating the setting and unchanged
after that
2020-09-16 13:53:08 +02:00
regnat 3c525d1590 Complete the toJSON instance for Setting<T>
Don't let it just contain the value, but also the other fields of the
setting (description, aliases, etc..)
2020-09-16 13:53:08 +02:00
regnat 3b57181f8e Separate the instantiation and initialisation of the stores
Add a new `init()` method to the `Store` class that is supposed to
handle all the effectful initialisation needed to set-up the store.
The constructor should remain side-effect free and just initialize the
c++ data structure.

The goal behind that is that we can create “dummy” instances of each
store to query static properties about it (the parameters it accepts for
example)
2020-09-16 13:53:08 +02:00
regnat fa32560169 Fix the registration of stores 2020-09-16 13:53:08 +02:00
regnat 7d5bdf8b56 Make the store plugins more introspectable
Directly register the store classes rather than a function to build an
instance of them.
This gives the possibility to introspect static members of the class or
choose different ways of instantiating them.
2020-09-16 13:53:08 +02:00
John Ericson 3a5cdd737c Rename Derivation::pathOpt to Derivation::path
We no longer need the `*Opt` to disambiguate.
2020-09-15 15:21:39 +00:00
John Ericson 6387550d58 Get rid of confusing std::optional<bool> for validity 2020-09-15 15:19:45 +00:00
John Ericson c4bf219b55 Don't link deriver until after any delayed exception is thrown
Otherwise, we will associate fixed-output derivations with outputs that
they did indeed produce, but which had the wrong hash. That's no good.
2020-09-15 14:28:06 +00:00
John Ericson 3ba552b245 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-15 14:17:06 +00:00
John Ericson c08c9f08c7 Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-15 14:08:35 +00:00
Eelco Dolstra 2a80170920
Merge pull request #4014 from tweag/fix-queryDrvOutputMaps-for-old-daemons
gracefully handle old daemon versions
2020-09-15 11:25:19 +02:00
regnat 057c6203b5 gracefully handle old daemon versions
Add a fallback path in `queryPartialDerivationOutputMap` for daemons
that don't support it.

Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this
is needed to handle the fallback path.
2020-09-15 09:58:01 +02:00
regnat 250f8a4bba Escape ${ in strings when printing Nix expressions
Otherwise the result of the printing can't be parsed back correctly by
Nix (because the unescaped `${` will be parsed as the begining of an
anti-quotation).

Fix #3989
2020-09-14 17:19:25 +02:00
Eelco Dolstra a59e77d9e5 nix-daemon: Lower verbosity of restricted setting warning
Fixes #3992.
2020-09-14 13:48:51 +02:00
Eelco Dolstra 35ba092019
Merge pull request #4005 from Infinisil/fix-autoArgs
Fix auto argument passing for more auto arguments than formals
2020-09-14 13:31:23 +02:00
aszlig 525b38eee8
Fix unspecified behaviour in readStorePathCAMap
When deploying a Hydra instance with current Nix master, most builds
would not run because of errors like this:

  queue monitor: error: --- Error --- hydra-queue-runner
  error: --- UsageError --- nix-daemon
  not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv

The last error message is from parseContentAddress, which expects a
colon-separated string, however what we got here is a store path.

Looking at the worker protocol, the following message sent to the Nix
daemon caused the error above:

  0x1E -> wopQuerySubstitutablePathInfos
  0x01 -> Number of paths
  0x16 -> Length of string
  "/nix/store/...-somedrv"
  0x00 -> Length of string
  ""

Looking at writeStorePathCAMap, the store path is indeed the first field
that's transmitted. However, readStorePathCAMap expects it to be the
*second* field *on my machine*, since expression evaluation order is a
classic form of unspecified behaviour[1] in C++.

This has been introduced in https://github.com/NixOS/nix/pull/3689,
specifically in commit 66a62b3189.

[1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions

Signed-off-by: aszlig <aszlig@nix.build>
2020-09-13 02:40:51 +02:00
Silvan Mosberger 8dbd57a6a5
Fix auto argument passing for more auto arguments than formals
The change in 626200713b didn't account
for when the number of auto arguments is bigger than the number of
formal arguments. This causes the following:

  $ nix-instantiate --eval -E '{ ... }@args: args.foo' --argstr foo foo
  nix-instantiate: src/libexpr/attr-set.hh:55: void nix::Bindings::push_back(const nix::Attr&): Assertion `size_ < capacity_' failed.
  Aborted (core dumped)
2020-09-11 20:41:51 +02:00
Gabriel Gonzalez ee5906243a Add nix-shell support for preserving PS1
Fixes https://github.com/NixOS/nix/issues/1268

`nix-shell` will now preserve `PS1` if the `NIX_SHELL_PRESERVE_PROMPT`
environment variable is set.
2020-09-04 20:05:43 -07:00
John Ericson 98dfd7531d Fix querying outputs for CA derivations some more
If we resolve using the known path of a derivation whose output we
didn't have, we previously blew up. Now we just fail gracefully,
returning the map of all outputs unknown.
2020-09-04 18:33:58 +00:00
John Ericson 075d399e3f Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-04 16:04:35 +00:00
John Ericson 5aed6f9b25 Document mkOutputString 2020-09-04 15:58:42 +00:00
John Ericson e9fad3006b Fix some of the issues raised by @edolstra
- More and better comments

 - The easier renames
2020-09-04 15:18:25 +00:00
John Ericson e86dd59dcc
Apply suggestions from code review
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-09-04 10:48:50 -04:00
John Ericson c9f1ed912c Don't chmod symlink before moving outputs around
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-09-04 14:41:53 +00:00
John Ericson 25f7ff16fa Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-09-04 02:40:36 +00:00
John Ericson aad4abcc9c Fix floating CA tests
We will sometimes try to query the outputs of derivations we can't
resolve. That's fine; it just means we don't know what those outputs are
yet.
2020-09-04 01:17:38 +00:00
John Ericson 975a47f7fe Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-03 22:09:04 +00:00
John Ericson e7d93e7ece Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-03 15:43:17 +00:00
Eelco Dolstra 8a945d6ddb
Remove gc-check-reachability 2020-09-03 11:30:15 +02:00
Eelco Dolstra 82b77a7726
addPermRoot(): Remove indirect flag 2020-09-03 11:29:34 +02:00
Eelco Dolstra 00d25e8457
Remove the --indirect flag
All GC roots are now indirect.
2020-09-03 11:22:00 +02:00
Eelco Dolstra b07167be5a
createGeneration(): Always create an indirect root
This means profiles outside of /nix/var/nix/profiles don't get
garbage-collected. It also means we don't need to scan
/nix/var/nix/profiles for GC roots anymore, except for compatibility
with previously existing generations.
2020-09-03 11:13:53 +02:00
Eelco Dolstra b74f5cdd23
createGeneration(): Take a StorePath 2020-09-03 11:06:56 +02:00
Eelco Dolstra 94a043ff3b
EvalCache: Fix caching of strings
This was broken in 50f13b06fb. Once
again it turns out that putting a bool in a std::variant is a bad
idea, since pointers get silently cast to them...
2020-09-02 14:16:44 +02:00
Domen Kožar dd4b56c87f
Allow HTTP binary cache to request absolute uris 2020-09-01 21:35:48 +02:00
John Ericson ef278d00f9 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-01 18:01:48 +00:00
Eelco Dolstra 6d7f7efb89
github: Use access token when calling .../commits API 2020-09-01 15:29:04 +02:00
Eelco Dolstra 84f5cabbea Merge remote-tracking branch 'origin/master' into markdown 2020-08-31 14:24:26 +02:00
John Ericson 02e0001fc0 Merge branch 'ca-floating-upstream' of github.com:obsidiansystems/nix into ca-floating-upstream 2020-08-28 21:23:09 +00:00
John Ericson 421ed527c7
Update src/libstore/build.cc
Thanks for catching, @regnat.
2020-08-28 17:22:57 -04:00
John Ericson 48e8828050 Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-08-28 20:38:25 +00:00
John Ericson 8017fe7487 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-28 19:59:14 +00:00
Eelco Dolstra f15651303f nix develop: Add convenience flags for running specific phases
For example, for building the Nix flake, you would do:

  $ nix develop --configure
  $ nix develop --install
  $ nix develop --installcheck
2020-08-28 19:24:29 +02:00
Eelco Dolstra 3156560d41 nix develop: Set output paths to writable locations
Currently, they're set to $(pwd)/outputs/$outputName. This allows
commands like 'make install' to work.
2020-08-28 18:16:03 +02:00
Eelco Dolstra 691a1bd717 Merge branch 'minimal-logger' of https://github.com/Ma27/nix 2020-08-28 10:50:52 +02:00
Griffin Smith 626200713b Pass all args when auto-calling a function with an ellipsis
The command line options --arg and --argstr that are used by a bunch of
CLI commands to pass arguments to top-level functions in files go
through the same code-path as auto-calling top-level functions with
their default arguments - this, however, was only passing the arguments
that were *explicitly* mentioned in the formals of the function - in the
case of an as-pattern with an ellipsis (eg args @ { ... }) extra passed
arguments would get omitted. This fixes that to instead pass *all*
specified auto args in the case that our function has an ellipsis.

Fixes #598
2020-08-27 12:38:25 -04:00
Eelco Dolstra eb75282b8d
Merge pull request #3434 from Ericson2314/derivation-header-include-order
Revise division of labor in deserialization of derivations
2020-08-27 16:39:28 +02:00
Eelco Dolstra e915fd6d2a Typo 2020-08-27 14:51:50 +02:00
Eelco Dolstra a0f19d9f3a RemoteStore::addToStore(): Fix race between stderrThread and NAR writer
As pointed out by @B4dM4n, the call to to.flush() on stderrThread is
unsafe because the NAR writer thread is also writing to 'to'.

Fixes #3943.
2020-08-27 14:50:51 +02:00
Eelco Dolstra c3efef9275 Remove obsolete comment 2020-08-26 09:28:10 +02:00
Eelco Dolstra 4bf5faf416 Merge remote-tracking branch 'origin/master' into markdown 2020-08-25 19:47:34 +02:00
Eelco Dolstra 7a02865b94
Move import docs 2020-08-25 14:06:01 +02:00
Eelco Dolstra f53b5f1058
Add getDoc() function 2020-08-25 13:31:11 +02:00
Eelco Dolstra 2a2121d264
Use RegisterPrimOp for some undocumented primops 2020-08-25 11:25:01 +02:00
Eelco Dolstra b8416779e3
Document some primops 2020-08-25 11:16:45 +02:00
Eelco Dolstra 24b1c2c66b
Fix tests 2020-08-25 10:51:14 +02:00
Eelco Dolstra b42789f013
Fix clang build 2020-08-24 21:13:39 +02:00
John Ericson 8eb73a8724 CA derivations that depend on other CA derivations
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
2020-08-24 19:01:15 +00:00
John Ericson e0b0e18905 Add constructor for BasicDerivation -> Derivation 2020-08-24 19:01:07 +00:00
John Ericson 59979e7053 Fix bad debug format string 2020-08-24 19:01:07 +00:00
Eelco Dolstra d9a8619762
Don't barf if corepkgs is in the store but not a valid path
This can happen when using a dummy store (or indeed any non-local store).
2020-08-24 19:15:17 +02:00
Eelco Dolstra 6a67e57019
Add DummyStore (dummy://)
DummyStore does not allow building or adding paths. This is useful for
evaluation tests when you don't want to initialize a "proper" store.
2020-08-24 18:54:16 +02:00
Eelco Dolstra d0690bc311
nix repl ':doc': Render using lowdown 2020-08-24 18:10:33 +02:00
Eelco Dolstra 0f314f3c25
Generate builtins section of the manual 2020-08-24 14:49:30 +02:00
Eelco Dolstra a990f063ff
Move primop docs inline
This makes them available to 'nix repl'.
2020-08-24 14:31:10 +02:00
Eelco Dolstra 33b1679d75
Allow primops to have Markdown documentation 2020-08-24 13:16:02 +02:00
Eelco Dolstra 88d5c9ec58
Fix tests 2020-08-24 10:37:10 +02:00
Eelco Dolstra 2530cc6827
Merge pull request #3950 from obsidiansystems/tabs-spaces
tabs -> spaces
2020-08-24 10:20:28 +02:00
John Ericson b0b59fd05a Merge remote-tracking branch 'obsidian/write-derivation-borrow' into HEAD 2020-08-23 15:11:10 +00:00
John Ericson 35e6288be1 writeDerivation just needs a plain store reference 2020-08-23 15:01:11 +00:00
John Ericson 3a7b330b64 "Downstream placeholders" should not be store paths
Insead they should be opaque `/<hash>` like the placeholders we already
have.
2020-08-21 19:35:35 +00:00
John Ericson 422affe102 tabs -> spaces
Sorry I let the tab sneak in there in the first place.
2020-08-21 19:19:24 +00:00
John Ericson c265e0ea82 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-08-20 20:24:22 +00:00
John Ericson 27a3f82c0b Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-20 18:28:17 +00:00
John Ericson 45a2f1baab Rename drv output querying functions, like master
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap
2020-08-20 18:14:12 +00:00
Eelco Dolstra 4d77513d97
Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing
`queryDerivationOutputMap` no longer assumes all outputs have a mapping
2020-08-20 16:49:23 +02:00
John Ericson 9a9d834dc7 Rename drv output querying functions
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`
2020-08-20 14:12:51 +00:00
Eelco Dolstra 25ecfffdc3
Remove PrimOp constructor 2020-08-20 12:34:04 +02:00
Eelco Dolstra dc2f278c95
Allow 'nix' subcommands to provide docs in Markdown format 2020-08-20 12:21:46 +02:00
Eelco Dolstra 3c4f8c9175
List deprecated option aliases in the docs 2020-08-20 11:13:17 +02:00
Eelco Dolstra acb99f03f9
Config: Use nlohmann/json 2020-08-20 11:02:16 +02:00
John Ericson 3df78858f2 Fix max fd calc and add test 2020-08-20 05:08:50 +00:00
Ryan Mulligan f36793c7b9
fix spelling 2020-08-19 20:31:01 -07:00
John Ericson a83694c7a1 Use RemoteStore to open connection for proxying daemon
Removes duplicate websocket opening code, and also means we should be
able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.
2020-08-19 19:34:47 +00:00
Eelco Dolstra c8fa39324a
Generate the nix.conf docs from the source code
This means we don't have two (divergent) sets of option descriptions
anymore.
2020-08-19 18:28:04 +02:00
John Ericson d5af5763cf Merge branch 'master' of github.com:NixOS/nix into remove-storetype-delegate-regStore 2020-08-19 15:54:50 +00:00
Eelco Dolstra 34b22e0123
Change option descriptions to Markdown 2020-08-19 14:21:27 +02:00
John Ericson be0d429b95 Merge branch 'master' of github.com:NixOS/nix into templated-daemon-protocol 2020-08-19 03:17:41 +00:00
John Ericson 950ddfdb82 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-18 14:36:44 +00:00
Eelco Dolstra 1c8b550e34
Merge pull request #3917 from obsidiansystems/output-env-var-unconditional
Simplify code as output env vars are unconditional
2020-08-18 16:21:17 +02:00
Eelco Dolstra dfeb76dbf9
Merge pull request #3930 from obsidiansystems/legacy-ssh-build-paths
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths`
2020-08-18 16:07:40 +02:00
Eelco Dolstra 069340179e
Improve nix.1 manpage generator 2020-08-18 15:15:35 +02:00
Eelco Dolstra 6f19c776db
Start generation of the nix.1 manpage 2020-08-17 19:33:18 +02:00
Eelco Dolstra a72a20d68f
Add 'nix dump-args' to dump all commands/flags for manpage generation 2020-08-17 17:44:52 +02:00
Eelco Dolstra 7cdc739ece
Merge remote-tracking branch 'origin/master' into markdown 2020-08-17 13:43:39 +02:00
Eelco Dolstra e849b19872
Merge pull request #3932 from chkno/no-show-signature
Don't try to parse signature check as commit timestamp
2020-08-17 11:00:24 +02:00
Eelco Dolstra 847a5392f4 Merge branch 'ca-no-need-trust' of https://github.com/obsidiansystems/nix 2020-08-17 10:57:02 +02:00
Eelco Dolstra 3c619f6290 Merge branch 'test-RemoteStore-buildDerivation' of https://github.com/obsidiansystems/nix 2020-08-17 10:53:18 +02:00
John Ericson 6f7ac5e865 Remove extra closing paren 2020-08-14 21:59:31 +00:00
John Ericson f899a7c6d7 Work around clang bug 2020-08-14 18:51:31 +00:00
John Ericson 3c8b5b6219 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-14 17:00:13 +00:00
John Ericson 4b571ea321
Update src/libstore/daemon.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2020-08-14 11:52:37 -04:00
Eelco Dolstra 13e49be660
Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt
Offer a safer interface for path and pathOpt
2020-08-14 17:19:19 +02:00
Eelco Dolstra 7714d9a943
Merge pull request #3924 from obsidiansystems/features-per-store
Make `system-features` a store setting
2020-08-14 17:13:07 +02:00
Eelco Dolstra 9b9d5297ba
Merge pull request #3909 from matthewbauer/readd-hashed-mirrors
Add hashed-mirrors back
2020-08-14 17:05:47 +02:00