qyriad/lix
1
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
10123 commits 57 branches 0 tags 73 MiB
Commit graph

6399 commits

Author SHA1 Message Date
Eelco Dolstra 0bd060f23a
Merge pull request #4308 from tweag/properly-test-early-cutoff 
Properly test the early cutoff for CA derivations
 2020-12-03 14:45:29 +01:00
Eelco Dolstra 4f25644a13
Merge pull request #4304 from NixOS/separate-manpages 
Separate manpages for 'nix' subcommands
 2020-12-03 13:38:29 +01:00
regnat 0afab668fa Don't fail early when -j0 is passed 
If the build closure contains some CA derivations, then we can't know
ahead-of-time that we won't build anything as early-cutoff might come-in
at a laster stage
 2020-12-03 13:24:36 +01:00
Eelco Dolstra df552a2645 nix eval: Add option to write a directory 
This is useful for generating the nix manpages, but it may have other
applications (like generating configuration files without a Nix store).
 2020-12-02 23:23:23 +01:00
Eelco Dolstra 148608ba6d Add 'nix help' 2020-12-02 23:23:23 +01:00
Greg Hale d8fc1bb7b0 fix tokens documentation 2020-12-02 10:15:18 -05:00
regnat 0d9e1af695 Remove an unknown pragma gcc warning 2020-12-02 14:33:20 +01:00
regnat a8a96dbaf8 Add forgotten override annotation 2020-12-02 14:23:38 +01:00
Eelco Dolstra 1b79b5b983 read(): Use char * instead of unsigned char * 
This gets rid of some pointless casts.
 2020-12-02 14:17:27 +01:00
Eelco Dolstra faa31f4084 Sink: Use std::string_view 2020-12-02 14:17:27 +01:00
Eelco Dolstra aa68486112 writeFull/writeFile: Use std::string_view 2020-12-02 14:17:27 +01:00
Eelco Dolstra e5cf501c77
Merge pull request #4284 from tweag/fixed-output-depending-on-ca 
Allow fixed-output derivations to depend on (floating) content-addressed ones
 2020-12-01 20:25:41 +01:00
regnat 438977731c shut up clang warnings 
- Fix some class/struct discrepancies
- Explicit the overloading of `run` in the `Cmd*` classes
- Ignore a warning in the generated lexer
 2020-12-01 15:04:03 +01:00
Eelco Dolstra 88798613ee replaceStrings(): Use std::string_view 2020-12-01 13:45:43 +01:00
Eelco Dolstra c0d1354b7d Macro hygiene 2020-12-01 13:45:06 +01:00
Eelco Dolstra 3b7e00ce22 Move primeCache() to Worker::run() 
We need the missing path info to communicate the worker's remaining
goals to the progress bar.
 2020-12-01 13:44:48 +01:00
Eelco Dolstra 5927624473 Lower verbosity for 'Failed to find a machine' message 2020-12-01 13:43:36 +01:00
Eelco Dolstra e224c16d28 Macro hygiene 2020-12-01 13:43:33 +01:00
regnat 9bd8184f1f Allow fixed-output derivations to depend on (floating) content-addressed ones 
Fix an overlook of https://github.com/NixOS/nix/pull/4056
 2020-11-27 15:39:24 +01:00
Eelco Dolstra 05d9442f68 builtins.fetchGit: Fix shortRev attribute for dirty trees 2020-11-26 21:45:28 +01:00
Dominique Martinet 1fd13d67e8 archive: disable preallocate-contents by default 
using fallocate() to preallocate files space does more harm than good:
 - breaks compression on btrfs
 - has been called "not the right thing to do" by xfs developers
(because delayed allocation that most filesystems implement leads to smarter
allocation than what the filesystem needs to do if we upfront fallocate files)
 2020-11-26 14:26:57 +01:00
Eelco Dolstra 8252a44e96 Move to separate file 2020-11-26 13:16:36 +01:00
Eelco Dolstra 9a586e34ac Record trusted/untrusted settings in ~/.local/share/nix 2020-11-26 13:11:07 +01:00
Eelco Dolstra 0287f83057 Ask for confirmation before allowing flake Nix configuration settings 2020-11-26 12:37:23 +01:00
regnat 13c557fe82 fix the hash rewriting for ca-derivations 2020-11-25 11:33:00 +01:00
Lily Ballard 437189e446 Escape filename given to nix-shell in shebang mode 
This prevents spaces or other metacharacters from causing nix-shell to
execute the wrong path.

Fixes #4229.
 2020-11-24 15:08:37 -08:00
Luke Granger-Brown 226116f482 fetchMercurial: set HGPLAIN when invoking hg 
Without setting HGPLAIN, the user's environment leaks into
hg invocations, which means that the output may not be in the
expected format.

HGPLAIN is the Mercurial-recommended solution for this in that
it's intended for uses by scripts and programs which are looking
to parse Mercurial's output in a consistent manner.
 2020-11-23 16:12:33 +00:00
Wil Taylor 07603890d2 Removed reference check from bundler command 2020-11-23 21:19:40 +10:00
Wil Taylor c3bad73e27 Added switch 2020-11-21 14:28:49 +10:00
Eelco Dolstra 4dcb183af3 AttrCursor::getStringWithContext(): Force re-evaluation if the cached context is not valid 
Fixes #4236.
 2020-11-19 20:59:36 +01:00
Eelco Dolstra 0327580e54 Fix assertion failure in LockFile::LockFile() 
Fixes #4241.
 2020-11-19 20:31:30 +01:00
John Ericson 2113ae2d85 Make drv hash modulo memo table thread-safe 
Let's get one step closer to the daemon not needing to fork.
 2020-11-19 16:50:06 +00:00
Eelco Dolstra 4dbd05e933
Merge pull request #4189 from edolstra/flake-config 
Allow nix.conf options to be set in flake.nix
 2020-11-17 15:39:14 +01:00
Eelco Dolstra f4e790cc85
Merge pull request #4182 from mkenigs/fix-1930 
Print built derivations as json for build
 2020-11-17 14:59:49 +01:00
Eelco Dolstra df5c69a94e
Merge pull request #4180 from Ma27/ssh-ng-substitute 
Allow substituting paths when building remotely using `ssh-ng://`
 2020-11-17 14:01:04 +01:00
Eelco Dolstra e6b7c7b79c Cleanup 2020-11-17 13:58:55 +01:00
Eelco Dolstra bccff827dc Fix deadlock in IFD through the daemon 
Fixes #4235.
 2020-11-17 13:50:36 +01:00
regnat 7de21f6664 Make the sql debug statements more useful 
Print the expanded sql query (with the variables bound to their value) rather
than the original one in case of error
 2020-11-17 10:05:45 +01:00
Eelco Dolstra ef84c780bb
filterANSIEscapes(): Handle UTF-8 characters 2020-11-16 16:41:53 +01:00
Eelco Dolstra 0d6419ad87
Merge pull request #4262 from jbaum98/master 
Fix deadlock in nix-store when max-connections=1
 2020-11-16 10:20:36 +01:00
Jake Waksbaum 01db455733 Fix deadlock in nix-store when max-connections=1 
This fixes a bug I encountered where `nix-store -qR` will deadlock when
the `--include-outputs` flag is passed and `max-connections=1`.

The deadlock occurs because `RemoteStore::queryDerivationOutputs` takes
the only connection from the connection pool and uses it to check the
daemon version. If the version is new enough, it calls
`Store::queryDerivationOutputs`, which eventually calls
`RemoteStore::queryPartialDerivationOutputMap`, where we take another
connection from the connection pool to check the version again. Because
we still haven't released the connection from the caller, this waits for
a connection to be available, causing a deadlock.

This diff solves the issue by using `getProtocol` to check the protocol
version in the caller `RemoteStore::queryDerivationOutputs`, which
immediately frees the connection back to the pool before returning the
protocol version. That way we've already freed the connection by the
time we call `RemoteStore::queryPartialDerivationOutputMap`.
 2020-11-16 02:35:50 -05:00
Robert Hensing ac5081d280 nix-build: Fix #4197 output order regression 2020-11-13 17:49:27 +01:00
Christian Höppner c4c3c15c19 Fix default nix-path 
The default nix-path values for nixpkgs and root channels were
incorrect.
 2020-11-12 15:46:08 +00:00
Ben Burdette b327de9c2d change message 2020-11-11 11:09:59 -07:00
Ben Burdette 8895fa70a4 pare down the error message 2020-11-11 11:05:21 -07:00
Ben Burdette 3edfe6090e missing argument error 2020-11-11 09:29:32 -07:00
Matthew Kenigsberg 8abb80a478 Print built derivations as json for build 
Add --json option to nix build to allow machine readable output on
stdout with all built derivations

Fixes #1930
 2020-11-11 10:27:02 -06:00
Ben Burdette 7d9037035e usage example location 2020-11-11 09:21:26 -07:00
Eelco Dolstra 4badb6943f Fix use of dirty Git/Mercurial inputs with chroot stores 
Fixes:

  $ nix build --store /tmp/nix /home/eelco/Dev/patchelf#hydraJobs.build.x86_64-linux
  warning: Git tree '/home/eelco/Dev/patchelf' is dirty
  error: --- RestrictedPathError ------------------------------------------------------------------------------------------- nix
  access to path '/tmp/nix/nix/store/xmkvfmffk7xfnazykb5kx999aika8an4-source/flake.nix' is forbidden in restricted mode
  (use '--show-trace' to show detailed location information)
 2020-11-10 23:22:45 +01:00
Eelco Dolstra cdc840d60b
Merge pull request #4242 from wizeman/fix-stack-overflow 
Fix stack overflow introduced in #4206
 2020-11-10 11:51:00 +01:00
Eelco Dolstra 3f680c1dcc
Merge pull request #4233 from Kha/master 
nix develop: Preserve stdin with `-c`
 2020-11-10 10:51:48 +01:00
Ricardo M. Correia 108a2dab7e Fix stack overflow introduced in #4206 2020-11-10 04:25:24 +01:00
Ben Burdette d8ef423a18 error message formatting 2020-11-09 19:16:50 -07:00
Ben Burdette 9f2b25ce55 remove unused ftn; reformat line breaks 2020-11-09 17:17:47 -07:00
Ben Burdette 6c2933a8d7 add position 2020-11-09 17:04:52 -07:00
Ben Burdette 107c91f5fe auto-call error 2020-11-09 16:48:35 -07:00
Sebastian Ullrich fb7735e4cf nix develop: Preserve stdin with -c 2020-11-09 22:43:14 +01:00
Eelco Dolstra b87f84cf55
Fix appending to Setting<StringSet> 
Fixes: warning: unknown setting 'extra-sandbox-paths'
 2020-11-09 15:04:34 +01:00
Maximilian Bosch 3a63fc6cd5
Allow substituting paths when building remotely using ssh-ng:// 
Until now, it was not possible to substitute missing paths from e.g.
`https://cache.nixos.org` on a remote server when building on it using
the new `ssh-ng` protocol.

This is because every store implementation except legacy `ssh://`
ignores the substitution flag passed to `Store::queryValidPaths` while
the `legacy-ssh-store` substitutes the remote store using
`cmdQueryValidPaths` when the remote store is opened with `nix-store
--serve`.

This patch slightly modifies the daemon protocol to allow passing an
integer value suggesting whether to substitute missing paths during
`wopQueryValidPaths`. To implement this on the daemon-side, the
substitution logic from `nix-store --serve` has been moved into a
protected method named `Store::substitutePaths` which gets currently
called from `LocalStore::queryValidPaths` and `Store::queryValidPaths`
if `maybeSubstitute` is `true`.

Fixes #2770
 2020-11-05 20:12:37 +01:00
Eelco Dolstra 387f824cab
Merge pull request #4206 from hercules-ci/fix-coroutine-gc 
Fix memory corruption caused by GC-invisible coroutine stacks
 2020-11-05 10:18:31 +01:00
Eelco Dolstra 5e6eabe155 Fix error message 'assertion failed at' 2020-11-03 14:45:24 +01:00
Eelco Dolstra e8c379555f LocalStore: Get rid of recursive_mutex 2020-11-03 14:45:24 +01:00
Eelco Dolstra 797a52e31d Add FIXME 2020-11-03 14:45:24 +01:00
Eelco Dolstra 550e11f077 nix repl: Fix handling of multi-line expressions 2020-11-02 19:07:37 +01:00
Eelco Dolstra 7cf874c17d Don't use readDerivation() in addValidPath() 
readDerivation() requires a valid path.

Fixes #4210.
 2020-11-02 18:46:44 +01:00
Eelco Dolstra ab2ef851b6
Merge pull request #4207 from hercules-ci/fix-RemoteStore-filterSource-deadlock 
Fix RemoteStore pool deadlock in filterSource etc
 2020-11-02 14:46:10 +01:00
Eelco Dolstra db5424bf09 Don't send eval-related settings to the daemon 2020-11-02 13:57:58 +01:00
Robert Hensing e8a45d07bc Restore RestrictedStore.addToStoreFromDump implementation 
It was accidentally removed in commit ca30abb3fb
 2020-10-31 23:56:03 +01:00
Robert Hensing b43c13a916 BoehmGCStackAllocator: increase stack size to 8MB 
The default stack size was not based on the normal stack size and
was too small.
 2020-10-30 23:18:26 +01:00
Robert Hensing 2192cac634 Fix RemoteStore pool deadlock in filterSource etc 2020-10-30 21:47:34 +01:00
Robert Hensing c4d903ddb0 Fix memory corruption caused by GC-invisible coroutine stacks 
Crucially this introduces BoehmGCStackAllocator, but it also
adds a bunch of wiring to avoid making libutil depend on bdw-gc.

Part of the solutions for #4178, #4200
 2020-10-30 21:21:59 +01:00
Eelco Dolstra dc5696b84f Fix test 2020-10-30 12:00:53 +01:00
Eelco Dolstra 7f56cf67ba Fix assertion failure in tab completion for --option 2020-10-29 18:26:35 +01:00
Eelco Dolstra ff4dea63c9 Generalize extra-* settings 
This removes the extra-substituters and extra-sandbox-paths settings
and instead makes every array setting extensible by setting
"extra-<name> = <value>" in the configuration file or passing
"--<name> <value>" on the command line.
 2020-10-29 18:17:39 +01:00
Eelco Dolstra bb8e837e4c
Merge pull request #4199 from stefanjaax/replaceWantedBySpecified 
Alter "wanted:" to "specified:" in hash mismatch output
 2020-10-29 07:38:38 +01:00
Eelco Dolstra 662e67f8de
Merge pull request #4198 from mkenigs/capitalize-JSON 
Capitalize JSON for consistency
 2020-10-29 07:34:34 +01:00
stev 869c0321ff Alter "wanted:" to "specified:" in hash mismatch output 
This makes it even clearer which of the two hashes was specified in the
nix files. Some may think that "wanted" and "got" is obvious, but:
"got" could mean "got in nix file" and "wanted" could mean "want to see in nix file".
 2020-10-29 00:33:14 +01:00
Matthew Kenigsberg 6a4bf535d8 Capitalize JSON for consistency 2020-10-28 17:54:28 -05:00
Eelco Dolstra a5019f0508 Consistency 2020-10-28 20:45:57 +01:00
Eelco Dolstra 02a1facbdc
Merge pull request #4056 from tweag/non-ca-depending-on-ca 
Allow non-CA derivations to depend on CA ones
 2020-10-27 17:38:29 +01:00
regnat bc081bcd81 Inline unkownHashes 
See https://github.com/NixOS/nix/pull/4056#discussion_r493661632
 2020-10-27 07:29:25 +01:00
regnat c092fa4702 Allow non-CA derivations to depend on CA derivations 2020-10-27 07:29:23 +01:00
Eelco Dolstra 343239fc8a Allow nix.conf options to be set in flake.nix 
This makes it possible to have per-project configuration in flake.nix,
e.g. binary caches and other stuff:

  nixConfig.bash-prompt-suffix = "ngi# ";
  nixConfig.substituters = [ "https://cache.ngi0.nixos.org/" ];
 2020-10-26 20:45:39 +01:00
Eelco Dolstra 731edf0d9b isTrivial(): Support trivial lists 2020-10-26 20:37:11 +01:00
Eelco Dolstra 14aecbb288 BaseSetting<StringMap>::set(): Don't append to previous value 2020-10-26 20:36:46 +01:00
Eelco Dolstra b875b8f45c Remove edition field 2020-10-26 17:59:36 +01:00
Eelco Dolstra 9d5e9ef0da Move Explicit 2020-10-26 17:01:20 +01:00
Eelco Dolstra dc7d1322ef Make the prompt used in development shells configurable 2020-10-26 14:24:25 +01:00
Eelco Dolstra 750ce500c2 Fix clang build 2020-10-22 13:40:30 +02:00
Eelco Dolstra f9438fb64a nix develop: Add --redirect flag to redirect dependencies 
This is primarily useful if you're hacking simultaneously on a package
and one of its dependencies. E.g. if you're hacking on Hydra and Nix,
you would start a dev shell for Nix, and then a dev shell for Hydra as
follows:

  $ nix develop \
    --redirect .#hydraJobs.build.x86_64-linux.nix ~/Dev/nix/outputs/out \
    --redirect .#hydraJobs.build.x86_64-linux.nix.dev ~/Dev/nix/outputs/dev

(This assumes hydraJobs.build.x86_64-linux has a passthru.nix
attribute. You can also use a store path.)

This causes all references in the environment to those store paths to
be rewritten to ~/Dev/nix/outputs/{out,dev}. Note: unfortunately, you
may need to set LD_LIBRARY_PATH=~/Dev/nix/outputs/out/lib because
Nixpkgs' ld-wrapper only adds -rpath entries for -L flags that point
to the Nix store.
 2020-10-22 13:40:30 +02:00
Eelco Dolstra e556a1beb7 nix develop: Handle 'declare -ax' in bash output 
Fixes 'nix develop nixpkgs#qpdfview'.
 2020-10-21 17:54:21 +02:00
Christian Kampka 461cf2b856
Add NIX_CONFIG env var for applying nix.conf overrides 2020-10-21 13:41:26 +02:00
Matthew Kenigsberg f6aaac2b59 Make bash non-interactive for nix develop --phase 
Fix #3975: Currently if Ctrl-C is pressed during a phase, the interactive subshell
is not exited. Removing --rcfile when --phase is present makes bash
non-interactive
 2020-10-20 12:00:30 -05:00
Eelco Dolstra 9635fb77bd
Merge pull request #4080 from kquick/kwq/flake-int-doc 
Add some internal documentation for flake support objects.
 2020-10-19 11:29:12 +02:00
Eelco Dolstra fda835b231
Merge pull request #4143 from obsidiansystems/typed-goal-maps 
Properly type the derivation and substitution goal maps
 2020-10-18 18:12:21 +02:00
Robert Hensing bd9eb5c743 DerivationGoal: only retry if output closure incomplete is only problem 2020-10-18 14:26:37 +02:00
Robert Hensing 94f1e4a441 Typo 2020-10-18 14:26:37 +02:00
John Ericson 7ed46c1574 Explain that upcast_goal is still a static cast 2020-10-17 21:50:12 +00:00
John Ericson 57d0432b39 Just use auto in two places. 2020-10-17 21:47:52 +00:00
John Ericson e6f8ae56d8 tab -> space 2020-10-17 21:45:31 +00:00
John Ericson 619d262c97 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-17 21:44:27 +00:00
John Ericson 64be1c15c2 Add missing include for MAX_PATH 
And remove one that we didn't actually need to add
 2020-10-15 19:05:17 +00:00
John Ericson fccef6a7fa Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-10-15 18:55:03 +00:00
John Ericson 0fefc2a439 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-14 20:49:01 +00:00
Eelco Dolstra 11882d7c7c Create /etc/passwd *after* figuring out the sandbox uid/gid 
Fixes build failures like

  # nix log /nix/store/gjaa0psfcmqvw7ivggsncx9w364p3s8s-sshd.conf-validated.drv
  No user exists for uid 30012
 2020-10-14 12:20:58 +02:00
John Ericson 55592b253f Add some more docs 2020-10-13 18:04:24 +00:00
John Ericson 13804f126e Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-13 18:02:32 +00:00
Eelco Dolstra 2653801939 Merge branch 'split_build_cc' of https://github.com/obsidiansystems/nix 2020-10-13 15:36:55 +02:00
John Ericson 1b8ebe92dc Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps 2020-10-12 20:47:22 +00:00
John Ericson a73959e6be Merge remote-tracking branch 'upstream/master' into split_build_cc 2020-10-12 17:20:46 +00:00
John Ericson d334fd4882 Merge branches 'split_build_hh-0', 'split_build_hh-1', 'split_build_hh-2', 'split_build_hh-3' and 'split_build_hh-4' into split_build_cc 2020-10-12 17:20:12 +00:00
John Ericson 542972f029 Trim worker.hh 2020-10-12 17:16:48 +00:00
John Ericson 0e2306204a Rename to hand-hold git (worker.hh) 2020-10-12 17:16:48 +00:00
John Ericson 3ffa3546bd Trim substitution-goal.hh 2020-10-12 17:16:25 +00:00
John Ericson e77a2344d5 Rename to hand-hold git (substitution-goal.hh) 2020-10-12 17:16:25 +00:00
John Ericson d585b4c54f Trim hook-instance.hh 2020-10-12 17:16:13 +00:00
John Ericson 10b749a156 Rename to hand-hold git (hook-instance.hh) 2020-10-12 17:16:13 +00:00
John Ericson 8067d32f2a Trim goal.hh 2020-10-12 17:16:00 +00:00
John Ericson 0d0e345cdc Rename to hand-hold git (goal.hh) 2020-10-12 17:16:00 +00:00
John Ericson 2ce726947a Trim derivation-goal.hh 2020-10-12 17:15:32 +00:00
John Ericson 4eb8c69853 Rename to hand-hold git (derivation-goal.hh) 2020-10-12 17:15:32 +00:00
John Ericson f7099965bf Change .cc files to use split build headers 2020-10-12 17:08:52 +00:00
John Ericson 5a97621d6d Prepare for build/*.hh headers 2020-10-12 17:07:51 +00:00
volth eee18f88dd
Handle amount of disk space saved by hard linking being negative 
Fixes bogus messages like "currently hard linking saves 17592186044416.00 MiB".
 2020-10-12 16:06:38 +00:00
John Ericson 15fdb7cc6b Split out uds-remote-store.{cc.hh} 2020-10-11 17:37:05 +00:00
John Ericson 38e3897162 Copy {uds-,}remote-store.{cc,hh} 
This prepares for the splitting that happens in the next commit.
 2020-10-11 17:18:26 +00:00
John Ericson 5c74a6147b Properly type the derivation and substitution goal maps 
As a bonus, Worker::removeGoal is less inefficient.
 2020-10-11 17:07:14 +00:00
Eelco Dolstra b0fbf3a653
Merge pull request #4135 from obsidiansystems/split-local-fs-store-header 
Split out `local-fs-store.hh`
 2020-10-11 18:49:40 +02:00
John Ericson 8cc510fb79 Merge branches 'build-1', 'build-2', 'build-3', 'build-4', 'build-5', 'build-6', 'build-7' and 'build-8' into split_build_cc 2020-10-11 16:44:34 +00:00
John Ericson bcb67e1ed8 Trim lock.cc 2020-10-11 16:44:19 +00:00
John Ericson e0be04129b Rename to hand-hold git (lock.cc) 2020-10-11 16:44:14 +00:00
John Ericson dbc588651c Trim lock.hh 2020-10-11 16:44:07 +00:00
John Ericson d0004bfcab Rename to hand-hold git (lock.hh) 2020-10-11 16:43:59 +00:00
John Ericson eed53ed87a Trim build/worker.cc 2020-10-11 16:43:51 +00:00
John Ericson 904e315dae Rename to hand-hold git (build/worker.cc) 2020-10-11 16:43:42 +00:00
John Ericson d24ffe0eb1 Trim build/substitution-goal.cc 2020-10-11 16:43:24 +00:00
John Ericson 3633b3572b Rename to hand-hold git (build/substitution-goal.cc) 2020-10-11 16:43:18 +00:00
John Ericson 4bdff7d1b0 Trim build/local-store-build.cc 2020-10-11 16:43:12 +00:00
John Ericson dc5225cde5 Rename to hand-hold git (build/local-store-build.cc) 2020-10-11 16:43:05 +00:00
John Ericson 159054f730 Trim build/hook-instance.cc 2020-10-11 16:42:35 +00:00
John Ericson f0b8987299 Rename to hand-hold git (build/hook-instance.cc) 2020-10-11 16:42:08 +00:00
John Ericson 819fe848ac Trim build/goal.cc 2020-10-11 16:41:58 +00:00
John Ericson 184bfc301e Rename to hand-hold git (build/goal.cc) 2020-10-11 16:41:18 +00:00
John Ericson 3bab1c5bb0 Trim build/derivation-goal.cc 2020-10-11 16:41:11 +00:00
John Ericson 9629290eda Rename to hand-hold git (build/derivation-goal.cc) 2020-10-11 16:40:52 +00:00
John Ericson a4f0fecb03 Trim build.hh 2020-10-11 16:40:34 +00:00
John Ericson fc72cb0760 Rename to hand-hold git (build.hh) 2020-10-11 16:40:14 +00:00
John Ericson 428536fd75 Prepare for build/* files 2020-10-11 16:39:08 +00:00
John Ericson aef44cbaa9 Split out commonChildInit 2020-10-11 16:38:46 +00:00
John Ericson 6cc1541782 Split out local-fs-store.hh 
This matches the already-existing `local-fs-store.cc`.
 2020-10-09 20:18:08 +00:00
Eelco Dolstra 725488b892 nix develop: Unset $HOSTNAME 
This is set to "localhost" by stdenv which is probably not what you
want.
 2020-10-09 22:03:18 +02:00
Eelco Dolstra 44349064f7 nix develop: Source ~/.bashrc 
Fixes #4104.
 2020-10-09 22:02:00 +02:00
Eelco Dolstra 59bd6e87a4 Completions::add(): Guard against newlines 2020-10-09 21:55:59 +02:00
Eelco Dolstra ea4b2b985f
Merge pull request #4128 from tweag/extended-completions 
Add a zsh completion script
 2020-10-09 21:53:12 +02:00
John Ericson 39de73550d Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-10-09 18:26:47 +00:00
Eelco Dolstra e845d19ae3 Remove Lazy 
This fixes a crash during startup when compiling Nix as a single
compilation unit.
 2020-10-09 17:54:59 +02:00
Eelco Dolstra 87157b2bd3 writeFile(): Add error context to writeFull() failure 
Issue #4092.
 2020-10-09 16:02:53 +02:00
Eelco Dolstra 636ec17139
Remove stray DerivationOutputsAndPaths type 2020-10-09 15:41:24 +02:00
regnat 04e5d0e704 Add a description in the completion outputs 
Make nix output completions in the form `completion\tdescription`.
This can't be used by bash (afaik), but other shells like zsh or fish
can display it along the completion choices
 2020-10-09 09:39:51 +02:00
Eelco Dolstra 97ffc1e013 Dynamically disable user namespaces if CLONE_NEWUSER fails 
This makes builds work inside nixos-enter.

Fixes #3145.
 2020-10-07 22:46:01 +02:00
Eelco Dolstra 6aa64627c8 Support user namespaces being disabled 
If max_user_namespaces is set to 0, then don't run the build in a user
namespace.

Fixes #4092.
 2020-10-07 22:02:36 +02:00
Eelco Dolstra f66bbd8c7b Doh 2020-10-07 21:25:06 +02:00
Eelco Dolstra e705c24294 Tweak error messages 2020-10-07 17:28:43 +02:00
Eelco Dolstra c43e882f54 Serialize exceptions from the daemon to the client 2020-10-07 17:13:54 +02:00
Eelco Dolstra be149acfda Serialize exceptions from the sandbox process to the parent 
Fixes #4118.
 2020-10-07 16:34:03 +02:00
Eelco Dolstra 27ca87c46a Formatting 2020-10-07 16:33:19 +02:00
John Ericson 57d960dcd1 Remove generic std::optional<T> suppport from worker proto 
See comment for rational; I think it's good to leave a comment lest
anyone is tempted to add such a sum-type instance again.

Fixes #4113
 2020-10-07 12:50:37 +00:00
Maximilian Bosch 59f2dd8e8d
libfetchers/github: allow slashes in refs 
Refs #4061
 2020-10-06 20:08:51 +02:00
Eelco Dolstra d761485010 Prevent a deadlock when user namespace setup fails 
Observed on Centos 7 when user namespaces are disabled:
DerivationGoal::startBuilder() throws an exception, ~DerivationGoal()
waits for the child process to exit, but the child process hangs
forever in drainFD(userNamespaceSync.readSide.get()) in
DerivationGoal::runChild(). Not sure why the SIGKILL doesn't get
through.

Issue #4092.
 2020-10-06 18:57:23 +02:00
Eelco Dolstra ad143c5b3b Shut up some clang warnings 2020-10-06 14:52:30 +02:00
Eelco Dolstra 85c8be6286 Remove static variable name clashes 
This was useful for an experiment with building Nix as a single
compilation unit. It's not very useful otherwise but also doesn't
hurt...
 2020-10-06 13:49:20 +02:00
Eelco Dolstra 0419cd2695 Remove unneeded -lboost_* flags 2020-10-06 13:34:58 +02:00
Eelco Dolstra 636455c471 Remove 'using namespace fetchers' 2020-10-06 11:16:32 +02:00
Eelco Dolstra 6691256e79 Factor out common showBytes() 2020-10-06 10:40:49 +02:00
Eelco Dolstra d0bb544128 Add missing #pragma once 2020-10-06 10:40:07 +02:00
Eelco Dolstra 88a667e49e
Fix s3:// store 
Fixes https://github.com/NixOS/nixos-org-configurations/issues/123.
 2020-10-05 17:53:30 +02:00
Eelco Dolstra 51c299213b
Merge pull request #3935 from obsidiansystems/binary-cache-addToStoreFromDump 
Get rid of Hash::dummy from BinaryCacheStore
 2020-10-05 14:41:28 +02:00
Eelco Dolstra f3aba88737
Merge pull request #3895 from obsidiansystems/templated-daemon-protocol 
More templated STL support for the daemon protocol
 2020-10-05 14:40:27 +02:00
Fabian Möller d5d196b0a1
Fix profile update in nix command 2020-10-02 12:10:31 +02:00
DavHau f3280004e2 add more examples to --help of nix run 2020-10-01 11:34:13 +07:00
Eelco Dolstra 20a1e20d91 Style 2020-09-30 13:35:56 +02:00
Eelco Dolstra 274357eb6a Simplify 2020-09-30 12:09:18 +02:00
Eelco Dolstra 002ce8449d Merge branch 'access-tokens' of https://github.com/kquick/nix 2020-09-30 11:35:15 +02:00
John Ericson 69afaeace3 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-30 00:42:28 +00:00
John Ericson b759701652 nix::worker_proto -> worker_proto 2020-09-30 00:41:18 +00:00
John Ericson 45a0ed82f0 Revert "Use template structs instead of phantoms" 
This reverts commit 9ab07e99f5.
 2020-09-30 00:39:06 +00:00
Kevin Quick 5e7838512e
Remove github-access-token in favor of access-token. 2020-09-29 16:26:34 -07:00
Eelco Dolstra 76f000712e Merge branch 'kwq/path-dir-query' of https://github.com/kquick/nix 2020-09-29 23:42:43 +02:00
Eelco Dolstra 64e9b3c83b nix registry list: Show 'dir' attribute 
Issue #4050.
 2020-09-29 23:33:16 +02:00
Kevin Quick 66c3959e8c
Merge branch 'master' into access-tokens 2020-09-29 08:32:06 -07:00
Eelco Dolstra 5999978a05 Make Headers an optional argument 2020-09-29 13:05:19 +02:00
Eelco Dolstra de86abbf3f Cleanup 2020-09-29 12:55:06 +02:00
Eelco Dolstra cebd2fc35d Merge branch 'github-api-token' of https://github.com/imalsogreg/nix 2020-09-29 12:17:00 +02:00
John Ericson 00135e13f4 Clarify comment a bit 2020-09-28 18:19:10 +00:00
Kevin Quick 887be7b6f2
Switch comment format from '// ...' to '/* ... */' for consistency. 2020-09-28 09:37:26 -07:00
Kevin Quick 128c98ab09
Clarification in the description of the FlakeInput. 2020-09-28 09:34:23 -07:00
Kevin Quick 5ae164b7cf
Update description of FlakeRef, incorporating suggestion. 2020-09-28 09:23:05 -07:00
Kevin Quick bcb3da3b6b
Fix spelling error. 2020-09-28 08:58:14 -07:00
John Ericson 80e335bb58 Use drvPath2 and give it a better name 2020-09-28 15:43:56 +00:00
John Ericson 10202bbf29 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-28 15:39:11 +00:00
John Ericson 6c31297d80
Update src/libstore/binary-cache-store.cc 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-09-28 11:32:58 -04:00
Eelco Dolstra 649c465873
Merge pull request #4064 from serokell/balsoft/fix-max-jobs 
Fix max-jobs option
 2020-09-28 10:42:02 +02:00
Eelco Dolstra 5df687c316
Merge pull request #4077 from Ma27/left-whitespace-log 
libmain/progress-bar: don't trim whitespace on the left
 2020-09-28 10:40:00 +02:00
Kevin Quick 5885b0cfd8
Miscellaneous spelling fixes in comments. (#4071) 2020-09-27 20:04:06 +00:00
Kevin Quick bd5328814f
Add some internal documentation for flake support objects. 2020-09-26 14:32:58 -07:00
Maximilian Bosch a76fb07314
libmain/progress-bar: don't trim whitespace on the left 
When running `nix build -L` it can be fairly hard to read the output if
the build program intentionally renders whitespace on the left. A
typical example is `g++` displaying compilation errors.

With this patch, the whitespace on the left is retained to make the log
more readable:

```
foo> no configure script, doing nothing
foo> building
foo> foobar.cc: In function 'int main()':
foo> foobar.cc:5:5: error: 'wrong_func' was not declared in this scope
foo>     5 |     wrong_func(1);
foo>       |     ^~~~~~~~~~
error: --- Error ------------------------------------------------------------------------------------- nix
error: --- Error --- nix-daemon
builder for '/nix/store/i1q76cw6cyh91raaqg5p5isd1l2x6rx2-foo-1.0.drv' failed with exit code 1
```
 2020-09-26 17:38:11 +02:00
John Ericson 25fffdda86
Remove redundant nar hash and size setting 
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2020-09-26 10:17:30 -04:00
John Ericson 1832436526 Fix up BinaryCacheStore::addToStore taking a path 2020-09-26 04:56:29 +00:00
John Ericson 5db83dd771 BinaryCacheStore::addTextToStore include CA field 2020-09-26 03:21:36 +00:00
John Ericson c40c832f19 Merge remote-tracking branch 'obsidian/master' into binary-cache-addToStoreFromDump 2020-09-26 00:21:45 +00:00
Kevin Quick cb186f1e75
Use "?dir=..." portion of "registry add" local path specification. 
The registry targets generally follow a URL formatting schema with
support for a query parameter of "?dir=subpath" to specify a sub-path
location below the URL root.

Alternatively, an absolute path can be specified.  This specification
mode accepts the query parameter but ignores/drops it.  It would
probably be better to either (a) disallow the query parameter for the
path form, or (b) recognize the query parameter and add to the path.

This patch implements (b) for consistency, and to make it easier for
tooling that might switch between a remote git reference and a local
path reference.

See also issue #4050.
 2020-09-25 09:36:18 -07:00
Gregory Hale faa5607f54 Merge remote-tracking branch 'origin/master' into github-api-token 2020-09-25 12:10:58 -04:00
John Ericson cfe791a638 stdout_ -> cout 
Better to get creative than just sprinkle arbitrary underscores.
 2020-09-25 11:30:04 -04:00
Eelco Dolstra 7d81582488
Merge pull request #3953 from obsidiansystems/basic-derivation-goal-outputs 
Deduplicate basic derivation goals too
 2020-09-25 17:21:19 +02:00
Kevin Quick 5a35cc29bf
Re-add support for github-access-token, but mark as deprecated. 2020-09-25 08:09:56 -07:00
Kevin Quick ef2a14be19
Fix reference to older name for access-tokens config value. 2020-09-25 08:08:27 -07:00
Eelco Dolstra cbb9862cd9
Merge pull request #3626 from W95Psp/master 
Make `functionArgs` primitive accept primops (fix #3624)
 2020-09-25 15:14:18 +02:00
Eelco Dolstra 7b2ae472ff
expectArg(): Respect the 'optional' flag 2020-09-25 10:27:40 +02:00
Kevin Quick 8fba2a8b54
Update to use access-tokens configuration for github/gitlab access. 
This change provides support for using access tokens with other
instances of GitHub and GitLab beyond just github.com and
gitlab.com (especially company-specific or foundation-specific
instances).

This change also provides the ability to specify the type of access
token being used, where different types may have different handling,
based on the forge type.
 2020-09-24 22:49:44 -07:00
Kevin Quick c2f48cfcee
Complete conversion of "url" to "host" with associated variable renaming. 
Completes the change begun in commit 56f1e0d to consistently use the
"host" attribute for "github" and "gitlab" inputs instead of a "url"
attribute.
 2020-09-24 22:46:03 -07:00
Kevin Quick a439e9488d
Support StringMap configuration settings. 
Allows Configuration values that are space-separated key=value pairs.
 2020-09-24 22:42:59 -07:00
Paul Opiyo 4d863a9fcb Remove redundant value checks 
std::optional had redundant checks for whether it had a value.
An object is emplaced either way so it can be dereferenced
without repeating a value check
 2020-09-24 18:32:03 -05:00
Alexander Bantyev ed218e1d6c
Fix max-jobs option 
After 0ed946aa61, max-jobs setting (-j/--max-jobs)
stopped working.

The reason was that nrLocalBuilds (which compared to maxBuildJobs to figure
out whether the limit is reached or not) is not incremented yet when tryBuild
is started; So, the solution is to move the check to tryLocalBuild.

Closes https://github.com/nixos/nix/issues/3763
 2020-09-25 00:07:42 +03:00
Kevin Quick bd5f3dbe11
Fixes fall-through to report correct description of hash-file command. 2020-09-24 12:30:03 -07:00
Eelco Dolstra 8d9402f411
Merge pull request #4054 from edolstra/fix-4021 
registerOutputs(): Don't call canonicalisePathMetaData() twice
 2020-09-23 21:57:53 +02:00
Eelco Dolstra 4ce8a3ed45 Hopefully fix EPERM on macOS 2020-09-23 21:29:10 +02:00
Eelco Dolstra 9a24ece122 Fix exception 2020-09-23 20:21:08 +02:00
Eelco Dolstra 236d9ee7f7 lstat() cleanup 2020-09-23 19:17:28 +02:00
Eelco Dolstra 688bd4fb50 After rewriting a path, make it read-only 2020-09-23 19:10:16 +02:00
Dominique Martinet 2548347bba libutil/archive: add preallocate-contents option 
Make archive preallocation (fallocate) optional because some filesystems
like btrfs do not behave as expected with fallocate.

See #3550.
 2020-09-23 18:49:11 +02:00
Eelco Dolstra cec9473871 DerivationGoal::registerOutputs(): Don't canonicalize twice 
Fixes #4021.
 2020-09-23 18:21:22 +02:00
Eelco Dolstra d4f8163d10 canonicalisePathMetaData_(): Change assertion to error message 2020-09-23 18:21:22 +02:00
Eelco Dolstra e8f0b1e996 DerivationGoal::registerOutputs(): Fix bad format string 2020-09-23 18:21:22 +02:00
John Ericson 3f226f71c1 Return more info from BinaryCacheStore::addToStoreCommon 
We don't need it yet, but we could/should in the future, and it's a
cost-free change since we already have the reference. I like it.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2020-09-23 14:40:41 +00:00
John Ericson 412b3a54fb
Clarify FIXME in BinaryCacheStore::addToStoreCommon 
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2020-09-23 10:36:55 -04:00
Eelco Dolstra 8a2e10827f Remove unused Flake::vOutputs field 2020-09-23 14:08:52 +02:00
Domen Kožar dac8a1a5e1
Merge pull request #4052 from ujjwaljainnn/fix-typo 
fixed typo
 2020-09-23 09:58:30 +02:00
John Ericson 9fbc31a65b Get rid of Hash::dummy from BinaryCacheStore 2020-09-23 04:56:04 +00:00
ujjwal a2f5c921d4 fixed typo 2020-09-22 23:37:06 +05:30
John Ericson 993229cdaf Deduplicate basic derivation goals too 
See comments for security concerns.

Also optimize goal creation by not traversing map twice.
 2020-09-22 17:13:59 +00:00
John Ericson e9fc2031f0 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 14:18:31 +00:00
regnat 67cc356bb3 Merge remote-tracking branch 'origin/master' into non-ca-depending-on-ca 2020-09-22 16:11:25 +02:00
Eelco Dolstra 980edd1f3a RemoteStore::addCAToStore(): Don't hold connection while calling queryPathInfo() 
This leads to a deadlock if we're at the connection limit.
 2020-09-22 15:28:20 +02:00
Eelco Dolstra 5b107f2c5f
Merge pull request #4038 from maljub01/master 
Add a nix.conf option for allowing a symlinked store
 2020-09-22 13:19:22 +02:00
Eelco Dolstra 35a0ac1838 Style fixes 2020-09-22 11:40:19 +02:00
Eelco Dolstra 92ac8df0ec Merge branch 'add-ca-to-store' of https://github.com/hercules-ci/nix 2020-09-22 11:31:33 +02:00
Eelco Dolstra e7f1109f06
Merge pull request #4040 from OmnipotentEntity/master 
Fix compatibility with nlohmann-json 3.9.1
 2020-09-22 11:09:25 +02:00
Eelco Dolstra 7dd8baafe1
Merge pull request #4041 from cole-h/enum-stringify 
Serialize SandboxMode enum to string for JSON
 2020-09-22 11:08:56 +02:00
regnat c1e79f870c Silence a compiler warning in serialise.hh 
Explicitely cast to `uint64_t` in `readNum` to avoid a "comparison
between signed and unsigned" warning
 2020-09-22 10:39:29 +02:00
John Ericson 3786a801c3 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-22 04:15:55 +00:00
John Ericson b92d3b2edd Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 00:45:55 +00:00
Michael Reilly d860295e11 Bump nlohmann-json version to 3.9.1 2020-09-21 14:24:43 -04:00
Cole Helbling ba37299a03
Serialize SandboxMode enum to string for JSON 
Rather than showing an integer as the default, instead show the boolean
referenced in the description.

The nix.conf.5 manpage used to show "default: 0", which is unnecessarily
opaque and confusing (doesn't 0 mean false, even though the default is
true?); now it properly shows that the default is true.
 2020-09-21 10:36:45 -07:00
Eelco Dolstra ecc8672aa0 fmt.hh: Don't include boost/algorithm/string/replace.hpp 
This cuts compilation time by ~49s.

Issue #4045.
 2020-09-21 19:07:55 +02:00
Eelco Dolstra 557d2427ee Random header cleanup 2020-09-21 18:59:02 +02:00
Eelco Dolstra 0716adaa8b abstractsettingtojson.hh -> abstract-setting-to-json.hh 2020-09-21 18:49:43 +02:00
Eelco Dolstra 340ca382c4 Don't include nlohmann/json.hpp in globals.hh 
This reduces compilation time by 207s.

Issue #4045.
 2020-09-21 18:47:18 +02:00
Eelco Dolstra d51ba43047 Move Callback into its own header 
This gets rid of the inclusion of <future> in util.hh, cutting
compilation time by ~20s (CPU time).

Issue #4045.
 2020-09-21 18:42:21 +02:00
Marwan Aljubeh f80ffeb8c9 Update the variable name accordingly 2020-09-21 17:29:08 +01:00
Eelco Dolstra e8e1d420f3 Don't include <regex> in header files 
This reduces compilation time by ~15 seconds (CPU time).

Issue #4045.
 2020-09-21 18:22:45 +02:00
Marwan Aljubeh 4e1a04733d Use a better name for the config option 2020-09-21 16:32:22 +01:00
Eelco Dolstra cbe0bb29f4
Merge pull request #4035 from Ma27/url-attr 
libfetchers/github: allow `url` attribute
 2020-09-21 17:26:36 +02:00
Maximilian Bosch 56f1e0df05
libfetchers/github: rename url to host 2020-09-21 16:29:08 +02:00
Robert Hensing ca30abb3fb Document addCAToStore/addToStoreFromDump source drainage 
Also checked that all usages satisfy the requirement and
removed dead code.
 2020-09-21 07:55:47 +02:00
Robert Hensing fa08db5c4c wopAddToStore: return ValidPathInfo 
A ValidPathInfo is created anyway. By returning it we can save a
roundtrip and we have a nicer interface.
 2020-09-21 07:55:47 +02:00
Robert Hensing 7c68264085 wopAddToStore: add RepairFlag 2020-09-21 07:55:47 +02:00
Robert Hensing fbf509c113 parseContentAddressMethodPrefix: use string_view 
Co-authored-by: John Ericson <git@JohnEricson.me>
 2020-09-21 07:55:47 +02:00
Robert Hensing 8279178b07 Move FramedSink next to FramedSource 2020-09-21 07:55:47 +02:00
Robert Hensing ecc8088cb7 wopAddToStore: Throw to clarify unused refs 
Co-authored-by: John Ericson <git@JohnEricson.me>
 2020-09-21 07:55:47 +02:00
Robert Hensing c602ebfb34 Refactor wopAddToStore to make wopAddTextToStore obsolete 2020-09-21 07:55:45 +02:00
Robert Hensing e34fe47d0c Overhaul wopAddToStore 2020-09-21 07:54:05 +02:00
Marwan Aljubeh e40772cd35 Lint issue: replacing tabs with spaces 2020-09-18 17:18:45 +01:00
Marwan Aljubeh c00e078343 Add a nix.conf option for allowing a symlinked store 2020-09-18 17:10:39 +01:00
Bryan Richter 5fe375a8f1
nix-prefetch-url: Add --executable flag 
pkgs.fetchurl supports an executable argument, which is especially nice
when downloading a large executable. This patch adds the same option to
nix-prefetch-url.

I have tested this to work on the simple case of prefetching a little
executable:

1. nix-prefetch-url --executable https://my/little/script
2. Paste the hash into a pkgs.fetchurl-based package, script-pkg.nix
3. Delete the output from the store to avoid any misidentified artifacts
4. Realise the package script-pkg.nix
5. Run the executable

I repeated the above while using --name, as well.

I suspect --executable would have no meaningful effect if combined with
--unpack, but I have not tried it.
 2020-09-18 19:09:45 +03:00
Maximilian Bosch 2bcf8cbe7a
libfetchers/github: allow url attribute 
Since 108debef6f we allow a
`url`-attribute for the `github`-fetcher to fetch tarballs from
self-hosted `gitlab`/`github` instances.

However it's not used when defining e.g. a flake-input

    foobar = {
        type = "github";
        url = "gitlab.myserver";
        /* ... */
    }

and breaks with an evaluation-error:

    error: --- Error --------------------------------------nix
    unsupported input attribute 'url'
    (use '--show-trace' to show detailed location information)

This patch allows flake-inputs to be fetched from self-hosted instances
as well.
 2020-09-18 14:10:45 +02:00
Eelco Dolstra 958bf57123 nix build: find() -> get() 
find() returns an iterator so "!attr" doesn't work.
 2020-09-18 13:10:42 +02:00
Robert Hensing 14b30b3f3d Move FramedSource and FramedSink, extract withFramedSink 2020-09-17 20:21:04 +02:00
Robert Hensing dfa547c6a8 Add ContentAddressMethod and parse/render it 2020-09-17 20:21:04 +02:00
Robert Hensing 29c82ccc77 Add Source.drainInto(Sink) 2020-09-17 20:21:04 +02:00
Robert Hensing 9ee3122ec7 Remove redundant import 2020-09-17 20:21:04 +02:00
Robert Hensing 3f93bc0d39 Typo 2020-09-17 20:21:04 +02:00
John Ericson b7df353f27 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-17 16:33:10 +00:00
Eelco Dolstra 649d3aaf24
Merge pull request #3829 from obsidiansystems/remove-storetype-delegate-regStore 
Remove storetype delegate reg store -- contains #3736
 2020-09-17 13:55:01 +02:00
Eelco Dolstra b94a35ef40
Merge pull request #4027 from tweag/fix-gc-of-ca-derivations 
Fix garbage collection of CA derivations
 2020-09-17 13:46:26 +02:00
regnat 520895b1da Fix garbage collection of CA derivations 
Fix #4026
 2020-09-17 13:36:58 +02:00
Eelco Dolstra 787469c7b6 Remove corepkgs/unpack-channel.nix 2020-09-17 09:41:02 +02:00
Eelco Dolstra 10d1865f5f Remove corepkgs/derivation.nix 2020-09-17 09:41:02 +02:00
John Ericson f60b380a7f Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-16 22:35:24 +00:00
John Ericson 2741fffa35 Ensure resolved CA derivations are written 
so we can link outputs to deriver and thus properly cache.
 2020-09-16 17:57:32 +00:00
John Ericson 7fdbb377ba Start to fix floating CA + remote building 2020-09-16 17:50:48 +00:00
John Ericson c5ccebae00 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-16 17:50:40 +00:00
Greg Hale a303c0b6dc Fetch commits from github/gitlab using Auth header 
`nix flake info` calls the github 'commits' API, which requires
authorization when the repository is private. Currently this request
fails with a 404.

This commit adds an authorization header when calling the 'commits' API.
It also changes the way that the 'tarball' API authenticates, moving the
user's token from a query parameter into the Authorization header.

The query parameter method is recently deprecated and will be disallowed
in November 2020. Using them today triggers a warning email.
 2020-09-16 13:46:48 -04:00
Eelco Dolstra 5080d4e7b2 Merge branch 'document-store-options' of https://github.com/tweag/nix 2020-09-16 17:02:30 +02:00
Eelco Dolstra 0066ef6c59 Fix doc generation 2020-09-16 16:56:28 +02:00
Eelco Dolstra 2eacc1bc00 builtins.toFile: Fix indentation 2020-09-16 14:18:46 +02:00
Théophane Hufschmitt 77a0e2c5be
Remove useless exception copy 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-09-16 14:00:21 +02:00
regnat d72927aa7a Fix the s3 store 
Add some necessary casts in the initialisation of the store's config
 2020-09-16 13:53:09 +02:00
regnat e0817cbcdc Don't include nlohmann/json.hpp in config.hh 
Instead make a separate header with the template implementation of
`BaseSetting<T>::toJSONObj` that can be included where needed
 2020-09-16 13:53:09 +02:00
regnat 93c0e14a30 Include the full nlohmann/json header in config.hh 
It is apparently required for using `toJSONObject()`, which we do inside
the header file (because it's in a template).

This was accidentally working when building Nix itself (presumably because
`config.hh` was always included after `nlohman/json.hpp`) but caused a
(pretty dirty) build failure in the perl bindings package.
 2020-09-16 13:53:09 +02:00
regnat fc2d31c423 Add (StoreConfig*) casts to work around a GCC bug 
Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was
already there in the code but was accidentally removed in the last
commits
 2020-09-16 13:53:09 +02:00
regnat a1e82ba450 fixup! Add a default value for the settings 2020-09-16 13:53:09 +02:00
regnat 888f7afe9f Fix build issues with gcc 2020-09-16 13:53:09 +02:00
regnat 634cb2a5ae Add a markdown output to nix describe-stores 2020-09-16 13:53:09 +02:00
regnat b73adacc1e Add a name to the stores 
So that it can be printed by `nix describe-stores`
 2020-09-16 13:53:09 +02:00
regnat f24f0888f9 Document the new store hierarchy 2020-09-16 13:53:09 +02:00
regnat d65962db4d Make uri schemes grammar more RFC-compliant 
Allow `-` and `.` in the RFC schemes as stated by
[RFC3986](https://tools.ietf.org/html/rfc3986#section-3.1).

Practically, this is needed so that `ssh-ng` is a valid URI scheme
 2020-09-16 13:53:09 +02:00
regnat 29a632386e fixup! Make the store plugins more introspectable 2020-09-16 13:53:09 +02:00
regnat 1129913c4e fixup! Correctly call all the parent contructors of the stores 2020-09-16 13:53:09 +02:00
regnat 7f103dcddd Properly filter the stores according to their declared uriSchemes 
When opening a store, only try the stores whose `uriSchemes()` include
the current one
 2020-09-16 13:53:09 +02:00
regnat 5895184df4 Correctly call all the parent contructors of the stores 
Using virtual inheritance means that only the default constructors of
the parent classes will be called, which isn't what we want
 2020-09-16 13:53:09 +02:00
regnat dae39f0a7a Make nix describe-stores functional 
Using the `*Config` class hierarchy
 2020-09-16 13:53:08 +02:00
regnat 22afa8fb4d Separate store configs from the implems 
Rework the `Store` hierarchy so that there's now one hierarchy for the
store configs and one for the implementations (where each implementation
extends the corresponding config). So a class hierarchy like

```
StoreConfig-------->Store
    |                 |
    v                 v
SubStoreConfig----->SubStore
    |                 |
    v                 v
SubSubStoreConfig-->SubSubStore
```

(with virtual inheritance to prevent DDD).

The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
 2020-09-16 13:53:08 +02:00
regnat aa4eac3788 fixup! Separate the instantiation and initialisation of the stores 2020-09-16 13:53:08 +02:00
regnat 35042c9623 Add a default value for the settings 
The default value is initialized when creating the setting and unchanged
after that
 2020-09-16 13:53:08 +02:00
regnat 3c525d1590 Complete the toJSON instance for Setting<T> 
Don't let it just contain the value, but also the other fields of the
setting (description, aliases, etc..)
 2020-09-16 13:53:08 +02:00
regnat 3b57181f8e Separate the instantiation and initialisation of the stores 
Add a new `init()` method to the `Store` class that is supposed to
handle all the effectful initialisation needed to set-up the store.
The constructor should remain side-effect free and just initialize the
c++ data structure.

The goal behind that is that we can create “dummy” instances of each
store to query static properties about it (the parameters it accepts for
example)
 2020-09-16 13:53:08 +02:00
regnat fa32560169 Fix the registration of stores 2020-09-16 13:53:08 +02:00
regnat 7d5bdf8b56 Make the store plugins more introspectable 
Directly register the store classes rather than a function to build an
instance of them.
This gives the possibility to introspect static members of the class or
choose different ways of instantiating them.
 2020-09-16 13:53:08 +02:00
John Ericson 3a5cdd737c Rename Derivation::pathOpt to Derivation::path 
We no longer need the `*Opt` to disambiguate.
 2020-09-15 15:21:39 +00:00
John Ericson 6387550d58 Get rid of confusing std::optional<bool> for validity 2020-09-15 15:19:45 +00:00
John Ericson c4bf219b55 Don't link deriver until after any delayed exception is thrown 
Otherwise, we will associate fixed-output derivations with outputs that
they did indeed produce, but which had the wrong hash. That's no good.
 2020-09-15 14:28:06 +00:00
John Ericson 3ba552b245 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-15 14:17:06 +00:00
John Ericson c08c9f08c7 Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-15 14:08:35 +00:00
Eelco Dolstra 2a80170920
Merge pull request #4014 from tweag/fix-queryDrvOutputMaps-for-old-daemons 
gracefully handle old daemon versions
 2020-09-15 11:25:19 +02:00
regnat 057c6203b5 gracefully handle old daemon versions 
Add a fallback path in `queryPartialDerivationOutputMap` for daemons
that don't support it.

Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this
is needed to handle the fallback path.
 2020-09-15 09:58:01 +02:00
regnat 250f8a4bba Escape ${ in strings when printing Nix expressions 
Otherwise the result of the printing can't be parsed back correctly by
Nix (because the unescaped `${` will be parsed as the begining of an
anti-quotation).

Fix #3989
 2020-09-14 17:19:25 +02:00
Eelco Dolstra a59e77d9e5 nix-daemon: Lower verbosity of restricted setting warning 
Fixes #3992.
 2020-09-14 13:48:51 +02:00
Eelco Dolstra 35ba092019
Merge pull request #4005 from Infinisil/fix-autoArgs 
Fix auto argument passing for more auto arguments than formals
 2020-09-14 13:31:23 +02:00
aszlig 525b38eee8
Fix unspecified behaviour in readStorePathCAMap 
When deploying a Hydra instance with current Nix master, most builds
would not run because of errors like this:

  queue monitor: error: --- Error --- hydra-queue-runner
  error: --- UsageError --- nix-daemon
  not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv

The last error message is from parseContentAddress, which expects a
colon-separated string, however what we got here is a store path.

Looking at the worker protocol, the following message sent to the Nix
daemon caused the error above:

  0x1E -> wopQuerySubstitutablePathInfos
  0x01 -> Number of paths
  0x16 -> Length of string
  "/nix/store/...-somedrv"
  0x00 -> Length of string
  ""

Looking at writeStorePathCAMap, the store path is indeed the first field
that's transmitted. However, readStorePathCAMap expects it to be the
*second* field *on my machine*, since expression evaluation order is a
classic form of unspecified behaviour[1] in C++.

This has been introduced in https://github.com/NixOS/nix/pull/3689,
specifically in commit 66a62b3189.

[1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions

Signed-off-by: aszlig <aszlig@nix.build>
 2020-09-13 02:40:51 +02:00
Silvan Mosberger 8dbd57a6a5
Fix auto argument passing for more auto arguments than formals 
The change in 626200713b didn't account
for when the number of auto arguments is bigger than the number of
formal arguments. This causes the following:

  $ nix-instantiate --eval -E '{ ... }@args: args.foo' --argstr foo foo
  nix-instantiate: src/libexpr/attr-set.hh:55: void nix::Bindings::push_back(const nix::Attr&): Assertion `size_ < capacity_' failed.
  Aborted (core dumped)
 2020-09-11 20:41:51 +02:00
Gabriel Gonzalez ee5906243a Add nix-shell support for preserving PS1 
Fixes https://github.com/NixOS/nix/issues/1268

`nix-shell` will now preserve `PS1` if the `NIX_SHELL_PRESERVE_PROMPT`
environment variable is set.
 2020-09-04 20:05:43 -07:00
John Ericson 98dfd7531d Fix querying outputs for CA derivations some more 
If we resolve using the known path of a derivation whose output we
didn't have, we previously blew up. Now we just fail gracefully,
returning the map of all outputs unknown.
 2020-09-04 18:33:58 +00:00
John Ericson 075d399e3f Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-04 16:04:35 +00:00
John Ericson 5aed6f9b25 Document mkOutputString 2020-09-04 15:58:42 +00:00
John Ericson e9fad3006b Fix some of the issues raised by @edolstra 
- More and better comments

 - The easier renames
 2020-09-04 15:18:25 +00:00
John Ericson e86dd59dcc
Apply suggestions from code review 
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-09-04 10:48:50 -04:00
John Ericson c9f1ed912c Don't chmod symlink before moving outputs around 
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
 2020-09-04 14:41:53 +00:00
John Ericson 25f7ff16fa Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-09-04 02:40:36 +00:00
John Ericson aad4abcc9c Fix floating CA tests 
We will sometimes try to query the outputs of derivations we can't
resolve. That's fine; it just means we don't know what those outputs are
yet.
 2020-09-04 01:17:38 +00:00
John Ericson 975a47f7fe Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-03 22:09:04 +00:00
John Ericson e7d93e7ece Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-03 15:43:17 +00:00
Eelco Dolstra 8a945d6ddb
Remove gc-check-reachability 2020-09-03 11:30:15 +02:00
Eelco Dolstra 82b77a7726
addPermRoot(): Remove indirect flag 2020-09-03 11:29:34 +02:00
Eelco Dolstra 00d25e8457
Remove the --indirect flag 
All GC roots are now indirect.
 2020-09-03 11:22:00 +02:00
Eelco Dolstra b07167be5a
createGeneration(): Always create an indirect root 
This means profiles outside of /nix/var/nix/profiles don't get
garbage-collected. It also means we don't need to scan
/nix/var/nix/profiles for GC roots anymore, except for compatibility
with previously existing generations.
 2020-09-03 11:13:53 +02:00
Eelco Dolstra b74f5cdd23
createGeneration(): Take a StorePath 2020-09-03 11:06:56 +02:00
Eelco Dolstra 94a043ff3b
EvalCache: Fix caching of strings 
This was broken in 50f13b06fb. Once
again it turns out that putting a bool in a std::variant is a bad
idea, since pointers get silently cast to them...
 2020-09-02 14:16:44 +02:00
Domen Kožar dd4b56c87f
Allow HTTP binary cache to request absolute uris 2020-09-01 21:35:48 +02:00
John Ericson ef278d00f9 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-01 18:01:48 +00:00
Eelco Dolstra 6d7f7efb89
github: Use access token when calling .../commits API 2020-09-01 15:29:04 +02:00