qyriad/lix
1
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
11862 commits 57 branches 0 tags 73 MiB
Commit graph

7458 commits

Author SHA1 Message Date
Robert Hensing ac5081d280 nix-build: Fix #4197 output order regression 2020-11-13 17:49:27 +01:00
Christian Höppner c4c3c15c19 Fix default nix-path 
The default nix-path values for nixpkgs and root channels were
incorrect.
 2020-11-12 15:46:08 +00:00
Ben Burdette b327de9c2d change message 2020-11-11 11:09:59 -07:00
Ben Burdette 8895fa70a4 pare down the error message 2020-11-11 11:05:21 -07:00
Ben Burdette 3edfe6090e missing argument error 2020-11-11 09:29:32 -07:00
Matthew Kenigsberg 8abb80a478 Print built derivations as json for build 
Add --json option to nix build to allow machine readable output on
stdout with all built derivations

Fixes #1930
 2020-11-11 10:27:02 -06:00
Ben Burdette 7d9037035e usage example location 2020-11-11 09:21:26 -07:00
Eelco Dolstra 4badb6943f Fix use of dirty Git/Mercurial inputs with chroot stores 
Fixes:

  $ nix build --store /tmp/nix /home/eelco/Dev/patchelf#hydraJobs.build.x86_64-linux
  warning: Git tree '/home/eelco/Dev/patchelf' is dirty
  error: --- RestrictedPathError ------------------------------------------------------------------------------------------- nix
  access to path '/tmp/nix/nix/store/xmkvfmffk7xfnazykb5kx999aika8an4-source/flake.nix' is forbidden in restricted mode
  (use '--show-trace' to show detailed location information)
 2020-11-10 23:22:45 +01:00
Eelco Dolstra cdc840d60b
Merge pull request #4242 from wizeman/fix-stack-overflow 
Fix stack overflow introduced in #4206
 2020-11-10 11:51:00 +01:00
Eelco Dolstra 3f680c1dcc
Merge pull request #4233 from Kha/master 
nix develop: Preserve stdin with `-c`
 2020-11-10 10:51:48 +01:00
Ricardo M. Correia 108a2dab7e Fix stack overflow introduced in #4206 2020-11-10 04:25:24 +01:00
Ben Burdette d8ef423a18 error message formatting 2020-11-09 19:16:50 -07:00
Ben Burdette 9f2b25ce55 remove unused ftn; reformat line breaks 2020-11-09 17:17:47 -07:00
Ben Burdette 6c2933a8d7 add position 2020-11-09 17:04:52 -07:00
Ben Burdette 107c91f5fe auto-call error 2020-11-09 16:48:35 -07:00
Sebastian Ullrich fb7735e4cf nix develop: Preserve stdin with -c 2020-11-09 22:43:14 +01:00
Eelco Dolstra b87f84cf55
Fix appending to Setting<StringSet> 
Fixes: warning: unknown setting 'extra-sandbox-paths'
 2020-11-09 15:04:34 +01:00
Maximilian Bosch 3a63fc6cd5
Allow substituting paths when building remotely using ssh-ng:// 
Until now, it was not possible to substitute missing paths from e.g.
`https://cache.nixos.org` on a remote server when building on it using
the new `ssh-ng` protocol.

This is because every store implementation except legacy `ssh://`
ignores the substitution flag passed to `Store::queryValidPaths` while
the `legacy-ssh-store` substitutes the remote store using
`cmdQueryValidPaths` when the remote store is opened with `nix-store
--serve`.

This patch slightly modifies the daemon protocol to allow passing an
integer value suggesting whether to substitute missing paths during
`wopQueryValidPaths`. To implement this on the daemon-side, the
substitution logic from `nix-store --serve` has been moved into a
protected method named `Store::substitutePaths` which gets currently
called from `LocalStore::queryValidPaths` and `Store::queryValidPaths`
if `maybeSubstitute` is `true`.

Fixes #2770
 2020-11-05 20:12:37 +01:00
Eelco Dolstra 387f824cab
Merge pull request #4206 from hercules-ci/fix-coroutine-gc 
Fix memory corruption caused by GC-invisible coroutine stacks
 2020-11-05 10:18:31 +01:00
Eelco Dolstra 5e6eabe155 Fix error message 'assertion failed at' 2020-11-03 14:45:24 +01:00
Eelco Dolstra e8c379555f LocalStore: Get rid of recursive_mutex 2020-11-03 14:45:24 +01:00
Eelco Dolstra 797a52e31d Add FIXME 2020-11-03 14:45:24 +01:00
Eelco Dolstra 550e11f077 nix repl: Fix handling of multi-line expressions 2020-11-02 19:07:37 +01:00
Eelco Dolstra 7cf874c17d Don't use readDerivation() in addValidPath() 
readDerivation() requires a valid path.

Fixes #4210.
 2020-11-02 18:46:44 +01:00
Eelco Dolstra ab2ef851b6
Merge pull request #4207 from hercules-ci/fix-RemoteStore-filterSource-deadlock 
Fix RemoteStore pool deadlock in filterSource etc
 2020-11-02 14:46:10 +01:00
Eelco Dolstra db5424bf09 Don't send eval-related settings to the daemon 2020-11-02 13:57:58 +01:00
Robert Hensing e8a45d07bc Restore RestrictedStore.addToStoreFromDump implementation 
It was accidentally removed in commit ca30abb3fb
 2020-10-31 23:56:03 +01:00
Robert Hensing b43c13a916 BoehmGCStackAllocator: increase stack size to 8MB 
The default stack size was not based on the normal stack size and
was too small.
 2020-10-30 23:18:26 +01:00
Robert Hensing 2192cac634 Fix RemoteStore pool deadlock in filterSource etc 2020-10-30 21:47:34 +01:00
Robert Hensing c4d903ddb0 Fix memory corruption caused by GC-invisible coroutine stacks 
Crucially this introduces BoehmGCStackAllocator, but it also
adds a bunch of wiring to avoid making libutil depend on bdw-gc.

Part of the solutions for #4178, #4200
 2020-10-30 21:21:59 +01:00
Eelco Dolstra dc5696b84f Fix test 2020-10-30 12:00:53 +01:00
Eelco Dolstra 7f56cf67ba Fix assertion failure in tab completion for --option 2020-10-29 18:26:35 +01:00
Eelco Dolstra ff4dea63c9 Generalize extra-* settings 
This removes the extra-substituters and extra-sandbox-paths settings
and instead makes every array setting extensible by setting
"extra-<name> = <value>" in the configuration file or passing
"--<name> <value>" on the command line.
 2020-10-29 18:17:39 +01:00
Eelco Dolstra bb8e837e4c
Merge pull request #4199 from stefanjaax/replaceWantedBySpecified 
Alter "wanted:" to "specified:" in hash mismatch output
 2020-10-29 07:38:38 +01:00
Eelco Dolstra 662e67f8de
Merge pull request #4198 from mkenigs/capitalize-JSON 
Capitalize JSON for consistency
 2020-10-29 07:34:34 +01:00
stev 869c0321ff Alter "wanted:" to "specified:" in hash mismatch output 
This makes it even clearer which of the two hashes was specified in the
nix files. Some may think that "wanted" and "got" is obvious, but:
"got" could mean "got in nix file" and "wanted" could mean "want to see in nix file".
 2020-10-29 00:33:14 +01:00
Matthew Kenigsberg 6a4bf535d8 Capitalize JSON for consistency 2020-10-28 17:54:28 -05:00
Eelco Dolstra a5019f0508 Consistency 2020-10-28 20:45:57 +01:00
Eelco Dolstra 02a1facbdc
Merge pull request #4056 from tweag/non-ca-depending-on-ca 
Allow non-CA derivations to depend on CA ones
 2020-10-27 17:38:29 +01:00
regnat bc081bcd81 Inline unkownHashes 
See https://github.com/NixOS/nix/pull/4056#discussion_r493661632
 2020-10-27 07:29:25 +01:00
regnat c092fa4702 Allow non-CA derivations to depend on CA derivations 2020-10-27 07:29:23 +01:00
Eelco Dolstra 343239fc8a Allow nix.conf options to be set in flake.nix 
This makes it possible to have per-project configuration in flake.nix,
e.g. binary caches and other stuff:

  nixConfig.bash-prompt-suffix = "ngi# ";
  nixConfig.substituters = [ "https://cache.ngi0.nixos.org/" ];
 2020-10-26 20:45:39 +01:00
Eelco Dolstra 731edf0d9b isTrivial(): Support trivial lists 2020-10-26 20:37:11 +01:00
Eelco Dolstra 14aecbb288 BaseSetting<StringMap>::set(): Don't append to previous value 2020-10-26 20:36:46 +01:00
Eelco Dolstra b875b8f45c Remove edition field 2020-10-26 17:59:36 +01:00
Eelco Dolstra 9d5e9ef0da Move Explicit 2020-10-26 17:01:20 +01:00
Eelco Dolstra dc7d1322ef Make the prompt used in development shells configurable 2020-10-26 14:24:25 +01:00
Eelco Dolstra 750ce500c2 Fix clang build 2020-10-22 13:40:30 +02:00
Eelco Dolstra f9438fb64a nix develop: Add --redirect flag to redirect dependencies 
This is primarily useful if you're hacking simultaneously on a package
and one of its dependencies. E.g. if you're hacking on Hydra and Nix,
you would start a dev shell for Nix, and then a dev shell for Hydra as
follows:

  $ nix develop \
    --redirect .#hydraJobs.build.x86_64-linux.nix ~/Dev/nix/outputs/out \
    --redirect .#hydraJobs.build.x86_64-linux.nix.dev ~/Dev/nix/outputs/dev

(This assumes hydraJobs.build.x86_64-linux has a passthru.nix
attribute. You can also use a store path.)

This causes all references in the environment to those store paths to
be rewritten to ~/Dev/nix/outputs/{out,dev}. Note: unfortunately, you
may need to set LD_LIBRARY_PATH=~/Dev/nix/outputs/out/lib because
Nixpkgs' ld-wrapper only adds -rpath entries for -L flags that point
to the Nix store.
 2020-10-22 13:40:30 +02:00
Eelco Dolstra e556a1beb7 nix develop: Handle 'declare -ax' in bash output 
Fixes 'nix develop nixpkgs#qpdfview'.
 2020-10-21 17:54:21 +02:00
Christian Kampka 461cf2b856
Add NIX_CONFIG env var for applying nix.conf overrides 2020-10-21 13:41:26 +02:00
Matthew Kenigsberg f6aaac2b59 Make bash non-interactive for nix develop --phase 
Fix #3975: Currently if Ctrl-C is pressed during a phase, the interactive subshell
is not exited. Removing --rcfile when --phase is present makes bash
non-interactive
 2020-10-20 12:00:30 -05:00
Eelco Dolstra 9635fb77bd
Merge pull request #4080 from kquick/kwq/flake-int-doc 
Add some internal documentation for flake support objects.
 2020-10-19 11:29:12 +02:00
Eelco Dolstra fda835b231
Merge pull request #4143 from obsidiansystems/typed-goal-maps 
Properly type the derivation and substitution goal maps
 2020-10-18 18:12:21 +02:00
Robert Hensing bd9eb5c743 DerivationGoal: only retry if output closure incomplete is only problem 2020-10-18 14:26:37 +02:00
Robert Hensing 94f1e4a441 Typo 2020-10-18 14:26:37 +02:00
John Ericson 7ed46c1574 Explain that upcast_goal is still a static cast 2020-10-17 21:50:12 +00:00
John Ericson 57d0432b39 Just use auto in two places. 2020-10-17 21:47:52 +00:00
John Ericson e6f8ae56d8 tab -> space 2020-10-17 21:45:31 +00:00
John Ericson 619d262c97 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-17 21:44:27 +00:00
John Ericson 64be1c15c2 Add missing include for MAX_PATH 
And remove one that we didn't actually need to add
 2020-10-15 19:05:17 +00:00
John Ericson fccef6a7fa Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-10-15 18:55:03 +00:00
John Ericson 0fefc2a439 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-14 20:49:01 +00:00
Eelco Dolstra 11882d7c7c Create /etc/passwd *after* figuring out the sandbox uid/gid 
Fixes build failures like

  # nix log /nix/store/gjaa0psfcmqvw7ivggsncx9w364p3s8s-sshd.conf-validated.drv
  No user exists for uid 30012
 2020-10-14 12:20:58 +02:00
John Ericson 55592b253f Add some more docs 2020-10-13 18:04:24 +00:00
John Ericson 13804f126e Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-13 18:02:32 +00:00
Eelco Dolstra 2653801939 Merge branch 'split_build_cc' of https://github.com/obsidiansystems/nix 2020-10-13 15:36:55 +02:00
John Ericson 1b8ebe92dc Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps 2020-10-12 20:47:22 +00:00
John Ericson a73959e6be Merge remote-tracking branch 'upstream/master' into split_build_cc 2020-10-12 17:20:46 +00:00
John Ericson d334fd4882 Merge branches 'split_build_hh-0', 'split_build_hh-1', 'split_build_hh-2', 'split_build_hh-3' and 'split_build_hh-4' into split_build_cc 2020-10-12 17:20:12 +00:00
John Ericson 542972f029 Trim worker.hh 2020-10-12 17:16:48 +00:00
John Ericson 0e2306204a Rename to hand-hold git (worker.hh) 2020-10-12 17:16:48 +00:00
John Ericson 3ffa3546bd Trim substitution-goal.hh 2020-10-12 17:16:25 +00:00
John Ericson e77a2344d5 Rename to hand-hold git (substitution-goal.hh) 2020-10-12 17:16:25 +00:00
John Ericson d585b4c54f Trim hook-instance.hh 2020-10-12 17:16:13 +00:00
John Ericson 10b749a156 Rename to hand-hold git (hook-instance.hh) 2020-10-12 17:16:13 +00:00
John Ericson 8067d32f2a Trim goal.hh 2020-10-12 17:16:00 +00:00
John Ericson 0d0e345cdc Rename to hand-hold git (goal.hh) 2020-10-12 17:16:00 +00:00
John Ericson 2ce726947a Trim derivation-goal.hh 2020-10-12 17:15:32 +00:00
John Ericson 4eb8c69853 Rename to hand-hold git (derivation-goal.hh) 2020-10-12 17:15:32 +00:00
John Ericson f7099965bf Change .cc files to use split build headers 2020-10-12 17:08:52 +00:00
John Ericson 5a97621d6d Prepare for build/*.hh headers 2020-10-12 17:07:51 +00:00
volth eee18f88dd
Handle amount of disk space saved by hard linking being negative 
Fixes bogus messages like "currently hard linking saves 17592186044416.00 MiB".
 2020-10-12 16:06:38 +00:00
John Ericson 15fdb7cc6b Split out uds-remote-store.{cc.hh} 2020-10-11 17:37:05 +00:00
John Ericson 38e3897162 Copy {uds-,}remote-store.{cc,hh} 
This prepares for the splitting that happens in the next commit.
 2020-10-11 17:18:26 +00:00
John Ericson 5c74a6147b Properly type the derivation and substitution goal maps 
As a bonus, Worker::removeGoal is less inefficient.
 2020-10-11 17:07:14 +00:00
Eelco Dolstra b0fbf3a653
Merge pull request #4135 from obsidiansystems/split-local-fs-store-header 
Split out `local-fs-store.hh`
 2020-10-11 18:49:40 +02:00
John Ericson 8cc510fb79 Merge branches 'build-1', 'build-2', 'build-3', 'build-4', 'build-5', 'build-6', 'build-7' and 'build-8' into split_build_cc 2020-10-11 16:44:34 +00:00
John Ericson bcb67e1ed8 Trim lock.cc 2020-10-11 16:44:19 +00:00
John Ericson e0be04129b Rename to hand-hold git (lock.cc) 2020-10-11 16:44:14 +00:00
John Ericson dbc588651c Trim lock.hh 2020-10-11 16:44:07 +00:00
John Ericson d0004bfcab Rename to hand-hold git (lock.hh) 2020-10-11 16:43:59 +00:00
John Ericson eed53ed87a Trim build/worker.cc 2020-10-11 16:43:51 +00:00
John Ericson 904e315dae Rename to hand-hold git (build/worker.cc) 2020-10-11 16:43:42 +00:00
John Ericson d24ffe0eb1 Trim build/substitution-goal.cc 2020-10-11 16:43:24 +00:00
John Ericson 3633b3572b Rename to hand-hold git (build/substitution-goal.cc) 2020-10-11 16:43:18 +00:00
John Ericson 4bdff7d1b0 Trim build/local-store-build.cc 2020-10-11 16:43:12 +00:00
John Ericson dc5225cde5 Rename to hand-hold git (build/local-store-build.cc) 2020-10-11 16:43:05 +00:00
John Ericson 159054f730 Trim build/hook-instance.cc 2020-10-11 16:42:35 +00:00
John Ericson f0b8987299 Rename to hand-hold git (build/hook-instance.cc) 2020-10-11 16:42:08 +00:00
John Ericson 819fe848ac Trim build/goal.cc 2020-10-11 16:41:58 +00:00
John Ericson 184bfc301e Rename to hand-hold git (build/goal.cc) 2020-10-11 16:41:18 +00:00
John Ericson 3bab1c5bb0 Trim build/derivation-goal.cc 2020-10-11 16:41:11 +00:00
John Ericson 9629290eda Rename to hand-hold git (build/derivation-goal.cc) 2020-10-11 16:40:52 +00:00
John Ericson a4f0fecb03 Trim build.hh 2020-10-11 16:40:34 +00:00
John Ericson fc72cb0760 Rename to hand-hold git (build.hh) 2020-10-11 16:40:14 +00:00
John Ericson 428536fd75 Prepare for build/* files 2020-10-11 16:39:08 +00:00
John Ericson aef44cbaa9 Split out commonChildInit 2020-10-11 16:38:46 +00:00
John Ericson 6cc1541782 Split out local-fs-store.hh 
This matches the already-existing `local-fs-store.cc`.
 2020-10-09 20:18:08 +00:00
Eelco Dolstra 725488b892 nix develop: Unset $HOSTNAME 
This is set to "localhost" by stdenv which is probably not what you
want.
 2020-10-09 22:03:18 +02:00
Eelco Dolstra 44349064f7 nix develop: Source ~/.bashrc 
Fixes #4104.
 2020-10-09 22:02:00 +02:00
Eelco Dolstra 59bd6e87a4 Completions::add(): Guard against newlines 2020-10-09 21:55:59 +02:00
Eelco Dolstra ea4b2b985f
Merge pull request #4128 from tweag/extended-completions 
Add a zsh completion script
 2020-10-09 21:53:12 +02:00
John Ericson 39de73550d Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-10-09 18:26:47 +00:00
Eelco Dolstra e845d19ae3 Remove Lazy 
This fixes a crash during startup when compiling Nix as a single
compilation unit.
 2020-10-09 17:54:59 +02:00
Eelco Dolstra 87157b2bd3 writeFile(): Add error context to writeFull() failure 
Issue #4092.
 2020-10-09 16:02:53 +02:00
Eelco Dolstra 636ec17139
Remove stray DerivationOutputsAndPaths type 2020-10-09 15:41:24 +02:00
regnat 04e5d0e704 Add a description in the completion outputs 
Make nix output completions in the form `completion\tdescription`.
This can't be used by bash (afaik), but other shells like zsh or fish
can display it along the completion choices
 2020-10-09 09:39:51 +02:00
Eelco Dolstra 97ffc1e013 Dynamically disable user namespaces if CLONE_NEWUSER fails 
This makes builds work inside nixos-enter.

Fixes #3145.
 2020-10-07 22:46:01 +02:00
Eelco Dolstra 6aa64627c8 Support user namespaces being disabled 
If max_user_namespaces is set to 0, then don't run the build in a user
namespace.

Fixes #4092.
 2020-10-07 22:02:36 +02:00
Eelco Dolstra f66bbd8c7b Doh 2020-10-07 21:25:06 +02:00
Eelco Dolstra e705c24294 Tweak error messages 2020-10-07 17:28:43 +02:00
Eelco Dolstra c43e882f54 Serialize exceptions from the daemon to the client 2020-10-07 17:13:54 +02:00
Eelco Dolstra be149acfda Serialize exceptions from the sandbox process to the parent 
Fixes #4118.
 2020-10-07 16:34:03 +02:00
Eelco Dolstra 27ca87c46a Formatting 2020-10-07 16:33:19 +02:00
John Ericson 57d960dcd1 Remove generic std::optional<T> suppport from worker proto 
See comment for rational; I think it's good to leave a comment lest
anyone is tempted to add such a sum-type instance again.

Fixes #4113
 2020-10-07 12:50:37 +00:00
Maximilian Bosch 59f2dd8e8d
libfetchers/github: allow slashes in refs 
Refs #4061
 2020-10-06 20:08:51 +02:00
Eelco Dolstra d761485010 Prevent a deadlock when user namespace setup fails 
Observed on Centos 7 when user namespaces are disabled:
DerivationGoal::startBuilder() throws an exception, ~DerivationGoal()
waits for the child process to exit, but the child process hangs
forever in drainFD(userNamespaceSync.readSide.get()) in
DerivationGoal::runChild(). Not sure why the SIGKILL doesn't get
through.

Issue #4092.
 2020-10-06 18:57:23 +02:00
Eelco Dolstra ad143c5b3b Shut up some clang warnings 2020-10-06 14:52:30 +02:00
Eelco Dolstra 85c8be6286 Remove static variable name clashes 
This was useful for an experiment with building Nix as a single
compilation unit. It's not very useful otherwise but also doesn't
hurt...
 2020-10-06 13:49:20 +02:00
Eelco Dolstra 0419cd2695 Remove unneeded -lboost_* flags 2020-10-06 13:34:58 +02:00
Eelco Dolstra 636455c471 Remove 'using namespace fetchers' 2020-10-06 11:16:32 +02:00
Eelco Dolstra 6691256e79 Factor out common showBytes() 2020-10-06 10:40:49 +02:00
Eelco Dolstra d0bb544128 Add missing #pragma once 2020-10-06 10:40:07 +02:00
Eelco Dolstra 88a667e49e
Fix s3:// store 
Fixes https://github.com/NixOS/nixos-org-configurations/issues/123.
 2020-10-05 17:53:30 +02:00
Eelco Dolstra 51c299213b
Merge pull request #3935 from obsidiansystems/binary-cache-addToStoreFromDump 
Get rid of Hash::dummy from BinaryCacheStore
 2020-10-05 14:41:28 +02:00
Eelco Dolstra f3aba88737
Merge pull request #3895 from obsidiansystems/templated-daemon-protocol 
More templated STL support for the daemon protocol
 2020-10-05 14:40:27 +02:00
Fabian Möller d5d196b0a1
Fix profile update in nix command 2020-10-02 12:10:31 +02:00
DavHau f3280004e2 add more examples to --help of nix run 2020-10-01 11:34:13 +07:00
Eelco Dolstra 20a1e20d91 Style 2020-09-30 13:35:56 +02:00
Eelco Dolstra 274357eb6a Simplify 2020-09-30 12:09:18 +02:00
Eelco Dolstra 002ce8449d Merge branch 'access-tokens' of https://github.com/kquick/nix 2020-09-30 11:35:15 +02:00
John Ericson 69afaeace3 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-30 00:42:28 +00:00
John Ericson b759701652 nix::worker_proto -> worker_proto 2020-09-30 00:41:18 +00:00
John Ericson 45a0ed82f0 Revert "Use template structs instead of phantoms" 
This reverts commit 9ab07e99f5.
 2020-09-30 00:39:06 +00:00
Kevin Quick 5e7838512e
Remove github-access-token in favor of access-token. 2020-09-29 16:26:34 -07:00
Eelco Dolstra 76f000712e Merge branch 'kwq/path-dir-query' of https://github.com/kquick/nix 2020-09-29 23:42:43 +02:00
Eelco Dolstra 64e9b3c83b nix registry list: Show 'dir' attribute 
Issue #4050.
 2020-09-29 23:33:16 +02:00
Kevin Quick 66c3959e8c
Merge branch 'master' into access-tokens 2020-09-29 08:32:06 -07:00
Eelco Dolstra 5999978a05 Make Headers an optional argument 2020-09-29 13:05:19 +02:00
Eelco Dolstra de86abbf3f Cleanup 2020-09-29 12:55:06 +02:00
Eelco Dolstra cebd2fc35d Merge branch 'github-api-token' of https://github.com/imalsogreg/nix 2020-09-29 12:17:00 +02:00
John Ericson 00135e13f4 Clarify comment a bit 2020-09-28 18:19:10 +00:00
Kevin Quick 887be7b6f2
Switch comment format from '// ...' to '/* ... */' for consistency. 2020-09-28 09:37:26 -07:00
Kevin Quick 128c98ab09
Clarification in the description of the FlakeInput. 2020-09-28 09:34:23 -07:00
Kevin Quick 5ae164b7cf
Update description of FlakeRef, incorporating suggestion. 2020-09-28 09:23:05 -07:00
Kevin Quick bcb3da3b6b
Fix spelling error. 2020-09-28 08:58:14 -07:00
John Ericson 80e335bb58 Use drvPath2 and give it a better name 2020-09-28 15:43:56 +00:00
John Ericson 10202bbf29 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-28 15:39:11 +00:00
John Ericson 6c31297d80
Update src/libstore/binary-cache-store.cc 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-09-28 11:32:58 -04:00
Eelco Dolstra 649c465873
Merge pull request #4064 from serokell/balsoft/fix-max-jobs 
Fix max-jobs option
 2020-09-28 10:42:02 +02:00
Eelco Dolstra 5df687c316
Merge pull request #4077 from Ma27/left-whitespace-log 
libmain/progress-bar: don't trim whitespace on the left
 2020-09-28 10:40:00 +02:00
Kevin Quick 5885b0cfd8
Miscellaneous spelling fixes in comments. (#4071) 2020-09-27 20:04:06 +00:00
Kevin Quick bd5328814f
Add some internal documentation for flake support objects. 2020-09-26 14:32:58 -07:00
Maximilian Bosch a76fb07314
libmain/progress-bar: don't trim whitespace on the left 
When running `nix build -L` it can be fairly hard to read the output if
the build program intentionally renders whitespace on the left. A
typical example is `g++` displaying compilation errors.

With this patch, the whitespace on the left is retained to make the log
more readable:

```
foo> no configure script, doing nothing
foo> building
foo> foobar.cc: In function 'int main()':
foo> foobar.cc:5:5: error: 'wrong_func' was not declared in this scope
foo>     5 |     wrong_func(1);
foo>       |     ^~~~~~~~~~
error: --- Error ------------------------------------------------------------------------------------- nix
error: --- Error --- nix-daemon
builder for '/nix/store/i1q76cw6cyh91raaqg5p5isd1l2x6rx2-foo-1.0.drv' failed with exit code 1
```
 2020-09-26 17:38:11 +02:00
John Ericson 25fffdda86
Remove redundant nar hash and size setting 
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2020-09-26 10:17:30 -04:00
John Ericson 1832436526 Fix up BinaryCacheStore::addToStore taking a path 2020-09-26 04:56:29 +00:00
John Ericson 5db83dd771 BinaryCacheStore::addTextToStore include CA field 2020-09-26 03:21:36 +00:00
John Ericson c40c832f19 Merge remote-tracking branch 'obsidian/master' into binary-cache-addToStoreFromDump 2020-09-26 00:21:45 +00:00
Kevin Quick cb186f1e75
Use "?dir=..." portion of "registry add" local path specification. 
The registry targets generally follow a URL formatting schema with
support for a query parameter of "?dir=subpath" to specify a sub-path
location below the URL root.

Alternatively, an absolute path can be specified.  This specification
mode accepts the query parameter but ignores/drops it.  It would
probably be better to either (a) disallow the query parameter for the
path form, or (b) recognize the query parameter and add to the path.

This patch implements (b) for consistency, and to make it easier for
tooling that might switch between a remote git reference and a local
path reference.

See also issue #4050.
 2020-09-25 09:36:18 -07:00
Gregory Hale faa5607f54 Merge remote-tracking branch 'origin/master' into github-api-token 2020-09-25 12:10:58 -04:00
John Ericson cfe791a638 stdout_ -> cout 
Better to get creative than just sprinkle arbitrary underscores.
 2020-09-25 11:30:04 -04:00
Eelco Dolstra 7d81582488
Merge pull request #3953 from obsidiansystems/basic-derivation-goal-outputs 
Deduplicate basic derivation goals too
 2020-09-25 17:21:19 +02:00
Kevin Quick 5a35cc29bf
Re-add support for github-access-token, but mark as deprecated. 2020-09-25 08:09:56 -07:00
Kevin Quick ef2a14be19
Fix reference to older name for access-tokens config value. 2020-09-25 08:08:27 -07:00
Eelco Dolstra cbb9862cd9
Merge pull request #3626 from W95Psp/master 
Make `functionArgs` primitive accept primops (fix #3624)
 2020-09-25 15:14:18 +02:00
Eelco Dolstra 7b2ae472ff
expectArg(): Respect the 'optional' flag 2020-09-25 10:27:40 +02:00
Kevin Quick 8fba2a8b54
Update to use access-tokens configuration for github/gitlab access. 
This change provides support for using access tokens with other
instances of GitHub and GitLab beyond just github.com and
gitlab.com (especially company-specific or foundation-specific
instances).

This change also provides the ability to specify the type of access
token being used, where different types may have different handling,
based on the forge type.
 2020-09-24 22:49:44 -07:00
Kevin Quick c2f48cfcee
Complete conversion of "url" to "host" with associated variable renaming. 
Completes the change begun in commit 56f1e0d to consistently use the
"host" attribute for "github" and "gitlab" inputs instead of a "url"
attribute.
 2020-09-24 22:46:03 -07:00
Kevin Quick a439e9488d
Support StringMap configuration settings. 
Allows Configuration values that are space-separated key=value pairs.
 2020-09-24 22:42:59 -07:00
Paul Opiyo 4d863a9fcb Remove redundant value checks 
std::optional had redundant checks for whether it had a value.
An object is emplaced either way so it can be dereferenced
without repeating a value check
 2020-09-24 18:32:03 -05:00
Alexander Bantyev ed218e1d6c
Fix max-jobs option 
After 0ed946aa61, max-jobs setting (-j/--max-jobs)
stopped working.

The reason was that nrLocalBuilds (which compared to maxBuildJobs to figure
out whether the limit is reached or not) is not incremented yet when tryBuild
is started; So, the solution is to move the check to tryLocalBuild.

Closes https://github.com/nixos/nix/issues/3763
 2020-09-25 00:07:42 +03:00
Kevin Quick bd5f3dbe11
Fixes fall-through to report correct description of hash-file command. 2020-09-24 12:30:03 -07:00
Eelco Dolstra 8d9402f411
Merge pull request #4054 from edolstra/fix-4021 
registerOutputs(): Don't call canonicalisePathMetaData() twice
 2020-09-23 21:57:53 +02:00
Eelco Dolstra 4ce8a3ed45 Hopefully fix EPERM on macOS 2020-09-23 21:29:10 +02:00
Eelco Dolstra 9a24ece122 Fix exception 2020-09-23 20:21:08 +02:00
Eelco Dolstra 236d9ee7f7 lstat() cleanup 2020-09-23 19:17:28 +02:00
Eelco Dolstra 688bd4fb50 After rewriting a path, make it read-only 2020-09-23 19:10:16 +02:00
Dominique Martinet 2548347bba libutil/archive: add preallocate-contents option 
Make archive preallocation (fallocate) optional because some filesystems
like btrfs do not behave as expected with fallocate.

See #3550.
 2020-09-23 18:49:11 +02:00
Eelco Dolstra cec9473871 DerivationGoal::registerOutputs(): Don't canonicalize twice 
Fixes #4021.
 2020-09-23 18:21:22 +02:00
Eelco Dolstra d4f8163d10 canonicalisePathMetaData_(): Change assertion to error message 2020-09-23 18:21:22 +02:00
Eelco Dolstra e8f0b1e996 DerivationGoal::registerOutputs(): Fix bad format string 2020-09-23 18:21:22 +02:00
John Ericson 3f226f71c1 Return more info from BinaryCacheStore::addToStoreCommon 
We don't need it yet, but we could/should in the future, and it's a
cost-free change since we already have the reference. I like it.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2020-09-23 14:40:41 +00:00
John Ericson 412b3a54fb
Clarify FIXME in BinaryCacheStore::addToStoreCommon 
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
 2020-09-23 10:36:55 -04:00
Eelco Dolstra 8a2e10827f Remove unused Flake::vOutputs field 2020-09-23 14:08:52 +02:00
Domen Kožar dac8a1a5e1
Merge pull request #4052 from ujjwaljainnn/fix-typo 
fixed typo
 2020-09-23 09:58:30 +02:00
John Ericson 9fbc31a65b Get rid of Hash::dummy from BinaryCacheStore 2020-09-23 04:56:04 +00:00
ujjwal a2f5c921d4 fixed typo 2020-09-22 23:37:06 +05:30
John Ericson 993229cdaf Deduplicate basic derivation goals too 
See comments for security concerns.

Also optimize goal creation by not traversing map twice.
 2020-09-22 17:13:59 +00:00
John Ericson e9fc2031f0 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 14:18:31 +00:00
regnat 67cc356bb3 Merge remote-tracking branch 'origin/master' into non-ca-depending-on-ca 2020-09-22 16:11:25 +02:00
Eelco Dolstra 980edd1f3a RemoteStore::addCAToStore(): Don't hold connection while calling queryPathInfo() 
This leads to a deadlock if we're at the connection limit.
 2020-09-22 15:28:20 +02:00
Eelco Dolstra 5b107f2c5f
Merge pull request #4038 from maljub01/master 
Add a nix.conf option for allowing a symlinked store
 2020-09-22 13:19:22 +02:00
Eelco Dolstra 35a0ac1838 Style fixes 2020-09-22 11:40:19 +02:00
Eelco Dolstra 92ac8df0ec Merge branch 'add-ca-to-store' of https://github.com/hercules-ci/nix 2020-09-22 11:31:33 +02:00
Eelco Dolstra e7f1109f06
Merge pull request #4040 from OmnipotentEntity/master 
Fix compatibility with nlohmann-json 3.9.1
 2020-09-22 11:09:25 +02:00
Eelco Dolstra 7dd8baafe1
Merge pull request #4041 from cole-h/enum-stringify 
Serialize SandboxMode enum to string for JSON
 2020-09-22 11:08:56 +02:00
regnat c1e79f870c Silence a compiler warning in serialise.hh 
Explicitely cast to `uint64_t` in `readNum` to avoid a "comparison
between signed and unsigned" warning
 2020-09-22 10:39:29 +02:00
John Ericson 3786a801c3 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-22 04:15:55 +00:00
John Ericson b92d3b2edd Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 00:45:55 +00:00
Michael Reilly d860295e11 Bump nlohmann-json version to 3.9.1 2020-09-21 14:24:43 -04:00
Cole Helbling ba37299a03
Serialize SandboxMode enum to string for JSON 
Rather than showing an integer as the default, instead show the boolean
referenced in the description.

The nix.conf.5 manpage used to show "default: 0", which is unnecessarily
opaque and confusing (doesn't 0 mean false, even though the default is
true?); now it properly shows that the default is true.
 2020-09-21 10:36:45 -07:00
Eelco Dolstra ecc8672aa0 fmt.hh: Don't include boost/algorithm/string/replace.hpp 
This cuts compilation time by ~49s.

Issue #4045.
 2020-09-21 19:07:55 +02:00
Eelco Dolstra 557d2427ee Random header cleanup 2020-09-21 18:59:02 +02:00
Eelco Dolstra 0716adaa8b abstractsettingtojson.hh -> abstract-setting-to-json.hh 2020-09-21 18:49:43 +02:00
Eelco Dolstra 340ca382c4 Don't include nlohmann/json.hpp in globals.hh 
This reduces compilation time by 207s.

Issue #4045.
 2020-09-21 18:47:18 +02:00
Eelco Dolstra d51ba43047 Move Callback into its own header 
This gets rid of the inclusion of <future> in util.hh, cutting
compilation time by ~20s (CPU time).

Issue #4045.
 2020-09-21 18:42:21 +02:00
Marwan Aljubeh f80ffeb8c9 Update the variable name accordingly 2020-09-21 17:29:08 +01:00
Eelco Dolstra e8e1d420f3 Don't include <regex> in header files 
This reduces compilation time by ~15 seconds (CPU time).

Issue #4045.
 2020-09-21 18:22:45 +02:00
Marwan Aljubeh 4e1a04733d Use a better name for the config option 2020-09-21 16:32:22 +01:00
Eelco Dolstra cbe0bb29f4
Merge pull request #4035 from Ma27/url-attr 
libfetchers/github: allow `url` attribute
 2020-09-21 17:26:36 +02:00
Maximilian Bosch 56f1e0df05
libfetchers/github: rename url to host 2020-09-21 16:29:08 +02:00
Robert Hensing ca30abb3fb Document addCAToStore/addToStoreFromDump source drainage 
Also checked that all usages satisfy the requirement and
removed dead code.
 2020-09-21 07:55:47 +02:00
Robert Hensing fa08db5c4c wopAddToStore: return ValidPathInfo 
A ValidPathInfo is created anyway. By returning it we can save a
roundtrip and we have a nicer interface.
 2020-09-21 07:55:47 +02:00
Robert Hensing 7c68264085 wopAddToStore: add RepairFlag 2020-09-21 07:55:47 +02:00
Robert Hensing fbf509c113 parseContentAddressMethodPrefix: use string_view 
Co-authored-by: John Ericson <git@JohnEricson.me>
 2020-09-21 07:55:47 +02:00
Robert Hensing 8279178b07 Move FramedSink next to FramedSource 2020-09-21 07:55:47 +02:00
Robert Hensing ecc8088cb7 wopAddToStore: Throw to clarify unused refs 
Co-authored-by: John Ericson <git@JohnEricson.me>
 2020-09-21 07:55:47 +02:00
Robert Hensing c602ebfb34 Refactor wopAddToStore to make wopAddTextToStore obsolete 2020-09-21 07:55:45 +02:00
Robert Hensing e34fe47d0c Overhaul wopAddToStore 2020-09-21 07:54:05 +02:00
Marwan Aljubeh e40772cd35 Lint issue: replacing tabs with spaces 2020-09-18 17:18:45 +01:00
Marwan Aljubeh c00e078343 Add a nix.conf option for allowing a symlinked store 2020-09-18 17:10:39 +01:00
Bryan Richter 5fe375a8f1
nix-prefetch-url: Add --executable flag 
pkgs.fetchurl supports an executable argument, which is especially nice
when downloading a large executable. This patch adds the same option to
nix-prefetch-url.

I have tested this to work on the simple case of prefetching a little
executable:

1. nix-prefetch-url --executable https://my/little/script
2. Paste the hash into a pkgs.fetchurl-based package, script-pkg.nix
3. Delete the output from the store to avoid any misidentified artifacts
4. Realise the package script-pkg.nix
5. Run the executable

I repeated the above while using --name, as well.

I suspect --executable would have no meaningful effect if combined with
--unpack, but I have not tried it.
 2020-09-18 19:09:45 +03:00
Maximilian Bosch 2bcf8cbe7a
libfetchers/github: allow url attribute 
Since 108debef6f we allow a
`url`-attribute for the `github`-fetcher to fetch tarballs from
self-hosted `gitlab`/`github` instances.

However it's not used when defining e.g. a flake-input

    foobar = {
        type = "github";
        url = "gitlab.myserver";
        /* ... */
    }

and breaks with an evaluation-error:

    error: --- Error --------------------------------------nix
    unsupported input attribute 'url'
    (use '--show-trace' to show detailed location information)

This patch allows flake-inputs to be fetched from self-hosted instances
as well.
 2020-09-18 14:10:45 +02:00
Eelco Dolstra 958bf57123 nix build: find() -> get() 
find() returns an iterator so "!attr" doesn't work.
 2020-09-18 13:10:42 +02:00
Robert Hensing 14b30b3f3d Move FramedSource and FramedSink, extract withFramedSink 2020-09-17 20:21:04 +02:00
Robert Hensing dfa547c6a8 Add ContentAddressMethod and parse/render it 2020-09-17 20:21:04 +02:00
Robert Hensing 29c82ccc77 Add Source.drainInto(Sink) 2020-09-17 20:21:04 +02:00
Robert Hensing 9ee3122ec7 Remove redundant import 2020-09-17 20:21:04 +02:00
Robert Hensing 3f93bc0d39 Typo 2020-09-17 20:21:04 +02:00
John Ericson b7df353f27 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-17 16:33:10 +00:00
Eelco Dolstra 649d3aaf24
Merge pull request #3829 from obsidiansystems/remove-storetype-delegate-regStore 
Remove storetype delegate reg store -- contains #3736
 2020-09-17 13:55:01 +02:00
Eelco Dolstra b94a35ef40
Merge pull request #4027 from tweag/fix-gc-of-ca-derivations 
Fix garbage collection of CA derivations
 2020-09-17 13:46:26 +02:00
regnat 520895b1da Fix garbage collection of CA derivations 
Fix #4026
 2020-09-17 13:36:58 +02:00
Eelco Dolstra 787469c7b6 Remove corepkgs/unpack-channel.nix 2020-09-17 09:41:02 +02:00
Eelco Dolstra 10d1865f5f Remove corepkgs/derivation.nix 2020-09-17 09:41:02 +02:00
John Ericson f60b380a7f Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-16 22:35:24 +00:00
John Ericson 2741fffa35 Ensure resolved CA derivations are written 
so we can link outputs to deriver and thus properly cache.
 2020-09-16 17:57:32 +00:00
John Ericson 7fdbb377ba Start to fix floating CA + remote building 2020-09-16 17:50:48 +00:00
John Ericson c5ccebae00 Merge remote-tracking branch 'upstream/master' into ca-floating-upstream 2020-09-16 17:50:40 +00:00
Greg Hale a303c0b6dc Fetch commits from github/gitlab using Auth header 
`nix flake info` calls the github 'commits' API, which requires
authorization when the repository is private. Currently this request
fails with a 404.

This commit adds an authorization header when calling the 'commits' API.
It also changes the way that the 'tarball' API authenticates, moving the
user's token from a query parameter into the Authorization header.

The query parameter method is recently deprecated and will be disallowed
in November 2020. Using them today triggers a warning email.
 2020-09-16 13:46:48 -04:00
Eelco Dolstra 5080d4e7b2 Merge branch 'document-store-options' of https://github.com/tweag/nix 2020-09-16 17:02:30 +02:00
Eelco Dolstra 0066ef6c59 Fix doc generation 2020-09-16 16:56:28 +02:00
Eelco Dolstra 2eacc1bc00 builtins.toFile: Fix indentation 2020-09-16 14:18:46 +02:00
Théophane Hufschmitt 77a0e2c5be
Remove useless exception copy 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-09-16 14:00:21 +02:00
regnat d72927aa7a Fix the s3 store 
Add some necessary casts in the initialisation of the store's config
 2020-09-16 13:53:09 +02:00
regnat e0817cbcdc Don't include nlohmann/json.hpp in config.hh 
Instead make a separate header with the template implementation of
`BaseSetting<T>::toJSONObj` that can be included where needed
 2020-09-16 13:53:09 +02:00
regnat 93c0e14a30 Include the full nlohmann/json header in config.hh 
It is apparently required for using `toJSONObject()`, which we do inside
the header file (because it's in a template).

This was accidentally working when building Nix itself (presumably because
`config.hh` was always included after `nlohman/json.hpp`) but caused a
(pretty dirty) build failure in the perl bindings package.
 2020-09-16 13:53:09 +02:00
regnat fc2d31c423 Add (StoreConfig*) casts to work around a GCC bug 
Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was
already there in the code but was accidentally removed in the last
commits
 2020-09-16 13:53:09 +02:00
regnat a1e82ba450 fixup! Add a default value for the settings 2020-09-16 13:53:09 +02:00
regnat 888f7afe9f Fix build issues with gcc 2020-09-16 13:53:09 +02:00
regnat 634cb2a5ae Add a markdown output to nix describe-stores 2020-09-16 13:53:09 +02:00
regnat b73adacc1e Add a name to the stores 
So that it can be printed by `nix describe-stores`
 2020-09-16 13:53:09 +02:00
regnat f24f0888f9 Document the new store hierarchy 2020-09-16 13:53:09 +02:00
regnat d65962db4d Make uri schemes grammar more RFC-compliant 
Allow `-` and `.` in the RFC schemes as stated by
[RFC3986](https://tools.ietf.org/html/rfc3986#section-3.1).

Practically, this is needed so that `ssh-ng` is a valid URI scheme
 2020-09-16 13:53:09 +02:00
regnat 29a632386e fixup! Make the store plugins more introspectable 2020-09-16 13:53:09 +02:00
regnat 1129913c4e fixup! Correctly call all the parent contructors of the stores 2020-09-16 13:53:09 +02:00
regnat 7f103dcddd Properly filter the stores according to their declared uriSchemes 
When opening a store, only try the stores whose `uriSchemes()` include
the current one
 2020-09-16 13:53:09 +02:00
regnat 5895184df4 Correctly call all the parent contructors of the stores 
Using virtual inheritance means that only the default constructors of
the parent classes will be called, which isn't what we want
 2020-09-16 13:53:09 +02:00
regnat dae39f0a7a Make nix describe-stores functional 
Using the `*Config` class hierarchy
 2020-09-16 13:53:08 +02:00
regnat 22afa8fb4d Separate store configs from the implems 
Rework the `Store` hierarchy so that there's now one hierarchy for the
store configs and one for the implementations (where each implementation
extends the corresponding config). So a class hierarchy like

```
StoreConfig-------->Store
    |                 |
    v                 v
SubStoreConfig----->SubStore
    |                 |
    v                 v
SubSubStoreConfig-->SubSubStore
```

(with virtual inheritance to prevent DDD).

The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
 2020-09-16 13:53:08 +02:00
regnat aa4eac3788 fixup! Separate the instantiation and initialisation of the stores 2020-09-16 13:53:08 +02:00
regnat 35042c9623 Add a default value for the settings 
The default value is initialized when creating the setting and unchanged
after that
 2020-09-16 13:53:08 +02:00
regnat 3c525d1590 Complete the toJSON instance for Setting<T> 
Don't let it just contain the value, but also the other fields of the
setting (description, aliases, etc..)
 2020-09-16 13:53:08 +02:00
regnat 3b57181f8e Separate the instantiation and initialisation of the stores 
Add a new `init()` method to the `Store` class that is supposed to
handle all the effectful initialisation needed to set-up the store.
The constructor should remain side-effect free and just initialize the
c++ data structure.

The goal behind that is that we can create “dummy” instances of each
store to query static properties about it (the parameters it accepts for
example)
 2020-09-16 13:53:08 +02:00
regnat fa32560169 Fix the registration of stores 2020-09-16 13:53:08 +02:00
regnat 7d5bdf8b56 Make the store plugins more introspectable 
Directly register the store classes rather than a function to build an
instance of them.
This gives the possibility to introspect static members of the class or
choose different ways of instantiating them.
 2020-09-16 13:53:08 +02:00
John Ericson 3a5cdd737c Rename Derivation::pathOpt to Derivation::path 
We no longer need the `*Opt` to disambiguate.
 2020-09-15 15:21:39 +00:00
John Ericson 6387550d58 Get rid of confusing std::optional<bool> for validity 2020-09-15 15:19:45 +00:00
John Ericson c4bf219b55 Don't link deriver until after any delayed exception is thrown 
Otherwise, we will associate fixed-output derivations with outputs that
they did indeed produce, but which had the wrong hash. That's no good.
 2020-09-15 14:28:06 +00:00
John Ericson 3ba552b245 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-15 14:17:06 +00:00
John Ericson c08c9f08c7 Merge remote-tracking branch 'upstream/master' into remove-storetype-delegate-regStore 2020-09-15 14:08:35 +00:00
Eelco Dolstra 2a80170920
Merge pull request #4014 from tweag/fix-queryDrvOutputMaps-for-old-daemons 
gracefully handle old daemon versions
 2020-09-15 11:25:19 +02:00
regnat 057c6203b5 gracefully handle old daemon versions 
Add a fallback path in `queryPartialDerivationOutputMap` for daemons
that don't support it.

Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this
is needed to handle the fallback path.
 2020-09-15 09:58:01 +02:00
regnat 250f8a4bba Escape ${ in strings when printing Nix expressions 
Otherwise the result of the printing can't be parsed back correctly by
Nix (because the unescaped `${` will be parsed as the begining of an
anti-quotation).

Fix #3989
 2020-09-14 17:19:25 +02:00
Eelco Dolstra a59e77d9e5 nix-daemon: Lower verbosity of restricted setting warning 
Fixes #3992.
 2020-09-14 13:48:51 +02:00
Eelco Dolstra 35ba092019
Merge pull request #4005 from Infinisil/fix-autoArgs 
Fix auto argument passing for more auto arguments than formals
 2020-09-14 13:31:23 +02:00
aszlig 525b38eee8
Fix unspecified behaviour in readStorePathCAMap 
When deploying a Hydra instance with current Nix master, most builds
would not run because of errors like this:

  queue monitor: error: --- Error --- hydra-queue-runner
  error: --- UsageError --- nix-daemon
  not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv

The last error message is from parseContentAddress, which expects a
colon-separated string, however what we got here is a store path.

Looking at the worker protocol, the following message sent to the Nix
daemon caused the error above:

  0x1E -> wopQuerySubstitutablePathInfos
  0x01 -> Number of paths
  0x16 -> Length of string
  "/nix/store/...-somedrv"
  0x00 -> Length of string
  ""

Looking at writeStorePathCAMap, the store path is indeed the first field
that's transmitted. However, readStorePathCAMap expects it to be the
*second* field *on my machine*, since expression evaluation order is a
classic form of unspecified behaviour[1] in C++.

This has been introduced in https://github.com/NixOS/nix/pull/3689,
specifically in commit 66a62b3189.

[1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions

Signed-off-by: aszlig <aszlig@nix.build>
 2020-09-13 02:40:51 +02:00
Silvan Mosberger 8dbd57a6a5
Fix auto argument passing for more auto arguments than formals 
The change in 626200713b didn't account
for when the number of auto arguments is bigger than the number of
formal arguments. This causes the following:

  $ nix-instantiate --eval -E '{ ... }@args: args.foo' --argstr foo foo
  nix-instantiate: src/libexpr/attr-set.hh:55: void nix::Bindings::push_back(const nix::Attr&): Assertion `size_ < capacity_' failed.
  Aborted (core dumped)
 2020-09-11 20:41:51 +02:00
Gabriel Gonzalez ee5906243a Add nix-shell support for preserving PS1 
Fixes https://github.com/NixOS/nix/issues/1268

`nix-shell` will now preserve `PS1` if the `NIX_SHELL_PRESERVE_PROMPT`
environment variable is set.
 2020-09-04 20:05:43 -07:00
John Ericson 98dfd7531d Fix querying outputs for CA derivations some more 
If we resolve using the known path of a derivation whose output we
didn't have, we previously blew up. Now we just fail gracefully,
returning the map of all outputs unknown.
 2020-09-04 18:33:58 +00:00
John Ericson 075d399e3f Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-04 16:04:35 +00:00
John Ericson 5aed6f9b25 Document mkOutputString 2020-09-04 15:58:42 +00:00
John Ericson e9fad3006b Fix some of the issues raised by @edolstra 
- More and better comments

 - The easier renames
 2020-09-04 15:18:25 +00:00
John Ericson e86dd59dcc
Apply suggestions from code review 
Thanks!

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-09-04 10:48:50 -04:00
John Ericson c9f1ed912c Don't chmod symlink before moving outputs around 
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
 2020-09-04 14:41:53 +00:00
John Ericson 25f7ff16fa Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-09-04 02:40:36 +00:00
John Ericson aad4abcc9c Fix floating CA tests 
We will sometimes try to query the outputs of derivations we can't
resolve. That's fine; it just means we don't know what those outputs are
yet.
 2020-09-04 01:17:38 +00:00
John Ericson 975a47f7fe Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-09-03 22:09:04 +00:00
John Ericson e7d93e7ece Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-03 15:43:17 +00:00
Eelco Dolstra 8a945d6ddb
Remove gc-check-reachability 2020-09-03 11:30:15 +02:00
Eelco Dolstra 82b77a7726
addPermRoot(): Remove indirect flag 2020-09-03 11:29:34 +02:00
Eelco Dolstra 00d25e8457
Remove the --indirect flag 
All GC roots are now indirect.
 2020-09-03 11:22:00 +02:00
Eelco Dolstra b07167be5a
createGeneration(): Always create an indirect root 
This means profiles outside of /nix/var/nix/profiles don't get
garbage-collected. It also means we don't need to scan
/nix/var/nix/profiles for GC roots anymore, except for compatibility
with previously existing generations.
 2020-09-03 11:13:53 +02:00
Eelco Dolstra b74f5cdd23
createGeneration(): Take a StorePath 2020-09-03 11:06:56 +02:00
Eelco Dolstra 94a043ff3b
EvalCache: Fix caching of strings 
This was broken in 50f13b06fb. Once
again it turns out that putting a bool in a std::variant is a bad
idea, since pointers get silently cast to them...
 2020-09-02 14:16:44 +02:00
Domen Kožar dd4b56c87f
Allow HTTP binary cache to request absolute uris 2020-09-01 21:35:48 +02:00
John Ericson ef278d00f9 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-09-01 18:01:48 +00:00
Eelco Dolstra 6d7f7efb89
github: Use access token when calling .../commits API 2020-09-01 15:29:04 +02:00
Eelco Dolstra 84f5cabbea Merge remote-tracking branch 'origin/master' into markdown 2020-08-31 14:24:26 +02:00
John Ericson 02e0001fc0 Merge branch 'ca-floating-upstream' of github.com:obsidiansystems/nix into ca-floating-upstream 2020-08-28 21:23:09 +00:00
John Ericson 421ed527c7
Update src/libstore/build.cc 
Thanks for catching, @regnat.
 2020-08-28 17:22:57 -04:00
John Ericson 48e8828050 Merge remote-tracking branch 'obsidian/single-ca-drv-build' into ca-floating-upstream 2020-08-28 20:38:25 +00:00
John Ericson 8017fe7487 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-28 19:59:14 +00:00
Eelco Dolstra f15651303f nix develop: Add convenience flags for running specific phases 
For example, for building the Nix flake, you would do:

  $ nix develop --configure
  $ nix develop --install
  $ nix develop --installcheck
 2020-08-28 19:24:29 +02:00
Eelco Dolstra 3156560d41 nix develop: Set output paths to writable locations 
Currently, they're set to $(pwd)/outputs/$outputName. This allows
commands like 'make install' to work.
 2020-08-28 18:16:03 +02:00
Eelco Dolstra 691a1bd717 Merge branch 'minimal-logger' of https://github.com/Ma27/nix 2020-08-28 10:50:52 +02:00
Griffin Smith 626200713b Pass all args when auto-calling a function with an ellipsis 
The command line options --arg and --argstr that are used by a bunch of
CLI commands to pass arguments to top-level functions in files go
through the same code-path as auto-calling top-level functions with
their default arguments - this, however, was only passing the arguments
that were *explicitly* mentioned in the formals of the function - in the
case of an as-pattern with an ellipsis (eg args @ { ... }) extra passed
arguments would get omitted. This fixes that to instead pass *all*
specified auto args in the case that our function has an ellipsis.

Fixes #598
 2020-08-27 12:38:25 -04:00
Eelco Dolstra eb75282b8d
Merge pull request #3434 from Ericson2314/derivation-header-include-order 
Revise division of labor in deserialization of derivations
 2020-08-27 16:39:28 +02:00
Eelco Dolstra e915fd6d2a Typo 2020-08-27 14:51:50 +02:00
Eelco Dolstra a0f19d9f3a RemoteStore::addToStore(): Fix race between stderrThread and NAR writer 
As pointed out by @B4dM4n, the call to to.flush() on stderrThread is
unsafe because the NAR writer thread is also writing to 'to'.

Fixes #3943.
 2020-08-27 14:50:51 +02:00
Eelco Dolstra c3efef9275 Remove obsolete comment 2020-08-26 09:28:10 +02:00
Eelco Dolstra 4bf5faf416 Merge remote-tracking branch 'origin/master' into markdown 2020-08-25 19:47:34 +02:00
Eelco Dolstra 7a02865b94
Move import docs 2020-08-25 14:06:01 +02:00
Eelco Dolstra f53b5f1058
Add getDoc() function 2020-08-25 13:31:11 +02:00
Eelco Dolstra 2a2121d264
Use RegisterPrimOp for some undocumented primops 2020-08-25 11:25:01 +02:00
Eelco Dolstra b8416779e3
Document some primops 2020-08-25 11:16:45 +02:00
Eelco Dolstra 24b1c2c66b
Fix tests 2020-08-25 10:51:14 +02:00
Eelco Dolstra b42789f013
Fix clang build 2020-08-24 21:13:39 +02:00
John Ericson 8eb73a8724 CA derivations that depend on other CA derivations 
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
 2020-08-24 19:01:15 +00:00
John Ericson e0b0e18905 Add constructor for BasicDerivation -> Derivation 2020-08-24 19:01:07 +00:00
John Ericson 59979e7053 Fix bad debug format string 2020-08-24 19:01:07 +00:00
Eelco Dolstra d9a8619762
Don't barf if corepkgs is in the store but not a valid path 
This can happen when using a dummy store (or indeed any non-local store).
 2020-08-24 19:15:17 +02:00
Eelco Dolstra 6a67e57019
Add DummyStore (dummy://) 
DummyStore does not allow building or adding paths. This is useful for
evaluation tests when you don't want to initialize a "proper" store.
 2020-08-24 18:54:16 +02:00
Eelco Dolstra d0690bc311
nix repl ':doc': Render using lowdown 2020-08-24 18:10:33 +02:00
Eelco Dolstra 0f314f3c25
Generate builtins section of the manual 2020-08-24 14:49:30 +02:00
Eelco Dolstra a990f063ff
Move primop docs inline 
This makes them available to 'nix repl'.
 2020-08-24 14:31:10 +02:00
Eelco Dolstra 33b1679d75
Allow primops to have Markdown documentation 2020-08-24 13:16:02 +02:00
Eelco Dolstra 88d5c9ec58
Fix tests 2020-08-24 10:37:10 +02:00
Eelco Dolstra 2530cc6827
Merge pull request #3950 from obsidiansystems/tabs-spaces 
tabs -> spaces
 2020-08-24 10:20:28 +02:00
John Ericson b0b59fd05a Merge remote-tracking branch 'obsidian/write-derivation-borrow' into HEAD 2020-08-23 15:11:10 +00:00
John Ericson 35e6288be1 writeDerivation just needs a plain store reference 2020-08-23 15:01:11 +00:00
John Ericson 3a7b330b64 "Downstream placeholders" should not be store paths 
Insead they should be opaque `/<hash>` like the placeholders we already
have.
 2020-08-21 19:35:35 +00:00
John Ericson 422affe102 tabs -> spaces 
Sorry I let the tab sneak in there in the first place.
 2020-08-21 19:19:24 +00:00
John Ericson c265e0ea82 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-08-20 20:24:22 +00:00
John Ericson 27a3f82c0b Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-20 18:28:17 +00:00
John Ericson 45a2f1baab Rename drv output querying functions, like master 
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap
 2020-08-20 18:14:12 +00:00
Eelco Dolstra 4d77513d97
Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing 
`queryDerivationOutputMap` no longer assumes all outputs have a mapping
 2020-08-20 16:49:23 +02:00
John Ericson 9a9d834dc7 Rename drv output querying functions 
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`
 2020-08-20 14:12:51 +00:00
Eelco Dolstra 25ecfffdc3
Remove PrimOp constructor 2020-08-20 12:34:04 +02:00
Eelco Dolstra dc2f278c95
Allow 'nix' subcommands to provide docs in Markdown format 2020-08-20 12:21:46 +02:00
Eelco Dolstra 3c4f8c9175
List deprecated option aliases in the docs 2020-08-20 11:13:17 +02:00
Eelco Dolstra acb99f03f9
Config: Use nlohmann/json 2020-08-20 11:02:16 +02:00
John Ericson 3df78858f2 Fix max fd calc and add test 2020-08-20 05:08:50 +00:00
Ryan Mulligan f36793c7b9
fix spelling 2020-08-19 20:31:01 -07:00
John Ericson a83694c7a1 Use RemoteStore to open connection for proxying daemon 
Removes duplicate websocket opening code, and also means we should be
able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.
 2020-08-19 19:34:47 +00:00
Eelco Dolstra c8fa39324a
Generate the nix.conf docs from the source code 
This means we don't have two (divergent) sets of option descriptions
anymore.
 2020-08-19 18:28:04 +02:00
John Ericson d5af5763cf Merge branch 'master' of github.com:NixOS/nix into remove-storetype-delegate-regStore 2020-08-19 15:54:50 +00:00
Eelco Dolstra 34b22e0123
Change option descriptions to Markdown 2020-08-19 14:21:27 +02:00
John Ericson be0d429b95 Merge branch 'master' of github.com:NixOS/nix into templated-daemon-protocol 2020-08-19 03:17:41 +00:00
John Ericson 950ddfdb82 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-18 14:36:44 +00:00
Eelco Dolstra 1c8b550e34
Merge pull request #3917 from obsidiansystems/output-env-var-unconditional 
Simplify code as output env vars are unconditional
 2020-08-18 16:21:17 +02:00
Eelco Dolstra dfeb76dbf9
Merge pull request #3930 from obsidiansystems/legacy-ssh-build-paths 
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths`
 2020-08-18 16:07:40 +02:00
Eelco Dolstra 069340179e
Improve nix.1 manpage generator 2020-08-18 15:15:35 +02:00
Eelco Dolstra 6f19c776db
Start generation of the nix.1 manpage 2020-08-17 19:33:18 +02:00
Eelco Dolstra a72a20d68f
Add 'nix dump-args' to dump all commands/flags for manpage generation 2020-08-17 17:44:52 +02:00
Eelco Dolstra 7cdc739ece
Merge remote-tracking branch 'origin/master' into markdown 2020-08-17 13:43:39 +02:00
Eelco Dolstra e849b19872
Merge pull request #3932 from chkno/no-show-signature 
Don't try to parse signature check as commit timestamp
 2020-08-17 11:00:24 +02:00
Eelco Dolstra 847a5392f4 Merge branch 'ca-no-need-trust' of https://github.com/obsidiansystems/nix 2020-08-17 10:57:02 +02:00
Eelco Dolstra 3c619f6290 Merge branch 'test-RemoteStore-buildDerivation' of https://github.com/obsidiansystems/nix 2020-08-17 10:53:18 +02:00
John Ericson 6f7ac5e865 Remove extra closing paren 2020-08-14 21:59:31 +00:00
John Ericson f899a7c6d7 Work around clang bug 2020-08-14 18:51:31 +00:00
John Ericson 3c8b5b6219 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-14 17:00:13 +00:00
John Ericson 4b571ea321
Update src/libstore/daemon.cc 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-08-14 11:52:37 -04:00
Eelco Dolstra 13e49be660
Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt 
Offer a safer interface for path and pathOpt
 2020-08-14 17:19:19 +02:00
Eelco Dolstra 7714d9a943
Merge pull request #3924 from obsidiansystems/features-per-store 
Make `system-features` a store setting
 2020-08-14 17:13:07 +02:00
Eelco Dolstra 9b9d5297ba
Merge pull request #3909 from matthewbauer/readd-hashed-mirrors 
Add hashed-mirrors back
 2020-08-14 17:05:47 +02:00
Eelco Dolstra d81f13f7cb
Merge pull request #3899 from obsidiansystems/make-narHash-not-optional 
Make narHash in ValidPathInfo not optional
 2020-08-14 17:00:18 +02:00
Chuck ed026f7206 Don't try to parse signature check as commit timestamp 
When the log.showSignature git setting is enabled, the output of
"git log" contains signature verification information in addition to the
timestamp GitInputScheme::fetch wants:

  $ git log -1 --format=%ct
  gpg: Signature made Sat 07 Sep 2019 02:02:03 PM PDT
  gpg:                using RSA key 0123456789ABCDEF0123456789ABCDEF01234567
  gpg:                issuer "user@example.com"
  gpg: Good signature from "User <user@example.com>" [ultimate] 1567890123
  1567890123

For folks that had log.showSignature set, this caused all nix operations
on flakes to fail:

  $ nix build
  error: stoull
 2020-08-13 17:44:42 -07:00
John Ericson e1308b1211 Define LegacySSHStore::buildPaths using cmdBuildPaths 
Evidentally this was never implemented because Nix switched to using
`buildDerivation` exclusively before `build-remote.pl` was rewritten.

The `nix-copy-ssh` test (already) tests this.
 2020-08-13 21:27:55 +00:00
John Ericson 5ccd94501d Allow trustless building of CA derivations 
Include a long comment explaining the policy. Perhaps this can be moved
to the manual at some point in the future.

Also bump the daemon protocol minor version, so clients can tell whether
`wopBuildDerivation` supports trustless CA derivation building. I hope
to take advantage of this in a follow-up PR to support trustless remote
building with the minimal sending of derivation closures.
 2020-08-13 18:15:57 +00:00
John Ericson 85aacbee64 Use TeeSink and TeeSouce in a few more places 2020-08-13 14:51:17 +00:00
John Ericson d2f2be0f70 Test RemoteStore::buildDerivation 
Fix `wopNarFromPath` which needed a `toRealPath`.
 2020-08-13 04:07:14 +00:00
John Ericson 5d67f18c86 Merge branch 'daemon-auth-cleanup' of github.com:obsidiansystems/nix into HEAD 2020-08-12 18:22:31 +00:00
John Ericson 4720853129 Make system-features a store setting 
This seems more correct. It also means one can specify the features a
store should support with --store and remote-store=..., which is useful.
I use this to clean up the build remotes test.
 2020-08-12 18:13:00 +00:00
John Ericson 8d4162ff9e Separate auth and logic for the daemon 
Before, processConnection wanted to know a user name and user id, and
`nix-daemon --stdio`, when it isn't proxying to an underlying daemon,
would just assume "root" and 0. But `nix-daemon --stdio` (no proxying)
shouldn't make guesses about who holds the other end of its standard
streams.

Now processConnection takes an "auth hook", so `nix-daemon` can provide
the appropriate policy and daemon.cc doesn't need to know or care what
it is.
 2020-08-12 15:22:33 +00:00
John Ericson 5f80aea795 Break out lambda so output can be matched just once 
This is much better.
 2020-08-12 02:23:31 +00:00
John Ericson 18834f7764 Recheck path validity after acquiring lock 
It might have changed, and in any event this is how the cod used to work
so let's just keep it.
 2020-08-11 23:44:02 +00:00
John Ericson 2de201254e Don't assume a total output map in two places in build.cc 
Thanks @regnat for catching one of them. The other follows for many of
the same reasons. I'm find fixing others on a need-to-fix basis,
provided their are no regressions.
 2020-08-11 23:07:50 +00:00
John Ericson 4c6aac8fdf Clarify comment on sandbox and temp fresh paths 2020-08-11 22:46:05 +00:00
John Ericson 6d57139050 Clarify outputReferences variable with self-describing type 
Thanks for the idea, @Regnat!
 2020-08-11 22:34:09 +00:00
John Ericson 8a068bd025 Remove redundant equality check 2020-08-11 21:25:40 +00:00
John Ericson 07e3466eb4 Float comment to out describe gaveUpOnSubstitution in general 2020-08-11 21:16:14 +00:00
John Ericson d0f6e338dd
Apply suggestions from code review 
Thanks!!

Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
 2020-08-11 16:49:10 -04:00
Ben Burdette 1a281ec07f demote remote build message to Info 2020-08-11 10:29:43 -06:00
Eelco Dolstra 7a911b6783
Merge pull request #3914 from obsidiansystems/small-drv-serialize-cleanup 
Two small derivation serialization cleanups
 2020-08-11 07:14:38 +02:00
John Ericson d3fa8c04c6 Simplify code as output env vars are unconditional 
Since the jsonObject unique ptr is reset to flush the string to make
`__json`, all these `!jsonObject` conditions will always be true.
 2020-08-11 01:13:26 +00:00
John Ericson 2a0902634e Fix error in merge breaking floating CA drvs 
Forgot to add this hunk!
 2020-08-11 00:13:19 +00:00
Maximilian Bosch 5f8ae16c8b
Always reset ANSI colors in progress-bar line 
When having a message like `waiting for a machine to build X` and
building with `nix build -L`, the log-prefix is always colored yellow[1]
on a small terminal-width as everything (including the ANSI color-reset) is
stripped away.

To work around that problem, this patch explicitly adds an `ANSI_NORMAL`
to the end of the line.

[1] https://imgur.com/a/FjtJOk3
 2020-08-10 17:44:17 +02:00
John Ericson 1b5c24662b Merge branch 'small-drv-serialize-cleanup' of github.com:obsidiansystems/nix into single-ca-drv-build 2020-08-10 01:57:54 +00:00
John Ericson 581183d4d5 Deduplicate parsing and reading derivations 2020-08-10 01:40:50 +00:00
John Ericson bcd0629c2e Remove name parameter from writeDerivation 
The name is now stored with the derivation itself.
 2020-08-10 01:35:59 +00:00
John Ericson e913a2989f Squashed get CA derivations building 2020-08-07 19:51:55 +00:00
Eelco Dolstra edfd676e05 Fix .ls file names in binary caches 
These are not supposed to include the 'name' part of the store
path. This was broken by 759947bf72.
 2020-08-07 21:18:29 +02:00
John Ericson 8f92bb5ad9 Merge branch 'drv-outputs-map-allow-missing' of github.com:obsidiansystems/nix into templated-daemon-protocol 2020-08-07 18:51:01 +00:00
John Ericson f7ba16f9cb Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-07 17:46:39 +00:00
John Ericson 47644e49ca Specialize std::optional<StorePath> so this is backwards compatible 
While I am cautious to break parametricity, I think it's OK in this
cases---we're not about to try to do some crazy polymorphic protocol
anytime soon.
 2020-08-07 17:05:14 +00:00
John Ericson 39ae9a3d4a Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-08-07 14:44:47 +00:00
Eelco Dolstra 3c75ddc16b nix build (and others): Force re-evaluation of cached errors 
Fixes #3872.

This is a bit hacky. Ideally we would automatically re-evaluate the
failed attribute iff we need to print the error message (so in
commands like 'nix search' we wouldn't re-evaluate because we're
suppressing errors).
 2020-08-07 14:47:23 +02:00
Eelco Dolstra 2ffc058950 Make --no-eval-cache a global setting 2020-08-07 14:13:24 +02:00
Matthew Bauer 96c158d6e1 Fix build 2020-08-06 21:04:31 -05:00
Matthew Bauer 8abc577cc2 Merge remote-tracking branch 'origin/master' into readd-hashed-mirrors 2020-08-06 18:58:59 -05:00
Carlo Nucera 46f9dd56da Fix bug due to non-deterministic arg eval order 2020-08-06 19:30:05 -04:00
Matthew Bauer 641c950701 Add hashed-mirrors back 
Some users have their own hashed-mirrors setup, that is used to mirror
things in addition to what’s available on tarballs.nixos.org. Although
this should be feasable to do with a Binary Cache, it’s not always
easy, since you have to remember what "name" each of the tarballs has.
Continuing to support hashed-mirrors is cheap, so it’s best to leave
support in Nix. Note that NIX_HASHED_MIRRORS is also supported in
Nixpkgs through fetchurl.nix.

Note that this excludes tarballs.nixos.org from the default, as in
\#3689. All of these are available on cache.nixos.org.
 2020-08-06 18:19:09 -05:00
Carlo Nucera 9ab07e99f5 Use template structs instead of phantoms 2020-08-06 18:04:13 -04:00
Carlo Nucera 3d8240c32e Remove leftover commented code 2020-08-06 16:04:18 -04:00
Carlo Nucera f795f0fabc Merge branch 'drv-outputs-map-allow-missing-namespace' of github.com:obsidiansystems/nix into templated-daemon-protocol 2020-08-06 15:53:09 -04:00
John Ericson e89b5bd0bf Minimize the usage of Hash::dummy 2020-08-06 18:31:48 +00:00
John Ericson 5e59b25a23 Merge remote-tracking branch 'upstream/master' into make-narHash-not-optional 2020-08-06 15:34:15 +00:00
Eelco Dolstra 6146447842
Merge pull request #3856 from obsidiansystems/buildable-variant 
Make `Buildable` a `std::variant`
 2020-08-06 14:14:06 +02:00
Eelco Dolstra 59067f0f58 repl.cc: Check for HAVE_BOEHMGC 
Fixes #3906.
 2020-08-06 11:40:41 +02:00
Carlo Nucera 8b175f58d1 Simplify the namespace 2020-08-05 17:57:07 -04:00
Carlo Nucera 0739d428e0 Solve template deduction problem 
We had to predeclare our template functions
 2020-08-05 17:49:45 -04:00
John Ericson 6c66331d5c WIP: Put the worker protocol read and write in a namespace to disambig 2020-08-05 20:37:48 +00:00
Carlo Nucera 1d2e80ddd6 Merge branch 'master' of github.com:NixOS/nix into new-interface-for-path-pathOpt 2020-08-05 15:45:33 -04:00
John Ericson ed96e603e1 Proxy -> Phantom to match Rust 
Sorry, Haskell.
 2020-08-05 19:44:08 +00:00
Carlo Nucera 8241e660ba Remove Hash::operator bool () 
Since the hash is not optional anymore
 2020-08-05 15:30:38 -04:00
Carlo Nucera be6e1c6457 Merge branch 'master' of github.com:NixOS/nix into make-narHash-not-optional 2020-08-05 15:14:47 -04:00
Carlo Nucera 1ad6394b33 Add Hash::dummy to signal default value 
We did this in the same spirit of the dummy value that's present in
libstore/path.hh
 2020-08-05 15:11:49 -04:00
Carlo Nucera 1d71028f4d Remove optionality in ValidPathInfo::narInfo 2020-08-05 14:42:48 -04:00
John Ericson 7302761f64 Merge remote-tracking branch 'obsidian/drv-outputs-map-allow-missing' into templated-daemon-protocol 2020-08-05 17:53:24 +00:00
Ben Burdette 31f1af0cab don't crash if there's no drvPath 2020-08-05 11:26:06 -06:00
Ben Burdette e4eae078a5 add derivation path to hint 2020-08-05 11:21:36 -06:00
Ben Burdette f1a47a96b6 error messages for issue 2238 2020-08-05 10:58:00 -06:00
John Ericson cf939055c8 Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-05 16:43:30 +00:00
John Ericson a9bbfaa851 Fix --profile with multiple opaque paths 2020-08-05 16:27:15 +00:00
John Ericson d89472a912 Merge remote-tracking branch 'upstream/master' into buildable-variant 2020-08-05 15:41:57 +00:00
John Ericson b3e73547a0
Update src/libexpr/primops.cc 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-08-05 11:05:46 -04:00
John Ericson 0559ff3d8b Merge branch 'master' of github.com:NixOS/nix into derivation-primop-floating-output 2020-08-05 14:56:40 +00:00
John Ericson 92ad550e96 Merge remote-tracking branch 'obsidian/misc-ca' into derivation-primop-floating-output 2020-08-05 14:51:41 +00:00
Eelco Dolstra 790b694be7 Style fix 2020-08-05 16:51:06 +02:00
Eelco Dolstra e48f944e9d Merge branch 'misc-ca' of https://github.com/obsidiansystems/nix 2020-08-05 16:50:05 +02:00
John Ericson b9ebe373bb Sed some names to perhaps avoid conflicts 2020-08-05 14:49:25 +00:00
Eelco Dolstra 25f7912156 Style fix 2020-08-05 16:47:48 +02:00
John Ericson e561a13a58 Reanme DerivationType::Regular defintion too 
This is the one non-prefixed occurence
 2020-08-05 14:45:56 +00:00
John Ericson e7b0847f2d Make names more consistent 2020-08-05 14:44:39 +00:00
John Ericson 839f0fe095 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:40:01 +00:00
John Ericson 03f4fafc27 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:36:25 +00:00
Eelco Dolstra b91dc7ebad
Merge pull request #3730 from obsidiansystems/better-ca-parse-errors 
Improve hash parsing and errors
 2020-08-05 16:33:07 +02:00
Eelco Dolstra 75f220a595
Merge pull request #3864 from obsidiansystems/more-topo-sort 
Abstract out topo sorting logic
 2020-08-05 16:07:29 +02:00
Eelco Dolstra 088dcea0e8 Typo 2020-08-05 15:41:51 +02:00
John Ericson 16c98bf57c Get rid of some unneeded temporaries 2020-08-04 22:36:31 +00:00
John Ericson 1dfcbebc95 Organize and format code a bit 2020-08-04 22:28:10 +00:00
John Ericson 45b6fdb22b Remove unused functions 2020-08-04 22:10:13 +00:00
John Ericson 1bab8a321f Remove unneeded definition 
Template instantiations will cover this case fine.
 2020-08-04 21:56:42 +00:00
John Ericson 2f2ae993dc WIP systematize more of the worker protocol 
This refactor should *not* change the wire protocol.
 2020-08-04 19:02:05 +00:00
John Ericson fbeb8695fb Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-04 18:35:33 +00:00
John Ericson d3452a5ed6 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-08-04 16:13:20 +00:00
Eelco Dolstra 5b22a2c0d4
Merge pull request #3894 from obsidiansystems/json-test-order-agnostic 
Make JSON equality tests agnostic to ordering
 2020-08-04 17:24:46 +02:00
Eelco Dolstra 4e7f1c7f11 S3BinaryCacheStore: Fix size determination 2020-08-04 16:01:13 +02:00
Eelco Dolstra dfe66420e7 Revert "Remove putBytes" 
This reverts commit b8eea7e81a.
 2020-08-04 15:56:10 +02:00
Eelco Dolstra 327b1bf378 BinaryCacheStore: Explicitly flush file sink 
The file sink is also flushed in its destructor, but we ignore any
exceptions in the destructor.

Issue #3886.
 2020-08-04 14:54:07 +02:00
Eelco Dolstra ed52cf632b
Merge pull request #3892 from obsidiansystems/path-info-header 
Move ValidPathInfo to its own header
 2020-08-04 10:44:47 +02:00
John Ericson fe7e57a80d tab -> space 2020-08-04 03:47:09 +00:00
John Ericson 062533f7cd Merge remote-tracking branch 'upstream/master' into path-info-header 2020-08-03 21:02:28 +00:00
Eelco Dolstra 24e07c428f Delete compressed NARs 
Fixes #3891.
 2020-08-03 18:34:52 +02:00
John Ericson 54281f3ac1 addToStore in terms of addToStoreFromDump is not local-store-specific 2020-08-03 04:13:45 +00:00
John Ericson 9357512d73 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-08-01 19:38:35 +00:00
John Ericson c4ada76e86 Fix error message and avoid recalculation 2020-08-01 16:22:50 +00:00
John Ericson bc165e28ae Embelish documentation of new Hash functions 2020-08-01 15:32:20 +00:00
John Ericson 3cbee1e840 Convert to C-style comments 2020-08-01 15:26:57 +00:00
Carlo Nucera b6d97fdbf4 Merge branch 'master' of github.com:NixOS/nix into drv-outputs-map-allow-missing 2020-07-31 13:12:51 -04:00
Eelco Dolstra a3f9625818 Tweak description 2020-07-31 17:32:40 +02:00
Eelco Dolstra 935723eb3b Merge branch 'nix-command-build-check' of https://github.com/Ma27/nix 2020-07-31 17:30:55 +02:00
Eelco Dolstra f56dfce734 nix bundle: Set category 2020-07-31 17:30:12 +02:00
Maximilian Bosch 5a09eb86f1
nix/build: add --rebuild option 
Occasionally, `nix-build --check` is fairly helpful and I'd like to be
able to use this feature for flakes that need to be built with `nix
build` as well.
 2020-07-31 17:17:03 +02:00
Eelco Dolstra bf290c2306
Merge remote-tracking branch 'origin/master' into markdown 2020-07-31 16:07:04 +02:00
Eelco Dolstra 1d0a7b54fa
Enable syntax highlighting 2020-07-31 15:43:25 +02:00
Eelco Dolstra 0604cfd0eb
Merge pull request #3880 from matthewbauer/nix-bundle 
Add "bundle" command to Nix
 2020-07-31 11:45:53 +02:00
John Ericson e3a2154f5a Fix indentation 2020-07-31 01:07:59 +00:00
Carlo Nucera eee6ef86cd Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-30 18:27:25 -04:00
Matthew Bauer cdc2386644 Make expectedHash optional in prim_path 
This fixes an error found in builtins.path that looks like:

store path mismatch in (possibly filtered) path added from '/private/tmp/nix-shell.CyXViH/nix-test/filter-source/filterin'

when no hash is specified
 2020-07-30 16:40:40 -05:00
Matthew Bauer fa2d1fb36e Pass system to bundler 2020-07-30 15:37:05 -05:00
Matthew Bauer 22fcfdf18a Address misc review 2020-07-30 15:18:57 -05:00
Matthew Bauer 1a705637ce Remove single file restriction for bundler 2020-07-30 15:16:29 -05:00
Matthew Bauer 52407f83a1 exporter -> bundler 2020-07-30 15:16:23 -05:00
Matthew Bauer d7ffe327ae Merge remote-tracking branch 'origin/master' into fix-and-ci-static-builds 2020-07-30 14:59:57 -05:00
Matthew Bauer 05ac4db39a Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-30 12:38:24 -05:00
Matthew Bauer 5d04a4db9b Handle exporters checking correctly 2020-07-30 11:45:47 -05:00
Matthew Bauer 2f4250a416 Add "export" to Nix 
This adds a ‘nix export’ command which hooks into nix-bundle. It can
be used in a similar way as nix-bundle, with the benefit of hooking
into the new “app” functionality. For instance,

$ nix export nixpkgs#jq
$ ./jq --help
jq - commandline JSON processor [version 1.6]
...
$ scp jq machine-without-nix:
$ ssh machine-without-nix ./jq
jq - commandline JSON processor [version 1.6]
...

Note that nix-bundle currently requires Linux to run. Other exporters
might not have that requirement.

“exporters” are meant to be reusable, so that, other repos can
implement their own bundling.

Fixes #3705
 2020-07-30 11:33:22 -05:00
Eelco Dolstra a785b3eddf Fix build 2020-07-30 15:27:28 +02:00
Eelco Dolstra 3f6e88a552 unsigned long long -> uint64_t 2020-07-30 13:34:04 +02:00
Eelco Dolstra ebee2b7852 receiveContents(): unsigned int -> size_t 2020-07-30 13:00:30 +02:00
Mat Marini d9e23bfee2 Prevent nix-build from accepting --run/--command 2020-07-29 20:53:02 -04:00
Mat Marini e4f6f8da77 Prevent nix-build from accepting --packages 2020-07-29 20:50:06 -04:00
Eelco Dolstra 0c94c17644 Merge remote-tracking branch 'origin/master' into markdown 2020-07-29 18:08:57 +02:00
Eelco Dolstra f63839bfa4 Cleanup 2020-07-29 18:04:51 +02:00
Eelco Dolstra 39311e7009
Merge pull request #3549 from Ma27/fetchgit-hash 
Merge legacy `fetchGit`-builtin with the generic `fetchTree`-function
 2020-07-29 11:33:27 +02:00
Eelco Dolstra 7c097275c4
Merge pull request #3877 from matthewbauer/develop-continuous-regex 
Set regex_constants::match_continuous for quicker search in nix develop
 2020-07-29 10:07:52 +02:00
Matthew Bauer fa8515d7ec Set regex_constants::match_continuous for quicker search 
match_continuous limits the search to the current start position,
instead of searching the entire file.

On libc++, this improves performance dramatically:

$ time /nix/store/70ai68dfm6xbzwn26j5n4li9di52ylia-nix-3.0pre20200728_c159f48/bin/nix print-dev-env >/dev/null
/nix/store/70ai68dfm6xbzwn26j5n4li9di52ylia-nix-3.0pre20200728_c159f48/bin/ni  2.39s user 0.19s system 64% cpu 4.032 total
$ time /nix/store/cwjfxxlp83zln4mfyy1d2dbsx7f6s962-nix-3.0pre20200728_dirty/bin/nix print-dev-env >/dev/null
/nix/store/cwjfxxlp83zln4mfyy1d2dbsx7f6s962-nix-3.0pre20200728_dirty/bin/nix   0.09s user 0.05s system 65% cpu 0.204 total

Fixes #3874
 2020-07-29 00:57:20 -05:00
Eelco Dolstra 4c0077a07d Fix RemoteStore::addToStore() latency 
Since 6185d25e52, this was very
latency-bound since it required a round-trip for every 32 KiB. So for
example copying a 514 MiB closure over a virtual ethernet device with
a articial delay of just 1 ms took 343s. Now it takes 2.7s.

Fixes #3372.
 2020-07-29 00:48:39 +02:00
Eelco Dolstra c159f48a39 Cleanup 2020-07-29 00:24:55 +02:00
John Ericson df707d05d1 Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca 2020-07-28 21:12:36 +00:00
Maximilian Bosch e4940e90f3
Restore backwards-compat for current builtins.fetchGit 
If a repo is dirty, it used to return a `rev` object with an "empty"
sha1 (0000000000000000000000000000000000000000). Please note that this
only applies for `builtins.fetchGit` and *not* for `builtins.fetchTree{
type = "git"; }`.
 2020-07-28 22:46:39 +02:00
Carlo Nucera 2980b244b7 Use assert(false) instead of abort() 2020-07-28 15:39:45 -04:00
Carlo Nucera cfe96bd7ce Merge branch 'master' of github.com:NixOS/nix into ca-drv 2020-07-28 15:36:45 -04:00
Carlo Nucera c318d398f3 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into new-interface-for-path-pathOpt 2020-07-28 14:22:24 -04:00
Carlo Nucera 7ef1e3cd14 Use the new interface 2020-07-28 13:59:24 -04:00
Carlo Nucera d564ac1c50 Offer a safer interface for pathOpt 
The new interface we offer provides a way of getting all the
DerivationOutputs with the storePaths directly, based on the observation
that it's the most common usecase.
 2020-07-28 13:55:15 -04:00
Eelco Dolstra b8eea7e81a Remove putBytes 
istream->tellg() returns -1 so we can't get the number of bytes
written.

Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.
 2020-07-28 13:27:56 +02:00
Maximilian Bosch f742438465
Merge legacy fetchGit-builtin with the generic fetchTree-function 
The original idea was to implement a git-fetcher in Nix's core that
supports content hashes[1]. In #3549[2] it has been suggested to
actually use `fetchTree` for this since it's a fairly generic wrapper
over the new fetcher-API[3] and already supports content-hashes.

This patch implements a new git-fetcher based on `fetchTree` by
incorporating the following changes:

* Removed the original `fetchGit`-implementation and replaced it with an
  alias on the `fetchTree` implementation.

* Ensured that the `git`-fetcher from `libfetchers` always computes a
  content-hash and returns an "empty" revision on dirty trees (the
  latter one is needed to retain backwards-compatibility).

* The hash-mismatch error in the fetcher-API exits with code 102 as it
  usually happens whenever a hash-mismatch is detected by Nix.

* Removed the `flakes`-feature-flag: I didn't see a reason why this API
  is so tightly coupled to the flakes-API and at least `fetchGit` should
  remain usable without any feature-flags.

* It's only possible to specify a `narHash` for a `git`-tree if either a
  `ref` or a `rev` is given[4].

* It's now possible to specify an URL without a protocol. If it's missing,
  `file://` is automatically added as it was the case in the original
  `fetchGit`-implementation.

[1] https://github.com/NixOS/nix/pull/3216
[2] https://github.com/NixOS/nix/pull/3549#issuecomment-625194383
[3] https://github.com/NixOS/nix/pull/3459
[4] https://github.com/NixOS/nix/pull/3216#issuecomment-553956703
 2020-07-28 00:44:39 +02:00
John Ericson 8065c6d160 Abstract out topo sorting logic 2020-07-27 20:45:34 +00:00
John Ericson 951415b568 Require ca-derivations everywhere we create a CA derivation 
"create" as in read one in from a serialized form, or build one from
scratch in memory.
 2020-07-27 17:56:36 +00:00
John Ericson e32a9e124b Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-27 17:50:06 +00:00
John Ericson 7cf978440c Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-27 17:47:40 +00:00
John Ericson e6115ca004 Merge remote-tracking branch 'upstream/master' into ca-derivation-data-types 2020-07-27 17:41:43 +00:00
John Ericson 1d7d94ceea Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-27 16:17:50 +00:00
John Ericson 43f2bd8dc5 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 16:13:57 +00:00
John Ericson 387699fe7f Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-27 15:00:28 +00:00
John Ericson 78466bcb2f Merge branch 'optional-derivation-output-storepath' into ca-derivation-data-types 2020-07-27 14:40:08 +00:00
John Ericson d5bb67cfa4 Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-27 14:29:32 +00:00
John Ericson 699fc89b39 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 14:22:45 +00:00
Maximilian Bosch 6ccfdb79c7
libutil/logging: extend internal-json logger to make it more machine-readable 
The new error-format is pretty nice from a UX point-of-view, however
it's fairly hard to parse the output e.g. for editor plugins such as
vim-ale[1] that use `nix-instantiate --parse` to determine syntax errors in
Nix expression files.

This patch extends the `internal-json` logger by adding the fields
`line`, `column` and `file` to easily locate an error in a file and the
field `raw_msg` which contains the error-message itself without
code-lines and additional helpers.

An exemplary output may look like this:

```
[nix-shell]$ ./inst/bin/nix-instantiate ~/test.nix --log-format minimal
{"action":"msg","column":1,"file":"/home/ma27/test.nix","level":0,"line":4,"raw_msg":"syntax error, unexpected IF, expecting $end","msg":"<full error-msg with code-lines etc>"}
```

[1] https://github.com/dense-analysis/ale
 2020-07-26 02:29:57 +02:00
John Ericson 2c7557481b queryDerivationOutputMap no longer assumes all outputs have a mapping 
This assumption is broken by CA derivations. Making a PR now to do the
breaking daemon change as soon as possible (if it is already too late,
we can bump protocol intead).
 2020-07-24 21:14:06 +00:00
Eelco Dolstra a71d1cedff
printVersion(): Show system types 2020-07-24 11:34:01 +02:00
Eelco Dolstra 2292814049
createUnixDomainSocket(): Fix off-by-one error in copying the socket path 
Reported by Kane York.
 2020-07-24 11:19:17 +02:00
John Ericson 58e55c0923 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-23 19:17:52 +00:00
John Ericson a2af25c55a Merge remote-tracking branch 'upstream/master' into misc-ca 2020-07-23 19:14:17 +00:00
John Ericson e1de1fe0d8 Make Buildable a std::variant 
I think this better captures the intent of what's going on: we either
have an opaque store path, or a drv path with some outputs.

Having this structure will also help us support CA derivations: we'll
have to allow the outpath paths to be optional, so the structure we gain
now makes up for the structure we loose then.
 2020-07-23 19:02:57 +00:00
John Ericson c36916eca0 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-23 15:06:56 +00:00
John Ericson 6fcd9966bf Merge remote-tracking branch 'upstream/master' into misc-ca 2020-07-23 15:02:29 +00:00
John Ericson 66a2067288 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-07-23 14:56:35 +00:00
John Ericson 2274f63453 Get rid of basicDerivation::findOutput 
It's a tiny function which is:

 - hardly worth abstrating over, and also only used once.

 - doesn't work once we get CA drvs

I rewrote the one callsite to be forwards compatable with CA
derivations, and also potentially more performant: instead of reading in
the derivation it can ust consult the SQLite DB in the common case.
 2020-07-23 14:39:02 +00:00
Fabian Möller b9ead08ca8
Save changes made by "nix registry pin" to user registry 2020-07-23 14:21:27 +02:00
John Ericson 9423f64ee2 Parse CA derivations using new output variants 
We no longer need `ParsedDerivation` because everything libstore needs
to know about is in the `BasicDerivation` proper.
 2020-07-22 23:59:25 +00:00
John Ericson c56356bacc Separate concerns in scanForReferences with TeeSink 
This also will make it easier to use a `HashModuloSink` instead for CA
derivations.
 2020-07-22 22:13:44 +00:00
Eelco Dolstra ff314f186e
Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option 
Allow relative paths in --store option
 2020-07-21 19:43:07 +02:00
Eelco Dolstra 0951330680
Merge pull request #3714 from obsidiansystems/add-body-to-network-errors 
Add response body to network errors
 2020-07-21 18:09:42 +02:00
John Ericson 02639716ea Merge branch 'allow-relative-paths-in-store-option' into remove-storetype-delegate-regStore 2020-07-21 15:55:27 +00:00
John Ericson 6cce32c8e8 Change logic for deciding what is a relative path for the local store 
The was Eelco's prefered logic, and it looks good to me!
 2020-07-21 15:39:47 +00:00
Carlo Nucera 922a845ffc Update chunkSize to the suggested value 
This was a suggested course of action in a review in one of our earlier
commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079
 2020-07-21 10:24:19 -04:00
John Ericson 54e507a7aa Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors 2020-07-21 14:17:59 +00:00
John Ericson 5ce95b9529
Update src/libstore/build.cc 2020-07-21 09:47:40 -04:00
John Ericson cdb3f39b64 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-07-21 13:43:20 +00:00
John Ericson 5055c595bd Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into ca-derivation-data-types 2020-07-21 01:20:53 +00:00
John Ericson 5a37413796 Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream 2020-07-21 01:20:14 +00:00
John Ericson bf0dca7790 Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into misc-ca 2020-07-21 01:18:33 +00:00
Carlo Nucera 9aae179f34 Correct bug, thoroughly document addToStoreSlow 2020-07-20 20:18:12 -04:00
Carlo Nucera 0ca9744694 Use heuristics to decide when to show the response 
Due to https://github.com/NixOS/nix/issues/3841 we don't know how print
different messages for different verbosity levels.
 2020-07-20 15:57:58 -04:00
Carlo Nucera 6357b1b0fb Add another Unimplemented case 2020-07-20 14:17:25 -04:00
Carlo Nucera 007bf6d17a Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-20 14:14:39 -04:00
Carlo Nucera 362ae93851 Add UnimplementedError to ease grepping for these 2020-07-20 14:13:37 -04:00
John Ericson c58c6165c5 Remove period at the end of the exception message 2020-07-20 17:43:19 +00:00
John Ericson bf61871271 parser.hh -> split.hh 2020-07-20 17:42:34 +00:00
John Ericson eab945f591 Merge remote-tracking branch 'upstream/master' into from-dump-stream 2020-07-20 17:29:23 +00:00
Eelco Dolstra 1c5f8bbfb5
Merge pull request #3822 from obsidiansystems/dump-thrice-fixme 
Optimize `addToStoreSlow` and remove `TeeParseSink`
 2020-07-20 18:55:05 +02:00
John Ericson ac2fc7ba1f
Apply suggestions from code review 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-07-20 11:29:46 -04:00
Carlo Nucera 0aa79dcc6f Remove StoreType abstraction and delegate regStore 
to each Store implementation. The generic regStore implementation will
only be for the ambiguous shorthands, like "" and "auto".

This also could get us close to simplifying the daemon command.
 2020-07-17 17:24:02 -04:00
John Ericson 6756cecfcf Add DerivationType::CAFloating 2020-07-17 19:55:41 +00:00
Carlo Nucera fefd6c9e5f Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option 2020-07-17 14:34:20 -04:00
Carlo Nucera bbc633c98c Revert "Don't anticipate CA but not fixed outputs for now" 
This reverts commit 3a9e4c3262.
 2020-07-17 13:10:32 -04:00
Carlo Nucera 205dcd140d Revert "Don't anticipate multiple CA outputs for now" 
This reverts commit 74b251b2f3.
 2020-07-17 12:43:46 -04:00
Carlo Nucera 1feb8981df Revert "Don't anticipate hash algo without hash in derivation for now" 
This reverts commit 3804e3df9b.
 2020-07-17 12:05:54 -04:00
Lars Mühmel bc73590151 nix edit: call restoreSignals() before execvp-ing the $EDITOR 
Currently resizing of the terminal doesn't play nicely with
nix edit when using kakoune as the editor, as it relies on the
SIGWINCH signal which is trapped by nix. How this is not a problem
with e.g. vim is beyond me.

Virtually all other exec* calls are following a call to
restoreSignals(). This commit adds this behavior to nix edit
as well.
 2020-07-17 17:35:59 +02:00
Carlo Nucera 487c5751c6 Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca 2020-07-17 11:32:59 -04:00
Carlo Nucera a395c12830 Merge branch 'master' of github.com:NixOS/nix into misc-ca 2020-07-17 11:06:54 -04:00
Carlo Nucera 5cb840541b Merge branch 'multi-output-hashDerivationModulo' of github.com:Ericson2314/nix into misc-ca 2020-07-17 10:28:33 -04:00
Eelco Dolstra 17f75f9cc4 parseFlakeRef(): Only search for the top-level directory for CLI flakerefs 2020-07-17 14:54:21 +02:00
Eelco Dolstra 52c8be38e0 nix profile diff-closures: Don't inherit EvalCommand 2020-07-17 12:36:12 +02:00
Carlo Nucera 745a03cef5 Merge branch 'optional-derivation-output-storepath' of github.com:obsidiansystems/nix into ca-derivation-data-types 2020-07-16 13:36:01 -04:00
Carlo Nucera 048e916f64 Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath 2020-07-16 13:32:28 -04:00
John Ericson cc0d77f8c9 Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-16 17:28:52 +00:00
Carlo Nucera e3a3406db8 Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors 2020-07-16 12:04:10 -04:00
Eelco Dolstra 16c9f6762d Add command 'nix profile diff-closure' 
This shows all changes between generations of a profile. E.g.

  $ nix profile diff-closures --profile /nix/var/nix/profiles/system
  Generation 654 -> 655:
    nix: 2.4pre20200617_5d69bbf → 2.4pre20200701_6ff9aa8, +42.2 KiB

  Generation 655 -> 656:
    blender-bin: 2.83.0 → 2.83.1, -294.2 KiB

  Generation 656 -> 657:
    curl: 7.68.0 → 7.70.0, +19.1 KiB
    firmware-linux-nonfree: 2020-01-22 → 2020-05-19, +30827.7 KiB
    ibus: -21.8 KiB
    initrd-linux: 5.4.46 → 5.4.49
    ...
 2020-07-16 17:00:42 +02:00
John Ericson 5ea817dace Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-16 14:58:53 +00:00
John Ericson 3dcca18c30 Fix bug in TeeSource 
We use this to simplify `LocalStore::addToStoreFromDump`.

Also, hope I fixed build error with old clang (used in Darwin CI).
 2020-07-16 13:39:27 +00:00
Carlo Nucera e3b394b6e8 Small namespace fix 2020-07-16 09:36:02 -04:00
Eelco Dolstra 5517eee17e Generations API cleanup 2020-07-16 15:18:21 +02:00
Eelco Dolstra 8807ff902e nix diff-closures: Fix build 2020-07-16 14:25:51 +02:00
Eelco Dolstra 716eafaca4 Merge branch 'diff-closures' 2020-07-16 14:19:35 +02:00
Eelco Dolstra 3f264916db Merge remote-tracking branch 'origin/flakes' 2020-07-16 14:07:32 +02:00
John Ericson 68dfb8c6ae Optimize addToStoreSlow and remove TeeParseSink 2020-07-16 05:09:41 +00:00
John Ericson 5602637d9e Revert "LocalStore::addToStoreFromDump copy in chunks" 
This reverts commit 592851fb67. We don't
need this extra feature anymore
 2020-07-15 23:37:49 +00:00
John Ericson bc109648c4 Get rid of LocalStore::addToStoreCommon 
I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking
a store and then doing nothing too fancy with it.

`LocalStore::addToStore(...Path...)` is now just a simple wrapper with a
bare-bones sinkToSource of the right dump command.
 2020-07-15 23:14:30 +00:00
Carlo Nucera 455bdee205 Merge branch 'master' of github.com:NixOS/nix into derivation-header-include-order 2020-07-15 17:58:30 -04:00
John Ericson 64b7421741 Merge branch 'handle-flat-case' into from-dump-stream 2020-07-15 21:22:55 +00:00
Carlo Nucera d090562348 Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-15 17:21:01 -04:00
John Ericson d087cf4855 Revert "Revert "LocalStore::addToStore(srcPath): Handle the flat case"" 
This reverts commit cff2157185.
 2020-07-15 21:10:33 +00:00
John Ericson aff69e229c Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream 2020-07-15 21:05:29 +00:00
Eelco Dolstra 36a1242603 nix why-depends: Fix shortest path calculation 
This was completely broken since d8972317fc.
 2020-07-15 21:11:07 +02:00
Eelco Dolstra 2d6d53bc87 nix: Fix examples 2020-07-15 20:28:16 +02:00
Eelco Dolstra 3624c042ac nix: Add --derivation flag to operate on .drv paths 
For instance, 'nix why-depends --use-derivation nixpkgs#hello
nixpkgs#glibc' shows why hello's .drv depends on glibc's .drv.
 2020-07-15 20:25:10 +02:00
Eelco Dolstra dfe8f3ebc6 nix why-depends: Fix misleading message 2020-07-15 20:09:50 +02:00
Eelco Dolstra 94eb5fad76 Clean up RealiseMode 2020-07-15 20:05:42 +02:00
Eelco Dolstra e3c2b00237 Make InstallableStorePath behave consistently with InstallableValue 
That is, the commands 'nix path-info nixpkgs#hello' and 'nix path-info
/nix/store/00ls0qi49qkqpqblmvz5s1ajl3gc63lr-hello-2.10.drv' now do the
same thing (i.e. build the derivation and operate on the output store
path, rather than the .drv path).
 2020-07-15 19:50:32 +02:00
Carlo Nucera 2d2a10e79a Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors 2020-07-15 13:06:08 -04:00
Eelco Dolstra 0356f14459 Add 'nix diff-closures' command 
This command makes it easier to see what changed between two closures,
i.e. what packages/versions got added or removed, and whether there
were any notable changes in path size.

For example:

  $ nix diff-closures /nix/var/nix/profiles/system-655-link /nix/var/nix/profiles/system-658-link
  blender-bin: 2.83.0 → 2.83.2, -294.2 KiB
  curl: 7.68.0 → 7.70.0, +19.1 KiB
  firmware-linux-nonfree: 2020-01-22 → 2020-05-19, +30827.7 KiB
  ibus: -21.8 KiB
  initrd-linux: 5.4.46 → 5.4.51, +16.9 KiB
  libexif: 0.6.21 → 0.6.22, +497.6 KiB
  linux: 5.4.46 → 5.4.51, +13.2 KiB
  mesa: 19.3.3 → 19.3.5, -183.9 KiB
  nix: 2.4pre20200701_6ff9aa8 → 2.4pre20200708_9223603, +9.7 KiB
  nix-bash-completions: 0.6.8 → ∅, -57.6 KiB
  nixos-system-hagbard: 20.03.20200615.a84b797 → 20.03.20200713.add5529
  nvidia-persistenced: 440.82 → 440.100
  nvidia-settings: 440.82 → 440.100
  nvidia-x11: 440.82-5.4.46 → 440.100-5.4.51, +664.7 KiB
  pcre: 8.43 → 8.44
  php: 7.3.16 → 7.3.20, -26.2 KiB
  python3.7-youtube-dl: 2020.06.06 → 2020.06.16.1, +8.4 KiB
  samba: 4.11.5 → 4.11.9, +30.1 KiB
  sane-backends: 1.0.28 → 1.0.30, +680.5 KiB
  source: -182.0 KiB
  zfs-kernel: 0.8.3-5.4.46 → 0.8.4-5.4.51, +9.9 KiB
  zfs-user: 0.8.3 → 0.8.4, +20.1 KiB
 2020-07-15 16:24:25 +02:00
Eelco Dolstra cff2157185 Revert "LocalStore::addToStore(srcPath): Handle the flat case" 
This reverts commit a2c27022e9. See
addToStoreSlow(), we don't need to handle this case efficiently
anymore. In fact, we can almost remove the method/hashAlgo arguments
since the non-recursive and/or non-SHA256 are almost not used anymore.
 2020-07-15 12:49:03 +02:00
John Ericson 650c2c6558 Rename variable nar -> dump according to TODO 2020-07-14 21:28:50 +00:00
John Ericson 8173e7bfef Fix localhost::addToStore(...Path...) 
We were calculating the nar hash wrong when the file ingestion method
was flat. I don't think there's anything we can do in that case but dump
the file again, so that's what I do.

As an optomization, we again could reuse the original dump for just the
recursive and non-sha256 case, but I rather do that after this fix, and
after my other PRs which deduplicate this code.
 2020-07-14 21:12:11 +00:00
John Ericson 592851fb67 LocalStore::addToStoreFromDump copy in chunks 
Rather than copying byte-by-byte, we let the coroutine know how much
data we would like it to send back to us.
 2020-07-14 13:56:36 +00:00
John Ericson 9de96ef7d4 Dedup LocalStore::addToStore* 
The downsides is that the coroutine has byte-by-byte loop transfer. Will
fix that next.
 2020-07-14 13:56:36 +00:00
John Ericson c86fc3a965 Crudely make addToStoreFromDump take Source not string 
I just as little beyond the type as possible, so the implementation
changes this enables can be reviewed separately.
 2020-07-14 13:56:36 +00:00
John Ericson 9ec10046e0 Narrow scope of temporary value 2020-07-14 13:54:29 +00:00
Eelco Dolstra 298ff6af8f
Merge pull request #3809 from Ma27/gitlab-refs 
Fix gitlab-fetcher to obtain tags and branches
 2020-07-14 15:22:52 +02:00
Eelco Dolstra da3aea291d EvalCache: Ignore SQLite errors 
Fixes #3794.
 2020-07-14 15:17:38 +02:00
Eelco Dolstra 832e111494 Merge remote-tracking branch 'origin/master' into flakes 2020-07-14 13:56:18 +02:00
Eelco Dolstra 926c3a6664 Doh 2020-07-14 11:55:54 +02:00
Eelco Dolstra 43b8e96d30 Fix 'nix verify --all' on a binary cache (cached case) 2020-07-13 20:17:00 +02:00
Eelco Dolstra 9502c0e2eb nix verify: Show correct path when using --all on a binary cache 2020-07-13 20:12:44 +02:00
Eelco Dolstra 7c2fef0a81 Make 'nix copy' to s3:// binary caches run in constant memory 2020-07-13 20:07:19 +02:00
Maximilian Bosch cf9f33995b
Fix gitlab-fetcher to obtain tags and branches 
Until now, the `gitlab`-fetcher determined the source's rev by checking
the latest commit of the given `ref` using the
`/repository/branches`-API.

This breaks however when trying to fetch a gitlab-repo by its tag:

```
$ nix repl
nix-repl> builtins.fetchTree gitlab:Ma27/nvim.nix/0.2.0
error: --- Error ------------------------------------------------------------------------------------- nix
unable to download 'https://gitlab.com/api/v4/projects/Ma27%2Fnvim.nix/repository/branches/0.2.0': HTTP error 404 ('')
```

When using the `/commits?ref_name`-endpoint[1] you can pass any kind of
valid ref to the `gitlab`-fetcher.

Please note that this fetches the only first 20 commits on a ref,
unfortunately there's currently no endpoint which only retrieves the
latest commit of any kind of `ref`.

[1] https://docs.gitlab.com/ee/api/commits.html#list-repository-commits
 2020-07-13 19:22:59 +02:00
Eelco Dolstra 493961b689 Remove istringstream_nocopy 2020-07-13 18:31:19 +02:00
Eelco Dolstra 545bb2ed03 Remove 'accessor' from addToStore() 
This is only used by hydra-queue-runner and it's better to implement
it there.
 2020-07-13 18:31:19 +02:00
Eelco Dolstra 0a9da00a10 NarAccessor: Run in constant memory 2020-07-13 17:30:42 +02:00
Eelco Dolstra fc84c358d9 Make 'nix copy' to file:// binary caches run in constant memory 2020-07-13 16:28:45 +02:00
Eelco Dolstra 400f1a9b59 Store::pathInfoToJSON(): Use consistent format for downloadHash 2020-07-13 16:26:09 +02:00
Eelco Dolstra c0dd05131e toStorePath(): Return a StorePath and the suffix 2020-07-13 16:25:48 +02:00
Eelco Dolstra 1d01ae816b Fix 'nix verify --all' on a binary cache and add a test 2020-07-13 14:35:01 +02:00
John Ericson c466cb2091 Merge branch 'hash-always-has-type' into better-ca-parse-errors 2020-07-13 03:02:09 +00:00
John Ericson 4415765385 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-13 03:01:44 +00:00
John Ericson 230c9b4329 Change types to prepare the way for CA derivations 
We've added the variant to `DerivationOutput` to support them, but made
`DerivationOutput::path` partial to avoid actually implementing them.

With this chage, we can all collaborate on "just" removing
`DerivationOutput::path` calls to implement CA derivations.
 2020-07-12 23:53:33 +00:00
John Ericson fedfc913ad Use more std::visit to prepare for new variant 
N.B. not using `std::visit` for fetchurl because there is no attempt to
handle all the cases (e.g. no `else`) and lambda complicates early
return.
 2020-07-12 23:51:40 +00:00
John Ericson ffc18583b1 Move C++17 "pattern matching" boilerplat to utils.hh 2020-07-12 22:15:14 +00:00
John Ericson 503b425690 DerivationOutputExtensional -> DerivationOutputInputAddressed 
Thanks @regnat for the great name.
 2020-07-12 15:56:20 +00:00
John Ericson a8d4707107 Undo erroneous indentation change 2020-07-12 15:54:12 +00:00
John Ericson 18152406ce String .drv suffix to create derivation name 2020-07-12 15:40:14 +00:00
John Ericson 5d0b75e5b6 std::string_view for new derivation name parameters 2020-07-12 15:02:36 +00:00
Maximilian Bosch 64f03635d7
Fix ANSI color constants 
The `m` acts as termination-symbol when declaring graphics. Because
of this, the `;1m` doesn't have any effect and is directly printed to
the console:

```
$ nix repl
> builtins.fetchGit { /* ... */ }
{ outPath = "/nix/store/s0f0iz4a41cxx2h055lmh6p2d5k5bc6r-source"; rev = "e73e45b723a9a6eecb98bd5f3df395d9ab3633b6"; revCount = ;1m428; shortRev = "e73e45b"; submodules = ;1mfalse; }
```

Introduced by 6403508f5a.
 2020-07-12 16:52:20 +02:00
John Ericson 13ec627e0a Set derivation name in dervationStrict 2020-07-12 03:03:12 +00:00
John Ericson 1c9bec226f Don't improperly assume path is store path 2020-07-12 02:38:03 +00:00
John Ericson abea26a968 BasicDerivation::findOutput cannot return reference anymore 2020-07-12 01:57:06 +00:00
John Ericson 442d43178b Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-11 02:27:45 +00:00
Matthew Bauer acb74d4d94 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-10 18:13:45 -04:00
Eelco Dolstra 8efa23bb99 Avoid a redundant hash 2020-07-10 15:56:24 +02:00
Eelco Dolstra 5dff49f661 Factor out commonality between nix-prefetch-url and nix-store --add-fixed 2020-07-10 13:21:37 +02:00
Eelco Dolstra 7f1a86d57c nix-store --add-fixed: Run in constant memory 2020-07-10 12:51:56 +02:00
Eelco Dolstra 06e3dd9005 nix-prefetch-url: Run in constant memory when using RemoteStore 
Fixes #3684.
 2020-07-10 11:22:48 +02:00
John Ericson 323d51a7a0 Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-09 19:07:06 +00:00
Matthew Bauer a7884970c5 Fix DerivationOutputExtensional name 2020-07-09 11:37:18 -04:00
Eelco Dolstra a2c27022e9 LocalStore::addToStore(srcPath): Handle the flat case 
This helps nix-prefetch-url when using a local store.
 2020-07-09 15:54:32 +02:00
Matthew Bauer 06a4e15478 Fix build.cc on linux 2020-07-08 19:27:51 -04:00
Matthew Bauer 8e0d0689be Only store hash of fixed derivation output 
we don’t need a full storepath for a fixedoutput derivation. So just
putting the ingestion method + the hash is sufficient.
 2020-07-08 19:11:39 -04:00
Eelco Dolstra b981e5aacf Cleanup 2020-07-08 22:07:21 +02:00
Eelco Dolstra 34f25124ba Make LocalStore::addToStore(srcPath) run in constant memory 
This reduces memory consumption of

  nix-instantiate \
    -E 'with import <nixpkgs> {}; runCommand "foo" { src = ./blender; } "echo foo"' \
    --option nar-buffer-size 10000

(where ./blender is a 1.1 GiB tree) from 1716 to 36 MiB, while still
ensuring that we don't do any write I/O for small source paths (up to
'nar-buffer-size' bytes). The downside is that large paths are now
always written to a temporary location in the store, even if they
produce an already valid store path. Thus, adding large paths might be
slower and run out of disk space. ¯\_(ツ)_/¯ Of course, you can always
restore the old behaviour by setting 'nar-buffer-size' to a very high
value.
 2020-07-08 22:07:21 +02:00
Matthew Bauer af95a7c16b Add name to BasicDerivation 
We always have a name for BasicDerivation, since we have a derivation
store path that has a name.
 2020-07-08 15:38:01 -04:00
Eelco Dolstra 9223603908 Merge remote-tracking branch 'origin/master' into flakes 2020-07-08 15:55:19 +02:00
Eelco Dolstra 16ec7785ca Fix 'got unknown message type 1 from Nix daemon' 
Example:

  $ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" { x = runCommand "bar" {} "exit 1"; } "echo foo; exit 1"'
  warning: unknown setting 'auto-allocate-uids'
  these 2 derivations will be built:
    /nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv
    /nix/store/k4fsvrjl7cp2xpz7927iv7g0dqj1zyhs-foo.drv
  warning: unknown setting 'auto-allocate-uids'
  building '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv'...
  error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-daemon
  builder for '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv' failed with exit code 1
  error: --- Error ------------------------------------------------------------------------------------------------------------------------------------------------------------------ nix-build
  got unknown message type 1 from Nix daemon
 2020-07-08 15:53:14 +02:00
Eelco Dolstra 1ab9da9154 Merge remote-tracking branch 'origin/master' into flakes 2020-07-07 14:38:57 +02:00
Eelco Dolstra 4055cfee36 Fix coverage build 2020-07-07 14:37:47 +02:00
Eelco Dolstra 7c9ece5dca exportReferencesGraph: Fix support for non-top-level store paths 
Fixes #3471.
 2020-07-07 14:25:43 +02:00
Carlo Nucera 0f05a36e20 Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-06 16:57:09 -04:00
Ben Burdette efd6a8b230 bump 2020-07-06 11:54:53 -06:00
Ben Burdette 75bfcf8d15 revamp trace code and test 2020-07-06 10:51:48 -06:00
Eelco Dolstra 68f524d717 nix develop: Support derivations with multiple outputs 2020-07-06 18:34:58 +02:00
Eelco Dolstra cd8eb8a7d1 nix develop: Fall back to "bash" if nixpkgs#bashInteractive is unavailable 2020-07-06 17:08:54 +02:00
Eelco Dolstra 54712aaf8a Merge remote-tracking branch 'origin/master' into flakes 2020-07-06 16:40:10 +02:00
John Ericson a38ab99d57 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-07-05 21:49:01 +00:00
Ben Burdette a168224464 spacing 2020-07-04 18:30:49 -06:00
John Ericson 465daa9396 Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors 2020-07-03 17:08:39 +00:00
John Ericson 3134db1a83 Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-03 14:12:38 +00:00
John Ericson dbffd309fe Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-03 14:11:38 +00:00
Ben Burdette b29a4ea1dc Merge branch 'master' into add-trace 2020-07-03 07:57:36 -06:00
Eelco Dolstra c3c7aedbb5 nix develop: Fix bad regex 
This was accepted by libstdc++ but not libc++.

https://hydra.nixos.org/build/123569154
 2020-07-03 14:58:58 +02:00
Eelco Dolstra 6f8fd3a3f2 Shut up a clang warning 2020-07-03 14:50:07 +02:00
regnat 223fbe644a Shorten the path to the test root 
Fix a socket length failure on the OSX builders
 2020-07-03 09:20:01 +02:00
John Ericson 13796be78d Have splitPrefix and splitPrefixTo parser helpers 2020-07-02 23:18:22 +00:00
John Ericson a7cd7425d9 Move getParsedTypeAndSRI to a more suitable location 
Also mark it static
 2020-07-02 23:10:11 +00:00
John Ericson 2f93d9f2ba Merge branch 'hash-always-has-type' into HEAD 2020-07-02 21:47:51 +00:00
John Ericson 1be279af26 Fix Narinfo corruption detection bug 
The aim of this check was just to ensure each key occurs once.
 2020-07-02 21:46:10 +00:00
Eelco Dolstra b5e4253697 Fix abort in 'nix develop' 2020-07-02 18:24:11 +02:00
Carlo Nucera 1fc835aa22 Tighten parsing for drv files and pathinfo 2020-07-02 11:57:21 -04:00
Ben Burdette 5818271c6e spacing 2020-07-02 09:41:54 -06:00
Carlo Nucera b6b10b1d4c Write the implementation for parseNonSRIUnprefixed 2020-07-02 11:34:40 -04:00
Carlo Nucera ea48e3a5b5 Abstract common parsing functionality 2020-07-02 11:29:33 -04:00
Carlo Nucera 36cbc74689 Inline and simplify in parseAnyPrefixed 2020-07-02 11:21:00 -04:00
Matthew Bauer fc2ab42e86 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-02 11:14:04 -04:00
Matthew Bauer d2e8b9ff0e Store subPath in SubstitutionGoal 2020-07-02 11:12:05 -04:00
Carlo Nucera 9462d8a50b Rename fromSRI to parseSRI for constistency 2020-07-02 11:11:18 -04:00
Carlo Nucera f61bc45d19 Get rid of the std::pair 2020-07-02 11:09:04 -04:00
Ben Burdette bf2788e4c1 move showTrace to new loggerSettings 2020-07-02 09:04:31 -06:00
Carlo Nucera 27c8029573 Inline newFunction 2020-07-02 11:01:10 -04:00
Matthew Bauer 1f9cb06db2 Try next when no ca exists and have different store dirs 2020-07-02 10:59:24 -04:00
Carlo Nucera 343d1569b1 Fix test suite 2020-07-02 10:48:47 -04:00
Ben Burdette 5ae498872a assert for invalid fileorigin 2020-07-02 07:14:40 -06:00
Carlo Nucera 263ccdd489 Rename two hash constructors to proper functions 2020-07-01 18:34:18 -04:00
Carlo Nucera c8c4bcf90e Inline Hash::init() 2020-07-01 18:03:35 -04:00
Carlo Nucera d63a5ded76 Remove unused import 2020-07-01 17:53:24 -04:00
Carlo Nucera 6faeec3b2a Keep the previous name, for diffing 2020-07-01 17:50:34 -04:00
Carlo Nucera 274a8136fb Correct FIXMEs in libfetchers 2020-07-01 17:47:15 -04:00
Carlo Nucera c2e7f7a712 Fixed build, we still have test errors 2020-07-01 17:32:06 -04:00
Ben Burdette 8497891b99 spacing 2020-07-01 13:50:18 -06:00
Eelco Dolstra 6ff9aa8df7 Don't process an option if any of its arguments need completion 2020-07-01 20:31:39 +02:00
Eelco Dolstra d746503e5c Add --inputs-from to use flake inputs as registry entries 
This allows you to refer to an input from another flake. For example,

  $ nix run --inputs-from /path/to/hydra nixpkgs#hello

runs 'hello' from the 'nixpkgs' inputs of the 'hydra' flake.

Fixes #3769.
 2020-07-01 20:25:13 +02:00
Ben Burdette a295b2ea96 if no errLoc, no Loc. 2020-07-01 12:02:02 -06:00
Ben Burdette 3629b0585a don't include errpos for addErrorContext 2020-07-01 11:49:01 -06:00
Ben Burdette 2a39c083dc non-pos trace test 2020-07-01 10:37:31 -06:00
Eelco Dolstra 38ccf2e241 Cleanup 2020-07-01 15:31:34 +02:00
Eelco Dolstra 86a4aba6c4 Merge branch 'remote-query-outputs' of https://github.com/tweag/nix 2020-07-01 15:10:29 +02:00
Eelco Dolstra 7d554f295c Support building flakes from a shallow Git repo 
Fixes #3756.
 2020-07-01 14:57:59 +02:00
Ben Burdette a7d5d26443 fix tests with the 'from string' change 2020-06-30 22:05:21 -06:00
Ben Burdette 9159dfe3d8 comments and cleanup 2020-06-30 16:31:55 -06:00
Ben Burdette 70bcb39d3f double addtrace for 'called from' 2020-06-30 15:44:19 -06:00
Matthew Bauer 09833126f3 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-30 16:35:08 -04:00
Ben Burdette ddb81ca126 Merge branch 'master' into add-trace 2020-06-30 12:21:45 -06:00
Carlo Nucera b798efb829 WIP initial design 2020-06-30 14:10:30 -04:00
Ben Burdette a0705e0dd1 invalid pos check 2020-06-30 11:01:46 -06:00
Ben Burdette e72a16a339 check for a null symbol 2020-06-30 11:00:51 -06:00
Carlo Nucera a1f66d1d9e Factor the prefix splitting in hash 2020-06-30 12:49:00 -04:00
Carlo Nucera 77b51f4598 Factor the prefix splitting in content-address 2020-06-30 11:57:46 -04:00
Carlo Nucera 7ba0fae0dd Create the spitPrefix function in parser.hh 2020-06-30 11:57:09 -04:00
Ben Burdette c484a67914 trace formatting 2020-06-29 15:46:21 -06:00
John Ericson 24da034bc3 Add possibly missing <string> include 2020-06-29 21:22:00 +00:00
Eelco Dolstra 2b834d48aa NAR parser: Fix missing name field check 
Discovered by @Kloenk.
 2020-06-29 22:45:41 +02:00
Matthew Bauer 289558dffb Add unordered_set to globals.cc header 2020-06-29 20:34:45 +00:00
Matthew Bauer da77331cb7 Remove lazy lookup in getHome 
this seems to break in Musl/Static with:

terminate called after throwing an instance of 'std::bad_function_call'
  what():  bad_function_call
 2020-06-29 20:34:26 +00:00
Matthew Bauer 88cf6ffce3 Rename logging->stdout to logging->stdout_ 
musl doesn't like this identifier
 2020-06-29 20:34:26 +00:00
John Ericson a83566e5bc Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-06-29 18:40:34 +00:00
John Ericson 987a4a0be9 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-06-29 18:03:21 +00:00
Eelco Dolstra 26cf0c674f nix run: Use packages/legacyPackages as fallback if there is no app definition 
'nix run' will try to run $out/bin/<name>, where <name> is the
derivation name (excluding the version). This often works well:

  $ nix run nixpkgs#hello
  Hello, world!

  $ nix run nix -- --version
  nix (Nix) 2.4pre20200626_adf2fbb

  $ nix run patchelf -- --version
  patchelf 0.11.20200623.e61654b

  $ nix run nixpkgs#firefox -- --version
  Mozilla Firefox 77.0.1

  $ nix run nixpkgs#gimp -- --version
  GNU Image Manipulation Program version 2.10.14

though not always:

  $ nix run nixpkgs#git
  error: unable to execute '/nix/store/kp7wp760l4gryq9s36x481b2x4rfklcy-git-2.25.4/bin/git-minimal': No such file or directory
 2020-06-29 19:08:50 +02:00
Eelco Dolstra 50f13b06fb EvalCache: Store string contexts 2020-06-29 19:08:37 +02:00
Ben Burdette 8f81fae116 showTrace flag in loggers 2020-06-29 10:20:51 -06:00
Eelco Dolstra b681408879 Factor out EvalCache::forceDerivation() 2020-06-29 16:39:41 +02:00
Eelco Dolstra ca946860ce Fix bash completion 2020-06-29 14:37:22 +02:00
Eelco Dolstra bc03c6f23d Move App 2020-06-29 14:14:23 +02:00
Eelco Dolstra 58bc3b6578
Merge pull request #3729 from obsidiansystems/simpler-hased-mirror 
hashed-mirrors: Use parsed derivation output rather than reconstructing it
 2020-06-29 14:04:12 +02:00
Eelco Dolstra 64232f3ea6
Merge pull request #3749 from rodarima/master 
Fall back to copyPath if link fails with EPERM
 2020-06-29 13:31:24 +02:00
Ben Burdette ef24a0835d showtrace as function arg 2020-06-27 12:19:31 -06:00
Carlo Nucera 278a679782 Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option 2020-06-26 10:40:13 -04:00
Eelco Dolstra adf2fbbdc2 Merge remote-tracking branch 'origin/master' into flakes 2020-06-26 08:46:46 +02:00
John Ericson d92d4f85a5 Move ValidPathInfo to its own header 2020-06-25 17:48:49 +00:00
Ben Burdette bc9e87412c 'string' makes more sense in nix repl 2020-06-25 09:56:32 -06:00
Ben Burdette 9ab808c926 showTrace flag for ErrorInfo; showTrace test. 2020-06-25 09:23:12 -06:00
Eelco Dolstra de2641ae99 Fix empty std::optional dereference in writeDerivation() 
https://hydra.nixos.org/build/123017579
 2020-06-25 15:50:30 +02:00
Rodrigo 3a642187c3 Fall back to copyPath if link fails with EPERM 
BeeGFS doesn't allow hard-links and returns EPERM, so we fall back
to copyPath. See https://github.com/NixOS/nix/issues/3748
 2020-06-25 12:03:26 +02:00
Ben Burdette 9c0e1fd4f1 add trace test; error formatting refinements 2020-06-24 18:31:28 -06:00
Ben Burdette 6359d71d6b re-enable --show-trace check 2020-06-24 18:28:20 -06:00
Ben Burdette 023912def3 convenience form of addTrace 2020-06-24 13:46:25 -06:00
Ben Burdette 93e9307329 repl indenting 2020-06-24 13:14:49 -06:00
Ben Burdette b18ed02b76 repl indenting 2020-06-24 13:10:41 -06:00
regnat d38f860c3e Add a way to get all the outputs of a derivation with their label 
Generalize `queryDerivationOutputNames` and `queryDerivationOutputs` by
adding a `queryDerivationOutputMap` that returns the map
`outputName=>outputPath`

(not that this is not equivalent to merging the results of
`queryDerivationOutputs` and `queryDerivationOutputNames` as sets don't
preserve the order, so we would end up with an incorrect mapping).

squash! Add a way to get all the outputs of a derivation with their label

Rename StorePathMap to OutputPathMap
 2020-06-24 20:38:40 +02:00
Ben Burdette 6fe660acf9 re-remove 2020-06-24 12:33:05 -06:00
Matthew Bauer bcf8582091 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-24 12:52:09 -04:00
John Ericson d9fb9e9f26 Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-24 15:41:29 +00:00
Ben Burdette 00fe653ea5 nixCode -> LinesOfCode 2020-06-24 08:33:53 -06:00
Ben Burdette 1d43a6e123 use plain errPos instead of nixCode; fix tests 2020-06-23 15:30:13 -06:00
Carlo Nucera 8d51d38e4c Fix test suite 2020-06-23 17:16:20 -04:00
John Ericson ec3a857307 Fix and clean up hash parser 2020-06-23 18:19:40 +00:00
John Ericson 98e5d1af03 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-06-23 17:03:37 +00:00
Ben Burdette d0e78fbb03 re-add Pos origin in tests 2020-06-23 10:51:58 -06:00
Ben Burdette abe0552504 Merge remote-tracking branch 'upstream/master' into add-trace 2020-06-23 09:40:28 -06:00
Ben Burdette 13e87535ff traces to bottom 2020-06-23 09:36:58 -06:00
Carlo Nucera e197bc6229 Enable the --store option to take relative paths 
In nix commands which accept --store options, we can now specify a
relative path, which will be canonicalized.
 2020-06-23 11:13:18 -04:00
Eelco Dolstra 09fc06daab nix flake init: Use git add --force 2020-06-23 16:25:32 +02:00
Eelco Dolstra 015e1c2131
Merge pull request #3724 from bburdette/hintfmt-percent 
Hintfmt percent test, and fix
 2020-06-23 12:26:00 +02:00
John Ericson 55d4bd6e0e Improve content address parsing 
- Ensure hash is in form <algo>-<prefix> and not SRI.

 - Better errors if something goes wrong

 - string_view for no coppying
 2020-06-22 18:08:27 +00:00
Ben Burdette 9d1cb0c5e6 with normaltxt, elide yellow color code instead of canceling it; use normaltxt on plain_string hintfmt 2020-06-22 11:32:20 -06:00
Matthew Bauer 66a62b3189 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-22 13:08:11 -04:00
Ben Burdette 28b079067f
Update src/libutil/fmt.hh 
Co-authored-by: John Ericson <git@JohnEricson.me>
 2020-06-22 10:00:37 -06:00
John Ericson f4a5913125 hashed-mirrors: Use parsed derivation output rather than reconstructing it 
Now the derivation outputs are parsed up front, we can avoid a reparse
by doing it. Also, this just feels a bit better as the `output*` env
vars are more of a `libnixexpr` interface than `libnixstore` interface:
ultimately, it's the derivation outputs that decide whether the
derivation is fixed-output.

Yes, hashed mirrors might go away with #3689, but this bit of code would
be moved rather than deleted, so it's worth doing a cleanup anyways I
think.
 2020-06-22 15:17:20 +00:00
Eelco Dolstra 334e26bfc2 nix flake check: Don't build apps 
This was inconsistent since we're not building 'packages' or
'defaultPackage' either.

Closes #3726.
 2020-06-22 11:31:07 +02:00
John Ericson 93bbe6e8ab Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-21 21:11:58 +00:00
John Ericson 3804e3df9b Don't anticipate hash algo without hash in derivation for now 
When we merge with master, the new lack of string types make this case
impossible (after parsing). Later, when we actually implemenent
CA-derivations, we'll change the types to allow that.
 2020-06-21 21:05:37 +00:00
John Ericson 8313f0e939 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-06-21 20:39:10 +00:00
John Ericson bf9f040112 Tweak declaration 
I think this is clearer
 2020-06-21 16:51:39 +00:00
John Ericson fdeabf7160 Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo 2020-06-21 16:43:17 +00:00
Ben Burdette be4f444175 tidying up 2020-06-19 16:58:12 -06:00
Ben Burdette 0309488a66 fmt -> hintfmt test 2020-06-19 16:46:49 -06:00
Matthew Bauer f2a6cee334 Update worker protocol to support sending storepath maps 
We need to also send the ca to daemon in addition to the path.
 2020-06-19 18:06:19 -04:00
Ben Burdette 397dbe114e remove formathelper 2020-06-19 15:57:19 -06:00
John Ericson 507aa48739 WIP: Make Hash always store a valid hash type 2020-06-19 21:48:57 +00:00
Ben Burdette b193aca4ae escape percents 2020-06-19 15:29:19 -06:00
Ben Burdette db475f9e7e too few, too many args 2020-06-19 15:28:13 -06:00
Ben Burdette cdddf24f25 add hintfmt test 2020-06-19 14:54:41 -06:00
Ben Burdette 54e8f550c9 addErrorTrace 2020-06-19 13:44:08 -06:00
Matthew Bauer e8e1f5282f Replace error message in getDerivationCA 2020-06-19 15:19:09 -04:00
Matthew Bauer 0c9c1b8826 Return map of StorePaths in copyPaths 
This allows the caller to know what values were actually added to the
store.
 2020-06-19 14:48:57 -04:00
John Ericson e288c0987a Merge remote-tracking branch 'upstream/master' into validPathInfo-ca-proper-datatype 2020-06-19 18:44:24 +00:00
Matthew Bauer 7e11cf3399 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-19 14:40:49 -04:00
Matthew Bauer 2796b01e86 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-19 14:22:06 -04:00
John Ericson 68294746ae Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown 2020-06-19 17:53:34 +00:00
John Ericson c98081d270 Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown 2020-06-19 17:50:05 +00:00
John Ericson c1892a5316 tabs -> spaces 2020-06-19 17:49:57 +00:00
John Ericson 911fc88bcb More designated initializers 2020-06-19 17:42:56 +00:00
John Ericson 2f0e395c99 Merge remote-tracking branch 'me/no-stringly-typed-derivation-output' into validPathInfo-ca-proper-datatype 2020-06-19 15:26:59 +00:00
John Ericson fb39a5e00c Remove unneeded constructor for DerivationOutputHash 2020-06-19 15:11:11 +00:00
John Ericson b90cac3bad Remove uneeded = default for Hash 2020-06-19 15:00:38 +00:00
John Ericson 01dc8b0bab Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-19 14:59:05 +00:00
John Ericson 145d88cb2a Use designated initializers for DerivationOutputHash 2020-06-19 14:58:30 +00:00
John Ericson 237d88c97e FileSystemHash -> DerivationOutputHash 2020-06-19 14:47:10 +00:00
Eelco Dolstra 2886c92aef
Merge pull request #3669 from gilligan/add-compression-tests 
Add compression unit tests
 2020-06-19 13:59:04 +02:00
John Ericson 3fc58a9638 Remove some Base:: that crept in 2020-06-19 00:24:47 +00:00
John Ericson 3f8dcfe3fd Merge branch 'validPathInfo-temp' into validPathInfo-ca-proper-datatype 2020-06-18 23:01:58 +00:00
John Ericson 669c3992e8 Merge branch 'no-hash-type-unknown' into validPathInfo-temp 2020-06-18 22:33:07 +00:00
John Ericson 15abb2aa2b Revert the enum struct change 
Not a regular git revert as there have been many merges and things.
 2020-06-18 22:11:26 +00:00
John Ericson bbbf3602a3 Merge branch 'enum-class' into no-hash-type-unknown 2020-06-18 22:11:19 +00:00
John Ericson 40526fbea5 Merge remote-tracking branch 'upstream/master' into enum-class 2020-06-18 21:38:15 +00:00
Ben Burdette 4d1a4f0217 addTrace 2020-06-18 15:25:26 -06:00
Ben Burdette e6f93b94fc Merge branch 'master' into caveman-LOCs 2020-06-18 13:07:53 -06:00
John Ericson 1b23fe4afb Fix bugs 
- Bad dynamic cast target ...classic

 - std::shared_ptr need explicit deref
 2020-06-18 19:03:10 +00:00
John Ericson fb432b0bd2 Merge remote-tracking branch 'obsidian/fix-url-format' into add-body-to-network-errors 2020-06-18 18:08:02 +00:00
John Ericson 639e20dc3e Prevent '%' in URL from causing crashes 
We have a larger problem that passsing computed strings to the first
variable argument of many exception constructors is unsafe because that
first variable argument is interpreted not as a plain string, but format
string, and if it contains '%' boost::format will abort, since there are
no arguments to the format string.

In this particular instance '%' was used as part of an escape code in a
URL, which, when the download failed, caused Nix to abort displaying the
`FileTransferError`.
 2020-06-18 17:54:16 +00:00
Eelco Dolstra 6c000eed80
Merge pull request #3709 from expipiplus1/master 
Mention number of derivations to be build/fetched in output
 2020-06-18 19:03:05 +02:00
John Ericson 74b219ef6e Adjust FileTransferError message to use opt response 2020-06-18 14:48:45 +00:00
Eelco Dolstra 5771c8bbf2 Don't provide 'getFlake' if the 'flakes' feature is not enabled 
(cherry picked from commit 0a1d3c1dd3)
 2020-06-18 14:03:00 +02:00
Eelco Dolstra 2a61bbf77f Some backports from the flakes branch 2020-06-18 14:03:00 +02:00
Eelco Dolstra 377345e26f Remove unneeded #include 2020-06-18 13:47:05 +02:00
Eelco Dolstra 7083d33efe Make constant primops lazy 
(cherry picked from commit aa0e2a2e70)
 2020-06-18 13:42:47 +02:00
Eelco Dolstra 3d492199bb github: Respect default branch 2020-06-18 13:25:08 +02:00
regnat 4fef2ba7e4 Rename content-addressed-paths into ca-derivations 
See <https://github.com/NixOS/nix/pull/3710#issuecomment-645480333>
 2020-06-18 09:25:55 +02:00
Carlo Nucera 004570a377 Add HTTP responses to FileTransferErrors 2020-06-17 22:34:31 +00:00
Carlo Nucera a835c740ca Replace TransferItem::status with a local variable 
Everywhere seems to use `getHTTPStatus` now.
 2020-06-17 22:16:24 +00:00
Carlo Nucera 289b9b8dcf Create a new TeeSink abstraction 
This is a bit complex because we want to expose extra functionality the
wrapped class has. Perhaps there is some inheritancy trickery to do this
nicer, but I don't know it, and this is the first thing we tried after a
series of attempts that did build.

This design is kind of like that of Rust's Writer, Reader, or Iter
adapters, which impliment more traits based on what the inner type
implements.
 2020-06-17 22:09:32 +00:00
Carlo Nucera fb05a6adcf Eliminate old TeeSink abstraction 
This was introduced in fa125b9b28, and
then "reverted" in 1cf4801108, except that
revert left the struct around doing nothing useful.

We're removing it all the way now because we want to make a new
`TeeSink` complementing the already-exiting `TeeSource`, that is
actually a completely different concept as far as the class hierarchy is
concerned.
 2020-06-17 22:07:22 +00:00
Eelco Dolstra 2b8f33bf5f
Merge pull request #3713 from matthewbauer/cleanup-warnings 
Cleanup class StorePath warning
 2020-06-17 21:19:30 +02:00
Matthew Bauer 5e631e3304 Add StorePathCAMap for querySubstitutablePathInfos 
I’m not 100% sure this is wanted since it kind of makes everything
have to know about ca even if they don’t really want to. But it also
make things easier in dealing with looking up ca.
 2020-06-17 15:03:05 -04:00
Matthew Bauer be50de1142 Make sure references are empty for store path replacing 
also copy info2 instead of casting
 2020-06-17 14:15:08 -04:00
Matthew Bauer 8974755d19 Add assert for replaced storePath 2020-06-17 14:04:46 -04:00
Eelco Dolstra 3078404e35
Merge pull request #3712 from obsidiansystems/make-http-successful-states-coherent 
Make successful states coherent
 2020-06-17 19:37:46 +02:00
Matthew Bauer 7bd88cc1dc Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-17 13:32:29 -04:00
Matthew Bauer 22d7d36703 Remove unused narInfoFile in binary-cache-store 2020-06-17 13:27:19 -04:00
Matthew Bauer f767bedfac Replace struct StorePath with class StorePath 
also a similar case with struct Goal
 2020-06-17 13:26:37 -04:00
Carlo Nucera 4930cb48a2 Include review comments 2020-06-17 12:58:59 -04:00
Matthew Bauer 6438ba1e99 Update strings from review comment 2020-06-17 12:31:01 -04:00
Eelco Dolstra 2f51cd8dc9
Merge pull request #3710 from tweag/reserve_ca_derivations 
Reserve the `__contentAddressed` derivation parameter
 2020-06-17 18:28:26 +02:00
Eelco Dolstra 4d5169bdd5
Merge pull request #3707 from p01arst0rm/outdated-function-fix 
replaced uncaught_exception with uncaught_exceptions
 2020-06-17 18:26:01 +02:00
Eelco Dolstra 2e4bd78211 nix eval: Add --apply flag for post-processing the result 2020-06-17 18:12:24 +02:00
regnat 480b54e1c6 fixup! Reserve the __contentAddressed derivation parameter 2020-06-17 17:37:04 +02:00
Carlo Nucera 079c6e87de Make successful states coherent 
The successful states used in these two places in the code were slightly
different. Should they be the same list?
 2020-06-17 11:16:16 -04:00
Eelco Dolstra 5332c439d0 InstallableFlake: Show all possible attribute names 
E.g.

  $ nix run nixpkgs#hello
  error: --- Error ---------- nix
  flake 'flake:nixpkgs' does not provide attribute 'apps.x86_64-linux.hello' or 'hello'

instead of

  $ nix run nixpkgs#hello
  error: --- Error ---------- nix
  flake 'flake:nixpkgs' does not provide attribute 'hello'
 2020-06-17 17:13:01 +02:00
John Ericson 6403508f5a Use ansicolor.hh in nix repl rather than duplicates 2020-06-17 15:13:00 +00:00
Eelco Dolstra ad66fb0a37 getFlake -> builtins.getFlake 2020-06-17 17:05:08 +02:00
Eelco Dolstra 0a1d3c1dd3 Don't provide 'getFlake' if the 'flakes' feature is not enabled 2020-06-17 16:54:32 +02:00
regnat 56d75bf4fc Reserve the __contentAddressed derivation parameter 
Not implementing anything here, just throwing an error if a derivation
sets `__contentAddressed = true` without
`--experimental-features content-addressed-paths`
(and also with it as there's nothing implemented yet)
 2020-06-17 15:41:17 +02:00
Eelco Dolstra fdff09e57c Fix coverage build 2020-06-17 15:18:10 +02:00
Eelco Dolstra ccbea8255c
Merge pull request #3657 from obsidiansystems/sligthly-improve-store-path-documentation 
Clarify the description of StorePath inputs
 2020-06-17 14:54:37 +02:00
Joe Hermaszewski da8aac6ce8 Mention number of derivations to be build/fetched in output 
Also correct grammar for the case of a single derivation.
 2020-06-17 20:27:27 +08:00
Eelco Dolstra 1524752c17 Merge remote-tracking branch 'origin/master' into flakes 2020-06-17 10:26:52 +02:00
John Ericson 517f5980e2 Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-17 04:58:43 +00:00
John Ericson 18493fd9c4 Move some Store functions from derivations.cc to store-api.cc 
This further continues with the dependency inverstion. Also I just went
ahead and exposed `parseDerivation`: it seems like the more proper
building block, and not a bad thing to expose if we are trying to be
less wedded to drv files on disk anywas.
 2020-06-17 04:30:16 +00:00
John Ericson 7130f0a3a6 Don't need abstract struct Derivation in local-store 2020-06-17 04:05:38 +00:00
John Ericson 21ef342172 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-06-17 03:52:01 +00:00
John Ericson 02928f76fd Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo 2020-06-17 03:51:02 +00:00
John Ericson 49308ef039 Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-17 03:39:39 +00:00
p01arst0rm e9970a34e8 appended ' __attribute__((weak)); ' to 'extern char * * environ ' 2020-06-17 03:25:34 +01:00
p01arst0rm c9d06558b6 replaced uncaught_exception with uncaught_exceptions 2020-06-17 03:15:47 +01:00
Eelco Dolstra 29542865ce Remove StorePath::clone() and related functions 2020-06-16 22:20:18 +02:00
Eelco Dolstra df4da4f5da
Merge pull request #3702 from NixOS/store-path-cxx 
Rewrite StorePath class in C++
 2020-06-16 21:37:26 +02:00
John Ericson fbf90bd693 Add another missing #include 2020-06-16 14:19:49 +00:00
Eelco Dolstra 759947bf72 StorePath: Rewrite in C++ 
On nix-env -qa -f '<nixpkgs>', this reduces maximum RSS by 20970 KiB
and runtime by 0.8%. This is mostly because we're not parsing the hash
part as a hash anymore (just validating that it consists of base-32
characters).

Also, replace storePathToHash() by StorePath::hashPart().
 2020-06-16 14:28:41 +02:00
Eelco Dolstra 72e17290d4 Fix FTP support 
Fixes #3618.
 2020-06-16 11:53:04 +02:00
Tobias Pflug cd8214c398 Fix logging unit tests 2020-06-16 10:23:15 +02:00
John Ericson 7e7e3b71f3 Add mising #include for strerror 2020-06-15 23:35:07 +00:00
Eelco Dolstra a588b6b19d Print only one error message if a build fails 
E.g. instead of

  error: --- BuildError ----------------------------------------------- nix
  builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
  error: --- Error ---------------------------------------------------- nix
  build of '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed

we now get

  error: --- Error ---------------------------------------------------- nix
  builder for '/nix/store/03nk0a3n8h2948k4lqfgnnmym7knkcma-foo.drv' failed with exit code 1
 2020-06-15 19:35:31 +02:00
Eelco Dolstra 24a3208247 Include only the base name of the program in error messages 2020-06-15 19:35:31 +02:00
Eelco Dolstra 8b099812ea Respect terminal width printing error messages 2020-06-15 19:35:31 +02:00
Eelco Dolstra f20bb983ca Cleanup 2020-06-15 18:16:03 +02:00
Eelco Dolstra 4e995bc8a6 Always hide the progress bar on exit 2020-06-15 18:01:05 +02:00
Eelco Dolstra 31707735b6 Remove unnecessary amDone() overrides 2020-06-15 16:47:21 +02:00
Eelco Dolstra ccfa6b3eee Give better error message about <...> in pure eval mode 2020-06-15 16:12:27 +02:00
Eelco Dolstra 5ed5d7acbd Improve "waiting for locks" messages 
These are now shown in the progress bar.

Closes #3577.
 2020-06-15 16:03:29 +02:00
Eelco Dolstra e14e62fddd Remove trailing whitespace 2020-06-15 14:12:39 +02:00
Eelco Dolstra 1fb762d11f Get rid of explicit ErrorInfo constructors 2020-06-15 14:06:58 +02:00
Eelco Dolstra fd64e4fb96 Disambiguate BaseError(Args) constructor 
This means that 'throw Error({ ... ErrorInfo ... })' now works.
 2020-06-15 13:50:33 +02:00
Eelco Dolstra 7a77762961 Merge branch 'errors-phase-2' of https://github.com/bburdette/nix 2020-06-15 11:46:31 +02:00
Matthew Bauer c214cda940 Correctly substitute from different storeDir 
Originally, the test was only checking for different “real” storeDir.
That’s an easy case to handle, but the much harder one is if different
virtual store dirs are used. To do this, we need the SubstitutionGoal
to know about the ca, so it can recalculate the path to copy it over.
An important note here is that the store path passed to copyStorePath
needs to be one for srcStore - so that queryPathInfo works properly.

This also adds an error message when the store path from queryPathInfo
is different from the one we requested.
 2020-06-13 00:07:42 -05:00
Matthew Bauer b2cb288cdd Add makeFixedOutputPathFromCA function 
This puts what we are already doing into a shared method. It just
needs a path name and a ca and produces a store path.
 2020-06-12 16:36:35 -05:00
John Ericson f6f01416b7 Use std::string_view in a few more places 2020-06-12 21:32:30 +00:00
Matthew Bauer 88120442d2 Debug when storePath changes 
these rewrites should be transparent, but they are important to know
about when debugging
 2020-06-12 15:32:52 -05:00
Matthew Bauer e3cb536f19 Fix add-to-store --flat to put in correct hash 2020-06-12 15:25:45 -05:00
Matthew Bauer 5b386b05f5 Recompute storePath based on isContentAddressed 2020-06-12 14:57:45 -05:00
Matthew Bauer 85d01e1f65 Don’t use makeStorePath 2020-06-12 14:53:30 -05:00
Matthew Bauer 9077b9e6b2 Separate dstStore path from srcStore path 2020-06-12 14:27:28 -05:00
Matthew Bauer 2f2ac850b5 Compute new store path correctly 2020-06-12 14:03:17 -05:00
Matthew Bauer 1c55f16a16 Add --flat to nix add-to-store 
This can be used to add flat hashes to the nix store.
 2020-06-12 12:42:54 -05:00
Eelco Dolstra 2853ba4ab2 Fix build 2020-06-12 19:00:48 +02:00
Eelco Dolstra 00fa7e2205
Merge pull request #3674 from matthewbauer/allow-empty-hash2 
Allow empty hash in derivations
 2020-06-12 18:18:12 +02:00
Matthew Bauer ea0d29d99a Provide base argument to to_string 2020-06-12 10:18:27 -05:00
Matthew Bauer 11c97070f3 Fix storeDir != storeDir condition 
this needs to only continue if the path replacement fails.
 2020-06-12 10:14:03 -05:00
Matthew Bauer b260c9ee03 Add newHashAllowEmpty helper function 
This replaces the copy&paste with a helper function in hash.hh.
 2020-06-12 10:11:16 -05:00
Matthew Bauer 3e3eaa90dd Remove hashed-mirrors 2020-06-12 09:51:44 -05:00
Matthew Bauer 79c169d1c6 Allow substituting from different storeDir 
Substituters can substitute from one store dir to another with a
little bit of help. The store api just needs to have a CA so it can
recompute the store path based on the new store dir. We can only do
this for fixed output derivations with no references, though.
 2020-06-12 09:49:09 -05:00
Eelco Dolstra 9f736dd89d Add Store::readDerivation() convenience function 2020-06-12 13:04:52 +02:00
Eelco Dolstra 045b07200c Remove Store::queryDerivationOutputNames() 
This function was used in only one place, where it could easily be
replaced by readDerivation() since it's not
performance-critical. (This function appears to have been modelled
after queryDerivationOutputs(), which exists only to make the garbage
collector faster.)
 2020-06-12 12:46:33 +02:00
Eelco Dolstra 4a4c063222
Merge pull request #3670 from gilligan/add-pool-tests 
Add tests for pool.hh
 2020-06-12 11:19:05 +02:00
Eelco Dolstra 7db879e65e Check 'follows' inputs 2020-06-12 00:52:56 +02:00
Ben Burdette ef1b3f21b6 Merge remote-tracking branch 'upstream/master' into errors-phase-2 2020-06-11 14:06:35 -06:00
Eelco Dolstra d15c20efd5 diffLockFiles(): Show 'follows' changes 2020-06-11 22:00:58 +02:00
Eelco Dolstra ac4d43a31b
Merge pull request #3073 from tweag/machine-logs 
Add an option to print the logs in a machine-readable format
 2020-06-11 15:45:18 +02:00
Eelco Dolstra dd9bb11d0d Move names.{cc,hh} to libstore 2020-06-11 15:42:18 +02:00
Eelco Dolstra 95eb064062 Shut up warning 2020-06-11 15:39:30 +02:00
Eelco Dolstra 8bd892117a Style fixes 2020-06-11 15:39:08 +02:00
Eelco Dolstra 0c62b4ad0f Represent 'follows' inputs explicitly in the lock file 
This fixes an issue where lockfile generation was not idempotent:
after updating a lockfile, a "follows" node would end up pointing to a
new copy of the node, rather than to the original node.
 2020-06-11 14:40:21 +02:00
Tobias Pflug 4750d98bbd Add tests for pool.hh 2020-06-10 22:29:50 +02:00
Eelco Dolstra 195ed43b60 Preserve 'isFlake' when not updating a lock file entry 2020-06-10 16:24:05 +02:00
Eelco Dolstra 2226e97ec2 Combine lock file update messages 2020-06-10 15:22:12 +02:00
Eelco Dolstra fc6c7af424 Add helper function printInputPath() 2020-06-10 15:20:00 +02:00
Eelco Dolstra b9ae1bdd7a
Merge pull request #3655 from zimbatm/hash-encoding-prepare 
libutils/hash: remove default encoding
 2020-06-10 11:48:38 +02:00
Matthew Bauer b2c8061b44
Disable extra-platforms = i686-linux on wsl1 (#3676) 
WSL1 doesn’t support i686-linux emulation, see https://github.com/microsoft/wsl/issues/2468
 2020-06-09 21:53:53 +00:00
Matthew Bauer 19aa892f20 Support empty hash in fetchers 
fetchTarball, fetchTree, and fetchGit all have *optional* hash attrs.
This means that we need to be careful with what we allow to avoid
accidentally making these defaults. When ‘hash = ""’ we assume the
empty hash is wanted.
 2020-06-09 11:10:54 -05:00
Eelco Dolstra 29e0748847 Show HTTP status message 
For example:

  warning: unable to download 'https://api.github.com/repos/edolstra/dwarffs/commits/master': HTTP error 403 ('rate limit exceeded'); using cached version
 2020-06-09 14:20:22 +02:00
Eelco Dolstra 447ea52b07 FileTransfer: Don't store status since curl already does that 2020-06-09 14:05:15 +02:00
Eelco Dolstra 6cfc2db494 Fix applyOverride() for github 2020-06-09 13:45:07 +02:00
Matthew Bauer 762273f1fd Allow empty hash in derivations 
follow up of https://github.com/NixOS/nix/pull/3544

This allows hash="" so that it can be used for debugging purposes. For
instance, this gives you an error message like:

  warning: found empty hash, assuming you wanted 'sha256:0000000000000000000000000000000000000000000000000000'
  hash mismatch in fixed-output derivation '/nix/store/asx6qw1r1xk6iak6y6jph4n58h4hdmbm-nix':
    wanted: sha256:0000000000000000000000000000000000000000000000000000
    got:    sha256:0fpfhipl9v1mfzw2ffmxiyyzqwlkvww22bh9wcy4qrfslb4jm429
 2020-06-09 01:23:37 -05:00
Ben Burdette 2f19650768 add file origin to Pos in stests 2020-06-08 11:21:17 -06:00
Ben Burdette b1c53b034c Merge branch 'errors-phase-2' into caveman-LOCs 2020-06-08 11:10:13 -06:00
regnat 801112de1a Move progress-bar.cc to libmain 
Needed so that we can include it as a logger in loggers.cc without
adding a dependency on nix

This also requires moving names.hh to libutil to prevent a circular
dependency between libmain and libexpr
 2020-06-08 17:16:52 +02:00
Eelco Dolstra e073f2c584 nix flake: Require 'flakes' feature 2020-06-08 16:23:54 +02:00
Eelco Dolstra 6470450ab4 Add completion for --update-input 2020-06-08 16:20:00 +02:00
Eelco Dolstra c27f92698b Style fixes 2020-06-08 13:24:01 +02:00
Tobias Pflug cd6dbf951a Add compression unit tests 2020-06-08 11:34:37 +02:00
regnat 4983401440 Unify the printing of the logs between bar-with-logs and raw 
Make the printing of the build logs systematically go through the
logger, and replicate the behavior of `no-build-output` by having two
different loggers (one that prints the build logs and one that doesn't)
 2020-06-08 09:31:15 +02:00
regnat 170e86dff5 Make the logger customisable 
Add a new `--log-format` cli argument to change the format of the logs.
The possible values are
- raw (the default one for old-style commands)
- bar (the default one for new-style commands)
- bar-with-logs (equivalent to `--print-build-logs`)
- internal-json (the internal machine-readable json format)
 2020-06-08 09:31:15 +02:00
Ben Burdette 94c347577e set verbosity levels 2020-06-07 07:24:49 -06:00
Tobias Pflug e60747b5fb Remove error-demo/error-demo.cc 
The logging.hh superseeds the demo
 2020-06-06 10:23:12 +02:00
Tobias Pflug 952e72c804 Add tests for logging.hh 2020-06-06 10:22:32 +02:00
Eelco Dolstra 9ef6048d78 diffLockFiles(): Fix assertion failure 
There are some cases where this inequality didn't hold, in particular
due to the Input / TreeInfo merge, where we're not always showing
narHash.
 2020-06-05 20:52:23 +02:00
Eelco Dolstra d558fb98f6
Merge pull request #3656 from obsidiansystems/handle-unknown-file-ingestion 
Add error message when FileIngestionMethod is out of bounds
 2020-06-05 17:18:12 +02:00
Eelco Dolstra 488ff83e6b Fix completion of --template 2020-06-05 14:09:12 +02:00
Eelco Dolstra 39e84c35d0 Fix log-prefix of nix build -L 
Alternative fix to #3661. The cause was that 'name' is a
std::string_view into a temporary which could get overwritten.
 2020-06-05 10:45:05 +02:00
Carlo Nucera d614166cb6 Fix condition error and make test suite pass 2020-06-04 17:21:21 -04:00
John Ericson e5cc1ebc5d Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-04 21:04:35 +00:00
John Ericson a7b82fd006 Remove file which shouldn't be committed 2020-06-04 21:04:20 +00:00
John Ericson 94ddea9e2f Use readString rather than >> temporary 
Fixed the rest of these before, but this one slipped through.
 2020-06-04 20:55:08 +00:00
John Ericson 744ce9ce16 Merge branch 'master' of github.com:NixOS/nix into validPathInfo-ca-proper-datatype 2020-06-04 20:46:58 +00:00
John Ericson 2041499b5e Flip boolean 
Thanks Matt!
 2020-06-04 20:42:25 +00:00
John Ericson ed86acf02a Use some std::optional::has_value for clarity 2020-06-04 20:42:02 +00:00
John Ericson 574d5460f0 Make sure info.ca tag bit is set in nix add-to-store 2020-06-04 20:33:28 +00:00
Eelco Dolstra ab54031e04 getDefaultUrl() -> value_or() 2020-06-04 20:24:28 +02:00
Eelco Dolstra 810b2c6a48 nix flake init: Add a '--template' flag 
The initial contents of the flake is specified by the
'templates.<name>' or 'defaultTemplate' output of another flake. E.g.

  outputs = { self }: {

    templates = {

      nixos-container = {
        path = ./nixos-container;
        description = "An example of a NixOS container";
      };

    };

  };

allows

  $ nix flake init -t templates#nixos-container

Also add a command 'nix flake new', which is identical to 'nix flake
init' except that it initializes a specified directory rather than the
current directory.
 2020-06-04 20:22:25 +02:00
Ben Burdette 94427ffee3 add some comments 2020-06-04 11:53:19 -06:00
Matthew Bauer 2299ef705c Add error message when FileIngestionMethod is out of bounds 
bool coerces anything >0 to true, but in the future we may have other
file ingestion methods. This shows a better error message when the
“recursive” byte isn’t 1.
 2020-06-04 11:32:39 -05:00
Eelco Dolstra dc305500c3
Merge pull request #3660 from Kloenk/selfhost-gitlab 
add support for selfhosted gitlab/github
 2020-06-04 14:59:33 +02:00
Eelco Dolstra d746ef4a81 Disable eval cache with --impure 
Fixes

  $ nix build nixpkgs#zoom-us
  error: Package ‘zoom-us-5.0.399860.0429’ in /nix/store/m79v7h75b69fkk8d2qcwm555l3wq6fmv-source/pkgs/applications/networking/instant-messengers/zoom-us/default.nix:126 has an unfree license (‘unfree’), refusing to evaluate.

  $ nix build nixpkgs#zoom-us --impure
  error: cached failure of attribute 'legacyPackages.x86_64-linux.zoom-us.drvPath'
 2020-06-04 14:53:51 +02:00
Finn Behrens 108debef6f
add support for selfhosted gitlab/github 2020-06-04 14:42:39 +02:00
Eelco Dolstra f85606c431 Merge remote-tracking branch 'origin/master' into flakes 2020-06-04 13:16:28 +02:00
Eelco Dolstra 0f44b60e6d Make 'nix dev-shell' a deprecated alias for 'nix develop' 2020-06-04 11:14:19 +02:00
Eelco Dolstra 61e3d598b6 Rename 'nix dev-shell' to 'nix develop' 
Fixes #3648.
 2020-06-04 10:57:40 +02:00
John Ericson 53bc8ff152 No C++ designated initializers yet with Clang 7 2020-06-03 20:45:14 -04:00
Ben Burdette 4335ba999b Merge remote-tracking branch 'upstream/master' into errors-phase-2 2020-06-03 17:00:00 -06:00
Ben Burdette f97576c5d9 newline-as-prefix; no final newline in output. 2020-06-03 14:47:00 -06:00
Carlo Nucera 132d6f2c24 Clarify the description of StorePath construction 2020-06-03 16:08:32 -04:00
John Ericson 74b251b2f3 Don't anticipate multiple CA outputs for now 2020-06-03 18:53:04 +00:00
John Ericson 3a9e4c3262 Don't anticipate CA but not fixed outputs for now 2020-06-03 18:50:45 +00:00
John Ericson 2500403059 Use enum and predicates rather than bitfile for derivation type 2020-06-03 17:38:54 +00:00
John Ericson 6b7f4ec4ab Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-03 16:36:06 +00:00
Eelco Dolstra c20591ddc3 Merge remote-tracking branch 'origin/master' into flakes 2020-06-03 16:15:22 +02:00
Jonas Chevalier 6ee03b8444
libutils/hash: remove default encoding 
This will make it easier to reason about the hash encoding and switch to
SRI everywhere where possible.
 2020-06-03 13:49:51 +02:00
John Ericson 01572c2198
Missing #include <cassert> in lru-cache.hh (#3654) 
This was a latent bug that just appeared because of the tests that were
added. Remember to wait for CI! :)
 2020-06-03 10:15:22 +00:00
John Ericson 3c78ac348c Merge remote-tracking branch 'obsidian/no-hash-type-unknown' into validPathInfo-ca-proper-datatype 2020-06-03 04:44:24 +00:00
John Ericson fecff16a6e Merge remote-tracking branch 'obsidian/missing-include-0' into validPathInfo-ca-proper-datatype 2020-06-02 23:23:30 +00:00
John Ericson 39ba87be9b Missing #include <cassert> in lru-cache.hh 
This was a latent bug that just appeared because of the tests that were
added. Remember to wait for CI! :)
 2020-06-02 21:36:53 +00:00
John Ericson 406dbb7fce outputHashAlgo can be blank so parse accordingly 
It is blank for SRI hashes.
 2020-06-02 21:09:15 +00:00
John Ericson 1fcd3afc38 Fix hashes 2020-06-02 20:35:17 +00:00
Carlo Nucera 75d2581390 Typo 2020-06-02 16:21:18 -04:00
Carlo Nucera 78f137e931 Validate text version instead, throw Errors 2020-06-02 16:20:22 -04:00
Carlo Nucera a5cdf1867e Add assertions for SHA256 in fixed case 2020-06-02 16:13:08 -04:00
Carlo Nucera fd2eb41e64 Move file-hash to content-address 2020-06-02 15:44:58 -04:00
Carlo Nucera 343c20a404 WIP Completed implementation 2020-06-02 15:23:21 -04:00
John Ericson c664e68b87 Fix to-base --type handler to correctly set std::optional flag 
Now that we have a separate flag function, also describe why it is
optional.
 2020-06-02 18:25:32 +00:00
Carlo Nucera 390bf64858 WIP 2020-06-02 14:15:58 -04:00
John Ericson c502119fd3 to-base supports parsing SRI hashes, so make type flag optional 2020-06-02 18:05:26 +00:00
John Ericson a33270ce1d Clean up ValidPathInfo::isContentAddressed with std::visit 2020-06-02 17:04:21 +00:00
John Ericson 25e61812f3
Apply suggestions from code review 
Co-authored-by: Matthew Bauer <mjbauer95@gmail.com>
 2020-06-02 12:47:18 -04:00
John Ericson d73dbc8e4c Remove hashingWithUnknownAlgoExits 
A valid hash type must be provided now. The hash itself can still be
invalid, but that doesn't cause an `abort()`.
 2020-06-02 16:28:54 +00:00
John Ericson 64cffb804a Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown 2020-06-02 16:07:25 +00:00
John Ericson 450dcf2c1b Remove HashType::Unknown 
Instead, `Hash` uses `std::optional<HashType>`. In the future, we may
also make `Hash` itself require a known hash type, encoraging people to
use `std::optional<Hash>` instead.
 2020-06-02 15:52:13 +00:00
Carlo Nucera a5d820a0a3 Change parseCa(Opt) to parseContentAddress(Opt) 2020-06-02 11:00:10 -04:00
Ben Burdette 156d4f8bc8 remove extra space in SysErrors 2020-06-02 08:45:37 -06:00
John Ericson 1b6461f671 Merge remote-tracking branch 'upstream/master' into validPathInfo-ca-proper-datatype 2020-06-02 14:31:18 +00:00
Ben Burdette d82d230b40 elide the 'ErrorInfo' in logError and logWarning calls 2020-06-02 08:22:24 -06:00
Eelco Dolstra bfa1acd85c
Merge pull request #3639 from obsidiansystems/do-fixme-store-removes 
Remove `addToStore` variant as requested by `FIXME`
 2020-06-02 15:39:07 +02:00
Eelco Dolstra c16fdda3a6 Merge branch 'lru-tests' of https://github.com/gilligan/nix 2020-06-02 12:07:48 +02:00
Eelco Dolstra e9fee8e6a7
src/libutil/tests/lru-cache.cc: Check erase() 
Co-authored-by: James Lee <jbit@jbit.net>
 2020-06-02 12:06:59 +02:00
John Ericson efcd30da89 WIP 2020-06-02 00:37:43 +00:00
John Ericson 754c910953 WIP more progress 2020-06-01 19:26:40 -04:00
Carlo Nucera da39092a39 WIP 2020-06-01 18:53:31 -04:00
Carlo Nucera 0e9438b6d3 Create new file-hash files 2020-06-01 17:32:40 -04:00
Carlo Nucera 0cb67ecbd3 Merge branch 'derivation-header-include-order' of github.com:Ericson2314/nix into validPathInfo-ca-proper-datatype 2020-06-01 17:13:11 -04:00
Carlo Nucera f4b89e11a4 Merge branch 'no-stringly-typed-derivation-output' of github.com:Ericson2314/nix into validPathInfo-ca-proper-datatype 2020-06-01 17:12:50 -04:00
Matthew Kenigsberg c254254a80 use Tree ctor 2020-06-01 12:32:17 -06:00
Matthew Kenigsberg 7680993506 Tree ctors 2020-06-01 09:01:37 -06:00
Matthew Kenigsberg ff1320b850 fetchOrSubstituteTree improvements 
Caches tree in addition to lockedRef, and explicitly writes out the logic for different combinations of cached/uncached flakes and indirect/resolved/locked flakes. This eliminates uneccessary calls to lookupInFlakeCache, fetchTree, maybeLookupFlake, and flakeCache.push_back
 2020-06-01 02:57:22 -06:00
Tobias Pflug eca1ff7a9f Add tests for lru-cache.hh 2020-05-31 01:05:05 +02:00
Nikola Knezevic fb38459d6e Ensure we restrict refspec interpretation while fetching 
As `git fetch` may chose to interpret refspec to it's liking, ensure that we
only pass refs that begin with `refs/` as is, otherwise, prepend them with
`refs/heads`. Otherwise, branches named `heads/foo` (I know it's bad, but it's
allowed), would be fetched as `foo`, instead of `heads/foo`.
 2020-05-30 12:33:38 +02:00
Nikola Knezevic 77007d4eab Improve ref validity checking in fetchGit 
The previous regex was too strict and did not match what git was allowing. It
could lead to `fetchGit` not accepting valid branch names, even though they
exist in a repository (for example, branch names containing `/`, which are
pretty standard, like `release/1.0` branches).

The new regex defines what a branch name should **NOT** contain. It takes the
definitions from `refs.c` in https://github.com/git/git and `git help
check-ref-format` pages.

This change also introduces a test for ref name validity checking, which
compares the result from Nix with the result of `git check-ref-format --branch`.
 2020-05-30 12:29:35 +02:00
Eelco Dolstra 89e0b3e2d6 Move substitution into Input::fetch() 
Closes #3520.
 2020-05-30 01:16:53 +02:00
Eelco Dolstra 0e7f77a59a Check revCount / lastModified input attributes if specified 2020-05-30 00:59:13 +02:00
Eelco Dolstra 950b46821f Remove TreeInfo 
The attributes previously stored in TreeInfo (narHash, revCount,
lastModified) are now stored in Input. This makes it less arbitrary
what attributes are stored where.

As a result, the lock file format has changed. An entry like

    "info": {
      "lastModified": 1585405475,
      "narHash": "sha256-bESW0n4KgPmZ0luxvwJ+UyATrC6iIltVCsGdLiphVeE="
    },
    "locked": {
      "owner": "NixOS",
      "repo": "nixpkgs",
      "rev": "b88ff468e9850410070d4e0ccd68c7011f15b2be",
      "type": "github"
    },

is now stored as

    "locked": {
      "owner": "NixOS",
      "repo": "nixpkgs",
      "rev": "b88ff468e9850410070d4e0ccd68c7011f15b2be",
      "type": "github",
      "lastModified": 1585405475,
      "narHash": "sha256-bESW0n4KgPmZ0luxvwJ+UyATrC6iIltVCsGdLiphVeE="
    },

The 'Input' class is now a dumb set of attributes. All the fetcher
implementations subclass InputScheme, not Input. This simplifies the
API.

Also, fix substitution of flake inputs. This was broken since lazy
flake fetching started using fetchTree internally.
 2020-05-30 00:44:11 +02:00
John Ericson fac0c2d54a Remove addToStore variant as requested by FIXME 
The idea is it's always more flexible to consumer a `Source` than a
plain string, and it might even reduce memory consumption.

I also looked at `addToStoreFromDump` with its `// FIXME: remove?`, but
the worked needed for that is far more up for interpretation, so I
punted for now.
 2020-05-29 17:02:32 -04:00
Ben Burdette 734283d636 Merge remote-tracking branch 'upstream/master' into errors-phase-2 2020-05-29 09:51:37 -06:00
Eelco Dolstra 5633c0975b Factor out GitHub / GitLab commonality 2020-05-29 14:23:32 +02:00
Finn Behrens 5256bc77ca
add gitlab libfetcher 2020-05-28 23:00:08 +02:00
Carlo Nucera 6dd471ebf6 Fixing the result of merge 2020-05-28 12:14:36 -04:00
Carlo Nucera 4f597fb901 Merge branch 'master' of github.com:NixOS/nix into enum-class 2020-05-28 10:58:22 -04:00
John Ericson 5b4cd84bc2 Merge remote-tracking branch 'me/more-rust-ffi' into no-stringly-typed-derivation-output 2020-05-28 10:35:53 -04:00
John Ericson ef71caba29 Merge remote-tracking branch 'upstream/master' into more-rust-ffi 2020-05-28 10:31:46 -04:00
Eelco Dolstra f60ce4fa20
Merge pull request #3631 from andir/libutil-config-tests 
Add unit tests for config.cc
 2020-05-28 13:51:37 +02:00
Eelco Dolstra de141fcb79
Merge pull request #3455 from Ericson2314/enum-FileIngestionMethod 
Replace some `bool recursive` with a new `FileIngestionMethod` enum
 2020-05-28 13:50:06 +02:00
Eelco Dolstra 17ca997fc6 Merge remote-tracking branch 'origin/master' into flakes 2020-05-28 12:55:24 +02:00
Eelco Dolstra c3eff22f46 Merge branch 'store-visited' of https://github.com/mkenigs/nix into flakes 2020-05-28 12:15:39 +02:00
Eelco Dolstra 6286272371 nixpkgsFlakeRef(): Use locked nixpkgs 2020-05-28 12:13:13 +02:00
John Ericson 0f96f45061 Use FileIngestionMethod for nix hash 
There was an enum there that matched in perfectly.
 2020-05-27 23:50:11 -04:00
Matthew Bauer c66441a646 Rename some variables named “recursive” to “method” 
This is much less confusing since recursive is no longer a boolean.
 2020-05-27 13:21:26 -05:00
Matthew Bauer 7873fd175d Don’t use FileIngestionMethod for StorePathsCommand 
This is a different recursive than used in makeFixedOutputPath.
 2020-05-27 13:21:11 -05:00
Andreas Rammhold fc137d2f00
config.hh: Add documentation 
Provides some general overview on the mechanics of Config/Setting and
comments for the public methods of Config.
 2020-05-27 17:47:18 +02:00
Andreas Rammhold 9df3d8ccd7
tests/config.cc: add tests for Config::applyConfig 2020-05-27 17:47:18 +02:00
Andreas Rammhold e1b8c64c04
config.cc: extract parts of applyConfigFile into applyConfig 
This moves the actual parsing of configuration contents into applyConfig
which applyConfigFile is then going to call. By changing this we can now
test the configuration file parsing without actually create a file on
disk.
 2020-05-27 17:47:18 +02:00
Andreas Rammhold 93129cf1dd
Add unit tests for config.cc 2020-05-27 17:47:17 +02:00
Eelco Dolstra 66d3ac94c9 Merge pull request #3621 from gilligan/add-json-tests 
Add unit tests for "json.hh"
 2020-05-27 11:08:12 +02:00
Eelco Dolstra dae6a267a8
Merge pull request #3625 from gilligan/xml-writer-tests 
Add unit tests for xml-writer
 2020-05-27 11:07:53 +02:00
Carlo Nucera f3f520c14c Change syntax for CI 2020-05-26 12:51:28 -04:00
Carlo Nucera 89a5ac9d3b Merge remote-tracking branch 'john-ericson/more-rust-ffi' into no-stringly-typed-derivation-output 2020-05-26 12:31:26 -04:00
Carlo Nucera d49e65ba9d Merge remote-tracking branch 'john-ericson/enum-FileIngestionMethod' into no-stringly-typed-derivation-output 2020-05-26 12:30:48 -04:00
Carlo Nucera 0f3f901071 Merge remote-tracking branch 'origin/master' into more-rust-ffi 2020-05-26 11:46:42 -04:00
Carlo Nucera c2f33edd1f
Update src/libutil/rust-ffi.hh 
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
 2020-05-26 11:43:18 -04:00
Carlo Nucera b90241ceb1 Change remaining bools with FileIngestionMethod 2020-05-26 11:32:41 -04:00
Carlo Nucera 6d73c10041 Merge remote-tracking branch 'origin/master' into enum-FileIngestionMethod 2020-05-26 11:14:08 -04:00
Lucas Franceschino b2748c6e99 Make functionArgs primitive accept primops 2020-05-25 19:07:38 +02:00
Tobias Pflug 4b388e8431 Add unit tests for xml-writer 2020-05-25 18:34:55 +02:00
Tobias Pflug c284700867 Add unit tests for "json.hh" 2020-05-25 11:57:45 +02:00
Tobias Pflug ecc5c90dfc Add unit tests for hashing functions 2020-05-25 11:50:41 +02:00
Matthew Kenigsberg 8d67794da1 handle circular flake dependencies in list-inputs 2020-05-21 17:06:11 -06:00
Ben Burdette b7057fa627 remove error-demo from make; clean up comment 2020-05-21 16:04:18 -06:00
Ben Burdette 0e49de6a2b position for stdin, string; (string) for trace; fix tests 2020-05-21 14:28:45 -06:00
Ben Burdette 6a420d672c print LOC for stdin, string args 2020-05-20 22:18:26 -06:00
Ben Burdette 85ce455b85 get code lines from the nix file 2020-05-20 17:25:02 -06:00
Tobias Pflug a73a820a5d Add unit testes for url.cc 
This adds tests for

- parseURL
- percentDecode
- decodeQuery
 2020-05-20 16:37:35 +02:00
Eelco Dolstra 5ef64f05e6 Cleanup 2020-05-18 15:50:29 +02:00
Eelco Dolstra 0ed946aa61 Merge branch 'wait-for-builders' of https://github.com/serokell/nix 2020-05-18 13:48:45 +02:00
Matthew Kenigsberg c4beded32e rm includes 2020-05-16 11:19:41 -06:00
Matthew Kenigsberg ba7d7ed2e3 Create bashInteractive InstallableFlake 2020-05-16 11:03:06 -06:00
Matthew Kenigsberg 0858793604 Call lockFlake once and store in _lockedFlake 2020-05-16 11:03:06 -06:00
Matthew Kenigsberg 04821bc171 use flake's nixpkgs to find bashInteractive 2020-05-16 11:03:06 -06:00
Matthew Kenigsberg 8fbc8540d3 use nixpkgs#bashInteractive for dev-shell 2020-05-16 11:03:06 -06:00
Ben Burdette 92123c6c79 Merge remote-tracking branch 'upstream/master' into errors-phase-2 2020-05-15 07:00:36 -06:00
Eelco Dolstra 5f64655ff4 Move registry-related commands from 'nix flake' to 'nix registry' 
This makes 'nix flake' less cluttered and more consistent (it's only
subcommands that operator on a flake). Also, the registry is not
inherently flake-related (e.g. fetchTree could also use it to remap
inputs).
 2020-05-15 14:38:10 +02:00
Ben Burdette 19694aa213 fix compile errors 2020-05-14 12:28:18 -06:00
Ben Burdette 4daccb279c formatting 2020-05-14 10:28:17 -06:00
Alexander Bantyev 183dd28266
Don't lock a user while doing remote builds 2020-05-14 17:00:54 +03:00
Ben Burdette ef9dd9f9bc formatting and a few minor changes 2020-05-13 15:56:39 -06:00
Ben Burdette c79d4addab consistent capitalization 2020-05-13 10:02:18 -06:00
Ben Burdette bfca5fc395 change status messages to info level 2020-05-13 09:52:36 -06:00
Eelco Dolstra 849d3968db
Update src/libfetchers/git.cc 
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
 2020-05-13 10:41:21 +02:00
Ben Burdette ecbb8e9c0a no blank line if no LOC 2020-05-12 14:41:30 -06:00
Ben Burdette 960d4362ed hint only 2020-05-12 13:54:18 -06:00
Ben Burdette 72ecccee57 convert to logWarning format 2020-05-12 12:19:34 -06:00
Ben Burdette 19cffc29c9 remove unused extra json fields 2020-05-12 12:09:12 -06:00
Ben Burdette 2a19bf8619 move pos to the first arg, to indicate its not used in a fmt template 2020-05-12 11:27:37 -06:00
Ben Burdette ec870b9c85 new pos format for more errors 2020-05-12 10:52:26 -06:00
Eelco Dolstra 215f09d765
Merge pull request #3587 from NixOS/bash-completion 
Generic shell completion support for the 'nix' command
 2020-05-12 18:26:13 +02:00
Eelco Dolstra ebc024df22
Show hint how to enable experimental features 2020-05-12 15:47:09 +02:00
Eelco Dolstra 268ecf5b3f
nix: Don't require --experimental-features=nix-command for some subcommands 2020-05-12 15:47:09 +02:00
Eelco Dolstra b8b2dbf272 Fix InstallableCommand 2020-05-12 11:53:32 +02:00
Eelco Dolstra 437614b479 Fix macOS build 
macOS doesn't have GLOB_ONLYDIR.
 2020-05-12 11:09:09 +02:00
Ben Burdette 7c3138844c more pos reporting 2020-05-11 17:34:57 -06:00
Ben Burdette 631642c5b4 new format for pos 2020-05-11 16:58:08 -06:00
Ben Burdette b93c1bf3d6 fixes to merged code 2020-05-11 15:52:15 -06:00
Ben Burdette 59b1f5c701 Merge branch 'master' into errors-phase-2 2020-05-11 14:35:30 -06:00
Eelco Dolstra 649c2db308 nix flake: Add completion support 2020-05-11 22:10:33 +02:00
Eelco Dolstra 27d34ef770 When completing flakerefs, only return directories 2020-05-11 22:04:13 +02:00
Ben Burdette 536bbf53e1 comments and cleanup 2020-05-11 13:58:38 -06:00
Eelco Dolstra 259ff74bde Add completion for installables 
This completes flakerefs using the registry (e.g. 'nix<TAB>' => 'nix
nixpkgs') and flake output attributes by evaluating the flake
(e.g. 'dwarffs#nix<TAB>' => 'dwarffs#nixosModules').
 2020-05-11 21:49:02 +02:00
Eelco Dolstra e917332d63 Shut up warnings while running completers 2020-05-11 21:38:17 +02:00
Ben Burdette 958e81987b switch from printError warnings to logWarnings 2020-05-11 13:02:16 -06:00
Domen Kožar 5bdb67c843
Merge pull request #3568 from kolloch/outputHashModeError 
libstore/build.cc: more explicit error about form of output
 2020-05-11 18:14:32 +02:00
Domen Kožar 1d8144e36b
Update src/libstore/build.cc 2020-05-11 18:14:23 +02:00
Eelco Dolstra 4c3c638a05 Cleanup 2020-05-11 15:57:45 +02:00
Eelco Dolstra 0884f180f5 Simplify 2020-05-10 21:50:32 +02:00
Eelco Dolstra e0c19ee620 Add completion for paths 2020-05-10 21:35:07 +02:00
Eelco Dolstra 91ddee6bf0 nix: Implement basic bash completion 2020-05-10 20:32:21 +02:00
Matthew Kenigsberg 73ee1afffe Reorder to build 
This reverts commit 883948d7a0add742ccae58e9845d769a8064371c.
 2020-05-09 14:42:32 -06:00
Matthew Kenigsberg 9f4cfbb2e7 Refactor installables 
InstallableValue has children InstallableFlake and InstallableAttrPath, but InstallableFlake was overriding toDerivations, and usage was changed so that InstallableFlake didn't need cmd. So these changes were made:
InstallableValue::toDerivations() -> InstalllableAttrPath::toDerivations()
InstallableValue::cmd -> InstallableAttrPath::cmd

InstallableValue uses state instead of cmd

toBuildables() and toDerivations() were made abstract
 2020-05-09 14:42:32 -06:00
Matthew Kenigsberg bf81dd40e9 InstallableExpr unused 2020-05-09 10:16:00 -06:00
Benjamin Hipple 146f9c114f doc: consistently refer to 'fixed-output' with a dash 
General cleanup that makes it easier to search for the term.
 2020-05-09 10:58:43 -04:00
Ben Burdette 55eb717148 add pos to errorinfo, remove from hints 2020-05-08 18:18:28 -06:00
Eelco Dolstra d3d8186c9c
Merge pull request #3571 from gilligan/nix-unit-testing 
Add unit tests
 2020-05-08 17:02:25 +02:00
Tobias Pflug 181a47d884 Enable toLower umlauts test 
Update comment and enable the test
 2020-05-08 15:13:55 +02:00
Tobias Pflug 2191141274 Enable baseNameOf test 
Add note about removal of trailing slashes in the doc comment of
baseNameOf and enabled the test.
 2020-05-08 15:07:40 +02:00
Tobias Pflug e3df9c2a6e Enable dirOf test 
Adjusted the doc comment for `dirOf` to reflect the implementation
behavior.
 2020-05-08 15:03:44 +02:00
Eelco Dolstra 5b8883faac
configure: Look for gtest 2020-05-08 12:09:37 +02:00
Eelco Dolstra ca657525b8
Don't install unit tests 2020-05-08 12:03:27 +02:00
Eelco Dolstra 7898cdb75a
make check: Run unit tests 2020-05-08 11:49:40 +02:00
Eelco Dolstra 72b9d971bc
Fix warning 2020-05-08 11:35:57 +02:00
Eelco Dolstra 7cc7cef950
Move unit tests to sr/libutil/tests, use mk make rules 2020-05-08 11:34:09 +02:00
Alexander Bantyev 772e5db828
Mention build users in the 'waiting for' message 2020-05-08 12:29:00 +03:00
Alexander Bantyev 14073fb76b
Don't block while waiting for build users 2020-05-08 12:22:39 +03:00
Ben Burdette 1b801cec40 pretending to be const 2020-05-07 16:43:36 -06:00
Tobias Pflug 1f3602a2c9 Remove replaceInSet 
The function isn't being used anywhere so it seems safe to remove
 2020-05-07 18:15:13 +02:00
Eelco Dolstra 14a3a62bfc
Update src/nix/search.cc 
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
 2020-05-07 12:13:16 +02:00
Ben Burdette e3901638b5 todo removal 2020-05-06 15:01:13 -06:00
Ben Burdette e76ad2e48a implement SysError errno handling 2020-05-06 14:07:20 -06:00
Eelco Dolstra 1ad71bc62c Remove support for old lockfiles and the epoch/uri attributes 2020-05-06 17:48:18 +02:00
Eelco Dolstra b0e9b07e80 Remove obsolete FIXME 2020-05-06 17:40:01 +02:00
Eelco Dolstra ff394ff206 Remove the nixpkgs.<attr> compatibility hack 
Since we've changed a lot of things in the 'nix' command (e.g. rename
'nix run') there is not much point in keeping this around.
 2020-05-06 17:20:23 +02:00
Domen Kožar 672985531c
Merge pull request #3570 from Mic92/nix-search 
nix/search: no error for empty search results if json is enabled
 2020-05-06 16:14:58 +02:00
Jörg Thalheim 85c1932c94
nix/search: no error for empty search results if json is enabled 
- result list will be always empty if --json is passed
- for scripts an empty search result is not really an error,
  we rather want to distinguish between evaluation errors and empty results
 2020-05-06 14:43:54 +01:00
Eelco Dolstra 2f8ee4578f
Merge remote-tracking branch 'origin/master' into flakes 2020-05-06 12:01:40 +02:00
Eelco Dolstra 74a1bfdcab
Merge pull request #3546 from guibou/nix_readfile_on_0_sized_files 
builtins.readFile: do not truncate content
 2020-05-06 11:33:55 +02:00
Eelco Dolstra 272c4ba36d
Merge pull request #3557 from Ma27/nix-repl-help 
Improve help-message for nix-repl
 2020-05-06 11:25:47 +02:00
Peter Kolloch 9be46859a9 libstore/build.cc: more explicit about form of output 
Be more explicit about why we expect a regular file as output
when outputHashMode=flat for a fixed output derivation.
 2020-05-06 11:21:12 +02:00
Eelco Dolstra 02c5914ea4
Merge pull request #3562 from pikajude/master 
Use fragment size for autoGC capacity calculation
 2020-05-06 11:19:30 +02:00
Eelco Dolstra fd4911269f
Revert "Merge pull request #3558 from LnL7/ssh-ng-stderr" 
This reverts commit 3ebfbecdd1, reversing
changes made to c089c52d5f.

https://github.com/NixOS/nix/pull/3558
 2020-05-06 10:54:18 +02:00
Eelco Dolstra 6f3244ce45 Merge remote-tracking branch 'origin/master' into flakes 2020-05-05 18:59:33 +02:00
Eelco Dolstra 909b4a8820 nix doctor: Consistency 2020-05-05 15:27:47 +02:00
Eelco Dolstra f132d82a79 nix --help: Group commands 2020-05-05 15:18:23 +02:00
Alexander Bantyev 04967dee9d
Wait for build users when none are available 2020-05-05 13:04:36 +03:00
Ben Burdette 7ffb5efdbc appending to hints; remove _printError 2020-05-04 16:19:57 -06:00
Ben Burdette f30de61578 add normaltxt, yellowify->yellowtxt 2020-05-04 16:19:20 -06:00
Jude Taylor e2fc575c61 nix auto-gc: use fragment size 2020-05-04 14:42:06 -07:00
Ben Burdette 8c8f2b74ec log as warning 2020-05-04 14:44:42 -06:00
Ben Burdette afaa541013 affinity operator<< 2020-05-04 14:44:00 -06:00
Eelco Dolstra a721a0b114 Flag: Use designated initializers 2020-05-04 22:40:19 +02:00
Ben Burdette 9c5ece44a7 separate msgs instead of appending to what() 2020-05-04 13:46:15 -06:00
Eelco Dolstra 3ebfbecdd1
Merge pull request #3558 from LnL7/ssh-ng-stderr 
remote-store: don't log raw stderr by default
 2020-05-04 13:02:33 +02:00
Ben Burdette ab6f0b9641 convert some printError calls to logError 2020-05-03 08:01:25 -06:00
Daiderd Jordan 4769eea5e2
logging: handle build log lines in simple logger 
The raw stderr output isn't logged anymore so the build logs need to be
printed by the default logger in order for the old commands like
nix-build to still show build output.
 2020-05-02 23:40:53 +02:00
Daiderd Jordan f16e24f95e
remote-store: don't log raw stderr by default 
For remote stores the log messages are already forwarded as structured
STDERR_RESULT messages so the old format is duplicate information.  But
still included with -vvv since it could be useful for debugging
problems.

    $ nix build -L /nix/store/nl71b2niws857ffiaggyrkjwgx9jjzc0-foo.drv --store ssh-ng://localhost
    Hello World!
    foo> Hello World!
    [1/0/1 built] building foo

Fixes #3556
 2020-05-02 23:40:50 +02:00
Maximilian Bosch 2aeb874e83
Improve help-message for nix-repl 
* Remove obsolete `printHelp` function
* Add an example to demonstrate how to list all available commands
  within the REPL
 2020-05-01 23:32:01 +02:00
Ben Burdette 4b99c09f5c convert some errors 2020-05-01 14:32:06 -06:00
Eelco Dolstra c089c52d5f Fix build 2020-05-01 12:42:39 +02:00
Matthew Kenigsberg d6b4047c2f rename nix run to nix shell and nix app to nix run 
(cherry picked from commit 5d8504b978)
 2020-05-01 12:17:36 +02:00
Eelco Dolstra 941f95284a Merge remote-tracking branch 'origin/master' into flakes 2020-05-01 11:59:56 +02:00
Ben Burdette a3030e3c31 fix error calls 2020-04-30 17:56:26 -06:00
Ben Burdette f5d3215c87 logError 2020-04-30 16:31:47 -06:00
Ben Burdette 171b4ce85c typo 2020-04-30 09:57:01 -06:00
Eelco Dolstra 0135fd6ec4
nix dev-shell: Unset shellHook 
This avoids inheriting the caller's shellHook, which can happen when
running a dev-shell inside a dev-shell.
 2020-04-30 14:47:56 +02:00
Eelco Dolstra efe6c186ea
nix dev-shell: Support structured attrs 
Tested against https://github.com/NixOS/nixpkgs/pull/72074.

Fixes #3540.
 2020-04-30 14:47:47 +02:00
Eelco Dolstra 2fcfc6c2c6
nix dev-shell: Refactor script for getting the environment 2020-04-30 13:05:29 +02:00
Domen Kožar df8e9d691c
Merge pull request #3548 from Ma27/fetchtarball-pos 
Fix displaying error-position in `builtins.fetch{Tree,Tarball}`
 2020-04-30 10:33:12 +02:00
Ben Burdette 39ff80d031 errorinfo constructor test 2020-04-29 18:57:05 -06:00
Matthew Kenigsberg 5d8504b978 rename nix run to nix shell and nix app to nix run 2020-04-29 15:45:10 -06:00
Maximilian Bosch d1229859c2
Fix displaying error-position in builtins.fetch{Tree,Tarball} 
Without dereferencing this pointer, you'd get an error like this:

```
error: unsupported argument 'abc' to 'fetchTarball', at 0x13627e8
```
 2020-04-29 22:53:39 +02:00
Guillaume Bouchard 2e5be2a749 StringSink pre allocate 
When used with `readFile`, we have a pretty good heuristic of the file
size, so `reserve` this in the `string`. This will save some allocation
/ copy when the string is growing.
 2020-04-29 18:44:01 +02:00
Guillaume Bouchard 7afcb5af98 Remove the drain argument from readFile 
Now it is always `drain` (see previous commit).
 2020-04-29 18:43:45 +02:00
Ben Burdette e2f61263eb uncrustify formatting 2020-04-29 10:14:32 -06:00
Eelco Dolstra 9570036146 nix copy: Build derivations 
Fixes

  $ nix copy .#hydraJobs.vendoredCrates --to /tmp/nix
  error: path '/nix/store/...' is not valid
 2020-04-29 15:51:45 +02:00
Eelco Dolstra 9c4e05766b nix copy: Move --from / --to check 
This means you now get an error message *before* stuff gets built:

  $ nix copy .#hydraJobs.vendoredCrates
  error: you must pass '--from' and/or '--to'
  Try 'nix --help' for more information.
 2020-04-29 15:50:59 +02:00
Antoine Eiche ca93b26db6 Only call grantpt on MacOS systems 
The commit 3cc1125595 adds a `grantpt`
call on the builder pseudo terminal fd. This call is actually only
required for MacOS, but it however requires a RW access to /dev/pts
which is only RO bindmounted in the Bazel Linux sandbox. So, Nix can
not be actually run in the Bazel Linux sandbox for unneeded reasons.
 2020-04-29 15:43:20 +02:00
Eelco Dolstra 70bcd6a55c Evaluation cache: Don't barf in read-only mode 
Fixes

  $ nix copy
  warning: Git tree '/home/eelco/Dev/nix-flake' is dirty
  nix: src/nix/installables.cc:348: std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, nix::FlakeRef, nix::InstallableValue::DerivationInfo> nix::InstallableFlake::toDerivation(): Assertion `state->store->isValidPath(drvPath)' failed.
  Aborted (core dumped)
 2020-04-29 15:42:53 +02:00
Guillaume Bouchard 5a34a473dd builtins.readFile: do not truncate content 
This closes #3026 by allowing `builtins.readFile` to read a file with a
wrongly reported file size, for example, files in `/proc` may report a
file size of 0. Reading file in `/proc` is not a good enough motivation,
however I do think it just makes nix more robust by allowing more file
to be read.  Especially, I do considerer the previous behavior to be
dangerous because nix was previously reading truncated files. Examples
of file system which incorrectly report file size may be network file
system or dynamic file system (for performance reason, a dynamic file
system such as FUSE may generate the content of the file on demand).

```
nix-repl> builtins.readFile "/proc/version"
""
```

With this commit:

```
nix-repl> builtins.readFile "/proc/version"
"Linux version 5.6.7 (nixbld@localhost) (gcc version 9.3.0 (GCC)) #1-NixOS SMP Thu Apr 23 08:38:27 UTC 2020\n"
```

Here is a summary of the behavior changes:

- If the reported size is smaller, previous implementation
was silently returning a truncated file content. The new implementation
is returning the correct file content.

- If a file had a bigger reported file size, previous implementation was
failing with an exception, but the new implementation is returning the
correct file content. This change of behavior is coherent with this pull
request.

Open questions

- The behavior is unchanged for correctly reported file size, however
performances may vary because it uses the more complex sink interface.
Considering that sink is used a lot, I don't think this impacts the
performance a lot.
- `builtins.readFile` on an infinite file, such as `/dev/random` may
fill the memory.
- it does not support adding file to store, such as `${/proc/version}`.
 2020-04-29 14:50:52 +02:00
Eelco Dolstra 5ada0831cf Merge remote-tracking branch 'origin/master' into flakes 2020-04-29 13:32:27 +02:00
Ben Burdette 22e6490311 Error classname as name 2020-04-28 21:06:08 -06:00
Eelco Dolstra 06849c3090
Merge pull request #3542 from mkenigs/gcroots 
Set GCROOT to store path to prevent garbage collection
 2020-04-28 21:04:06 +02:00
Matthew Kenigsberg 6d40fe573c rename to NIX_GCROOT 2020-04-28 11:18:54 -06:00
Eelco Dolstra 52a3ca823d Tweak warning message 2020-04-28 17:56:01 +02:00
Eelco Dolstra 6a8cba83bb Merge branch 'nix-env-warn-unmatched' of https://github.com/lheckemann/nix 2020-04-28 17:45:25 +02:00
Ben Burdette e51a757720 astyle format 2020-04-27 15:15:08 -06:00
Eelco Dolstra 6521c92ce8 Improve path:// handling 
In particular, doing 'nix build /path/to/dir' now works if
/path/to/dir is not a Git tree (it only has to contain a flake.nix
file).

Also, 'nix flake init' no longer requires a Git tree (but it will do a
'git add flake.nix' if it's a Git tree)
 2020-04-27 22:53:11 +02:00
Eelco Dolstra 829dcb35d5 flake-template.nix: Add defaultPackage 2020-04-27 22:52:49 +02:00
Alyssa Ross c05e20daa1
Fix long paths permanently breaking GC 
Suppose I have a path /nix/store/[hash]-[name]/a/a/a/a/a/[...]/a,
long enough that everything after "/nix/store/" is longer than 4096
(MAX_PATH) bytes.

Nix will happily allow such a path to be inserted into the store,
because it doesn't look at all the nested structure.  It just cares
about the /nix/store/[hash]-[name] part.  But, when the path is deleted,
we encounter a problem.  Nix will move the path to /nix/store/trash, but
then when it's trying to recursively delete the trash directory, it will
at some point try to unlink
/nix/store/trash/[hash]-[name]/a/a/a/a/a/[...]/a.  This will fail,
because the path is too long.  After this has failed, any store deletion
operation will never work again, because Nix needs to delete the trash
directory before recreating it to move new things to it.  (I assume this
is because otherwise a path being deleted could already exist in the
trash, and then moving it would fail.)

This means that if I can trick somebody into just fetching a tarball
containing a path of the right length, they won't be able to delete
store paths or garbage collect ever again, until the offending path is
manually removed from /nix/store/trash.  (And even fixing this manually
is quite difficult if you don't understand the issue, because the
absolute path that Nix says it failed to remove is also too long for
rm(1).)

This patch fixes the issue by making Nix's recursive delete operation
use unlinkat(2).  This function takes a relative path and a directory
file descriptor.  We ensure that the relative path is always just the
name of the directory entry, and therefore its length will never exceed
255 bytes.  This means that it will never even come close to AX_PATH,
and Nix will therefore be able to handle removing arbitrarily deep
directory hierachies.

Since the directory file descriptor is used for recursion after being
used in readDirectory, I made a variant of readDirectory that takes an
already open directory stream, to avoid the directory being opened
multiple times.  As we have seen from this issue, the less we have to
interact with paths, the better, and so it's good to reuse file
descriptors where possible.

I left _deletePath as succeeding even if the parent directory doesn't
exist, even though that feels wrong to me, because without that early
return, the linux-sandbox test failed.

Reported-by: Alyssa Ross <hi@alyssa.is>
Thanks-to: Puck Meerburg <puck@puckipedia.com>
Tested-by: Puck Meerburg <puck@puckipedia.com>
Reviewed-by: Puck Meerburg <puck@puckipedia.com>
 2020-04-27 20:50:17 +00:00
Matthew Kenigsberg 9e95b95a5d comment 2020-04-27 13:18:26 -06:00
Matthew Kenigsberg a3bc695e7d Set GCROOT to store path to prevent garbage collection 2020-04-27 11:22:20 -06:00
Eelco Dolstra b51dff431c Improve error message when an argument is not a flake 2020-04-27 18:55:20 +02:00
Eelco Dolstra b4e23dcd9e nix search: Search legacyPackages recursively 2020-04-27 16:29:26 +02:00
Ben Burdette 1ff42722ce error.hh 2020-04-26 14:47:41 -06:00
Ben Burdette d4fd7b543e print dashes instead of empty name string 2020-04-25 12:05:26 -06:00
Linus Heckemann f59404e1a6 nix-env: refactor uninstallDerivations 
Reduces the number of store queries it performs. Also prints a warning
if any of the selectors did not match any installed derivations.

UX Caveats:
- Will print a warning that nothing matched if a previous selector
  already removed the path
- Will not do anything if no selectors were provided (no change from
  before).

Fixes #3531
 2020-04-25 16:30:42 +02:00
Ben Burdette cdac083dc5 don't print blank lines for blank description 2020-04-24 21:40:13 -06:00
Ben Burdette d8d4844b88 all things error to error.hh 2020-04-24 14:57:51 -06:00
Ben Burdette d9632765a8 add has_value check; remove obslete friend class 2020-04-24 12:44:23 -06:00
Eelco Dolstra 0469795978 nix search: Show version 2020-04-24 14:42:17 +02:00
Ben Burdette 833501f6f1 'what' string 2020-04-23 15:55:34 -06:00
Eelco Dolstra ef4d3fc111 Merge remote-tracking branch 'origin/master' into flakes 2020-04-23 15:16:18 +02:00
Ben Burdette 3bc9155dfc a few more 'format's rremoved 2020-04-22 15:00:11 -06:00
Eelco Dolstra c9d0cf7e02
Don't include error.hh in util.hh to prevent header bloat 2020-04-22 15:29:27 +02:00
Eelco Dolstra 16e3bf4537
Merge branch 'error-format' of https://github.com/bburdette/nix 2020-04-22 15:29:10 +02:00
Eelco Dolstra 2ea4d45449
Path fetcher: Fix store path name 
(cherry picked from commit c7af247bea)
 2020-04-22 15:27:06 +02:00
Eelco Dolstra c7af247bea Path fetcher: Fix store path name 2020-04-22 10:15:32 +02:00
Dustin DeWeese c0d940978a Replace select() with poll() to allow waiting on more than FD_SETSIZE fds 2020-04-21 16:21:28 -07:00
Ben Burdette e4fb9a3849 remove 'format' from Error constructor calls 2020-04-21 17:07:07 -06:00
Ben Burdette d3052197fe add ErrorInfo to BaseError 2020-04-21 13:25:41 -06:00
Eelco Dolstra 8c75621da6 Fix typo 2020-04-20 15:28:56 +02:00
Eelco Dolstra b69323f8c9 Revive 'nix search' 
It uses the evaluation cache now rather than the ad hoc JSON cache.
 2020-04-20 15:27:09 +02:00
Eelco Dolstra 42a12f9232 Move eval-cache.{cc,hh} 2020-04-20 13:14:59 +02:00
Eelco Dolstra 539a9c1c5f Get rid of the old eval cache 2020-04-20 13:13:52 +02:00
Ben Burdette 15e9564fd1 logEI for tunnelLogger and progressbar 2020-04-19 17:16:51 -06:00
Eelco Dolstra 0725ab2fd7 Store more stuff in the evaluation cache 
In particular, we store whether an attribute failed to evaluate (threw
an exception) or was an unsupported type. This is to ensure that a
repeated 'nix flake show' never has to evaluate anything, so it can
execute without fetching the flake.

With this, 'nix flake show nixpkgs/nixos-20.03 --legacy' executes in
0.6s (was 3.4s).
 2020-04-19 23:07:06 +02:00
Eelco Dolstra 3738bcb05e Eval cache: Don't replace real attributes with placeholders 2020-04-18 15:12:31 +02:00
Domen Kožar 25ed842725
Merge pull request #3502 from NixOS/more-pos 
pass Pos to forceValue to improve infinite recursion error
 2020-04-18 14:05:21 +02:00
Ben Burdette 4697552948 demoing other error levels than warn/error; rename line and file fields in errPos 2020-04-17 15:50:46 -06:00
Eelco Dolstra 69cb9f7eee Wrap eval cache creation in a giant transaction 
This speeds up the creation of the cache for the nixpkgs flake from
21.2s to 10.2s. Oddly, it also speeds up querying the cache
(i.e. running 'nix flake show nixpkgs/nixos-20.03 --legacy') from 4.2s
to 3.4s.

(For comparison, running with --no-eval-cache takes 9.5s, so the
overhead of building the SQLite cache is only 0.7s.)
 2020-04-17 23:17:21 +02:00
Ben Burdette 3d5b1032a1 logError, logWarning; Logger functions; switch to Verbosity enum 2020-04-17 15:07:44 -06:00
Eelco Dolstra aaa109565e Use a more space/time-efficient representation for the eval cache 2020-04-17 23:04:21 +02:00
Eelco Dolstra bdb3226607 Add flag to disable the eval cache 2020-04-17 14:30:04 +02:00
Eelco Dolstra aa34c0ef51 nix flake show: Speed up eval cache bigly 
In the fully cached case for the 'nixpkgs' flake, it went from 101s to
4.6s. Populating the cache went from 132s to 17.4s (which could
probably be improved further by combining INSERTs).
 2020-04-17 13:57:02 +02:00
Eelco Dolstra 9ea4f93f88 nix flake show: Support apps 2020-04-17 01:21:24 +02:00
Eelco Dolstra a6c4fd044c Hide progress bar on exit 2020-04-17 01:13:13 +02:00
Eelco Dolstra 12b7eefbc5 nix flake show: Use evaluation cache 2020-04-17 01:02:29 +02:00
Eelco Dolstra 7a9687ba30 SQLiteStmt: Use std::string_view 2020-04-17 01:00:56 +02:00
Eelco Dolstra 3b489e8843 Add 'nix flake show' command 2020-04-16 19:52:39 +02:00
Eelco Dolstra 29043e7e9e Fix 2020-04-16 19:01:49 +02:00
Ben Burdette 12814806ef iomanip no longer needed 2020-04-16 10:48:15 -06:00
Eelco Dolstra c277231b7d Use RootValue 2020-04-16 18:33:34 +02:00
Eelco Dolstra f89349f07e Merge remote-tracking branch 'origin/master' into flakes 2020-04-16 18:33:10 +02:00
Eelco Dolstra 0858738355 Merge remote-tracking branch 'origin/master' into flakes 2020-04-16 18:27:37 +02:00
Eelco Dolstra efaffaa9d1 Use Logger::stdout() 
(cherry picked from commit 8f41847394)
 2020-04-16 18:14:01 +02:00
Eelco Dolstra 67a5941472 Logger: Add method for writing to stdout 
Usually this just writes to stdout, but for ProgressBar, we need to
clear the current line, write the line to stdout, and then redraw the
progress bar.

(cherry picked from commit 696c026006)
 2020-04-16 18:03:38 +02:00
Eelco Dolstra fcd048a526 Use RootValue 2020-04-16 18:02:59 +02:00
Ben Burdette 96262e744e switch to structs, which don't need public: 2020-04-16 09:55:38 -06:00
Eelco Dolstra 9f46f54de4 JSONSax: Use a RootValue 
More #3377.
 2020-04-16 17:30:18 +02:00
Eelco Dolstra 10e17eaa58 ValueMap, VectorVector: Use traceable_allocator 
We want to *trace* the 'Value *' arrays, not garbage-collect them!
Otherwise the vectors/maps can end up pointing to nowhere.

Fixes #3377. Closes #3384.
 2020-04-16 17:30:13 +02:00
Eelco Dolstra b3e5eea4a9 Add function to allocate a Value in traceable memory 2020-04-16 17:30:05 +02:00
Eelco Dolstra 1290411c2d fetchMercurial: Use inputFromAttrs() 2020-04-16 17:29:30 +02:00
Eelco Dolstra 8f41847394 Use Logger::stdout() 2020-04-16 13:47:59 +02:00
Eelco Dolstra 696c026006 Logger: Add method for writing to stdout 
Usually this just writes to stdout, but for ProgressBar, we need to
clear the current line, write the line to stdout, and then redraw the
progress bar.
 2020-04-16 13:47:59 +02:00
Domen Kožar b865b5b40c
pass Pos to forceValue to improve infinite recursion error 2020-04-16 12:32:07 +02:00
Ben Burdette 057e5b6b2e move implementation to cc 2020-04-15 10:09:43 -06:00
Ben Burdette adf03b0b8e Merge branch 'initializer-style' into error-format 2020-04-15 10:06:20 -06:00
Eelco Dolstra 2f9789c2e6
Merge pull request #3492 from andir/nix-build-gc-free 
SourceExprCommand: allocate the vSourceExpr via uncollectable memory
 2020-04-15 13:01:04 +02:00
Eelco Dolstra a118293bd0
Merge pull request #3458 from zimbatm/nix-user-conf-dir 
NIX_USER_CONF_FILES
 2020-04-15 13:00:28 +02:00
Jonas Chevalier 895516cadf
add NIX_USER_CONF_FILES 
Motivation: maintain project-level configuration files.

Document the whole situation a bit better so that it corresponds to the
implementation, and add NIX_USER_CONF_FILES that allows overriding
which user files Nix will load during startup.
 2020-04-14 18:45:06 +02:00
Eelco Dolstra 3729df34da Make Registry::read() more robust 2020-04-14 17:25:39 +02:00
Eelco Dolstra c0c2cb871d Merge remote-tracking branch 'origin/master' into flakes 2020-04-14 13:02:55 +02:00
Andreas Rammhold d2c371927e SourceExprCommand: allocate the vSourceExpr via uncollectable memory 
Previously the memory would occasionally be collected during eval since
the GC doesn't consider the member variable as alive / doesn't scan the
region of memory where the pointer lives.

By using the traceable_allocator<T> allocator provided by Boehm GC we
can ensure the memory isn't collected. It should be properly freed when
SourceExprCommand goes out of scope.
 2020-04-13 21:23:54 +02:00
Daiderd Jordan 4d9db420ff
never use /var/folders for TMPDIR on darwin 
This doesn't just cause problems for nix-store --serve but also results
in certain build failures. Builds that use unix domain sockets in their
tests often fail because the /var/folders prefix already consumes more
than half of the maximum length of socket paths.

    struct sockaddr_un {
       sa_family_t sun_family;               /* AF_UNIX */
       char        sun_path[108];            /* Pathname */
    };
 2020-04-12 09:57:22 +02:00
Domen Kožar ea2148f47c
Merge pull request #2688 from tollb/fix/build_check_keep_failed_sandbox_perms 
Fix nix-build --check -K in sandbox w/o root
 2020-04-11 20:03:17 +02:00
Domen Kožar fc144242d5
Merge pull request #3447 from DavHau/improve-tofile-error-msg 
improve toFile error message when containing potential drv path
 2020-04-11 13:12:11 +02:00
DavHau fc14213d2d improve toFile error message when containing potential drv path 2020-04-11 10:54:26 +00:00
Bruce Toll 8132d0a12e Fix nix-build --check -K in sandbox w/o root 
Temporarily add user-write permission to build directory so that it
can be moved out of the sandbox to the store with a .check suffix.

This is necessary because the build directory has already had its
permissions set read-only, but write permission is required
to update the directory's parent link to move it out of the sandbox.

Updated the related --check "derivation may not be deterministic"
messages to consistently use the real store paths.

Added test for non-root sandbox nix-build --check -K to demonstrate
issue and help prevent regressions.
 2020-04-10 16:23:10 -04:00
Eelco Dolstra e5ea01c1a8
Remove flake 'edition' field 
Future editions of flakes or the Nix language can be supported by
renaming flake.nix (e.g. flake-v2.nix). This avoids a bootstrap
problem where we don't know which grammar to use to parse
flake*.nix. It also allows a project to support multiple flake
editions, in theory.
 2020-04-10 10:24:09 +02:00
Domen Kožar db25a6d7bb
Merge pull request #2689 from tollb/fix/delete_tmp_dir_when_build_check_ok 
Delete temporary directory on successful build
 2020-04-10 09:47:57 +02:00
Bruce Toll 16a4864759 Delete temporary directory on successful build 
With --check and the --keep-failed (-K) flag, the temporary directory
was being retained regardless of whether the build was successful and
reproducible.  This removes the temporary directory, as expected, on
a reproducible check build.

Added tests to verify that temporary build directories are not
retained unnecessarily, particularly when using --check with
--keep-failed.
 2020-04-09 16:37:41 -04:00
Philipp Middendorf 04bedda0b6 gc.cc: Ignore hidden files in temproots 2020-04-09 17:05:29 +02:00
Eelco Dolstra 3aaceeb7e2
Fix build 2020-04-09 16:00:43 +02:00
Eelco Dolstra d103c79144
Merge remote-tracking branch 'origin/master' into flakes 2020-04-09 15:54:21 +02:00
Domen Kožar a693a9fa4b
Attach pos to if expression errors 2020-04-09 09:45:15 +02:00
Nikola Knezevic 7867685dcd after flake rebase 2020-04-08 22:26:57 +02:00
Nikola Knezevic f5095594e7 datatransfer.{cc,hh} -> filetransfer.{cc,hh} 2020-04-08 22:26:57 +02:00
Nikola Knezevic c330109bfa DataTransfer -> FileTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic 7848372b0f Add upload method 2020-04-08 22:26:57 +02:00
Nikola Knezevic a0c5931208 actDownload -> actDataTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic c4c1ae0a00 DownloadError -> DataTransferError 2020-04-08 22:26:57 +02:00
Nikola Knezevic 213d124277 DownloadItem -> TransferItem 2020-04-08 22:26:57 +02:00
Nikola Knezevic cd391206e6 {get,make,new}Downloader -> DataTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic 2df2741ec6 enqueueDownload -> enqueueDataTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic 142ed7fe45 DownloadResult -> DataTransferResult 2020-04-08 22:26:57 +02:00
Nikola Knezevic e5cc53beec DownloadSettings -> DataTransferSettings 2020-04-08 22:26:57 +02:00
Nikola Knezevic 741e9012d3 Rename src/lib/download.* to src/lib/datatransfer.* 2020-04-08 22:26:57 +02:00
Nikola Knezevic 65ef57e0cb DownloadRequest -> DataTransferRequest 2020-04-08 22:26:57 +02:00
Eelco Dolstra 5449ff7d8a
Merge pull request #3477 from Ninlives/nix-run-using-env 
`nix run` using $SHELL as default command
 2020-04-08 19:35:39 +02:00
Ben Burdette 805ffe1bc9 indention 2020-04-08 11:33:46 -06:00
Ben Burdette 8c2bf15c4f format -> fmt 2020-04-08 11:17:02 -06:00
Domen Kožar bf81b31559
build.cc: improve message if home directory exists 2020-04-08 18:27:10 +02:00
Ben Burdette 555baa8fb0 comments 2020-04-08 09:56:10 -06:00
Ben Burdette 54f91923c8 return of NixCode 2020-04-08 09:48:21 -06:00
Eelco Dolstra 96f3c36709
Merge pull request #3478 from edolstra/ignore-failed-data 
Downloader: Only write data to the sink on a 200 response
 2020-04-08 17:15:53 +02:00
Ben Burdette 47ed067d45 initializer style 2020-04-08 09:07:58 -06:00
Eelco Dolstra 9ed097db7b
Merge pull request #3468 from Infinisil/functionArgsPositions 
Make function arguments retain position info
 2020-04-08 15:29:39 +02:00
Eelco Dolstra 1ab8d6ac18
Downloader: Only write data to the sink on a 200 response 
Hopefully fixes #3278.
 2020-04-08 15:27:09 +02:00
Ben Burdette 00c507cc52 columnRange -> column 2020-04-07 14:36:32 -06:00
Ben Burdette 20c0984a46 remove columnrange; switch to fmt in error.cc 2020-04-07 10:14:15 -06:00
Eelco Dolstra 03a4a3c95c Merge branch 'flakes' of github.com:NixOS/nix into flakes 2020-04-07 14:30:25 +02:00
Eelco Dolstra 54955867a6 Merge remote-tracking branch 'origin/master' into flakes 2020-04-07 14:29:45 +02:00
mlatus 9d04b5da17 nix run using $SHELL as default command 2020-04-07 20:29:40 +08:00
Eelco Dolstra 55cefd41d6 Merge branch 'fetchgit-recursive' of https://github.com/blitz/nix 2020-04-07 13:45:17 +02:00
Eelco Dolstra cd39709003 Cleanup 2020-04-07 09:27:17 +02:00
Eelco Dolstra 26aeeb7653 Add FIXME 
(cherry picked from commit 2f494531b7)
 2020-04-07 09:09:43 +02:00
Eelco Dolstra a6dfa3cb85 PathInput: Add some methods 
(cherry picked from commit 78ad5b3d91)
 2020-04-07 09:09:43 +02:00
Eelco Dolstra f58a9b0e62 Respect the narHash attribute in more input types 
(cherry picked from commit a6ff66b658)
 2020-04-07 09:09:18 +02:00
Eelco Dolstra 670feb000a Add 'path' fetcher 
This fetchers copies a plain directory (i.e. not a Git/Mercurial
repository) to the store (or does nothing if the path is already a
store path).

One use case is to pin the 'nixpkgs' flake used to build the current
NixOS system, and prevent it from being garbage-collected, via a
system registry entry like this:

  {
      "from": {
          "id": "nixpkgs",
          "type": "indirect"
      },
      "to": {
          "type": "path",
          "path": "/nix/store/rralhl3wj4rdwzjn16g7d93mibvlr521-source",
          "lastModified": 1585388205,
          "rev": "b0c285807d6a9f1b7562ec417c24fa1a30ecc31a"
      },
      "exact": true
  }

Note the fake "lastModified" and "rev" attributes that ensure that the
flake gives the same evaluation results as the corresponding
Git/GitHub inputs.

(cherry picked from commit 12f9379123)
 2020-04-07 09:08:51 +02:00
Eelco Dolstra 462421d345 Backport libfetchers from the flakes branch 
This provides a pluggable mechanism for defining new fetchers. It adds
a builtin function 'fetchTree' that generalizes existing fetchers like
'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a
set of attributes, e.g.

  fetchTree {
    type = "git";
    url = "https://example.org/repo.git";
    ref = "some-branch";
    rev = "abcdef...";
  }

The existing fetchers are just wrappers around this. Note that the
input attributes to fetchTree are the same as flake input
specifications and flake lock file entries.

All fetchers share a common cache stored in
~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching
mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}).

This also adds support for Git worktrees (c169ea5904).
 2020-04-07 09:03:14 +02:00
Ben Burdette 55c96b64e4 comment cleanup 2020-04-06 20:14:48 -06:00
Ben Burdette ec449c8450 constructor style basically working 2020-04-06 19:43:22 -06:00
Ben Burdette 85f14c4582 add libutil, libexpr include dirs 2020-04-06 11:15:01 -06:00
Ben Burdette 216263c36f Merge branch 'master' into error-format 2020-04-06 10:00:00 -06:00
Eelco Dolstra 68b43e01dd
nix flake info: Show resolved URL 
This is useful for finding out what a registry lookup resolves to, e.g

  $ nix flake info patchelf
  Resolved URL:  github:NixOS/patchelf
  Locked URL:    github:NixOS/patchelf/cd7955af31698c571c30b7a0f78e59fd624d0229
 2020-04-06 14:56:13 +02:00
Eelco Dolstra ce3173edc1
nix flake info --json: Don't evaluate 
This makes its behaviour consistent with the non-json
variant. Querying the outputs should be done by another command
(e.g. 'nix search')
 2020-04-06 14:39:47 +02:00
Eelco Dolstra 2f494531b7
Add FIXME 2020-04-06 14:28:37 +02:00
Cole Helbling c976cb0b8a
Don't retry on "unsupported protocol" error 
When encountering an unsupported protocol, there's no need to retry.
Chances are, it won't suddenly be supported between retry attempts;
error instead. Otherwise, you see something like the following:

    $ nix-env -i -f git://git@github.com/foo/bar
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 335 ms
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 604 ms
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 1340 ms
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 2685 ms

With this change, you now see:

    $ nix-env -i -f git://git@github.com/foo/bar
    error: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1)
 2020-04-05 09:00:34 -07:00
Ben Burdette 1221ae3dd0 libexpr 2020-04-05 07:12:16 -06:00
Ben Burdette 9a8b3e9747 move out of tests/ 2020-04-03 14:55:26 -06:00
Ben Burdette 9bb528d392 handle Pos instead of individual file/line/columnrange args 2020-04-03 13:15:59 -06:00
Eelco Dolstra 63fa92605b nix-env: Refuse to operate on a new-style profile 
This prevents users from accidentally nuking their profile via
nix-env.

(cherry picked from commit 021634e3e3)
 2020-04-03 20:14:34 +02:00
Ben Burdette 7b7801d3f0 variadic args for hint format 2020-04-03 08:48:20 -06:00
Eelco Dolstra 6e7f252ea6 Make --override-input sticky 
When we do something like 'nix flake update --override-input nixpkgs
...', the override is now kept on subsequent calls. (If you don't want
this behaviour, you can use --no-write-lock-file.)
 2020-04-03 14:06:29 +02:00
Eelco Dolstra 9c78f7f196 getFlake: In pure mode, check that the argument is an immutable flakeref 2020-04-03 13:07:05 +02:00
Ben Burdette c6b3fcddb0 formatted with astyle 2020-04-02 16:02:40 -06:00
Ben Burdette 1c329ca433 indenting 2020-04-02 14:25:43 -06:00
Eelco Dolstra 485a87f22f Don't barf on registry parse errors 2020-04-02 19:09:17 +02:00
Eelco Dolstra a6ff66b658 Respect the narHash attribute in more input types 
call-flake.nix now passes node.info.narHash to fetchTree. This ensures
that dirty Git trees work even in pure mode.
 2020-04-02 19:04:33 +02:00
Eelco Dolstra 78ad5b3d91 PathInput: Add some methods 2020-04-02 19:04:27 +02:00
Eelco Dolstra e1fc9f6690 Improve error message 2020-04-02 19:03:32 +02:00
Eelco Dolstra ab47868639 Change lastModified to the number of seconds in the epoch 
'lastModifiedDate' is now a string representing the equivalent
date/time.
 2020-04-02 18:39:41 +02:00
Eelco Dolstra 6d6467d376 Move parseTreeInfo() 2020-04-02 18:26:39 +02:00
Andreas Rammhold 4fc4eb6c93 libexpr: remove unused attrError 
The attrError variable is no longer used but still allocated on every
call to the findAlongAttrPath function.
 2020-04-02 17:04:00 +02:00
Eelco Dolstra 6cf91d6fbd fetchTree: Support integer attributes 2020-04-02 16:51:56 +02:00
Eelco Dolstra 12f9379123 Add 'path' fetcher 
This fetchers copies a plain directory (i.e. not a Git/Mercurial
repository) to the store (or does nothing if the path is already a
store path).

One use case is to pin the 'nixpkgs' flake used to build the current
NixOS system, and prevent it from being garbage-collected, via a
system registry entry like this:

  {
      "from": {
          "id": "nixpkgs",
          "type": "indirect"
      },
      "to": {
          "type": "path",
          "path": "/nix/store/rralhl3wj4rdwzjn16g7d93mibvlr521-source",
          "lastModified": 1585388205,
          "rev": "b0c285807d6a9f1b7562ec417c24fa1a30ecc31a"
      },
      "exact": true
  }

Note the fake "lastModified" and "rev" attributes that ensure that the
flake gives the same evaluation results as the corresponding
Git/GitHub inputs.
 2020-04-02 14:56:20 +02:00
Eelco Dolstra 00e1400eb7 Doh 2020-04-02 11:55:41 +02:00
Eelco Dolstra 021634e3e3 nix-env: Refuse to operate on a new-style profile 
This prevents users from accidentally nuking their profile via
nix-env.
 2020-04-02 11:54:48 +02:00
Eelco Dolstra ed13457dbf nix flake info --json: Show TreeInfo 2020-04-02 11:51:34 +02:00
Silvan Mosberger c34e96f7e0
Make function arguments retain position info 
This allows querying the location of function arguments. E.g.

  builtins.unsafeGetAttrPos "x" (builtins.functionArgs ({ x }: null))

  => { column = 57; file = "/home/infinisil/src/nix/inst/test.nix"; line = 1; }
 2020-04-02 05:52:52 +02:00
Ben Burdette e697884f65 using std:: everywhere; fix a formatting error; add exception flags 2020-04-01 21:30:19 -06:00
Ben Burdette dd7b8183a5 indenting 2020-04-01 16:20:20 -06:00
Ben Burdette 8713aeac5e remove using std::*, switch to include guard 2020-04-01 15:51:14 -06:00
Eelco Dolstra 74024515a3 Support registry entries that must match exactly 
An example use is for pinning the "nixpkgs" entry the system-wide
registry to a particular store path. Inexact matches
(e.g. "nixpkgs/master") should still use the global registry.
 2020-04-01 23:12:45 +02:00
Eelco Dolstra bd10a07d17 Registry: Use a struct instead of a tuple for entries 2020-04-01 23:03:27 +02:00
Eelco Dolstra 77ffaea4fa Add a system-wide flake registry /etc/nix/registry.json 
One application for this is pinning the 'nixpkgs' flake to the exact
revision used to build the NixOS system, e.g.

  {
      "flakes": [
          {
              "from": {
                  "id": "nixpkgs",
                  "type": "indirect"
              },
              "to": {
                  "owner": "NixOS",
                  "repo": "nixpkgs",
                  "type": "github",
                  "rev": "b0c285807d6a9f1b7562ec417c24fa1a30ecc31a"
              }
          }
      ],
      "version": 2
  }
 2020-04-01 22:56:50 +02:00
Eelco Dolstra 4fdec5f61d Merge remote-tracking branch 'origin/master' into flakes 2020-04-01 00:03:50 +02:00
Eelco Dolstra 03b56e96bf Typo 2020-03-31 23:55:07 +02:00
Ben Burdette 5b3aefff85 add some explanatory comments 2020-03-31 12:42:41 -06:00
Ben Burdette 9e7b89bf10 rename errors/warnings 2020-03-31 11:56:37 -06:00
Ben Burdette 09652f597c enum style 2020-03-31 09:36:20 -06:00
Eelco Dolstra a7540294cf
Merge pull request #3460 from NixOS/dev-shell 
Backport 'nix dev-shell' from the flakes branch
 2020-03-31 14:46:15 +02:00
Eelco Dolstra 3166b97174 nix shell -> nix dev-shell 2020-03-31 13:45:28 +02:00
mlatus 12556e5709 fix placeholder not substituted in passAsFile 2020-03-31 19:40:16 +08:00
John Ericson 7e9a2718f0 s/outputHashRecursive/ingestionMethod/c 2020-03-30 22:36:15 +00:00
John Ericson 51afea3af2 Never cast FileIngestionMethod to or from boolean 2020-03-30 22:31:51 +00:00
John Ericson c251b011cd Merge remote-tracking branch 'upstream/master' into enum-FileIngestionMethod 2020-03-30 18:16:44 -04:00
John Ericson bbbb7c1bc7 Use auto with some FileIngestionMethod local variables 2020-03-30 18:15:55 -04:00
Eelco Dolstra e1a94ad852 Backport 'nix dev-shell' from the flakes branch 
This also adds a '--profile' option to 'nix build' (replacing 'nix-env
--set').
 2020-03-30 19:16:45 +02:00
John Ericson 832bd534dc Store parsed hashes in DerivationOutput 
It's best to detect invalid data as soon as possible, with data types
that make storing it impossible.
 2020-03-30 11:33:35 -04:00
Ben Burdette 35c7bab09a build with make 2020-03-30 09:14:29 -06:00
John Ericson f5494d9442 Merge remote-tracking branch 'me/enum-FileIngestionMethod' into HEAD 2020-03-30 11:08:13 -04:00
Eelco Dolstra 367577d9a6 Fix macOS build 2020-03-30 17:00:40 +02:00
Eelco Dolstra d15d91cad1 Makefile cleanup 2020-03-30 16:50:13 +02:00
Eelco Dolstra e322a16523 Remove global -I flags 
(cherry picked from commit 2c692a3b14)
 2020-03-30 15:30:19 +02:00
Eelco Dolstra 2c692a3b14 Remove global -I flags 2020-03-30 14:39:33 +02:00
Eelco Dolstra e0a0ae0467 Move fetchers from libstore to libfetchers 2020-03-30 14:04:53 +02:00
Eelco Dolstra 2287cc6486 Fix segfault 2020-03-30 13:31:55 +02:00
Julian Stecklina 40c023ecfe fetchGit: don't use std::filesystem to filter git repos 
Using std::filesystem means also having to link with -lstdc++fs on
some platforms and it's hard to discover for what platforms this is
needed. As all the functionality is already implemented as utilities,
use those instead.
 2020-03-30 00:32:42 +02:00
Bjørn Forsman f686efeed4 fetchGit: fix submodule corner case by fetching all refs from cacheDir 
Due to fetchGit not checking if rev is an ancestor of ref (there is even
a FIXME comment about it in the code), the cache repo might not have the
ref even though it has the rev. This doesn't matter when submodule =
false, but the submodule = true code blows up because it tries to fetch
the (missing) ref from the cache repo.

Fix this in the simplest way possible: fetch all refs from the local
cache repo when submodules = true.

TODO: Add tests.
 2020-03-29 22:29:58 +02:00
Bjørn Forsman cc522d0d23 fetchGit: fix submodules = true for dirty trees 2020-03-29 22:29:58 +02:00
Bjørn Forsman b306b7039e fetchGit: checkout rev instead of latest ref 
Major bugfix for the submodules = true code path.

TODO: Add tests.
 2020-03-29 22:29:58 +02:00
Bjørn Forsman 369fffd6f1 fetchGit: add submodules attribute to the .link file 
The .link file is used as a lock, so I think we should put the
"submodule" attribute in there since turning on submodules creates a new
.link file path.
 2020-03-29 22:29:58 +02:00
Bjørn Forsman 6864ad7cf5 fetchGit: fix submodule output attribute 
Before this change it would be false for all evaluations but the first.
Now it follows the input argument (as it should).
 2020-03-29 22:29:58 +02:00
Bjørn Forsman 002a3a95dc fetchGit: fix "fatal: couldn't find remote ref refs/heads/master" issue with submodules 2020-03-29 22:29:58 +02:00
Julian Stecklina cc4fe977e5 Link to stdc++fs 
Some platforms seem to still require linking with stdc++fs to enable
STL std::filesystem support.
 2020-03-29 22:29:58 +02:00
Julian Stecklina 435366ed3c Rename fetchGit fetchSubmodules to just submodules 2020-03-29 22:29:58 +02:00
Julian Stecklina c8d33de777 Add git submodule fixes from @bjornfor 
This fixes fetching repositories with no submodules and also cleans up
.git files in checkouts.
 2020-03-29 22:29:57 +02:00
Julian Stecklina f58604ac32 Add fetchSubmodules to builtins.fetchGit 
There are some downsides to this features:

 - Submodules are not cached (unlike the root repo),
 - Full checkouts are created in a temporary directory.
 2020-03-29 22:29:57 +02:00
John Ericson 225e62a56a Replace some bool recursive with a new FileIngestionMethod enum 2020-03-29 15:16:20 -04:00
John Ericson 87b32bab05 Use enum struct and drop prefixes 
This does a few enums; the rest will be gotten in subsequent commits.
 2020-03-29 11:23:15 -04:00
Eelco Dolstra 4989c04dd2 nix flake info --json: Dump attr-style flakeref 2020-03-28 22:59:38 +01:00
Eelco Dolstra 2fccef0c59 Warn about --override-input / --update-input flags that don't match an input 2020-03-28 19:09:36 +01:00
Eelco Dolstra 2287e2f279 nix flake info: Show flake subdirectory 2020-03-28 18:05:50 +01:00
Eelco Dolstra 015f8f1c13 Improve lock file generation 
This is now done in a single pass. Also fixes some issues when
updating flakes with circular dependencies. Finally, when using
'--recreate-lock-file --commit-lock-file', the commit message now
correctly shows the differences.
 2020-03-27 21:08:41 +01:00
Ben Burdette 759f39800b remove util.hh from deps 2020-03-27 10:55:09 -06:00
Ben Burdette 00eb3fcb7a more cleanup 2020-03-27 10:13:46 -06:00
Ben Burdette a3ef00be6c camelcase; optional hint 2020-03-27 10:03:02 -06:00
Eelco Dolstra 3fa1e7dace Fix diffLockFiles() 2020-03-27 16:15:50 +01:00
John Ericson e433d4af4c Extend Rust FFI 
Do idiomatic C++ copy and move constructors for a few things, so
wrapping structs' defaults can work.
 2020-03-25 16:12:14 -04:00
Ben Burdette d44c9c5581 some colors 2020-03-25 11:20:44 -06:00
Ben Burdette 3582dc3c88 programName as static member var 2020-03-25 10:52:03 -06:00
John Ericson bcde5456cc Flip dependency so store-api.hh includes derivations.hh 
I think it makes more sense to define the data model (derivations),
before the operations (store api).
 2020-03-24 20:39:45 +00:00
Ben Burdette fc310eda3a switch to one level of builder function, not subobject functions 2020-03-24 14:24:57 -06:00
Ben Burdette 0166e7ab6d MkNixCode, MkErrLine approach 2020-03-24 11:21:35 -06:00
Ben Burdette 4171ab4bbd renaming 2020-03-24 09:18:23 -06:00
Eelco Dolstra 0a10854f85 Misc changes from the flakes branch 2020-03-24 14:34:47 +01:00
Eelco Dolstra c85097da7c Fix --refresh with --no-net 
https://hydra.nixos.org/build/110879699
(cherry picked from commit 5bbe793abf)
 2020-03-24 14:26:23 +01:00
Eelco Dolstra 6b824c78f1 nix: Add --refresh as an alias for --tarball-ttl 0 
(cherry picked from commit e721f99817)
 2020-03-24 14:26:23 +01:00
Eelco Dolstra 777e21e596 nix path-info --json: Print hash in SRI format 
(cherry picked from commit 442e665d6d)
 2020-03-24 14:26:23 +01:00
Eelco Dolstra 7a8de57d3e Pretty-print 'nix why-depends' / 'nix-store -q --tree' output 
Extracted from 678301072f.
 2020-03-24 14:26:23 +01:00
Eelco Dolstra 4260a22a55 absPath(): Use std::optional 
(cherry picked from commit 1bf9eb21b7)
 2020-03-24 14:25:28 +01:00
Eelco Dolstra f9611c7ae4 buildenv: Eliminate global variables, other cleanup 
(cherry picked from commit b82f75464d)
 2020-03-24 14:06:47 +01:00
Eelco Dolstra 76e7d958ed Fix coverage build 
https://hydra.nixos.org/build/110757285
(cherry picked from commit b430a81a1f)
 2020-03-24 14:06:47 +01:00
Eelco Dolstra 231a8aa2c2 nix edit: Support non-derivation attributes 
E.g.

  $ nix edit .#nixosConfigurations.bla

now works.

(cherry picked from commit d2032edb2f)
 2020-03-24 14:06:47 +01:00
Eelco Dolstra c1ca4f0acc findAlongAttrPath(): Return position 
(cherry picked from commit 0b013a54dc)
 2020-03-24 14:06:47 +01:00
Eelco Dolstra 1eb952d27a findAlongAttrPath(): Throw AttrPathNotFound 
(cherry picked from commit 6b0ca8e803)
 2020-03-24 14:06:47 +01:00
Eelco Dolstra edc34cc1a2 Add function for quoting strings 
(cherry picked from commit 7dcf5b011a)
 2020-03-24 13:44:04 +01:00
Eelco Dolstra 5a7e7fc35f Use std::string_view 
(cherry picked from commit 6529490cc1)
 2020-03-24 13:26:37 +01:00
Eelco Dolstra c34a20e1f6 EvalState::allocAttr(): Add convenience method 
(cherry picked from commit c02da99757)
 2020-03-24 13:26:37 +01:00
Ben Burdette aadd59d005 error test 2020-03-23 15:29:49 -06:00
Eelco Dolstra 7abe3bde8a Merge remote-tracking branch 'origin/master' into flakes 2020-03-23 13:27:54 +01:00
Eelco Dolstra 5885e20404
Merge pull request #3429 from LnL7/darwin-sandbox 
darwin sandbox
 2020-03-23 09:13:36 +01:00
Ben Burdette f694f43d7d straightforward port of rust mockup code 2020-03-22 12:25:47 -06:00
Daiderd Jordan 2e9bc1245c
sandbox: fix /bin/sh on catalina 
Sadly 10.15 changed /bin/sh to a shim which executes bash, this means it
can't be used anymore without also opening up the sandbox to allow bash.

    Failed to exec /bin/bash as variant for /bin/sh (1: Operation not permitted).
 2020-03-20 22:12:30 +01:00
Daiderd Jordan f6c122aaeb
sandbox: allow pty devices 
Nix now runs builds with a pseudo-terminal to enable colored build
output.
 2020-03-20 21:58:45 +01:00
Daiderd Jordan 7f2df903d9
libstore: relax default sandbox-paths on darwin 2020-03-20 21:31:20 +01:00
Daiderd Jordan afb78ebd34
libstore: disable resolve-system-dependencies hook 
This is used to determine the dependency tree of impure libraries so nix
knows what paths to open in the sandbox.  With the less restrictive
defaults it isn't needed anymore.
 2020-03-20 21:21:56 +01:00
Eelco Dolstra 1537e270fb Merge remote-tracking branch 'origin/master' into flakes 2020-03-20 13:15:11 +01:00
John Ericson e317324236
Apply suggestions from code review 2020-03-19 23:38:51 -04:00
John Ericson d5b3328dd1
Apply suggestions from code review 
Co-Authored-By: Cole Helbling <cole.e.helbling@outlook.com>
 2020-03-19 23:37:52 -04:00
jakobrs c5a488afc0 Remove the --delete option for --gc 
Running `nix-store --gc --delete` will, as of Nix 2.3.3, simply fail
because the --delete option conflicts with the --delete operation.

  $ nix-store --gc --delete
  error: only one operation may be specified
  Try 'nix-store --help' for more information.

Furthermore, it has been broken since at least Nix 0.16 (which was
released sometime in 2010), which means that any scripts which depend
on it should have been broken at least nine years ago. This commit
simply formally removes the option. There should be no actual difference
in behaviour as far as the user is concerned: it errors with the exact
same error message. The manual has been edited to remove any references
to the (now gone) --delete option.

Other information:
* Path for Nix 0.16 used:
  /nix/store/rp3sgmskn0p0pj1ia2qwd5al6f6pinz4-nix-0.16
 2020-03-19 19:41:23 +01:00
John Ericson f1cf3ab870 hashDerivationModulo: Generalize for multiple fixed ouputs per drv 
See documentattion in header and comments in implementation for details.

This is actually done in preparation for floating ca derivations, not
multi-output fixed ca derivations, but the distinction doesn't yet
mattter.

Thanks @cole-h for finding and fixing a bunch of typos.
 2020-03-19 10:30:49 -04:00
Eelco Dolstra ef74fafc03
nix repl: Put EvalState on the heap 
See 0629601da1.
 2020-03-19 13:52:28 +01:00
Eelco Dolstra b244e65cdb
nix repl: Scan NixRepl for GC roots 
Fixes #3175.
 2020-03-19 13:50:01 +01:00
Eelco Dolstra 8aa354fdfd
Register flake-registry.json as a GC root again 2020-03-19 11:45:34 +01:00
Eelco Dolstra 1c127e6a82
downloadFile(): Use expired file if the download fails 2020-03-19 11:42:50 +01:00
John Ericson 049179ba07
Fix typos 
Thanks @asymmetric

I failed to do them all in one batch

Co-Authored-By: asymmetric <lorenzo@mailbox.org>
 2020-03-18 19:07:05 -04:00
Eelco Dolstra f6ddf48882
Get rid of downloadCached() 
Everything uses the generic caching system now.
 2020-03-18 17:24:19 +01:00
Eelco Dolstra c5ec95e2c7
tarball.cc: Use ETags 2020-03-18 15:14:23 +01:00
Eelco Dolstra 1b49479836
Remove flake closure caching 
This is not compatible with lazy flake input fetching.
 2020-03-18 14:11:58 +01:00
Eelco Dolstra 1e7ce1d6da
tarball / github fetchers: Use generic caching system 2020-03-18 14:08:25 +01:00
Eelco Dolstra 38e360154d
Git: Use unified caching system 2020-03-17 22:35:29 +01:00
Eelco Dolstra d1165d8791
Require shallow clones to be requested explicitly 
If you do a fetchTree on a Git repository, whether the result contains
a revCount attribute should not depend on whether that repository
happens to be a shallow clone or not. That would complicate caching a
lot and would be semantically messy. So applying fetchTree/fetchGit to
a shallow repository is now an error unless you pass the attribute
'shallow = true'. If 'shallow = true', we don't return revCount, even
if the repository is not actually shallow.

Note that Nix itself is not doing shallow clones at the moment. But it
could do so as an optimisation if the user specifies 'shallow = true'.

Issue #2988.
 2020-03-17 22:35:29 +01:00
Eelco Dolstra 2a4e4f6a6e
Unified fetcher caching system 2020-03-17 22:35:29 +01:00
John Ericson e5178fd22d
Fix typos 
Thanks @asymmetric!

Co-Authored-By: asymmetric <lorenzo@mailbox.org>
 2020-03-16 16:40:13 -04:00
Eelco Dolstra fbcb897e21
Add a test for shallow Git clones 
Also, don't return a revCount anymore for shallow or dirty Git trees,
since it's incorrect.

Closes #2988.
 2020-03-16 13:20:32 +01:00
John Ericson 2be64efb02 Generalize isFixedOutput in preparation for CA drvs 
Today's fixed output derivations and regular derivations differ in a few
ways which are largely orthogonal. This replaces `isFixedOutput` with a
`type` that returns an enum of possible combinations.
 2020-03-15 11:05:37 -04:00
John Ericson 68fe0d9809 Add missing #include <regex> 2020-03-13 21:24:35 -04:00
Eelco Dolstra 779ef8f5ef
Merge pull request #3380 from contrun/no-attr-path-for-installed 
display attr-path only when queried available
 2020-03-13 19:26:20 +01:00
YI b6d794fb8d display attr-path only when queried available 2020-03-14 00:36:26 +08:00
Eelco Dolstra 0c2088d438 Merge remote-tracking branch 'origin/master' into flakes 2020-03-13 17:03:30 +01:00
Eelco Dolstra b816515f61
Fix ca-references feature check 
Fixes #3406.
 2020-03-13 13:15:51 +01:00
Eelco Dolstra ae9119167e Change the lock file to a graph 
This enables support for cycles between flakes.
 2020-03-12 22:06:57 +01:00
Eelco Dolstra d048577909
Merge pull request #3403 from hercules-ci/issue-3398-path-info-cache-ttls 
pathInfoCache: Respect disk cache TTLs #3398
 2020-03-12 11:43:31 +01:00
Robert Hensing 3f55f8a8fb pathInfoCache: Respect disk cache TTLs #3398 2020-03-12 10:30:28 +01:00
Will Dietz 15edd2349e local.mk: fix user-env.cc dep on buildenv.nix.gen.hh, resolve occasional build failure 2020-03-12 00:51:56 +01:00
Eelco Dolstra e188fe7c6d Move call-flake.nix into libexpr 2020-03-11 17:04:51 +01:00
Eelco Dolstra 35f6651735 Merge remote-tracking branch 'origin/master' into flakes 2020-03-11 17:03:38 +01:00
Eelco Dolstra 9950cdec35 Move some corepkgs into the nix binary 2020-03-11 16:57:48 +01:00
Eelco Dolstra e02481ded2 parseExprFromString(): Use std::string_view 2020-03-11 16:56:29 +01:00
Eelco Dolstra 97d1c7f932 Merge remote-tracking branch 'origin/master' into flakes 2020-03-11 15:17:21 +01:00
Eelco Dolstra cfc38257cf Fix flake subdirectory handling 2020-03-10 19:21:47 +01:00
Eelco Dolstra 8a1d8701f6
nix-store -q --graph: Fix edges 
Fixes #3389.
 2020-03-10 11:11:46 +01:00
Eelco Dolstra 983fab7ea9
dotgraph.cc: Remove dead code 2020-03-10 11:06:55 +01:00
Eelco Dolstra 73769b28e3 Move calling flakes into a Nix helper function (call-flake.nix) 2020-03-09 15:28:41 +01:00
Eelco Dolstra 5a1514adb8 Add 'flake' attribute to lock files to denote non-flakes 2020-03-09 15:27:49 +01:00
Eelco Dolstra 73b6d87e17
Merge remote-tracking branch 'origin/master' into flakes 2020-03-04 13:58:42 +01:00
Eelco Dolstra d37dc71e3c
nix-build: Fix !<output> handling 
This was broken by 22a754c091.

https://hydra.nixos.org/eval/1573669
 2020-03-04 13:56:17 +01:00
Eelco Dolstra 75db069f92
Optimise Derivation::unparse() 
In

  nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux

this reduces time spent in unparse() from 9.15% to 4.31%. The main
culprit was appending characters one at a time to the destination
string. Even though the string has enough capacity, push_back() still
needs to check this on every call.
 2020-03-04 11:44:45 +01:00
Eelco Dolstra 401b5bc541
builtins.cache: Cache regular expressions 
The evaluator was spending about 1% of its time compiling a small
number of regexes over and over again.
 2020-03-04 11:44:33 +01:00
Eelco Dolstra 22a754c091
Fix GC failures on bad store path names 
It failed on names like '/nix/store/9ip48nkc9rfy0a4yaw98lp6gipqlib1a-'.
 2020-02-28 18:07:10 +01:00
Eelco Dolstra f443d5ca19 Merge remote-tracking branch 'origin/master' into flakes 2020-02-27 15:19:58 +01:00
Eelco Dolstra 2672a28bb4 nix dev-shell: Add --command option 
Note: like 'nix run', and unlike 'nix-shell', this takes an argv
vector rather than a shell command. So

  nix dev-shell -c 'echo $PATH'

doesn't work. Instead you need to do

  nix dev-shell -c bash -c 'echo $PATH'
 2020-02-27 15:17:37 +01:00
Eelco Dolstra 4a4521f462 Fix nlohmann::json exception 2020-02-20 23:47:02 +01:00
Eelco Dolstra 73c9840569 Restore subdir -> dir 
Got this mixed up somewhere.
 2020-02-20 23:44:06 +01:00
Eelco Dolstra d068f9ffff Restore subdir support in registries 
Hacky...
 2020-02-20 22:14:44 +01:00
Eelco Dolstra 890df325c7
fetchTree: Use a feature flag 2020-02-20 13:36:16 +01:00
Eelco Dolstra 2e953b567e
Merge pull request #3325 from xzfc/clean-tmpdir 
nix-shell: clean up the tmpDir and escape variables
 2020-02-19 21:29:18 +01:00
Eelco Dolstra 4ad5826a18 nix eval-hydra-jobs: Remove 
On second thought, let's move this back to Hydra.
 2020-02-19 16:12:49 +01:00
Albert Safin f2a03acf3f nix-shell: clean up the tmpDir and escape variables 
The problem fixed: each nix-shell invocation creates a new temporary
directory (`/tmp/nix-shell-*`) and never cleans up.

And while I'm here, shellescape all variables inlined into the rcfile.
See what might happen without escaping:

    $ export TZ="';echo pwned'"
    $ nix-shell -p hello --run hello
    pwned
    Hello, world!
 2020-02-19 14:28:49 +00:00
Eelco Dolstra 1d99c4ab25
Merge pull request #3229 from Ma27/flakes-fetchgit-worktree-support 
builtins.fetchGit: Fix build when fetching a git worktree
 2020-02-19 14:11:18 +01:00
Maximilian Bosch c169ea5904
builtins.fetchGit: Fix build when fetching a git worktree 
Worktrees[1] are a feature of git which allow you to check out a ref in
a different directory.

While playing around with flakes I realized that git repositories in a
worktree checkout break when trying to build a flake:

```
$ git worktree add ../nixpkgs-flakes nixpkgs-flakes
$ cd ../nixpkgs-flakes
$ nix build .#hello
error: opening directory '/home/ma27/Projects/nixpkgs-flakes/.git/refs/heads': Not a directory
```

This issue has been fixed by determining with `git rev-parse --git-common-dir`
where the actual `.git` directory is.

Please note that this issue only exists on the `flakes` branch, fetching
worktree checkouts with Nix master seems to work fine.

[1] https://git-scm.com/docs/git-worktree
 2020-02-19 14:00:36 +01:00
Eelco Dolstra 2a14c28669
Merge pull request #3357 from carlosdagos/pure-nix-shell-proxy-env 
Pass through http proxy env vars in pure shell
 2020-02-19 13:02:02 +01:00
Eelco Dolstra f3505a7899 Merge remote-tracking branch 'origin/master' into flakes 2020-02-19 12:57:45 +01:00
Eelco Dolstra 30c8297ded nix eval-hydra-jobs: Add feature 2020-02-19 12:35:03 +01:00
Eelco Dolstra c4d3674de6
Merge pull request #3353 from tbsmoest/priv_tobias_pr_set_deathsig-1.4 
Fix PR_SET_PDEATHSIG results in Broken pipe (#2395)
 2020-02-19 12:29:12 +01:00
Eelco Dolstra 95468e3c1e Fix nixpkgs.<attr> warning 2020-02-19 00:09:42 +01:00
Eelco Dolstra edee6169bf nix eval-hydra-jobs: Fix aggregate derivation name 2020-02-18 22:05:49 +01:00
Eelco Dolstra 1351101c28 nix eval-hydra-jobs: Check aggregate jobs in --dry-run mode 2020-02-18 19:26:11 +01:00
Eelco Dolstra 553e584f92
LocalStore::checkDerivationOutputs(): Improve error message 2020-02-18 17:51:48 +01:00
Eelco Dolstra d8fd31f50f
Disable the progress bar if $TERM == dumb or unset 
Fixes #3363.
 2020-02-18 17:51:18 +01:00
Eelco Dolstra 8f9dcfc671
Disable the progress bar if $TERM == dumb or unset 
Fixes #3363.
 2020-02-18 17:47:53 +01:00
Eelco Dolstra b5e3c04c03
Fix URL parser 
Fixes #3361.
 2020-02-18 12:51:26 +01:00
Eelco Dolstra 6529490cc1 nix eval-hydra-jobs: Support job names as aggregate constituents 
Fixes https://github.com/NixOS/hydra/issues/715.
 2020-02-17 15:53:59 +01:00
Eelco Dolstra 8a78bcf6a2 LocalStore::checkDerivationOutputs(): Improve error message 2020-02-17 15:46:07 +01:00
Eelco Dolstra b0336e7cf7 nix eval-hydra-job: Progress indicator 2020-02-17 14:33:10 +01:00
Eelco Dolstra c6e63065f3 nix eval-hydra-jobs: Improve error handling 2020-02-17 14:31:29 +01:00
Eelco Dolstra eb19ff3b82 nix eval-hydra-jobs: Support flakes 2020-02-14 23:23:41 +01:00
Eelco Dolstra 4c24263967 nix eval-hydra-jobs: Support parallel evaluation 
Example usage:

  $ nix eval-hydra-jobs -f '<nixpkgs/pkgs/top-level/release.nix>' '' \
      --max-memory-size 2048 --workers 8
 2020-02-14 23:05:49 +01:00
Eelco Dolstra e375da6899 Add 'nix eval-hydra-jobs' command 2020-02-14 23:00:42 +01:00
Eelco Dolstra 46a284263f Fix build 2020-02-14 22:45:33 +01:00
Eelco Dolstra 6208d24c38 Merge remote-tracking branch 'origin/master' into flakes 2020-02-14 22:42:35 +01:00
Tobias Möst 3e347220c8 Fix PR_SET_PDEATHSIG results in Broken pipe (#2395) 
The ssh client is lazily started by the first worker thread, that
requires a ssh connection. To avoid the ssh client to be killed, when
the worker process is stopped, do not set PR_SET_PDEATHSIG.
 2020-02-14 07:51:44 +01:00
Carlos D d78141a886 Pass through http proxy env vars in pure shell 2020-02-14 16:11:22 +11:00
Eelco Dolstra 9af10b753c Bindings::get(): std::optional<Attr *> -> Attr * 
Returning a nullable type in an optional is silly.
 2020-02-13 17:15:05 +01:00
Eelco Dolstra d8972317fc Prevent uninitialized StorePath creation 2020-02-13 16:12:16 +01:00
Eelco Dolstra 26dacc0983 Add fetchTree builtin function 
This allows all supported fetchers to be used, e.g.

  builtins.fetchTree {
    type = "github";
    owner = "NixOS";
    repo = "nix";
    rev = "d4df99a3349cf2228a8ee78dea320afef86eb3ba";
  }
 2020-02-11 23:53:46 +01:00