lunaphied/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
9170 commits 2 branches 3 tags 72 MiB
Commit graph

5705 commits

Author SHA1 Message Date
Eelco Dolstra f63839bfa4 Cleanup 2020-07-29 18:04:51 +02:00
Eelco Dolstra 39311e7009
Merge pull request #3549 from Ma27/fetchgit-hash 
Merge legacy `fetchGit`-builtin with the generic `fetchTree`-function
 2020-07-29 11:33:27 +02:00
Eelco Dolstra 7c097275c4
Merge pull request #3877 from matthewbauer/develop-continuous-regex 
Set regex_constants::match_continuous for quicker search in nix develop
 2020-07-29 10:07:52 +02:00
Matthew Bauer fa8515d7ec Set regex_constants::match_continuous for quicker search 
match_continuous limits the search to the current start position,
instead of searching the entire file.

On libc++, this improves performance dramatically:

$ time /nix/store/70ai68dfm6xbzwn26j5n4li9di52ylia-nix-3.0pre20200728_c159f48/bin/nix print-dev-env >/dev/null
/nix/store/70ai68dfm6xbzwn26j5n4li9di52ylia-nix-3.0pre20200728_c159f48/bin/ni  2.39s user 0.19s system 64% cpu 4.032 total
$ time /nix/store/cwjfxxlp83zln4mfyy1d2dbsx7f6s962-nix-3.0pre20200728_dirty/bin/nix print-dev-env >/dev/null
/nix/store/cwjfxxlp83zln4mfyy1d2dbsx7f6s962-nix-3.0pre20200728_dirty/bin/nix   0.09s user 0.05s system 65% cpu 0.204 total

Fixes #3874
 2020-07-29 00:57:20 -05:00
Eelco Dolstra 4c0077a07d Fix RemoteStore::addToStore() latency 
Since 6185d25e52, this was very
latency-bound since it required a round-trip for every 32 KiB. So for
example copying a 514 MiB closure over a virtual ethernet device with
a articial delay of just 1 ms took 343s. Now it takes 2.7s.

Fixes #3372.
 2020-07-29 00:48:39 +02:00
Eelco Dolstra c159f48a39 Cleanup 2020-07-29 00:24:55 +02:00
John Ericson df707d05d1 Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca 2020-07-28 21:12:36 +00:00
Maximilian Bosch e4940e90f3
Restore backwards-compat for current builtins.fetchGit 
If a repo is dirty, it used to return a `rev` object with an "empty"
sha1 (0000000000000000000000000000000000000000). Please note that this
only applies for `builtins.fetchGit` and *not* for `builtins.fetchTree{
type = "git"; }`.
 2020-07-28 22:46:39 +02:00
Carlo Nucera 2980b244b7 Use assert(false) instead of abort() 2020-07-28 15:39:45 -04:00
Carlo Nucera cfe96bd7ce Merge branch 'master' of github.com:NixOS/nix into ca-drv 2020-07-28 15:36:45 -04:00
Carlo Nucera c318d398f3 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into new-interface-for-path-pathOpt 2020-07-28 14:22:24 -04:00
Carlo Nucera 7ef1e3cd14 Use the new interface 2020-07-28 13:59:24 -04:00
Carlo Nucera d564ac1c50 Offer a safer interface for pathOpt 
The new interface we offer provides a way of getting all the
DerivationOutputs with the storePaths directly, based on the observation
that it's the most common usecase.
 2020-07-28 13:55:15 -04:00
Eelco Dolstra b8eea7e81a Remove putBytes 
istream->tellg() returns -1 so we can't get the number of bytes
written.

Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.
 2020-07-28 13:27:56 +02:00
Maximilian Bosch f742438465
Merge legacy fetchGit-builtin with the generic fetchTree-function 
The original idea was to implement a git-fetcher in Nix's core that
supports content hashes[1]. In #3549[2] it has been suggested to
actually use `fetchTree` for this since it's a fairly generic wrapper
over the new fetcher-API[3] and already supports content-hashes.

This patch implements a new git-fetcher based on `fetchTree` by
incorporating the following changes:

* Removed the original `fetchGit`-implementation and replaced it with an
  alias on the `fetchTree` implementation.

* Ensured that the `git`-fetcher from `libfetchers` always computes a
  content-hash and returns an "empty" revision on dirty trees (the
  latter one is needed to retain backwards-compatibility).

* The hash-mismatch error in the fetcher-API exits with code 102 as it
  usually happens whenever a hash-mismatch is detected by Nix.

* Removed the `flakes`-feature-flag: I didn't see a reason why this API
  is so tightly coupled to the flakes-API and at least `fetchGit` should
  remain usable without any feature-flags.

* It's only possible to specify a `narHash` for a `git`-tree if either a
  `ref` or a `rev` is given[4].

* It's now possible to specify an URL without a protocol. If it's missing,
  `file://` is automatically added as it was the case in the original
  `fetchGit`-implementation.

[1] https://github.com/NixOS/nix/pull/3216
[2] https://github.com/NixOS/nix/pull/3549#issuecomment-625194383
[3] https://github.com/NixOS/nix/pull/3459
[4] https://github.com/NixOS/nix/pull/3216#issuecomment-553956703
 2020-07-28 00:44:39 +02:00
John Ericson 8065c6d160 Abstract out topo sorting logic 2020-07-27 20:45:34 +00:00
John Ericson 951415b568 Require ca-derivations everywhere we create a CA derivation 
"create" as in read one in from a serialized form, or build one from
scratch in memory.
 2020-07-27 17:56:36 +00:00
John Ericson e32a9e124b Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-27 17:50:06 +00:00
John Ericson 7cf978440c Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-27 17:47:40 +00:00
John Ericson e6115ca004 Merge remote-tracking branch 'upstream/master' into ca-derivation-data-types 2020-07-27 17:41:43 +00:00
John Ericson 1d7d94ceea Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-27 16:17:50 +00:00
John Ericson 43f2bd8dc5 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 16:13:57 +00:00
John Ericson 387699fe7f Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-27 15:00:28 +00:00
John Ericson 78466bcb2f Merge branch 'optional-derivation-output-storepath' into ca-derivation-data-types 2020-07-27 14:40:08 +00:00
John Ericson d5bb67cfa4 Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-27 14:29:32 +00:00
John Ericson 699fc89b39 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 14:22:45 +00:00
Maximilian Bosch 6ccfdb79c7
libutil/logging: extend internal-json logger to make it more machine-readable 
The new error-format is pretty nice from a UX point-of-view, however
it's fairly hard to parse the output e.g. for editor plugins such as
vim-ale[1] that use `nix-instantiate --parse` to determine syntax errors in
Nix expression files.

This patch extends the `internal-json` logger by adding the fields
`line`, `column` and `file` to easily locate an error in a file and the
field `raw_msg` which contains the error-message itself without
code-lines and additional helpers.

An exemplary output may look like this:

```
[nix-shell]$ ./inst/bin/nix-instantiate ~/test.nix --log-format minimal
{"action":"msg","column":1,"file":"/home/ma27/test.nix","level":0,"line":4,"raw_msg":"syntax error, unexpected IF, expecting $end","msg":"<full error-msg with code-lines etc>"}
```

[1] https://github.com/dense-analysis/ale
 2020-07-26 02:29:57 +02:00
John Ericson 2c7557481b queryDerivationOutputMap no longer assumes all outputs have a mapping 
This assumption is broken by CA derivations. Making a PR now to do the
breaking daemon change as soon as possible (if it is already too late,
we can bump protocol intead).
 2020-07-24 21:14:06 +00:00
Eelco Dolstra a71d1cedff
printVersion(): Show system types 2020-07-24 11:34:01 +02:00
Eelco Dolstra 2292814049
createUnixDomainSocket(): Fix off-by-one error in copying the socket path 
Reported by Kane York.
 2020-07-24 11:19:17 +02:00
John Ericson 58e55c0923 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-23 19:17:52 +00:00
John Ericson a2af25c55a Merge remote-tracking branch 'upstream/master' into misc-ca 2020-07-23 19:14:17 +00:00
John Ericson e1de1fe0d8 Make Buildable a std::variant 
I think this better captures the intent of what's going on: we either
have an opaque store path, or a drv path with some outputs.

Having this structure will also help us support CA derivations: we'll
have to allow the outpath paths to be optional, so the structure we gain
now makes up for the structure we loose then.
 2020-07-23 19:02:57 +00:00
John Ericson c36916eca0 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into derivation-primop-floating-output 2020-07-23 15:06:56 +00:00
John Ericson 6fcd9966bf Merge remote-tracking branch 'upstream/master' into misc-ca 2020-07-23 15:02:29 +00:00
John Ericson 66a2067288 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-07-23 14:56:35 +00:00
John Ericson 2274f63453 Get rid of basicDerivation::findOutput 
It's a tiny function which is:

 - hardly worth abstrating over, and also only used once.

 - doesn't work once we get CA drvs

I rewrote the one callsite to be forwards compatable with CA
derivations, and also potentially more performant: instead of reading in
the derivation it can ust consult the SQLite DB in the common case.
 2020-07-23 14:39:02 +00:00
Fabian Möller b9ead08ca8
Save changes made by "nix registry pin" to user registry 2020-07-23 14:21:27 +02:00
John Ericson 9423f64ee2 Parse CA derivations using new output variants 
We no longer need `ParsedDerivation` because everything libstore needs
to know about is in the `BasicDerivation` proper.
 2020-07-22 23:59:25 +00:00
John Ericson c56356bacc Separate concerns in scanForReferences with TeeSink 
This also will make it easier to use a `HashModuloSink` instead for CA
derivations.
 2020-07-22 22:13:44 +00:00
Eelco Dolstra ff314f186e
Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option 
Allow relative paths in --store option
 2020-07-21 19:43:07 +02:00
Eelco Dolstra 0951330680
Merge pull request #3714 from obsidiansystems/add-body-to-network-errors 
Add response body to network errors
 2020-07-21 18:09:42 +02:00
John Ericson 6cce32c8e8 Change logic for deciding what is a relative path for the local store 
The was Eelco's prefered logic, and it looks good to me!
 2020-07-21 15:39:47 +00:00
Carlo Nucera 922a845ffc Update chunkSize to the suggested value 
This was a suggested course of action in a review in one of our earlier
commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079
 2020-07-21 10:24:19 -04:00
John Ericson 54e507a7aa Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors 2020-07-21 14:17:59 +00:00
John Ericson 5ce95b9529
Update src/libstore/build.cc 2020-07-21 09:47:40 -04:00
John Ericson cdb3f39b64 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-07-21 13:43:20 +00:00
John Ericson 5055c595bd Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into ca-derivation-data-types 2020-07-21 01:20:53 +00:00
John Ericson 5a37413796 Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream 2020-07-21 01:20:14 +00:00
John Ericson bf0dca7790 Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into misc-ca 2020-07-21 01:18:33 +00:00
Carlo Nucera 9aae179f34 Correct bug, thoroughly document addToStoreSlow 2020-07-20 20:18:12 -04:00
Carlo Nucera 0ca9744694 Use heuristics to decide when to show the response 
Due to https://github.com/NixOS/nix/issues/3841 we don't know how print
different messages for different verbosity levels.
 2020-07-20 15:57:58 -04:00
Carlo Nucera 6357b1b0fb Add another Unimplemented case 2020-07-20 14:17:25 -04:00
Carlo Nucera 007bf6d17a Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-20 14:14:39 -04:00
Carlo Nucera 362ae93851 Add UnimplementedError to ease grepping for these 2020-07-20 14:13:37 -04:00
John Ericson c58c6165c5 Remove period at the end of the exception message 2020-07-20 17:43:19 +00:00
John Ericson bf61871271 parser.hh -> split.hh 2020-07-20 17:42:34 +00:00
John Ericson eab945f591 Merge remote-tracking branch 'upstream/master' into from-dump-stream 2020-07-20 17:29:23 +00:00
Eelco Dolstra 1c5f8bbfb5
Merge pull request #3822 from obsidiansystems/dump-thrice-fixme 
Optimize `addToStoreSlow` and remove `TeeParseSink`
 2020-07-20 18:55:05 +02:00
John Ericson ac2fc7ba1f
Apply suggestions from code review 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-07-20 11:29:46 -04:00
John Ericson 6756cecfcf Add DerivationType::CAFloating 2020-07-17 19:55:41 +00:00
Carlo Nucera fefd6c9e5f Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option 2020-07-17 14:34:20 -04:00
Carlo Nucera bbc633c98c Revert "Don't anticipate CA but not fixed outputs for now" 
This reverts commit 3a9e4c3262.
 2020-07-17 13:10:32 -04:00
Carlo Nucera 205dcd140d Revert "Don't anticipate multiple CA outputs for now" 
This reverts commit 74b251b2f3.
 2020-07-17 12:43:46 -04:00
Carlo Nucera 1feb8981df Revert "Don't anticipate hash algo without hash in derivation for now" 
This reverts commit 3804e3df9b.
 2020-07-17 12:05:54 -04:00
Lars Mühmel bc73590151 nix edit: call restoreSignals() before execvp-ing the $EDITOR 
Currently resizing of the terminal doesn't play nicely with
nix edit when using kakoune as the editor, as it relies on the
SIGWINCH signal which is trapped by nix. How this is not a problem
with e.g. vim is beyond me.

Virtually all other exec* calls are following a call to
restoreSignals(). This commit adds this behavior to nix edit
as well.
 2020-07-17 17:35:59 +02:00
Carlo Nucera 487c5751c6 Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca 2020-07-17 11:32:59 -04:00
Carlo Nucera a395c12830 Merge branch 'master' of github.com:NixOS/nix into misc-ca 2020-07-17 11:06:54 -04:00
Carlo Nucera 5cb840541b Merge branch 'multi-output-hashDerivationModulo' of github.com:Ericson2314/nix into misc-ca 2020-07-17 10:28:33 -04:00
Eelco Dolstra 17f75f9cc4 parseFlakeRef(): Only search for the top-level directory for CLI flakerefs 2020-07-17 14:54:21 +02:00
Eelco Dolstra 52c8be38e0 nix profile diff-closures: Don't inherit EvalCommand 2020-07-17 12:36:12 +02:00
Carlo Nucera 745a03cef5 Merge branch 'optional-derivation-output-storepath' of github.com:obsidiansystems/nix into ca-derivation-data-types 2020-07-16 13:36:01 -04:00
Carlo Nucera 048e916f64 Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath 2020-07-16 13:32:28 -04:00
John Ericson cc0d77f8c9 Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-16 17:28:52 +00:00
Carlo Nucera e3a3406db8 Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors 2020-07-16 12:04:10 -04:00
Eelco Dolstra 16c9f6762d Add command 'nix profile diff-closure' 
This shows all changes between generations of a profile. E.g.

  $ nix profile diff-closures --profile /nix/var/nix/profiles/system
  Generation 654 -> 655:
    nix: 2.4pre20200617_5d69bbf → 2.4pre20200701_6ff9aa8, +42.2 KiB

  Generation 655 -> 656:
    blender-bin: 2.83.0 → 2.83.1, -294.2 KiB

  Generation 656 -> 657:
    curl: 7.68.0 → 7.70.0, +19.1 KiB
    firmware-linux-nonfree: 2020-01-22 → 2020-05-19, +30827.7 KiB
    ibus: -21.8 KiB
    initrd-linux: 5.4.46 → 5.4.49
    ...
 2020-07-16 17:00:42 +02:00
John Ericson 5ea817dace Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-16 14:58:53 +00:00
John Ericson 3dcca18c30 Fix bug in TeeSource 
We use this to simplify `LocalStore::addToStoreFromDump`.

Also, hope I fixed build error with old clang (used in Darwin CI).
 2020-07-16 13:39:27 +00:00
Carlo Nucera e3b394b6e8 Small namespace fix 2020-07-16 09:36:02 -04:00
Eelco Dolstra 5517eee17e Generations API cleanup 2020-07-16 15:18:21 +02:00
Eelco Dolstra 8807ff902e nix diff-closures: Fix build 2020-07-16 14:25:51 +02:00
Eelco Dolstra 716eafaca4 Merge branch 'diff-closures' 2020-07-16 14:19:35 +02:00
Eelco Dolstra 3f264916db Merge remote-tracking branch 'origin/flakes' 2020-07-16 14:07:32 +02:00
John Ericson 68dfb8c6ae Optimize addToStoreSlow and remove TeeParseSink 2020-07-16 05:09:41 +00:00
John Ericson 5602637d9e Revert "LocalStore::addToStoreFromDump copy in chunks" 
This reverts commit 592851fb67. We don't
need this extra feature anymore
 2020-07-15 23:37:49 +00:00
John Ericson bc109648c4 Get rid of LocalStore::addToStoreCommon 
I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking
a store and then doing nothing too fancy with it.

`LocalStore::addToStore(...Path...)` is now just a simple wrapper with a
bare-bones sinkToSource of the right dump command.
 2020-07-15 23:14:30 +00:00
Carlo Nucera 455bdee205 Merge branch 'master' of github.com:NixOS/nix into derivation-header-include-order 2020-07-15 17:58:30 -04:00
John Ericson 64b7421741 Merge branch 'handle-flat-case' into from-dump-stream 2020-07-15 21:22:55 +00:00
Carlo Nucera d090562348 Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-15 17:21:01 -04:00
John Ericson d087cf4855 Revert "Revert "LocalStore::addToStore(srcPath): Handle the flat case"" 
This reverts commit cff2157185.
 2020-07-15 21:10:33 +00:00
John Ericson aff69e229c Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream 2020-07-15 21:05:29 +00:00
Eelco Dolstra 36a1242603 nix why-depends: Fix shortest path calculation 
This was completely broken since d8972317fc.
 2020-07-15 21:11:07 +02:00
Eelco Dolstra 2d6d53bc87 nix: Fix examples 2020-07-15 20:28:16 +02:00
Eelco Dolstra 3624c042ac nix: Add --derivation flag to operate on .drv paths 
For instance, 'nix why-depends --use-derivation nixpkgs#hello
nixpkgs#glibc' shows why hello's .drv depends on glibc's .drv.
 2020-07-15 20:25:10 +02:00
Eelco Dolstra dfe8f3ebc6 nix why-depends: Fix misleading message 2020-07-15 20:09:50 +02:00
Eelco Dolstra 94eb5fad76 Clean up RealiseMode 2020-07-15 20:05:42 +02:00
Eelco Dolstra e3c2b00237 Make InstallableStorePath behave consistently with InstallableValue 
That is, the commands 'nix path-info nixpkgs#hello' and 'nix path-info
/nix/store/00ls0qi49qkqpqblmvz5s1ajl3gc63lr-hello-2.10.drv' now do the
same thing (i.e. build the derivation and operate on the output store
path, rather than the .drv path).
 2020-07-15 19:50:32 +02:00
Carlo Nucera 2d2a10e79a Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors 2020-07-15 13:06:08 -04:00
Eelco Dolstra 0356f14459 Add 'nix diff-closures' command 
This command makes it easier to see what changed between two closures,
i.e. what packages/versions got added or removed, and whether there
were any notable changes in path size.

For example:

  $ nix diff-closures /nix/var/nix/profiles/system-655-link /nix/var/nix/profiles/system-658-link
  blender-bin: 2.83.0 → 2.83.2, -294.2 KiB
  curl: 7.68.0 → 7.70.0, +19.1 KiB
  firmware-linux-nonfree: 2020-01-22 → 2020-05-19, +30827.7 KiB
  ibus: -21.8 KiB
  initrd-linux: 5.4.46 → 5.4.51, +16.9 KiB
  libexif: 0.6.21 → 0.6.22, +497.6 KiB
  linux: 5.4.46 → 5.4.51, +13.2 KiB
  mesa: 19.3.3 → 19.3.5, -183.9 KiB
  nix: 2.4pre20200701_6ff9aa8 → 2.4pre20200708_9223603, +9.7 KiB
  nix-bash-completions: 0.6.8 → ∅, -57.6 KiB
  nixos-system-hagbard: 20.03.20200615.a84b797 → 20.03.20200713.add5529
  nvidia-persistenced: 440.82 → 440.100
  nvidia-settings: 440.82 → 440.100
  nvidia-x11: 440.82-5.4.46 → 440.100-5.4.51, +664.7 KiB
  pcre: 8.43 → 8.44
  php: 7.3.16 → 7.3.20, -26.2 KiB
  python3.7-youtube-dl: 2020.06.06 → 2020.06.16.1, +8.4 KiB
  samba: 4.11.5 → 4.11.9, +30.1 KiB
  sane-backends: 1.0.28 → 1.0.30, +680.5 KiB
  source: -182.0 KiB
  zfs-kernel: 0.8.3-5.4.46 → 0.8.4-5.4.51, +9.9 KiB
  zfs-user: 0.8.3 → 0.8.4, +20.1 KiB
 2020-07-15 16:24:25 +02:00
Eelco Dolstra cff2157185 Revert "LocalStore::addToStore(srcPath): Handle the flat case" 
This reverts commit a2c27022e9. See
addToStoreSlow(), we don't need to handle this case efficiently
anymore. In fact, we can almost remove the method/hashAlgo arguments
since the non-recursive and/or non-SHA256 are almost not used anymore.
 2020-07-15 12:49:03 +02:00
John Ericson 650c2c6558 Rename variable nar -> dump according to TODO 2020-07-14 21:28:50 +00:00
John Ericson 8173e7bfef Fix localhost::addToStore(...Path...) 
We were calculating the nar hash wrong when the file ingestion method
was flat. I don't think there's anything we can do in that case but dump
the file again, so that's what I do.

As an optomization, we again could reuse the original dump for just the
recursive and non-sha256 case, but I rather do that after this fix, and
after my other PRs which deduplicate this code.
 2020-07-14 21:12:11 +00:00
John Ericson 592851fb67 LocalStore::addToStoreFromDump copy in chunks 
Rather than copying byte-by-byte, we let the coroutine know how much
data we would like it to send back to us.
 2020-07-14 13:56:36 +00:00
John Ericson 9de96ef7d4 Dedup LocalStore::addToStore* 
The downsides is that the coroutine has byte-by-byte loop transfer. Will
fix that next.
 2020-07-14 13:56:36 +00:00
John Ericson c86fc3a965 Crudely make addToStoreFromDump take Source not string 
I just as little beyond the type as possible, so the implementation
changes this enables can be reviewed separately.
 2020-07-14 13:56:36 +00:00
John Ericson 9ec10046e0 Narrow scope of temporary value 2020-07-14 13:54:29 +00:00
Eelco Dolstra 298ff6af8f
Merge pull request #3809 from Ma27/gitlab-refs 
Fix gitlab-fetcher to obtain tags and branches
 2020-07-14 15:22:52 +02:00
Eelco Dolstra da3aea291d EvalCache: Ignore SQLite errors 
Fixes #3794.
 2020-07-14 15:17:38 +02:00
Eelco Dolstra 832e111494 Merge remote-tracking branch 'origin/master' into flakes 2020-07-14 13:56:18 +02:00
Eelco Dolstra 926c3a6664 Doh 2020-07-14 11:55:54 +02:00
Eelco Dolstra 43b8e96d30 Fix 'nix verify --all' on a binary cache (cached case) 2020-07-13 20:17:00 +02:00
Eelco Dolstra 9502c0e2eb nix verify: Show correct path when using --all on a binary cache 2020-07-13 20:12:44 +02:00
Eelco Dolstra 7c2fef0a81 Make 'nix copy' to s3:// binary caches run in constant memory 2020-07-13 20:07:19 +02:00
Maximilian Bosch cf9f33995b
Fix gitlab-fetcher to obtain tags and branches 
Until now, the `gitlab`-fetcher determined the source's rev by checking
the latest commit of the given `ref` using the
`/repository/branches`-API.

This breaks however when trying to fetch a gitlab-repo by its tag:

```
$ nix repl
nix-repl> builtins.fetchTree gitlab:Ma27/nvim.nix/0.2.0
error: --- Error ------------------------------------------------------------------------------------- nix
unable to download 'https://gitlab.com/api/v4/projects/Ma27%2Fnvim.nix/repository/branches/0.2.0': HTTP error 404 ('')
```

When using the `/commits?ref_name`-endpoint[1] you can pass any kind of
valid ref to the `gitlab`-fetcher.

Please note that this fetches the only first 20 commits on a ref,
unfortunately there's currently no endpoint which only retrieves the
latest commit of any kind of `ref`.

[1] https://docs.gitlab.com/ee/api/commits.html#list-repository-commits
 2020-07-13 19:22:59 +02:00
Eelco Dolstra 493961b689 Remove istringstream_nocopy 2020-07-13 18:31:19 +02:00
Eelco Dolstra 545bb2ed03 Remove 'accessor' from addToStore() 
This is only used by hydra-queue-runner and it's better to implement
it there.
 2020-07-13 18:31:19 +02:00
Eelco Dolstra 0a9da00a10 NarAccessor: Run in constant memory 2020-07-13 17:30:42 +02:00
Eelco Dolstra fc84c358d9 Make 'nix copy' to file:// binary caches run in constant memory 2020-07-13 16:28:45 +02:00
Eelco Dolstra 400f1a9b59 Store::pathInfoToJSON(): Use consistent format for downloadHash 2020-07-13 16:26:09 +02:00
Eelco Dolstra c0dd05131e toStorePath(): Return a StorePath and the suffix 2020-07-13 16:25:48 +02:00
Eelco Dolstra 1d01ae816b Fix 'nix verify --all' on a binary cache and add a test 2020-07-13 14:35:01 +02:00
John Ericson c466cb2091 Merge branch 'hash-always-has-type' into better-ca-parse-errors 2020-07-13 03:02:09 +00:00
John Ericson 4415765385 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-13 03:01:44 +00:00
John Ericson 230c9b4329 Change types to prepare the way for CA derivations 
We've added the variant to `DerivationOutput` to support them, but made
`DerivationOutput::path` partial to avoid actually implementing them.

With this chage, we can all collaborate on "just" removing
`DerivationOutput::path` calls to implement CA derivations.
 2020-07-12 23:53:33 +00:00
John Ericson fedfc913ad Use more std::visit to prepare for new variant 
N.B. not using `std::visit` for fetchurl because there is no attempt to
handle all the cases (e.g. no `else`) and lambda complicates early
return.
 2020-07-12 23:51:40 +00:00
John Ericson ffc18583b1 Move C++17 "pattern matching" boilerplat to utils.hh 2020-07-12 22:15:14 +00:00
John Ericson 503b425690 DerivationOutputExtensional -> DerivationOutputInputAddressed 
Thanks @regnat for the great name.
 2020-07-12 15:56:20 +00:00
John Ericson a8d4707107 Undo erroneous indentation change 2020-07-12 15:54:12 +00:00
John Ericson 18152406ce String .drv suffix to create derivation name 2020-07-12 15:40:14 +00:00
John Ericson 5d0b75e5b6 std::string_view for new derivation name parameters 2020-07-12 15:02:36 +00:00
Maximilian Bosch 64f03635d7
Fix ANSI color constants 
The `m` acts as termination-symbol when declaring graphics. Because
of this, the `;1m` doesn't have any effect and is directly printed to
the console:

```
$ nix repl
> builtins.fetchGit { /* ... */ }
{ outPath = "/nix/store/s0f0iz4a41cxx2h055lmh6p2d5k5bc6r-source"; rev = "e73e45b723a9a6eecb98bd5f3df395d9ab3633b6"; revCount = ;1m428; shortRev = "e73e45b"; submodules = ;1mfalse; }
```

Introduced by 6403508f5a.
 2020-07-12 16:52:20 +02:00
John Ericson 13ec627e0a Set derivation name in dervationStrict 2020-07-12 03:03:12 +00:00
John Ericson 1c9bec226f Don't improperly assume path is store path 2020-07-12 02:38:03 +00:00
John Ericson abea26a968 BasicDerivation::findOutput cannot return reference anymore 2020-07-12 01:57:06 +00:00
John Ericson 442d43178b Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-11 02:27:45 +00:00
Matthew Bauer acb74d4d94 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-10 18:13:45 -04:00
Eelco Dolstra 8efa23bb99 Avoid a redundant hash 2020-07-10 15:56:24 +02:00
Eelco Dolstra 5dff49f661 Factor out commonality between nix-prefetch-url and nix-store --add-fixed 2020-07-10 13:21:37 +02:00
Eelco Dolstra 7f1a86d57c nix-store --add-fixed: Run in constant memory 2020-07-10 12:51:56 +02:00
Eelco Dolstra 06e3dd9005 nix-prefetch-url: Run in constant memory when using RemoteStore 
Fixes #3684.
 2020-07-10 11:22:48 +02:00
John Ericson 323d51a7a0 Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-09 19:07:06 +00:00
Matthew Bauer a7884970c5 Fix DerivationOutputExtensional name 2020-07-09 11:37:18 -04:00
Eelco Dolstra a2c27022e9 LocalStore::addToStore(srcPath): Handle the flat case 
This helps nix-prefetch-url when using a local store.
 2020-07-09 15:54:32 +02:00
Matthew Bauer 06a4e15478 Fix build.cc on linux 2020-07-08 19:27:51 -04:00
Matthew Bauer 8e0d0689be Only store hash of fixed derivation output 
we don’t need a full storepath for a fixedoutput derivation. So just
putting the ingestion method + the hash is sufficient.
 2020-07-08 19:11:39 -04:00
Eelco Dolstra b981e5aacf Cleanup 2020-07-08 22:07:21 +02:00
Eelco Dolstra 34f25124ba Make LocalStore::addToStore(srcPath) run in constant memory 
This reduces memory consumption of

  nix-instantiate \
    -E 'with import <nixpkgs> {}; runCommand "foo" { src = ./blender; } "echo foo"' \
    --option nar-buffer-size 10000

(where ./blender is a 1.1 GiB tree) from 1716 to 36 MiB, while still
ensuring that we don't do any write I/O for small source paths (up to
'nar-buffer-size' bytes). The downside is that large paths are now
always written to a temporary location in the store, even if they
produce an already valid store path. Thus, adding large paths might be
slower and run out of disk space. ¯\_(ツ)_/¯ Of course, you can always
restore the old behaviour by setting 'nar-buffer-size' to a very high
value.
 2020-07-08 22:07:21 +02:00
Matthew Bauer af95a7c16b Add name to BasicDerivation 
We always have a name for BasicDerivation, since we have a derivation
store path that has a name.
 2020-07-08 15:38:01 -04:00
Eelco Dolstra 9223603908 Merge remote-tracking branch 'origin/master' into flakes 2020-07-08 15:55:19 +02:00
Eelco Dolstra 16ec7785ca Fix 'got unknown message type 1 from Nix daemon' 
Example:

  $ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" { x = runCommand "bar" {} "exit 1"; } "echo foo; exit 1"'
  warning: unknown setting 'auto-allocate-uids'
  these 2 derivations will be built:
    /nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv
    /nix/store/k4fsvrjl7cp2xpz7927iv7g0dqj1zyhs-foo.drv
  warning: unknown setting 'auto-allocate-uids'
  building '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv'...
  error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-daemon
  builder for '/nix/store/v4fbdbhcdi949929a67g8farwf72zgam-bar.drv' failed with exit code 1
  error: --- Error ------------------------------------------------------------------------------------------------------------------------------------------------------------------ nix-build
  got unknown message type 1 from Nix daemon
 2020-07-08 15:53:14 +02:00
Eelco Dolstra 1ab9da9154 Merge remote-tracking branch 'origin/master' into flakes 2020-07-07 14:38:57 +02:00
Eelco Dolstra 4055cfee36 Fix coverage build 2020-07-07 14:37:47 +02:00
Eelco Dolstra 7c9ece5dca exportReferencesGraph: Fix support for non-top-level store paths 
Fixes #3471.
 2020-07-07 14:25:43 +02:00
Carlo Nucera 0f05a36e20 Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-06 16:57:09 -04:00
Ben Burdette efd6a8b230 bump 2020-07-06 11:54:53 -06:00
Ben Burdette 75bfcf8d15 revamp trace code and test 2020-07-06 10:51:48 -06:00
Eelco Dolstra 68f524d717 nix develop: Support derivations with multiple outputs 2020-07-06 18:34:58 +02:00
Eelco Dolstra cd8eb8a7d1 nix develop: Fall back to "bash" if nixpkgs#bashInteractive is unavailable 2020-07-06 17:08:54 +02:00
Eelco Dolstra 54712aaf8a Merge remote-tracking branch 'origin/master' into flakes 2020-07-06 16:40:10 +02:00
John Ericson a38ab99d57 Merge remote-tracking branch 'upstream/master' into derivation-header-include-order 2020-07-05 21:49:01 +00:00
Ben Burdette a168224464 spacing 2020-07-04 18:30:49 -06:00
John Ericson 465daa9396 Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors 2020-07-03 17:08:39 +00:00
John Ericson 3134db1a83 Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-03 14:12:38 +00:00
John Ericson dbffd309fe Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-03 14:11:38 +00:00
Ben Burdette b29a4ea1dc Merge branch 'master' into add-trace 2020-07-03 07:57:36 -06:00
Eelco Dolstra c3c7aedbb5 nix develop: Fix bad regex 
This was accepted by libstdc++ but not libc++.

https://hydra.nixos.org/build/123569154
 2020-07-03 14:58:58 +02:00
Eelco Dolstra 6f8fd3a3f2 Shut up a clang warning 2020-07-03 14:50:07 +02:00
regnat 223fbe644a Shorten the path to the test root 
Fix a socket length failure on the OSX builders
 2020-07-03 09:20:01 +02:00
John Ericson 13796be78d Have splitPrefix and splitPrefixTo parser helpers 2020-07-02 23:18:22 +00:00
John Ericson a7cd7425d9 Move getParsedTypeAndSRI to a more suitable location 
Also mark it static
 2020-07-02 23:10:11 +00:00
John Ericson 2f93d9f2ba Merge branch 'hash-always-has-type' into HEAD 2020-07-02 21:47:51 +00:00
John Ericson 1be279af26 Fix Narinfo corruption detection bug 
The aim of this check was just to ensure each key occurs once.
 2020-07-02 21:46:10 +00:00
Eelco Dolstra b5e4253697 Fix abort in 'nix develop' 2020-07-02 18:24:11 +02:00
Carlo Nucera 1fc835aa22 Tighten parsing for drv files and pathinfo 2020-07-02 11:57:21 -04:00
Ben Burdette 5818271c6e spacing 2020-07-02 09:41:54 -06:00
Carlo Nucera b6b10b1d4c Write the implementation for parseNonSRIUnprefixed 2020-07-02 11:34:40 -04:00
Carlo Nucera ea48e3a5b5 Abstract common parsing functionality 2020-07-02 11:29:33 -04:00
Carlo Nucera 36cbc74689 Inline and simplify in parseAnyPrefixed 2020-07-02 11:21:00 -04:00
Matthew Bauer fc2ab42e86 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-02 11:14:04 -04:00
Matthew Bauer d2e8b9ff0e Store subPath in SubstitutionGoal 2020-07-02 11:12:05 -04:00
Carlo Nucera 9462d8a50b Rename fromSRI to parseSRI for constistency 2020-07-02 11:11:18 -04:00
Carlo Nucera f61bc45d19 Get rid of the std::pair 2020-07-02 11:09:04 -04:00
Ben Burdette bf2788e4c1 move showTrace to new loggerSettings 2020-07-02 09:04:31 -06:00
Carlo Nucera 27c8029573 Inline newFunction 2020-07-02 11:01:10 -04:00
Matthew Bauer 1f9cb06db2 Try next when no ca exists and have different store dirs 2020-07-02 10:59:24 -04:00
Carlo Nucera 343d1569b1 Fix test suite 2020-07-02 10:48:47 -04:00
Ben Burdette 5ae498872a assert for invalid fileorigin 2020-07-02 07:14:40 -06:00
Carlo Nucera 263ccdd489 Rename two hash constructors to proper functions 2020-07-01 18:34:18 -04:00
Carlo Nucera c8c4bcf90e Inline Hash::init() 2020-07-01 18:03:35 -04:00
Carlo Nucera d63a5ded76 Remove unused import 2020-07-01 17:53:24 -04:00
Carlo Nucera 6faeec3b2a Keep the previous name, for diffing 2020-07-01 17:50:34 -04:00
Carlo Nucera 274a8136fb Correct FIXMEs in libfetchers 2020-07-01 17:47:15 -04:00
Carlo Nucera c2e7f7a712 Fixed build, we still have test errors 2020-07-01 17:32:06 -04:00
Ben Burdette 8497891b99 spacing 2020-07-01 13:50:18 -06:00
Eelco Dolstra 6ff9aa8df7 Don't process an option if any of its arguments need completion 2020-07-01 20:31:39 +02:00
Eelco Dolstra d746503e5c Add --inputs-from to use flake inputs as registry entries 
This allows you to refer to an input from another flake. For example,

  $ nix run --inputs-from /path/to/hydra nixpkgs#hello

runs 'hello' from the 'nixpkgs' inputs of the 'hydra' flake.

Fixes #3769.
 2020-07-01 20:25:13 +02:00
Ben Burdette a295b2ea96 if no errLoc, no Loc. 2020-07-01 12:02:02 -06:00
Ben Burdette 3629b0585a don't include errpos for addErrorContext 2020-07-01 11:49:01 -06:00
Ben Burdette 2a39c083dc non-pos trace test 2020-07-01 10:37:31 -06:00
Eelco Dolstra 38ccf2e241 Cleanup 2020-07-01 15:31:34 +02:00