jade/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
10342 commits 10 branches 3 tags 77 MiB
Commit graph

569 commits

Author SHA1 Message Date
regnat f24f0888f9 Document the new store hierarchy 2020-09-16 13:53:09 +02:00
regnat 5895184df4 Correctly call all the parent contructors of the stores 
Using virtual inheritance means that only the default constructors of
the parent classes will be called, which isn't what we want
 2020-09-16 13:53:09 +02:00
John Ericson 98dfd7531d Fix querying outputs for CA derivations some more 
If we resolve using the known path of a derivation whose output we
didn't have, we previously blew up. Now we just fail gracefully,
returning the map of all outputs unknown.
 2020-09-04 18:33:58 +00:00
John Ericson aad4abcc9c Fix floating CA tests 
We will sometimes try to query the outputs of derivations we can't
resolve. That's fine; it just means we don't know what those outputs are
yet.
 2020-09-04 01:17:38 +00:00
John Ericson 8eb73a8724 CA derivations that depend on other CA derivations 
Co-authored-by: Théophane Hufschmitt <regnat@users.noreply.github.com>
 2020-08-24 19:01:15 +00:00
John Ericson 45a2f1baab Rename drv output querying functions, like master 
- `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap`
 - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap
 2020-08-20 18:14:12 +00:00
John Ericson 3c8b5b6219 Merge remote-tracking branch 'upstream/master' into single-ca-drv-build 2020-08-14 17:00:13 +00:00
Eelco Dolstra 13e49be660
Merge pull request #3875 from obsidiansystems/new-interface-for-path-pathOpt 
Offer a safer interface for path and pathOpt
 2020-08-14 17:19:19 +02:00
Eelco Dolstra d81f13f7cb
Merge pull request #3899 from obsidiansystems/make-narHash-not-optional 
Make narHash in ValidPathInfo not optional
 2020-08-14 17:00:18 +02:00
John Ericson 85aacbee64 Use TeeSink and TeeSouce in a few more places 2020-08-13 14:51:17 +00:00
John Ericson e913a2989f Squashed get CA derivations building 2020-08-07 19:51:55 +00:00
John Ericson e89b5bd0bf Minimize the usage of Hash::dummy 2020-08-06 18:31:48 +00:00
Carlo Nucera 1d2e80ddd6 Merge branch 'master' of github.com:NixOS/nix into new-interface-for-path-pathOpt 2020-08-05 15:45:33 -04:00
Carlo Nucera 8241e660ba Remove Hash::operator bool () 
Since the hash is not optional anymore
 2020-08-05 15:30:38 -04:00
Carlo Nucera be6e1c6457 Merge branch 'master' of github.com:NixOS/nix into make-narHash-not-optional 2020-08-05 15:14:47 -04:00
Carlo Nucera 1d71028f4d Remove optionality in ValidPathInfo::narInfo 2020-08-05 14:42:48 -04:00
John Ericson cf939055c8 Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-05 16:43:30 +00:00
Eelco Dolstra 790b694be7 Style fix 2020-08-05 16:51:06 +02:00
John Ericson e7b0847f2d Make names more consistent 2020-08-05 14:44:39 +00:00
John Ericson 839f0fe095 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:40:01 +00:00
John Ericson 03f4fafc27 Merge remote-tracking branch 'upstream/master' into misc-ca 2020-08-05 14:36:25 +00:00
John Ericson fbeb8695fb Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-04 18:35:33 +00:00
John Ericson d3452a5ed6 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-08-04 16:13:20 +00:00
John Ericson 54281f3ac1 addToStore in terms of addToStoreFromDump is not local-store-specific 2020-08-03 04:13:45 +00:00
Carlo Nucera b6d97fdbf4 Merge branch 'master' of github.com:NixOS/nix into drv-outputs-map-allow-missing 2020-07-31 13:12:51 -04:00
Carlo Nucera eee6ef86cd Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-30 18:27:25 -04:00
Matthew Bauer 05ac4db39a Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-30 12:38:24 -05:00
Carlo Nucera c318d398f3 Merge branch 'misc-ca' of github.com:obsidiansystems/nix into new-interface-for-path-pathOpt 2020-07-28 14:22:24 -04:00
Carlo Nucera 7ef1e3cd14 Use the new interface 2020-07-28 13:59:24 -04:00
John Ericson 7cf978440c Merge branch 'ca-derivation-data-types' of github.com:obsidiansystems/nix into misc-ca 2020-07-27 17:47:40 +00:00
John Ericson 1d7d94ceea Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-27 16:17:50 +00:00
John Ericson 43f2bd8dc5 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 16:13:57 +00:00
John Ericson d5bb67cfa4 Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-27 14:29:32 +00:00
John Ericson 699fc89b39 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 14:22:45 +00:00
John Ericson 2c7557481b queryDerivationOutputMap no longer assumes all outputs have a mapping 
This assumption is broken by CA derivations. Making a PR now to do the
breaking daemon change as soon as possible (if it is already too late,
we can bump protocol intead).
 2020-07-24 21:14:06 +00:00
John Ericson 6fcd9966bf Merge remote-tracking branch 'upstream/master' into misc-ca 2020-07-23 15:02:29 +00:00
John Ericson 66a2067288 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-07-23 14:56:35 +00:00
Carlo Nucera 922a845ffc Update chunkSize to the suggested value 
This was a suggested course of action in a review in one of our earlier
commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079
 2020-07-21 10:24:19 -04:00
John Ericson cdb3f39b64 Merge remote-tracking branch 'upstream/master' into better-ca-parse-errors 2020-07-21 13:43:20 +00:00
Carlo Nucera 6357b1b0fb Add another Unimplemented case 2020-07-20 14:17:25 -04:00
John Ericson eab945f591 Merge remote-tracking branch 'upstream/master' into from-dump-stream 2020-07-20 17:29:23 +00:00
Carlo Nucera 205dcd140d Revert "Don't anticipate multiple CA outputs for now" 
This reverts commit 74b251b2f3.
 2020-07-17 12:43:46 -04:00
Carlo Nucera 487c5751c6 Merge branch 'ca-drv' of github.com:Ericson2314/nix into misc-ca 2020-07-17 11:32:59 -04:00
Carlo Nucera 5cb840541b Merge branch 'multi-output-hashDerivationModulo' of github.com:Ericson2314/nix into misc-ca 2020-07-17 10:28:33 -04:00
Carlo Nucera 048e916f64 Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath 2020-07-16 13:32:28 -04:00
John Ericson cc0d77f8c9 Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-16 17:28:52 +00:00
John Ericson 5ea817dace Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-16 14:58:53 +00:00
John Ericson 3dcca18c30 Fix bug in TeeSource 
We use this to simplify `LocalStore::addToStoreFromDump`.

Also, hope I fixed build error with old clang (used in Darwin CI).
 2020-07-16 13:39:27 +00:00
Eelco Dolstra 3f264916db Merge remote-tracking branch 'origin/flakes' 2020-07-16 14:07:32 +02:00
John Ericson 5602637d9e Revert "LocalStore::addToStoreFromDump copy in chunks" 
This reverts commit 592851fb67. We don't
need this extra feature anymore
 2020-07-15 23:37:49 +00:00
John Ericson bc109648c4 Get rid of LocalStore::addToStoreCommon 
I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking
a store and then doing nothing too fancy with it.

`LocalStore::addToStore(...Path...)` is now just a simple wrapper with a
bare-bones sinkToSource of the right dump command.
 2020-07-15 23:14:30 +00:00
Carlo Nucera d090562348 Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-15 17:21:01 -04:00
John Ericson aff69e229c Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream 2020-07-15 21:05:29 +00:00
Eelco Dolstra cff2157185 Revert "LocalStore::addToStore(srcPath): Handle the flat case" 
This reverts commit a2c27022e9. See
addToStoreSlow(), we don't need to handle this case efficiently
anymore. In fact, we can almost remove the method/hashAlgo arguments
since the non-recursive and/or non-SHA256 are almost not used anymore.
 2020-07-15 12:49:03 +02:00
John Ericson 650c2c6558 Rename variable nar -> dump according to TODO 2020-07-14 21:28:50 +00:00
John Ericson 8173e7bfef Fix localhost::addToStore(...Path...) 
We were calculating the nar hash wrong when the file ingestion method
was flat. I don't think there's anything we can do in that case but dump
the file again, so that's what I do.

As an optomization, we again could reuse the original dump for just the
recursive and non-sha256 case, but I rather do that after this fix, and
after my other PRs which deduplicate this code.
 2020-07-14 21:12:11 +00:00
John Ericson 592851fb67 LocalStore::addToStoreFromDump copy in chunks 
Rather than copying byte-by-byte, we let the coroutine know how much
data we would like it to send back to us.
 2020-07-14 13:56:36 +00:00
John Ericson 9de96ef7d4 Dedup LocalStore::addToStore* 
The downsides is that the coroutine has byte-by-byte loop transfer. Will
fix that next.
 2020-07-14 13:56:36 +00:00
John Ericson c86fc3a965 Crudely make addToStoreFromDump take Source not string 
I just as little beyond the type as possible, so the implementation
changes this enables can be reviewed separately.
 2020-07-14 13:56:36 +00:00
Eelco Dolstra 832e111494 Merge remote-tracking branch 'origin/master' into flakes 2020-07-14 13:56:18 +02:00
Eelco Dolstra 545bb2ed03 Remove 'accessor' from addToStore() 
This is only used by hydra-queue-runner and it's better to implement
it there.
 2020-07-13 18:31:19 +02:00
John Ericson 4415765385 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-13 03:01:44 +00:00
John Ericson 442d43178b Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-11 02:27:45 +00:00
Matthew Bauer acb74d4d94 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-10 18:13:45 -04:00
John Ericson 323d51a7a0 Merge branch 'master' of github.com:NixOS/nix into better-ca-parse-errors 2020-07-09 19:07:06 +00:00
Eelco Dolstra a2c27022e9 LocalStore::addToStore(srcPath): Handle the flat case 
This helps nix-prefetch-url when using a local store.
 2020-07-09 15:54:32 +02:00
Matthew Bauer 8e0d0689be Only store hash of fixed derivation output 
we don’t need a full storepath for a fixedoutput derivation. So just
putting the ingestion method + the hash is sufficient.
 2020-07-08 19:11:39 -04:00
Eelco Dolstra b981e5aacf Cleanup 2020-07-08 22:07:21 +02:00
Eelco Dolstra 34f25124ba Make LocalStore::addToStore(srcPath) run in constant memory 
This reduces memory consumption of

  nix-instantiate \
    -E 'with import <nixpkgs> {}; runCommand "foo" { src = ./blender; } "echo foo"' \
    --option nar-buffer-size 10000

(where ./blender is a 1.1 GiB tree) from 1716 to 36 MiB, while still
ensuring that we don't do any write I/O for small source paths (up to
'nar-buffer-size' bytes). The downside is that large paths are now
always written to a temporary location in the store, even if they
produce an already valid store path. Thus, adding large paths might be
slower and run out of disk space. ¯\_(ツ)_/¯ Of course, you can always
restore the old behaviour by setting 'nar-buffer-size' to a very high
value.
 2020-07-08 22:07:21 +02:00
Eelco Dolstra 54712aaf8a Merge remote-tracking branch 'origin/master' into flakes 2020-07-06 16:40:10 +02:00
John Ericson 3134db1a83 Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix into better-ca-parse-errors 2020-07-03 14:12:38 +00:00
John Ericson dbffd309fe Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-03 14:11:38 +00:00
Matthew Bauer fc2ab42e86 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-02 11:14:04 -04:00
Carlo Nucera 263ccdd489 Rename two hash constructors to proper functions 2020-07-01 18:34:18 -04:00
Eelco Dolstra adf2fbbdc2 Merge remote-tracking branch 'origin/master' into flakes 2020-06-26 08:46:46 +02:00
regnat d38f860c3e Add a way to get all the outputs of a derivation with their label 
Generalize `queryDerivationOutputNames` and `queryDerivationOutputs` by
adding a `queryDerivationOutputMap` that returns the map
`outputName=>outputPath`

(not that this is not equivalent to merging the results of
`queryDerivationOutputs` and `queryDerivationOutputNames` as sets don't
preserve the order, so we would end up with an incorrect mapping).

squash! Add a way to get all the outputs of a derivation with their label

Rename StorePathMap to OutputPathMap
 2020-06-24 20:38:40 +02:00
John Ericson d9fb9e9f26 Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-24 15:41:29 +00:00
John Ericson 98e5d1af03 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-06-23 17:03:37 +00:00
Matthew Bauer 66a62b3189 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-22 13:08:11 -04:00
John Ericson 93bbe6e8ab Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-21 21:11:58 +00:00
John Ericson fdeabf7160 Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo 2020-06-21 16:43:17 +00:00
John Ericson 507aa48739 WIP: Make Hash always store a valid hash type 2020-06-19 21:48:57 +00:00
Matthew Bauer 7e11cf3399 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-19 14:40:49 -04:00
Matthew Bauer 2796b01e86 Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-19 14:22:06 -04:00
John Ericson 68294746ae Merge remote-tracking branch 'upstream/master' into no-hash-type-unknown 2020-06-19 17:53:34 +00:00
John Ericson 911fc88bcb More designated initializers 2020-06-19 17:42:56 +00:00
John Ericson 2f0e395c99 Merge remote-tracking branch 'me/no-stringly-typed-derivation-output' into validPathInfo-ca-proper-datatype 2020-06-19 15:26:59 +00:00
John Ericson 3f8dcfe3fd Merge branch 'validPathInfo-temp' into validPathInfo-ca-proper-datatype 2020-06-18 23:01:58 +00:00
John Ericson 669c3992e8 Merge branch 'no-hash-type-unknown' into validPathInfo-temp 2020-06-18 22:33:07 +00:00
John Ericson 15abb2aa2b Revert the enum struct change 
Not a regular git revert as there have been many merges and things.
 2020-06-18 22:11:26 +00:00
John Ericson bbbf3602a3 Merge branch 'enum-class' into no-hash-type-unknown 2020-06-18 22:11:19 +00:00
John Ericson 40526fbea5 Merge remote-tracking branch 'upstream/master' into enum-class 2020-06-18 21:38:15 +00:00
Matthew Bauer 5e631e3304 Add StorePathCAMap for querySubstitutablePathInfos 
I’m not 100% sure this is wanted since it kind of makes everything
have to know about ca even if they don’t really want to. But it also
make things easier in dealing with looking up ca.
 2020-06-17 15:03:05 -04:00
Matthew Bauer be50de1142 Make sure references are empty for store path replacing 
also copy info2 instead of casting
 2020-06-17 14:15:08 -04:00
Matthew Bauer 8974755d19 Add assert for replaced storePath 2020-06-17 14:04:46 -04:00
Matthew Bauer 7bd88cc1dc Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-06-17 13:32:29 -04:00
Eelco Dolstra 1524752c17 Merge remote-tracking branch 'origin/master' into flakes 2020-06-17 10:26:52 +02:00
John Ericson 517f5980e2 Merge remote-tracking branch 'upstream/master' into no-stringly-typed-derivation-output 2020-06-17 04:58:43 +00:00
John Ericson 02928f76fd Merge remote-tracking branch 'upstream/master' into multi-output-hashDerivationModulo 2020-06-17 03:51:02 +00:00
John Ericson 49308ef039 Merge remote-tracking branch 'upstream/master' into ca-drv 2020-06-17 03:39:39 +00:00