puck/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
8895 commits 19 branches 0 tags 68 MiB
Commit graph

2419 commits

Author SHA1 Message Date
John Ericson 2f2ae993dc WIP systematize more of the worker protocol 
This refactor should *not* change the wire protocol.
 2020-08-04 19:02:05 +00:00
John Ericson fbeb8695fb Merge remote-tracking branch 'upstream/master' into drv-outputs-map-allow-missing 2020-08-04 18:35:33 +00:00
Eelco Dolstra 4e7f1c7f11 S3BinaryCacheStore: Fix size determination 2020-08-04 16:01:13 +02:00
Eelco Dolstra dfe66420e7 Revert "Remove putBytes" 
This reverts commit b8eea7e81a.
 2020-08-04 15:56:10 +02:00
Eelco Dolstra 327b1bf378 BinaryCacheStore: Explicitly flush file sink 
The file sink is also flushed in its destructor, but we ignore any
exceptions in the destructor.

Issue #3886.
 2020-08-04 14:54:07 +02:00
Eelco Dolstra ed52cf632b
Merge pull request #3892 from obsidiansystems/path-info-header 
Move ValidPathInfo to its own header
 2020-08-04 10:44:47 +02:00
John Ericson 062533f7cd Merge remote-tracking branch 'upstream/master' into path-info-header 2020-08-03 21:02:28 +00:00
Eelco Dolstra 24e07c428f Delete compressed NARs 
Fixes #3891.
 2020-08-03 18:34:52 +02:00
John Ericson 54281f3ac1 addToStore in terms of addToStoreFromDump is not local-store-specific 2020-08-03 04:13:45 +00:00
Carlo Nucera b6d97fdbf4 Merge branch 'master' of github.com:NixOS/nix into drv-outputs-map-allow-missing 2020-07-31 13:12:51 -04:00
Matthew Bauer 05ac4db39a Merge remote-tracking branch 'origin/master' into substitute-other-storedir 2020-07-30 12:38:24 -05:00
Eelco Dolstra 3f6e88a552 unsigned long long -> uint64_t 2020-07-30 13:34:04 +02:00
Eelco Dolstra ebee2b7852 receiveContents(): unsigned int -> size_t 2020-07-30 13:00:30 +02:00
Eelco Dolstra f63839bfa4 Cleanup 2020-07-29 18:04:51 +02:00
Eelco Dolstra 4c0077a07d Fix RemoteStore::addToStore() latency 
Since 6185d25e52, this was very
latency-bound since it required a round-trip for every 32 KiB. So for
example copying a 514 MiB closure over a virtual ethernet device with
a articial delay of just 1 ms took 343s. Now it takes 2.7s.

Fixes #3372.
 2020-07-29 00:48:39 +02:00
Eelco Dolstra c159f48a39 Cleanup 2020-07-29 00:24:55 +02:00
Eelco Dolstra b8eea7e81a Remove putBytes 
istream->tellg() returns -1 so we can't get the number of bytes
written.

Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.
 2020-07-28 13:27:56 +02:00
John Ericson 43f2bd8dc5 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 16:13:57 +00:00
John Ericson d5bb67cfa4 Merge remote-tracking branch 'upstream/master' into optional-derivation-output-storepath 2020-07-27 14:29:32 +00:00
John Ericson 699fc89b39 Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-27 14:22:45 +00:00
John Ericson 2c7557481b queryDerivationOutputMap no longer assumes all outputs have a mapping 
This assumption is broken by CA derivations. Making a PR now to do the
breaking daemon change as soon as possible (if it is already too late,
we can bump protocol intead).
 2020-07-24 21:14:06 +00:00
John Ericson 2274f63453 Get rid of basicDerivation::findOutput 
It's a tiny function which is:

 - hardly worth abstrating over, and also only used once.

 - doesn't work once we get CA drvs

I rewrote the one callsite to be forwards compatable with CA
derivations, and also potentially more performant: instead of reading in
the derivation it can ust consult the SQLite DB in the common case.
 2020-07-23 14:39:02 +00:00
John Ericson c56356bacc Separate concerns in scanForReferences with TeeSink 
This also will make it easier to use a `HashModuloSink` instead for CA
derivations.
 2020-07-22 22:13:44 +00:00
Eelco Dolstra ff314f186e
Merge pull request #3736 from obsidiansystems/allow-relative-paths-in-store-option 
Allow relative paths in --store option
 2020-07-21 19:43:07 +02:00
Eelco Dolstra 0951330680
Merge pull request #3714 from obsidiansystems/add-body-to-network-errors 
Add response body to network errors
 2020-07-21 18:09:42 +02:00
John Ericson 6cce32c8e8 Change logic for deciding what is a relative path for the local store 
The was Eelco's prefered logic, and it looks good to me!
 2020-07-21 15:39:47 +00:00
Carlo Nucera 922a845ffc Update chunkSize to the suggested value 
This was a suggested course of action in a review in one of our earlier
commits, https://github.com/NixOS/nix/pull/3801#discussion_r457557079
 2020-07-21 10:24:19 -04:00
John Ericson 54e507a7aa Merge remote-tracking branch 'upstream/master' into add-body-to-network-errors 2020-07-21 14:17:59 +00:00
John Ericson 5a37413796 Merge branch 'fix-and-document-addToStoreSlow' of github.com:obsidiansystems/nix into from-dump-stream 2020-07-21 01:20:14 +00:00
Carlo Nucera 9aae179f34 Correct bug, thoroughly document addToStoreSlow 2020-07-20 20:18:12 -04:00
Carlo Nucera 0ca9744694 Use heuristics to decide when to show the response 
Due to https://github.com/NixOS/nix/issues/3841 we don't know how print
different messages for different verbosity levels.
 2020-07-20 15:57:58 -04:00
John Ericson eab945f591 Merge remote-tracking branch 'upstream/master' into from-dump-stream 2020-07-20 17:29:23 +00:00
Eelco Dolstra 1c5f8bbfb5
Merge pull request #3822 from obsidiansystems/dump-thrice-fixme 
Optimize `addToStoreSlow` and remove `TeeParseSink`
 2020-07-20 18:55:05 +02:00
John Ericson ac2fc7ba1f
Apply suggestions from code review 
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
 2020-07-20 11:29:46 -04:00
Carlo Nucera fefd6c9e5f Merge branch 'master' of github.com:NixOS/nix into allow-relative-paths-in-store-option 2020-07-17 14:34:20 -04:00
Carlo Nucera 048e916f64 Merge branch 'master' of github.com:NixOS/nix into optional-derivation-output-storepath 2020-07-16 13:32:28 -04:00
Carlo Nucera e3a3406db8 Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors 2020-07-16 12:04:10 -04:00
John Ericson 5ea817dace Merge remote-tracking branch 'upstream/master' into hash-always-has-type 2020-07-16 14:58:53 +00:00
John Ericson 3dcca18c30 Fix bug in TeeSource 
We use this to simplify `LocalStore::addToStoreFromDump`.

Also, hope I fixed build error with old clang (used in Darwin CI).
 2020-07-16 13:39:27 +00:00
Eelco Dolstra 5517eee17e Generations API cleanup 2020-07-16 15:18:21 +02:00
Eelco Dolstra 3f264916db Merge remote-tracking branch 'origin/flakes' 2020-07-16 14:07:32 +02:00
John Ericson 68dfb8c6ae Optimize addToStoreSlow and remove TeeParseSink 2020-07-16 05:09:41 +00:00
John Ericson 5602637d9e Revert "LocalStore::addToStoreFromDump copy in chunks" 
This reverts commit 592851fb67. We don't
need this extra feature anymore
 2020-07-15 23:37:49 +00:00
John Ericson bc109648c4 Get rid of LocalStore::addToStoreCommon 
I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking
a store and then doing nothing too fancy with it.

`LocalStore::addToStore(...Path...)` is now just a simple wrapper with a
bare-bones sinkToSource of the right dump command.
 2020-07-15 23:14:30 +00:00
Carlo Nucera d090562348 Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type 2020-07-15 17:21:01 -04:00
John Ericson aff69e229c Merge remote-tracking branch 'obsidian/fix-add-to-star-narhash' into from-dump-stream 2020-07-15 21:05:29 +00:00
Carlo Nucera 2d2a10e79a Merge branch 'master' of github.com:NixOS/nix into add-body-to-network-errors 2020-07-15 13:06:08 -04:00
Eelco Dolstra cff2157185 Revert "LocalStore::addToStore(srcPath): Handle the flat case" 
This reverts commit a2c27022e9. See
addToStoreSlow(), we don't need to handle this case efficiently
anymore. In fact, we can almost remove the method/hashAlgo arguments
since the non-recursive and/or non-SHA256 are almost not used anymore.
 2020-07-15 12:49:03 +02:00
John Ericson 650c2c6558 Rename variable nar -> dump according to TODO 2020-07-14 21:28:50 +00:00
John Ericson 8173e7bfef Fix localhost::addToStore(...Path...) 
We were calculating the nar hash wrong when the file ingestion method
was flat. I don't think there's anything we can do in that case but dump
the file again, so that's what I do.

As an optomization, we again could reuse the original dump for just the
recursive and non-sha256 case, but I rather do that after this fix, and
after my other PRs which deduplicate this code.
 2020-07-14 21:12:11 +00:00