lunaphied/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
6579 commits 2 branches 3 tags 72 MiB
Commit graph

127 commits

Author SHA1 Message Date
Daiderd Jordan 898823b67d
s3: make scheme configurable 
This enables using for http for S3 request for debugging or
implementations that don't have https configured.  This is not a problem
for binary caches since they should not contain sensitive information.
Both package signatures and AWS auth already protect against tampering.
 2018-12-07 23:55:38 +01:00
Eelco Dolstra ebeea068d5
Merge pull request #2216 from dtzWill/fix/curl-760-timeout 
download: if there are active requests, never sleep for 10s
 2018-11-20 19:07:56 +01:00
Eelco Dolstra 9750430003
Ensure download thread liveness 
* Don't wait forever for the client to remove data from the
  buffer. This does mean that the buffer can grow without bounds
  (e.g. when downloading is faster than writing to disk), but meh.

* Don't hold the state lock while calling the sink. The sink could
  take any amount of time to process the data (in particular when it's
  actually a coroutine), so we don't want to block the download
  thread.
 2018-09-26 21:47:34 +02:00
Will Dietz 28418af920 download: fix size reported to progress bar 2018-09-03 14:12:54 -05:00
Eelco Dolstra d3761f5f8b
Fix Brotli decompression in 'nix log' 
This didn't work anymore since decompression was only done in the
non-coroutine case.

Decompressors are now sinks, just like compressors.

Also fixed a bug in bzip2 API handling (we have to handle BZ_RUN_OK
rather than BZ_OK), which we didn't notice because there was a missing
'throw':

  if (ret != BZ_OK)
      CompressionError("error while compressing bzip2 file");
 2018-08-06 15:40:29 +02:00
Eelco Dolstra 4361a4331f
Fix reporting of HTTP body size when a result callback is used 2018-08-06 11:31:14 +02:00
Graham Christensen 49a53c1d3f s3 binary cache: support specifying an endpoint 
Works for uploading and not downloading.
 2018-08-02 08:19:50 -04:00
Eelco Dolstra 4b4adbc93d
Fix race downloading into a sink 
This fixes 'error 10 while decompressing xz file'.

https://hydra.nixos.org/build/78308551
 2018-07-31 17:26:04 +02:00
Eelco Dolstra b920b90857
Detect redirect loops 
Fixes #2225.
 2018-06-18 10:36:19 +02:00
Will Dietz 6bf7dc1baf download: if there are active requests, never sleep for 10s 
In some versions/configurations libcurl doesn't handle timeouts
(especially DNS timeouts) in a way that wakes curl_multi_wait.

This doesn't appear to be a problem if using c-ares, FWIW.
 2018-06-08 21:12:43 -05:00
Eelco Dolstra 691b7582c7
Handle HTTP 410 Gone errors 2018-06-05 16:03:32 +02:00
Eelco Dolstra ca06a9cea7
Improve binary cache fallback test 2018-06-05 15:26:12 +02:00
Eelco Dolstra a2ec7a3bfd
Further improve upload messages 2018-06-05 14:37:26 +02:00
Will Dietz 54b1c59643 download.cc: fix readCallback return type (int -> size_t) 
I'm not sure if curl ever asks for enough data at once
for truncation to occur but better safe than sorry.
 2018-06-01 10:59:26 -05:00
Eelco Dolstra 73357500ac
Improve binary cache upload messages 
Don't say "download" when we mean "upload".
 2018-06-01 14:14:22 +02:00
Eelco Dolstra 5a654fd7dd
getDownloader(): Simplify 2018-05-30 13:42:30 +02:00
Eelco Dolstra e87e4a60d6
Make HttpBinaryCacheStore::narFromPath() run in constant memory 
This reduces memory consumption of

  nix copy --from https://cache.nixos.org --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 176 MiB to 82 MiB. (The remaining memory is probably due to xz
decompression overhead.)

Issue https://github.com/NixOS/nix/issues/1681.
Issue https://github.com/NixOS/nix/issues/1969.
 2018-05-30 13:42:29 +02:00
Eelco Dolstra 81ea8bd5ce
Simplify the callback mechanism 2018-05-30 13:34:37 +02:00
Eelco Dolstra c1d445ecec
Move some Download-specific settings to download.cc 2018-05-30 13:29:50 +02:00
Will Dietz ca9c6cb95d libstore/download: remove unused function "getTime()" 2018-05-25 19:49:45 -05:00
Yorick b9289e4855
make sure not to use cached channels for nix-channel --update 
fixes #1964
 2018-05-09 16:18:20 +02:00
Shea Levy 58f9ae0dcd
Merge branch 'hash-mismatch' of git://github.com/LnL7/nix 2018-04-23 08:40:34 -04:00
Peter Simons 040acdcee2
Merge pull request #1707 from dtzWill/fix/issue-1692 
ignore "interrupted" exception in progress callback
 2018-04-21 16:51:19 +02:00
Daiderd Jordan 13d4d1c0a2
download: make hash mismatch error consistent with fetchurl 2018-04-20 21:07:32 +02:00
Will Dietz 74da813912 download: improve error for hash mismatch ("store mismatch") 
Fixes #1905
 2018-03-22 15:22:06 -05:00
Will Dietz 59b32403f2 download.cc: fix path for cached files 2018-03-22 15:05:46 -05:00
Eelco Dolstra eb75bc5afb
Don't retry on CURLE_SSL_CACERT_BADFILE 
The certificates won't get any better if we retry.
 2018-03-16 12:03:18 +01:00
Shea Levy cfdbfa6b2c
Catch more possible instances of passing NULL to memcpy. 
Actually fixes #1976.
 2018-03-14 23:53:43 -04:00
Eelco Dolstra 64e486ab63
Fix downloadCached() with a chroot store 
E.g.

  nix run --store ~/my-nix -f channel:nixos-17.03 hello -c hello

This problem was mentioned in #1897.
 2018-02-23 13:00:42 +01:00
Eelco Dolstra 88f4f0231b Fix building without aws-sdk-cpp 2018-01-31 22:34:51 +01:00
Eelco Dolstra 6270b2e50f
Merge branch 'http-binary-cache-put-upsert' of https://github.com/adelbertc/nix 2018-01-31 15:10:12 +01:00
Eelco Dolstra 478e3e4649
Indent properly 2018-01-31 13:46:38 +01:00
Shea Levy 1d5d277ac7
HttpBinaryCacheStore: Support upsertFile with PUT. 
Some servers, such as Artifactory, allow uploading with PUT and BASIC
auth. This allows nix copy to work to upload binaries to those
servers.

Worked on together with @adelbertc
 2018-01-26 11:12:30 -08:00
Eelco Dolstra 2896bb6826
Don't retry CURLE_URL_MALFORMAT 2018-01-19 14:05:08 +01:00
Benjamin Hipple d76c4fe770 Fix for builds with system libcurl < 7.30 
CentOS 7.4 and RHEL 7.4 ship with libcurl-devel-7.29.0-42.el7.x86_64; this flag
was added in 7.30.0
https://curl.haxx.se/libcurl/c/CURLMOPT_MAX_TOTAL_CONNECTIONS.html
 2017-12-09 20:02:21 -05:00
Will Dietz 79f5c296c0 ignore "interrupted" exception in progress callback 
Context/discusson:

https://github.com/NixOS/nix/issues/1692#issuecomment-348282301
 2017-11-30 12:45:54 -06:00
Eelco Dolstra 8956ae1987
Add a "profile" option to S3BinaryCacheStore 
This allows specifying the AWS configuration profile to use. E.g.

  nix copy --from s3://my-cache?profile=aws-dev-account /nix/store/cf3isrlqavvd5w7rpky1fa8j9lcnlggm-...
 2017-11-15 14:18:20 +01:00
Eelco Dolstra 1969f357b7
Add fetchMercurial primop 
E.g.

  $ nix eval '(fetchMercurial https://www.mercurial-scm.org/repo/hello)'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "82e55d328c8ca4ee16520036c0aaace03a5beb65"; revCount = 1; shortRev = "82e55d328c8c"; }

  $ nix eval '(fetchMercurial { url = https://www.mercurial-scm.org/repo/hello; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; })'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; revCount = 0; shortRev = "0a04b987be5a"; }

  $ nix eval '(fetchMercurial /tmp/unclean-hg-tree)'
  { branch = "default"; outPath = "/nix/store/cm750cdw1x8wfpm3jq7mz09r30l9r024-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "000000000000"; }
 2017-11-01 17:45:32 +01:00
Eelco Dolstra 66ddbef754
fetchurl/fetchTarball: Respect name changes 
The computation of urlHash didn't take the name into account, so
subsequent fetchurl calls with the same URL but a different name would
resolve to the same cached store path.
 2017-10-30 10:22:58 +01:00
Dan Peebles 476493dbf5 Reverse retry logic to retry in all but a few cases 
It was getting too much like whac-a-mole listing all the retriable error
conditions, so we now retry by default and list the cases where retrying
is almost certainly hopeless.
 2017-10-02 23:22:02 -04:00
Eelco Dolstra aca4f7dff0
Don't remove Content-Encoding in fetchurl / nix-prefetch-url 
Fixes #1568.
 2017-09-18 11:07:28 +02:00
Eelco Dolstra 8215b75d36
Abort curl downloads if there is no progress for 5 minutes 
Maybe this will fix the curl hangs on macOS. (We could also use
CURLOPT_TIMEOUT but that seems more of a sledgehammer.)
 2017-09-01 16:51:26 +02:00
Eelco Dolstra fd73c1e20a
Add an activity for binary cache queries 2017-08-31 15:25:58 +02:00
Eelco Dolstra 2cc345b95f
Give activities a verbosity level again 
And print them (separately from the progress bar) given sufficient -v
flags.
 2017-08-28 19:13:24 +02:00
Eelco Dolstra c137c0a5eb
Allow activities to be nested 
In particular, this allows more relevant activities ("substituting X")
to supersede inferior ones ("downloading X").
 2017-08-25 17:49:40 +02:00
Eelco Dolstra 40bffe0a43
Progress indicator: Cleanup 2017-08-16 20:56:03 +02:00
Eelco Dolstra c36467ad2e
Improve substitution progress indicator 
E.g.

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
 2017-08-16 20:56:02 +02:00
Jörg Thalheim 2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII 
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
 2017-07-30 12:32:45 +01:00
Eelco Dolstra 8e8caf7f3e fetchTarball: Prevent concurrent downloads of the same file 
Fixes #849.
 2017-07-11 23:21:24 +02:00
Eelco Dolstra c0015e87af
Support base-64 hashes 
Also simplify the Hash API.

Fixes #1437.
 2017-07-04 15:07:41 +02:00