Eelco Dolstra
15f4d4fd43
Drop libbz2 / zlib / lzma dependency + style fixes
2021-04-15 13:55:22 +02:00
Yorick
8a0c00b856
Use libarchive for all compression
2021-03-10 22:34:29 +01:00
Eelco Dolstra
faa31f4084
Sink: Use std::string_view
2020-12-02 14:17:27 +01:00
Ben Burdette
e4fb9a3849
remove 'format' from Error constructor calls
2020-04-21 17:07:07 -06:00
Eelco Dolstra
e6bd88878e
Improve gzip error message
2019-12-13 19:05:26 +01:00
Tom Bereknyei
c6295a3afd
Initial gzip support
...
Closes #3256
2019-12-13 03:34:15 -05:00
Eelco Dolstra
901dfc7978
Fix build failure if parallel xz is not available
2018-09-17 16:36:11 +02:00
Eelco Dolstra
6317c65937
Fix warnings in compression.cc
2018-08-21 15:20:23 +02:00
Eelco Dolstra
d3761f5f8b
Fix Brotli decompression in 'nix log'
...
This didn't work anymore since decompression was only done in the
non-coroutine case.
Decompressors are now sinks, just like compressors.
Also fixed a bug in bzip2 API handling (we have to handle BZ_RUN_OK
rather than BZ_OK), which we didn't notice because there was a missing
'throw':
if (ret != BZ_OK)
CompressionError("error while compressing bzip2 file");
2018-08-06 15:40:29 +02:00
Eelco Dolstra
fa4def3d46
Require libbrotli
2018-08-06 14:06:54 +02:00
Eelco Dolstra
4a2c948943
Fix bzip2 compression of files > 4 GiB
...
Bzip2's 'avail_in' parameter is declared as an unsigned int, so
assigning a size_t length to it led to silent truncation.
Fixes #2111 .
2018-05-02 12:54:30 +02:00
Eelco Dolstra
28eb9621cc
Fix build
2018-03-22 13:46:19 +01:00
Eelco Dolstra
3e6b194d78
decompress(): Use a Source and Sink
...
This allows decompression to happen in O(1) memory.
2018-03-16 20:35:59 +01:00
Will Dietz
c6209030c4
compression: make parallel sink separate class
2018-02-11 13:50:28 -06:00
Will Dietz
a0bdc96726
compression: print warning if parallel requested but not supported
2018-02-11 13:03:47 -06:00
Will Dietz
5a082ad15a
configure.ac: check if lzma has MT support, fix deb build/etc.
2018-02-09 21:02:25 -06:00
AmineChikhaoui
55ecdfe2a8
make multi threaded compression configurable and use single threaded
...
by default.
2018-02-07 17:54:08 +01:00
AmineChikhaoui
9d1e22f743
set block size to 0 to let the lzma lib choose the right one, add
...
some comments about possible improvements wrt memory usage/threading.
2018-02-07 11:18:55 +01:00
AmineChikhaoui
bc7e3a4dd6
support multi threaded xz encoder, this might be particularly useful in
...
the case of hydra where the overhead of single threaded encoding is more
noticeable e.g most of the time spent in "Sending inputs"/"Receiving outputs"
is due to compression while the actual upload to the binary cache seems
to be negligible.
2018-02-06 22:42:02 +01:00
Will Dietz
9dd2b8ac7b
use libbrotli directly when available
...
* Look for both 'brotli' and 'bro' as external command,
since upstream has renamed it in newer versions.
If neither are found, current runtime behavior
is preserved: try to find 'bro' on PATH.
* Limit amount handed to BrotliEncoderCompressStream
to ensure interrupts are processed in a timely manner.
Testing shows negligible performance impact.
(Other compression sinks don't seem to require this)
2017-12-30 20:26:33 -06:00
Jörg Thalheim
2fd8f8bb99
Replace Unicode quotes in user-facing strings by ASCII
...
Relevant RFC: NixOS/rfcs#4
$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
Eelco Dolstra
0a7ca24c26
Fix xz decompression
...
Fixes #1285 .
2017-03-22 11:53:33 +01:00
Eelco Dolstra
895a74a814
LocalFSStore::getBuildLog(): Handle corrupted logs
2017-03-21 19:23:35 +01:00
Eelco Dolstra
ed5c0f69f2
Don't hang in decompression if bzip2 data ends prematurely
2017-03-21 19:23:35 +01:00
Eelco Dolstra
25dff2b7db
runProgram(): Distinguish between empty input and no input
...
For example, if we call brotli with an empty input, it shouldn't read
from the caller's stdin.
2017-03-15 16:50:19 +01:00
Eelco Dolstra
8b1d65bebe
S3BinaryCacheStore: Support compression of narinfo and log files
...
You can now set the store parameter "text-compression=br" to compress
textual files in the binary cache (i.e. narinfo and logs) using
Brotli. This sets the Content-Encoding header; the extension of
compressed files is unchanged.
You can separately specify the compression of log files using
"log-compression=br". This is useful when you don't want to compress
narinfo files for backward compatibility.
2017-03-15 16:49:28 +01:00
Eelco Dolstra
2691498b5c
Fix assertion failure on SIGINT
...
nix: src/libutil/compression.cc:142: virtual nix::XzSink::~XzSink(): Assertion `finished' failed.
2017-03-15 16:49:06 +01:00
Eelco Dolstra
e8186085e0
Add support for brotli compression
...
Build logs on cache.nixos.org are compressed using Brotli (since this
allows them to be decompressed automatically by Chrome and Firefox),
so it's handy if "nix log" can decompress them.
2017-03-15 16:49:06 +01:00
Eelco Dolstra
215b70f51e
Revert "Get rid of unicode quotes ( #1140 )"
...
This reverts commit f78126bfd6
. There
really is no need for such a massive change...
2016-11-26 00:38:01 +01:00
Guillaume Maudoux
f78126bfd6
Get rid of unicode quotes ( #1140 )
2016-11-25 15:48:27 +01:00
Eelco Dolstra
0d4a10e910
Do compression in a sink
2016-05-04 16:16:53 +02:00
Eelco Dolstra
538a64e8c3
Add a Store::addToStore() variant that accepts a NAR
...
As a side effect, this ensures that signatures are propagated when
copying paths between stores.
Also refactored import/export to make use of this.
2016-05-04 16:15:54 +02:00
Eelco Dolstra
33de2bc080
Fix segfault in xz compression/decompression
2016-05-04 11:50:12 +02:00
Eelco Dolstra
4dde0b0562
BinaryCacheStore: Support bzip2 compression
2016-04-29 17:43:37 +02:00
Eelco Dolstra
5acb691402
BinaryCacheStore: Support "none" compression method
2016-04-29 17:02:57 +02:00
Eelco Dolstra
58c84cda3b
Make compression interruptible
2016-04-22 18:15:02 +02:00
Eelco Dolstra
d1b0909894
BinaryCacheStore::readFile(): Return a shared_ptr to a string
...
This allows readFile() to indicate that a file doesn't exist, and
might eliminate some large string copying.
2016-04-15 15:39:48 +02:00
Eelco Dolstra
eff5021eaa
Add xz compression function
...
This is used by the Hydra queue runner, but since it may also be
useful for the C++ rewrite of nix-push, I'm putting it here.
2016-02-15 21:45:56 +01:00
Eelco Dolstra
bfdacb712c
decompressXZ: Ensure that lzma_end() is called
...
Otherwise we might leak memory.
2016-02-15 21:26:28 +01:00
FrankHB
607d91517e
Fixed unresolved 'BUFSIZ'
...
`BUFSIZ` is defined in header `<cstdio>`.
2016-01-05 13:33:40 +01:00
Eelco Dolstra
1f735a3440
<nix/fetchurl.nix>: Support xz-compressed NARs
2015-10-30 12:34:30 +01:00