puck/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
4621 commits 19 branches 0 tags 68 MiB
Commit graph

2615 commits

Author SHA1 Message Date
Eelco Dolstra 8cffec8485 Remove failed build caching 
This feature was implemented for Hydra, but Hydra no longer uses it.
 2016-04-08 18:19:04 +02:00
Eelco Dolstra f398949b40 Make LocalStore thread-safe 
Necessary for multi-threaded commands like "nix verify-paths".
 2016-04-08 18:07:13 +02:00
Eelco Dolstra 05fbc606fc nix verify-paths: Add ‘--sigs-needed <N>’ flag 
This specifies the number of distinct signatures required to consider
each path "trusted".

Also renamed ‘--no-sigs’ to ‘--no-trust’ for the flag that disables
verifying whether a path is trusted (since a path can also be trusted
if it has no signatures, but was built locally).
 2016-04-07 15:16:57 +02:00
Eelco Dolstra 6b2ae52808 Use secret-key-files for verifying 2016-04-07 15:16:57 +02:00
Eelco Dolstra e39999ed48 Sign locally-built paths 
Locally-built paths are now signed automatically using the secret keys
specified by the ‘secret-key-files’ option.
 2016-04-07 15:16:57 +02:00
Eelco Dolstra dc82160164 Fix "tar: This does not look like a tar archive" with fetchTarball / -I http:// 
The 304 Not Modified was not handled correctly, so the empty result
from the conditional request would overwrite the previous tarball.
 2016-04-06 16:57:20 +02:00
Eelco Dolstra b654381eb3 Add "nix sign-paths" command 
E.g.

  $ nix sign-paths -k ./secret -r $(type -p geeqie)

signs geeqie and all its dependencies using the key in ./secret.
 2016-04-05 16:39:29 +02:00
Eelco Dolstra d0f5719c2a Add "nix copy-sigs" command 
This imports signatures from one store into another. E.g.

  $ nix copy-sigs -r /run/current-system -s https://cache.nixos.org/
  imported 595 signatures
 2016-04-05 16:39:23 +02:00
Eelco Dolstra 80da7a6375 Probably fix SQLITE_BUSY errors 2016-04-05 15:29:56 +02:00
Eelco Dolstra 37a337bcec throwSQLiteError(): Check for SIGINT so we don't loop forever 2016-04-04 15:08:08 +02:00
Eelco Dolstra 6e120b76ee Add missing -pthread 
https://hydra.nixos.org/build/33908385
 2016-03-31 12:42:48 +02:00
Eelco Dolstra 9cee600c88 LocalStore: Keep track of ultimately trusted paths 
These are content-addressed paths or outputs of locally performed
builds. They are trusted even if they don't have signatures, so "nix
verify-paths" won't complain about them.
 2016-03-30 17:35:48 +02:00
Eelco Dolstra 3d119f0a3b Improve the SQLite wrapper API 
In particular, this eliminates a bunch of boilerplate code.
 2016-03-30 15:50:45 +02:00
Eelco Dolstra d9c5e3bbf0 Factour out SQLite handling 2016-03-30 13:27:25 +02:00
Eelco Dolstra 2ae43ced9a Turn retrying SQLite transactions into a higher-order function 2016-03-30 12:04:27 +02:00
Eelco Dolstra 39a6abc0bc nix verify: Support checking against signatures in other stores 
Typical usage is to check local paths using the signatures from a
binary cache:

  $ nix verify-paths -r /run/current-system -s https://cache.nixos.org
  path ‘/nix/store/c1k4zqfb74wba5sn4yflb044gvap0x6k-nixos-system-mandark-16.03.git.fc2d7a5M’ is untrusted
  ...
  checked 844 paths, 119 untrusted
 2016-03-30 11:39:34 +02:00
Eelco Dolstra 88541569a2 HttpBinaryCacheStore: Treat 403 errors as 404 2016-03-30 11:17:51 +02:00
Eelco Dolstra de88004a9d CurlDownloader: Fix HTTP error processing 2016-03-30 11:17:35 +02:00
Eelco Dolstra ab3ce1cc13 Improve SIGINT handling in multi-threaded programs 
The flag remembering whether an Interrupted exception was thrown is
now thread-local. Thus, all threads will (eventually) throw
Interrupted. Previously, one thread would throw Interrupted, and then
the other threads wouldn't see that they were supposed to quit.
 2016-03-29 16:37:16 +02:00
Eelco Dolstra 4f34c40398 Add "nix verify-store" command 
Like "nix-store --verify --check-contents", but with the same
advantages as "nix verify-paths".
 2016-03-29 16:37:16 +02:00
Eelco Dolstra 784ee35c80 Add "nix verify-paths" command 
Unlike "nix-store --verify-path", this command verifies signatures in
addition to store path contents, is multi-threaded (especially useful
when verifying binary caches), and has a progress indicator.

Example use:

$ nix verify-paths --store https://cache.nixos.org -r $(type -p thunderbird)
...
[17/132 checked] checking ‘/nix/store/rawakphadqrqxr6zri2rmnxh03gqkrl3-autogen-5.18.6’
 2016-03-29 16:37:16 +02:00
Eelco Dolstra 0ebe69dc67 Re-enable sync_with_stdio 
Otherwise writing to std::cerr is not thread-safe (in particular,
lines will be randomly duplicated).
 2016-03-29 16:37:16 +02:00
Eelco Dolstra 374198ad6d Move signature support from NarInfo to ValidPathInfo 2016-03-24 11:41:00 +01:00
Eelco Dolstra 11525377e1 Typos 2016-03-24 11:27:58 +01:00
Eelco Dolstra 8b7839b608 HttpBinaryCacheStore: Make thread-safe 2016-03-24 11:10:05 +01:00
Eelco Dolstra c7d44bad00 Drop support for daemon socket path >= 108 characters 
Doing a chdir() is a bad idea in multi-threaded programs, leading to
failures such as

  error: cannot connect to daemon at ‘/nix/var/nix/daemon-socket/socket’: No such file or directory

Since Linux doesn't have a connectat() syscall like FreeBSD, there is
no way we can support this in a race-free way.
 2016-03-23 17:16:16 +01:00
Eelco Dolstra 056b3ecfa4 LocalStoreAccessor::stat: Handle ENOTDIR 
Closes https://github.com/NixOS/hydra/pull/286.
 2016-03-23 11:17:46 +01:00
Eelco Dolstra bb1034316d Don't overload dumpPath() 2016-03-22 14:21:45 +01:00
Eelco Dolstra 712b616a84 Move signatures from NarInfo to ValidPathInfo 
This allows queryPathInfo() to return signatures.
 2016-03-21 18:05:47 +01:00
Eelco Dolstra cebc150b7c nix: Add --store flag 
This is a bit user-friendlier than using $NIX_REMOTE.
 2016-03-21 18:03:36 +01:00
Eelco Dolstra 1c5f73f529 Add Store::dumpPath() method 
This allows applying nix-store --verify-path to binary cache stores:

  NIX_REMOTE=https://cache.nixos.org nix-store --verify-path /nix/store/s5c7...
 2016-03-21 17:55:57 +01:00
Eelco Dolstra 87295b9844 Drop support for upgrading from Nix <= 0.12 2016-03-21 15:09:03 +01:00
Eelco Dolstra 02654f782f Fix Darwin build 
http://hydra.nixos.org/build/33279996
 2016-03-15 12:11:27 +01:00
Eelco Dolstra 00a75b1cd2 Fix another mismatched tag 
http://hydra.nixos.org/build/33279570
 2016-03-14 15:32:34 +01:00
Eelco Dolstra dc4a71aae5 Fix build on clang due to -Wmismatched-tags 
http://hydra.nixos.org/build/33073389
 2016-03-14 12:37:30 +01:00
Eelco Dolstra e7c76f7274 BinaryCacheStore::isValidPath(): Use .narinfo cache 
If a path is in the .narinfo cache, obviously it's valid.
 2016-03-14 12:33:06 +01:00
Eelco Dolstra 674c5ff64f Merge pull request #834 from abbradar/master 
Propagate path context via builtins.readFile
 2016-03-10 20:17:03 +01:00
Eelco Dolstra b4e0335d4d Add option binary-cache-secret-key-file for signing binary caches 2016-03-04 17:45:22 +01:00
Eelco Dolstra af7cdb1096 BinaryCacheStore: Remove publicKeyFile argument 
The public key can be derived from the secret key, so there's no need
for the user to supply it separately.
 2016-03-04 17:45:22 +01:00
Eelco Dolstra 42bc395b63 Eliminate some large string copying 2016-03-04 16:49:56 +01:00
Eelco Dolstra ce113c32d2 Add warning about "nix" being experimental 2016-03-04 15:54:41 +01:00
Eelco Dolstra 7c9d7a253c Merge branch 'new-cli' 2016-03-04 15:21:23 +01:00
Nikolay Amiantov f7f0116dd7 Propagate path context via builtins.readFile 2016-03-04 14:04:51 +03:00
Eelco Dolstra 76f1ba4f3b Add file missing from 201b48de60 2016-03-03 18:03:34 +01:00
Nathan Zadoks fe2be8f016 build.cc: fs.h doesn't appear to be necessary anymore 2016-03-03 14:11:00 +01:00
Eelco Dolstra 5a8455c85e Provide function required by Hydra 2016-03-02 18:21:48 +01:00
Eelco Dolstra 25eea97873 Force stack trace for boost format errors 2016-03-02 15:46:07 +01:00
Eelco Dolstra 201b48de60 Add an HTTP binary cache store 
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
 2016-03-02 15:46:07 +01:00
Eelco Dolstra 0402b6398d Eliminate local-binary-cache-store.hh 2016-03-02 15:46:07 +01:00
Eelco Dolstra 68a5414982 Make store implementations pluggable 
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
 2016-03-02 15:46:07 +01:00
Eelco Dolstra 6055d84beb Fix reading symlinks 
The st_size field of a symlink doesn't have to be correct, e.g. for
/dev/fd symlinks.
 2016-03-02 15:46:07 +01:00
Eelco Dolstra 0a62d9b3d7 Remove bad assertion 2016-03-02 15:46:07 +01:00
Eelco Dolstra 89a2fa68ac FdSource: track number of bytes read 2016-03-02 15:46:07 +01:00
Eelco Dolstra cca4a8dc1a importPaths(): Optionally add NARs to binary cache accessor 
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
 2016-03-02 15:46:07 +01:00
Eelco Dolstra e9c50064b5 Add an HTTP binary cache store 
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
 2016-02-29 18:15:20 +01:00
Eelco Dolstra 6170bb474b Eliminate local-binary-cache-store.hh 2016-02-29 16:14:39 +01:00
Eelco Dolstra 0b907321cc Make store implementations pluggable 
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
 2016-02-29 16:11:11 +01:00
Eelco Dolstra 012f8d187c Fix reading symlinks 
The st_size field of a symlink doesn't have to be correct, e.g. for
/dev/fd symlinks.
 2016-02-28 03:27:34 +01:00
Eelco Dolstra d0344dd2c1 Remove bad assertion 2016-02-26 21:43:59 +01:00
Eelco Dolstra 581e1bc5b4 FdSource: track number of bytes read 2016-02-26 16:16:08 +01:00
Eelco Dolstra bcc9943cee importPaths(): Optionally add NARs to binary cache accessor 
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
 2016-02-26 15:20:10 +01:00
Eelco Dolstra 00b2c05749 nix: Add commands to query contents of NARs / binary caches 
For example,

  $ NIX_REMOTE=file:///my-cache nix ls-store -lR /nix/store/f4kbgl8shhyy76rkk3nbxr0lz8d2ip7q-binutils-2.23.1
  dr-xr-xr-x                    0 ./bin
  -r-xr-xr-x                30748 ./bin/addr2line
  -r-xr-xr-x                66973 ./bin/ar
  ...

Similarly, "nix ls-nar" lists the contents of a NAR file, "nix
cat-nar" extracts a file from a NAR file, and "nix cat-store" extract
a file from a Nix store.
 2016-02-25 17:57:00 +01:00
Eelco Dolstra 24a8f9e27b Merge branch 'master' into new-cli 2016-02-25 17:48:35 +01:00
Eelco Dolstra 1042c10fd0 Add NAR / Store accessor abstraction 
This is primary to allow hydra-queue-runner to extract files like
"nix-support/hydra-build-products" from NARs in binary caches.
 2016-02-25 17:43:19 +01:00
Eelco Dolstra c5bc571861 Fix short boolean flags 2016-02-25 13:31:34 +01:00
Eelco Dolstra 7873cfb18d Fix build 2016-02-25 11:55:05 +01:00
Eelco Dolstra f1bdeac986 Merge branch 'master' into new-cli 2016-02-25 11:25:11 +01:00
Eelco Dolstra 152b1d6bf9 deletePath(): Succeed if path doesn't exist 
Also makes it robust against concurrent deletions.
 2016-02-24 17:44:55 +01:00
Eelco Dolstra 28e7e29abd Eliminate reserveSpace flag 2016-02-24 17:44:55 +01:00
Eelco Dolstra 5a64e66268 LocalStore::addTextToStore(): Don't read the path we just wrote 
This eliminates some unnecessary (presumably cached) I/O.
 2016-02-24 17:11:31 +01:00
Eelco Dolstra 9ccbd55c5b BinaryCacheStore: Implement addToStore() 
So now you can do

  $ NIX_REMOTE=file:///tmp/binary-cache nix-instantiate '<nixpkgs>' -A hello

and lots of other operations.
 2016-02-24 16:52:28 +01:00
Eelco Dolstra 30e9d01516 openStoreAt(): Initialise the binary cache 2016-02-24 16:52:07 +01:00
Eelco Dolstra ba0a81d14f Show a proper error message for unimplemented binary cache operations 2016-02-24 14:57:30 +01:00
Eelco Dolstra 45c83e5f9b Typo 2016-02-24 14:49:28 +01:00
Eelco Dolstra 263187a2ec Move BinaryCacheStore / LocalBinaryCacheStore from Hydra 
So you can now do:

  $ NIX_REMOTE=file:///tmp/binary-cache nix-store -qR /nix/store/...
 2016-02-24 14:48:16 +01:00
Eelco Dolstra b584a0e7de Merge pull request #816 from tsion/incomplete-parse 
Throw a specific error for incomplete parse errors.
 2016-02-24 13:34:56 +01:00
Eelco Dolstra bf2adf72c4 std::condition_variable_any -> std::condition_variable 
The latter is supposed to be more efficient.
 2016-02-24 13:31:46 +01:00
Eelco Dolstra ccdbf589a4 C++ templates are just a glorified macro facility 2016-02-24 13:07:32 +01:00
Eelco Dolstra 5f862658c3 Remove bad daemon connections from the pool 
This is necessary for long-running processes like hydra-queue-runner:
if a nix-daemon worker is killed, we need to stop reusing that
connection.
 2016-02-24 11:39:56 +01:00
Scott Olson 6498adb002 Throw a specific error for incomplete parse errors. 
`nix-repl` will use this for deciding whether to keep waiting for input or
error out right away.
 2016-02-24 04:32:21 -06:00
Eelco Dolstra d5626bf4c1 Pool<T>: Allow a maximum pool size 2016-02-23 16:40:16 +01:00
Eelco Dolstra e292144d46 RemoteStore: Make thread-safe 
This allows a RemoteStore object to be used safely from multiple
threads concurrently. It will make multiple daemon connections if
necessary.

Note: pool.hh and sync.hh have been copied from the Hydra source tree.
 2016-02-23 15:00:59 +01:00
Vladimír Čunát 03cbb9ad59 nix-env: respect meta.outputsToInstall 
Discussed on https://github.com/NixOS/nixpkgs/pull/12653#discussion_r51601849
 2016-02-23 14:19:14 +01:00
Eelco Dolstra c0b7a8a0b5 Move ref<t> into a separate header 2016-02-23 13:53:31 +01:00
Eelco Dolstra 8f71bc33d5 Doh 2016-02-22 17:33:27 +01:00
Eelco Dolstra 840056af04 Make OpenSSL usage thread-safe 
OpenSSL can randomly segfault unless we register a callback function
to do locking.

https://www.openssl.org/docs/manmaster/crypto/threads.html
 2016-02-22 15:51:14 +01:00
Eelco Dolstra d361901bfe curl: Set CURLOPT_NOSIGNAL 
Otherwise using curl is not safe in multi-threaded applications
because it installs a SIGALRM handler.
 2016-02-22 15:51:14 +01:00
Eelco Dolstra 7a173a7be1 JSONObject: Support floats and booleans 2016-02-19 14:24:04 +01:00
Eelco Dolstra b49d323ce2 Fix build without sodium 
http://hydra.nixos.org/build/32085949
 2016-02-17 12:42:14 +01:00
Eelco Dolstra c4d22997f3 Add C++ functions for .narinfo processing / signing 
This is currently only used by the Hydra queue runner rework, but like
eff5021eaa it presumably will be useful
for the C++ rewrite of nix-push and
download-from-binary-cache. (@shlevy)
 2016-02-16 16:38:44 +01:00
Eelco Dolstra 5ac27053e9 Rename ValidPathInfo::hash -> narHash for consistency 2016-02-16 11:49:12 +01:00
Tristan Hume 9b05d5848c Fix to-base16 description 2016-02-16 10:15:58 +01:00
Eelco Dolstra eff5021eaa Add xz compression function 
This is used by the Hydra queue runner, but since it may also be
useful for the C++ rewrite of nix-push, I'm putting it here.
 2016-02-15 21:45:56 +01:00
Eelco Dolstra bfdacb712c decompressXZ: Ensure that lzma_end() is called 
Otherwise we might leak memory.
 2016-02-15 21:26:28 +01:00
Eelco Dolstra c8f4d89a34 Expose the export magic value and move LocalStore::queryReferences to Store 2016-02-15 15:01:26 +01:00
Eelco Dolstra e03d6e0998 Fix broken number parsing in fromJSON 
The call to tmp_number.append had its arguments mixed up. Also, JSON
does not allow a trailing "," after array/object members.
 2016-02-15 15:01:26 +01:00
Eelco Dolstra d089372565 Add function to extract hash part of a store path 2016-02-15 15:01:26 +01:00
Eelco Dolstra b3e8d72770 Merge pull request #762 from ctheune/ctheune-floats 
Implement floats
 2016-02-12 12:49:59 +01:00
Eelco Dolstra ae4a3cfa03 Move addPermRoot into Store 2016-02-11 16:14:42 +01:00
Eelco Dolstra fd205fb6f8 ref<T>: Add cast operator 2016-02-11 15:32:48 +01:00