philiptaron/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
4795 commits 26 branches 4 tags 76 MiB
Commit graph

1224 commits

Author SHA1 Message Date
Eelco Dolstra dead8189ac Typo 2016-06-01 16:56:22 +02:00
Eelco Dolstra e24e2caaaf Skip substituters with an incompatible store directory 2016-06-01 16:40:49 +02:00
Eelco Dolstra cf198952d0 HttpBinaryCacheStore: Fix caching of WantMassQuery 
Also, test HttpBinaryCacheStore in addition to LocalBinaryCacheStore.
 2016-06-01 16:24:17 +02:00
Eelco Dolstra 7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra a9fa5e050a Shut up some clang warnings 2016-05-31 13:31:04 +02:00
Eelco Dolstra 10f3a2e5f2 Fix clang build failure 
Apparently opinion is divided on whether [[noreturn]] is allowed on a
lambda: http://stackoverflow.com/questions/26888805/how-to-declare-a-lambdas-operator-as-noreturn

http://hydra.nixos.org/build/36462100
 2016-05-31 13:23:54 +02:00
Eelco Dolstra c2d27d30cf nix-copy-closure / build-remote.pl: Disable signature checking 
This restores the Nix 1.11 behaviour.
 2016-05-31 11:48:05 +02:00
Eelco Dolstra cd35f0280c Fix build 2016-05-30 16:12:50 +02:00
Eelco Dolstra 57d33013ce Check signatures before downloading the substitute 2016-05-30 15:18:12 +02:00
Eelco Dolstra bac123ddd9 Test the NAR info cache 2016-05-30 15:18:12 +02:00
Eelco Dolstra 3593c8285d Re-implement binary cache signature checking 
This is now done in LocalStore::addToStore(), rather than in the
binary cache substituter (which no longer exists).
 2016-05-30 15:18:12 +02:00
Eelco Dolstra 12ddbad458 LocalStore::addToStore: Verify hash of the imported path 2016-05-30 15:18:12 +02:00
Eelco Dolstra e222484401 Re-implement the WantMassQuery property of binary caches 2016-05-30 15:18:12 +02:00
Eelco Dolstra b66ab6cdbc Fix repair during substitution 2016-05-30 15:18:12 +02:00
Eelco Dolstra 3be2e71ab3 BinaryCacheStore: Remove buildPaths() / ensurePath() 2016-05-30 15:18:12 +02:00
Eelco Dolstra 75d2492f20 Make the aws-cpp-sdk dependency optional 2016-05-04 17:16:48 +02:00
Eelco Dolstra 6c75cf69c3 Cleanup: Remove singleton() 2016-05-04 16:16:53 +02:00
Eelco Dolstra 0d4a10e910 Do compression in a sink 2016-05-04 16:16:53 +02:00
Eelco Dolstra c6a21aed07 More GC fixes 2016-05-04 16:16:53 +02:00
Eelco Dolstra 538a64e8c3 Add a Store::addToStore() variant that accepts a NAR 
As a side effect, this ensures that signatures are propagated when
copying paths between stores.

Also refactored import/export to make use of this.
 2016-05-04 16:15:54 +02:00
Eelco Dolstra f435f82475 Remove OpenSSL-based signing 2016-05-04 11:01:48 +02:00
Eelco Dolstra dfebfc835f Add a copyStorePath() utility function 2016-05-04 11:01:48 +02:00
Eelco Dolstra 38539b943a Add fetchgit builtin 
The function builtins.fetchgit fetches Git repositories at evaluation
time, similar to builtins.fetchTarball. (Perhaps the name should be
changed, being confusing with respect to Nixpkgs's fetchgit function,
with works at build time.)

Example:

  (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello

or

  (import (builtins.fetchgit {
    url = git://github.com/NixOS/nixpkgs-channels;
    rev = "nixos-16.03";
  }) {}).hello

Note that the result does not contain a .git directory.
 2016-04-29 20:47:36 +02:00
Eelco Dolstra 4dde0b0562 BinaryCacheStore: Support bzip2 compression 2016-04-29 17:43:37 +02:00
Eelco Dolstra 6963de2091 nix verify --all: Support local binary caches 2016-04-29 17:34:31 +02:00
Eelco Dolstra 0dd988d2e3 Better error message 2016-04-29 17:33:22 +02:00
Eelco Dolstra 5acb691402 BinaryCacheStore: Support "none" compression method 2016-04-29 17:02:57 +02:00
Eelco Dolstra 8e065c6b3e BinaryCacheStore: Make the signing key a parameter 2016-04-29 16:47:20 +02:00
Eelco Dolstra f6aee2f477 BinaryCacheStore::queryPathInfo: Don't check signatures 
Other stores don't do this either. It's up to the caller to check
signatures.
 2016-04-29 16:28:57 +02:00
Eelco Dolstra 95d20dfde9 Allow parameters in store URIs 
This is to allow store-specific configuration,
e.g. s3://my-cache?compression=bzip2&secret-key=/path/to/key.
 2016-04-29 16:26:16 +02:00
Eelco Dolstra aa3bc3d5dc Eliminate the substituter mechanism 
Substitution is now simply a Store -> Store copy operation, most
typically from BinaryCacheStore to LocalStore.
 2016-04-29 13:57:08 +02:00
Eelco Dolstra 21e9d183cc Really handle carriage return 2016-04-28 14:27:00 +02:00
Eelco Dolstra ce5776758d Fix error message 2016-04-28 14:12:10 +02:00
Eelco Dolstra cc804d0dc6 Handle carriage return 2016-04-25 19:18:45 +02:00
Eelco Dolstra 5761827d5b Show the log tail when a build fails 
If --no-build-output is given (which will become the default for the
"nix" command at least), show the last 10 lines of the build output if
the build fails.
 2016-04-25 19:18:45 +02:00
Eelco Dolstra 6e1b099279 Remove --print-build-trace 
This was added to support Hydra, but Hydra no longer uses it.
 2016-04-25 19:18:45 +02:00
Eelco Dolstra 9eba2c3945 Fix "path is not in the Nix store" during GC 2016-04-25 19:18:45 +02:00
Eelco Dolstra 41633f9f73 Improved logging abstraction 
This also gets rid of --log-type, since the nested log type isn't
useful in a multi-threaded situation, and nobody cares about the
"pretty" log type.
 2016-04-25 19:18:45 +02:00
Eelco Dolstra 0207272b28 BinaryCacheStore: When adding a path, ensure the references are valid 
This prevents copying a partial closure to a binary cache.
 2016-04-22 12:15:06 +02:00
Eelco Dolstra 21ef1670b3 Fix test failures 2016-04-21 18:21:25 +02:00
Eelco Dolstra 7d14f5c331 Implement S3BinaryCacheStore::queryAllValidPaths() 
This allows commands like "nix verify --all" or "nix path-info --all"
to work on S3 caches.

Unfortunately, this requires some ugly hackery: when querying the
contents of the bucket, we don't want to have to read every .narinfo
file. But the S3 bucket keys only include the hash part of each store
path, not the name part. So as a special exception
queryAllValidPaths() can now return store paths *without* the name
part, and queryPathInfo() accepts such store paths (returning a
ValidPathInfo object containing the full name).
 2016-04-21 17:53:47 +02:00
Eelco Dolstra d155d80155 Move S3BinaryCacheStore from Hydra 
This allows running arbitrary Nix commands against an S3 binary cache.

To do: make this a compile time option to prevent a dependency on
aws-sdk-cpp.
 2016-04-21 16:08:51 +02:00
Eelco Dolstra ddea253ff8 RemoteStore: Propagate InvalidPath exceptions from the daemon 2016-04-20 15:28:07 +02:00
Eelco Dolstra c0c4ddcd9c BinaryCacheStore: Insert new paths into the disk cache 2016-04-20 15:27:48 +02:00
Eelco Dolstra 451ebf24ce Cache path info lookups in SQLite 
This re-implements the binary cache database in C++, allowing it to be
used by other Store backends, in particular the S3 backend.
 2016-04-20 14:12:38 +02:00
Eelco Dolstra e0204f8d46 Move path info caching from BinaryCacheStore to Store 
Caching path info is generally useful. For instance, it speeds up "nix
path-info -rS /run/current-system" (i.e. showing the closure sizes of
all paths in the closure of the current system) from 5.6s to 0.15s.

This also eliminates some APIs like Store::queryDeriver() and
Store::queryReferences().
 2016-04-19 18:52:53 +02:00
Dan Peebles 608b0265e1 Print out all bad references/requisites at once 
Also updates tests to check for new information. Fixes #799
 2016-04-16 19:58:26 -04:00
Eelco Dolstra 0423787086 Make the .narinfo cache bigger 2016-04-15 15:39:48 +02:00
Eelco Dolstra a7d8eaba54 BinaryCacheStore: Do negative caching of .narinfo lookups 2016-04-15 15:39:48 +02:00
Eelco Dolstra d1b0909894 BinaryCacheStore::readFile(): Return a shared_ptr to a string 
This allows readFile() to indicate that a file doesn't exist, and
might eliminate some large string copying.
 2016-04-15 15:39:48 +02:00
Eelco Dolstra c045630522 Support channel:<channel-name> URIs 
For convenience, you can now say

  $ nix-env -f channel:nixos-16.03 -iA hello

instead of

  $ nix-env -f https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz -iA hello

Similarly,

  $ nix-shell -I channel:nixpkgs-unstable -p hello
  $ nix-build channel:nixos-15.09 -A hello

Abstracting over the NixOS/Nixpkgs channels location also allows us to
use a more efficient transport (e.g. Git) in the future.
 2016-04-14 17:26:57 +02:00
Eelco Dolstra 3c1c6b8f00 Set RLIMIT_CORE to 0, infinity in builders 
This prevents the builder from being affected by whatever the host
system limits happen to be.
 2016-04-14 13:39:14 +02:00
Eelco Dolstra 96515b0c0d Fix std::atomic_flag build failure 
http://hydra.nixos.org/build/34453794
 2016-04-14 12:50:01 +02:00
Eelco Dolstra 867967265b Remove manifest support 
Manifests have been superseded by binary caches for years. This also
gets rid of nix-pull, nix-generate-patches and bsdiff/bspatch.
 2016-04-11 16:20:15 +02:00
Eelco Dolstra 8cffec8485 Remove failed build caching 
This feature was implemented for Hydra, but Hydra no longer uses it.
 2016-04-08 18:19:04 +02:00
Eelco Dolstra f398949b40 Make LocalStore thread-safe 
Necessary for multi-threaded commands like "nix verify-paths".
 2016-04-08 18:07:13 +02:00
Eelco Dolstra 05fbc606fc nix verify-paths: Add ‘--sigs-needed <N>’ flag 
This specifies the number of distinct signatures required to consider
each path "trusted".

Also renamed ‘--no-sigs’ to ‘--no-trust’ for the flag that disables
verifying whether a path is trusted (since a path can also be trusted
if it has no signatures, but was built locally).
 2016-04-07 15:16:57 +02:00
Eelco Dolstra 6b2ae52808 Use secret-key-files for verifying 2016-04-07 15:16:57 +02:00
Eelco Dolstra e39999ed48 Sign locally-built paths 
Locally-built paths are now signed automatically using the secret keys
specified by the ‘secret-key-files’ option.
 2016-04-07 15:16:57 +02:00
Eelco Dolstra dc82160164 Fix "tar: This does not look like a tar archive" with fetchTarball / -I http:// 
The 304 Not Modified was not handled correctly, so the empty result
from the conditional request would overwrite the previous tarball.
 2016-04-06 16:57:20 +02:00
Eelco Dolstra b654381eb3 Add "nix sign-paths" command 
E.g.

  $ nix sign-paths -k ./secret -r $(type -p geeqie)

signs geeqie and all its dependencies using the key in ./secret.
 2016-04-05 16:39:29 +02:00
Eelco Dolstra d0f5719c2a Add "nix copy-sigs" command 
This imports signatures from one store into another. E.g.

  $ nix copy-sigs -r /run/current-system -s https://cache.nixos.org/
  imported 595 signatures
 2016-04-05 16:39:23 +02:00
Eelco Dolstra 80da7a6375 Probably fix SQLITE_BUSY errors 2016-04-05 15:29:56 +02:00
Eelco Dolstra 37a337bcec throwSQLiteError(): Check for SIGINT so we don't loop forever 2016-04-04 15:08:08 +02:00
Eelco Dolstra 9cee600c88 LocalStore: Keep track of ultimately trusted paths 
These are content-addressed paths or outputs of locally performed
builds. They are trusted even if they don't have signatures, so "nix
verify-paths" won't complain about them.
 2016-03-30 17:35:48 +02:00
Eelco Dolstra 3d119f0a3b Improve the SQLite wrapper API 
In particular, this eliminates a bunch of boilerplate code.
 2016-03-30 15:50:45 +02:00
Eelco Dolstra d9c5e3bbf0 Factour out SQLite handling 2016-03-30 13:27:25 +02:00
Eelco Dolstra 2ae43ced9a Turn retrying SQLite transactions into a higher-order function 2016-03-30 12:04:27 +02:00
Eelco Dolstra 88541569a2 HttpBinaryCacheStore: Treat 403 errors as 404 2016-03-30 11:17:51 +02:00
Eelco Dolstra de88004a9d CurlDownloader: Fix HTTP error processing 2016-03-30 11:17:35 +02:00
Eelco Dolstra 784ee35c80 Add "nix verify-paths" command 
Unlike "nix-store --verify-path", this command verifies signatures in
addition to store path contents, is multi-threaded (especially useful
when verifying binary caches), and has a progress indicator.

Example use:

$ nix verify-paths --store https://cache.nixos.org -r $(type -p thunderbird)
...
[17/132 checked] checking ‘/nix/store/rawakphadqrqxr6zri2rmnxh03gqkrl3-autogen-5.18.6’
 2016-03-29 16:37:16 +02:00
Eelco Dolstra 374198ad6d Move signature support from NarInfo to ValidPathInfo 2016-03-24 11:41:00 +01:00
Eelco Dolstra 11525377e1 Typos 2016-03-24 11:27:58 +01:00
Eelco Dolstra 8b7839b608 HttpBinaryCacheStore: Make thread-safe 2016-03-24 11:10:05 +01:00
Eelco Dolstra c7d44bad00 Drop support for daemon socket path >= 108 characters 
Doing a chdir() is a bad idea in multi-threaded programs, leading to
failures such as

  error: cannot connect to daemon at ‘/nix/var/nix/daemon-socket/socket’: No such file or directory

Since Linux doesn't have a connectat() syscall like FreeBSD, there is
no way we can support this in a race-free way.
 2016-03-23 17:16:16 +01:00
Eelco Dolstra 056b3ecfa4 LocalStoreAccessor::stat: Handle ENOTDIR 
Closes https://github.com/NixOS/hydra/pull/286.
 2016-03-23 11:17:46 +01:00
Eelco Dolstra bb1034316d Don't overload dumpPath() 2016-03-22 14:21:45 +01:00
Eelco Dolstra 712b616a84 Move signatures from NarInfo to ValidPathInfo 
This allows queryPathInfo() to return signatures.
 2016-03-21 18:05:47 +01:00
Eelco Dolstra 1c5f73f529 Add Store::dumpPath() method 
This allows applying nix-store --verify-path to binary cache stores:

  NIX_REMOTE=https://cache.nixos.org nix-store --verify-path /nix/store/s5c7...
 2016-03-21 17:55:57 +01:00
Eelco Dolstra 87295b9844 Drop support for upgrading from Nix <= 0.12 2016-03-21 15:09:03 +01:00
Eelco Dolstra 02654f782f Fix Darwin build 
http://hydra.nixos.org/build/33279996
 2016-03-15 12:11:27 +01:00
Eelco Dolstra 00a75b1cd2 Fix another mismatched tag 
http://hydra.nixos.org/build/33279570
 2016-03-14 15:32:34 +01:00
Eelco Dolstra e7c76f7274 BinaryCacheStore::isValidPath(): Use .narinfo cache 
If a path is in the .narinfo cache, obviously it's valid.
 2016-03-14 12:33:06 +01:00
Eelco Dolstra b4e0335d4d Add option binary-cache-secret-key-file for signing binary caches 2016-03-04 17:45:22 +01:00
Eelco Dolstra af7cdb1096 BinaryCacheStore: Remove publicKeyFile argument 
The public key can be derived from the secret key, so there's no need
for the user to supply it separately.
 2016-03-04 17:45:22 +01:00
Eelco Dolstra 42bc395b63 Eliminate some large string copying 2016-03-04 16:49:56 +01:00
Eelco Dolstra 7c9d7a253c Merge branch 'new-cli' 2016-03-04 15:21:23 +01:00
Eelco Dolstra 76f1ba4f3b Add file missing from 201b48de60 2016-03-03 18:03:34 +01:00
Nathan Zadoks fe2be8f016 build.cc: fs.h doesn't appear to be necessary anymore 2016-03-03 14:11:00 +01:00
Eelco Dolstra 5a8455c85e Provide function required by Hydra 2016-03-02 18:21:48 +01:00
Eelco Dolstra 201b48de60 Add an HTTP binary cache store 
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
 2016-03-02 15:46:07 +01:00
Eelco Dolstra 0402b6398d Eliminate local-binary-cache-store.hh 2016-03-02 15:46:07 +01:00
Eelco Dolstra 68a5414982 Make store implementations pluggable 
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
 2016-03-02 15:46:07 +01:00
Eelco Dolstra 0a62d9b3d7 Remove bad assertion 2016-03-02 15:46:07 +01:00
Eelco Dolstra cca4a8dc1a importPaths(): Optionally add NARs to binary cache accessor 
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
 2016-03-02 15:46:07 +01:00
Eelco Dolstra e9c50064b5 Add an HTTP binary cache store 
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
 2016-02-29 18:15:20 +01:00
Eelco Dolstra 6170bb474b Eliminate local-binary-cache-store.hh 2016-02-29 16:14:39 +01:00
Eelco Dolstra 0b907321cc Make store implementations pluggable 
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
 2016-02-29 16:11:11 +01:00
Eelco Dolstra d0344dd2c1 Remove bad assertion 2016-02-26 21:43:59 +01:00
Eelco Dolstra bcc9943cee importPaths(): Optionally add NARs to binary cache accessor 
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
 2016-02-26 15:20:10 +01:00