qyriad/lix
1
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
4963 commits 57 branches 0 tags 73 MiB
Commit graph

1312 commits

Author SHA1 Message Date
Shea Levy 59124228b3 nix-channel: implement in c++ 2016-08-11 11:34:43 -04:00
Eelco Dolstra 66adbdfd97 HttpBinaryCacheStore: Retry on transient HTTP errors 
This makes us more robust against 500 errors from CloudFront or S3
(assuming the 500 error isn't cached by CloudFront...).
 2016-08-10 18:08:23 +02:00
Eelco Dolstra 9204ea7294 Show HTTP error codes 2016-08-10 18:05:35 +02:00
Eelco Dolstra f294623d1d SQLite:: Add some convenience 2016-08-10 18:05:35 +02:00
Eelco Dolstra 6cb4bdf152 FSAccessor: Throw InvalidPath 2016-08-10 18:05:35 +02:00
Eelco Dolstra f43823f676 NarAccessor: Fix handling of non-executable files 2016-08-10 18:05:35 +02:00
Eelco Dolstra d961c29c9c Mark content-addressed paths in the Nix database and in .narinfo 
This allows such paths to be imported without signatures.
 2016-08-10 18:05:35 +02:00
Eelco Dolstra 36a51ecab3 LocalFSStore::narFromPath(): Handle the diverted store case 2016-08-10 18:05:35 +02:00
Eelco Dolstra 3eb6217508 Add a "root" parameter to local stores 
This makes it easier to create a diverted store, i.e.

  NIX_REMOTE="local?root=/tmp/root"

instead of

  NIX_REMOTE="local?real=/tmp/root/nix/store&state=/tmp/root/nix/var/nix" NIX_LOG_DIR=/tmp/root/nix/var/log
 2016-08-10 18:05:35 +02:00
Eelco Dolstra 2fad86f361 Remove $NIX_DB_DIR 
This variable has no reason to exist, given $NIX_STATE_DIR.
 2016-08-10 18:05:35 +02:00
Eelco Dolstra be64fbb501 RemoteStore / nix-daemon: Drop support for Nix < 1.0 2016-08-10 18:05:35 +02:00
Eelco Dolstra ee22a91ab8 makeFixedOutputPath(): Drop superfluous HashType argument 2016-07-26 21:25:52 +02:00
Eelco Dolstra 06bbfb6004 builtins.{fetchurl,fetchTarball}: Support a sha256 attribute 
Also, allow builtins.{fetchurl,fetchTarball} in restricted mode if a
hash is specified.
 2016-07-26 21:16:52 +02:00
Shea Levy ee3032e4de Merge branch 'find-runtime-roots-c++' 2016-07-24 07:33:49 -04:00
Shea Levy 18b0808475 Respect --keep-going when a substituter fails. 
Fixes #977
 2016-07-23 09:16:08 -04:00
Eelco Dolstra e19c90fc6b Fix random failures caused by CurlDownloader modifying a shared string 2016-07-21 18:35:31 +02:00
Eelco Dolstra ea8e8df6c7 NarInfo::NarInfo(): Ensure that we get a NAR size/hash 2016-07-21 18:35:12 +02:00
Eelco Dolstra d57981bac4 Store::queryMissing(): Use a thread pool 
For one particular NixOS configuration, this cut the runtime of
"nix-store -r --dry-run" from 6m51s to 3.4s. It also fixes a bug in
the size calculation that was causing certain paths to be counted
twice, e.g. before:

  these paths will be fetched (1249.98 MiB download, 2995.74 MiB unpacked):

and after:

  these paths will be fetched (1219.56 MiB download, 2862.17 MiB unpacked):
 2016-07-21 17:50:39 +02:00
Eelco Dolstra a728780fbd Store::computeFSClosure(): Use thread pool 
This speeds up queries against the binary cache.
 2016-07-21 14:13:35 +02:00
Shea Levy 3c46fe62b8 find-runtime-roots: fold into gc.cc 2016-07-21 07:04:41 -04:00
Shea Levy cb5e7254b6 Modernize AutoCloseFD 2016-07-11 15:44:44 -04:00
Eelco Dolstra 74dd603495 Re-implement negative binary cache lookup caching 2016-06-20 17:39:05 +02:00
Eelco Dolstra c68e5913c7 Run builds in a user namespace 
This way, all builds appear to have a uid/gid of 0 inside the
chroot. In the future, this may allow using programs like
systemd-nspawn inside builds, but that will require assigning a larger
UID/GID map to the build.

Issue #625.
 2016-06-09 18:30:39 +02:00
Eelco Dolstra 202683a4fc Use O_CLOEXEC in most places 2016-06-09 16:37:08 +02:00
Eelco Dolstra 9bdd949cfd Fix "creating statement: table ValidPaths has no column named ultimate" 2016-06-03 16:06:42 +02:00
Eelco Dolstra 5e51ffb1c2 Support sandbox builds by non-root users 
This allows an unprivileged user to perform builds on a diverted store
(i.e. where the physical store location differs from the logical
location).

Example:

  $ NIX_LOG_DIR=/tmp/log NIX_REMOTE="local?real=/tmp/store&state=/tmp/var" nix-build -E \
    'with import <nixpkgs> {}; runCommand "foo" { buildInputs = [procps nettools]; } "id; ps; ifconfig; echo $out > $out"'

will do a build in the Nix store physically in /tmp/store but
logically in /nix/store (and thus using substituters for the latter).
 2016-06-03 15:59:28 +02:00
Eelco Dolstra 2f8b0e557b Consistent quotes 2016-06-03 14:03:09 +02:00
Eelco Dolstra 080d607975 Don't call shared_from_this() from a destructor 2016-06-03 14:02:44 +02:00
Eelco Dolstra 30c8c3ba2d Fix some more references to storeDir 2016-06-03 14:02:44 +02:00
Eelco Dolstra a8dfdc52b8 Merge pull request #925 from abbradar/master 
Show both cycle ends
 2016-06-03 13:58:51 +02:00
Nikolay Amiantov c87a56f4d0 Show both cycle ends 2016-06-03 14:25:37 +03:00
Eelco Dolstra 75361b6dce Fix narrowing conversion from int64_t to bool 
http://hydra.nixos.org/build/36613774
 2016-06-03 13:20:46 +02:00
Eelco Dolstra d64e0c1b64 Make derivationFromPath work on diverted stores 2016-06-02 18:43:36 +02:00
Eelco Dolstra a24f2c9b84 nix run: Mount the Nix store in a private namespace 
This is a convenience command to allow users who are not privileged to
create /nix/store to use Nix with regular binary caches. For example,

  $ NIX_REMOTE="local?state=$HOME/nix/var&real=/$HOME/nix/store" nix run firefox bashInteractive

will download Firefox and bash from cache.nixos.org, then start a
shell in which $HOME/nix/store is mounted on /nix/store.
 2016-06-02 16:51:43 +02:00
Eelco Dolstra 4494000e04 LocalStore: Allow the physical and logical store directories to differ 
This is primarily to subsume the functionality of the
copy-from-other-stores substituter. For example, in the NixOS
installer, we can now do (assuming we're in the target chroot, and the
Nix store of the installation CD is bind-mounted on /tmp/nix):

  $ nix-build ... --option substituters 'local?state=/tmp/nix/var&real=/tmp/nix/store'

However, unlike copy-from-other-stores, this also allows write access
to such a store. One application might be fetching substitutes for
/nix/store in a situation where the user doesn't have sufficient
privileges to create /nix, e.g.:

  $ NIX_REMOTE="local?state=/home/alice/nix/var&real=/home/alice/nix/store" nix-build ...
 2016-06-02 16:02:48 +02:00
Eelco Dolstra 064816ab98 Respect build-use-substitutes 2016-06-02 16:02:48 +02:00
Eelco Dolstra 812c0dfbe2 Allow setting the state directory as a store parameter 
E.g. "local?store=/tmp/store&state=/tmp/var".
 2016-06-02 16:02:48 +02:00
Eelco Dolstra f2682e6e18 <nix/fetchurl.nix>: Use tarballs.nixos.org 
This ensures that the disappearance of upstream bootstrap tarballs
(e.g. https://github.com/NixOS/nixpkgs-channels/pull/1) doesn't break
stdenv rebuilds.
 2016-06-01 17:11:51 +02:00
Eelco Dolstra dead8189ac Typo 2016-06-01 16:56:22 +02:00
Eelco Dolstra e24e2caaaf Skip substituters with an incompatible store directory 2016-06-01 16:40:49 +02:00
Eelco Dolstra cf198952d0 HttpBinaryCacheStore: Fix caching of WantMassQuery 
Also, test HttpBinaryCacheStore in addition to LocalBinaryCacheStore.
 2016-06-01 16:24:17 +02:00
Eelco Dolstra 7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra a9fa5e050a Shut up some clang warnings 2016-05-31 13:31:04 +02:00
Eelco Dolstra 10f3a2e5f2 Fix clang build failure 
Apparently opinion is divided on whether [[noreturn]] is allowed on a
lambda: http://stackoverflow.com/questions/26888805/how-to-declare-a-lambdas-operator-as-noreturn

http://hydra.nixos.org/build/36462100
 2016-05-31 13:23:54 +02:00
Eelco Dolstra c2d27d30cf nix-copy-closure / build-remote.pl: Disable signature checking 
This restores the Nix 1.11 behaviour.
 2016-05-31 11:48:05 +02:00
Eelco Dolstra cd35f0280c Fix build 2016-05-30 16:12:50 +02:00
Eelco Dolstra 57d33013ce Check signatures before downloading the substitute 2016-05-30 15:18:12 +02:00
Eelco Dolstra bac123ddd9 Test the NAR info cache 2016-05-30 15:18:12 +02:00
Eelco Dolstra 3593c8285d Re-implement binary cache signature checking 
This is now done in LocalStore::addToStore(), rather than in the
binary cache substituter (which no longer exists).
 2016-05-30 15:18:12 +02:00
Eelco Dolstra 12ddbad458 LocalStore::addToStore: Verify hash of the imported path 2016-05-30 15:18:12 +02:00
Eelco Dolstra e222484401 Re-implement the WantMassQuery property of binary caches 2016-05-30 15:18:12 +02:00
Eelco Dolstra b66ab6cdbc Fix repair during substitution 2016-05-30 15:18:12 +02:00
Eelco Dolstra 3be2e71ab3 BinaryCacheStore: Remove buildPaths() / ensurePath() 2016-05-30 15:18:12 +02:00
Eelco Dolstra 75d2492f20 Make the aws-cpp-sdk dependency optional 2016-05-04 17:16:48 +02:00
Eelco Dolstra 6c75cf69c3 Cleanup: Remove singleton() 2016-05-04 16:16:53 +02:00
Eelco Dolstra 0d4a10e910 Do compression in a sink 2016-05-04 16:16:53 +02:00
Eelco Dolstra c6a21aed07 More GC fixes 2016-05-04 16:16:53 +02:00
Eelco Dolstra 538a64e8c3 Add a Store::addToStore() variant that accepts a NAR 
As a side effect, this ensures that signatures are propagated when
copying paths between stores.

Also refactored import/export to make use of this.
 2016-05-04 16:15:54 +02:00
Eelco Dolstra f435f82475 Remove OpenSSL-based signing 2016-05-04 11:01:48 +02:00
Eelco Dolstra dfebfc835f Add a copyStorePath() utility function 2016-05-04 11:01:48 +02:00
Eelco Dolstra 38539b943a Add fetchgit builtin 
The function builtins.fetchgit fetches Git repositories at evaluation
time, similar to builtins.fetchTarball. (Perhaps the name should be
changed, being confusing with respect to Nixpkgs's fetchgit function,
with works at build time.)

Example:

  (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello

or

  (import (builtins.fetchgit {
    url = git://github.com/NixOS/nixpkgs-channels;
    rev = "nixos-16.03";
  }) {}).hello

Note that the result does not contain a .git directory.
 2016-04-29 20:47:36 +02:00
Eelco Dolstra 4dde0b0562 BinaryCacheStore: Support bzip2 compression 2016-04-29 17:43:37 +02:00
Eelco Dolstra 6963de2091 nix verify --all: Support local binary caches 2016-04-29 17:34:31 +02:00
Eelco Dolstra 0dd988d2e3 Better error message 2016-04-29 17:33:22 +02:00
Eelco Dolstra 5acb691402 BinaryCacheStore: Support "none" compression method 2016-04-29 17:02:57 +02:00
Eelco Dolstra 8e065c6b3e BinaryCacheStore: Make the signing key a parameter 2016-04-29 16:47:20 +02:00
Eelco Dolstra f6aee2f477 BinaryCacheStore::queryPathInfo: Don't check signatures 
Other stores don't do this either. It's up to the caller to check
signatures.
 2016-04-29 16:28:57 +02:00
Eelco Dolstra 95d20dfde9 Allow parameters in store URIs 
This is to allow store-specific configuration,
e.g. s3://my-cache?compression=bzip2&secret-key=/path/to/key.
 2016-04-29 16:26:16 +02:00
Eelco Dolstra aa3bc3d5dc Eliminate the substituter mechanism 
Substitution is now simply a Store -> Store copy operation, most
typically from BinaryCacheStore to LocalStore.
 2016-04-29 13:57:08 +02:00
Eelco Dolstra 21e9d183cc Really handle carriage return 2016-04-28 14:27:00 +02:00
Eelco Dolstra ce5776758d Fix error message 2016-04-28 14:12:10 +02:00
Eelco Dolstra cc804d0dc6 Handle carriage return 2016-04-25 19:18:45 +02:00
Eelco Dolstra 5761827d5b Show the log tail when a build fails 
If --no-build-output is given (which will become the default for the
"nix" command at least), show the last 10 lines of the build output if
the build fails.
 2016-04-25 19:18:45 +02:00
Eelco Dolstra 6e1b099279 Remove --print-build-trace 
This was added to support Hydra, but Hydra no longer uses it.
 2016-04-25 19:18:45 +02:00
Eelco Dolstra 9eba2c3945 Fix "path is not in the Nix store" during GC 2016-04-25 19:18:45 +02:00
Eelco Dolstra 41633f9f73 Improved logging abstraction 
This also gets rid of --log-type, since the nested log type isn't
useful in a multi-threaded situation, and nobody cares about the
"pretty" log type.
 2016-04-25 19:18:45 +02:00
Eelco Dolstra 0207272b28 BinaryCacheStore: When adding a path, ensure the references are valid 
This prevents copying a partial closure to a binary cache.
 2016-04-22 12:15:06 +02:00
Eelco Dolstra 21ef1670b3 Fix test failures 2016-04-21 18:21:25 +02:00
Eelco Dolstra 7d14f5c331 Implement S3BinaryCacheStore::queryAllValidPaths() 
This allows commands like "nix verify --all" or "nix path-info --all"
to work on S3 caches.

Unfortunately, this requires some ugly hackery: when querying the
contents of the bucket, we don't want to have to read every .narinfo
file. But the S3 bucket keys only include the hash part of each store
path, not the name part. So as a special exception
queryAllValidPaths() can now return store paths *without* the name
part, and queryPathInfo() accepts such store paths (returning a
ValidPathInfo object containing the full name).
 2016-04-21 17:53:47 +02:00
Eelco Dolstra d155d80155 Move S3BinaryCacheStore from Hydra 
This allows running arbitrary Nix commands against an S3 binary cache.

To do: make this a compile time option to prevent a dependency on
aws-sdk-cpp.
 2016-04-21 16:08:51 +02:00
Eelco Dolstra ddea253ff8 RemoteStore: Propagate InvalidPath exceptions from the daemon 2016-04-20 15:28:07 +02:00
Eelco Dolstra c0c4ddcd9c BinaryCacheStore: Insert new paths into the disk cache 2016-04-20 15:27:48 +02:00
Eelco Dolstra 451ebf24ce Cache path info lookups in SQLite 
This re-implements the binary cache database in C++, allowing it to be
used by other Store backends, in particular the S3 backend.
 2016-04-20 14:12:38 +02:00
Eelco Dolstra e0204f8d46 Move path info caching from BinaryCacheStore to Store 
Caching path info is generally useful. For instance, it speeds up "nix
path-info -rS /run/current-system" (i.e. showing the closure sizes of
all paths in the closure of the current system) from 5.6s to 0.15s.

This also eliminates some APIs like Store::queryDeriver() and
Store::queryReferences().
 2016-04-19 18:52:53 +02:00
Dan Peebles 608b0265e1 Print out all bad references/requisites at once 
Also updates tests to check for new information. Fixes #799
 2016-04-16 19:58:26 -04:00
Eelco Dolstra 0423787086 Make the .narinfo cache bigger 2016-04-15 15:39:48 +02:00
Eelco Dolstra a7d8eaba54 BinaryCacheStore: Do negative caching of .narinfo lookups 2016-04-15 15:39:48 +02:00
Eelco Dolstra d1b0909894 BinaryCacheStore::readFile(): Return a shared_ptr to a string 
This allows readFile() to indicate that a file doesn't exist, and
might eliminate some large string copying.
 2016-04-15 15:39:48 +02:00
Eelco Dolstra c045630522 Support channel:<channel-name> URIs 
For convenience, you can now say

  $ nix-env -f channel:nixos-16.03 -iA hello

instead of

  $ nix-env -f https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz -iA hello

Similarly,

  $ nix-shell -I channel:nixpkgs-unstable -p hello
  $ nix-build channel:nixos-15.09 -A hello

Abstracting over the NixOS/Nixpkgs channels location also allows us to
use a more efficient transport (e.g. Git) in the future.
 2016-04-14 17:26:57 +02:00
Eelco Dolstra 3c1c6b8f00 Set RLIMIT_CORE to 0, infinity in builders 
This prevents the builder from being affected by whatever the host
system limits happen to be.
 2016-04-14 13:39:14 +02:00
Eelco Dolstra 96515b0c0d Fix std::atomic_flag build failure 
http://hydra.nixos.org/build/34453794
 2016-04-14 12:50:01 +02:00
Eelco Dolstra 867967265b Remove manifest support 
Manifests have been superseded by binary caches for years. This also
gets rid of nix-pull, nix-generate-patches and bsdiff/bspatch.
 2016-04-11 16:20:15 +02:00
Eelco Dolstra 8cffec8485 Remove failed build caching 
This feature was implemented for Hydra, but Hydra no longer uses it.
 2016-04-08 18:19:04 +02:00
Eelco Dolstra f398949b40 Make LocalStore thread-safe 
Necessary for multi-threaded commands like "nix verify-paths".
 2016-04-08 18:07:13 +02:00
Eelco Dolstra 05fbc606fc nix verify-paths: Add ‘--sigs-needed <N>’ flag 
This specifies the number of distinct signatures required to consider
each path "trusted".

Also renamed ‘--no-sigs’ to ‘--no-trust’ for the flag that disables
verifying whether a path is trusted (since a path can also be trusted
if it has no signatures, but was built locally).
 2016-04-07 15:16:57 +02:00
Eelco Dolstra 6b2ae52808 Use secret-key-files for verifying 2016-04-07 15:16:57 +02:00
Eelco Dolstra e39999ed48 Sign locally-built paths 
Locally-built paths are now signed automatically using the secret keys
specified by the ‘secret-key-files’ option.
 2016-04-07 15:16:57 +02:00
Eelco Dolstra dc82160164 Fix "tar: This does not look like a tar archive" with fetchTarball / -I http:// 
The 304 Not Modified was not handled correctly, so the empty result
from the conditional request would overwrite the previous tarball.
 2016-04-06 16:57:20 +02:00
Eelco Dolstra b654381eb3 Add "nix sign-paths" command 
E.g.

  $ nix sign-paths -k ./secret -r $(type -p geeqie)

signs geeqie and all its dependencies using the key in ./secret.
 2016-04-05 16:39:29 +02:00
Eelco Dolstra d0f5719c2a Add "nix copy-sigs" command 
This imports signatures from one store into another. E.g.

  $ nix copy-sigs -r /run/current-system -s https://cache.nixos.org/
  imported 595 signatures
 2016-04-05 16:39:23 +02:00
Eelco Dolstra 80da7a6375 Probably fix SQLITE_BUSY errors 2016-04-05 15:29:56 +02:00
Eelco Dolstra 37a337bcec throwSQLiteError(): Check for SIGINT so we don't loop forever 2016-04-04 15:08:08 +02:00
Eelco Dolstra 9cee600c88 LocalStore: Keep track of ultimately trusted paths 
These are content-addressed paths or outputs of locally performed
builds. They are trusted even if they don't have signatures, so "nix
verify-paths" won't complain about them.
 2016-03-30 17:35:48 +02:00
Eelco Dolstra 3d119f0a3b Improve the SQLite wrapper API 
In particular, this eliminates a bunch of boilerplate code.
 2016-03-30 15:50:45 +02:00
Eelco Dolstra d9c5e3bbf0 Factour out SQLite handling 2016-03-30 13:27:25 +02:00
Eelco Dolstra 2ae43ced9a Turn retrying SQLite transactions into a higher-order function 2016-03-30 12:04:27 +02:00
Eelco Dolstra 88541569a2 HttpBinaryCacheStore: Treat 403 errors as 404 2016-03-30 11:17:51 +02:00
Eelco Dolstra de88004a9d CurlDownloader: Fix HTTP error processing 2016-03-30 11:17:35 +02:00
Eelco Dolstra 784ee35c80 Add "nix verify-paths" command 
Unlike "nix-store --verify-path", this command verifies signatures in
addition to store path contents, is multi-threaded (especially useful
when verifying binary caches), and has a progress indicator.

Example use:

$ nix verify-paths --store https://cache.nixos.org -r $(type -p thunderbird)
...
[17/132 checked] checking ‘/nix/store/rawakphadqrqxr6zri2rmnxh03gqkrl3-autogen-5.18.6’
 2016-03-29 16:37:16 +02:00
Eelco Dolstra 374198ad6d Move signature support from NarInfo to ValidPathInfo 2016-03-24 11:41:00 +01:00
Eelco Dolstra 11525377e1 Typos 2016-03-24 11:27:58 +01:00
Eelco Dolstra 8b7839b608 HttpBinaryCacheStore: Make thread-safe 2016-03-24 11:10:05 +01:00
Eelco Dolstra c7d44bad00 Drop support for daemon socket path >= 108 characters 
Doing a chdir() is a bad idea in multi-threaded programs, leading to
failures such as

  error: cannot connect to daemon at ‘/nix/var/nix/daemon-socket/socket’: No such file or directory

Since Linux doesn't have a connectat() syscall like FreeBSD, there is
no way we can support this in a race-free way.
 2016-03-23 17:16:16 +01:00
Eelco Dolstra 056b3ecfa4 LocalStoreAccessor::stat: Handle ENOTDIR 
Closes https://github.com/NixOS/hydra/pull/286.
 2016-03-23 11:17:46 +01:00
Eelco Dolstra bb1034316d Don't overload dumpPath() 2016-03-22 14:21:45 +01:00
Eelco Dolstra 712b616a84 Move signatures from NarInfo to ValidPathInfo 
This allows queryPathInfo() to return signatures.
 2016-03-21 18:05:47 +01:00
Eelco Dolstra 1c5f73f529 Add Store::dumpPath() method 
This allows applying nix-store --verify-path to binary cache stores:

  NIX_REMOTE=https://cache.nixos.org nix-store --verify-path /nix/store/s5c7...
 2016-03-21 17:55:57 +01:00
Eelco Dolstra 87295b9844 Drop support for upgrading from Nix <= 0.12 2016-03-21 15:09:03 +01:00
Eelco Dolstra 02654f782f Fix Darwin build 
http://hydra.nixos.org/build/33279996
 2016-03-15 12:11:27 +01:00
Eelco Dolstra 00a75b1cd2 Fix another mismatched tag 
http://hydra.nixos.org/build/33279570
 2016-03-14 15:32:34 +01:00
Eelco Dolstra e7c76f7274 BinaryCacheStore::isValidPath(): Use .narinfo cache 
If a path is in the .narinfo cache, obviously it's valid.
 2016-03-14 12:33:06 +01:00
Eelco Dolstra b4e0335d4d Add option binary-cache-secret-key-file for signing binary caches 2016-03-04 17:45:22 +01:00
Eelco Dolstra af7cdb1096 BinaryCacheStore: Remove publicKeyFile argument 
The public key can be derived from the secret key, so there's no need
for the user to supply it separately.
 2016-03-04 17:45:22 +01:00
Eelco Dolstra 42bc395b63 Eliminate some large string copying 2016-03-04 16:49:56 +01:00
Eelco Dolstra 7c9d7a253c Merge branch 'new-cli' 2016-03-04 15:21:23 +01:00
Eelco Dolstra 76f1ba4f3b Add file missing from 201b48de60 2016-03-03 18:03:34 +01:00
Nathan Zadoks fe2be8f016 build.cc: fs.h doesn't appear to be necessary anymore 2016-03-03 14:11:00 +01:00
Eelco Dolstra 5a8455c85e Provide function required by Hydra 2016-03-02 18:21:48 +01:00
Eelco Dolstra 201b48de60 Add an HTTP binary cache store 
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
 2016-03-02 15:46:07 +01:00
Eelco Dolstra 0402b6398d Eliminate local-binary-cache-store.hh 2016-03-02 15:46:07 +01:00
Eelco Dolstra 68a5414982 Make store implementations pluggable 
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
 2016-03-02 15:46:07 +01:00
Eelco Dolstra 0a62d9b3d7 Remove bad assertion 2016-03-02 15:46:07 +01:00
Eelco Dolstra cca4a8dc1a importPaths(): Optionally add NARs to binary cache accessor 
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
 2016-03-02 15:46:07 +01:00
Eelco Dolstra e9c50064b5 Add an HTTP binary cache store 
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
 2016-02-29 18:15:20 +01:00
Eelco Dolstra 6170bb474b Eliminate local-binary-cache-store.hh 2016-02-29 16:14:39 +01:00
Eelco Dolstra 0b907321cc Make store implementations pluggable 
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
 2016-02-29 16:11:11 +01:00
Eelco Dolstra d0344dd2c1 Remove bad assertion 2016-02-26 21:43:59 +01:00
Eelco Dolstra bcc9943cee importPaths(): Optionally add NARs to binary cache accessor 
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
 2016-02-26 15:20:10 +01:00
Eelco Dolstra 1042c10fd0 Add NAR / Store accessor abstraction 
This is primary to allow hydra-queue-runner to extract files like
"nix-support/hydra-build-products" from NARs in binary caches.
 2016-02-25 17:43:19 +01:00
Eelco Dolstra 152b1d6bf9 deletePath(): Succeed if path doesn't exist 
Also makes it robust against concurrent deletions.
 2016-02-24 17:44:55 +01:00
Eelco Dolstra 28e7e29abd Eliminate reserveSpace flag 2016-02-24 17:44:55 +01:00
Eelco Dolstra 5a64e66268 LocalStore::addTextToStore(): Don't read the path we just wrote 
This eliminates some unnecessary (presumably cached) I/O.
 2016-02-24 17:11:31 +01:00
Eelco Dolstra 9ccbd55c5b BinaryCacheStore: Implement addToStore() 
So now you can do

  $ NIX_REMOTE=file:///tmp/binary-cache nix-instantiate '<nixpkgs>' -A hello

and lots of other operations.
 2016-02-24 16:52:28 +01:00
Eelco Dolstra 30e9d01516 openStoreAt(): Initialise the binary cache 2016-02-24 16:52:07 +01:00
Eelco Dolstra ba0a81d14f Show a proper error message for unimplemented binary cache operations 2016-02-24 14:57:30 +01:00
Eelco Dolstra 263187a2ec Move BinaryCacheStore / LocalBinaryCacheStore from Hydra 
So you can now do:

  $ NIX_REMOTE=file:///tmp/binary-cache nix-store -qR /nix/store/...
 2016-02-24 14:48:16 +01:00
Eelco Dolstra 5f862658c3 Remove bad daemon connections from the pool 
This is necessary for long-running processes like hydra-queue-runner:
if a nix-daemon worker is killed, we need to stop reusing that
connection.
 2016-02-24 11:39:56 +01:00
Eelco Dolstra d5626bf4c1 Pool<T>: Allow a maximum pool size 2016-02-23 16:40:16 +01:00
Eelco Dolstra e292144d46 RemoteStore: Make thread-safe 
This allows a RemoteStore object to be used safely from multiple
threads concurrently. It will make multiple daemon connections if
necessary.

Note: pool.hh and sync.hh have been copied from the Hydra source tree.
 2016-02-23 15:00:59 +01:00
Eelco Dolstra d361901bfe curl: Set CURLOPT_NOSIGNAL 
Otherwise using curl is not safe in multi-threaded applications
because it installs a SIGALRM handler.
 2016-02-22 15:51:14 +01:00