Commit graph

446 commits

Author SHA1 Message Date
Eelco Dolstra d74c8a3f4e Fix 32-bit build 2016-08-30 17:38:09 +02:00
Eelco Dolstra c0a7b84748 nix path-info: Add --json flag
Also, factor out JSON generation from value-to-json.{cc,hh}, and
support producing indented JSON.
2016-08-29 17:29:24 +02:00
Dmitry Kalinkin f91748ba73 override rx directory permissions in deletePath()
This fixes instantiation of pythonPackages.pytest that produces a
directory with less permissions during one of it's tests that leads to
a nix error like:

error: opening directory ‘/tmp/nix-build-python2.7-pytest-2.9.2.drv-0/pytest-of-user/pytest-0/testdir/test_cache_failure_warns0/.cache’: Permission denied
2016-07-25 18:11:46 -04:00
Eelco Dolstra d3cd0f5856 Fix assertion failure 2016-07-21 18:39:32 +02:00
Eelco Dolstra ea8e8df6c7 NarInfo::NarInfo(): Ensure that we get a NAR size/hash 2016-07-21 18:35:12 +02:00
Eelco Dolstra e682a8e138 Fix assertion failure in ThreadPool::enqueue() 2016-07-21 18:14:16 +02:00
Shea Levy 06068b353d FdSink: Restore move constructor/assignment 2016-07-13 06:27:41 -04:00
Shea Levy b33e85229d Make Buffered{Source,Sink} move-safe 2016-07-13 06:03:37 -04:00
Shea Levy cb5e7254b6 Modernize AutoCloseFD 2016-07-11 15:44:44 -04:00
Eelco Dolstra 202683a4fc Use O_CLOEXEC in most places 2016-06-09 16:37:08 +02:00
Eelco Dolstra a424ab0444 createTempDir(): Don't do a chown on Linux
It's not needed and can cause problems in a user namespace.
2016-06-02 18:17:30 +02:00
Eelco Dolstra 812c0dfbe2 Allow setting the state directory as a store parameter
E.g. "local?store=/tmp/store&state=/tmp/var".
2016-06-02 16:02:48 +02:00
Dmitry Kalinkin 78b00bbd8a use $(LIBLZMA_LIBS) instead of -llzma
This is needed in case of non-standard lzma installation path that will
be specified in pkgconfig manifest as extra -L option for LDFLAGS.
2016-05-31 03:20:11 -04:00
Eelco Dolstra 6c75cf69c3 Cleanup: Remove singleton() 2016-05-04 16:16:53 +02:00
Eelco Dolstra 0d4a10e910 Do compression in a sink 2016-05-04 16:16:53 +02:00
Eelco Dolstra 538a64e8c3 Add a Store::addToStore() variant that accepts a NAR
As a side effect, this ensures that signatures are propagated when
copying paths between stores.

Also refactored import/export to make use of this.
2016-05-04 16:15:54 +02:00
Eelco Dolstra 33de2bc080 Fix segfault in xz compression/decompression 2016-05-04 11:50:12 +02:00
Eelco Dolstra dfebfc835f Add a copyStorePath() utility function 2016-05-04 11:01:48 +02:00
Eelco Dolstra d8bf0d4859 Support Git repos in the Nix path
E.g.

  $ nix-build -I nixpkgs=git://github.com/NixOS/nixpkgs '<nixpkgs>' -A hello

This is not extremely useful yet because you can't specify a
branch/revision.
2016-04-29 21:07:47 +02:00
Eelco Dolstra 4dde0b0562 BinaryCacheStore: Support bzip2 compression 2016-04-29 17:43:37 +02:00
Eelco Dolstra 5acb691402 BinaryCacheStore: Support "none" compression method 2016-04-29 17:02:57 +02:00
Eelco Dolstra aa3bc3d5dc Eliminate the substituter mechanism
Substitution is now simply a Store -> Store copy operation, most
typically from BinaryCacheStore to LocalStore.
2016-04-29 13:57:08 +02:00
Eelco Dolstra 41633f9f73 Improved logging abstraction
This also gets rid of --log-type, since the nested log type isn't
useful in a multi-threaded situation, and nobody cares about the
"pretty" log type.
2016-04-25 19:18:45 +02:00
Eelco Dolstra c879a20850 Factor out parallel processing of work items that have dependencies 2016-04-22 20:50:06 +02:00
Eelco Dolstra b2ce6fde5a ThreadPool: Start doing work as soon as work items are enqueued 2016-04-22 18:19:17 +02:00
Eelco Dolstra 58c84cda3b Make compression interruptible 2016-04-22 18:15:02 +02:00
Eelco Dolstra 1b0088ebb2 nix --help: Show short flags 2016-04-21 14:34:46 +02:00
Eelco Dolstra 451ebf24ce Cache path info lookups in SQLite
This re-implements the binary cache database in C++, allowing it to be
used by other Store backends, in particular the S3 backend.
2016-04-20 14:12:38 +02:00
Eelco Dolstra e0204f8d46 Move path info caching from BinaryCacheStore to Store
Caching path info is generally useful. For instance, it speeds up "nix
path-info -rS /run/current-system" (i.e. showing the closure sizes of
all paths in the closure of the current system) from 5.6s to 0.15s.

This also eliminates some APIs like Store::queryDeriver() and
Store::queryReferences().
2016-04-19 18:52:53 +02:00
Eelco Dolstra d1b0909894 BinaryCacheStore::readFile(): Return a shared_ptr to a string
This allows readFile() to indicate that a file doesn't exist, and
might eliminate some large string copying.
2016-04-15 15:39:48 +02:00
Eelco Dolstra f398949b40 Make LocalStore thread-safe
Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
Eelco Dolstra 6e120b76ee Add missing -pthread
https://hydra.nixos.org/build/33908385
2016-03-31 12:42:48 +02:00
Eelco Dolstra ab3ce1cc13 Improve SIGINT handling in multi-threaded programs
The flag remembering whether an Interrupted exception was thrown is
now thread-local. Thus, all threads will (eventually) throw
Interrupted. Previously, one thread would throw Interrupted, and then
the other threads wouldn't see that they were supposed to quit.
2016-03-29 16:37:16 +02:00
Eelco Dolstra 784ee35c80 Add "nix verify-paths" command
Unlike "nix-store --verify-path", this command verifies signatures in
addition to store path contents, is multi-threaded (especially useful
when verifying binary caches), and has a progress indicator.

Example use:

$ nix verify-paths --store https://cache.nixos.org -r $(type -p thunderbird)
...
[17/132 checked] checking ‘/nix/store/rawakphadqrqxr6zri2rmnxh03gqkrl3-autogen-5.18.6’
2016-03-29 16:37:16 +02:00
Eelco Dolstra 02654f782f Fix Darwin build
http://hydra.nixos.org/build/33279996
2016-03-15 12:11:27 +01:00
Eelco Dolstra 42bc395b63 Eliminate some large string copying 2016-03-04 16:49:56 +01:00
Eelco Dolstra 6055d84beb Fix reading symlinks
The st_size field of a symlink doesn't have to be correct, e.g. for
/dev/fd symlinks.
2016-03-02 15:46:07 +01:00
Eelco Dolstra 89a2fa68ac FdSource: track number of bytes read 2016-03-02 15:46:07 +01:00
Eelco Dolstra 24a8f9e27b Merge branch 'master' into new-cli 2016-02-25 17:48:35 +01:00
Eelco Dolstra 1042c10fd0 Add NAR / Store accessor abstraction
This is primary to allow hydra-queue-runner to extract files like
"nix-support/hydra-build-products" from NARs in binary caches.
2016-02-25 17:43:19 +01:00
Eelco Dolstra c5bc571861 Fix short boolean flags 2016-02-25 13:31:34 +01:00
Eelco Dolstra f1bdeac986 Merge branch 'master' into new-cli 2016-02-25 11:25:11 +01:00
Eelco Dolstra 152b1d6bf9 deletePath(): Succeed if path doesn't exist
Also makes it robust against concurrent deletions.
2016-02-24 17:44:55 +01:00
Eelco Dolstra 9ccbd55c5b BinaryCacheStore: Implement addToStore()
So now you can do

  $ NIX_REMOTE=file:///tmp/binary-cache nix-instantiate '<nixpkgs>' -A hello

and lots of other operations.
2016-02-24 16:52:28 +01:00
Eelco Dolstra 45c83e5f9b Typo 2016-02-24 14:49:28 +01:00
Eelco Dolstra 263187a2ec Move BinaryCacheStore / LocalBinaryCacheStore from Hydra
So you can now do:

  $ NIX_REMOTE=file:///tmp/binary-cache nix-store -qR /nix/store/...
2016-02-24 14:48:16 +01:00
Eelco Dolstra bf2adf72c4 std::condition_variable_any -> std::condition_variable
The latter is supposed to be more efficient.
2016-02-24 13:31:46 +01:00
Eelco Dolstra ccdbf589a4 C++ templates are just a glorified macro facility 2016-02-24 13:07:32 +01:00
Eelco Dolstra 5f862658c3 Remove bad daemon connections from the pool
This is necessary for long-running processes like hydra-queue-runner:
if a nix-daemon worker is killed, we need to stop reusing that
connection.
2016-02-24 11:39:56 +01:00
Eelco Dolstra d5626bf4c1 Pool<T>: Allow a maximum pool size 2016-02-23 16:40:16 +01:00
Eelco Dolstra e292144d46 RemoteStore: Make thread-safe
This allows a RemoteStore object to be used safely from multiple
threads concurrently. It will make multiple daemon connections if
necessary.

Note: pool.hh and sync.hh have been copied from the Hydra source tree.
2016-02-23 15:00:59 +01:00
Eelco Dolstra c0b7a8a0b5 Move ref<t> into a separate header 2016-02-23 13:53:31 +01:00
Eelco Dolstra eff5021eaa Add xz compression function
This is used by the Hydra queue runner, but since it may also be
useful for the C++ rewrite of nix-push, I'm putting it here.
2016-02-15 21:45:56 +01:00
Eelco Dolstra bfdacb712c decompressXZ: Ensure that lzma_end() is called
Otherwise we might leak memory.
2016-02-15 21:26:28 +01:00
Eelco Dolstra b3e8d72770 Merge pull request #762 from ctheune/ctheune-floats
Implement floats
2016-02-12 12:49:59 +01:00
Eelco Dolstra fd205fb6f8 ref<T>: Add cast operator 2016-02-11 15:32:48 +01:00
Eelco Dolstra cd2196b089 Start of new Nix command-line interface 2016-02-09 21:28:29 +01:00
Eelco Dolstra 0db9e6cd1a New command line parsing infrastructure 2016-02-09 21:07:48 +01:00
Eelco Dolstra c10c61449f Eliminate the "store" global variable
Also, move a few free-standing functions into StoreAPI and Derivation.

Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
2016-02-04 14:28:26 +01:00
Eelco Dolstra 7899fc959d Define HashType's size to allow it to be forward-declared 2016-01-27 17:46:01 +01:00
Eelco Dolstra 739bab0be7 Trivia 2016-01-27 17:18:31 +01:00
Eelco Dolstra d45ad8fcf5 Make hashLength32() a method of Hash 2016-01-27 17:18:20 +01:00
Eelco Dolstra 5b8c09c124 string2Int: Barf on negative numbers for unsigned types 2016-01-20 16:45:05 +01:00
Eelco Dolstra 9a4706eb19 Fix compilation 2016-01-05 14:05:11 +01:00
Eelco Dolstra 6298afc047 Merge pull request #685 from vizanto/master
POSIX compliant directory access (fixes build on Solaris)
2016-01-05 13:49:55 +01:00
FrankHB 607d91517e Fixed unresolved 'BUFSIZ'
`BUFSIZ` is defined in header `<cstdio>`.
2016-01-05 13:33:40 +01:00
Christian Theune 14ebde5289 First hit at providing support for floats in the language. 2016-01-05 00:40:40 +01:00
Eelco Dolstra 8f67325a7c Build sandbox support etc. unconditionally on Linux
Also, use "#if __APPLE__" instead of "#if SANDBOX_ENABLED" to prevent
ambiguity.
2015-12-10 11:47:17 +01:00
Danny Wilson 69b9d8fdbd Use DT_UNKNOWN when dirent d_type extension is not supported.
edolstra:
“…since callers of readDirectory have to handle the possibility of
 DT_UNKNOWN anyway, and we don't want to do a stat call for every
 directory entry unless it's really needed.”
2015-11-25 21:57:19 +01:00
Shea Levy 1d3529e93a Default arguments belong at declaration, not definition 2015-11-16 05:55:55 -05:00
Shea Levy 9b4cd20752 Fix copy-paste error 2015-11-16 05:54:34 -05:00
Shea Levy 58d2fac91d AutoDelete: Add default constructor with deletion disabled 2015-11-16 05:53:10 -05:00
Eelco Dolstra 7759a56bed Fix namespace issue 2015-11-09 21:28:25 +01:00
Danny Wilson cdb346c65e Fix build on Solaris
d_type is not part of the POSIX spec unfortunately.
2015-11-07 04:51:33 +01:00
Eelco Dolstra 6f1743b1a5 Support SHA-512 hashes
Fixes #679.

Note: on x86_64, SHA-512 is considerably faster than SHA-256 (198 MB/s
versus 131 MB/s).
2015-11-04 16:37:49 +01:00
Eelco Dolstra a6ca68a70c Require OpenSSL 2015-11-04 16:37:49 +01:00
Eelco Dolstra 1f735a3440 <nix/fetchurl.nix>: Support xz-compressed NARs 2015-10-30 12:34:30 +01:00
Eelco Dolstra 5c28943e8f int2String() -> std::to_string() 2015-10-29 13:26:55 +01:00
Jude Taylor af1218a86a revert libutil change 2015-10-21 14:40:35 -07:00
Jude Taylor e770f941d6 make sandbox builds more permissive 2015-10-21 12:38:52 -07:00
Eelco Dolstra bec3c31608 nix-prefetch-url: Rewrite in C++ 2015-10-01 16:47:43 +02:00
Eelco Dolstra 4ba6bc184c Shut up clang warnings 2015-09-18 01:22:35 +02:00
Eelco Dolstra 71a5161365 Implement buildDerivation() via the daemon 2015-09-03 12:56:59 +02:00
Kirill Elagin 3b0f60e5c2 baseNameOf: Enhance basename compatibility
* If the path ends with a slash, drop it.
* If the remaining path doesn’t contain slashes, just return it.

Fixes #574.
2015-08-07 03:35:02 +02:00
Eelco Dolstra db55940d9e Support systemd log severity prefixes
This is mostly useful for hydra-queue-runner.
2015-07-20 01:39:48 +02:00
Eelco Dolstra b3491c781c More cleanup 2015-07-20 01:16:16 +02:00
Eelco Dolstra 6bd2c7bb38 OCD: foreach -> C++11 ranged for 2015-07-17 20:13:56 +02:00
Eelco Dolstra 1511aa9f48 Allow remote builds without sending the derivation closure
Previously, to build a derivation remotely, we had to copy the entire
closure of the .drv file to the remote machine, even though we only
need the top-level derivation. This is very wasteful: the closure can
contain thousands of store paths, and in some Hydra use cases, include
source paths that are very large (e.g. Git/Mercurial checkouts).

So now there is a new operation, StoreAPI::buildDerivation(), that
performs a build from an in-memory representation of a derivation
(BasicDerivation) rather than from a on-disk .drv file. The only files
that need to be in the Nix store are the sources of the derivation
(drv.inputSrcs), and the needed output paths of the dependencies (as
described by drv.inputDrvs). "nix-store --serve" exposes this
interface.

Note that this is a privileged operation, because you can construct a
derivation that builds any store path whatsoever. Fixing this will
require changing the hashing scheme (i.e., the output paths should be
computed from the other fields in BasicDerivation, allowing them to be
verified without access to other derivations). However, this would be
quite nice because it would allow .drv-free building (e.g. "nix-env
-i" wouldn't have to write any .drv files to disk).

Fixes #173.
2015-07-17 17:57:40 +02:00
Eelco Dolstra 65f17cd330 Support URLs in $NIX_PATH
This didn't work (despite claims in the manual), because the colon in
"http://" was parsed as a element separator. So handle "://"
specially.
2015-06-17 16:20:11 +02:00
Eelco Dolstra 7c4501886d Use std::vector::data() 2015-06-09 10:54:46 +02:00
Rok Garbas dad754843a cygwin: looks like stdout/stdin are reserved words 2015-05-13 09:37:56 +02:00
Eelco Dolstra 60340ce3e2 Implement caching of fetchurl/fetchTarball results
ETags are used to prevent redownloading unchanged files.
2015-04-09 11:42:04 +02:00
Eelco Dolstra a596c525ad Add base64 encoder/decoder 2015-02-10 11:33:33 +01:00
Eelco Dolstra e0def5bc4b Use libsodium instead of OpenSSL for binary cache signing
Sodium's Ed25519 signatures are much shorter than OpenSSL's RSA
signatures. Public keys are also much shorter, so they're now
specified directly in the nix.conf option ‘binary-cache-public-keys’.

The new command ‘nix-store --generate-binary-cache-key’ generates and
prints a public and secret key.
2015-02-04 17:10:31 +01:00
Eelco Dolstra 0d1dafa0c4 Simplify parseHash32 2015-02-03 18:56:47 +01:00
Eelco Dolstra db2ec59903 Simplify printHash32 2015-02-03 18:35:24 +01:00
aszlig 8b88d25cda libutil: Limit readLink() error to only overflows.
Let's not just improve the error message itself, but also the behaviour
to actually work around the ntfs-3g symlink bug. If the readlink() call
returns a smaller size than the stat() call, this really isn't a problem
even if the symlink target really has changed between the calls.

So if stat() reports the size for the absolute path, it's most likely
that the relative path is smaller and thus it should also work for file
system bugs as mentioned in 93002d69fc58c2b71e2dfad202139230c630c53a.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Tested-by: John Ericson <Ericson2314@Yahoo.com>
2015-01-02 12:53:42 +01:00
aszlig bbd45ac80f libutil: Improve errmsg on readLink size mismatch.
A message like "error: reading symbolic link `...' : Success" really is
quite confusing, so let's not indicate "success" but rather point out
the real issue.

We could also limit the check of this to just check for non-negative
values, but this would introduce a race condition between stat() and
readlink() if the link target changes between those two calls, thus
leading to a buffer overflow vulnerability.

Reported by @Ericson2314 on IRC. Happened due to a possible ntfs-3g bug
where a relative symlink returned the absolute path (st_)size in stat()
while readlink() returned the relative size.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Tested-by: John Ericson <Ericson2314@Yahoo.com>
2015-01-02 12:53:42 +01:00
Eelco Dolstra 4e0607369e Pedantry 2014-12-14 01:51:14 +01:00
Eelco Dolstra 8bdff8c100 Merge branch 'cygwin-master' of https://github.com/ternaris/nix 2014-12-14 01:49:14 +01:00