Commit graph

3231 commits

Author SHA1 Message Date
Eelco Dolstra c5f23f10a8
Replace readline by linenoise
Using linenoise avoids a license compatibility issue (#1356), is a lot
smaller and doesn't pull in ncurses.
2017-05-10 18:37:42 +02:00
Linus Heckemann d48edcc3a5 nix-shell: use appropriate prompt terminator
If running nix-shell as root, the terminator should be # and not $.
2017-05-10 12:03:45 +01:00
Eelco Dolstra 03ae5e6459
Add "nix edit" command
This is a little convenience command that opens the Nix expression of
the specified package. For example,

  nix edit nixpkgs.perlPackages.Moose

opens <nixpkgs/pkgs/top-level/perl-packages.nix> in $EDITOR (at the
right line number for some editors).

This requires the package to have a meta.position attribute.
2017-05-08 18:42:30 +02:00
Eelco Dolstra 7689181e4f
Minor cleanup 2017-05-08 15:56:52 +02:00
Eelco Dolstra 00b286275c
Linux sandbox: Fix compatibility with older kernels 2017-05-08 15:42:59 +02:00
Eelco Dolstra ebfceeb333
build-remote: Check remote build status 2017-05-08 14:27:12 +02:00
Eelco Dolstra 0a97eb6bd7
Remove superfluous #ifdef 2017-05-08 11:27:20 +02:00
Eelco Dolstra bb50c89319
Make the location of the build directory in the sandbox configurable
This is mostly for use in the sandbox tests, since if the Nix store is
under /build, then we can't use /build as the build directory.
2017-05-05 17:45:22 +02:00
Eelco Dolstra 465cb68244
Figure out the user's home directory if $HOME is not set 2017-05-05 17:08:23 +02:00
Eelco Dolstra eba840c8a1
Linux sandbox: Use /build instead of /tmp as $TMPDIR
There is a security issue when a build accidentally stores its $TMPDIR
in some critical place, such as an RPATH. If
TMPDIR=/tmp/nix-build-..., then any user on the system can recreate
that directory and inject libraries into the RPATH of programs
executed by other users. Since /build probably doesn't exist (or isn't
world-writable), this mitigates the issue.
2017-05-04 16:57:03 +02:00
Eelco Dolstra 2da6a42448
nix dump-path: Add
This is primarily useful for extracting NARs from other stores (like
binary caches), which "nix-store --dump" cannot do.
2017-05-04 14:21:22 +02:00
Eelco Dolstra 72fb2a7edc
Fix build on gcc 4.9
http://hydra.nixos.org/build/52408843
2017-05-03 16:08:48 +02:00
Eelco Dolstra 08355643ab
nix-shell: Implement passAsFile 2017-05-03 15:01:15 +02:00
Eelco Dolstra 782c0bff45
nix eval: Add a --raw flag
Similar to "jq -r", this prints the evaluation result (which must be a
string value) unquoted.
2017-05-03 14:08:18 +02:00
Eelco Dolstra cef8c169b1
Fix "nix ... --all"
When "--all" is used, we should not fill in a default installable.
2017-05-02 15:46:10 +02:00
Eelco Dolstra c5bea16611
LocalStoreAccessor: Fix handling of diverted stores 2017-05-02 15:46:09 +02:00
Eelco Dolstra 7f6837a0f6
Replace $NIX_REMOTE_SYSTEMS with an option "builder-files"
Also, to unify with hydra-queue-runner, allow it to be a list of
files.
2017-05-02 15:46:09 +02:00
Eelco Dolstra cd4d2705ec
build-remote: Fix fallback to other machines when connecting fails
Opening an SSHStore or LegacySSHStore does not actually establish a
connection, so the try/catch block here did nothing. Added a
Store::connect() method to test whether a connection can be
established.
2017-05-02 15:46:09 +02:00
Eelco Dolstra 1a68710d4d
Add an option for specifying remote builders
This is useful for one-off situations where you want to specify a
builder on the command line instead of having to mess with
nix.machines. E.g.

  $ nix-build -A hello --argstr system x86_64-darwin \
    --option builders 'root@macstadium1 x86_64-darwin'

will perform the specified build on "macstadium1".

It also removes the need for a separate nix.machines file since you
can specify builders in nix.conf directly. (In fact nix.machines is
yet another hack that predates the general nix.conf configuration
file, IIRC.)

Note: this option is supported by the daemon for trusted users. The
fact that this allows trusted users to specify paths to SSH keys to
which they don't normally have access is maybe a bit too much trust...
2017-05-02 15:42:43 +02:00
Eelco Dolstra ebc9f36a81
Factor out machines.conf parsing
This allows hydra-queue-runner to use it.
2017-05-02 13:17:37 +02:00
Eelco Dolstra 174b68a2a2
build-hook: If there are no machines defined, quit permanently 2017-05-02 12:16:29 +02:00
Eelco Dolstra feefcb3a98
build-remote: Ugly hackery to get build logs to work
The build hook mechanism expects build log output to go to file
descriptor 4, so do that.
2017-05-02 12:02:23 +02:00
Eelco Dolstra 3a5f04f48c
build-remote: Don't require signatures
This restores the old behaviour.
2017-05-01 20:03:25 +02:00
Eelco Dolstra 031d70e500
Support arbitrary store URIs in nix.machines
For backwards compatibility, if the URI is just a hostname, ssh://
(i.e. LegacySSHStore) is prepended automatically.

Also, all fields except the URI are now optional. For example, this is
a valid nix.machines file:

  local?root=/tmp/nix

This is useful for testing the remote build machinery since you don't
have to mess around with ssh.
2017-05-01 17:35:30 +02:00
Eelco Dolstra 3e4bdfedee
Minor cleanup 2017-05-01 17:30:17 +02:00
Eelco Dolstra deac171925
Implement LegacySSHStore::buildDerivation()
This makes LegacySSHStore usable by build-remote and
hydra-queue-runner.
2017-05-01 17:30:16 +02:00
Eelco Dolstra 3f5b98e65a
Chomp log output from the build hook 2017-05-01 17:30:16 +02:00
Eelco Dolstra d7653dfc6d
Remove $NIX_BUILD_HOOK and $NIX_CURRENT_LOAD
This is to simplify remote build configuration. These environment
variables predate nix.conf.

The build hook now has a sensible default (namely build-remote).

The current load is kept in the Nix state directory now.
2017-05-01 17:30:16 +02:00
Eelco Dolstra ca9f589a93
build-remote: Don't copy the .drv closure
Since build-remote uses buildDerivation() now, we don't need to copy
the .drv file anymore. This greatly reduces the set of input paths
copied to the remote side (e.g. from 392 to 51 store paths for GNU
hello on x86_64-darwin).
2017-05-01 17:30:16 +02:00
Eelco Dolstra b986c7f8b1
Pass verbosity level to build hook 2017-05-01 14:43:14 +02:00
Eelco Dolstra 227a48f86f
Reduce severity of EMLINK warnings
Fixes #1357.
2017-05-01 14:36:56 +02:00
Eelco Dolstra 0dddcf867a
Add a dummy Store::buildPaths() method
This default implementation of buildPaths() does nothing if all
requested paths are already valid, and throws an "unsupported
operation" error otherwise. This fixes a regression introduced by
c30330df6f in binary cache and legacy
SSH stores.
2017-05-01 13:43:34 +02:00
Guillaume Maudoux a143014d73 lexer: remove catch-all rules hiding real errors
With catch-all rules, we hide potential errors.
It turns out that a4744254 made one cath-all useless. Flex detected that
is was impossible to reach.
The other is more subtle, as it can only trigger on unfinished escapes
in unfinished strings, which only occurs at EOF.
2017-05-01 01:18:06 +02:00
Guillaume Maudoux a474425425 Fix lexer to support $' in multiline strings. 2017-05-01 01:15:40 +02:00
Eelco Dolstra 2f21d522c2
Hopefully fix the Darwin build
http://hydra.nixos.org/build/52080911
2017-04-28 17:13:55 +02:00
Eelco Dolstra 895f00c372
Suppress warning about ssh-auth-sock 2017-04-28 16:55:52 +02:00
Eelco Dolstra 73bba12d8b
Check for libreadline 2017-04-28 16:53:56 +02:00
Eelco Dolstra a1a5e63e14
Fix brainfart 2017-04-28 16:21:54 +02:00
Eelco Dolstra 41c4558afe
Fix hash computation when importing NARs greater than 4 GiB
This caused "nix-store --import" to compute an incorrect hash on NARs
that don't fit in an unsigned int. The import would succeed, but
"nix-store --verify-path" or subsequent exports would detect an
incorrect hash.

A deeper issue is that the export/import format does not contain a
hash, so we can't detect such issues early.

Also, I learned that -Wall does not warn about this.
2017-04-28 15:24:05 +02:00
Eelco Dolstra 39b08f4c0c Merge pull request #1358 from shlevy/store-nesting
Add Store nesting to fix import-from-derivation within filterSource
2017-04-26 20:28:49 +02:00
Shea Levy 4bc00760f9 Add Store nesting to fix import-from-derivation within filterSource 2017-04-26 14:15:47 -04:00
Eelco Dolstra 45ce2c7413
Doh 2017-04-26 17:58:09 +02:00
Eelco Dolstra 6734c18c99
nix repl: Fix Ctrl-C 2017-04-25 19:19:48 +02:00
Eelco Dolstra 23aa1619da
Minor cleanup 2017-04-25 19:10:47 +02:00
Eelco Dolstra 536f061765
"using namespace std" considered harmful 2017-04-25 18:58:02 +02:00
Eelco Dolstra 5bd8795e1f
nix repl: Use $XDG_DATA_HOME for the readline history 2017-04-25 18:56:29 +02:00
Eelco Dolstra 921a2aeb05
Make "nix repl" build 2017-04-25 18:48:40 +02:00
Eelco Dolstra c31000bc93
Merge nix-repl repository 2017-04-25 18:14:13 +02:00
Eelco Dolstra 40daf0d800
Cleanup in preparation of merging nix-repl repo into nix repo 2017-04-25 18:13:23 +02:00
Eelco Dolstra c30330df6f
StorePathCommands: Build installables
So for instance "nix copy --to ... nixpkgs.hello" will build
nixpkgs.hello first. It's debatable whether this is a good idea. It
seems desirable for commands like "nix copy" but maybe not for
commands like "nix path-info".
2017-04-25 16:19:22 +02:00
Eelco Dolstra d48c973ece
Set default installable
Thus

  $ nix build -f foo.nix

will build foo.nix.

And

  $ nix build

will build default.nix. However, this may not be a good idea because
it's kind of inconsistent, given that "nix build foo" will build the
"foo" attribute from the default installation source (i.e. the
synthesis of $NIX_PATH), rather than ./default.nix. So I may revert
this.
2017-04-25 15:18:05 +02:00
Eelco Dolstra 0b6220fbd6
Interpret any installable containing a slash as a path
So "nix path-info ./result" now works.
2017-04-25 14:09:01 +02:00
Eelco Dolstra 7ee81f3887
Make StorePathsCommand a subclass of InstallablesCommand
This allows commands like 'nix path-info', 'nix copy', 'nix verify'
etc. to work on arbitrary installables. E.g. to copy geeqie to a
binary cache:

  $ nix copy -r --to file:///tmp/binary-cache nixpkgs.geeqie

Or to get the closure size of thunderbird:

  $ nix path-info -S nixpkgs.thunderbird
2017-04-25 13:20:26 +02:00
Eelco Dolstra c769841bc4
Move code around 2017-04-25 12:07:31 +02:00
Eelco Dolstra 6267d74889
Add "nix eval" command
This replaces "nix-instantiate --eval". The result is evaluated
strictly since this seems more useful.
2017-04-25 11:23:47 +02:00
Eelco Dolstra bcecc99007
Restructure installables handling in the "nix" command 2017-04-25 11:20:37 +02:00
Eelco Dolstra 1bb87c0487
Remove debug statement 2017-04-24 15:01:28 +02:00
Eelco Dolstra 66577a1c64
Factor out --json 2017-04-24 14:21:36 +02:00
Eelco Dolstra 9b63bb88c8
nix-shell -p: Use runCommandCC
This restores pre-17.03 behaviour by making gcc available.
2017-04-24 12:04:01 +02:00
David McFarland 804ac52489 add helper function to set 'interruptThrown'
this fixes a linker failure on cygwin 64 due to some bad
interaction between tls and shared libraries.

see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64697
2017-04-21 11:28:14 -03:00
Eelco Dolstra 749696e71c
Detect lsof
Also, don't use lsof on Linux since it's not needed.

Fixes #1328.
2017-04-20 19:11:45 +02:00
Eelco Dolstra efa4bdbfcd
Improve nix show-config --json
In particular, show descriptions. This could be used for manpage
generation etc.
2017-04-20 17:34:47 +02:00
Eelco Dolstra 4410e9d995
Setting: Remove "Tag" template argument 2017-04-20 16:52:53 +02:00
Eelco Dolstra f05d5f89ff
Read per-user settings from ~/.config/nix/nix.conf 2017-04-20 14:58:16 +02:00
Eelco Dolstra 562585e901
binary-caches-parallel-connections -> http-connections 2017-04-20 14:04:00 +02:00
Eelco Dolstra 76cb3c702c
Reimplement trusted-substituters (aka trusted-binary-caches) 2017-04-20 13:41:29 +02:00
Eelco Dolstra 9cc8047f44
Reimplement connect-timeout
Fixes #1339.
2017-04-19 14:54:52 +02:00
Eelco Dolstra b0cb117226
getDerivations(): Filter out packages with bad derivation names
In particular, this disallows attribute names containing dots or
starting with dots. Hydra already disallowed these. This affects the
following packages in Nixpkgs master:

  2048-in-terminal
  2bwm
  389-ds-base
  90secondportraits
  lispPackages.3bmd
  lispPackages.hu.dwim.asdf
  lispPackages.hu.dwim.def

Closes #1342.
2017-04-19 14:18:26 +02:00
Eelco Dolstra 67fe3e07b2 Merge pull request #1321 from shlevy/channel-direct-tarball-error
nix-channel: error out if direct tarball unpack fails.
2017-04-18 16:51:46 +02:00
Eelco Dolstra f8a2e8a552
Shut up some warnings 2017-04-14 14:42:20 +02:00
Eelco Dolstra dd3714f6ef
Doh 2017-04-14 14:42:08 +02:00
Eelco Dolstra 6520b757c5
Fix 32-bit build
http://hydra.nixos.org/build/51569816
2017-04-14 13:59:39 +02:00
Eelco Dolstra 3872371f25
Minor cleanup 2017-04-14 13:42:22 +02:00
Eelco Dolstra 01dcdfcf33
nix-daemon: Don't set untrusted-* settings
These are no longer used anywhere.
2017-04-14 11:59:51 +02:00
Eelco Dolstra 1673c373c9
nix-daemon: Don't die if the user sends an unknown setting 2017-04-14 11:57:02 +02:00
Eelco Dolstra 872ba75d8b
Add "nix show-config" command
This dumps the entire Nix configuration, including all options that
have default values.
2017-04-13 20:59:38 +02:00
Eelco Dolstra ba9ad29fdb
Convert Settings to the new config system
This makes all config options self-documenting.

Unknown or unparseable config settings and --option flags now cause a
warning.
2017-04-13 20:53:23 +02:00
Eelco Dolstra 6bd9576aeb
Support arbitrary numeric types for settings 2017-04-13 17:54:05 +02:00
Eelco Dolstra 0bf34de43b
Validate Boolean settings better 2017-04-13 16:31:28 +02:00
Eelco Dolstra 1860070548
Merge branch 'rework-options' of https://github.com/copumpkin/nix 2017-04-13 16:15:51 +02:00
Eelco Dolstra 2040240e23
Add a Config class to simplify adding configuration settings
The typical use is to inherit Config and add Setting<T> members:

  class MyClass : private Config
  {
    Setting<int> foo{this, 123, "foo", "the number of foos to use"};
    Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"};

    MyClass() : Config(readConfigFile("/etc/my-app.conf"))
    {
      std::cout << foo << "\n"; // will print 123 unless overriden
    }
  };

Currently, this is used by Store and its subclasses for store
parameters. You now get a warning if you specify a non-existant store
parameter in a store URI.
2017-04-13 16:03:31 +02:00
Eelco Dolstra 568a099c88
canonPath(): Check against empty paths 2017-04-13 16:03:31 +02:00
Eelco Dolstra 6d97d81656
Add warn function 2017-04-13 16:03:31 +02:00
Eelco Dolstra 31cc9366fc
Initialise logger 2017-04-13 16:03:31 +02:00
Eelco Dolstra 23304f527a Merge pull request #1302 from dtzWill/fix/nix-options
Process nix.conf options in "new" commands, add test
2017-04-12 11:03:19 +02:00
Shea Levy 503cc4431b nix-channel: error out if direct tarball unpack fails.
It's very unlikely a path ending in .tar.gz is a directory

Fixes #1318
2017-04-10 18:16:46 -04:00
Eelco Dolstra 1fe1976e0d Merge pull request #1316 from copumpkin/nix-retries-default
Default to 5 download retries
2017-04-10 15:58:57 +02:00
Dan Peebles d1fdade755 Add CURLE_WRITE_ERROR as a transient error condition
We've observed it failing downloads in the wild and retrying the same URL
a few moments later seemed to fix it.
2017-04-10 09:28:44 -04:00
Dan Peebles e43e8be8e7 Default to 5 download retries
This should help certain downloaders that don't request anything special
for the number of retries, like nix-channel.
2017-04-10 09:22:24 -04:00
Eelco Dolstra 105f8ffc98
Minor cleanup
Also, possible fix for #1310 on 32-bit systems.
2017-04-10 11:27:33 +02:00
Eelco Dolstra 95295482ea
Allow "auto" as a store URI
Using the empty string is likely to be ambiguous in some contexts.
2017-04-10 11:27:29 +02:00
Neil Mayhew f12a048a05 Propagate NIX_BUILD_CORES to nix-shell environments 2017-04-09 08:21:52 -06:00
Will Dietz 30f89e0d65 Process nix.conf options in "new" nix commands, add test.
Without this (minor) change, the options set using "--option"
or read from nix.conf were parsed but not used.
2017-04-08 12:59:42 -05:00
Dan Peebles 98283915f5 Retry downloads on transient SSL errors too 2017-04-06 18:18:43 +00:00
Eelco Dolstra ba20730b3f
Implement RemoteStore::queryMissing()
This provides a significant speedup, e.g. 64 s -> 12 s for

  nix-build --dry-run -I nixpkgs=channel:nixos-16.03 '<nixpkgs/nixos/tests/misc.nix>' -A test

on a cold local and CloudFront cache.

The alternative is to use lots of concurrent daemon connections but
that seems wasteful.
2017-04-06 18:40:19 +02:00
Eelco Dolstra 963f2bf12b
Fix bogus "unexpected Nix daemon error: interrupted by the user" 2017-04-06 17:19:32 +02:00
Eelco Dolstra 6b5e271163
Add a method to allow hydra-queue-runner to flush the path info cache 2017-04-06 15:22:37 +02:00
Eelco Dolstra 256940fc48
nix-daemon: Disable path info cache
This is useless because the client also caches path info, and can
cause problems for long-running clients like hydra-queue-runner
(i.e. it may return cached info about paths that have been
garbage-collected).
2017-04-06 14:30:31 +02:00
Eelco Dolstra 8decb07c31
Allow default sandbox paths to be overriden
E.g. you can now redirect /etc/resolv.conf to a different file.
2017-04-04 17:54:16 +02:00
Eelco Dolstra 488792a87d
Make /var/run/nscd/socket optional
Not every distribution uses nscd.
2017-04-04 17:40:50 +02:00
Eelco Dolstra 29d35805c6
Sandbox: Fix /dev/ptmx on recent kernels
This fixes "No such file or directory" when opening /dev/ptmx
(e.g. http://hydra.nixos.org/build/51094249).

The reason appears to be some changes to /dev/ptmx / /dev/pts handling
between Linux 4.4 and 4.9. See
https://patchwork.kernel.org/patch/7832531/.

The fix is to go back to mounting a proper /dev/pts instance inside
the sandbox. Happily, this now works inside user namespaces, even for
unprivileged users. So

  NIX_REMOTE=local?root=/tmp/nix nix-build \
    '<nixpkgs/nixos/tests/misc.nix>' -A test

works for non-root users.

The downside is that the fix breaks sandbox builds on older kernels
(probably pre-4.6), since mounting a devpts fails inside user
namespaces for some reason I've never been able to figure out. Builds
on those systems will fail with

  error: while setting up the build environment: mounting /dev/pts: Invalid argument

Ah well.
2017-03-31 18:20:19 +02:00
Shea Levy 3ecb09a40a builtins.exec: Make the argument just a list 2017-03-31 11:58:41 -04:00
Shea Levy d299bd710a Merge branch 'builtins.exec' 2017-03-31 11:22:39 -04:00
Eelco Dolstra e1509adbbb
Retry curl error 16 2017-03-31 15:50:11 +02:00
Eelco Dolstra c0745a2531
Merge branch 'remove-perl' of https://github.com/shlevy/nix 2017-03-31 14:13:32 +02:00
Shea Levy 0bb8db257d Add exec primop behind allow-unsafe-native-code-during-evaluation.
Execute a given program with the (optional) given arguments as the
user running the evaluation, parsing stdout as an expression to be
evaluated.

There are many use cases for nix that would benefit from being able to
run arbitrary code during evaluation, including but not limited to:

* Automatic git fetching to get a sha256 from a git revision
* git rev-parse HEAD
* Automatic extraction of information from build specifications from
  other tools, particularly language-specific package managers like
  cabal or npm
* Secrets decryption (e.g. with nixops)
* Private repository fetching

Ideally, we would add this functionality in a more principled way to
nix, but in the mean time 'builtins.exec' can be used to get these
tasks done.

The primop is only available when the
'allow-unsafe-native-code-during-evaluation' nix option is true. That
flag also enables the 'importNative' primop, which is strictly more
powerful but less convenient (since it requires compiling a plugin
against the running version of nix).
2017-03-30 08:04:21 -04:00
Eelco Dolstra c60715e937
Ignore broken "Deriver: unknown-deriver" fields in .narinfo
These were generated by a legacy tool.
2017-03-28 13:08:13 +02:00
Daiderd Jordan 023217f07c
use std::tuple for ValueMap allocator 2017-03-24 23:05:49 +01:00
Eelco Dolstra 0a7ca24c26
Fix xz decompression
Fixes #1285.
2017-03-22 11:53:33 +01:00
Eelco Dolstra 895a74a814
LocalFSStore::getBuildLog(): Handle corrupted logs 2017-03-21 19:23:35 +01:00
Eelco Dolstra ed5c0f69f2
Don't hang in decompression if bzip2 data ends prematurely 2017-03-21 19:23:35 +01:00
Eelco Dolstra e1e49c58e1
Only use cache.nixos.org when the store is /nix/store
This is consistent with the behaviour of the old
download-from-binary-cache substituter.
2017-03-21 17:59:18 +01:00
Eelco Dolstra ecbc3fedd3
Require signatures by default
This corresponds to the NixOS default.
2017-03-21 15:10:48 +01:00
Eelco Dolstra 4bb38591e5
Restore cache.nixos.org as the default substituter
Fixes #1283.
2017-03-21 15:06:46 +01:00
Eelco Dolstra f8b84a3b8c
Move istringstream_nocopy to a separate file 2017-03-21 14:43:03 +01:00
Eelco Dolstra 3229f85585
Honor $NIX_SSHOPTS again
NixOps needs this.
2017-03-21 14:35:50 +01:00
Eelco Dolstra 20df50d8e1
Merge branch 'darwin-s3-binary-cache-store' of https://github.com/shlevy/nix 2017-03-20 17:46:35 +01:00
Eelco Dolstra dc931fe1cd Merge pull request #1275 from steveeJ/patch-1
nix-shell/pure: keep environment variable SHLVL
2017-03-20 17:38:40 +01:00
Eelco Dolstra 558eda0115
nix copy: Make -r option use the "from" store
Previously, we tried to compute the closure in the local store, which
obviously doesn't work.
2017-03-16 14:25:54 +01:00
Eelco Dolstra 287084d688
ssh:// -> ssh-ng://, legacy-ssh:// -> ssh:// 2017-03-16 14:19:32 +01:00
Eelco Dolstra c5b83d8913
copyPaths(): Use queryValidPaths() to reduce SSH latency 2017-03-16 13:50:01 +01:00
Eelco Dolstra 91d67692cf
copyPaths(): Don't query path info for a path the target already has
For example, this cuts "nix-copy-closure --from" on a NixOS system
closure from 15.9s to 0.5s.
2017-03-16 12:05:51 +01:00
Eelco Dolstra ea7fa88131
LegacySSHStore: Provide a faster implementation of computeFSClosure()
This avoids the latency of the standard implementation, which can make
a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16 11:44:52 +01:00
Eelco Dolstra 7a716ef2a5
Fix nix-copy-closure --to 2017-03-16 10:58:48 +01:00
Eelco Dolstra 0ec7f47b00
Remove "killing process <pid>" messages
They convey no useful information.
2017-03-16 10:52:28 +01:00
Eelco Dolstra 43f158bb08
nix-copy-closure: Fix assertion failure
$ ./inst/bin/nix-copy-closure --to bla $(type -p firefox)
  nix-copy-closure: src/libstore/store-api.cc:80: std::__cxx11::string nix::storePathToHash(const Path&): Assertion `base.size() >= storePathHashLen' failed.
2017-03-16 10:45:45 +01:00
Eelco Dolstra 3f35612c04
More precise compression settings 2017-03-15 17:20:19 +01:00
Eelco Dolstra 25dff2b7db
runProgram(): Distinguish between empty input and no input
For example, if we call brotli with an empty input, it shouldn't read
from the caller's stdin.
2017-03-15 16:50:19 +01:00
Eelco Dolstra 042975ea8e
Compress NAR listings using the "text-compression" method
So if "text-compression=br", the .ls file in S3 will get a
Content-Encoding of "br". Brotli appears to compress better than xz
for this kind of file and is natively supported by browsers.
2017-03-15 16:50:19 +01:00
Eelco Dolstra 45c70382ac
S3BinaryCacheStore: Set Content-Type
This is necessary for serving log files to browsers.
2017-03-15 16:50:19 +01:00
Eelco Dolstra 8b1d65bebe
S3BinaryCacheStore: Support compression of narinfo and log files
You can now set the store parameter "text-compression=br" to compress
textual files in the binary cache (i.e. narinfo and logs) using
Brotli. This sets the Content-Encoding header; the extension of
compressed files is unchanged.

You can separately specify the compression of log files using
"log-compression=br". This is useful when you don't want to compress
narinfo files for backward compatibility.
2017-03-15 16:49:28 +01:00
Eelco Dolstra 2691498b5c
Fix assertion failure on SIGINT
nix: src/libutil/compression.cc:142: virtual nix::XzSink::~XzSink(): Assertion `finished' failed.
2017-03-15 16:49:06 +01:00
Eelco Dolstra fbbc4d8dda
Fix deadlock in runProgram() when input is larger than the pipe buffer size 2017-03-15 16:49:06 +01:00
Eelco Dolstra e8186085e0
Add support for brotli compression
Build logs on cache.nixos.org are compressed using Brotli (since this
allows them to be decompressed automatically by Chrome and Firefox),
so it's handy if "nix log" can decompress them.
2017-03-15 16:49:06 +01:00
Eelco Dolstra 73d7a51ee6
Remove dependency on "curl" binary 2017-03-15 16:48:29 +01:00
Eelco Dolstra 532d73d5d8
BinaryCacheStore: Implement getBuildLog()
We assume that build logs are stored under log/<drv>, e.g.

  /nix/store/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv

maps to

  https://cache.nixos.org/log/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv
2017-03-15 16:48:29 +01:00
Eelco Dolstra 5b86451f02
Add a "nix log" command
This replaces "nix-store --read-log". It checks the local store and
any configured substituters for the requested logs.
2017-03-15 16:48:29 +01:00
Eelco Dolstra 0afeb7f51e
Store: Add a method for getting build logs
This allows various Store implementations to provide different ways to
get build logs. For example, BinaryCacheStore can get the build logs
from the binary cache.

Also, remove the log-servers option since we can use substituters for
this.
2017-03-15 16:48:29 +01:00
Shea Levy 96443e94a1 Merge branch 'configurable-aws-region' 2017-03-12 10:01:11 -04:00
Stefan Junker f628ca2a1f nix-shell/pure: keep environment variable SHLVL 2017-03-12 01:04:21 +01:00
Eelco Dolstra 86227390c5
Merge branch 'allow-import-from-derivation' of https://github.com/shlevy/nix 2017-03-09 13:36:30 +01:00
Adrien Devresse d853877ce9 Add missing header <sstream> 2017-03-08 22:24:10 +01:00
Shea Levy 93f863be96 Add option to disable import-from-derivation completely, even if the drv is already realized 2017-03-08 08:46:12 -05:00
Shea Levy 3cc18d3753 Properly set the caFile for aws-sdk-cpp s3 2017-03-06 14:30:35 -05:00
Shea Levy 4fc30922cf istringstream_nocopy: Implement in a standards-compliant way.
Fixes the problem mentioned in e6a61b8da7

See #1135
2017-03-06 13:03:02 -05:00
Eelco Dolstra 1cf4801108
Revert fa125b9b28
This causes quadratic performance.
2017-03-06 13:56:20 +01:00
Shea Levy 5789eaa3f4 Add aws-region param to S3 store URLs 2017-03-03 16:12:17 -05:00
Eelco Dolstra d1158bb816
Cache connection failures 2017-03-03 19:36:01 +01:00
Eelco Dolstra 8490ee37a6
SSHMaster: Make thread-safe 2017-03-03 19:28:27 +01:00
Eelco Dolstra d3eb1cf3bb
build-remote: Don't use a SSH master
This is unnecessary because we make only one connection.
2017-03-03 19:23:20 +01:00
Eelco Dolstra d140c75530
Pool: Don't hang if creating a connection fails 2017-03-03 19:21:43 +01:00
Eelco Dolstra 577ebeaefb
Improve SSH handling
* Unify SSH code in SSHStore and LegacySSHStore.

* Fix a race starting the SSH master. We now wait synchronously for
  the SSH master to finish starting. This prevents the SSH clients
  from starting their own connections.

* Don't use a master if max-connections == 1.

* Add a "max-connections" store parameter.

* Add a "compress" store parameter.
2017-03-03 19:05:50 +01:00
Eelco Dolstra 7f62be1bcd
build-remote: Fix passing SSH key 2017-03-03 16:33:18 +01:00
Eelco Dolstra 5a1fb03b8f
build-remote: Misc cleanup 2017-03-03 16:18:49 +01:00
Eelco Dolstra 2f992692e2
Fix fatal "broken pipe" error when $NIX_BUILD_HOOK is missing 2017-03-03 15:40:06 +01:00
Eelco Dolstra ffcf9d24a6
builtins.fetchgit: Support paths 2017-03-02 11:46:28 +01:00
Eelco Dolstra fbf17f1ad7
builtins.fetchgit: Fix bad format string 2017-03-02 11:40:11 +01:00
Eelco Dolstra fa125b9b28
TeeSink: Pre-reserve string space
When receiving a very large file, this can prevent the string from
having tobe copied, which temporarily doubles memory consumption.
2017-03-01 16:16:04 +01:00
Eelco Dolstra f61f67ddee
RemoteStore::addToStore(): Send NAR rather than string containing NAR
This allows the NAR to be streamed in the future (though we're not
doing that yet).
2017-03-01 16:07:15 +01:00
Eelco Dolstra 374908726b
readString(): Read directly into std::string
When reading a huge string, this halves memory consumption.

(Strictly speaking, this appears only valid in C++17, but who cares...)
2017-03-01 14:54:11 +01:00
Eelco Dolstra 07a0b8ca67
Tweak message 2017-03-01 14:52:16 +01:00
Eelco Dolstra 56e19d970d
nix-store --import: Fix importing unsigned paths 2017-03-01 14:47:52 +01:00
Eelco Dolstra e321551d54
Fix assertion failure in nix-store --export
Fixes #1173.
2017-03-01 14:41:54 +01:00
Eelco Dolstra c4a40949d9
Handle importing NARs containing files greater than 4 GiB
Also templatize readInt() to work for various integer types.
2017-03-01 13:52:54 +01:00
Eelco Dolstra 0780805246
Fix building against older curl versions
http://hydra.nixos.org/build/49490928
2017-02-28 14:03:43 +01:00
Eelco Dolstra fd86dd93dd
Improve SQLite busy handling 2017-02-28 13:59:11 +01:00
Eelco Dolstra 34b12bad59
NarInfoDiskCache: Handle SQLite busy errors 2017-02-28 13:44:11 +01:00
Eelco Dolstra 80027144ae
In SQLite errors, include the database path
This is necessary because we have multiple SQLite databases (e.g. the
binary cache cache).
2017-02-28 13:20:12 +01:00
Eelco Dolstra 7251d048fa
Support auto-configuration of build-max-jobs
"build-max-jobs" and the "-j" option can now be set to "auto" to use
the number of CPUs in the system. (Unlike build-cores, it doesn't use
0 to imply auto-configuration, because a) magic values are a bad idea
in general; b) 0 is a legitimate value used to disable local
building.)

Fixes #1198.
2017-02-28 12:54:50 +01:00
Eelco Dolstra 3fab1f04a7
_SC_NPROCESSORS_ONLN -> std:🧵:hardware_concurrency() 2017-02-27 16:01:54 +01:00
Eelco Dolstra 3cb0387d3f
Retry downloads on HTTP/2 stream errors
Issue #1254.
2017-02-27 14:36:09 +01:00
Eelco Dolstra d3e1aad421
nix-shell: Better error message when the shell can't be started 2017-02-24 17:25:00 +01:00
Eelco Dolstra e76df9bd52
Register content-addressability assertion for fixed outputs 2017-02-24 16:39:53 +01:00
Eelco Dolstra 1c718f80d3
Verify content-addressability assertions at registration time 2017-02-24 16:38:14 +01:00
Eelco Dolstra 668fef2e4f
nix-shell: Overwrite environment variables
Need to remember that std::map::insert() and emplace() don't overwrite
existing entries...

This fixes a regression relative to 1.11 that in particular triggers
in nested nix-shells.

Before:

  $ nativeBuildInputs=/foo nix-shell -p hello --run 'hello'
  build input /foo does not exist

After:

  $ nativeBuildInputs=/foo nix-shell -p hello --run 'hello'
  Hello, world!
2017-02-24 13:31:46 +01:00
Dan Peebles e7cb2847ab Explicitly model all settings and fail on unrecognized ones
Previously, the Settings class allowed other code to query for string
properties, which led to a proliferation of code all over the place making
up new options without any sort of central registry of valid options. This
commit pulls all those options back into the central Settings class and
removes the public get() methods, to discourage future abuses like that.

Furthermore, because we know the full set of options ahead of time, we
now fail loudly if someone enters an unrecognized option, thus preventing
subtle typos. With some template fun, we could probably also dump the full
set of options (with documentation, defaults, etc.) to the command line,
but I'm not doing that yet here.
2017-02-22 20:19:25 -05:00
Eelco Dolstra f023f64f40
RemoteStore::addToStore(): Pass content-addressability assertion
... and use this in Downloader::downloadCached(). This fixes

  $ nix-build https://nixos.org/channels/nixos-16.09-small/nixexprs.tar.xz -A hello
  error: cannot import path ‘/nix/store/csfbp1s60dkgmk9f8g0zk0mwb7hzgabd-nixexprs.tar.xz’ because it lacks a valid signature
2017-02-22 16:58:00 +01:00
Eelco Dolstra 8b1b5f9a12
Handle CURLE_RECV_ERROR as a transient error
This fixes

  unable to download ‘https://cache.nixos.org/nar/077h8ji74y9b0qx7rjk71xd80vjqp6q5gy137r553jlvdlxdcdlk.nar.xz’: HTTP error 200 (curl error: Failure when receiving data from the peer)
2017-02-21 16:04:47 +01:00
Eelco Dolstra 99bbddedb1
Fix building without S3 support
http://hydra.nixos.org/build/49031196/nixlog/2/raw
2017-02-21 13:15:07 +01:00
Eelco Dolstra df66d346df
Log AWS retries 2017-02-21 11:50:31 +01:00
Eelco Dolstra 302386f775
Support netrc in <nix/fetchurl.nix>
This allows <nix/fetchurl.nix> to fetch private Git/Mercurial
repositories, e.g.

  import <nix/fetchurl.nix> {
    url = https://edolstra@bitbucket.org/edolstra/my-private-repo/get/80a14018daed.tar.bz2;
    sha256 = "1mgqzn7biqkq3hf2697b0jc4wabkqhmzq2srdymjfa6sb9zb6qs7";
  }

where /etc/nix/netrc contains:

  machine bitbucket.org
  login edolstra
  password blabla...

This works even when sandboxing is enabled.

To do: add unpacking support (i.e. fetchzip functionality).
2017-02-16 15:51:50 +01:00
Eelco Dolstra cde4b60919
Move netrcFile to Settings
Also get rid of Settings::processEnvironment(), it appears to be
useless.
2017-02-16 14:50:41 +01:00
Eelco Dolstra 40f0e3b366
Include "curl" in the User-Agent header
Some sites (e.g. BitBucket) give a helpful 401 error when trying to
download a private archive if the User-Agent contains "curl", but give
a redirect to a login page otherwise (so for instance
"nix-prefetch-url" will succeed but produce useless output).
2017-02-16 13:55:43 +01:00
Eelco Dolstra d1139ff36b Merge pull request #1215 from k0001/netrc-1
Add netrc-file support
2017-02-16 12:45:44 +01:00
Eelco Dolstra 9ff9c3f2f8
Add support for s3:// URIs
This adds support for s3:// URIs in all places where Nix allows URIs,
e.g. in builtins.fetchurl, builtins.fetchTarball, <nix/fetchurl.nix>
and NIX_PATH. It allows fetching resources from private S3 buckets,
using credentials obtained from the standard places (i.e. AWS_*
environment variables, ~/.aws/credentials and the EC2 metadata
server). This may not be super-useful in general, but since we already
depend on aws-sdk-cpp, it's a cheap feature to add.
2017-02-14 14:20:00 +01:00
Eelco Dolstra 62ff5ad424 Merge pull request #1224 from dezgeg/configh
Unbreak 32-bit builds by always implicitly including config.h
2017-02-13 14:44:58 +01:00
Tuomas Tynkkynen 649a81bcd6 nix-daemon: Don't splice with len=SIZE_MAX
Currently, 'nix-daemon --stdio' is always failing for me, due to the
splice call always failing with (on a 32-bit host):

splice(0, NULL, 3, NULL, 4294967295, SPLICE_F_MOVE) = -1 EINVAL (Invalid argument)

With a bit of ftracing (and luck) the problem seems to be that splice()
always fails with EINVAL if the len cast as ssize_t is negative:
http://lxr.free-electrons.com/source/fs/read_write.c?v=4.4#L384

So use SSIZE_MAX instead of SIZE_MAX.
2017-02-13 15:14:44 +02:00
Renzo Carbonara e2257d4eeb Documentation. 2017-02-09 18:16:09 +01:00
Tuomas Tynkkynen 2cd468874f Include config.h implicitly with '-include config.h' in CFLAGS
Because config.h can #define things like _FILE_OFFSET_BITS=64 and not
every compilation unit includes config.h, we currently compile half of
Nix with _FILE_OFFSET_BITS=64 and other half with _FILE_OFFSET_BITS
unset. This causes major havoc with the Settings class on e.g. 32-bit ARM,
where different compilation units disagree with the struct layout.

E.g.:

diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc
@@ -166,6 +166,8 @@ void Settings::update()
     _get(useSubstitutes, "build-use-substitutes");
+    fprintf(stderr, "at Settings::update(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
     _get(buildUsersGroup, "build-users-group");
diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc
+++ b/src/libstore/remote-store.cc
@@ -138,6 +138,8 @@ void RemoteStore::initConnection(Connection & conn)
 void RemoteStore::setOptions(Connection & conn)
 {
+    fprintf(stderr, "at RemoteStore::setOptions(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
     conn.to << wopSetOptions

Gave me:

at Settings::update(): &useSubstitutes = 0xb6e5c5cb
at RemoteStore::setOptions(): &useSubstitutes = 0xb6e5c5c7

That was not a fun one to debug!
2017-02-08 21:51:02 +02:00
Shea Levy 418a837897 Remove perl dependency.
Fixes #341
2017-02-07 15:56:32 -05:00
Eelco Dolstra 4724903c78
nix-copy-closure: Use computeFSClosure() and LegacySSHStore 2017-02-07 20:55:47 +01:00
Eelco Dolstra 6f4682ad36
Merge branch 'nix-copy-closure-c++' of https://github.com/shlevy/nix 2017-02-07 20:47:45 +01:00
Eelco Dolstra caa5793b4a
Add a LegacySSHStore that uses nix-store --serve
This is useful for nix-copy-closure.
2017-02-07 19:29:21 +01:00
Eelco Dolstra f38224e924
copyStorePath(): Don't require signatures for "trusted" stores
For example, SSH stores could be trusted.
2017-02-07 19:29:21 +01:00
Eelco Dolstra fa07558a06
Provide default implementations for a couple of Store methods 2017-02-07 19:29:21 +01:00
Eelco Dolstra ddb5577f2e
Move SavingSourceAdapter to serialise.hh 2017-02-07 19:29:20 +01:00
Eelco Dolstra ce4d8e3ef8
Remove unnecessary call to topoSortPaths()
exportPaths() already does this.
2017-02-07 19:20:41 +01:00
Eelco Dolstra 7a58ad0ef5
SSHStore: uri -> host 2017-02-07 19:20:15 +01:00
Eelco Dolstra 612aeb2df5
Cleanup 2017-02-07 19:16:25 +01:00
Eelco Dolstra c54814b175
Remove download-via-ssh
Replaced by SSHStore.
2017-02-07 18:54:33 +01:00