Commit graph

3404 commits

Author SHA1 Message Date
Eelco Dolstra 44272d8719
Rename "use-substitutes" to "substitute"
Commit c2154d4c84 renamed
"build-use-substitutes" to "use-substitutes", but that broke
"nix-copy-closure --use-substitutes".
2018-01-04 16:58:39 +01:00
Shea Levy 689b2783fc
Add hasContext primop 2018-01-02 12:25:14 -05:00
Will Dietz 9dd2b8ac7b use libbrotli directly when available
* Look for both 'brotli' and 'bro' as external command,
  since upstream has renamed it in newer versions.
  If neither are found, current runtime behavior
  is preserved: try to find 'bro' on PATH.
* Limit amount handed to BrotliEncoderCompressStream
  to ensure interrupts are processed in a timely manner.
  Testing shows negligible performance impact.
  (Other compression sinks don't seem to require this)
2017-12-30 20:26:33 -06:00
Shea Levy 6a0dd63508
Merge branch 'fix/issue-1757' of git://github.com/dtzWill/nix 2017-12-27 18:51:05 -05:00
Will Dietz bd17ccf1d8 nix repl: use linenoiseKeyType to differentiate ^C and ^D
Fixes #1757.
2017-12-26 19:25:50 -06:00
Will Dietz 2e6f06c37e fetchGit: Fix handling of local repo when not using 'master' branch
Add tests checking this behavior.
2017-12-22 15:29:52 -06:00
Eelco Dolstra aa43cbb764
Check aws-sdk-cpp version 2017-12-22 12:05:13 +01:00
Will Dietz 71987b18d4 linenoise.cpp: allow completions from empty input
Fixes #1742.
2017-12-19 12:52:10 -06:00
Eelco Dolstra 8efe937a35 Fix build on gcc 7
Fixes #1738.
2017-12-16 00:49:31 +01:00
Ben Gamari 626a94d70e edit: Catch stoi exceptions from line number parsing 2017-12-14 19:11:56 -05:00
Ben Gamari f9bcbddef2 json-to-value: Throw sensible error message on invalid numbers 2017-12-14 19:09:45 -05:00
Eelco Dolstra 5d5b931fb1 Fix build against current aws-sdk-cpp 2017-12-14 16:45:40 +01:00
Eelco Dolstra 1ca8e95178
Merge pull request #1722 from bhipple/fix-for-older-libcurl
Fix for builds with system libcurl < 7.30
2017-12-12 11:48:08 +01:00
Will Dietz 00e0c416ff Mark comparison call operator as const 2017-12-11 12:08:40 -06:00
Will Dietz e4d43f1f94 logging.cc: add missing 'override' 2017-12-11 12:08:40 -06:00
Will Dietz c800f47395 Fix forward-decl of Bindings as "class", match definition.
(appease clang -Wmismatched-tags warning)
2017-12-11 11:51:55 -06:00
Shea Levy 5ba5ca7888
Merge branch 'feature/linenoise-ng' of git://github.com/dtzWill/nix 2017-12-11 07:52:20 -05:00
Benjamin Hipple d76c4fe770 Fix for builds with system libcurl < 7.30
CentOS 7.4 and RHEL 7.4 ship with libcurl-devel-7.29.0-42.el7.x86_64; this flag
was added in 7.30.0
https://curl.haxx.se/libcurl/c/CURLMOPT_MAX_TOTAL_CONNECTIONS.html
2017-12-09 20:02:21 -05:00
Eelco Dolstra 2df9cbeb47 Provide random access to cached NARs
E.g.

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \
    /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null
  real    0m4.139s

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \
    /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null
  real    0m0.024s

(Before, the second call took ~0.220s.)

This will use a NAR listing in
/tmp/nars/b0w2hafndl09h64fhb86kw6bmhbmnpm1.ls containing all metadata,
including the offsets of regular files inside the NAR. Thus, we don't
need to read the entire NAR. (We do read the entire listing, but
that's generally pretty small. We could use a SQLite DB by borrowing
some more code from nixos-channel-scripts/file-cache.hh.)

This is primarily useful when Hydra is serving files from an S3 binary
cache, in particular when you have giant NARs. E.g. we had some 12 GiB
NARs, so accessing individuals files was pretty slow.
2017-12-07 01:07:07 +01:00
Eelco Dolstra 338f29dbd4 nix ls-{nar,store}: Return offset of files in the NAR if known
E.g.

  $ nix ls-store --json --recursive --store https://cache.nixos.org /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79 \
    | jq .entries.bin.entries.blender.narOffset
  400
2017-12-07 01:07:07 +01:00
Daiderd Jordan 2cb8aaa610
ssh-store: fix length when writing nar
This fixes nix copy and other things that use copyStorePath.
2017-12-06 11:41:08 +01:00
Shea Levy 11a7f8ce14
Merge branch 'fetchGit-fast-revision-update' 2017-12-05 11:16:39 -05:00
Eelco Dolstra 7f2c324ed1
Simplify build by including nlohmann/json.hpp 2017-12-04 17:11:36 +01:00
Eelco Dolstra f1efb97075
Fix #1635. 2017-12-04 13:19:16 +01:00
Linus Heckemann f986a44980 Fix "Unexpected EOF reading a line" error
propagated-user-env-packages files in nixpkgs aren't all terminated by
newlines, as buildenv expected. Now it does not require a terminating
newline; note that this introduces a behaviour change: propagated user
env packages may now be spread across multiple lines. However, nix
1.11.x still expects them to be on a single line so this shouldn't be
used in nixpkgs for now.
2017-11-29 11:56:15 +00:00
Eelco Dolstra b52846ab5b
Show log tail when a remote build fails 2017-11-28 15:36:28 +01:00
Will Dietz 964349e44d no "linenoiseFree" in linenoise-ng 2017-11-27 18:30:05 -06:00
Will Dietz 21f515724c replace vendored linenoise with linenoise-ng 2017-11-27 18:27:28 -06:00
aszlig 6567ab95a0
build-remote: Fix missing log output
The storeUri variable in the build-remote hook is declared very much to
the start of the main function and a bunch of lines later, the same
variable gets checked via hasPrefix() but it gets assigned *after* that
check when the most suitable machine for the build was choosen.

So I guess this was just a typo in d16fd24973
and what we really want is to either checkd the prefix *after* assigning
storeUri or use bestMachine->storeUri directly.

I choose the latter, because the former could introduce even more
regressions if the try block where the variable gets assigned terminates
early.

Nevertheless, the reason why the log output didn't work is because
hasPrefix() checked for "ssh://" in front of storeUri, but if the
storeUri isn't set correctly (or at all), we don't get the log file
descriptor set up properly, leading to no log output.

I've adjusted the remote-builds test to include a regression test for
this, so that we can make sure we get a build output when using remote
builds.

In addition to that I've tested this with two of my build farms and the
build logs are emitted correctly again.

Signed-off-by: aszlig <aszlig@nix.build>
2017-11-25 01:34:18 +01:00
Eelco Dolstra 549c3706a5
nix path-info: Show URL of NARs in binary caches 2017-11-24 18:08:50 +01:00
Eelco Dolstra 90948a4e3a
nix-shell/nix-build: Support .drv files again
Fixes #1663.

Also handle '!<output-name>' (#1694).
2017-11-24 18:08:35 +01:00
Shea Levy eedbc4e06c
fetchGit: Ignore tarballTtl if rev is set and not in the repo.
Fixes #1697.
2017-11-24 06:09:24 -05:00
Eelco Dolstra d7da6c9ea9
fetchGit/fetchMercurial: Fix directory inclusion check
E.g. the existence of .gitignore would cause .git to be included.
2017-11-21 19:34:46 +01:00
Eelco Dolstra 6cdaa858d0
Propagate flags like --sandbox to the daemon properly 2017-11-21 18:50:56 +01:00
Eelco Dolstra f5f29dc2b7
nix eval --raw: Use coerceToString()
Thus,

  $ nix eval --raw '("foo")'
  foo

  $ nix eval --raw nixpkgs.hello
  /nix/store/1y6ckg6khrdsvll54s5spcmf3w6ka9k4-hello-2.10

  $ nix eval --raw '(/etc/resolv.conf)'
  /nix/store/vml92ama92i8mz013nny461mlvg8mvap-resolv.conf
2017-11-20 18:27:29 +01:00
Eelco Dolstra 1ff01187e2
nix run: Fix "flag '--command' requires 2 argument(s)" 2017-11-20 18:07:58 +01:00
Eelco Dolstra 4eb9e20028
nix run: Fix accidental removal of /nix/store existence check
Parenthetical to #1686, we don't need to create a new root if we can
just bind-mount on top of the existing /nix/store.
2017-11-20 17:59:32 +01:00
Eelco Dolstra a3aa850f0f
nix run: Ignore non-directories while setting up the chroot
Fixes #1686.
2017-11-20 17:58:47 +01:00
Eelco Dolstra 91a1987607
signed-binary-caches -> require-sigs
Unlike signed-binary-caches (which could only be '*' or ''),
require-sigs is a proper Boolean option. The default is true.
2017-11-20 17:44:07 +01:00
Eelco Dolstra d0b88db441
Cleanup 2017-11-20 17:32:58 +01:00
Eelco Dolstra 7a2b64e55c
binary-cache-public-keys -> trusted-public-keys
The name had become a misnomer since it's not only for substitution
from binary caches, but when adding/copying any
(non-content-addressed) path to a store.
2017-11-20 17:32:34 +01:00
Eelco Dolstra 72804dc0bd
Merge pull request #1645 from twhitehead/stat-race
Fix (highly unlikely) race condition in readLink
2017-11-20 17:32:27 +01:00
Eelco Dolstra 7474ac871b
nix copy: Abbreviate "daemon" 2017-11-20 15:17:11 +01:00
Eelco Dolstra 8956ae1987
Add a "profile" option to S3BinaryCacheStore
This allows specifying the AWS configuration profile to use. E.g.

  nix copy --from s3://my-cache?profile=aws-dev-account /nix/store/cf3isrlqavvd5w7rpky1fa8j9lcnlggm-...
2017-11-15 14:18:20 +01:00
Will Dietz 9d7ce0bf45 anchor nix::Exit exception 2017-11-14 12:39:30 -06:00
Eelco Dolstra ec5b04862b
nix sign-paths: Support binary caches 2017-11-14 18:44:05 +01:00
Eelco Dolstra 2c39e4eca0
Revert "Don't parse "x:x" as a URI"
This reverts commit f90f660b24.

This broke Hydra's release.nix, which contained

  preCheck = ''export LOGNAME=${LOGNAME:-foo}'';
2017-11-14 15:10:52 +01:00
Eelco Dolstra 5773d667ee
nix ls-{nar,store}: Don't abort on missing files 2017-11-14 14:49:06 +01:00
Eelco Dolstra 4db0a9555e
nix ls-{nar,store} --json: Respect -R 2017-11-14 14:31:38 +01:00
Eelco Dolstra c8155e9f5f
Don't indent JSON output 2017-11-14 14:28:03 +01:00