Commit graph

2173 commits

Author SHA1 Message Date
Ben Burdette 15e9564fd1 logEI for tunnelLogger and progressbar 2020-04-19 17:16:51 -06:00
Ben Burdette 3d5b1032a1 logError, logWarning; Logger functions; switch to Verbosity enum 2020-04-17 15:07:44 -06:00
Eelco Dolstra a118293bd0
Merge pull request #3458 from zimbatm/nix-user-conf-dir
NIX_USER_CONF_FILES
2020-04-15 13:00:28 +02:00
Jonas Chevalier 895516cadf
add NIX_USER_CONF_FILES
Motivation: maintain project-level configuration files.

Document the whole situation a bit better so that it corresponds to the
implementation, and add NIX_USER_CONF_FILES that allows overriding
which user files Nix will load during startup.
2020-04-14 18:45:06 +02:00
Bruce Toll 8132d0a12e Fix nix-build --check -K in sandbox w/o root
Temporarily add user-write permission to build directory so that it
can be moved out of the sandbox to the store with a .check suffix.

This is necessary because the build directory has already had its
permissions set read-only, but write permission is required
to update the directory's parent link to move it out of the sandbox.

Updated the related --check "derivation may not be deterministic"
messages to consistently use the real store paths.

Added test for non-root sandbox nix-build --check -K to demonstrate
issue and help prevent regressions.
2020-04-10 16:23:10 -04:00
Domen Kožar db25a6d7bb
Merge pull request #2689 from tollb/fix/delete_tmp_dir_when_build_check_ok
Delete temporary directory on successful build
2020-04-10 09:47:57 +02:00
Bruce Toll 16a4864759 Delete temporary directory on successful build
With --check and the --keep-failed (-K) flag, the temporary directory
was being retained regardless of whether the build was successful and
reproducible.  This removes the temporary directory, as expected, on
a reproducible check build.

Added tests to verify that temporary build directories are not
retained unnecessarily, particularly when using --check with
--keep-failed.
2020-04-09 16:37:41 -04:00
Philipp Middendorf 04bedda0b6 gc.cc: Ignore hidden files in temproots 2020-04-09 17:05:29 +02:00
Nikola Knezevic f5095594e7 datatransfer.{cc,hh} -> filetransfer.{cc,hh} 2020-04-08 22:26:57 +02:00
Nikola Knezevic c330109bfa DataTransfer -> FileTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic 7848372b0f Add upload method 2020-04-08 22:26:57 +02:00
Nikola Knezevic a0c5931208 actDownload -> actDataTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic c4c1ae0a00 DownloadError -> DataTransferError 2020-04-08 22:26:57 +02:00
Nikola Knezevic 213d124277 DownloadItem -> TransferItem 2020-04-08 22:26:57 +02:00
Nikola Knezevic cd391206e6 {get,make,new}Downloader -> DataTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic 2df2741ec6 enqueueDownload -> enqueueDataTransfer 2020-04-08 22:26:57 +02:00
Nikola Knezevic 142ed7fe45 DownloadResult -> DataTransferResult 2020-04-08 22:26:57 +02:00
Nikola Knezevic e5cc53beec DownloadSettings -> DataTransferSettings 2020-04-08 22:26:57 +02:00
Nikola Knezevic 741e9012d3 Rename src/lib/download.* to src/lib/datatransfer.* 2020-04-08 22:26:57 +02:00
Nikola Knezevic 65ef57e0cb DownloadRequest -> DataTransferRequest 2020-04-08 22:26:57 +02:00
Domen Kožar bf81b31559
build.cc: improve message if home directory exists 2020-04-08 18:27:10 +02:00
Eelco Dolstra 1ab8d6ac18
Downloader: Only write data to the sink on a 200 response
Hopefully fixes #3278.
2020-04-08 15:27:09 +02:00
Eelco Dolstra cd39709003 Cleanup 2020-04-07 09:27:17 +02:00
Eelco Dolstra 462421d345 Backport libfetchers from the flakes branch
This provides a pluggable mechanism for defining new fetchers. It adds
a builtin function 'fetchTree' that generalizes existing fetchers like
'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a
set of attributes, e.g.

  fetchTree {
    type = "git";
    url = "https://example.org/repo.git";
    ref = "some-branch";
    rev = "abcdef...";
  }

The existing fetchers are just wrappers around this. Note that the
input attributes to fetchTree are the same as flake input
specifications and flake lock file entries.

All fetchers share a common cache stored in
~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching
mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}).

This also adds support for Git worktrees (c169ea5904).
2020-04-07 09:03:14 +02:00
Cole Helbling c976cb0b8a
Don't retry on "unsupported protocol" error
When encountering an unsupported protocol, there's no need to retry.
Chances are, it won't suddenly be supported between retry attempts;
error instead. Otherwise, you see something like the following:

    $ nix-env -i -f git://git@github.com/foo/bar
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 335 ms
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 604 ms
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 1340 ms
    warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 2685 ms

With this change, you now see:

    $ nix-env -i -f git://git@github.com/foo/bar
    error: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1)
2020-04-05 09:00:34 -07:00
mlatus 12556e5709 fix placeholder not substituted in passAsFile 2020-03-31 19:40:16 +08:00
John Ericson 51afea3af2 Never cast FileIngestionMethod to or from boolean 2020-03-30 22:31:51 +00:00
John Ericson c251b011cd Merge remote-tracking branch 'upstream/master' into enum-FileIngestionMethod 2020-03-30 18:16:44 -04:00
John Ericson 832bd534dc Store parsed hashes in DerivationOutput
It's best to detect invalid data as soon as possible, with data types
that make storing it impossible.
2020-03-30 11:33:35 -04:00
John Ericson f5494d9442 Merge remote-tracking branch 'me/enum-FileIngestionMethod' into HEAD 2020-03-30 11:08:13 -04:00
Eelco Dolstra e322a16523 Remove global -I flags
(cherry picked from commit 2c692a3b14)
2020-03-30 15:30:19 +02:00
John Ericson 225e62a56a Replace some bool recursive with a new FileIngestionMethod enum 2020-03-29 15:16:20 -04:00
John Ericson 87b32bab05 Use enum struct and drop prefixes
This does a few enums; the rest will be gotten in subsequent commits.
2020-03-29 11:23:15 -04:00
John Ericson e433d4af4c Extend Rust FFI
Do idiomatic C++ copy and move constructors for a few things, so
wrapping structs' defaults can work.
2020-03-25 16:12:14 -04:00
John Ericson bcde5456cc Flip dependency so store-api.hh includes derivations.hh
I think it makes more sense to define the data model (derivations),
before the operations (store api).
2020-03-24 20:39:45 +00:00
Eelco Dolstra 0a10854f85 Misc changes from the flakes branch 2020-03-24 14:34:47 +01:00
Eelco Dolstra 777e21e596 nix path-info --json: Print hash in SRI format
(cherry picked from commit 442e665d6d)
2020-03-24 14:26:23 +01:00
Eelco Dolstra f9611c7ae4 buildenv: Eliminate global variables, other cleanup
(cherry picked from commit b82f75464d)
2020-03-24 14:06:47 +01:00
Eelco Dolstra edc34cc1a2 Add function for quoting strings
(cherry picked from commit 7dcf5b011a)
2020-03-24 13:44:04 +01:00
Eelco Dolstra 5a7e7fc35f Use std::string_view
(cherry picked from commit 6529490cc1)
2020-03-24 13:26:37 +01:00
Eelco Dolstra 5885e20404
Merge pull request #3429 from LnL7/darwin-sandbox
darwin sandbox
2020-03-23 09:13:36 +01:00
Daiderd Jordan 2e9bc1245c
sandbox: fix /bin/sh on catalina
Sadly 10.15 changed /bin/sh to a shim which executes bash, this means it
can't be used anymore without also opening up the sandbox to allow bash.

    Failed to exec /bin/bash as variant for /bin/sh (1: Operation not permitted).
2020-03-20 22:12:30 +01:00
Daiderd Jordan f6c122aaeb
sandbox: allow pty devices
Nix now runs builds with a pseudo-terminal to enable colored build
output.
2020-03-20 21:58:45 +01:00
Daiderd Jordan 7f2df903d9
libstore: relax default sandbox-paths on darwin 2020-03-20 21:31:20 +01:00
Daiderd Jordan afb78ebd34
libstore: disable resolve-system-dependencies hook
This is used to determine the dependency tree of impure libraries so nix
knows what paths to open in the sandbox.  With the less restrictive
defaults it isn't needed anymore.
2020-03-20 21:21:56 +01:00
Eelco Dolstra b816515f61
Fix ca-references feature check
Fixes #3406.
2020-03-13 13:15:51 +01:00
Robert Hensing 3f55f8a8fb pathInfoCache: Respect disk cache TTLs #3398 2020-03-12 10:30:28 +01:00
Eelco Dolstra d37dc71e3c
nix-build: Fix !<output> handling
This was broken by 22a754c091.

https://hydra.nixos.org/eval/1573669
2020-03-04 13:56:17 +01:00
Eelco Dolstra 75db069f92
Optimise Derivation::unparse()
In

  nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux

this reduces time spent in unparse() from 9.15% to 4.31%. The main
culprit was appending characters one at a time to the destination
string. Even though the string has enough capacity, push_back() still
needs to check this on every call.
2020-03-04 11:44:45 +01:00
Eelco Dolstra 22a754c091
Fix GC failures on bad store path names
It failed on names like '/nix/store/9ip48nkc9rfy0a4yaw98lp6gipqlib1a-'.
2020-02-28 18:07:10 +01:00
Eelco Dolstra c4d3674de6
Merge pull request #3353 from tbsmoest/priv_tobias_pr_set_deathsig-1.4
Fix PR_SET_PDEATHSIG results in Broken pipe (#2395)
2020-02-19 12:29:12 +01:00
Eelco Dolstra 553e584f92
LocalStore::checkDerivationOutputs(): Improve error message 2020-02-18 17:51:48 +01:00
Tobias Möst 3e347220c8 Fix PR_SET_PDEATHSIG results in Broken pipe (#2395)
The ssh client is lazily started by the first worker thread, that
requires a ssh connection. To avoid the ssh client to be killed, when
the worker process is stopped, do not set PR_SET_PDEATHSIG.
2020-02-14 07:51:44 +01:00
Eelco Dolstra d8972317fc Prevent uninitialized StorePath creation 2020-02-13 16:12:16 +01:00
Eelco Dolstra d82b78bf51
Fix segfault in gcc on i686-linux
src/libstore/ssh-store.cc: In constructor 'nix::SSHStore::SSHStore(const string&, const Params&)':
  src/libstore/ssh-store.cc:31:21: internal compiler error: Segmentation fault
               compress)
                       ^
  Please submit a full bug report,
  with preprocessed source if appropriate.

https://hydra.nixos.org/build/111545609
2020-02-07 13:01:48 +01:00
Eelco Dolstra db88cb401b
Merge pull request #3344 from LnL7/ssh-ng-remote-params
ssh-store: add remote-store and remote-program query params
2020-02-04 10:10:08 +01:00
Daiderd Jordan 8745c63d3c
ssh-store: add remote-store and remote-program query params
Brings the functionality of ssh-ng:// in sync with the legacy ssh://
implementation.  Specifying the remote store uri enables various useful
things. eg.

    $ nix copy --to ssh-ng://cache?remote-store=file://mnt/cache --all
2020-02-03 23:22:28 +01:00
Eelco Dolstra 7be1a07a45
Merge pull request #3335 from domenkozar/retry-429
retry on HTTP status code 429
2020-01-29 16:22:46 +01:00
Domen Kožar 48ddb8e481
retry on HTTP status code 429 2020-01-29 11:47:39 +01:00
Robin Gloster f8dbde0813
structured-attrs: chown .attrs.* files to builder
Otherwise `chmod .`'ing the build directory doesn't work anymore, which
is done in nixpkgs if sourceRoot is set to '.'.
2020-01-23 17:38:07 +01:00
Eelco Dolstra d506bd587a Fix clang warning 2020-01-22 21:20:01 +01:00
Eelco Dolstra aef635da78 Fix derivation computation with __structuredAttrs and multiple outputs
Fixes

  error: derivation '/nix/store/klivma7r7h5lndb99f7xxmlh5whyayvg-zlib-1.2.11.drv' has incorrect output '/nix/store/fv98nnx5ykgbq8sqabilkgkbc4169q05-zlib-1.2.11-dev', should be '/nix/store/adm7pilzlj3z5k249s8b4wv3scprhzi1-zlib-1.2.11-dev'
2020-01-21 21:14:13 +01:00
Daiderd Jordan 8b3217f832
build: remove warning when in sandboxing test mode
Introduced in 66fccd5832, but somehow
breaks the secure-drv-outputs test.
2020-01-13 22:09:18 +01:00
John Ericson d64ab5131c unbreak build without pch 2020-01-13 21:45:33 +01:00
Eelco Dolstra bc22a7ee6a Fix use of uninitialized store path
Fixes 'building of '/nix/store/00000000000000000000000000000000-': ...'.
2020-01-06 22:20:10 +01:00
Eelco Dolstra e2988f48a1
Merge pull request #3303 from LnL7/darwin-sandbox
build: fix sandboxing on darwin
2020-01-06 20:56:35 +01:00
Daiderd Jordan 66fccd5832
build: fix sandboxing on darwin
Starting ba87b08f85 getEnv now returns an
std::optional which means these getEnv() != "" conditions no longer happen
if the variables are not defined.
2020-01-05 20:23:52 +01:00
Daiderd Jordan 7d448bc966
build: fix path repairing when hash rewriting is required
Handle store path repairing on darwin when sandboxing is enabled. Unlike
on linux sandboxing on darwin still requires hash rewriting.
2020-01-04 20:25:25 +01:00
Daiderd Jordan b33fefcb92
build: recover store path when replacing fails
This shouldn't happen in normal circumstances, but just in case
attempt to move the temporary path back if possible.
2020-01-04 20:24:27 +01:00
edef c65a6fa86a passAsFile: leave out the hash prefix
Having a colon in the path may cause issues, and having the hash
function indicated isn't actually necessary. We now verify the path 
format in the tests to prevent regressions.
2020-01-02 23:56:06 +00:00
Puck Meerburg 515c0a263e passAsFile: hash the attribute name instead of numbering sequentially
This makes the paths consistent without relying on ordering.

Co-authored-by: edef <edef@edef.eu>
2020-01-02 22:56:03 +00:00
Graham Christensen c502831a1d
exportReferencesGraph: support working
Before, we would get:

    [deploy@bastion:~]$ nix-store -r /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv
    these derivations will be built:
      /nix/store/3ka4ihvwh6wsyhpd2qa9f59506mnxvx1-initrd-linux-4.19.88.drv
      /nix/store/ssxwmll7v21did1c8j027q0m8w6pg41i-unit-prometheus-alertmanager-irc-notifier.service.drv
      /nix/store/mvyvkj46ay7pp7b1znqbkck2mq98k0qd-unit-script-network-local-commands-start.drv
      /nix/store/vsl1y9mz38qfk6pyirjwnfzfggz5akg6-unit-network-local-commands.service.drv
      /nix/store/wi5ighfwwb83fdmav6z6n2fw6npm9ffl-unit-prometheus-hydra-exporter.service.drv
      /nix/store/x0qkv535n75pbl3xn6nn1w7qkrg9wwyg-unit-prometheus-packet-sd.service.drv
      /nix/store/lv491znsjxdf51xnfxh9ld7r1zg14d52-unit-script-packet-sd-env-key-pre-start.drv
      /nix/store/nw4nzlca49agsajvpibx7zg5b873gk9f-unit-script-packet-sd-env-key-start.drv
      /nix/store/x674wwabdwjrkhnykair4c8mpxa9532w-unit-packet-sd-env-key.service.drv
      /nix/store/ywivz64ilb1ywlv652pkixw3vxzfvgv8-unit-wireguard-wg0.service.drv
      /nix/store/v3b648293g3zl8pnn0m1345nvmyd8dwb-unit-script-acme-selfsigned-status.nixos.org-start.drv
      /nix/store/zci5d3zvr6fgdicz6k7jjka6lmx0v3g4-unit-acme-selfsigned-status.nixos.org.service.drv
      /nix/store/f6pwvnm63d0kw5df0v7sipd1rkhqxk5g-system-units.drv
      /nix/store/iax8071knxk9c7krpm9jqg0lcrawf4lc-etc.drv
      /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv
    error: invalid file name 'closure-init-0' in 'exportReferencesGraph'

This was tough to debug, I didn't figure out which one was broken until I did:

    nix-store -r /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv 2>&1 | grep  nix/store | xargs -n1 nix-store -r

and then looking at the remaining build graph:

    $ nix-store -r /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv
    these derivations will be built:
      /nix/store/3ka4ihvwh6wsyhpd2qa9f59506mnxvx1-initrd-linux-4.19.88.drv
      /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv
    error: invalid file name 'closure-init-0' in 'exportReferencesGraph'

and knowing the initrd build is before the system, then:

    $ nix show-derivation /nix/store/3ka4ihvwh6wsyhpd2qa9f59506mnxvx1-initrd-linux-4.19.88.drv
    {
      "/nix/store/3ka4ihvwh6wsyhpd2qa9f59506mnxvx1-initrd-linux-4.19.88.drv": {
        [...]
        "exportReferencesGraph": "closure-init-0 /nix/store/...-stage-1-init.sh closure-mdadm.conf-1 /nix/store/...-mdadm.conf closure-ubuntu.conf-2 ...",
        [...]
      }
    }

I then searched the repo for "in 'exportReferencesGraph'", found this
recently updated regex, and realized it was missing a "-".
2020-01-01 20:50:40 -05:00
Brian McKenna d25923263e Disable use-sqlite-wal under WSL
Before:

    $ nix-channel --update
    unpacking channels...
    warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy (SQLITE_PROTOCOL)
    warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy (SQLITE_PROTOCOL)
    warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy (SQLITE_PROTOCOL)
    warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy (SQLITE_PROTOCOL)
    warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy (SQLITE_PROTOCOL)

After:

    $ inst/bin/nix-channel --update
    unpacking channels...
    created 1 symlinks in user environment

I've seen complaints that "sandbox" caused problems under WSL but I'm
having no problems. I think recent changes could have fixed the issue.
2019-12-21 08:14:19 +11:00
Eelco Dolstra ee235e764c
Merge branch 'libarchive' of https://github.com/yorickvP/nix 2019-12-19 14:47:18 +01:00
Eelco Dolstra f8abbdd456 Add priority setting to stores
This allows overriding the priority of substituters, e.g.

  $ nix-store --store ~/my-nix/ -r /nix/store/df3m4da96d84ljzxx4mygfshm1p0r2n3-geeqie-1.4 \
    --substituters 'http://cache.nixos.org?priority=100 daemon?priority=10'

Fixes #3264.
2019-12-17 17:17:53 +01:00
Eelco Dolstra 54bf5ba422 nix-store -r: Handle symlinks to store paths
Fixes #3270.
2019-12-16 19:11:47 +01:00
Eelco Dolstra e8aa2290ed Only install *.sb files on macOS 2019-12-13 14:42:55 +01:00
Eelco Dolstra d1b238ec3c Simplify 2019-12-13 12:53:20 +01:00
Profpatsch 38b29fb72c libstore/ssh: Improve error message on failing execvp
If the `throw` is reached, this means that execvp into `ssh` wasn’t
successful. We can hint at a usual problem, which is a missing `ssh`
executable.

Test with:

```
env PATH= ./result/bin/nix-copy-closure --builders '' unusedhost
```

and the bash version with

```
env PATH= ./result/bin/nix-copy-closure --builders '' localhost
```
2019-12-12 15:32:17 +01:00
Eelco Dolstra bbe97dff8b Make the Store API more type-safe
Most functions now take a StorePath argument rather than a Path (which
is just an alias for std::string). The StorePath constructor ensures
that the path is syntactically correct (i.e. it looks like
<store-dir>/<base32-hash>-<name>). Similarly, functions like
buildPaths() now take a StorePathWithOutputs, rather than abusing Path
by adding a '!<outputs>' suffix.

Note that the StorePath type is implemented in Rust. This involves
some hackery to allow Rust values to be used directly in C++, via a
helper type whose destructor calls the Rust type's drop()
function. The main issue is the dynamic nature of C++ move semantics:
after we have moved a Rust value, we should not call the drop function
on the original value. So when we move a value, we set the original
value to bitwise zero, and the destructor only calls drop() if the
value is not bitwise zero. This should be sufficient for most types.

Also lots of minor cleanups to the C++ API to make it more modern
(e.g. using std::optional and std::string_view in some places).
2019-12-10 22:06:05 +01:00
Eelco Dolstra 61cc9f34d2 Remove UserLock self-lock check
This is no longer needed since we're not using POSIX locks anymore.
2019-12-09 23:57:33 +01:00
Yorick 9ff5f6492f
libarchive proof of concept 2019-12-07 22:35:14 +07:00
Eelco Dolstra 47a937d512 Show hash mismatch warnings in SRI format
(cherry picked from commit 63c5c91cc0)
2019-12-05 20:32:25 +01:00
Eelco Dolstra ac2bc721d8 Merge remote-tracking branch 'origin/recursive-nix' 2019-12-02 12:34:46 +01:00
Brian Wignall 8737980e75 Fix typos 2019-11-30 19:04:14 -05:00
Eelco Dolstra 8918bae098 Drop remaining uses of external "tar"
Also, fetchGit now runs in O(1) memory since we pipe the output of
'git archive' directly into unpackTarball() (rather than first reading
it all into memory).
2019-11-26 22:07:28 +01:00
Eelco Dolstra d33dd6e6c0 Move code around 2019-11-26 22:07:28 +01:00
Eelco Dolstra b7fba16613 Move code around 2019-11-26 22:07:28 +01:00
Eelco Dolstra f738cd4d97 More Rust FFI adventures
We can now convert Rust Errors to C++ exceptions. At the Rust->C++ FFI
boundary, Result<T, Error> will cause Error to be converted to and
thrown as a C++ exception.
2019-11-26 22:07:28 +01:00
Eelco Dolstra 8110b4ebb2 Rust cleanup 2019-11-26 22:07:28 +01:00
Eelco Dolstra 045708db43 Make <nix/unpack-channel.nix> a builtin builder
This was the last function using a shell script, so this allows us to
get rid of tar, coreutils, bash etc.
2019-11-26 22:07:28 +01:00
Eelco Dolstra 313106d549
Fix clang warnings 2019-11-26 21:07:44 +01:00
Eelco Dolstra 425991883a
Merge pull request #3141 from xbreak/nocafile
Downloader: Log configured CA file
2019-11-26 20:52:25 +01:00
Eelco Dolstra 7c8d7c17f8
Merge pull request #3144 from matthewbauer/fix-sandbox-fallback
Fix sandbox fallback settings
2019-11-26 20:51:52 +01:00
Eelco Dolstra ec5e7b44ff
Simplify 2019-11-26 20:26:22 +01:00
Eelco Dolstra 96e1c39bb7
Merge branch 'repair-bad-links' of https://github.com/chkno/nix 2019-11-26 20:21:48 +01:00
Eelco Dolstra c13193017f
Disallow empty store path names
Fixes #3239.
2019-11-26 20:12:15 +01:00
Eelco Dolstra 89db9353d7
Doh 2019-11-26 20:08:25 +01:00
Eelco Dolstra 1ec6e6e11e
Add feature to disable URL literals
E.g.

  $ nix-build '<nixpkgs>' -A hello --experimental-features no-url-literals
  error: URL literals are disabled, at /nix/store/vsjamkzh15r3c779q2711az826hqgvzr-nixpkgs-20.03pre194957.bef773ed53f/nixpkgs/pkgs/top-level/all-packages.nix:1236:11

Helps with implementing https://github.com/NixOS/rfcs/pull/45.
2019-11-26 19:48:34 +01:00
Eelco Dolstra ba87b08f85
getEnv(): Return std::optional
This allows distinguishing between an empty value and no value.
2019-11-22 16:18:13 +01:00
Chuck 3e2c77d001 Check for and repair bad .links entries
A corrupt entry in .links prevents adding a fixed version of that file
to the store in any path.  The user experience is that corruption
present in the store 'spreads' to new paths added to the store:

(With store optimisation enabled)

1. A file in the store gets corrupted somehow (eg: filesystem bug).
2. The user tries to add a thing to the store which contains a good copy
   of the corrupted file.
3. The file being added to the store is hashed, found to match the bad
   .links entry, and is replaced by a link to the bad .links entry.
   (The .links entry's hash is not verified during add -- this would
   impose a substantial performance burden.)
4. The user observes that the thing in the store that is supposed to be
   a copy of what they were trying to add is not a correct copy -- some
   files have different contents!  Running "nix-store --verify
   --check-contents --repair" does not fix the problem.

This change makes "nix-store --verify --check-contents --repair" fix
this problem.  Bad .links entries are simply removed, allowing future
attempts to insert a good copy of the file to succeed.
2019-11-15 11:55:36 -08:00
Eelco Dolstra 5ee23c35b9
Merge pull request #3219 from Ericson2314/semicolons
Fix extra semicolons warnings
2019-11-11 12:13:51 +01:00
John Ericson 8669db1dcc Clean up semicolon and comma
Thanks @bhipple for catching!
2019-11-10 16:21:59 -05:00
John Ericson 96e6e680c1 Fix extra ; warnings involving MakeError 2019-11-10 11:24:47 -05:00
Eelco Dolstra 04bf9acd22
Remove #include 2019-11-07 10:12:35 +01:00
Eelco Dolstra 39a2e166dd
Cleanup 2019-11-06 16:53:02 +01:00
Eelco Dolstra 69326f3637
Recursive Nix: Handle concurrent client connections 2019-11-06 00:55:03 +01:00
Eelco Dolstra c119ab9db0
Enable recursive Nix using a feature
Derivations that want to use recursion should now set

  requiredSystemFeatures = [ "recursive-nix" ];

to make the daemon socket appear.

Also, Nix should be configured with "experimental-features =
recursive-nix".
2019-11-06 00:55:03 +01:00
Eelco Dolstra c921074c19
RestrictedStore: Implement addToStore() 2019-11-06 00:55:03 +01:00
Eelco Dolstra c4d7c76b64
Recursive Nix support
This allows Nix builders to call Nix to build derivations, with some
limitations.

Example:

  let nixpkgs = fetchTarball channel:nixos-18.03; in

  with import <nixpkgs> {};

  runCommand "foo"
    {
      buildInputs = [ nix jq ];
      NIX_PATH = "nixpkgs=${nixpkgs}";
    }
    ''
      hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })')

      $hello/bin/hello

      mkdir -p $out/bin
      ln -s $hello/bin/hello $out/bin/hello

      nix path-info -r --json $hello | jq .
    ''

This derivation makes a recursive Nix call to build GNU Hello and
symlinks it from its $out, i.e.

  # ll ./result/bin/
  lrwxrwxrwx 1 root root 63 Jan  1  1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello

  # nix-store -qR ./result
  /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131
  /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5
  /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo

This is implemented as follows:

* Before running the outer builder, Nix creates a Unix domain socket
  '.nix-socket' in the builder's temporary directory and sets
  $NIX_REMOTE to point to it. It starts a thread to process
  connections to this socket. (Thus you don't need to have nix-daemon
  running.)

* The daemon thread uses a wrapper store (RestrictedStore) to keep
  track of paths added through recursive Nix calls, to implement some
  restrictions (see below), and to do some censorship (e.g. for
  purity, queryPathInfo() won't return impure information such as
  signatures and timestamps).

* After the build finishes, the output paths are scanned for
  references to the paths added through recursive Nix calls (in
  addition to the inputs closure). Thus, in the example above, $out
  has a reference to $hello.

The main restriction on recursive Nix calls is that they cannot do
arbitrary substitutions. For example, doing

  nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10

is forbidden unless /nix/store/kmwd... is in the inputs closure or
previously built by a recursive Nix call. This is to prevent
irreproducible derivations that have hidden dependencies on
substituters or the current store contents. Building a derivation is
fine, however, and Nix will use substitutes if available. In other
words, the builder has to present proof that it knows how to build a
desired store path from scratch by constructing a derivation graph for
that path.

Probably we should also disallow instantiating/building fixed-output
derivations (specifically, those that access the network, but
currently we have no way to mark fixed-output derivations that don't
access the network). Otherwise sandboxed derivations can bypass
sandbox restrictions and access the network.

When sandboxing is enabled, we make paths appear in the sandbox of the
builder by entering the mount namespace of the builder and
bind-mounting each path. This is tricky because we do a pivot_root()
in the builder to change the root directory of its mount namespace,
and thus the host /nix/store is not visible in the mount namespace of
the builder. To get around this, just before doing pivot_root(), we
branch a second mount namespace that shares its /nix/store mountpoint
with the parent.

Recursive Nix currently doesn't work on macOS in sandboxed mode
(because we can't change the sandbox policy of a running build) and in
non-root mode (because setns() barfs).
2019-11-06 00:52:38 +01:00
Eelco Dolstra 6bff1aa46d
Merge pull request #3182 from bhipple/fixup/comments
Minor updates to inline comments
2019-10-31 14:14:35 +01:00
Benjamin Hipple 80d5ec6ff4 Minor updates to inline comments
Add missing docstring on InstallableCommand. Also, some of these were wrapped
when they're right next to a line longer than the unwrapped line, so we can just
unwrap them to save vertical space.
2019-10-31 05:56:37 -04:00
Will Dietz 0e9b72e097 minor: fix mismatch of struct/class forward decl of 'Source'
Fixes the following warning and the indicate potential issue:

src/libstore/worker-protocol.hh:66:1: warning: class 'Source' was previously declared as a struct; this is valid, but may result in linker errors
under the Microsoft C++ ABI [-Wmismatched-tags]

(cherry picked from commit 6e1bb04870b1b723282d32182af286646f13bf3c)
2019-10-30 14:39:01 -05:00
Eelco Dolstra e5319a87ce
queryPathInfoUncached(): Return const ValidPathInfo 2019-10-29 13:53:04 +01:00
Eelco Dolstra 992a2ad475
Move addToStoreFromDump to Store 2019-10-29 13:38:24 +01:00
Eelco Dolstra 05819d013f
Don't create a Store in processConnection() 2019-10-29 13:36:19 +01:00
Eelco Dolstra 2d37e88319
Move most of the daemon implementation to libstore 2019-10-29 13:25:33 +01:00
Eelco Dolstra 95c727caef
Remove the check against concurrent builds in the same process 2019-10-29 12:43:20 +01:00
Eelco Dolstra f7ce80f90a
Factor out linkOrCopy() 2019-10-27 18:19:13 +01:00
Eelco Dolstra 629b9b0049 Mark content-addressable paths with references as experimental 2019-10-21 18:05:31 +02:00
Eelco Dolstra 0abb3ad537 Allow content-addressable paths to have references
This adds a command 'nix make-content-addressable' that rewrites the
specified store paths into content-addressable paths. The advantage of
such paths is that 1) they can be imported without signatures; 2) they
can enable deduplication in cases where derivation changes do not
cause output changes (apart from store path hashes).

For example,

  $ nix make-content-addressable -r nixpkgs.cowsay
  rewrote '/nix/store/g1g31ah55xdia1jdqabv1imf6mcw0nb1-glibc-2.25-49' to '/nix/store/48jfj7bg78a8n4f2nhg269rgw1936vj4-glibc-2.25-49'
  ...
  rewrote '/nix/store/qbi6rzpk0bxjw8lw6azn2mc7ynnn455q-cowsay-3.03+dfsg1-16' to '/nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16'

We can then copy the resulting closure to another store without
signatures:

  $ nix copy --trusted-public-keys '' ---to ~/my-nix /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16

In order to support self-references in content-addressable paths,
these paths are hashed "modulo" self-references, meaning that
self-references are zeroed out during hashing. Somewhat annoyingly,
this means that the NAR hash stored in the Nix database is no longer
necessarily equal to the output of "nix hash-path"; for
content-addressable paths, you need to pass the --modulo flag:

  $ nix path-info --json /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16  | jq -r .[].narHash
  sha256:0ri611gdilz2c9rsibqhsipbfs9vwcqvs811a52i2bnkhv7w9mgw

  $ nix hash-path --type sha256 --base32 /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16
  1ggznh07khq0hz6id09pqws3a8q9pn03ya3c03nwck1kwq8rclzs

  $ nix hash-path --type sha256 --base32 /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16 --modulo iq6g2x4q62xp7y7493bibx0qn5w7xz67
  0ri611gdilz2c9rsibqhsipbfs9vwcqvs811a52i2bnkhv7w9mgw
2019-10-21 17:47:24 +02:00
Eelco Dolstra aabf5c86c9
Add experimental-features setting
Experimental features are now opt-in. There is currently one
experimental feature: "nix-command" (which enables the "nix"
command. This will allow us to merge experimental features more
quickly, without committing to supporting them indefinitely.

Typical usage:

$ nix build --experimental-features 'nix-command flakes' nixpkgs#hello

(cherry picked from commit 8e478c2341,
without the "flakes" feature)
2019-10-21 13:34:44 +02:00
xbreak 7c568d4c6e Downloader: Warn if no trusted CA file has been configured 2019-10-18 19:08:33 +00:00
Matthew Bauer 96c84937c4 Move tmpDirInSandbox to initTmpDir 2019-10-13 16:41:49 -04:00
Matthew Bauer 499b038875 Fix sandbox fallback settings
The tmpDirInSandbox is different when in sandboxed vs. non-sandboxed.
Since we don’t know ahead of time here whether sandboxing is enabled,
we need to reset all of the env vars we’ve set previously. This fixes
the issue encountered in https://github.com/NixOS/nixpkgs/issues/70856.
2019-10-12 19:22:13 -04:00
Eelco Dolstra 906d56a96b
ssh-ng: Don't set CPU affinity on the remote
Fixes #3138.
2019-10-11 18:49:46 +02:00
Eelco Dolstra bda64a2b0f
Doh
https://hydra.nixos.org/build/102803044
2019-10-10 00:12:30 +02:00
Eelco Dolstra 20eec802ff
Force per-user group to a known value 2019-10-09 23:35:02 +02:00
Eelco Dolstra d7bae5680f
Go back to 755 permission on per-user directories
700 is pointless since the store is world-readable anyway. And
per-user/root/channels must be world-readable.
2019-10-09 23:35:02 +02:00
Eelco Dolstra 9348f9291e
nix-env: Create ~/.nix-profile automatically 2019-10-09 23:35:01 +02:00
Eelco Dolstra 5a303093dc
Remove world-writability from per-user directories
'nix-daemon' now creates subdirectories for users when they first
connect.

Fixes #509 (CVE-2019-17365).
Should also fix #3127.
2019-10-09 23:34:48 +02:00
Eelco Dolstra 99b73fb507
OCD performance fix: {find,count}+insert => insert 2019-10-09 16:06:29 +02:00
Matthew Bauer 199e888785 Handle empty sandbox_shell
Previously, SANDBOX_SHELL was set to empty when unavailable. This
caused issues when actually generating the sandbox. Instead, just set
SANDBOX_SHELL when --with-sandbox-shell= is non-empty. Alternative
implementation to https://github.com/NixOS/nix/pull/3038.
2019-10-08 23:12:54 -04:00
Eelco Dolstra 93b1ce1ac5
Revert "std::uncaught_exception() -> std::uncaught_exceptions()"
This reverts commit 6b83174fff because
it doesn't work on macOS yet.

https://hydra.nixos.org/build/102617587
2019-10-04 16:34:59 +02:00
Domen Kožar 2d2769f68c
Merge pull request #2338 from bobvanderlinden/pr-cannot-delete-alive-why
mention `nix-store --query --roots` when a path cannot be deleted
2019-09-30 14:06:52 +02:00
Eelco Dolstra 5038e1bec4
Merge pull request #3103 from bhipple/fix/spelling
Fix spelling in comment
2019-09-23 10:46:40 +02:00
Eelco Dolstra 02b4632e77
Merge pull request #3104 from zimbatm/no-show-trace-forwarding
libstore: don't forward --show-trace
2019-09-23 10:44:42 +02:00
Eelco Dolstra 3a022d4599 Shut up some warnings
(cherry picked from commit 99e8e58f2d)
2019-09-22 21:57:05 +02:00
Eelco Dolstra bd79c1f6f6 Don't catch exceptions by value
(cherry picked from commit 893be6f5e3)
2019-09-22 21:56:56 +02:00
Jonas Chevalier e63c9e73e3
libstore: don't forward --show-trace 2019-09-22 10:57:20 +00:00
Benjamin Hipple c6a542f22a Fix spelling in comment 2019-09-21 18:53:15 -04:00
Eelco Dolstra 6b83174fff std::uncaught_exception() -> std::uncaught_exceptions()
The former is deprecated in C++17. Fixes a clang warning.
2019-09-13 20:05:44 +02:00
Eelco Dolstra 5dafde28db
BinaryCacheStore: Add index-debug-info option
This integrates the functionality of the index-debuginfo program in
nixos-channel-scripts to maintain an index of DWARF debuginfo files in
a format usable by dwarffs. Thus the debug info index is updated by
Hydra rather than by the channel mirroring script.

Example usage:

  $ nix copy --to 'file:///tmp/binary-cache?index-debug-info=true' /nix/store/vr9mhcch3fljzzkjld3kvkggvpq38cva-nix-2.2.2-debug

  $ cat /tmp/binary-cache/debuginfo/036b210b03bad75ab2d8fc80b7a146f98e7f1ecf.debug
  {"archive":"../nar/0313h2kdhk4v73xna9ysiksp2v8xrsk5xsw79mmwr3rg7byb4ka8.nar.xz","member":"lib/debug/.build-id/03/6b210b03bad75ab2d8fc80b7a146f98e7f1ecf.debug"}

Fixes #3083.
2019-09-04 19:28:26 +02:00
Eelco Dolstra e07ec8d27e
Support allowSubstitutes attribute in structured attribute derivations
Hopefully fixes #3081 (didn't test).
2019-09-03 16:03:49 +02:00
Eelco Dolstra f186000367
Add some noexcepts
This is to assert that callback functions should never throw (since
the context in which they're called may not be able to handle the
exception).
2019-09-03 13:45:35 +02:00
Eelco Dolstra 7348653ff4
Ensure that Callback is called only once
Also, make Callback movable but uncopyable.
2019-09-03 13:45:35 +02:00
Eelco Dolstra 8c4ea7a451
Downloader: Remove a possible double call to Callback 2019-09-03 13:45:32 +02:00
Eelco Dolstra 84de821004
Merge pull request #3069 from matthewbauer/max-name
Set maximum name length in Nix
2019-08-29 15:22:36 +02:00
Eelco Dolstra a2c4fcd5e9 Don't rely on st_blocks
It doesn't seem very reliable on ZFS.
2019-08-29 14:49:58 +02:00
Eelco Dolstra f27e53f77e Cleanup 2019-08-29 12:09:58 +02:00
Matthew Bauer 693e68e09c Set maximum name length in Nix
Previously we allowed any length of name for Nix derivations. This is
bad because different file systems have different max lengths. To make
things predictable, I have picked a max. This was done by trying to
build this derivation:

  derivation {
    name = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
    builder = "/no-such-path";
    system = "x86_64-linux";
  }

Take off one a and it will not lead to file name too long. That ends
up being 212 a’s. An even smaller max could be picked if we want to
support more file systems.

Working backwards, this is why:

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-${name}.drv.chroot

> 255 - 32 - 1 - 4 - 7 = 211
2019-08-28 12:32:54 -04:00
Eelco Dolstra 7298a38a07
Don't send certain setting overrides to the daemon
These are already handled separately. This fixes warnings like

  warning: ignoring the user-specified setting 'max-jobs', because it is a restricted setting and you are not a trusted user

when using the -j flag.
2019-08-28 16:29:44 +02:00
Eelco Dolstra e5b397b2c7 Merge branch 'test-sandboxing' of https://github.com/matthewbauer/nix 2019-08-27 20:58:47 +02:00
Matthew Bauer 5c06a8d328 Reset tmpDirInSandbox for unsandboxed 2019-08-23 20:24:39 -04:00
Eelco Dolstra 35ebae198f
Merge pull request #3031 from grahamc/low-speed-limit
conf: stalled-download-timeout: make tunable
2019-08-08 22:06:26 +02:00
Graham Christensen a02457db71
conf: stalled-download-timeout: make tunable
Make curl's low speed limit configurable via stalled-download-timeout.
Before, this limit was five minutes without receiving a single byte.
This is much too long as if the remote end may not have even
acknowledged the HTTP request.
2019-08-08 10:22:13 -04:00
Eelco Dolstra f9021c4c6c
Merge pull request #3030 from dtzWill/fix/missing-include-ocloexec
pathlocks: add include to fcntl.h for O_CLOEXEC
2019-08-07 22:03:09 +02:00
Eelco Dolstra 56df30cd3f
Merge pull request #2995 from tweag/post-build-hook
Add a post build hook
2019-08-07 15:02:29 +02:00
Will Dietz c3fefd1a6e
pathlocks: add include to fcntl.h for O_CLOEXEC 2019-08-07 07:41:22 -05:00
Eelco Dolstra 399b6f3c46
nix-store --verify: Don't repair while holding the GC lock 2019-08-02 18:48:26 +02:00
Eelco Dolstra a2597d5f27
Simplify
With BSD locks we don't have to guard against reading our own
temproots.
2019-08-02 18:39:16 +02:00
Eelco Dolstra e349f2c0a3
Use BSD instead of POSIX file locks
POSIX file locks are essentially incompatible with multithreading. BSD
locks have much saner semantics. We need this now that there can be
multiple concurrent LocalStore::buildPaths() invocations.
2019-08-02 18:39:16 +02:00
Eelco Dolstra ec415d7166
Add a test for auto-GC
This currently fails because we're using POSIX file locks. So when the
garbage collector opens and closes its own temproots file, it causes
the lock to be released and then deleted by another GC instance.
2019-08-02 18:39:16 +02:00
regnat 7c5596734f
Add a post-build-hook
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:

    DRV_PATH=/nix/store/drv-that-has-been-built.drv
    OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev

This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).

This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:

    [nix]$ ./inst/bin/nix-build ./test.nix
    these derivations will be built:
      /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
    building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
    hello!
    bye!
    running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + printf 'very important stuff'
    /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation

    [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
    my-example-derivation> hello!
    my-example-derivation> bye!
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + printf 'very important stuff'
    [1 built, 0.0 MiB DL]

Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-08-02 10:48:15 -04:00
Matthew Bauer 9a0855bbb6 Don’t rely on EPERM
startProcess does not appear to send the exit code to the helper
correctly. Not sure why this is, but it is probably safe to just
fallback on all sandbox errors.
2019-07-30 17:53:37 -04:00
Bas van Dijk ee1e3132ca Disable findRuntimeRoots on darwin when running tests because lsof is slow
See: https://github.com/NixOS/nix/issues/3011
2019-07-30 11:29:03 +02:00
Matthew Bauer 11d8534629 Use sandbox fallback when cloning fails in builder
When sandbox-fallback = true (the default), the Nix builder will fall
back to disabled sandbox mode when the kernel doesn’t allow users to
set it up. This prevents hard errors from occuring in tricky places,
especially the initial installer. To restore the previous behavior,
users can set:

  sandbox-fallback = false

in their /etc/nix/nix.conf configuration.
2019-07-25 14:42:30 -04:00
Matthew Bauer d171090530 Disable CLONE_NEWUSER when it’s unavailable
Some kernels disable "unpriveleged user namespaces". This is
unfortunate, but we can still use mount namespaces. Anyway, since each
builder has its own nixbld user, we already have most of the benefits
of user namespaces.
2019-07-25 14:42:25 -04:00
Eelco Dolstra 2f853b20df
Merge pull request #2975 from matthewbauer/fix-nsswitch-issue
Don’t use entire /etc/nsswitch.conf file
2019-07-13 17:08:02 +02:00
Eelco Dolstra 53247d6b11
Resume NAR downloads
This is a much simpler fix to the 'error 9 while decompressing xz
file' problem than 78fa47a7f0. We just
do a ranged HTTP request starting after the data that we previously
wrote into the sink.

Fixes #2952, #379.
2019-07-10 23:12:17 +02:00
Eelco Dolstra 00f6fafad6
HttpBinaryCacheStore: Use default number of retries for NARs 2019-07-10 23:05:04 +02:00
Eelco Dolstra f76b2a7fdd
Downloader: Use warn() 2019-07-10 22:27:50 +02:00
Eelco Dolstra 03f09e1d18
Revert "Fix 'error 9 while decompressing xz file'"
This reverts commit 78fa47a7f0.
2019-07-10 19:46:15 +02:00
Eelco Dolstra 7e1c85c5fb
Merge pull request #2779 from LnL7/build-exit-codes
build: add exit code for hash and check mismatches
2019-07-02 17:37:49 +02:00
Eelco Dolstra db700f730e
Merge pull request #2974 from grahamc/invalid-name
checkStoreName: give more precise/verbose error information
2019-07-02 16:12:01 +02:00
Graham Christensen 17d3ec3405
checkStoreName: give more precise/verbose error information
$ sudo ./inst/bin/nix-instantiate -E '"${./.git}"'
error: The path name '.git' is invalid: it is illegal to start the
name with a period. Path names are alphanumeric and can include the
symbols +-._?= and must not begin with a period. Note: If '.git' is a
source file and you cannot rename it on disk,
builtins.path { name = ... } can be used to give it an alternative
name.
2019-07-02 08:41:53 -04:00
Daiderd Jordan a52c331edb
build: replace 100 offset for build exit codes 2019-07-02 00:12:38 +02:00
Daiderd Jordan cbf84bcce7
build: use binary mask for build status flags
If multiple builds with fail with different errors it will be reflected
in the status code.

eg.

	103 => timeout + hash mismatch
	105 => timeout + check mismatch
	106 => hash mismatch + check mismatch
	107 => timeout + hash mismatch + check mismatch
2019-07-02 00:12:38 +02:00
Daiderd Jordan 97baf32fbc
build: add exit code for hash and check mismatches
Makes it easier to identify the failure reason in other tooling, eg.
differentiate between a non-deterministic --check vs a failed build.

	$ nix-build '<nix/fetchurl.nix>' --argstr url http://example.org --argstr sha256 0000000000000000000000000000000000000000000000000000
	hash mismatch in fixed-output derivation '/nix/store/nzi9ck45rwlxzcwr25is7qlf3hs5xl83-example.org':
	  wanted: sha256:0000000000000000000000000000000000000000000000000000
	  got:    sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm
	$ echo $?
	102

	$ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" {} "date +%s > $out"' --check
	warning: rewriting hashes in '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo'; cross fingers
	error: derivation '/nix/store/mggc8dz13ackb49qca6m23zq4fpq132q-foo.drv' may not be deterministic: output '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo' differs
	$ echo $?
	104
2019-07-02 00:12:34 +02:00
Aniket Deshpande ec58ba38c5
Fix http2 = false having no effect. Fixes #2971.
Setting `http2 = false` in nix config (e.g. /etc/nix/nix.conf)
had no effect, and `nix-env -vvvvv -i hello` still downloaded .nar
packages using HTTP/2.

In `src/libstore/download.cc`, the `CURL_HTTP_VERSION_2TLS` option was
being explicitly set when `downloadSettings.enableHttp2` was `true`,
but, `CURL_HTTP_VERSION_1_1` option was not being explicitly set when
`downloadSettings.enableHttp2` was `false`.

This may be because `https://curl.haxx.se/libcurl/c/libcurl-env.html` states:
"You have to set this option if you want to use libcurl's HTTP/2 support."
but, also, in the changelog, states:
"DEFAULT
Since curl 7.62.0: CURL_HTTP_VERSION_2TLS
Before that: CURL_HTTP_VERSION_1_1"

So, the default setting for `libcurl` is HTTP/2 for version >= 7.62.0.

In this commit, option `CURLOPT_HTTP_VERSION` is explicitly set to
`CURL_HTTP_VERSION_1_1` when `downloadSettings.enableHttp2` nix config
setting is `false`.

This can be tested by running `nix-env -vvvvv -i hello | grep HTTP`
2019-06-28 20:44:46 +05:30
Matthew Bauer ec0087df0a Don’t use entire /etc/nsswitch.conf file
The default nsswitch.conf(5) file in most distros can handle many
different things including host name, user names, groups, etc. In Nix,
we want to limit the amount of impurities that come from these things.
As a result, we should only allow nss to be used for gethostbyname(3)
and getservent(3).

/cc @Ericson2314
2019-06-27 14:31:29 -04:00
Eelco Dolstra 88571219d9
nix-channel: Don't fetch binary-cache-url
This has been ignored since the Perl->C++ rewrite.
2019-06-25 13:27:16 +02:00
Eelco Dolstra 64ec087f58
Fix 32-bit overflow with --no-net
--no-net causes tarballTtl to be set to the largest 32-bit integer,
which causes comparison like 'time + tarballTtl < other_time' to
fail on 32-bit systems. So cast them to 64-bit first.

https://hydra.nixos.org/build/95076624
(cherry picked from commit 29ccb2e969)
2019-06-24 22:16:43 +02:00
Eelco Dolstra f8b30338ac
Refactor downloadCached() interface
(cherry picked from commit df3f5a78d5)
2019-06-24 22:12:26 +02:00
Eelco Dolstra 7b9c68766d
Add '--no-net' convenience flag
This flag

* Disables substituters.

* Sets the tarball-ttl to infinity (ensuring e.g. that the flake
  registry and any downloaded flakes are considered current).

* Disables retrying downloads and sets the connection timeout to the
  minimum. (So it doesn't completely disable downloads at the moment.)

(cherry picked from commit 8ea842260b)
2019-06-24 22:07:29 +02:00
Eelco Dolstra 78fa47a7f0
Fix 'error 9 while decompressing xz file'
Once we've started writing data to a Sink, we can't restart a download
request, because then we end up writing duplicate data to the
Sink. Therefore we shouldn't handle retries in Downloader but at a
higher level (in particular, in copyStorePath()).

Fixes #2952.

(cherry picked from commit a67cf5a358)
2019-06-24 21:59:51 +02:00
Eelco Dolstra 2fef4dd296
Downloader: Propagate exceptions from decompressionSink->finish()
(cherry picked from commit 15fa70cd1b)
2019-06-24 21:59:31 +02:00
Eelco Dolstra b43e1e186e
CachedDownloadResult: Include store path
Also, make fetchGit and fetchMercurial update allowedPaths properly.

(Maybe the evaluator, rather than the caller of the evaluator, should
apply toRealPath(), but that's a bigger change.)

(cherry picked from commit 5c34d66538)
2019-06-24 21:59:27 +02:00
Eelco Dolstra dc29e9fb47
downloadCached: Return ETag
(cherry picked from commit 529add316c)
2019-06-24 21:58:33 +02:00
Eelco Dolstra 3cc1125595
Another attempt at getting pseudoterminals to work on macOS 2019-06-17 08:08:04 +02:00
Eelco Dolstra 2743bf0bb1 Hopefully fix macOS tests 2019-06-16 20:02:40 +02:00
Eelco Dolstra 82ca6ef390 Set $TERM 2019-06-16 17:36:30 +02:00
Eelco Dolstra e84c265645 Run builds in a pseudo-terminal
This allows many programs (e.g. gcc, clang, cmake) to print colorized
log output (assuming $TERM is set to a value like "xterm").

There are other ways to get colors, in particular setting
CLICOLOR_FORCE, but they're less widely supported and can break
programs that parse tool output.
2019-06-16 17:36:30 +02:00
Eelco Dolstra aec545c20b Fix segfault in builtin fetchurl with hashed mirrors + SRI hashes 2019-06-01 15:27:43 +02:00
Eelco Dolstra 5f6840fbb4
Merge pull request #2802 from LnL7/fix-needs-hashrewrite
build: move needsHashRewrite initialization to startBuilder
2019-05-13 08:53:48 +02:00
Daiderd Jordan ce02fc74b2
build: make needsHashRewrite a method 2019-05-12 22:56:27 +02:00
Graham Christensen 73b797c207
handleDiffHook: stop passing allowVfork 2019-05-12 13:44:22 -04:00
Graham Christensen b4a05edbfe
runProgram: support gid, uid, chdir 2019-05-12 13:17:27 -04:00
Graham Christensen dde8eeb39a
chdir, setgroups 2019-05-12 13:17:26 -04:00
Graham Christensen 6df61db060
diff hook: execute as the build user, and pass the temp dir 2019-05-12 13:17:26 -04:00