Commit graph

4672 commits

Author SHA1 Message Date
Eelco Dolstra 8918bae098 Drop remaining uses of external "tar"
Also, fetchGit now runs in O(1) memory since we pipe the output of
'git archive' directly into unpackTarball() (rather than first reading
it all into memory).
2019-11-26 22:07:28 +01:00
Eelco Dolstra d33dd6e6c0 Move code around 2019-11-26 22:07:28 +01:00
Eelco Dolstra b7fba16613 Move code around 2019-11-26 22:07:28 +01:00
Eelco Dolstra f738cd4d97 More Rust FFI adventures
We can now convert Rust Errors to C++ exceptions. At the Rust->C++ FFI
boundary, Result<T, Error> will cause Error to be converted to and
thrown as a C++ exception.
2019-11-26 22:07:28 +01:00
Eelco Dolstra 8110b4ebb2 Rust cleanup 2019-11-26 22:07:28 +01:00
Eelco Dolstra 045708db43 Make <nix/unpack-channel.nix> a builtin builder
This was the last function using a shell script, so this allows us to
get rid of tar, coreutils, bash etc.
2019-11-26 22:07:28 +01:00
Eelco Dolstra e60f6bd4ce Enable Rust code to call C++ Source objects 2019-11-26 22:07:28 +01:00
Eelco Dolstra 11da5b2816 Add some Rust code 2019-11-26 22:07:28 +01:00
Eelco Dolstra abb8ef619b
Fix macOS build
https://hydra.nixos.org/build/107457009
2019-11-26 21:08:56 +01:00
Eelco Dolstra 313106d549
Fix clang warnings 2019-11-26 21:07:44 +01:00
Eelco Dolstra 425991883a
Merge pull request #3141 from xbreak/nocafile
Downloader: Log configured CA file
2019-11-26 20:52:25 +01:00
Eelco Dolstra 7c8d7c17f8
Merge pull request #3144 from matthewbauer/fix-sandbox-fallback
Fix sandbox fallback settings
2019-11-26 20:51:52 +01:00
Eelco Dolstra ec5e7b44ff
Simplify 2019-11-26 20:26:22 +01:00
Eelco Dolstra 96e1c39bb7
Merge branch 'repair-bad-links' of https://github.com/chkno/nix 2019-11-26 20:21:48 +01:00
Eelco Dolstra 872740cf60
Merge pull request #3238 from puckipedia/attrset-overrides-dynamic
Ensure enough space in attrset bindings
2019-11-26 20:14:55 +01:00
Eelco Dolstra c13193017f
Disallow empty store path names
Fixes #3239.
2019-11-26 20:12:15 +01:00
Eelco Dolstra 89db9353d7
Doh 2019-11-26 20:08:25 +01:00
Eelco Dolstra 1ec6e6e11e
Add feature to disable URL literals
E.g.

  $ nix-build '<nixpkgs>' -A hello --experimental-features no-url-literals
  error: URL literals are disabled, at /nix/store/vsjamkzh15r3c779q2711az826hqgvzr-nixpkgs-20.03pre194957.bef773ed53f/nixpkgs/pkgs/top-level/all-packages.nix:1236:11

Helps with implementing https://github.com/NixOS/rfcs/pull/45.
2019-11-26 19:48:34 +01:00
puck cd55f91ad2 Ensure enough space in attrset bindings when using both __overrides and dynamic attributes 2019-11-25 12:37:14 +00:00
Eelco Dolstra d12d69ea1a
Turn NIX_PATH into a config setting
This allows it to be set in nix.conf.
2019-11-22 23:07:35 +01:00
Eelco Dolstra ec9dd9a5ae
Provide a default value for NIX_PATH 2019-11-22 22:08:51 +01:00
Eelco Dolstra ba87b08f85
getEnv(): Return std::optional
This allows distinguishing between an empty value and no value.
2019-11-22 16:18:13 +01:00
Chuck 3e2c77d001 Check for and repair bad .links entries
A corrupt entry in .links prevents adding a fixed version of that file
to the store in any path.  The user experience is that corruption
present in the store 'spreads' to new paths added to the store:

(With store optimisation enabled)

1. A file in the store gets corrupted somehow (eg: filesystem bug).
2. The user tries to add a thing to the store which contains a good copy
   of the corrupted file.
3. The file being added to the store is hashed, found to match the bad
   .links entry, and is replaced by a link to the bad .links entry.
   (The .links entry's hash is not verified during add -- this would
   impose a substantial performance burden.)
4. The user observes that the thing in the store that is supposed to be
   a copy of what they were trying to add is not a correct copy -- some
   files have different contents!  Running "nix-store --verify
   --check-contents --repair" does not fix the problem.

This change makes "nix-store --verify --check-contents --repair" fix
this problem.  Bad .links entries are simply removed, allowing future
attempts to insert a good copy of the file to succeed.
2019-11-15 11:55:36 -08:00
Eelco Dolstra 0352c1a4f8
Typo 2019-11-13 17:18:17 +01:00
Eelco Dolstra 5ee23c35b9
Merge pull request #3219 from Ericson2314/semicolons
Fix extra semicolons warnings
2019-11-11 12:13:51 +01:00
John Ericson 8669db1dcc Clean up semicolon and comma
Thanks @bhipple for catching!
2019-11-10 16:21:59 -05:00
John Ericson 4c34054673 Remove unneeded semicolons 2019-11-10 11:24:47 -05:00
John Ericson 96e6e680c1 Fix extra ; warnings involving MakeError 2019-11-10 11:24:47 -05:00
Peter Kolloch 2ba9f22715
De-duplicate struct PrimOp forward declaration 2019-11-10 10:02:22 +01:00
Eelco Dolstra 0d6774468c
Move editorFor srom libutil to nix
libutil should not depend on libexpr.
2019-11-08 15:13:32 +01:00
Eelco Dolstra 4145cd2da0
Use upstream nlohmann_json 2019-11-07 18:23:17 +01:00
Eelco Dolstra 04bf9acd22
Remove #include 2019-11-07 10:12:35 +01:00
Eelco Dolstra 39a2e166dd
Cleanup 2019-11-06 16:53:02 +01:00
Eelco Dolstra 69326f3637
Recursive Nix: Handle concurrent client connections 2019-11-06 00:55:03 +01:00
Eelco Dolstra c119ab9db0
Enable recursive Nix using a feature
Derivations that want to use recursion should now set

  requiredSystemFeatures = [ "recursive-nix" ];

to make the daemon socket appear.

Also, Nix should be configured with "experimental-features =
recursive-nix".
2019-11-06 00:55:03 +01:00
Eelco Dolstra c921074c19
RestrictedStore: Implement addToStore() 2019-11-06 00:55:03 +01:00
Eelco Dolstra c4d7c76b64
Recursive Nix support
This allows Nix builders to call Nix to build derivations, with some
limitations.

Example:

  let nixpkgs = fetchTarball channel:nixos-18.03; in

  with import <nixpkgs> {};

  runCommand "foo"
    {
      buildInputs = [ nix jq ];
      NIX_PATH = "nixpkgs=${nixpkgs}";
    }
    ''
      hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })')

      $hello/bin/hello

      mkdir -p $out/bin
      ln -s $hello/bin/hello $out/bin/hello

      nix path-info -r --json $hello | jq .
    ''

This derivation makes a recursive Nix call to build GNU Hello and
symlinks it from its $out, i.e.

  # ll ./result/bin/
  lrwxrwxrwx 1 root root 63 Jan  1  1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello

  # nix-store -qR ./result
  /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131
  /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5
  /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo

This is implemented as follows:

* Before running the outer builder, Nix creates a Unix domain socket
  '.nix-socket' in the builder's temporary directory and sets
  $NIX_REMOTE to point to it. It starts a thread to process
  connections to this socket. (Thus you don't need to have nix-daemon
  running.)

* The daemon thread uses a wrapper store (RestrictedStore) to keep
  track of paths added through recursive Nix calls, to implement some
  restrictions (see below), and to do some censorship (e.g. for
  purity, queryPathInfo() won't return impure information such as
  signatures and timestamps).

* After the build finishes, the output paths are scanned for
  references to the paths added through recursive Nix calls (in
  addition to the inputs closure). Thus, in the example above, $out
  has a reference to $hello.

The main restriction on recursive Nix calls is that they cannot do
arbitrary substitutions. For example, doing

  nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10

is forbidden unless /nix/store/kmwd... is in the inputs closure or
previously built by a recursive Nix call. This is to prevent
irreproducible derivations that have hidden dependencies on
substituters or the current store contents. Building a derivation is
fine, however, and Nix will use substitutes if available. In other
words, the builder has to present proof that it knows how to build a
desired store path from scratch by constructing a derivation graph for
that path.

Probably we should also disallow instantiating/building fixed-output
derivations (specifically, those that access the network, but
currently we have no way to mark fixed-output derivations that don't
access the network). Otherwise sandboxed derivations can bypass
sandbox restrictions and access the network.

When sandboxing is enabled, we make paths appear in the sandbox of the
builder by entering the mount namespace of the builder and
bind-mounting each path. This is tricky because we do a pivot_root()
in the builder to change the root directory of its mount namespace,
and thus the host /nix/store is not visible in the mount namespace of
the builder. To get around this, just before doing pivot_root(), we
branch a second mount namespace that shares its /nix/store mountpoint
with the parent.

Recursive Nix currently doesn't work on macOS in sandboxed mode
(because we can't change the sandbox policy of a running build) and in
non-root mode (because setns() barfs).
2019-11-06 00:52:38 +01:00
Eelco Dolstra d823381c0a
Merge branch 'fix/nix-doctor-output' of https://github.com/bhipple/nix 2019-11-05 16:04:40 +01:00
Eelco Dolstra 852554bb16
Merge branch 'nix-repl-e' of https://github.com/zimbatm/nix 2019-11-05 11:20:53 +01:00
Eelco Dolstra 1b600ecd14
Don't use SOCK_CLOEXEC on macOS
https://hydra.nixos.org/build/105428308
2019-11-05 10:25:09 +01:00
Eelco Dolstra f5a46ef0b1
Merge pull request #3202 from kraem/master
Update nix eval --help msg to not include deprecated command
2019-11-04 09:34:30 +01:00
Harald van Dijk c935ad3f02
Fix progress bar when nix-prefetch-url is piped.
The intent of the code was that if the window size cannot be determined,
it would be treated as having the maximum possible size. Because of a
missing assignment, it was actually treated as having a width of 0.

The reason the width could not be determined was because it was obtained
from stdout, not stderr, even though the printing was done to stderr.

This commit addresses both issues.
2019-11-03 21:46:59 +00:00
kraem dcd7a26063
Update nix eval --help msg to not include deprecated command 2019-11-03 18:47:28 +01:00
ng0 b811bd2172 include netinet/in.h in src/nix/main.cc
Fixes #3186
2019-11-01 14:09:42 +00:00
Eelco Dolstra 6bff1aa46d
Merge pull request #3182 from bhipple/fixup/comments
Minor updates to inline comments
2019-10-31 14:14:35 +01:00
Benjamin Hipple 80d5ec6ff4 Minor updates to inline comments
Add missing docstring on InstallableCommand. Also, some of these were wrapped
when they're right next to a line longer than the unwrapped line, so we can just
unwrap them to save vertical space.
2019-10-31 05:56:37 -04:00
Will Dietz 0e9b72e097 minor: fix mismatch of struct/class forward decl of 'Source'
Fixes the following warning and the indicate potential issue:

src/libstore/worker-protocol.hh:66:1: warning: class 'Source' was previously declared as a struct; this is valid, but may result in linker errors
under the Microsoft C++ ABI [-Wmismatched-tags]

(cherry picked from commit 6e1bb04870b1b723282d32182af286646f13bf3c)
2019-10-30 14:39:01 -05:00
Eelco Dolstra e5319a87ce
queryPathInfoUncached(): Return const ValidPathInfo 2019-10-29 13:53:04 +01:00
Eelco Dolstra 992a2ad475
Move addToStoreFromDump to Store 2019-10-29 13:38:24 +01:00
Eelco Dolstra 05819d013f
Don't create a Store in processConnection() 2019-10-29 13:36:19 +01:00
Eelco Dolstra 63b99af85a
Move Unix domain socket creation to libutil
Also drop multithread-unfriendly hacks like doing a temporary
chmod/umask.
2019-10-29 13:30:51 +01:00
Eelco Dolstra 2d37e88319
Move most of the daemon implementation to libstore 2019-10-29 13:25:33 +01:00
Eelco Dolstra 95c727caef
Remove the check against concurrent builds in the same process 2019-10-29 12:43:20 +01:00
Jonas Chevalier 9a25059656
findDerivationFilename: add FIXME 2019-10-28 21:40:02 +01:00
Jonas Chevalier d407f4d15f
nix repl: also handle lambda edit 2019-10-28 21:37:22 +01:00
Jonas Chevalier 3774fe55fd
editorFor: take a pos object instead 2019-10-28 21:36:34 +01:00
Jonas Chevalier ec448f8bb6
libexpr: findDerivationFilename return Pos instead of tuple 2019-10-28 21:29:54 +01:00
Eelco Dolstra f7ce80f90a
Factor out linkOrCopy() 2019-10-27 18:19:13 +01:00
Eelco Dolstra f1c0b2c0e1
Add O(1)-memory copyPath() function 2019-10-27 18:18:58 +01:00
Eelco Dolstra 3913afdd69
Simplification 2019-10-27 18:00:09 +01:00
Eelco Dolstra 0e459d79a6
Merge branch 'issue-3147-inNixShell-arg' of https://github.com/hercules-ci/nix 2019-10-27 17:10:19 +01:00
Robert Hensing 9d612c393a Add inNixShell = true to nix-shell auto-call
This is an alternative to the IN_NIX_SHELL environment variable,
allowing the expression to adapt itself to nix-shell without
triggering those adaptations when used as a dependency of another
shell.

Closes #3147
2019-10-27 13:16:02 +01:00
Eelco Dolstra e012384fe9
Merge branch 'tojson-tostring-fix' of https://github.com/mayflower/nix 2019-10-27 12:18:35 +01:00
Robin Gloster e583df5280
builtins.toJSON: fix __toString usage 2019-10-27 10:15:51 +01:00
Jonas Chevalier 59c7249769
libexpr: add findDerivationFilename
extract the derivation to filename:lineno heuristic
2019-10-23 17:21:16 +02:00
Jonas Chevalier 207a537343
libutil: add editorFor heuristic 2019-10-23 16:48:28 +02:00
Jonas Chevalier 73ff84f6a8
nix repl: add :edit command
This allows to have a repl-centric workflow to working on nixpkgs.

Usage:

    :edit <package> - heuristic that find the package file path

    :edit <path> - just open the editor on the file path

Once invoked, `nix repl` will open $EDITOR on that file path. Once the
editor exits, `nix repl` will automatically reload itself.
2019-10-23 16:09:42 +02:00
Eelco Dolstra 629b9b0049 Mark content-addressable paths with references as experimental 2019-10-21 18:05:31 +02:00
Eelco Dolstra e68736936a nix make-content-addressable: Add examples 2019-10-21 17:58:17 +02:00
Eelco Dolstra d77970fde7 Fix build 2019-10-21 17:49:16 +02:00
Eelco Dolstra 0abb3ad537 Allow content-addressable paths to have references
This adds a command 'nix make-content-addressable' that rewrites the
specified store paths into content-addressable paths. The advantage of
such paths is that 1) they can be imported without signatures; 2) they
can enable deduplication in cases where derivation changes do not
cause output changes (apart from store path hashes).

For example,

  $ nix make-content-addressable -r nixpkgs.cowsay
  rewrote '/nix/store/g1g31ah55xdia1jdqabv1imf6mcw0nb1-glibc-2.25-49' to '/nix/store/48jfj7bg78a8n4f2nhg269rgw1936vj4-glibc-2.25-49'
  ...
  rewrote '/nix/store/qbi6rzpk0bxjw8lw6azn2mc7ynnn455q-cowsay-3.03+dfsg1-16' to '/nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16'

We can then copy the resulting closure to another store without
signatures:

  $ nix copy --trusted-public-keys '' ---to ~/my-nix /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16

In order to support self-references in content-addressable paths,
these paths are hashed "modulo" self-references, meaning that
self-references are zeroed out during hashing. Somewhat annoyingly,
this means that the NAR hash stored in the Nix database is no longer
necessarily equal to the output of "nix hash-path"; for
content-addressable paths, you need to pass the --modulo flag:

  $ nix path-info --json /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16  | jq -r .[].narHash
  sha256:0ri611gdilz2c9rsibqhsipbfs9vwcqvs811a52i2bnkhv7w9mgw

  $ nix hash-path --type sha256 --base32 /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16
  1ggznh07khq0hz6id09pqws3a8q9pn03ya3c03nwck1kwq8rclzs

  $ nix hash-path --type sha256 --base32 /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16 --modulo iq6g2x4q62xp7y7493bibx0qn5w7xz67
  0ri611gdilz2c9rsibqhsipbfs9vwcqvs811a52i2bnkhv7w9mgw
2019-10-21 17:47:24 +02:00
Eelco Dolstra aabf5c86c9
Add experimental-features setting
Experimental features are now opt-in. There is currently one
experimental feature: "nix-command" (which enables the "nix"
command. This will allow us to merge experimental features more
quickly, without committing to supporting them indefinitely.

Typical usage:

$ nix build --experimental-features 'nix-command flakes' nixpkgs#hello

(cherry picked from commit 8e478c2341,
without the "flakes" feature)
2019-10-21 13:34:44 +02:00
Eelco Dolstra 389a2cebed
SourceExprCommand::getSourceExpr(): Allocate more space
Fixes #3140.
2019-10-21 13:14:39 +02:00
xbreak 7c568d4c6e Downloader: Warn if no trusted CA file has been configured 2019-10-18 19:08:33 +00:00
Matthew Bauer 96c84937c4 Move tmpDirInSandbox to initTmpDir 2019-10-13 16:41:49 -04:00
Matthew Bauer 499b038875 Fix sandbox fallback settings
The tmpDirInSandbox is different when in sandboxed vs. non-sandboxed.
Since we don’t know ahead of time here whether sandboxing is enabled,
we need to reset all of the env vars we’ve set previously. This fixes
the issue encountered in https://github.com/NixOS/nixpkgs/issues/70856.
2019-10-12 19:22:13 -04:00
Eelco Dolstra 906d56a96b
ssh-ng: Don't set CPU affinity on the remote
Fixes #3138.
2019-10-11 18:49:46 +02:00
Eelco Dolstra 95cf23ee7c
nix verify: Fix uninitialized variable 2019-10-10 15:03:01 +02:00
Eelco Dolstra c3aaf3b8da
nix-env: Ignore failures creating ~/.nix-profile and ~/.nix-defexpr
https://hydra.nixos.org/build/102803093
2019-10-10 09:14:05 +02:00
Eelco Dolstra bda64a2b0f
Doh
https://hydra.nixos.org/build/102803044
2019-10-10 00:12:30 +02:00
Eelco Dolstra 20eec802ff
Force per-user group to a known value 2019-10-09 23:35:02 +02:00
Eelco Dolstra d7bae5680f
Go back to 755 permission on per-user directories
700 is pointless since the store is world-readable anyway. And
per-user/root/channels must be world-readable.
2019-10-09 23:35:02 +02:00
Eelco Dolstra c9159f86cc
nix-env: Create ~/.nix-defexpr automatically 2019-10-09 23:35:02 +02:00
Eelco Dolstra 9348f9291e
nix-env: Create ~/.nix-profile automatically 2019-10-09 23:35:01 +02:00
Eelco Dolstra 5a303093dc
Remove world-writability from per-user directories
'nix-daemon' now creates subdirectories for users when they first
connect.

Fixes #509 (CVE-2019-17365).
Should also fix #3127.
2019-10-09 23:34:48 +02:00
Eelco Dolstra 4331eeb13d
Filter ANSI escape sequences in -L output
Otherwise, builds like NixOS VM tests may leave the terminal in a
weird state and do resets.
2019-10-09 23:25:06 +02:00
Eelco Dolstra 55bba8e4f5
Make std::uncaught_exception warning less noisy 2019-10-09 23:04:11 +02:00
Eelco Dolstra 926d3e5bb0
Fix Bison 2.4 warning 2019-10-09 22:57:37 +02:00
Eelco Dolstra 99b73fb507
OCD performance fix: {find,count}+insert => insert 2019-10-09 16:06:29 +02:00
Eelco Dolstra e6e61f0a54
getSourceExpr(): Handle channels
Fixes #1892.
Fixes #1865.
Fixes #3119.
2019-10-09 15:36:51 +02:00
Eelco Dolstra 08ad9714e1
Merge pull request #3132 from matthewbauer/handle-sandbox-shell
Handle empty sandbox_shell
2019-10-09 14:52:51 +02:00
Eelco Dolstra 7c74f075f4
nix search: Don't quietly ignore errors 2019-10-09 14:46:58 +02:00
Eelco Dolstra 64d8872900
nix-build: Fix compilation 2019-10-09 14:46:44 +02:00
Matthew Bauer 199e888785 Handle empty sandbox_shell
Previously, SANDBOX_SHELL was set to empty when unavailable. This
caused issues when actually generating the sandbox. Instead, just set
SANDBOX_SHELL when --with-sandbox-shell= is non-empty. Alternative
implementation to https://github.com/NixOS/nix/pull/3038.
2019-10-08 23:12:54 -04:00
Matthew Bauer 65f6d5db6f Don’t source bashrc in pure mode
Pure mode should not try to source the user’s bashrc file. These may
have many impurities that the user does not expect to get into their
shell.

Fixes #3090
2019-10-08 22:41:59 -04:00
Sam Doshi 6f6cb5e388 nix search: remove verbose example 2019-10-07 11:40:42 +01:00
Benjamin Hipple c5bd564c69 nix doctor: add more logging output to checks
When running nix doctor on a healthy system, it just prints the store URI and
nothing else. This makes it unclear whether the system is in a good state and
what check(s) it actually ran, since some of the checks are optional depending
on the store type.

This commit updates nix doctor to print an colored log message for every check
that it does, and explicitly state whether that check was a PASS or FAIL to make
it clear to the user whether the system passed its checkup with the doctor.

Fixes #3084
2019-10-06 16:57:57 -04:00
Eelco Dolstra 93b1ce1ac5
Revert "std::uncaught_exception() -> std::uncaught_exceptions()"
This reverts commit 6b83174fff because
it doesn't work on macOS yet.

https://hydra.nixos.org/build/102617587
2019-10-04 16:34:59 +02:00
Eelco Dolstra 15e70c662e
Fix indentation 2019-10-02 16:26:15 +02:00
Eelco Dolstra 168a887916
Fix fetchTarball with chroot stores
Fixes #2405.
2019-10-01 07:51:06 +00:00
Domen Kožar 2d2769f68c
Merge pull request #2338 from bobvanderlinden/pr-cannot-delete-alive-why
mention `nix-store --query --roots` when a path cannot be deleted
2019-09-30 14:06:52 +02:00
Eelco Dolstra 5038e1bec4
Merge pull request #3103 from bhipple/fix/spelling
Fix spelling in comment
2019-09-23 10:46:40 +02:00
Eelco Dolstra 02b4632e77
Merge pull request #3104 from zimbatm/no-show-trace-forwarding
libstore: don't forward --show-trace
2019-09-23 10:44:42 +02:00
Eelco Dolstra 3a022d4599 Shut up some warnings
(cherry picked from commit 99e8e58f2d)
2019-09-22 21:57:05 +02:00
Eelco Dolstra bd79c1f6f6 Don't catch exceptions by value
(cherry picked from commit 893be6f5e3)
2019-09-22 21:56:56 +02:00
Jonas Chevalier e63c9e73e3
libstore: don't forward --show-trace 2019-09-22 10:57:20 +00:00
Benjamin Hipple c6a542f22a Fix spelling in comment 2019-09-21 18:53:15 -04:00
Jonas Chevalier 619cc4af85
function-trace: always show the trace
If the user invokes nix with --trace-function-calls it means that they
want to see the trace.
2019-09-18 23:23:21 +02:00
Eelco Dolstra 6b83174fff std::uncaught_exception() -> std::uncaught_exceptions()
The former is deprecated in C++17. Fixes a clang warning.
2019-09-13 20:05:44 +02:00
Eelco Dolstra a56b51a0ba
Disable OpenSSL lock callback on OpenSSL >= 1.1.1 2019-09-04 21:45:01 +02:00
Eelco Dolstra 5dafde28db
BinaryCacheStore: Add index-debug-info option
This integrates the functionality of the index-debuginfo program in
nixos-channel-scripts to maintain an index of DWARF debuginfo files in
a format usable by dwarffs. Thus the debug info index is updated by
Hydra rather than by the channel mirroring script.

Example usage:

  $ nix copy --to 'file:///tmp/binary-cache?index-debug-info=true' /nix/store/vr9mhcch3fljzzkjld3kvkggvpq38cva-nix-2.2.2-debug

  $ cat /tmp/binary-cache/debuginfo/036b210b03bad75ab2d8fc80b7a146f98e7f1ecf.debug
  {"archive":"../nar/0313h2kdhk4v73xna9ysiksp2v8xrsk5xsw79mmwr3rg7byb4ka8.nar.xz","member":"lib/debug/.build-id/03/6b210b03bad75ab2d8fc80b7a146f98e7f1ecf.debug"}

Fixes #3083.
2019-09-04 19:28:26 +02:00
Eelco Dolstra e07ec8d27e
Support allowSubstitutes attribute in structured attribute derivations
Hopefully fixes #3081 (didn't test).
2019-09-03 16:03:49 +02:00
Eelco Dolstra f186000367
Add some noexcepts
This is to assert that callback functions should never throw (since
the context in which they're called may not be able to handle the
exception).
2019-09-03 13:45:35 +02:00
Eelco Dolstra 7348653ff4
Ensure that Callback is called only once
Also, make Callback movable but uncopyable.
2019-09-03 13:45:35 +02:00
Eelco Dolstra 8c4ea7a451
Downloader: Remove a possible double call to Callback 2019-09-03 13:45:32 +02:00
Eelco Dolstra 84de821004
Merge pull request #3069 from matthewbauer/max-name
Set maximum name length in Nix
2019-08-29 15:22:36 +02:00
Eelco Dolstra a2c4fcd5e9 Don't rely on st_blocks
It doesn't seem very reliable on ZFS.
2019-08-29 14:49:58 +02:00
Eelco Dolstra f27e53f77e Cleanup 2019-08-29 12:09:58 +02:00
Eelco Dolstra 7ef2645f45
Merge pull request #2921 from matthewbauer/handle-sigwinch
Handle SIGWINCH in main thread
2019-08-28 21:48:14 +02:00
Matthew Bauer 693e68e09c Set maximum name length in Nix
Previously we allowed any length of name for Nix derivations. This is
bad because different file systems have different max lengths. To make
things predictable, I have picked a max. This was done by trying to
build this derivation:

  derivation {
    name = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
    builder = "/no-such-path";
    system = "x86_64-linux";
  }

Take off one a and it will not lead to file name too long. That ends
up being 212 a’s. An even smaller max could be picked if we want to
support more file systems.

Working backwards, this is why:

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-${name}.drv.chroot

> 255 - 32 - 1 - 4 - 7 = 211
2019-08-28 12:32:54 -04:00
Eelco Dolstra 7298a38a07
Don't send certain setting overrides to the daemon
These are already handled separately. This fixes warnings like

  warning: ignoring the user-specified setting 'max-jobs', because it is a restricted setting and you are not a trusted user

when using the -j flag.
2019-08-28 16:29:44 +02:00
Eelco Dolstra e5b397b2c7 Merge branch 'test-sandboxing' of https://github.com/matthewbauer/nix 2019-08-27 20:58:47 +02:00
Matthew Bauer 5c06a8d328 Reset tmpDirInSandbox for unsandboxed 2019-08-23 20:24:39 -04:00
Jonas Chevalier b226b5cd97
nix-store: fix out of sync protocol
If a NAR is already in the store, addToStore doesn't read the source
which makes the protocol go out of sync. This happens for example when
two client try to nix-copy-closure the same derivation at the same time.
2019-08-16 15:05:45 +02:00
Jonas Chevalier 91b00b145f
libutil: add SizedSource
Introduce the SizeSource which allows to bound how much data is being
read from a source. It also contains a drainAll() function to discard
the rest of the source, useful to keep the nix protocol in sync.
2019-08-16 15:05:40 +02:00
Eelco Dolstra 477f82e5a7
Merge pull request #2782 from grahamc/flames
Track function start and end
2019-08-15 14:20:42 +02:00
Graham Christensen ee9c988a1b
Track function start and ends for flame graphs
With this patch, and this file I called `log.py`:

    #!/usr/bin/env nix-shell
    #!nix-shell -i python3 -p python3 --pure

    import sys
    from pprint import pprint

    stack = []
    timestack = []

    for line in open(sys.argv[1]):
        components = line.strip().split(" ", 2)
        if components[0] != "function-trace":
            continue

        direction = components[1]
        components = components[2].rsplit(" ", 2)

        loc = components[0]
        _at = components[1]
        time = int(components[2])

        if direction == "entered":
            stack.append(loc)
            timestack.append(time)
        elif direction == "exited":
            dur = time - timestack.pop()
            vst = ";".join(stack)
            print(f"{vst} {dur}")
            stack.pop()

and:

    nix-instantiate --trace-function-calls -vvvv ../nixpkgs/pkgs/top-level/release.nix -A unstable > log.matthewbauer 2>&1
    ./log.py ./log.matthewbauer > log.matthewbauer.folded
    flamegraph.pl --title matthewbauer-post-pr log.matthewbauer.folded > log.matthewbauer.folded.svg

I can make flame graphs like: http://gsc.io/log.matthewbauer.folded.svg

---

Includes test cases around function call failures and tryEval. Uses
RAII so the finish is always called at the end of the function.
2019-08-14 16:09:35 -04:00
Eelco Dolstra 35ebae198f
Merge pull request #3031 from grahamc/low-speed-limit
conf: stalled-download-timeout: make tunable
2019-08-08 22:06:26 +02:00
Graham Christensen a02457db71
conf: stalled-download-timeout: make tunable
Make curl's low speed limit configurable via stalled-download-timeout.
Before, this limit was five minutes without receiving a single byte.
This is much too long as if the remote end may not have even
acknowledged the HTTP request.
2019-08-08 10:22:13 -04:00
Eelco Dolstra f9021c4c6c
Merge pull request #3030 from dtzWill/fix/missing-include-ocloexec
pathlocks: add include to fcntl.h for O_CLOEXEC
2019-08-07 22:03:09 +02:00
Eelco Dolstra 56df30cd3f
Merge pull request #2995 from tweag/post-build-hook
Add a post build hook
2019-08-07 15:02:29 +02:00
Will Dietz c3fefd1a6e
pathlocks: add include to fcntl.h for O_CLOEXEC 2019-08-07 07:41:22 -05:00
Eelco Dolstra 399b6f3c46
nix-store --verify: Don't repair while holding the GC lock 2019-08-02 18:48:26 +02:00
Eelco Dolstra a2597d5f27
Simplify
With BSD locks we don't have to guard against reading our own
temproots.
2019-08-02 18:39:16 +02:00
Eelco Dolstra e349f2c0a3
Use BSD instead of POSIX file locks
POSIX file locks are essentially incompatible with multithreading. BSD
locks have much saner semantics. We need this now that there can be
multiple concurrent LocalStore::buildPaths() invocations.
2019-08-02 18:39:16 +02:00
Eelco Dolstra ec415d7166
Add a test for auto-GC
This currently fails because we're using POSIX file locks. So when the
garbage collector opens and closes its own temproots file, it causes
the lock to be released and then deleted by another GC instance.
2019-08-02 18:39:16 +02:00
regnat 7c5596734f
Add a post-build-hook
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:

    DRV_PATH=/nix/store/drv-that-has-been-built.drv
    OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev

This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).

This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:

    [nix]$ ./inst/bin/nix-build ./test.nix
    these derivations will be built:
      /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
    building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
    hello!
    bye!
    running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + printf 'very important stuff'
    /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation

    [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
    my-example-derivation> hello!
    my-example-derivation> bye!
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + printf 'very important stuff'
    [1 built, 0.0 MiB DL]

Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-08-02 10:48:15 -04:00
Matthew Bauer 9a0855bbb6 Don’t rely on EPERM
startProcess does not appear to send the exit code to the helper
correctly. Not sure why this is, but it is probably safe to just
fallback on all sandbox errors.
2019-07-30 17:53:37 -04:00
Eelco Dolstra 41d010fff6
Merge pull request #3009 from codedownio/add-pname-and-version-to-json
Add pname and version to nix-env -q --json
2019-07-30 11:43:45 +02:00
Eelco Dolstra 219d645987
Merge pull request #3013 from basvandijk/disable-lsof-for-darwin-tests
Disable findRuntimeRoots on darwin when running tests because lsof is slow
2019-07-30 11:34:18 +02:00
Bas van Dijk ee1e3132ca Disable findRuntimeRoots on darwin when running tests because lsof is slow
See: https://github.com/NixOS/nix/issues/3011
2019-07-30 11:29:03 +02:00
Bas van Dijk 89865144c3 Allow builtins.pathExists to check the existence of /nix/store paths
This makes it consitent with builtins.readDir.
2019-07-30 11:27:35 +02:00
Tom McLaughlin cd933b22d2 Add pname and version to nix-env -q --json 2019-07-27 19:40:51 -07:00
Matthew Bauer 11d8534629 Use sandbox fallback when cloning fails in builder
When sandbox-fallback = true (the default), the Nix builder will fall
back to disabled sandbox mode when the kernel doesn’t allow users to
set it up. This prevents hard errors from occuring in tricky places,
especially the initial installer. To restore the previous behavior,
users can set:

  sandbox-fallback = false

in their /etc/nix/nix.conf configuration.
2019-07-25 14:42:30 -04:00
Matthew Bauer d171090530 Disable CLONE_NEWUSER when it’s unavailable
Some kernels disable "unpriveleged user namespaces". This is
unfortunate, but we can still use mount namespaces. Anyway, since each
builder has its own nixbld user, we already have most of the benefits
of user namespaces.
2019-07-25 14:42:25 -04:00
Eelco Dolstra 2f853b20df
Merge pull request #2975 from matthewbauer/fix-nsswitch-issue
Don’t use entire /etc/nsswitch.conf file
2019-07-13 17:08:02 +02:00
Eelco Dolstra 53247d6b11
Resume NAR downloads
This is a much simpler fix to the 'error 9 while decompressing xz
file' problem than 78fa47a7f0. We just
do a ranged HTTP request starting after the data that we previously
wrote into the sink.

Fixes #2952, #379.
2019-07-10 23:12:17 +02:00
Eelco Dolstra 00f6fafad6
HttpBinaryCacheStore: Use default number of retries for NARs 2019-07-10 23:05:04 +02:00
Eelco Dolstra f76b2a7fdd
Downloader: Use warn() 2019-07-10 22:27:50 +02:00
Eelco Dolstra 03f09e1d18
Revert "Fix 'error 9 while decompressing xz file'"
This reverts commit 78fa47a7f0.
2019-07-10 19:46:15 +02:00
Eelco Dolstra aa739e7839
nix copy: Rename --substitute to --substitute-on-destination
'--substitute' was being shadowed by the regular '--substitute' (the
short-hand for '--option substitute true').

Fixes #2983.
2019-07-10 11:28:37 +02:00
Eelco Dolstra 7d6ba1dc90
Merge branch 'autoconf-ubuntu-16.04-fixes' of https://github.com/nh2/nix 2019-07-03 08:02:45 +02:00
Niklas Hambüchen a96006d97f Get BOOST_LDFLAGS from autoconf, fix Ubuntu 16.04 build.
Our use of boost::coroutine2 depends on -lboost_context,
which in turn depends on `-lboost_thread`, which in turn depends
on `-lboost_system`.

I suspect that this builds on nix only because of low-level hacks
like NIX_LDFLAGS.

This commit passes the proper linker flags, thus fixing bootstrap
builds on non-nix distributions like Ubuntu 16.04.

With these changes, I can build Nix on Ubuntu 16.04 using:

    ./bootstrap.sh
    ./configure --prefix=$HOME/editline-prefix \
      --disable-doc-gen \
      CXX=g++-7 \
      --with-boost=$HOME/boost-prefix \
      EDITLINE_CFLAGS=-I$HOME/editline-prefix/include \
      EDITLINE_LIBS=-leditline \
      LDFLAGS=-L$HOME/editline-prefix/lib
    make

where

* g++-7 comes from gcc-7 from
  https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test,
* editline 1.14 from https://github.com/troglobit/editline/releases/tag/1.14.0
	was installed into `$HOME/editline-prefix`
  (because Ubuntu 16.04's `editline` is too old to have the function nix uses),
* boost 1.66 from
	https://www.boost.org/doc/libs/1_66_0/more/getting_started/unix-variants.html
	was installed into $HOME/boost-prefix (because Ubuntu 16.04 only has 1.58)
2019-07-03 04:32:25 +02:00
Niklas Hambüchen d203c554fa Fix C++ compatibility with older editline versions.
For example, Ubuntu 16.04 and many similar long-term-support distros
have older versions.
2019-07-03 04:32:25 +02:00
Eelco Dolstra 7e1c85c5fb
Merge pull request #2779 from LnL7/build-exit-codes
build: add exit code for hash and check mismatches
2019-07-02 17:37:49 +02:00
Eelco Dolstra db700f730e
Merge pull request #2974 from grahamc/invalid-name
checkStoreName: give more precise/verbose error information
2019-07-02 16:12:01 +02:00
Eelco Dolstra 33db1d35ae
Merge pull request #2582 from LnL7/fetchgit-refs
fetchGit: allow fetching explicit refs
2019-07-02 15:44:31 +02:00
Graham Christensen 17d3ec3405
checkStoreName: give more precise/verbose error information
$ sudo ./inst/bin/nix-instantiate -E '"${./.git}"'
error: The path name '.git' is invalid: it is illegal to start the
name with a period. Path names are alphanumeric and can include the
symbols +-._?= and must not begin with a period. Note: If '.git' is a
source file and you cannot rename it on disk,
builtins.path { name = ... } can be used to give it an alternative
name.
2019-07-02 08:41:53 -04:00
Daiderd Jordan a52c331edb
build: replace 100 offset for build exit codes 2019-07-02 00:12:38 +02:00
Daiderd Jordan cbf84bcce7
build: use binary mask for build status flags
If multiple builds with fail with different errors it will be reflected
in the status code.

eg.

	103 => timeout + hash mismatch
	105 => timeout + check mismatch
	106 => hash mismatch + check mismatch
	107 => timeout + hash mismatch + check mismatch
2019-07-02 00:12:38 +02:00
Daiderd Jordan 97baf32fbc
build: add exit code for hash and check mismatches
Makes it easier to identify the failure reason in other tooling, eg.
differentiate between a non-deterministic --check vs a failed build.

	$ nix-build '<nix/fetchurl.nix>' --argstr url http://example.org --argstr sha256 0000000000000000000000000000000000000000000000000000
	hash mismatch in fixed-output derivation '/nix/store/nzi9ck45rwlxzcwr25is7qlf3hs5xl83-example.org':
	  wanted: sha256:0000000000000000000000000000000000000000000000000000
	  got:    sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm
	$ echo $?
	102

	$ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" {} "date +%s > $out"' --check
	warning: rewriting hashes in '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo'; cross fingers
	error: derivation '/nix/store/mggc8dz13ackb49qca6m23zq4fpq132q-foo.drv' may not be deterministic: output '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo' differs
	$ echo $?
	104
2019-07-02 00:12:34 +02:00
Aniket Deshpande ec58ba38c5
Fix http2 = false having no effect. Fixes #2971.
Setting `http2 = false` in nix config (e.g. /etc/nix/nix.conf)
had no effect, and `nix-env -vvvvv -i hello` still downloaded .nar
packages using HTTP/2.

In `src/libstore/download.cc`, the `CURL_HTTP_VERSION_2TLS` option was
being explicitly set when `downloadSettings.enableHttp2` was `true`,
but, `CURL_HTTP_VERSION_1_1` option was not being explicitly set when
`downloadSettings.enableHttp2` was `false`.

This may be because `https://curl.haxx.se/libcurl/c/libcurl-env.html` states:
"You have to set this option if you want to use libcurl's HTTP/2 support."
but, also, in the changelog, states:
"DEFAULT
Since curl 7.62.0: CURL_HTTP_VERSION_2TLS
Before that: CURL_HTTP_VERSION_1_1"

So, the default setting for `libcurl` is HTTP/2 for version >= 7.62.0.

In this commit, option `CURLOPT_HTTP_VERSION` is explicitly set to
`CURL_HTTP_VERSION_1_1` when `downloadSettings.enableHttp2` nix config
setting is `false`.

This can be tested by running `nix-env -vvvvv -i hello | grep HTTP`
2019-06-28 20:44:46 +05:30
Eelco Dolstra 6847c92788
Fix macOS build failure
Issue #2976.
2019-06-28 15:38:23 +02:00
Matthew Bauer ec0087df0a Don’t use entire /etc/nsswitch.conf file
The default nsswitch.conf(5) file in most distros can handle many
different things including host name, user names, groups, etc. In Nix,
we want to limit the amount of impurities that come from these things.
As a result, we should only allow nss to be used for gethostbyname(3)
and getservent(3).

/cc @Ericson2314
2019-06-27 14:31:29 -04:00
Eelco Dolstra 324a5dc92f
ProgressBar: Fix updating
'updateCV.notify_one()' does nothing if the update thread is not
waiting for updateCV (in particular this happens when it is sleeping
on quitCV). So also set a variable to ensure that the update isn't
lost.
2019-06-25 21:59:20 +02:00
Eelco Dolstra 88571219d9
nix-channel: Don't fetch binary-cache-url
This has been ignored since the Perl->C++ rewrite.
2019-06-25 13:27:16 +02:00
Eelco Dolstra 09dde33c19
Automatically use --no-net if there are no network interfaces
(cherry picked from commit 04a5976996)
2019-06-25 12:45:59 +02:00
Eelco Dolstra 5600b070a7
Add "warning" verbosity level
This ensures that "nix" shows warnings. Previously these were hidden
because they were at "info" level.

(cherry picked from commit 615a9d031d)
2019-06-25 12:44:17 +02:00
Eelco Dolstra 64ec087f58
Fix 32-bit overflow with --no-net
--no-net causes tarballTtl to be set to the largest 32-bit integer,
which causes comparison like 'time + tarballTtl < other_time' to
fail on 32-bit systems. So cast them to 64-bit first.

https://hydra.nixos.org/build/95076624
(cherry picked from commit 29ccb2e969)
2019-06-24 22:16:43 +02:00
Eelco Dolstra f8b30338ac
Refactor downloadCached() interface
(cherry picked from commit df3f5a78d5)
2019-06-24 22:12:26 +02:00
Eelco Dolstra 7b9c68766d
Add '--no-net' convenience flag
This flag

* Disables substituters.

* Sets the tarball-ttl to infinity (ensuring e.g. that the flake
  registry and any downloaded flakes are considered current).

* Disables retrying downloads and sets the connection timeout to the
  minimum. (So it doesn't completely disable downloads at the moment.)

(cherry picked from commit 8ea842260b)
2019-06-24 22:07:29 +02:00
Eelco Dolstra 78fa47a7f0
Fix 'error 9 while decompressing xz file'
Once we've started writing data to a Sink, we can't restart a download
request, because then we end up writing duplicate data to the
Sink. Therefore we shouldn't handle retries in Downloader but at a
higher level (in particular, in copyStorePath()).

Fixes #2952.

(cherry picked from commit a67cf5a358)
2019-06-24 21:59:51 +02:00
Eelco Dolstra 2fef4dd296
Downloader: Propagate exceptions from decompressionSink->finish()
(cherry picked from commit 15fa70cd1b)
2019-06-24 21:59:31 +02:00
Eelco Dolstra b43e1e186e
CachedDownloadResult: Include store path
Also, make fetchGit and fetchMercurial update allowedPaths properly.

(Maybe the evaluator, rather than the caller of the evaluator, should
apply toRealPath(), but that's a bigger change.)

(cherry picked from commit 5c34d66538)
2019-06-24 21:59:27 +02:00
Eelco Dolstra dc29e9fb47
downloadCached: Return ETag
(cherry picked from commit 529add316c)
2019-06-24 21:58:33 +02:00
Eelco Dolstra 94f11d0a61
Fix abort in fromTOML
Fixes #2969.
2019-06-24 17:09:09 +02:00
Eelco Dolstra 3cc1125595
Another attempt at getting pseudoterminals to work on macOS 2019-06-17 08:08:04 +02:00
Eelco Dolstra 2743bf0bb1 Hopefully fix macOS tests 2019-06-16 20:02:40 +02:00
Eelco Dolstra 82ca6ef390 Set $TERM 2019-06-16 17:36:30 +02:00
Eelco Dolstra e84c265645 Run builds in a pseudo-terminal
This allows many programs (e.g. gcc, clang, cmake) to print colorized
log output (assuming $TERM is set to a value like "xterm").

There are other ways to get colors, in particular setting
CLICOLOR_FORCE, but they're less widely supported and can break
programs that parse tool output.
2019-06-16 17:36:30 +02:00
Eelco Dolstra b693029ca0
Style fix 2019-06-16 09:43:20 +02:00
Eelco Dolstra 26bc876ae6
nix: Add -L alias for --print-build-logs 2019-06-15 16:45:00 +02:00
Eelco Dolstra 34fa8ce917
nix: Support -j flag 2019-06-15 16:34:06 +02:00
Félix Baylac-Jacqué 9e0f5f803f
Daemon: warn when an untrusted user cannot override a setting
In a daemon-based Nix setup, some options cannot be overridden by a
client unless the client's user is considered trusted.

Currently, if an untrusted user tries to override one of those
options, we are silently ignoring it.

This can be pretty confusing in certain situations.

e.g. a user thinks he disabled the sandbox when in reality he did not.

We are now sending a warning message letting know the user some options
have been ignored.

Related to #1761.
2019-06-15 03:59:12 -04:00
Matthew Bauer 5011a52cf3 Just enable hack on macOS
This is not needed on linux at all! Tried to explain as much as I
understand with the problem.
2019-06-05 20:20:50 -04:00
Matthew Bauer 4a3e96281d Handle SIGWINCH in main thread
For the SIGWINCH signal to be caught, it needs to be set in sigaction
on the main thread. Previously, this was broken, and updateWindowSize
was never being called. Tested on macOS 10.14.
2019-06-05 00:40:45 -04:00
Eelco Dolstra aec545c20b Fix segfault in builtin fetchurl with hashed mirrors + SRI hashes 2019-06-01 15:27:43 +02:00
Ding Xiang Fei abdedcdb38 bump cpptoml to v0.1.1 2019-05-29 17:01:39 +08:00
Eelco Dolstra 22f2744afd
Iterate over references 2019-05-28 23:05:08 +02:00
Eelco Dolstra 9eaebbf575
Merge branch 'attrPaths' of https://github.com/NinjaTrappeur/nix 2019-05-28 22:59:05 +02:00
Eelco Dolstra 4d829916e7
Merge pull request #2800 from flokli/progress-bar-hide-unknown-expected
progress-bar: hide expected if expected is 0 (unknown)
2019-05-21 13:29:28 +02:00
Eelco Dolstra 8f6c72faee
Merge pull request #2810 from NixOS/print-build-logs
nix: Add --print-build-logs flag
2019-05-15 20:38:38 +02:00
Eelco Dolstra 66b8a62101 nix: Add --print-build-logs flag
This causes 'nix' to print build log output to stderr rather than
showing the last log line in the progress bar. Log lines are prefixed
by the name of the derivation (minus the version string), e.g.

  binutils> make[1]: Leaving directory '/build/binutils-2.31.1'
  binutils-wrapper> unpacking sources
  binutils-wrapper> patching sources
  ...
  binutils-wrapper> Using dynamic linker: '/nix/store/kr51dlsj9v5cr4n8700jliyz8v5b2q7q-bootstrap-stage0-glibc/lib/ld-linux-x86-64.so.2'
  bootstrap-stage2-gcc-wrapper> unpacking sources
  ...
  linux-headers> unpacking sources
  linux-headers> unpacking source archive /nix/store/8javli69jhj3bkql2c35gsj5vl91p382-linux-4.19.16.tar.xz
2019-05-15 17:33:56 +02:00
Eelco Dolstra 5f6840fbb4
Merge pull request #2802 from LnL7/fix-needs-hashrewrite
build: move needsHashRewrite initialization to startBuilder
2019-05-13 08:53:48 +02:00
Graham Christensen f1b8e9efe7
runProgram: Uncomment chdir support 2019-05-12 17:03:01 -04:00
Daiderd Jordan ce02fc74b2
build: make needsHashRewrite a method 2019-05-12 22:56:27 +02:00
Graham Christensen 73b797c207
handleDiffHook: stop passing allowVfork 2019-05-12 13:44:22 -04:00
Graham Christensen b4a05edbfe
runProgram: support gid, uid, chdir 2019-05-12 13:17:27 -04:00
Graham Christensen dde8eeb39a
chdir, setgroups 2019-05-12 13:17:26 -04:00
Graham Christensen 6df61db060
diff hook: execute as the build user, and pass the temp dir 2019-05-12 13:17:26 -04:00
Graham Christensen c78686e411
build: run diff-hook under --check and document diff-hook 2019-05-12 13:17:24 -04:00
Daiderd Jordan ff6867ab94
build: move needsHashRewrite initialization to startBuilder
The value of useChroot is not set yet in the constructor, resulting in
hash rewriting being enabled in certain cases where it should not be.

Fixes #2801
2019-05-12 15:53:40 +02:00
Florian Klink 6ade7ec022 progress-bar: hide expected if expected is 0 (unknown)
Sometimes, "expected" can be "0", but in fact means "unknown".

This is for example the case when downloading a file while the http
server doesn't send the `Content-Length` header, like when running `nix
build` pointing to a nixpkgs checkout streamed from GitHub:

⇒  nix build -f https://github.com/NixOS/nixpkgs/archive/master.tar.gz hello
[1.8/0.0 MiB DL] downloading 'https://github.com/NixOS/nixpkgs/archive/master.tar.gz'

In that case, don't show that weird progress bar, but only the (slowly
increasing) downloaded size ("done").

⇒  nix build -f https://github.com/NixOS/nixpkgs/archive/master.tar.gz hello
[1.8 MiB DL] downloading 'https://github.com/NixOS/nixpkgs/archive/master.tar.gz'

This commit also updates fmt calls with three numbers (when something is
currently 'running' too) - I'm not sure if this can be provoked, but
showing "0" as expected doesn't make any sense, as we're obviously doing
more than nothing.
2019-05-11 12:34:39 +02:00
Eelco Dolstra 7c6391ddc7
nix-shell: Don't absolutize '-p' expressions
This prevents spurious syscalls like

  25011 lstat("/home/eelco/with import <nixpkgs> { }; (pkgs.runCommandCC or pkgs.runCommand) \"shell\" { buildInputs = [ (hello) ]; } \"\"", 0x7ffe9c67f580) = -1 ENOENT (No such file or directory)
2019-05-08 14:29:36 +02:00
Eelco Dolstra 92caa60c49
Merge branch 'repl/ctrlc' of https://github.com/xbreak/nix 2019-05-08 10:18:45 +02:00
Eelco Dolstra 3e940bbf2d
Merge pull request #2790 from samueldr/fix/minor-comment-NIX_ROOT_FINDER
findRootsNoTemp: fixes comment about findRuntimeRoots
2019-05-08 10:13:09 +02:00
Daniel Schaefer 3f192ac80c Add builtins.hashFile
For text files it is possible to do it like so:
`builtins.hashString "sha256" (builtins.readFile /tmp/a)`
but that doesn't work for binary files.

With builtins.hashFile any kind of file can be conveniently hashed.
2019-05-03 17:23:36 +02:00
Eelco Dolstra f9a2ea4486
Fix "Bad system call" running i686-linux binaries on x86_64-linux
To determine which seccomp filters to install, we were incorrectly
using settings.thisSystem, which doesn't denote the actual system when
--system is used.

Fixes #2791.
2019-05-03 10:48:33 +02:00
Samuel Dionne-Riel cbc7d9a412 findRootsNoTemp: fixes comment about findRuntimeRoots
The NIX_ROOT_FINDER environment variable was removed in
3c46fe62b8 when porting from perl to C.
2019-04-30 22:43:24 -04:00
Daiderd Jordan b614e0e53d
build: only skip hash rewriting for sandboxing on linux
The sandbox on darwin, and possibly other future platforms, doesn't have
enough isolation to redirect outputs without hash rewriting.
2019-04-17 23:41:23 +02:00
Eelco Dolstra bb6e6923f2 Add environment variable NIX_SHOW_SYMBOLS for dumping the symbol table 2019-04-11 23:04:13 +02:00
Eelco Dolstra 41ba5135e0 primeCache(): Barf if builds are needed but not allowed
Fixes #2753.
2019-04-01 21:09:49 +02:00
Eelco Dolstra 2bc6304793 getMachines(): Cache result 2019-04-01 21:09:35 +02:00
Eelco Dolstra f32fbf952d Fix Bison 3.3 warning 2019-03-27 21:09:31 +01:00
Eelco Dolstra 5c05c238e6
Merge pull request #1828 from zimbatm/isPath
Add isPath primop
2019-03-25 19:58:11 +01:00
Will Dietz 42e2d5e7b7
store-api.hh: add missing include for unordered_map 2019-03-25 09:18:30 +01:00
Jonas Chevalier 514b3c7f83
Add isPath primop
this is added for completeness' sake since all the other possible
`builtins.typeOf` results have a corresponding `builtins.is<Type>`
2019-03-24 11:36:49 +01:00
xbreak fcd7660976 repl: Restore CTRL-C behaviour
Install signal handler during `readline` to handle SIGINT to abort
partially typed expressions.
2019-03-24 09:39:48 +00:00
Linus Heckemann 2aa89daab3 eval: improve type description for primops and applied primops
This can make type errors a little easier to understand.
2019-03-21 15:31:46 +01:00
Chaker Benhamed 81a23fa7e2 Add a 5 seconds of timeouts to connect to S3.
The default is 1000ms, but we can hit it a lot of we don't have direct
link to AWS (e.g. using VPN).
2019-03-15 13:23:58 +01:00
Eelco Dolstra 1f64f4c7c8
pkg-config files: Use c++17 2019-03-14 14:11:12 +01:00
Eelco Dolstra ef52ccf035
experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Eelco Dolstra 86f3b94c8c
nix-store --gc --print-roots: Sort output 2019-03-14 13:59:10 +01:00
Eelco Dolstra 578ed7a259
findRoots(): Don't censor for trusted users
They're pretty much root anyway.
2019-03-14 13:53:34 +01:00
Eelco Dolstra 53522cb6ac
findRoots(): Add 'censor' parameter
This is less brittle than filtering paths after the fact in
nix-daemon.
2019-03-14 13:53:34 +01:00
Eelco Dolstra a3f37d87ea
findRuntimeRoots: Simplify/fix handling of /proc files
Scanning of /proc/<pid>/{exe,cwd} was broken because '{memory:' was
prepended twice. Also, get rid of the whole '{memory:...}' thing
because it's unnecessary, we can just list the file in /proc directly.
2019-03-14 13:30:25 +01:00
Guillaume Maudoux 4f4391193c Fix warning about unused variable 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 38ee16ae9c Unify internal findRootsXxx() api 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 9d7221183a unify naming of roots as links & targets 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 5c56570726 Also obfuscate the number of memory roots 2019-03-10 00:56:09 +01:00
Guillaume Maudoux fc02b1b3ee Also print rooted path in nix-store -q --roots 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 8574b70342 fixup! Make roots a map of store paths to pinning links 2019-03-10 00:56:09 +01:00
Guillaume Maudoux ebc86550f9 Make roots a map of store paths to pinning links
This new structure makes more sense as there may be many sources rooting
the same store path. Many profiles can reference the same path but this
is even more true with /proc/<pid>/maps where distinct pids can and
often do map the same store path.
This implementation is also more efficient as the `Roots` map contains
only one entry per rooted store path.
2019-03-10 00:56:09 +01:00
Guillaume Maudoux a17f86ce3a Obfuscate memory roots for non-root users 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 43331d6344 detail memory roots 2019-03-10 00:56:09 +01:00
volth fff8db205c
canBuildLocally: check for features
It could happen that the local builder match the system but lacks some features.
Now it results a failure.
The fix gracefully excludes the local builder from the set of available builders for derivation which requires the feature, so the derivation is built on remote builders only (as though it has incompatible system, like ```aarch64-linux``` when local is x86)
2019-03-06 05:03:25 +00:00
Will Dietz 0963479741
archive.cc: ignore more posix_fallocate "not supported" error codes
Fixes w/musl.
2019-03-01 10:31:17 +08:00
Dzmitry Zaitsau 06d6335987 fix indentation 2019-02-26 11:07:37 +01:00
Dzmitry Zaitsau ac200c3678 Apply param values on S3Helper initialization 2019-02-25 18:06:19 +01:00
Dzmitry Zaitsau 56c18c67d9 Extend S3 URL parsing with parameters extraction 2019-02-25 18:00:55 +01:00
Dzmitry Zaitsau 07f992a74b Extract and expose splitUriAndParams function
which splits a URL into localtor and parameter parts
2019-02-25 17:59:26 +01:00
Eelco Dolstra d8fe447139
Merge pull request #2579 from catern/dumpdb
nix-store: make --dump-db take a list of paths to dump
2019-02-14 13:08:02 +01:00
Eelco Dolstra 7a7ec22298
Merge pull request #2628 from shlevy/context-introspection
Context introspection
2019-02-12 17:57:35 +01:00
Eelco Dolstra 01d07b1e92
Revert "Restore parent mount namespace before executing a child process"
This reverts commit a0ef21262f. This
doesn't work in 'nix run' and nix-shell because setns() fails in
multithreaded programs, and Boehm GC mark threads are uncancellable.

Fixes #2646.
2019-02-05 10:49:19 +01:00
Shea Levy b30be6b450
Add builtins.appendContext.
A partner of builtins.getContext, useful for the same reasons.
2019-01-31 08:52:23 -05:00
Spencer Baugh 5f1891b795 nix-store: make --dump-db take a list of paths to dump
Inside a derivation, exportReferencesGraph already provides a way to
dump the Nix database for a specific closure. On the command line,
--dump-db gave us the same information, but only for the entire Nix
database at once.

With this change, one can now pass a list of paths to --dump-db to get
the Nix database dumped for just those paths. (The user is responsible
for ensuring this is a closure, like for --export).

Among other things, this is useful for deploying a closure to a new
host without using --import/--export; one can use tar to transfer the
store paths, and --dump-db/--load-db to transfer the validity
information. This is useful if the new host doesn't actually have Nix
yet, and the closure that is being deployed itself contains Nix.
2019-01-23 01:24:34 +00:00
Eelco Dolstra 7cc1a2593e
unsupported(): Show the name of the unsupported operation 2019-01-18 13:34:53 +01:00
Shea Levy 1d757292d0
Add builtins.getContext.
This can be very helpful when debugging, as well as enabling complex
black magic like surgically removing a single dependency from a
string's context.
2019-01-14 11:27:10 -05:00
Shea Levy 087be7281a
Treat plain derivation paths in context as normal paths.
Previously, plain derivation paths in the string context (e.g. those
that arose from builtins.storePath on a drv file, not those that arose
from accessing .drvPath of a derivation) were treated somewhat like
derivaiton paths derived from .drvPath, except their dependencies
weren't recursively added to the input set. With this change, such
plain derivation paths are simply treated as paths and added to the
source inputs set accordingly, simplifying context handling code and
removing the inconsistency. If drvPath-like behavior is desired, the
.drv file can be imported and then .drvPath can be accessed.

This is a backwards-incompatibility, but storePath is never used on
drv files within nixpkgs and almost never used elsewhere.
2019-01-13 11:29:55 -05:00
Eelco Dolstra c7bf1cdb4e
Merge pull request #2608 from dtzWill/fix/issue-2546
EvalState::resetFileCache: clear parse cache as well as eval cache
2019-01-10 20:56:31 +01:00
Eelco Dolstra 30906122e2
Merge pull request #2617 from veprbl/pr/V547_nix_repl.cc
remove some dead code in nix/repl.cc
2019-01-10 20:55:27 +01:00
Eelco Dolstra 013dd28b15
Merge pull request #2601 from dtzWill/update/json-3.5.0
Update nlohmann-json 3.0.1 -> 3.5.0
2019-01-07 23:13:35 +01:00
John Ericson fef9f5653b Remove mentions of libformat, it no longer exists 2019-01-05 14:31:29 -05:00
John Ericson e10d6ed2a7 brotli is only used as a library now 2019-01-05 14:25:54 -05:00
Will Dietz 21ea00d3ec EvalState::resetFileCache: clear parse cache as well as eval cache
Fixes #2546.

(at least the basic reproduction I've been testing)
2018-12-31 10:18:28 -06:00
Dmitry Kalinkin 93c9ba3e78 remove some dead code in nix/repl.cc
Fixes a static analyzer warning:

Expression 'isDrv' is always false.

Fixes: 3beb6f6e7 ('Show derivations more concisely')
2018-12-25 22:38:40 +03:00
Will Dietz 3f8b78a84d nlohmann-json: 3.4.0 -> 3.5.0
https://github.com/nlohmann/json/releases/tag/v3.5.0
2018-12-21 22:38:06 -06:00
Will Dietz e78511743e nlohmann: 3.0.1 -> 3.4.0
```
$ curl -L
"https://github.com/nlohmann/json/releases/download/v3.4.0/json.hpp" -o
src/nlohmann/json.hpp
```
2018-12-21 22:36:53 -06:00
Will Dietz c1112ae9a2 nix-store: remove debugging print 2018-12-15 09:59:47 -06:00
Daiderd Jordan 7e35e914c1
fetchGit: allow fetching explicit refs
Trying to fetch refs that are not in refs/heads currently fails because
it looks for refs/heads/refs/foo instead of refs/foo.

eg.

	builtins.fetchGit {
	  url = https://github.com/NixOS/nixpkgs.git;
	  ref = "refs/pull/1024/head;
	}
2018-12-14 20:12:22 +01:00
Eelco Dolstra 6024dc1d97
Support SRI hashes
SRI hashes (https://www.w3.org/TR/SRI/) combine the hash algorithm and
a base-64 hash. This allows more concise and standard hash
specifications. For example, instead of

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    sha256 = "5d22dad058d5c800d65a115f919da22938c50dd6ba98c5e3a183172d149840a4";
  };

you can write

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    hash = "sha256-XSLa0FjVyADWWhFfkZ2iKTjFDda6mMXjoYMXLRSYQKQ=";
  };

In fixed-output derivations, the outputHashAlgo is no longer mandatory
if outputHash specifies the hash (either as an SRI or in the old
"<type>:<hash>" format).

'nix hash-{file,path}' now print hashes in SRI format by default. I
also reverted them to use SHA-256 by default because that's what we're
using most of the time in Nixpkgs.

Suggested by @zimbatm.
2018-12-13 14:30:52 +01:00
Eelco Dolstra 378e89360d
Merge pull request #2576 from LnL7/repl-no-link
nix repl: don't create result symlinks
2018-12-13 10:40:39 +01:00
volth 21d494da83
probably typo
...at least MSVC unable to compile this
2018-12-13 02:45:50 +00:00
Daiderd Jordan 0e6c84a771
nix repl: don't create result symlinks 2018-12-12 23:00:07 +01:00
Jonas Chevalier 5e6fa9092f
libstore: improve hash mismatch error messages
Use the same output ordering and format everywhere.

This is such a common issue that we trade the single-line error message for
more readability.

Old message:
```
fixed-output derivation produced path '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com' with sha256 hash '08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm' instead of the expected hash '1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m'
```

New message:
```
hash mismatch in fixed-output derivation '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com':
  wanted: sha256:1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m
  got:    sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm

```
2018-12-12 17:42:00 +01:00
Eelco Dolstra f7425d55df
Fix assertion failure in NarInfoDiskCache
https://hydra.nixos.org/build/85827920
2018-12-12 17:13:22 +01:00
Daiderd Jordan 6f89053108
nar-info-disk-cache: include ca in the cache entries
Without this information the content addressable state and hashes are
lost after the first request, this causes signatures to be required for
everything even tho the path could be verified without signing.
2018-12-12 01:08:47 +01:00
Daiderd Jordan 898823b67d
s3: make scheme configurable
This enables using for http for S3 request for debugging or
implementations that don't have https configured.  This is not a problem
for binary caches since they should not contain sensitive information.
Both package signatures and AWS auth already protect against tampering.
2018-12-07 23:55:38 +01:00
CHEIKH Chawki fa5143c722 Solve hg "abandoned transaction" issue 2018-12-06 13:57:59 +01:00
Eelco Dolstra 4aee93d5ce
fetchGit: Drop unnecessary localRef 2018-11-20 20:59:44 +01:00
Eelco Dolstra 3f4de91d80
Merge branch 'better-git-cache' of https://github.com/graham-at-target/nix 2018-11-20 20:41:19 +01:00
Eelco Dolstra ebeea068d5
Merge pull request #2216 from dtzWill/fix/curl-760-timeout
download: if there are active requests, never sleep for 10s
2018-11-20 19:07:56 +01:00
Eelco Dolstra 5e64470b19
Fix typo 2018-11-20 16:54:17 +01:00
Kai Harries de5997332d repl: give user the choice between libeditline and libreadline
The goal is to support libeditline AND libreadline and let the user
decide at compile time which one to use.

Add a compile time option to use libreadline instead of
libeditline. If compiled against libreadline completion functionality
is lost because of a incompatibility between libeditlines and
libreadlines completion function. Completion with libreadline is
possible and can be added later.

To use libreadline instead of libeditline the environment
variables 'EDITLINE_LIBS' and 'EDITLINE_CFLAGS' have to been set
during the ./configure step.

Example:

  EDITLINE_LIBS="/usr/lib/x86_64-linux-gnu/libhistory.so /usr/lib/x86_64-linux-gnu/libreadline.so"
  EDITLINE_CFLAGS="-DREADLINE"

The reason for this change is that for example on Debian already three
different editline libraries exist but none of those is compatible the
flavor used by nix. My hope is that with this change it would be
easier to port nix to systems that have already libreadline available.
2018-11-20 15:43:21 +01:00
Kai Harries b289d86cd1 repl: Remove code that was commented out 2018-11-20 15:42:58 +01:00
Linus Heckemann 40e0c9e925 nix ls-nar: allow reading from FIFOs
fixes #2528
2018-11-16 16:24:36 +01:00
Eelco Dolstra a32ff2573b
Fix 'Read-only file system' when building a derivation 2018-11-15 13:20:39 +01:00
Eelco Dolstra 6c6bbeb439 Merge branch 'feature/editline-pr' of https://github.com/dtzWill/nix 2018-11-13 20:42:13 +01:00
Eelco Dolstra a0ef21262f
Restore parent mount namespace before executing a child process
This ensures that they can't write to /nix/store. Fixes #2535.
2018-11-13 16:15:30 +01:00
Eelco Dolstra 7751616645
nix-prefetch-url: Stop progress bar before printing results 2018-11-09 10:34:12 +01:00
Eelco Dolstra 220c79ec22
Merge branch 'feature/prefetch-progress' of https://github.com/dtzWill/nix 2018-11-09 10:29:59 +01:00
Eelco Dolstra 812e39313c
Enable sandboxing by default
Closes #179.
2018-11-07 17:08:28 +01:00
Eelco Dolstra 6323b0729a
Disable the S3 content-encoding patch
Since we're not using multi-part uploads at the moment, we can drop
this patch.
2018-11-01 15:18:38 +01:00
Jan Path d1b049c4ea Fix typo in comments 2018-10-31 20:50:18 +01:00
Eelco Dolstra fb35aaa422
Merge pull request #2501 from mayflower/xdg_config_dirs
config: use all of XDG_CONFIG_DIRS
2018-10-31 12:37:23 +01:00
Linus Heckemann f3b8173a93 config: use all of XDG_CONFIG_DIRS
Previously, config would only be read from XDG_CONFIG_HOME. This change
allows reading config from additional directories, which enables e.g.
per-project binary caches or chroot stores with the help of direnv.
2018-10-31 09:24:43 +01:00
Eelco Dolstra 38d76d6d78
Remove redundant message 2018-10-30 19:28:42 +01:00
Eelco Dolstra 9f99d62480
S3BinaryCacheStore: Allow disabling multipart uploads
The use of TransferManager has several issues, including that it
doesn't allow setting a Content-Encoding without a patch, and it
doesn't handle exceptions in worker threads (causing termination on
memory allocation failure).

Fixes #2493.
2018-10-30 14:25:00 +01:00
Eelco Dolstra 0163e8928c
Fix broken uploadProgressCallback closure
Since the callback is global we can't refer to 'path' in it. This
could cause a segfault or printing of arbitrary data.
2018-10-30 11:45:31 +01:00
Eelco Dolstra 056c3fbbfc
Merge pull request #2494 from FPtje/fix-copy-from-old
copyStorePath: Fix hash errors when copying from older store
2018-10-29 20:45:25 +01:00
Falco Peijnenburg 49e272f647 copyStorePath: Fix hash errors when copying from older store
This commit partially reverts 48662d151b. When
copying from an older store (in my case a store running Nix 1.11.7), nix would
throw errors about there being no hash. This is fixed by recalculating the hash.
2018-10-29 20:24:37 +01:00
Will Dietz 9f998096d2 repl: complete if all matches share prefix 2018-10-29 08:45:04 -05:00
Will Dietz 3d974d31fa editline: wip 2018-10-29 08:44:58 -05:00
Daiderd Jordan 8e6bf49297
nix doctor: return nonzero exitcode if a check fails
This makes it easier to use this when testing the installer or when
running the checks with other automated tooling.
2018-10-29 12:09:22 +00:00
Eelco Dolstra 18b4c53f71
Restore old (dis)allowedRequisites behaviour for self-references
stdenv relies on this. So ignore self-references (but only in legacy non-structured attributes mode).
2018-10-27 15:41:53 +02:00
Eelco Dolstra 1427958b3c
Merge branch 'simplify-lexer' of https://github.com/layus/nix 2018-10-27 13:42:06 +02:00
Eelco Dolstra 0c61515be1
Fix signedness warning 2018-10-27 13:07:26 +02:00
Eelco Dolstra 63575ffa38
Merge branch 'nix-doctor' of https://github.com/LnL7/nix 2018-10-27 12:54:22 +02:00
Eelco Dolstra 2cf98218c8
Merge pull request #2488 from dtzWill/fix/struct-class-minor
remote-store.hh: ConnectionHandle is struct, minor fix warning
2018-10-27 12:50:16 +02:00
Guillaume Maudoux 6a5bf9b143 simplify handling of extra '}' 2018-10-27 00:14:51 +02:00
Will Dietz 9d24b5d56e nix-prefetch-url: progressbar 2018-10-26 13:19:10 -05:00
Will Dietz 3283c0dc45 remote-store.hh: ConnectionHandle is struct, minor fix warning 2018-10-26 13:15:48 -05:00
Will Dietz 27c2fcd4c0 src/nix/local.mk: fix typos in names of symlinks 2018-10-26 13:15:07 -05:00
Eelco Dolstra f6a3dfe4e0
Merge all nix-* binaries into nix
These are all symlinks to 'nix' now, reducing the installed size by
about ~1.7 MiB.
2018-10-26 12:54:00 +02:00
Eelco Dolstra c47e14ee45
Merge pull request #2483 from mayflower/tail-config
Promote verbose-build and log-lines
2018-10-23 16:59:52 +02:00
Linus Heckemann a25abe823f Promote log-lines to a fully-qualified option
This allows commands like

nix build --log-lines 30 nixpkgs.hello

in order to obtain more information in case of a failure.
2018-10-23 16:54:24 +02:00
Eelco Dolstra 1a08ad75ea
Merge pull request #2479 from nlewo/graphml
Add --graphml option to the nix-store --query command
2018-10-23 15:56:05 +02:00
Eelco Dolstra 3cd15c5b1f
Per-output reference and closure size checks
In structured-attributes derivations, you can now specify per-output
checks such as:

  outputChecks."out" = {
    # The closure of 'out' must not be larger than 256 MiB.
    maxClosureSize = 256 * 1024 * 1024;

    # It must not refer to C compiler or to the 'dev' output.
    disallowedRequisites = [ stdenv.cc "dev" ];
  };

  outputChecks."dev" = {
    # The 'dev' output must not be larger than 128 KiB.
    maxSize = 128 * 1024;
  };

Also fixed a bug in allowedRequisites that caused it to ignore
self-references.
2018-10-23 01:29:16 +02:00
Antoine Eiche d506342aa2 Remove the --xml query command option
The `--graphml` option can be used instead.
2018-10-20 09:48:53 +02:00
Antoine Eiche 73c2ae43f0 Add --graphml option to the nix-store --query command
This prints the references graph of the store paths in the graphML
format [1]. The graphML format is supported by several graph tools
such as the Python Networkx library or the Apache Thinkerpop project.

[1] http://graphml.graphdrawing.org
2018-10-20 09:48:34 +02:00
Eelco Dolstra bd78544f66 Fix assertion failure in Store::queryPathInfo()
$ nix-store -qR /nix/store/fnord
  nix-store: src/libstore/store-api.cc:80: std::__cxx11::string nix::storePathToHash(const Path&): Assertion `base.size() >= storePathHashLen' failed.
  Aborted
2018-10-16 23:39:36 +02:00
Eelco Dolstra 79e358ce6d RemoteStore: Close connection if an exception occurs
Fixes #2075.
2018-10-16 23:36:15 +02:00
Eelco Dolstra ba51100d64 Get rid of UDSRemoteStore::Connection
Since its superclass RemoteStore::Connection contains 'to' and 'from'
fields that refer to the file descriptor maintained in the subclass,
it was possible for the flush() call in Connection::~Connection() to
write to a closed file descriptor (or worse, a file descriptor now
referencing another file). So make sure that the file descriptor
survives 'to' and 'from'.
2018-10-16 22:58:01 +02:00
Eelco Dolstra 01bd66bf83
Merge pull request #2452 from ElvishJerricco/verify-sigs-overflow
Fix overflow when verifying signatures of content addressable paths
2018-10-08 12:03:03 +02:00
Eelco Dolstra 0fda9b22c7
Remove unnecessary typecast 2018-10-04 13:16:30 +02:00
Eelco Dolstra 1e7b8deea7
Check requiredSystemFeatures for local builds
For example, this prevents a "kvm" build on machines that don't have
KVM.

Fixes #2012.
2018-09-28 16:10:27 +02:00
Eelco Dolstra 7ae7a38c9a
Move structured attrs handling into a separate class
This is primarily because Derivation::{can,will}BuildLocally() depends
on attributes like preferLocalBuild and requiredSystemFeatures, but it
can't handle them properly because it doesn't have access to the
structured attributes.
2018-09-28 14:32:46 +02:00
Eelco Dolstra 99d4bb2d4c
Remove obsolete check on system features 2018-09-28 13:28:15 +02:00
Eelco Dolstra c9ba33870e
Support special attributes in structured attributes derivations
E.g. __noChroot and allowedReferences now work correctly. We also now
check that the attribute type is correct. For instance, instead of

  allowedReferences = "out";

you have to write

  allowedReferences = [ "out" ];

Fixes #2453.
2018-09-28 12:43:01 +02:00
Eelco Dolstra 63786cbd3b
S3BinaryCacheStore: Don't create buckets
This meant that making a typo in an s3:// URI would cause a bucket to
be created. Also it didn't handle eventual consistency very well. Now
it's up to the user to create the bucket.
2018-09-27 21:34:41 +02:00
Eelco Dolstra f74e0b4786
Merge pull request #2455 from grahamc/add-interactive-path
nix-shell: add bashInteractive to the start of the PATH, set SHELL
2018-09-27 21:15:50 +02:00
Graham Christensen f11acbaf17
nix-shell: add bashInteractive to the start of the PATH, set SHELL
Tools which re-exec `$SHELL` or `$0` or `basename $SHELL` or even just
`bash` will otherwise  get the non-interactive bash, providing a
broken shell for the same reasons described in
https://github.com/NixOS/nixpkgs/issues/27493.

Extends c94f3d5575
2018-09-27 15:12:56 -04:00
Eelco Dolstra 7cba4214a8
Don't talk about a "current folder build output"
Presumably this refers to ./default.nix but the support for that in
'nix' is tenuous. Also folders are a Mac thing.
2018-09-27 21:01:19 +02:00
Daiderd Jordan ed25753501
nix doctor: reimplement profile warning without gcroot check
Calculating roots seems significantly slower on darwin compared to
linux.  Checking for /profile/ links could show some false positives but
should still catch most issues.
2018-09-26 23:13:17 +02:00
Eelco Dolstra 9750430003
Ensure download thread liveness
* Don't wait forever for the client to remove data from the
  buffer. This does mean that the buffer can grow without bounds
  (e.g. when downloading is faster than writing to disk), but meh.

* Don't hold the state lock while calling the sink. The sink could
  take any amount of time to process the data (in particular when it's
  actually a coroutine), so we don't want to block the download
  thread.
2018-09-26 21:47:34 +02:00
Eelco Dolstra 98b2cc2e6e
Untabify 2018-09-26 21:39:06 +02:00
Eelco Dolstra ed78582847
sinkToSource(): Start the coroutine lazily
In particular this causes copyStorePath() from HttpBinaryCacheStore to
only start a download if needed. E.g. if the destination LocalStore
goes to sleep waiting for the path lock and another process creates
the path, then LocalStore::addToStore() will never read from the
source so we don't have to do the download.
2018-09-26 21:19:34 +02:00
Eelco Dolstra 44e86304b6
Make NAR header check more robust
Changes

  std::bad_alloc

into

  bad archive: input doesn't look like a Nix archive
2018-09-26 12:03:58 +02:00
Will Fancher 0ae8d4033d Fix overflow when verifying signatures of content addressable paths 2018-09-25 21:19:24 -04:00
Will Dietz 8df367a92d nix-daemon: allow setting builders to "" by any user (untrusted)
(cherry picked from commit a94a2eb1cb1c81e90a7529be5fecac27899a3442)
2018-09-25 03:22:08 -05:00
Eelco Dolstra 42e4ea2b6d
Merge pull request #2312 from dtzWill/fix/matched-names-should-still-print-version
search: include version for matches too!
2018-09-24 13:03:28 +02:00
Eelco Dolstra 52f6d541b9
nix-env: Fix segfault if -f argument is not a directory or a Nix expression
Fixes #2425.
2018-09-17 16:36:30 +02:00
Eelco Dolstra 901dfc7978
Fix build failure if parallel xz is not available 2018-09-17 16:36:11 +02:00
Eelco Dolstra 8327a7a8fa
Merge branch 'dirOf-relative' of https://github.com/lheckemann/nix 2018-09-13 14:33:12 +02:00
Ding Xiang Fei d6ac762bf7 auto args on repl 2018-09-10 15:43:17 +08:00
Eelco Dolstra bba3f0a308
If fallback is enabled, temporarily disable substituters after a failure
Otherwise, we just keep asking the substituter for other .narinfo
files, which can take a very long time due to retries/timeouts.
2018-09-07 17:08:43 +02:00
Eelco Dolstra 33c3f91885
Handle queryPathInfo() failure from substituters when fallback is enabled
Fixes #1990.
2018-09-07 16:35:48 +02:00
Eelco Dolstra 91405986f4
Convert NIX_COUNT_CALLS to JSON too 2018-09-05 21:57:54 +02:00
Eelco Dolstra 0a2545f95c
Log stats to stderr
We shouldn't pollute stdout.
2018-09-05 21:35:58 +02:00
Eelco Dolstra c4ba5f4be9
Merge branch 'improve-stats' of https://github.com/cleverca22/nix 2018-09-05 21:28:05 +02:00
Eelco Dolstra 2d91012754
fetchurl: Respect unpack
Fixes #2393.
2018-09-05 21:22:37 +02:00
Matthew Bauer 74f6d8767d Get effective user in Nix commands
‘geteuid’ gives us the user that the command is being run as,
including in setuid modes. By using geteuid to determind id, we can
avoid the ‘sudo -i’ hack when upgrading Nix. So now, upgrading Nix on
macOS is as simple as:

$ sudo nix-channel --update
$ sudo nix-env -u

$ sudo launchctl stop org.nixos.nix-daemon
$ sudo launchctl start org.nixos.nix-daemon
or
$ sudo systemctl restart nix-daemon
2018-09-04 19:32:39 -05:00
Will Dietz 28418af920 download: fix size reported to progress bar 2018-09-03 14:12:54 -05:00
Michael Bishop 4b034f390c remove the old text format output 2018-09-02 18:25:23 -03:00
Daiderd Jordan 80a4b44d3d
nix doctor: only perform path/profile checks with a daemon/local store
Not all store types LegacySSHStore support these operations and it
doesn't really make sense to check those.
2018-09-02 13:41:55 +02:00
Daiderd Jordan bfdca55868
nix doctor: add check for profile roots
In most cases profiles that are in PATH should have a gcroot.
2018-09-02 12:56:28 +02:00
Daiderd Jordan 0f18dc5479
nix doctor: add warning for multiple versions
It's pretty easy to unintentionally install a second version of nix
into the user profile when using a daemon install.  In this case it
looks like nix was upgraded while the nix-daemon is probably still
unning an older version.
2018-09-02 12:56:28 +02:00
Daiderd Jordan 246acf93f2
nix doctor: handle serve protocol
The serve protocol used by LegacySSHStore has a different major and
shouldn't be compared to PROTOCOL_VERSION.
2018-09-02 12:54:42 +02:00
Daiderd Jordan 7314dc7f07
nix doctor: add warning if client/daemon protocol mismatches
A protocol mismatch can sometimes cause problems when using specific
features with an older daemon. For example:

Nix 2.0 changed the way files are compied to the store.  The daemon is
backwards compatible and can still handle older clients, however a 1.11
nix-daemon isn't forwards compatible.
2018-09-02 12:54:39 +02:00
Daiderd Jordan 070823baa4
Store: expose the protocol version used by a store 2018-09-02 12:54:29 +02:00
Daiderd Jordan c9a08540c3
nix doctor: add command
Inspired by the homebrew command, shows a combination of debugging
information and warnings with potential issues with a nix installation.
2018-09-02 12:54:20 +02:00
Michael Bishop 2fd1008c70 add JSON to NIX_SHOW_STATS 2018-09-01 20:05:06 -03:00
Michael Bishop c29e5fbb13 improve the stats when profiling 2018-09-01 17:11:56 -03:00
Graham Christensen 87702532d2
nix-build: Print stats even in failing builds 2018-08-31 21:03:32 -04:00
Graham Christensen b7bb627f67
nix-build: print stats on sucessful builds 2018-08-31 20:57:07 -04:00
Eelco Dolstra 475a0a54a9
fetchGit/fetchMercurial: Don't absolutize paths
This is already done by coerceToString(), provided that the argument
is a path (e.g. 'fetchGit ./bla'). It fixes the handling of URLs like
git@github.com:owner/repo.git. It breaks 'fetchGit "./bla"', but that
was never intended to work anyway and is inconsistent with other
builtin functions (e.g. 'readFile "./bla"' fails).
2018-09-01 00:19:49 +02:00
Eelco Dolstra b7409c5754
nix path-info: Remove trailing spaces
Fixes #2390.
2018-09-01 00:04:57 +02:00
Eelco Dolstra d1f36e8787
Really fix the build 2018-08-31 19:20:08 +02:00
Eelco Dolstra ef09da58f2
nix path-info: Hopefully fix macOS build failure
https://hydra.nixos.org/build/80480356
2018-08-31 17:30:45 +02:00
Eelco Dolstra c0c31b58a4
Add localhost alias for ::1 to the sandbox 2018-08-31 11:12:51 +02:00
Michael Bishop 0767e402f1 fix error: unknown serve command 9 2018-08-30 20:10:08 -03:00
Eelco Dolstra 39f1722f36
nix upgrade-nix: Handle .nix-profile being a link to .../profiles/per-user/...
Also some cosmetic improvements.
2018-08-30 21:18:56 +02:00
Eelco Dolstra f08b14c9d0
nix upgrade-nix: Improve error message if 'nix' is not in a profile
E.g.

  $ nix upgrade-nix
  error: directory '/home/eelco/Dev/nix/inst/bin' does not appear to be part of a Nix profile

instead of

  $ nix upgrade-nix
  error: '/home/eelco/Dev/nix/inst' is not a symlink
2018-08-30 17:14:17 +02:00
Eelco Dolstra ddeda0b62e
Merge branch 'nix-upgrade-profile' of https://github.com/LnL7/nix 2018-08-30 17:10:59 +02:00
Eelco Dolstra 264e66f696
Add simple test for nix upgrade-nix 2018-08-30 17:10:28 +02:00
Eelco Dolstra 99828245f8
printSize() fixes
Fix a 32-bit overflow that resulted in negative numbers being printed;
use fmt() instead of boost::format(); change -H to -h for consistency
with 'ls' and 'du'; make the columns narrower (since they can't be
bigger than 1024.0).
2018-08-30 16:22:04 +02:00
Benjamin Hipple c908df881f
Avoid overflow and use boost::format
If the user has an object greater than 1024 yottabytes, it'll just display it as
N yottabytes instead of overflowing.

Swaps to use boost::format strings instead of std::setw and std::setprecision.
2018-08-30 16:22:04 +02:00
Benjamin Hipple 3407a5d936
Add human readable closure sizes to nix path-info
Unfortunately, -h is already taken as a short option by --help, so we have to
use a different letter or the capitalized version.

Resolves #2363
2018-08-30 16:22:04 +02:00
Eelco Dolstra 5e83b0227f
nix: Remove the -h flag 2018-08-30 16:14:19 +02:00
Eelco Dolstra 1286d86b95
Merge pull request #2336 from ivan/repl-trailing-spaces
repl: don't add trailing spaces to history lines
2018-08-29 18:13:29 +02:00
aszlig 0ad643ed5c
libexpr: Use int64_t for NixInt
Using a 64bit integer on 32bit systems will come with a bit of a
performance overhead, but given that Nix doesn't use a lot of integers
compared to other types, I think the overhead is negligible also
considering that 32bit systems are in decline.

The biggest advantage however is that when we use a consistent integer
size across all platforms it's less likely that we miss things that we
break due to that. One example would be:

https://github.com/NixOS/nixpkgs/pull/44233

On Hydra it will evaluate, because the evaluator runs on a 64bit
machine, but when evaluating the same on a 32bit machine it will fail,
so using 64bit integers should make that consistent.

While the change of the type in value.hh is rather easy to do, we have a
few more options available for doing the conversion in the lexer:

  * Via an #ifdef on the architecture and using strtol() or strtoll()
    accordingly depending on which architecture we are. For the #ifdef
    we would need another AX_COMPILE_CHECK_SIZEOF in configure.ac.
  * Using istringstream, which would involve copying the value.
  * As we're already using boost, lexical_cast might be a good idea.

Spoiler: I went for the latter, first of all because lexical_cast does
have an overload for const char* and second of all, because it doesn't
involve copying around the input string. Also, because istringstream
seems to come with a bigger overhead than boost::lexical_cast:

https://www.boost.org/doc/libs/release/doc/html/boost_lexical_cast/performance.html

The first method (still using strtol/strtoll) also wasn't something I
pursued further, because it is also locale-aware which I doubt is what
we want, given that the regex for int is [0-9]+.

Signed-off-by: aszlig <aszlig@nix.build>
Fixes: #2339
2018-08-29 01:05:52 +02:00
Daiderd Jordan d85bb4814f
upgrade-nix: resolve profile symlinks
The profile present in PATH is not necessarily the actual profile
location. User profiles are generally added as $HOME/.nix-profile
in which case the indirect profile link needs to be resolved first.

/home/user/.nix-profile -> /nix/var/nix/profiles/per-user/user/profile
/nix/var/nix/profiles/per-user/user/profile -> profile-15-link
/nix/var/nix/profiles/per-user/user/profile-14-link -> /nix/store/hyi4kkjh3bwi2z3wfljrkfymz9904h62-user-environment
/nix/var/nix/profiles/per-user/user/profile-15-link -> /nix/store/6njpl3qvihz46vj911pwx7hfcvwhifl9-user-environment

To upgrade nix here we want /nix/var/nix/profiles/per-user/user/profile-16-link
instead of /home/user/.nix-profile-1-link. The latter is not a gcroot
and would be garbage collected, resulting in a broken profile.

Fixes #2175
2018-08-25 20:50:44 +02:00
Daiderd Jordan 414397759a
upgrade-nix: add --dry-run 2018-08-25 20:25:43 +02:00
Eelco Dolstra c651b7bdc9
Revert "Fix parser/lexer generation with parallel make"
This reverts commit d277442df5.

Make sucks.
2018-08-23 00:23:35 +02:00
Eelco Dolstra 1b01954a3d
Merge pull request #2368 from dezgeg/drop-dead-code
Drop dead code
2018-08-22 21:32:49 +02:00
Eelco Dolstra ebe3d2d370 Improve 'coroutine has finished' error message 2018-08-21 15:22:04 +02:00
Eelco Dolstra 6317c65937 Fix warnings in compression.cc 2018-08-21 15:20:23 +02:00
Eelco Dolstra 17a92dfb7d Fix another 'coroutine has finished' during decompression
https://hydra.nixos.org/build/79867739
2018-08-21 15:19:20 +02:00
Tuomas Tynkkynen 2894197de7 Drop all references to NIX_INDENT_MAKE
Dead code since 6669a3b477
2018-08-20 02:01:59 +03:00
Eelco Dolstra cc7b4386b1
nix run: Restore CPU affinity
Fixes #2359.
2018-08-19 12:05:08 +02:00
Eelco Dolstra 9b1bdf2db8
FIx floating point evaluation
Fixes #2361.
2018-08-19 11:59:49 +02:00
Graham Christensen 02098d2073 fetchGit: use a better caching scheme
The current usage technically works by putting multiple different
repos in to the same git directory. However, it is very slow as
Git tries very hard to find common commits between the two
repositories. If the two repositories are large (like Nixpkgs and
another long-running project,) it is maddeningly slow.

This change busts the cache for existing deployments, but users
will be promptly repaid in per-repository performance.
2018-08-17 11:27:34 -04:00
Eelco Dolstra d277442df5
Fix parser/lexer generation with parallel make
Fun fact: rules with multiple targets don't work properly with 'make
-j'. For example, a rule like

  a b: c
    touch a b

is equivalent to

  a: c
    touch a b

  b: c
    touch a b

so with 'make -j', the 'touch' command will be run twice. See
e.g. https://stackoverflow.com/questions/2973445/gnu-makefile-rule-generating-a-few-targets-from-a-single-source-file.
2018-08-17 12:59:23 +02:00
Symphorien Gibol 5b19a6663b ignore when listxattr fails with ENODATA
This happens on CIFS and means the remote filesystem has no extended
attributes.
2018-08-13 21:11:36 +02:00
Linus Heckemann d7402c9cd5 dirOf: allow use on non-absolute paths 2018-08-13 11:27:35 +02:00
Eelco Dolstra f72c907ad8
Slightly questionable workaround for #2342 2018-08-10 11:34:41 +02:00
Eelco Dolstra ada4e90267
S3BinaryCacheStore: Share TransferManager
TransferManager allocates a lot of memory (50 MiB by default), and it
might leak but I'm not sure about that. In any case it was causing
OOMs in hydra-queue-runner. So allocate only one TransferManager per
S3BinaryCacheStore.

Hopefully fixes https://github.com/NixOS/hydra/issues/586.
2018-08-09 20:48:45 +02:00
Eelco Dolstra c87f4b9324
nix run: Respect propagated-user-env-packages
Also, add $path/bin to $PATH even if it doesn't exist. This makes
'man' work properly (since it looks for ../share/man relative to $PATH
entries).
2018-08-09 13:01:03 +02:00
Eelco Dolstra a0b971dd9c
S3BinaryCacheStore: Don't use the transfer status callback
This callback is executed on a different thread, so exceptions thrown
from the callback are not caught:

  Aug 08 16:25:48 chef hydra-queue-runner[11967]: terminate called after throwing an instance of 'nix::Error'
  Aug 08 16:25:48 chef hydra-queue-runner[11967]:   what():  AWS error: failed to upload 's3://nix-cache/19dbddlfb0vp68g68y19p9fswrgl0bg7.ls'

Therefore, just check the transfer status after it completes. Also
include the S3 error message in the exception.
2018-08-08 21:39:11 +02:00
Bob van der Linden 58a85fa462 mention nix-store --query --roots when a path cannot be deleted 2018-08-08 21:21:21 +02:00
Ivan Kozik ec49ea28dc repl: don't add trailing spaces to history lines 2018-08-07 03:59:34 +00:00
Eelco Dolstra 103ad1598c
Merge pull request #2334 from dtzWill/fix/revert-progress-bar-refresh-for-now
Revert "progress-bar: re-draw last update if nothing new for 1sec."
2018-08-06 19:01:08 +02:00
Eelco Dolstra ed6c646f44
Doh 2018-08-06 17:27:08 +02:00
Eelco Dolstra 7de3e00ad9
Fix setting Content-Encoding in S3 uploads
Fixes https://github.com/NixOS/nix/issues/2333 and https://github.com/NixOS/nixpkgs/issues/44337.
2018-08-06 17:20:23 +02:00
Eelco Dolstra d3761f5f8b
Fix Brotli decompression in 'nix log'
This didn't work anymore since decompression was only done in the
non-coroutine case.

Decompressors are now sinks, just like compressors.

Also fixed a bug in bzip2 API handling (we have to handle BZ_RUN_OK
rather than BZ_OK), which we didn't notice because there was a missing
'throw':

  if (ret != BZ_OK)
      CompressionError("error while compressing bzip2 file");
2018-08-06 15:40:29 +02:00
Eelco Dolstra fa4def3d46
Require libbrotli 2018-08-06 14:06:54 +02:00
Eelco Dolstra 4361a4331f
Fix reporting of HTTP body size when a result callback is used 2018-08-06 11:31:14 +02:00
Eelco Dolstra 2825e05d21
Make adding paths via nix-store --serve run in constant memory
It adds a new operation, cmdAddToStoreNar, that does the same thing as
the corresponding nix-daemon operation, i.e. call addToStore(). This
replaces cmdImportPaths, which has the major issue that it sends the
NAR first and the store path second, thus requiring us to store the
incoming NAR either in memory or on disk until we decide what to do
with it.

For example, this reduces the memory usage of

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 267 MiB to 12 MiB.

Probably fixes #1988.
2018-08-03 21:21:16 +02:00
Eelco Dolstra eeebe4cdc5
cmdDumpStorePath: Support chroot stores 2018-08-03 21:20:38 +02:00
Eelco Dolstra 848a9375c3
Support escaping in store URIs 2018-08-03 21:20:38 +02:00
Eelco Dolstra 4e7d5f660c
SSHMaster: Bypass SSH when connecting to localhost
This is primarily useful for testing since it removes the need to have
SSH working.
2018-08-03 21:20:38 +02:00
Eelco Dolstra e268bbc054
LegacySSHStore: Add remote-store option
This is primarily useful for testing, e.g.

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' ...
2018-08-03 21:20:38 +02:00
Eelco Dolstra bc65e02d96
Merge pull request #2326 from aszlig/fix-symlink-leak
Fix symlink leak in restricted eval mode
2018-08-03 17:01:34 +02:00
Eelco Dolstra 122e1a61f8
Merge pull request #2323 from samueldr/feature/selective-impurity
Allows selectively adding environment variables to pure shells.
2018-08-03 16:58:18 +02:00
aszlig 43e28a1b75
Fix symlink leak in restricted eval mode
In EvalState::checkSourcePath, the path is checked against the list of
allowed paths first and later it's checked again *after* resolving
symlinks.

The resolving of the symlinks is done via canonPath, which also strips
out "../" and "./". However after the canonicalisation the error message
pointing out that the path is not allowed prints the symlink target in
the error message.

Even if we'd suppress the message, symlink targets could still be leaked
if the symlink target doesn't exist (in this case the error is thrown in
canonPath).

So instead, we now do canonPath() without symlink resolving first before
even checking against the list of allowed paths and then later do the
symlink resolving and checking the allowed paths again.

The first call to canonPath() should get rid of all the "../" and "./",
so in theory the only way to leak a symlink if the attacker is able to
put a symlink in one of the paths allowed by restricted evaluation mode.

For the latter I don't think this is part of the threat model, because
if the attacker can write to that path, the attack vector is even
larger.

Signed-off-by: aszlig <aszlig@nix.build>
2018-08-03 06:46:43 +02:00
Samuel Dionne-Riel 438e02529d Allows selectively adding environment variables to pure shells.
Includes documentation and test.
2018-08-02 08:21:30 -04:00
Graham Christensen 49a53c1d3f s3 binary cache: support specifying an endpoint
Works for uploading and not downloading.
2018-08-02 08:19:50 -04:00
Samuel Dionne-Riel fb6907e3d6 Removes unused variable from nix-build/nix-shell
This particular `shell` variable wasn't used, since a new one was
declared in the only side of the `if` branch that used a `shell`
variable.

It could realistically confuse developers thinking it could use `$SHELL`
under some situations.
2018-08-01 18:30:03 -04:00
Eelco Dolstra 45bcf5416a
Merge branch 'prim_mapAttr-fix' of https://github.com/volth/nix 2018-07-31 20:05:07 +02:00
Eelco Dolstra 4b4adbc93d
Fix race downloading into a sink
This fixes 'error 10 while decompressing xz file'.

https://hydra.nixos.org/build/78308551
2018-07-31 17:26:04 +02:00
Eelco Dolstra d9e8ab0ff2
Add a check for broken Boost versions
In some Boost versions, coroutines don't propagate exceptions
properly, causing Nix to fail with the exception 'coroutine has
finished'.
2018-07-31 16:35:05 +02:00
Will Dietz 0d72f4e0ca search: include version 2018-07-27 10:45:35 -05:00
Linus Heckemann 9ac1a79882 copyPathsToStore: honour keep-going 2018-07-24 17:05:06 +02:00
Eelco Dolstra a7fb7d3cde
Merge pull request #2303 from volth/patch-4
parser.y: fix assoc of -> and < > <= >=
2018-07-23 11:38:15 +02:00
volth deaa6e9a34
parser.y: right-associativity of -> 2018-07-23 07:28:48 +00:00
volth 85fe4a819c
parser.y: fix assoc of -> and < > <= >=
The parser allowed senseless `a > b > c` but disallowed `a -> b -> c` which seems valid
It might be a typo
2018-07-21 15:24:51 +00:00
volth e2b114cfe1
prim_foldlStrict: call forceValue() before value is copied
forceValue() were called after a value is copied effectively forcing only one of the copies keeping another copy not evaluated.
This resulted in its evaluation of the same lazy value more than once (the number of hits is not big though)
2018-07-21 06:44:42 +00:00
Will Dietz 4495560d6d Revert "progress-bar: re-draw last update if nothing new for 1sec."
Not ready for this yet, causes the prompt to disappear in nix repl
and more generally can overwrite non-progress-bar messages.

This reverts commit 44de71a396.
2018-07-19 10:29:31 -05:00
Eelco Dolstra 1b34b69b45
nix-prefetch-url: Download file in constant memory
Before:

  $ command time nix-prefetch-url https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.17.6.tar.xz
  1.19user 1.02system 0:41.96elapsed 5%CPU (0avgtext+0avgdata 182720maxresident)k

After:

  1.38user 1.05system 0:39.73elapsed 6%CPU (0avgtext+0avgdata 16204maxresident)k

Note however that addToStore() can still take a lot of memory
(e.g. RemoteStore::addToStore() is constant space, but
LocalStore::addToStore() isn't; that's fixed by
c94b4fc7ee
though).

Fixes #1400.
2018-07-12 18:48:41 +02:00
Eelco Dolstra c2de2ff385
Fix build on 32-bit systems and macOS
Apparently, on macOS, 'long' != 'int64_t'.

https://hydra.nixos.org/build/77100756
2018-07-11 21:12:18 +02:00
Eelco Dolstra efd04888ca
Shup up a warning 2018-07-11 21:05:09 +02:00
Eelco Dolstra 875cd9da2b
Remove unused function printStats2()
Closes #2282.
2018-07-11 20:29:18 +02:00
volth c03d73c1cf
prim_map, prim_genList: no need to force function argument which will be stored in mkApp 2018-07-07 23:30:17 +00:00
volth 627e28ba33 prim_mapAttrs: `f' must be evaluated lazily to avoid infinite recursion 2018-07-06 21:52:54 +00:00
Eelco Dolstra aa64e95bc8
Merge pull request #2250 from dtzWill/feature/repl-progress-bar-builds
repl: use `nix build` for building instead of `nix-store -r`
2018-07-06 21:36:37 +02:00
volth 1515c65616 prim_concatMap: no need to force value 2018-07-05 15:33:33 +00:00
volth e6bf1a79d7 prim_mapAttrs: must be lazy to avoid infinite recursion 2018-07-05 15:33:12 +00:00
volth 841747b0e6
prim_concatMap: allocate intermediate list on stack 2018-07-05 12:37:37 +00:00
volth ee218f99ca
primops.cc: fix comment 2018-07-05 11:58:15 +00:00
volth 403a76a18f lib.concatMap and lib.mapAttrs to be builtins 2018-07-05 02:54:09 +00:00
Eelco Dolstra e388739098
nix-daemon: Respect --store
For example, this allows you to do run nix-daemon as a non-privileged
user:

  eelco$ NIX_STATE_DIR=~/my-nix/nix/var nix-daemon --store ~/my-nix/

The NIX_STATE_DIR is still needed because settings.nixDaemonSocketFile
is not derived from settings.storeUri (and we can't derive it from the
store's state directory because we don't want to open the store in the
parent process).
2018-07-03 19:52:32 +02:00
Eelco Dolstra ea3c9dab5f
Include cpptoml for build simplicity 2018-07-03 18:39:36 +02:00
Eelco Dolstra 3b1f54cf06
Add a fromTOML primop
This is primarily useful for processing Cargo.lock files.
2018-07-03 18:39:36 +02:00
Eelco Dolstra a92ed973e5
Store floating point numbers in double precision
Even on 32-bit systems, Value has enough space to hold a double.
2018-07-03 18:39:32 +02:00
Maximilian Bosch 8a98ffc276
nix why-depends: render output into $PAGER 2018-07-03 12:49:12 +02:00
Eelco Dolstra dd98683e65
Merge pull request #2158 from Ma27/improve-search-ux
search.cc: improve UX for `nix search`
2018-07-02 11:16:21 +02:00
Maximilian Bosch 6b74fdac27
search.cc: improve UX for nix search
As proposed in #1634 the `nix search` command could use some
improvements. Initially 0413aeb35d added
some basic sorting behavior using `std::map`, a next step would be an
improvement of the output.

This patch includes the following changes:

* Use `$PAGER` for outputs with `RunPager` from `shared.hh`:
  The same behavior is defined for `nix-env --query`, furthermore it
  makes searching huge results way easier.

* Simplified result blocks:
  The new output is heavily inspired by the output from `nox`, the first
  line shows the attribute path and the derivaiton name
  (`attribute path (derivation name)`) and the description in the second
  line.
2018-07-02 11:05:59 +02:00
Will Dietz 6a24e49ba8 repl: use nix build for building instead of nix-store -r
progress bar!
2018-06-23 07:56:26 -05:00
Will Dietz 44de71a396 progress-bar: re-draw last update if nothing new for 1sec.
Slightly nicer behavior when updates are somewhat far apart
(during a long linking step, perhaps) ensuring things
don't appear unresponsive.

If we wait the maximum amount for the update,
don't bother waiting another 50ms (for rate-limiting purposes)
and just check if we should quit.

This also ensures we'll notice the request to quit within 1s
if quit is signalled but there is not an udpate.
(I'm not sure if this happens or not)
2018-06-18 17:54:09 -05:00
Will Dietz f601bc0492 sync.hh: return cv_status instead of dropping it, in case useful. 2018-06-18 17:31:00 -05:00
Eelco Dolstra b920b90857
Detect redirect loops
Fixes #2225.
2018-06-18 10:36:19 +02:00
Eelco Dolstra 2e244fb68f
Merge pull request #2234 from symphorien/esrch2
libstore/gc.cc: ignore ESRCH when reading symlinks in /proc
2018-06-18 09:01:55 +02:00
Symphorien Gibol ef8122064b libstore/gc.cc: ignore ESRCH when reading symlinks in /proc
readlink is also affected by the problem fixed for regular files in
c567afe355df
2018-06-16 18:04:09 +02:00
Matthew Justin Bauer 6d09e4400c
Fix optimise-store.cc to skip more paths on macOS
I hate to make this such a large check but the lack of documentation means we really have no idea what's allowed. All of them reported so far have been within ".app/Contents" directories. That appears to be a safe starting point. However, I would not be surprised to also find more paths that are disallowed for instance in .framework or .bundle directories.

Fixes #2031
Fixes #2229
2018-06-14 12:34:55 -04:00
Eelco Dolstra 85530bcc0b
nix: Remove special handling of .drv files
This makes 'nix copy' and 'nix path-info' work on .drv store
paths. Removing special treatment of .drv files seems the most
future-proof approach given the possible removal of .drv files in the
future.

Note that 'nix build' will still build (rather than substitute) .drv
paths due to the unfortunate overloading in Store::buildPaths().
2018-06-13 18:04:22 +02:00
Eelco Dolstra 7902fccf6f
hashDerivationModulo(): Handle chroot stores
This makes 'nix copy --to /mnt /nix/store/bla.drv' work.
2018-06-13 18:00:54 +02:00
Eelco Dolstra 3c5d9f478d
std::random_shuffle -> std::shuffle
The former is removed in C++17.
2018-06-13 16:56:19 +02:00
Eelco Dolstra 0629601da1
Move EvalState from the stack to the heap
EvalState contains a few counters (e.g. nrValues) that increase
quickly enough that they end up being interpreted as pointers by the
garbage collector. Moving it to the heap makes them invisible to the
garbage collector.

This reduces the max RSS doing 100 evaluations of
nixos.tests.firefox.x86_64-linux.drvPath from 455 MiB to 292 MiB.

Note: ideally, allocations would be much further up in the 64-bit
address space to reduce the odds of an integer being misinterpreted as
a pointer. Maybe we can use some linker magic to move the .bss segment
to a higher address.
2018-06-12 17:49:55 +02:00
Eelco Dolstra c905d8b0a8
GC_malloc -> GC_MALLOC
This makes it possible to build with -DGC_DEBUG.
2018-06-12 17:49:55 +02:00
Eelco Dolstra 455d1f01d0
Don't scan for roots in dynamic libraries
This reduces the risk of object liveness misdetection. For example,
Glibc has an internal variable "mp_" that often points to a Boehm
object, keeping it alive unnecessarily. Since we don't store any
actual roots in global variables, we can just disable data segment
scanning.

With this, the max RSS doing 100 evaluations of
nixos.tests.firefox.x86_64-linux.drvPath went from 718 MiB to 455 MiB.
2018-06-12 17:49:55 +02:00
Eelco Dolstra 30964103dc
Add temporary stats 2018-06-12 17:49:55 +02:00
Eelco Dolstra 24c6806994
Cache parse trees
This prevents EvalState::resetFileCache() from parsing everything all
over again.
2018-06-12 17:49:55 +02:00
Eelco Dolstra 6ad0a2f749
Remove duplicate definition of allocBytes() 2018-06-12 17:49:51 +02:00
Yorick 72a78beb34
Fix #2162: use getaddrinfo instead of curl to preload NSS 2018-06-12 13:05:14 +02:00
Symphorien Gibol 8c567afe35 libstore/gc.cc: ignore ESRCH when reading /proc
If a process disappears between the time /proc/[pid]/maps is opened and
the time it is read, the read() syscall will return ESRCH. This should be ignored.
2018-06-11 16:29:47 +02:00
Will Dietz 6bf7dc1baf download: if there are active requests, never sleep for 10s
In some versions/configurations libcurl doesn't handle timeouts
(especially DNS timeouts) in a way that wakes curl_multi_wait.

This doesn't appear to be a problem if using c-ares, FWIW.
2018-06-08 21:12:43 -05:00
AmineChikhaoui 170165ee5a
use debug instead of printTalkative to avoid spamming hydra queue
runner logs.
2018-06-08 11:43:32 +01:00
Eelco Dolstra 4ac4f675df
Don't require --fallback to recover from disappeared binary cache NARs 2018-06-05 16:04:41 +02:00
Eelco Dolstra 691b7582c7
Handle HTTP 410 Gone errors 2018-06-05 16:03:32 +02:00
Eelco Dolstra ca06a9cea7
Improve binary cache fallback test 2018-06-05 15:26:12 +02:00
Eelco Dolstra a2ec7a3bfd
Further improve upload messages 2018-06-05 14:37:26 +02:00
Will Dietz 54b1c59643 download.cc: fix readCallback return type (int -> size_t)
I'm not sure if curl ever asks for enough data at once
for truncation to occur but better safe than sorry.
2018-06-01 10:59:26 -05:00
Eelco Dolstra 73357500ac
Improve binary cache upload messages
Don't say "download" when we mean "upload".
2018-06-01 14:14:22 +02:00
Peter Simons 93aa3bea2e
Merge pull request #767 from mogorman/garbage_collect_keep_last_few
Implement --delete-generations + flag for keeping last N number of gens
2018-05-31 10:00:21 +02:00
Eelco Dolstra 5d4a9d5677
Fix static assertion failure on 32-bit systems 2018-05-30 17:47:30 +02:00
Eelco Dolstra 3c2de9830d
RemoteStore::addToStore(): Ensure capacity is decreased on exceptions 2018-05-30 13:48:31 +02:00
Eelco Dolstra a4c1618876
Fix deadlock in RemoteStore::queryPathInfoUncached() 2018-05-30 13:46:14 +02:00
Eelco Dolstra 7d21863bb3
Make <nix/fetchurl.nix> run in constant memory
E.g.

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv"; }'

now runs in 17 MiB (was 70 MiB), while

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0d2fxljdih3nc5dqx41hjzic3141ajil94m8kdbpryq569dpsbvb"; unpack = true; }'

runs in 17 MiB (was 346 MiB).
2018-05-30 13:42:51 +02:00
Eelco Dolstra 5a654fd7dd
getDownloader(): Simplify 2018-05-30 13:42:30 +02:00
Eelco Dolstra e87e4a60d6
Make HttpBinaryCacheStore::narFromPath() run in constant memory
This reduces memory consumption of

  nix copy --from https://cache.nixos.org --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 176 MiB to 82 MiB. (The remaining memory is probably due to xz
decompression overhead.)

Issue https://github.com/NixOS/nix/issues/1681.
Issue https://github.com/NixOS/nix/issues/1969.
2018-05-30 13:42:29 +02:00
Eelco Dolstra 08ec757726
Make LocalBinaryCacheStore::narFromPath() run in constant memory
This reduces memory consumption of

  nix copy --from file://... --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 514 MiB to 18 MiB for an uncompressed binary cache, and from 192
MiB to 53 MiB for a bzipped binary cache. It may also be faster
because fetching can happen concurrently with decompression/writing.

Continuation of 48662d151b.

Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30 13:34:57 +02:00
Eelco Dolstra 81ea8bd5ce
Simplify the callback mechanism 2018-05-30 13:34:37 +02:00
Eelco Dolstra 1672bcd230
Move evaluator-specific settings out of libstore 2018-05-30 13:29:50 +02:00
Eelco Dolstra c1d445ecec
Move some Download-specific settings to download.cc 2018-05-30 13:29:50 +02:00
Eelco Dolstra 737ed88f35
Modularize config settings
Allow global config settings to be defined in multiple Config
classes. For example, this means that libutil can have settings and
evaluator settings can be moved out of libstore. The Config classes
are registered in a new GlobalConfig class to which config files
etc. are applied.

Relevant to https://github.com/NixOS/nix/issues/2009 in that it
removes the need for ad hoc handling of useCaseHack, which was the
underlying cause of that issue.
2018-05-30 13:28:01 +02:00
Eelco Dolstra e606cd412f
Fix assertion failure in storePathToHash()
Fixes https://github.com/NixOS/nix/issues/2015.
2018-05-30 13:27:19 +02:00
Eelco Dolstra 6185d25e52
Make 'nix copy --to daemon' run in constant memory (daemon side)
Continuation of 97002b684c. This makes
the daemon use constant memory. For example, it reduces the daemon's
maximum RSS on

  $ nix copy --from ~/my-nix --to daemon /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 264 MiB to 7 MiB.

We now use a TunnelSource to prevent the connection from ending up in
an undefined state if an exception is thrown while the NAR is being
sent.

Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30 13:26:48 +02:00
Eelco Dolstra 23d6bb583a
LocalStore::addToStore: Throw an exception if no hash is given 2018-05-30 13:25:40 +02:00
Eelco Dolstra 3cab639e21
Use extended SQLite error codes 2018-05-30 13:25:03 +02:00
Eelco Dolstra bd56b5fe3f
Merge pull request #2187 from bgamari/stoi-exceptions
json-to-value: Use strtol instead of strtoi
2018-05-28 14:54:49 +02:00
Ben Gamari 7f560b81ea json-to-value: Use strtol instead of strtoi
NixInt is long, so strtoi is too restrictive.
2018-05-26 18:43:46 -04:00
Will Dietz 93ae90de0f eval.cc: add message to static_assert, message can be omitted w/c++17 2018-05-26 14:04:41 -05:00
Will Dietz 14c464b6c8 nix-daemon: remove unused "pendingMsgs" variable 2018-05-26 13:09:55 -05:00
Will Dietz ca9c6cb95d libstore/download: remove unused function "getTime()" 2018-05-25 19:49:45 -05:00
Eelco Dolstra 743359bc8a
Merge pull request #2157 from volth/bitwise
add builtins: __bitAnd,  __bitOr,  __bitXor
2018-05-24 15:00:39 +02:00
Eelco Dolstra 4bb8741b98
Make Env self-describing
If the Env denotes a 'with', then values[0] may be an Expr* cast to a
Value*. For code that generically traverses Values/Envs, it's useful
to know this.
2018-05-22 16:02:32 +02:00
Eelco Dolstra 9fd7cf98db
Memoise checkSourcePath()
This prevents hydra-eval-jobs from statting the same files over and
over again.
2018-05-22 13:02:14 +02:00
Will Dietz b08923b4a0 serialise: fix buffer size used, hide method for internal use only
Fixes #2169.
2018-05-21 17:30:38 -05:00
volth 6cc28c0589
add `mod' and bitwise builtins: camel-case function names 2018-05-16 10:52:19 +00:00
volth f3c090f91c
add mod' and bitwise builtins: remove mod' and shifts 2018-05-16 06:57:11 +00:00
volth 49b7cf1813
add `mod' and bitwise builtins: remove infix functions 2018-05-16 06:55:24 +00:00
Eelco Dolstra 966407bcf1
Merge pull request #2145 from serokell/uncached-channel
make sure not to use cached channels for nix-channel --update
2018-05-15 14:57:18 +02:00
volth 8a6a14e1f5 add `mod' and bitwise builtins 2018-05-12 16:50:39 +00:00
Eelco Dolstra 1ad19232c4
Don't return negative numbers from the flex tokenizer
Fixes #1374.
Closes #2129.
2018-05-11 12:05:12 +02:00
Eelco Dolstra f3c85f9eb3
Revert "Throw a specific error for incomplete parse errors."
This reverts commit 6498adb002. We don't
actually use IncompleteParseError in 'nix repl'.
2018-05-11 11:40:50 +02:00
Eelco Dolstra 2b3c1b3a88
Remove mention of default since we already know the default 2018-05-10 12:06:28 +02:00
Eelco Dolstra 38def17627
Merge branch 's3-multipart-uploads' of https://github.com/AmineChikhaoui/nix 2018-05-10 12:06:02 +02:00
Yorick b9289e4855
make sure not to use cached channels for nix-channel --update
fixes #1964
2018-05-09 16:18:20 +02:00
Eelco Dolstra a91c4ca01f
In restricted eval mode, allow access to the closure of store paths
E.g. this makes

  nix eval --restrict-eval -I /nix/store/foo '(builtins.readFile "/nix/store/foo/symlink/bla")'

(where /nix/store/foo/symlink is a symlink to another path in the
closure of /nix/store/foo) succeed.

This fixes a regression in Hydra compared to Nix 1.x (where there were
no restrictions at all on access to the Nix store).
2018-05-09 15:45:05 +02:00
AmineChikhaoui 854c0860f4
share the executor between multiple copy threads. 2018-05-07 15:07:00 +01:00
AmineChikhaoui 591e75cd01
add a FIXME note to find a way to abort the multipart uploads in
case the nix command is interrupted.
2018-05-07 14:27:53 +01:00
AmineChikhaoui 129394fb95
Support multi-part uploads for large NARs that exceed the size of
5Gb.
2018-05-07 14:23:51 +01:00
Eelco Dolstra a1adcdf087
nix-shell: Only download requested outputs
Fixes #2116.
2018-05-07 14:58:25 +02:00
Eelco Dolstra c717d8e3bd
Don't set a CA assertion for paths with references
Really fixes #2133.
2018-05-03 14:49:56 +02:00
Eelco Dolstra 80a7b16593
LocalStore::addValidPath(): Throw error when the CA assertion is wrong
Closes #2133.
2018-05-03 11:02:40 +02:00
Eelco Dolstra 53ec5ac69f
Fix some random -Wconversion warnings 2018-05-02 13:56:34 +02:00
Eelco Dolstra 4a2c948943
Fix bzip2 compression of files > 4 GiB
Bzip2's 'avail_in' parameter is declared as an unsigned int, so
assigning a size_t length to it led to silent truncation.

Fixes #2111.
2018-05-02 12:54:30 +02:00
Will Dietz 8e7d77d494 globals.hh: don't use '==' to compare string literals
Saw this in logs, also reported here:

8e6108ff71 (r28707288)
2018-04-23 20:57:05 -05:00
Shea Levy 8e6108ff71
Merge branch 'aarch64-armv7' of git://github.com/lheckemann/nix
Support extra compatible architectures (#1916)
2018-04-23 08:48:22 -04:00
Shea Levy e2b028353b
Merge branch 'pos-crash-fix' of git://github.com/dezgeg/nix 2018-04-23 08:43:54 -04:00
Shea Levy 58f9ae0dcd
Merge branch 'hash-mismatch' of git://github.com/LnL7/nix 2018-04-23 08:40:34 -04:00
Andrew Dunham 3a918014b2 Fix library ordering in Makefiles
The existing ordering linked `libutil` before `libstore`, which causes
link failures when building statically. This is due to `libstore` using
functions from `libutil`, and the fact that symbol resolution works
"forward" - i.e. if you pass `-lfoo -lbar -lbaz`, any symbols that
`libbar` uses from `libbaz` will be resolved, but symbols from `libfoo`
will not since it comes first in the command line.

All this to say: this commit reorders the libraries which fixes the link
errors.
2018-04-21 21:10:52 -07:00
Peter Simons 040acdcee2
Merge pull request #1707 from dtzWill/fix/issue-1692
ignore "interrupted" exception in progress callback
2018-04-21 16:51:19 +02:00
Daiderd Jordan 13d4d1c0a2
download: make hash mismatch error consistent with fetchurl 2018-04-20 21:07:32 +02:00
Will Dietz 6d9129014d build.cc: fix bind-mount of /dev/{pts,ptmx} fallback
Don't bind-mount these to themselves,
mount them into the chroot directory.

Fixes pty issues when using sandbox on CentOS 7.4.
(build of perlPackages.IOTty fails before this change)
2018-04-20 12:24:23 -05:00
Daniel Poelzleithner b712d4674b Allow multiple search experssions in nix search
The common use case is to search for packages containing multiple words
like a "git" "frontend". Having only one expressions makes this simple regular
use case very complicated. Instead, search accepts multiple regular epressions
which all need to match.

nix search git 'gui|frontend'

returns a list of all git uis for example
2018-04-18 21:08:35 +02:00
Félix Baylac-Jacqué 00584bb091
parser: Allow mixed nested and top-level attrpaths
Fixes #2077.
2018-04-18 18:44:38 +02:00
Shea Levy b37f5ae31d
isFunction: True on primops.
Fixes #2073
2018-04-17 14:33:12 -04:00
Eelco Dolstra a4aac7f88c
Handle arguments in $EDITOR
Fixes #2079.
2018-04-17 12:16:04 +02:00
Eelco Dolstra c74f838620
exportReferences: Check whether the path is in the input closure
Fixes #2082.
2018-04-17 12:03:27 +02:00
Félix Baylac-Jacqué a91fb422fe
nix repl: Fix multiline SIGINT handling.
Fixes #2076
2018-04-16 16:09:30 +02:00
Eelco Dolstra d34fa2bcc3
Fix #1921 2018-04-13 15:42:35 +02:00
Shea Levy dc0a542c9f
initPlugins: Fix dlopen error message. 2018-04-11 21:02:50 -04:00
Eelco Dolstra 1839a5542a
Fix #2057 2018-04-11 11:42:17 +02:00
Eelco Dolstra d6aaa35478
Fix perl-bindings
https://hydra.nixos.org/build/72636781
2018-04-10 13:11:43 +02:00
Shea Levy a38fe5c1a8
Export required C++ version in pkgconfig. 2018-04-09 11:32:43 -04:00
Shea Levy 7459388448
Merge branch 'fix/ruby-shebang' of git://github.com/samueldr/nix 2018-04-09 10:54:39 -04:00
Shea Levy fd98fca7bb
Make prim_exec and prim_importNative available to plugins 2018-04-09 10:26:50 -04:00
Shea Levy 346c0ac361
nix-daemon: Exit successfully when interrupted.
Fixes #2058.
2018-04-09 08:05:54 -04:00
Samuel Dionne-Riel 399f43c3d5 nix-shell: Fixes use with ruby shebangs.
The ported code in 80ebc553ec was incorrectly ported.

```
-            $envCommand = "exec $execArgs $interpreter -e 'load(\"$script\")' -- ${\(join ' ', (map shellEscape, @savedArgs))}";
...
+                    envCommand = (format("exec %1% %2% -e 'load(\"%3%\") -- %4%") % execArgs % interpreter % script % joined.str()).str();
```

The single-quote finishing the small ruby snippet was lost in
translation.
2018-04-07 22:38:45 -04:00
AmineChikhaoui 62d75ad3e1
rename the options to mention it's a narinfo TTL as disk cache is used all over the place for other operations 2018-04-06 11:05:15 +01:00
AmineChikhaoui 86930ed414
add documentation for the local disk cache TTL config 2018-04-06 10:51:18 +01:00
AmineChikhaoui 2855c3d965
Make the TTL for disk cache configurable, we can now completely disable
disk cache lookup for example by doing:
  nix copy --from <binary-cahe> <store-path> --option \
  positive-disk-cache-ttl 0

Issues: #1885 #2035
2018-04-06 10:51:18 +01:00
Eelco Dolstra 1d9742b95f
<nix/buildenv.nix>: Ignore dangling symlinks
Fixes

  error: getting status of '/nix/store/j8p0vv89k1pf0cn7kmfsdcs7bshwga1i-firefox-52.7.2esr/share/icons/hicolor/48x48/apps/firefox.png': No such file or directory

https://github.com/NixOS/nix/issues/1934

Also improve error message on directory/non-directory collisions.
2018-04-05 20:04:42 +02:00
Eelco Dolstra 5c904e10eb
<nix/buildenv.nix>: Skip non-directory top-level paths
Fixes https://github.com/NixOS/nix/issues/1934 (at least the "error:
opening directory '/nix/store/...-linux-config-4.4.14': Not a
directory" issue).
2018-04-05 19:34:56 +02:00
Tuomas Tynkkynen af86132e1a libexpr: Make unsafeGetAttrPos not crash on noPos
Currently e.g. `builtins.unsafeGetAttrPos "abort" builtins` will
eventually segfault because pos->file is an unset Symbol.

Found by afl-fuzz.
2018-04-03 15:54:42 +03:00
Jonas Chevalier 865ca2402f nix copy: add an example with S3
I couldn't find a good example how to use it with non-us-east-1 buckets.
2018-03-30 02:31:14 +01:00
Eelco Dolstra 446bb88f13
Remove reference to non-existent manpages 2018-03-29 12:29:10 +02:00
Eelco Dolstra 2b61c74922
Process --option use-case-hack properly
Fixes https://github.com/NixOS/nix/issues/2009.
2018-03-29 12:28:29 +02:00
Eelco Dolstra 9d1220a01d
ValidPathInfo::isContentAddressed(): Ensure there are no references 2018-03-29 12:27:42 +02:00
Will Dietz 74da813912 download: improve error for hash mismatch ("store mismatch")
Fixes #1905
2018-03-22 15:22:06 -05:00
Will Dietz 59b32403f2 download.cc: fix path for cached files 2018-03-22 15:05:46 -05:00
Eelco Dolstra 28eb9621cc
Fix build 2018-03-22 13:46:19 +01:00
Eelco Dolstra dce8af59a6
Tweak 2018-03-22 13:32:18 +01:00
Eelco Dolstra f87e286e82
Merge branch 'fix/avoid-large-stack-buffers' of https://github.com/dtzWill/nix 2018-03-22 13:19:25 +01:00
Shea Levy c24e0439b2
Merge branch 'fix/execl-null-sentinel' of git://github.com/dtzWill/nix 2018-03-21 23:25:53 -04:00
Eelco Dolstra ec91840e70 Also make the backwards compatible case of RemoteStore::addToStore run in constant memory
Just because we can.
2018-03-21 23:46:03 +01:00
Eelco Dolstra 97002b684c Make 'nix copy --to daemon' run in constant memory 2018-03-21 23:42:52 +01:00
Will Dietz f66ae22288 shared.cc: style: prefer "nullptr" over '(char *)NULL' 2018-03-21 17:17:37 -05:00
Eelco Dolstra 11898d6a37 Make 'nix copy --to ssh://...' run in constant memory 2018-03-21 23:13:06 +01:00
Eelco Dolstra 47f7e5585b Make 'nix copy --from ssh://...' run in constant memory
For instance, this reduced the memory consumption of

  $ nix copy --from ssh://localhost --to ~/my-nix /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 632 MiB to 16 MiB.
2018-03-21 23:10:40 +01:00
Eelco Dolstra 8615cfb130
LegacySSHStore: Allow overriding the path to nix-store 2018-03-21 15:10:46 +01:00
Will Dietz 02d36fdab7 execl: cast NULL sentinel to (char *), per man page and compiler warning
From exec(3):

> The list of arguments must be terminated by a null pointer, and, since these
> are variadic functions, this pointer must be cast (char *) NULL
2018-03-20 13:31:58 -05:00
Eelco Dolstra 03d8136b02
Remove unused channel-cache directory 2018-03-20 18:22:49 +01:00
Eelco Dolstra e0c1597910
Style fix 2018-03-20 17:33:45 +01:00
Eelco Dolstra 668ac3ea2c
Make <nix/buildenv.nix> a builtin builder
This avoids sandbox annoyances.
2018-03-20 17:28:09 +01:00
Eelco Dolstra 9d40787938
Move builtinFetchurl to its own file 2018-03-20 17:01:36 +01:00
Eelco Dolstra 5df427f35b
Slight simplification 2018-03-20 16:32:59 +01:00
Eelco Dolstra 21ebf6a160
SSHMaster: Optionally pass -v to ssh 2018-03-20 15:30:55 +01:00
Eelco Dolstra 1aca195e52
Hack to get SSH error messages from build-remote
E.g.

  cannot build on 'ssh://mac1': cannot connect to 'mac1': bash: nix-store: command not found
  cannot build on 'ssh://mac2': cannot connect to 'mac2': Host key verification failed.
  cannot build on 'ssh://mac3': cannot connect to 'mac3': Received disconnect from 213... port 6001:2: Too many authentication failures
  Authentication failed.
2018-03-20 15:25:26 +01:00
Will Dietz f5a2136a40 serialise.cc: remove pessimising move
from clang6:

src/libutil/serialise.cc:189:23: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
2018-03-19 19:18:47 -05:00
Will Dietz 9b33201e72 util: rename stdout/stdin members to avoid conflicts w/standard macro
(cherry picked from commit c389a7fb617ed7bcd617efa68c6a48c00405310d)
2018-03-19 11:18:56 -05:00
Eelco Dolstra 138af2e554
Shut up signedness warning 2018-03-19 11:57:15 +01:00
Linus Heckemann 639c166647 build-remote: take extra-platforms into account 2018-03-16 22:50:57 +00:00
Linus Heckemann 637701b604 rename build-extra-platforms -> extra-platforms
also document it
2018-03-16 22:50:27 +00:00
Linus Heckemann 12913ccf45 nix-shell: allow symlinks to .drvs
This makes persistent shell environments easier to use.
2018-03-16 22:15:24 +00:00
Eelco Dolstra 48662d151b
Reduce substitution memory consumption
copyStorePath() now pipes the output of srcStore->narFromPath()
directly into dstStore->addToStore(). The sink used by the former is
converted into a source usable by the latter using
boost::coroutine2. This is based on [1].

This reduces the maximum resident size of

  $ nix build --store ~/my-nix/ /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1 --substituters file:///tmp/binary-cache-xz/ --no-require-sigs

from 418592 KiB to 53416 KiB. (The previous commit also reduced the
runtime from ~4.2s to ~3.4s, not sure why.) A further improvement will
be to download files into a Sink.

[1] https://github.com/NixOS/nix/compare/master...Mathnerd314:dump-fix-coroutine#diff-dcbcac55a634031f9cc73707da6e4b18

Issue #1969.
2018-03-16 20:35:59 +01:00
Eelco Dolstra 3e6b194d78
decompress(): Use a Source and Sink
This allows decompression to happen in O(1) memory.
2018-03-16 20:35:59 +01:00
Eelco Dolstra 64441f0551
Merge pull request #1939 from dezgeg/lexer-fix
libexpr: Recognize newline in more places in lexer
2018-03-16 12:14:58 +01:00
Eelco Dolstra eb75bc5afb
Don't retry on CURLE_SSL_CACERT_BADFILE
The certificates won't get any better if we retry.
2018-03-16 12:03:18 +01:00
Eelco Dolstra 4868721506
Filter ANSI colors when not writing to a terminal
Fixes https://github.com/NixOS/nixpkgs/issues/37114.
2018-03-15 16:08:07 +01:00
Eelco Dolstra 7b8914825a
Merge pull request #1965 from masaeedu/master
Wrap thread local in function for Cygwin
2018-03-15 14:56:21 +01:00
Shea Levy cfdbfa6b2c
Catch more possible instances of passing NULL to memcpy.
Actually fixes #1976.
2018-03-14 23:53:43 -04:00
Shea Levy d25d9f7cec
Merge branch 'fix/errno-sighandler' of git://github.com/dtzWill/nix 2018-03-14 23:45:35 -04:00
Shea Levy e2088febf3
concatLists: Don't pass NULL pointers to memcpy.
This is UB, even if the size is 0. See #1976.

Fixes #1976.
2018-03-14 23:44:02 -04:00
Will Dietz d98755b0c2 nix-daemon: preserve errno in signal handler (thanks tsan) 2018-03-14 21:39:46 -05:00
Eelco Dolstra 55aa622fb1
Fix compatibility with latest boost::format 2018-03-14 19:25:09 +01:00
Eelco Dolstra ca14b14200
Use boost::format from the boost package
Note that this only requires headers from boost so it doesn't add a
runtime dependency.

Also, use Nixpkgs 18.03.
2018-03-14 19:24:04 +01:00
Shea Levy 56f2ed0081
Merge pull request #1906 from dtzWill/fix/nix-search
nix search: tests and fix #1893 and part of #1892
2018-03-13 18:43:47 -06:00
Eelco Dolstra 7afdc8d4a1
build-remote: Don't substitute the build result 2018-03-13 17:56:51 +01:00
Eelco Dolstra 6e60141a80
Typo 2018-03-13 17:48:09 +01:00
Guillaume Maudoux 80735c4cc9
fetchGit: Fix debug message 2018-03-13 10:28:23 +01:00
Asad Saeeduddin be54f4a0b6 Wrap thread local in function for Cygwin
Fixes #1826. See #1352 for a previous instance of a similar change.
2018-03-12 00:56:41 -04:00
Eelco Dolstra 24b739817f
Fix double free in Store::queryPathInfo()
It was holding on to a Value* (i.e. a std::shared_ptr<ValidPathInfo>*)
outside of the pathInfoCache lock, so the std::shared_ptr could be
destroyed between the release of the lock and the decrement of the
std::shared_ptr refcount. This can happen if more than
'path-info-cache-size' paths are added in the meantime, *or* if
clearPathInfoCache() is called. The hydra-queue-runner queue monitor
thread periodically calls the later, so is likely to trigger a crash.

Fixes https://github.com/NixOS/hydra/issues/542.
2018-03-09 14:19:51 +01:00
Tim Engler 383afab9ad Modified MakeBinOp to no longer produce its name using concatenation and "##".
Doing so prevents emacs tags from working, as well as makes the code extremely
confusing for a newbie.

In the prior state, if someone wants to find the definition of "ExprApp" for
example, a grep through the code reveals nothing. Since the definition could be
hiding in numerous ".h" files, it's really difficult to find. This personally
took me several hours to figure out.
2018-03-09 03:16:33 +00:00
Eelco Dolstra 16551f54c9
Fix #1957 2018-03-07 11:40:50 +01:00
Eelco Dolstra 4452f6e855
Merge pull request #1955 from dtzWill/fix/rand-random
rand() -> random(), since we use srandom().
2018-03-07 11:00:08 +01:00
Will Dietz 56253bb08f rand() -> random(), since we use srandom().
rand() requires we call srand() instead,
but might as well use random().
2018-03-06 17:44:25 -06:00
Will Dietz e917c05628 logging: Don't throw from Activity destructor
Move definition of destructor to logging.cc for
access to util.hh's ignoreException.
2018-03-06 10:42:58 -06:00
Shea Levy 088ef81759
ssh-ng: Don't forward options to the daemon.
This can be iterated on and currently leaves out settings we know we
want to forward, but it fixes #1713 and fixes #1935 and isn't
fundamentally broken like the status quo. Future changes are suggested
in a comment.
2018-03-05 07:42:15 -05:00
Will Dietz 6b9a03f5d8 hoist vector out of loop just in case 2018-03-02 10:52:05 -06:00
Will Dietz c89a3d5368 don't allocate large buffers on the stack 2018-03-02 10:52:04 -06:00
Shea Levy 3748a0ca1e
Merge branch 'improve-search-algorithm' of git://github.com/Ma27/nix 2018-03-02 11:41:08 -05:00
Shea Levy aa8bbbf69d
Merge branch 'write-failure-fixes' of git://github.com/lheckemann/nix 2018-03-02 10:59:59 -05:00
Tuomas Tynkkynen a0e38c16bc libexpr: Recognize newline in more places in lexer
Flex's regexes have an annoying feature: the dot matches everything
except a newline. This causes problems for expressions like:

"${0}\
"

where the backslash-newline combination matches this rule instead of the
intended one mentioned in the comment:

    <STRING>\$|\\|\$\\ {
                    /* This can only occur when we reach EOF, otherwise the above
                    (...|\$[^\{\"\\]|\\.|\$\\.)+ would have triggered.
                    This is technically invalid, but we leave the problem to the
                    parser who fails with exact location. */
                    return STR;
                }
However, the parser actually accepts the resulting token sequence
('"' DOLLAR_CURLY 0 '}' STR '"'), which is a problem because the lexer
rule didn't assign anything to yylval. Ultimately this leads to a crash
when dereferencing a NULL pointer in ExprConcatStrings::bindVars().

The fix does change the syntax of the language in some corner cases
but I think it's only turning previously invalid (or crashing) syntax
to valid syntax. E.g.

"a\
b"

and

''a''\
b''

were previously syntax errors but now both result in "a\nb".

Found by afl-fuzz.
2018-03-02 17:30:48 +02:00
Eelco Dolstra 939cf4cceb
Fix error message 2018-03-02 14:32:00 +01:00
Matthew O'Gorman 467fdd8ca4 only delete things older than current gen and update logic in doc as
well
2018-03-01 22:59:00 -05:00
Matthew O'Gorman 3c16044cb0 remove unused variable and make sure to check that the current
generation is not the one we are deleting
2018-03-01 22:22:02 -05:00
Matt O'Gorman 12fe2249e1 Update nix-env.cc
missing comma
2018-03-01 21:47:57 -05:00
Matthew O'Gorman 0312d30315 this updates issues that were addressed by people in pr 2018-03-01 21:47:57 -05:00