Commit graph

6221 commits

Author SHA1 Message Date
Eelco Dolstra dea18ff999 nix dev-shell: Execute shellHook 2019-05-02 21:13:19 +02:00
Eelco Dolstra 2919c496ea nix dev-shell: Use 'provides.devShell' by default
Thus

  $ nix dev-shell

will now build the 'provides.devShell' attribute from the flake in the
current directory. If it doesn't exist, it falls back to
'provides.defaultPackage'.
2019-05-02 21:10:13 +02:00
Eelco Dolstra 7dcf5b011a Add function for quoting strings 2019-05-02 21:09:52 +02:00
Eelco Dolstra 2b8c63f303 Add 'nix dev-shell' and 'nix print-dev-env' command
'nix dev-shell' is intended to replace nix-shell. It supports flakes,
e.g.

  $ nix dev-shell nixpkgs:hello

starts a bash shell providing an environment for building 'hello'.

Like Lorri (and unlike nix-shell), it computes the build environment
by building a modified top-level derivation that writes the
environment after running $stdenv/setup to $out and exits. This
provides some caching, so it's faster than nix-shell in some cases
(especially for packages with lots of dependencies, where the setup
script takes a long time).

There also is a command 'nix print-dev-env' that prints out shell code
for setting up the build environment in an existing shell, e.g.

  $ . <(nix print-dev-env nixpkgs:hello)

https://github.com/tweag/nix/issues/21
2019-05-02 20:22:14 +02:00
Nick Van den Broeck e0d4aa75fc Fixed compile errors 2019-05-02 08:40:00 +02:00
Eelco Dolstra 5d6e8c008b Allow 'dir' parameter in github: URIs
E.g. 'github:edolstra/dwarffs/flake?dir=foo/bar'.
2019-05-01 20:44:30 +02:00
Eelco Dolstra a37436d792 Accept empty directories 2019-05-01 20:44:30 +02:00
Eelco Dolstra fa88f71520 Validate 'dir=' parameters
We reject any path element starting with a '.' (mostly to reject '.'
and '..').
2019-05-01 20:44:30 +02:00
Eelco Dolstra ab9e47284a Improve error message 2019-05-01 20:44:30 +02:00
Nick Van den Broeck 43408d3cd6 flake.lock now uses flakeRef.subdir 2019-05-01 16:24:33 +02:00
Eelco Dolstra 00db8d4549 Support 'dir' parameters in https and ssh flake URIs 2019-05-01 14:24:22 +02:00
Nick Van den Broeck 9b3069a88c Fuzzymatching
Fixed issue #61
2019-05-01 12:59:35 +02:00
Nick Van den Broeck eba85e2367 WIP: still need to adapt flakeref parsing 2019-05-01 12:59:12 +02:00
Samuel Dionne-Riel cbc7d9a412 findRootsNoTemp: fixes comment about findRuntimeRoots
The NIX_ROOT_FINDER environment variable was removed in
3c46fe62b8 when porting from perl to C.
2019-04-30 22:43:24 -04:00
Eelco Dolstra 4588a6ff3c Merge remote-tracking branch 'tweag/flakeFlags' into flakes 2019-04-30 12:46:54 +02:00
Nick Van den Broeck 24b35bf9e7 Fixed issue #13 2019-04-30 11:26:45 +02:00
Nick Van den Broeck 35d1c95f7f Fix flag registry order 2019-04-30 11:04:31 +02:00
Eelco Dolstra e6109ec765 Merge remote-tracking branch 'tweag/fixFlakeList' into flakes 2019-04-25 14:05:40 +02:00
Nick Van den Broeck d867e1804a Fix printing FlakeList 2019-04-25 11:43:16 +02:00
Nick Van den Broeck e9c42c06ef Fixed lookupFlake bug 2019-04-25 10:50:59 +02:00
Eelco Dolstra 21d5abfc14 Merge remote-tracking branch 'tweag/flake-clone' into flakes 2019-04-24 12:47:35 +02:00
Eelco Dolstra cc51e37ad0 Use "." as the default installable
This makes e.g. "nix build" do something more or less reasonable
(namely, build the default package of the flake in the current
directory).
2019-04-19 16:16:04 +02:00
Eelco Dolstra 4fb594a375 Merge remote-tracking branch 'tweag/resolvedFlake' into flakes 2019-04-19 15:56:17 +02:00
Eelco Dolstra f8a52cc598 nlohmann-json: 3.5.0 -> 3.6.1
https://github.com/nlohmann/json/releases/tag/v3.6.1

This fixes some clang warnings.
2019-04-19 14:53:58 +02:00
Eelco Dolstra 3392f1b778 Shut up clang warning 2019-04-19 14:41:06 +02:00
Nick Van den Broeck e51abb6631 Changed some names 2019-04-19 14:23:35 +02:00
Eelco Dolstra 50ec2bed9e nix flake info: Show revcount 2019-04-19 14:19:46 +02:00
Eelco Dolstra bc259192b4 fetchGit: Return revCount for dirty working trees 2019-04-19 14:15:51 +02:00
Eelco Dolstra 0cbda84f5b exportGit: Don't clone local repositories
This ensures that commands like 'nix flake info /my/nixpkgs' don't
copy a gigabyte of crap to ~/.cache/nix.

Fixes #60.
2019-04-19 14:10:57 +02:00
Eelco Dolstra 160ce18a0e Improve missing flake.nix error message 2019-04-19 11:43:56 +02:00
Eelco Dolstra 6960ee929d Clean up exportGit argument handling 2019-04-19 11:34:23 +02:00
Eelco Dolstra 46cb15df9b Fix assertion failure in FlakeRef::to_string() 2019-04-19 11:16:14 +02:00
Eelco Dolstra 6e4210d8ce Fix assertion failure 2019-04-19 10:58:08 +02:00
Daiderd Jordan b614e0e53d
build: only skip hash rewriting for sandboxing on linux
The sandbox on darwin, and possibly other future platforms, doesn't have
enough isolation to redirect outputs without hash rewriting.
2019-04-17 23:41:23 +02:00
Eelco Dolstra 160b974fb0 Fix mutability check 2019-04-17 13:54:06 +02:00
Nick Van den Broeck b42ba08fc8 Add command flake clone 2019-04-17 13:31:09 +02:00
Eelco Dolstra 939bee06cd Pass a flake to itself as "self" 2019-04-16 16:29:44 +02:00
Eelco Dolstra 3ddb6d1833 Allow refs to start with a digit
E.g. we want to accept "19.03" as a ref.
2019-04-16 16:24:51 +02:00
Eelco Dolstra 260527a90c Use the lock file 2019-04-16 16:18:47 +02:00
Eelco Dolstra 3c28cb1b8f Improve 'nix flake info' a bit
Example:

  $ nix flake info dwarffs
  ID:          dwarffs
  URI:         github:edolstra/dwarffs/a83d182fe3fe528ed6366a5cec3458bcb1a5f6e1
  Description: A filesystem that fetches DWARF debug info from the Internet on demand
  Revision:    a83d182fe3fe528ed6366a5cec3458bcb1a5f6e1
  Path:        /nix/store/grgd14kxxk8q4n503j87mpz48gcqpqw7-source
2019-04-16 16:18:19 +02:00
Eelco Dolstra 3d0e81051f Fix lock file generation
Before:

   "requires": {
        "nixpkgs": {
            "uri": "nixpkgs"
        }
    },

After:

    "requires": {
        "nixpkgs": {
            "uri": "github:edolstra/nixpkgs/f10e8a02eb7fa2b4a070f30cf87f4efcc7f3186d"
        }
    },
2019-04-16 15:43:55 +02:00
Eelco Dolstra 8c4e759efd updateLockFile(): Make sure Git can see flake.lock 2019-04-16 15:11:17 +02:00
Eelco Dolstra cfca793a20 Remove unneeded pureEval flags 2019-04-16 15:06:40 +02:00
Eelco Dolstra 60834492ae Update lock files from InstallableFlake::toValue()
This ensures that the lock file is updated *before* evaluating it, and
that it gets updated for any nix command, not just 'nix build'.

Also, while computing the lock file, allow arbitrary registry lookups,
not just at top-level.

Also, improve some error messages slightly.
2019-04-16 15:02:02 +02:00
Eelco Dolstra 7b312a8762 Pass stuff by reference 2019-04-16 14:27:54 +02:00
Eelco Dolstra e1d73edb10 writeLockFile(): Emit empty objects rather than null 2019-04-16 14:23:10 +02:00
Eelco Dolstra ed9d725392 getFlake(): Use impureIsAllowed
This fixes 'nix build nixpkgs:hello' without --impure.
2019-04-16 14:16:20 +02:00
Eelco Dolstra ba66455636 Improve incremental build 2019-04-16 14:10:05 +02:00
Eelco Dolstra aecf07b1d6 Remove dead function 2019-04-16 14:08:14 +02:00
Eelco Dolstra 035ac44354 Fix makeFlakeValue() 2019-04-16 13:56:08 +02:00
Eelco Dolstra d8fa2fc429 Add FIXME for pureEval 2019-04-16 12:26:17 +02:00
Eelco Dolstra 3a5493bfe8 Merge remote-tracking branch 'tweag/privateGithub' into flakes 2019-04-16 12:23:56 +02:00
Nick Van den Broeck 7587d62d02 Fixed flake pin issues 2019-04-16 08:38:17 +02:00
Nick Van den Broeck b3d33b02e3 Added support for private github repositories 2019-04-16 08:14:46 +02:00
Eelco Dolstra b4e367bf4a FlakeRef::to_string(): Drop the "flake:" prefix
This is unnecessary in most contexts and makes 'nix flake list' output
less readable.
2019-04-15 14:13:10 +02:00
Eelco Dolstra c179f668e5 Slight cleanup 2019-04-15 14:08:18 +02:00
Eelco Dolstra f6d684b5e2 getFlakeRegistries(): Return registries regardless of pureEval
This makes e.g. 'nix flake list' work.
2019-04-15 13:48:56 +02:00
Eelco Dolstra 84c12dbd7c Move --impure to MixEvalArgs 2019-04-15 13:45:51 +02:00
Eelco Dolstra a12cd53567 Merge remote-tracking branch 'tweag/automatedUpdate' into flakes 2019-04-15 12:11:16 +02:00
Eelco Dolstra bb6e6923f2 Add environment variable NIX_SHOW_SYMBOLS for dumping the symbol table 2019-04-11 23:04:13 +02:00
Nick Van den Broeck 4bf3a8226b Automated lockfile updating with nix build 2019-04-11 14:34:04 +02:00
Nick Van den Broeck 4ad4e48668 FlakeRegistry = FlakeRef -> FlakeRef 2019-04-11 07:08:04 +02:00
Nick Van den Broeck c64f98b883 FlakeAlias is implemented 2019-04-09 10:16:02 +02:00
Nick Van den Broeck f39670c631 Took ref and rev out of FlakeRef 2019-04-09 09:51:42 +02:00
Nick Van den Broeck 3ec0c82fab Fixed dependency resolution 2019-04-09 09:51:42 +02:00
Nick Van den Broeck 641db127be FlakeIds are now properly looked up in registries 2019-04-09 09:51:36 +02:00
Nick Van den Broeck 18c019b616 Added nonFlakeRequires and the command nix flake deps 2019-04-09 08:38:09 +02:00
Eelco Dolstra 87033f2c4e Whitespace 2019-04-08 23:58:33 +02:00
Eelco Dolstra c996e04aca Allow relative paths in flakerefs
Also allow "." as an installable to refer to the flake in the current
directory. E.g.

  $ nix build .

will build 'provides.defaultPackage' in the flake in the current
directory.
2019-04-08 23:52:09 +02:00
Eelco Dolstra 507da65900 Move flake template into a separate file 2019-04-08 23:39:38 +02:00
Eelco Dolstra 47727252ff Add "nix flake init" command for creating a flake 2019-04-08 23:36:12 +02:00
Eelco Dolstra ee1254d4f5 nix: Add --impure as a shorter alias of --no-pure-eval 2019-04-08 23:19:19 +02:00
Eelco Dolstra 6a4c7fb975 Add path flakeref variant
Unlike file://<path>, this allows the path to be a dirty Git tree, so

  nix build /path/to/flake:attr

is a convenient way to test building a local flake.
2019-04-08 23:09:18 +02:00
Eelco Dolstra 4023ae4cdf nix: Support nixpkgs.<attrpath> for compatibility 2019-04-08 16:22:04 +02:00
Eelco Dolstra 101d964a59 nix: Make -f work for compatibility 2019-04-08 16:11:17 +02:00
Eelco Dolstra 9b7eac332b Merge remote-tracking branch 'origin/master' into flakes 2019-04-08 14:23:48 +02:00
Eelco Dolstra 154244adc6 nix: New installables syntax
The general syntax for an installable is now
<flakeref>:<attrpath>. The attrpath is relative to the flake's
'provides.packages' or 'provides' if the former doesn't yield a
result. E.g.

  $ nix build nixpkgs:hello

is equivalent to

  $ nix build nixpkgs:packages.hello

Also, '<flakeref>:' can be omitted, in which case it defaults to
'nixpkgs', e.g.

  $ nix build hello
2019-04-08 14:21:13 +02:00
Eelco Dolstra 6b0ca8e803 findAlongAttrPath(): Throw AttrPathNotFound 2019-04-08 14:20:58 +02:00
Eelco Dolstra 41ba5135e0 primeCache(): Barf if builds are needed but not allowed
Fixes #2753.
2019-04-01 21:09:49 +02:00
Eelco Dolstra 2bc6304793 getMachines(): Cache result 2019-04-01 21:09:35 +02:00
Eelco Dolstra f32fbf952d Fix Bison 3.3 warning 2019-03-27 21:09:31 +01:00
Eelco Dolstra c101b29133 Merge remote-tracking branch 'tweag/flake-registries' into flakes 2019-03-27 14:40:36 +01:00
Eelco Dolstra be7fd63595 Remove debug line 2019-03-26 14:25:43 +01:00
Eelco Dolstra f9c7176a87 nix flake add: Handle ~/.config/nix not existing
Fixes

  $ nix flake add fnord github:edolstra/fnord
  error: opening file '/home/eelco/.config/nix/registry.json': No such file or directory
2019-03-26 12:48:57 +01:00
Eelco Dolstra 42be60c6af Merge remote-tracking branch 'tweag/flake-add' into flakes 2019-03-26 12:35:27 +01:00
Eelco Dolstra edb3836696 Merge remote-tracking branch 'tweag/auto-update-flake' into flakes 2019-03-26 11:50:22 +01:00
Eelco Dolstra b5565a7081 Merge remote-tracking branch 'origin/master' into flakes 2019-03-26 11:44:14 +01:00
Eelco Dolstra 5c05c238e6
Merge pull request #1828 from zimbatm/isPath
Add isPath primop
2019-03-25 19:58:11 +01:00
Will Dietz 42e2d5e7b7
store-api.hh: add missing include for unordered_map 2019-03-25 09:18:30 +01:00
Jonas Chevalier 514b3c7f83
Add isPath primop
this is added for completeness' sake since all the other possible
`builtins.typeOf` results have a corresponding `builtins.is<Type>`
2019-03-24 11:36:49 +01:00
xbreak fcd7660976 repl: Restore CTRL-C behaviour
Install signal handler during `readline` to handle SIGINT to abort
partially typed expressions.
2019-03-24 09:39:48 +00:00
Nick Van den Broeck a554f523db Combining registries properly 2019-03-23 13:42:48 +01:00
Nick Van den Broeck 5e4d92d267 Issue #15 is finished 2019-03-22 15:22:18 +01:00
Nick Van den Broeck e007f367bd Fixed minor things 2019-03-22 11:21:49 +01:00
Nick Van den Broeck d4ee8afd59 Implemented --flake flag for nix build
Also fixed Eelco's PR comments
2019-03-22 11:16:20 +01:00
Linus Heckemann 2aa89daab3 eval: improve type description for primops and applied primops
This can make type errors a little easier to understand.
2019-03-21 15:31:46 +01:00
Chaker Benhamed 81a23fa7e2 Add a 5 seconds of timeouts to connect to S3.
The default is 1000ms, but we can hit it a lot of we don't have direct
link to AWS (e.g. using VPN).
2019-03-15 13:23:58 +01:00
Eelco Dolstra 1f64f4c7c8
pkg-config files: Use c++17 2019-03-14 14:11:12 +01:00
Eelco Dolstra ef52ccf035
experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Eelco Dolstra 86f3b94c8c
nix-store --gc --print-roots: Sort output 2019-03-14 13:59:10 +01:00
Eelco Dolstra 578ed7a259
findRoots(): Don't censor for trusted users
They're pretty much root anyway.
2019-03-14 13:53:34 +01:00
Eelco Dolstra 53522cb6ac
findRoots(): Add 'censor' parameter
This is less brittle than filtering paths after the fact in
nix-daemon.
2019-03-14 13:53:34 +01:00
Eelco Dolstra a3f37d87ea
findRuntimeRoots: Simplify/fix handling of /proc files
Scanning of /proc/<pid>/{exe,cwd} was broken because '{memory:' was
prepended twice. Also, get rid of the whole '{memory:...}' thing
because it's unnecessary, we can just list the file in /proc directly.
2019-03-14 13:30:25 +01:00
Guillaume Maudoux 4f4391193c Fix warning about unused variable 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 38ee16ae9c Unify internal findRootsXxx() api 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 9d7221183a unify naming of roots as links & targets 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 5c56570726 Also obfuscate the number of memory roots 2019-03-10 00:56:09 +01:00
Guillaume Maudoux fc02b1b3ee Also print rooted path in nix-store -q --roots 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 8574b70342 fixup! Make roots a map of store paths to pinning links 2019-03-10 00:56:09 +01:00
Guillaume Maudoux ebc86550f9 Make roots a map of store paths to pinning links
This new structure makes more sense as there may be many sources rooting
the same store path. Many profiles can reference the same path but this
is even more true with /proc/<pid>/maps where distinct pids can and
often do map the same store path.
This implementation is also more efficient as the `Roots` map contains
only one entry per rooted store path.
2019-03-10 00:56:09 +01:00
Guillaume Maudoux a17f86ce3a Obfuscate memory roots for non-root users 2019-03-10 00:56:09 +01:00
Guillaume Maudoux 43331d6344 detail memory roots 2019-03-10 00:56:09 +01:00
Nick Van den Broeck 6542de98c2 Implemented writeRegistry 2019-03-07 12:00:10 +01:00
Nick Van den Broeck 9ff1a9ea65 Implemented json flag for nix flake info 2019-03-07 11:56:43 +01:00
volth fff8db205c
canBuildLocally: check for features
It could happen that the local builder match the system but lacks some features.
Now it results a failure.
The fix gracefully excludes the local builder from the set of available builders for derivation which requires the feature, so the derivation is built on remote builders only (as though it has incompatible system, like ```aarch64-linux``` when local is x86)
2019-03-06 05:03:25 +00:00
Will Dietz 0963479741
archive.cc: ignore more posix_fallocate "not supported" error codes
Fixes w/musl.
2019-03-01 10:31:17 +08:00
Nick Van den Broeck cfb6ab80ce Implemented "nix flake info" 2019-02-26 13:12:51 +01:00
Dzmitry Zaitsau 06d6335987 fix indentation 2019-02-26 11:07:37 +01:00
Dzmitry Zaitsau ac200c3678 Apply param values on S3Helper initialization 2019-02-25 18:06:19 +01:00
Dzmitry Zaitsau 56c18c67d9 Extend S3 URL parsing with parameters extraction 2019-02-25 18:00:55 +01:00
Dzmitry Zaitsau 07f992a74b Extract and expose splitUriAndParams function
which splits a URL into localtor and parameter parts
2019-02-25 17:59:26 +01:00
Eelco Dolstra d342de02b9
fetchFlake: Use infinite TTL when the revision is specified 2019-02-25 23:23:45 +08:00
Eelco Dolstra 529add316c
downloadCached: Return ETag
This allows fetchFlake() to return the Git revision of a GitHub
archive.
2019-02-25 23:20:50 +08:00
Eelco Dolstra d8fe447139
Merge pull request #2579 from catern/dumpdb
nix-store: make --dump-db take a list of paths to dump
2019-02-14 13:08:02 +01:00
Eelco Dolstra 6e9182fbc2
Add basic flake lock file support 2019-02-12 22:43:22 +01:00
Eelco Dolstra e38ec77de8
Interpret all file:// URIs as Git repositories 2019-02-12 22:06:19 +01:00
Eelco Dolstra beab05851b
nix: Add --flake flag
This allows using an arbitrary "provides" attribute from the specified
flake. For example:

  nix build --flake nixpkgs packages.hello

(Maybe provides.packages should be used for consistency...)
2019-02-12 21:55:43 +01:00
Eelco Dolstra 272b58220d
Enforce use of immutable flakes in pure mode
... plus a temporary hack to allow impure flakes at top-level for the
default installation source.
2019-02-12 21:07:48 +01:00
Eelco Dolstra ba05f29838
nix: Enable pure mode by default
We want to encourage a brave new world of hermetic evaluation for
source-level reproducibility, so flakes should not poke around in the
filesystem outside of their explicit dependencies.

Note that the default installation source remains impure in that it
can refer to mutable flakes, so "nix build nixpkgs.hello" still works
(and fetches the latest nixpkgs, unless it has been pinned by the
user).

A problem with pure evaluation is that builtins.currentSystem is
unavailable. For the moment, I've hard-coded "x86_64-linux" in the
nixpkgs flake. Eventually, "system" should be a flake function
argument.
2019-02-12 20:35:03 +01:00
Eelco Dolstra 91a6a47b0e
Improve flake references 2019-02-12 18:23:11 +01:00
Eelco Dolstra 7a7ec22298
Merge pull request #2628 from shlevy/context-introspection
Context introspection
2019-02-12 17:57:35 +01:00
Eelco Dolstra 0cd7f2cd8d
pkg-config files: Use c++17 2019-02-12 13:44:20 +01:00
Eelco Dolstra c8a0b9d5cb
experimental/optional -> optional 2019-02-12 13:43:32 +01:00
Eelco Dolstra 52419f8db3
Add flake registry
This will eventually be moved to nixos.org.
2019-02-11 13:01:39 +01:00
Eelco Dolstra dcae46ab14
Add github URIs
For example, github:edolstra/dwarffs is more-or-less equivalent to
https://github.com/edolstra/dwarffs.git. It's a much faster way to get
GitHub repositories: it fetches tarballs rather than entire Git
repositories. It also allows fetching specific revisions by hash
without specifying a ref (e.g. a branch name):

  github:edolstra/dwarffs/41c0c1bf292ea3ac3858ff393b49ca1123dbd553
2019-02-11 12:05:17 +01:00
Eelco Dolstra ef4cf4e681
Introduce flake URIs 2019-02-11 12:05:13 +01:00
Eelco Dolstra 7a5cf31060
Initial flake support 2019-02-11 12:00:13 +01:00
Eelco Dolstra f216c76c56
Bindings::get(): Add convenience method
This allows writing attribute lookups as

    if (auto name = value.attrs->get(state.sName))
      ...
2019-02-11 11:55:35 +01:00
Eelco Dolstra c02da99757
EvalState::allocAttr(): Add convenience method 2019-02-11 11:55:29 +01:00
Eelco Dolstra f70434b1fb
Move Command and MultiCommand to libutil 2019-02-11 11:55:25 +01:00
Eelco Dolstra 15a16e5c05
MultiCommand: Simplify construction 2019-02-11 11:55:22 +01:00
Eelco Dolstra aa0e2a2e70
Make constant primops lazy 2019-02-11 11:52:07 +01:00
Eelco Dolstra 01d07b1e92
Revert "Restore parent mount namespace before executing a child process"
This reverts commit a0ef21262f. This
doesn't work in 'nix run' and nix-shell because setns() fails in
multithreaded programs, and Boehm GC mark threads are uncancellable.

Fixes #2646.
2019-02-05 10:49:19 +01:00
Shea Levy b30be6b450
Add builtins.appendContext.
A partner of builtins.getContext, useful for the same reasons.
2019-01-31 08:52:23 -05:00
Spencer Baugh 5f1891b795 nix-store: make --dump-db take a list of paths to dump
Inside a derivation, exportReferencesGraph already provides a way to
dump the Nix database for a specific closure. On the command line,
--dump-db gave us the same information, but only for the entire Nix
database at once.

With this change, one can now pass a list of paths to --dump-db to get
the Nix database dumped for just those paths. (The user is responsible
for ensuring this is a closure, like for --export).

Among other things, this is useful for deploying a closure to a new
host without using --import/--export; one can use tar to transfer the
store paths, and --dump-db/--load-db to transfer the validity
information. This is useful if the new host doesn't actually have Nix
yet, and the closure that is being deployed itself contains Nix.
2019-01-23 01:24:34 +00:00
Eelco Dolstra 7cc1a2593e
unsupported(): Show the name of the unsupported operation 2019-01-18 13:34:53 +01:00
Shea Levy 1d757292d0
Add builtins.getContext.
This can be very helpful when debugging, as well as enabling complex
black magic like surgically removing a single dependency from a
string's context.
2019-01-14 11:27:10 -05:00
Shea Levy 087be7281a
Treat plain derivation paths in context as normal paths.
Previously, plain derivation paths in the string context (e.g. those
that arose from builtins.storePath on a drv file, not those that arose
from accessing .drvPath of a derivation) were treated somewhat like
derivaiton paths derived from .drvPath, except their dependencies
weren't recursively added to the input set. With this change, such
plain derivation paths are simply treated as paths and added to the
source inputs set accordingly, simplifying context handling code and
removing the inconsistency. If drvPath-like behavior is desired, the
.drv file can be imported and then .drvPath can be accessed.

This is a backwards-incompatibility, but storePath is never used on
drv files within nixpkgs and almost never used elsewhere.
2019-01-13 11:29:55 -05:00
Eelco Dolstra c7bf1cdb4e
Merge pull request #2608 from dtzWill/fix/issue-2546
EvalState::resetFileCache: clear parse cache as well as eval cache
2019-01-10 20:56:31 +01:00
Eelco Dolstra 30906122e2
Merge pull request #2617 from veprbl/pr/V547_nix_repl.cc
remove some dead code in nix/repl.cc
2019-01-10 20:55:27 +01:00
Eelco Dolstra 013dd28b15
Merge pull request #2601 from dtzWill/update/json-3.5.0
Update nlohmann-json 3.0.1 -> 3.5.0
2019-01-07 23:13:35 +01:00
John Ericson fef9f5653b Remove mentions of libformat, it no longer exists 2019-01-05 14:31:29 -05:00
John Ericson e10d6ed2a7 brotli is only used as a library now 2019-01-05 14:25:54 -05:00
Will Dietz 21ea00d3ec EvalState::resetFileCache: clear parse cache as well as eval cache
Fixes #2546.

(at least the basic reproduction I've been testing)
2018-12-31 10:18:28 -06:00
Dmitry Kalinkin 93c9ba3e78 remove some dead code in nix/repl.cc
Fixes a static analyzer warning:

Expression 'isDrv' is always false.

Fixes: 3beb6f6e7 ('Show derivations more concisely')
2018-12-25 22:38:40 +03:00
Will Dietz 3f8b78a84d nlohmann-json: 3.4.0 -> 3.5.0
https://github.com/nlohmann/json/releases/tag/v3.5.0
2018-12-21 22:38:06 -06:00
Will Dietz e78511743e nlohmann: 3.0.1 -> 3.4.0
```
$ curl -L
"https://github.com/nlohmann/json/releases/download/v3.4.0/json.hpp" -o
src/nlohmann/json.hpp
```
2018-12-21 22:36:53 -06:00
Will Dietz c1112ae9a2 nix-store: remove debugging print 2018-12-15 09:59:47 -06:00
Daiderd Jordan 7e35e914c1
fetchGit: allow fetching explicit refs
Trying to fetch refs that are not in refs/heads currently fails because
it looks for refs/heads/refs/foo instead of refs/foo.

eg.

	builtins.fetchGit {
	  url = https://github.com/NixOS/nixpkgs.git;
	  ref = "refs/pull/1024/head;
	}
2018-12-14 20:12:22 +01:00
Eelco Dolstra 6024dc1d97
Support SRI hashes
SRI hashes (https://www.w3.org/TR/SRI/) combine the hash algorithm and
a base-64 hash. This allows more concise and standard hash
specifications. For example, instead of

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    sha256 = "5d22dad058d5c800d65a115f919da22938c50dd6ba98c5e3a183172d149840a4";
  };

you can write

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    hash = "sha256-XSLa0FjVyADWWhFfkZ2iKTjFDda6mMXjoYMXLRSYQKQ=";
  };

In fixed-output derivations, the outputHashAlgo is no longer mandatory
if outputHash specifies the hash (either as an SRI or in the old
"<type>:<hash>" format).

'nix hash-{file,path}' now print hashes in SRI format by default. I
also reverted them to use SHA-256 by default because that's what we're
using most of the time in Nixpkgs.

Suggested by @zimbatm.
2018-12-13 14:30:52 +01:00
Eelco Dolstra 378e89360d
Merge pull request #2576 from LnL7/repl-no-link
nix repl: don't create result symlinks
2018-12-13 10:40:39 +01:00
volth 21d494da83
probably typo
...at least MSVC unable to compile this
2018-12-13 02:45:50 +00:00
Daiderd Jordan 0e6c84a771
nix repl: don't create result symlinks 2018-12-12 23:00:07 +01:00
Jonas Chevalier 5e6fa9092f
libstore: improve hash mismatch error messages
Use the same output ordering and format everywhere.

This is such a common issue that we trade the single-line error message for
more readability.

Old message:
```
fixed-output derivation produced path '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com' with sha256 hash '08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm' instead of the expected hash '1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m'
```

New message:
```
hash mismatch in fixed-output derivation '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com':
  wanted: sha256:1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m
  got:    sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm

```
2018-12-12 17:42:00 +01:00
Eelco Dolstra f7425d55df
Fix assertion failure in NarInfoDiskCache
https://hydra.nixos.org/build/85827920
2018-12-12 17:13:22 +01:00
Daiderd Jordan 6f89053108
nar-info-disk-cache: include ca in the cache entries
Without this information the content addressable state and hashes are
lost after the first request, this causes signatures to be required for
everything even tho the path could be verified without signing.
2018-12-12 01:08:47 +01:00
Daiderd Jordan 898823b67d
s3: make scheme configurable
This enables using for http for S3 request for debugging or
implementations that don't have https configured.  This is not a problem
for binary caches since they should not contain sensitive information.
Both package signatures and AWS auth already protect against tampering.
2018-12-07 23:55:38 +01:00
CHEIKH Chawki fa5143c722 Solve hg "abandoned transaction" issue 2018-12-06 13:57:59 +01:00
Eelco Dolstra 4aee93d5ce
fetchGit: Drop unnecessary localRef 2018-11-20 20:59:44 +01:00
Eelco Dolstra 3f4de91d80
Merge branch 'better-git-cache' of https://github.com/graham-at-target/nix 2018-11-20 20:41:19 +01:00
Eelco Dolstra ebeea068d5
Merge pull request #2216 from dtzWill/fix/curl-760-timeout
download: if there are active requests, never sleep for 10s
2018-11-20 19:07:56 +01:00
Eelco Dolstra 5e64470b19
Fix typo 2018-11-20 16:54:17 +01:00
Kai Harries de5997332d repl: give user the choice between libeditline and libreadline
The goal is to support libeditline AND libreadline and let the user
decide at compile time which one to use.

Add a compile time option to use libreadline instead of
libeditline. If compiled against libreadline completion functionality
is lost because of a incompatibility between libeditlines and
libreadlines completion function. Completion with libreadline is
possible and can be added later.

To use libreadline instead of libeditline the environment
variables 'EDITLINE_LIBS' and 'EDITLINE_CFLAGS' have to been set
during the ./configure step.

Example:

  EDITLINE_LIBS="/usr/lib/x86_64-linux-gnu/libhistory.so /usr/lib/x86_64-linux-gnu/libreadline.so"
  EDITLINE_CFLAGS="-DREADLINE"

The reason for this change is that for example on Debian already three
different editline libraries exist but none of those is compatible the
flavor used by nix. My hope is that with this change it would be
easier to port nix to systems that have already libreadline available.
2018-11-20 15:43:21 +01:00
Kai Harries b289d86cd1 repl: Remove code that was commented out 2018-11-20 15:42:58 +01:00
Linus Heckemann 40e0c9e925 nix ls-nar: allow reading from FIFOs
fixes #2528
2018-11-16 16:24:36 +01:00
Eelco Dolstra a32ff2573b
Fix 'Read-only file system' when building a derivation 2018-11-15 13:20:39 +01:00
Eelco Dolstra 6c6bbeb439 Merge branch 'feature/editline-pr' of https://github.com/dtzWill/nix 2018-11-13 20:42:13 +01:00
Eelco Dolstra a0ef21262f
Restore parent mount namespace before executing a child process
This ensures that they can't write to /nix/store. Fixes #2535.
2018-11-13 16:15:30 +01:00
Eelco Dolstra 7751616645
nix-prefetch-url: Stop progress bar before printing results 2018-11-09 10:34:12 +01:00
Eelco Dolstra 220c79ec22
Merge branch 'feature/prefetch-progress' of https://github.com/dtzWill/nix 2018-11-09 10:29:59 +01:00
Eelco Dolstra 812e39313c
Enable sandboxing by default
Closes #179.
2018-11-07 17:08:28 +01:00
Eelco Dolstra 6323b0729a
Disable the S3 content-encoding patch
Since we're not using multi-part uploads at the moment, we can drop
this patch.
2018-11-01 15:18:38 +01:00
Jan Path d1b049c4ea Fix typo in comments 2018-10-31 20:50:18 +01:00
Eelco Dolstra fb35aaa422
Merge pull request #2501 from mayflower/xdg_config_dirs
config: use all of XDG_CONFIG_DIRS
2018-10-31 12:37:23 +01:00
Linus Heckemann f3b8173a93 config: use all of XDG_CONFIG_DIRS
Previously, config would only be read from XDG_CONFIG_HOME. This change
allows reading config from additional directories, which enables e.g.
per-project binary caches or chroot stores with the help of direnv.
2018-10-31 09:24:43 +01:00
Eelco Dolstra 38d76d6d78
Remove redundant message 2018-10-30 19:28:42 +01:00
Eelco Dolstra 9f99d62480
S3BinaryCacheStore: Allow disabling multipart uploads
The use of TransferManager has several issues, including that it
doesn't allow setting a Content-Encoding without a patch, and it
doesn't handle exceptions in worker threads (causing termination on
memory allocation failure).

Fixes #2493.
2018-10-30 14:25:00 +01:00
Eelco Dolstra 0163e8928c
Fix broken uploadProgressCallback closure
Since the callback is global we can't refer to 'path' in it. This
could cause a segfault or printing of arbitrary data.
2018-10-30 11:45:31 +01:00
Eelco Dolstra 056c3fbbfc
Merge pull request #2494 from FPtje/fix-copy-from-old
copyStorePath: Fix hash errors when copying from older store
2018-10-29 20:45:25 +01:00
Falco Peijnenburg 49e272f647 copyStorePath: Fix hash errors when copying from older store
This commit partially reverts 48662d151b. When
copying from an older store (in my case a store running Nix 1.11.7), nix would
throw errors about there being no hash. This is fixed by recalculating the hash.
2018-10-29 20:24:37 +01:00
Will Dietz 9f998096d2 repl: complete if all matches share prefix 2018-10-29 08:45:04 -05:00
Will Dietz 3d974d31fa editline: wip 2018-10-29 08:44:58 -05:00
Daiderd Jordan 8e6bf49297
nix doctor: return nonzero exitcode if a check fails
This makes it easier to use this when testing the installer or when
running the checks with other automated tooling.
2018-10-29 12:09:22 +00:00
Eelco Dolstra 18b4c53f71
Restore old (dis)allowedRequisites behaviour for self-references
stdenv relies on this. So ignore self-references (but only in legacy non-structured attributes mode).
2018-10-27 15:41:53 +02:00
Eelco Dolstra 1427958b3c
Merge branch 'simplify-lexer' of https://github.com/layus/nix 2018-10-27 13:42:06 +02:00
Eelco Dolstra 0c61515be1
Fix signedness warning 2018-10-27 13:07:26 +02:00
Eelco Dolstra 63575ffa38
Merge branch 'nix-doctor' of https://github.com/LnL7/nix 2018-10-27 12:54:22 +02:00
Eelco Dolstra 2cf98218c8
Merge pull request #2488 from dtzWill/fix/struct-class-minor
remote-store.hh: ConnectionHandle is struct, minor fix warning
2018-10-27 12:50:16 +02:00
Guillaume Maudoux 6a5bf9b143 simplify handling of extra '}' 2018-10-27 00:14:51 +02:00
Will Dietz 9d24b5d56e nix-prefetch-url: progressbar 2018-10-26 13:19:10 -05:00
Will Dietz 3283c0dc45 remote-store.hh: ConnectionHandle is struct, minor fix warning 2018-10-26 13:15:48 -05:00
Will Dietz 27c2fcd4c0 src/nix/local.mk: fix typos in names of symlinks 2018-10-26 13:15:07 -05:00
Eelco Dolstra f6a3dfe4e0
Merge all nix-* binaries into nix
These are all symlinks to 'nix' now, reducing the installed size by
about ~1.7 MiB.
2018-10-26 12:54:00 +02:00
Eelco Dolstra c47e14ee45
Merge pull request #2483 from mayflower/tail-config
Promote verbose-build and log-lines
2018-10-23 16:59:52 +02:00
Linus Heckemann a25abe823f Promote log-lines to a fully-qualified option
This allows commands like

nix build --log-lines 30 nixpkgs.hello

in order to obtain more information in case of a failure.
2018-10-23 16:54:24 +02:00
Eelco Dolstra 1a08ad75ea
Merge pull request #2479 from nlewo/graphml
Add --graphml option to the nix-store --query command
2018-10-23 15:56:05 +02:00
Eelco Dolstra 3cd15c5b1f
Per-output reference and closure size checks
In structured-attributes derivations, you can now specify per-output
checks such as:

  outputChecks."out" = {
    # The closure of 'out' must not be larger than 256 MiB.
    maxClosureSize = 256 * 1024 * 1024;

    # It must not refer to C compiler or to the 'dev' output.
    disallowedRequisites = [ stdenv.cc "dev" ];
  };

  outputChecks."dev" = {
    # The 'dev' output must not be larger than 128 KiB.
    maxSize = 128 * 1024;
  };

Also fixed a bug in allowedRequisites that caused it to ignore
self-references.
2018-10-23 01:29:16 +02:00
Antoine Eiche d506342aa2 Remove the --xml query command option
The `--graphml` option can be used instead.
2018-10-20 09:48:53 +02:00
Antoine Eiche 73c2ae43f0 Add --graphml option to the nix-store --query command
This prints the references graph of the store paths in the graphML
format [1]. The graphML format is supported by several graph tools
such as the Python Networkx library or the Apache Thinkerpop project.

[1] http://graphml.graphdrawing.org
2018-10-20 09:48:34 +02:00
Eelco Dolstra bd78544f66 Fix assertion failure in Store::queryPathInfo()
$ nix-store -qR /nix/store/fnord
  nix-store: src/libstore/store-api.cc:80: std::__cxx11::string nix::storePathToHash(const Path&): Assertion `base.size() >= storePathHashLen' failed.
  Aborted
2018-10-16 23:39:36 +02:00
Eelco Dolstra 79e358ce6d RemoteStore: Close connection if an exception occurs
Fixes #2075.
2018-10-16 23:36:15 +02:00
Eelco Dolstra ba51100d64 Get rid of UDSRemoteStore::Connection
Since its superclass RemoteStore::Connection contains 'to' and 'from'
fields that refer to the file descriptor maintained in the subclass,
it was possible for the flush() call in Connection::~Connection() to
write to a closed file descriptor (or worse, a file descriptor now
referencing another file). So make sure that the file descriptor
survives 'to' and 'from'.
2018-10-16 22:58:01 +02:00
Eelco Dolstra 01bd66bf83
Merge pull request #2452 from ElvishJerricco/verify-sigs-overflow
Fix overflow when verifying signatures of content addressable paths
2018-10-08 12:03:03 +02:00
Eelco Dolstra 0fda9b22c7
Remove unnecessary typecast 2018-10-04 13:16:30 +02:00
Eelco Dolstra 1e7b8deea7
Check requiredSystemFeatures for local builds
For example, this prevents a "kvm" build on machines that don't have
KVM.

Fixes #2012.
2018-09-28 16:10:27 +02:00
Eelco Dolstra 7ae7a38c9a
Move structured attrs handling into a separate class
This is primarily because Derivation::{can,will}BuildLocally() depends
on attributes like preferLocalBuild and requiredSystemFeatures, but it
can't handle them properly because it doesn't have access to the
structured attributes.
2018-09-28 14:32:46 +02:00
Eelco Dolstra 99d4bb2d4c
Remove obsolete check on system features 2018-09-28 13:28:15 +02:00
Eelco Dolstra c9ba33870e
Support special attributes in structured attributes derivations
E.g. __noChroot and allowedReferences now work correctly. We also now
check that the attribute type is correct. For instance, instead of

  allowedReferences = "out";

you have to write

  allowedReferences = [ "out" ];

Fixes #2453.
2018-09-28 12:43:01 +02:00
Eelco Dolstra 63786cbd3b
S3BinaryCacheStore: Don't create buckets
This meant that making a typo in an s3:// URI would cause a bucket to
be created. Also it didn't handle eventual consistency very well. Now
it's up to the user to create the bucket.
2018-09-27 21:34:41 +02:00
Eelco Dolstra f74e0b4786
Merge pull request #2455 from grahamc/add-interactive-path
nix-shell: add bashInteractive to the start of the PATH, set SHELL
2018-09-27 21:15:50 +02:00
Graham Christensen f11acbaf17
nix-shell: add bashInteractive to the start of the PATH, set SHELL
Tools which re-exec `$SHELL` or `$0` or `basename $SHELL` or even just
`bash` will otherwise  get the non-interactive bash, providing a
broken shell for the same reasons described in
https://github.com/NixOS/nixpkgs/issues/27493.

Extends c94f3d5575
2018-09-27 15:12:56 -04:00
Eelco Dolstra 7cba4214a8
Don't talk about a "current folder build output"
Presumably this refers to ./default.nix but the support for that in
'nix' is tenuous. Also folders are a Mac thing.
2018-09-27 21:01:19 +02:00
Daiderd Jordan ed25753501
nix doctor: reimplement profile warning without gcroot check
Calculating roots seems significantly slower on darwin compared to
linux.  Checking for /profile/ links could show some false positives but
should still catch most issues.
2018-09-26 23:13:17 +02:00
Eelco Dolstra 9750430003
Ensure download thread liveness
* Don't wait forever for the client to remove data from the
  buffer. This does mean that the buffer can grow without bounds
  (e.g. when downloading is faster than writing to disk), but meh.

* Don't hold the state lock while calling the sink. The sink could
  take any amount of time to process the data (in particular when it's
  actually a coroutine), so we don't want to block the download
  thread.
2018-09-26 21:47:34 +02:00
Eelco Dolstra 98b2cc2e6e
Untabify 2018-09-26 21:39:06 +02:00
Eelco Dolstra ed78582847
sinkToSource(): Start the coroutine lazily
In particular this causes copyStorePath() from HttpBinaryCacheStore to
only start a download if needed. E.g. if the destination LocalStore
goes to sleep waiting for the path lock and another process creates
the path, then LocalStore::addToStore() will never read from the
source so we don't have to do the download.
2018-09-26 21:19:34 +02:00
Eelco Dolstra 44e86304b6
Make NAR header check more robust
Changes

  std::bad_alloc

into

  bad archive: input doesn't look like a Nix archive
2018-09-26 12:03:58 +02:00
Will Fancher 0ae8d4033d Fix overflow when verifying signatures of content addressable paths 2018-09-25 21:19:24 -04:00
Will Dietz 8df367a92d nix-daemon: allow setting builders to "" by any user (untrusted)
(cherry picked from commit a94a2eb1cb1c81e90a7529be5fecac27899a3442)
2018-09-25 03:22:08 -05:00
Eelco Dolstra 42e4ea2b6d
Merge pull request #2312 from dtzWill/fix/matched-names-should-still-print-version
search: include version for matches too!
2018-09-24 13:03:28 +02:00
Eelco Dolstra 52f6d541b9
nix-env: Fix segfault if -f argument is not a directory or a Nix expression
Fixes #2425.
2018-09-17 16:36:30 +02:00
Eelco Dolstra 901dfc7978
Fix build failure if parallel xz is not available 2018-09-17 16:36:11 +02:00
Eelco Dolstra 8327a7a8fa
Merge branch 'dirOf-relative' of https://github.com/lheckemann/nix 2018-09-13 14:33:12 +02:00
Ding Xiang Fei d6ac762bf7 auto args on repl 2018-09-10 15:43:17 +08:00
Eelco Dolstra bba3f0a308
If fallback is enabled, temporarily disable substituters after a failure
Otherwise, we just keep asking the substituter for other .narinfo
files, which can take a very long time due to retries/timeouts.
2018-09-07 17:08:43 +02:00
Eelco Dolstra 33c3f91885
Handle queryPathInfo() failure from substituters when fallback is enabled
Fixes #1990.
2018-09-07 16:35:48 +02:00
Eelco Dolstra 91405986f4
Convert NIX_COUNT_CALLS to JSON too 2018-09-05 21:57:54 +02:00
Eelco Dolstra 0a2545f95c
Log stats to stderr
We shouldn't pollute stdout.
2018-09-05 21:35:58 +02:00
Eelco Dolstra c4ba5f4be9
Merge branch 'improve-stats' of https://github.com/cleverca22/nix 2018-09-05 21:28:05 +02:00
Eelco Dolstra 2d91012754
fetchurl: Respect unpack
Fixes #2393.
2018-09-05 21:22:37 +02:00
Matthew Bauer 74f6d8767d Get effective user in Nix commands
‘geteuid’ gives us the user that the command is being run as,
including in setuid modes. By using geteuid to determind id, we can
avoid the ‘sudo -i’ hack when upgrading Nix. So now, upgrading Nix on
macOS is as simple as:

$ sudo nix-channel --update
$ sudo nix-env -u

$ sudo launchctl stop org.nixos.nix-daemon
$ sudo launchctl start org.nixos.nix-daemon
or
$ sudo systemctl restart nix-daemon
2018-09-04 19:32:39 -05:00
Will Dietz 28418af920 download: fix size reported to progress bar 2018-09-03 14:12:54 -05:00
Michael Bishop 4b034f390c remove the old text format output 2018-09-02 18:25:23 -03:00
Daiderd Jordan 80a4b44d3d
nix doctor: only perform path/profile checks with a daemon/local store
Not all store types LegacySSHStore support these operations and it
doesn't really make sense to check those.
2018-09-02 13:41:55 +02:00
Daiderd Jordan bfdca55868
nix doctor: add check for profile roots
In most cases profiles that are in PATH should have a gcroot.
2018-09-02 12:56:28 +02:00
Daiderd Jordan 0f18dc5479
nix doctor: add warning for multiple versions
It's pretty easy to unintentionally install a second version of nix
into the user profile when using a daemon install.  In this case it
looks like nix was upgraded while the nix-daemon is probably still
unning an older version.
2018-09-02 12:56:28 +02:00
Daiderd Jordan 246acf93f2
nix doctor: handle serve protocol
The serve protocol used by LegacySSHStore has a different major and
shouldn't be compared to PROTOCOL_VERSION.
2018-09-02 12:54:42 +02:00
Daiderd Jordan 7314dc7f07
nix doctor: add warning if client/daemon protocol mismatches
A protocol mismatch can sometimes cause problems when using specific
features with an older daemon. For example:

Nix 2.0 changed the way files are compied to the store.  The daemon is
backwards compatible and can still handle older clients, however a 1.11
nix-daemon isn't forwards compatible.
2018-09-02 12:54:39 +02:00
Daiderd Jordan 070823baa4
Store: expose the protocol version used by a store 2018-09-02 12:54:29 +02:00
Daiderd Jordan c9a08540c3
nix doctor: add command
Inspired by the homebrew command, shows a combination of debugging
information and warnings with potential issues with a nix installation.
2018-09-02 12:54:20 +02:00
Michael Bishop 2fd1008c70 add JSON to NIX_SHOW_STATS 2018-09-01 20:05:06 -03:00
Michael Bishop c29e5fbb13 improve the stats when profiling 2018-09-01 17:11:56 -03:00
Graham Christensen 87702532d2
nix-build: Print stats even in failing builds 2018-08-31 21:03:32 -04:00
Graham Christensen b7bb627f67
nix-build: print stats on sucessful builds 2018-08-31 20:57:07 -04:00
Eelco Dolstra 475a0a54a9
fetchGit/fetchMercurial: Don't absolutize paths
This is already done by coerceToString(), provided that the argument
is a path (e.g. 'fetchGit ./bla'). It fixes the handling of URLs like
git@github.com:owner/repo.git. It breaks 'fetchGit "./bla"', but that
was never intended to work anyway and is inconsistent with other
builtin functions (e.g. 'readFile "./bla"' fails).
2018-09-01 00:19:49 +02:00
Eelco Dolstra b7409c5754
nix path-info: Remove trailing spaces
Fixes #2390.
2018-09-01 00:04:57 +02:00
Eelco Dolstra d1f36e8787
Really fix the build 2018-08-31 19:20:08 +02:00
Eelco Dolstra ef09da58f2
nix path-info: Hopefully fix macOS build failure
https://hydra.nixos.org/build/80480356
2018-08-31 17:30:45 +02:00
Eelco Dolstra c0c31b58a4
Add localhost alias for ::1 to the sandbox 2018-08-31 11:12:51 +02:00
Michael Bishop 0767e402f1 fix error: unknown serve command 9 2018-08-30 20:10:08 -03:00
Eelco Dolstra 39f1722f36
nix upgrade-nix: Handle .nix-profile being a link to .../profiles/per-user/...
Also some cosmetic improvements.
2018-08-30 21:18:56 +02:00
Eelco Dolstra f08b14c9d0
nix upgrade-nix: Improve error message if 'nix' is not in a profile
E.g.

  $ nix upgrade-nix
  error: directory '/home/eelco/Dev/nix/inst/bin' does not appear to be part of a Nix profile

instead of

  $ nix upgrade-nix
  error: '/home/eelco/Dev/nix/inst' is not a symlink
2018-08-30 17:14:17 +02:00
Eelco Dolstra ddeda0b62e
Merge branch 'nix-upgrade-profile' of https://github.com/LnL7/nix 2018-08-30 17:10:59 +02:00
Eelco Dolstra 264e66f696
Add simple test for nix upgrade-nix 2018-08-30 17:10:28 +02:00
Eelco Dolstra 99828245f8
printSize() fixes
Fix a 32-bit overflow that resulted in negative numbers being printed;
use fmt() instead of boost::format(); change -H to -h for consistency
with 'ls' and 'du'; make the columns narrower (since they can't be
bigger than 1024.0).
2018-08-30 16:22:04 +02:00
Benjamin Hipple c908df881f
Avoid overflow and use boost::format
If the user has an object greater than 1024 yottabytes, it'll just display it as
N yottabytes instead of overflowing.

Swaps to use boost::format strings instead of std::setw and std::setprecision.
2018-08-30 16:22:04 +02:00
Benjamin Hipple 3407a5d936
Add human readable closure sizes to nix path-info
Unfortunately, -h is already taken as a short option by --help, so we have to
use a different letter or the capitalized version.

Resolves #2363
2018-08-30 16:22:04 +02:00
Eelco Dolstra 5e83b0227f
nix: Remove the -h flag 2018-08-30 16:14:19 +02:00
Eelco Dolstra 1286d86b95
Merge pull request #2336 from ivan/repl-trailing-spaces
repl: don't add trailing spaces to history lines
2018-08-29 18:13:29 +02:00
aszlig 0ad643ed5c
libexpr: Use int64_t for NixInt
Using a 64bit integer on 32bit systems will come with a bit of a
performance overhead, but given that Nix doesn't use a lot of integers
compared to other types, I think the overhead is negligible also
considering that 32bit systems are in decline.

The biggest advantage however is that when we use a consistent integer
size across all platforms it's less likely that we miss things that we
break due to that. One example would be:

https://github.com/NixOS/nixpkgs/pull/44233

On Hydra it will evaluate, because the evaluator runs on a 64bit
machine, but when evaluating the same on a 32bit machine it will fail,
so using 64bit integers should make that consistent.

While the change of the type in value.hh is rather easy to do, we have a
few more options available for doing the conversion in the lexer:

  * Via an #ifdef on the architecture and using strtol() or strtoll()
    accordingly depending on which architecture we are. For the #ifdef
    we would need another AX_COMPILE_CHECK_SIZEOF in configure.ac.
  * Using istringstream, which would involve copying the value.
  * As we're already using boost, lexical_cast might be a good idea.

Spoiler: I went for the latter, first of all because lexical_cast does
have an overload for const char* and second of all, because it doesn't
involve copying around the input string. Also, because istringstream
seems to come with a bigger overhead than boost::lexical_cast:

https://www.boost.org/doc/libs/release/doc/html/boost_lexical_cast/performance.html

The first method (still using strtol/strtoll) also wasn't something I
pursued further, because it is also locale-aware which I doubt is what
we want, given that the regex for int is [0-9]+.

Signed-off-by: aszlig <aszlig@nix.build>
Fixes: #2339
2018-08-29 01:05:52 +02:00
Daiderd Jordan d85bb4814f
upgrade-nix: resolve profile symlinks
The profile present in PATH is not necessarily the actual profile
location. User profiles are generally added as $HOME/.nix-profile
in which case the indirect profile link needs to be resolved first.

/home/user/.nix-profile -> /nix/var/nix/profiles/per-user/user/profile
/nix/var/nix/profiles/per-user/user/profile -> profile-15-link
/nix/var/nix/profiles/per-user/user/profile-14-link -> /nix/store/hyi4kkjh3bwi2z3wfljrkfymz9904h62-user-environment
/nix/var/nix/profiles/per-user/user/profile-15-link -> /nix/store/6njpl3qvihz46vj911pwx7hfcvwhifl9-user-environment

To upgrade nix here we want /nix/var/nix/profiles/per-user/user/profile-16-link
instead of /home/user/.nix-profile-1-link. The latter is not a gcroot
and would be garbage collected, resulting in a broken profile.

Fixes #2175
2018-08-25 20:50:44 +02:00
Daiderd Jordan 414397759a
upgrade-nix: add --dry-run 2018-08-25 20:25:43 +02:00
Eelco Dolstra c651b7bdc9
Revert "Fix parser/lexer generation with parallel make"
This reverts commit d277442df5.

Make sucks.
2018-08-23 00:23:35 +02:00
Eelco Dolstra 1b01954a3d
Merge pull request #2368 from dezgeg/drop-dead-code
Drop dead code
2018-08-22 21:32:49 +02:00
Eelco Dolstra ebe3d2d370 Improve 'coroutine has finished' error message 2018-08-21 15:22:04 +02:00
Eelco Dolstra 6317c65937 Fix warnings in compression.cc 2018-08-21 15:20:23 +02:00
Eelco Dolstra 17a92dfb7d Fix another 'coroutine has finished' during decompression
https://hydra.nixos.org/build/79867739
2018-08-21 15:19:20 +02:00
Tuomas Tynkkynen 2894197de7 Drop all references to NIX_INDENT_MAKE
Dead code since 6669a3b477
2018-08-20 02:01:59 +03:00
Eelco Dolstra cc7b4386b1
nix run: Restore CPU affinity
Fixes #2359.
2018-08-19 12:05:08 +02:00
Eelco Dolstra 9b1bdf2db8
FIx floating point evaluation
Fixes #2361.
2018-08-19 11:59:49 +02:00
Graham Christensen 02098d2073 fetchGit: use a better caching scheme
The current usage technically works by putting multiple different
repos in to the same git directory. However, it is very slow as
Git tries very hard to find common commits between the two
repositories. If the two repositories are large (like Nixpkgs and
another long-running project,) it is maddeningly slow.

This change busts the cache for existing deployments, but users
will be promptly repaid in per-repository performance.
2018-08-17 11:27:34 -04:00
Eelco Dolstra d277442df5
Fix parser/lexer generation with parallel make
Fun fact: rules with multiple targets don't work properly with 'make
-j'. For example, a rule like

  a b: c
    touch a b

is equivalent to

  a: c
    touch a b

  b: c
    touch a b

so with 'make -j', the 'touch' command will be run twice. See
e.g. https://stackoverflow.com/questions/2973445/gnu-makefile-rule-generating-a-few-targets-from-a-single-source-file.
2018-08-17 12:59:23 +02:00
Symphorien Gibol 5b19a6663b ignore when listxattr fails with ENODATA
This happens on CIFS and means the remote filesystem has no extended
attributes.
2018-08-13 21:11:36 +02:00
Linus Heckemann d7402c9cd5 dirOf: allow use on non-absolute paths 2018-08-13 11:27:35 +02:00
Eelco Dolstra f72c907ad8
Slightly questionable workaround for #2342 2018-08-10 11:34:41 +02:00
Eelco Dolstra ada4e90267
S3BinaryCacheStore: Share TransferManager
TransferManager allocates a lot of memory (50 MiB by default), and it
might leak but I'm not sure about that. In any case it was causing
OOMs in hydra-queue-runner. So allocate only one TransferManager per
S3BinaryCacheStore.

Hopefully fixes https://github.com/NixOS/hydra/issues/586.
2018-08-09 20:48:45 +02:00
Eelco Dolstra c87f4b9324
nix run: Respect propagated-user-env-packages
Also, add $path/bin to $PATH even if it doesn't exist. This makes
'man' work properly (since it looks for ../share/man relative to $PATH
entries).
2018-08-09 13:01:03 +02:00
Eelco Dolstra a0b971dd9c
S3BinaryCacheStore: Don't use the transfer status callback
This callback is executed on a different thread, so exceptions thrown
from the callback are not caught:

  Aug 08 16:25:48 chef hydra-queue-runner[11967]: terminate called after throwing an instance of 'nix::Error'
  Aug 08 16:25:48 chef hydra-queue-runner[11967]:   what():  AWS error: failed to upload 's3://nix-cache/19dbddlfb0vp68g68y19p9fswrgl0bg7.ls'

Therefore, just check the transfer status after it completes. Also
include the S3 error message in the exception.
2018-08-08 21:39:11 +02:00
Bob van der Linden 58a85fa462 mention nix-store --query --roots when a path cannot be deleted 2018-08-08 21:21:21 +02:00
Ivan Kozik ec49ea28dc repl: don't add trailing spaces to history lines 2018-08-07 03:59:34 +00:00
Eelco Dolstra 103ad1598c
Merge pull request #2334 from dtzWill/fix/revert-progress-bar-refresh-for-now
Revert "progress-bar: re-draw last update if nothing new for 1sec."
2018-08-06 19:01:08 +02:00
Eelco Dolstra ed6c646f44
Doh 2018-08-06 17:27:08 +02:00
Eelco Dolstra 7de3e00ad9
Fix setting Content-Encoding in S3 uploads
Fixes https://github.com/NixOS/nix/issues/2333 and https://github.com/NixOS/nixpkgs/issues/44337.
2018-08-06 17:20:23 +02:00
Eelco Dolstra d3761f5f8b
Fix Brotli decompression in 'nix log'
This didn't work anymore since decompression was only done in the
non-coroutine case.

Decompressors are now sinks, just like compressors.

Also fixed a bug in bzip2 API handling (we have to handle BZ_RUN_OK
rather than BZ_OK), which we didn't notice because there was a missing
'throw':

  if (ret != BZ_OK)
      CompressionError("error while compressing bzip2 file");
2018-08-06 15:40:29 +02:00
Eelco Dolstra fa4def3d46
Require libbrotli 2018-08-06 14:06:54 +02:00
Eelco Dolstra 4361a4331f
Fix reporting of HTTP body size when a result callback is used 2018-08-06 11:31:14 +02:00
Eelco Dolstra 2825e05d21
Make adding paths via nix-store --serve run in constant memory
It adds a new operation, cmdAddToStoreNar, that does the same thing as
the corresponding nix-daemon operation, i.e. call addToStore(). This
replaces cmdImportPaths, which has the major issue that it sends the
NAR first and the store path second, thus requiring us to store the
incoming NAR either in memory or on disk until we decide what to do
with it.

For example, this reduces the memory usage of

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 267 MiB to 12 MiB.

Probably fixes #1988.
2018-08-03 21:21:16 +02:00
Eelco Dolstra eeebe4cdc5
cmdDumpStorePath: Support chroot stores 2018-08-03 21:20:38 +02:00
Eelco Dolstra 848a9375c3
Support escaping in store URIs 2018-08-03 21:20:38 +02:00
Eelco Dolstra 4e7d5f660c
SSHMaster: Bypass SSH when connecting to localhost
This is primarily useful for testing since it removes the need to have
SSH working.
2018-08-03 21:20:38 +02:00
Eelco Dolstra e268bbc054
LegacySSHStore: Add remote-store option
This is primarily useful for testing, e.g.

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' ...
2018-08-03 21:20:38 +02:00
Eelco Dolstra bc65e02d96
Merge pull request #2326 from aszlig/fix-symlink-leak
Fix symlink leak in restricted eval mode
2018-08-03 17:01:34 +02:00
Eelco Dolstra 122e1a61f8
Merge pull request #2323 from samueldr/feature/selective-impurity
Allows selectively adding environment variables to pure shells.
2018-08-03 16:58:18 +02:00
aszlig 43e28a1b75
Fix symlink leak in restricted eval mode
In EvalState::checkSourcePath, the path is checked against the list of
allowed paths first and later it's checked again *after* resolving
symlinks.

The resolving of the symlinks is done via canonPath, which also strips
out "../" and "./". However after the canonicalisation the error message
pointing out that the path is not allowed prints the symlink target in
the error message.

Even if we'd suppress the message, symlink targets could still be leaked
if the symlink target doesn't exist (in this case the error is thrown in
canonPath).

So instead, we now do canonPath() without symlink resolving first before
even checking against the list of allowed paths and then later do the
symlink resolving and checking the allowed paths again.

The first call to canonPath() should get rid of all the "../" and "./",
so in theory the only way to leak a symlink if the attacker is able to
put a symlink in one of the paths allowed by restricted evaluation mode.

For the latter I don't think this is part of the threat model, because
if the attacker can write to that path, the attack vector is even
larger.

Signed-off-by: aszlig <aszlig@nix.build>
2018-08-03 06:46:43 +02:00
Samuel Dionne-Riel 438e02529d Allows selectively adding environment variables to pure shells.
Includes documentation and test.
2018-08-02 08:21:30 -04:00
Graham Christensen 49a53c1d3f s3 binary cache: support specifying an endpoint
Works for uploading and not downloading.
2018-08-02 08:19:50 -04:00
Samuel Dionne-Riel fb6907e3d6 Removes unused variable from nix-build/nix-shell
This particular `shell` variable wasn't used, since a new one was
declared in the only side of the `if` branch that used a `shell`
variable.

It could realistically confuse developers thinking it could use `$SHELL`
under some situations.
2018-08-01 18:30:03 -04:00
Eelco Dolstra 45bcf5416a
Merge branch 'prim_mapAttr-fix' of https://github.com/volth/nix 2018-07-31 20:05:07 +02:00
Eelco Dolstra 4b4adbc93d
Fix race downloading into a sink
This fixes 'error 10 while decompressing xz file'.

https://hydra.nixos.org/build/78308551
2018-07-31 17:26:04 +02:00
Eelco Dolstra d9e8ab0ff2
Add a check for broken Boost versions
In some Boost versions, coroutines don't propagate exceptions
properly, causing Nix to fail with the exception 'coroutine has
finished'.
2018-07-31 16:35:05 +02:00
Will Dietz 0d72f4e0ca search: include version 2018-07-27 10:45:35 -05:00
Linus Heckemann 9ac1a79882 copyPathsToStore: honour keep-going 2018-07-24 17:05:06 +02:00
Eelco Dolstra a7fb7d3cde
Merge pull request #2303 from volth/patch-4
parser.y: fix assoc of -> and < > <= >=
2018-07-23 11:38:15 +02:00
volth deaa6e9a34
parser.y: right-associativity of -> 2018-07-23 07:28:48 +00:00
volth 85fe4a819c
parser.y: fix assoc of -> and < > <= >=
The parser allowed senseless `a > b > c` but disallowed `a -> b -> c` which seems valid
It might be a typo
2018-07-21 15:24:51 +00:00
volth e2b114cfe1
prim_foldlStrict: call forceValue() before value is copied
forceValue() were called after a value is copied effectively forcing only one of the copies keeping another copy not evaluated.
This resulted in its evaluation of the same lazy value more than once (the number of hits is not big though)
2018-07-21 06:44:42 +00:00
Will Dietz 4495560d6d Revert "progress-bar: re-draw last update if nothing new for 1sec."
Not ready for this yet, causes the prompt to disappear in nix repl
and more generally can overwrite non-progress-bar messages.

This reverts commit 44de71a396.
2018-07-19 10:29:31 -05:00
Eelco Dolstra 1b34b69b45
nix-prefetch-url: Download file in constant memory
Before:

  $ command time nix-prefetch-url https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.17.6.tar.xz
  1.19user 1.02system 0:41.96elapsed 5%CPU (0avgtext+0avgdata 182720maxresident)k

After:

  1.38user 1.05system 0:39.73elapsed 6%CPU (0avgtext+0avgdata 16204maxresident)k

Note however that addToStore() can still take a lot of memory
(e.g. RemoteStore::addToStore() is constant space, but
LocalStore::addToStore() isn't; that's fixed by
c94b4fc7ee
though).

Fixes #1400.
2018-07-12 18:48:41 +02:00
Eelco Dolstra c2de2ff385
Fix build on 32-bit systems and macOS
Apparently, on macOS, 'long' != 'int64_t'.

https://hydra.nixos.org/build/77100756
2018-07-11 21:12:18 +02:00
Eelco Dolstra efd04888ca
Shup up a warning 2018-07-11 21:05:09 +02:00
Eelco Dolstra 875cd9da2b
Remove unused function printStats2()
Closes #2282.
2018-07-11 20:29:18 +02:00
volth c03d73c1cf
prim_map, prim_genList: no need to force function argument which will be stored in mkApp 2018-07-07 23:30:17 +00:00
volth 627e28ba33 prim_mapAttrs: `f' must be evaluated lazily to avoid infinite recursion 2018-07-06 21:52:54 +00:00
Eelco Dolstra aa64e95bc8
Merge pull request #2250 from dtzWill/feature/repl-progress-bar-builds
repl: use `nix build` for building instead of `nix-store -r`
2018-07-06 21:36:37 +02:00
volth 1515c65616 prim_concatMap: no need to force value 2018-07-05 15:33:33 +00:00
volth e6bf1a79d7 prim_mapAttrs: must be lazy to avoid infinite recursion 2018-07-05 15:33:12 +00:00
volth 841747b0e6
prim_concatMap: allocate intermediate list on stack 2018-07-05 12:37:37 +00:00
volth ee218f99ca
primops.cc: fix comment 2018-07-05 11:58:15 +00:00
volth 403a76a18f lib.concatMap and lib.mapAttrs to be builtins 2018-07-05 02:54:09 +00:00
Eelco Dolstra e388739098
nix-daemon: Respect --store
For example, this allows you to do run nix-daemon as a non-privileged
user:

  eelco$ NIX_STATE_DIR=~/my-nix/nix/var nix-daemon --store ~/my-nix/

The NIX_STATE_DIR is still needed because settings.nixDaemonSocketFile
is not derived from settings.storeUri (and we can't derive it from the
store's state directory because we don't want to open the store in the
parent process).
2018-07-03 19:52:32 +02:00
Eelco Dolstra ea3c9dab5f
Include cpptoml for build simplicity 2018-07-03 18:39:36 +02:00
Eelco Dolstra 3b1f54cf06
Add a fromTOML primop
This is primarily useful for processing Cargo.lock files.
2018-07-03 18:39:36 +02:00
Eelco Dolstra a92ed973e5
Store floating point numbers in double precision
Even on 32-bit systems, Value has enough space to hold a double.
2018-07-03 18:39:32 +02:00
Maximilian Bosch 8a98ffc276
nix why-depends: render output into $PAGER 2018-07-03 12:49:12 +02:00
Eelco Dolstra dd98683e65
Merge pull request #2158 from Ma27/improve-search-ux
search.cc: improve UX for `nix search`
2018-07-02 11:16:21 +02:00
Maximilian Bosch 6b74fdac27
search.cc: improve UX for nix search
As proposed in #1634 the `nix search` command could use some
improvements. Initially 0413aeb35d added
some basic sorting behavior using `std::map`, a next step would be an
improvement of the output.

This patch includes the following changes:

* Use `$PAGER` for outputs with `RunPager` from `shared.hh`:
  The same behavior is defined for `nix-env --query`, furthermore it
  makes searching huge results way easier.

* Simplified result blocks:
  The new output is heavily inspired by the output from `nox`, the first
  line shows the attribute path and the derivaiton name
  (`attribute path (derivation name)`) and the description in the second
  line.
2018-07-02 11:05:59 +02:00
Will Dietz 6a24e49ba8 repl: use nix build for building instead of nix-store -r
progress bar!
2018-06-23 07:56:26 -05:00
Will Dietz 44de71a396 progress-bar: re-draw last update if nothing new for 1sec.
Slightly nicer behavior when updates are somewhat far apart
(during a long linking step, perhaps) ensuring things
don't appear unresponsive.

If we wait the maximum amount for the update,
don't bother waiting another 50ms (for rate-limiting purposes)
and just check if we should quit.

This also ensures we'll notice the request to quit within 1s
if quit is signalled but there is not an udpate.
(I'm not sure if this happens or not)
2018-06-18 17:54:09 -05:00
Will Dietz f601bc0492 sync.hh: return cv_status instead of dropping it, in case useful. 2018-06-18 17:31:00 -05:00
Eelco Dolstra b920b90857
Detect redirect loops
Fixes #2225.
2018-06-18 10:36:19 +02:00
Eelco Dolstra 2e244fb68f
Merge pull request #2234 from symphorien/esrch2
libstore/gc.cc: ignore ESRCH when reading symlinks in /proc
2018-06-18 09:01:55 +02:00
Symphorien Gibol ef8122064b libstore/gc.cc: ignore ESRCH when reading symlinks in /proc
readlink is also affected by the problem fixed for regular files in
c567afe355df
2018-06-16 18:04:09 +02:00
Matthew Justin Bauer 6d09e4400c
Fix optimise-store.cc to skip more paths on macOS
I hate to make this such a large check but the lack of documentation means we really have no idea what's allowed. All of them reported so far have been within ".app/Contents" directories. That appears to be a safe starting point. However, I would not be surprised to also find more paths that are disallowed for instance in .framework or .bundle directories.

Fixes #2031
Fixes #2229
2018-06-14 12:34:55 -04:00
Eelco Dolstra 85530bcc0b
nix: Remove special handling of .drv files
This makes 'nix copy' and 'nix path-info' work on .drv store
paths. Removing special treatment of .drv files seems the most
future-proof approach given the possible removal of .drv files in the
future.

Note that 'nix build' will still build (rather than substitute) .drv
paths due to the unfortunate overloading in Store::buildPaths().
2018-06-13 18:04:22 +02:00
Eelco Dolstra 7902fccf6f
hashDerivationModulo(): Handle chroot stores
This makes 'nix copy --to /mnt /nix/store/bla.drv' work.
2018-06-13 18:00:54 +02:00
Eelco Dolstra 3c5d9f478d
std::random_shuffle -> std::shuffle
The former is removed in C++17.
2018-06-13 16:56:19 +02:00
Eelco Dolstra 0629601da1
Move EvalState from the stack to the heap
EvalState contains a few counters (e.g. nrValues) that increase
quickly enough that they end up being interpreted as pointers by the
garbage collector. Moving it to the heap makes them invisible to the
garbage collector.

This reduces the max RSS doing 100 evaluations of
nixos.tests.firefox.x86_64-linux.drvPath from 455 MiB to 292 MiB.

Note: ideally, allocations would be much further up in the 64-bit
address space to reduce the odds of an integer being misinterpreted as
a pointer. Maybe we can use some linker magic to move the .bss segment
to a higher address.
2018-06-12 17:49:55 +02:00
Eelco Dolstra c905d8b0a8
GC_malloc -> GC_MALLOC
This makes it possible to build with -DGC_DEBUG.
2018-06-12 17:49:55 +02:00
Eelco Dolstra 455d1f01d0
Don't scan for roots in dynamic libraries
This reduces the risk of object liveness misdetection. For example,
Glibc has an internal variable "mp_" that often points to a Boehm
object, keeping it alive unnecessarily. Since we don't store any
actual roots in global variables, we can just disable data segment
scanning.

With this, the max RSS doing 100 evaluations of
nixos.tests.firefox.x86_64-linux.drvPath went from 718 MiB to 455 MiB.
2018-06-12 17:49:55 +02:00
Eelco Dolstra 30964103dc
Add temporary stats 2018-06-12 17:49:55 +02:00
Eelco Dolstra 24c6806994
Cache parse trees
This prevents EvalState::resetFileCache() from parsing everything all
over again.
2018-06-12 17:49:55 +02:00
Eelco Dolstra 6ad0a2f749
Remove duplicate definition of allocBytes() 2018-06-12 17:49:51 +02:00
Yorick 72a78beb34
Fix #2162: use getaddrinfo instead of curl to preload NSS 2018-06-12 13:05:14 +02:00
Symphorien Gibol 8c567afe35 libstore/gc.cc: ignore ESRCH when reading /proc
If a process disappears between the time /proc/[pid]/maps is opened and
the time it is read, the read() syscall will return ESRCH. This should be ignored.
2018-06-11 16:29:47 +02:00
Will Dietz 6bf7dc1baf download: if there are active requests, never sleep for 10s
In some versions/configurations libcurl doesn't handle timeouts
(especially DNS timeouts) in a way that wakes curl_multi_wait.

This doesn't appear to be a problem if using c-ares, FWIW.
2018-06-08 21:12:43 -05:00
AmineChikhaoui 170165ee5a
use debug instead of printTalkative to avoid spamming hydra queue
runner logs.
2018-06-08 11:43:32 +01:00
Eelco Dolstra 4ac4f675df
Don't require --fallback to recover from disappeared binary cache NARs 2018-06-05 16:04:41 +02:00
Eelco Dolstra 691b7582c7
Handle HTTP 410 Gone errors 2018-06-05 16:03:32 +02:00
Eelco Dolstra ca06a9cea7
Improve binary cache fallback test 2018-06-05 15:26:12 +02:00
Eelco Dolstra a2ec7a3bfd
Further improve upload messages 2018-06-05 14:37:26 +02:00
Will Dietz 54b1c59643 download.cc: fix readCallback return type (int -> size_t)
I'm not sure if curl ever asks for enough data at once
for truncation to occur but better safe than sorry.
2018-06-01 10:59:26 -05:00
Eelco Dolstra 73357500ac
Improve binary cache upload messages
Don't say "download" when we mean "upload".
2018-06-01 14:14:22 +02:00
Peter Simons 93aa3bea2e
Merge pull request #767 from mogorman/garbage_collect_keep_last_few
Implement --delete-generations + flag for keeping last N number of gens
2018-05-31 10:00:21 +02:00
Eelco Dolstra 5d4a9d5677
Fix static assertion failure on 32-bit systems 2018-05-30 17:47:30 +02:00
Eelco Dolstra 3c2de9830d
RemoteStore::addToStore(): Ensure capacity is decreased on exceptions 2018-05-30 13:48:31 +02:00
Eelco Dolstra a4c1618876
Fix deadlock in RemoteStore::queryPathInfoUncached() 2018-05-30 13:46:14 +02:00
Eelco Dolstra 7d21863bb3
Make <nix/fetchurl.nix> run in constant memory
E.g.

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv"; }'

now runs in 17 MiB (was 70 MiB), while

  nix-build --store ~/my-nix/ -E 'import <nix/fetchurl.nix> { url = https://cache.nixos.org/nar/0nwi996rgq4b914qyx0mv2wq4k80hjac7xilikavagw7kxmn2iiv.nar.xz; sha256 = "0d2fxljdih3nc5dqx41hjzic3141ajil94m8kdbpryq569dpsbvb"; unpack = true; }'

runs in 17 MiB (was 346 MiB).
2018-05-30 13:42:51 +02:00
Eelco Dolstra 5a654fd7dd
getDownloader(): Simplify 2018-05-30 13:42:30 +02:00
Eelco Dolstra e87e4a60d6
Make HttpBinaryCacheStore::narFromPath() run in constant memory
This reduces memory consumption of

  nix copy --from https://cache.nixos.org --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 176 MiB to 82 MiB. (The remaining memory is probably due to xz
decompression overhead.)

Issue https://github.com/NixOS/nix/issues/1681.
Issue https://github.com/NixOS/nix/issues/1969.
2018-05-30 13:42:29 +02:00
Eelco Dolstra 08ec757726
Make LocalBinaryCacheStore::narFromPath() run in constant memory
This reduces memory consumption of

  nix copy --from file://... --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 514 MiB to 18 MiB for an uncompressed binary cache, and from 192
MiB to 53 MiB for a bzipped binary cache. It may also be faster
because fetching can happen concurrently with decompression/writing.

Continuation of 48662d151b.

Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30 13:34:57 +02:00
Eelco Dolstra 81ea8bd5ce
Simplify the callback mechanism 2018-05-30 13:34:37 +02:00
Eelco Dolstra 1672bcd230
Move evaluator-specific settings out of libstore 2018-05-30 13:29:50 +02:00
Eelco Dolstra c1d445ecec
Move some Download-specific settings to download.cc 2018-05-30 13:29:50 +02:00
Eelco Dolstra 737ed88f35
Modularize config settings
Allow global config settings to be defined in multiple Config
classes. For example, this means that libutil can have settings and
evaluator settings can be moved out of libstore. The Config classes
are registered in a new GlobalConfig class to which config files
etc. are applied.

Relevant to https://github.com/NixOS/nix/issues/2009 in that it
removes the need for ad hoc handling of useCaseHack, which was the
underlying cause of that issue.
2018-05-30 13:28:01 +02:00
Eelco Dolstra e606cd412f
Fix assertion failure in storePathToHash()
Fixes https://github.com/NixOS/nix/issues/2015.
2018-05-30 13:27:19 +02:00
Eelco Dolstra 6185d25e52
Make 'nix copy --to daemon' run in constant memory (daemon side)
Continuation of 97002b684c. This makes
the daemon use constant memory. For example, it reduces the daemon's
maximum RSS on

  $ nix copy --from ~/my-nix --to daemon /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 264 MiB to 7 MiB.

We now use a TunnelSource to prevent the connection from ending up in
an undefined state if an exception is thrown while the NAR is being
sent.

Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30 13:26:48 +02:00
Eelco Dolstra 23d6bb583a
LocalStore::addToStore: Throw an exception if no hash is given 2018-05-30 13:25:40 +02:00
Eelco Dolstra 3cab639e21
Use extended SQLite error codes 2018-05-30 13:25:03 +02:00
Eelco Dolstra bd56b5fe3f
Merge pull request #2187 from bgamari/stoi-exceptions
json-to-value: Use strtol instead of strtoi
2018-05-28 14:54:49 +02:00
Ben Gamari 7f560b81ea json-to-value: Use strtol instead of strtoi
NixInt is long, so strtoi is too restrictive.
2018-05-26 18:43:46 -04:00
Will Dietz 93ae90de0f eval.cc: add message to static_assert, message can be omitted w/c++17 2018-05-26 14:04:41 -05:00
Will Dietz 14c464b6c8 nix-daemon: remove unused "pendingMsgs" variable 2018-05-26 13:09:55 -05:00
Will Dietz ca9c6cb95d libstore/download: remove unused function "getTime()" 2018-05-25 19:49:45 -05:00
Eelco Dolstra 743359bc8a
Merge pull request #2157 from volth/bitwise
add builtins: __bitAnd,  __bitOr,  __bitXor
2018-05-24 15:00:39 +02:00
Eelco Dolstra 4bb8741b98
Make Env self-describing
If the Env denotes a 'with', then values[0] may be an Expr* cast to a
Value*. For code that generically traverses Values/Envs, it's useful
to know this.
2018-05-22 16:02:32 +02:00
Eelco Dolstra 9fd7cf98db
Memoise checkSourcePath()
This prevents hydra-eval-jobs from statting the same files over and
over again.
2018-05-22 13:02:14 +02:00
Will Dietz b08923b4a0 serialise: fix buffer size used, hide method for internal use only
Fixes #2169.
2018-05-21 17:30:38 -05:00
volth 6cc28c0589
add `mod' and bitwise builtins: camel-case function names 2018-05-16 10:52:19 +00:00
volth f3c090f91c
add mod' and bitwise builtins: remove mod' and shifts 2018-05-16 06:57:11 +00:00
volth 49b7cf1813
add `mod' and bitwise builtins: remove infix functions 2018-05-16 06:55:24 +00:00
Eelco Dolstra 966407bcf1
Merge pull request #2145 from serokell/uncached-channel
make sure not to use cached channels for nix-channel --update
2018-05-15 14:57:18 +02:00
volth 8a6a14e1f5 add `mod' and bitwise builtins 2018-05-12 16:50:39 +00:00
Eelco Dolstra 1ad19232c4
Don't return negative numbers from the flex tokenizer
Fixes #1374.
Closes #2129.
2018-05-11 12:05:12 +02:00
Eelco Dolstra f3c85f9eb3
Revert "Throw a specific error for incomplete parse errors."
This reverts commit 6498adb002. We don't
actually use IncompleteParseError in 'nix repl'.
2018-05-11 11:40:50 +02:00
Eelco Dolstra 2b3c1b3a88
Remove mention of default since we already know the default 2018-05-10 12:06:28 +02:00
Eelco Dolstra 38def17627
Merge branch 's3-multipart-uploads' of https://github.com/AmineChikhaoui/nix 2018-05-10 12:06:02 +02:00
Yorick b9289e4855
make sure not to use cached channels for nix-channel --update
fixes #1964
2018-05-09 16:18:20 +02:00
Eelco Dolstra a91c4ca01f
In restricted eval mode, allow access to the closure of store paths
E.g. this makes

  nix eval --restrict-eval -I /nix/store/foo '(builtins.readFile "/nix/store/foo/symlink/bla")'

(where /nix/store/foo/symlink is a symlink to another path in the
closure of /nix/store/foo) succeed.

This fixes a regression in Hydra compared to Nix 1.x (where there were
no restrictions at all on access to the Nix store).
2018-05-09 15:45:05 +02:00
AmineChikhaoui 854c0860f4
share the executor between multiple copy threads. 2018-05-07 15:07:00 +01:00
AmineChikhaoui 591e75cd01
add a FIXME note to find a way to abort the multipart uploads in
case the nix command is interrupted.
2018-05-07 14:27:53 +01:00
AmineChikhaoui 129394fb95
Support multi-part uploads for large NARs that exceed the size of
5Gb.
2018-05-07 14:23:51 +01:00
Eelco Dolstra a1adcdf087
nix-shell: Only download requested outputs
Fixes #2116.
2018-05-07 14:58:25 +02:00
Eelco Dolstra c717d8e3bd
Don't set a CA assertion for paths with references
Really fixes #2133.
2018-05-03 14:49:56 +02:00
Eelco Dolstra 80a7b16593
LocalStore::addValidPath(): Throw error when the CA assertion is wrong
Closes #2133.
2018-05-03 11:02:40 +02:00
Eelco Dolstra 53ec5ac69f
Fix some random -Wconversion warnings 2018-05-02 13:56:34 +02:00
Eelco Dolstra 4a2c948943
Fix bzip2 compression of files > 4 GiB
Bzip2's 'avail_in' parameter is declared as an unsigned int, so
assigning a size_t length to it led to silent truncation.

Fixes #2111.
2018-05-02 12:54:30 +02:00
Will Dietz 8e7d77d494 globals.hh: don't use '==' to compare string literals
Saw this in logs, also reported here:

8e6108ff71 (r28707288)
2018-04-23 20:57:05 -05:00
Shea Levy 8e6108ff71
Merge branch 'aarch64-armv7' of git://github.com/lheckemann/nix
Support extra compatible architectures (#1916)
2018-04-23 08:48:22 -04:00
Shea Levy e2b028353b
Merge branch 'pos-crash-fix' of git://github.com/dezgeg/nix 2018-04-23 08:43:54 -04:00
Shea Levy 58f9ae0dcd
Merge branch 'hash-mismatch' of git://github.com/LnL7/nix 2018-04-23 08:40:34 -04:00
Andrew Dunham 3a918014b2 Fix library ordering in Makefiles
The existing ordering linked `libutil` before `libstore`, which causes
link failures when building statically. This is due to `libstore` using
functions from `libutil`, and the fact that symbol resolution works
"forward" - i.e. if you pass `-lfoo -lbar -lbaz`, any symbols that
`libbar` uses from `libbaz` will be resolved, but symbols from `libfoo`
will not since it comes first in the command line.

All this to say: this commit reorders the libraries which fixes the link
errors.
2018-04-21 21:10:52 -07:00
Peter Simons 040acdcee2
Merge pull request #1707 from dtzWill/fix/issue-1692
ignore "interrupted" exception in progress callback
2018-04-21 16:51:19 +02:00
Daiderd Jordan 13d4d1c0a2
download: make hash mismatch error consistent with fetchurl 2018-04-20 21:07:32 +02:00
Will Dietz 6d9129014d build.cc: fix bind-mount of /dev/{pts,ptmx} fallback
Don't bind-mount these to themselves,
mount them into the chroot directory.

Fixes pty issues when using sandbox on CentOS 7.4.
(build of perlPackages.IOTty fails before this change)
2018-04-20 12:24:23 -05:00
Daniel Poelzleithner b712d4674b Allow multiple search experssions in nix search
The common use case is to search for packages containing multiple words
like a "git" "frontend". Having only one expressions makes this simple regular
use case very complicated. Instead, search accepts multiple regular epressions
which all need to match.

nix search git 'gui|frontend'

returns a list of all git uis for example
2018-04-18 21:08:35 +02:00
Félix Baylac-Jacqué 00584bb091
parser: Allow mixed nested and top-level attrpaths
Fixes #2077.
2018-04-18 18:44:38 +02:00
Shea Levy b37f5ae31d
isFunction: True on primops.
Fixes #2073
2018-04-17 14:33:12 -04:00
Eelco Dolstra a4aac7f88c
Handle arguments in $EDITOR
Fixes #2079.
2018-04-17 12:16:04 +02:00
Eelco Dolstra c74f838620
exportReferences: Check whether the path is in the input closure
Fixes #2082.
2018-04-17 12:03:27 +02:00
Félix Baylac-Jacqué a91fb422fe
nix repl: Fix multiline SIGINT handling.
Fixes #2076
2018-04-16 16:09:30 +02:00
Eelco Dolstra d34fa2bcc3
Fix #1921 2018-04-13 15:42:35 +02:00
Shea Levy dc0a542c9f
initPlugins: Fix dlopen error message. 2018-04-11 21:02:50 -04:00
Eelco Dolstra 1839a5542a
Fix #2057 2018-04-11 11:42:17 +02:00
Eelco Dolstra d6aaa35478
Fix perl-bindings
https://hydra.nixos.org/build/72636781
2018-04-10 13:11:43 +02:00
Shea Levy a38fe5c1a8
Export required C++ version in pkgconfig. 2018-04-09 11:32:43 -04:00
Shea Levy 7459388448
Merge branch 'fix/ruby-shebang' of git://github.com/samueldr/nix 2018-04-09 10:54:39 -04:00
Shea Levy fd98fca7bb
Make prim_exec and prim_importNative available to plugins 2018-04-09 10:26:50 -04:00
Shea Levy 346c0ac361
nix-daemon: Exit successfully when interrupted.
Fixes #2058.
2018-04-09 08:05:54 -04:00
Samuel Dionne-Riel 399f43c3d5 nix-shell: Fixes use with ruby shebangs.
The ported code in 80ebc553ec was incorrectly ported.

```
-            $envCommand = "exec $execArgs $interpreter -e 'load(\"$script\")' -- ${\(join ' ', (map shellEscape, @savedArgs))}";
...
+                    envCommand = (format("exec %1% %2% -e 'load(\"%3%\") -- %4%") % execArgs % interpreter % script % joined.str()).str();
```

The single-quote finishing the small ruby snippet was lost in
translation.
2018-04-07 22:38:45 -04:00
AmineChikhaoui 62d75ad3e1
rename the options to mention it's a narinfo TTL as disk cache is used all over the place for other operations 2018-04-06 11:05:15 +01:00
AmineChikhaoui 86930ed414
add documentation for the local disk cache TTL config 2018-04-06 10:51:18 +01:00
AmineChikhaoui 2855c3d965
Make the TTL for disk cache configurable, we can now completely disable
disk cache lookup for example by doing:
  nix copy --from <binary-cahe> <store-path> --option \
  positive-disk-cache-ttl 0

Issues: #1885 #2035
2018-04-06 10:51:18 +01:00
Eelco Dolstra 1d9742b95f
<nix/buildenv.nix>: Ignore dangling symlinks
Fixes

  error: getting status of '/nix/store/j8p0vv89k1pf0cn7kmfsdcs7bshwga1i-firefox-52.7.2esr/share/icons/hicolor/48x48/apps/firefox.png': No such file or directory

https://github.com/NixOS/nix/issues/1934

Also improve error message on directory/non-directory collisions.
2018-04-05 20:04:42 +02:00
Eelco Dolstra 5c904e10eb
<nix/buildenv.nix>: Skip non-directory top-level paths
Fixes https://github.com/NixOS/nix/issues/1934 (at least the "error:
opening directory '/nix/store/...-linux-config-4.4.14': Not a
directory" issue).
2018-04-05 19:34:56 +02:00
Tuomas Tynkkynen af86132e1a libexpr: Make unsafeGetAttrPos not crash on noPos
Currently e.g. `builtins.unsafeGetAttrPos "abort" builtins` will
eventually segfault because pos->file is an unset Symbol.

Found by afl-fuzz.
2018-04-03 15:54:42 +03:00
Jonas Chevalier 865ca2402f nix copy: add an example with S3
I couldn't find a good example how to use it with non-us-east-1 buckets.
2018-03-30 02:31:14 +01:00
Eelco Dolstra 446bb88f13
Remove reference to non-existent manpages 2018-03-29 12:29:10 +02:00
Eelco Dolstra 2b61c74922
Process --option use-case-hack properly
Fixes https://github.com/NixOS/nix/issues/2009.
2018-03-29 12:28:29 +02:00
Eelco Dolstra 9d1220a01d
ValidPathInfo::isContentAddressed(): Ensure there are no references 2018-03-29 12:27:42 +02:00
Will Dietz 74da813912 download: improve error for hash mismatch ("store mismatch")
Fixes #1905
2018-03-22 15:22:06 -05:00
Will Dietz 59b32403f2 download.cc: fix path for cached files 2018-03-22 15:05:46 -05:00
Eelco Dolstra 28eb9621cc
Fix build 2018-03-22 13:46:19 +01:00
Eelco Dolstra dce8af59a6
Tweak 2018-03-22 13:32:18 +01:00
Eelco Dolstra f87e286e82
Merge branch 'fix/avoid-large-stack-buffers' of https://github.com/dtzWill/nix 2018-03-22 13:19:25 +01:00
Shea Levy c24e0439b2
Merge branch 'fix/execl-null-sentinel' of git://github.com/dtzWill/nix 2018-03-21 23:25:53 -04:00
Eelco Dolstra ec91840e70 Also make the backwards compatible case of RemoteStore::addToStore run in constant memory
Just because we can.
2018-03-21 23:46:03 +01:00
Eelco Dolstra 97002b684c Make 'nix copy --to daemon' run in constant memory 2018-03-21 23:42:52 +01:00
Will Dietz f66ae22288 shared.cc: style: prefer "nullptr" over '(char *)NULL' 2018-03-21 17:17:37 -05:00
Eelco Dolstra 11898d6a37 Make 'nix copy --to ssh://...' run in constant memory 2018-03-21 23:13:06 +01:00
Eelco Dolstra 47f7e5585b Make 'nix copy --from ssh://...' run in constant memory
For instance, this reduced the memory consumption of

  $ nix copy --from ssh://localhost --to ~/my-nix /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 632 MiB to 16 MiB.
2018-03-21 23:10:40 +01:00
Eelco Dolstra 8615cfb130
LegacySSHStore: Allow overriding the path to nix-store 2018-03-21 15:10:46 +01:00
Will Dietz 02d36fdab7 execl: cast NULL sentinel to (char *), per man page and compiler warning
From exec(3):

> The list of arguments must be terminated by a null pointer, and, since these
> are variadic functions, this pointer must be cast (char *) NULL
2018-03-20 13:31:58 -05:00
Eelco Dolstra 03d8136b02
Remove unused channel-cache directory 2018-03-20 18:22:49 +01:00
Eelco Dolstra e0c1597910
Style fix 2018-03-20 17:33:45 +01:00
Eelco Dolstra 668ac3ea2c
Make <nix/buildenv.nix> a builtin builder
This avoids sandbox annoyances.
2018-03-20 17:28:09 +01:00
Eelco Dolstra 9d40787938
Move builtinFetchurl to its own file 2018-03-20 17:01:36 +01:00
Eelco Dolstra 5df427f35b
Slight simplification 2018-03-20 16:32:59 +01:00
Eelco Dolstra 21ebf6a160
SSHMaster: Optionally pass -v to ssh 2018-03-20 15:30:55 +01:00
Eelco Dolstra 1aca195e52
Hack to get SSH error messages from build-remote
E.g.

  cannot build on 'ssh://mac1': cannot connect to 'mac1': bash: nix-store: command not found
  cannot build on 'ssh://mac2': cannot connect to 'mac2': Host key verification failed.
  cannot build on 'ssh://mac3': cannot connect to 'mac3': Received disconnect from 213... port 6001:2: Too many authentication failures
  Authentication failed.
2018-03-20 15:25:26 +01:00
Will Dietz f5a2136a40 serialise.cc: remove pessimising move
from clang6:

src/libutil/serialise.cc:189:23: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
2018-03-19 19:18:47 -05:00
Will Dietz 9b33201e72 util: rename stdout/stdin members to avoid conflicts w/standard macro
(cherry picked from commit c389a7fb617ed7bcd617efa68c6a48c00405310d)
2018-03-19 11:18:56 -05:00
Eelco Dolstra 138af2e554
Shut up signedness warning 2018-03-19 11:57:15 +01:00
Linus Heckemann 639c166647 build-remote: take extra-platforms into account 2018-03-16 22:50:57 +00:00
Linus Heckemann 637701b604 rename build-extra-platforms -> extra-platforms
also document it
2018-03-16 22:50:27 +00:00
Linus Heckemann 12913ccf45 nix-shell: allow symlinks to .drvs
This makes persistent shell environments easier to use.
2018-03-16 22:15:24 +00:00
Eelco Dolstra 48662d151b
Reduce substitution memory consumption
copyStorePath() now pipes the output of srcStore->narFromPath()
directly into dstStore->addToStore(). The sink used by the former is
converted into a source usable by the latter using
boost::coroutine2. This is based on [1].

This reduces the maximum resident size of

  $ nix build --store ~/my-nix/ /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1 --substituters file:///tmp/binary-cache-xz/ --no-require-sigs

from 418592 KiB to 53416 KiB. (The previous commit also reduced the
runtime from ~4.2s to ~3.4s, not sure why.) A further improvement will
be to download files into a Sink.

[1] https://github.com/NixOS/nix/compare/master...Mathnerd314:dump-fix-coroutine#diff-dcbcac55a634031f9cc73707da6e4b18

Issue #1969.
2018-03-16 20:35:59 +01:00
Eelco Dolstra 3e6b194d78
decompress(): Use a Source and Sink
This allows decompression to happen in O(1) memory.
2018-03-16 20:35:59 +01:00
Eelco Dolstra 64441f0551
Merge pull request #1939 from dezgeg/lexer-fix
libexpr: Recognize newline in more places in lexer
2018-03-16 12:14:58 +01:00
Eelco Dolstra eb75bc5afb
Don't retry on CURLE_SSL_CACERT_BADFILE
The certificates won't get any better if we retry.
2018-03-16 12:03:18 +01:00
Eelco Dolstra 4868721506
Filter ANSI colors when not writing to a terminal
Fixes https://github.com/NixOS/nixpkgs/issues/37114.
2018-03-15 16:08:07 +01:00
Eelco Dolstra 7b8914825a
Merge pull request #1965 from masaeedu/master
Wrap thread local in function for Cygwin
2018-03-15 14:56:21 +01:00
Shea Levy cfdbfa6b2c
Catch more possible instances of passing NULL to memcpy.
Actually fixes #1976.
2018-03-14 23:53:43 -04:00
Shea Levy d25d9f7cec
Merge branch 'fix/errno-sighandler' of git://github.com/dtzWill/nix 2018-03-14 23:45:35 -04:00
Shea Levy e2088febf3
concatLists: Don't pass NULL pointers to memcpy.
This is UB, even if the size is 0. See #1976.

Fixes #1976.
2018-03-14 23:44:02 -04:00
Will Dietz d98755b0c2 nix-daemon: preserve errno in signal handler (thanks tsan) 2018-03-14 21:39:46 -05:00
Eelco Dolstra 55aa622fb1
Fix compatibility with latest boost::format 2018-03-14 19:25:09 +01:00
Eelco Dolstra ca14b14200
Use boost::format from the boost package
Note that this only requires headers from boost so it doesn't add a
runtime dependency.

Also, use Nixpkgs 18.03.
2018-03-14 19:24:04 +01:00
Shea Levy 56f2ed0081
Merge pull request #1906 from dtzWill/fix/nix-search
nix search: tests and fix #1893 and part of #1892
2018-03-13 18:43:47 -06:00
Eelco Dolstra 7afdc8d4a1
build-remote: Don't substitute the build result 2018-03-13 17:56:51 +01:00
Eelco Dolstra 6e60141a80
Typo 2018-03-13 17:48:09 +01:00
Guillaume Maudoux 80735c4cc9
fetchGit: Fix debug message 2018-03-13 10:28:23 +01:00
Asad Saeeduddin be54f4a0b6 Wrap thread local in function for Cygwin
Fixes #1826. See #1352 for a previous instance of a similar change.
2018-03-12 00:56:41 -04:00
Eelco Dolstra 24b739817f
Fix double free in Store::queryPathInfo()
It was holding on to a Value* (i.e. a std::shared_ptr<ValidPathInfo>*)
outside of the pathInfoCache lock, so the std::shared_ptr could be
destroyed between the release of the lock and the decrement of the
std::shared_ptr refcount. This can happen if more than
'path-info-cache-size' paths are added in the meantime, *or* if
clearPathInfoCache() is called. The hydra-queue-runner queue monitor
thread periodically calls the later, so is likely to trigger a crash.

Fixes https://github.com/NixOS/hydra/issues/542.
2018-03-09 14:19:51 +01:00
Tim Engler 383afab9ad Modified MakeBinOp to no longer produce its name using concatenation and "##".
Doing so prevents emacs tags from working, as well as makes the code extremely
confusing for a newbie.

In the prior state, if someone wants to find the definition of "ExprApp" for
example, a grep through the code reveals nothing. Since the definition could be
hiding in numerous ".h" files, it's really difficult to find. This personally
took me several hours to figure out.
2018-03-09 03:16:33 +00:00
Eelco Dolstra 16551f54c9
Fix #1957 2018-03-07 11:40:50 +01:00
Eelco Dolstra 4452f6e855
Merge pull request #1955 from dtzWill/fix/rand-random
rand() -> random(), since we use srandom().
2018-03-07 11:00:08 +01:00
Will Dietz 56253bb08f rand() -> random(), since we use srandom().
rand() requires we call srand() instead,
but might as well use random().
2018-03-06 17:44:25 -06:00
Will Dietz e917c05628 logging: Don't throw from Activity destructor
Move definition of destructor to logging.cc for
access to util.hh's ignoreException.
2018-03-06 10:42:58 -06:00
Shea Levy 088ef81759
ssh-ng: Don't forward options to the daemon.
This can be iterated on and currently leaves out settings we know we
want to forward, but it fixes #1713 and fixes #1935 and isn't
fundamentally broken like the status quo. Future changes are suggested
in a comment.
2018-03-05 07:42:15 -05:00
Will Dietz 6b9a03f5d8 hoist vector out of loop just in case 2018-03-02 10:52:05 -06:00
Will Dietz c89a3d5368 don't allocate large buffers on the stack 2018-03-02 10:52:04 -06:00
Shea Levy 3748a0ca1e
Merge branch 'improve-search-algorithm' of git://github.com/Ma27/nix 2018-03-02 11:41:08 -05:00
Shea Levy aa8bbbf69d
Merge branch 'write-failure-fixes' of git://github.com/lheckemann/nix 2018-03-02 10:59:59 -05:00
Tuomas Tynkkynen a0e38c16bc libexpr: Recognize newline in more places in lexer
Flex's regexes have an annoying feature: the dot matches everything
except a newline. This causes problems for expressions like:

"${0}\
"

where the backslash-newline combination matches this rule instead of the
intended one mentioned in the comment:

    <STRING>\$|\\|\$\\ {
                    /* This can only occur when we reach EOF, otherwise the above
                    (...|\$[^\{\"\\]|\\.|\$\\.)+ would have triggered.
                    This is technically invalid, but we leave the problem to the
                    parser who fails with exact location. */
                    return STR;
                }
However, the parser actually accepts the resulting token sequence
('"' DOLLAR_CURLY 0 '}' STR '"'), which is a problem because the lexer
rule didn't assign anything to yylval. Ultimately this leads to a crash
when dereferencing a NULL pointer in ExprConcatStrings::bindVars().

The fix does change the syntax of the language in some corner cases
but I think it's only turning previously invalid (or crashing) syntax
to valid syntax. E.g.

"a\
b"

and

''a''\
b''

were previously syntax errors but now both result in "a\nb".

Found by afl-fuzz.
2018-03-02 17:30:48 +02:00
Eelco Dolstra 939cf4cceb
Fix error message 2018-03-02 14:32:00 +01:00
Matthew O'Gorman 467fdd8ca4 only delete things older than current gen and update logic in doc as
well
2018-03-01 22:59:00 -05:00
Matthew O'Gorman 3c16044cb0 remove unused variable and make sure to check that the current
generation is not the one we are deleting
2018-03-01 22:22:02 -05:00
Matt O'Gorman 12fe2249e1 Update nix-env.cc
missing comma
2018-03-01 21:47:57 -05:00
Matthew O'Gorman 0312d30315 this updates issues that were addressed by people in pr 2018-03-01 21:47:57 -05:00
Matthew O'Gorman 429154b74c Implement --delete-generations + flag for keeping last N number of generations 2018-03-01 21:47:57 -05:00
Will Dietz e89d02bf03 fetchGit: use "HEAD" as default ref 2018-02-28 16:34:34 -06:00
Shea Levy 14ca85688c
Actually fix nixDataDir in non-canonical path 2018-02-28 06:19:40 -05:00
Linus Heckemann 919c3c20b3 seccomp: add 32-bit ARM on aarch64-linux
This allows building armv[67]l-linux derivations on compatible aarch64
machines. Failure to add the architecture may result from missing
hardware support, in which case we can't run 32-bit binaries and don't
need to restrict them with seccomp anyway,
2018-02-27 00:32:15 +00:00
Linus Heckemann eb03a296c1 Add build-extra-platforms setting
This allows specifying additional systems that a machine is able to
build for. This may apply on some armv7-capable aarch64 processors, or
on systems using qemu-user with binfmt-misc to support transparent
execution of foreign-arch programs.

This removes the previous hard-coded assumptions about which systems are
ABI-compatible with which other systems, and instead relies on the user
to specify any additional platforms that they have ensured compatibility
for and wish to build for locally.

NixOS should probably add i686-linux on x86_64-linux systems for this
setting by default.
2018-02-27 00:27:58 +00:00
Tuomas Tynkkynen 77e9e1ed91 libexpr: Fix prim_replaceStrings() to work on an empty source string
Otherwise, running e.g.

nix-instantiate --eval -E --strict 'builtins.replaceStrings [""] ["X"] "abc"'

would just hang in an infinite loop.

Found by afl-fuzz.

First attempt of this was reverted in e2d71bd186 because it caused
another infinite loop, which is fixed now and a test added.
2018-02-26 19:49:13 +02:00
Eelco Dolstra 24ec750003
nix run: Fix segfault on macOS
Note that clearenv() is not available on macOS.

Fixes #1907.
2018-02-26 18:29:40 +01:00
Will Dietz 009cf9cd23 nix search: explicitly handle empty search string, fixes #1893
This is important since this is given as an example.
Other patterns containing "empty search string" will still
be handled differently on different platforms ("asdf|")
but that's less of an issue.
2018-02-25 16:40:56 -06:00
Will Dietz 3cac8734ac nix search: fix bug where we wrote to cache when shouldn't, breaking
This is exposed by the tests added previously,
and resolves the error reported in #1892:
"expected JSON value".
2018-02-25 16:40:05 -06:00
Eelco Dolstra 64e486ab63
Fix downloadCached() with a chroot store
E.g.

  nix run --store ~/my-nix -f channel:nixos-17.03 hello -c hello

This problem was mentioned in #1897.
2018-02-23 13:00:42 +01:00
Eelco Dolstra 179b896acb
Merge branch 'data-dir-non-canon' of https://github.com/shlevy/nix 2018-02-22 14:20:43 +01:00
Eelco Dolstra e8d53bfdc9
Revert "Enable sandbox builds on Linux by default"
This reverts commit ddc58e7896.

https://hydra.nixos.org/eval/1435322
2018-02-22 14:20:07 +01:00
Shea Levy ddbcd01c83
Fix restricted mode when installing in non-canonical data dir 2018-02-22 07:18:14 -05:00
Eelco Dolstra ddc58e7896
Enable sandbox builds on Linux by default
The overhead of sandbox builds is a problem on NixOS (since building a
NixOS configuration involves a lot of small derivations) but not for
typical non-NixOS use cases. So outside of NixOS we can enable it.

Issue #179.
2018-02-22 12:27:25 +01:00
Eelco Dolstra de4c03d201
Merge branch 'fix/dry-run-partially' of https://github.com/dtzWill/nix 2018-02-22 12:18:20 +01:00
Eelco Dolstra 0d54671b7b
Manual: Update chapter on remote builds
Alos add a command "nix ping-store" to make it easier to see if Nix
can connect to a remote builder (e.g. 'nix ping-store --store
ssh://mac').
2018-02-21 16:24:26 +01:00
Eelco Dolstra e2d71bd186
Revert "libexpr: Fix prim_replaceStrings() to work on an empty source string"
This reverts commit 4ea9707591.

It causes an infinite loop in Nixpkgs evaluation,
e.g. "nix-instantiate -A hello" hung.

PR #1886.
2018-02-21 15:35:28 +01:00
Shea Levy a6c497f526
Merge branch 'nix-copy' of git://github.com/Mic92/nix-1 2018-02-20 21:05:17 -05:00
Jörg Thalheim fa7fd76c5e nix-copy: fix examples
maybe a left-over from nix-store -r ?
2018-02-21 01:13:46 +00:00
Tuomas Tynkkynen 546f98dace libutil: Fix invalid assert on decoding base64 hashes
The assertion is broken because there is no one-to-one mapping from
length of a base64 string to the length of the output.

E.g.

"1q69lz7Empb06nzfkj651413n9icx0njmyr3xzq1j9q=" results in a 32-byte output.
"1q69lz7Empb06nzfkj651413n9icx0njmyr3xzq1j9qy" results in a 33-byte output.

To reproduce, evaluate:

builtins.derivationStrict {
    name = "0";
    builder = "0";
    system = "0";
    outputHashAlgo = "sha256";
    outputHash = "1q69lz7Empb06nzfkj651413n9icx0njmyr3xzq1j9qy";
}

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Tuomas Tynkkynen 4ea9707591 libexpr: Fix prim_replaceStrings() to work on an empty source string
Otherwise, running e.g.

nix-instantiate --eval -E --strict 'builtins.replaceStrings [""] ["X"] "abc"'

would just hang in an infinite loop.

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Tuomas Tynkkynen 1d0e42879f libutil: Fix infinite loop in filterANSIEscapes on '\r'
E.g. nix-instantiate --eval -E 'abort "\r"' hangs.

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Tuomas Tynkkynen 056d28a601 libexpr: Don't create lots of temporary strings in Bindings::lexicographicOrder
Avoids ~180,000 string temporaries created when evaluating a headless
NixOS system.
2018-02-19 22:47:25 +02:00
Eelco Dolstra d4e93532e2
Fix incorrect (and unnecessary) format string
https://hydra.nixos.org/eval/1434547#tabs-now-fail
2018-02-19 20:46:39 +01:00
Eelco Dolstra 70eb64147e
Update release notes
Also add some examples to nix --help.
2018-02-19 20:38:53 +01:00
Eelco Dolstra 623fcb071e
Merge pull request #1882 from shlevy/no-seccomp-no-filterSyscalls
Don't silently succeed seccomp setup when !HAVE_SECCOMP.
2018-02-19 17:39:46 +01:00
Shea Levy e1eb63a586
Merge branch 'perf-fixes' of git://github.com/dezgeg/nix 2018-02-19 10:11:52 -05:00
Shea Levy e59a8a63e1
Don't silently succeed seccomp setup when !HAVE_SECCOMP.
Running Nix with build users without seccomp on Linux is dangerous,
and administrators should very explicitly opt-in to it.
2018-02-19 09:56:24 -05:00
Eelco Dolstra ed73d40c3b
Config::handleUnknownSettings(): Remove unused 'fatal' argument 2018-02-19 14:00:34 +01:00
Eelco Dolstra 75a1d96cfd
Merge branch 'register-settings' of https://github.com/shlevy/nix 2018-02-19 13:58:34 +01:00
Eelco Dolstra 7fe5910bf8
Merge pull request #1857 from dtzWill/fix/check-for-lzma-mt
configure.ac: check if lzma has MT support, fix deb build/etc.
2018-02-19 13:06:14 +01:00
Shea Levy 690ac7c90b
configure: Add a flag to disable seccomp.
This is needed for new arches where libseccomp support doesn't exist
yet.

Fixes #1878.
2018-02-18 02:35:01 -05:00
Tuomas Tynkkynen 37264ed0ad libexpr: Avoid an unnecessary string copy in prim_derivationStrict 2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen 66eeff3345 libexpr: Remove unnecessary drvName assignment in prim_derivationStrict
drvName is already assigned to the same value right at the start of the
function.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen 7e0360504d libexpr: Optimize prim_derivationStrict by using more symbol comparisons 2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen 0845cdf944 libexpr: Rely on Boehm returning zeroed memory in EvalState::allocEnv()
Boehm guarantees that memory returned by GC_malloc() is zeroed, so take
advantage of that.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen b8bed7da14 libexpr: Optimize prim_attrNames a bit
Instead of having lexicographicOrder() create a temporary sorted array
of Attr*:s and copying attr names from that, copy the attr names
first and then sort that.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen f67a7007a2 libexpr: Pre-reserve space in string in unescapeStr()
Avoids some malloc() traffic.
2018-02-16 04:39:43 +02:00
Eelco Dolstra 3a5a241b32
Merge pull request #1873 from matthewbauer/fix-manpath
Set MANPATH in case man path isn’t set correctly.
2018-02-15 17:37:21 +01:00
Matthew Bauer 8f186722a9 Set backup MANPATH in case man path isn’t set correctly.
Previously, this would fail at startup for non-NixOS installs:

nix-env --help

The fix for this is to just use "nixManDir" as the value for MANPATH
when spawning "man".

To test this, I’m using the following:

$ nix-build release.nix -A build
$ MANPATH= ./result/bin/nix-env --help

Fixes #1627
2018-02-15 10:20:02 -06:00
Shea Levy ac973a6d3c
monitor-fds: Fix on macOS.
Fixes #1871.
2018-02-14 18:26:37 -05:00
Shea Levy b095c06139
Add splitVersion primop.
Fixes #1868.
2018-02-14 09:55:43 -05:00
Shea Levy de4934ab3b
Allow plugins to define new settings. 2018-02-13 14:43:32 -05:00
Eelco Dolstra 52c777a793
Merge pull request #1863 from shlevy/conf-includes
Allow includes from nix.conf
2018-02-13 17:33:07 +01:00
Shea Levy b8739f2fb3
Enable specifying directories in plugin-files. 2018-02-13 11:25:01 -05:00
Linus Heckemann 78ac3eb4eb Fix #1762
nix-store --export, nix-store --dump, and nix dump-path would previously
fail silently if writing the data out failed, because
 a) FdSink::write ignored exceptions, and
 b) the commands relied on FdSink's destructor, which ignores
    exceptions, to flush the data out.

This could cause rather opaque issues with installing nixos, because
nix-store --export would happily proceed even if it couldn't write its
data out (e.g. if nix-store --import on the other side of the pipe
failed).

This commit adds tests that expose these issues in the nix-store
commands, and fixes them for all three.
2018-02-13 14:26:19 +00:00
Shea Levy 6eb1040e90
Allow includes from nix.conf 2018-02-13 08:16:32 -05:00
Eelco Dolstra 7828dca9e8
Merge branch 'register-constant' of https://github.com/shlevy/nix 2018-02-13 12:24:48 +01:00
Eelco Dolstra 1c10a74c73
Merge branch 'plugins' of https://github.com/shlevy/nix 2018-02-13 12:15:27 +01:00
Eelco Dolstra c5cc57e962
Merge pull request #1859 from FRidh/flatten
Nix stats: flatten statistics
2018-02-13 11:11:12 +01:00
Eelco Dolstra 9bcb4d2dd9 Fix hang in build-remote 2018-02-12 22:48:55 +01:00
Eelco Dolstra 4f09ce7940
Fix 'deadlock: trying to re-acquire self-held lock'
This was caused by derivations with 'allowSubstitutes = false'. Such
derivations will be built locally. However, if there is another
SubstitionGoal that has the output of the first derivation in its
closure, then the path will be simultaneously built and substituted.

There was a check to catch this situation (via pathIsLockedByMe()),
but it no longer worked reliably because substitutions are now done in
another thread. (Thus the comment 'It can't happen between here and
the lockPaths() call below because we're not allowing multi-threading'
was no longer valid.)

The fix is to handle the path already being locked in both
SubstitutionGoal and DerivationGoal.
2018-02-12 17:06:06 +01:00
Eelco Dolstra 35fd31770c
toBuildables -> build 2018-02-12 17:06:06 +01:00
Will Dietz c6209030c4 compression: make parallel sink separate class 2018-02-11 13:50:28 -06:00
Will Dietz a0bdc96726 compression: print warning if parallel requested but not supported 2018-02-11 13:03:47 -06:00
Frederik Rietdijk 60eca58533 Nix stats: flatten statistics
Flattens the list of statistics as suggested in
https://github.com/NixOS/ofborg/issues/67. This makes it easier to work
with.
2018-02-11 14:37:50 +01:00
Will Dietz 5a082ad15a configure.ac: check if lzma has MT support, fix deb build/etc. 2018-02-09 21:02:25 -06:00
Eelco Dolstra 960e9c560e
nix: Ensure that the user sees errors from substituters 2018-02-09 15:06:47 +01:00
Eelco Dolstra aa02cdc33c
getDefaultSubstituters(): Skip broken substituters
Fixes #1340.
2018-02-09 15:06:47 +01:00
Eelco Dolstra 3d2d207aad
Merge pull request #1848 from AmineChikhaoui/parallel-xz
support multi threaded xz encoder
2018-02-09 15:03:25 +01:00
Shea Levy 081f14a169
Allow using RegisterPrimop to define constants.
This enables plugins to add new constants, as well as new primops.
2018-02-08 14:35:50 -05:00
Shea Levy 88cd2d41ac
Add plugins to make Nix more extensible.
All plugins in plugin-files will be dlopened, allowing them to
statically construct instances of the various Register* types Nix
supports.
2018-02-08 12:44:37 -05:00
Eelco Dolstra ad97a21834
nix-env: Fix parsing of --system
https://hydra.nixos.org/build/68827814
2018-02-08 15:25:03 +01:00
AmineChikhaoui 0685a6480a
Merge branch 'master' of https://github.com/NixOS/nix into parallel-xz 2018-02-08 12:24:48 +01:00
Will Dietz 98031b6050 nix build: Don't create output links with --dry-run.
Fixes #1849.
2018-02-07 15:20:54 -06:00
Will Dietz f201b7733e More completely fix recursive nix, unbreak tests
See:
88b5d0c8e8 (commitcomment-27406365)
2018-02-07 15:19:28 -06:00
AmineChikhaoui a56637205a
Merge branch 'master' of https://github.com/NixOS/nix into parallel-xz 2018-02-07 21:09:04 +01:00
AmineChikhaoui 47ad88099b
move the parallel-compression setting to binary-cache-store, the setting
can be done now from the url e.g s3://nix-cache?parallel-compression=1
instead of nix.conf.
2018-02-07 21:06:11 +01:00
Eelco Dolstra 88b5d0c8e8
Prevent accidental recursive Nix 2018-02-07 19:07:38 +01:00
AmineChikhaoui 55ecdfe2a8
make multi threaded compression configurable and use single threaded
by default.
2018-02-07 17:54:08 +01:00
AmineChikhaoui 163e39547a
Merge branch 'master' of https://github.com/NixOS/nix into parallel-xz 2018-02-07 17:53:50 +01:00
Eelco Dolstra 84989d3af2
Improve filtering of ANSI escape sequences in build logs
All ANSI sequences except color setting are now filtered out. In
particular, terminal resets (such as from NixOS VM tests) are filtered
out.

Also, fix the completely broken tab character handling.
2018-02-07 15:23:57 +01:00
Eelco Dolstra abe6be578b
Merge pull request #1816 from shlevy/add-path
Add path primop.
2018-02-07 13:32:35 +01:00
AmineChikhaoui 9d1e22f743
set block size to 0 to let the lzma lib choose the right one, add
some comments about possible improvements wrt memory usage/threading.
2018-02-07 11:18:55 +01:00
Shea Levy 69d82e5c58
Add path primop.
builtins.path allows specifying the name of a path (which makes paths
with store-illegal names now addable), allows adding paths with flat
instead of recursive hashes, allows specifying a filter (so is a
generalization of filterSource), and allows specifying an expected
hash (enabling safe path adding in pure mode).
2018-02-06 16:48:08 -05:00
AmineChikhaoui bc7e3a4dd6
support multi threaded xz encoder, this might be particularly useful in
the case of hydra where the overhead of single threaded encoding is more
noticeable e.g most of the time spent in "Sending inputs"/"Receiving outputs"
is due to compression while the actual upload to the binary cache seems
to be negligible.
2018-02-06 22:42:02 +01:00
Eelco Dolstra 6f6bfc8205 Update the progress bar at most 20 times per second
Fixes #1834.
2018-02-06 20:51:37 +01:00
Eelco Dolstra 43f8ef73c6
realiseContext(): Add derivation outputs to the allowed paths
This makes import-from-derivation work in restricted mode again.
2018-02-06 15:38:45 +01:00
Eelco Dolstra f24e726ba5
checkURI(): Check file URIs against allowedPaths
This makes e.g. 'fetchGit ./.' work (assuming that ./. is an allowed
path).
2018-02-06 14:35:33 +01:00
Eelco Dolstra 55012ec0b9
Tweak progress bar message
Say "copying" instead of "fetching" when copying from another local
store. Nice for nixos-install.
2018-02-05 18:32:23 +01:00
Eelco Dolstra bb1d046f5c
Allow substituters to be marked as trusted
This is needed by nixos-install, which uses the Nix store on the
installation CD as a substituter. We don't want to disable signature
checking entirely because substitutes from cache.nixos.org should
still be checked. So now we can pas "local?trusted=1" to mark only the
Nix store in /nix as not requiring signatures.

Fixes #1819.
2018-02-05 18:08:30 +01:00
Eelco Dolstra 2175eee9fe
Fix segfault using non-binary cache stores as substituters 2018-02-05 17:46:43 +01:00
Eelco Dolstra 84722d67d2
Remove nix-build --hash
Instead, if a fixed-output derivation produces has an incorrect output
hash, we now unconditionally move the outputs to the path
corresponding with the actual hash and register it as valid. Thus,
after correcting the hash in the Nix expression (e.g. in a fetchurl
call), the fixed-output derivation doesn't have to be built again.

It would still be good to have a command for reporting the actual hash
of a fixed-output derivation (instead of throwing an error), but
"nix-build --hash" didn't do that.
2018-02-03 10:08:05 +01:00
Shea Levy de96daf54f
Merge branch 'master' of git://github.com/catern/nix 2018-02-01 13:21:45 -05:00
Eelco Dolstra 19477e8815
nix-build: Ignore --indirect
Note that nix-build always creates indirect roots.

Fixes #1830.
2018-02-01 17:09:56 +01:00
Eelco Dolstra 855699855f
Remove obsolete references to manifests
Closes #323.
2018-02-01 10:39:16 +01:00
Spencer Baugh e5432574e2 document ability to set NIX_REMOTE=unix://path/to/socket 2018-01-31 22:47:16 +00:00
Eelco Dolstra c129fc6ee8
Merge pull request #1801 from catern/master
remote_store: register for NIX_REMOTE=unix://path
2018-01-31 23:12:39 +01:00
Eelco Dolstra 88f4f0231b Fix building without aws-sdk-cpp 2018-01-31 22:34:51 +01:00
Eelco Dolstra c287d73121
Rename 1.12 -> 2.0
Following discussion with Shea and Graham. It's a big enough change
from the last release. Also, from a semver perspective, 2.0 makes more
sense because we did remove some interfaces (like nix-pull/nix-push).
2018-01-31 18:58:45 +01:00
Eelco Dolstra 6fa690291a
Add 'nix upgrade-nix' command
This command upgrades Nix to the latest stable version by installing a
store path obtained from

  https://github.com/NixOS/nixpkgs/raw/master/nixos/modules/installer/tools/nix-fallback-paths.nix

which is the same store path that the installer at
https://nixos.org/nix/install.sh uses.

The upgrade fails if Nix is not installed in a profile (e.g. on NixOS,
or when installed outside of the Nix store).
2018-01-31 16:24:43 +01:00
Eelco Dolstra 30370f168f
Cleanup 2018-01-31 15:14:03 +01:00
Eelco Dolstra 6270b2e50f
Merge branch 'http-binary-cache-put-upsert' of https://github.com/adelbertc/nix 2018-01-31 15:10:12 +01:00
Eelco Dolstra 478e3e4649
Indent properly 2018-01-31 13:46:38 +01:00
Spencer Baugh 746f8aed86 remote_store: register for NIX_REMOTE=unix://path
This allows overriding the socket path so the daemon may be listening at
an arbitrary Unix domain socket location.

Fixes #1800
2018-01-26 22:05:16 +00:00
Shea Levy 1d5d277ac7
HttpBinaryCacheStore: Support upsertFile with PUT.
Some servers, such as Artifactory, allow uploading with PUT and BASIC
auth. This allows nix copy to work to upload binaries to those
servers.

Worked on together with @adelbertc
2018-01-26 11:12:30 -08:00
Eelco Dolstra e09161d05c
Remove signed-binary-caches as the default for require-sigs
This was for backward compatibility. However, with security-related
configuration settings, it's best not to have any confusion.

Issue #495.
2018-01-26 17:12:15 +01:00
Eelco Dolstra 479692a068
Merge pull request #1811 from dtzWill/update/json-3.0.1
nlohmann-json: 2.1.1 -> 3.0.1
2018-01-24 17:11:25 +01:00
Dan Peebles d43a8b25f0 Fix obscure corner case in name resolution for builtin:fetchurl in sandboxed environments 2018-01-23 14:45:50 -05:00
Will Dietz f7c26365eb nlohmann-json: 2.1.1 -> 3.0.1 2018-01-22 12:19:50 -06:00
Eelco Dolstra 89a2a11d9f
Don't use [[noreturn]] 2018-01-19 15:00:38 +01:00
Eelco Dolstra 3c4c30eadd
Rewrite builtin derivation environment
Also add a test.

Fixes #1803.
Closes #1805.
2018-01-19 15:00:38 +01:00
Eelco Dolstra 2896bb6826
Don't retry CURLE_URL_MALFORMAT 2018-01-19 14:05:08 +01:00
Eelco Dolstra 0c95776c3e
Don't define builtins.{currentSystem,currentTime} in pure mode
This makes it easier to provide a default, e.g.

  system = builtins.currentSystem or "x86_64-linux";
2018-01-18 16:38:48 +01:00
Eelco Dolstra 27b510af5c
nix eval: Stop progress bar before printing the result 2018-01-17 12:04:44 +01:00
Eelco Dolstra 16e0287556
nix eval: Take only one argument
Thus --json no longer produces a list.
2018-01-17 12:03:06 +01:00
Eelco Dolstra d8b4cfad82
Typo 2018-01-17 11:53:08 +01:00
Eelco Dolstra 75b9670df6
Make show-trace a config setting 2018-01-16 19:24:23 +01:00
Eelco Dolstra d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Eelco Dolstra 23fa7e3606
parseExprFromFile -> evalFile
parseExprFromFile() should be avoided since it doesn't cache anything.
2018-01-16 17:11:58 +01:00
Eelco Dolstra ba75c69e00
Barf when using a diverted store on macOS
Fixes #1792.
2018-01-15 12:14:43 +01:00
Will Dietz 6454ca393a nix log: use pager 2018-01-12 15:05:26 -06:00
Eelco Dolstra 74f75c8558
import, builtins.readFile: Handle diverted stores
Fixes #1791
2018-01-12 17:31:08 +01:00
Renzo Carbonara b0328c244d nix.conf: builders-use-substitutes
Fixes #937
2018-01-09 22:40:07 +01:00
Will Dietz 428680b307 fetchGit: fix creation of uninitialized cache dir, let git create it
fetchGit test (as modified in previous commit) now passes.
2018-01-09 09:05:18 -06:00
Eelco Dolstra 7b9583680e
Improve error message with --repair for untrusted users 2018-01-08 19:13:48 +01:00
Eelco Dolstra 44272d8719
Rename "use-substitutes" to "substitute"
Commit c2154d4c84 renamed
"build-use-substitutes" to "use-substitutes", but that broke
"nix-copy-closure --use-substitutes".
2018-01-04 16:58:39 +01:00
Shea Levy 689b2783fc
Add hasContext primop 2018-01-02 12:25:14 -05:00
Will Dietz 9dd2b8ac7b use libbrotli directly when available
* Look for both 'brotli' and 'bro' as external command,
  since upstream has renamed it in newer versions.
  If neither are found, current runtime behavior
  is preserved: try to find 'bro' on PATH.
* Limit amount handed to BrotliEncoderCompressStream
  to ensure interrupts are processed in a timely manner.
  Testing shows negligible performance impact.
  (Other compression sinks don't seem to require this)
2017-12-30 20:26:33 -06:00
Shea Levy 6a0dd63508
Merge branch 'fix/issue-1757' of git://github.com/dtzWill/nix 2017-12-27 18:51:05 -05:00
Will Dietz bd17ccf1d8 nix repl: use linenoiseKeyType to differentiate ^C and ^D
Fixes #1757.
2017-12-26 19:25:50 -06:00
Will Dietz 2e6f06c37e fetchGit: Fix handling of local repo when not using 'master' branch
Add tests checking this behavior.
2017-12-22 15:29:52 -06:00
Eelco Dolstra aa43cbb764
Check aws-sdk-cpp version 2017-12-22 12:05:13 +01:00
Will Dietz 71987b18d4 linenoise.cpp: allow completions from empty input
Fixes #1742.
2017-12-19 12:52:10 -06:00
Eelco Dolstra 8efe937a35 Fix build on gcc 7
Fixes #1738.
2017-12-16 00:49:31 +01:00
Ben Gamari 626a94d70e edit: Catch stoi exceptions from line number parsing 2017-12-14 19:11:56 -05:00
Ben Gamari f9bcbddef2 json-to-value: Throw sensible error message on invalid numbers 2017-12-14 19:09:45 -05:00
Eelco Dolstra 5d5b931fb1 Fix build against current aws-sdk-cpp 2017-12-14 16:45:40 +01:00
Maximilian Bosch 0413aeb35d
search.cc: sort attribute names with std::map 2017-12-12 15:13:36 +01:00
Eelco Dolstra 1ca8e95178
Merge pull request #1722 from bhipple/fix-for-older-libcurl
Fix for builds with system libcurl < 7.30
2017-12-12 11:48:08 +01:00
Will Dietz 00e0c416ff Mark comparison call operator as const 2017-12-11 12:08:40 -06:00
Will Dietz e4d43f1f94 logging.cc: add missing 'override' 2017-12-11 12:08:40 -06:00
Will Dietz c800f47395 Fix forward-decl of Bindings as "class", match definition.
(appease clang -Wmismatched-tags warning)
2017-12-11 11:51:55 -06:00
Shea Levy 5ba5ca7888
Merge branch 'feature/linenoise-ng' of git://github.com/dtzWill/nix 2017-12-11 07:52:20 -05:00
Benjamin Hipple d76c4fe770 Fix for builds with system libcurl < 7.30
CentOS 7.4 and RHEL 7.4 ship with libcurl-devel-7.29.0-42.el7.x86_64; this flag
was added in 7.30.0
https://curl.haxx.se/libcurl/c/CURLMOPT_MAX_TOTAL_CONNECTIONS.html
2017-12-09 20:02:21 -05:00
Eelco Dolstra 2df9cbeb47 Provide random access to cached NARs
E.g.

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \
    /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null
  real    0m4.139s

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \
    /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null
  real    0m0.024s

(Before, the second call took ~0.220s.)

This will use a NAR listing in
/tmp/nars/b0w2hafndl09h64fhb86kw6bmhbmnpm1.ls containing all metadata,
including the offsets of regular files inside the NAR. Thus, we don't
need to read the entire NAR. (We do read the entire listing, but
that's generally pretty small. We could use a SQLite DB by borrowing
some more code from nixos-channel-scripts/file-cache.hh.)

This is primarily useful when Hydra is serving files from an S3 binary
cache, in particular when you have giant NARs. E.g. we had some 12 GiB
NARs, so accessing individuals files was pretty slow.
2017-12-07 01:07:07 +01:00
Eelco Dolstra 338f29dbd4 nix ls-{nar,store}: Return offset of files in the NAR if known
E.g.

  $ nix ls-store --json --recursive --store https://cache.nixos.org /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79 \
    | jq .entries.bin.entries.blender.narOffset
  400
2017-12-07 01:07:07 +01:00
Daiderd Jordan 2cb8aaa610
ssh-store: fix length when writing nar
This fixes nix copy and other things that use copyStorePath.
2017-12-06 11:41:08 +01:00
Shea Levy 11a7f8ce14
Merge branch 'fetchGit-fast-revision-update' 2017-12-05 11:16:39 -05:00
Eelco Dolstra 7f2c324ed1
Simplify build by including nlohmann/json.hpp 2017-12-04 17:11:36 +01:00
Eelco Dolstra f1efb97075
Fix #1635. 2017-12-04 13:19:16 +01:00
Will Dietz 79f5c296c0 ignore "interrupted" exception in progress callback
Context/discusson:

https://github.com/NixOS/nix/issues/1692#issuecomment-348282301
2017-11-30 12:45:54 -06:00
Linus Heckemann f986a44980 Fix "Unexpected EOF reading a line" error
propagated-user-env-packages files in nixpkgs aren't all terminated by
newlines, as buildenv expected. Now it does not require a terminating
newline; note that this introduces a behaviour change: propagated user
env packages may now be spread across multiple lines. However, nix
1.11.x still expects them to be on a single line so this shouldn't be
used in nixpkgs for now.
2017-11-29 11:56:15 +00:00
Eelco Dolstra b52846ab5b
Show log tail when a remote build fails 2017-11-28 15:36:28 +01:00
Will Dietz 964349e44d no "linenoiseFree" in linenoise-ng 2017-11-27 18:30:05 -06:00
Will Dietz 21f515724c replace vendored linenoise with linenoise-ng 2017-11-27 18:27:28 -06:00
aszlig 6567ab95a0
build-remote: Fix missing log output
The storeUri variable in the build-remote hook is declared very much to
the start of the main function and a bunch of lines later, the same
variable gets checked via hasPrefix() but it gets assigned *after* that
check when the most suitable machine for the build was choosen.

So I guess this was just a typo in d16fd24973
and what we really want is to either checkd the prefix *after* assigning
storeUri or use bestMachine->storeUri directly.

I choose the latter, because the former could introduce even more
regressions if the try block where the variable gets assigned terminates
early.

Nevertheless, the reason why the log output didn't work is because
hasPrefix() checked for "ssh://" in front of storeUri, but if the
storeUri isn't set correctly (or at all), we don't get the log file
descriptor set up properly, leading to no log output.

I've adjusted the remote-builds test to include a regression test for
this, so that we can make sure we get a build output when using remote
builds.

In addition to that I've tested this with two of my build farms and the
build logs are emitted correctly again.

Signed-off-by: aszlig <aszlig@nix.build>
2017-11-25 01:34:18 +01:00
Eelco Dolstra 549c3706a5
nix path-info: Show URL of NARs in binary caches 2017-11-24 18:08:50 +01:00
Eelco Dolstra 90948a4e3a
nix-shell/nix-build: Support .drv files again
Fixes #1663.

Also handle '!<output-name>' (#1694).
2017-11-24 18:08:35 +01:00
Shea Levy eedbc4e06c
fetchGit: Ignore tarballTtl if rev is set and not in the repo.
Fixes #1697.
2017-11-24 06:09:24 -05:00
Eelco Dolstra d7da6c9ea9
fetchGit/fetchMercurial: Fix directory inclusion check
E.g. the existence of .gitignore would cause .git to be included.
2017-11-21 19:34:46 +01:00
Eelco Dolstra 6cdaa858d0
Propagate flags like --sandbox to the daemon properly 2017-11-21 18:50:56 +01:00
Eelco Dolstra f5f29dc2b7
nix eval --raw: Use coerceToString()
Thus,

  $ nix eval --raw '("foo")'
  foo

  $ nix eval --raw nixpkgs.hello
  /nix/store/1y6ckg6khrdsvll54s5spcmf3w6ka9k4-hello-2.10

  $ nix eval --raw '(/etc/resolv.conf)'
  /nix/store/vml92ama92i8mz013nny461mlvg8mvap-resolv.conf
2017-11-20 18:27:29 +01:00
Eelco Dolstra 1ff01187e2
nix run: Fix "flag '--command' requires 2 argument(s)" 2017-11-20 18:07:58 +01:00
Eelco Dolstra 4eb9e20028
nix run: Fix accidental removal of /nix/store existence check
Parenthetical to #1686, we don't need to create a new root if we can
just bind-mount on top of the existing /nix/store.
2017-11-20 17:59:32 +01:00
Eelco Dolstra a3aa850f0f
nix run: Ignore non-directories while setting up the chroot
Fixes #1686.
2017-11-20 17:58:47 +01:00
Eelco Dolstra 91a1987607
signed-binary-caches -> require-sigs
Unlike signed-binary-caches (which could only be '*' or ''),
require-sigs is a proper Boolean option. The default is true.
2017-11-20 17:44:07 +01:00
Eelco Dolstra d0b88db441
Cleanup 2017-11-20 17:32:58 +01:00
Eelco Dolstra 7a2b64e55c
binary-cache-public-keys -> trusted-public-keys
The name had become a misnomer since it's not only for substitution
from binary caches, but when adding/copying any
(non-content-addressed) path to a store.
2017-11-20 17:32:34 +01:00
Eelco Dolstra 72804dc0bd
Merge pull request #1645 from twhitehead/stat-race
Fix (highly unlikely) race condition in readLink
2017-11-20 17:32:27 +01:00
Eelco Dolstra 7474ac871b
nix copy: Abbreviate "daemon" 2017-11-20 15:17:11 +01:00
Eelco Dolstra 8956ae1987
Add a "profile" option to S3BinaryCacheStore
This allows specifying the AWS configuration profile to use. E.g.

  nix copy --from s3://my-cache?profile=aws-dev-account /nix/store/cf3isrlqavvd5w7rpky1fa8j9lcnlggm-...
2017-11-15 14:18:20 +01:00
Will Dietz 9d7ce0bf45 anchor nix::Exit exception 2017-11-14 12:39:30 -06:00
Eelco Dolstra ec5b04862b
nix sign-paths: Support binary caches 2017-11-14 18:44:05 +01:00
Eelco Dolstra 2c39e4eca0
Revert "Don't parse "x:x" as a URI"
This reverts commit f90f660b24.

This broke Hydra's release.nix, which contained

  preCheck = ''export LOGNAME=${LOGNAME:-foo}'';
2017-11-14 15:10:52 +01:00
Eelco Dolstra 5773d667ee
nix ls-{nar,store}: Don't abort on missing files 2017-11-14 14:49:06 +01:00
Eelco Dolstra 4db0a9555e
nix ls-{nar,store} --json: Respect -R 2017-11-14 14:31:38 +01:00
Eelco Dolstra c8155e9f5f
Don't indent JSON output 2017-11-14 14:28:03 +01:00
Eelco Dolstra bac8055652
nix ls-{store,nar}: Add --json flag 2017-11-14 14:23:53 +01:00
Eelco Dolstra 5ef5d70b5e
Make config options available to legacy commands 2017-11-14 14:04:09 +01:00
Eelco Dolstra 9d87d03331
Remove extraneous comment 2017-11-08 16:17:32 +01:00
Eelco Dolstra 513b143cd8
Merge pull request #1650 from copumpkin/darwin-sandbox-unix-socket
Always allow builds to use unix domain sockets in Darwin sandbox
2017-11-08 16:16:42 +01:00
Eelco Dolstra dc30856141
Merge pull request #1632 from AmineChikhaoui/sigint-copy
run query paths in parallel during nix copy and handle SIGINT
2017-11-06 13:36:28 +01:00
Eelco Dolstra 7a4d9574d9
fetchgit.cc -> fetchGit.cc 2017-11-03 13:55:31 +01:00
Eelco Dolstra 6cf7c6a6b0
Remove git:// support in NIX_PATH
This didn't support specifying a revision/branch, and was restricted
to git:// URIs (since https:// or ssh:// would be ambiguous).
2017-11-03 13:55:31 +01:00
Eelco Dolstra ee6ac38848
fetchGit/fetchMercurial: Filter out directories with untracked files 2017-11-03 13:55:31 +01:00
Eelco Dolstra 4dee01da7c
fetchGit: Add a test 2017-11-03 13:55:30 +01:00
Eelco Dolstra 0e77aa3982
fetchGit: Don't do a remote fetch if we already have the rev 2017-11-03 13:55:30 +01:00
Eelco Dolstra 9ccea31dc2
Merge pull request #1651 from LnL7/darwin-sandbox-getpwuid
Allow getpwuid in the darwin sandbox
2017-11-03 10:55:31 +01:00
Daiderd Jordan 453f675810
Allow getpwuid in the darwin sandbox. 2017-11-03 10:50:49 +01:00
Eelco Dolstra e104842f8b
Merge pull request #1660 from 4z3/patch-1
fetchMercurial: fix error message
2017-11-03 10:49:57 +01:00
Eelco Dolstra 4070264613
Merge pull request #1655 from copumpkin/patch-1
Don't freak out if we get a 403 from S3
2017-11-03 10:49:38 +01:00
Daniel Peebles 3105679226 Don't freak out if we get a 403 from S3
As far as we're concerned, not being able to access a file just means
the file is missing. Plus, AWS explicitly goes out of its way to
return a 403 if the file is missing and the requester doesn't have
permission to list the bucket.

Also getting rid of an old hack that Eelco said was only relevant
to an older AWS SDK.
2017-11-03 01:31:42 +01:00
tv 5ab37f0e44
fetchMercurial: fix error message 2017-11-02 23:37:42 +01:00
Eelco Dolstra 212e72c609 Fix build
https://hydra.nixos.org/build/63172338
2017-11-01 21:32:30 +01:00
Eelco Dolstra e026bc3b05
fetchMercurial: Don't fetch hashes we already have 2017-11-01 18:43:11 +01:00
Eelco Dolstra 1969f357b7
Add fetchMercurial primop
E.g.

  $ nix eval '(fetchMercurial https://www.mercurial-scm.org/repo/hello)'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "82e55d328c8ca4ee16520036c0aaace03a5beb65"; revCount = 1; shortRev = "82e55d328c8c"; }

  $ nix eval '(fetchMercurial { url = https://www.mercurial-scm.org/repo/hello; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; })'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; revCount = 0; shortRev = "0a04b987be5a"; }

  $ nix eval '(fetchMercurial /tmp/unclean-hg-tree)'
  { branch = "default"; outPath = "/nix/store/cm750cdw1x8wfpm3jq7mz09r30l9r024-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "000000000000"; }
2017-11-01 17:45:32 +01:00
Eelco Dolstra cd532a9251
Fix filterSource 2017-10-31 16:18:32 +01:00
Dan Peebles bc6b3f7e8f Always allow builds to use unix domain sockets in Darwin sandbox 2017-10-31 15:33:57 +01:00
Eelco Dolstra 72cd52c3cd
builtins.fetchgit: Support importing a working tree
For example, you can write

  src = fetchgit ./.;

and if ./. refers to an unclean working tree, that tree will be copied
to the Nix store. This removes the need for "cleanSource".
2017-10-30 19:59:25 +01:00
Eelco Dolstra 197922ea4e
Merge pull request #1646 from copumpkin/optional-sandbox-local-network
Allow optional localhost network access to sandboxed derivations
2017-10-30 18:54:40 +01:00
Dan Peebles 4a4a009f78 Allow optional localhost network access to sandboxed derivations
This will allow bind and connect to 127.0.0.1, which can reduce purity/
security (if you're running a vulnerable service on localhost) but is
also needed for a ton of test suites, so I'm leaving it turned off by
default but allowing certain derivations to turn it on as needed.

It also allows DNS resolution of arbitrary hostnames but I haven't found
a way to avoid that. In principle I'd just want to allow resolving
localhost but that doesn't seem to be possible.

I don't think this belongs under `build-use-sandbox = relaxed` because we
want it on Hydra and I don't think it's the end of the world.
2017-10-30 17:59:12 +01:00
Eelco Dolstra f90f660b24
Don't parse "x:x" as a URI
URIs now have to contain "://" or start with "channel:".
2017-10-30 17:58:01 +01:00
Tyson Whitehead 07d2c6d213
Fix (highly unlikely) race condition in readLink
Used to determine symlink size with stat and value with readlink.
This could technically result in garbage if symlink changed between
calls.  Also gets around the broken stat implementation in our
network filesystem (returns size + 1 giving a byte of garbage).
2017-10-30 11:49:55 -04:00
Eelco Dolstra 12991152be
nix-build: Fix --hash 2017-10-30 13:31:05 +01:00
Eelco Dolstra 63c80ae26f
Make "fetchGit /path" work 2017-10-30 13:18:28 +01:00
Eelco Dolstra a5c392a80e
fetchGit: Fix broken assertion
Different URIs can map to the same cache entry if they have the same
revision.
2017-10-30 12:55:46 +01:00
Eelco Dolstra 812e027e1d
Add option allowed-uris
This allows network access in restricted eval mode.
2017-10-30 12:41:49 +01:00
Eelco Dolstra e38382895d
builtins.fetchGit: Return an attrset with revision info
This adds rev, shortRev and revCount attributes, equal to what Hydra
provides. E.g.

  $ nix eval '(fetchGit https://github.com/NixOS/patchelf.git)'
  { outPath = "/nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source"; rev = "29c085fd9d3fc972f75b3961905d6b4ecce7eb2b"; revCount = 303; shortRev = "29c085f"; }
2017-10-30 11:49:03 +01:00
Eelco Dolstra f9686885be
enable-http2 -> http2 2017-10-30 11:00:59 +01:00
Eelco Dolstra 049322702b
fetchgit -> fetchGit
Almost all other primops are camelCase so no reason not to use that
here.
2017-10-30 10:25:08 +01:00
Eelco Dolstra 23ce4b3393
fetchTarball: Use "source" as the default name
This ensures that it produces the same output as fetchgit:

  $ nix eval --raw '(builtins.fetchgit https://github.com/NixOS/patchelf.git)'
  /nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source

  $ nix eval --raw '(fetchTarball https://github.com/NixOS/patchelf/archive/master.tar.gz)'
  /nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source
2017-10-30 10:22:58 +01:00
Eelco Dolstra 66ddbef754
fetchurl/fetchTarball: Respect name changes
The computation of urlHash didn't take the name into account, so
subsequent fetchurl calls with the same URL but a different name would
resolve to the same cached store path.
2017-10-30 10:22:58 +01:00
Eelco Dolstra f25791c196
builtins.fetchgit: Don't mess up the progress indicator 2017-10-30 10:22:58 +01:00
Eelco Dolstra 65b5f177b5
builtins.fetchgit: Support a "name" attribute
The "name" attribute defaults to "source", which we should use for all
similar functions (e.g. fetchTarball and in Hydra) to ensure that we
get a consistent store path regardless of how the tree is fetched.

"source" is not necessarily a correct label, but using an empty name
is problematic: you get an ugly store path ending in a dash, and it's
impossible to have a fixed-output derivation that produces that path
because ".drv" is not a valid store name.

Fixes #904.
2017-10-30 10:22:07 +01:00
Eelco Dolstra c1ae18941a
nix repl :t: Indicate strings that have a context 2017-10-29 17:11:22 +01:00
AmineChikhaoui 0f9a7225ab respect SIGINT in nix copy during the paths queries #1629 2017-10-25 16:51:45 +01:00
AmineChikhaoui 54a2cd9ce4 Merge branch 'master' of github.com:NixOS/nix into sigint-copy 2017-10-25 16:50:52 +01:00
Eelco Dolstra 9971d875a4
Fix building on clang
https://hydra.nixos.org/build/62945761
2017-10-25 17:20:47 +02:00
AmineChikhaoui 9f01a3f0a8 attempt to fix #1630: make the queries of store paths run in parallel using a thread pool 2017-10-25 16:13:49 +01:00
Eelco Dolstra 82327e3cc4
exportReferencesGraph: Allow exporting a list of store paths 2017-10-25 15:18:49 +02:00
Eelco Dolstra 8191992c83
Bump language version 2017-10-25 15:18:37 +02:00
Eelco Dolstra 3395e3bbc4
Fix exportReferencesGraph in the structured attrs case 2017-10-25 14:08:29 +02:00
Eelco Dolstra 2d5b1b24bf
Pass lists/attrsets to bash as (associative) arrays 2017-10-25 13:01:50 +02:00
Eelco Dolstra 86aab6e9a0
Add --store argument to legacy commands 2017-10-24 15:41:11 +02:00
Eelco Dolstra d16fd24973
Allow shorter syntax for chroot stores
You can now say '--store /tmp/nix' instead of '--store local?root=/tmp/nix'.
2017-10-24 15:32:38 +02:00
Eelco Dolstra 3460e4cf00
More progress indicator improvements
In particular, don't show superfluous "fetching path" and "building
path(s)" messages, and show the current round (with --repeat).
2017-10-24 15:32:38 +02:00
Eelco Dolstra 96051dd057
More progress indicator improvements
Fixes #1599.
2017-10-24 14:47:23 +02:00
Eelco Dolstra be220702a7
Progress indicator: Show on what machine we're building
E.g.

  $ nix build nixpkgs.hello --builders 'root@wendy'
  [1/0/1 built] building hello-2.10 on ssh://root@wendy: checking for minix/config.h... no
2017-10-24 14:24:57 +02:00
Eelco Dolstra f8624762ac
build-remote: Work properly on a chrooted store 2017-10-24 14:14:28 +02:00
Eelco Dolstra fe9d2f974d
Handle log messages from build-remote
This makes the progress indicator show statuses like "connecting to
'root@machine'".
2017-10-24 14:05:22 +02:00
Eelco Dolstra 0d59f1ca49
nix: Respect -I, --arg, --argstr
Also, random cleanup to argument handling.
2017-10-24 12:58:34 +02:00
Eelco Dolstra 25f32625e2
Remove the remote-builds option
This is superfluous since you can now just set "builders" to empty,
e.g. "--builders ''".
2017-10-24 11:00:16 +02:00
Eelco Dolstra af241ae7d3
Remove the builder-files option
You can now include files via the "builders" option, using the syntax
"@<filename>". Having only one option makes it easier to override
builders completely.

For backward compatibility, the default is "@/etc/nix/machines", or
"@<filename>" for each file name in NIX_REMOTE_SYSTEMS.
2017-10-24 10:54:43 +02:00
Eelco Dolstra 37fbfffd8e
Pass all settings to build-remote
This ensures that command line flags such as --builders get passed
correctly.
2017-10-23 20:50:28 +02:00
Eelco Dolstra f32cdc4fab
Turn $NIX_REMOTE into a configuration option 2017-10-23 20:50:28 +02:00
Eelco Dolstra 863a45f1f3
build-remote: Put current load under the store state directory
Fixes the error

  error: opening lock file '/nix/var/nix/current-load/main-lock': Permission denied

when using a chroot store.
2017-10-23 20:50:24 +02:00
Eelco Dolstra f31c66d392 Merge pull request #1616 from copumpkin/separate-darwin-sandbox
Shift Darwin sandbox to separate installed files
2017-10-20 12:00:27 +02:00
Shea Levy c7af84ce84
nix-shell: Fix file name resolution in shebangs 2017-10-19 15:02:40 -04:00
Eelco Dolstra ef1d0142a0
Revert "Let's not populate the NAR cache from hydra-queue-runner for now"
This reverts commit 908590dc6c. Since
hydra-server can have a different store URI from hydra-queue-runner
now, we don't really need this.
2017-10-19 12:11:07 +02:00
Eelco Dolstra 908590dc6c
Let's not populate the NAR cache from hydra-queue-runner for now 2017-10-18 15:45:39 +02:00
Eelco Dolstra 75cd75b1ae
Errors writing to the NAR cache should not be fatal 2017-10-18 15:41:14 +02:00
Eelco Dolstra d8306148e0
Suppress "copying 0 paths" message 2017-10-18 15:02:58 +02:00
Dan Peebles 6e5165b773 Shift Darwin sandbox to separate installed files
This makes it slightly more manageable to see at a glance what in a
build's sandbox profile is unique to the build and what is standard. Also
a first step to factoring more of our Darwin logic into scheme functions
that will allow us a bit more flexibility. And of course less of that
nasty codegen in C++! 😀
2017-10-17 19:15:51 -04:00
Eelco Dolstra b24b8ef77c BinaryCacheStore::addToStore(): Add NARs to the local cache 2017-10-17 21:39:48 +02:00
Eelco Dolstra ca580bec35 BinaryCacheStore: Support local caching of NARs
This speeds up commands like "nix cat-store". For example:

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nar-cache /nix/store/i60yncmq6w9dyv37zd2k454g0fkl3arl-systemd-234/etc/udev/udev.conf
  real    0m4.336s

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nar-cache /nix/store/i60yncmq6w9dyv37zd2k454g0fkl3arl-systemd-234/etc/udev/udev.conf
  real    0m0.045s

The primary motivation is to allow hydra-server to serve files from S3
binary caches. Previously Hydra had a hack to do "nix-store -r
<path>", but that fetches the entire closure so is prohibitively
expensive.

There is no garbage collection of the NAR cache yet. Also, the entire
NAR is read when accessing a single member file. We could generate the
NAR listing to provide random access.

Note: the NAR cache is indexed by the store path hash, not the content
hash, so NAR caches should not be shared between binary caches, unless
you're sure that all your builds are binary-reproducible.
2017-10-17 21:15:33 +02:00
Eelco Dolstra 11ba4302e3 Remove BinaryCacheStoreAccessor
Probably as a result of a bad merge in
4b8f1b0ec0, we had both a
BinaryCacheStoreAccessor and a
RemoteFSAccessor. BinaryCacheStore::getFSAccessor() returned the
latter, but BinaryCacheStore::addToStore() checked for the
former. This probably caused hydra-queue-runner to download paths that
it just uploaded.
2017-10-17 20:51:42 +02:00
Shea Levy cbc216911d
fetchgit: Remove incomplete/unneeded ref check.
This check fails for tags and branches, and is made redundant by the
checks git itself will do when fetching the repo.
2017-10-17 12:30:54 -04:00
Shea Levy 4e58294ae6
fetchgit: Remove incomplete/unneeded isURI check.
This check spuriously fails for e.g. git@github.com:NixOS/nixpkgs.git,
and even for ssh://git@github.com/NixOS/nixpkgs.git, and is made
redundant by the checks git itself will do when fetching the repo. We
instead pass a -- before passing the URI to git to avoid injection.
2017-10-16 14:35:28 -04:00
Eelco Dolstra 1dd29d7aeb
Add option to disable the seccomp filter
I needed this to test ACL/xattr removal in
canonicalisePathMetaData(). Might also be useful if you need to build
old Nixpkgs that doesn't have the required patches to remove
setuid/setgid creation.
2017-10-12 18:21:55 +02:00
Eelco Dolstra 97307811ee Merge pull request #1571 from teto/improve_priority_msg
Improve error message for conflicting priorities
2017-10-12 13:10:44 +02:00
Eelco Dolstra 177aee07c6 Merge pull request #1580 from copumpkin/retry-after-timeout
Retry in all error cases but a few
2017-10-12 13:05:45 +02:00
Eelco Dolstra fda7b95cb0
Fix a hang in ThreadPool
The worker threads could exit prematurely if they finished processing
all items while the main thread was still adding items. In particular,
this caused hanging nix-store --serve processes in the build farm.

Also, process items from the main thread.
2017-10-09 15:41:09 +02:00
Dan Peebles 476493dbf5 Reverse retry logic to retry in all but a few cases
It was getting too much like whac-a-mole listing all the retriable error
conditions, so we now retry by default and list the cases where retrying
is almost certainly hopeless.
2017-10-02 23:22:02 -04:00
Eelco Dolstra f3e0d46821
nix run: Restore signals
Otherwise Ctrl-C doesn't work.
2017-09-28 17:58:59 +02:00
Eelco Dolstra 5421ad243e
nix copy: make recursive by default 2017-09-27 18:28:54 +02:00
Eelco Dolstra b8875213dc
Tweak message 2017-09-27 13:21:25 +02:00
Eelco Dolstra bfef3ec855 nix build: Fix realising a store path 2017-09-27 13:14:45 +02:00
Eelco Dolstra d1c6e0fe8c
nix why-depends: Write to stdout 2017-09-25 16:59:16 +02:00
Eelco Dolstra e8d6ee7c1b
Add "nix show-derivation"
This debug command prints a store derivation in JSON format. For
example:

  $ nix show-derivation nixpkgs.hello
  {
    "/nix/store/ayjwpwwiyy04nh9z71rsdgd3q7bra7ch-hello-2.10.drv": {
      "outputs": {
        "out": {
          "path": "/nix/store/w5w4v29ql0qwqhczkdxs94ix2lh7ibgs-hello-2.10"
        }
      },
      "inputSrcs": [
        "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
      ],
      "inputDrvs": {
        "/nix/store/13839aqdf6x4k3b785rw5f2l7857l6y3-bash-4.4-p12.drv": [
          "out"
        ],
        "/nix/store/vgdx7fdc7d4iirmnwj2py1nrvr5qwzj7-hello-2.10.tar.gz.drv": [
          "out"
        ],
        "/nix/store/x3kkd0vsqfflbvwf1055l9mr39bg0ms0-stdenv.drv": [
          "out"
        ]
      },
      "platform": "x86_64-linux",
      "builder": "/nix/store/qp5fw57d38bd1n07ss4zxh88zg67c3vg-bash-4.4-p12/bin/bash",
      "args": [
        "-e",
        "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
      ],
      "env": {
        "buildInputs": "",
        "builder": "/nix/store/qp5fw57d38bd1n07ss4zxh88zg67c3vg-bash-4.4-p12/bin/bash",
        "configureFlags": "",
        "doCheck": "1",
        "name": "hello-2.10",
        "nativeBuildInputs": "",
        "out": "/nix/store/w5w4v29ql0qwqhczkdxs94ix2lh7ibgs-hello-2.10",
        "propagatedBuildInputs": "",
        "propagatedNativeBuildInputs": "",
        "src": "/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz",
        "stdenv": "/nix/store/6zngq1rdh0ans9qyckqimqibgnlvlfrm-stdenv",
        "system": "x86_64-linux"
      }
    }
  }

This removes the need for pp-aterm.
2017-09-25 13:43:35 +02:00
Eelco Dolstra e350671737
Doh 2017-09-25 13:25:55 +02:00
Matthieu Coudron 6920c23701 Improve error message for conflicting priorities
I find the error message 'nix-env --set-flag priority NUMBER PKGNAME'
not as helpful as it could be :
- doesn't share the current priorities
- doesn't say that the command must be run on the already installed
PKGNAME (which is confusing the first time)
- the doc needs careful reading:
"If there are multiple derivations matching a name in args that have the same name (e.g., gcc-3.3.6 and gcc-4.1.1), then the derivation with the highest priority is used."
if one stops reading there, he is screwed. Salvation comes with reading "A derivation can define a priority by declaring the meta.priority attribute. This attribute should be a number, with a higher value denoting a lower priority. The default priority is 0."
To sum it up, lower number wins. I tried to convey this idea in the
message too.
2017-09-21 10:25:20 +09:00
Eelco Dolstra 84f112b1c8
nix-shell: Ignore readFile() errors
Fixes #1563.
2017-09-18 13:28:00 +02:00
Eelco Dolstra aca4f7dff0
Don't remove Content-Encoding in fetchurl / nix-prefetch-url
Fixes #1568.
2017-09-18 11:07:28 +02:00
Eelco Dolstra 89dc62c174
RemoteStore: Add option to drop old connections from the pool
This is a hack to make hydra-queue-runner free its temproots
periodically, thereby ensuring that garbage collection of the
corresponding paths is not blocked until the queue runner is
restarted.

It would be better if temproots could be released earlier than at
process exit. I started working on a RAII object returned by functions
like addToStore() that releases temproots. However, this would be a
pretty massive change so I gave up on it for now.
2017-09-14 18:10:38 +02:00
Eelco Dolstra 308ecf6361
Show the pid of temporary roots 2017-09-14 15:02:52 +02:00
Eelco Dolstra da1e4fdfb5
nix-store -q --roots / --gc --print-roots: Print temporary / in-memory roots
For example,

  $ nix-store -q --roots /nix/store/7phd2sav7068nivgvmj2vpm3v47fd27l-patchelf-0.8pre845_0315148
  {temp:1}

denotes that the path is only being kept alive by a temporary root
(i.e. /nix/var/nix/temproots/). Similarly,

  $ nix-store --gc --print-roots
  ...
  {memory:9} -> /nix/store/094gpjn9f15ip17wzxhma4r51nvsj17p-curl-7.53.1

shows that curl is being used by some process.
2017-09-14 14:38:36 +02:00
Eelco Dolstra 970366266b
Add "nix add-to-store" command 2017-09-14 13:29:33 +02:00
Eelco Dolstra 1c58ad2ffa
nix why-depends: Fix showing self-references 2017-09-14 11:17:32 +02:00
Eelco Dolstra fc0ded3408
nix why-depends: Add option to show all edges causing a dependency
For example, without --all:

  $ nix why-depends nixpkgs.nixUnstable nixpkgs.libssh2
  /nix/store/s9n5gvj2l49b4n19nz6xl832654nf7n7-nix-1.12pre5511_c94f3d55
  └───lib/libnixstore.so: …/lib:/nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0/lib…
      => /nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0
      └───lib/libcurl.la: …ib -L/nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0/l…
          => /nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0

but with --all:

  $ nix why-depends -a nixpkgs.nixUnstable nixpkgs.libssh2
  /nix/store/s9n5gvj2l49b4n19nz6xl832654nf7n7-nix-1.12pre5511_c94f3d55
  ├───lib/libnixstore.so: …/lib:/nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0/lib…
  │   => /nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0
  │   └───lib/libcurl.la: …ib -L/nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0/l…
  │       lib/libcurl.so.4.4.0: …/lib:/nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0/l…
  │       => /nix/store/4mbayl1y5hpjbjzkx8ndyhkv98kqw1wi-libssh2-1.8.0
  └───lib/libnixstore.so: …/lib:/nix/store/bx2i9vi76lps6w9rr73fxf6my31s4dg5-aws-sdk-cpp-1.0…
      => /nix/store/bx2i9vi76lps6w9rr73fxf6my31s4dg5-aws-sdk-cpp-1.0.153
      └───lib/libaws-cpp-sdk-core.so: …e.so./nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0/lib…
          lib/libaws-cpp-sdk-s3.so: …/lib:/nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0/lib…
          => /nix/store/w9ykqpl5v0r3vfwsgn408jqhs72cx96x-curl-7.55.0
2017-09-11 16:21:27 +02:00
Eelco Dolstra d41c5eb13f Add command "nix why-depends"
This command shows why a package has another package in its runtime
closure. For example, to see why VLC has libdrm.dev in its closure:

  $ nix why-depends nixpkgs.vlc nixpkgs.libdrm.dev
  /nix/store/g901z9pcj0n5yy5n6ykxk3qm4ina1d6z-vlc-2.2.5.1:
    lib/libvlccore.so.8.0.0: …nfig:/nix/store/405lmx6jl8lp0ad1vrr6j498chrqhz8g-libdrm-2.4.75-d…

  /nix/store/s3nm7kd8hlcg0facn2q1ff2n7wrwdi2l-mesa-noglu-17.0.7-dev:
    nix-support/propagated-native-build-inputs: …-dev /nix/store/405lmx6jl8lp0ad1vrr6j498chrqhz8g-libdrm-2.4.75-d…

Thus, VLC's lib/libvlccore.so.8.0.0 as well as mesa-noglu's
nix-support/propagated-native-build-inputs cause the dependency.
2017-09-10 22:40:33 +02:00
Eelco Dolstra 8af704eef1 Typo 2017-09-10 22:40:33 +02:00
Eelco Dolstra ad228d84e5 nix build: Only download the requested derivation outputs
Also some refactoring.
2017-09-10 22:40:33 +02:00
Eelco Dolstra 3ed8290e53
LegacySSHStore: Include signatures etc. 2017-09-08 16:55:27 +02:00
Eelco Dolstra e02edb1483
nix copy: Add --substitute flag 2017-09-08 15:32:07 +02:00
Eelco Dolstra b7376edf06
ThreadPool: On exception, interrupt the other worker threads 2017-09-08 15:31:24 +02:00
Eelco Dolstra 6a888ec29a
copyStorePath(): Fill in missing narHash regardless of checkSigs
I don't remember what the reasoning was here, but security is provided
by the signatures, not by whether the hash is provided by the other
store.
2017-09-08 14:48:08 +02:00
Eelco Dolstra 3cf1705583
nix copy: Don't open the --from store twice 2017-09-08 14:46:55 +02:00
Eelco Dolstra 8f6b347abd
ThreadPool: Improve exception handling
In particular, process() won't return as long as there are active
items. This prevents work item lambdas from referring to stack frames
that no longer exist.
2017-09-08 14:42:09 +02:00
Eelco Dolstra a2740c9ca2 nix copy: Add examples 2017-09-08 11:33:46 +02:00
Eelco Dolstra 88e6bb76de Communicate the structured log FD to builders
Since we may use a dedicated file descriptor in the future, this
allows us to change it. So builders can do

  if [[ -n $NIX_LOG_FD ]]; then
    echo "@nix { message... }" >&$NIX_LOG_FD
  fi
2017-09-08 11:27:10 +02:00
Eelco Dolstra 24e23a1a73 nix search: Add examples 2017-09-07 20:42:11 +02:00
Eelco Dolstra 4967f0509a nix eval: Add examples 2017-09-07 20:37:46 +02:00
Eelco Dolstra 2ebeffcfd4 nix log: Add examples 2017-09-07 20:18:29 +02:00
Eelco Dolstra 17afc42895 nix build: Add examples 2017-09-07 20:14:04 +02:00
Eelco Dolstra 2e9b7c4cb2 nix run: Add some examples 2017-09-07 20:09:04 +02:00
Eelco Dolstra dff440aab3
nix build: Add --out-link and --no-link options 2017-09-06 16:20:34 +02:00
Eelco Dolstra df4342bc17
nix build: Create result symlinks 2017-09-06 16:03:22 +02:00
Eelco Dolstra 1277aab219
Fix abort when the GC thread gets an exception 2017-09-06 11:37:58 +02:00
Eelco Dolstra 0b606aad46
Add automatic garbage collection
Nix can now automatically run the garbage collector during builds or
while adding paths to the store. The option "min-free = <bytes>"
specifies that Nix should run the garbage collector whenever free
space in the Nix store drops below <bytes>. It will then delete
garbage until "max-free" bytes are available.

Garbage collection during builds is asynchronous; running builds are
not paused and new builds are not blocked. However, there also is a
synchronous GC run prior to the first build/substitution.

Currently, no old GC roots are deleted (as in "nix-collect-garbage
-d").
2017-09-05 20:43:42 +02:00
Eelco Dolstra b932ea58ec
GC: Don't delete own temproots file
Since file locks are per-process rather than per-file-descriptor, the
garbage collector would always acquire a lock on its own temproots
file and conclude that it's stale.
2017-09-05 20:39:57 +02:00
Eelco Dolstra 8215b75d36
Abort curl downloads if there is no progress for 5 minutes
Maybe this will fix the curl hangs on macOS. (We could also use
CURLOPT_TIMEOUT but that seems more of a sledgehammer.)
2017-09-01 16:51:26 +02:00
Eelco Dolstra 7a108d904e
Fix verbosity level for nix build --dry-run 2017-09-01 12:37:09 +02:00
Sergei Trofimovich c9857ef262 src/libmain/stack.cc: fix 'ucontext' usage on glibc-2.26
Build fails as:

$ make
  CXX    src/libmain/stack.o
src/libmain/stack.cc: In function 'void nix::sigsegvHandler(int, siginfo_t*, void*)':
src/libmain/stack.cc:21:21: error: 'ucontext' was not declared in this scope
     sp = (char *) ((ucontext *) ctx)->uc_mcontext.gregs[REG_RSP];
                     ^~~~~~~~
src/libmain/stack.cc:21:21: note: suggested alternative: 'ucontext_t'
     sp = (char *) ((ucontext *) ctx)->uc_mcontext.gregs[REG_RSP];
                     ^~~~~~~~
                     ucontext_t

It's caused by upstream rename:
https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=251287734e89a52da3db682a8241eb6bccc050c9

which basically changes
    typedef struct ucontext {} ucontext_t;
to
    typedef struct ucontext_t {} ucontext_t;

The change uses ucontext_t.

Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
2017-08-31 21:41:36 +01:00
Eelco Dolstra bbdf08bc0f
Call queryMissing() prior to building
Without this, substitute info is fetched sequentially, which is
superslow. In the old UI (e.g. nix-build), we call printMissing(),
which calls queryMissing(), thereby preheating the binary cache
cache. But the new UI doesn't do that.
2017-08-31 16:05:07 +02:00
Eelco Dolstra fd73c1e20a
Add an activity for binary cache queries 2017-08-31 15:25:58 +02:00
Eelco Dolstra c2154d4c84
Rename a few configuration options
In particular, drop the "build-" and "gc-" prefixes which are
pointless. So now you can say

  nix build --no-sandbox

instead of

  nix build --no-build-use-sandbox
2017-08-31 14:28:25 +02:00
Eelco Dolstra 7d4a7136db
More macOS build fixes 2017-08-31 12:52:07 +02:00
Eelco Dolstra fabde432dc
Fix build failure on non-Linux
https://hydra.nixos.org/build/59649086
2017-08-31 11:05:18 +02:00
Eelco Dolstra 9f47eac92b
Fix mismatched tag warning
https://hydra.nixos.org/build/59649086
2017-08-31 11:04:52 +02:00
Eelco Dolstra e9c07a3b26
nix edit / log: Operate on a single Installable 2017-08-29 16:18:00 +02:00
Eelco Dolstra 9b82ecbae0
nix search: Warn about cached results 2017-08-29 15:22:05 +02:00
Eelco Dolstra c8235c5313
nix run: Flush the progress bar before starting the command 2017-08-29 15:13:30 +02:00
Eelco Dolstra 05d68a6e23
nix run: Add some flags for clearing/keeping the environment
This is useful for testing commands in isolation.

For example,

  $ nix run nixpkgs.geeqie -i -k DISPLAY -k XAUTHORITY -c geeqie

runs geeqie in an empty environment, except for $DISPLAY and
$XAUTHORITY.
2017-08-29 15:00:08 +02:00
Eelco Dolstra 5cc8609e30
nix run: Allow passing a command to execute
E.g.

  nix run nixpkgs.hello -c hello --greeting Hallo

Note that unlike "nix-shell --command", no quoting of arguments is
necessary.

"-c" (short for "--command") cannot be combined with "--" because they
both consume all remaining arguments. But since installables shouldn't
start with a dash, this is unlikely to cause problems.
2017-08-29 14:42:48 +02:00
Eelco Dolstra 93a5ef0516
nix run: Fix chroot execution
Running "nix run" with a diverted store, e.g.

  $ nix run --store local?root=/tmp/nix nixpkgs.hello

stopped working when Nix became multithreaded, because
unshare(CLONE_NEWUSER) doesn't work in multithreaded processes. The
obvious solution is to terminate all other threads first, but 1) there
is no way to terminate Boehm GC marker threads; and 2) it appears that
the kernel has a race where unshare(CLONE_NEWUSER) will still fail for
some indeterminate amount of time after joining other threads.

So instead, "nix run" will now exec() a single-threaded helper ("nix
__run_in_chroot") that performs the actual unshare()/chroot()/exec().
2017-08-29 13:21:07 +02:00
Eelco Dolstra 1c58e13bee
Hide commands that don't have a description
These are assumed to be internal.
2017-08-29 11:52:55 +02:00
Eelco Dolstra 05c45f301d
readLink(): Handle symlinks in /proc
Symlinks like /proc/self/exe report a stat() size of 0, so use a
buffer of at least PATH_MAX instead.
2017-08-29 11:52:34 +02:00
Eelco Dolstra 2cc345b95f
Give activities a verbosity level again
And print them (separately from the progress bar) given sufficient -v
flags.
2017-08-28 19:13:24 +02:00
Eelco Dolstra cfc8132391
Don't send progress messages to older clients 2017-08-28 18:54:23 +02:00
Eelco Dolstra fe34b91289
Tunnel progress messages from the daemon to the client
This makes the progress bar work for non-root users.
2017-08-28 18:49:42 +02:00
Eelco Dolstra e681b1f064
Simplify 2017-08-28 14:30:35 +02:00
Eelco Dolstra 8fff3e7bb5
Make TunnelLogger thread-safe
Now that we use threads in lots of places, it's possible for
TunnelLogger::log() to be called asynchronously from other threads
than the main loop. So we need to ensure that STDERR_NEXT messages
don't clobber other messages.
2017-08-28 14:17:07 +02:00
Eelco Dolstra 94a0548dc4
Simplify 2017-08-25 21:26:37 +02:00
Eelco Dolstra 9b845e6936
Doh 2017-08-25 20:52:34 +02:00
Eelco Dolstra 0ac35b67b8
Allow derivations to update the build phase
So the progress bar can show

  [1/0/1 built, 0.0 MiB DL] building hello-2.10 (configuring): checking whether pread is declared without a macro... yes
2017-08-25 18:04:05 +02:00
Eelco Dolstra c137c0a5eb
Allow activities to be nested
In particular, this allows more relevant activities ("substituting X")
to supersede inferior ones ("downloading X").
2017-08-25 17:49:40 +02:00
Eelco Dolstra f194629f96
Fix Debian build
https://hydra.nixos.org/build/59390148
2017-08-25 16:11:18 +02:00
Eelco Dolstra db1d45037c
Handle SIGWINCH 2017-08-25 15:59:03 +02:00
Eelco Dolstra ec9e0c03c3
When truncating the progress bar, take ANSI escape sequences into account 2017-08-25 15:59:03 +02:00
Eelco Dolstra a3015db6c3
Typo 2017-08-25 15:59:03 +02:00
Eelco Dolstra 0e9ddcc306
Restore activity metadata
This allows the progress bar to display "building perl-5.22.3" instead
of "building /nix/store/<hash>-perl-5.22.3.drv".
2017-08-25 15:58:35 +02:00
Eelco Dolstra 1f56235438
Clean up JSON construction 2017-08-21 12:18:46 +02:00
Eelco Dolstra 4af2611bd1
Allow builders to create activities
Actually, currently they can only create download activities. Thus,
downloads by builtins.fetchurl show up in the progress bar.
2017-08-21 12:18:46 +02:00
Eelco Dolstra 4c6a26539c
Remove debug line 2017-08-21 12:18:46 +02:00
Eelco Dolstra 9400cb36b7
Disallow accidental copy construction 2017-08-21 12:18:42 +02:00
Andy Wingo 6bb4e3e8fe Remove unused decodeOctalEscaped
Besides being unused, this function has a bug that it will incorrectly
decode the path component Ubuntu\04016.04.2\040LTS\040amd64 as
"Ubuntu.04.2 LTS amd64" instead of "Ubuntu 16.04.2 LTS amd64".
2017-08-18 11:07:00 +02:00
Eelco Dolstra 2ee1b9359b Merge branch 'tokenize' of https://github.com/nbp/nix 2017-08-16 21:21:36 +02:00
Eelco Dolstra c2cab20732
nix verify: Restore the progress indicator 2017-08-16 20:56:03 +02:00
Eelco Dolstra b4ed97e3a3
nix optimise-store: Show how much space has been freed 2017-08-16 20:56:03 +02:00
Eelco Dolstra 23b8b7e096
nix optimise-store: Add
This replaces "nix-store --optimise". Main difference is that it has a
progress indicator.
2017-08-16 20:56:03 +02:00
Eelco Dolstra 40bffe0a43
Progress indicator: Cleanup 2017-08-16 20:56:03 +02:00
Eelco Dolstra dff12b38f9
Progress indicator: More improvements 2017-08-16 20:56:03 +02:00
Eelco Dolstra bf1f123b09
Progress indicator: Show number of active items 2017-08-16 20:56:03 +02:00
Eelco Dolstra 0e0dcf2c7e
Progress indicator: Unify "copying" and "substituting"
They're the same thing after all.

Example:

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 49/98 copied, 16.3/92.8 MiB DL, 55.8/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/0pl9li1jigcj2dany47hpmn0r3r48wc4nz48v5mqhh426lgz3bz6.nar.xz'
2017-08-16 20:56:03 +02:00
Eelco Dolstra c36467ad2e
Improve substitution progress indicator
E.g.

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-16 20:56:02 +02:00
Eelco Dolstra b29b6feaba
nix copy: Improve progress indicator
It now shows the amount of data copied:

  [8/1038 copied, 160.4/1590.9 MiB copied] copying path '...'
2017-08-16 20:56:02 +02:00
Eelco Dolstra c5e4404580
nix copy: Revive progress bar 2017-08-16 20:56:02 +02:00
Eelco Dolstra dffc3fe43b
nix copy: Add --no-check-sigs flag 2017-08-16 20:56:02 +02:00
Eelco Dolstra e4bd42f98f
Disallow SSH host names starting with a dash 2017-08-16 20:55:58 +02:00
Nicolas B. Pierron b8867a0239 Add builtins.string function.
The function 'builtins.split' takes a POSIX extended regular expression
and an arbitrary string. It returns a list of non-matching substring
interleaved by lists of matched groups of the regular expression.

```nix
with builtins;
assert split "(a)b" "abc"      == [ "" [ "a" ] "c" ];
assert split "([ac])" "abc"    == [ "" [ "a" ] "b" [ "c" ] "" ];
assert split "(a)|(c)" "abc"   == [ "" [ "a" null ] "b" [ null "c" ] "" ];
assert split "([[:upper:]]+)" "  FOO   "
                               == [ "  " [ "FOO" ] "   " ];
```
2017-08-15 20:04:11 +00:00
Eelco Dolstra b6ee5e5bf0
Style fix 2017-08-10 13:51:07 +02:00
Eelco Dolstra af765a8eab
Use /proc/self/fd to efficiently close all FDs on Linux
Issue #1506.
2017-08-09 16:22:05 +02:00
Eelco Dolstra c6184dec6c
nix repl: Support printing floating-point numbers 2017-08-09 15:17:29 +02:00
Brian McKenna b39cc4fc81 Include missing <cstdlib> for abort()
This is needed to get Nix compiled using Android NDK.
2017-08-03 07:03:22 +10:00
Eelco Dolstra a2778988f2
Merge branch 'macOS' of https://github.com/davidak/nix 2017-07-31 10:31:51 +02:00
Jörg Thalheim 2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
davidak fcb8d6a7a0 replace "OS X" with "macOS" 2017-07-30 12:28:50 +02:00
davidak 92bcb61127 replace "Mac OS X" with "macOS"
except in older release notes where the name was actually Mac OS X.
2017-07-30 12:26:17 +02:00
Eelco Dolstra c7654bc491
nix-build: Fix regression causing all outputs to be built 2017-07-28 15:17:52 +02:00
Eelco Dolstra af4689f9e9
nix-prefetch-url: Fix regression in hash printing 2017-07-28 14:56:39 +02:00
Eelco Dolstra 7480f4f9a4
builtins.fetchgit: Support specifying commit hashes
This adds an argument "rev" specififying the Git commit hash. The
existing argument "rev" is renamed to "ref". The default value for
"ref" is "master". When specifying a hash, it's necessary to specify a
ref since we're not cloning the entire repository but only fetching a
specific ref.

Example usage:

  builtins.fetchgit {
    url = https://github.com/NixOS/nixpkgs.git;
    ref = "release-16.03";
    rev = "c1c0484041ab6f9c6858c8ade80a8477c9ae4442";
  };
2017-07-27 18:08:23 +02:00
Eelco Dolstra 9f64cb89cb
builtins.fetchgit: Respect tarball-ttl
I.e. if the local ref is more recent than tarball-ttl seconds, then
don't check the remote.
2017-07-27 17:23:29 +02:00
Eelco Dolstra 69deca194e
builtins.fetchgit: Use proper refs locally 2017-07-27 17:02:25 +02:00
Eelco Dolstra 6d7de7f3de
builtins.fetchgit: Cache hash -> store path mappings
This prevents an expensive call to addToStore() in the cached case.
2017-07-27 16:16:08 +02:00
Eelco Dolstra 57b9505731
nix search: Add a cache
The package list is now cached in
~/.cache/nix/package-search.json. This gives a substantial speedup to
"nix search" queries. For example (on an SSD):

First run: (no package search cache, cold page cache)

  $ time nix search blender
  Attribute name: nixpkgs.blender
  Package name: blender
  Version: 2.78c
  Description: 3D Creation/Animation/Publishing System

  real    0m6.516s

Second run: (package search cache populated)

  $ time nix search blender
  Attribute name: nixpkgs.blender
  Package name: blender
  Version: 2.78c
  Description: 3D Creation/Animation/Publishing System

  real    0m0.143s
2017-07-26 17:29:10 +02:00
Eelco Dolstra 4c9ff89c26
nix-build/nix-shell: Eliminate call to nix-instantiate / nix-store
Note that this removes the need for a derivation symlink, so the
--drv-path and --add-drv-link flags now do nothing.
2017-07-26 17:29:09 +02:00
Eelco Dolstra c94f3d5575
nix-shell: Use bashInteractive from <nixpkgs>
This adds about 0.1s to nix-shell runtime in the case where
bashInteractive already exists.

See discussion at https://github.com/NixOS/nixpkgs/issues/27493.
2017-07-20 13:50:25 +02:00
Eelco Dolstra 57a30e101b
nix search: Ignore top-level eval errors
$NIX_PATH may contain elements that don't evaluate to an attrset (like
"nixos-config"), so ignore those.
2017-07-20 13:33:13 +02:00
Eelco Dolstra fc3568e263
FD_SETSIZE check: BuildError -> Error
BuildError denotes a permanent build failure, which is not the case
here.
2017-07-20 13:33:13 +02:00
Eelco Dolstra b144c4d617
nix search: Add --json flag 2017-07-20 13:33:13 +02:00
Eelco Dolstra 90825dea51
Add "nix search" command 2017-07-20 13:33:13 +02:00
Dmitry Kalinkin d5e1bffd2a
Do not try to fill fd_set with fd>=FD_SETSIZE
This is UB and causes buffer overflow and crash on linux.
2017-07-18 17:51:50 -04:00
Eelco Dolstra bf6792c0df
Always use base-16 for hashed mirror lookups
In particular, don't use base-64, which we don't support. (We do have
base-32 redirects for hysterical reasons.)

Also, add a test for the hashed mirror feature.
2017-07-17 13:13:18 +02:00
Eelco Dolstra 49304bae81
Make the hashes mirrors used by builtins.fetchurl configurable
In particular, this allows it to be disabled in our tests.
2017-07-17 13:07:08 +02:00
Eelco Dolstra 4ec6eb1fdf
Fix accidental printError 2017-07-17 11:38:15 +02:00
Eelco Dolstra 766ad5db3b
nix path-info: Show download sizes for binary cache stores
E.g.

  $ nix path-info --json --store https://cache.nixos.org nixpkgs.thunderbird -S
  ...
      "downloadHash": "sha256:1jlixpzi225wwa0f4xdrwrqgi47ip1qpj9p06fyxxg07sfmyi4q0",
      "downloadSize": 43047620,
      "closureDownloadSize": 84745960
    }
  ]
2017-07-14 18:29:10 +02:00
Eelco Dolstra fdc9da034f
Avoid a call to derivationFromPath()
This doesn't work in read-only mode, ensuring that operations like

  nix path-info --store https://cache.nixos.org -S nixpkgs.hello

(asking for the closure size of nixpkgs.hello in cache.nixos.org) work
when nixpkgs.hello doesn't exist in the local store.
2017-07-14 18:29:10 +02:00
Eelco Dolstra 3908d3929c
nix path-info: Don't barf on invalid paths
Now you get

  [
    {
      "path": "/nix/store/fzvliz4j5xzvnd0w5zgw2l0ksqh578yk-bla",
      "valid": false
    }
  ]
2017-07-14 18:29:10 +02:00
Eelco Dolstra 6438ba22af
StorePathsCommand: Don't build installables
On second though this was annoying. E.g. "nix log nixpkgs.hello" would
build/download Hello first, even though the log can be fetched
directly from the binary cache.

May need to revisit this.
2017-07-14 18:29:07 +02:00
Eelco Dolstra 112ff7833d
nix: Show help when no arguments are given
Fixes #1464.
2017-07-14 13:44:45 +02:00
Eelco Dolstra 0681f8c907
Shut up a memory leak warning 2017-07-14 11:40:57 +02:00
Eelco Dolstra 2965d40612 replaceSymlink(): Handle the case where the temporary file already exists
Not really necessary anymore for #849, but still nice to have.
2017-07-11 23:21:40 +02:00
Eelco Dolstra 8e8caf7f3e fetchTarball: Prevent concurrent downloads of the same file
Fixes #849.
2017-07-11 23:21:24 +02:00
Eelco Dolstra 9c00fa4179 Merge pull request #1422 from nh2/fix-potential-hash-comparison-crash
Fix potential crash/wrong result two hashes of unequal length are compared
2017-07-10 18:09:49 +02:00
Eelco Dolstra 1762b9616c Merge pull request #1428 from rimmington/clearer-regex-space-error
Clearer error message when regex exceeds space limit
2017-07-10 11:45:05 +02:00
Rhys 17bb00d378 Clearer error message when regex exceeds space limit 2017-07-10 09:35:53 +10:00
Eelco Dolstra d3713716b6 Merge pull request #1445 from matthewbauer/macos-skip-hardlink
Don’t hardlink disallowed paths in OS X.
2017-07-07 11:05:21 +02:00
Matthew Bauer 72e80c59b5 Don’t hardlink disallowed paths in OS X.
Fixes #1443
2017-07-06 19:30:19 -07:00
Eelco Dolstra a3dc1e65ab
Add X32 to the seccomp filter
Fixes #1432.
2017-07-04 19:00:51 +02:00
Eelco Dolstra 42c5774e78
Sort substituters by priority
Fixes #1438.
2017-07-04 16:34:53 +02:00
Eelco Dolstra b7203e853e
getDefaultSubstituters(): Simplify initialisation
As shlevy pointed out, static variables in C++11 have thread-safe
initialisation.
2017-07-04 16:26:48 +02:00
Eelco Dolstra 6cf23c3e8f
Add allow-new-privileges option
This allows builds to call setuid binaries. This was previously
possible until we started using seccomp. Turns out that seccomp by
default disallows processes from acquiring new privileges. Generally,
any use of setuid binaries (except those created by the builder
itself) is by definition impure, but some people were relying on this
ability for certain tests.

Example:

  $ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --no-allow-new-privileges
  builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 2 log lines:
    cannot raise the capability into the Ambient set
    : Operation not permitted

  $ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --allow-new-privileges
  builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 6 log lines:
    PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=15.2 ms

Fixes #1429.
2017-07-04 15:48:25 +02:00
Eelco Dolstra ad8b96f1f2
Fix handling of expression installables with a / in them 2017-07-04 15:38:23 +02:00
Eelco Dolstra c0015e87af
Support base-64 hashes
Also simplify the Hash API.

Fixes #1437.
2017-07-04 15:07:41 +02:00
Eelco Dolstra 0a5a867758
nix-shell: Respect --dry-run
Fixes #824.
2017-07-03 11:54:30 +02:00
Eelco Dolstra fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00
Eelco Dolstra 90da34e421
processGraph(): Call getEdges in parallel 2017-07-03 11:38:08 +02:00
Niklas Hambüchen b591536e93 Fix potential crash/wrong result two hashes of unequal length are compared 2017-06-24 02:17:45 +02:00
David McFarland 596b0e0a04 Call SetDllDirectory("") after sqlite3 init on cygwin
Cygwin sqlite3 is patched to call SetDllDirectory("/usr/bin") on init, which
affects the current process and is inherited by child processes.  It causes
DLLs to be loaded from /usr/bin/ before $PATH, which breaks all sorts of
things.  A typical failures would be header/lib version mismatches (e.g.
openssl when running checkPhase on openssh).  We'll just set it back to the
default value.

Note that this is a problem with the cygwin version of sqlite3 (currently
3.18.0).  nixpkgs doesn't have the problematic patch.
2017-06-20 10:59:13 -03:00
Eelco Dolstra c7346a275c
Restore thunks on any exception
There's no reason to restrict this to Error exceptions. This shouldn't
matter to #1407 since the repl doesn't catch non-Error exceptions
anyway, but you never know...
2017-06-20 12:13:17 +02:00
Eelco Dolstra a1355917ec
Disable use of virtual hosting in aws-sdk-cpp
Recently aws-sdk-cpp quietly switched to using S3 virtual host URIs
(https://github.com/aws/aws-sdk-cpp/commit/69d9c53882), i.e. it sends
requests to http://<bucket>.<region>.s3.amazonaws.com rather than
http://<region>.s3.amazonaws.com/<bucket>. However this interacts
badly with curl connection reuse. For example, if we do the following:

1) Check whether a bucket exists using GetBucketLocation.
2) If it doesn't, create it using CreateBucket.
3) Do operations on the bucket.

then 3) will fail for a minute or so with a NoSuchBucket exception,
presumably because the server being hit is a fallback for cases when
buckets don't exist.

Disabling the use of virtual hosts ensures that 3) succeeds
immediately. (I don't know what S3's consistency guarantees are for
bucket creation, but in practice buckets appear to be available
immediately.)
2017-06-19 18:51:33 +02:00
Eelco Dolstra 82a0d614cf
Support creating S3 caches in other regions than us-east-1 2017-06-19 18:51:33 +02:00
Eelco Dolstra b33621d425
Handle S3Errors::RESOURCE_NOT_FOUND from aws-sdk-cpp
This is returned by recent versions. Also handle NO_SUCH_KEY even
though the library doesn't actually return that at the moment.
2017-06-19 18:51:32 +02:00
Eelco Dolstra 1c969611ba
Suppress "will retry in N ms" for non-retriable errors
Newer versions of aws-sdk-cpp call CalculateDelayBeforeNextRetry()
even for non-retriable errors (like NoSuchKey) whih causes log spam in
hydra-queue-runner.
2017-06-19 18:51:32 +02:00
Eelco Dolstra 00aa7c6705
Show aws-sdk-cpp log messages 2017-06-19 18:51:32 +02:00
Eelco Dolstra 1888f7889b
macOS: Ugly hack to make the tests succeed
Sandboxes cannot be nested, so if Nix's build runs inside a sandbox,
it cannot use a sandbox itself. I don't see a clean way to detect
whether we're in a sandbox, so use a test-specific hack.

https://github.com/NixOS/nix/issues/1413
2017-06-19 14:26:05 +02:00
Eelco Dolstra b5bdfdef73
macOS: Remove flags
In particular, UF_IMMUTABLE (uchg) needs to be cleared to allow the
path to be garbage-collected or optimised.

See https://github.com/NixOS/nixpkgs/issues/25819.
+       the file from being garbage-collected.
2017-06-19 14:19:21 +02:00
Eelco Dolstra 38b7d55af1
Remove redundant debug line 2017-06-14 13:45:38 +02:00
Eelco Dolstra 88b291ffc4
canonicalisePathMetaData(): Ignore security.selinux attribute
Untested, hopefully fixes #1406.
2017-06-14 11:41:03 +02:00
Eelco Dolstra 177f3996e2
Suppress spurious "killing process N: Operation not permitted" on macOS 2017-06-12 18:34:48 +02:00
Eelco Dolstra 25230a17a9
On macOS, don't use /var/folders for TMPDIR
This broke "nix-store --serve".
2017-06-12 17:43:19 +02:00
Eelco Dolstra 847f19a5f7
Provide a builtin default for $NIX_SSL_CERT_FILE
This is mostly to ensure that when Nix is started on macOS via a
launchd service or sshd (for a remote build), it gets a certificate
bundle.
2017-06-12 16:44:43 +02:00
Eelco Dolstra 7f5b750b40
Don't run pre-build-hook if we don't have a derivation
This fixes a build failure on OS X when using Hydra or Nix 1.12's
build-remote (since they don't copy the derivation to the build
machine).
2017-06-12 16:07:34 +02:00
Eelco Dolstra 186571965d
Don't show flags from config settings in "nix --help" 2017-06-07 18:41:20 +02:00
Eelco Dolstra aa952d5f0b
nix: Add --help-config flag 2017-06-07 16:49:54 +02:00
Eelco Dolstra b8283773bd
nix: Make all options available as flags
Thus, instead of ‘--option <name> <value>’, you can write ‘--<name>
<value>’. So

  --option http-connections 100

becomes

  --http-connections 100

Apart from brevity, the difference is that it's not an error to set a
non-existent option via --option, but unrecognized arguments are
fatal.

Boolean options have special treatment: they're mapped to the
argument-less flags ‘--<name>’ and ‘--no-<name>’. E.g.

  --option auto-optimise-store false

becomes

  --no-auto-optimise-store
2017-06-07 16:17:17 +02:00
Eelco Dolstra c8cc50d46e
Disable the build user mechanism on all platforms except Linux and OS X 2017-06-06 18:52:15 +02:00
Eelco Dolstra 85e93d7b87
Always use the Darwin sandbox
Even with "build-use-sandbox = false", we now use sandboxing with a
permissive profile that allows everything except the creation of
setuid/setgid binaries.
2017-06-06 18:44:49 +02:00
Eelco Dolstra 52fec8dde8
Remove listxattr assertion
It appears that sometimes, listxattr() returns a different value for
the query case (i.e. when the buffer size is 0).
2017-05-31 20:43:47 +02:00
Eelco Dolstra c96e8cd097
OS X sandbox: Improve builtin sandbox profile
Also, add rules to allow fixed-output derivations to access the
network.

These rules are sufficient to build stdenvDarwin without any
__sandboxProfile magic.
2017-05-31 17:25:51 +02:00
Eelco Dolstra 5ea8161b55 resolve-system-dependencies: Misc fixes
This fixes

  Could not find any mach64 blobs in file ‘/usr/lib/libSystem.B.dylib’, continuing...
2017-05-31 16:10:10 +02:00
Eelco Dolstra c368e079ca resolve-system-dependencies: Simplify 2017-05-31 15:34:03 +02:00
Eelco Dolstra 44f3f8048f OS X sandbox: Don't use a deterministic $TMPDIR
This doesn't work because the OS X sandbox cannot bind-mount
path to a different location.
2017-05-31 14:09:00 +02:00
Eelco Dolstra c740c3ce50 OS X sandbox: Store .sb file in $TMPDIR rather than the Nix store
The filename used was not unique and owned by the build user, so
builds could fail with

error: while setting up the build environment: cannot unlink ‘/nix/store/99i210ihnsjacajaw8r33fmgjvzpg6nr-bison-3.0.4.drv.sb’: Permission denied
2017-05-31 14:09:00 +02:00
Eelco Dolstra 683a499ebb
resolve-system-dependencies: Fix another segfault
runResolver() was barfing on directories like
/System/Library/Frameworks/Security.framework/Versions/Current/PlugIns. It
should probably do something sophisticated for frameworks, but let's
ignore them for now.
2017-05-30 20:39:40 +02:00
Eelco Dolstra acc889c821
Darwin sandbox: Use sandbox-defaults.sb
Issue #759.

Also, remove nix.conf from the sandbox since I don't really see a
legitimate reason for builders to access the Nix configuration.
2017-05-30 17:40:12 +02:00
Eelco Dolstra 53a1644187
Darwin sandbox: Disallow creating setuid/setgid binaries
Suggested by Daiderd Jordan.
2017-05-30 17:17:17 +02:00
Eelco Dolstra 83eec5a997 resolve-system-dependencies: Several fixes
This fixes

  error: getting attributes of path ‘Versions/Current/CoreFoundation’: No such file or directory

when /System/Library/Frameworks/CoreFoundation.framework/CoreFoundation is a symlink.

Also fixes a segfault when encounting a file that is not a MACH binary (such
as /dev/null, which is included in __impureHostDeps in Nixpkgs).

Possibly fixes #786.
2017-05-30 16:03:37 +02:00
Eelco Dolstra fe08d17934
Fix seccomp build failure on clang
Fixes

  src/libstore/build.cc:2321:45: error: non-constant-expression cannot be narrowed from type 'int' to 'scmp_datum_t' (aka 'unsigned long') in initializer list [-Wc++11-narrowing]
2017-05-30 14:37:24 +02:00
Eelco Dolstra d552d38758
Shut up some clang warnings 2017-05-30 14:35:50 +02:00
Eelco Dolstra 2ac99a32da
Add a seccomp rule to disallow setxattr() 2017-05-30 13:59:24 +02:00
Eelco Dolstra d798349ede
canonicalisePathMetaData(): Remove extended attributes / ACLs
EAs/ACLs are not part of the NAR canonicalisation. Worse, setting an
ACL allows a builder to create writable files in the Nix store. So get
rid of them.

Closes #185.
2017-05-30 13:47:41 +02:00
Eelco Dolstra ff6becafa8
Require seccomp only in multi-user setups 2017-05-30 12:37:04 +02:00
Eelco Dolstra cf93397d3f
Fix seccomp initialisation on i686-linux 2017-05-29 16:14:10 +02:00
Eelco Dolstra 6cc6c15a2d
Add a seccomp filter to prevent creating setuid/setgid binaries
This prevents builders from setting the S_ISUID or S_ISGID bits,
preventing users from using a nixbld* user to create a setuid/setgid
binary to interfere with subsequent builds under the same nixbld* uid.

This is based on aszlig's seccomp code
(47f587700d).

Reported by Linus Heckemann.
2017-05-29 16:14:10 +02:00
Eelco Dolstra 6e01ecd112
Fix nix-copy-closure test
Fixes

  client# error: size mismatch importing path ‘/nix/store/ywf5fihjlxwijm6ygh6s0a353b5yvq4d-libidn2-0.16’; expected 0, got 120264

This is mostly an artifact of the NixOS VM test environment, where the
Nix database doesn't contain hashes/sizes.

http://hydra.nixos.org/build/53537471
2017-05-29 16:08:56 +02:00
Eelco Dolstra 588dad4084
Fix build failure on Debian/Ubuntu
http://hydra.nixos.org/build/53537463
2017-05-29 15:59:18 +02:00
Eelco Dolstra 63145be2a5
Fix typo 2017-05-29 15:52:36 +02:00
Eelco Dolstra a7e55151a8
Fix #1314
Also, make nix-shell respect --option. (Previously it only passed it
along to nix-instantiate and nix-build.)
2017-05-24 11:33:42 +02:00
Eelco Dolstra fbe9fe0e75
Merge branch 'topic/cores-master' of https://github.com/neilmayhew/nix 2017-05-24 11:28:36 +02:00
Eelco Dolstra 9711524188
Fix #1380
It lacked a backslash. Use a raw string and single quotes around PS1
to simplify this.
2017-05-24 11:23:28 +02:00
Eelco Dolstra 01200d07d2
Merge branch 'prompt-terminator' of https://github.com/lheckemann/nix 2017-05-24 11:22:48 +02:00
Eelco Dolstra edbb105e98
Merge branch 'nar-accessor-tree' of https://github.com/bennofs/nix 2017-05-24 11:04:43 +02:00
Eelco Dolstra e46090edb1
builtins.match: Improve error message for bad regular expression
Issue #1331.
2017-05-17 11:58:01 +02:00
Eelco Dolstra b01d62285c
Improve progress indicator 2017-05-16 16:09:57 +02:00
Benno Fünfstück a1f428b13b nar-accessor.cc: remove unused member NarIndexer::currentName 2017-05-15 19:41:59 +02:00
Benno Fünfstück 5ee06e612a nar-accessor: non-recursive NarMember::find
This avoids a possible stack overflow if directories are very deeply nested.
2017-05-15 19:34:18 +02:00
Eelco Dolstra e80257f122
Simplify fixed-output check 2017-05-15 18:50:54 +02:00
Eelco Dolstra c05d9ae7a5
Disallow outputHash being null or an empty string
Fixes #1384.
2017-05-15 18:47:13 +02:00
Eelco Dolstra a2d92bb20e
Add --with-sandbox-shell configure flag
And add a 116 KiB ash shell from busybox to the release build. This
helps to make sandbox builds work out of the box on non-NixOS systems
and with diverted stores.
2017-05-15 17:36:32 +02:00
Eelco Dolstra b30f5784d0
Linux sandbox: Don't barf on invalid paths
This is useful when we're using a diverted store (e.g. "--store
local?root=/tmp/nix") in conjunction with a statically-linked sh from
the host store (e.g. "sandbox-paths =/bin/sh=/nix/store/.../bin/busybox").
2017-05-15 17:36:32 +02:00
Eelco Dolstra 2b761d5f50
Make fmt() non-recursive 2017-05-15 17:36:32 +02:00
Benno Fünfstück 06880d7ed8 nix ls: support '/' for the root directory 2017-05-15 10:25:55 +02:00
Benno Fünfstück 75a1d9849d nar-accessor: use tree, fixes readDirectory missing children
Previously, if a directory `foo` existed and a file `foo-` (where `-` is any character that is sorted before `/`), then  `readDirectory` would return an empty list.

To fix this, we now use a tree where we can just access the children of the node, and do not need to rely on sorting behavior to list the contents of a directory.
2017-05-15 10:23:16 +02:00
Eelco Dolstra 510bc1735b
Add an option for extending the user agent header
This is useful e.g. for distinguishing traffic to a binary cache
(e.g. certain machines can use a different tag in the user agent).
2017-05-11 15:10:22 +02:00
Eelco Dolstra 62d476c7ee
Fix typo 2017-05-11 14:02:03 +02:00
Eelco Dolstra ea65ae0f9c
Tweak error message 2017-05-11 13:59:47 +02:00
Eelco Dolstra 1a8e15053a
Don't allow untrusted users to set info.ultimate
Note that a trusted signature was still required in this case so it
was not a huge deal.
2017-05-11 13:58:09 +02:00
Eelco Dolstra 6f245bf24a
Change the meaning of info.ultimate
It now means "paths that were built locally". It no longer includes
paths that were added locally. For those we don't need info.ultimate,
since we have the content-addressability assertion (info.ca).
2017-05-11 13:31:23 +02:00
Eelco Dolstra 45d7b1a9e9
LocalStore::addToStore(): Check info.narSize
It allowed the client to specify bogus narSize values. In particular,
Downloader::downloadCached wasn't setting narSize at all.
2017-05-11 13:26:03 +02:00
Eelco Dolstra c5f23f10a8
Replace readline by linenoise
Using linenoise avoids a license compatibility issue (#1356), is a lot
smaller and doesn't pull in ncurses.
2017-05-10 18:37:42 +02:00
Linus Heckemann d48edcc3a5 nix-shell: use appropriate prompt terminator
If running nix-shell as root, the terminator should be # and not $.
2017-05-10 12:03:45 +01:00
Eelco Dolstra 03ae5e6459
Add "nix edit" command
This is a little convenience command that opens the Nix expression of
the specified package. For example,

  nix edit nixpkgs.perlPackages.Moose

opens <nixpkgs/pkgs/top-level/perl-packages.nix> in $EDITOR (at the
right line number for some editors).

This requires the package to have a meta.position attribute.
2017-05-08 18:42:30 +02:00
Eelco Dolstra 7689181e4f
Minor cleanup 2017-05-08 15:56:52 +02:00
Eelco Dolstra 00b286275c
Linux sandbox: Fix compatibility with older kernels 2017-05-08 15:42:59 +02:00
Eelco Dolstra ebfceeb333
build-remote: Check remote build status 2017-05-08 14:27:12 +02:00
Eelco Dolstra 0a97eb6bd7
Remove superfluous #ifdef 2017-05-08 11:27:20 +02:00
Eelco Dolstra bb50c89319
Make the location of the build directory in the sandbox configurable
This is mostly for use in the sandbox tests, since if the Nix store is
under /build, then we can't use /build as the build directory.
2017-05-05 17:45:22 +02:00
Eelco Dolstra 465cb68244
Figure out the user's home directory if $HOME is not set 2017-05-05 17:08:23 +02:00
Eelco Dolstra eba840c8a1
Linux sandbox: Use /build instead of /tmp as $TMPDIR
There is a security issue when a build accidentally stores its $TMPDIR
in some critical place, such as an RPATH. If
TMPDIR=/tmp/nix-build-..., then any user on the system can recreate
that directory and inject libraries into the RPATH of programs
executed by other users. Since /build probably doesn't exist (or isn't
world-writable), this mitigates the issue.
2017-05-04 16:57:03 +02:00
Eelco Dolstra 2da6a42448
nix dump-path: Add
This is primarily useful for extracting NARs from other stores (like
binary caches), which "nix-store --dump" cannot do.
2017-05-04 14:21:22 +02:00
Eelco Dolstra 72fb2a7edc
Fix build on gcc 4.9
http://hydra.nixos.org/build/52408843
2017-05-03 16:08:48 +02:00
Eelco Dolstra 08355643ab
nix-shell: Implement passAsFile 2017-05-03 15:01:15 +02:00
Eelco Dolstra 782c0bff45
nix eval: Add a --raw flag
Similar to "jq -r", this prints the evaluation result (which must be a
string value) unquoted.
2017-05-03 14:08:18 +02:00
Eelco Dolstra cef8c169b1
Fix "nix ... --all"
When "--all" is used, we should not fill in a default installable.
2017-05-02 15:46:10 +02:00
Eelco Dolstra c5bea16611
LocalStoreAccessor: Fix handling of diverted stores 2017-05-02 15:46:09 +02:00
Eelco Dolstra 7f6837a0f6
Replace $NIX_REMOTE_SYSTEMS with an option "builder-files"
Also, to unify with hydra-queue-runner, allow it to be a list of
files.
2017-05-02 15:46:09 +02:00
Eelco Dolstra cd4d2705ec
build-remote: Fix fallback to other machines when connecting fails
Opening an SSHStore or LegacySSHStore does not actually establish a
connection, so the try/catch block here did nothing. Added a
Store::connect() method to test whether a connection can be
established.
2017-05-02 15:46:09 +02:00
Eelco Dolstra 1a68710d4d
Add an option for specifying remote builders
This is useful for one-off situations where you want to specify a
builder on the command line instead of having to mess with
nix.machines. E.g.

  $ nix-build -A hello --argstr system x86_64-darwin \
    --option builders 'root@macstadium1 x86_64-darwin'

will perform the specified build on "macstadium1".

It also removes the need for a separate nix.machines file since you
can specify builders in nix.conf directly. (In fact nix.machines is
yet another hack that predates the general nix.conf configuration
file, IIRC.)

Note: this option is supported by the daemon for trusted users. The
fact that this allows trusted users to specify paths to SSH keys to
which they don't normally have access is maybe a bit too much trust...
2017-05-02 15:42:43 +02:00
Eelco Dolstra ebc9f36a81
Factor out machines.conf parsing
This allows hydra-queue-runner to use it.
2017-05-02 13:17:37 +02:00
Eelco Dolstra 174b68a2a2
build-hook: If there are no machines defined, quit permanently 2017-05-02 12:16:29 +02:00
Eelco Dolstra feefcb3a98
build-remote: Ugly hackery to get build logs to work
The build hook mechanism expects build log output to go to file
descriptor 4, so do that.
2017-05-02 12:02:23 +02:00
Eelco Dolstra 3a5f04f48c
build-remote: Don't require signatures
This restores the old behaviour.
2017-05-01 20:03:25 +02:00
Eelco Dolstra 031d70e500
Support arbitrary store URIs in nix.machines
For backwards compatibility, if the URI is just a hostname, ssh://
(i.e. LegacySSHStore) is prepended automatically.

Also, all fields except the URI are now optional. For example, this is
a valid nix.machines file:

  local?root=/tmp/nix

This is useful for testing the remote build machinery since you don't
have to mess around with ssh.
2017-05-01 17:35:30 +02:00
Eelco Dolstra 3e4bdfedee
Minor cleanup 2017-05-01 17:30:17 +02:00
Eelco Dolstra deac171925
Implement LegacySSHStore::buildDerivation()
This makes LegacySSHStore usable by build-remote and
hydra-queue-runner.
2017-05-01 17:30:16 +02:00
Eelco Dolstra 3f5b98e65a
Chomp log output from the build hook 2017-05-01 17:30:16 +02:00
Eelco Dolstra d7653dfc6d
Remove $NIX_BUILD_HOOK and $NIX_CURRENT_LOAD
This is to simplify remote build configuration. These environment
variables predate nix.conf.

The build hook now has a sensible default (namely build-remote).

The current load is kept in the Nix state directory now.
2017-05-01 17:30:16 +02:00
Eelco Dolstra ca9f589a93
build-remote: Don't copy the .drv closure
Since build-remote uses buildDerivation() now, we don't need to copy
the .drv file anymore. This greatly reduces the set of input paths
copied to the remote side (e.g. from 392 to 51 store paths for GNU
hello on x86_64-darwin).
2017-05-01 17:30:16 +02:00
Eelco Dolstra b986c7f8b1
Pass verbosity level to build hook 2017-05-01 14:43:14 +02:00
Eelco Dolstra 227a48f86f
Reduce severity of EMLINK warnings
Fixes #1357.
2017-05-01 14:36:56 +02:00
Eelco Dolstra 0dddcf867a
Add a dummy Store::buildPaths() method
This default implementation of buildPaths() does nothing if all
requested paths are already valid, and throws an "unsupported
operation" error otherwise. This fixes a regression introduced by
c30330df6f in binary cache and legacy
SSH stores.
2017-05-01 13:43:34 +02:00
Guillaume Maudoux a143014d73 lexer: remove catch-all rules hiding real errors
With catch-all rules, we hide potential errors.
It turns out that a4744254 made one cath-all useless. Flex detected that
is was impossible to reach.
The other is more subtle, as it can only trigger on unfinished escapes
in unfinished strings, which only occurs at EOF.
2017-05-01 01:18:06 +02:00
Guillaume Maudoux a474425425 Fix lexer to support $' in multiline strings. 2017-05-01 01:15:40 +02:00
Eelco Dolstra 2f21d522c2
Hopefully fix the Darwin build
http://hydra.nixos.org/build/52080911
2017-04-28 17:13:55 +02:00
Eelco Dolstra 895f00c372
Suppress warning about ssh-auth-sock 2017-04-28 16:55:52 +02:00
Eelco Dolstra 73bba12d8b
Check for libreadline 2017-04-28 16:53:56 +02:00
Eelco Dolstra a1a5e63e14
Fix brainfart 2017-04-28 16:21:54 +02:00
Eelco Dolstra 41c4558afe
Fix hash computation when importing NARs greater than 4 GiB
This caused "nix-store --import" to compute an incorrect hash on NARs
that don't fit in an unsigned int. The import would succeed, but
"nix-store --verify-path" or subsequent exports would detect an
incorrect hash.

A deeper issue is that the export/import format does not contain a
hash, so we can't detect such issues early.

Also, I learned that -Wall does not warn about this.
2017-04-28 15:24:05 +02:00
Eelco Dolstra 39b08f4c0c Merge pull request #1358 from shlevy/store-nesting
Add Store nesting to fix import-from-derivation within filterSource
2017-04-26 20:28:49 +02:00
Shea Levy 4bc00760f9 Add Store nesting to fix import-from-derivation within filterSource 2017-04-26 14:15:47 -04:00
Eelco Dolstra 45ce2c7413
Doh 2017-04-26 17:58:09 +02:00
Eelco Dolstra 6734c18c99
nix repl: Fix Ctrl-C 2017-04-25 19:19:48 +02:00
Eelco Dolstra 23aa1619da
Minor cleanup 2017-04-25 19:10:47 +02:00
Eelco Dolstra 536f061765
"using namespace std" considered harmful 2017-04-25 18:58:02 +02:00
Eelco Dolstra 5bd8795e1f
nix repl: Use $XDG_DATA_HOME for the readline history 2017-04-25 18:56:29 +02:00
Eelco Dolstra 921a2aeb05
Make "nix repl" build 2017-04-25 18:48:40 +02:00
Eelco Dolstra c31000bc93
Merge nix-repl repository 2017-04-25 18:14:13 +02:00
Eelco Dolstra 40daf0d800
Cleanup in preparation of merging nix-repl repo into nix repo 2017-04-25 18:13:23 +02:00
Eelco Dolstra c30330df6f
StorePathCommands: Build installables
So for instance "nix copy --to ... nixpkgs.hello" will build
nixpkgs.hello first. It's debatable whether this is a good idea. It
seems desirable for commands like "nix copy" but maybe not for
commands like "nix path-info".
2017-04-25 16:19:22 +02:00
Eelco Dolstra d48c973ece
Set default installable
Thus

  $ nix build -f foo.nix

will build foo.nix.

And

  $ nix build

will build default.nix. However, this may not be a good idea because
it's kind of inconsistent, given that "nix build foo" will build the
"foo" attribute from the default installation source (i.e. the
synthesis of $NIX_PATH), rather than ./default.nix. So I may revert
this.
2017-04-25 15:18:05 +02:00
Eelco Dolstra 0b6220fbd6
Interpret any installable containing a slash as a path
So "nix path-info ./result" now works.
2017-04-25 14:09:01 +02:00
Eelco Dolstra 7ee81f3887
Make StorePathsCommand a subclass of InstallablesCommand
This allows commands like 'nix path-info', 'nix copy', 'nix verify'
etc. to work on arbitrary installables. E.g. to copy geeqie to a
binary cache:

  $ nix copy -r --to file:///tmp/binary-cache nixpkgs.geeqie

Or to get the closure size of thunderbird:

  $ nix path-info -S nixpkgs.thunderbird
2017-04-25 13:20:26 +02:00
Eelco Dolstra c769841bc4
Move code around 2017-04-25 12:07:31 +02:00
Eelco Dolstra 6267d74889
Add "nix eval" command
This replaces "nix-instantiate --eval". The result is evaluated
strictly since this seems more useful.
2017-04-25 11:23:47 +02:00
Eelco Dolstra bcecc99007
Restructure installables handling in the "nix" command 2017-04-25 11:20:37 +02:00
Eelco Dolstra 1bb87c0487
Remove debug statement 2017-04-24 15:01:28 +02:00
Eelco Dolstra 66577a1c64
Factor out --json 2017-04-24 14:21:36 +02:00
Eelco Dolstra 9b63bb88c8
nix-shell -p: Use runCommandCC
This restores pre-17.03 behaviour by making gcc available.
2017-04-24 12:04:01 +02:00
David McFarland 804ac52489 add helper function to set 'interruptThrown'
this fixes a linker failure on cygwin 64 due to some bad
interaction between tls and shared libraries.

see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64697
2017-04-21 11:28:14 -03:00
Eelco Dolstra 749696e71c
Detect lsof
Also, don't use lsof on Linux since it's not needed.

Fixes #1328.
2017-04-20 19:11:45 +02:00
Eelco Dolstra efa4bdbfcd
Improve nix show-config --json
In particular, show descriptions. This could be used for manpage
generation etc.
2017-04-20 17:34:47 +02:00
Eelco Dolstra 4410e9d995
Setting: Remove "Tag" template argument 2017-04-20 16:52:53 +02:00
Eelco Dolstra f05d5f89ff
Read per-user settings from ~/.config/nix/nix.conf 2017-04-20 14:58:16 +02:00
Eelco Dolstra 562585e901
binary-caches-parallel-connections -> http-connections 2017-04-20 14:04:00 +02:00
Eelco Dolstra 76cb3c702c
Reimplement trusted-substituters (aka trusted-binary-caches) 2017-04-20 13:41:29 +02:00
Eelco Dolstra 9cc8047f44
Reimplement connect-timeout
Fixes #1339.
2017-04-19 14:54:52 +02:00
Eelco Dolstra b0cb117226
getDerivations(): Filter out packages with bad derivation names
In particular, this disallows attribute names containing dots or
starting with dots. Hydra already disallowed these. This affects the
following packages in Nixpkgs master:

  2048-in-terminal
  2bwm
  389-ds-base
  90secondportraits
  lispPackages.3bmd
  lispPackages.hu.dwim.asdf
  lispPackages.hu.dwim.def

Closes #1342.
2017-04-19 14:18:26 +02:00
Eelco Dolstra 67fe3e07b2 Merge pull request #1321 from shlevy/channel-direct-tarball-error
nix-channel: error out if direct tarball unpack fails.
2017-04-18 16:51:46 +02:00
Eelco Dolstra f8a2e8a552
Shut up some warnings 2017-04-14 14:42:20 +02:00
Eelco Dolstra dd3714f6ef
Doh 2017-04-14 14:42:08 +02:00
Eelco Dolstra 6520b757c5
Fix 32-bit build
http://hydra.nixos.org/build/51569816
2017-04-14 13:59:39 +02:00
Eelco Dolstra 3872371f25
Minor cleanup 2017-04-14 13:42:22 +02:00
Eelco Dolstra 01dcdfcf33
nix-daemon: Don't set untrusted-* settings
These are no longer used anywhere.
2017-04-14 11:59:51 +02:00
Eelco Dolstra 1673c373c9
nix-daemon: Don't die if the user sends an unknown setting 2017-04-14 11:57:02 +02:00
Eelco Dolstra 872ba75d8b
Add "nix show-config" command
This dumps the entire Nix configuration, including all options that
have default values.
2017-04-13 20:59:38 +02:00
Eelco Dolstra ba9ad29fdb
Convert Settings to the new config system
This makes all config options self-documenting.

Unknown or unparseable config settings and --option flags now cause a
warning.
2017-04-13 20:53:23 +02:00
Eelco Dolstra 6bd9576aeb
Support arbitrary numeric types for settings 2017-04-13 17:54:05 +02:00
Eelco Dolstra 0bf34de43b
Validate Boolean settings better 2017-04-13 16:31:28 +02:00
Eelco Dolstra 1860070548
Merge branch 'rework-options' of https://github.com/copumpkin/nix 2017-04-13 16:15:51 +02:00
Eelco Dolstra 2040240e23
Add a Config class to simplify adding configuration settings
The typical use is to inherit Config and add Setting<T> members:

  class MyClass : private Config
  {
    Setting<int> foo{this, 123, "foo", "the number of foos to use"};
    Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"};

    MyClass() : Config(readConfigFile("/etc/my-app.conf"))
    {
      std::cout << foo << "\n"; // will print 123 unless overriden
    }
  };

Currently, this is used by Store and its subclasses for store
parameters. You now get a warning if you specify a non-existant store
parameter in a store URI.
2017-04-13 16:03:31 +02:00
Eelco Dolstra 568a099c88
canonPath(): Check against empty paths 2017-04-13 16:03:31 +02:00
Eelco Dolstra 6d97d81656
Add warn function 2017-04-13 16:03:31 +02:00
Eelco Dolstra 31cc9366fc
Initialise logger 2017-04-13 16:03:31 +02:00
Eelco Dolstra 23304f527a Merge pull request #1302 from dtzWill/fix/nix-options
Process nix.conf options in "new" commands, add test
2017-04-12 11:03:19 +02:00
Shea Levy 503cc4431b nix-channel: error out if direct tarball unpack fails.
It's very unlikely a path ending in .tar.gz is a directory

Fixes #1318
2017-04-10 18:16:46 -04:00
Eelco Dolstra 1fe1976e0d Merge pull request #1316 from copumpkin/nix-retries-default
Default to 5 download retries
2017-04-10 15:58:57 +02:00
Dan Peebles d1fdade755 Add CURLE_WRITE_ERROR as a transient error condition
We've observed it failing downloads in the wild and retrying the same URL
a few moments later seemed to fix it.
2017-04-10 09:28:44 -04:00
Dan Peebles e43e8be8e7 Default to 5 download retries
This should help certain downloaders that don't request anything special
for the number of retries, like nix-channel.
2017-04-10 09:22:24 -04:00
Eelco Dolstra 105f8ffc98
Minor cleanup
Also, possible fix for #1310 on 32-bit systems.
2017-04-10 11:27:33 +02:00
Eelco Dolstra 95295482ea
Allow "auto" as a store URI
Using the empty string is likely to be ambiguous in some contexts.
2017-04-10 11:27:29 +02:00
Neil Mayhew f12a048a05 Propagate NIX_BUILD_CORES to nix-shell environments 2017-04-09 08:21:52 -06:00
Will Dietz 30f89e0d65 Process nix.conf options in "new" nix commands, add test.
Without this (minor) change, the options set using "--option"
or read from nix.conf were parsed but not used.
2017-04-08 12:59:42 -05:00
Dan Peebles 98283915f5 Retry downloads on transient SSL errors too 2017-04-06 18:18:43 +00:00
Eelco Dolstra ba20730b3f
Implement RemoteStore::queryMissing()
This provides a significant speedup, e.g. 64 s -> 12 s for

  nix-build --dry-run -I nixpkgs=channel:nixos-16.03 '<nixpkgs/nixos/tests/misc.nix>' -A test

on a cold local and CloudFront cache.

The alternative is to use lots of concurrent daemon connections but
that seems wasteful.
2017-04-06 18:40:19 +02:00
Eelco Dolstra 963f2bf12b
Fix bogus "unexpected Nix daemon error: interrupted by the user" 2017-04-06 17:19:32 +02:00
Eelco Dolstra 6b5e271163
Add a method to allow hydra-queue-runner to flush the path info cache 2017-04-06 15:22:37 +02:00
Eelco Dolstra 256940fc48
nix-daemon: Disable path info cache
This is useless because the client also caches path info, and can
cause problems for long-running clients like hydra-queue-runner
(i.e. it may return cached info about paths that have been
garbage-collected).
2017-04-06 14:30:31 +02:00
Eelco Dolstra 8decb07c31
Allow default sandbox paths to be overriden
E.g. you can now redirect /etc/resolv.conf to a different file.
2017-04-04 17:54:16 +02:00
Eelco Dolstra 488792a87d
Make /var/run/nscd/socket optional
Not every distribution uses nscd.
2017-04-04 17:40:50 +02:00
Eelco Dolstra 29d35805c6
Sandbox: Fix /dev/ptmx on recent kernels
This fixes "No such file or directory" when opening /dev/ptmx
(e.g. http://hydra.nixos.org/build/51094249).

The reason appears to be some changes to /dev/ptmx / /dev/pts handling
between Linux 4.4 and 4.9. See
https://patchwork.kernel.org/patch/7832531/.

The fix is to go back to mounting a proper /dev/pts instance inside
the sandbox. Happily, this now works inside user namespaces, even for
unprivileged users. So

  NIX_REMOTE=local?root=/tmp/nix nix-build \
    '<nixpkgs/nixos/tests/misc.nix>' -A test

works for non-root users.

The downside is that the fix breaks sandbox builds on older kernels
(probably pre-4.6), since mounting a devpts fails inside user
namespaces for some reason I've never been able to figure out. Builds
on those systems will fail with

  error: while setting up the build environment: mounting /dev/pts: Invalid argument

Ah well.
2017-03-31 18:20:19 +02:00
Shea Levy 3ecb09a40a builtins.exec: Make the argument just a list 2017-03-31 11:58:41 -04:00
Shea Levy d299bd710a Merge branch 'builtins.exec' 2017-03-31 11:22:39 -04:00
Eelco Dolstra e1509adbbb
Retry curl error 16 2017-03-31 15:50:11 +02:00
Eelco Dolstra c0745a2531
Merge branch 'remove-perl' of https://github.com/shlevy/nix 2017-03-31 14:13:32 +02:00
Shea Levy 0bb8db257d Add exec primop behind allow-unsafe-native-code-during-evaluation.
Execute a given program with the (optional) given arguments as the
user running the evaluation, parsing stdout as an expression to be
evaluated.

There are many use cases for nix that would benefit from being able to
run arbitrary code during evaluation, including but not limited to:

* Automatic git fetching to get a sha256 from a git revision
* git rev-parse HEAD
* Automatic extraction of information from build specifications from
  other tools, particularly language-specific package managers like
  cabal or npm
* Secrets decryption (e.g. with nixops)
* Private repository fetching

Ideally, we would add this functionality in a more principled way to
nix, but in the mean time 'builtins.exec' can be used to get these
tasks done.

The primop is only available when the
'allow-unsafe-native-code-during-evaluation' nix option is true. That
flag also enables the 'importNative' primop, which is strictly more
powerful but less convenient (since it requires compiling a plugin
against the running version of nix).
2017-03-30 08:04:21 -04:00
Eelco Dolstra c60715e937
Ignore broken "Deriver: unknown-deriver" fields in .narinfo
These were generated by a legacy tool.
2017-03-28 13:08:13 +02:00
Daiderd Jordan 023217f07c
use std::tuple for ValueMap allocator 2017-03-24 23:05:49 +01:00
Eelco Dolstra 0a7ca24c26
Fix xz decompression
Fixes #1285.
2017-03-22 11:53:33 +01:00
Eelco Dolstra 895a74a814
LocalFSStore::getBuildLog(): Handle corrupted logs 2017-03-21 19:23:35 +01:00
Eelco Dolstra ed5c0f69f2
Don't hang in decompression if bzip2 data ends prematurely 2017-03-21 19:23:35 +01:00
Eelco Dolstra e1e49c58e1
Only use cache.nixos.org when the store is /nix/store
This is consistent with the behaviour of the old
download-from-binary-cache substituter.
2017-03-21 17:59:18 +01:00
Eelco Dolstra ecbc3fedd3
Require signatures by default
This corresponds to the NixOS default.
2017-03-21 15:10:48 +01:00
Eelco Dolstra 4bb38591e5
Restore cache.nixos.org as the default substituter
Fixes #1283.
2017-03-21 15:06:46 +01:00
Eelco Dolstra f8b84a3b8c
Move istringstream_nocopy to a separate file 2017-03-21 14:43:03 +01:00
Eelco Dolstra 3229f85585
Honor $NIX_SSHOPTS again
NixOps needs this.
2017-03-21 14:35:50 +01:00
Eelco Dolstra 20df50d8e1
Merge branch 'darwin-s3-binary-cache-store' of https://github.com/shlevy/nix 2017-03-20 17:46:35 +01:00
Eelco Dolstra dc931fe1cd Merge pull request #1275 from steveeJ/patch-1
nix-shell/pure: keep environment variable SHLVL
2017-03-20 17:38:40 +01:00
Eelco Dolstra 558eda0115
nix copy: Make -r option use the "from" store
Previously, we tried to compute the closure in the local store, which
obviously doesn't work.
2017-03-16 14:25:54 +01:00
Eelco Dolstra 287084d688
ssh:// -> ssh-ng://, legacy-ssh:// -> ssh:// 2017-03-16 14:19:32 +01:00
Eelco Dolstra c5b83d8913
copyPaths(): Use queryValidPaths() to reduce SSH latency 2017-03-16 13:50:01 +01:00
Eelco Dolstra 91d67692cf
copyPaths(): Don't query path info for a path the target already has
For example, this cuts "nix-copy-closure --from" on a NixOS system
closure from 15.9s to 0.5s.
2017-03-16 12:05:51 +01:00
Eelco Dolstra ea7fa88131
LegacySSHStore: Provide a faster implementation of computeFSClosure()
This avoids the latency of the standard implementation, which can make
a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16 11:44:52 +01:00
Eelco Dolstra 7a716ef2a5
Fix nix-copy-closure --to 2017-03-16 10:58:48 +01:00
Eelco Dolstra 0ec7f47b00
Remove "killing process <pid>" messages
They convey no useful information.
2017-03-16 10:52:28 +01:00
Eelco Dolstra 43f158bb08
nix-copy-closure: Fix assertion failure
$ ./inst/bin/nix-copy-closure --to bla $(type -p firefox)
  nix-copy-closure: src/libstore/store-api.cc:80: std::__cxx11::string nix::storePathToHash(const Path&): Assertion `base.size() >= storePathHashLen' failed.
2017-03-16 10:45:45 +01:00
Eelco Dolstra 3f35612c04
More precise compression settings 2017-03-15 17:20:19 +01:00
Eelco Dolstra 25dff2b7db
runProgram(): Distinguish between empty input and no input
For example, if we call brotli with an empty input, it shouldn't read
from the caller's stdin.
2017-03-15 16:50:19 +01:00
Eelco Dolstra 042975ea8e
Compress NAR listings using the "text-compression" method
So if "text-compression=br", the .ls file in S3 will get a
Content-Encoding of "br". Brotli appears to compress better than xz
for this kind of file and is natively supported by browsers.
2017-03-15 16:50:19 +01:00
Eelco Dolstra 45c70382ac
S3BinaryCacheStore: Set Content-Type
This is necessary for serving log files to browsers.
2017-03-15 16:50:19 +01:00
Eelco Dolstra 8b1d65bebe
S3BinaryCacheStore: Support compression of narinfo and log files
You can now set the store parameter "text-compression=br" to compress
textual files in the binary cache (i.e. narinfo and logs) using
Brotli. This sets the Content-Encoding header; the extension of
compressed files is unchanged.

You can separately specify the compression of log files using
"log-compression=br". This is useful when you don't want to compress
narinfo files for backward compatibility.
2017-03-15 16:49:28 +01:00
Eelco Dolstra 2691498b5c
Fix assertion failure on SIGINT
nix: src/libutil/compression.cc:142: virtual nix::XzSink::~XzSink(): Assertion `finished' failed.
2017-03-15 16:49:06 +01:00
Eelco Dolstra fbbc4d8dda
Fix deadlock in runProgram() when input is larger than the pipe buffer size 2017-03-15 16:49:06 +01:00
Eelco Dolstra e8186085e0
Add support for brotli compression
Build logs on cache.nixos.org are compressed using Brotli (since this
allows them to be decompressed automatically by Chrome and Firefox),
so it's handy if "nix log" can decompress them.
2017-03-15 16:49:06 +01:00
Eelco Dolstra 73d7a51ee6
Remove dependency on "curl" binary 2017-03-15 16:48:29 +01:00
Eelco Dolstra 532d73d5d8
BinaryCacheStore: Implement getBuildLog()
We assume that build logs are stored under log/<drv>, e.g.

  /nix/store/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv

maps to

  https://cache.nixos.org/log/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv
2017-03-15 16:48:29 +01:00
Eelco Dolstra 5b86451f02
Add a "nix log" command
This replaces "nix-store --read-log". It checks the local store and
any configured substituters for the requested logs.
2017-03-15 16:48:29 +01:00
Eelco Dolstra 0afeb7f51e
Store: Add a method for getting build logs
This allows various Store implementations to provide different ways to
get build logs. For example, BinaryCacheStore can get the build logs
from the binary cache.

Also, remove the log-servers option since we can use substituters for
this.
2017-03-15 16:48:29 +01:00
Shea Levy 96443e94a1 Merge branch 'configurable-aws-region' 2017-03-12 10:01:11 -04:00
Stefan Junker f628ca2a1f nix-shell/pure: keep environment variable SHLVL 2017-03-12 01:04:21 +01:00
Eelco Dolstra 86227390c5
Merge branch 'allow-import-from-derivation' of https://github.com/shlevy/nix 2017-03-09 13:36:30 +01:00
Adrien Devresse d853877ce9 Add missing header <sstream> 2017-03-08 22:24:10 +01:00
Shea Levy 93f863be96 Add option to disable import-from-derivation completely, even if the drv is already realized 2017-03-08 08:46:12 -05:00
Shea Levy 3cc18d3753 Properly set the caFile for aws-sdk-cpp s3 2017-03-06 14:30:35 -05:00
Shea Levy 4fc30922cf istringstream_nocopy: Implement in a standards-compliant way.
Fixes the problem mentioned in e6a61b8da7

See #1135
2017-03-06 13:03:02 -05:00
Eelco Dolstra 1cf4801108
Revert fa125b9b28
This causes quadratic performance.
2017-03-06 13:56:20 +01:00
Shea Levy 5789eaa3f4 Add aws-region param to S3 store URLs 2017-03-03 16:12:17 -05:00
Eelco Dolstra d1158bb816
Cache connection failures 2017-03-03 19:36:01 +01:00
Eelco Dolstra 8490ee37a6
SSHMaster: Make thread-safe 2017-03-03 19:28:27 +01:00
Eelco Dolstra d3eb1cf3bb
build-remote: Don't use a SSH master
This is unnecessary because we make only one connection.
2017-03-03 19:23:20 +01:00
Eelco Dolstra d140c75530
Pool: Don't hang if creating a connection fails 2017-03-03 19:21:43 +01:00
Eelco Dolstra 577ebeaefb
Improve SSH handling
* Unify SSH code in SSHStore and LegacySSHStore.

* Fix a race starting the SSH master. We now wait synchronously for
  the SSH master to finish starting. This prevents the SSH clients
  from starting their own connections.

* Don't use a master if max-connections == 1.

* Add a "max-connections" store parameter.

* Add a "compress" store parameter.
2017-03-03 19:05:50 +01:00
Eelco Dolstra 7f62be1bcd
build-remote: Fix passing SSH key 2017-03-03 16:33:18 +01:00
Eelco Dolstra 5a1fb03b8f
build-remote: Misc cleanup 2017-03-03 16:18:49 +01:00
Eelco Dolstra 2f992692e2
Fix fatal "broken pipe" error when $NIX_BUILD_HOOK is missing 2017-03-03 15:40:06 +01:00
Eelco Dolstra ffcf9d24a6
builtins.fetchgit: Support paths 2017-03-02 11:46:28 +01:00
Eelco Dolstra fbf17f1ad7
builtins.fetchgit: Fix bad format string 2017-03-02 11:40:11 +01:00
Eelco Dolstra fa125b9b28
TeeSink: Pre-reserve string space
When receiving a very large file, this can prevent the string from
having tobe copied, which temporarily doubles memory consumption.
2017-03-01 16:16:04 +01:00
Eelco Dolstra f61f67ddee
RemoteStore::addToStore(): Send NAR rather than string containing NAR
This allows the NAR to be streamed in the future (though we're not
doing that yet).
2017-03-01 16:07:15 +01:00
Eelco Dolstra 374908726b
readString(): Read directly into std::string
When reading a huge string, this halves memory consumption.

(Strictly speaking, this appears only valid in C++17, but who cares...)
2017-03-01 14:54:11 +01:00
Eelco Dolstra 07a0b8ca67
Tweak message 2017-03-01 14:52:16 +01:00
Eelco Dolstra 56e19d970d
nix-store --import: Fix importing unsigned paths 2017-03-01 14:47:52 +01:00
Eelco Dolstra e321551d54
Fix assertion failure in nix-store --export
Fixes #1173.
2017-03-01 14:41:54 +01:00
Eelco Dolstra c4a40949d9
Handle importing NARs containing files greater than 4 GiB
Also templatize readInt() to work for various integer types.
2017-03-01 13:52:54 +01:00
Eelco Dolstra 0780805246
Fix building against older curl versions
http://hydra.nixos.org/build/49490928
2017-02-28 14:03:43 +01:00
Eelco Dolstra fd86dd93dd
Improve SQLite busy handling 2017-02-28 13:59:11 +01:00
Eelco Dolstra 34b12bad59
NarInfoDiskCache: Handle SQLite busy errors 2017-02-28 13:44:11 +01:00
Eelco Dolstra 80027144ae
In SQLite errors, include the database path
This is necessary because we have multiple SQLite databases (e.g. the
binary cache cache).
2017-02-28 13:20:12 +01:00
Eelco Dolstra 7251d048fa
Support auto-configuration of build-max-jobs
"build-max-jobs" and the "-j" option can now be set to "auto" to use
the number of CPUs in the system. (Unlike build-cores, it doesn't use
0 to imply auto-configuration, because a) magic values are a bad idea
in general; b) 0 is a legitimate value used to disable local
building.)

Fixes #1198.
2017-02-28 12:54:50 +01:00
Eelco Dolstra 3fab1f04a7
_SC_NPROCESSORS_ONLN -> std:🧵:hardware_concurrency() 2017-02-27 16:01:54 +01:00
Eelco Dolstra 3cb0387d3f
Retry downloads on HTTP/2 stream errors
Issue #1254.
2017-02-27 14:36:09 +01:00
Eelco Dolstra d3e1aad421
nix-shell: Better error message when the shell can't be started 2017-02-24 17:25:00 +01:00
Eelco Dolstra e76df9bd52
Register content-addressability assertion for fixed outputs 2017-02-24 16:39:53 +01:00
Eelco Dolstra 1c718f80d3
Verify content-addressability assertions at registration time 2017-02-24 16:38:14 +01:00
Eelco Dolstra 668fef2e4f
nix-shell: Overwrite environment variables
Need to remember that std::map::insert() and emplace() don't overwrite
existing entries...

This fixes a regression relative to 1.11 that in particular triggers
in nested nix-shells.

Before:

  $ nativeBuildInputs=/foo nix-shell -p hello --run 'hello'
  build input /foo does not exist

After:

  $ nativeBuildInputs=/foo nix-shell -p hello --run 'hello'
  Hello, world!
2017-02-24 13:31:46 +01:00
Dan Peebles e7cb2847ab Explicitly model all settings and fail on unrecognized ones
Previously, the Settings class allowed other code to query for string
properties, which led to a proliferation of code all over the place making
up new options without any sort of central registry of valid options. This
commit pulls all those options back into the central Settings class and
removes the public get() methods, to discourage future abuses like that.

Furthermore, because we know the full set of options ahead of time, we
now fail loudly if someone enters an unrecognized option, thus preventing
subtle typos. With some template fun, we could probably also dump the full
set of options (with documentation, defaults, etc.) to the command line,
but I'm not doing that yet here.
2017-02-22 20:19:25 -05:00
Eelco Dolstra f023f64f40
RemoteStore::addToStore(): Pass content-addressability assertion
... and use this in Downloader::downloadCached(). This fixes

  $ nix-build https://nixos.org/channels/nixos-16.09-small/nixexprs.tar.xz -A hello
  error: cannot import path ‘/nix/store/csfbp1s60dkgmk9f8g0zk0mwb7hzgabd-nixexprs.tar.xz’ because it lacks a valid signature
2017-02-22 16:58:00 +01:00
Eelco Dolstra 8b1b5f9a12
Handle CURLE_RECV_ERROR as a transient error
This fixes

  unable to download ‘https://cache.nixos.org/nar/077h8ji74y9b0qx7rjk71xd80vjqp6q5gy137r553jlvdlxdcdlk.nar.xz’: HTTP error 200 (curl error: Failure when receiving data from the peer)
2017-02-21 16:04:47 +01:00
Eelco Dolstra 99bbddedb1
Fix building without S3 support
http://hydra.nixos.org/build/49031196/nixlog/2/raw
2017-02-21 13:15:07 +01:00
Eelco Dolstra df66d346df
Log AWS retries 2017-02-21 11:50:31 +01:00
Eelco Dolstra 302386f775
Support netrc in <nix/fetchurl.nix>
This allows <nix/fetchurl.nix> to fetch private Git/Mercurial
repositories, e.g.

  import <nix/fetchurl.nix> {
    url = https://edolstra@bitbucket.org/edolstra/my-private-repo/get/80a14018daed.tar.bz2;
    sha256 = "1mgqzn7biqkq3hf2697b0jc4wabkqhmzq2srdymjfa6sb9zb6qs7";
  }

where /etc/nix/netrc contains:

  machine bitbucket.org
  login edolstra
  password blabla...

This works even when sandboxing is enabled.

To do: add unpacking support (i.e. fetchzip functionality).
2017-02-16 15:51:50 +01:00
Eelco Dolstra cde4b60919
Move netrcFile to Settings
Also get rid of Settings::processEnvironment(), it appears to be
useless.
2017-02-16 14:50:41 +01:00
Eelco Dolstra 40f0e3b366
Include "curl" in the User-Agent header
Some sites (e.g. BitBucket) give a helpful 401 error when trying to
download a private archive if the User-Agent contains "curl", but give
a redirect to a login page otherwise (so for instance
"nix-prefetch-url" will succeed but produce useless output).
2017-02-16 13:55:43 +01:00
Eelco Dolstra d1139ff36b Merge pull request #1215 from k0001/netrc-1
Add netrc-file support
2017-02-16 12:45:44 +01:00
Eelco Dolstra 9ff9c3f2f8
Add support for s3:// URIs
This adds support for s3:// URIs in all places where Nix allows URIs,
e.g. in builtins.fetchurl, builtins.fetchTarball, <nix/fetchurl.nix>
and NIX_PATH. It allows fetching resources from private S3 buckets,
using credentials obtained from the standard places (i.e. AWS_*
environment variables, ~/.aws/credentials and the EC2 metadata
server). This may not be super-useful in general, but since we already
depend on aws-sdk-cpp, it's a cheap feature to add.
2017-02-14 14:20:00 +01:00
Eelco Dolstra 62ff5ad424 Merge pull request #1224 from dezgeg/configh
Unbreak 32-bit builds by always implicitly including config.h
2017-02-13 14:44:58 +01:00
Tuomas Tynkkynen 649a81bcd6 nix-daemon: Don't splice with len=SIZE_MAX
Currently, 'nix-daemon --stdio' is always failing for me, due to the
splice call always failing with (on a 32-bit host):

splice(0, NULL, 3, NULL, 4294967295, SPLICE_F_MOVE) = -1 EINVAL (Invalid argument)

With a bit of ftracing (and luck) the problem seems to be that splice()
always fails with EINVAL if the len cast as ssize_t is negative:
http://lxr.free-electrons.com/source/fs/read_write.c?v=4.4#L384

So use SSIZE_MAX instead of SIZE_MAX.
2017-02-13 15:14:44 +02:00
Renzo Carbonara e2257d4eeb Documentation. 2017-02-09 18:16:09 +01:00
Tuomas Tynkkynen 2cd468874f Include config.h implicitly with '-include config.h' in CFLAGS
Because config.h can #define things like _FILE_OFFSET_BITS=64 and not
every compilation unit includes config.h, we currently compile half of
Nix with _FILE_OFFSET_BITS=64 and other half with _FILE_OFFSET_BITS
unset. This causes major havoc with the Settings class on e.g. 32-bit ARM,
where different compilation units disagree with the struct layout.

E.g.:

diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc
@@ -166,6 +166,8 @@ void Settings::update()
     _get(useSubstitutes, "build-use-substitutes");
+    fprintf(stderr, "at Settings::update(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
     _get(buildUsersGroup, "build-users-group");
diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc
+++ b/src/libstore/remote-store.cc
@@ -138,6 +138,8 @@ void RemoteStore::initConnection(Connection & conn)
 void RemoteStore::setOptions(Connection & conn)
 {
+    fprintf(stderr, "at RemoteStore::setOptions(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
     conn.to << wopSetOptions

Gave me:

at Settings::update(): &useSubstitutes = 0xb6e5c5cb
at RemoteStore::setOptions(): &useSubstitutes = 0xb6e5c5c7

That was not a fun one to debug!
2017-02-08 21:51:02 +02:00
Shea Levy 418a837897 Remove perl dependency.
Fixes #341
2017-02-07 15:56:32 -05:00
Eelco Dolstra 4724903c78
nix-copy-closure: Use computeFSClosure() and LegacySSHStore 2017-02-07 20:55:47 +01:00
Eelco Dolstra 6f4682ad36
Merge branch 'nix-copy-closure-c++' of https://github.com/shlevy/nix 2017-02-07 20:47:45 +01:00
Eelco Dolstra caa5793b4a
Add a LegacySSHStore that uses nix-store --serve
This is useful for nix-copy-closure.
2017-02-07 19:29:21 +01:00
Eelco Dolstra f38224e924
copyStorePath(): Don't require signatures for "trusted" stores
For example, SSH stores could be trusted.
2017-02-07 19:29:21 +01:00
Eelco Dolstra fa07558a06
Provide default implementations for a couple of Store methods 2017-02-07 19:29:21 +01:00
Eelco Dolstra ddb5577f2e
Move SavingSourceAdapter to serialise.hh 2017-02-07 19:29:20 +01:00
Eelco Dolstra ce4d8e3ef8
Remove unnecessary call to topoSortPaths()
exportPaths() already does this.
2017-02-07 19:20:41 +01:00
Eelco Dolstra 7a58ad0ef5
SSHStore: uri -> host 2017-02-07 19:20:15 +01:00
Eelco Dolstra 612aeb2df5
Cleanup 2017-02-07 19:16:25 +01:00
Eelco Dolstra c54814b175
Remove download-via-ssh
Replaced by SSHStore.
2017-02-07 18:54:33 +01:00
Renzo Carbonara e6e74f987f Add netrc-file support 2017-02-02 13:24:20 +01:00
Eelco Dolstra 1351b0df87
exportReferencesGraph: Only export in JSON format when in structured mode
This prevents breaking compatibility with builders that read
"closure.*", since they would accidentally pick up the new JSON files.
2017-02-02 12:20:28 +01:00
Eelco Dolstra 7a65b2470e
Restore default signal handling in child processes
In particular, this fixes Ctrl-C in nix-shell sessions.
2017-02-01 13:00:21 +01:00
Eelco Dolstra cb1951e746
Periodically purge binary-cache.sqlite 2017-01-27 15:19:33 +01:00
Eelco Dolstra 211bc7f0e6
Implement TTL for binary cache lookups 2017-01-27 13:17:08 +01:00
Eelco Dolstra f57a38b109
Remove unused NARExistence table 2017-01-27 12:57:49 +01:00
Eelco Dolstra c2b0d8749f
exportReferencesGraph: Export more complete info in JSON format
This writes info about every path in the closure in the same format as
‘nix path-info --json’. Thus it also includes NAR hashes and sizes.

Example:

  [
    {
      "path": "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
      "narHash": "sha256:0ckdc4z20kkmpqdilx0wl6cricxv90lh85xpv2qljppcmz6vzcxl",
      "narSize": 197648,
      "references": [
        "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
        "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24"
      ],
      "closureSize": 20939776
    },
    {
      "path": "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24",
      "narHash": "sha256:1nfn3m3p98y1c0kd0brp80dn9n5mycwgrk183j17rajya0h7gax3",
      "narSize": 20742128,
      "references": [
        "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24"
      ],
      "closureSize": 20742128
    }
  ]

Fixes #1134.
2017-01-26 20:41:08 +01:00
Eelco Dolstra 6de33a9c67
Add support for passing structured data to builders
Previously, all derivation attributes had to be coerced into strings
so that they could be passed via the environment. This is lossy
(e.g. lists get flattened, necessitating configureFlags
vs. configureFlagsArray, of which the latter cannot be specified as an
attribute), doesn't support attribute sets at all, and has size
limitations (necessitating hacks like passAsFile).

This patch adds a new mode for passing attributes to builders, namely
encoded as a JSON file ".attrs.json" in the current directory of the
builder. This mode is activated via the special attribute

  __structuredAttrs = true;

(The idea is that one day we can set this in stdenv.mkDerivation.)

For example,

  stdenv.mkDerivation {
    __structuredAttrs = true;
    name = "foo";
    buildInputs = [ pkgs.hello pkgs.cowsay ];
    doCheck = true;
    hardening.format = false;
  }

results in a ".attrs.json" file containing (sans the indentation):

  {
    "buildInputs": [],
    "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash",
    "configureFlags": [
      "--with-foo",
      "--with-bar=1 2"
    ],
    "doCheck": true,
    "hardening": {
      "format": false
    },
    "name": "foo",
    "nativeBuildInputs": [
      "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
      "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03"
    ],
    "propagatedBuildInputs": [],
    "propagatedNativeBuildInputs": [],
    "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv",
    "system": "x86_64-linux"
  }

"passAsFile" is ignored in this mode because it's not needed - large
strings are included directly in the JSON representation.

It is up to the builder to do something with the JSON
representation. For example, in bash-based builders, lists/attrsets of
string values could be mapped to bash (associative) arrays.
2017-01-26 20:40:33 +01:00
Eelco Dolstra 54801ed6ad
Bindings: Add a method for iterating in lexicographically sorted order 2017-01-26 20:40:33 +01:00
Eelco Dolstra b1f001538e
Fix assertion failure when a path is locked
Fixes:

  nix-store: src/libstore/build.cc:3649: void nix::Worker::run(const Goals&): Assertion `!awake.empty()' failed.
2017-01-26 20:40:33 +01:00
Eelco Dolstra 83ae6503e8
Fix interrupt handling 2017-01-26 20:40:33 +01:00
Eelco Dolstra 951357e5fb
UserLock: Fix multi-threaded access to a global variable 2017-01-26 20:40:33 +01:00
Eelco Dolstra a55f589720
openLockFile: Return an AutoCloseFD 2017-01-26 20:40:33 +01:00
Eelco Dolstra c0f2f4eeef
UserLock: Make more RAII-ish 2017-01-26 20:40:33 +01:00
Eelco Dolstra a529c740d2
Moving more code out of DerivationGoal::startBuilder() 2017-01-26 20:40:33 +01:00
Eelco Dolstra e8c43abd9a
On HTTP errors, also show the curl error
This is a hopefully temporary measure to diagnose the intermittent
"HTTP error 200" failures.
2017-01-26 20:40:32 +01:00
Eelco Dolstra 4425a5c547
Move exportReferencesGraph into a separate method
startBuilder() is getting rather obese.
2017-01-26 20:40:32 +01:00
Eelco Dolstra 4e6a2fbc56
Revert "Propagate path context via builtins.readFile"
This reverts commit f7f0116dd7.

Issue #1174.
2017-01-24 15:31:54 +01:00
Eelco Dolstra fa738e50bc
Revert "builtins.readFile: Put the references of the file, not those needed to realize the file, into the context"
Reverting commit 451c223dee for now
because it breaks http://hydra.nixos.org/build/46805136, not clear
why.
2017-01-24 15:29:46 +01:00
Eelco Dolstra 4b6d3c5a28
Hopefully fix build on older GCC
http://hydra.nixos.org/build/46805140
2017-01-24 15:28:50 +01:00
Eelco Dolstra 19ce732a13
Fix typo 2017-01-24 13:57:26 +01:00
Eelco Dolstra e3bf228c92
Enable verbose curl output
Closes #1182.
2017-01-24 13:57:01 +01:00
Shea Levy 3b4a15bd48 build-remote: Use futimes instead of futimens on APPLE 2017-01-24 06:22:02 -05:00
Eelco Dolstra e5641dfe1e
Work around a bug in clang and older versions of gcc
http://hydra.nixos.org/build/46597440

https://llvm.org/bugs/show_bug.cgi?id=28096
2017-01-24 10:57:24 +01:00
Shea Levy bfa41eb671 nix-copy-closure: Implement in C++.
Tests fail currently because the database is not given proper hashes in the VM
2017-01-20 09:47:58 -05:00
Eelco Dolstra 8af062f372 Merge pull request #981 from shlevy/build-remote-c++
build-remote: Implement in C++
2017-01-19 18:21:55 +01:00
Eelco Dolstra 21948deed9
Kill builds when we get EOF on the log FD
This closes a long-time bug that allowed builds to hang Nix
indefinitely (regardless of timeouts) simply by doing

  exec > /dev/null 2>&1; while true; do true; done

Now, on EOF, we just send SIGKILL to the child to make sure it's
really gone.
2017-01-19 17:16:14 +01:00
Eelco Dolstra 63e10b4d28
Cleanup 2017-01-19 17:06:04 +01:00
Eelco Dolstra 2579e32c2b
Use std::unique_ptr for HookInstance 2017-01-19 17:06:04 +01:00
Eelco Dolstra cc3b93c991
Handle SIGINT etc. via a sigwait() signal handler thread
This allows other threads to install callbacks that run in a regular,
non-signal context. In particular, we can use this to signal the
downloader thread to quit.

Closes #1183.
2017-01-17 18:21:02 +01:00
Jude Taylor c0d55f9183 assign environ to a temp variable to ensure liveness 2017-01-16 23:52:44 +01:00
Eelco Dolstra 8079ab87a2 AutoCloseDir: Use std::unique_ptr 2017-01-16 22:39:27 +01:00
Eelco Dolstra 2b9d0a99cb AutoDeleteArray -> std::unique_ptr
Also, switch to C++14 for std::make_unique.
2017-01-16 22:24:49 +01:00
Eelco Dolstra 40dfac968a nix-build: Use showManPage() and printVersion() 2017-01-16 22:11:17 +01:00
Shea Levy 451c223dee builtins.readFile: Put the references of the file, not those needed to realize the file, into the context 2017-01-10 12:22:22 -05:00
Shea Levy 28db297862 build-remote: Don't use C++ streams to read the conf file 2017-01-10 10:36:26 -05:00
Shea Levy d771c28613 build-remote: Use std::set for feature sets 2017-01-10 10:29:06 -05:00
Eelco Dolstra 7adb986e35 Merge pull request #1139 from Mic92/master
Simplify remouting with MS_PRIVATE in sandbox build
2017-01-09 16:46:59 +01:00
Tuomas Tynkkynen 3890de049d nix-shell: Fix 'nix-shell --command' doing nothing without TTY
Regression from a5f2750e ("Fix early removal of rc-file for nix-shell").
The removal of BASH_ENV causes nothing to be executed by bash if it
detects itself in a non-interactive context. Instead, just
use the same condition used by bash to launch bash differently.

According to bash sources, the condition (stdin and stder both
must be TTYs) is specified by POSIX so this should be pretty
safe to rely on.

Fixes #1171 on master, needs a backport to the Perl code in 1.11.
2017-01-07 19:08:28 +02:00
Eelco Dolstra c287e797a8
nix-shell: In #! mode, pass the last argument
"i < argc - 1" should be "i < argc".
2017-01-03 11:40:51 +01:00
Tuomas Tynkkynen 018f884ffd nix-shell: Don't act interactive in shebangs
I had observed that 'bash --rcfile' would do nothing in a
non-interactive context and cause nothing to be executed if a script
using nix-shell shebangs were run in a non-interactive context.
2017-01-03 10:11:09 +02:00
Tuomas Tynkkynen c698ec1f22 nix-shell: Fix 'nix-shell -i'
The 'args' variable here is shadowing one in the outer scope and its
contents end up unused. This causes any '#! nix-shell' lines to
effectively be ignored. The intention here was to clear the args vector,
as far as I can tell (and it seems to work).
2017-01-03 10:11:09 +02:00
Eelco Dolstra b6b142b4b1
Provide /var/run/nscd/socket in the sandbox
Otherwise sandbox builds can fail, e.g.

  $ NIX_REMOTE=local?root=/tmp/nix nix-build '<nixpkgs>' -A hello --option build-use-substitutes false
  ...
  downloading ‘http://ftpmirror.gnu.org/bash/bash-4.3-patches/bash43-047’...
  error: unable to download ‘http://ftpmirror.gnu.org/bash/bash-4.3-patches/bash43-047’: Couldn't resolve host name (6)
2017-01-02 14:46:37 +01:00
Eelco Dolstra d0a2db17d9
Call Aws::InitAPI
This is required now.
2016-12-22 17:39:49 +01:00
Eelco Dolstra 786ee585b8
Add comment 2016-12-22 17:39:49 +01:00
Eelco Dolstra 3a4bd320c2
Revert "Merge branch 'seccomp' of https://github.com/aszlig/nix"
This reverts commit 9f3f2e21ed, reversing
changes made to 47f587700d.
2016-12-19 11:52:57 +01:00
Eelco Dolstra 05862209de
Revert "Give root a valid home directory"
This reverts commit ec7d498b72.
2016-12-19 11:49:03 +01:00
Eelco Dolstra ec7d498b72
Give root a valid home directory
Some programs barf if the current user has a non-writable home
directory, e.g. http://hydra.nixos.org/build/44818144.
2016-12-15 15:56:08 +01:00
Eelco Dolstra 5278bb7c16
Merge branch 'master' of github.com:NixOS/nix 2016-12-15 12:31:52 +01:00
Eelco Dolstra 9f3f2e21ed
Merge branch 'seccomp' of https://github.com/aszlig/nix 2016-12-15 12:04:45 +01:00
Shea Levy f867f090ed Merge branch 'base32-overflow' of git://github.com/vcunat/nix 2016-12-13 09:41:02 -05:00
Eelco Dolstra 47f587700d
Probably fix a segfault in PathLocks 2016-12-09 13:26:43 +01:00
Eelco Dolstra b30d1e7ada
Don't delete .check directories of running builds
We need to keep them around for diffoscope.
2016-12-08 21:38:58 +01:00
Eelco Dolstra 88ef77226e
Fix warning on 32-bit systems
http://hydra.nixos.org/build/44628517
2016-12-08 20:37:58 +01:00
Eelco Dolstra e629a17cc1
Fix build
http://hydra.nixos.org/build/44628517
2016-12-08 20:36:14 +01:00
Eelco Dolstra fe1162a805
S3BinaryCacheStore: Ensure it only builds on Linux 2016-12-08 15:35:46 +01:00
Eelco Dolstra e6a61b8da7
Fix S3BinaryCacheStore
It failed with

   AWS error uploading ‘6gaxphsyhg66mz0a00qghf9nqf7majs2.ls.xz’: Unable to parse ExceptionName: MissingContentLength Message: You must provide the Content-Length HTTP header.

possibly because the istringstream_nocopy introduced in
0d2ebb4373 doesn't supply the seek
method that the AWS library expects. So bring back the old version,
but only for S3BinaryCacheStore.
2016-12-08 15:31:27 +01:00
Eelco Dolstra 9a313469a4
Add a hook to run diffoscope when non-determinism is detected 2016-12-07 17:57:35 +01:00
Eelco Dolstra b07060688a
Keep track of the exact build start/stop times 2016-12-07 16:09:38 +01:00
Eelco Dolstra dadfddfa7c
Bail out early when non-determinism is detected 2016-12-07 15:31:18 +01:00
Eelco Dolstra cea4e2c618
Expose enforce-determinism and the result to Hydra 2016-12-07 13:43:40 +01:00
Eelco Dolstra 8bdf83f936
Add an option to make non-determinism non-fatal
That is, when build-repeat > 0, and the output of two rounds differ,
then print a warning rather than fail the build. This is primarily to
let Hydra check reproducibility of all packages.
2016-12-07 13:16:06 +01:00
Eelco Dolstra ceeedb58d2
Use a steady clock for timeouts
Fixes #1146.
2016-12-06 21:58:04 +01:00
Eelco Dolstra 7a3e7d0e61
nix-store --serve: Suppress log output on stderr when repeating a build 2016-12-06 17:43:39 +01:00
Eelco Dolstra 6069b946ad
nix-store --serve: Support setting build-repeat
This allows Hydra to test whether builds are reproducible.
2016-12-06 17:19:22 +01:00
Eelco Dolstra 603f08506e
Tweak error message 2016-12-06 17:18:40 +01:00
Guillaume Maudoux e4b82af387 Improve error message on trailing path slashes 2016-11-27 17:48:46 +01:00
Jörg Thalheim 855abd85d8
Simplify remouting with MS_PRIVATE in sandbox build
also fix race condition if mounts are added after mountinfo is read.
2016-11-25 00:15:39 +01:00
Eelco Dolstra 7ee43df862
nix-channel: Fix --update <CHANNELS>
This unbreaks "nixos-rebuild --upgrade".
2016-11-21 15:54:19 +01:00
Eelco Dolstra 349e988903
Typo 2016-11-17 17:55:41 +01:00
Shea Levy 00b8bce4d0 Fix binary-cache-store build 2016-11-17 11:48:10 -05:00
Shea Levy 0d2ebb4373 istringstream_nocopy: Implement in a standards-compliant way
Fixes #1135.
2016-11-17 10:10:32 -05:00
Shea Levy 3f4d3f8a1a nix-daemon: Fix splice faking on non-Linux 2016-11-17 08:10:12 -05:00
Shea Levy 8bf378e999 Update darwin build for optional sandbox paths
Fixes #1132
2016-11-17 08:06:32 -05:00
aszlig 4e1a2cd537
seccomp: Forge return values for *chown32
These syscalls are only available in 32bit architectures, but libseccomp
should handle them correctly even if we're on native architectures that
do not have these syscalls.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 17:29:36 +01:00
aszlig ed64976cec
seccomp: Forge return codes for POSIX ACL syscalls
Commands such as "cp -p" also use fsetxattr() in addition to fchown(),
so we need to make sure these syscalls always return successful as well
in order to avoid nasty "Invalid value" errors.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 17:29:14 +01:00
aszlig b90a435332
libstore/build: Forge chown() to return success
What we basically want is a seccomp mode 2 BPF program like this but for
every architecture:

  BPF_STMT(BPF_LD+BPF_W+BPF_ABS, offsetof(struct seccomp_data, nr)),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_chown, 4, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchown, 3, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchownat, 2, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_lchown, 1, 0),
  BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ALLOW),
  BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ERRNO)

However, on 32 bit architectures we do have chown32, lchown32 and
fchown32, so we'd need to add all the architecture blurb which
libseccomp handles for us.

So we only need to make sure that we add the 32bit seccomp arch while
we're on x86_64 and otherwise we just stay at the native architecture
which was set during seccomp_init(), which more or less replicates
setting 32bit personality during runChild().

The FORCE_SUCCESS() macro here could be a bit less ugly but I think
repeating the seccomp_rule_add() all over the place is way uglier.

Another way would have been to create a vector of syscalls to iterate
over, but that would make error messages uglier because we can either
only print the (libseccomp-internal) syscall number or use
seccomp_syscall_resolve_num_arch() to get the name or even make the
vector a pair number/name, essentially duplicating everything again.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:30 +01:00
aszlig 1c52e344c4
Add build dependency for libseccomp
We're going to use libseccomp instead of creating the raw BPF program,
because we have different syscall numbers on different architectures.

Although our initial seccomp rules will be quite small it really doesn't
make sense to generate the raw BPF program because we need to duplicate
it and/or make branches on every single architecture we want to suuport.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:26 +01:00
aszlig e8838713df
Run builds as root in user namespace again
This reverts commit ff0c0b645c.

We're going to use seccomp to allow "cp -p" and force chown-related
syscalls to always return 0.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:08 +01:00
Eelco Dolstra 4be4f6de56
S3BinaryCacheStore:: Eliminate a string copy while uploading
This cuts hydra-queue-runner's peak memory usage by about a third.
2016-11-16 16:21:30 +01:00
Eelco Dolstra 10ae8fabf1 buildPaths(): Handle ecIncompleteClosure
buildPaths() on a non-derivation would incorrectly not throw an error
if the path didn't have a substitute.
2016-11-14 15:00:17 +01:00
Eelco Dolstra b77fb8acb5 Don't rely on %m 2016-11-14 13:37:16 +01:00
Ludovic Courtès ccb1022022 daemon: Do not error out when deduplication fails due to ENOSPC.
This solves a problem whereby if /gnu/store/.links had enough entries,
ext4's directory index would be full, leading to link(2) returning
ENOSPC.

* nix/libstore/optimise-store.cc (LocalStore::optimisePath_): Upon
ENOSPC from link(2), print a message and return instead of throwing a
'SysError'.
2016-11-14 13:35:03 +01:00
Guillaume Maudoux a5e761dddb Fix comments parsing
Fixed the parsing of multiline strings ending with an even number of
stars, like /** this **/.
Added test cases for comments.
2016-11-13 17:20:34 +01:00
Eelco Dolstra dd77f7d593 Store::computeFSClosure(): Support a set of paths
This way, callers can exploits the parallelism of computeFSClosure()
when they have multiple paths that they need the (combined) closure of.
2016-11-10 17:45:04 +01:00
Shea Levy bff3ad767e build-remote: replace strtoull with stoull to take advantage of C++ error handling 2016-11-10 11:09:15 -05:00
Shea Levy 167d12b02c build-remote: Implement in C++ 2016-11-10 11:09:15 -05:00
Eelco Dolstra e4655b166c copyClosure() / copyStorePath(): Expose dontCheckSigs
Needed by Hydra.
2016-11-09 19:08:20 +01:00
Eelco Dolstra 4b8f1b0ec0 Merge branch 'ssh-store' of https://github.com/shlevy/nix 2016-11-09 18:57:22 +01:00
Eelco Dolstra 21c55ab3b5 Implement backwards-compatible RemoteStore::addToStore()
The SSHStore PR adds this functionality to the daemon, but we have to
handle the case where the Nix daemon is 1.11.

Also, don't require signatures for trusted users. This restores 1.11
behaviour.

Fixes https://github.com/NixOS/hydra/issues/398.
2016-11-09 18:45:06 +01:00
Eelco Dolstra a83b10f84c Typo 2016-11-08 20:19:02 +01:00
Eelco Dolstra b3ba762dbf Add missing #include 2016-11-07 14:35:47 +01:00
Vladimír Čunát 818aad3ec4
Detect and disallow base32 hash overflow
Example (before this commit):
$ nix-hash --type sha256 --to-base16 4n0igfxbd3kqvvj2k2xgysrp63l4v2gd110fwkk4apfpm0hvzwh0 \
    | xargs nix-hash --type sha256 --to-base32
0n0igfxbd3kqvvj2k2xgysrp63l4v2gd110fwkk4apfpm0hvzwh0

It's a real-life example:
https://github.com/NixOS/nixpkgs/pull/20208/files#r86695567
2016-11-06 22:17:37 +01:00
Eelco Dolstra 18b7363a69 Support optional sandbox paths
For example, you can now set

  build-sandbox-paths = /dev/nvidiactl?

to specify that /dev/nvidiactl should only be mounted in the sandbox
if it exists in the host filesystem. This is useful e.g. for EC2
images that should support both CUDA and non-CUDA instances.
2016-10-31 17:09:52 +01:00
Shea Levy c4969aebaf Add nix.conf options for -k and -K
Fixes #1084
2016-10-27 12:49:34 -04:00
Tuomas Tynkkynen f0fc3dd88b Fix SIGFPE from integer overflow during division
On some architectures (like x86_64 or i686, but not ARM for example)
overflow during integer division causes a crash due to SIGFPE.
Reproduces on a 64-bit system with:

    nix-instantiate --eval -E '(-9223372036854775807 - 1) / -1'

The only way this can happen is when the smallest possible integer is
divided by -1, so just special-case that.
2016-10-26 18:41:59 +03:00
Eelco Dolstra 5c0bd51d49 Fix build log output in nix-store --serve 2016-10-26 17:13:38 +02:00
Eelco Dolstra d2c58ba605 Restore the CachedFailure status code
The removal of CachedFailure caused the value of TimedOut to change,
which broke timed-out handling in Hydra (so timed-out builds would
show up as "aborted" and would be retried, e.g. at
http://hydra.nixos.org/build/42537427).
2016-10-26 15:02:38 +02:00
Eelco Dolstra fdbbcc4492 Remove addPathToAccessor 2016-10-21 18:09:30 +02:00
Eelco Dolstra 542ae5c8f8 BinaryCacheStore: Optionally write a NAR listing
The store parameter "write-nar-listing=1" will cause BinaryCacheStore
to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’
added to the binary cache. This file contains an XZ-compressed JSON
file describing the contents of the NAR, excluding the contents of
regular files.

E.g.

  {
    "version": 1,
    "root": {
      "type": "directory",
      "entries": {
        "lib": {
          "type": "directory",
          "entries": {
            "Mcrt1.o": {
              "type": "regular",
              "size": 1288
            },
            "Scrt1.o": {
              "type": "regular",
              "size": 3920
            },
          }
        }
      }
      ...
    }
  }

(The actual file has no indentation.)

This is intended to speed up the NixOS channels programs index
generator [1], since fetching gazillions of large NARs from
cache.nixos.org is currently a bottleneck for updating the regular
(non-small) channel.

[1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-21 16:50:28 +02:00
Eelco Dolstra 307cc8c33d Merge branch 'priorityqueue' of https://github.com/groxxda/nix 2016-10-19 16:37:04 +02:00
Eelco Dolstra 795d9b8668 Merge branch 'regex' of https://github.com/groxxda/nix 2016-10-19 16:32:24 +02:00
Alexander Ried c935e8eeaf fixup! replace own regex class with std::regex 2016-10-19 16:03:53 +02:00
Eelco Dolstra 19c278de89 Fix Darwin build
Done slightly differently from https://github.com/NixOS/nix/pull/1093.
2016-10-19 15:21:18 +02:00
Eelco Dolstra 29727ff944 Shut up clang warning 2016-10-19 15:02:48 +02:00
Eelco Dolstra e5346c54df Fix uninitialised variable 2016-10-19 15:02:38 +02:00
Alexander Ried b05b98df75 replace own regex class with std::regex 2016-10-18 20:22:25 +02:00
Alexander Ried efb938468c downloader: use priority_queue 2016-10-18 15:45:51 +02:00
Eelco Dolstra ae8884b949 Fix /dev/ptmx in sandboxes
This was broken since ff0c0b645c. Since
I can't figure out how to mount a devpts instance in the sandbox,
let's just bind-mount the host devpts.
2016-10-14 15:42:13 +02:00
Eelco Dolstra 2f9d60ed7f CURL_AT_LEAST_VERSION -> LIBCURL_VERSION_NUM
http://hydra.nixos.org/build/42025230
2016-10-14 14:44:15 +02:00
Eelco Dolstra 5e61b422c5 Fix HTTP/2 support
Commit 86e8c67efc broke it, because
CURL_* are not actually #defines.
2016-10-13 17:25:42 +02:00
Eelco Dolstra fb2dd32100 SSL_CERT_FILE -> NIX_SSL_CERT_FILE
This prevents collisions with the "native" OpenSSL, in particular on
OS X.

Fixes #921.
2016-10-13 17:09:10 +02:00
Eelco Dolstra 816d3e5724 Shut up some warnings 2016-10-12 15:53:38 +02:00
Eelco Dolstra 82e2a070e0 Add some functions needed by hydra 2016-10-12 15:49:37 +02:00
Eelco Dolstra edf9eb8181 querySubstitutablePaths(): Don't query paths for which we already have a substituter 2016-10-07 19:57:47 +02:00
Eelco Dolstra 0c85ef7090 LocalStore::querySubstitutablePaths(): Implement using queryValidPaths() 2016-10-07 19:46:56 +02:00
Eelco Dolstra 844219f364 Store::queryValidPaths(): Use async queryPathInfo()
This allows the binary cache substituter to pipeline requests.
2016-10-07 19:43:36 +02:00
Eelco Dolstra c663b84573 Implement generic Store::queryValidPaths() 2016-10-07 19:20:47 +02:00
Eelco Dolstra 35db4f65a0 Add copyClosure utility function for Hydra 2016-10-07 19:15:25 +02:00
Eelco Dolstra 629ab80022 importPaths(): Fix accessor support for Hydra 2016-10-07 18:13:40 +02:00
Eelco Dolstra b0f7f9c98f toJSON(): Support some more types 2016-10-06 17:00:52 +02:00
Eelco Dolstra ae522f930d Fix getS3Stats() 2016-10-06 17:00:52 +02:00
Eelco Dolstra cd128f4bad Merge pull request #998 from veprbl/rx_chmod_fix
override rx directory permissions in deletePath()
2016-10-06 11:08:21 +02:00
Eelco Dolstra c6a929986a Use std::random_device 2016-10-05 21:29:18 +02:00
Eelco Dolstra 86e8c67efc Fix build with older versions of libcurl 2016-10-04 14:43:23 +02:00
Eelco Dolstra 4bd51d74af Handle the case where signed-binary-caches consists of whitespace 2016-09-22 15:59:25 +02:00
Eelco Dolstra 818ab58cc6 Add sandbox-dev-shm-size option
Fixes #1069.
2016-09-21 16:54:53 +02:00
Eelco Dolstra 4546be1b3e nix-build, nix-shell: Don't print error message if nix-store/nix-instantiate fails 2016-09-21 16:54:53 +02:00
Eelco Dolstra c55bf085eb printMsg(lvlError, ...) -> printError(...) etc. 2016-09-21 16:54:53 +02:00
Eelco Dolstra 4036185cb4 Some notational convenience for formatting strings
We can now write

  throw Error("file '%s' not found", path);

instead of

  throw Error(format("file '%s' not found") % path);

and similarly

  printError("file '%s' not found", path);

instead of

  printMsg(lvlError, format("file '%s' not found") % path);
2016-09-21 16:54:53 +02:00
Eelco Dolstra beaefdf706 Tweak 2016-09-20 17:49:31 +02:00
Eelco Dolstra f12d56b27b Improve robustness 2016-09-20 17:25:44 +02:00
Eelco Dolstra 794e4ab475 Fix "Promise already satisfied" error 2016-09-20 17:25:12 +02:00
Eelco Dolstra 4de0639105 nix-shell: Fix $PATH handling in the impure case
We were passing "p=$PATH" rather than "p=$PATH;", resulting in some
invalid shell code.

Also, construct a separate environment for the child rather than
overwriting the parent's.
2016-09-20 15:41:41 +02:00
Eelco Dolstra 9fc4cb2ae9 nix-shell: Restore CPU affinity
Otherwise the shell and its children will be bound to one CPU core...
2016-09-20 15:41:41 +02:00
Eric Litak 0d38b4c792 armv5tel can be built by armv6l and armv7l (#1063) 2016-09-16 22:03:34 +02:00
Eelco Dolstra 75989bdca7 Make computeFSClosure() single-threaded again
The fact that queryPathInfo() is synchronous meant that we needed a
thread for every concurrent binary cache lookup, even though they end
up being handled by the same download thread. Requiring hundreds of
threads is not a good idea. So now there is an asynchronous version of
queryPathInfo() that takes a callback function to process the
result. Similarly, enqueueDownload() now takes a callback rather than
returning a future.

Thus, a command like

  nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5

that returns 4941 paths now takes 1.87s using only 2 threads (the main
thread and the downloader thread). (This is with a prewarmed
CloudFront.)
2016-09-16 18:54:14 +02:00
Eelco Dolstra 054be50257 printMsg(): Don't check for interrupts
Having the logger function potentially throw exceptions is
Heisenbuggy.
2016-09-16 18:52:42 +02:00
Eelco Dolstra 2e1493037b nix path-info: Add some more examples 2016-09-14 18:20:11 +02:00
Eelco Dolstra 6656ef7b5b Revive binary-caches-parallel-connections
It's a slight misnomer now because it actually limits *all* downloads,
not just binary cache lookups.

Also add a "enable-http2" option to allow disabling use of HTTP/2
(enabled by default).
2016-09-14 16:38:26 +02:00
Eelco Dolstra 90ad02bf62 Enable HTTP/2 support
The binary cache store can now use HTTP/2 to do lookups. This is much
more efficient than HTTP/1.1 due to multiplexing: we can issue many
requests in parallel over a single TCP connection. Thus it's no longer
necessary to use a bunch of concurrent TCP connections (25 by
default).

For example, downloading 802 .narinfo files from
https://cache.nixos.org/, using a single TCP connection, takes 11.8s
with HTTP/1.1, but only 0.61s with HTTP/2.

This did require a fairly substantial rewrite of the Downloader class
to use the curl multi interface, because otherwise curl wouldn't be
able to do multiplexing for us. As a bonus, we get connection reuse
even with HTTP/1.1. All downloads are now handled by a single worker
thread. Clients call Downloader::enqueueDownload() to tell the worker
thread to start the download, getting a std::future to the result.
2016-09-14 16:36:02 +02:00
Eelco Dolstra a75d11a7e6 Add a toLower utility function 2016-09-14 14:58:49 +02:00
Eelco Dolstra e07c0dcf5c Move some .drv parsing functions out of util 2016-09-14 10:54:57 +02:00
Shea Levy b99c6e0e29 nix-daemon: Fix error message 2016-09-12 08:09:41 -04:00
Shea Levy 196815f700 ssh-store: Start master on-demand 2016-09-12 08:07:50 -04:00
Shea Levy ab31f9986c Inline ssh-store.hh into ssh-store.cc 2016-09-12 08:03:29 -04:00
Eelco Dolstra e5949b5ce8 Fix build on GCC 4.9
GCC 4.9 doesn't like reassigning a std::stringstream.

http://hydra.nixos.org/build/40371644
2016-09-12 13:22:23 +02:00
Eelco Dolstra 5039d3b9de Write "T x" instead of "auto x = T"
That's just silly. Hopefully this also fixes the Debian build failure:

http://hydra.nixos.org/build/40371644
2016-09-12 12:06:13 +02:00
Eelco Dolstra ff0c0b645c Don't run builds as root in the user namespace
This largely reverts c68e5913c7. Running
builds as root breaks "cp -p", since when running as root, "cp -p"
assumes that it can succesfully chown() files. But that's not actually
the case since the user namespace doesn't provide a complete uid
mapping. So it barfs with a fatal error message ("cp: failed to
preserve ownership for 'foo': Invalid argument").
2016-09-08 18:16:23 +02:00
Eelco Dolstra e4bdd49022 Give a better error message if a path in build-sandbox-paths is invalid 2016-09-08 17:30:25 +02:00
Eelco Dolstra 0babc4ff31 Fix assertion failure after early build setup failure 2016-09-08 17:29:50 +02:00
Shea Levy a5f2750ebb Fix early removal of rc-file for nix-shell
BASH_ENV causes all non-interactive shells called via eg. /etc/bashrc to
remove the rc-file before the main shell gets to run it. Completion
scripts will often do this. Fixes #976.

Adapted from and fixes #1034.
2016-09-08 09:09:52 -04:00
Shea Levy 584f8a62de Implement nar-based addToStore for remote-store 2016-09-02 14:33:58 -04:00
Shea Levy ecba88de93 Add ssh store implementation 2016-09-02 14:31:38 -04:00
Shea Levy b4b5e9ce2f Add narFromPath op to nix daemon 2016-09-02 14:26:02 -04:00
Shea Levy a705e8ce0a Factor a general remote FS accessor out of BinaryCacheStore 2016-09-02 14:24:34 -04:00
Shea Levy 0f39633290 Factor out the unix domain socket-specific code from RemoteStore 2016-09-02 14:15:04 -04:00
Shea Levy 7d4ccd9b17 nix-daemon: add --stdio flag for handling connections over stdin/stdout 2016-09-02 13:30:28 -04:00
Shea Levy 53b27ddce2 Factor a function to get the store type from a URI out of the main RegisterStoreImplementation 2016-09-02 06:39:29 -04:00
Shea Levy a91954f0c6 Merge openStore and openStoreAt with default arguments 2016-09-02 06:35:48 -04:00
Shea Levy 87b189c2b3 Merge branch 'nix-build-c++' 2016-08-31 12:10:21 -04:00
Shea Levy 821380c77b nix-build: Clean up a bit 2016-08-31 10:08:00 -04:00
Shea Levy dfe0938614 download.hh: Fix conflicts from nix-channel-c++ merge 2016-08-31 09:57:56 -04:00
Shea Levy 572aba284a Merge branch 'nix-channel-c++' 2016-08-31 09:49:24 -04:00
Eelco Dolstra aa1ea0d1e4 Merge pull request #1030 from pikajude/rsd-cc
Implement resolve-system-dependencies in C++
2016-08-31 15:19:10 +02:00
Eelco Dolstra d74c8a3f4e Fix 32-bit build 2016-08-30 17:38:09 +02:00
Eelco Dolstra 6631a6e1a1 Increase the sleep time between download retries 2016-08-30 15:48:24 +02:00
Eelco Dolstra 97b1af1cbe Fix --fallback
This fixes an assertion failure in "assert(goal);" in
Worker::waitForInput() after a substitution goal is cancelled by the
termination of another goal. The problem was the line

  //worker.childTerminated(shared_from_this()); // FIXME

in the SubstitutionGoal destructor. This was disabled because
shared_from_this() obviously doesn't work from a destructor. So we now
use a real pointer for object identity.
2016-08-30 15:45:39 +02:00
Eelco Dolstra 67364a5795 Fix Fedora build 2016-08-30 13:56:22 +02:00
Eelco Dolstra 92d917b71a Fix GC build 2016-08-30 13:12:12 +02:00
Eelco Dolstra c42d1acfeb forceBool(): Show position info 2016-08-29 19:37:19 +02:00
Eelco Dolstra 26d92017d3 Add builtin function "partition"
The implementation of "partition" in Nixpkgs is O(n^2) (because of the
use of ++), and for some reason was causing stack overflows in
multi-threaded evaluation (not sure why).

This reduces "nix-env -qa --drv-path" runtime by 0.197s and memory
usage by 298 MiB (in non-Boehm mode).
2016-08-29 19:36:54 +02:00
Eelco Dolstra c0a7b84748 nix path-info: Add --json flag
Also, factor out JSON generation from value-to-json.{cc,hh}, and
support producing indented JSON.
2016-08-29 17:29:24 +02:00
Shea Levy 9fa21765e7 callFunction: Copy functors to the heap
Normally it's impossible to take a reference to the function passed to
callFunction, so some callers (e.g. ExprApp::eval) allocate that value
on the stack. For functors, a reference to the functor itself may be
kept, so we need to have it on the heap.

Fixes #1045
2016-08-29 07:36:28 -04:00
Eelco Dolstra a9d4f2b303 Fix queryPathFromHashPart()
The inner lambda was returning a SQLite-internal char * rather than a
std::string, leading to Hydra errors liks

  Caught exception in Hydra::Controller::Root->narinfo "path ‘ø˜£â€™ is not in the Nix store at /nix/store/6mvvyb8fgwj23miyal5mdr8ik4ixk15w-hydra-0.1.1234.abcdef/libexec/hydra/lib/Hydra/Controller/Root.pm line 352."
2016-08-24 14:53:33 +02:00
Eelco Dolstra d74236d1f2 nix build: Use Nix search path
That is, unless --file is specified, the Nix search path is
synthesized into an attribute set. Thus you can say

  $ nix build nixpkgs.hello

assuming $NIX_PATH contains an entry of the form "nixpkgs=...". This
is more verbose than

  $ nix build hello

but is less ambiguous.
2016-08-23 17:11:19 +02:00
Jude Taylor 50c3b5df32 iterate through fat_headers correctly 2016-08-17 10:24:11 -07:00
Eelco Dolstra 22d6e31fc6 Add a mechanism for derivation attributes to reference the derivation's outputs
For example, you can now say:

  configureFlags = "--prefix=${placeholder "out"} --includedir=${placeholder "dev"}";

The strings returned by the ‘placeholder’ builtin are replaced at
build time by the actual store paths corresponding to the specified
outputs.

Previously, you had to work around the inability to self-reference by doing stuff like:

  preConfigure = ''
    configureFlags+=" --prefix $out --includedir=$dev"
  '';

or rely on ad-hoc variable interpolation semantics in Autoconf or Make
(e.g. --prefix=\$(out)), which doesn't always work.
2016-08-17 17:19:32 +02:00
Jude Taylor ca0bce2851 remove double casting 2016-08-16 14:09:57 -07:00
Jude Taylor cfb77d6e5b account for unknown magic numbers 2016-08-16 12:52:49 -07:00
Jude Taylor 90516c5a7b switch to mmap strategy 2016-08-16 12:45:55 -07:00
Eelco Dolstra ac841a4679 Merge pull request #1031 from abbradar/replacestrings-context
Allow contexted strings in replaceStrings
2016-08-16 12:47:20 +02:00
Nikolay Amiantov f1b8dd43be Allow contexted strings in replaceStrings 2016-08-16 13:20:56 +03:00
Jude Taylor 7b006122ae ensure presence of directories 2016-08-15 10:46:21 -07:00
Jude Taylor adf0216d98 code review comments 2016-08-15 10:43:14 -07:00
Shea Levy d52d391164 builtins.fetch{url,tarball}: Allow name attribute 2016-08-15 07:37:11 -04:00
Jude Taylor 60f4b25d7d make inclusion conditional 2016-08-14 19:10:38 -07:00
Jude Taylor f37b6fd07e add a missing load command 2016-08-14 18:54:40 -07:00
Jude Taylor 596e4a5693 remove old traces of resolve-system-dependencies 2016-08-13 15:27:49 -07:00
Jude Taylor 2df9a972fc resolve-system-dependencies: implement in C++ 2016-08-13 11:36:22 -07:00
Shea Levy 59124228b3 nix-channel: implement in c++ 2016-08-11 11:34:43 -04:00
Vladimír Čunát 1bffd83e1a nix-build: extend the meaning of $IN_NIX_SHELL
An equivalent was originally filed against the perl version:
https://github.com/NixOS/nix/pull/933
2016-08-11 17:14:09 +02:00
Eelco Dolstra b0ebad88cd Merge branch 'buildenv-c++' of https://github.com/shlevy/nix 2016-08-11 17:11:17 +02:00
Shea Levy 91978e3b9a buildenv: perl -> c++ 2016-08-11 07:58:33 -04:00
Eelco Dolstra 66adbdfd97 HttpBinaryCacheStore: Retry on transient HTTP errors
This makes us more robust against 500 errors from CloudFront or S3
(assuming the 500 error isn't cached by CloudFront...).
2016-08-10 18:08:23 +02:00
Eelco Dolstra 9204ea7294 Show HTTP error codes 2016-08-10 18:05:35 +02:00
Eelco Dolstra f294623d1d SQLite:: Add some convenience 2016-08-10 18:05:35 +02:00
Eelco Dolstra 6cb4bdf152 FSAccessor: Throw InvalidPath 2016-08-10 18:05:35 +02:00
Eelco Dolstra f43823f676 NarAccessor: Fix handling of non-executable files 2016-08-10 18:05:35 +02:00
Eelco Dolstra d961c29c9c Mark content-addressed paths in the Nix database and in .narinfo
This allows such paths to be imported without signatures.
2016-08-10 18:05:35 +02:00
Eelco Dolstra 36a51ecab3 LocalFSStore::narFromPath(): Handle the diverted store case 2016-08-10 18:05:35 +02:00
Eelco Dolstra 3eb6217508 Add a "root" parameter to local stores
This makes it easier to create a diverted store, i.e.

  NIX_REMOTE="local?root=/tmp/root"

instead of

  NIX_REMOTE="local?real=/tmp/root/nix/store&state=/tmp/root/nix/var/nix" NIX_LOG_DIR=/tmp/root/nix/var/log
2016-08-10 18:05:35 +02:00
Eelco Dolstra 2fad86f361 Remove $NIX_DB_DIR
This variable has no reason to exist, given $NIX_STATE_DIR.
2016-08-10 18:05:35 +02:00
Eelco Dolstra be64fbb501 RemoteStore / nix-daemon: Drop support for Nix < 1.0 2016-08-10 18:05:35 +02:00
Shea Levy 80ebc553ec nix-build: Port to c++
This was a dumb line-for-line rewrite, because nix build/nix run/etc.
will replace it.
2016-08-09 07:42:20 -04:00
Eelco Dolstra ee22a91ab8 makeFixedOutputPath(): Drop superfluous HashType argument 2016-07-26 21:25:52 +02:00
Eelco Dolstra 06bbfb6004 builtins.{fetchurl,fetchTarball}: Support a sha256 attribute
Also, allow builtins.{fetchurl,fetchTarball} in restricted mode if a
hash is specified.
2016-07-26 21:16:52 +02:00
Dmitry Kalinkin f91748ba73 override rx directory permissions in deletePath()
This fixes instantiation of pythonPackages.pytest that produces a
directory with less permissions during one of it's tests that leads to
a nix error like:

error: opening directory ‘/tmp/nix-build-python2.7-pytest-2.9.2.drv-0/pytest-of-user/pytest-0/testdir/test_cache_failure_warns0/.cache’: Permission denied
2016-07-25 18:11:46 -04:00
Shea Levy ee3032e4de Merge branch 'find-runtime-roots-c++' 2016-07-24 07:33:49 -04:00
Shea Levy 18b0808475 Respect --keep-going when a substituter fails.
Fixes #977
2016-07-23 09:16:08 -04:00
Eelco Dolstra d3cd0f5856 Fix assertion failure 2016-07-21 18:39:32 +02:00
Eelco Dolstra e19c90fc6b Fix random failures caused by CurlDownloader modifying a shared string 2016-07-21 18:35:31 +02:00
Eelco Dolstra ea8e8df6c7 NarInfo::NarInfo(): Ensure that we get a NAR size/hash 2016-07-21 18:35:12 +02:00
Eelco Dolstra e682a8e138 Fix assertion failure in ThreadPool::enqueue() 2016-07-21 18:14:16 +02:00
Eelco Dolstra d57981bac4 Store::queryMissing(): Use a thread pool
For one particular NixOS configuration, this cut the runtime of
"nix-store -r --dry-run" from 6m51s to 3.4s. It also fixes a bug in
the size calculation that was causing certain paths to be counted
twice, e.g. before:

  these paths will be fetched (1249.98 MiB download, 2995.74 MiB unpacked):

and after:

  these paths will be fetched (1219.56 MiB download, 2862.17 MiB unpacked):
2016-07-21 17:50:39 +02:00
Eelco Dolstra a728780fbd Store::computeFSClosure(): Use thread pool
This speeds up queries against the binary cache.
2016-07-21 14:13:35 +02:00
Shea Levy 3c46fe62b8 find-runtime-roots: fold into gc.cc 2016-07-21 07:04:41 -04:00
Shea Levy 06068b353d FdSink: Restore move constructor/assignment 2016-07-13 06:27:41 -04:00
Shea Levy b33e85229d Make Buffered{Source,Sink} move-safe 2016-07-13 06:03:37 -04:00
Shea Levy cb5e7254b6 Modernize AutoCloseFD 2016-07-11 15:44:44 -04:00
Eelco Dolstra 74dd603495 Re-implement negative binary cache lookup caching 2016-06-20 17:39:05 +02:00
Eelco Dolstra c68e5913c7 Run builds in a user namespace
This way, all builds appear to have a uid/gid of 0 inside the
chroot. In the future, this may allow using programs like
systemd-nspawn inside builds, but that will require assigning a larger
UID/GID map to the build.

Issue #625.
2016-06-09 18:30:39 +02:00
Eelco Dolstra 202683a4fc Use O_CLOEXEC in most places 2016-06-09 16:37:08 +02:00
Eelco Dolstra 9bdd949cfd Fix "creating statement: table ValidPaths has no column named ultimate" 2016-06-03 16:06:42 +02:00
Eelco Dolstra 5e51ffb1c2 Support sandbox builds by non-root users
This allows an unprivileged user to perform builds on a diverted store
(i.e. where the physical store location differs from the logical
location).

Example:

  $ NIX_LOG_DIR=/tmp/log NIX_REMOTE="local?real=/tmp/store&state=/tmp/var" nix-build -E \
    'with import <nixpkgs> {}; runCommand "foo" { buildInputs = [procps nettools]; } "id; ps; ifconfig; echo $out > $out"'

will do a build in the Nix store physically in /tmp/store but
logically in /nix/store (and thus using substituters for the latter).
2016-06-03 15:59:28 +02:00
Eelco Dolstra 2f8b0e557b Consistent quotes 2016-06-03 14:03:09 +02:00
Eelco Dolstra 080d607975 Don't call shared_from_this() from a destructor 2016-06-03 14:02:44 +02:00
Eelco Dolstra 30c8c3ba2d Fix some more references to storeDir 2016-06-03 14:02:44 +02:00
Eelco Dolstra a8dfdc52b8 Merge pull request #925 from abbradar/master
Show both cycle ends
2016-06-03 13:58:51 +02:00
Nikolay Amiantov c87a56f4d0 Show both cycle ends 2016-06-03 14:25:37 +03:00
Eelco Dolstra 75361b6dce Fix narrowing conversion from int64_t to bool
http://hydra.nixos.org/build/36613774
2016-06-03 13:20:46 +02:00
Eelco Dolstra f8a8b4d8f8 nix run: Set a reasonable uid/gid 2016-06-02 19:04:09 +02:00
Eelco Dolstra d64e0c1b64 Make derivationFromPath work on diverted stores 2016-06-02 18:43:36 +02:00
Eelco Dolstra eda2aaae92 nix run: Handle the case where the /nix/store mount point doesn't exist 2016-06-02 18:24:51 +02:00
Eelco Dolstra a424ab0444 createTempDir(): Don't do a chown on Linux
It's not needed and can cause problems in a user namespace.
2016-06-02 18:17:30 +02:00
Eelco Dolstra a24f2c9b84 nix run: Mount the Nix store in a private namespace
This is a convenience command to allow users who are not privileged to
create /nix/store to use Nix with regular binary caches. For example,

  $ NIX_REMOTE="local?state=$HOME/nix/var&real=/$HOME/nix/store" nix run firefox bashInteractive

will download Firefox and bash from cache.nixos.org, then start a
shell in which $HOME/nix/store is mounted on /nix/store.
2016-06-02 16:51:43 +02:00
Eelco Dolstra 6f2d51287c Add basic "nix run" command 2016-06-02 16:29:49 +02:00
Eelco Dolstra 4494000e04 LocalStore: Allow the physical and logical store directories to differ
This is primarily to subsume the functionality of the
copy-from-other-stores substituter. For example, in the NixOS
installer, we can now do (assuming we're in the target chroot, and the
Nix store of the installation CD is bind-mounted on /tmp/nix):

  $ nix-build ... --option substituters 'local?state=/tmp/nix/var&real=/tmp/nix/store'

However, unlike copy-from-other-stores, this also allows write access
to such a store. One application might be fetching substitutes for
/nix/store in a situation where the user doesn't have sufficient
privileges to create /nix, e.g.:

  $ NIX_REMOTE="local?state=/home/alice/nix/var&real=/home/alice/nix/store" nix-build ...
2016-06-02 16:02:48 +02:00
Eelco Dolstra 064816ab98 Respect build-use-substitutes 2016-06-02 16:02:48 +02:00
Eelco Dolstra 812c0dfbe2 Allow setting the state directory as a store parameter
E.g. "local?store=/tmp/store&state=/tmp/var".
2016-06-02 16:02:48 +02:00
Eelco Dolstra f2682e6e18 <nix/fetchurl.nix>: Use tarballs.nixos.org
This ensures that the disappearance of upstream bootstrap tarballs
(e.g. https://github.com/NixOS/nixpkgs-channels/pull/1) doesn't break
stdenv rebuilds.
2016-06-01 17:11:51 +02:00
Eelco Dolstra dead8189ac Typo 2016-06-01 16:56:22 +02:00
Eelco Dolstra e24e2caaaf Skip substituters with an incompatible store directory 2016-06-01 16:40:49 +02:00
Eelco Dolstra cf198952d0 HttpBinaryCacheStore: Fix caching of WantMassQuery
Also, test HttpBinaryCacheStore in addition to LocalBinaryCacheStore.
2016-06-01 16:24:17 +02:00
Eelco Dolstra 7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra a9fa5e050a Shut up some clang warnings 2016-05-31 13:31:04 +02:00
Eelco Dolstra 10f3a2e5f2 Fix clang build failure
Apparently opinion is divided on whether [[noreturn]] is allowed on a
lambda: http://stackoverflow.com/questions/26888805/how-to-declare-a-lambdas-operator-as-noreturn

http://hydra.nixos.org/build/36462100
2016-05-31 13:23:54 +02:00
Eelco Dolstra c2d27d30cf nix-copy-closure / build-remote.pl: Disable signature checking
This restores the Nix 1.11 behaviour.
2016-05-31 11:48:05 +02:00
Dmitry Kalinkin 78b00bbd8a use $(LIBLZMA_LIBS) instead of -llzma
This is needed in case of non-standard lzma installation path that will
be specified in pkgconfig manifest as extra -L option for LDFLAGS.
2016-05-31 03:20:11 -04:00
Eelco Dolstra cd35f0280c Fix build 2016-05-30 16:12:50 +02:00
Eelco Dolstra 57d33013ce Check signatures before downloading the substitute 2016-05-30 15:18:12 +02:00
Eelco Dolstra bac123ddd9 Test the NAR info cache 2016-05-30 15:18:12 +02:00
Eelco Dolstra 3593c8285d Re-implement binary cache signature checking
This is now done in LocalStore::addToStore(), rather than in the
binary cache substituter (which no longer exists).
2016-05-30 15:18:12 +02:00
Eelco Dolstra 12ddbad458 LocalStore::addToStore: Verify hash of the imported path 2016-05-30 15:18:12 +02:00
Eelco Dolstra e222484401 Re-implement the WantMassQuery property of binary caches 2016-05-30 15:18:12 +02:00
Eelco Dolstra b66ab6cdbc Fix repair during substitution 2016-05-30 15:18:12 +02:00
Eelco Dolstra 3be2e71ab3 BinaryCacheStore: Remove buildPaths() / ensurePath() 2016-05-30 15:18:12 +02:00
Eelco Dolstra 75d2492f20 Make the aws-cpp-sdk dependency optional 2016-05-04 17:16:48 +02:00
Eelco Dolstra 6c75cf69c3 Cleanup: Remove singleton() 2016-05-04 16:16:53 +02:00
Eelco Dolstra 0d4a10e910 Do compression in a sink 2016-05-04 16:16:53 +02:00
Eelco Dolstra c6a21aed07 More GC fixes 2016-05-04 16:16:53 +02:00
Eelco Dolstra 538a64e8c3 Add a Store::addToStore() variant that accepts a NAR
As a side effect, this ensures that signatures are propagated when
copying paths between stores.

Also refactored import/export to make use of this.
2016-05-04 16:15:54 +02:00
Eelco Dolstra b6c768fb6a nix-daemon: Fix queryPathInfo() backward compatibility 2016-05-04 12:53:24 +02:00
Eelco Dolstra 33de2bc080 Fix segfault in xz compression/decompression 2016-05-04 11:50:12 +02:00
Eelco Dolstra f435f82475 Remove OpenSSL-based signing 2016-05-04 11:01:48 +02:00
Eelco Dolstra dfebfc835f Add a copyStorePath() utility function 2016-05-04 11:01:48 +02:00
Eelco Dolstra d8bf0d4859 Support Git repos in the Nix path
E.g.

  $ nix-build -I nixpkgs=git://github.com/NixOS/nixpkgs '<nixpkgs>' -A hello

This is not extremely useful yet because you can't specify a
branch/revision.
2016-04-29 21:07:47 +02:00
Eelco Dolstra 38539b943a Add fetchgit builtin
The function builtins.fetchgit fetches Git repositories at evaluation
time, similar to builtins.fetchTarball. (Perhaps the name should be
changed, being confusing with respect to Nixpkgs's fetchgit function,
with works at build time.)

Example:

  (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello

or

  (import (builtins.fetchgit {
    url = git://github.com/NixOS/nixpkgs-channels;
    rev = "nixos-16.03";
  }) {}).hello

Note that the result does not contain a .git directory.
2016-04-29 20:47:36 +02:00
Eelco Dolstra 83258225e6 nix-store -l: Simplify implementation 2016-04-29 17:47:03 +02:00
Eelco Dolstra 4dde0b0562 BinaryCacheStore: Support bzip2 compression 2016-04-29 17:43:37 +02:00
Eelco Dolstra 6963de2091 nix verify --all: Support local binary caches 2016-04-29 17:34:31 +02:00
Eelco Dolstra 0dd988d2e3 Better error message 2016-04-29 17:33:22 +02:00
Eelco Dolstra 5acb691402 BinaryCacheStore: Support "none" compression method 2016-04-29 17:02:57 +02:00
Eelco Dolstra 8e065c6b3e BinaryCacheStore: Make the signing key a parameter 2016-04-29 16:47:20 +02:00
Eelco Dolstra f6aee2f477 BinaryCacheStore::queryPathInfo: Don't check signatures
Other stores don't do this either. It's up to the caller to check
signatures.
2016-04-29 16:28:57 +02:00
Eelco Dolstra 95d20dfde9 Allow parameters in store URIs
This is to allow store-specific configuration,
e.g. s3://my-cache?compression=bzip2&secret-key=/path/to/key.
2016-04-29 16:26:16 +02:00
Eelco Dolstra aa3bc3d5dc Eliminate the substituter mechanism
Substitution is now simply a Store -> Store copy operation, most
typically from BinaryCacheStore to LocalStore.
2016-04-29 13:57:08 +02:00
Eelco Dolstra 21e9d183cc Really handle carriage return 2016-04-28 14:27:00 +02:00
Eelco Dolstra ce5776758d Fix error message 2016-04-28 14:12:10 +02:00
Eelco Dolstra cc804d0dc6 Handle carriage return 2016-04-25 19:18:45 +02:00
Eelco Dolstra ca9de88a51 nix: Disable verbose builds 2016-04-25 19:18:45 +02:00
Eelco Dolstra 5761827d5b Show the log tail when a build fails
If --no-build-output is given (which will become the default for the
"nix" command at least), show the last 10 lines of the build output if
the build fails.
2016-04-25 19:18:45 +02:00
Eelco Dolstra 6e1b099279 Remove --print-build-trace
This was added to support Hydra, but Hydra no longer uses it.
2016-04-25 19:18:45 +02:00
Eelco Dolstra 9eba2c3945 Fix "path is not in the Nix store" during GC 2016-04-25 19:18:45 +02:00
Eelco Dolstra 697e0b3a6f Remove nix-log2xml 2016-04-25 19:18:45 +02:00
Eelco Dolstra 41633f9f73 Improved logging abstraction
This also gets rid of --log-type, since the nested log type isn't
useful in a multi-threaded situation, and nobody cares about the
"pretty" log type.
2016-04-25 19:18:45 +02:00
Eelco Dolstra c879a20850 Factor out parallel processing of work items that have dependencies 2016-04-22 20:50:06 +02:00
Eelco Dolstra 91539d305f nix copy: Parallelise 2016-04-22 18:19:48 +02:00
Eelco Dolstra b2ce6fde5a ThreadPool: Start doing work as soon as work items are enqueued 2016-04-22 18:19:17 +02:00
Eelco Dolstra 58c84cda3b Make compression interruptible 2016-04-22 18:15:02 +02:00
Eelco Dolstra 456179018a Fold "nix query-path-sigs" into "nix path-info" 2016-04-22 14:39:37 +02:00
Eelco Dolstra 95abf9c402 Add "nix copy" command
This replaces nix-push. For example,

  $ nix copy --to file:///tmp/cache -r $(type -p firefox)

copies the closure of firefox to the specified binary cache. And

  $ nix copy --from file:///tmp/cache --to s3://my-cache /nix/store/abcd...

copies between two binary caches.

It will also replace nix-copy-closure, once we have an SSHStore class,
e.g.

  $ nix copy --from ssh://alice@machine /nix/store/abcd...
2016-04-22 14:33:01 +02:00
Eelco Dolstra 0207272b28 BinaryCacheStore: When adding a path, ensure the references are valid
This prevents copying a partial closure to a binary cache.
2016-04-22 12:15:06 +02:00
Eelco Dolstra 21ef1670b3 Fix test failures 2016-04-21 18:21:25 +02:00
Eelco Dolstra 7d14f5c331 Implement S3BinaryCacheStore::queryAllValidPaths()
This allows commands like "nix verify --all" or "nix path-info --all"
to work on S3 caches.

Unfortunately, this requires some ugly hackery: when querying the
contents of the bucket, we don't want to have to read every .narinfo
file. But the S3 bucket keys only include the hash part of each store
path, not the name part. So as a special exception
queryAllValidPaths() can now return store paths *without* the name
part, and queryPathInfo() accepts such store paths (returning a
ValidPathInfo object containing the full name).
2016-04-21 17:53:47 +02:00
Eelco Dolstra d155d80155 Move S3BinaryCacheStore from Hydra
This allows running arbitrary Nix commands against an S3 binary cache.

To do: make this a compile time option to prevent a dependency on
aws-sdk-cpp.
2016-04-21 16:08:51 +02:00
Eelco Dolstra 1a71495273 nix path-info: Add
Forgot to commit this earlier...
2016-04-21 15:00:45 +02:00
Eelco Dolstra 69e3ffb076 nix --help: Show usage examples 2016-04-21 14:58:32 +02:00
Eelco Dolstra 1b0088ebb2 nix --help: Show short flags 2016-04-21 14:34:46 +02:00
Eelco Dolstra ddea253ff8 RemoteStore: Propagate InvalidPath exceptions from the daemon 2016-04-20 15:28:07 +02:00
Eelco Dolstra c0c4ddcd9c BinaryCacheStore: Insert new paths into the disk cache 2016-04-20 15:27:48 +02:00
Eelco Dolstra 451ebf24ce Cache path info lookups in SQLite
This re-implements the binary cache database in C++, allowing it to be
used by other Store backends, in particular the S3 backend.
2016-04-20 14:12:38 +02:00
Eelco Dolstra e0204f8d46 Move path info caching from BinaryCacheStore to Store
Caching path info is generally useful. For instance, it speeds up "nix
path-info -rS /run/current-system" (i.e. showing the closure sizes of
all paths in the closure of the current system) from 5.6s to 0.15s.

This also eliminates some APIs like Store::queryDeriver() and
Store::queryReferences().
2016-04-19 18:52:53 +02:00
Dan Peebles 608b0265e1 Print out all bad references/requisites at once
Also updates tests to check for new information. Fixes #799
2016-04-16 19:58:26 -04:00
Eelco Dolstra 0423787086 Make the .narinfo cache bigger 2016-04-15 15:39:48 +02:00
Eelco Dolstra a7d8eaba54 BinaryCacheStore: Do negative caching of .narinfo lookups 2016-04-15 15:39:48 +02:00
Eelco Dolstra d1b0909894 BinaryCacheStore::readFile(): Return a shared_ptr to a string
This allows readFile() to indicate that a file doesn't exist, and
might eliminate some large string copying.
2016-04-15 15:39:48 +02:00
Eelco Dolstra 99851c6f06 Unify "nix verify-paths" and "nix verify-store"
"verify-store" is now simply an "--all" flag to "nix verify". This
flag can be used for any other store path command as well (e.g. "nix
path-info", "nix copy-sigs", ...).
2016-04-15 15:39:48 +02:00
Eelco Dolstra 31a1a8ed3c Merge pull request #815 from vcunat/p/outputsToInstall
nix-env: respect meta.outputsToInstall
2016-04-14 20:55:41 +02:00
Eelco Dolstra 5169a6da98 Make $NIX_PATH parsing more robust 2016-04-14 17:29:08 +02:00
Eelco Dolstra c045630522 Support channel:<channel-name> URIs
For convenience, you can now say

  $ nix-env -f channel:nixos-16.03 -iA hello

instead of

  $ nix-env -f https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz -iA hello

Similarly,

  $ nix-shell -I channel:nixpkgs-unstable -p hello
  $ nix-build channel:nixos-15.09 -A hello

Abstracting over the NixOS/Nixpkgs channels location also allows us to
use a more efficient transport (e.g. Git) in the future.
2016-04-14 17:26:57 +02:00
Eelco Dolstra 363f37d084 Make the search path lazier with non-fatal errors
Thus, -I / $NIX_PATH entries are now downloaded only when they are
needed for evaluation. An error to download an entry is a non-fatal
warning (just like non-existant paths).

This does change the semantics of builtins.nixPath, which now returns
the original, rather than resulting path. E.g., before we had

  [ { path = "/nix/store/hgm3yxf1lrrwa3z14zpqaj5p9vs0qklk-nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

but now

  [ { path = "https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

Fixes #792.
2016-04-14 15:32:24 +02:00
Eelco Dolstra 3c1c6b8f00 Set RLIMIT_CORE to 0, infinity in builders
This prevents the builder from being affected by whatever the host
system limits happen to be.
2016-04-14 13:39:14 +02:00
Eelco Dolstra 12b257f045 Make primop registration pluggable
This way we don't have to put all primops in one giant file.
2016-04-14 12:50:01 +02:00
Eelco Dolstra 96515b0c0d Fix std::atomic_flag build failure
http://hydra.nixos.org/build/34453794
2016-04-14 12:50:01 +02:00
Eelco Dolstra 867967265b Remove manifest support
Manifests have been superseded by binary caches for years. This also
gets rid of nix-pull, nix-generate-patches and bsdiff/bspatch.
2016-04-11 16:20:15 +02:00
Eelco Dolstra 8cffec8485 Remove failed build caching
This feature was implemented for Hydra, but Hydra no longer uses it.
2016-04-08 18:19:04 +02:00
Eelco Dolstra f398949b40 Make LocalStore thread-safe
Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
Eelco Dolstra 05fbc606fc nix verify-paths: Add ‘--sigs-needed <N>’ flag
This specifies the number of distinct signatures required to consider
each path "trusted".

Also renamed ‘--no-sigs’ to ‘--no-trust’ for the flag that disables
verifying whether a path is trusted (since a path can also be trusted
if it has no signatures, but was built locally).
2016-04-07 15:16:57 +02:00
Eelco Dolstra 6b2ae52808 Use secret-key-files for verifying 2016-04-07 15:16:57 +02:00
Eelco Dolstra e39999ed48 Sign locally-built paths
Locally-built paths are now signed automatically using the secret keys
specified by the ‘secret-key-files’ option.
2016-04-07 15:16:57 +02:00
Eelco Dolstra dc82160164 Fix "tar: This does not look like a tar archive" with fetchTarball / -I http://
The 304 Not Modified was not handled correctly, so the empty result
from the conditional request would overwrite the previous tarball.
2016-04-06 16:57:20 +02:00
Eelco Dolstra b654381eb3 Add "nix sign-paths" command
E.g.

  $ nix sign-paths -k ./secret -r $(type -p geeqie)

signs geeqie and all its dependencies using the key in ./secret.
2016-04-05 16:39:29 +02:00
Eelco Dolstra d0f5719c2a Add "nix copy-sigs" command
This imports signatures from one store into another. E.g.

  $ nix copy-sigs -r /run/current-system -s https://cache.nixos.org/
  imported 595 signatures
2016-04-05 16:39:23 +02:00
Eelco Dolstra 80da7a6375 Probably fix SQLITE_BUSY errors 2016-04-05 15:29:56 +02:00
Eelco Dolstra 37a337bcec throwSQLiteError(): Check for SIGINT so we don't loop forever 2016-04-04 15:08:08 +02:00
Eelco Dolstra 6e120b76ee Add missing -pthread
https://hydra.nixos.org/build/33908385
2016-03-31 12:42:48 +02:00
Eelco Dolstra 9cee600c88 LocalStore: Keep track of ultimately trusted paths
These are content-addressed paths or outputs of locally performed
builds. They are trusted even if they don't have signatures, so "nix
verify-paths" won't complain about them.
2016-03-30 17:35:48 +02:00
Eelco Dolstra 3d119f0a3b Improve the SQLite wrapper API
In particular, this eliminates a bunch of boilerplate code.
2016-03-30 15:50:45 +02:00
Eelco Dolstra d9c5e3bbf0 Factour out SQLite handling 2016-03-30 13:27:25 +02:00
Eelco Dolstra 2ae43ced9a Turn retrying SQLite transactions into a higher-order function 2016-03-30 12:04:27 +02:00
Eelco Dolstra 39a6abc0bc nix verify: Support checking against signatures in other stores
Typical usage is to check local paths using the signatures from a
binary cache:

  $ nix verify-paths -r /run/current-system -s https://cache.nixos.org
  path ‘/nix/store/c1k4zqfb74wba5sn4yflb044gvap0x6k-nixos-system-mandark-16.03.git.fc2d7a5M’ is untrusted
  ...
  checked 844 paths, 119 untrusted
2016-03-30 11:39:34 +02:00
Eelco Dolstra 88541569a2 HttpBinaryCacheStore: Treat 403 errors as 404 2016-03-30 11:17:51 +02:00
Eelco Dolstra de88004a9d CurlDownloader: Fix HTTP error processing 2016-03-30 11:17:35 +02:00
Eelco Dolstra ab3ce1cc13 Improve SIGINT handling in multi-threaded programs
The flag remembering whether an Interrupted exception was thrown is
now thread-local. Thus, all threads will (eventually) throw
Interrupted. Previously, one thread would throw Interrupted, and then
the other threads wouldn't see that they were supposed to quit.
2016-03-29 16:37:16 +02:00
Eelco Dolstra 4f34c40398 Add "nix verify-store" command
Like "nix-store --verify --check-contents", but with the same
advantages as "nix verify-paths".
2016-03-29 16:37:16 +02:00
Eelco Dolstra 784ee35c80 Add "nix verify-paths" command
Unlike "nix-store --verify-path", this command verifies signatures in
addition to store path contents, is multi-threaded (especially useful
when verifying binary caches), and has a progress indicator.

Example use:

$ nix verify-paths --store https://cache.nixos.org -r $(type -p thunderbird)
...
[17/132 checked] checking ‘/nix/store/rawakphadqrqxr6zri2rmnxh03gqkrl3-autogen-5.18.6’
2016-03-29 16:37:16 +02:00
Eelco Dolstra 0ebe69dc67 Re-enable sync_with_stdio
Otherwise writing to std::cerr is not thread-safe (in particular,
lines will be randomly duplicated).
2016-03-29 16:37:16 +02:00
Eelco Dolstra 374198ad6d Move signature support from NarInfo to ValidPathInfo 2016-03-24 11:41:00 +01:00
Eelco Dolstra 11525377e1 Typos 2016-03-24 11:27:58 +01:00
Eelco Dolstra 8b7839b608 HttpBinaryCacheStore: Make thread-safe 2016-03-24 11:10:05 +01:00
Eelco Dolstra c7d44bad00 Drop support for daemon socket path >= 108 characters
Doing a chdir() is a bad idea in multi-threaded programs, leading to
failures such as

  error: cannot connect to daemon at ‘/nix/var/nix/daemon-socket/socket’: No such file or directory

Since Linux doesn't have a connectat() syscall like FreeBSD, there is
no way we can support this in a race-free way.
2016-03-23 17:16:16 +01:00
Eelco Dolstra 056b3ecfa4 LocalStoreAccessor::stat: Handle ENOTDIR
Closes https://github.com/NixOS/hydra/pull/286.
2016-03-23 11:17:46 +01:00
Eelco Dolstra bb1034316d Don't overload dumpPath() 2016-03-22 14:21:45 +01:00
Eelco Dolstra 712b616a84 Move signatures from NarInfo to ValidPathInfo
This allows queryPathInfo() to return signatures.
2016-03-21 18:05:47 +01:00
Eelco Dolstra cebc150b7c nix: Add --store flag
This is a bit user-friendlier than using $NIX_REMOTE.
2016-03-21 18:03:36 +01:00
Eelco Dolstra 1c5f73f529 Add Store::dumpPath() method
This allows applying nix-store --verify-path to binary cache stores:

  NIX_REMOTE=https://cache.nixos.org nix-store --verify-path /nix/store/s5c7...
2016-03-21 17:55:57 +01:00
Eelco Dolstra 87295b9844 Drop support for upgrading from Nix <= 0.12 2016-03-21 15:09:03 +01:00
Eelco Dolstra 02654f782f Fix Darwin build
http://hydra.nixos.org/build/33279996
2016-03-15 12:11:27 +01:00
Eelco Dolstra 00a75b1cd2 Fix another mismatched tag
http://hydra.nixos.org/build/33279570
2016-03-14 15:32:34 +01:00
Eelco Dolstra dc4a71aae5 Fix build on clang due to -Wmismatched-tags
http://hydra.nixos.org/build/33073389
2016-03-14 12:37:30 +01:00
Eelco Dolstra e7c76f7274 BinaryCacheStore::isValidPath(): Use .narinfo cache
If a path is in the .narinfo cache, obviously it's valid.
2016-03-14 12:33:06 +01:00
Eelco Dolstra 674c5ff64f Merge pull request #834 from abbradar/master
Propagate path context via builtins.readFile
2016-03-10 20:17:03 +01:00
Eelco Dolstra b4e0335d4d Add option binary-cache-secret-key-file for signing binary caches 2016-03-04 17:45:22 +01:00
Eelco Dolstra af7cdb1096 BinaryCacheStore: Remove publicKeyFile argument
The public key can be derived from the secret key, so there's no need
for the user to supply it separately.
2016-03-04 17:45:22 +01:00
Eelco Dolstra 42bc395b63 Eliminate some large string copying 2016-03-04 16:49:56 +01:00
Eelco Dolstra ce113c32d2 Add warning about "nix" being experimental 2016-03-04 15:54:41 +01:00
Eelco Dolstra 7c9d7a253c Merge branch 'new-cli' 2016-03-04 15:21:23 +01:00
Nikolay Amiantov f7f0116dd7 Propagate path context via builtins.readFile 2016-03-04 14:04:51 +03:00
Eelco Dolstra 76f1ba4f3b Add file missing from 201b48de60 2016-03-03 18:03:34 +01:00
Nathan Zadoks fe2be8f016 build.cc: fs.h doesn't appear to be necessary anymore 2016-03-03 14:11:00 +01:00
Eelco Dolstra 5a8455c85e Provide function required by Hydra 2016-03-02 18:21:48 +01:00
Eelco Dolstra 25eea97873 Force stack trace for boost format errors 2016-03-02 15:46:07 +01:00
Eelco Dolstra 201b48de60 Add an HTTP binary cache store
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
2016-03-02 15:46:07 +01:00
Eelco Dolstra 0402b6398d Eliminate local-binary-cache-store.hh 2016-03-02 15:46:07 +01:00
Eelco Dolstra 68a5414982 Make store implementations pluggable
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
2016-03-02 15:46:07 +01:00
Eelco Dolstra 6055d84beb Fix reading symlinks
The st_size field of a symlink doesn't have to be correct, e.g. for
/dev/fd symlinks.
2016-03-02 15:46:07 +01:00
Eelco Dolstra 0a62d9b3d7 Remove bad assertion 2016-03-02 15:46:07 +01:00
Eelco Dolstra 89a2fa68ac FdSource: track number of bytes read 2016-03-02 15:46:07 +01:00
Eelco Dolstra cca4a8dc1a importPaths(): Optionally add NARs to binary cache accessor
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
2016-03-02 15:46:07 +01:00
Eelco Dolstra e9c50064b5 Add an HTTP binary cache store
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
2016-02-29 18:15:20 +01:00
Eelco Dolstra 6170bb474b Eliminate local-binary-cache-store.hh 2016-02-29 16:14:39 +01:00
Eelco Dolstra 0b907321cc Make store implementations pluggable
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
2016-02-29 16:11:11 +01:00
Eelco Dolstra 012f8d187c Fix reading symlinks
The st_size field of a symlink doesn't have to be correct, e.g. for
/dev/fd symlinks.
2016-02-28 03:27:34 +01:00
Eelco Dolstra d0344dd2c1 Remove bad assertion 2016-02-26 21:43:59 +01:00
Eelco Dolstra 581e1bc5b4 FdSource: track number of bytes read 2016-02-26 16:16:08 +01:00
Eelco Dolstra bcc9943cee importPaths(): Optionally add NARs to binary cache accessor
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
2016-02-26 15:20:10 +01:00
Eelco Dolstra 00b2c05749 nix: Add commands to query contents of NARs / binary caches
For example,

  $ NIX_REMOTE=file:///my-cache nix ls-store -lR /nix/store/f4kbgl8shhyy76rkk3nbxr0lz8d2ip7q-binutils-2.23.1
  dr-xr-xr-x                    0 ./bin
  -r-xr-xr-x                30748 ./bin/addr2line
  -r-xr-xr-x                66973 ./bin/ar
  ...

Similarly, "nix ls-nar" lists the contents of a NAR file, "nix
cat-nar" extracts a file from a NAR file, and "nix cat-store" extract
a file from a Nix store.
2016-02-25 17:57:00 +01:00
Eelco Dolstra 24a8f9e27b Merge branch 'master' into new-cli 2016-02-25 17:48:35 +01:00
Eelco Dolstra 1042c10fd0 Add NAR / Store accessor abstraction
This is primary to allow hydra-queue-runner to extract files like
"nix-support/hydra-build-products" from NARs in binary caches.
2016-02-25 17:43:19 +01:00
Eelco Dolstra c5bc571861 Fix short boolean flags 2016-02-25 13:31:34 +01:00
Eelco Dolstra 7873cfb18d Fix build 2016-02-25 11:55:05 +01:00
Eelco Dolstra f1bdeac986 Merge branch 'master' into new-cli 2016-02-25 11:25:11 +01:00
Eelco Dolstra 152b1d6bf9 deletePath(): Succeed if path doesn't exist
Also makes it robust against concurrent deletions.
2016-02-24 17:44:55 +01:00
Eelco Dolstra 28e7e29abd Eliminate reserveSpace flag 2016-02-24 17:44:55 +01:00
Eelco Dolstra 5a64e66268 LocalStore::addTextToStore(): Don't read the path we just wrote
This eliminates some unnecessary (presumably cached) I/O.
2016-02-24 17:11:31 +01:00
Eelco Dolstra 9ccbd55c5b BinaryCacheStore: Implement addToStore()
So now you can do

  $ NIX_REMOTE=file:///tmp/binary-cache nix-instantiate '<nixpkgs>' -A hello

and lots of other operations.
2016-02-24 16:52:28 +01:00
Eelco Dolstra 30e9d01516 openStoreAt(): Initialise the binary cache 2016-02-24 16:52:07 +01:00
Eelco Dolstra ba0a81d14f Show a proper error message for unimplemented binary cache operations 2016-02-24 14:57:30 +01:00
Eelco Dolstra 45c83e5f9b Typo 2016-02-24 14:49:28 +01:00
Eelco Dolstra 263187a2ec Move BinaryCacheStore / LocalBinaryCacheStore from Hydra
So you can now do:

  $ NIX_REMOTE=file:///tmp/binary-cache nix-store -qR /nix/store/...
2016-02-24 14:48:16 +01:00
Eelco Dolstra b584a0e7de Merge pull request #816 from tsion/incomplete-parse
Throw a specific error for incomplete parse errors.
2016-02-24 13:34:56 +01:00
Eelco Dolstra bf2adf72c4 std::condition_variable_any -> std::condition_variable
The latter is supposed to be more efficient.
2016-02-24 13:31:46 +01:00
Eelco Dolstra ccdbf589a4 C++ templates are just a glorified macro facility 2016-02-24 13:07:32 +01:00
Eelco Dolstra 5f862658c3 Remove bad daemon connections from the pool
This is necessary for long-running processes like hydra-queue-runner:
if a nix-daemon worker is killed, we need to stop reusing that
connection.
2016-02-24 11:39:56 +01:00
Scott Olson 6498adb002 Throw a specific error for incomplete parse errors.
`nix-repl` will use this for deciding whether to keep waiting for input or
error out right away.
2016-02-24 04:32:21 -06:00
Eelco Dolstra d5626bf4c1 Pool<T>: Allow a maximum pool size 2016-02-23 16:40:16 +01:00
Eelco Dolstra e292144d46 RemoteStore: Make thread-safe
This allows a RemoteStore object to be used safely from multiple
threads concurrently. It will make multiple daemon connections if
necessary.

Note: pool.hh and sync.hh have been copied from the Hydra source tree.
2016-02-23 15:00:59 +01:00
Vladimír Čunát 03cbb9ad59 nix-env: respect meta.outputsToInstall
Discussed on https://github.com/NixOS/nixpkgs/pull/12653#discussion_r51601849
2016-02-23 14:19:14 +01:00
Eelco Dolstra c0b7a8a0b5 Move ref<t> into a separate header 2016-02-23 13:53:31 +01:00
Eelco Dolstra 8f71bc33d5 Doh 2016-02-22 17:33:27 +01:00
Eelco Dolstra 840056af04 Make OpenSSL usage thread-safe
OpenSSL can randomly segfault unless we register a callback function
to do locking.

https://www.openssl.org/docs/manmaster/crypto/threads.html
2016-02-22 15:51:14 +01:00
Eelco Dolstra d361901bfe curl: Set CURLOPT_NOSIGNAL
Otherwise using curl is not safe in multi-threaded applications
because it installs a SIGALRM handler.
2016-02-22 15:51:14 +01:00
Eelco Dolstra 7a173a7be1 JSONObject: Support floats and booleans 2016-02-19 14:24:04 +01:00
Eelco Dolstra b49d323ce2 Fix build without sodium
http://hydra.nixos.org/build/32085949
2016-02-17 12:42:14 +01:00
Eelco Dolstra c4d22997f3 Add C++ functions for .narinfo processing / signing
This is currently only used by the Hydra queue runner rework, but like
eff5021eaa it presumably will be useful
for the C++ rewrite of nix-push and
download-from-binary-cache. (@shlevy)
2016-02-16 16:38:44 +01:00
Eelco Dolstra 5ac27053e9 Rename ValidPathInfo::hash -> narHash for consistency 2016-02-16 11:49:12 +01:00
Tristan Hume 9b05d5848c Fix to-base16 description 2016-02-16 10:15:58 +01:00
Eelco Dolstra eff5021eaa Add xz compression function
This is used by the Hydra queue runner, but since it may also be
useful for the C++ rewrite of nix-push, I'm putting it here.
2016-02-15 21:45:56 +01:00
Eelco Dolstra bfdacb712c decompressXZ: Ensure that lzma_end() is called
Otherwise we might leak memory.
2016-02-15 21:26:28 +01:00
Eelco Dolstra c8f4d89a34 Expose the export magic value and move LocalStore::queryReferences to Store 2016-02-15 15:01:26 +01:00
Eelco Dolstra e03d6e0998 Fix broken number parsing in fromJSON
The call to tmp_number.append had its arguments mixed up. Also, JSON
does not allow a trailing "," after array/object members.
2016-02-15 15:01:26 +01:00
Eelco Dolstra d089372565 Add function to extract hash part of a store path 2016-02-15 15:01:26 +01:00
Eelco Dolstra b3e8d72770 Merge pull request #762 from ctheune/ctheune-floats
Implement floats
2016-02-12 12:49:59 +01:00
Eelco Dolstra ae4a3cfa03 Move addPermRoot into Store 2016-02-11 16:14:42 +01:00
Eelco Dolstra fd205fb6f8 ref<T>: Add cast operator 2016-02-11 15:32:48 +01:00
Eelco Dolstra 206bbb5dc9 Add basic "nix build" command
Currently only builds by attribute from <nixpkgs> or the specified
file, e.g. "nix build hello".
2016-02-09 21:34:24 +01:00
Eelco Dolstra cd2196b089 Start of new Nix command-line interface 2016-02-09 21:28:29 +01:00
Eelco Dolstra 0db9e6cd1a New command line parsing infrastructure 2016-02-09 21:07:48 +01:00
Eelco Dolstra c780c1124e More of the same 2016-02-04 15:10:47 +01:00
Eelco Dolstra fa7cd5369b StoreAPI -> Store
Calling a class an API is a bit redundant...
2016-02-04 14:48:42 +01:00
Eelco Dolstra c10c61449f Eliminate the "store" global variable
Also, move a few free-standing functions into StoreAPI and Derivation.

Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
2016-02-04 14:28:26 +01:00
Eelco Dolstra d367b8e787 Add build mode to compute fixed-output derivation hashes
For example,

  $ nix-build --hash -A nix-repl.src

will build the fixed-output derivation nix-repl.src (a fetchFromGitHub
call), but instead of *verifying* the hash given in the Nix
expression, it prints out the resulting hash, and then moves the
result to its content-addressed location in the Nix store. E.g

  build produced path ‘/nix/store/504a4k6zi69dq0yjc0bm12pa65bccxam-nix-repl-8a2f5f0607540ffe56b56d52db544373e1efb980-src’ with sha256 hash ‘0cjablz01i0g9smnavhf86imwx1f9mnh5flax75i615ml71gsr88’

The goal of this is to make all nix-prefetch-* scripts unnecessary: we
can just let Nix run the real thing (i.e., the corresponding fetch*
derivation).

Another example:

  $ nix-build --hash -E 'with import <nixpkgs> {}; fetchgit { url = "https://github.com/NixOS/nix.git"; sha256 = "ffffffffffffffffffffffffffffffffffffffffffffffffffff"; }'
  ...
  git revision is 9e7c1a4bbd
  ...
  build produced path ‘/nix/store/gmsnh9i7x4mb7pyd2ns7n3c9l90jfsi1-nix’ with sha256 hash ‘1188xb621diw89n25rifqg9lxnzpz7nj5bfh4i1y3dnis0dmc0zp’

(Having to specify a fake sha256 hash is a bit annoying...)
2016-01-31 12:18:09 +01:00
Eelco Dolstra 9e7c1a4bbd Use the daemon when we don't have write access to the Nix database 2016-01-31 10:28:48 +01:00
Eelco Dolstra 4fa08f3edb Untabify 2016-01-28 16:03:32 +01:00
Eelco Dolstra 26566cd28e Untabify 2016-01-28 16:01:01 +01:00
Eelco Dolstra 9f14d7d33a printMissing(): Propagate store argument 2016-01-28 15:16:06 +01:00
Eelco Dolstra 7899fc959d Define HashType's size to allow it to be forward-declared 2016-01-27 17:46:01 +01:00
Eelco Dolstra 739bab0be7 Trivia 2016-01-27 17:18:31 +01:00
Eelco Dolstra d45ad8fcf5 Make hashLength32() a method of Hash 2016-01-27 17:18:20 +01:00
Eelco Dolstra 5b8c09c124 string2Int: Barf on negative numbers for unsigned types 2016-01-20 16:45:05 +01:00
Eelco Dolstra 5d8b7eb3e1 Revert "Revert "next try for "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751"""
This reverts commit b669d3d2e8.
2016-01-20 16:34:42 +01:00
Eelco Dolstra b669d3d2e8 Revert "next try for "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751""
This reverts commit ed23c8568e. Let's
merge this *after* the 1.11.1 release.
2016-01-20 00:05:28 +01:00
Fabian Schmitthenner ed23c8568e next try for "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751"
This reverts commit 8120b6fb8a and fixes the regression introduced in
8d22b26448.
2016-01-19 20:35:35 +00:00
Eelco Dolstra 8120b6fb8a Revert "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751"
This reverts commit 8d22b26448. It
breaks Nixpkgs:

$ nix-env -qa
error: syntax error, unexpected IND_STR, expecting '}', at /home/eelco/Dev/nixpkgs-stable/pkgs/top-level/python-packages.nix:7605:8
2016-01-19 20:33:32 +01:00
Shea Levy a6d52fcd95 Merge branch 'lexer' of git://github.com/fkz/nix 2016-01-17 10:08:03 -05:00
Fabian Schmitthenner 8d22b26448 don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751 2016-01-12 20:40:41 +00:00
Eelco Dolstra 786046cf13 --option build-repeat: Keep the differing output if -K is given
Similar to 00903fa799. Regardless of -K,
we now also print which output differs.
2016-01-12 18:26:24 +01:00
Eelco Dolstra 8906eda2f9 Canonicalize gids to 0
Previously files in the Nix store were owned by root or by nixbld,
depending on whether they were created by a substituter or by a
builder. This doesn't matter much, but causes spurious diffoscope
differences. So use root everywhere.
2016-01-12 17:27:40 +01:00
Eelco Dolstra 00903fa799 --check: Keep the differing output if -K is given
This makes it easier to investigate the non-determinism, e.g.

  $ nix-build pkgs/stdenv/linux -A stage1.pkgs.zlib --check -K
  error: derivation ‘/nix/store/l54i8wlw22656i4pk05c52ngv9rpl39q-zlib-1.2.8.drv’ may not be deterministic: output ‘/nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8’ differs from ‘/nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8-check’

  $ diffoscope /nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8 /nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8-check
  ...
  ├── lib/libz.a
  │   ├── metadata
  │   │ @@ -1,15 +1,15 @@
  │   │ -rw-r--r-- 30001/30000   3096 Jan 12 15:20 2016 adler32.o
  ...
  │   │ +rw-r--r-- 30001/30000   3096 Jan 12 15:28 2016 adler32.o
  ...
2016-01-12 16:44:26 +01:00
Eelco Dolstra 0cad1f8049 --check: Fix "failed to produce output path"
This occured when sandbox building is disabled, at least one output
exists, and at least one other output does not.
2016-01-12 15:13:47 +01:00
Eelco Dolstra 1c57ab8b31 --check: Fix assertion failure when some outputs are missing
E.g.

  $ nix-build pkgs/stdenv/linux/ -A stage1.pkgs.perl --check
  nix-store: src/libstore/build.cc:1323: void nix::DerivationGoal::tryToBuild(): Assertion `buildMode != bmCheck || validPaths.size() == drv->outputs.size()' failed.

when perl.out exists but perl.man doesn't. The fix is to only check
the outputs that exist. Note that "nix-build -A stage1.pkgs.all
--check" will still give a (proper) error in this case.
2016-01-12 14:54:39 +01:00
Eelco Dolstra 458711e4ee Fix "Bad address" executing build hook
This was observed in the deb_debian7x86_64 build:
http://hydra.nixos.org/build/29973215

Calling c_str() on a temporary should be fine because the temporary
shouldn't be destroyed until after the execl() call, but who knows...
2016-01-07 15:10:14 +01:00
Eelco Dolstra 9aac1861f7 Fix some signedness warnings 2016-01-07 14:37:39 +01:00
Eelco Dolstra 02a66b3fd7 nix-store --repair-path: Rebuild if there is no substituter 2016-01-06 22:07:59 +01:00
Eelco Dolstra caaaff3954 Fix --repair failure on multiple-output derivations
If repair found a corrupted/missing path that depended on a
multiple-output derivation, and some of the outputs of the latter were
not present, it failed with a message like

  error: path ‘/nix/store/cnfn9d5fjys1y93cz9shld2xwaibd7nn-bash-4.3-p42-doc’ is not valid
2016-01-06 21:49:32 +01:00
Christian Theune 934642155c @eelco's feedback: downgrade to regular float for size, remove unused function. 2016-01-06 08:25:58 +01:00
Fabian Schmitthenner 39d1da7b51 Better error message
Also show types when nix cannot compare values of different types.
This is also more consistent since types are already shown when comparing values of the same not comparable type.
2016-01-05 14:13:54 +01:00
Eelco Dolstra 9a4706eb19 Fix compilation 2016-01-05 14:05:11 +01:00
Shea Levy 14080f3e4b Use __toString when coercing sets to strings.
For example, "${{ foo = "bar"; __toString = x: x.foo; }}" evaluates
to "bar".

With this, we can delay calling functions like mkDerivation,
buildPythonPackage, etc. until we actually need a derivation, enabling
overrides and other modifications to happen by simple attribute set
update.
2016-01-05 13:55:06 +01:00
Eelco Dolstra 6298afc047 Merge pull request #685 from vizanto/master
POSIX compliant directory access (fixes build on Solaris)
2016-01-05 13:49:55 +01:00
FrankHB 607d91517e Fixed unresolved 'BUFSIZ'
`BUFSIZ` is defined in header `<cstdio>`.
2016-01-05 13:33:40 +01:00
Eelco Dolstra 743e310046 Fix non-Darwin build 2016-01-05 13:31:15 +01:00
Vladimír Čunát 29c0199d5c manual: document builtins.functionArgs
The text is just a conversion of comment from src/libexpr/primops.cc
2016-01-05 13:29:17 +01:00
Tuomas Tynkkynen f770b9e6c5 libstore: mmap() returns MAP_FAILED, not NULL on failure 2016-01-05 13:26:35 +01:00
Christian Theune a12a43046b Edge condition: parser did not pick up floats starting exactly with 0. 2016-01-05 09:54:49 +01:00
Christian Theune f872262e08 Fix up float parsing. 2016-01-05 09:46:37 +01:00
Christian Theune 494fc5acbb Try a simplified version of float lexing that didn't work.
The last one I tried was botchered anyway ...
2016-01-05 00:53:22 +01:00
Christian Theune 14ebde5289 First hit at providing support for floats in the language. 2016-01-05 00:40:40 +01:00
Eelco Dolstra 71a93a5f0e Don't allow sandbox profile except in relaxed mode
This makes Darwin consistent with Linux: Nix expressions can't break
out of the sandbox unless relaxed sandbox mode is enabled.

For the normal sandbox mode this will require fixing #759 however.
2016-01-04 20:01:13 +01:00
Eelco Dolstra 77ad443bd1 ~PathLocks(): Handle exceptions
Otherwise, since the call to write a "d" character to the lock file
can fail with ENOSPC, we can get an unhandled exception resulting in a
call to terminate().
2016-01-04 11:34:36 +01:00
Eelco Dolstra b8258a4475 Fix regression in passAsFile
Caused by 8063fc497a. If tmpDir !=
tmpDirInSandbox (typically when there are multiple concurrent builds
with the same name), the *Path attribute would not point to an
existing file. This caused Nixpkgs' writeTextFile to write an empty
file. In particular this showed up as hanging VM builds (because it
would run an empty run-nixos-vm script and then wait for it to finish
booting).
2015-12-29 15:28:20 +01:00
Eelco Dolstra 52120123a5 Handle /tmp being a symlink
Hopefully fixes Darwin sandbox regression introduced in
8063fc497a.
2015-12-22 17:16:17 +01:00
Eelco Dolstra 68fd01f42a Don't ignore sodium_init() return value 2015-12-22 17:14:04 +01:00
Eelco Dolstra f696af0fab Fix bad error message in Darwin chroots 2015-12-22 17:05:29 +01:00
Eelco Dolstra a9faa7bbce showId: Handle empty attribute names
We should probably disallow these, but until then, we shouldn't barf
with an assertion failure.

Fixes #738.
2015-12-17 15:15:28 +01:00
Eelco Dolstra 8f67325a7c Build sandbox support etc. unconditionally on Linux
Also, use "#if __APPLE__" instead of "#if SANDBOX_ENABLED" to prevent
ambiguity.
2015-12-10 11:47:17 +01:00
Bjørn Forsman 65bd82d42a Clarify error message for hash mismatches (again)
This is arguably nitpicky, but I think this new formulation is even
clearer. My thinking is that it's easier to comprehend when the
calculated hash value is displayed close to the output path. (I think it
is somewhat similar to eliminating double negatives in logic
statements.)

The formulation is inspired / copied from the OpenEmbedded build tool,
bitbake.
2015-12-08 19:50:25 +01:00
Ludovic Courtès d1e3bf01bc daemon: Add 'buildMode' parameter to 'buildPaths' RPC 2015-12-02 18:14:49 +01:00
Eelco Dolstra 8063fc497a Use deterministic $TMPDIR in sandbox
Rather than using $<host-TMPDIR>/nix-build-<drvname>-<number>, the
temporary directory is now always /tmp/nix-build-<drvname>-0. This
improves bitwise-exact reproducibility for builds that store $TMPDIR
in their build output. (Of course, those should still be fixed...)
2015-12-02 15:04:00 +01:00
Danny Wilson 69b9d8fdbd Use DT_UNKNOWN when dirent d_type extension is not supported.
edolstra:
“…since callers of readDirectory have to handle the possibility of
 DT_UNKNOWN anyway, and we don't want to do a stat call for every
 directory entry unless it's really needed.”
2015-11-25 21:57:19 +01:00
Shea Levy f327970129 Merge branch 'auto-call-functor'
autoCallFunction now auto-calls functors
2015-11-25 11:57:31 -05:00
Shea Levy 9533532ce2 autoCallFunction: Auto-call functors 2015-11-25 11:56:14 -05:00
Eelco Dolstra 0ab4d905e7 Merge branch 'p/sandbox-rename-minimal' of https://github.com/vcunat/nix 2015-11-25 14:53:42 +01:00
Eelco Dolstra a0f0733413 Fix build failure introduced by #704
Also, make the FreeBSD checks conditional on FreeBSD.
2015-11-25 14:41:19 +01:00
Eelco Dolstra afef347157 Merge pull request #712 from pSub/print-meta-license
Print license information on '--xml --meta'
2015-11-25 14:00:52 +01:00
Eelco Dolstra cad40adce5 Merge pull request #704 from ysangkok/freebsd-support
FreeBSD support with knowledge about Linux emulation
2015-11-24 19:24:21 +01:00
Jude Taylor 279fa8f618 reintroduce host deps in tandem with sandbox profiles 2015-11-21 15:57:06 -08:00
Shea Levy e0bd114e09 Revert "remove sandbox-defaults.sb"
As discussed in NixOS/nixpkgs#11001, we still need some of the old
sandbox mechanism.

This reverts commit d760c2638c.
2015-11-21 16:40:24 -05:00
Pascal Wittmann 4921223160 Print license information on '--xml --meta'
The nixpkgs manual prescribes the use of values from stdenv.lib.licenses
for the meta.license attribute. Those values are attribute sets and
currently skipped when running nix-env with '--xml --meta'. This has the
consequence that also nixpkgs-lint will report missing licenses.

With this commit nix-env with '--xml --meta' will print all attributes
of an attribute set that are of type tString. For example the output for
the package nixpkgs.hello is

    <meta name="license" type="strings">
      <string type="url" value="http://spdx.org/licenses/GPL-3.0+" />
      <string type="shortName" value="gpl3Plus" />
      <string type="fullName" value="GNU General Public License v3.0 or later" />
      <string type="spdxId" value="GPL-3.0+" />
    </meta>

This commit fixes nixpkgs-lint, too.
2015-11-21 11:43:44 +01:00
Jude Taylor b9b7bb1806 re-fix permissions for GHC 2015-11-19 16:06:21 -08:00
Shea Levy 5deb7fbdfb Merge branch 'sandbox-profiles' of git://github.com/pikajude/nix
Temporarily allow derivations to describe their full sandbox profile.
This will be eventually scaled back to a more secure setup, see the
discussion at #695
2015-11-19 17:44:11 -05:00
Peter Simons 6ad10591ce src/libstore/build.cc: clarify error message for hash mismatches
Nix reports a hash mismatch saying:

  output path ‘foo’ should have sha256 hash ‘abc’, instead has ‘xyz’

That message is slightly ambiguous and some people read that statement
to mean the exact opposite of what it is supposed to mean. After this
patch, the message will be:

  Nix expects output path ‘foo’ to have sha256 hash ‘abc’, instead it has ‘xyz’
2015-11-19 12:42:37 +01:00
janus 8a74a125bc FreeBSD can build Linux 32-bit binaries 2015-11-17 14:16:08 +00:00
Shea Levy 1d3529e93a Default arguments belong at declaration, not definition 2015-11-16 05:55:55 -05:00
Shea Levy 9b4cd20752 Fix copy-paste error 2015-11-16 05:54:34 -05:00
Shea Levy 58d2fac91d AutoDelete: Add default constructor with deletion disabled 2015-11-16 05:53:10 -05:00
Shea Levy 4390142315 Use AutoDelete for sandbox profile file 2015-11-15 06:08:50 -05:00
Jude Taylor bd09a4c967 simplify build.cc using modern C++ features 2015-11-14 14:11:03 -08:00
Jude Taylor 4876bb012e simplify build permissions 2015-11-14 14:11:03 -08:00
Jude Taylor d760c2638c remove sandbox-defaults.sb 2015-11-14 14:11:03 -08:00
Jude Taylor 8a7f0dfd68 use per-derivation sandbox profiles 2015-11-14 14:10:43 -08:00
Vladimír Čunát b39622a487 rename chroot to sandbox (fixes #656, close #682)
- rename options but leav old names as lower-priority aliases,
  also "-dirs" -> "-paths" to get closer to the meaning
- update docs to reflect the new names (old aliases are not documented),
  including a new file with release notes
- tests need an update after corresponding changes to nixpkgs
- __noChroot is left as it is (after discussion on the PR)
2015-11-10 22:32:51 +01:00
Eelco Dolstra 8fdd156a65 Add option to verify build determinism
Passing "--option build-repeat <N>" will cause every build to be
repeated N times. If the build output differs between any round, the
build is rejected, and the output paths are not registered as
valid. This is primarily useful to verify build determinism. (We
already had a --check option to repeat a previously succeeded
build. However, with --check, non-deterministic builds are registered
in the DB. Preventing that is useful for Hydra to ensure that
non-deterministic builds don't end up getting published at all.)
2015-11-09 23:16:24 +01:00
Eelco Dolstra 96c2ebf004 Revert "Allow using /bin and /usr/bin as impure prefixes on non-darwin by default"
This reverts commit 79ca503332. Ouch,
never noticed this. We definitely don't want to allow builds to have
arbitrary access to /bin and /usr/bin, because then they can (for
instance) bring in a bunch of setuid programs. Also, we shouldn't be
encouraging the use of impurities in the default configuration.
2015-11-09 21:28:38 +01:00
Eelco Dolstra 4384bbd2e1 optimizePath(): Detect some .links corruption
If automatic store optimisation is enabled, and a hard-linked file in
the store gets corrupted, then the corresponding .links entry will
also be corrupted. In that case, trying to repair with --repair or
--repair-path won't work, because the new "good" file will be replaced
by a hard link to the corrupted file. We can catch most of these cases
by doing a sanity-check on the file sizes.
2015-11-09 21:28:25 +01:00
Eelco Dolstra 7759a56bed Fix namespace issue 2015-11-09 21:28:25 +01:00
Danny Wilson cdb346c65e Fix build on Solaris
d_type is not part of the POSIX spec unfortunately.
2015-11-07 04:51:33 +01:00
Shea Levy 5bc12454fd Merge branch 'master' of https://github.com/pikajude/nix
> I made this change for two reasons:

> 1. Darwin's locale data doesn't appear to be open source
> 2. Privileged processes will always use /usr/share/locale regardless of environment variables
2015-11-04 17:27:19 -05:00
Eelco Dolstra 6f1743b1a5 Support SHA-512 hashes
Fixes #679.

Note: on x86_64, SHA-512 is considerably faster than SHA-256 (198 MB/s
versus 131 MB/s).
2015-11-04 16:37:49 +01:00
Eelco Dolstra a6ca68a70c Require OpenSSL 2015-11-04 16:37:49 +01:00
Jude Taylor c10ef2d134 fix syntax error 2015-11-03 09:03:32 -08:00
Jude Taylor 3b52e54e31 darwin: allow reading system locale and zoneinfo 2015-11-03 08:43:09 -08:00
Jude Taylor d6bd51ffbe allow reading ICU data 2015-10-31 02:52:42 -07:00
Jude Taylor 95c66cc950 add special devices to sandbox-defaults 2015-10-30 20:32:11 -07:00
Eelco Dolstra 1f735a3440 <nix/fetchurl.nix>: Support xz-compressed NARs 2015-10-30 12:34:30 +01:00
Eelco Dolstra dae5dc7ade <nix/fetchurl.nix>: Support downloading and unpacking NARs
This removes the need to have multiple downloads in the stdenv
bootstrap process (like a separate busybox binary for Linux, or
curl/mkdir/sh/bzip2 for Darwin). Now all those files can be combined
into a single NAR.
2015-10-30 11:27:47 +01:00
Eelco Dolstra 5c28943e8f int2String() -> std::to_string() 2015-10-29 13:26:55 +01:00
Eelco Dolstra 45b2f32cc1 Merge pull request #668 from svanderburg/master
Fix compilation error due to missing ENOENT on cygwin
2015-10-26 11:12:01 +01:00
Jude Taylor 9155fbd386 use nixDataDir instead of appending /share to PREFIX 2015-10-21 14:44:44 -07:00
Jude Taylor af1218a86a revert libutil change 2015-10-21 14:40:35 -07:00
Jude Taylor 992cda1b11 clarifying comment 2015-10-21 14:39:16 -07:00
Jude Taylor 76f3ba42fd move preBuildHook defaulting to globals.cc 2015-10-21 14:39:08 -07:00
Jude Taylor 2dc8e19a17 restore old DEFAULT_ALLOWED_IMPURE_PREFIXES 2015-10-21 14:38:49 -07:00
Jude Taylor ff6953cb03 Add resolve-system-dependencies.pl 2015-10-21 12:38:52 -07:00
Jude Taylor 15c7a79ed3 remove usr paths from allowed inputs 2015-10-21 12:38:52 -07:00
Jude Taylor 5bf4085000 allow access to SystemVersion for python builders 2015-10-21 12:38:52 -07:00
Jude Taylor 6a8cee19b3 fix line reading in preBuildHook 2015-10-21 12:38:52 -07:00
Jude Taylor 1fc0fe1baf remove sandbox defaults into a new file 2015-10-21 12:38:52 -07:00
Jude Taylor d4cac051f7 restore allowed impure prefixes 2015-10-21 12:38:52 -07:00
Jude Taylor 4b8c71b4ef remove an unneeded default impure-dep 2015-10-21 12:38:52 -07:00
Jude Taylor e770f941d6 make sandbox builds more permissive 2015-10-21 12:38:52 -07:00
Jude Taylor 1ea2aa9e99 add a few more permissions 2015-10-21 12:38:52 -07:00
Eelco Dolstra f5b32ff47d Allow builtin fetchurl regardless of the derivation's system attribute 2015-10-21 15:14:42 +02:00
Eelco Dolstra 01615b5f63 Show progress indicator for builtin fetchurl 2015-10-21 15:14:42 +02:00
Eelco Dolstra 5db358d4d7 Disable TLS verification for builtin fetchurl
This makes it consistent with the Nixpkgs fetchurl and makes it work
in chroots. We don't need verification because the hash of the result
is checked anyway.
2015-10-21 15:14:42 +02:00
Eelco Dolstra 357d31b339 Fix segfault in builtin fetchurl
The stack allocated for the builder was way too small (32 KB). This is
sufficient for normal derivations, because they just do some setup and
then exec() the actual builder. But for the fetchurl builtin
derivation it's not enough. Also, allocating the stack on the caller's
stack was fishy business.
2015-10-21 15:14:42 +02:00
Sander van der Burg 058d1950c0 Fix compilation error due to missing ENOENT on cygwin 2015-10-18 19:04:24 +00:00
Tuomas Tynkkynen 2075ec83e3 Allow building ARMv6 stuff on ARMv7
This allows building a Raspberry Pi image on modern, faster boards.
2015-10-08 13:32:28 +02:00
Eelco Dolstra 51cf4455b2 isFunctor: Simplify 2015-10-08 13:22:11 +02:00
Mathnerd314 8a87521636 forceFunction: allow functors as well 2015-10-08 13:19:44 +02:00
Eelco Dolstra 14f7e1fc42 Revert to CURLOPT_PROGRESSFUNCTION
CURLOPT_XFERINFOFUNCTION isn't widely supported yet.

http://hydra.nixos.org/build/26679495
2015-10-08 13:15:21 +02:00
Eelco Dolstra 21ecd106ed Show progress during downloads 2015-10-07 17:31:50 +02:00
Eelco Dolstra f08449ccbd nix-prefetch-url: Add --name option
This allows overriding the name component of the resulting Nix store
path, which is necessary if the base name of the URI contains
"illegal" characters.
2015-10-07 14:55:33 +02:00
Eelco Dolstra e4abf8610b nix-prefetch-url -A: Use "name" attribute from Nix expression
This is in particular useful for fetchFromGitHub et al., ensuring that
the store path produced by nix-prefetch-url corresponds to what those
functions expect.
2015-10-07 14:47:39 +02:00
Eelco Dolstra b54f447df9 nix-prefetch-url: Support unpacking tarballs
This allows nix-prefetch-url to prefetch the output of fetchzip and
its wrappers (like fetchFromGitHub). For example:

  $ nix-prefetch-url --unpack https://github.com/NixOS/patchelf/archive/0.8.tar.gz

or from a Nix expression:

  $ nix-prefetch-url -A nix-repl.src

In the latter case, --unpack can be omitted because nix-repl.src is a
fetchFromGitHub derivation and thus has "outputHashMode" set to
"recursive".
2015-10-07 14:40:10 +02:00
Manuel Jacob 7888b2ba15 Define SOL_LOCAL if not defined already (e.g. on FreeBSD).
Some evidence that defining it to be 0 is right:
* OS X headers define it to be 0.
* Other code uses 0 instead of SOL_LOCAL to check for peer credentials
  (e.g. FreeBSD's implementation of getpeereid).
2015-10-06 22:28:30 +02:00
Manuel Jacob 12c1776df9 Don't try to link libdl on FreeBSD.
There is no libdl on FreeBSD.  Instead the symbols are included in libc.
2015-10-06 22:28:30 +02:00
Manuel Jacob cbc177ca01 Use pkg-config-provided LDFLAGS for libsqlite3 and libcurl.
Previously, pkg-config was already queried for libsqlite3's and
libcurl's link flags.  However they were not used, but hardcoded
instead.  This commit replaces the hardcoded LDFLAGS by the ones
provided by pkg-config in a similar pattern as already used for
libsodium.
2015-10-06 22:28:30 +02:00
Eelco Dolstra 1abda8e173 nix-store --serve: Implement log size limit 2015-10-06 17:33:30 +02:00
Eelco Dolstra 06f29fafe8 nix-prefetch-url: Support prefetching from a Nix expression
For example,

  $ nix-prefetch-url -A hello.src

will prefetch the file specified by the fetchurl call in the attribute
‘hello.src’ from the Nix expression in the current directory. This
differs from ‘nix-build -A hello.src’ in that it doesn't verify the
hash.

You can also specify a path to the Nix expression:

  $ nix-prefetch-url ~/Dev/nixpkgs -A hello.src

List elements (typically used in ‘patches’ attributes) also work:

  $ nix-prefetch-url -A portmidi.patches.0
2015-10-01 18:07:56 +02:00
Eelco Dolstra a3c4eb0964 nix-prefetch-url: $PRINT_PATH -> --print-path 2015-10-01 16:53:07 +02:00
Eelco Dolstra bec3c31608 nix-prefetch-url: Rewrite in C++ 2015-10-01 16:47:43 +02:00
Eelco Dolstra 4ba6bc184c Shut up clang warnings 2015-09-18 01:22:35 +02:00
Vladimír Čunát 42808fa281 nix-env --upgrade: show "downgrading" when doing so
It was strange to show "upgrading" when the version was getting lower.
This is left on "upgrading" when the versions are the same,
as I can't see any better wording.
2015-09-17 12:09:54 +02:00
Vladimír Čunát 64f9b511be nix-env --upgrade: avoid unexpected downgrades
Until now, if one explicitly installed a low-priority version,
nix-env --upgrade would downgrade it by default and even with --leq.
Let's never accept an upgrade with version not matching the upgradeType.
Additionally, let's never decrease the priority of an installed package;
you can use --install to force that.

Also refactor to use variable bestVersion instead of bestName,
as only version was used from it.
2015-09-17 12:06:26 +02:00
Eelco Dolstra 71a5161365 Implement buildDerivation() via the daemon 2015-09-03 12:56:59 +02:00
Eelco Dolstra 73332fd342 Filter build-chroot-dirs entries that conflict with derivation outputs
Fixes https://github.com/NixOS/nixpkgs/issues/9504.

Note that this means we may have a non-functional /bin/sh in the
chroot while rebuilding Bash or one of its dependencies. Ideally those
packages don't rely on /bin/sh though.
2015-09-02 14:54:12 +02:00
Eelco Dolstra d711fe8f5e Remove unused variable 2015-09-02 14:46:04 +02:00
Eelco Dolstra e12cf82782 Prevent .chroot from being GC'ed when using LocalStore::buildDerivation()
Fixes #616.
2015-08-24 11:13:31 +02:00
Eelco Dolstra eadb86f447 nix-collect-garbage: Revive --max-freed
Fixes #609.
2015-08-21 13:57:53 +02:00
Manolis Ragkousis 26221e44eb Remove unneeded HAVE_UNSHARE.
* src/libstore/build.cc (CHROOT_ENABLED): Remove HAVE_UNSHARE.
2015-08-19 14:11:15 +03:00
Kirill Elagin 3b0f60e5c2 baseNameOf: Enhance basename compatibility
* If the path ends with a slash, drop it.
* If the remaining path doesn’t contain slashes, just return it.

Fixes #574.
2015-08-07 03:35:02 +02:00
Eelco Dolstra 34dfbd9394 Don't include <iostream> before config.h
This breaks the build on 32-bit systems.

http://hydra.nixos.org/build/24373658
2015-08-04 11:12:31 +02:00
Eelco Dolstra 30d19a2bdc Handle debug messages from runChild()
Turns out that "nix-build -vvv" with chroots enabled has been broken
for some time, because some debug message got interpreted as an error.
2015-08-03 18:04:32 +02:00
Iwan Aucamp 3db950aab7 Removed unnecessary included 2015-08-03 11:48:34 +02:00
Eelco Dolstra 2bac04c5ff Fix stack consumption 2015-07-31 20:28:25 +02:00
Iwan Aucamp 75837651f1 Output line number on infinite recursion 2015-07-31 20:26:44 +02:00
Eelco Dolstra 76cc8e97a2 Add sort primop 2015-07-28 18:39:39 +02:00
Eelco Dolstra 50807f3dd5 Add primop genList
This can be used to implement functions like ‘imap’ (or for that
matter, ‘map’) without the quadratic complexity incurred by calling
‘++’ repeatedly.
2015-07-28 17:28:35 +02:00
Eelco Dolstra f3dda728a4 Remove unnecessary parentheses 2015-07-26 12:07:14 +02:00
Eelco Dolstra d6d5885c15 Add replaceStrings primop
This is a generalisation of replaceChars in Nixpkgs.
2015-07-24 15:32:24 +02:00
Eelco Dolstra 2e8fd4c5cd Add concatStringsSep as a primop
This fixes the quadratic behaviour of concatStrings/concatStringsSep
in Nixpkgs.
2015-07-24 02:38:09 +02:00
Eelco Dolstra cb4320c1a0 Cleanup 2015-07-23 23:14:07 +02:00
Eelco Dolstra c8bb2371eb Optimize empty sets
This reduces the number of Bindings allocations by about 10%.
2015-07-23 23:11:08 +02:00
Eelco Dolstra 16c9935fa9 Fix fetchurl of executable file
Pointed out by @cstrahan, thanks!
2015-07-23 22:25:04 +02:00
Eelco Dolstra 19eddecc0f Merge branch 'attr-set-hh' of https://github.com/nbp/nix
Conflicts:
	src/libexpr/eval.cc
2015-07-23 22:16:01 +02:00
Eelco Dolstra b83801f8b3 Optimize small lists
The value pointers of lists with 1 or 2 elements are now stored in the
list value itself. In particular, this makes the "concatMap (x: if
cond then [(f x)] else [])" idiom cheaper.
2015-07-23 22:05:09 +02:00
Eelco Dolstra 14be783676 Add primops all and any
These are used thousands of times during NixOS evaluation, so it's
useful to speed them up.
2015-07-23 19:23:11 +02:00
Shea Levy 39e27a04b8 Importing derivations: Add name attribute to make a valid drv 2015-07-23 17:04:07 +02:00
Shea Levy 1ed55234d9 Allow derivations-as-srcs in the context of builtins.toFile files 2015-07-23 17:03:47 +02:00
Eelco Dolstra 61af14a921 Add foldl' primop 2015-07-23 17:03:02 +02:00
Eelco Dolstra 887bb5fa5a --version: Print some config info
Such as whether Nix is built with signed binary cache support, and the
location of the configuration file.
2015-07-23 14:38:00 +02:00
Eelco Dolstra 1993b10d11 Fix Darwin build
Turns out getgrouplist() is not POSIX.

http://hydra.nixos.org/build/23881243
2015-07-21 14:45:24 +02:00
Eelco Dolstra 0a2bee307b Make <nix/fetchurl.nix> a builtin builder
This ensures that 1) the derivation doesn't change when Nix changes;
2) the derivation closure doesn't contain Nix and its dependencies; 3)
we don't have to rely on ugly chroot hacks.
2015-07-20 04:38:46 +02:00
Eelco Dolstra eda2f36c2a Provide more detailed info about build status to hydra-queue-runner
In particular, hydra-queue-runner can now distinguish between remote
build / substitution / already-valid. For instance, if a path already
existed on the remote side, we don't want to store a log file.
2015-07-20 03:20:03 +02:00
Eelco Dolstra ccf31dbc25 nix-copy-closure: Add -v flag
And make exportPath() less spammy by default.
2015-07-20 01:52:07 +02:00
Eelco Dolstra db55940d9e Support systemd log severity prefixes
This is mostly useful for hydra-queue-runner.
2015-07-20 01:39:48 +02:00
Eelco Dolstra b3491c781c More cleanup 2015-07-20 01:16:16 +02:00
Eelco Dolstra 6bd2c7bb38 OCD: foreach -> C++11 ranged for 2015-07-17 20:13:56 +02:00
Eelco Dolstra 1511aa9f48 Allow remote builds without sending the derivation closure
Previously, to build a derivation remotely, we had to copy the entire
closure of the .drv file to the remote machine, even though we only
need the top-level derivation. This is very wasteful: the closure can
contain thousands of store paths, and in some Hydra use cases, include
source paths that are very large (e.g. Git/Mercurial checkouts).

So now there is a new operation, StoreAPI::buildDerivation(), that
performs a build from an in-memory representation of a derivation
(BasicDerivation) rather than from a on-disk .drv file. The only files
that need to be in the Nix store are the sources of the derivation
(drv.inputSrcs), and the needed output paths of the dependencies (as
described by drv.inputDrvs). "nix-store --serve" exposes this
interface.

Note that this is a privileged operation, because you can construct a
derivation that builds any store path whatsoever. Fixing this will
require changing the hashing scheme (i.e., the output paths should be
computed from the other fields in BasicDerivation, allowing them to be
verified without access to other derivations). However, this would be
quite nice because it would allow .drv-free building (e.g. "nix-env
-i" wouldn't have to write any .drv files to disk).

Fixes #173.
2015-07-17 17:57:40 +02:00
Eelco Dolstra f39979c6d3 Make printValue() interruptible
Fixes #572.
2015-07-17 11:33:39 +02:00
Eelco Dolstra 7c9d0a5969 nix-collect-garbage: Handle ENOENT
Don't barf trying to read a link that just got deleted.

Fixes #575.
2015-07-17 11:24:25 +02:00
Nicolas B. Pierron db21cfa688 Move attribute set data structures into their own header file.
This modification moves Attr and Bindings structures into their own header
file which is dedicated to the attribute set representation. The goal of to
isolate pieces of code which are related to the attribute set
representation. Thus future modifications of the attribute set
representation will only have to modify these files, and not every other
file across the evaluator.
2015-07-14 19:23:17 +02:00
Guillaume Maudoux 467977f203 Fix the parsing of "$"'s in strings. 2015-07-03 14:09:58 +02:00
Guillaume Maudoux 65e4dcd69b Fix the hack that resets the scanner state. 2015-07-03 13:53:36 +02:00
Eelco Dolstra dd48c06bb6 Typo 2015-07-02 00:30:16 +02:00
Ludovic Courtès 9aed117395 Preserve supplementary groups of build users
The following patch is an attempt to address this bug (see
<http://bugs.gnu.org/18994>) by preserving the supplementary groups of
build users in the build environment.

In practice, I would expect that supplementary groups would contain only
one or two groups: the build users group, and possibly the “kvm” group.

[Changed &at(0) to data() and removed tabs - Eelco]
2015-07-01 14:57:48 +02:00
Eelco Dolstra ff4de4cb27 GC: Handle ENOSPC creating/moving to the trash directory
Issue #564.
2015-06-30 21:41:26 +02:00
Eelco Dolstra 2bc9c84327 Use posix_fallocate to create /nix/var/nix/db/reserved 2015-06-22 15:54:55 +02:00
Eelco Dolstra ba63ec6f39 Make /nix/var/nix/db/reserved bigger
Issue #564.
2015-06-22 15:47:40 +02:00
Eelco Dolstra 65f17cd330 Support URLs in $NIX_PATH
This didn't work (despite claims in the manual), because the colon in
"http://" was parsed as a element separator. So handle "://"
specially.
2015-06-17 16:20:11 +02:00
Eelco Dolstra b65875f859 Export outputPaths function
This is useful for the new hydra-queue-runner.
2015-06-10 16:17:06 +02:00
Eelco Dolstra 7c4501886d Use std::vector::data() 2015-06-09 10:54:46 +02:00
Eelco Dolstra b64988bb35 Allow substitutes for builds that have preferLocalBuild set
Not substituting builds with "preferLocalBuild = true" was a bad idea,
because it didn't take the cost of dependencies into account. For
instance, if we can't substitute a fetchgit call, then we have to
download/build git and all its dependencies.

Partially reverts 5558652709 and adds a
new derivation attribute "allowSubstitutes" to specify whether a
derivation may be substituted.
2015-06-04 16:30:22 +02:00
Eelco Dolstra 07d7e7df84 Chown files created for passAsFile
Nixpkgs' writeTextAsFile does this:

  mv "$textPath" "$n"

Since $textPath was owned by root, if $textPath is on the same
filesystem as $n, $n will be owned as root. As a result, the build
result was rejected as having suspicious ownership.

http://hydra.nixos.org/build/22836807
2015-06-04 14:07:43 +02:00
Eelco Dolstra d8ddf994e7 Don't let unprivileged users repair paths 2015-06-02 02:21:54 +02:00
Eelco Dolstra 7106bb0611 Use StoreAPI::verifyStore() 2015-06-02 02:21:15 +02:00
Ludovic Courtès b755752f76 Add a ‘verifyStore’ RPC
Hello!

The patch below adds a ‘verifyStore’ RPC with the same signature as the
current LocalStore::verifyStore method.

Thanks,
Ludo’.

>From aef46c03ca77eb6344f4892672eb6d9d06432041 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Mon, 1 Jun 2015 23:17:10 +0200
Subject: [PATCH] Add a 'verifyStore' remote procedure call.
2015-06-02 02:14:24 +02:00
Eelco Dolstra bc51175dc0 Add tarball tests 2015-06-01 16:18:23 +02:00
Eelco Dolstra da196ec68f Document tarball downloading 2015-06-01 15:14:44 +02:00
Eelco Dolstra c1323b53e3 Fix fetchurl/fetchTarball 2015-06-01 15:08:09 +02:00
Eelco Dolstra 1c88e100e7 readFile: Check against nul bytes 2015-06-01 15:07:42 +02:00
Eelco Dolstra 920f5fd4dd Fix import-from-derivation in restricted eval mode
This relaxes restricted mode to allow access to anything in the
store. In the future, it would be better to allow access to only paths
that have been constructed in the current evaluation (so a hard-coded
/nix/store/blabla in a Nix expression would still be
rejected). However, note that reading /nix/store itself is still
rejected, so you can't use this so get access to things you don't know
about.
2015-05-22 12:18:23 +02:00
Eelco Dolstra 7a411e01cf Remove dead code 2015-05-22 11:29:22 +02:00
Eelco Dolstra 4ca5a9dcfd nix-collect-garbage: Don't call nix-env
Also, make sure --delete-older-than doesn't delete the current
generation.
2015-05-21 16:28:30 +02:00
Eelco Dolstra 22b1a8d43f Move profiles.{cc,hh} to libstore 2015-05-21 15:42:59 +02:00
Eelco Dolstra 13493ef97c nix-collect-garbage: Call collectGarbage() internally 2015-05-21 15:21:38 +02:00
Eelco Dolstra 4441e4cc13 nix-collect-garbage: Don't barf on unreadable directories
And don't try to delete generations from unwritable directories.
2015-05-21 15:04:05 +02:00
Eelco Dolstra 8d813fe3e0 nix-collect-garbage: Remove redundant call to getFileType 2015-05-21 14:09:34 +02:00
Eelco Dolstra a1c1bf3a56 Merge branch 'submit/sparse-generation-symlinks' of https://github.com/ctheune/nix 2015-05-21 12:04:54 +02:00
Christian Theune 12a888894b Mis-read Eelko's request to not make this an option: now, let's not make
it an option. :)
2015-05-20 17:29:52 +02:00
Christian Theune ea39c98d41 Implement alternative to lazy generations:
* only the last generation can be lazy
* depend on the '--lazy-generation' flag to be set
2015-05-19 20:03:36 +02:00
Eelco Dolstra 1529db702b Don't install nix-worker symlink
It has been obsolete since Nix 1.2.

Closes #417.
2015-05-19 16:43:33 +02:00
Christian Theune 3d83188702 Enable lazy/sparse allocation of generation symlinks: avoid creating
new generations if a generation already exists.

Alternatively or additionally I propose a mode where only the *last* generation will be sparse.
2015-05-18 08:38:49 +02:00
Eelco Dolstra a010c0ae05 Fix "error: deriver of path ‘’ is not known" 2015-05-13 18:03:22 +02:00
Rok Garbas dad754843a cygwin: looks like stdout/stdin are reserved words 2015-05-13 09:37:56 +02:00
Rok Garbas 000de699e9 cygwin: explicitly include required c headers 2015-05-13 09:37:12 +02:00
Shea Levy 71083f9e5e Don't try to map /bin/sh to a store path on non-Linux 2015-05-12 16:36:15 -04:00
Luca Bruno 8972f7c14a nix-collect-garbage: Do not pass an empty argument. Closes #530 2015-05-06 15:17:23 +00:00
Eelco Dolstra 6519f06f39 nix-env/nix-instantiate/nix-build: Support URIs
For instance, you can install Firefox from a specific Nixpkgs revision
like this:

  $ nix-env -f 63def04891.tar.gz -iA firefox

Or build a package from the latest nixpkgs-unstable channel:

  $ nix-build https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz -A hello
2015-05-06 14:54:31 +02:00
William A. Kennington III 0705d04dfa nix-collect-garbage: Fix deleting old generations
The call to nix-env expects a string which represents how old the
derivations are or just "old" which means any generations other than
the current one in use. Currently nix-collect-garbage passes an empty
string to nix-env when using the -d option. This patch corrects the call
to nix-env such that it follows the old behavior.
2015-05-06 11:02:36 +02:00
Eelco Dolstra 9451ef3731 Allow URLs in the Nix search path
E.g. to install "hello" from the latest Nixpkgs:

  $ nix-build '<nixpkgs>' -A hello -I nixpkgs=https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz

Or to install a specific version of NixOS:

  $ nixos-rebuild switch -I nixpkgs=63def04891.tar.gz
2015-05-05 17:09:42 +02:00
Eelco Dolstra 35d30d67eb Make downloads interruptable 2015-05-05 14:39:48 +02:00
Luca Bruno ab2b3d6668 nix-collect-garbage: translate to C++ 2015-04-22 15:08:48 +00:00
Shea Levy 4d652875bd Add the pre-build hook.
This hook can be used to set system-specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files through the interface, but it also has full
access to the chroot root.

The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-support system calls. For example,
on OS X there is a set of system-provided frameworks that can reliably
be accessed by any program linked to them, no matter the version the
program is running on. Unfortunately, those frameworks do not
necessarily live in the same locations on each version of OS X, nor do
their dependencies, and thus nix needs to know the specific version of
OS X currently running in order to make those frameworks available. The
pre-build hook is a perfect mechanism for doing just that.
2015-04-18 16:56:02 -04:00
Shea Levy fd6774e285 Revert "Add the pre-build hook."
Going to reimplement differently.

This reverts commit 1e4a4a2e9f.
2015-04-18 14:59:58 -04:00
Eelco Dolstra 035aeb9547 Fix using restricted mode with chroots 2015-04-16 18:46:17 +02:00
Shea Levy 1e4a4a2e9f Add the pre-build hook.
This hook can be used to set system specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files.

The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-supported system calls. For
example, on OS X there is a set of system-provided frameworks that can
reliably be accessed by any program linked to them, no matter the
version the program is running on. Unfortunately, those frameworks do
not necessarily live in the same locations on each version of OS X, nor
do their dependencies, and thus nix needs to know the specific version
of OS X currently running in order to make those frameworks available.
The pre-build hook is a perfect mechanism for doing just that.
2015-04-12 12:56:38 -04:00
Eelco Dolstra 4ed2187377 Use cached result if there is a network error 2015-04-09 12:49:13 +02:00
Eelco Dolstra 1fc905ad4c Move curl stuff into a separate file 2015-04-09 12:12:50 +02:00
Eelco Dolstra c1f04fae35 Implement a TTL on cached fetchurl/fetchTarball results
This is because we don't want to do HTTP requests on every evaluation,
even though we can prevent a full redownload via the cached ETag. The
default is one hour.
2015-04-09 11:55:36 +02:00
Eelco Dolstra 60340ce3e2 Implement caching of fetchurl/fetchTarball results
ETags are used to prevent redownloading unchanged files.
2015-04-09 11:42:04 +02:00
Eelco Dolstra 1711679ea5 Revert /nix/store permission back to 01775
This broke NixOS VM tests.

Mostly reverts 27b7b94923,
5ce50cd99e,
afa433e58c.
2015-04-07 13:21:26 +02:00
Eelco Dolstra afa433e58c Chroot builds: Provide world-readable /nix/store
This was causing NixOS VM tests to fail mysteriously since
5ce50cd99e. Nscd could (sometimes) no
longer read /etc/hosts:

open("/etc/hosts", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)

Probably there was some wacky interaction between the guest kernel and
the 9pfs implementation in QEMU.
2015-04-02 16:59:40 +02:00
Eelco Dolstra 000b5a000f Add fetchTarball builtin
This function downloads and unpacks the given URL at evaluation
time. This is primarily intended to make it easier to deal with Nix
expressions that have external dependencies. For instance, to fetch
Nixpkgs 14.12:

  with import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/nixos-14.12.tar.gz) {};

Or to fetch a specific revision:

  with import (fetchTarball 2766a4b44e.tar.gz) {};

This patch also adds a ‘fetchurl’ builtin that downloads but doesn't
unpack its argument. Not sure if it's useful though.
2015-03-25 17:29:09 +01:00
Eelco Dolstra 7ea6ecf855 addToStore(): Take explicit name argument 2015-03-25 17:06:12 +01:00
Eelco Dolstra 5114a07d95 Improve setting the default chroot dirs 2015-03-24 11:57:46 +01:00
Eelco Dolstra fd89f97be9 Add the closure of store paths to the chroot
Thus, for example, to get /bin/sh in a chroot, you only need to
specify /bin/sh=${pkgs.bash}/bin/sh in build-chroot-dirs. The
dependencies of sh will be added automatically.
2015-03-24 11:52:34 +01:00
Eelco Dolstra 5ce50cd99e Tighten permissions on chroot directories 2015-03-24 11:35:53 +01:00
Eelco Dolstra 6f0c6e20e0 Don't rely on __noChroot for corepkgs
This doesn't work anymore if the "strict" chroot mode is
enabled. Instead, add Nix's store path as a dependency. This ensures
that its closure is present in the chroot.
2015-03-24 11:15:45 +01:00
Eelco Dolstra b005e63ccf Disable scanning for interior pointers
This may remove the "Repeated allocation of very large block"
warnings.
2015-03-19 20:10:08 +01:00
Eelco Dolstra 726f7f7fc9 Fix Boehm API violation
We were calling GC_INIT() after doing an allocation (in the baseEnv
construction), which is not allowed.
2015-03-19 20:02:37 +01:00
Eelco Dolstra da6b704b19 Check return values from malloc/strdup 2015-03-19 14:11:35 +01:00
Eelco Dolstra fa47279440 Print some Boehm GC stats 2015-03-18 16:24:54 +01:00
Eelco Dolstra 7a84143910 valueSize(): Take into account list/bindings/env size 2015-03-18 14:41:28 +01:00
Daniel Hahler e659978ced Fix typos: s/the the/the/ 2015-03-06 16:43:22 +01:00
Eelco Dolstra 17c71334e1 forceValueDeep: Add to error prefix 2015-03-06 15:10:12 +01:00
Eelco Dolstra 5badc8f975 Improve error message 2015-03-06 14:24:08 +01:00
Eelco Dolstra 71b0a3a86b Add option to hide display of missing paths 2015-03-04 15:43:04 +01:00
Eelco Dolstra 75ede65e3d Don't use vfork() before clone()
I'm seeing hangs in Glibc's setxid_mark_thread() again. This is
probably because the use of an intermediate process to make clone()
safe from a multi-threaded program (see
524f89f139) is defeated by the use of
vfork(), since the intermediate process will have a copy of Glibc's
threading data structures due to the vfork(). So use a regular fork()
again.
2015-03-04 15:13:10 +01:00
Shea Levy c2699be93b Merge branch 'allow-system-library' of git://github.com/copumpkin/nix
Make the default impure prefix include all of /System/Library
2015-03-03 15:01:09 -05:00
Dan Peebles 336c4270c6 Make the default impure prefix (not actual allowed impurities!) include all of /System/Library, since we also want PrivateFrameworks from there and (briefly) TextEncodings, and who knows what else. Yay infectious impurities? 2015-03-02 23:01:24 -05:00
Dan Peebles 66d612f1da Allow local networking in the darwin sandbox to appease tests 2015-03-02 22:55:42 -05:00
Eelco Dolstra 885bebf13b More graceful fallback for chroots on Linux < 2.13 2015-02-23 15:54:31 +01:00
Eelco Dolstra 99897f6979 Use chroots for all derivations
If ‘build-use-chroot’ is set to ‘true’, fixed-output derivations are
now also chrooted. However, unlike normal derivations, they don't get
a private network namespace, so they can still access the
network. Also, the use of the ‘__noChroot’ derivation attribute is
no longer allowed.

Setting ‘build-use-chroot’ to ‘relaxed’ gives the old behaviour.
2015-02-23 15:54:31 +01:00
Eelco Dolstra 15d2d3c34e Add restricted evaluation mode
If ‘--option restrict-eval true’ is given, the evaluator will throw an
exception if an attempt is made to access any file outside of the Nix
search path. This is primarily intended for Hydra, where we don't want
people doing ‘builtins.readFile ~/.ssh/id_dsa’ or stuff like that.
2015-02-23 15:54:31 +01:00
Eelco Dolstra c33244d7c1 Merge branch 'tilde-paths' of https://github.com/shlevy/nix 2015-02-19 14:55:06 +01:00
Shea Levy c4653afbcd tilde paths: The rest of the string has to start with a slash anyway 2015-02-19 08:52:13 -05:00
Shea Levy e3e38a048e tilde paths: construct the entire path at parse time 2015-02-19 08:51:21 -05:00
Shea Levy 4b7c9f834c tilde paths: get HOME at parse time 2015-02-19 08:49:10 -05:00
Eelco Dolstra 9bedd9b09b Remove obsolete reference to ~ operator 2015-02-19 14:41:22 +01:00
Shea Levy 4646e94610 ExprConcatStrings: canonicalize concatenated paths 2015-02-19 08:39:25 -05:00
Shea Levy e0953d53de Allow the leading component of a path to be a ~ 2015-02-19 08:05:16 -05:00
Eelco Dolstra 147deb236e nix-store --generate-binary-cache-key: Write key to disk
This ensures proper permissions for the secret key.
2015-02-18 11:19:44 +01:00
Eelco Dolstra bd91064150 Use $<attr>Path instead of $<attr> for passAsFile 2015-02-17 16:42:54 +01:00
Eelco Dolstra a70d275f3d Allow passing attributes via files instead of environment variables
Closes #473.
2015-02-17 14:42:15 +01:00
Harald van Dijk 5451b8db9d Use pivot_root in addition to chroot when possible
chroot only changes the process root directory, not the mount namespace root
directory, and it is well-known that any process with chroot capability can
break out of a chroot "jail". By using pivot_root as well, and unmounting the
original mount namespace root directory, breaking out becomes impossible.

Non-root processes typically have no ability to use chroot() anyway, but they
can gain that capability through the use of clone() or unshare(). For security
reasons, these syscalls are limited in functionality when used inside a normal
chroot environment. Using pivot_root() this way does allow those syscalls to be
put to their full use.
2015-02-16 12:18:19 +01:00
Eelco Dolstra 1c972cba14 Make libsodium an optional dependency 2015-02-10 11:54:06 +01:00
Eelco Dolstra a596c525ad Add base64 encoder/decoder 2015-02-10 11:33:33 +01:00
Eelco Dolstra 2be7f79fd4 Remove tab 2015-02-05 17:21:30 +01:00
Shea Levy d66d9e8425 Require linux 3.13 or later for chroot
Fixes #453
2015-02-04 18:15:56 +01:00
Eelco Dolstra e0def5bc4b Use libsodium instead of OpenSSL for binary cache signing
Sodium's Ed25519 signatures are much shorter than OpenSSL's RSA
signatures. Public keys are also much shorter, so they're now
specified directly in the nix.conf option ‘binary-cache-public-keys’.

The new command ‘nix-store --generate-binary-cache-key’ generates and
prints a public and secret key.
2015-02-04 17:10:31 +01:00
Eelco Dolstra 0d1dafa0c4 Simplify parseHash32 2015-02-03 18:56:47 +01:00
Eelco Dolstra db2ec59903 Simplify printHash32 2015-02-03 18:35:24 +01:00
Shea Levy 73bf32ce94 Merge remote-tracking branch 'shlevy/baseNameOf-no-copy'
baseNameOf: Don't copy paths to the store first
2015-01-29 03:29:09 -05:00
Daniel Peebles f46e329a13 Make inputs writeable in the sandbox (builds still can’t actually write due to user permissions) 2015-01-18 23:25:29 -05:00
Eelco Dolstra c2a8b5c42d Fix assertion failure in nix-env
$ nix-env -f ~/Dev/nixops/ -iA foo
  nix-env: src/libexpr/eval.hh:57: void nix::Bindings::push_back(const nix::Attr&): Assertion `size_ < capacity' failed.
  Aborted
2015-01-15 12:15:22 +01:00
Shea Levy 79ca503332 Allow using /bin and /usr/bin as impure prefixes on non-darwin by default
These directories are generally world-readable anyway, and give us the two
most common linux impurities (env and sh)
2015-01-13 15:41:46 +01:00
Eelco Dolstra fcf57aad27 SysError -> Error 2015-01-13 11:17:56 +01:00
Eelco Dolstra 100961e370 Don't resolve symlinks while checking __impureHostDeps
Since these come from untrusted users, we shouldn't do any I/O on them
before we've checked that they're in an allowed prefix.
2015-01-13 11:16:32 +01:00
Daniel Peebles f1151a3373 Add basic Apple sandbox support 2015-01-12 12:00:01 +01:00
Eelco Dolstra 2a3b1df423 Fix builtins.readDir on XFS
The DT_UNKNOWN fallback code was getting the type of the wrong path,
causing readDir to report "directory" as the type of every file.

Reported by deepfire on IRC.
2015-01-09 14:56:25 +01:00
Eelco Dolstra 57d64d24aa Doh^2 2015-01-08 16:59:22 +01:00
Eelco Dolstra 57b82256b0 Doh 2015-01-08 16:49:31 +01:00
Eelco Dolstra 27b7b94923 Set /nix/store permission to 1737
I.e., not readable to the nixbld group. This improves purity a bit for
non-chroot builds, because it prevents a builder from enumerating
store paths (i.e. it can only access paths it knows about).
2015-01-08 16:39:07 +01:00
Eelco Dolstra 153a943de7 Show position info for failing <...> lookups 2015-01-07 13:43:55 +01:00
Eelco Dolstra 6fec43ccb3 Remove quotes around filenames in position info 2015-01-07 12:08:10 +01:00
Eelco Dolstra df05f49dcd Fix building on Darwin
Fixes #433.
2015-01-06 10:49:44 +01:00
Eelco Dolstra 8027083c3a Allow $NIX_PAGER to override $PAGER 2015-01-02 15:26:56 +01:00
aszlig 8b88d25cda libutil: Limit readLink() error to only overflows.
Let's not just improve the error message itself, but also the behaviour
to actually work around the ntfs-3g symlink bug. If the readlink() call
returns a smaller size than the stat() call, this really isn't a problem
even if the symlink target really has changed between the calls.

So if stat() reports the size for the absolute path, it's most likely
that the relative path is smaller and thus it should also work for file
system bugs as mentioned in 93002d69fc58c2b71e2dfad202139230c630c53a.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Tested-by: John Ericson <Ericson2314@Yahoo.com>
2015-01-02 12:53:42 +01:00
aszlig bbd45ac80f libutil: Improve errmsg on readLink size mismatch.
A message like "error: reading symbolic link `...' : Success" really is
quite confusing, so let's not indicate "success" but rather point out
the real issue.

We could also limit the check of this to just check for non-negative
values, but this would introduce a race condition between stat() and
readlink() if the link target changes between those two calls, thus
leading to a buffer overflow vulnerability.

Reported by @Ericson2314 on IRC. Happened due to a possible ntfs-3g bug
where a relative symlink returned the absolute path (st_)size in stat()
while readlink() returned the relative size.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Tested-by: John Ericson <Ericson2314@Yahoo.com>
2015-01-02 12:53:42 +01:00
Shea Levy 3d97b8d1e7 LocalStore initialization: Don't die if build-users-group doesn't exist
See NixOS/nixpkgs@9245516
2014-12-29 14:40:13 +01:00
Eelco Dolstra bd0f362d2f Revive running builds in a PID namespace 2014-12-23 17:25:06 +01:00
Eelco Dolstra 4e0607369e Pedantry 2014-12-14 01:51:14 +01:00
Eelco Dolstra 8bdff8c100 Merge branch 'cygwin-master' of https://github.com/ternaris/nix 2014-12-14 01:49:14 +01:00
Eelco Dolstra fa2063ca35 Better error message 2014-12-13 16:54:40 +01:00
Eelco Dolstra b77037b8fd Silence some warnings on GCC 4.9 2014-12-12 17:14:28 +01:00
Eelco Dolstra 46f3eb6fdd Shut up a Valgrind warning 2014-12-12 15:10:02 +01:00
Eelco Dolstra f52b6c944e Fix some memory leaks 2014-12-12 15:01:16 +01:00
Eelco Dolstra 28f22b4653 Ensure we're writing to stderr in the builder
http://hydra.nixos.org/build/17862041
2014-12-12 14:35:44 +01:00
Eelco Dolstra 5a2d451648 Don't abort if we get a signal while waiting for the pager 2014-12-12 14:05:23 +01:00
Eelco Dolstra 54616be64f Get rid of unnecessary "interrupted by the user" message with -vvv 2014-12-12 14:01:14 +01:00
Eelco Dolstra 4acca1a5b9 Remove chatty message
This broke building with "-vv", because the builder is not allowed to
write to stderr at this point.
2014-12-12 13:43:59 +01:00
Eelco Dolstra ad790022fd Doh 2014-12-12 13:41:00 +01:00
Eelco Dolstra dcdb254281 Remove tabs 2014-12-12 12:39:50 +01:00
Eelco Dolstra df319047f4 Remove dead code 2014-12-12 12:30:38 +01:00
Eelco Dolstra 8aedaf111e Remove canary stuff 2014-12-12 10:59:50 +01:00
Eelco Dolstra c56bfbe863 Provide default pagers
Borrowed from systemd.
2014-12-10 18:16:05 +01:00
Eelco Dolstra 851b47bd7d Don't do vfork in conjunction with setuid 2014-12-10 18:01:01 +01:00
Eelco Dolstra 0e8fc118b3 Use vfork 2014-12-10 17:25:39 +01:00
Eelco Dolstra b5ed5b6e66 Rename function 2014-12-10 17:25:38 +01:00
Eelco Dolstra e529823635 Don't wait for PID -1
The pid field can be -1 if forking the substituter process failed.
2014-12-10 13:53:04 +01:00
Eelco Dolstra ad332e1718 Revert "Use posix_spawn to run the pager"
This reverts commit d34d2b2bbf.
2014-12-10 13:48:50 +01:00
Shea Levy 50c3352811 builtins.readFile: realise context associated with the path 2014-12-10 12:26:53 +01:00
Marko Durkovic 629e8da3aa Explicitly include required C headers 2014-12-09 13:00:59 +01:00
Eelco Dolstra 8d169b2b75 Define ‘environ’
http://hydra.nixos.org/build/17690555
2014-12-05 21:05:24 +01:00
Eelco Dolstra d34d2b2bbf Use posix_spawn to run the pager
In low memory environments, "nix-env -qa" failed because the fork to
run the pager hit the kernel's overcommit limits. Using posix_spawn
gets around this. (Actually, you have to use posix_spawn with the
undocumented POSIX_SPAWN_USEVFORK flag, otherwise it just uses
fork/exec...)
2014-12-05 20:34:41 +01:00
Shea Levy 608110804c Make all ExternalValueBase functions const 2014-12-02 10:27:10 -05:00
Shea Levy 320659b0cd Allow external code using libnixexpr to add types
Code that links to libnixexpr (e.g. plugins loaded with importNative, or
nix-exec) may want to provide custom value types and operations on
values of those types. For example, nix-exec is currently using sets
where a custom IO value type would be more appropriate. This commit
provides a generic hook for such types in the form of tExternal and the
ExternalBase virtual class, which contains all functions necessary for
libnixexpr's type-polymorphic functions (e.g. `showType`) to be
implemented.
2014-12-02 10:27:04 -05:00
Eelco Dolstra 976df480c9 Add a primop for regular expression pattern matching
The function ‘builtins.match’ takes a POSIX extended regular
expression and an arbitrary string. It returns ‘null’ if the string
does not match the regular expression. Otherwise, it returns a list
containing substring matches corresponding to parenthesis groups in
the regex. The regex must match the entire string (i.e. there is an
implied "^<pat>$" around the regex).  For example:

  match "foo" "foobar" => null
  match "foo" "foo" => []
  match "f(o+)(.*)" "foooobar" => ["oooo" "bar"]
  match "(.*/)?([^/]*)" "/dir/file.nix" => ["/dir/" "file.nix"]
  match "(.*/)?([^/]*)" "file.nix" => [null "file.nix"]

The following example finds all regular files with extension .nix or
.patch underneath the current directory:

  let

    findFiles = pat: dir: concatLists (mapAttrsToList (name: type:
      if type == "directory" then
        findFiles pat (dir + "/" + name)
      else if type == "regular" && match pat name != null then
        [(dir + "/" + name)]
      else []) (readDir dir));

  in findFiles ".*\\.(nix|patch)" (toString ./.)
2014-11-25 11:47:06 +01:00
Eelco Dolstra 4e340a983f forceString(): Accept pos argument 2014-11-25 10:23:36 +01:00
Eelco Dolstra b7b6e3ddec Build derivations in a more predictable order
Derivations are now built in order of derivation name, so a package
named "aardvark" is built before "baboon".

Fixes #399.
2014-11-24 16:50:46 +01:00
Eelco Dolstra 9e3389c337 Don't create unnecessary substitution goals for derivations 2014-11-24 16:50:46 +01:00
Shea Levy b0c5c2ac34 import derivation: cleanup
Before this there was a bug where a `find` was being called on a
not-yet-sorted set. The code was just a mess before anyway, so I cleaned
it up while fixing it.
2014-11-20 22:48:12 -05:00
Eelco Dolstra 8299aaf079 Disable vacuuming the DB after garbage collection
Especially in WAL mode on a highly loaded machine, this is not a good
idea because it results in a WAL file of approximately the same size
ad the database, which apparently cannot be deleted while anybody is
accessing it.
2014-11-19 18:14:24 +01:00
Eelco Dolstra a3e5c99d66 nix-daemon: Call exit(), not _exit()
This was preventing destructors from running. In particular, it was
preventing the deletion of the temproot file for each worker
process. It may also have been responsible for the excessive WAL
growth on Hydra (due to the SQLite database not being closed
properly).

Apparently broken by accident in
8e9140cfde.
2014-11-19 17:09:27 +01:00
Eelco Dolstra 1256ab3b44 Clean up temp roots in a more C++ way 2014-11-19 17:07:29 +01:00
Eelco Dolstra 35aad73bb6 Fix message 2014-11-17 01:00:39 +01:00
Shea Levy 2719627bbe realiseContext: Handle all context types
Avoids an assertion
2014-11-15 21:43:51 -05:00
Shea Levy 997defa166 Add functors (callable attribute sets).
With this, attribute sets with a `__functor` attribute can be applied
just like normal functions. This can be used to attach arbitrary
metadata to a function without callers needing to treat it specially.
2014-11-15 16:12:05 -05:00
Eelco Dolstra 8cfe939b0f Don't use ADDR_LIMIT_3GB
This gives 32-bit builds on x86_64-linux more memory.
2014-11-14 14:16:20 +01:00
Eelco Dolstra bab8d9b52a Make ~DerivationGoal more reliable 2014-11-12 11:35:53 +01:00
Eelco Dolstra 06a86aee15 nix-store --gc: Don't warn about missing manifests directory 2014-11-04 10:41:29 +01:00
Eelco Dolstra 526811c87a nix-daemon: Get peer credentials on Mac OS X
This makes allowed-users and trusted-users work on Mac OS X.
2014-10-31 10:08:59 +01:00
Eelco Dolstra e389f4ea55 Improve error message if the daemon worker fails to start 2014-10-31 09:36:09 +01:00
Eelco Dolstra 3a9b4a1467 Fix more warnings 2014-10-31 08:49:15 +01:00
Eelco Dolstra 80893a13a7 Shut up a clang warning 2014-10-31 08:49:15 +01:00
Eelco Dolstra 11aef17a77 Remove comments claiming we use a private PID namespace
This is no longer the case since
524f89f139.
2014-10-29 15:49:34 +01:00
Shea Levy 6062b12160 Fix build on gcc < 4.7 2014-10-20 12:15:50 -04:00
Shea Levy f040159f77 Revert "Drop support for pre-c++11 compilers."
The breakage this fixed can be worked around without removing support.

This reverts commit 84a13dc576.
2014-10-20 11:33:48 -04:00
Eelco Dolstra ecc2c8f464 Improve printing of ASTs 2014-10-20 09:13:21 +02:00
Shea Levy 84a13dc576 Drop support for pre-c++11 compilers.
In particular, gcc 4.6's std::exception::~exception has an exception
specification in c++0x mode, which requires us to use that deprecated
feature in nix (and led to breakage after some recent changes that were
valid c++11).

nix already uses several c++11 features and gcc 4.7 has been around for
over 2 years.
2014-10-18 22:44:59 -04:00
Shea Levy 0ee1ca628a baseNameOf: Don't copy paths to the store first 2014-10-18 20:28:28 -04:00
Shea Levy d16e3c7f09 Export realiseContext in libnixexpr
Useful for importNative plugins
2014-10-17 22:15:09 -04:00
Eelco Dolstra d04e1ff12a nix-store -q: Check for conflicting flags
Fixes #364.
2014-10-14 12:15:39 +02:00
Eelco Dolstra c6849e2dee Remove redundant space in usage errors 2014-10-14 11:41:04 +02:00
Eelco Dolstra 2a4dbcff74 Improved error message when encountering unsupported file types
Fixes #269.
2014-10-14 10:51:19 +02:00
Eelco Dolstra 6bb4c0b712 mkList: Scrub better
Clearing v.app.right was not enough, because the length field of a
list only takes 32 bits, so the most significant 32 bits of v.app.left
(a.k.a. v.thunk.env) would remain. This could cause Boehm GC to
interpret it as a valid pointer.

This change reduces maximum RSS for evaluating the ‘tested’ job in
nixos/release-small.nix from 1.33 GiB to 0.80 GiB, and runtime by
about 8%.
2014-10-09 13:08:53 +02:00
Eelco Dolstra 986fbd6fab Typo 2014-10-09 11:34:48 +02:00
Eelco Dolstra b6809608cc Get rid of some unnecessary ExprConcatStrings nodes in dynamic attrs
This gives a ~18% speedup in NixOS evaluation (after converting
most calls to hasAttr/getAttr to dynamic attrs).
2014-10-05 01:04:58 +02:00
Eelco Dolstra 1418806969 Show total allocations 2014-10-05 00:39:28 +02:00
Eelco Dolstra c3f0a489f9 Add primop ‘catAttrs’ 2014-10-04 18:15:03 +02:00
Eelco Dolstra d4fcbe1687 Add primop ‘attrValues’ 2014-10-04 16:41:24 +02:00
Eelco Dolstra 58d8a213b0 Tweak 2014-10-04 11:27:23 +02:00
Eelco Dolstra 3f8576a6ab Remove some duplicate code 2014-10-03 22:37:51 +02:00
Shea Levy c08c802bf3 Add readDir primop 2014-10-03 22:32:11 +02:00
Eelco Dolstra 3b5fa8d50c Don't recompile the same regex over and over 2014-10-03 21:29:40 +02:00
Eelco Dolstra 104e55bb7f nix-env: Add regular expression support in selectors
So you can now do things like:

  $ nix-env -qa '.*zip.*'
  $ nix-env -qa '.*(firefox|chromium).*'
2014-10-03 21:29:40 +02:00
Eelco Dolstra 3800f441e4 createDirs(): Handle ‘path’ being a symlink
In particular, this fixes "nix-build -o /tmp/result" on Mac OS X
(where /tmp is a symlink).
2014-10-03 16:53:28 +02:00
Eelco Dolstra a17c23426e printValue(): Don't print <CYCLE> for repeated values 2014-10-01 15:54:40 +02:00
Eelco Dolstra d61853430a Support control characters in JSON output 2014-09-30 00:41:18 +02:00
Eelco Dolstra f77be20c16 printMissing(): Print derivations in approximate build order 2014-09-26 14:09:20 +02:00
Eelco Dolstra 9b146a52f1 nix-daemon: Close unnecessary fd 2014-09-25 18:45:43 +02:00
Eelco Dolstra 0ed1b924be Bindings: Remove copy constructor 2014-09-24 15:29:05 +02:00
Shell Turner cf72a61af2 Add --force-name support for --set in nix-env, to support nix-install-package --set 2014-09-23 15:11:24 +02:00
Eelco Dolstra ebb1dbb3e1 Add missing static 2014-09-23 15:08:27 +02:00
Eelco Dolstra 53b044c2f6 Don't evaluate inside a "throw"
Workaround for
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=41174. This caused
hydra-eval-jobs to ignore SIGINT.
2014-09-22 19:18:05 +02:00
Eelco Dolstra 0cd6596b0e Add ‘deepSeq’ primop
Note that unlike ‘lib.deepSeq’ in Nixpkgs, this handles cycles.
2014-09-22 16:05:00 +02:00
Eelco Dolstra 831fc8ea21 Make forceValueDeep work on values with cycles 2014-09-22 16:05:00 +02:00
Eelco Dolstra 1e0a799bef Rename strictForceValue -> forceValueDeep 2014-09-22 16:05:00 +02:00
Eelco Dolstra 022618c794 Handle cycles when printing a value
So this no longer crashes with a stack overflow:

  nix-instantiate -E --eval 'let as = { x = as; }; in as'

Instead it prints:

  { x = { x = <CYCLE>; }; }
2014-09-22 16:05:00 +02:00
Eelco Dolstra a54c263402 Add ‘seq’ primop 2014-09-22 16:05:00 +02:00
Eelco Dolstra eff120d1b9 Add a function ‘valueSize’
It returns the size of value, including all other values and
environments reachable from it. It is intended for debugging memory
consumption issues.
2014-09-22 16:05:00 +02:00
Eelco Dolstra 2d6cd8aafd attrNames: Don't allocate duplicates of the symbols 2014-09-19 18:11:46 +02:00
Eelco Dolstra ea525a261f Fix off-by-one 2014-09-19 18:08:14 +02:00
Eelco Dolstra 93e4f01ee3 Inline Bindings::find() 2014-09-19 16:56:13 +02:00
Eelco Dolstra 5b58991a71 Store Attrs inside Bindings
This prevents a double allocation per attribute set.
2014-09-19 16:49:41 +02:00
Eelco Dolstra 0342eb1705 Remove bogus comment 2014-09-19 15:07:22 +02:00
Eelco Dolstra d4a71ec3bf Update spec file
http://hydra.nixos.org/build/14344391
2014-09-18 15:42:01 +02:00
Eelco Dolstra 8be9990cdb Install some pkgconfig files 2014-09-18 12:00:40 +02:00
Eelco Dolstra d98bfcbf81 On Linux, disable address space randomization 2014-09-17 17:21:13 +02:00
Eelco Dolstra 6e5b02bee4 Add some instrumentation for debugging GC leaks 2014-09-17 15:19:07 +02:00
Eelco Dolstra d37d012774 Settings: Add bool get() 2014-09-17 15:18:13 +02:00
Eelco Dolstra 9d65287b91 Fix dependency ordering 2014-09-04 20:02:08 +02:00
Eelco Dolstra 9472b4157d Fix boost::too_many_args error
Fixes #333.
2014-09-02 22:53:01 +02:00
Ludovic Courtès 8fb8c26b6d Add an 'optimiseStore' remote procedure call. 2014-09-01 23:53:26 +02:00
Eelco Dolstra b72e93bca8 Add disallowedReferences / disallowedRequisites
For the "stdenv accidentally referring to bootstrap-tools", it seems
easier to specify the path that we don't want to depend on, e.g.

  disallowedRequisites = [ bootstrapTools ];
2014-08-28 18:57:13 +02:00
Gergely Risko fd61069a42 Introduce allowedRequisites feature 2014-08-28 18:23:55 +02:00
Joel Taylor b224ac1520 fix disappearing bash arguments 2014-08-21 23:17:50 +02:00
Eelco Dolstra 809ca33806 Use PR_SET_PDEATHSIG to ensure child cleanup 2014-08-21 15:31:43 +02:00
Eelco Dolstra 524f89f139 Use unshare() instead of clone()
It turns out that using clone() to start a child process is unsafe in
a multithreaded program. It can cause the initialisation of a build
child process to hang in setgroups(), as seen several times in the
build farm:

The reason is that Glibc thinks that the other threads of the parent
exist in the child, so in setxid_mark_thread() it tries to get a futex
that has been acquired by another thread just before the clone(). With
fork(), Glibc runs pthread_atfork() handlers that take care of this
(in particular, __reclaim_stacks()). But clone() doesn't do that.

Fortunately, we can use fork()+unshare() instead of clone() to set up
private namespaces.

See also https://www.mail-archive.com/lxc-devel@lists.linuxcontainers.org/msg03434.html.
2014-08-21 14:08:09 +02:00
Eelco Dolstra fefd3650d4 Fix a segfault in ‘nix-env -qa’
This was triggered by 47e185847e, which
turned globals.state into a pointer.
2014-08-21 00:05:17 +02:00
Eelco Dolstra 65243ee833 Flush std::cout before closing stdout 2014-08-20 21:44:55 +02:00
Eelco Dolstra 43579c9799 Use pager for more commands 2014-08-20 21:44:55 +02:00
Eelco Dolstra 9481f77693 Provide reasonable default flags for $LESS
Borrowed from systemd.
2014-08-20 21:22:41 +02:00
Eelco Dolstra 11849a320e Use proper quotes everywhere 2014-08-20 18:03:48 +02:00
Eelco Dolstra 373fad75e1 Add some color 2014-08-20 16:50:17 +02:00
Eelco Dolstra 392430b2c4 nix-store -l: Automatically pipe output into $PAGER 2014-08-20 15:12:58 +02:00
Eelco Dolstra 954188af27 Filter Nix-specific ANSI escape sequences from stderr
The Nixpkgs stdenv prints some custom escape sequences to denote
nesting and stuff like that. Most terminals (e.g. xterm, konsole)
ignore them, but some do not (e.g. xfce4-terminal). So for the benefit
of the latter, filter them out.
2014-08-20 14:30:01 +02:00
Eelco Dolstra 029424d17d Make hook shutdown more reliable 2014-08-19 17:47:30 +02:00
Eelco Dolstra 02843ba4cb Fix --attr parsing 2014-08-18 15:48:23 +02:00
Eelco Dolstra 632f989b51 Doh 2014-08-18 11:35:50 +02:00
Eelco Dolstra c160ead82f Reduce verbosity 2014-08-17 19:11:50 +02:00
Eelco Dolstra 42e9ad8fd1 Propagate remote timeouts properly 2014-08-17 19:09:03 +02:00
Eelco Dolstra ad716378dc Fix download-via-ssh
http://hydra.nixos.org/build/13286020
2014-08-17 14:28:17 +02:00
Eelco Dolstra fde819f2e2 nix-log2xml: Handle newlines 2014-08-13 19:06:20 +02:00
Eelco Dolstra cb921f67c3 Remove log2html.xsl and friends
It's part of Hydra now.
2014-08-13 19:01:14 +02:00
Eelco Dolstra e9b609bf9a nix-log2xml: Handle UTF-8 characters
C++ chars can be negative...
2014-08-13 19:00:07 +02:00
Eelco Dolstra 9367046fef Use regular file GC roots if possible
This makes hydra-eval-jobs create roots as regular files. See
1c208f2b7e.
2014-08-13 17:44:41 +02:00
Eelco Dolstra 5f05197df4 Handle compound single dash options properly
So now

  nix-instantiate --eval -E '{x}: x' --argstr x -xyzzy

correctly prints "-xyzzy", rather than giving an error.

Issue NixOS/hydra#176.
2014-08-13 04:08:33 +02:00
Eelco Dolstra 47e185847e Refactor option handling 2014-08-13 03:50:44 +02:00
Eelco Dolstra 5bed74d1b0 Fix warning about non-existant -I directories 2014-08-13 02:57:59 +02:00
Eelco Dolstra 401ab50566 Remove pointless NIX_LOG_TYPE environment variable 2014-08-13 01:19:02 +02:00
Eelco Dolstra 98873ff7de Remove unnecessary call to addTempRoot() 2014-08-05 16:41:42 +02:00
Eelco Dolstra f02c52e3d4 Doh 2014-08-05 10:19:57 +02:00
Eelco Dolstra 988bf59421 Move some options out of globals 2014-08-04 18:13:14 +02:00
Eelco Dolstra daccd68999 Refactor 2014-08-04 18:02:29 +02:00
Eelco Dolstra d5a076c36f Add option ‘build-extra-chroot-dirs’
This is useful for extending (rather than overriding) the default set
of chroot paths.
2014-08-04 18:00:00 +02:00
Eelco Dolstra 4d73e2e893 Get rid of "killing <pid>" message for unused build hooks 2014-08-04 17:27:45 +02:00
Eelco Dolstra bb45092f72 Make chroot builds easier to set up
By default, we now include /bin/sh as a bind-mount of bash.
2014-08-04 17:09:26 +02:00
Eelco Dolstra 51485dcda2 Remove ugly hack for detecting build environment setup errors 2014-08-01 19:38:21 +02:00
Eelco Dolstra 4c4b219c07 Call commonChildInit() before doing chroot init
This ensures that daemon clients see error messages from the chroot
setup.
2014-08-01 19:29:03 +02:00
Eelco Dolstra eea0401d7a Eliminate redundant copy 2014-08-01 17:30:51 +02:00
Eelco Dolstra e0d7d0e45c findRoots(): Prevent a call to lstat()
This means that getting the roots from /nix/var/nix/.../hydra-roots
doesn't need any I/O other than reading the directory.
2014-08-01 17:20:25 +02:00
Eelco Dolstra daf3f2c11f Make readDirectory() return inode / file type 2014-08-01 17:14:28 +02:00
Eelco Dolstra 1c208f2b7e Allow regular files as GC roots
If a root is a regular file, then its name must denote a store
path. For instance, the existence of the file

  /nix/var/nix/gcroots/per-user/eelco/hydra-roots/wzc3cy1wwwd6d0dgxpa77ijr1yp50s6v-libxml2-2.7.7

would cause

  /nix/store/wzc3cy1wwwd6d0dgxpa77ijr1yp50s6v-libxml2-2.7.7

to be a root.

This is useful because it involves less I/O (no need for a readlink()
call) and takes up less disk space (the symlink target typically takes
up a full disk block, while directory entries are packed more
efficiently). This is particularly important for hydra.nixos.org,
which has hundreds of thousands of roots, and where reading the roots
can take 25 minutes.
2014-08-01 17:14:28 +02:00
Eelco Dolstra 50dc1f5b71 Restore default SIGPIPE handler before invoking ‘man’
Fixes NixOS/nixpkgs#3410.
2014-07-31 10:31:17 +02:00
Eelco Dolstra 3d221a7bb1 Rename nixPath to __nixPath
The name ‘nixPath’ breaks existing code.
2014-07-30 11:28:39 +02:00
Eelco Dolstra 67937907ca nix-daemon: Pass on the user's $SSH_AUTH_SOCK to the SSH substituter 2014-07-25 18:02:48 +02:00
Eelco Dolstra 71a20d4d95 Change the default for use-ssh-substituter to ‘true’
Now you only have to pass ‘--option ssh-substituter-hosts
nix-ssh@bla’ to enable SSH substitution.
2014-07-25 12:57:12 +02:00
Eelco Dolstra 1a44f1cb65 nix-copy-closure: Drop --bzip2, --xz, --show-progress
These are too difficult to implement via nix-store --serve.

‘--show-progress’ could be re-implemented fairly easily via a
sink/source wrapper class.
2014-07-24 17:11:54 +02:00
Eelco Dolstra 03103c0a36 Implement nix-copy-closure --from via nix-store --serve 2014-07-24 16:19:41 +02:00
Eelco Dolstra 62309a2c56 build-remote.pl: Be less verbose on failing builds 2014-07-24 12:43:59 +02:00
Eelco Dolstra d7a92ed4a9 nix-store --serve: Only monitor stdin during builds
Other operations cannot hang indefinitely (except when we're reading
from stdin, in which case we'll notice a client disconnect). But
monitoring works badly during compressed imports, since there the
client can close the connection before we've sent an ack.

http://hydra.nixos.org/build/12711638
2014-07-24 11:59:35 +02:00
Eelco Dolstra 0fae20c362 Use pthread_cancel instead of a signal
Signal handlers are process-wide, so sending SIGINT to the monitor
thread will cause the normal SIGINT handler to run. This sets the
isInterrupted flag, which is not what we want. So use pthread_cancel
instead.
2014-07-24 11:47:51 +02:00
Eelco Dolstra aa1560ca07 Fix bogus pass by reference
http://hydra.nixos.org/build/12711659
2014-07-24 09:58:07 +02:00
Eelco Dolstra 24c6d992c6 More debugging 2014-07-24 01:21:56 +02:00
Eelco Dolstra 1eb0af7ed5 Add some assertions 2014-07-24 00:16:06 +02:00
Eelco Dolstra d3c61d83be Remove some dead code 2014-07-24 00:00:53 +02:00
Eelco Dolstra 62ad3dfc43 Remove some obsolete files 2014-07-23 23:56:58 +02:00
Eelco Dolstra bd91453bb1 Pass -pthread only for programs that need it 2014-07-23 19:48:18 +02:00
Eelco Dolstra ece531d105 nix-daemon: Less verbosity 2014-07-23 19:43:46 +02:00
Eelco Dolstra d0fad1779f nix-daemon: Simplify stderr handling 2014-07-23 19:37:40 +02:00
Eelco Dolstra 78224cd871 nix-store --serve: Monitor for client disconnects
This is necessary because build-remote.pl now builds via ‘nix-store
--serve’. So if a build hangs without writing to stdout/stderr, and
the client disconnects, then we need to detect that.
2014-07-23 19:26:34 +02:00
Eelco Dolstra 49fe9592a4 nix-daemon: Use a thread instead of SIGPOLL to catch client disconnects
The thread calls poll() to wait until a HUP (or other error event)
happens on the client connection. If so, it sends SIGINT to the main
thread, which is then cleaned up normally. This is much nicer than
messing around with SIGPOLL.
2014-07-23 19:21:00 +02:00
Eelco Dolstra fdee1ced43 startProcess: Make writing error messages from the child more robust 2014-07-23 19:11:26 +02:00
Eelco Dolstra 5989966ed3 Remove dead code 2014-07-23 14:46:28 +02:00
Eelco Dolstra ee3c5d7916 Revert old useBuildHook behaviour 2014-07-19 02:25:47 +02:00
Eelco Dolstra 2e77bd70fa Better fix for strcasecmp on Darwin 2014-07-18 12:54:30 +02:00
Eelco Dolstra 8ddffe7aac Ugly hack to fix building on old Darwin
http://hydra.nixos.org/build/12580878
2014-07-17 23:57:17 +02:00
Eelco Dolstra 049c0eb49c nix-daemon: Add trusted-users and allowed-users options
‘trusted-users’ is a list of users and groups that have elevated
rights, such as the ability to specify binary caches. It defaults to
‘root’. A typical value would be ‘@wheel’ to specify all users in the
wheel group.

‘allowed-users’ is a list of users and groups that are allowed to
connect to the daemon. It defaults to ‘*’. A typical value would be
‘@users’ to specify the ‘users’ group.
2014-07-17 16:57:07 +02:00
Eelco Dolstra 0c730887c4 nix-daemon: Show name of connecting user 2014-07-17 15:49:33 +02:00
Eelco Dolstra 77c972c898 nix-daemon: Only print connection info if we have SO_PEERCRED 2014-07-17 15:44:51 +02:00
Eelco Dolstra 8f72e702a1 nix-daemon: Fix compat with older clients 2014-07-17 15:23:31 +02:00
Eelco Dolstra 2304a7dd21 Get rid of a compiler warning 2014-07-16 16:32:26 +02:00
Eelco Dolstra 985f1595fe Be more strict about file names in NARs 2014-07-16 16:30:50 +02:00
Eelco Dolstra 276a40b31f Handle case collisions on case-insensitive systems
When running NixOps under Mac OS X, we need to be able to import store
paths built on Linux into the local Nix store. However, HFS+ is
usually case-insensitive, so if there are directories with file names
that differ only in case, then importing will fail.

The solution is to add a suffix ("~nix~case~hack~<integer>") to
colliding files. For instance, if we have a directory containing
xt_CONNMARK.h and xt_connmark.h, then the latter will be renamed to
"xt_connmark.h~nix~case~hack~1". If a store path is dumped as a NAR,
the suffixes are removed. Thus, importing and exporting via a
case-insensitive Nix store is round-tripping. So when NixOps calls
nix-copy-closure to copy the path to a Linux machine, you get the
original file names back.

Closes #119.
2014-07-16 16:02:05 +02:00
Eelco Dolstra fa13d3f4f3 build-remote.pl: Fix building multiple output derivations
We were importing paths without sorting them topologically, leading to
"path is not valid" errors.

See e.g. http://hydra.nixos.org/build/12451761
2014-07-14 12:19:27 +02:00
Eelco Dolstra a5c6347ff0 build-remote.pl: Use ‘nix-store --serve’ on the remote side
This makes things more efficient (we don't need to use an SSH master
connection, and we only start a single remote process) and gets rid of
locking issues (the remote nix-store process will keep inputs and
outputs locked as long as they're needed).

It also makes it more or less secure to connect directly to the root
account on the build machine, using a forced command
(e.g. ‘command="nix-store --serve --write"’). This bypasses the Nix
daemon and is therefore more efficient.

Also, don't call nix-store to import the output paths.
2014-07-11 16:22:24 +02:00
Eelco Dolstra e196eecbe6 Allow $NIX_BUILD_HOOK to be relative to Nix libexec directory 2014-07-11 13:55:06 +02:00
Eelco Dolstra d0eb970fb4 Fix broken Pid constructor 2014-07-10 21:48:21 +02:00
Eelco Dolstra edbfe2232e Replace message "importing path <...>" with "exporting path <...>"
This causes nix-copy-closure to show what it's doing before rather
than after.
2014-07-10 21:30:22 +02:00
Eelco Dolstra 7bb632b024 nix-copy-closure -s: Do substitutions via ‘nix-store --serve’
This means we no longer need an SSH master connection, since we only
execute a single command on the remote host.
2014-07-10 20:43:04 +02:00
Eelco Dolstra 43b64f5038 Remove tabs 2014-07-10 17:32:21 +02:00
Eelco Dolstra 8e9140cfde Refactoring: Move all fork handling into a higher-order function
C++11 lambdas ftw.
2014-07-10 16:58:09 +02:00
Eelco Dolstra 1114c7bd57 nix-copy-closure: Restore compression and the progress viewer 2014-07-10 14:15:12 +02:00
Eelco Dolstra 7911e4c27a Remove maybeVfork 2014-07-10 13:35:44 +02:00
Eelco Dolstra 04170d06bf nix-copy-closure: Fix race condition
There is a long-standing race condition when copying a closure to a
remote machine, particularly affecting build-remote.pl: the client
first asks the remote machine which paths it already has, then copies
over the missing paths. If the garbage collector kicks in on the
remote machine between the first and second step, the already-present
paths may be deleted. The missing paths may then refer to deleted
paths, causing nix-copy-closure to fail. The client now performs both
steps using a single remote Nix call (using ‘nix-store --serve’),
locking all paths in the closure while querying.

I changed the --serve protocol a bit (getting rid of QueryCommand), so
this breaks the SSH substituter from older versions. But it was marked
experimental anyway.

Fixes #141.
2014-07-10 11:58:59 +02:00
Eelco Dolstra 2c3a8f787b Fix security hole in ‘nix-store --serve’
Since it didn't check that the path received from the client is a
store path, the client could dump any path in the file system.
2014-07-10 11:46:01 +02:00
Eelco Dolstra 0e5d0c1543 Fix compilation error on some versions of GCC
src/libexpr/primops.cc:42:8: error: looser throw specifier for 'virtual nix::InvalidPathError::~InvalidPathError()'
src/libexpr/nixexpr.hh:12:1: error:   overriding 'virtual nix::EvalError::~EvalError() noexcept (true)'

http://hydra.nixos.org/build/12385750
2014-07-09 12:14:40 +02:00
Eelco Dolstra beaf3e90af Add builtin function ‘fromJSON’
Fixes #294.
2014-07-04 13:34:15 +02:00
Eelco Dolstra e477f0e938 Style fix 2014-06-27 11:36:23 +02:00
Paul Colomiets 858b8f9760 Add --json argument to nix-instantiate 2014-06-27 11:23:40 +02:00
Eelco Dolstra 8504e7d604 allow-arbitrary-code-during-evaluation -> allow-unsafe-native-code-during-evaluation 2014-06-27 11:20:16 +02:00
Shea Levy d62f46e500 Only add the importNative primop if the allow-arbitrary-code-during-evaluation option is true (default false) 2014-06-24 10:50:03 -04:00
Shea Levy 5cd022d6c0 Add importNative primop
This can be used to import a dynamic shared object and return an
arbitrary value, including new primops. This can be used both to test
new primops without having to recompile nix every time, and to build
specialized primops that probably don't belong upstream (e.g. a function
that calls out to gpg to decrypt a nixops secret as-needed).

The imported function should initialize the Value & as needed. A single
import can define multiple values by creating an attrset or list, of
course.

An example initialization function might look like:

extern "C" void initialize(nix::EvalState & state, nix::Value & v)
{
    v.type = nix::tPrimOp;
    v.primOp = NEW nix::PrimOp(myFun, 1, state.symbols.create("myFun"));
}

Then `builtins.importNative ./example.so "initialize"` will evaluate to
the primop defined in the myFun function.
2014-06-17 12:08:01 -04:00
Eelco Dolstra 9d0709e8c4 Don't use member initialisers
They're a little bit too recent (only supported since GCC 4.7).

http://hydra.nixos.org/build/11851475
2014-06-12 17:30:37 +02:00
Eelco Dolstra 48495f67ed Fix bogus warnings about dumping large paths
Also, yay for C++11 non-static initialisers.
2014-06-12 13:15:35 +02:00
Eelco Dolstra 0960d674d4 Drop ImportError and FindError
We're not catching these anywhere.
2014-06-12 13:00:54 +02:00
Shea Levy 718f20da6d findFile: Realise the context of the path attributes 2014-06-12 12:57:14 +02:00
Shea Levy a8fb575c98 Share code between scopedImport and import
In addition to reducing duplication, this fixes both import from
derivation and import of derivation for scopedImport
2014-06-12 12:52:39 +02:00
Eelco Dolstra ee7fe64c0a == operator: Ignore string context
There really is no case I can think of where taking the context into
account is useful. Mostly it's just very inconvenient.
2014-06-10 14:02:56 +02:00
Eelco Dolstra b1beed97a0 Report daemon OOM better
When copying a large path causes the daemon to run out of memory, you
now get:

  error: Nix daemon out of memory

instead of:

  error: writing to file: Broken pipe
2014-06-10 13:45:50 +02:00
Eelco Dolstra 829af22759 Print a warning when loading a large path into memory
I.e. if you have a derivation with

  src = ./huge-directory;

you'll get a warning that this is not a good idea.
2014-06-10 13:30:09 +02:00
Eelco Dolstra 3c6b8a5215 nix-env -qa --json: Generate valid JSON even if there are invalid meta attrs 2014-06-02 17:58:43 +02:00
Eelco Dolstra becc2b0167 Sort nixPath attributes 2014-05-29 19:02:14 +02:00
Eelco Dolstra 54a34119f3 Use std::unordered_set 2014-05-26 17:53:17 +02:00
Eelco Dolstra d8c061e044 Remove ExprBuiltin
It's slower than ExprVar since it doesn't compute a static
displacement. Since we're not using the throw primop in the
implementation of <...> anymore, it's also not really needed.
2014-05-26 17:14:28 +02:00
Eelco Dolstra 62a6eeb1f3 Make the Nix search path declarative
Nix search path lookups like <nixpkgs> are now desugared to ‘findFile
nixPath <nixpkgs>’, where ‘findFile’ is a new primop. Thus you can
override the search path simply by saying

  let
    nixPath = [ { prefix = "nixpkgs"; path = "/my-nixpkgs"; } ];
  in ... <nixpkgs> ...

In conjunction with ‘scopedImport’ (commit
c273c15cb1), the Nix search path can be
propagated across imports, e.g.

  let

    overrides = {
      nixPath = [ ... ] ++ builtins.nixPath;
      import = fn: scopedImport overrides fn;
      scopedImport = attrs: fn: scopedImport (overrides // attrs) fn;
      builtins = builtins // overrides;
    };

  in scopedImport overrides ./nixos
2014-05-26 17:02:22 +02:00
Eelco Dolstra 39d72640c2 Ensure that -I flags get included in nixPath
Also fixes #261.
2014-05-26 16:52:31 +02:00
Eelco Dolstra a8edf185a9 Add constant ‘nixPath’
It contains the Nix expression search path as a list of { prefix, path
} sets, e.g.

  [ { path = "/nix/var/nix/profiles/per-user/root/channels/nixos"; prefix = ""; }
    { path = "/etc/nixos/configuration.nix"; prefix = "nixos-config"; }
    { path = "/home/eelco/Dev/nix/inst/share/nix/corepkgs"; prefix = "nix"; }
  ]
2014-05-26 14:55:47 +02:00
Eelco Dolstra c273c15cb1 Add primop ‘scopedImport’
‘scopedImport’ works like ‘import’, except that it takes a set of
attributes to be added to the lexical scope of the expression,
essentially extending or overriding the builtin variables.  For
instance, the expression

  scopedImport { x = 1; } ./foo.nix

where foo.nix contains ‘x’, will evaluate to 1.

This has a few applications:

* It allows getting rid of function argument specifications in package
  expressions. For instance, a package expression like:

    { stdenv, fetchurl, libfoo }:

    stdenv.mkDerivation { ... buildInputs = [ libfoo ]; }

  can now we written as just

    stdenv.mkDerivation { ... buildInputs = [ libfoo ]; }

  and imported in all-packages.nix as:

    bar = scopedImport pkgs ./bar.nix;

  So whereas we once had dependencies listed in three places
  (buildInputs, the function, and the call site), they now only need
  to appear in one place.

* It allows overriding builtin functions. For instance, to trace all
  calls to ‘map’:

  let
    overrides = {
      map = f: xs: builtins.trace "map called!" (map f xs);

      # Ensure that our override gets propagated by calls to
      # import/scopedImport.
      import = fn: scopedImport overrides fn;

      scopedImport = attrs: fn: scopedImport (overrides // attrs) fn;

      # Also update ‘builtins’.
      builtins = builtins // overrides;
    };
  in scopedImport overrides ./bla.nix

* Similarly, it allows extending the set of builtin functions. For
  instance, during Nixpkgs/NixOS evaluation, the Nixpkgs library
  functions could be added to the default scope.

There is a downside: calls to scopedImport are not memoized, unlike
import. So importing a file multiple times leads to multiple parsings
/ evaluations. It would be possible to construct the AST only once,
but that would require careful handling of variables/environments.
2014-05-26 14:26:29 +02:00
Eelco Dolstra f0fdbd0897 Shut up some signedness warnings 2014-05-26 12:34:15 +02:00
Eelco Dolstra 0321ef9bb2 Ugly hack to allow --argstr values starting with a dash
Fixes #265.
2014-05-23 14:43:58 +02:00
Eelco Dolstra 9f9080e2c0 nix-store -l: Fetch build logs from the Internet
If a build log is not available locally, then ‘nix-store -l’ will now
try to download it from the servers listed in the ‘log-servers’ option
in nix.conf. For instance, if you have:

  log-servers = http://hydra.nixos.org/log

then it will try to get logs from http://hydra.nixos.org/log/<base
name of the store path>. So you can do things like:

  $ nix-store -l $(which xterm)

and get a log even if xterm wasn't built locally.
2014-05-21 17:19:36 +02:00
Shea Levy eac5841970 Provide a more useful error message when a dynamic attr lookup fails 2014-05-15 17:56:24 +02:00
Eelco Dolstra 8d5f472f2c lvlInfo -> lvlTalkative 2014-05-15 11:37:44 +02:00
Eelco Dolstra 84813af5b9 nix-store --optimise: Remove bogus statistics 2014-05-15 11:33:46 +02:00
Eelco Dolstra 690adeb03d Remove tab 2014-05-15 11:19:16 +02:00
Eelco Dolstra a1b66f316e Merge branch 'master' of github.com:wmertens/nix 2014-05-15 11:18:29 +02:00
Wout Mertens 3b9ea8452f Shortcut store files before lstat
readdir() already returns the inode numbers, so we don't need to call
lstat to know if a file was already linked or not.
2014-05-15 09:02:22 +02:00
Wout Mertens d73ffc552f Use the inodes given by readdir directly 2014-05-14 22:52:10 +02:00
Wout Mertens e974f20c98 Preload linked hashes to speed up lookups
By preloading all inodes in the /nix/store/.links directory, we can
quickly determine of a hardlinked file was already linked to the hashed
links.
This is tolerant of removing the .links directory, it will simply
recalculate all hashes in the store.
2014-05-13 23:10:06 +02:00
Eelco Dolstra 95501c4dee nix-instantiate --eval: Apply auto-arguments if the result is a function
Fixes #254.
2014-05-13 12:56:48 +02:00
wmertens a84f503d86 Shortcut already-hardlinked files
If an inode in the Nix store has more than 1 link, it probably means that it was linked into .links/ by us. If so, skip.

There's a possibility that something else hardlinked the file, so it would be nice to be able to override this.

Also, by looking at the number of hardlinks for each of the files in .links/, you can get deduplication numbers and space savings.
2014-05-10 15:53:01 +02:00
Eelco Dolstra 696f960dee Set up directories and permissions for multi-user install automatically
This automatically creates /nix/var/nix/profiles/per-user and sets the
permissions/ownership on /nix/store to 1775 and root:nixbld.
2014-05-02 14:31:15 +02:00
Eelco Dolstra de4cdd0d47 Set build-max-jobs to the number of available cores by default
More zero configuration.
2014-05-02 12:51:43 +02:00
Eelco Dolstra ada3e3fa15 When running as root, use build users by default
This removes the need to have a nix.conf, and prevents people from
accidentally running Nix builds as root.
2014-05-02 12:46:03 +02:00
Ricardo M. Correia 700c678c2e nix-env: Minor change to '--delete-generations Nd' semantics
The option '--delete-generations Nd' deletes all generations older than N
days. However, most likely the user does not want to delete the
generation that was active N days ago.

For example, say that you have these 3 generations:

1: <30 days ago>
2: <15 days ago>
3: <1 hour ago>

If you do --delete-generations 7d (say, as part of a cron job), most
likely you still want to keep generation 2, i.e. the generation that was
active 7 days ago (and for most of the past 7 days, in fact).

This patch fixes this issue. Note that this also affects
'nix-collect-garbage --delete-older-than Nd'.

Thanks to @roconnor for noticing the issue!
2014-04-15 15:34:58 +02:00
Eelco Dolstra dfa2f77d2e If a .drv cannot be parsed, show its path
Otherwise you just get ‘expected string `Derive(['’ which isn't very helpful.
2014-04-08 19:24:29 +02:00
Eelco Dolstra 8e5fbf4d73 Show position info in attribute selection errors 2014-04-04 22:52:14 +02:00
Eelco Dolstra 4c5faad994 Show position info in Boolean operations 2014-04-04 22:43:52 +02:00
Eelco Dolstra bd9b1d97b4 Show position info in string concatenation / addition errors 2014-04-04 22:19:33 +02:00
Eelco Dolstra a5fe730940 forceString: Show position info 2014-04-04 21:14:11 +02:00
Eelco Dolstra 27b44b8cf7 forceAttrs: Show position info 2014-04-04 19:11:40 +02:00
Eelco Dolstra 96b695ccab forceList: Show position info 2014-04-04 19:05:36 +02:00
Eelco Dolstra b62d36963c forceInt: Show position info 2014-04-04 18:59:29 +02:00
Eelco Dolstra c28de6d96e Pass position information to primop calls
For example:

  error: `tail' called on an empty list, at
    /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/ex-2/default.nix:13:7
2014-04-04 18:59:29 +02:00
Eelco Dolstra 8b31ffd10d Remove unnecessary quotes around file names 2014-04-04 18:59:29 +02:00
Eelco Dolstra b72c8d2e5b Include position info in function application
This allows error messages like:

  error: the anonymous function at `/etc/nixos/configuration.nix:1:1'
    called without required argument `foo', at
    `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:77:59'
2014-04-04 18:59:29 +02:00
Danny Wilson ae6b631dc4 Fix compile errors on Illumos 2014-04-03 17:39:57 +02:00
Eelco Dolstra f0de86357c Tweak error message 2014-04-03 15:24:02 +02:00
Ludovic Courtès e7720aa10a Make sure /dev/pts/ptmx is world-writable
While running Python 3’s test suite, we noticed that on some systems
/dev/pts/ptmx is created with permissions 0 (that’s the case with my
Nixpkgs-originating 3.0.43 kernel, but someone with a Debian-originating
3.10-3 reported not having this problem.)

There’s still the problem that people without
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y are screwed (as noted in build.cc),
but I don’t see how we could work around it.
2014-04-03 13:42:45 +02:00
Eelco Dolstra ac6ceea764 Fix potential segfault
The newEnv variable was accessed (via the dynamicEnv) pointer after it
had gone out of scope.

Fixes #234.
2014-04-01 17:04:38 +02:00
Ricardo M. Correia 7ef7597f71 nix-env: Add support for --delete-generations 15d
It will delete all generations older than the specified number of days.
2014-03-30 00:54:16 +01:00
Eelco Dolstra 1c2550a2ae boost::shared_ptr -> std::shared_ptr 2014-03-30 00:49:23 +01:00
Eelco Dolstra 9becaa041f Drop pointless #include 2014-03-29 22:20:33 +01:00
Eelco Dolstra acb8facbbc Fix potential segfault in waitForInput()
Since the addition of build-max-log-size, a call to
handleChildOutput() can result in cancellation of a goal.  This
invalidated the "j" iterator in the waitForInput() loop, even though
it was still used afterwards.  Likewise for the maxSilentTime
handling.

Probably fixes #231.  At least it gets rid of the valgrind warnings.
2014-03-29 22:14:11 +01:00
Eelco Dolstra 90dc50b07c restoreSIGPIPE(): Fill in sa_mask
Issue #231.
2014-03-29 20:20:14 +01:00
Eelco Dolstra 49009573bc Don't interpret strings as format strings
Ludo reported this error:

  unexpected Nix daemon error: boost::too_few_args: format-string refered to more arguments than were passed

coming from this line:

  printMsg(lvlError, run.program + ": " + string(err, 0, p));

The problem here is that the string ends up implicitly converted to a
Boost format() object, so % characters are treated specially.  I
always assumed (wrongly) that strings are converted to a format object
that outputs the string as-is.

Since this assumption appears in several places that may be hard to
grep for, I've added some C++ type hackery to ensures that the right
thing happens.  So you don't have to worry about % in statements like

  printMsg(lvlError, "foo: " + s);

or

  throw Error("foo: " + s);
2014-03-28 16:59:26 +01:00
Ludovic Courtès 24cb65efc3 Make /dev/kvm optional
The daemon now creates /dev deterministically (thanks!).  However, it
expects /dev/kvm to be present.

The patch below restricts that requirement (1) to Linux-based systems,
and (2) to systems where /dev/kvm already exists.

I’m not sure about the way to handle (2).  We could special-case
/dev/kvm and create it (instead of bind-mounting it) in the chroot, so
it’s always available; however, it wouldn’t help much since most likely,
if /dev/kvm missing, then KVM support is missing.
2014-03-21 17:27:19 +01:00
Eelco Dolstra 3fc056927c Fix tabs 2014-03-18 23:23:55 +01:00
Ludovic Courtès 51800e06de Allow recovery from isValidPath RPCs with an invalid path
Currently, clients cannot recover from an isValidPath RPC with an
invalid path parameter because the daemon closes the connection when
that happens.

More precisely:

  1. in performOp, wopIsValidPath case, ‘readStorePath’ raises an
     ‘Error’ exception;

  2. that exception is caught by the handler in ‘processConnection’;

  3. the handler determines errorAllowed == false, and thus exits after
     sending the message.

This last part is fixed by calling ‘startWork’ early on, as in the patch
below.

The same reasoning could be applied to all the RPCs that take one or
more store paths as inputs, but isValidPath is, by definition, likely to
be passed invalid paths in the first place, so it’s important for this
one to allow recovery.
2014-03-18 23:22:49 +01:00
Eelco Dolstra f93e97517e Fix -j and other flags when using the daemon 2014-03-17 17:35:11 +01:00
Eelco Dolstra fb8d8f5428 Remove unnecessary null pointer checks
Fixes #225.
2014-03-12 14:42:25 +01:00
Eelco Dolstra 25386e5edc Fix passing meta attribute to buildenv.nix
Since the meta attributes were not sorted, attribute lookup could
fail, leading to package priorities and active flags not working
correctly.

Broken since 0f24400d90.
2014-03-11 17:34:02 +01:00
Shea Levy 049a379ec6 The expr of AttrNames/DynamicAttrDefs is always an ExprConcatStrings 2014-03-10 10:14:50 +01:00
Shea Levy 908e9ce259 If a dynamic attribute name evaluates to null, remove it from the set 2014-03-10 10:14:50 +01:00
Eelco Dolstra 2caab81660 Revert "Make ifs and asserts tail-recursive"
This reverts commit 273322c773.
2014-03-05 16:18:13 +01:00
Eelco Dolstra 1017bd68ea Set up a private /dev/pts in the chroot 2014-02-27 23:35:23 +01:00
Eelco Dolstra 3fd01b171a Set up a minimal /dev in chroots
Not bind-mounting the /dev from the host also solves the problem with
/dev/shm being a symlink to something not in the chroot.
2014-02-27 23:17:53 +01:00
Eelco Dolstra c9f6232304 Correctly detect infinite recursion in function application
If we're evaluating some application ‘v = f x’, we can't store ‘f’
temporarily in ‘v’, because if ‘f x’ refers to ‘v’, it will get ‘f’
rather than an infinite recursion error.

Unfortunately, this breaks the tail call optimisation introduced in
c897bac549.

Fixes #217.
2014-02-27 21:47:59 +01:00
Eelco Dolstra 29cde917fe Fix deadlock in SubstitutionGoal
We were relying on SubstitutionGoal's destructor releasing the lock,
but if a goal is a top-level goal, the destructor won't run in a
timely manner since its reference count won't drop to zero.  So
release it explicitly.

Fixes #178.
2014-02-27 13:34:13 +01:00
Eelco Dolstra 91f25f0510 And another one 2014-02-26 18:55:18 +01:00
Eelco Dolstra 432328cc55 Remove another unused function 2014-02-26 18:49:36 +01:00
Eelco Dolstra 509993e598 Remove unused function 2014-02-26 18:48:32 +01:00
Eelco Dolstra 6b9cd59a41 nix-store -r: Respect --add-root for non-derivations
Fixes #68.
Fixes #117.
2014-02-26 16:32:46 +01:00
Eelco Dolstra f14ef84a51 Warn about missing -I paths
Fixes #121.  Note that we don't warn about missing $NIX_PATH entries
because it's intended that some may be missing (cf. the default
$NIX_PATH on NixOS, which includes paths like /etc/nixos/nixpkgs for
backward compatibility).
2014-02-26 15:21:56 +01:00
Eelco Dolstra 42eb4afd7a Simplify getting use-ssh-substituter from untrusted users 2014-02-26 13:58:46 +01:00
Eelco Dolstra dcaea042fc Only start download-via-ssh if it's enabled 2014-02-26 13:41:03 +01:00
Shea Levy df5de9dfd7 Add use-ssh-substituter setting.
It defaults to false and can be overridden by RemoteStore.

Untested currently, just quickly put this together
2014-02-26 13:32:47 +01:00
Eelco Dolstra a897b58373 nix-instantiate: Allow --dry-run as a synonym for --readonly-mode
--dry-run is more consistent with nix-env and nix-store.
2014-02-19 16:46:33 +01:00
Eelco Dolstra e1cf40fa95 nix-instantiate: Rename --eval-only to --eval, --parse-only to --parse 2014-02-19 16:34:24 +01:00
Eelco Dolstra c31836008e nix-instantiate: Add a flag --expr / -E to read expressions from the command line
This is basically a shortcut for ‘echo 'expr...' | nix-instantiate -’.
Also supported by nix-build and nix-shell.
2014-02-19 16:30:19 +01:00
Shea Levy a7e70518b8 lexer-tab.o and parser-tab.o require each other's headers 2014-02-18 17:31:30 +01:00
Eelco Dolstra 8129cf33d9 Slight simplification 2014-02-18 10:46:30 +01:00
Eelco Dolstra 1aa19b24b2 Add a flag ‘--check’ to verify build determinism
The flag ‘--check’ to ‘nix-store -r’ or ‘nix-build’ will cause Nix to
redo the build of a derivation whose output paths are already valid.
If the new output differs from the original output, an error is
printed.  This makes it easier to test if a build is deterministic.
(Obviously this cannot catch all sources of non-determinism, but it
catches the most common one, namely the current time.)

For example:

  $ nix-build '<nixpkgs>' -A patchelf
  ...
  $ nix-build '<nixpkgs>' -A patchelf --check
  error: derivation `/nix/store/1ipvxsdnbhl1rw6siz6x92s7sc8nwkkb-patchelf-0.6' may not be deterministic: hash mismatch in output `/nix/store/4pc1dmw5xkwmc6q3gdc9i5nbjl4dkjpp-patchelf-0.6.drv'

The --check build fails if not all outputs are valid.  Thus the first
call to nix-build is necessary to ensure that all outputs are valid.

The current outputs are left untouched: the new outputs are either put
in a chroot or diverted to a different location in the store using
hash rewriting.
2014-02-18 01:01:14 +01:00
Eelco Dolstra b6def5b542 Make --repair work on Darwin
Mac OS X doesn't allow renaming a read-only directory.

http://hydra.nixos.org/build/9113895
2014-02-17 23:09:48 +01:00
Eelco Dolstra dfbcb7c403 Refactoring 2014-02-17 23:04:52 +01:00
Eelco Dolstra 71adb090f0 When using a build hook, only copy missing paths 2014-02-17 22:58:21 +01:00
Eelco Dolstra 69fe6c58fa Move some code around
In particular, do replacing of valid paths during repair later.  This
prevents us from replacing a valid path after the build fails.
2014-02-17 22:25:15 +01:00
Eelco Dolstra 1da6ae4f99 nix-store --gc --max-freed: Support a unit specifier
E.g. "--max-freed 10G" means "free ten gigabytes".
2014-02-17 14:48:50 +01:00
Eelco Dolstra 00d30496ca Heuristically detect if a build may have failed due to a full disk
This will allow Hydra to detect that a build should not be marked as
"permanently failed", allowing it to be retried later.
2014-02-17 14:15:56 +01:00
Eelco Dolstra a9d99ab55f download-via-ssh: Use readStorePath 2014-02-14 12:31:10 +01:00
Eelco Dolstra 4db572062c download-via-ssh: Show where we're downloading from 2014-02-14 12:20:12 +01:00
Eelco Dolstra dba33d4018 Minor style fixes 2014-02-14 11:48:42 +01:00
Shea Levy f67f52751f Indendation fix
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-12 07:33:07 -05:00
Shea Levy 62eb9eb76d Remove relic of old code
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-12 07:27:45 -05:00
Shea Levy 7438f0bc2b error messages start in lowercase
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-12 07:26:35 -05:00
Shea Levy 2246aa77d2 Remove using declarations from download-via-ssh
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-12 07:22:36 -05:00
Shea Levy c89d6b9b63 nix-store --serve: Use a versioned protocol
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 07:43:13 -05:00
Shea Levy 38c3beac1a Move StoreApi::serve into opServe
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:52:48 -05:00
Shea Levy 1614603165 Pass in params by const ref
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:49:37 -05:00
Shea Levy 78d979567f Clarify comment
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:43:29 -05:00
Shea Levy 64e23d0a38 Add download-via-ssh substituter
This substituter connects to a remote host, runs nix-store --serve
there, and then forwards substituter commands on to the remote host and
sends their results to the calling program. The ssh-substituter-hosts
option can be specified as a list of hosts to try.

This is an initial implementation and, while it works, it has some
limitations:

* Only the first host is used
* There is no caching of query results (all queries are sent to the
  remote machine)
* There is no informative output (such as progress bars)
* Some failure modes may cause unhelpful error messages
* There is no concept of trusted-ssh-substituter-hosts

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy 5671188eb2 nix-store --serve: Flush out after every loop
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy 73874629ef nix-store --serve: Use dump instead of export
Also remove signing support

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy 188f96500b nix-store --serve: Don't fail if asked for info about non-valid path
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Shea Levy 9488447594 nix-store --serve: Don't loop forever
nix-store --export takes a tmproot, which can only release by exiting.
Substituters don't currently work in a way that could take advantage of
the looping, anyway.

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:32 -05:00
Shea Levy 3a38d0f356 Add the nix-store --serve command
This is essentially the substituter API operating on the local store,
which will be used by the ssh substituter. It runs in a loop rather than
just taking one command so that in the future nix will be able to keep
one connection open for multiple instances of the substituter.

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:32 -05:00
Shea Levy 84a8b5e9af nix-instantiate --eval-only --read-write-mode: Don't depend on ordering
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-07 18:03:38 +01:00
Shea Levy e4058fab64 Rename --no-readonly-mode --read-write-mode
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-07 18:03:38 +01:00
Shea Levy 0c3e8a616e nix-instantiate: Add a --no-readonly-mode flag
This allows running nix-instantiate --eval-only without performing the
evaluation in readonly mode, letting features like import from
derivation and automatic substitution of builtins.storePath paths work.

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-07 18:03:38 +01:00
Eelco Dolstra d210cdc435 Fix assertion failure in ‘nix-store --load-db’
Namely:

  nix-store: derivations.cc:242: nix::Hash nix::hashDerivationModulo(nix::StoreAPI&, nix::Derivation): Assertion `store.isValidPath(i->first)' failed.

This happened because of the derivation output correctness check being
applied before the references of a derivation are valid.
2014-02-03 22:36:07 +01:00
Eelco Dolstra d6582c04c1 Give a friendly error message if the DB directory is not writable
Previously we would say "error: setting synchronous mode: unable to
open database file" which isn't very helpful.
2014-02-01 16:57:38 +01:00
Eelco Dolstra 844d83352c More "make dist" fixes 2014-02-01 15:18:48 +01:00
Eelco Dolstra 6ef32bddc1 Fix "make dist" 2014-02-01 14:38:12 +01:00
Eelco Dolstra 0c6d62cf27 Remove Automakefiles 2014-02-01 13:54:38 +01:00
Eelco Dolstra 16e7d69209 Update Makefile variable names 2014-02-01 13:54:38 +01:00
Eelco Dolstra e0234dfddc Rename Makefile -> local.mk 2014-01-30 12:11:06 +01:00
Eelco Dolstra 4a2ec9c659 Install nix-worker symlink 2014-01-30 12:08:26 +01:00
Eelco Dolstra 94f9c14d52 Fix some clang warnings 2014-01-21 18:29:55 +01:00
Eelco Dolstra 68cde53c47 Fix building against Bison 3.0.2 2014-01-21 15:34:04 +01:00
Eelco Dolstra 81628a6ccc Merge branch 'master' into make
Conflicts:
	src/libexpr/eval.cc
2014-01-21 15:30:01 +01:00
Eelco Dolstra 5ef8508a92 Remove unused type 2014-01-21 15:11:57 +01:00
Shea Levy f5e5793cd2 Bare dynamic attrs: Match interpolation semantics
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-01-14 14:00:15 +01:00
Shea Levy f9913f4422 Allow "bare" dynamic attrs
Now, in addition to a."${b}".c, you can write a.${b}.c (applicable
wherever dynamic attributes are valid).

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-01-14 14:00:15 +01:00
Eelco Dolstra b1db599dd0 Generate schema.sql.hh 2014-01-09 22:10:35 +01:00
Eelco Dolstra b4c684e0f9 Update Makefiles 2014-01-09 16:53:47 +01:00
Eelco Dolstra 11cb4bfb25 Fix checking of NAR hashes
*headdesk*
*headdesk*
*headdesk*

So since commit 22144afa8d, Nix hasn't
actually checked whether the content of a downloaded NAR matches the
hash specified in the manifest / NAR info file.  Urghhh...
2014-01-08 17:35:49 +01:00
Eelco Dolstra 7a61c88dbb Merge branch 'dynamic-attrs-no-sugar' of github.com:shlevy/nix 2014-01-06 15:46:18 +01:00
Domen Kožar 485f4740ee wording 2014-01-06 11:38:24 +01:00
Shea Levy cd49fe4f9b Don't use any syntactic sugar for dynamic attrs
This doesn't change any functionality but moves some behavior out of the
parser and into the evaluator in order to simplify the code.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-31 23:56:26 +00:00
Shea Levy 6f3a51809a Fold dynamic binds handling into addAttr
Since addAttr has to iterate through the AttrPath we pass it, it makes
more sense to just iterate through the AttrNames in addAttr instead. As
an added bonus, this allows attrsets where two dynamic attribute paths
have the same static leading part (see added test case for an example
that failed previously).

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-31 17:57:10 -05:00
Shea Levy 18fefacf7d Dynamic attrs
This adds new syntax for attribute names:

* attrs."${name}" => getAttr name attrs
* attrs ? "${name}" => isAttrs attrs && hasAttr attrs name
* attrs."${name}" or def => if attrs ? "${name}" then attrs."${name}" else def
* { "${name}" = value; } => listToAttrs [{ inherit name value; }]

Of course, it's a bit more complicated than that. The attribute chains
can be arbitrarily long and contain combinations of static and dynamic
parts (e.g. attrs."${foo}".bar."${baz}" or qux), which is relatively
straightforward for the getAttrs/hasAttrs cases but is more complex for
the listToAttrs case due to rules about duplicate attribute definitions.

For attribute sets with dynamic attribute names, duplicate static
attributes are detected at parse time while duplicate dynamic attributes
are detected when the attribute set is forced. So, for example, { a =
null; a.b = null; "${"c"}" = true; } will be a parse-time error, while
{ a = {}; "${"a"}".b = null; c = true; } will be an eval-time error
(technically that case could theoretically be detected at parse time,
but the general case would require full evaluation). Moreover, duplicate
dynamic attributes are not allowed even in cases where they would be
with static attributes ({ a.b.d = true; a.b.c = false; } is legal, but {
a."${"b"}".d = true; a."${"b"}".c = false; } is not). This restriction
might be relaxed in the future in cases where the static variant would
not be an error, but it is not obvious that that is desirable.

Finally, recursive attribute sets with dynamic attributes have the
static attributes in scope but not the dynamic ones. So rec { a = true;
"${"b"}" = a; } is equivalent to { a = true; b = true; } but rec {
"${"a"}" = true; b = a; } would be an error or use a from the
surrounding scope if it exists.

Note that the getAttr, getAttr or default, and hasAttr are all
implemented purely in the parser as syntactic sugar, while attribute
sets with dynamic attribute names required changes to the AST to be
implemented cleanly.

This is an alternative solution to and closes #167

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-31 20:59:49 +00:00
Shea Levy 136f2f7046 Add the ExprBuiltin Expr type to the AST
Certain desugaring schemes may require the parser to use some builtin
function to do some of the work (e.g. currently `throw` is used to
lazily cause an error if a `<>`-style path is not in the search path)
Unfortunately, these names are not reserved keywords, so an expression
that uses such a syntactic sugar will not see the expected behavior
(see tests/lang/eval-okay-redefine-builtin.nix for an example).

This adds the ExprBuiltin AST type, which when evaluated uses the value
from the rootmost variable scope (which of course is initialized
internally and can't shadow any of the builtins).

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-31 17:45:47 +00:00
Eelco Dolstra 8931bf7168 Doh 2013-12-20 13:09:12 +00:00
Eelco Dolstra 7d203faff6 nix-env --set-flag: Barf if a selector doesn't match any installed package
Fixes #184.
2013-12-20 13:56:42 +01:00
Eelco Dolstra 769f662165 nix-shell: Don't warn about the lack of a GC root 2013-12-20 12:19:10 +00:00
Eelco Dolstra 65a6452240 nix-shell: Handle --option correctly
Fixes #181.
2013-12-20 13:10:14 +01:00
Shea Levy e36229d27f Bump language version for new storePath feature
This will allow e.g. channel expressions to use builtins.storePath IFF
it is safe to do so without knowing if the path is valid yet.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-10 15:07:04 +01:00
Shea Levy 22d665019a builtins.storePath: Try to substitute the path if it is not yet valid
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-12-10 15:07:04 +01:00
Eelco Dolstra a6add93d73 Garbage collector: Release locks on temporary root files
This allows processes waiting for such locks to proceed during the
trash deletion phase of the garbage collector.
2013-12-10 13:13:59 +01:00
Eelco Dolstra c5b8fe3151 Print a trace message if a build fails due to the platform being unknown 2013-12-05 14:31:57 -05:00
Eelco Dolstra f3cf0436b5 Install bsdiff and bspatch in $(libexecdir)/nix 2013-12-02 20:21:31 +00:00
Eelco Dolstra 0202ce6b94 Add support for ‘make installcheck’ 2013-11-25 18:47:03 +01:00
Eelco Dolstra 7ce0e05ad8 Rename Makefile.new -> Makefile 2013-11-25 15:25:13 +00:00
Eelco Dolstra 1a1d8b073a Add a Makefile for bsdiff 2013-11-25 11:39:21 +00:00
Eelco Dolstra 2bd0fcc966 Use libnix as a prefix for all Nix libraries
In particular "libutil" was always a problem because it collides with
Glibc's libutil.  Even if we install into $(libdir)/nix, the linker
sometimes got confused (e.g. if a program links against libstore but
not libutil, then ld would report undefined symbols in libstore
because it was looking at Glibc's libutil).
2013-11-23 23:53:41 +00:00
Eelco Dolstra abb5bd66de Add Makefile for nix-log2xml 2013-11-23 23:42:50 +00:00
Eelco Dolstra c1f3a1a89b Disallow undefined symbols in dynamic libraries by default
This encourages that each library declares its own dependencies
properly.
2013-11-24 00:10:00 +01:00
Eelco Dolstra 5a1114ecdb Drop the dependency on libgc in libmain
Instead, libexpr now depends on libgc.  This means commands like
nix-store that don't do any evaluation no longer require libgc.
2013-11-23 20:19:36 +00:00
Eelco Dolstra 06a8ac96e7 Initialise Boehm GC only once 2013-11-23 20:15:57 +00:00
Eelco Dolstra 90dfb37f14 Allow (dynamic) libraries to depend on other libraries 2013-11-23 20:11:02 +00:00
Eelco Dolstra 611868a909 Implement basic ‘make install’ 2013-11-23 17:04:27 +00:00
Eelco Dolstra d1b3ca0b4a Improve building dynamic libraries
They now get a correct RPATH.
2013-11-23 17:26:43 +01:00
Eelco Dolstra 6dd74b48f8 Support building dynamic libraries 2013-11-22 20:56:58 +00:00
Eelco Dolstra 9b11a8bfbc Fix building without Boehm GC 2013-11-22 20:05:00 +00:00
Eelco Dolstra 6b5f89f2cf Drop the dependency on Automake 2013-11-22 19:30:24 +00:00
Eelco Dolstra 754c05ed6c Rename $(here) to $(d) for brevity, and remove trailing slash 2013-11-22 16:45:52 +00:00
Eelco Dolstra 62e35cc3a8 Add ‘make dist’ support 2013-11-22 16:42:25 +01:00
Eelco Dolstra eff6c4b791 Add missing #include 2013-11-22 15:41:48 +00:00
Eelco Dolstra 1474ecfe42 Generate the parser and the lexer 2013-11-22 16:22:31 +01:00
Eelco Dolstra b8e9efc476 New non-recursive, plain Make-based build system 2013-11-22 15:54:18 +01:00
Eelco Dolstra 709cbe4e76 Include <cstring> for memset
This should fix building on Illumos.
2013-11-22 10:00:43 +00:00
Eelco Dolstra 30b986908e Check meta values and warn about bad ones 2013-11-19 14:29:39 +01:00
Eelco Dolstra 0f24400d90 Generalise meta attributes 2013-11-19 14:09:14 +01:00
Eelco Dolstra 990126cde0 Shorter error message 2013-11-19 14:09:14 +01:00
Eelco Dolstra af94a70ba6 Drop support for user environment manifests in ATerm format 2013-11-19 14:09:14 +01:00
Eelco Dolstra 245e26408f nix-env -q: Add a --json flag 2013-11-19 00:41:45 +01:00
Eelco Dolstra 5fea98111b Refactor JSON output 2013-11-19 00:33:06 +01:00
Eelco Dolstra 77c13cdf56 Add a toJSON primop 2013-11-19 00:04:11 +01:00
Eelco Dolstra 285df765b9 Add a primop unsafeGetAttrPos to return the position of an attribute 2013-11-18 22:22:35 +01:00
Eelco Dolstra fc33fd86b7 Add a symbol __curPos that expands to the current source location
I.e. an attribute set { file = <string>; line = <int>; column = <int>; }.
2013-11-18 20:16:02 +01:00
Eelco Dolstra 90b5e69284 Support quoted attribute names in -A
This is requires if you have attribute names with dots in them.  So
you can now say:

  $ nix-instantiate '<nixos>' -A 'config.systemd.units."postgresql.service".text' --eval-only

Fixes #151.
2013-11-18 10:21:12 +00:00
Eelco Dolstra a478e8a7bb Remove nix-setuid-helper
AFAIK, nobody uses it, it's not maintained, and it has no tests.
2013-11-14 11:57:37 +01:00
Eelco Dolstra 89e6781cc5 Make function calls show up in stack traces again
Note that adding --show-trace prevents functions calls from being
tail-recursive, so an expression that evaluates without --show-trace
may fail with a stack overflow if --show-trace is given.
2013-11-12 12:51:59 +01:00
Eelco Dolstra c897bac549 Make function calls tail-recursive 2013-11-12 11:32:23 +00:00
Eelco Dolstra 273322c773 Make ifs and asserts tail-recursive
The local Value object prevented g++ from making a tail call.  Not
clear why.  In any case, not using a temporary makes g++ do the tail
call.
2013-11-12 11:32:23 +00:00
Eelco Dolstra 4badd7ed17 Get rid of an intermediary on the stack 2013-11-12 11:32:23 +00:00
Eelco Dolstra 8d6418d46e Fix building without a garbage collector
http://hydra.nixos.org/build/6695350
2013-10-28 22:51:12 +01:00
Eelco Dolstra dec2f19502 Fix a segfault in genericClosure
It kept temporary data in STL containers that were not scanned by
Boehm GC, so Nix programs using genericClosure could randomly crash if
the garbage collector kicked in at a bad time.

Also make it a bit more efficient by copying points to values rather
than values.
2013-10-28 18:52:26 +01:00
Eelco Dolstra ea6bf0c21f Slightly optimize listToAttrs 2013-10-28 07:34:44 +01:00
Eelco Dolstra 5bc41d78ff Rename "attribute sets" to "sets"
We don't have any other kind of sets so calling them attribute sets is
unnecessarily verbose.
2013-10-24 16:41:04 +02:00
Eelco Dolstra 69befd33a9 Remove unnecessary call to forceStringNoCtx 2013-10-24 03:08:34 +02:00
Eelco Dolstra 05d02f798f Add a typeOf primop
We already have some primops for determining the type of a value, such
as isString, but they're incomplete: for instance, there is no isPath.
Rather than adding more isBla functions, the generic typeOf function
returns a string representing the type of the argument (e.g. "int").
2013-10-24 02:49:13 +02:00
Eelco Dolstra 543d8a5942 Don't require NIX_SHOW_STATS for NIX_COUNT_CALLS 2013-10-24 02:20:54 +02:00
Eelco Dolstra fe95650487 Memoize evalFile() lookups under both the original and resolved name
Previously we only used the resolved name, causing repeated resolution
(e.g. /dir to /dir/default.nix).
2013-10-23 11:19:01 +00:00
Eelco Dolstra c086183843 For auto roots, show the intermediate link
I.e. "nix-store -q --roots" will now show (for example)

  /home/eelco/Dev/nixpkgs/result

rather than

  /nix/var/nix/gcroots/auto/53222qsppi12s2hkap8dm2lg8xhhyk6v
2013-10-22 11:39:10 +02:00
Eelco Dolstra 792fd51f41 Fold two stack trace messages in derivations
Combined with the previous changes, stack traces involving derivations
are now much less verbose, since something like

  while evaluating the builtin function `getAttr':
  while evaluating the builtin function `derivationStrict':
  while instantiating the derivation named `gtk+-2.24.20' at `/home/eelco/Dev/nixpkgs/pkgs/development/libraries/gtk+/2.x.nix:11:3':
  while evaluating the derivation attribute `propagatedNativeBuildInputs' at `/home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/default.nix:78:17':
  while evaluating the attribute `outPath' at `/nix/store/212ngf4ph63mp6p1np2bapkfikpakfv7-nix-1.6/share/nix/corepkgs/derivation.nix:18:9':
  ...

now reads

  while evaluating the attribute `propagatedNativeBuildInputs' of the derivation `gtk+-2.24.20' at `/home/eelco/Dev/nixpkgs/pkgs/development/libraries/gtk+/2.x.nix:11:3':
  ...
2013-10-17 11:57:25 +02:00
Eelco Dolstra f440558acc Don't show <nix/derivation.nix> in stack traces
Messages like

  while evaluating the attribute `outPath' at `/nix/store/212ngf4ph63mp6p1np2bapkfikpakfv7-nix-1.6/share/nix/corepkgs/derivation.nix:18:9':

are redundant, because Nix already shows that it's evaluating a derivation:

  while instantiating the derivation named `firefox-24.0' at `/home/eelco/Dev/nixpkgs/pkgs/applications/networking/browsers/firefox/default.nix:131:5':
  while evaluating the derivation attribute `nativeBuildInputs' at `/home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/default.nix:76:17':
2013-10-17 11:47:38 +02:00
Eelco Dolstra b8034e5581 Ensure proper type checking/coercion of "${expr}"
Now we only rewrite "${expr}" to expr if expr is a string literal.
2013-10-17 00:57:24 +02:00
Eelco Dolstra d6a7aa8f48 Revert the behaviour of antiquoted paths to pre-Nix 1.6
Commit 159e621d1a accidentally changed
the behaviour of antiquoted paths, e.g.

  "${/foo}/bar"

used to evaluate to "/nix/store/<hash>-foo/bar" (where /foo gets
copied to the store), but in Nix 1.6 it evaluates to "/foo/bar".  This
is inconsistent, since

  " ${/foo}/bar"

evaluates to " /nix/store/<hash>-foo/bar".  So revert to the old
behaviour.
2013-10-17 00:39:59 +02:00
Eelco Dolstra a737f51fd9 Retry all SQLite operations
To deal with SQLITE_PROTOCOL, we also need to retry read-only
operations.
2013-10-16 15:58:20 +02:00
Eelco Dolstra ff02f5336c Fix a race in registerFailedPath()
Registering the path as failed can fail if another process does the
same thing after the call to hasPathFailed().  This is extremely
unlikely though.
2013-10-16 14:55:53 +02:00
Eelco Dolstra 4bd5282573 Convenience macros for retrying a SQLite transaction 2013-10-16 14:46:35 +02:00
Eelco Dolstra bce14d0f61 Don't wrap read-only queries in a transaction
There is no risk of getting an inconsistent result here: if the ID
returned by queryValidPathId() is deleted from the database
concurrently, subsequent queries involving that ID will simply fail
(since IDs are never reused).
2013-10-16 14:36:53 +02:00
Eelco Dolstra 7cdefdbe73 Print a distinct warning for SQLITE_PROTOCOL 2013-10-16 14:27:36 +02:00
Eelco Dolstra d05bf04444 Treat SQLITE_PROTOCOL as SQLITE_BUSY
In the Hydra build farm we fairly regularly get SQLITE_PROTOCOL errors
(e.g., "querying path in database: locking protocol").  The docs for
this error code say that it "is returned if some other process is
messing with file locks and has violated the file locking protocol
that SQLite uses on its rollback journal files."  However, the SQLite
source code reveals that this error can also occur under high load:

  if( cnt>5 ){
    int nDelay = 1;                      /* Pause time in microseconds */
    if( cnt>100 ){
      VVA_ONLY( pWal->lockError = 1; )
      return SQLITE_PROTOCOL;
    }
    if( cnt>=10 ) nDelay = (cnt-9)*238;  /* Max delay 21ms. Total delay 996ms */
    sqlite3OsSleep(pWal->pVfs, nDelay);
  }

i.e. if certain locks cannot be not acquired, SQLite will retry a
number of times before giving up and returing SQLITE_PROTOCOL.  The
comments say:

  Circumstances that cause a RETRY should only last for the briefest
  instances of time.  No I/O or other system calls are done while the
  locks are held, so the locks should not be held for very long. But
  if we are unlucky, another process that is holding a lock might get
  paged out or take a page-fault that is time-consuming to resolve,
  during the few nanoseconds that it is holding the lock.  In that case,
  it might take longer than normal for the lock to free.
  ...
  The total delay time before giving up is less than 1 second.

On a heavily loaded machine like lucifer (the main Hydra server),
which often has dozens of processes waiting for I/O, it seems to me
that a page fault could easily take more than a second to resolve.
So, let's treat SQLITE_PROTOCOL as SQLITE_BUSY and retry the
transaction.

Issue NixOS/hydra#14.
2013-10-16 14:19:59 +02:00
Eelco Dolstra 7bdb85453d printStats(): Print the size of the symbol table in bytes 2013-10-08 15:37:08 +02:00
Eelco Dolstra 9deb822180 Deduplicate filenames in Pos
This saves ~4 MiB of RAM for NixOS system instantiation, and ~18 MiB
for "nix-env -qa".
2013-10-08 15:36:10 +02:00
Eelco Dolstra b1e3b1a4ac Treat undefined variable errors consistently
Previously, a undefined variable inside a "with" caused an EvalError
(which can be caught), while outside, it caused a ParseError (which
cannot be caught).  Now both cause an UndefinedVarError (which cannot
be caught).
2013-10-08 14:45:36 +02:00
Eelco Dolstra 6b47de580f Show the exact position of undefined variables
In particular, undefined variable errors in a "with" previously didn't
show *any* position information, so this should help a lot in those
cases.
2013-10-08 14:40:51 +02:00
Eelco Dolstra a5e0f64db3 Remove some unused functions 2013-10-08 12:30:23 +00:00
Eelco Dolstra 221a2daf34 Merge VarRef into ExprVar 2013-10-08 14:24:53 +02:00
Eelco Dolstra 176c666f36 Don't show calls to primops in stack traces
Since they don't have location information, they just give you crap
like:

  while evaluating the builtin function `getAttr':
  while evaluating the builtin function `derivationStrict':
  ...
2013-10-07 18:02:32 +02:00
Eelco Dolstra c945f015de Fix segfault in nix-repl / hydra-eval-jobs
If a "with" attribute set fails to evaluate, we have to make sure its
Env record remains unchanged.  Otherwise, repeated evaluation gives a
segfault:

  nix-repl> :a with 0; { a = x; b = x; }
  Added 2 variables.

  nix-repl> a
  error: value is an integer while an attribute set was expected

  nix-repl> b
  Segmentation fault
2013-10-02 15:24:45 +02:00
Eelco Dolstra 28e0742966 Report OOM errors better 2013-10-02 14:34:36 +02:00
Eelco Dolstra a5fb4b5b7c Fix typo 2013-10-02 14:22:49 +02:00
Eelco Dolstra f53574ebd6 RestoreSink: Slightly reduce the number of concurrent FDs 2013-09-17 12:06:59 +00:00
Eelco Dolstra 0220da3e10 Remove stray debug line 2013-09-06 17:20:19 +02:00
Eelco Dolstra 936f9d45ba Don't apply the CPU affinity hack to nix-shell (and other Perl programs)
As discovered by Todd Veldhuizen, the shell started by nix-shell has
its affinity set to a single CPU.  This is because nix-shell connects
to the Nix daemon, which causes the affinity hack to be applied.  So
we turn this off for Perl programs.
2013-09-06 16:36:56 +02:00
Eelco Dolstra 2c1ecf8e81 nix-env -i: Add a flag ‘--remove-all’ / ‘-r’
This is equivalent to running ‘nix-env -e '*'’ first, except that it
happens in a single transaction.  Thus, ‘nix-env -i pkgs...’ replaces
the profile with the specified set of packages.

The main motivation is to support declarative package management
(similar to environment.systemPackages in NixOS).  That is, if you
have a specification ‘profile.nix’ like this:

  with import <nixpkgs> {};
  [ thunderbird
    geeqie
    ...
  ]

then after any change to ‘profile.nix’, you can run:

  $ nix-env -f profile.nix -ir

to update the profile to match the specification.  (Without the ‘-r’
flag, if you remove a package from ‘profile.nix’, it won't be removed
from the actual profile.)

Suggested by @zefhemel.
2013-09-03 21:21:14 +02:00
Eelco Dolstra 88c07341a6 nix-env: Use wildcard match by default
That is, you don't need to pass '*' anymore, so

  nix-env -qa

is equivalent to

  nix-env -qa '*'
2013-09-03 16:35:49 +02:00
Eelco Dolstra 07a08bddf0 nix-env: Load files in ~/.nix-defexpr on demand
So if you do "nix-env -qa -A nixos", then other channels won't be
parsed/evaluated at all.
2013-09-03 15:45:32 +02:00
Eelco Dolstra c57ed84e28 Check for name collisions in the input Nix expressions 2013-09-03 15:25:51 +02:00
Eelco Dolstra ef4f5ba85e Work on Values instead of Exprs
This prevents some duplicate evaluation in nix-env and
nix-instantiate.

Also, when traversing ~/.nix-defexpr, only read regular files with the
extension .nix.  Previously it was reading files like
.../channels/binary-caches/<name>.  The only reason this didn't cause
problems is pure luck (namely, <name> shadows an actual Nix
expression, the binary-caches files happen to be syntactically valid
Nix expressions, and we iterate over the directory contents in just
the right order).
2013-09-03 13:17:51 +00:00
Eelco Dolstra 06bb2d95b4 Reformat 2013-09-03 11:04:21 +00:00
Eelco Dolstra 6f809194d7 Get rid of the parse tree cache
Since we already cache files in normal form (fileEvalCache), caching
parse trees is redundant.

Note that getting rid of this cache doesn't actually save much memory
at the moment, because parse trees are currently not freed / GC'ed.
2013-09-03 13:01:42 +02:00
Eelco Dolstra 57d18df7d0 Add some support code for nix-repl 2013-09-02 18:34:04 +02:00
Eelco Dolstra 92077b4547 Get rid of a signedness warning 2013-09-02 16:39:17 +02:00
Eelco Dolstra 33972629d7 Fix whitespace 2013-09-02 16:29:15 +02:00
Eelco Dolstra b29d3f4aee Only show trace messages when tracing is enabled 2013-09-02 12:01:04 +02:00
Eelco Dolstra efe4289464 Add an option to limit the log output of builders
This is mostly useful for Hydra to deal with builders that get stuck
in an infinite loop writing data to stdout/stderr.
2013-09-02 11:58:18 +02:00
Shea Levy afc6c1bad6 Simplify inherited attribute handling
This reduces the difference between inherited and non-inherited
attribute handling to the choice of which env to use (in recs and lets)
by setting the AttrDef::e to a new ExprVar in the parser rather than
carrying a separate AttrDef::v VarRef member.

As an added bonus, this allows inherited attributes that inherit from a
with to delay forcing evaluation of the with's attributes.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-08-26 11:31:56 +02:00
Ivan Kozik 34bb806f74 Fix typos, especially those that end up in the Nix manual 2013-08-26 11:15:22 +02:00
Gergely Risko c6c024ca6f Fix personality switching from x86_64 to i686
On Linux, Nix can build i686 packages even on x86_64 systems.  It's not
enough to recognize this situation by settings.thisSystem, we also have
to consult uname().  E.g. we can be running on a i686 Debian with an
amd64 kernel.  In that situation settings.thisSystem is i686-linux, but
we still need to change personality to i686 to make builds consistent.
2013-08-26 11:12:35 +02:00
Eelco Dolstra d308aeaf53 Store Nix integers as longs
So on 64-bit systems, integers are now 64-bit.

Fixes #158.
2013-08-19 12:35:03 +02:00
Eelco Dolstra 46222bbc43 Typo 2013-08-14 22:32:49 +02:00
Eelco Dolstra 3fb7ae0586 Respect MINSIGSTKSZ when allocating an alternative stack
http://hydra.nixos.org/build/5663577
2013-08-07 15:44:19 +00:00
Eelco Dolstra 161a2ccf7a Fix build on non-Linux
http://hydra.nixos.org/build/5662914
2013-08-07 17:26:41 +02:00
Eelco Dolstra a583a2bc59 Run the daemon worker on the same CPU as the client
On a system with multiple CPUs, running Nix operations through the
daemon is significantly slower than "direct" mode:

$ NIX_REMOTE= nix-instantiate '<nixos>' -A system
real    0m0.974s
user    0m0.875s
sys     0m0.088s

$ NIX_REMOTE=daemon nix-instantiate '<nixos>' -A system
real    0m2.118s
user    0m1.463s
sys     0m0.218s

The main reason seems to be that the client and the worker get moved
to a different CPU after every call to the worker.  This patch adds a
hack to lock them to the same CPU.  With this, the overhead of going
through the daemon is very small:

$ NIX_REMOTE=daemon nix-instantiate '<nixos>' -A system
real    0m1.074s
user    0m0.809s
sys     0m0.098s
2013-08-07 14:02:04 +02:00
Eelco Dolstra fd7d979c79 Remove obsolete reference to ATerms 2013-08-06 14:15:11 +02:00
Eelco Dolstra 46ffcce0c3 In the profiler output, show function names (if available) 2013-08-02 18:29:23 +00:00
Eelco Dolstra 8e74c0bfd1 Let the ordering operators also work on strings
E.g. ‘"foo" < "bar"’ now works.
2013-08-02 18:53:02 +02:00
Eelco Dolstra 3d77b28eac Add comparison operators ‘<’, ‘<=’, ‘>’ and ‘>=’ 2013-08-02 18:39:40 +02:00
Eelco Dolstra 47701677e8 Add integer ‘-’, ‘*’ and ‘/’ operators 2013-08-02 16:03:02 +00:00
Eelco Dolstra 5d147e125c Add a unary integer negation operator
This allows saying "-1" instead of "builtins.sub 0 1".
2013-08-02 15:43:31 +00:00
Eelco Dolstra 159e621d1a Overload the ‘+’ operator to support integer addition 2013-08-02 15:21:17 +00:00