Commit graph

1259 commits

Author SHA1 Message Date
Eelco Dolstra c179f668e5 Slight cleanup 2019-04-15 14:08:18 +02:00
Eelco Dolstra f6d684b5e2 getFlakeRegistries(): Return registries regardless of pureEval
This makes e.g. 'nix flake list' work.
2019-04-15 13:48:56 +02:00
Eelco Dolstra 84c12dbd7c Move --impure to MixEvalArgs 2019-04-15 13:45:51 +02:00
Eelco Dolstra bb6e6923f2 Add environment variable NIX_SHOW_SYMBOLS for dumping the symbol table 2019-04-11 23:04:13 +02:00
Nick Van den Broeck 4ad4e48668 FlakeRegistry = FlakeRef -> FlakeRef 2019-04-11 07:08:04 +02:00
Nick Van den Broeck c64f98b883 FlakeAlias is implemented 2019-04-09 10:16:02 +02:00
Nick Van den Broeck f39670c631 Took ref and rev out of FlakeRef 2019-04-09 09:51:42 +02:00
Nick Van den Broeck 3ec0c82fab Fixed dependency resolution 2019-04-09 09:51:42 +02:00
Nick Van den Broeck 641db127be FlakeIds are now properly looked up in registries 2019-04-09 09:51:36 +02:00
Nick Van den Broeck 18c019b616 Added nonFlakeRequires and the command nix flake deps 2019-04-09 08:38:09 +02:00
Eelco Dolstra c996e04aca Allow relative paths in flakerefs
Also allow "." as an installable to refer to the flake in the current
directory. E.g.

  $ nix build .

will build 'provides.defaultPackage' in the flake in the current
directory.
2019-04-08 23:52:09 +02:00
Eelco Dolstra ee1254d4f5 nix: Add --impure as a shorter alias of --no-pure-eval 2019-04-08 23:19:19 +02:00
Eelco Dolstra 6a4c7fb975 Add path flakeref variant
Unlike file://<path>, this allows the path to be a dirty Git tree, so

  nix build /path/to/flake:attr

is a convenient way to test building a local flake.
2019-04-08 23:09:18 +02:00
Eelco Dolstra 9b7eac332b Merge remote-tracking branch 'origin/master' into flakes 2019-04-08 14:23:48 +02:00
Eelco Dolstra 154244adc6 nix: New installables syntax
The general syntax for an installable is now
<flakeref>:<attrpath>. The attrpath is relative to the flake's
'provides.packages' or 'provides' if the former doesn't yield a
result. E.g.

  $ nix build nixpkgs:hello

is equivalent to

  $ nix build nixpkgs:packages.hello

Also, '<flakeref>:' can be omitted, in which case it defaults to
'nixpkgs', e.g.

  $ nix build hello
2019-04-08 14:21:13 +02:00
Eelco Dolstra 6b0ca8e803 findAlongAttrPath(): Throw AttrPathNotFound 2019-04-08 14:20:58 +02:00
Eelco Dolstra f32fbf952d Fix Bison 3.3 warning 2019-03-27 21:09:31 +01:00
Eelco Dolstra c101b29133 Merge remote-tracking branch 'tweag/flake-registries' into flakes 2019-03-27 14:40:36 +01:00
Eelco Dolstra f9c7176a87 nix flake add: Handle ~/.config/nix not existing
Fixes

  $ nix flake add fnord github:edolstra/fnord
  error: opening file '/home/eelco/.config/nix/registry.json': No such file or directory
2019-03-26 12:48:57 +01:00
Eelco Dolstra 42be60c6af Merge remote-tracking branch 'tweag/flake-add' into flakes 2019-03-26 12:35:27 +01:00
Eelco Dolstra edb3836696 Merge remote-tracking branch 'tweag/auto-update-flake' into flakes 2019-03-26 11:50:22 +01:00
Eelco Dolstra b5565a7081 Merge remote-tracking branch 'origin/master' into flakes 2019-03-26 11:44:14 +01:00
Jonas Chevalier 514b3c7f83
Add isPath primop
this is added for completeness' sake since all the other possible
`builtins.typeOf` results have a corresponding `builtins.is<Type>`
2019-03-24 11:36:49 +01:00
Nick Van den Broeck a554f523db Combining registries properly 2019-03-23 13:42:48 +01:00
Nick Van den Broeck 5e4d92d267 Issue #15 is finished 2019-03-22 15:22:18 +01:00
Nick Van den Broeck e007f367bd Fixed minor things 2019-03-22 11:21:49 +01:00
Nick Van den Broeck d4ee8afd59 Implemented --flake flag for nix build
Also fixed Eelco's PR comments
2019-03-22 11:16:20 +01:00
Linus Heckemann 2aa89daab3 eval: improve type description for primops and applied primops
This can make type errors a little easier to understand.
2019-03-21 15:31:46 +01:00
Eelco Dolstra 1f64f4c7c8
pkg-config files: Use c++17 2019-03-14 14:11:12 +01:00
Eelco Dolstra ef52ccf035
experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Nick Van den Broeck 6542de98c2 Implemented writeRegistry 2019-03-07 12:00:10 +01:00
Nick Van den Broeck 9ff1a9ea65 Implemented json flag for nix flake info 2019-03-07 11:56:43 +01:00
Nick Van den Broeck cfb6ab80ce Implemented "nix flake info" 2019-02-26 13:12:51 +01:00
Eelco Dolstra d342de02b9
fetchFlake: Use infinite TTL when the revision is specified 2019-02-25 23:23:45 +08:00
Eelco Dolstra 529add316c
downloadCached: Return ETag
This allows fetchFlake() to return the Git revision of a GitHub
archive.
2019-02-25 23:20:50 +08:00
Eelco Dolstra 6e9182fbc2
Add basic flake lock file support 2019-02-12 22:43:22 +01:00
Eelco Dolstra e38ec77de8
Interpret all file:// URIs as Git repositories 2019-02-12 22:06:19 +01:00
Eelco Dolstra beab05851b
nix: Add --flake flag
This allows using an arbitrary "provides" attribute from the specified
flake. For example:

  nix build --flake nixpkgs packages.hello

(Maybe provides.packages should be used for consistency...)
2019-02-12 21:55:43 +01:00
Eelco Dolstra 272b58220d
Enforce use of immutable flakes in pure mode
... plus a temporary hack to allow impure flakes at top-level for the
default installation source.
2019-02-12 21:07:48 +01:00
Eelco Dolstra ba05f29838
nix: Enable pure mode by default
We want to encourage a brave new world of hermetic evaluation for
source-level reproducibility, so flakes should not poke around in the
filesystem outside of their explicit dependencies.

Note that the default installation source remains impure in that it
can refer to mutable flakes, so "nix build nixpkgs.hello" still works
(and fetches the latest nixpkgs, unless it has been pinned by the
user).

A problem with pure evaluation is that builtins.currentSystem is
unavailable. For the moment, I've hard-coded "x86_64-linux" in the
nixpkgs flake. Eventually, "system" should be a flake function
argument.
2019-02-12 20:35:03 +01:00
Eelco Dolstra 91a6a47b0e
Improve flake references 2019-02-12 18:23:11 +01:00
Eelco Dolstra 0cd7f2cd8d
pkg-config files: Use c++17 2019-02-12 13:44:20 +01:00
Eelco Dolstra c8a0b9d5cb
experimental/optional -> optional 2019-02-12 13:43:32 +01:00
Eelco Dolstra 52419f8db3
Add flake registry
This will eventually be moved to nixos.org.
2019-02-11 13:01:39 +01:00
Eelco Dolstra dcae46ab14
Add github URIs
For example, github:edolstra/dwarffs is more-or-less equivalent to
https://github.com/edolstra/dwarffs.git. It's a much faster way to get
GitHub repositories: it fetches tarballs rather than entire Git
repositories. It also allows fetching specific revisions by hash
without specifying a ref (e.g. a branch name):

  github:edolstra/dwarffs/41c0c1bf292ea3ac3858ff393b49ca1123dbd553
2019-02-11 12:05:17 +01:00
Eelco Dolstra ef4cf4e681
Introduce flake URIs 2019-02-11 12:05:13 +01:00
Eelco Dolstra 7a5cf31060
Initial flake support 2019-02-11 12:00:13 +01:00
Eelco Dolstra f216c76c56
Bindings::get(): Add convenience method
This allows writing attribute lookups as

    if (auto name = value.attrs->get(state.sName))
      ...
2019-02-11 11:55:35 +01:00
Eelco Dolstra c02da99757
EvalState::allocAttr(): Add convenience method 2019-02-11 11:55:29 +01:00
Eelco Dolstra aa0e2a2e70
Make constant primops lazy 2019-02-11 11:52:07 +01:00
Shea Levy b30be6b450
Add builtins.appendContext.
A partner of builtins.getContext, useful for the same reasons.
2019-01-31 08:52:23 -05:00
Shea Levy 1d757292d0
Add builtins.getContext.
This can be very helpful when debugging, as well as enabling complex
black magic like surgically removing a single dependency from a
string's context.
2019-01-14 11:27:10 -05:00
Shea Levy 087be7281a
Treat plain derivation paths in context as normal paths.
Previously, plain derivation paths in the string context (e.g. those
that arose from builtins.storePath on a drv file, not those that arose
from accessing .drvPath of a derivation) were treated somewhat like
derivaiton paths derived from .drvPath, except their dependencies
weren't recursively added to the input set. With this change, such
plain derivation paths are simply treated as paths and added to the
source inputs set accordingly, simplifying context handling code and
removing the inconsistency. If drvPath-like behavior is desired, the
.drv file can be imported and then .drvPath can be accessed.

This is a backwards-incompatibility, but storePath is never used on
drv files within nixpkgs and almost never used elsewhere.
2019-01-13 11:29:55 -05:00
Eelco Dolstra c7bf1cdb4e
Merge pull request #2608 from dtzWill/fix/issue-2546
EvalState::resetFileCache: clear parse cache as well as eval cache
2019-01-10 20:56:31 +01:00
John Ericson fef9f5653b Remove mentions of libformat, it no longer exists 2019-01-05 14:31:29 -05:00
Will Dietz 21ea00d3ec EvalState::resetFileCache: clear parse cache as well as eval cache
Fixes #2546.

(at least the basic reproduction I've been testing)
2018-12-31 10:18:28 -06:00
Daiderd Jordan 7e35e914c1
fetchGit: allow fetching explicit refs
Trying to fetch refs that are not in refs/heads currently fails because
it looks for refs/heads/refs/foo instead of refs/foo.

eg.

	builtins.fetchGit {
	  url = https://github.com/NixOS/nixpkgs.git;
	  ref = "refs/pull/1024/head;
	}
2018-12-14 20:12:22 +01:00
Eelco Dolstra 6024dc1d97
Support SRI hashes
SRI hashes (https://www.w3.org/TR/SRI/) combine the hash algorithm and
a base-64 hash. This allows more concise and standard hash
specifications. For example, instead of

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    sha256 = "5d22dad058d5c800d65a115f919da22938c50dd6ba98c5e3a183172d149840a4";
  };

you can write

  import <nix/fetchurl.nl> {
    url = https://nixos.org/releases/nix/nix-2.1.3/nix-2.1.3.tar.xz;
    hash = "sha256-XSLa0FjVyADWWhFfkZ2iKTjFDda6mMXjoYMXLRSYQKQ=";
  };

In fixed-output derivations, the outputHashAlgo is no longer mandatory
if outputHash specifies the hash (either as an SRI or in the old
"<type>:<hash>" format).

'nix hash-{file,path}' now print hashes in SRI format by default. I
also reverted them to use SHA-256 by default because that's what we're
using most of the time in Nixpkgs.

Suggested by @zimbatm.
2018-12-13 14:30:52 +01:00
CHEIKH Chawki fa5143c722 Solve hg "abandoned transaction" issue 2018-12-06 13:57:59 +01:00
Eelco Dolstra 4aee93d5ce
fetchGit: Drop unnecessary localRef 2018-11-20 20:59:44 +01:00
Eelco Dolstra 3f4de91d80
Merge branch 'better-git-cache' of https://github.com/graham-at-target/nix 2018-11-20 20:41:19 +01:00
Jan Path d1b049c4ea Fix typo in comments 2018-10-31 20:50:18 +01:00
Guillaume Maudoux 6a5bf9b143 simplify handling of extra '}' 2018-10-27 00:14:51 +02:00
Eelco Dolstra 8327a7a8fa
Merge branch 'dirOf-relative' of https://github.com/lheckemann/nix 2018-09-13 14:33:12 +02:00
Eelco Dolstra 91405986f4
Convert NIX_COUNT_CALLS to JSON too 2018-09-05 21:57:54 +02:00
Eelco Dolstra 0a2545f95c
Log stats to stderr
We shouldn't pollute stdout.
2018-09-05 21:35:58 +02:00
Michael Bishop 4b034f390c remove the old text format output 2018-09-02 18:25:23 -03:00
Michael Bishop 2fd1008c70 add JSON to NIX_SHOW_STATS 2018-09-01 20:05:06 -03:00
Michael Bishop c29e5fbb13 improve the stats when profiling 2018-09-01 17:11:56 -03:00
Eelco Dolstra 475a0a54a9
fetchGit/fetchMercurial: Don't absolutize paths
This is already done by coerceToString(), provided that the argument
is a path (e.g. 'fetchGit ./bla'). It fixes the handling of URLs like
git@github.com:owner/repo.git. It breaks 'fetchGit "./bla"', but that
was never intended to work anyway and is inconsistent with other
builtin functions (e.g. 'readFile "./bla"' fails).
2018-09-01 00:19:49 +02:00
aszlig 0ad643ed5c
libexpr: Use int64_t for NixInt
Using a 64bit integer on 32bit systems will come with a bit of a
performance overhead, but given that Nix doesn't use a lot of integers
compared to other types, I think the overhead is negligible also
considering that 32bit systems are in decline.

The biggest advantage however is that when we use a consistent integer
size across all platforms it's less likely that we miss things that we
break due to that. One example would be:

https://github.com/NixOS/nixpkgs/pull/44233

On Hydra it will evaluate, because the evaluator runs on a 64bit
machine, but when evaluating the same on a 32bit machine it will fail,
so using 64bit integers should make that consistent.

While the change of the type in value.hh is rather easy to do, we have a
few more options available for doing the conversion in the lexer:

  * Via an #ifdef on the architecture and using strtol() or strtoll()
    accordingly depending on which architecture we are. For the #ifdef
    we would need another AX_COMPILE_CHECK_SIZEOF in configure.ac.
  * Using istringstream, which would involve copying the value.
  * As we're already using boost, lexical_cast might be a good idea.

Spoiler: I went for the latter, first of all because lexical_cast does
have an overload for const char* and second of all, because it doesn't
involve copying around the input string. Also, because istringstream
seems to come with a bigger overhead than boost::lexical_cast:

https://www.boost.org/doc/libs/release/doc/html/boost_lexical_cast/performance.html

The first method (still using strtol/strtoll) also wasn't something I
pursued further, because it is also locale-aware which I doubt is what
we want, given that the regex for int is [0-9]+.

Signed-off-by: aszlig <aszlig@nix.build>
Fixes: #2339
2018-08-29 01:05:52 +02:00
Eelco Dolstra c651b7bdc9
Revert "Fix parser/lexer generation with parallel make"
This reverts commit d277442df5.

Make sucks.
2018-08-23 00:23:35 +02:00
Eelco Dolstra 9b1bdf2db8
FIx floating point evaluation
Fixes #2361.
2018-08-19 11:59:49 +02:00
Graham Christensen 02098d2073 fetchGit: use a better caching scheme
The current usage technically works by putting multiple different
repos in to the same git directory. However, it is very slow as
Git tries very hard to find common commits between the two
repositories. If the two repositories are large (like Nixpkgs and
another long-running project,) it is maddeningly slow.

This change busts the cache for existing deployments, but users
will be promptly repaid in per-repository performance.
2018-08-17 11:27:34 -04:00
Eelco Dolstra d277442df5
Fix parser/lexer generation with parallel make
Fun fact: rules with multiple targets don't work properly with 'make
-j'. For example, a rule like

  a b: c
    touch a b

is equivalent to

  a: c
    touch a b

  b: c
    touch a b

so with 'make -j', the 'touch' command will be run twice. See
e.g. https://stackoverflow.com/questions/2973445/gnu-makefile-rule-generating-a-few-targets-from-a-single-source-file.
2018-08-17 12:59:23 +02:00
Linus Heckemann d7402c9cd5 dirOf: allow use on non-absolute paths 2018-08-13 11:27:35 +02:00
Eelco Dolstra bc65e02d96
Merge pull request #2326 from aszlig/fix-symlink-leak
Fix symlink leak in restricted eval mode
2018-08-03 17:01:34 +02:00
aszlig 43e28a1b75
Fix symlink leak in restricted eval mode
In EvalState::checkSourcePath, the path is checked against the list of
allowed paths first and later it's checked again *after* resolving
symlinks.

The resolving of the symlinks is done via canonPath, which also strips
out "../" and "./". However after the canonicalisation the error message
pointing out that the path is not allowed prints the symlink target in
the error message.

Even if we'd suppress the message, symlink targets could still be leaked
if the symlink target doesn't exist (in this case the error is thrown in
canonPath).

So instead, we now do canonPath() without symlink resolving first before
even checking against the list of allowed paths and then later do the
symlink resolving and checking the allowed paths again.

The first call to canonPath() should get rid of all the "../" and "./",
so in theory the only way to leak a symlink if the attacker is able to
put a symlink in one of the paths allowed by restricted evaluation mode.

For the latter I don't think this is part of the threat model, because
if the attacker can write to that path, the attack vector is even
larger.

Signed-off-by: aszlig <aszlig@nix.build>
2018-08-03 06:46:43 +02:00
Eelco Dolstra 45bcf5416a
Merge branch 'prim_mapAttr-fix' of https://github.com/volth/nix 2018-07-31 20:05:07 +02:00
Eelco Dolstra a7fb7d3cde
Merge pull request #2303 from volth/patch-4
parser.y: fix assoc of -> and < > <= >=
2018-07-23 11:38:15 +02:00
volth deaa6e9a34
parser.y: right-associativity of -> 2018-07-23 07:28:48 +00:00
volth 85fe4a819c
parser.y: fix assoc of -> and < > <= >=
The parser allowed senseless `a > b > c` but disallowed `a -> b -> c` which seems valid
It might be a typo
2018-07-21 15:24:51 +00:00
volth e2b114cfe1
prim_foldlStrict: call forceValue() before value is copied
forceValue() were called after a value is copied effectively forcing only one of the copies keeping another copy not evaluated.
This resulted in its evaluation of the same lazy value more than once (the number of hits is not big though)
2018-07-21 06:44:42 +00:00
Eelco Dolstra c2de2ff385
Fix build on 32-bit systems and macOS
Apparently, on macOS, 'long' != 'int64_t'.

https://hydra.nixos.org/build/77100756
2018-07-11 21:12:18 +02:00
Eelco Dolstra efd04888ca
Shup up a warning 2018-07-11 21:05:09 +02:00
Eelco Dolstra 875cd9da2b
Remove unused function printStats2()
Closes #2282.
2018-07-11 20:29:18 +02:00
volth c03d73c1cf
prim_map, prim_genList: no need to force function argument which will be stored in mkApp 2018-07-07 23:30:17 +00:00
volth 627e28ba33 prim_mapAttrs: `f' must be evaluated lazily to avoid infinite recursion 2018-07-06 21:52:54 +00:00
volth 1515c65616 prim_concatMap: no need to force value 2018-07-05 15:33:33 +00:00
volth e6bf1a79d7 prim_mapAttrs: must be lazy to avoid infinite recursion 2018-07-05 15:33:12 +00:00
volth 841747b0e6
prim_concatMap: allocate intermediate list on stack 2018-07-05 12:37:37 +00:00
volth ee218f99ca
primops.cc: fix comment 2018-07-05 11:58:15 +00:00
volth 403a76a18f lib.concatMap and lib.mapAttrs to be builtins 2018-07-05 02:54:09 +00:00
Eelco Dolstra ea3c9dab5f
Include cpptoml for build simplicity 2018-07-03 18:39:36 +02:00
Eelco Dolstra 3b1f54cf06
Add a fromTOML primop
This is primarily useful for processing Cargo.lock files.
2018-07-03 18:39:36 +02:00
Eelco Dolstra a92ed973e5
Store floating point numbers in double precision
Even on 32-bit systems, Value has enough space to hold a double.
2018-07-03 18:39:32 +02:00
Eelco Dolstra c905d8b0a8
GC_malloc -> GC_MALLOC
This makes it possible to build with -DGC_DEBUG.
2018-06-12 17:49:55 +02:00
Eelco Dolstra 455d1f01d0
Don't scan for roots in dynamic libraries
This reduces the risk of object liveness misdetection. For example,
Glibc has an internal variable "mp_" that often points to a Boehm
object, keeping it alive unnecessarily. Since we don't store any
actual roots in global variables, we can just disable data segment
scanning.

With this, the max RSS doing 100 evaluations of
nixos.tests.firefox.x86_64-linux.drvPath went from 718 MiB to 455 MiB.
2018-06-12 17:49:55 +02:00
Eelco Dolstra 30964103dc
Add temporary stats 2018-06-12 17:49:55 +02:00
Eelco Dolstra 24c6806994
Cache parse trees
This prevents EvalState::resetFileCache() from parsing everything all
over again.
2018-06-12 17:49:55 +02:00
Eelco Dolstra 6ad0a2f749
Remove duplicate definition of allocBytes() 2018-06-12 17:49:51 +02:00
Eelco Dolstra 5d4a9d5677
Fix static assertion failure on 32-bit systems 2018-05-30 17:47:30 +02:00
Eelco Dolstra 1672bcd230
Move evaluator-specific settings out of libstore 2018-05-30 13:29:50 +02:00
Eelco Dolstra bd56b5fe3f
Merge pull request #2187 from bgamari/stoi-exceptions
json-to-value: Use strtol instead of strtoi
2018-05-28 14:54:49 +02:00
Ben Gamari 7f560b81ea json-to-value: Use strtol instead of strtoi
NixInt is long, so strtoi is too restrictive.
2018-05-26 18:43:46 -04:00
Will Dietz 93ae90de0f eval.cc: add message to static_assert, message can be omitted w/c++17 2018-05-26 14:04:41 -05:00
Eelco Dolstra 743359bc8a
Merge pull request #2157 from volth/bitwise
add builtins: __bitAnd,  __bitOr,  __bitXor
2018-05-24 15:00:39 +02:00
Eelco Dolstra 4bb8741b98
Make Env self-describing
If the Env denotes a 'with', then values[0] may be an Expr* cast to a
Value*. For code that generically traverses Values/Envs, it's useful
to know this.
2018-05-22 16:02:32 +02:00
Eelco Dolstra 9fd7cf98db
Memoise checkSourcePath()
This prevents hydra-eval-jobs from statting the same files over and
over again.
2018-05-22 13:02:14 +02:00
volth 6cc28c0589
add `mod' and bitwise builtins: camel-case function names 2018-05-16 10:52:19 +00:00
volth f3c090f91c
add mod' and bitwise builtins: remove mod' and shifts 2018-05-16 06:57:11 +00:00
volth 49b7cf1813
add `mod' and bitwise builtins: remove infix functions 2018-05-16 06:55:24 +00:00
volth 8a6a14e1f5 add `mod' and bitwise builtins 2018-05-12 16:50:39 +00:00
Eelco Dolstra 1ad19232c4
Don't return negative numbers from the flex tokenizer
Fixes #1374.
Closes #2129.
2018-05-11 12:05:12 +02:00
Eelco Dolstra f3c85f9eb3
Revert "Throw a specific error for incomplete parse errors."
This reverts commit 6498adb002. We don't
actually use IncompleteParseError in 'nix repl'.
2018-05-11 11:40:50 +02:00
Eelco Dolstra a91c4ca01f
In restricted eval mode, allow access to the closure of store paths
E.g. this makes

  nix eval --restrict-eval -I /nix/store/foo '(builtins.readFile "/nix/store/foo/symlink/bla")'

(where /nix/store/foo/symlink is a symlink to another path in the
closure of /nix/store/foo) succeed.

This fixes a regression in Hydra compared to Nix 1.x (where there were
no restrictions at all on access to the Nix store).
2018-05-09 15:45:05 +02:00
Eelco Dolstra 53ec5ac69f
Fix some random -Wconversion warnings 2018-05-02 13:56:34 +02:00
Shea Levy e2b028353b
Merge branch 'pos-crash-fix' of git://github.com/dezgeg/nix 2018-04-23 08:43:54 -04:00
Félix Baylac-Jacqué 00584bb091
parser: Allow mixed nested and top-level attrpaths
Fixes #2077.
2018-04-18 18:44:38 +02:00
Shea Levy b37f5ae31d
isFunction: True on primops.
Fixes #2073
2018-04-17 14:33:12 -04:00
Shea Levy a38fe5c1a8
Export required C++ version in pkgconfig. 2018-04-09 11:32:43 -04:00
Shea Levy fd98fca7bb
Make prim_exec and prim_importNative available to plugins 2018-04-09 10:26:50 -04:00
Tuomas Tynkkynen af86132e1a libexpr: Make unsafeGetAttrPos not crash on noPos
Currently e.g. `builtins.unsafeGetAttrPos "abort" builtins` will
eventually segfault because pos->file is an unset Symbol.

Found by afl-fuzz.
2018-04-03 15:54:42 +03:00
Eelco Dolstra 138af2e554
Shut up signedness warning 2018-03-19 11:57:15 +01:00
Eelco Dolstra 64441f0551
Merge pull request #1939 from dezgeg/lexer-fix
libexpr: Recognize newline in more places in lexer
2018-03-16 12:14:58 +01:00
Shea Levy cfdbfa6b2c
Catch more possible instances of passing NULL to memcpy.
Actually fixes #1976.
2018-03-14 23:53:43 -04:00
Shea Levy e2088febf3
concatLists: Don't pass NULL pointers to memcpy.
This is UB, even if the size is 0. See #1976.

Fixes #1976.
2018-03-14 23:44:02 -04:00
Eelco Dolstra 55aa622fb1
Fix compatibility with latest boost::format 2018-03-14 19:25:09 +01:00
Guillaume Maudoux 80735c4cc9
fetchGit: Fix debug message 2018-03-13 10:28:23 +01:00
Tim Engler 383afab9ad Modified MakeBinOp to no longer produce its name using concatenation and "##".
Doing so prevents emacs tags from working, as well as makes the code extremely
confusing for a newbie.

In the prior state, if someone wants to find the definition of "ExprApp" for
example, a grep through the code reveals nothing. Since the definition could be
hiding in numerous ".h" files, it's really difficult to find. This personally
took me several hours to figure out.
2018-03-09 03:16:33 +00:00
Tuomas Tynkkynen a0e38c16bc libexpr: Recognize newline in more places in lexer
Flex's regexes have an annoying feature: the dot matches everything
except a newline. This causes problems for expressions like:

"${0}\
"

where the backslash-newline combination matches this rule instead of the
intended one mentioned in the comment:

    <STRING>\$|\\|\$\\ {
                    /* This can only occur when we reach EOF, otherwise the above
                    (...|\$[^\{\"\\]|\\.|\$\\.)+ would have triggered.
                    This is technically invalid, but we leave the problem to the
                    parser who fails with exact location. */
                    return STR;
                }
However, the parser actually accepts the resulting token sequence
('"' DOLLAR_CURLY 0 '}' STR '"'), which is a problem because the lexer
rule didn't assign anything to yylval. Ultimately this leads to a crash
when dereferencing a NULL pointer in ExprConcatStrings::bindVars().

The fix does change the syntax of the language in some corner cases
but I think it's only turning previously invalid (or crashing) syntax
to valid syntax. E.g.

"a\
b"

and

''a''\
b''

were previously syntax errors but now both result in "a\nb".

Found by afl-fuzz.
2018-03-02 17:30:48 +02:00
Will Dietz e89d02bf03 fetchGit: use "HEAD" as default ref 2018-02-28 16:34:34 -06:00
Shea Levy 14ca85688c
Actually fix nixDataDir in non-canonical path 2018-02-28 06:19:40 -05:00
Tuomas Tynkkynen 77e9e1ed91 libexpr: Fix prim_replaceStrings() to work on an empty source string
Otherwise, running e.g.

nix-instantiate --eval -E --strict 'builtins.replaceStrings [""] ["X"] "abc"'

would just hang in an infinite loop.

Found by afl-fuzz.

First attempt of this was reverted in e2d71bd186 because it caused
another infinite loop, which is fixed now and a test added.
2018-02-26 19:49:13 +02:00
Eelco Dolstra 179b896acb
Merge branch 'data-dir-non-canon' of https://github.com/shlevy/nix 2018-02-22 14:20:43 +01:00
Shea Levy ddbcd01c83
Fix restricted mode when installing in non-canonical data dir 2018-02-22 07:18:14 -05:00
Eelco Dolstra e2d71bd186
Revert "libexpr: Fix prim_replaceStrings() to work on an empty source string"
This reverts commit 4ea9707591.

It causes an infinite loop in Nixpkgs evaluation,
e.g. "nix-instantiate -A hello" hung.

PR #1886.
2018-02-21 15:35:28 +01:00
Tuomas Tynkkynen 4ea9707591 libexpr: Fix prim_replaceStrings() to work on an empty source string
Otherwise, running e.g.

nix-instantiate --eval -E --strict 'builtins.replaceStrings [""] ["X"] "abc"'

would just hang in an infinite loop.

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Tuomas Tynkkynen 056d28a601 libexpr: Don't create lots of temporary strings in Bindings::lexicographicOrder
Avoids ~180,000 string temporaries created when evaluating a headless
NixOS system.
2018-02-19 22:47:25 +02:00
Tuomas Tynkkynen 37264ed0ad libexpr: Avoid an unnecessary string copy in prim_derivationStrict 2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen 66eeff3345 libexpr: Remove unnecessary drvName assignment in prim_derivationStrict
drvName is already assigned to the same value right at the start of the
function.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen 7e0360504d libexpr: Optimize prim_derivationStrict by using more symbol comparisons 2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen 0845cdf944 libexpr: Rely on Boehm returning zeroed memory in EvalState::allocEnv()
Boehm guarantees that memory returned by GC_malloc() is zeroed, so take
advantage of that.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen b8bed7da14 libexpr: Optimize prim_attrNames a bit
Instead of having lexicographicOrder() create a temporary sorted array
of Attr*:s and copying attr names from that, copy the attr names
first and then sort that.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen f67a7007a2 libexpr: Pre-reserve space in string in unescapeStr()
Avoids some malloc() traffic.
2018-02-16 04:39:43 +02:00
Shea Levy b095c06139
Add splitVersion primop.
Fixes #1868.
2018-02-14 09:55:43 -05:00
Eelco Dolstra 7828dca9e8
Merge branch 'register-constant' of https://github.com/shlevy/nix 2018-02-13 12:24:48 +01:00
Frederik Rietdijk 60eca58533 Nix stats: flatten statistics
Flattens the list of statistics as suggested in
https://github.com/NixOS/ofborg/issues/67. This makes it easier to work
with.
2018-02-11 14:37:50 +01:00
Shea Levy 081f14a169
Allow using RegisterPrimop to define constants.
This enables plugins to add new constants, as well as new primops.
2018-02-08 14:35:50 -05:00
Eelco Dolstra abe6be578b
Merge pull request #1816 from shlevy/add-path
Add path primop.
2018-02-07 13:32:35 +01:00
Shea Levy 69d82e5c58
Add path primop.
builtins.path allows specifying the name of a path (which makes paths
with store-illegal names now addable), allows adding paths with flat
instead of recursive hashes, allows specifying a filter (so is a
generalization of filterSource), and allows specifying an expected
hash (enabling safe path adding in pure mode).
2018-02-06 16:48:08 -05:00
Eelco Dolstra 43f8ef73c6
realiseContext(): Add derivation outputs to the allowed paths
This makes import-from-derivation work in restricted mode again.
2018-02-06 15:38:45 +01:00
Eelco Dolstra f24e726ba5
checkURI(): Check file URIs against allowedPaths
This makes e.g. 'fetchGit ./.' work (assuming that ./. is an allowed
path).
2018-02-06 14:35:33 +01:00
Eelco Dolstra 89a2a11d9f
Don't use [[noreturn]] 2018-01-19 15:00:38 +01:00
Eelco Dolstra 0c95776c3e
Don't define builtins.{currentSystem,currentTime} in pure mode
This makes it easier to provide a default, e.g.

  system = builtins.currentSystem or "x86_64-linux";
2018-01-18 16:38:48 +01:00
Eelco Dolstra d8b4cfad82
Typo 2018-01-17 11:53:08 +01:00
Eelco Dolstra d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Eelco Dolstra 74f75c8558
import, builtins.readFile: Handle diverted stores
Fixes #1791
2018-01-12 17:31:08 +01:00
Will Dietz 428680b307 fetchGit: fix creation of uninitialized cache dir, let git create it
fetchGit test (as modified in previous commit) now passes.
2018-01-09 09:05:18 -06:00
Shea Levy 689b2783fc
Add hasContext primop 2018-01-02 12:25:14 -05:00
Will Dietz 2e6f06c37e fetchGit: Fix handling of local repo when not using 'master' branch
Add tests checking this behavior.
2017-12-22 15:29:52 -06:00
Ben Gamari f9bcbddef2 json-to-value: Throw sensible error message on invalid numbers 2017-12-14 19:09:45 -05:00
Shea Levy 11a7f8ce14
Merge branch 'fetchGit-fast-revision-update' 2017-12-05 11:16:39 -05:00
Eelco Dolstra 90948a4e3a
nix-shell/nix-build: Support .drv files again
Fixes #1663.

Also handle '!<output-name>' (#1694).
2017-11-24 18:08:35 +01:00
Shea Levy eedbc4e06c
fetchGit: Ignore tarballTtl if rev is set and not in the repo.
Fixes #1697.
2017-11-24 06:09:24 -05:00
Eelco Dolstra d7da6c9ea9
fetchGit/fetchMercurial: Fix directory inclusion check
E.g. the existence of .gitignore would cause .git to be included.
2017-11-21 19:34:46 +01:00
Eelco Dolstra 2c39e4eca0
Revert "Don't parse "x:x" as a URI"
This reverts commit f90f660b24.

This broke Hydra's release.nix, which contained

  preCheck = ''export LOGNAME=${LOGNAME:-foo}'';
2017-11-14 15:10:52 +01:00
Eelco Dolstra 7a4d9574d9
fetchgit.cc -> fetchGit.cc 2017-11-03 13:55:31 +01:00
Eelco Dolstra 6cf7c6a6b0
Remove git:// support in NIX_PATH
This didn't support specifying a revision/branch, and was restricted
to git:// URIs (since https:// or ssh:// would be ambiguous).
2017-11-03 13:55:31 +01:00
Eelco Dolstra ee6ac38848
fetchGit/fetchMercurial: Filter out directories with untracked files 2017-11-03 13:55:31 +01:00
Eelco Dolstra 4dee01da7c
fetchGit: Add a test 2017-11-03 13:55:30 +01:00
Eelco Dolstra 0e77aa3982
fetchGit: Don't do a remote fetch if we already have the rev 2017-11-03 13:55:30 +01:00
tv 5ab37f0e44
fetchMercurial: fix error message 2017-11-02 23:37:42 +01:00
Eelco Dolstra 212e72c609 Fix build
https://hydra.nixos.org/build/63172338
2017-11-01 21:32:30 +01:00
Eelco Dolstra e026bc3b05
fetchMercurial: Don't fetch hashes we already have 2017-11-01 18:43:11 +01:00
Eelco Dolstra 1969f357b7
Add fetchMercurial primop
E.g.

  $ nix eval '(fetchMercurial https://www.mercurial-scm.org/repo/hello)'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "82e55d328c8ca4ee16520036c0aaace03a5beb65"; revCount = 1; shortRev = "82e55d328c8c"; }

  $ nix eval '(fetchMercurial { url = https://www.mercurial-scm.org/repo/hello; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; })'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; revCount = 0; shortRev = "0a04b987be5a"; }

  $ nix eval '(fetchMercurial /tmp/unclean-hg-tree)'
  { branch = "default"; outPath = "/nix/store/cm750cdw1x8wfpm3jq7mz09r30l9r024-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "000000000000"; }
2017-11-01 17:45:32 +01:00
Eelco Dolstra cd532a9251
Fix filterSource 2017-10-31 16:18:32 +01:00
Eelco Dolstra 72cd52c3cd
builtins.fetchgit: Support importing a working tree
For example, you can write

  src = fetchgit ./.;

and if ./. refers to an unclean working tree, that tree will be copied
to the Nix store. This removes the need for "cleanSource".
2017-10-30 19:59:25 +01:00
Eelco Dolstra f90f660b24
Don't parse "x:x" as a URI
URIs now have to contain "://" or start with "channel:".
2017-10-30 17:58:01 +01:00
Eelco Dolstra 63c80ae26f
Make "fetchGit /path" work 2017-10-30 13:18:28 +01:00
Eelco Dolstra a5c392a80e
fetchGit: Fix broken assertion
Different URIs can map to the same cache entry if they have the same
revision.
2017-10-30 12:55:46 +01:00
Eelco Dolstra 812e027e1d
Add option allowed-uris
This allows network access in restricted eval mode.
2017-10-30 12:41:49 +01:00
Eelco Dolstra e38382895d
builtins.fetchGit: Return an attrset with revision info
This adds rev, shortRev and revCount attributes, equal to what Hydra
provides. E.g.

  $ nix eval '(fetchGit https://github.com/NixOS/patchelf.git)'
  { outPath = "/nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source"; rev = "29c085fd9d3fc972f75b3961905d6b4ecce7eb2b"; revCount = 303; shortRev = "29c085f"; }
2017-10-30 11:49:03 +01:00
Eelco Dolstra f9686885be
enable-http2 -> http2 2017-10-30 11:00:59 +01:00
Eelco Dolstra 049322702b
fetchgit -> fetchGit
Almost all other primops are camelCase so no reason not to use that
here.
2017-10-30 10:25:08 +01:00
Eelco Dolstra 23ce4b3393
fetchTarball: Use "source" as the default name
This ensures that it produces the same output as fetchgit:

  $ nix eval --raw '(builtins.fetchgit https://github.com/NixOS/patchelf.git)'
  /nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source

  $ nix eval --raw '(fetchTarball https://github.com/NixOS/patchelf/archive/master.tar.gz)'
  /nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source
2017-10-30 10:22:58 +01:00
Eelco Dolstra 66ddbef754
fetchurl/fetchTarball: Respect name changes
The computation of urlHash didn't take the name into account, so
subsequent fetchurl calls with the same URL but a different name would
resolve to the same cached store path.
2017-10-30 10:22:58 +01:00
Eelco Dolstra f25791c196
builtins.fetchgit: Don't mess up the progress indicator 2017-10-30 10:22:58 +01:00
Eelco Dolstra 65b5f177b5
builtins.fetchgit: Support a "name" attribute
The "name" attribute defaults to "source", which we should use for all
similar functions (e.g. fetchTarball and in Hydra) to ensure that we
get a consistent store path regardless of how the tree is fetched.

"source" is not necessarily a correct label, but using an empty name
is problematic: you get an ugly store path ending in a dash, and it's
impossible to have a fixed-output derivation that produces that path
because ".drv" is not a valid store name.

Fixes #904.
2017-10-30 10:22:07 +01:00
Eelco Dolstra c1ae18941a
nix repl :t: Indicate strings that have a context 2017-10-29 17:11:22 +01:00
Eelco Dolstra 8191992c83
Bump language version 2017-10-25 15:18:37 +02:00
Eelco Dolstra 2d5b1b24bf
Pass lists/attrsets to bash as (associative) arrays 2017-10-25 13:01:50 +02:00
Eelco Dolstra 0d59f1ca49
nix: Respect -I, --arg, --argstr
Also, random cleanup to argument handling.
2017-10-24 12:58:34 +02:00
Shea Levy cbc216911d
fetchgit: Remove incomplete/unneeded ref check.
This check fails for tags and branches, and is made redundant by the
checks git itself will do when fetching the repo.
2017-10-17 12:30:54 -04:00
Shea Levy 4e58294ae6
fetchgit: Remove incomplete/unneeded isURI check.
This check spuriously fails for e.g. git@github.com:NixOS/nixpkgs.git,
and even for ssh://git@github.com/NixOS/nixpkgs.git, and is made
redundant by the checks git itself will do when fetching the repo. We
instead pass a -- before passing the URI to git to avoid injection.
2017-10-16 14:35:28 -04:00
Nicolas B. Pierron b8867a0239 Add builtins.string function.
The function 'builtins.split' takes a POSIX extended regular expression
and an arbitrary string. It returns a list of non-matching substring
interleaved by lists of matched groups of the regular expression.

```nix
with builtins;
assert split "(a)b" "abc"      == [ "" [ "a" ] "c" ];
assert split "([ac])" "abc"    == [ "" [ "a" ] "b" [ "c" ] "" ];
assert split "(a)|(c)" "abc"   == [ "" [ "a" null ] "b" [ null "c" ] "" ];
assert split "([[:upper:]]+)" "  FOO   "
                               == [ "  " [ "FOO" ] "   " ];
```
2017-08-15 20:04:11 +00:00
Jörg Thalheim 2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
Eelco Dolstra 7480f4f9a4
builtins.fetchgit: Support specifying commit hashes
This adds an argument "rev" specififying the Git commit hash. The
existing argument "rev" is renamed to "ref". The default value for
"ref" is "master". When specifying a hash, it's necessary to specify a
ref since we're not cloning the entire repository but only fetching a
specific ref.

Example usage:

  builtins.fetchgit {
    url = https://github.com/NixOS/nixpkgs.git;
    ref = "release-16.03";
    rev = "c1c0484041ab6f9c6858c8ade80a8477c9ae4442";
  };
2017-07-27 18:08:23 +02:00
Eelco Dolstra 9f64cb89cb
builtins.fetchgit: Respect tarball-ttl
I.e. if the local ref is more recent than tarball-ttl seconds, then
don't check the remote.
2017-07-27 17:23:29 +02:00
Eelco Dolstra 69deca194e
builtins.fetchgit: Use proper refs locally 2017-07-27 17:02:25 +02:00
Eelco Dolstra 6d7de7f3de
builtins.fetchgit: Cache hash -> store path mappings
This prevents an expensive call to addToStore() in the cached case.
2017-07-27 16:16:08 +02:00
Eelco Dolstra 4c9ff89c26
nix-build/nix-shell: Eliminate call to nix-instantiate / nix-store
Note that this removes the need for a derivation symlink, so the
--drv-path and --add-drv-link flags now do nothing.
2017-07-26 17:29:09 +02:00
Eelco Dolstra c94f3d5575
nix-shell: Use bashInteractive from <nixpkgs>
This adds about 0.1s to nix-shell runtime in the case where
bashInteractive already exists.

See discussion at https://github.com/NixOS/nixpkgs/issues/27493.
2017-07-20 13:50:25 +02:00
Eelco Dolstra 90825dea51
Add "nix search" command 2017-07-20 13:33:13 +02:00
Eelco Dolstra 1762b9616c Merge pull request #1428 from rimmington/clearer-regex-space-error
Clearer error message when regex exceeds space limit
2017-07-10 11:45:05 +02:00
Rhys 17bb00d378 Clearer error message when regex exceeds space limit 2017-07-10 09:35:53 +10:00
Eelco Dolstra c0015e87af
Support base-64 hashes
Also simplify the Hash API.

Fixes #1437.
2017-07-04 15:07:41 +02:00
Eelco Dolstra fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00
Eelco Dolstra c7346a275c
Restore thunks on any exception
There's no reason to restrict this to Error exceptions. This shouldn't
matter to #1407 since the repl doesn't catch non-Error exceptions
anyway, but you never know...
2017-06-20 12:13:17 +02:00
Eelco Dolstra e46090edb1
builtins.match: Improve error message for bad regular expression
Issue #1331.
2017-05-17 11:58:01 +02:00
Eelco Dolstra b01d62285c
Improve progress indicator 2017-05-16 16:09:57 +02:00
Eelco Dolstra c05d9ae7a5
Disallow outputHash being null or an empty string
Fixes #1384.
2017-05-15 18:47:13 +02:00
Eelco Dolstra 465cb68244
Figure out the user's home directory if $HOME is not set 2017-05-05 17:08:23 +02:00
Guillaume Maudoux a143014d73 lexer: remove catch-all rules hiding real errors
With catch-all rules, we hide potential errors.
It turns out that a4744254 made one cath-all useless. Flex detected that
is was impossible to reach.
The other is more subtle, as it can only trigger on unfinished escapes
in unfinished strings, which only occurs at EOF.
2017-05-01 01:18:06 +02:00
Guillaume Maudoux a474425425 Fix lexer to support $' in multiline strings. 2017-05-01 01:15:40 +02:00
Eelco Dolstra bcecc99007
Restructure installables handling in the "nix" command 2017-04-25 11:20:37 +02:00
Eelco Dolstra b0cb117226
getDerivations(): Filter out packages with bad derivation names
In particular, this disallows attribute names containing dots or
starting with dots. Hydra already disallowed these. This affects the
following packages in Nixpkgs master:

  2048-in-terminal
  2bwm
  389-ds-base
  90secondportraits
  lispPackages.3bmd
  lispPackages.hu.dwim.asdf
  lispPackages.hu.dwim.def

Closes #1342.
2017-04-19 14:18:26 +02:00
Eelco Dolstra f8a2e8a552
Shut up some warnings 2017-04-14 14:42:20 +02:00
Eelco Dolstra 1860070548
Merge branch 'rework-options' of https://github.com/copumpkin/nix 2017-04-13 16:15:51 +02:00
Shea Levy 3ecb09a40a builtins.exec: Make the argument just a list 2017-03-31 11:58:41 -04:00
Shea Levy 0bb8db257d Add exec primop behind allow-unsafe-native-code-during-evaluation.
Execute a given program with the (optional) given arguments as the
user running the evaluation, parsing stdout as an expression to be
evaluated.

There are many use cases for nix that would benefit from being able to
run arbitrary code during evaluation, including but not limited to:

* Automatic git fetching to get a sha256 from a git revision
* git rev-parse HEAD
* Automatic extraction of information from build specifications from
  other tools, particularly language-specific package managers like
  cabal or npm
* Secrets decryption (e.g. with nixops)
* Private repository fetching

Ideally, we would add this functionality in a more principled way to
nix, but in the mean time 'builtins.exec' can be used to get these
tasks done.

The primop is only available when the
'allow-unsafe-native-code-during-evaluation' nix option is true. That
flag also enables the 'importNative' primop, which is strictly more
powerful but less convenient (since it requires compiling a plugin
against the running version of nix).
2017-03-30 08:04:21 -04:00
Daiderd Jordan 023217f07c
use std::tuple for ValueMap allocator 2017-03-24 23:05:49 +01:00
Shea Levy 93f863be96 Add option to disable import-from-derivation completely, even if the drv is already realized 2017-03-08 08:46:12 -05:00
Eelco Dolstra ffcf9d24a6
builtins.fetchgit: Support paths 2017-03-02 11:46:28 +01:00
Eelco Dolstra fbf17f1ad7
builtins.fetchgit: Fix bad format string 2017-03-02 11:40:11 +01:00
Dan Peebles e7cb2847ab Explicitly model all settings and fail on unrecognized ones
Previously, the Settings class allowed other code to query for string
properties, which led to a proliferation of code all over the place making
up new options without any sort of central registry of valid options. This
commit pulls all those options back into the central Settings class and
removes the public get() methods, to discourage future abuses like that.

Furthermore, because we know the full set of options ahead of time, we
now fail loudly if someone enters an unrecognized option, thus preventing
subtle typos. With some template fun, we could probably also dump the full
set of options (with documentation, defaults, etc.) to the command line,
but I'm not doing that yet here.
2017-02-22 20:19:25 -05:00
Tuomas Tynkkynen 2cd468874f Include config.h implicitly with '-include config.h' in CFLAGS
Because config.h can #define things like _FILE_OFFSET_BITS=64 and not
every compilation unit includes config.h, we currently compile half of
Nix with _FILE_OFFSET_BITS=64 and other half with _FILE_OFFSET_BITS
unset. This causes major havoc with the Settings class on e.g. 32-bit ARM,
where different compilation units disagree with the struct layout.

E.g.:

diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc
@@ -166,6 +166,8 @@ void Settings::update()
     _get(useSubstitutes, "build-use-substitutes");
+    fprintf(stderr, "at Settings::update(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
     _get(buildUsersGroup, "build-users-group");
diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc
+++ b/src/libstore/remote-store.cc
@@ -138,6 +138,8 @@ void RemoteStore::initConnection(Connection & conn)
 void RemoteStore::setOptions(Connection & conn)
 {
+    fprintf(stderr, "at RemoteStore::setOptions(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
     conn.to << wopSetOptions

Gave me:

at Settings::update(): &useSubstitutes = 0xb6e5c5cb
at RemoteStore::setOptions(): &useSubstitutes = 0xb6e5c5c7

That was not a fun one to debug!
2017-02-08 21:51:02 +02:00
Eelco Dolstra 6de33a9c67
Add support for passing structured data to builders
Previously, all derivation attributes had to be coerced into strings
so that they could be passed via the environment. This is lossy
(e.g. lists get flattened, necessitating configureFlags
vs. configureFlagsArray, of which the latter cannot be specified as an
attribute), doesn't support attribute sets at all, and has size
limitations (necessitating hacks like passAsFile).

This patch adds a new mode for passing attributes to builders, namely
encoded as a JSON file ".attrs.json" in the current directory of the
builder. This mode is activated via the special attribute

  __structuredAttrs = true;

(The idea is that one day we can set this in stdenv.mkDerivation.)

For example,

  stdenv.mkDerivation {
    __structuredAttrs = true;
    name = "foo";
    buildInputs = [ pkgs.hello pkgs.cowsay ];
    doCheck = true;
    hardening.format = false;
  }

results in a ".attrs.json" file containing (sans the indentation):

  {
    "buildInputs": [],
    "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash",
    "configureFlags": [
      "--with-foo",
      "--with-bar=1 2"
    ],
    "doCheck": true,
    "hardening": {
      "format": false
    },
    "name": "foo",
    "nativeBuildInputs": [
      "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
      "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03"
    ],
    "propagatedBuildInputs": [],
    "propagatedNativeBuildInputs": [],
    "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv",
    "system": "x86_64-linux"
  }

"passAsFile" is ignored in this mode because it's not needed - large
strings are included directly in the JSON representation.

It is up to the builder to do something with the JSON
representation. For example, in bash-based builders, lists/attrsets of
string values could be mapped to bash (associative) arrays.
2017-01-26 20:40:33 +01:00
Eelco Dolstra 54801ed6ad
Bindings: Add a method for iterating in lexicographically sorted order 2017-01-26 20:40:33 +01:00
Eelco Dolstra 4e6a2fbc56
Revert "Propagate path context via builtins.readFile"
This reverts commit f7f0116dd7.

Issue #1174.
2017-01-24 15:31:54 +01:00
Eelco Dolstra fa738e50bc
Revert "builtins.readFile: Put the references of the file, not those needed to realize the file, into the context"
Reverting commit 451c223dee for now
because it breaks http://hydra.nixos.org/build/46805136, not clear
why.
2017-01-24 15:29:46 +01:00
Shea Levy 451c223dee builtins.readFile: Put the references of the file, not those needed to realize the file, into the context 2017-01-10 12:22:22 -05:00
Eelco Dolstra 603f08506e
Tweak error message 2016-12-06 17:18:40 +01:00
Guillaume Maudoux e4b82af387 Improve error message on trailing path slashes 2016-11-27 17:48:46 +01:00
Guillaume Maudoux a5e761dddb Fix comments parsing
Fixed the parsing of multiline strings ending with an even number of
stars, like /** this **/.
Added test cases for comments.
2016-11-13 17:20:34 +01:00
Tuomas Tynkkynen f0fc3dd88b Fix SIGFPE from integer overflow during division
On some architectures (like x86_64 or i686, but not ARM for example)
overflow during integer division causes a crash due to SIGFPE.
Reproduces on a 64-bit system with:

    nix-instantiate --eval -E '(-9223372036854775807 - 1) / -1'

The only way this can happen is when the smallest possible integer is
divided by -1, so just special-case that.
2016-10-26 18:41:59 +03:00
Alexander Ried c935e8eeaf fixup! replace own regex class with std::regex 2016-10-19 16:03:53 +02:00
Alexander Ried b05b98df75 replace own regex class with std::regex 2016-10-18 20:22:25 +02:00
Eelco Dolstra c55bf085eb printMsg(lvlError, ...) -> printError(...) etc. 2016-09-21 16:54:53 +02:00
Eelco Dolstra 90ad02bf62 Enable HTTP/2 support
The binary cache store can now use HTTP/2 to do lookups. This is much
more efficient than HTTP/1.1 due to multiplexing: we can issue many
requests in parallel over a single TCP connection. Thus it's no longer
necessary to use a bunch of concurrent TCP connections (25 by
default).

For example, downloading 802 .narinfo files from
https://cache.nixos.org/, using a single TCP connection, takes 11.8s
with HTTP/1.1, but only 0.61s with HTTP/2.

This did require a fairly substantial rewrite of the Downloader class
to use the curl multi interface, because otherwise curl wouldn't be
able to do multiplexing for us. As a bonus, we get connection reuse
even with HTTP/1.1. All downloads are now handled by a single worker
thread. Clients call Downloader::enqueueDownload() to tell the worker
thread to start the download, getting a std::future to the result.
2016-09-14 16:36:02 +02:00
Eelco Dolstra 92d917b71a Fix GC build 2016-08-30 13:12:12 +02:00
Eelco Dolstra c42d1acfeb forceBool(): Show position info 2016-08-29 19:37:19 +02:00
Eelco Dolstra 26d92017d3 Add builtin function "partition"
The implementation of "partition" in Nixpkgs is O(n^2) (because of the
use of ++), and for some reason was causing stack overflows in
multi-threaded evaluation (not sure why).

This reduces "nix-env -qa --drv-path" runtime by 0.197s and memory
usage by 298 MiB (in non-Boehm mode).
2016-08-29 19:36:54 +02:00
Eelco Dolstra c0a7b84748 nix path-info: Add --json flag
Also, factor out JSON generation from value-to-json.{cc,hh}, and
support producing indented JSON.
2016-08-29 17:29:24 +02:00
Shea Levy 9fa21765e7 callFunction: Copy functors to the heap
Normally it's impossible to take a reference to the function passed to
callFunction, so some callers (e.g. ExprApp::eval) allocate that value
on the stack. For functors, a reference to the functor itself may be
kept, so we need to have it on the heap.

Fixes #1045
2016-08-29 07:36:28 -04:00
Eelco Dolstra d74236d1f2 nix build: Use Nix search path
That is, unless --file is specified, the Nix search path is
synthesized into an attribute set. Thus you can say

  $ nix build nixpkgs.hello

assuming $NIX_PATH contains an entry of the form "nixpkgs=...". This
is more verbose than

  $ nix build hello

but is less ambiguous.
2016-08-23 17:11:19 +02:00
Eelco Dolstra 22d6e31fc6 Add a mechanism for derivation attributes to reference the derivation's outputs
For example, you can now say:

  configureFlags = "--prefix=${placeholder "out"} --includedir=${placeholder "dev"}";

The strings returned by the ‘placeholder’ builtin are replaced at
build time by the actual store paths corresponding to the specified
outputs.

Previously, you had to work around the inability to self-reference by doing stuff like:

  preConfigure = ''
    configureFlags+=" --prefix $out --includedir=$dev"
  '';

or rely on ad-hoc variable interpolation semantics in Autoconf or Make
(e.g. --prefix=\$(out)), which doesn't always work.
2016-08-17 17:19:32 +02:00
Eelco Dolstra ac841a4679 Merge pull request #1031 from abbradar/replacestrings-context
Allow contexted strings in replaceStrings
2016-08-16 12:47:20 +02:00
Nikolay Amiantov f1b8dd43be Allow contexted strings in replaceStrings 2016-08-16 13:20:56 +03:00
Shea Levy d52d391164 builtins.fetch{url,tarball}: Allow name attribute 2016-08-15 07:37:11 -04:00
Eelco Dolstra ee22a91ab8 makeFixedOutputPath(): Drop superfluous HashType argument 2016-07-26 21:25:52 +02:00
Eelco Dolstra 06bbfb6004 builtins.{fetchurl,fetchTarball}: Support a sha256 attribute
Also, allow builtins.{fetchurl,fetchTarball} in restricted mode if a
hash is specified.
2016-07-26 21:16:52 +02:00
Eelco Dolstra 7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra 6c75cf69c3 Cleanup: Remove singleton() 2016-05-04 16:16:53 +02:00
Eelco Dolstra d8bf0d4859 Support Git repos in the Nix path
E.g.

  $ nix-build -I nixpkgs=git://github.com/NixOS/nixpkgs '<nixpkgs>' -A hello

This is not extremely useful yet because you can't specify a
branch/revision.
2016-04-29 21:07:47 +02:00
Eelco Dolstra 38539b943a Add fetchgit builtin
The function builtins.fetchgit fetches Git repositories at evaluation
time, similar to builtins.fetchTarball. (Perhaps the name should be
changed, being confusing with respect to Nixpkgs's fetchgit function,
with works at build time.)

Example:

  (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello

or

  (import (builtins.fetchgit {
    url = git://github.com/NixOS/nixpkgs-channels;
    rev = "nixos-16.03";
  }) {}).hello

Note that the result does not contain a .git directory.
2016-04-29 20:47:36 +02:00
Eelco Dolstra 41633f9f73 Improved logging abstraction
This also gets rid of --log-type, since the nested log type isn't
useful in a multi-threaded situation, and nobody cares about the
"pretty" log type.
2016-04-25 19:18:45 +02:00
Eelco Dolstra 31a1a8ed3c Merge pull request #815 from vcunat/p/outputsToInstall
nix-env: respect meta.outputsToInstall
2016-04-14 20:55:41 +02:00
Eelco Dolstra 5169a6da98 Make $NIX_PATH parsing more robust 2016-04-14 17:29:08 +02:00
Eelco Dolstra 363f37d084 Make the search path lazier with non-fatal errors
Thus, -I / $NIX_PATH entries are now downloaded only when they are
needed for evaluation. An error to download an entry is a non-fatal
warning (just like non-existant paths).

This does change the semantics of builtins.nixPath, which now returns
the original, rather than resulting path. E.g., before we had

  [ { path = "/nix/store/hgm3yxf1lrrwa3z14zpqaj5p9vs0qklk-nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

but now

  [ { path = "https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

Fixes #792.
2016-04-14 15:32:24 +02:00
Eelco Dolstra 12b257f045 Make primop registration pluggable
This way we don't have to put all primops in one giant file.
2016-04-14 12:50:01 +02:00
Nikolay Amiantov f7f0116dd7 Propagate path context via builtins.readFile 2016-03-04 14:04:51 +03:00
Eelco Dolstra e9c50064b5 Add an HTTP binary cache store
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
2016-02-29 18:15:20 +01:00
Scott Olson 6498adb002 Throw a specific error for incomplete parse errors.
`nix-repl` will use this for deciding whether to keep waiting for input or
error out right away.
2016-02-24 04:32:21 -06:00
Vladimír Čunát 03cbb9ad59 nix-env: respect meta.outputsToInstall
Discussed on https://github.com/NixOS/nixpkgs/pull/12653#discussion_r51601849
2016-02-23 14:19:14 +01:00
Eelco Dolstra 7a173a7be1 JSONObject: Support floats and booleans 2016-02-19 14:24:04 +01:00
Eelco Dolstra e03d6e0998 Fix broken number parsing in fromJSON
The call to tmp_number.append had its arguments mixed up. Also, JSON
does not allow a trailing "," after array/object members.
2016-02-15 15:01:26 +01:00
Eelco Dolstra b3e8d72770 Merge pull request #762 from ctheune/ctheune-floats
Implement floats
2016-02-12 12:49:59 +01:00
Eelco Dolstra fa7cd5369b StoreAPI -> Store
Calling a class an API is a bit redundant...
2016-02-04 14:48:42 +01:00
Eelco Dolstra c10c61449f Eliminate the "store" global variable
Also, move a few free-standing functions into StoreAPI and Derivation.

Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
2016-02-04 14:28:26 +01:00
Eelco Dolstra 5d8b7eb3e1 Revert "Revert "next try for "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751"""
This reverts commit b669d3d2e8.
2016-01-20 16:34:42 +01:00
Eelco Dolstra b669d3d2e8 Revert "next try for "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751""
This reverts commit ed23c8568e. Let's
merge this *after* the 1.11.1 release.
2016-01-20 00:05:28 +01:00
Fabian Schmitthenner ed23c8568e next try for "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751"
This reverts commit 8120b6fb8a and fixes the regression introduced in
8d22b26448.
2016-01-19 20:35:35 +00:00
Eelco Dolstra 8120b6fb8a Revert "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751"
This reverts commit 8d22b26448. It
breaks Nixpkgs:

$ nix-env -qa
error: syntax error, unexpected IND_STR, expecting '}', at /home/eelco/Dev/nixpkgs-stable/pkgs/top-level/python-packages.nix:7605:8
2016-01-19 20:33:32 +01:00
Fabian Schmitthenner 8d22b26448 don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751 2016-01-12 20:40:41 +00:00
Eelco Dolstra 9aac1861f7 Fix some signedness warnings 2016-01-07 14:37:39 +01:00
Christian Theune 934642155c @eelco's feedback: downgrade to regular float for size, remove unused function. 2016-01-06 08:25:58 +01:00
Fabian Schmitthenner 39d1da7b51 Better error message
Also show types when nix cannot compare values of different types.
This is also more consistent since types are already shown when comparing values of the same not comparable type.
2016-01-05 14:13:54 +01:00
Shea Levy 14080f3e4b Use __toString when coercing sets to strings.
For example, "${{ foo = "bar"; __toString = x: x.foo; }}" evaluates
to "bar".

With this, we can delay calling functions like mkDerivation,
buildPythonPackage, etc. until we actually need a derivation, enabling
overrides and other modifications to happen by simple attribute set
update.
2016-01-05 13:55:06 +01:00
Vladimír Čunát 29c0199d5c manual: document builtins.functionArgs
The text is just a conversion of comment from src/libexpr/primops.cc
2016-01-05 13:29:17 +01:00
Christian Theune a12a43046b Edge condition: parser did not pick up floats starting exactly with 0. 2016-01-05 09:54:49 +01:00
Christian Theune f872262e08 Fix up float parsing. 2016-01-05 09:46:37 +01:00
Christian Theune 494fc5acbb Try a simplified version of float lexing that didn't work.
The last one I tried was botchered anyway ...
2016-01-05 00:53:22 +01:00
Christian Theune 14ebde5289 First hit at providing support for floats in the language. 2016-01-05 00:40:40 +01:00
Eelco Dolstra a9faa7bbce showId: Handle empty attribute names
We should probably disallow these, but until then, we shouldn't barf
with an assertion failure.

Fixes #738.
2015-12-17 15:15:28 +01:00
Shea Levy f327970129 Merge branch 'auto-call-functor'
autoCallFunction now auto-calls functors
2015-11-25 11:57:31 -05:00
Shea Levy 9533532ce2 autoCallFunction: Auto-call functors 2015-11-25 11:56:14 -05:00
Eelco Dolstra cad40adce5 Merge pull request #704 from ysangkok/freebsd-support
FreeBSD support with knowledge about Linux emulation
2015-11-24 19:24:21 +01:00
Eelco Dolstra 5c28943e8f int2String() -> std::to_string() 2015-10-29 13:26:55 +01:00
Eelco Dolstra 51cf4455b2 isFunctor: Simplify 2015-10-08 13:22:11 +02:00
Mathnerd314 8a87521636 forceFunction: allow functors as well 2015-10-08 13:19:44 +02:00
Manuel Jacob 12c1776df9 Don't try to link libdl on FreeBSD.
There is no libdl on FreeBSD.  Instead the symbols are included in libc.
2015-10-06 22:28:30 +02:00
Eelco Dolstra 4ba6bc184c Shut up clang warnings 2015-09-18 01:22:35 +02:00
Iwan Aucamp 3db950aab7 Removed unnecessary included 2015-08-03 11:48:34 +02:00
Eelco Dolstra 2bac04c5ff Fix stack consumption 2015-07-31 20:28:25 +02:00
Iwan Aucamp 75837651f1 Output line number on infinite recursion 2015-07-31 20:26:44 +02:00
Eelco Dolstra 76cc8e97a2 Add sort primop 2015-07-28 18:39:39 +02:00
Eelco Dolstra 50807f3dd5 Add primop genList
This can be used to implement functions like ‘imap’ (or for that
matter, ‘map’) without the quadratic complexity incurred by calling
‘++’ repeatedly.
2015-07-28 17:28:35 +02:00
Eelco Dolstra d6d5885c15 Add replaceStrings primop
This is a generalisation of replaceChars in Nixpkgs.
2015-07-24 15:32:24 +02:00
Eelco Dolstra 2e8fd4c5cd Add concatStringsSep as a primop
This fixes the quadratic behaviour of concatStrings/concatStringsSep
in Nixpkgs.
2015-07-24 02:38:09 +02:00