Commit graph

2323 commits

Author SHA1 Message Date
Eelco Dolstra c56bfbe863 Provide default pagers
Borrowed from systemd.
2014-12-10 18:16:05 +01:00
Eelco Dolstra 851b47bd7d Don't do vfork in conjunction with setuid 2014-12-10 18:01:01 +01:00
Eelco Dolstra 0e8fc118b3 Use vfork 2014-12-10 17:25:39 +01:00
Eelco Dolstra b5ed5b6e66 Rename function 2014-12-10 17:25:38 +01:00
Eelco Dolstra e529823635 Don't wait for PID -1
The pid field can be -1 if forking the substituter process failed.
2014-12-10 13:53:04 +01:00
Eelco Dolstra ad332e1718 Revert "Use posix_spawn to run the pager"
This reverts commit d34d2b2bbf.
2014-12-10 13:48:50 +01:00
Shea Levy 50c3352811 builtins.readFile: realise context associated with the path 2014-12-10 12:26:53 +01:00
Marko Durkovic 629e8da3aa Explicitly include required C headers 2014-12-09 13:00:59 +01:00
Eelco Dolstra 8d169b2b75 Define ‘environ’
http://hydra.nixos.org/build/17690555
2014-12-05 21:05:24 +01:00
Eelco Dolstra d34d2b2bbf Use posix_spawn to run the pager
In low memory environments, "nix-env -qa" failed because the fork to
run the pager hit the kernel's overcommit limits. Using posix_spawn
gets around this. (Actually, you have to use posix_spawn with the
undocumented POSIX_SPAWN_USEVFORK flag, otherwise it just uses
fork/exec...)
2014-12-05 20:34:41 +01:00
Shea Levy 608110804c Make all ExternalValueBase functions const 2014-12-02 10:27:10 -05:00
Shea Levy 320659b0cd Allow external code using libnixexpr to add types
Code that links to libnixexpr (e.g. plugins loaded with importNative, or
nix-exec) may want to provide custom value types and operations on
values of those types. For example, nix-exec is currently using sets
where a custom IO value type would be more appropriate. This commit
provides a generic hook for such types in the form of tExternal and the
ExternalBase virtual class, which contains all functions necessary for
libnixexpr's type-polymorphic functions (e.g. `showType`) to be
implemented.
2014-12-02 10:27:04 -05:00
Eelco Dolstra 976df480c9 Add a primop for regular expression pattern matching
The function ‘builtins.match’ takes a POSIX extended regular
expression and an arbitrary string. It returns ‘null’ if the string
does not match the regular expression. Otherwise, it returns a list
containing substring matches corresponding to parenthesis groups in
the regex. The regex must match the entire string (i.e. there is an
implied "^<pat>$" around the regex).  For example:

  match "foo" "foobar" => null
  match "foo" "foo" => []
  match "f(o+)(.*)" "foooobar" => ["oooo" "bar"]
  match "(.*/)?([^/]*)" "/dir/file.nix" => ["/dir/" "file.nix"]
  match "(.*/)?([^/]*)" "file.nix" => [null "file.nix"]

The following example finds all regular files with extension .nix or
.patch underneath the current directory:

  let

    findFiles = pat: dir: concatLists (mapAttrsToList (name: type:
      if type == "directory" then
        findFiles pat (dir + "/" + name)
      else if type == "regular" && match pat name != null then
        [(dir + "/" + name)]
      else []) (readDir dir));

  in findFiles ".*\\.(nix|patch)" (toString ./.)
2014-11-25 11:47:06 +01:00
Eelco Dolstra 4e340a983f forceString(): Accept pos argument 2014-11-25 10:23:36 +01:00
Eelco Dolstra b7b6e3ddec Build derivations in a more predictable order
Derivations are now built in order of derivation name, so a package
named "aardvark" is built before "baboon".

Fixes #399.
2014-11-24 16:50:46 +01:00
Eelco Dolstra 9e3389c337 Don't create unnecessary substitution goals for derivations 2014-11-24 16:50:46 +01:00
Shea Levy b0c5c2ac34 import derivation: cleanup
Before this there was a bug where a `find` was being called on a
not-yet-sorted set. The code was just a mess before anyway, so I cleaned
it up while fixing it.
2014-11-20 22:48:12 -05:00
Eelco Dolstra 8299aaf079 Disable vacuuming the DB after garbage collection
Especially in WAL mode on a highly loaded machine, this is not a good
idea because it results in a WAL file of approximately the same size
ad the database, which apparently cannot be deleted while anybody is
accessing it.
2014-11-19 18:14:24 +01:00
Eelco Dolstra a3e5c99d66 nix-daemon: Call exit(), not _exit()
This was preventing destructors from running. In particular, it was
preventing the deletion of the temproot file for each worker
process. It may also have been responsible for the excessive WAL
growth on Hydra (due to the SQLite database not being closed
properly).

Apparently broken by accident in
8e9140cfde.
2014-11-19 17:09:27 +01:00
Eelco Dolstra 1256ab3b44 Clean up temp roots in a more C++ way 2014-11-19 17:07:29 +01:00
Eelco Dolstra 35aad73bb6 Fix message 2014-11-17 01:00:39 +01:00
Shea Levy 2719627bbe realiseContext: Handle all context types
Avoids an assertion
2014-11-15 21:43:51 -05:00
Shea Levy 997defa166 Add functors (callable attribute sets).
With this, attribute sets with a `__functor` attribute can be applied
just like normal functions. This can be used to attach arbitrary
metadata to a function without callers needing to treat it specially.
2014-11-15 16:12:05 -05:00
Eelco Dolstra 8cfe939b0f Don't use ADDR_LIMIT_3GB
This gives 32-bit builds on x86_64-linux more memory.
2014-11-14 14:16:20 +01:00
Eelco Dolstra bab8d9b52a Make ~DerivationGoal more reliable 2014-11-12 11:35:53 +01:00
Eelco Dolstra 06a86aee15 nix-store --gc: Don't warn about missing manifests directory 2014-11-04 10:41:29 +01:00
Eelco Dolstra 526811c87a nix-daemon: Get peer credentials on Mac OS X
This makes allowed-users and trusted-users work on Mac OS X.
2014-10-31 10:08:59 +01:00
Eelco Dolstra e389f4ea55 Improve error message if the daemon worker fails to start 2014-10-31 09:36:09 +01:00
Eelco Dolstra 3a9b4a1467 Fix more warnings 2014-10-31 08:49:15 +01:00
Eelco Dolstra 80893a13a7 Shut up a clang warning 2014-10-31 08:49:15 +01:00
Eelco Dolstra 11aef17a77 Remove comments claiming we use a private PID namespace
This is no longer the case since
524f89f139.
2014-10-29 15:49:34 +01:00
Shea Levy 6062b12160 Fix build on gcc < 4.7 2014-10-20 12:15:50 -04:00
Shea Levy f040159f77 Revert "Drop support for pre-c++11 compilers."
The breakage this fixed can be worked around without removing support.

This reverts commit 84a13dc576.
2014-10-20 11:33:48 -04:00
Eelco Dolstra ecc2c8f464 Improve printing of ASTs 2014-10-20 09:13:21 +02:00
Shea Levy 84a13dc576 Drop support for pre-c++11 compilers.
In particular, gcc 4.6's std::exception::~exception has an exception
specification in c++0x mode, which requires us to use that deprecated
feature in nix (and led to breakage after some recent changes that were
valid c++11).

nix already uses several c++11 features and gcc 4.7 has been around for
over 2 years.
2014-10-18 22:44:59 -04:00
Shea Levy 0ee1ca628a baseNameOf: Don't copy paths to the store first 2014-10-18 20:28:28 -04:00
Shea Levy d16e3c7f09 Export realiseContext in libnixexpr
Useful for importNative plugins
2014-10-17 22:15:09 -04:00
Eelco Dolstra d04e1ff12a nix-store -q: Check for conflicting flags
Fixes #364.
2014-10-14 12:15:39 +02:00
Eelco Dolstra c6849e2dee Remove redundant space in usage errors 2014-10-14 11:41:04 +02:00
Eelco Dolstra 2a4dbcff74 Improved error message when encountering unsupported file types
Fixes #269.
2014-10-14 10:51:19 +02:00
Eelco Dolstra 6bb4c0b712 mkList: Scrub better
Clearing v.app.right was not enough, because the length field of a
list only takes 32 bits, so the most significant 32 bits of v.app.left
(a.k.a. v.thunk.env) would remain. This could cause Boehm GC to
interpret it as a valid pointer.

This change reduces maximum RSS for evaluating the ‘tested’ job in
nixos/release-small.nix from 1.33 GiB to 0.80 GiB, and runtime by
about 8%.
2014-10-09 13:08:53 +02:00
Eelco Dolstra 986fbd6fab Typo 2014-10-09 11:34:48 +02:00
Eelco Dolstra b6809608cc Get rid of some unnecessary ExprConcatStrings nodes in dynamic attrs
This gives a ~18% speedup in NixOS evaluation (after converting
most calls to hasAttr/getAttr to dynamic attrs).
2014-10-05 01:04:58 +02:00
Eelco Dolstra 1418806969 Show total allocations 2014-10-05 00:39:28 +02:00
Eelco Dolstra c3f0a489f9 Add primop ‘catAttrs’ 2014-10-04 18:15:03 +02:00
Eelco Dolstra d4fcbe1687 Add primop ‘attrValues’ 2014-10-04 16:41:24 +02:00
Eelco Dolstra 58d8a213b0 Tweak 2014-10-04 11:27:23 +02:00
Eelco Dolstra 3f8576a6ab Remove some duplicate code 2014-10-03 22:37:51 +02:00
Shea Levy c08c802bf3 Add readDir primop 2014-10-03 22:32:11 +02:00
Eelco Dolstra 3b5fa8d50c Don't recompile the same regex over and over 2014-10-03 21:29:40 +02:00
Eelco Dolstra 104e55bb7f nix-env: Add regular expression support in selectors
So you can now do things like:

  $ nix-env -qa '.*zip.*'
  $ nix-env -qa '.*(firefox|chromium).*'
2014-10-03 21:29:40 +02:00
Eelco Dolstra 3800f441e4 createDirs(): Handle ‘path’ being a symlink
In particular, this fixes "nix-build -o /tmp/result" on Mac OS X
(where /tmp is a symlink).
2014-10-03 16:53:28 +02:00
Eelco Dolstra a17c23426e printValue(): Don't print <CYCLE> for repeated values 2014-10-01 15:54:40 +02:00
Eelco Dolstra d61853430a Support control characters in JSON output 2014-09-30 00:41:18 +02:00
Eelco Dolstra f77be20c16 printMissing(): Print derivations in approximate build order 2014-09-26 14:09:20 +02:00
Eelco Dolstra 9b146a52f1 nix-daemon: Close unnecessary fd 2014-09-25 18:45:43 +02:00
Eelco Dolstra 0ed1b924be Bindings: Remove copy constructor 2014-09-24 15:29:05 +02:00
Shell Turner cf72a61af2 Add --force-name support for --set in nix-env, to support nix-install-package --set 2014-09-23 15:11:24 +02:00
Eelco Dolstra ebb1dbb3e1 Add missing static 2014-09-23 15:08:27 +02:00
Eelco Dolstra 53b044c2f6 Don't evaluate inside a "throw"
Workaround for
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=41174. This caused
hydra-eval-jobs to ignore SIGINT.
2014-09-22 19:18:05 +02:00
Eelco Dolstra 0cd6596b0e Add ‘deepSeq’ primop
Note that unlike ‘lib.deepSeq’ in Nixpkgs, this handles cycles.
2014-09-22 16:05:00 +02:00
Eelco Dolstra 831fc8ea21 Make forceValueDeep work on values with cycles 2014-09-22 16:05:00 +02:00
Eelco Dolstra 1e0a799bef Rename strictForceValue -> forceValueDeep 2014-09-22 16:05:00 +02:00
Eelco Dolstra 022618c794 Handle cycles when printing a value
So this no longer crashes with a stack overflow:

  nix-instantiate -E --eval 'let as = { x = as; }; in as'

Instead it prints:

  { x = { x = <CYCLE>; }; }
2014-09-22 16:05:00 +02:00
Eelco Dolstra a54c263402 Add ‘seq’ primop 2014-09-22 16:05:00 +02:00
Eelco Dolstra eff120d1b9 Add a function ‘valueSize’
It returns the size of value, including all other values and
environments reachable from it. It is intended for debugging memory
consumption issues.
2014-09-22 16:05:00 +02:00
Eelco Dolstra 2d6cd8aafd attrNames: Don't allocate duplicates of the symbols 2014-09-19 18:11:46 +02:00
Eelco Dolstra ea525a261f Fix off-by-one 2014-09-19 18:08:14 +02:00
Eelco Dolstra 93e4f01ee3 Inline Bindings::find() 2014-09-19 16:56:13 +02:00
Eelco Dolstra 5b58991a71 Store Attrs inside Bindings
This prevents a double allocation per attribute set.
2014-09-19 16:49:41 +02:00
Eelco Dolstra 0342eb1705 Remove bogus comment 2014-09-19 15:07:22 +02:00
Eelco Dolstra d4a71ec3bf Update spec file
http://hydra.nixos.org/build/14344391
2014-09-18 15:42:01 +02:00
Eelco Dolstra 8be9990cdb Install some pkgconfig files 2014-09-18 12:00:40 +02:00
Eelco Dolstra d98bfcbf81 On Linux, disable address space randomization 2014-09-17 17:21:13 +02:00
Eelco Dolstra 6e5b02bee4 Add some instrumentation for debugging GC leaks 2014-09-17 15:19:07 +02:00
Eelco Dolstra d37d012774 Settings: Add bool get() 2014-09-17 15:18:13 +02:00
Eelco Dolstra 9d65287b91 Fix dependency ordering 2014-09-04 20:02:08 +02:00
Eelco Dolstra 9472b4157d Fix boost::too_many_args error
Fixes #333.
2014-09-02 22:53:01 +02:00
Ludovic Courtès 8fb8c26b6d Add an 'optimiseStore' remote procedure call. 2014-09-01 23:53:26 +02:00
Eelco Dolstra b72e93bca8 Add disallowedReferences / disallowedRequisites
For the "stdenv accidentally referring to bootstrap-tools", it seems
easier to specify the path that we don't want to depend on, e.g.

  disallowedRequisites = [ bootstrapTools ];
2014-08-28 18:57:13 +02:00
Gergely Risko fd61069a42 Introduce allowedRequisites feature 2014-08-28 18:23:55 +02:00
Joel Taylor b224ac1520 fix disappearing bash arguments 2014-08-21 23:17:50 +02:00
Eelco Dolstra 809ca33806 Use PR_SET_PDEATHSIG to ensure child cleanup 2014-08-21 15:31:43 +02:00
Eelco Dolstra 524f89f139 Use unshare() instead of clone()
It turns out that using clone() to start a child process is unsafe in
a multithreaded program. It can cause the initialisation of a build
child process to hang in setgroups(), as seen several times in the
build farm:

The reason is that Glibc thinks that the other threads of the parent
exist in the child, so in setxid_mark_thread() it tries to get a futex
that has been acquired by another thread just before the clone(). With
fork(), Glibc runs pthread_atfork() handlers that take care of this
(in particular, __reclaim_stacks()). But clone() doesn't do that.

Fortunately, we can use fork()+unshare() instead of clone() to set up
private namespaces.

See also https://www.mail-archive.com/lxc-devel@lists.linuxcontainers.org/msg03434.html.
2014-08-21 14:08:09 +02:00
Eelco Dolstra fefd3650d4 Fix a segfault in ‘nix-env -qa’
This was triggered by 47e185847e, which
turned globals.state into a pointer.
2014-08-21 00:05:17 +02:00
Eelco Dolstra 65243ee833 Flush std::cout before closing stdout 2014-08-20 21:44:55 +02:00
Eelco Dolstra 43579c9799 Use pager for more commands 2014-08-20 21:44:55 +02:00
Eelco Dolstra 9481f77693 Provide reasonable default flags for $LESS
Borrowed from systemd.
2014-08-20 21:22:41 +02:00
Eelco Dolstra 11849a320e Use proper quotes everywhere 2014-08-20 18:03:48 +02:00
Eelco Dolstra 373fad75e1 Add some color 2014-08-20 16:50:17 +02:00
Eelco Dolstra 392430b2c4 nix-store -l: Automatically pipe output into $PAGER 2014-08-20 15:12:58 +02:00
Eelco Dolstra 954188af27 Filter Nix-specific ANSI escape sequences from stderr
The Nixpkgs stdenv prints some custom escape sequences to denote
nesting and stuff like that. Most terminals (e.g. xterm, konsole)
ignore them, but some do not (e.g. xfce4-terminal). So for the benefit
of the latter, filter them out.
2014-08-20 14:30:01 +02:00
Eelco Dolstra 029424d17d Make hook shutdown more reliable 2014-08-19 17:47:30 +02:00
Eelco Dolstra 02843ba4cb Fix --attr parsing 2014-08-18 15:48:23 +02:00
Eelco Dolstra 632f989b51 Doh 2014-08-18 11:35:50 +02:00
Eelco Dolstra c160ead82f Reduce verbosity 2014-08-17 19:11:50 +02:00
Eelco Dolstra 42e9ad8fd1 Propagate remote timeouts properly 2014-08-17 19:09:03 +02:00
Eelco Dolstra ad716378dc Fix download-via-ssh
http://hydra.nixos.org/build/13286020
2014-08-17 14:28:17 +02:00
Eelco Dolstra fde819f2e2 nix-log2xml: Handle newlines 2014-08-13 19:06:20 +02:00
Eelco Dolstra cb921f67c3 Remove log2html.xsl and friends
It's part of Hydra now.
2014-08-13 19:01:14 +02:00
Eelco Dolstra e9b609bf9a nix-log2xml: Handle UTF-8 characters
C++ chars can be negative...
2014-08-13 19:00:07 +02:00
Eelco Dolstra 9367046fef Use regular file GC roots if possible
This makes hydra-eval-jobs create roots as regular files. See
1c208f2b7e.
2014-08-13 17:44:41 +02:00
Eelco Dolstra 5f05197df4 Handle compound single dash options properly
So now

  nix-instantiate --eval -E '{x}: x' --argstr x -xyzzy

correctly prints "-xyzzy", rather than giving an error.

Issue NixOS/hydra#176.
2014-08-13 04:08:33 +02:00
Eelco Dolstra 47e185847e Refactor option handling 2014-08-13 03:50:44 +02:00
Eelco Dolstra 5bed74d1b0 Fix warning about non-existant -I directories 2014-08-13 02:57:59 +02:00
Eelco Dolstra 401ab50566 Remove pointless NIX_LOG_TYPE environment variable 2014-08-13 01:19:02 +02:00
Eelco Dolstra 98873ff7de Remove unnecessary call to addTempRoot() 2014-08-05 16:41:42 +02:00
Eelco Dolstra f02c52e3d4 Doh 2014-08-05 10:19:57 +02:00
Eelco Dolstra 988bf59421 Move some options out of globals 2014-08-04 18:13:14 +02:00
Eelco Dolstra daccd68999 Refactor 2014-08-04 18:02:29 +02:00
Eelco Dolstra d5a076c36f Add option ‘build-extra-chroot-dirs’
This is useful for extending (rather than overriding) the default set
of chroot paths.
2014-08-04 18:00:00 +02:00
Eelco Dolstra 4d73e2e893 Get rid of "killing <pid>" message for unused build hooks 2014-08-04 17:27:45 +02:00
Eelco Dolstra bb45092f72 Make chroot builds easier to set up
By default, we now include /bin/sh as a bind-mount of bash.
2014-08-04 17:09:26 +02:00
Eelco Dolstra 51485dcda2 Remove ugly hack for detecting build environment setup errors 2014-08-01 19:38:21 +02:00
Eelco Dolstra 4c4b219c07 Call commonChildInit() before doing chroot init
This ensures that daemon clients see error messages from the chroot
setup.
2014-08-01 19:29:03 +02:00
Eelco Dolstra eea0401d7a Eliminate redundant copy 2014-08-01 17:30:51 +02:00
Eelco Dolstra e0d7d0e45c findRoots(): Prevent a call to lstat()
This means that getting the roots from /nix/var/nix/.../hydra-roots
doesn't need any I/O other than reading the directory.
2014-08-01 17:20:25 +02:00
Eelco Dolstra daf3f2c11f Make readDirectory() return inode / file type 2014-08-01 17:14:28 +02:00
Eelco Dolstra 1c208f2b7e Allow regular files as GC roots
If a root is a regular file, then its name must denote a store
path. For instance, the existence of the file

  /nix/var/nix/gcroots/per-user/eelco/hydra-roots/wzc3cy1wwwd6d0dgxpa77ijr1yp50s6v-libxml2-2.7.7

would cause

  /nix/store/wzc3cy1wwwd6d0dgxpa77ijr1yp50s6v-libxml2-2.7.7

to be a root.

This is useful because it involves less I/O (no need for a readlink()
call) and takes up less disk space (the symlink target typically takes
up a full disk block, while directory entries are packed more
efficiently). This is particularly important for hydra.nixos.org,
which has hundreds of thousands of roots, and where reading the roots
can take 25 minutes.
2014-08-01 17:14:28 +02:00
Eelco Dolstra 50dc1f5b71 Restore default SIGPIPE handler before invoking ‘man’
Fixes NixOS/nixpkgs#3410.
2014-07-31 10:31:17 +02:00
Eelco Dolstra 3d221a7bb1 Rename nixPath to __nixPath
The name ‘nixPath’ breaks existing code.
2014-07-30 11:28:39 +02:00
Eelco Dolstra 67937907ca nix-daemon: Pass on the user's $SSH_AUTH_SOCK to the SSH substituter 2014-07-25 18:02:48 +02:00
Eelco Dolstra 71a20d4d95 Change the default for use-ssh-substituter to ‘true’
Now you only have to pass ‘--option ssh-substituter-hosts
nix-ssh@bla’ to enable SSH substitution.
2014-07-25 12:57:12 +02:00
Eelco Dolstra 1a44f1cb65 nix-copy-closure: Drop --bzip2, --xz, --show-progress
These are too difficult to implement via nix-store --serve.

‘--show-progress’ could be re-implemented fairly easily via a
sink/source wrapper class.
2014-07-24 17:11:54 +02:00
Eelco Dolstra 03103c0a36 Implement nix-copy-closure --from via nix-store --serve 2014-07-24 16:19:41 +02:00
Eelco Dolstra 62309a2c56 build-remote.pl: Be less verbose on failing builds 2014-07-24 12:43:59 +02:00
Eelco Dolstra d7a92ed4a9 nix-store --serve: Only monitor stdin during builds
Other operations cannot hang indefinitely (except when we're reading
from stdin, in which case we'll notice a client disconnect). But
monitoring works badly during compressed imports, since there the
client can close the connection before we've sent an ack.

http://hydra.nixos.org/build/12711638
2014-07-24 11:59:35 +02:00
Eelco Dolstra 0fae20c362 Use pthread_cancel instead of a signal
Signal handlers are process-wide, so sending SIGINT to the monitor
thread will cause the normal SIGINT handler to run. This sets the
isInterrupted flag, which is not what we want. So use pthread_cancel
instead.
2014-07-24 11:47:51 +02:00
Eelco Dolstra aa1560ca07 Fix bogus pass by reference
http://hydra.nixos.org/build/12711659
2014-07-24 09:58:07 +02:00
Eelco Dolstra 24c6d992c6 More debugging 2014-07-24 01:21:56 +02:00
Eelco Dolstra 1eb0af7ed5 Add some assertions 2014-07-24 00:16:06 +02:00
Eelco Dolstra d3c61d83be Remove some dead code 2014-07-24 00:00:53 +02:00
Eelco Dolstra 62ad3dfc43 Remove some obsolete files 2014-07-23 23:56:58 +02:00
Eelco Dolstra bd91453bb1 Pass -pthread only for programs that need it 2014-07-23 19:48:18 +02:00
Eelco Dolstra ece531d105 nix-daemon: Less verbosity 2014-07-23 19:43:46 +02:00
Eelco Dolstra d0fad1779f nix-daemon: Simplify stderr handling 2014-07-23 19:37:40 +02:00
Eelco Dolstra 78224cd871 nix-store --serve: Monitor for client disconnects
This is necessary because build-remote.pl now builds via ‘nix-store
--serve’. So if a build hangs without writing to stdout/stderr, and
the client disconnects, then we need to detect that.
2014-07-23 19:26:34 +02:00
Eelco Dolstra 49fe9592a4 nix-daemon: Use a thread instead of SIGPOLL to catch client disconnects
The thread calls poll() to wait until a HUP (or other error event)
happens on the client connection. If so, it sends SIGINT to the main
thread, which is then cleaned up normally. This is much nicer than
messing around with SIGPOLL.
2014-07-23 19:21:00 +02:00
Eelco Dolstra fdee1ced43 startProcess: Make writing error messages from the child more robust 2014-07-23 19:11:26 +02:00
Eelco Dolstra 5989966ed3 Remove dead code 2014-07-23 14:46:28 +02:00
Eelco Dolstra ee3c5d7916 Revert old useBuildHook behaviour 2014-07-19 02:25:47 +02:00
Eelco Dolstra 2e77bd70fa Better fix for strcasecmp on Darwin 2014-07-18 12:54:30 +02:00
Eelco Dolstra 8ddffe7aac Ugly hack to fix building on old Darwin
http://hydra.nixos.org/build/12580878
2014-07-17 23:57:17 +02:00
Eelco Dolstra 049c0eb49c nix-daemon: Add trusted-users and allowed-users options
‘trusted-users’ is a list of users and groups that have elevated
rights, such as the ability to specify binary caches. It defaults to
‘root’. A typical value would be ‘@wheel’ to specify all users in the
wheel group.

‘allowed-users’ is a list of users and groups that are allowed to
connect to the daemon. It defaults to ‘*’. A typical value would be
‘@users’ to specify the ‘users’ group.
2014-07-17 16:57:07 +02:00
Eelco Dolstra 0c730887c4 nix-daemon: Show name of connecting user 2014-07-17 15:49:33 +02:00
Eelco Dolstra 77c972c898 nix-daemon: Only print connection info if we have SO_PEERCRED 2014-07-17 15:44:51 +02:00
Eelco Dolstra 8f72e702a1 nix-daemon: Fix compat with older clients 2014-07-17 15:23:31 +02:00
Eelco Dolstra 2304a7dd21 Get rid of a compiler warning 2014-07-16 16:32:26 +02:00
Eelco Dolstra 985f1595fe Be more strict about file names in NARs 2014-07-16 16:30:50 +02:00
Eelco Dolstra 276a40b31f Handle case collisions on case-insensitive systems
When running NixOps under Mac OS X, we need to be able to import store
paths built on Linux into the local Nix store. However, HFS+ is
usually case-insensitive, so if there are directories with file names
that differ only in case, then importing will fail.

The solution is to add a suffix ("~nix~case~hack~<integer>") to
colliding files. For instance, if we have a directory containing
xt_CONNMARK.h and xt_connmark.h, then the latter will be renamed to
"xt_connmark.h~nix~case~hack~1". If a store path is dumped as a NAR,
the suffixes are removed. Thus, importing and exporting via a
case-insensitive Nix store is round-tripping. So when NixOps calls
nix-copy-closure to copy the path to a Linux machine, you get the
original file names back.

Closes #119.
2014-07-16 16:02:05 +02:00
Eelco Dolstra fa13d3f4f3 build-remote.pl: Fix building multiple output derivations
We were importing paths without sorting them topologically, leading to
"path is not valid" errors.

See e.g. http://hydra.nixos.org/build/12451761
2014-07-14 12:19:27 +02:00
Eelco Dolstra a5c6347ff0 build-remote.pl: Use ‘nix-store --serve’ on the remote side
This makes things more efficient (we don't need to use an SSH master
connection, and we only start a single remote process) and gets rid of
locking issues (the remote nix-store process will keep inputs and
outputs locked as long as they're needed).

It also makes it more or less secure to connect directly to the root
account on the build machine, using a forced command
(e.g. ‘command="nix-store --serve --write"’). This bypasses the Nix
daemon and is therefore more efficient.

Also, don't call nix-store to import the output paths.
2014-07-11 16:22:24 +02:00
Eelco Dolstra e196eecbe6 Allow $NIX_BUILD_HOOK to be relative to Nix libexec directory 2014-07-11 13:55:06 +02:00
Eelco Dolstra d0eb970fb4 Fix broken Pid constructor 2014-07-10 21:48:21 +02:00
Eelco Dolstra edbfe2232e Replace message "importing path <...>" with "exporting path <...>"
This causes nix-copy-closure to show what it's doing before rather
than after.
2014-07-10 21:30:22 +02:00
Eelco Dolstra 7bb632b024 nix-copy-closure -s: Do substitutions via ‘nix-store --serve’
This means we no longer need an SSH master connection, since we only
execute a single command on the remote host.
2014-07-10 20:43:04 +02:00
Eelco Dolstra 43b64f5038 Remove tabs 2014-07-10 17:32:21 +02:00
Eelco Dolstra 8e9140cfde Refactoring: Move all fork handling into a higher-order function
C++11 lambdas ftw.
2014-07-10 16:58:09 +02:00
Eelco Dolstra 1114c7bd57 nix-copy-closure: Restore compression and the progress viewer 2014-07-10 14:15:12 +02:00
Eelco Dolstra 7911e4c27a Remove maybeVfork 2014-07-10 13:35:44 +02:00
Eelco Dolstra 04170d06bf nix-copy-closure: Fix race condition
There is a long-standing race condition when copying a closure to a
remote machine, particularly affecting build-remote.pl: the client
first asks the remote machine which paths it already has, then copies
over the missing paths. If the garbage collector kicks in on the
remote machine between the first and second step, the already-present
paths may be deleted. The missing paths may then refer to deleted
paths, causing nix-copy-closure to fail. The client now performs both
steps using a single remote Nix call (using ‘nix-store --serve’),
locking all paths in the closure while querying.

I changed the --serve protocol a bit (getting rid of QueryCommand), so
this breaks the SSH substituter from older versions. But it was marked
experimental anyway.

Fixes #141.
2014-07-10 11:58:59 +02:00
Eelco Dolstra 2c3a8f787b Fix security hole in ‘nix-store --serve’
Since it didn't check that the path received from the client is a
store path, the client could dump any path in the file system.
2014-07-10 11:46:01 +02:00
Eelco Dolstra 0e5d0c1543 Fix compilation error on some versions of GCC
src/libexpr/primops.cc:42:8: error: looser throw specifier for 'virtual nix::InvalidPathError::~InvalidPathError()'
src/libexpr/nixexpr.hh:12:1: error:   overriding 'virtual nix::EvalError::~EvalError() noexcept (true)'

http://hydra.nixos.org/build/12385750
2014-07-09 12:14:40 +02:00
Eelco Dolstra beaf3e90af Add builtin function ‘fromJSON’
Fixes #294.
2014-07-04 13:34:15 +02:00
Eelco Dolstra e477f0e938 Style fix 2014-06-27 11:36:23 +02:00
Paul Colomiets 858b8f9760 Add --json argument to nix-instantiate 2014-06-27 11:23:40 +02:00
Eelco Dolstra 8504e7d604 allow-arbitrary-code-during-evaluation -> allow-unsafe-native-code-during-evaluation 2014-06-27 11:20:16 +02:00
Shea Levy d62f46e500 Only add the importNative primop if the allow-arbitrary-code-during-evaluation option is true (default false) 2014-06-24 10:50:03 -04:00
Shea Levy 5cd022d6c0 Add importNative primop
This can be used to import a dynamic shared object and return an
arbitrary value, including new primops. This can be used both to test
new primops without having to recompile nix every time, and to build
specialized primops that probably don't belong upstream (e.g. a function
that calls out to gpg to decrypt a nixops secret as-needed).

The imported function should initialize the Value & as needed. A single
import can define multiple values by creating an attrset or list, of
course.

An example initialization function might look like:

extern "C" void initialize(nix::EvalState & state, nix::Value & v)
{
    v.type = nix::tPrimOp;
    v.primOp = NEW nix::PrimOp(myFun, 1, state.symbols.create("myFun"));
}

Then `builtins.importNative ./example.so "initialize"` will evaluate to
the primop defined in the myFun function.
2014-06-17 12:08:01 -04:00
Eelco Dolstra 9d0709e8c4 Don't use member initialisers
They're a little bit too recent (only supported since GCC 4.7).

http://hydra.nixos.org/build/11851475
2014-06-12 17:30:37 +02:00
Eelco Dolstra 48495f67ed Fix bogus warnings about dumping large paths
Also, yay for C++11 non-static initialisers.
2014-06-12 13:15:35 +02:00
Eelco Dolstra 0960d674d4 Drop ImportError and FindError
We're not catching these anywhere.
2014-06-12 13:00:54 +02:00
Shea Levy 718f20da6d findFile: Realise the context of the path attributes 2014-06-12 12:57:14 +02:00
Shea Levy a8fb575c98 Share code between scopedImport and import
In addition to reducing duplication, this fixes both import from
derivation and import of derivation for scopedImport
2014-06-12 12:52:39 +02:00
Eelco Dolstra ee7fe64c0a == operator: Ignore string context
There really is no case I can think of where taking the context into
account is useful. Mostly it's just very inconvenient.
2014-06-10 14:02:56 +02:00
Eelco Dolstra b1beed97a0 Report daemon OOM better
When copying a large path causes the daemon to run out of memory, you
now get:

  error: Nix daemon out of memory

instead of:

  error: writing to file: Broken pipe
2014-06-10 13:45:50 +02:00
Eelco Dolstra 829af22759 Print a warning when loading a large path into memory
I.e. if you have a derivation with

  src = ./huge-directory;

you'll get a warning that this is not a good idea.
2014-06-10 13:30:09 +02:00
Eelco Dolstra 3c6b8a5215 nix-env -qa --json: Generate valid JSON even if there are invalid meta attrs 2014-06-02 17:58:43 +02:00
Eelco Dolstra becc2b0167 Sort nixPath attributes 2014-05-29 19:02:14 +02:00
Eelco Dolstra 54a34119f3 Use std::unordered_set 2014-05-26 17:53:17 +02:00
Eelco Dolstra d8c061e044 Remove ExprBuiltin
It's slower than ExprVar since it doesn't compute a static
displacement. Since we're not using the throw primop in the
implementation of <...> anymore, it's also not really needed.
2014-05-26 17:14:28 +02:00
Eelco Dolstra 62a6eeb1f3 Make the Nix search path declarative
Nix search path lookups like <nixpkgs> are now desugared to ‘findFile
nixPath <nixpkgs>’, where ‘findFile’ is a new primop. Thus you can
override the search path simply by saying

  let
    nixPath = [ { prefix = "nixpkgs"; path = "/my-nixpkgs"; } ];
  in ... <nixpkgs> ...

In conjunction with ‘scopedImport’ (commit
c273c15cb1), the Nix search path can be
propagated across imports, e.g.

  let

    overrides = {
      nixPath = [ ... ] ++ builtins.nixPath;
      import = fn: scopedImport overrides fn;
      scopedImport = attrs: fn: scopedImport (overrides // attrs) fn;
      builtins = builtins // overrides;
    };

  in scopedImport overrides ./nixos
2014-05-26 17:02:22 +02:00
Eelco Dolstra 39d72640c2 Ensure that -I flags get included in nixPath
Also fixes #261.
2014-05-26 16:52:31 +02:00
Eelco Dolstra a8edf185a9 Add constant ‘nixPath’
It contains the Nix expression search path as a list of { prefix, path
} sets, e.g.

  [ { path = "/nix/var/nix/profiles/per-user/root/channels/nixos"; prefix = ""; }
    { path = "/etc/nixos/configuration.nix"; prefix = "nixos-config"; }
    { path = "/home/eelco/Dev/nix/inst/share/nix/corepkgs"; prefix = "nix"; }
  ]
2014-05-26 14:55:47 +02:00
Eelco Dolstra c273c15cb1 Add primop ‘scopedImport’
‘scopedImport’ works like ‘import’, except that it takes a set of
attributes to be added to the lexical scope of the expression,
essentially extending or overriding the builtin variables.  For
instance, the expression

  scopedImport { x = 1; } ./foo.nix

where foo.nix contains ‘x’, will evaluate to 1.

This has a few applications:

* It allows getting rid of function argument specifications in package
  expressions. For instance, a package expression like:

    { stdenv, fetchurl, libfoo }:

    stdenv.mkDerivation { ... buildInputs = [ libfoo ]; }

  can now we written as just

    stdenv.mkDerivation { ... buildInputs = [ libfoo ]; }

  and imported in all-packages.nix as:

    bar = scopedImport pkgs ./bar.nix;

  So whereas we once had dependencies listed in three places
  (buildInputs, the function, and the call site), they now only need
  to appear in one place.

* It allows overriding builtin functions. For instance, to trace all
  calls to ‘map’:

  let
    overrides = {
      map = f: xs: builtins.trace "map called!" (map f xs);

      # Ensure that our override gets propagated by calls to
      # import/scopedImport.
      import = fn: scopedImport overrides fn;

      scopedImport = attrs: fn: scopedImport (overrides // attrs) fn;

      # Also update ‘builtins’.
      builtins = builtins // overrides;
    };
  in scopedImport overrides ./bla.nix

* Similarly, it allows extending the set of builtin functions. For
  instance, during Nixpkgs/NixOS evaluation, the Nixpkgs library
  functions could be added to the default scope.

There is a downside: calls to scopedImport are not memoized, unlike
import. So importing a file multiple times leads to multiple parsings
/ evaluations. It would be possible to construct the AST only once,
but that would require careful handling of variables/environments.
2014-05-26 14:26:29 +02:00
Eelco Dolstra f0fdbd0897 Shut up some signedness warnings 2014-05-26 12:34:15 +02:00
Eelco Dolstra 0321ef9bb2 Ugly hack to allow --argstr values starting with a dash
Fixes #265.
2014-05-23 14:43:58 +02:00
Eelco Dolstra 9f9080e2c0 nix-store -l: Fetch build logs from the Internet
If a build log is not available locally, then ‘nix-store -l’ will now
try to download it from the servers listed in the ‘log-servers’ option
in nix.conf. For instance, if you have:

  log-servers = http://hydra.nixos.org/log

then it will try to get logs from http://hydra.nixos.org/log/<base
name of the store path>. So you can do things like:

  $ nix-store -l $(which xterm)

and get a log even if xterm wasn't built locally.
2014-05-21 17:19:36 +02:00
Shea Levy eac5841970 Provide a more useful error message when a dynamic attr lookup fails 2014-05-15 17:56:24 +02:00
Eelco Dolstra 8d5f472f2c lvlInfo -> lvlTalkative 2014-05-15 11:37:44 +02:00
Eelco Dolstra 84813af5b9 nix-store --optimise: Remove bogus statistics 2014-05-15 11:33:46 +02:00
Eelco Dolstra 690adeb03d Remove tab 2014-05-15 11:19:16 +02:00
Eelco Dolstra a1b66f316e Merge branch 'master' of github.com:wmertens/nix 2014-05-15 11:18:29 +02:00
Wout Mertens 3b9ea8452f Shortcut store files before lstat
readdir() already returns the inode numbers, so we don't need to call
lstat to know if a file was already linked or not.
2014-05-15 09:02:22 +02:00
Wout Mertens d73ffc552f Use the inodes given by readdir directly 2014-05-14 22:52:10 +02:00
Wout Mertens e974f20c98 Preload linked hashes to speed up lookups
By preloading all inodes in the /nix/store/.links directory, we can
quickly determine of a hardlinked file was already linked to the hashed
links.
This is tolerant of removing the .links directory, it will simply
recalculate all hashes in the store.
2014-05-13 23:10:06 +02:00
Eelco Dolstra 95501c4dee nix-instantiate --eval: Apply auto-arguments if the result is a function
Fixes #254.
2014-05-13 12:56:48 +02:00
wmertens a84f503d86 Shortcut already-hardlinked files
If an inode in the Nix store has more than 1 link, it probably means that it was linked into .links/ by us. If so, skip.

There's a possibility that something else hardlinked the file, so it would be nice to be able to override this.

Also, by looking at the number of hardlinks for each of the files in .links/, you can get deduplication numbers and space savings.
2014-05-10 15:53:01 +02:00
Eelco Dolstra 696f960dee Set up directories and permissions for multi-user install automatically
This automatically creates /nix/var/nix/profiles/per-user and sets the
permissions/ownership on /nix/store to 1775 and root:nixbld.
2014-05-02 14:31:15 +02:00
Eelco Dolstra de4cdd0d47 Set build-max-jobs to the number of available cores by default
More zero configuration.
2014-05-02 12:51:43 +02:00