artemist/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
lix/src/libstore/build.cc

3005 lines
91 KiB
C++
Raw Normal View History

* Include config.h before the C library headers, because it defines _FILE_OFFSET_BITS=64. Without it, functions like stat() fail on large file sizes. This happened with a Nix store on squashfs: $ nix-store --dump /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds > /dev/null error: getting attributes of path `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds': Value too large for defined data type $ stat /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds File: `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds' Size: 0 Blocks: 36028797018963968 IO Block: 1024 regular empty file (This is a bug in squashfs or mksquashfs, but it shouldn't cause Nix to fail.)
2009-12-17 14:12:44 +00:00
#include "config.h"
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 21:06:23 +00:00
#include "references.hh"
#include "pathlocks.hh"
#include "misc.hh"
#include "globals.hh"
* Refactoring. There is now an abstract interface class StoreAPI containing functions that operate on the Nix store. One implementation is LocalStore, which operates on the Nix store directly. The next step, to enable secure multi-user Nix, is to create a different implementation RemoteStore that talks to a privileged daemon process that uses LocalStore to perform the actual operations.
2006-11-30 17:43:04 +00:00
#include "local-store.hh"
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 21:06:23 +00:00
#include "util.hh"
* When building in a chroot, make a copy of a file if hard-linking fails. This is likely to happen after a `nix-store --optimise', because some files may have 32000 links (NIX-111).
2009-10-22 08:28:33 +00:00
#include "archive.hh"
Fix chroots builds Chroots are initialised by hard-linking inputs from the Nix store to the chroot. This doesn't work if the input has its immutable bit set, because it's forbidden to create hard links to immutable files. So temporarily clear the immutable bit when creating and destroying the chroot. Note that making regular files in the Nix store immutable isn't very reliable, since the bit can easily become cleared: for instance, if we run the garbage collector after running ‘nix-store --optimise’. So maybe we should only make directories immutable.
2012-02-18 00:23:52 +00:00
#include "immutable.hh"
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 21:06:23 +00:00
* Refactorings.
2003-07-20 19:29:38 +00:00
#include <map>
* db.hh shouldn't depend on the Berkeley DB headers.
2006-03-01 17:44:28 +00:00
#include <iostream>
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
#include <sstream>
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
#include <algorithm>
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
#include <boost/shared_ptr.hpp>
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
#include <boost/weak_ptr.hpp>
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
#include <boost/enable_shared_from_this.hpp>
* Refactorings.
2003-07-20 19:29:38 +00:00
* Kill a build if it has gone for more than a certain number of seconds without producing output on stdout or stderr (NIX-65). This timeout can be specified using the `--max-silent-time' option or the `build-max-silent-time' configuration setting. The default is infinity (0). * Fix a tricky race condition: if we kill the build user before the child has done its setuid() to the build user uid, then it won't be killed, and we'll potentially lock up in pid.wait(). So also send a conventional kill to the child.
2006-12-08 15:44:00 +00:00
#include <time.h>
* Not all platforms have sys/select.h.
2006-05-30 11:37:21 +00:00
#include <sys/time.h>
* include wait.h for WEXITSTATUS
2009-09-23 12:57:15 +00:00
#include <sys/wait.h>
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
* Include some missing headers.
2004-07-01 11:11:16 +00:00
#include <errno.h>
* Fix a build failure on Fedora 11. rename() needs <stdio.h>.
2009-09-24 07:21:29 +00:00
#include <stdio.h>
Include <cstring> to ensure that strcpy(), strlen(), and memset() are declared. An "using namespace std" was added locally in those functions that refer to names from <cstring>. That is not pretty, but it's a very portable solution, because strcpy() and friends will be found in both the 'std' and in the global namespace.
2010-06-24 17:51:04 +00:00
#include <cstring>
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
#include <pwd.h>
#include <grp.h>
Compress build logs on the fly using bzip2
2012-05-30 14:12:29 +00:00
#include <bzlib.h>
* On FreeBSD, sys/mount.h needs sys/param.h.
2007-10-29 10:03:07 +00:00
/* Includes required for chroot support. */
#if HAVE_SYS_PARAM_H
#include <sys/param.h>
#endif
* Detect whether chroot / bind-mount support is available.
2007-10-27 16:51:55 +00:00
#if HAVE_SYS_MOUNT_H
#include <sys/mount.h>
#endif
* Do chroot builds in a private namespace. This means that all the bind-mounts we do are only visible to the builder process and its children. So accidentally doing "rm -rf" on the chroot directory won't wipe out /nix/store and other bind-mounted directories anymore. Also, the bind-mounts in the private namespace disappear automatically when the builder exits.
2008-12-11 17:00:12 +00:00
#if HAVE_SCHED_H
#include <sched.h>
#endif
* Detect whether chroot / bind-mount support is available.
2007-10-27 16:51:55 +00:00
* Detect whether unshare() is available.
2008-12-12 13:41:36 +00:00
#define CHROOT_ENABLED HAVE_CHROOT && HAVE_UNSHARE && HAVE_SYS_MOUNT_H && defined(MS_BIND) && defined(CLONE_NEWNS)
* Detect whether chroot / bind-mount support is available.
2007-10-27 16:51:55 +00:00
In chroot builds, use a private network namespace On Linux it's possible to run a process in its own network namespace, meaning that it gets its own set of network interfaces, disjunct from the rest of the system. We use this to completely remove network access to chroot builds, except that they get a private loopback interface. This means that: - Builders cannot connect to the outside network or to other processes on the same machine, except processes within the same build. - Vice versa, other processes cannot connect to processes in a chroot build, and open ports/connections do not show up in "netstat". - If two concurrent builders try to listen on the same port (e.g. as part of a test), they no longer conflict with each other. This was inspired by the "PrivateNetwork" flag in systemd.
2012-06-23 04:28:35 +00:00
#if CHROOT_ENABLED
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <net/if.h>
#include <netinet/ip.h>
#endif
* Refactorings.
2003-07-20 19:29:38 +00:00
* Support i686-linux builds directly on x86_64-linux Nix installations. This is implemented using the personality() syscall, which causes uname to return "i686" in child processes.
2009-01-12 16:30:32 +00:00
#if HAVE_SYS_PERSONALITY_H
#include <sys/personality.h>
#define CAN_DO_LINUX32_BUILDS
#endif
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 21:06:23 +00:00
namespace nix {
using std::map;
* Refactorings.
2003-07-20 19:29:38 +00:00
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
static string pathNullDevice = "/dev/null";
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
static const uid_t rootUserId = 0;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Forward definition. */
class Worker;
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
class HookInstance;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* A pointer to a goal. */
class Goal;
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 21:06:23 +00:00
typedef boost::shared_ptr<Goal> GoalPtr;
typedef boost::weak_ptr<Goal> WeakGoalPtr;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
/* Set of goals. */
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
typedef set<GoalPtr> Goals;
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
typedef set<WeakGoalPtr> WeakGoals;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* A map of paths to goals (and the other way around). */
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
typedef map<Path, WeakGoalPtr> WeakGoalMap;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 21:06:23 +00:00
class Goal : public boost::enable_shared_from_this<Goal>
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
public:
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
typedef enum {ecBusy, ecSuccess, ecFailed, ecNoSubstituters} ExitCode;
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
protected:
/* Backlink to the worker. */
Worker & worker;
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
/* Goals that this goal is waiting for. */
Goals waitees;
/* Goals waiting for this one to finish. Must use weak pointers
here to prevent cycles. */
WeakGoals waiters;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Removed nrWaitees field. It was redundant with waitees.size() and could get out of sync if multiple input derivations mapped to the same closure expression (since waitees is a set).
2004-08-30 11:51:36 +00:00
/* Number of goals we are/were waiting for that have failed. */
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
unsigned int nrFailed;
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
/* Number of substitution goals we are/were waiting for that
failed because there are no substituters. */
unsigned int nrNoSubstituters;
* Subtle bug in the builder: if a subgoal that is instantiated multiple times is also a top-level goal, then the second and later instantiations would never be created because there would be a stable pointer to the first one that would keep it alive in the WeakGoalMap. * Some tracing code for debugging this kind of problem.
2005-02-18 09:50:20 +00:00
/* Name of this goal for debugging purposes. */
string name;
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
/* Whether the goal is finished. */
ExitCode exitCode;
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
Goal(Worker & worker) : worker(worker)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
nrFailed = nrNoSubstituters = 0;
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
exitCode = ecBusy;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
virtual ~Goal()
{
* Subtle bug in the builder: if a subgoal that is instantiated multiple times is also a top-level goal, then the second and later instantiations would never be created because there would be a stable pointer to the first one that would keep it alive in the WeakGoalMap. * Some tracing code for debugging this kind of problem.
2005-02-18 09:50:20 +00:00
trace("goal destroyed");
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
public:
virtual void work() = 0;
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
void addWaitee(GoalPtr waitee);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
virtual void waiteeDone(GoalPtr waitee, ExitCode result);
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
virtual void handleChildOutput(int fd, const string & data)
{
abort();
}
virtual void handleEOF(int fd)
* Write build logs to disk again.
2004-06-29 09:41:50 +00:00
{
abort();
}
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
void trace(const format & f);
* Subtle bug in the builder: if a subgoal that is instantiated multiple times is also a top-level goal, then the second and later instantiations would never be created because there would be a stable pointer to the first one that would keep it alive in the WeakGoalMap. * Some tracing code for debugging this kind of problem.
2005-02-18 09:50:20 +00:00
string getName()
{
return name;
}
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
ExitCode getExitCode()
{
return exitCode;
}
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
/* Cancel the goal. It should wake up its waiters, get rid of any
running child processes that are being monitored by the worker
(important!), etc. */
virtual void cancel() = 0;
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
protected:
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
void amDone(ExitCode result);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
};
/* A mapping used to remember for each child process to what goal it
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
belongs, and file descriptors for receiving log data and output
path creation commands. */
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
struct Child
{
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
WeakGoalPtr goal;
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
set<int> fds;
* Revert r19797, and use a simpler solution: just don't monitor build hooks for silence. It's unnecessary because the remote nix-store command is already monitoring the real build.
2010-02-03 21:38:41 +00:00
bool monitorForSilence;
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
bool inBuildSlot;
* Kill a build if it has gone for more than a certain number of seconds without producing output on stdout or stderr (NIX-65). This timeout can be specified using the `--max-silent-time' option or the `build-max-silent-time' configuration setting. The default is infinity (0). * Fix a tricky race condition: if we kill the build user before the child has done its setuid() to the build user uid, then it won't be killed, and we'll potentially lock up in pid.wait(). So also send a conventional kill to the child.
2006-12-08 15:44:00 +00:00
time_t lastOutput; /* time we last got output on stdout/stderr */
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
};
typedef map<pid_t, Child> Children;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* The worker class. */
class Worker
{
private:
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
/* Note: the worker should only have strong pointers to the
top-level goals. */
/* The top-level goals of the worker. */
Goals topGoals;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Goals that are ready to do some work. */
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
WeakGoals awake;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Goals waiting for a build slot. */
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
WeakGoals wantingToBuild;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Child processes currently running. */
Children children;
* Do a substitution even if --max-jobs == 0.
2009-03-31 21:14:07 +00:00
/* Number of build slots occupied. This includes local builds and
substitutions but not remote builds via the build hook. */
unsigned int nrLocalBuilds;
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
/* Maps used to prevent multiple instantiations of a goal for the
* Terminology fixes.
2005-01-20 16:01:07 +00:00
same derivation / path. */
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
WeakGoalMap derivationGoals;
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
WeakGoalMap substitutionGoals;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Fix a race condition with parallel builds where multiple fixed-output derivations or substitutions try to build the same store path at the same time. Locking generally catches this, but not between multiple goals in the same process. This happened especially often (actually, only) in the build farm with fetchurl downloads of the same file being executed on multiple machines and then copied back to the main machine where they would clobber each other (NIXBF-13). Solution: if a goal notices that the output path is already locked, then go to sleep until another goal finishes (hopefully the one locking the path) and try again.
2007-08-28 11:36:17 +00:00
/* Goals waiting for busy paths to be unlocked. */
WeakGoals waitingForAnyGoal;
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
/* Goals sleeping for a few seconds (polling a lock). */
WeakGoals waitingForAWhile;
/* Last time the goals in `waitingForAWhile' where woken up. */
time_t lastWokenUp;
Add support for the `build-timeout' and `--timeout' options.
2011-06-30 15:19:13 +00:00
/* Last time `waitForInput' was last called. */
time_t lastWait;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
public:
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
bool cacheFailure;
* nix-instantiate: return exit status 100 to denote a permanent build failure. The build hook can use this to distinguish between transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
/* Set if at least one derivation had a BuildError (i.e. permanent
failure). */
bool permanentFailure;
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
LocalStore & store;
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
boost::shared_ptr<HookInstance> hook;
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
Worker(LocalStore & store);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
~Worker();
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
/* Make a goal (with caching). */
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
GoalPtr makeDerivationGoal(const Path & drvPath);
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
GoalPtr makeSubstitutionGoal(const Path & storePath);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
/* Remove a dead goal. */
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
void removeGoal(GoalPtr goal);
/* Wake up a goal (i.e., there is something for it to do). */
void wakeUp(GoalPtr goal);
* Do a substitution even if --max-jobs == 0.
2009-03-31 21:14:07 +00:00
/* Return the number of local build and substitution processes
currently running (but not remote builds via the build
hook). */
unsigned int getNrLocalBuilds();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
/* Registers a running child process. `inBuildSlot' means that
the process counts towards the jobs limit. */
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
void childStarted(GoalPtr goal, pid_t pid,
* Revert r19797, and use a simpler solution: just don't monitor build hooks for silence. It's unnecessary because the remote nix-store command is already monitoring the real build.
2010-02-03 21:38:41 +00:00
const set<int> & fds, bool inBuildSlot, bool monitorForSilence);
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
/* Unregisters a running child process. `wakeSleepers' should be
false if there is no sense in waking up goals that are sleeping
because they can't run yet (e.g., there is no free build slot,
or the hook would still say `postpone'). */
* Don't go into a (sometimes infinite) loop calling the build hook.
2004-07-01 16:24:35 +00:00
void childTerminated(pid_t pid, bool wakeSleepers = true);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
/* Put `goal' to sleep until a build slot becomes available (which
might be right away). */
void waitForBuildSlot(GoalPtr goal);
* Fix a race condition with parallel builds where multiple fixed-output derivations or substitutions try to build the same store path at the same time. Locking generally catches this, but not between multiple goals in the same process. This happened especially often (actually, only) in the build farm with fetchurl downloads of the same file being executed on multiple machines and then copied back to the main machine where they would clobber each other (NIXBF-13). Solution: if a goal notices that the output path is already locked, then go to sleep until another goal finishes (hopefully the one locking the path) and try again.
2007-08-28 11:36:17 +00:00
/* Wait for any goal to finish. Pretty indiscriminate way to
wait for some resource that some other goal is holding. */
void waitForAnyGoal(GoalPtr goal);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
/* Wait for a few seconds and then retry this goal. Used when
waiting for a lock held by another process. This kind of
polling is inefficient, but POSIX doesn't really provide a way
to wait for multiple locks in the main select() loop. */
void waitForAWhile(GoalPtr goal);
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
/* Loop until the specified top-level goals have finished. */
void run(const Goals & topGoals);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Wait for input to become available. */
void waitForInput();
* nix-instantiate: return exit status 100 to denote a permanent build failure. The build hook can use this to distinguish between transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
unsigned int exitStatus();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
};
//////////////////////////////////////////////////////////////////////
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
void Goal::addWaitee(GoalPtr waitee)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
waitees.insert(waitee);
waitee->waiters.insert(shared_from_this());
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
void Goal::waiteeDone(GoalPtr waitee, ExitCode result)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
assert(waitees.find(waitee) != waitees.end());
waitees.erase(waitee);
* Subtle bug in the builder: if a subgoal that is instantiated multiple times is also a top-level goal, then the second and later instantiations would never be created because there would be a stable pointer to the first one that would keep it alive in the WeakGoalMap. * Some tracing code for debugging this kind of problem.
2005-02-18 09:50:20 +00:00
trace(format("waitee `%1%' done; %2% left") %
waitee->name % waitees.size());
* Added a switch `--fallback'. From the manual: Whenever Nix attempts to realise a derivation for which a closure is already known, but this closure cannot be realised, fall back on normalising the derivation. The most common scenario in which this is useful is when we have registered substitutes in order to perform binary distribution from, say, a network repository. If the repository is down, the realisation of the derivation will fail. When this option is specified, Nix will build the derivation instead. Thus, binary installation falls back on a source installation. This option is not the default since it is generally not desirable for a transient failure in obtaining the substitutes to lead to a full build from source (with the related consumption of resources).
2004-06-28 10:42:57 +00:00
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
if (result == ecFailed || result == ecNoSubstituters) ++nrFailed;
if (result == ecNoSubstituters) ++nrNoSubstituters;
* Added a switch `--fallback'. From the manual: Whenever Nix attempts to realise a derivation for which a closure is already known, but this closure cannot be realised, fall back on normalising the derivation. The most common scenario in which this is useful is when we have registered substitutes in order to perform binary distribution from, say, a network repository. If the repository is down, the realisation of the derivation will fail. When this option is specified, Nix will build the derivation instead. Thus, binary installation falls back on a source installation. This option is not the default since it is generally not desirable for a transient failure in obtaining the substitutes to lead to a full build from source (with the related consumption of resources).
2004-06-28 10:42:57 +00:00
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
if (waitees.empty() || (result == ecFailed && !keepGoing)) {
* Added a switch `--fallback'. From the manual: Whenever Nix attempts to realise a derivation for which a closure is already known, but this closure cannot be realised, fall back on normalising the derivation. The most common scenario in which this is useful is when we have registered substitutes in order to perform binary distribution from, say, a network repository. If the repository is down, the realisation of the derivation will fail. When this option is specified, Nix will build the derivation instead. Thus, binary installation falls back on a source installation. This option is not the default since it is generally not desirable for a transient failure in obtaining the substitutes to lead to a full build from source (with the related consumption of resources).
2004-06-28 10:42:57 +00:00
/* If we failed and keepGoing is not set, we remove all
remaining waitees. */
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (Goals::iterator, i, waitees) {
* Added a switch `--fallback'. From the manual: Whenever Nix attempts to realise a derivation for which a closure is already known, but this closure cannot be realised, fall back on normalising the derivation. The most common scenario in which this is useful is when we have registered substitutes in order to perform binary distribution from, say, a network repository. If the repository is down, the realisation of the derivation will fail. When this option is specified, Nix will build the derivation instead. Thus, binary installation falls back on a source installation. This option is not the default since it is generally not desirable for a transient failure in obtaining the substitutes to lead to a full build from source (with the related consumption of resources).
2004-06-28 10:42:57 +00:00
GoalPtr goal = *i;
WeakGoals waiters2;
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (WeakGoals::iterator, j, goal->waiters)
if (j->lock() != shared_from_this()) waiters2.insert(*j);
* Added a switch `--fallback'. From the manual: Whenever Nix attempts to realise a derivation for which a closure is already known, but this closure cannot be realised, fall back on normalising the derivation. The most common scenario in which this is useful is when we have registered substitutes in order to perform binary distribution from, say, a network repository. If the repository is down, the realisation of the derivation will fail. When this option is specified, Nix will build the derivation instead. Thus, binary installation falls back on a source installation. This option is not the default since it is generally not desirable for a transient failure in obtaining the substitutes to lead to a full build from source (with the related consumption of resources).
2004-06-28 10:42:57 +00:00
goal->waiters = waiters2;
}
waitees.clear();
* Don't go into a (sometimes infinite) loop calling the build hook.
2004-07-01 16:24:35 +00:00
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
worker.wakeUp(shared_from_this());
* Added a switch `--fallback'. From the manual: Whenever Nix attempts to realise a derivation for which a closure is already known, but this closure cannot be realised, fall back on normalising the derivation. The most common scenario in which this is useful is when we have registered substitutes in order to perform binary distribution from, say, a network repository. If the repository is down, the realisation of the derivation will fail. When this option is specified, Nix will build the derivation instead. Thus, binary installation falls back on a source installation. This option is not the default since it is generally not desirable for a transient failure in obtaining the substitutes to lead to a full build from source (with the related consumption of resources).
2004-06-28 10:42:57 +00:00
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
void Goal::amDone(ExitCode result)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
trace("done");
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
assert(exitCode == ecBusy);
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
assert(result == ecSuccess || result == ecFailed || result == ecNoSubstituters);
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
exitCode = result;
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (WeakGoals::iterator, i, waiters) {
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
GoalPtr goal = i->lock();
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
if (goal) goal->waiteeDone(shared_from_this(), result);
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
}
waiters.clear();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
worker.removeGoal(shared_from_this());
}
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
void Goal::trace(const format & f)
{
* Subtle bug in the builder: if a subgoal that is instantiated multiple times is also a top-level goal, then the second and later instantiations would never be created because there would be a stable pointer to the first one that would keep it alive in the WeakGoalMap. * Some tracing code for debugging this kind of problem.
2005-02-18 09:50:20 +00:00
debug(format("%1%: %2%") % name % f);
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Reduce gratuitous cut & pasting.
2004-06-22 10:21:44 +00:00
//////////////////////////////////////////////////////////////////////
/* Common initialisation performed in child processes. */
void commonChildInit(Pipe & logPipe)
{
* Use setsid instead of setpgrp in child processes. This not only creates a new process group but also a new session. New sessions have no controlling tty, so child processes like ssh cannot open /dev/tty (which is bad).
2006-02-03 14:20:59 +00:00
/* Put the child in a separate session (and thus a separate
process group) so that it has no controlling terminal (meaning
that e.g. ssh cannot open /dev/tty) and it doesn't receive
terminal signals. */
if (setsid() == -1)
throw SysError(format("creating a new session"));
* Reduce gratuitous cut & pasting.
2004-06-22 10:21:44 +00:00
/* Dup the write side of the logger pipe into stderr. */
if (dup2(logPipe.writeSide, STDERR_FILENO) == -1)
throw SysError("cannot pipe standard error into log file");
logPipe.readSide.close();
* Use setsid instead of setpgrp in child processes. This not only creates a new process group but also a new session. New sessions have no controlling tty, so child processes like ssh cannot open /dev/tty (which is bad).
2006-02-03 14:20:59 +00:00
/* Dup stderr to stdout. */
* Reduce gratuitous cut & pasting.
2004-06-22 10:21:44 +00:00
if (dup2(STDERR_FILENO, STDOUT_FILENO) == -1)
throw SysError("cannot dup stderr into stdout");
/* Reroute stdin to /dev/null. */
int fdDevNull = open(pathNullDevice.c_str(), O_RDWR);
if (fdDevNull == -1)
throw SysError(format("cannot open `%1%'") % pathNullDevice);
if (dup2(fdDevNull, STDIN_FILENO) == -1)
throw SysError("cannot dup null device into stdin");
Don't leak a file descriptor in commonChildInit()
2012-03-05 19:28:09 +00:00
close(fdDevNull);
* Reduce gratuitous cut & pasting.
2004-06-22 10:21:44 +00:00
}
* Multiple and/or failing substitutes now work.
2004-06-24 13:40:38 +00:00
/* Convert a string list to an array of char pointers. Careful: the
string list should outlive the array. */
* Reduce gratuitous cut & pasting.
2004-06-22 10:21:44 +00:00
const char * * strings2CharPtrs(const Strings & ss)
{
const char * * arr = new const char * [ss.size() + 1];
const char * * p = arr;
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (Strings::const_iterator, i, ss) *p++ = i->c_str();
* Reduce gratuitous cut & pasting.
2004-06-22 10:21:44 +00:00
*p = 0;
return arr;
}
* Restore SIGPIPE to SIG_DFL when running the builder. This prevents subtle and often hard-to-reproduce bugs where programs in pipes either barf with a "Broken pipe" message or not, depending on the exact timing conditions. This particularly happened in GNU M4 (and Bison, which uses M4).
2008-11-14 15:46:45 +00:00
/* Restore default handling of SIGPIPE, otherwise some programs will
randomly say "Broken pipe". */
static void restoreSIGPIPE()
{
struct sigaction act, oact;
act.sa_handler = SIG_DFL;
act.sa_flags = 0;
if (sigaction(SIGPIPE, &act, &oact)) throw SysError("resetting SIGPIPE");
}
* _exit() doesn't seem to work right on Cygwin.
2006-06-19 16:24:15 +00:00
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
//////////////////////////////////////////////////////////////////////
class UserLock
{
private:
/* POSIX locks suck. If we have a lock on a file, and we open and
close that file again (without closing the original file
descriptor), we lose the lock. So we have to be *very* careful
not to open a lock file on which we are holding a lock. */
static PathSet lockedPaths; /* !!! not thread-safe */
Path fnUserLock;
AutoCloseFD fdUserLock;
* Pass the actual build user to the setuid helper.
2006-12-07 11:27:32 +00:00
string user;
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
uid_t uid;
* Get rid of `build-users'. We'll just take all the members of `build-users-group'. This makes configuration easier: you can just add users in /etc/group.
2006-12-06 20:00:15 +00:00
gid_t gid;
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
public:
UserLock();
~UserLock();
void acquire();
void release();
* If not running as root, let the setuid helper kill the build user's processes before and after the build.
2006-12-07 16:33:31 +00:00
void kill();
* Pass the actual build user to the setuid helper.
2006-12-07 11:27:32 +00:00
string getUser() { return user; }
uid_t getUID() { return uid; }
uid_t getGID() { return gid; }
* Move killUser() to libutil so that the setuid helper can use it.
2006-12-07 00:16:07 +00:00
* Pass the actual build user to the setuid helper.
2006-12-07 11:27:32 +00:00
bool enabled() { return uid != 0; }
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
};
PathSet UserLock::lockedPaths;
UserLock::UserLock()
{
* Get rid of `build-users'. We'll just take all the members of `build-users-group'. This makes configuration easier: you can just add users in /etc/group.
2006-12-06 20:00:15 +00:00
uid = gid = 0;
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
}
UserLock::~UserLock()
{
release();
}
void UserLock::acquire()
{
assert(uid == 0);
* Get rid of `build-users'. We'll just take all the members of `build-users-group'. This makes configuration easier: you can just add users in /etc/group.
2006-12-06 20:00:15 +00:00
string buildUsersGroup = querySetting("build-users-group", "");
assert(buildUsersGroup != "");
/* Get the members of the build-users-group. */
struct group * gr = getgrnam(buildUsersGroup.c_str());
if (!gr)
throw Error(format("the group `%1%' specified in `build-users-group' does not exist")
% buildUsersGroup);
gid = gr->gr_gid;
/* Copy the result of getgrnam. */
Strings users;
for (char * * p = gr->gr_mem; *p; ++p) {
debug(format("found build user `%1%'") % *p);
users.push_back(*p);
}
if (users.empty())
throw Error(format("the build users group `%1%' has no members")
% buildUsersGroup);
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
* Get rid of `build-users'. We'll just take all the members of `build-users-group'. This makes configuration easier: you can just add users in /etc/group.
2006-12-06 20:00:15 +00:00
/* Find a user account that isn't currently in use for another
build. */
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (Strings::iterator, i, users) {
* Be less chatty.
2006-12-05 19:01:19 +00:00
debug(format("trying user `%1%'") % *i);
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
struct passwd * pw = getpwnam(i->c_str());
if (!pw)
* Get rid of `build-users'. We'll just take all the members of `build-users-group'. This makes configuration easier: you can just add users in /etc/group.
2006-12-06 20:00:15 +00:00
throw Error(format("the user `%1%' in the group `%2%' does not exist")
% *i % buildUsersGroup);
* Create some state directories automatically as a convenience.
2009-09-23 17:05:51 +00:00
createDirs(nixStateDir + "/userpool");
* Sanity check.
2006-12-07 00:19:27 +00:00
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
fnUserLock = (format("%1%/userpool/%2%") % nixStateDir % pw->pw_uid).str();
if (lockedPaths.find(fnUserLock) != lockedPaths.end())
/* We already have a lock on this one. */
continue;
AutoCloseFD fd = open(fnUserLock.c_str(), O_RDWR | O_CREAT, 0600);
if (fd == -1)
throw SysError(format("opening user lock `%1%'") % fnUserLock);
Set the close-on-exec flag on file descriptors
2012-03-05 19:29:00 +00:00
closeOnExec(fd);
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
if (lockFile(fd, ltWrite, false)) {
fdUserLock = fd.borrow();
lockedPaths.insert(fnUserLock);
* Pass the actual build user to the setuid helper.
2006-12-07 11:27:32 +00:00
user = *i;
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
uid = pw->pw_uid;
* Sanity check.
2006-12-07 00:19:27 +00:00
/* Sanity check... */
if (uid == getuid() || uid == geteuid())
throw Error(format("the Nix user should not be a member of `%1%'")
% buildUsersGroup);
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
return;
}
}
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
throw Error(format("all build users are currently in use; "
* Get rid of `build-users'. We'll just take all the members of `build-users-group'. This makes configuration easier: you can just add users in /etc/group.
2006-12-06 20:00:15 +00:00
"consider creating additional users and adding them to the `%1%' group")
% buildUsersGroup);
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
}
void UserLock::release()
{
if (uid == 0) return;
fdUserLock.close(); /* releases lock */
assert(lockedPaths.find(fnUserLock) != lockedPaths.end());
lockedPaths.erase(fnUserLock);
fnUserLock = "";
uid = 0;
}
* If not running as root, let the setuid helper kill the build user's processes before and after the build.
2006-12-07 16:33:31 +00:00
static void runSetuidHelper(const string & command,
const string & arg)
* Change the ownership of store paths to the Nix account before deleting them using the setuid helper.
2006-12-07 14:14:35 +00:00
{
* Override the setuid helper using NIX_SETUID_HELPER.
2007-03-20 22:04:25 +00:00
Path program = getEnv("NIX_SETUID_HELPER",
nixLibexecDir + "/nix-setuid-helper");
* Change the ownership of store paths to the Nix account before deleting them using the setuid helper.
2006-12-07 14:14:35 +00:00
/* Fork. */
Pid pid;
pid = fork();
switch (pid) {
case -1:
throw SysError("unable to fork");
case 0: /* child */
try {
* `nix-store --export --sign': sign the Nix archive using the RSA key in /nix/etc/nix/signing-key.sec
2007-02-21 14:31:42 +00:00
std::vector<const char *> args; /* careful with c_str()! */
* Change the ownership of store paths to the Nix account before deleting them using the setuid helper.
2006-12-07 14:14:35 +00:00
args.push_back(program.c_str());
* If not running as root, let the setuid helper kill the build user's processes before and after the build.
2006-12-07 16:33:31 +00:00
args.push_back(command.c_str());
args.push_back(arg.c_str());
* Change the ownership of store paths to the Nix account before deleting them using the setuid helper.
2006-12-07 14:14:35 +00:00
args.push_back(0);
* Restore SIGPIPE to SIG_DFL when running the builder. This prevents subtle and often hard-to-reproduce bugs where programs in pipes either barf with a "Broken pipe" message or not, depending on the exact timing conditions. This particularly happened in GNU M4 (and Bison, which uses M4).
2008-11-14 15:46:45 +00:00
restoreSIGPIPE();
* Change the ownership of store paths to the Nix account before deleting them using the setuid helper.
2006-12-07 14:14:35 +00:00
execve(program.c_str(), (char * *) &args[0], 0);
throw SysError(format("executing `%1%'") % program);
}
catch (std::exception & e) {
std::cerr << "error: " << e.what() << std::endl;
}
quickExit(1);
}
/* Parent. */
/* Wait for the child to finish. */
int status = pid.wait(true);
if (!statusOk(status))
throw Error(format("program `%1%' %2%")
% program % statusToString(status));
}
* If not running as root, let the setuid helper kill the build user's processes before and after the build.
2006-12-07 16:33:31 +00:00
void UserLock::kill()
{
assert(enabled());
if (amPrivileged())
killUser(uid);
else
runSetuidHelper("kill", user);
}
bool amPrivileged()
{
return geteuid() == 0;
}
bool haveBuildUsers()
{
return querySetting("build-users-group", "") != "";
}
void getOwnership(const Path & path)
{
runSetuidHelper("get-ownership", path);
}
* Use deletePathWrapped() in more places.
2006-12-09 00:26:24 +00:00
void deletePathWrapped(const Path & path,
* Some refactoring: put the GC options / results in separate structs. * The garbage collector now also prints the number of blocks freed.
2008-06-18 09:34:17 +00:00
unsigned long long & bytesFreed, unsigned long long & blocksFreed)
* Change the ownership of store paths to the Nix account before deleting them using the setuid helper.
2006-12-07 14:14:35 +00:00
{
* Use deletePathWrapped() in more places.
2006-12-09 00:26:24 +00:00
try {
/* First try to delete it ourselves. */
* Some refactoring: put the GC options / results in separate structs. * The garbage collector now also prints the number of blocks freed.
2008-06-18 09:34:17 +00:00
deletePath(path, bytesFreed, blocksFreed);
* Use deletePathWrapped() in more places.
2006-12-09 00:26:24 +00:00
} catch (SysError & e) {
/* If this failed due to a permission error, then try it with
the setuid helper. */
if (haveBuildUsers() && !amPrivileged()) {
getOwnership(path);
* Some refactoring: put the GC options / results in separate structs. * The garbage collector now also prints the number of blocks freed.
2008-06-18 09:34:17 +00:00
deletePath(path, bytesFreed, blocksFreed);
* Use deletePathWrapped() in more places.
2006-12-09 00:26:24 +00:00
} else
throw;
}
}
void deletePathWrapped(const Path & path)
{
* Some refactoring: put the GC options / results in separate structs. * The garbage collector now also prints the number of blocks freed.
2008-06-18 09:34:17 +00:00
unsigned long long dummy1, dummy2;
deletePathWrapped(path, dummy1, dummy2);
* Change the ownership of store paths to the Nix account before deleting them using the setuid helper.
2006-12-07 14:14:35 +00:00
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
//////////////////////////////////////////////////////////////////////
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
struct HookInstance
{
/* Pipes for talking to the build hook. */
Pipe toHook;
/* Pipe for the hook's standard output/error. */
Pipe fromHook;
* When using the build hook, distinguish between the stderr of the hook script proper, and the stdout/stderr of the builder. Only the latter should be saved in /nix/var/log/nix/drvs. * Allow the verbosity to be set through an option. * Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
/* Pipe for the builder's standard output/error. */
Pipe builderOut;
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
/* The process ID of the hook. */
Pid pid;
HookInstance();
~HookInstance();
};
HookInstance::HookInstance()
{
debug("starting build hook");
Path buildHook = absPath(getEnv("NIX_BUILD_HOOK"));
/* Create a pipe to get the output of the child. */
fromHook.create();
/* Create the communication pipes. */
toHook.create();
* When using the build hook, distinguish between the stderr of the hook script proper, and the stdout/stderr of the builder. Only the latter should be saved in /nix/var/log/nix/drvs. * Allow the verbosity to be set through an option. * Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
/* Create a pipe to get the output of the builder. */
builderOut.create();
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
/* Fork the hook. */
pid = fork();
switch (pid) {
case -1:
throw SysError("unable to fork");
case 0:
try { /* child */
commonChildInit(fromHook);
if (chdir("/") == -1) throw SysError("changing into `/");
/* Dup the communication pipes. */
toHook.writeSide.close();
if (dup2(toHook.readSide, STDIN_FILENO) == -1)
throw SysError("dupping to-hook read side");
* When using the build hook, distinguish between the stderr of the hook script proper, and the stdout/stderr of the builder. Only the latter should be saved in /nix/var/log/nix/drvs. * Allow the verbosity to be set through an option. * Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
/* Use fd 4 for the builder's stdout/stderr. */
builderOut.readSide.close();
if (dup2(builderOut.writeSide, 4) == -1)
throw SysError("dupping builder's stdout/stderr");
Add support for the `build-timeout' and `--timeout' options.
2011-06-30 15:19:13 +00:00
/* XXX: Pass `buildTimeout' to the hook? */
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
execl(buildHook.c_str(), buildHook.c_str(), thisSystem.c_str(),
* Always print hook output on stderr, even if --no-build-output is set. * In the build hook, print a trace message to allow Hydra to pick up the name of the remote machine used for the build.
2010-08-31 12:36:24 +00:00
(format("%1%") % maxSilentTime).str().c_str(),
(format("%1%") % printBuildTrace).str().c_str(),
NULL);
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
throw SysError(format("executing `%1%'") % buildHook);
} catch (std::exception & e) {
std::cerr << format("build hook error: %1%") % e.what() << std::endl;
}
quickExit(1);
}
/* parent */
pid.setSeparatePG(true);
pid.setKillSignal(SIGTERM);
fromHook.writeSide.close();
toHook.readSide.close();
}
HookInstance::~HookInstance()
{
try {
/* Cleanly shut down the hook by closing its stdin if it's not
already building. Otherwise pid's destructor will kill
it. */
if (pid != -1 && toHook.writeSide != -1) {
toHook.writeSide.close();
pid.wait(true);
}
} catch (...) {
ignoreException();
}
}
//////////////////////////////////////////////////////////////////////
typedef enum {rpAccept, rpDecline, rpPostpone} HookReply;
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
class SubstitutionGoal;
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
class DerivationGoal : public Goal
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
{
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
private:
* Terminology fixes.
2005-01-20 16:01:07 +00:00
/* The path of the derivation. */
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
Path drvPath;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
* Terminology fixes.
2005-01-20 16:01:07 +00:00
/* The derivation stored at drvPath. */
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
Derivation drv;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
/* The remainder is state held during the build. */
/* Locks on the output paths. */
PathLocks outputLocks;
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
/* All input paths (that is, the union of FS closures of the
immediate input paths). */
PathSet inputPaths;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
/* Referenceable paths (i.e., input and output paths). */
PathSet allPaths;
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
/* User selected for running the builder. */
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
UserLock buildUser;
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
/* The process ID of the builder. */
* Wrapper class around pids.
2004-06-22 09:51:44 +00:00
Pid pid;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
/* The temporary directory. */
Path tmpDir;
/* File descriptor for the log file. */
Compress build logs on the fly using bzip2
2012-05-30 14:12:29 +00:00
FILE * fLogFile;
BZFILE * bzLogFile;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
/* Pipe for the builder's standard output/error. */
* When using the build hook, distinguish between the stderr of the hook script proper, and the stdout/stderr of the builder. Only the latter should be saved in /nix/var/log/nix/drvs. * Allow the verbosity to be set through an option. * Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
Pipe builderOut;
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
/* The build hook. */
boost::shared_ptr<HookInstance> hook;
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 00:46:59 +00:00
/* Whether we're currently doing a chroot build. */
bool useChroot;
* Don't provide the whole Nix store in the chroot, but only the closure of the inputs. This really enforces that there can't be any undeclared dependencies on paths in the store. This is done by creating a fake Nix store and creating bind-mounts or hard-links in the fake store for all paths in the closure. After the build, the build output is moved from the fake store to the real store. TODO: the chroot has to be on the same filesystem as the Nix store for this to work, but this isn't enforced yet. (I.e. it only works currently if /tmp is on the same FS as /nix/store.)
2008-12-11 18:57:10 +00:00
Path chrootRootDir;
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 00:46:59 +00:00
* Do chroot builds in a private namespace. This means that all the bind-mounts we do are only visible to the builder process and its children. So accidentally doing "rm -rf" on the chroot directory won't wipe out /nix/store and other bind-mounted directories anymore. Also, the bind-mounts in the private namespace disappear automatically when the builder exits.
2008-12-11 17:00:12 +00:00
/* RAII object to delete the chroot directory. */
boost::shared_ptr<AutoDelete> autoDelChroot;
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
Fix chroots builds Chroots are initialised by hard-linking inputs from the Nix store to the chroot. This doesn't work if the input has its immutable bit set, because it's forbidden to create hard links to immutable files. So temporarily clear the immutable bit when creating and destroying the chroot. Note that making regular files in the Nix store immutable isn't very reliable, since the bit can easily become cleared: for instance, if we run the garbage collector after running ‘nix-store --optimise’. So maybe we should only make directories immutable.
2012-02-18 00:23:52 +00:00
/* All inputs that are regular files. */
PathSet regularInputPaths;
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
/* Whether this is a fixed-output derivation. */
bool fixedOutput;
* Delete the chroot directory automatically. * Removed some debug messages.
2007-10-27 16:06:38 +00:00
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
typedef void (DerivationGoal::*GoalState)();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
GoalState state;
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
/* Stuff we need to pass to initChild(). */
PathSet dirsInChroot;
typedef map<string, string> Environment;
Environment env;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
public:
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
DerivationGoal(const Path & drvPath, Worker & worker);
~DerivationGoal();
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
void cancel();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
void work();
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
Path getDrvPath()
{
return drvPath;
}
Fixed exportBuildReferenceGraph
2008-01-15 04:32:08 +00:00
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
private:
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* The states. */
void init();
* Remove ancient terminology.
2006-12-07 23:58:36 +00:00
void haveDerivation();
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 10:55:33 +00:00
void outputsSubstituted();
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
void inputsRealised();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
void tryToBuild();
void buildDone();
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
/* Is the build hook willing to perform the build? */
HookReply tryBuildHook();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Start building a derivation. */
void startBuilder();
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
/* Initialise the builder's process. */
void initChild();
friend int childEntry(void *);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Must be called after the output paths have become valid (either
due to a successful build or hook, or because they already
were). */
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
void computeClosure();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Open a log file and a pipe to it. */
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
Path openLogFile();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
Compress build logs on the fly using bzip2
2012-05-30 14:12:29 +00:00
/* Close the log file. */
void closeLogFile();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Delete the temporary directory, if we have one. */
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
void deleteTmpDir(bool force);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Write build logs to disk again.
2004-06-29 09:41:50 +00:00
/* Callback used by the worker to write to the log. */
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
void handleChildOutput(int fd, const string & data);
void handleEOF(int fd);
* Write build logs to disk again.
2004-06-29 09:41:50 +00:00
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 10:55:33 +00:00
/* Return the set of (in)valid paths. */
PathSet checkPathValidity(bool returnValid);
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
/* Abort the goal if `path' failed to build. */
bool pathFailed(const Path & path);
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
/* Forcibly kill the child process, if any. */
void killChild();
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
};
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
DerivationGoal::DerivationGoal(const Path & drvPath, Worker & worker)
: Goal(worker)
Compress build logs on the fly using bzip2
2012-05-30 14:12:29 +00:00
, fLogFile(0)
, bzLogFile(0)
Fix an uninitialised variable The variable ‘useChroot’ was not initialised properly. This caused random failures if using the build hook. Seen on Mac OS X 10.7 with Clang. Thanks to KolibriFX for finding this :-)
2012-02-24 19:57:54 +00:00
, useChroot(false)
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
{
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
this->drvPath = drvPath;
state = &DerivationGoal::init;
* Subtle bug in the builder: if a subgoal that is instantiated multiple times is also a top-level goal, then the second and later instantiations would never be created because there would be a stable pointer to the first one that would keep it alive in the WeakGoalMap. * Some tracing code for debugging this kind of problem.
2005-02-18 09:50:20 +00:00
name = (format("building of `%1%'") % drvPath).str();
trace("created");
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
}
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
DerivationGoal::~DerivationGoal()
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
{
/* Careful: we should never ever throw an exception from a
destructor. */
try {
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
killChild();
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
deleteTmpDir(false);
Compress build logs on the fly using bzip2
2012-05-30 14:12:29 +00:00
closeLogFile();
* Set a terminate() handler to ensure that we leave the BDB environment cleanly even when an exception is thrown from a destructor. We still crash, but we don't take all other Nix processes with us.
2007-05-01 15:16:17 +00:00
} catch (...) {
ignoreException();
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
}
}
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
void DerivationGoal::killChild()
{
if (pid != -1) {
worker.childTerminated(pid);
if (buildUser.enabled()) {
/* We can't use pid.kill(), since we may not have the
appropriate privilege. I.e., if we're not root, then
setuid helper should do it).
Also, if we're using a build user, then there is a
tricky race condition: if we kill the build user before
the child has done its setuid() to the build user uid,
then it won't be killed, and we'll potentially lock up
in pid.wait(). So also send a conventional kill to the
child. */
::kill(-pid, SIGKILL); /* ignore the result */
buildUser.kill();
pid.wait(true);
} else
pid.kill();
assert(pid == -1);
}
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
hook.reset();
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
}
void DerivationGoal::cancel()
{
killChild();
amDone(ecFailed);
}
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
void DerivationGoal::work()
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
{
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
(this->*state)();
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
}
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
void DerivationGoal::init()
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
{
* Started making Nix's diagnostic messages a bit more useful.
2004-06-22 14:48:59 +00:00
trace("init");
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
* Give a better error message when trying to build something and readOnlyMode is set.
2009-12-09 17:45:22 +00:00
if (readOnlyMode)
throw Error(format("cannot build derivation `%1%' - no write access to the Nix store") % drvPath);
* Terminology fixes.
2005-01-20 16:01:07 +00:00
/* The first thing to do is to make sure that the derivation
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
exists. If it doesn't, it may be created through a
substitute. */
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
addWaitee(worker.makeSubstitutionGoal(drvPath));
* normaliseFState() now locks all output paths prior to building, thus ensuring that simultaneous invocations of Nix don't clobber each other's builds. * Fixed a bug in `make install'.
2003-08-01 14:11:19 +00:00
* Remove ancient terminology.
2006-12-07 23:58:36 +00:00
state = &DerivationGoal::haveDerivation;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
}
* Remove ancient terminology.
2006-12-07 23:58:36 +00:00
void DerivationGoal::haveDerivation()
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
{
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
trace("loading derivation");
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
if (nrFailed != 0) {
* Don't provide the whole Nix store in the chroot, but only the closure of the inputs. This really enforces that there can't be any undeclared dependencies on paths in the store. This is done by creating a fake Nix store and creating bind-mounts or hard-links in the fake store for all paths in the closure. After the build, the build output is moved from the fake store to the real store. TODO: the chroot has to be on the same filesystem as the Nix store for this to work, but this isn't enforced yet. (I.e. it only works currently if /tmp is on the same FS as /nix/store.)
2008-12-11 18:57:10 +00:00
printMsg(lvlError, format("cannot build missing derivation `%1%'") % drvPath);
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
amDone(ecFailed);
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
return;
}
* Put chroots in the Nix store. This ensures that we can create hard links to the inputs.
2008-12-12 17:14:57 +00:00
/* `drvPath' should already be a root, but let's be on the safe
side: if the user forgot to make it a root, we wouldn't want
things being garbage collected while we're busy. */
worker.store.addTempRoot(drvPath);
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
assert(worker.store.isValidPath(drvPath));
* Refactorings.
2003-07-20 19:29:38 +00:00
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
/* Get the derivation. */
* Eliminate all uses of the global variable ‘store’ from libstore. This should also fix: nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed. which was caused by hashDerivationModulo() calling the ‘store’ object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
drv = derivationFromPath(worker.store, drvPath);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
* Cleanup.
2009-03-17 11:42:55 +00:00
foreach (DerivationOutputs::iterator, i, drv.outputs)
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
worker.store.addTempRoot(i->second.path);
* Start of concurrent garbage collection. Processes write temporary roots to a per-process temporary file in /nix/var/nix/temproots while holding a write lock on that file. The garbage collector acquires read locks on all those files, thus blocking further progress in other Nix processes, and reads the sets of temporary roots.
2005-01-31 10:27:25 +00:00
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 10:55:33 +00:00
/* Check what outputs paths are not already valid. */
PathSet invalidOutputs = checkPathValidity(false);
/* If they are all valid, then we're done. */
if (invalidOutputs.size() == 0) {
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
amDone(ecSuccess);
return;
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 10:55:33 +00:00
}
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
/* Check whether any output previously failed to build. If so,
don't bother. */
foreach (PathSet::iterator, i, invalidOutputs)
if (pathFailed(*i)) return;
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 10:55:33 +00:00
/* We are first going to try to create the invalid output paths
through substitutes. If that doesn't work, we'll build
them. */
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
if (queryBoolSetting("build-use-substitutes", true))
foreach (PathSet::iterator, i, invalidOutputs)
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 10:55:33 +00:00
addWaitee(worker.makeSubstitutionGoal(*i));
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 10:55:33 +00:00
if (waitees.empty()) /* to prevent hang (no wake-up event) */
outputsSubstituted();
else
state = &DerivationGoal::outputsSubstituted;
}
void DerivationGoal::outputsSubstituted()
{
trace("all outputs substituted (maybe)");
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
if (nrFailed > 0 && nrFailed > nrNoSubstituters && !tryFallback)
* Fix the `--fallback' switch. * Fix the substitutes tests.
2005-01-25 13:00:12 +00:00
throw Error(format("some substitutes for the outputs of derivation `%1%' failed; try `--fallback'") % drvPath);
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
nrFailed = nrNoSubstituters = 0;
* Fix the `--fallback' switch. * Fix the substitutes tests.
2005-01-25 13:00:12 +00:00
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 10:55:33 +00:00
if (checkPathValidity(false).size() == 0) {
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
amDone(ecSuccess);
return;
* execl() requires a terminating 0. * When a fast build wakes up a goal, try to start that goal in the same iteration of the startBuild() loop of run(). Otherwise no job might be started until the next job terminates.
2004-05-14 12:24:29 +00:00
}
* normaliseFState() now locks all output paths prior to building, thus ensuring that simultaneous invocations of Nix don't clobber each other's builds. * Fixed a bug in `make install'.
2003-08-01 14:11:19 +00:00
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 10:55:33 +00:00
/* Otherwise, at least one of the output paths could not be
produced using a substitute. So we have to build instead. */
/* The inputs must be built before we can build this goal. */
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
foreach (DerivationInputs::iterator, i, drv.inputDrvs)
addWaitee(worker.makeDerivationGoal(i->first));
* normaliseFState() now locks all output paths prior to building, thus ensuring that simultaneous invocations of Nix don't clobber each other's builds. * Fixed a bug in `make install'.
2003-08-01 14:11:19 +00:00
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (PathSet::iterator, i, drv.inputSrcs)
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
addWaitee(worker.makeSubstitutionGoal(*i));
* Handle the unlikely case where a derivation has no dependencies at all.
2010-08-24 11:45:44 +00:00
if (waitees.empty()) /* to prevent hang (no wake-up event) */
inputsRealised();
else
state = &DerivationGoal::inputsRealised;
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
}
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
void DerivationGoal::inputsRealised()
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
{
* Started making Nix's diagnostic messages a bit more useful.
2004-06-22 14:48:59 +00:00
trace("all inputs realised");
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
if (nrFailed != 0) {
printMsg(lvlError,
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
format("cannot build derivation `%1%': "
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
"%2% dependencies couldn't be built")
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
% drvPath % nrFailed);
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
amDone(ecFailed);
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
return;
}
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
/* Gather information necessary for computing the closure and/or
running the build hook. */
/* The outputs are referenceable paths. */
foreach (DerivationOutputs::iterator, i, drv.outputs) {
debug(format("building path `%1%'") % i->second.path);
allPaths.insert(i->second.path);
Fixed exportBuildReferenceGraph
2008-01-15 04:32:08 +00:00
}
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
/* Determine the full set of input paths. */
/* First, the input derivations. */
foreach (DerivationInputs::iterator, i, drv.inputDrvs) {
/* Add the relevant output closures of the input derivation
`*i' as input paths. Only add the closures of output paths
that are specified as inputs. */
assert(worker.store.isValidPath(i->first));
* Eliminate all uses of the global variable ‘store’ from libstore. This should also fix: nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed. which was caused by hashDerivationModulo() calling the ‘store’ object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
Derivation inDrv = derivationFromPath(worker.store, i->first);
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
foreach (StringSet::iterator, j, i->second)
if (inDrv.outputs.find(*j) != inDrv.outputs.end())
* Eliminate all uses of the global variable ‘store’ from libstore. This should also fix: nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed. which was caused by hashDerivationModulo() calling the ‘store’ object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
computeFSClosure(worker.store, inDrv.outputs[*j].path, inputPaths);
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
else
throw Error(
format("derivation `%1%' requires non-existent output `%2%' from input derivation `%3%'")
% drvPath % *j % i->first);
}
/* Second, the input sources. */
foreach (PathSet::iterator, i, drv.inputSrcs)
* Eliminate all uses of the global variable ‘store’ from libstore. This should also fix: nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed. which was caused by hashDerivationModulo() calling the ‘store’ object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
computeFSClosure(worker.store, *i, inputPaths);
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
debug(format("added input paths %1%") % showPaths(inputPaths));
allPaths.insert(inputPaths.begin(), inputPaths.end());
* The `fixedOutput' variable didn't get initialised when using a build hook, causing negative caching to fail randomly.
2009-03-28 16:12:27 +00:00
/* Is this a fixed-output derivation? */
fixedOutput = true;
foreach (DerivationOutputs::iterator, i, drv.outputs)
if (i->second.hash == "") fixedOutput = false;
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
/* Okay, try to build. Note that here we don't wait for a build
slot to become available, since we don't need one if there is a
build hook. */
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
state = &DerivationGoal::tryToBuild;
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
worker.wakeUp(shared_from_this());
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
}
* Refactoring.
2009-03-22 23:53:05 +00:00
PathSet outputPaths(const DerivationOutputs & outputs)
{
PathSet paths;
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (DerivationOutputs::const_iterator, i, outputs)
* Refactoring.
2009-03-22 23:53:05 +00:00
paths.insert(i->second.path);
return paths;
}
* Allow derivations to hint that they should not be built remotely using the build hook mechanism, by setting the derivation attribute "preferLocalBuild" to true. This has a few use cases: - The user environment builder. Since it just creates a bunch of symlinks without much computation, there is no reason to do it remotely. In fact, doing it remotely requires the entire closure of the user environment to be copied to the remote machine, which is extremely wasteful. - `fetchurl'. Performing the download on a remote machine and then copying it to the local machine involves twice as much network traffic as performing the download locally, and doesn't save any CPU cycles on the local machine.
2010-08-04 12:13:58 +00:00
static bool canBuildLocally(const string & platform)
{
return platform == thisSystem
#ifdef CAN_DO_LINUX32_BUILDS
|| (platform == "i686-linux" && thisSystem == "x86_64-linux")
#endif
;
}
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
void DerivationGoal::tryToBuild()
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
{
* Started making Nix's diagnostic messages a bit more useful.
2004-06-22 14:48:59 +00:00
trace("trying to build");
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Refactoring.
2009-03-22 23:53:05 +00:00
/* Check for the possibility that some other goal in this process
has locked the output since we checked in haveDerivation().
(It can't happen between here and the lockPaths() call below
because we're not allowing multi-threading.) If so, put this
goal to sleep until another goal finishes, then try again. */
foreach (DerivationOutputs::iterator, i, drv.outputs)
if (pathIsLockedByMe(i->second.path)) {
debug(format("putting derivation `%1%' to sleep because `%2%' is locked by another goal")
% drvPath % i->second.path);
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
worker.waitForAnyGoal(shared_from_this());
return;
* Refactoring.
2009-03-22 23:53:05 +00:00
}
/* Obtain locks on all output paths. The locks are automatically
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
released when we exit this function or Nix crashes. If we
can't acquire the lock, then continue; hopefully some other
goal can start a build, and if not, the main loop will sleep a
few seconds and then retry this goal. */
if (!outputLocks.lockPaths(outputPaths(drv.outputs), "", false)) {
worker.waitForAWhile(shared_from_this());
return;
}
* Refactoring.
2009-03-22 23:53:05 +00:00
/* Now check again whether the outputs are valid. This is because
another process may have started building in parallel. After
it has finished and released the locks, we can (and should)
reuse its results. (Strictly speaking the first check can be
omitted, but that would be less efficient.) Note that since we
now hold the locks on the output paths, no other process can
build this derivation, so no further checks are necessary. */
PathSet validPaths = checkPathValidity(true);
if (validPaths.size() == drv.outputs.size()) {
debug(format("skipping build of derivation `%1%', someone beat us to it")
% drvPath);
outputLocks.setDeletion(true);
amDone(ecSuccess);
return;
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
}
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
* Refactoring.
2009-03-22 23:53:05 +00:00
if (validPaths.size() > 0)
/* !!! fix this; try to delete valid paths */
throw Error(
format("derivation `%1%' is blocked by its output paths")
% drvPath);
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
* Refactoring.
2009-03-22 23:53:05 +00:00
/* If any of the outputs already exist but are not valid, delete
them. */
foreach (DerivationOutputs::iterator, i, drv.outputs) {
Path path = i->second.path;
if (worker.store.isValidPath(path))
throw Error(format("obstructed build: path `%1%' exists") % path);
if (pathExists(path)) {
debug(format("removing unregistered path `%1%'") % path);
deletePathWrapped(path);
}
}
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
/* Check again whether any output previously failed to build,
because some other process may have tried and failed before we
acquired the lock. */
foreach (DerivationOutputs::iterator, i, drv.outputs)
if (pathFailed(i->second.path)) return;
* Allow derivations to hint that they should not be built remotely using the build hook mechanism, by setting the derivation attribute "preferLocalBuild" to true. This has a few use cases: - The user environment builder. Since it just creates a bunch of symlinks without much computation, there is no reason to do it remotely. In fact, doing it remotely requires the entire closure of the user environment to be copied to the remote machine, which is extremely wasteful. - `fetchurl'. Performing the download on a remote machine and then copying it to the local machine involves twice as much network traffic as performing the download locally, and doesn't save any CPU cycles on the local machine.
2010-08-04 12:13:58 +00:00
/* Don't do a remote build if the derivation has the attribute
`preferLocalBuild' set. */
bool preferLocalBuild =
drv.env["preferLocalBuild"] == "1" && canBuildLocally(drv.platform);
* Refactoring.
2009-03-22 23:53:05 +00:00
/* Is the build hook willing to accept this job? */
* Allow derivations to hint that they should not be built remotely using the build hook mechanism, by setting the derivation attribute "preferLocalBuild" to true. This has a few use cases: - The user environment builder. Since it just creates a bunch of symlinks without much computation, there is no reason to do it remotely. In fact, doing it remotely requires the entire closure of the user environment to be copied to the remote machine, which is extremely wasteful. - `fetchurl'. Performing the download on a remote machine and then copying it to the local machine involves twice as much network traffic as performing the download locally, and doesn't save any CPU cycles on the local machine.
2010-08-04 12:13:58 +00:00
if (!preferLocalBuild) {
switch (tryBuildHook()) {
case rpAccept:
/* Yes, it has started doing so. Wait until we get
EOF from the hook. */
state = &DerivationGoal::buildDone;
return;
case rpPostpone:
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
/* Not now; wait until at least one child finishes or
the wake-up timeout expires. */
* Allow derivations to hint that they should not be built remotely using the build hook mechanism, by setting the derivation attribute "preferLocalBuild" to true. This has a few use cases: - The user environment builder. Since it just creates a bunch of symlinks without much computation, there is no reason to do it remotely. In fact, doing it remotely requires the entire closure of the user environment to be copied to the remote machine, which is extremely wasteful. - `fetchurl'. Performing the download on a remote machine and then copying it to the local machine involves twice as much network traffic as performing the download locally, and doesn't save any CPU cycles on the local machine.
2010-08-04 12:13:58 +00:00
worker.waitForAWhile(shared_from_this());
outputLocks.unlock();
return;
case rpDecline:
/* We should do it ourselves. */
break;
}
* Refactoring.
2009-03-22 23:53:05 +00:00
}
* Allow derivations to hint that they should not be built remotely using the build hook mechanism, by setting the derivation attribute "preferLocalBuild" to true. This has a few use cases: - The user environment builder. Since it just creates a bunch of symlinks without much computation, there is no reason to do it remotely. In fact, doing it remotely requires the entire closure of the user environment to be copied to the remote machine, which is extremely wasteful. - `fetchurl'. Performing the download on a remote machine and then copying it to the local machine involves twice as much network traffic as performing the download locally, and doesn't save any CPU cycles on the local machine.
2010-08-04 12:13:58 +00:00
/* Make sure that we are allowed to start a build. If this
derivation prefers to be done locally, do it even if
maxBuildJobs is 0. */
unsigned int curBuilds = worker.getNrLocalBuilds();
if (curBuilds >= maxBuildJobs && !(preferLocalBuild && curBuilds == 0)) {
* Refactoring.
2009-03-22 23:53:05 +00:00
worker.waitForBuildSlot(shared_from_this());
outputLocks.unlock();
return;
}
try {
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
/* Okay, we have to build. */
startBuilder();
* normaliseFState() now locks all output paths prior to building, thus ensuring that simultaneous invocations of Nix don't clobber each other's builds. * Fixed a bug in `make install'.
2003-08-01 14:11:19 +00:00
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
} catch (BuildError & e) {
printMsg(lvlError, e.msg());
* Release output locks as soon as possible, not when the destructor of the DerivationGoal runs. Otherwise, if a goal is a top-level goal, then the lock won't be released until nix-store finishes. With --keep-going and lots of top-level goals, it's possible to run out of file descriptors (this happened sometimes in the build farm for Nixpkgs). Also, for failed derivation, it won't be possible to build it again until the lock is released. * Idem for locks on build users: these weren't released in a timely manner for failed top-level derivation goals. So if there were more than (say) 10 such failed builds, you would get an error about having run out of build users.
2009-02-16 09:24:20 +00:00
outputLocks.unlock();
buildUser.release();
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
if (printBuildTrace)
* Refactoring.
2009-03-22 23:53:05 +00:00
printMsg(lvlError, format("@ build-failed %1% %2% %3% %4%")
% drvPath % drv.outputs["out"].path % 0 % e.msg());
* nix-instantiate: return exit status 100 to denote a permanent build failure. The build hook can use this to distinguish between transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
worker.permanentFailure = true;
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
amDone(ecFailed);
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
return;
}
/* This state will be reached when we get EOF on the child's
log pipe. */
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
state = &DerivationGoal::buildDone;
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
}
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
void DerivationGoal::buildDone()
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
{
* Started making Nix's diagnostic messages a bit more useful.
2004-06-22 14:48:59 +00:00
trace("build done");
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
/* Since we got an EOF on the logger pipe, the builder is presumed
to have terminated. In fact, the builder could also have
simply have closed its end of the pipe --- just don't do that
:-) */
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
int status;
pid_t savedPid;
if (hook) {
savedPid = hook->pid;
status = hook->pid.wait(true);
} else {
/* !!! this could block! security problem! solution: kill the
child */
savedPid = pid;
status = pid.wait(true);
}
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Also for convenience, change the ownership of the build output even in case of failure.
2006-12-08 00:19:50 +00:00
debug(format("builder process for `%1%' finished") % drvPath);
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
/* So the child is gone now. */
* Wrapper class around pids.
2004-06-22 09:51:44 +00:00
worker.childTerminated(savedPid);
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
* Also for convenience, change the ownership of the build output even in case of failure.
2006-12-08 00:19:50 +00:00
/* Close the read side of the logger pipe. */
* When using the build hook, distinguish between the stderr of the hook script proper, and the stdout/stderr of the builder. Only the latter should be saved in /nix/var/log/nix/drvs. * Allow the verbosity to be set through an option. * Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
if (hook) {
hook->builderOut.readSide.close();
hook->fromHook.readSide.close();
}
else builderOut.readSide.close();
* Also for convenience, change the ownership of the build output even in case of failure.
2006-12-08 00:19:50 +00:00
/* Close the log file. */
Compress build logs on the fly using bzip2
2012-05-30 14:12:29 +00:00
closeLogFile();
* Also for convenience, change the ownership of the build output even in case of failure.
2006-12-08 00:19:50 +00:00
* Also kill all processes of the build user after the build. This is critical to prevent certain kinds of 0wnage.
2005-10-17 17:43:21 +00:00
/* When running under a build user, make sure that all processes
running under that uid are gone. This is to prevent a
malicious user from leaving behind a process that keeps files
open and modifies them after they have been chown'ed to
root. */
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
if (buildUser.enabled()) buildUser.kill();
* Also kill all processes of the build user after the build. This is critical to prevent certain kinds of 0wnage.
2005-10-17 17:43:21 +00:00
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
try {
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
/* Some cleanup per path. We do this here and not in
computeClosure() for convenience when the build has
failed. */
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
foreach (DerivationOutputs::iterator, i, drv.outputs) {
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
Path path = i->second.path;
* Don't provide the whole Nix store in the chroot, but only the closure of the inputs. This really enforces that there can't be any undeclared dependencies on paths in the store. This is done by creating a fake Nix store and creating bind-mounts or hard-links in the fake store for all paths in the closure. After the build, the build output is moved from the fake store to the real store. TODO: the chroot has to be on the same filesystem as the Nix store for this to work, but this isn't enforced yet. (I.e. it only works currently if /tmp is on the same FS as /nix/store.)
2008-12-11 18:57:10 +00:00
if (useChroot && pathExists(chrootRootDir + path)) {
if (rename((chrootRootDir + path).c_str(), path.c_str()) == -1)
throw SysError(format("moving build output `%1%' from the chroot to the Nix store") % path);
}
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
if (!pathExists(path)) continue;
struct stat st;
if (lstat(path.c_str(), &st) == -1)
throw SysError(format("getting attributes of path `%1%'") % path);
* Also for convenience, change the ownership of the build output even in case of failure.
2006-12-08 00:19:50 +00:00
#ifndef __CYGWIN__
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
/* Check that the output is not group or world writable,
as that means that someone else can have interfered
with the build. Also, the output should be owned by
the build user. */
* New primop builtins.filterSource, which can be used to filter files from a source directory. All files for which a predicate function returns true are copied to the store. Typical example is to leave out the .svn directory: stdenv.mkDerivation { ... src = builtins.filterSource (path: baseNameOf (toString path) != ".svn") ./source-dir; # as opposed to # src = ./source-dir; } This is important because the .svn directory influences the hash in a rather unpredictable and variable way.
2006-12-12 23:05:01 +00:00
if ((!S_ISLNK(st.st_mode) && (st.st_mode & (S_IWGRP | S_IWOTH))) ||
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
(buildUser.enabled() && st.st_uid != buildUser.getUID()))
throw BuildError(format("suspicious ownership or permission on `%1%'; rejecting this build output") % path);
* Also for convenience, change the ownership of the build output even in case of failure.
2006-12-08 00:19:50 +00:00
#endif
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
/* Gain ownership of the build result using the setuid
wrapper if we're not root. If we *are* root, then
canonicalisePathMetaData() will take care of this later
on. */
if (buildUser.enabled() && !amPrivileged())
getOwnership(path);
}
* Also for convenience, change the ownership of the build output even in case of failure.
2006-12-08 00:19:50 +00:00
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
/* Check the exit status. */
if (!statusOk(status)) {
deleteTmpDir(false);
throw BuildError(format("builder for `%1%' %2%")
% drvPath % statusToString(status));
}
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
deleteTmpDir(true);
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Do chroot builds in a private namespace. This means that all the bind-mounts we do are only visible to the builder process and its children. So accidentally doing "rm -rf" on the chroot directory won't wipe out /nix/store and other bind-mounted directories anymore. Also, the bind-mounts in the private namespace disappear automatically when the builder exits.
2008-12-11 17:00:12 +00:00
/* Delete the chroot (if we were using one). */
autoDelChroot.reset(); /* this runs the destructor */
Fix chroots builds Chroots are initialised by hard-linking inputs from the Nix store to the chroot. This doesn't work if the input has its immutable bit set, because it's forbidden to create hard links to immutable files. So temporarily clear the immutable bit when creating and destroying the chroot. Note that making regular files in the Nix store immutable isn't very reliable, since the bit can easily become cleared: for instance, if we run the garbage collector after running ‘nix-store --optimise’. So maybe we should only make directories immutable.
2012-02-18 00:23:52 +00:00
/* Deleting the chroot will have caused the immutable bits on
hard-linked inputs to be cleared. So set them again. */
foreach (PathSet::iterator, i, regularInputPaths)
makeImmutable(*i);
* Do chroot builds in a private namespace. This means that all the bind-mounts we do are only visible to the builder process and its children. So accidentally doing "rm -rf" on the chroot directory won't wipe out /nix/store and other bind-mounted directories anymore. Also, the bind-mounts in the private namespace disappear automatically when the builder exits.
2008-12-11 17:00:12 +00:00
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
/* Compute the FS closure of the outputs and register them as
being valid. */
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
computeClosure();
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
* Make sure that lock files are cleaned up properly when building through the build hook.
2011-12-25 16:38:37 +00:00
/* It is now safe to delete the lock files, since all future
lockers will see that the output paths are valid; they will
not create new lock files with the same names as the old
(unlinked) lock files. */
outputLocks.setDeletion(true);
outputLocks.unlock();
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
} catch (BuildError & e) {
printMsg(lvlError, e.msg());
* Release output locks as soon as possible, not when the destructor of the DerivationGoal runs. Otherwise, if a goal is a top-level goal, then the lock won't be released until nix-store finishes. With --keep-going and lots of top-level goals, it's possible to run out of file descriptors (this happened sometimes in the build farm for Nixpkgs). Also, for failed derivation, it won't be possible to build it again until the lock is released. * Idem for locks on build users: these weren't released in a timely manner for failed top-level derivation goals. So if there were more than (say) 10 such failed builds, you would get an error about having run out of build users.
2009-02-16 09:24:20 +00:00
outputLocks.unlock();
buildUser.release();
* Don't cache transient build hook problems.
2009-04-15 06:25:02 +00:00
/* When using a build hook, the hook will return a remote
build failure using exit code 100. Anything else is a hook
problem. */
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
bool hookError = hook &&
* Don't cache transient build hook problems.
2009-04-15 06:25:02 +00:00
(!WIFEXITED(status) || WEXITSTATUS(status) != 100);
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
if (printBuildTrace) {
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
if (hook && hookError)
* When using a build hook, distinguish between transient failures (e.g. an SSH connection problem) and permanent failures (i.e. the builder failed). This matters to Hydra (it wants to know whether it makes sense to retry a build).
2009-01-13 11:39:09 +00:00
printMsg(lvlError, format("@ hook-failed %1% %2% %3% %4%")
% drvPath % drv.outputs["out"].path % status % e.msg());
else
printMsg(lvlError, format("@ build-failed %1% %2% %3% %4%")
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
% drvPath % drv.outputs["out"].path % 1 % e.msg());
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
}
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
/* Register the outputs of this build as "failed" so we won't
try to build them again (negative caching). However, don't
do this for fixed-output derivations, since they're likely
to fail for transient reasons (e.g., fetchurl not being
* Don't cache transient build hook problems.
2009-04-15 06:25:02 +00:00
able to access the network). Hook errors (like
communication problems with the remote machine) shouldn't
be cached either. */
if (worker.cacheFailure && !hookError && !fixedOutput)
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
foreach (DerivationOutputs::iterator, i, drv.outputs)
worker.store.registerFailedPath(i->second.path);
* nix-instantiate: return exit status 100 to denote a permanent build failure. The build hook can use this to distinguish between transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
worker.permanentFailure = !hookError && !fixedOutput;
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
amDone(ecFailed);
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
return;
}
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
/* Release the build user, if applicable. */
buildUser.release();
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
if (printBuildTrace) {
printMsg(lvlError, format("@ build-succeeded %1% %2%")
% drvPath % drv.outputs["out"].path);
}
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
amDone(ecSuccess);
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
}
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
HookReply DerivationGoal::tryBuildHook()
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
{
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
if (!useBuildHook || getEnv("NIX_BUILD_HOOK") == "") return rpDecline;
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
if (!worker.hook)
worker.hook = boost::shared_ptr<HookInstance>(new HookInstance);
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Experimental feature: allow a derivation to tell the build hook that it requires a certain feature on the build machine, e.g. requiredSystemFeatures = [ "kvm" ]; We need this in Hydra to make sure that builds that require KVM support are forwarded to machines that have KVM support. Probably this should also be enforced for local builds.
2010-08-27 13:18:13 +00:00
/* Tell the hook about system features (beyond the system type)
required from the build machine. (The hook could parse the
drv file itself, but this is easier.) */
Strings features = tokenizeString(drv.env["requiredSystemFeatures"]);
foreach (Strings::iterator, i, features) checkStoreName(*i); /* !!! abuse */
/* Send the request to the hook. */
writeLine(worker.hook->toHook.writeSide, (format("%1% %2% %3% %4%")
% (worker.getNrLocalBuilds() < maxBuildJobs ? "1" : "0")
% drv.platform % drvPath % concatStringsSep(",", features)).str());
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
/* Read the first line of input, which should be a word indicating
* Simplify communication with the hook a bit (don't use file descriptors 3/4, just use stdin/stderr).
2009-03-28 19:29:55 +00:00
whether the hook wishes to perform the build. */
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
string reply;
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
while (true) {
string s = readLine(worker.hook->fromHook.readSide);
if (string(s, 0, 2) == "# ") {
reply = string(s, 2);
break;
* Simplify communication with the hook a bit (don't use file descriptors 3/4, just use stdin/stderr).
2009-03-28 19:29:55 +00:00
}
* Always print hook output on stderr, even if --no-build-output is set. * In the build hook, print a trace message to allow Hydra to pick up the name of the remote machine used for the build.
2010-08-31 12:36:24 +00:00
s += "\n";
Use data() instead of c_str() where appropriate
2012-02-09 17:27:45 +00:00
writeToStderr((unsigned char *) s.data(), s.size());
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
}
debug(format("hook reply is `%1%'") % reply);
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
if (reply == "decline" || reply == "postpone")
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
return reply == "decline" ? rpDecline : rpPostpone;
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
else if (reply != "accept")
throw Error(format("bad hook reply `%1%'") % reply);
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
printMsg(lvlTalkative, format("using hook to build path(s) %1%")
% showPaths(outputPaths(drv.outputs)));
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
hook = worker.hook;
worker.hook.reset();
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
/* Tell the hook all the inputs that have to be copied to the
remote system. This unfortunately has to contain the entire
derivation closure to ensure that the validity invariant holds
on the remote system. (I.e., it's unfortunate that we have to
list it since the remote system *probably* already has it.) */
PathSet allInputs;
allInputs.insert(inputPaths.begin(), inputPaths.end());
* Eliminate all uses of the global variable ‘store’ from libstore. This should also fix: nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed. which was caused by hashDerivationModulo() calling the ‘store’ object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
computeFSClosure(worker.store, drvPath, allInputs);
* Export the references graph to the build hook.
2005-03-23 13:16:36 +00:00
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
string s;
foreach (PathSet::iterator, i, allInputs) s += *i + " ";
writeLine(hook->toHook.writeSide, s);
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
/* Tell the hooks the outputs that have to be copied back from the
remote system. */
s = "";
foreach (DerivationOutputs::iterator, i, drv.outputs)
s += i->second.path + " ";
writeLine(hook->toHook.writeSide, s);
hook->toHook.writeSide.close();
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
/* Create the log file and pipe. */
Path logFile = openLogFile();
* When using the build hook, distinguish between the stderr of the hook script proper, and the stdout/stderr of the builder. Only the latter should be saved in /nix/var/log/nix/drvs. * Allow the verbosity to be set through an option. * Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
set<int> fds;
fds.insert(hook->fromHook.readSide);
fds.insert(hook->builderOut.readSide);
worker.childStarted(shared_from_this(), hook->pid, fds, false, false);
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
if (printBuildTrace)
printMsg(lvlError, format("@ build-started %1% %2% %3% %4%")
% drvPath % drv.outputs["out"].path % drv.platform % logFile);
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
return rpAccept;
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
}
* Don't provide the whole Nix store in the chroot, but only the closure of the inputs. This really enforces that there can't be any undeclared dependencies on paths in the store. This is done by creating a fake Nix store and creating bind-mounts or hard-links in the fake store for all paths in the closure. After the build, the build output is moved from the fake store to the real store. TODO: the chroot has to be on the same filesystem as the Nix store for this to work, but this isn't enforced yet. (I.e. it only works currently if /tmp is on the same FS as /nix/store.)
2008-12-11 18:57:10 +00:00
void chmod(const Path & path, mode_t mode)
{
if (::chmod(path.c_str(), 01777) == -1)
throw SysError(format("setting permissions on `%1%'") % path);
}
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
int childEntry(void * arg)
{
((DerivationGoal *) arg)->initChild();
return 1;
}
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
void DerivationGoal::startBuilder()
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
{
* Started making Nix's diagnostic messages a bit more useful.
2004-06-22 14:48:59 +00:00
startNest(nest, lvlInfo,
* `nix-store --build' now builds its arguments in parallel instead of sequentially (within the limits set by `--jobs'). This should greatly improve the utilisation of the build farm when doing Nixpkgs builds.
2005-01-19 15:02:02 +00:00
format("building path(s) %1%") % showPaths(outputPaths(drv.outputs)))
* Started making Nix's diagnostic messages a bit more useful.
2004-06-22 14:48:59 +00:00
* A switch `-j NUMBER' to set the maximum number of parallel jobs (0 = no limit). * Add missing file to distribution.
2004-05-12 14:20:32 +00:00
/* Right platform? */
* Allow derivations to hint that they should not be built remotely using the build hook mechanism, by setting the derivation attribute "preferLocalBuild" to true. This has a few use cases: - The user environment builder. Since it just creates a bunch of symlinks without much computation, there is no reason to do it remotely. In fact, doing it remotely requires the entire closure of the user environment to be copied to the remote machine, which is extremely wasteful. - `fetchurl'. Performing the download on a remote machine and then copying it to the local machine involves twice as much network traffic as performing the download locally, and doesn't save any CPU cycles on the local machine.
2010-08-04 12:13:58 +00:00
if (!canBuildLocally(drv.platform))
* Acquire the locks on the output paths before trying to run the build hook. This fixes a problem with log files being partially or completely filled with 0's because another nix-store process truncates the log file. It should also be more efficient.
2009-03-18 14:48:42 +00:00
throw Error(
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
format("a `%1%' is required to build `%3%', but I am a `%2%'")
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
% drv.platform % thisSystem % drvPath);
* A switch `-j NUMBER' to set the maximum number of parallel jobs (0 = no limit). * Add missing file to distribution.
2004-05-12 14:20:32 +00:00
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
/* Construct the environment passed to the builder. */
* Most shells initialise PATH to some default (/bin:/usr/bin:...) when PATH is not set. We don't want this, so fill it in with some dummy value.
2003-08-18 14:54:54 +00:00
/* Most shells initialise PATH to some default (/bin:/usr/bin:...) when
PATH is not set. We don't want this, so we fill it in with some dummy
value. */
env["PATH"] = "/path-not-set";
* Remove write permission from output paths after they have been built. * Point $HOME to a non-existing path when building to prevent certain tools (such as wget) from falling back on /etc/passwd to locate the home directory (which we don't want them to look at since it's not declared as an input).
2003-08-22 20:12:44 +00:00
/* Set HOME to a non-existing path to prevent certain programs from using
/etc/passwd (or NIS, or whatever) to locate the home directory (for
example, wget looks for ~/.wgetrc). I.e., these tools use /etc/passwd
if HOME is not set, but they will just assume that the settings file
they are looking for does not exist if HOME is set but points to some
non-existing path. */
env["HOME"] = "/homeless-shelter";
* Set the NIX_STORE and NIX_BUILD_TOP environment variables in builders to point to the store and the temporary build directory, respectively. Useful for purity checking. * Also set TEMPDIR, TMPDIR, TEMP, and TEMP to NIX_BUILD_TOP to make sure that tools in the builder store temporary files in the right location.
2004-03-12 10:45:08 +00:00
/* Tell the builder where the Nix store is. Usually they
shouldn't care, but this is useful for purity checking (e.g.,
the compiler or linker might only want to accept paths to files
in the store or in the build directory). */
env["NIX_STORE"] = nixStore;
Added support for passing an (impure) NIX_BUILD_CORES variable to build expressions. This patch adds the configuration file variable "build-cores" and the command line argument "--cores". These settings specify the number of CPU cores to utilize for parallel building within a job, i.e. by passing an appropriate "-j" flag to GNU Make. The default value is 1, which means that parallel building is *disabled*. If the number of build cores is specified as 0 (synonymously: "guess" or "auto"), then the actual value is supposed to be auto-detected by builders at run-time, i.e by calling the nproc(1) utility from coreutils. The environment variable $NIX_BUILD_CORES is available to builders, but the contents of that variable does *not* influence the hash that goes into the $out store path, i.e. the number of build cores to be utilized can be changed at will without requiring any re-builds.
2010-06-23 14:34:08 +00:00
/* The maximum number of cores to utilize for parallel building. */
env["NIX_BUILD_CORES"] = (format("%d") % buildCores).str();
* Terminology fixes.
2005-01-20 16:01:07 +00:00
/* Add all bindings specified in the derivation. */
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (StringPairs::iterator, i, drv.env)
* Refactorings.
2003-07-20 19:29:38 +00:00
env[i->first] = i->second;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
/* Create a temporary directory where the build will take
place. */
* Use /tmp/nix-build-<drvpath>-<counter> instead of /tmp/nix-<pid>-<counter> for temporary build directories. This increases purity a bit: many packages store the temporary build path in their output, causing (generally unimportant) binary differences.
2008-03-27 13:45:17 +00:00
tmpDir = createTempDir("", "nix-build-" + baseNameOf(drvPath), false, false);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
/* For convenience, set an environment pointing to the top build
directory. */
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
env["NIX_BUILD_TOP"] = tmpDir;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
/* Also set TMPDIR and variants to point to this directory. */
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
env["TMPDIR"] = env["TEMPDIR"] = env["TMP"] = env["TEMP"] = tmpDir;
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
* Explicitly set PWD to prevent problems with chroot builds. In particular, dietlibc cannot figure out the cwd because the inode of the current directory doesn't appear in .. (because getdents returns the inode of the mount point).
2008-08-27 17:20:25 +00:00
/* Explicitly set PWD to prevent problems with chroot builds. In
particular, dietlibc cannot figure out the cwd because the
inode of the current directory doesn't appear in .. (because
getdents returns the inode of the mount point). */
env["PWD"] = tmpDir;
* Compatibility hack so that Nixpkgs can continue to do hash checking in `fetchurl' in Nix <= 0.7, but doesn't in Nix 0.8.
2005-02-22 15:23:24 +00:00
/* Compatibility hack with Nix <= 0.7: if this is a fixed-output
derivation, tell the builder, so that for instance `fetchurl'
can skip checking the output. On older Nixes, this environment
variable won't be set, so `fetchurl' will do the check. */
* The `fixedOutput' variable didn't get initialised when using a build hook, causing negative caching to fail randomly.
2009-03-28 16:12:27 +00:00
if (fixedOutput) env["NIX_OUTPUT_CHECKED"] = "1";
* For fixed-output derivations, pass the environment variables listed in the attribute variable `impureEnvVars' from the caller to the builder.
2006-05-31 09:51:45 +00:00
/* *Only* if this is a fixed-output derivation, propagate the
values of the environment variables specified in the
`impureEnvVars' attribute to the builder. This allows for
instance environment variables for proxy configuration such as
`http_proxy' to be easily passed to downloaders like
`fetchurl'. Passing such environment variables from the caller
to the builder is generally impure, but the output of
fixed-output derivations is by definition pure (since we
already know the cryptographic hash of the output). */
if (fixedOutput) {
Strings varNames = tokenizeString(drv.env["impureEnvVars"]);
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (Strings::iterator, i, varNames) env[*i] = getEnv(*i);
* For fixed-output derivations, pass the environment variables listed in the attribute variable `impureEnvVars' from the caller to the builder.
2006-05-31 09:51:45 +00:00
}
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
* Magic attribute `exportReferencesGraph' that allows the references graph to be passed to a builder. This attribute should be a list of pairs [name1 path1 name2 path2 ...]. The references graph of each `pathN' will be stored in a text file `nameN' in the temporary build directory. The text files have the format used by `nix-store --register-validity'. However, the deriver fields are left empty. `exportReferencesGraph' is useful for builders that want to do something with the closure of a store path. Examples: the builders that make initrds and ISO images for NixOS. `exportReferencesGraph' is entirely pure. It's necessary because otherwise the only way for a builder to get this information would be to call `nix-store' directly, which is not allowed (though unfortunately possible).
2006-11-13 18:18:13 +00:00
/* The `exportReferencesGraph' feature allows the references graph
to be passed to a builder. This attribute should be a list of
pairs [name1 path1 name2 path2 ...]. The references graph of
each `pathN' will be stored in a text file `nameN' in the
temporary build directory. The text files have the format used
by `nix-store --register-validity'. However, the deriver
fields are left empty. */
string s = drv.env["exportReferencesGraph"];
Strings ss = tokenizeString(s);
if (ss.size() % 2 != 0)
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
throw BuildError(format("odd number of tokens in `exportReferencesGraph': `%1%'") % s);
* Magic attribute `exportReferencesGraph' that allows the references graph to be passed to a builder. This attribute should be a list of pairs [name1 path1 name2 path2 ...]. The references graph of each `pathN' will be stored in a text file `nameN' in the temporary build directory. The text files have the format used by `nix-store --register-validity'. However, the deriver fields are left empty. `exportReferencesGraph' is useful for builders that want to do something with the closure of a store path. Examples: the builders that make initrds and ISO images for NixOS. `exportReferencesGraph' is entirely pure. It's necessary because otherwise the only way for a builder to get this information would be to call `nix-store' directly, which is not allowed (though unfortunately possible).
2006-11-13 18:18:13 +00:00
for (Strings::iterator i = ss.begin(); i != ss.end(); ) {
string fileName = *i++;
* exportReferencesGraph: work on paths within store paths as well.
2007-01-23 16:57:43 +00:00
checkStoreName(fileName); /* !!! abuse of this function */
/* Check that the store path is valid. */
* Magic attribute `exportReferencesGraph' that allows the references graph to be passed to a builder. This attribute should be a list of pairs [name1 path1 name2 path2 ...]. The references graph of each `pathN' will be stored in a text file `nameN' in the temporary build directory. The text files have the format used by `nix-store --register-validity'. However, the deriver fields are left empty. `exportReferencesGraph' is useful for builders that want to do something with the closure of a store path. Examples: the builders that make initrds and ISO images for NixOS. `exportReferencesGraph' is entirely pure. It's necessary because otherwise the only way for a builder to get this information would be to call `nix-store' directly, which is not allowed (though unfortunately possible).
2006-11-13 18:18:13 +00:00
Path storePath = *i++;
* exportReferencesGraph: work on paths within store paths as well.
2007-01-23 16:57:43 +00:00
if (!isInStore(storePath))
throw BuildError(format("`exportReferencesGraph' contains a non-store path `%1%'")
% storePath);
storePath = toStorePath(storePath);
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
if (!worker.store.isValidPath(storePath))
* exportReferencesGraph: work on paths within store paths as well.
2007-01-23 16:57:43 +00:00
throw BuildError(format("`exportReferencesGraph' contains an invalid path `%1%'")
* Magic attribute `exportReferencesGraph' that allows the references graph to be passed to a builder. This attribute should be a list of pairs [name1 path1 name2 path2 ...]. The references graph of each `pathN' will be stored in a text file `nameN' in the temporary build directory. The text files have the format used by `nix-store --register-validity'. However, the deriver fields are left empty. `exportReferencesGraph' is useful for builders that want to do something with the closure of a store path. Examples: the builders that make initrds and ISO images for NixOS. `exportReferencesGraph' is entirely pure. It's necessary because otherwise the only way for a builder to get this information would be to call `nix-store' directly, which is not allowed (though unfortunately possible).
2006-11-13 18:18:13 +00:00
% storePath);
* exportReferencesGraph: work on paths within store paths as well.
2007-01-23 16:57:43 +00:00
* Unify exportReferencesGraph and exportBuildReferencesGraph, and make sure that it works as expected when you pass it a derivation. That is, we have to make sure that all build-time dependencies are built, and that they are all in the input closure (otherwise remote builds might fail, for example). This is ensured at instantiation time by adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 17:36:42 +00:00
/* If there are derivations in the graph, then include their
outputs as well. This is useful if you want to do things
like passing all build-time dependencies of some path to a
derivation that builds a NixOS DVD image. */
PathSet paths, paths2;
* Eliminate all uses of the global variable ‘store’ from libstore. This should also fix: nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed. which was caused by hashDerivationModulo() calling the ‘store’ object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
computeFSClosure(worker.store, storePath, paths);
* Unify exportReferencesGraph and exportBuildReferencesGraph, and make sure that it works as expected when you pass it a derivation. That is, we have to make sure that all build-time dependencies are built, and that they are all in the input closure (otherwise remote builds might fail, for example). This is ensured at instantiation time by adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 17:36:42 +00:00
paths2 = paths;
foreach (PathSet::iterator, j, paths2) {
if (isDerivation(*j)) {
* Eliminate all uses of the global variable ‘store’ from libstore. This should also fix: nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed. which was caused by hashDerivationModulo() calling the ‘store’ object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
Derivation drv = derivationFromPath(worker.store, *j);
* Unify exportReferencesGraph and exportBuildReferencesGraph, and make sure that it works as expected when you pass it a derivation. That is, we have to make sure that all build-time dependencies are built, and that they are all in the input closure (otherwise remote builds might fail, for example). This is ensured at instantiation time by adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 17:36:42 +00:00
foreach (DerivationOutputs::iterator, k, drv.outputs)
* Eliminate all uses of the global variable ‘store’ from libstore. This should also fix: nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed. which was caused by hashDerivationModulo() calling the ‘store’ object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
computeFSClosure(worker.store, k->second.path, paths);
* Unify exportReferencesGraph and exportBuildReferencesGraph, and make sure that it works as expected when you pass it a derivation. That is, we have to make sure that all build-time dependencies are built, and that they are all in the input closure (otherwise remote builds might fail, for example). This is ensured at instantiation time by adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 17:36:42 +00:00
}
}
Added a way to include derivation with its outputs and dependencies - exportBuildReferenceGraph
2007-12-30 09:30:56 +00:00
/* Write closure info to `fileName'. */
* Added an option "fsync-metadata" to fsync() changes to /nix/var/nix/db. * Removed the function writeStringToFile since it does (almost) the same thing as writeFile.
2010-01-29 12:22:58 +00:00
writeFile(tmpDir + "/" + fileName,
* Store the size of a store path in the database (to be precise, the size of the NAR serialisation of the path, i.e., `nix-store --dump PATH'). This is useful for Hydra.
2010-11-16 17:11:46 +00:00
worker.store.makeValidityRegistration(paths, false, false));
Added a way to include derivation with its outputs and dependencies - exportBuildReferenceGraph
2007-12-30 09:30:56 +00:00
}
* Unify exportReferencesGraph and exportBuildReferencesGraph, and make sure that it works as expected when you pass it a derivation. That is, we have to make sure that all build-time dependencies are built, and that they are all in the input closure (otherwise remote builds might fail, for example). This is ensured at instantiation time by adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 17:36:42 +00:00
* For fixed-output derivations, pass the environment variables listed in the attribute variable `impureEnvVars' from the caller to the builder.
2006-05-31 09:51:45 +00:00
* Get rid of `build-users'. We'll just take all the members of `build-users-group'. This makes configuration easier: you can just add users in /etc/group.
2006-12-06 20:00:15 +00:00
/* If `build-users-group' is not empty, then we have to build as
one of the members of that group. */
* In the garbage collector, if deleting a path fails, try to fix its ownership, then try again.
2006-12-07 15:54:52 +00:00
if (haveBuildUsers()) {
* Prevent uids from being used for more than one build simultaneously. We do this using exclusive locks on uid files in /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
buildUser.acquire();
assert(buildUser.getUID() != 0);
* Get rid of `build-users'. We'll just take all the members of `build-users-group'. This makes configuration easier: you can just add users in /etc/group.
2006-12-06 20:00:15 +00:00
assert(buildUser.getGID() != 0);
* Before starting a build under some uid, kill all current processes running under that uid.
2005-10-17 17:35:37 +00:00
/* Make sure that no other processes are executing under this
uid. */
* If not running as root, let the setuid helper kill the build user's processes before and after the build.
2006-12-07 16:33:31 +00:00
buildUser.kill();
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
* Move killUser() to libutil so that the setuid helper can use it.
2006-12-07 00:16:07 +00:00
/* Change ownership of the temporary build directory, if we're
root. If we're not root, then the setuid helper will do it
just before it starts the builder. */
if (amPrivileged()) {
if (chown(tmpDir.c_str(), buildUser.getUID(), buildUser.getGID()) == -1)
throw SysError(format("cannot change ownership of `%1%'") % tmpDir);
}
* Don't use FIFOs to make Nix create the output path on behalf of the builder. Instead, require that the Nix store has sticky permission (S_ISVTX); everyone can created files in the Nix store, but they cannot delete, rename or modify files created by others.
2005-10-17 16:52:29 +00:00
/* Check that the Nix store has the appropriate permissions,
* Get rid of `build-users'. We'll just take all the members of `build-users-group'. This makes configuration easier: you can just add users in /etc/group.
2006-12-06 20:00:15 +00:00
i.e., owned by root and mode 1775 (sticky bit on so that
* Don't use FIFOs to make Nix create the output path on behalf of the builder. Instead, require that the Nix store has sticky permission (S_ISVTX); everyone can created files in the Nix store, but they cannot delete, rename or modify files created by others.
2005-10-17 16:52:29 +00:00
the builder can create its output but not mess with the
outputs of other processes). */
struct stat st;
if (stat(nixStore.c_str(), &st) == -1)
throw SysError(format("cannot stat `%1%'") % nixStore);
if (!(st.st_mode & S_ISVTX) ||
* Removed `build-allow-root'. * Added `build-users-group', the group under which builds are to be performed. * Check that /nix/store has 1775 permission and is owner by the build-users-group.
2006-12-03 15:32:38 +00:00
((st.st_mode & S_IRWXG) != S_IRWXG) ||
* Get rid of `build-users'. We'll just take all the members of `build-users-group'. This makes configuration easier: you can just add users in /etc/group.
2006-12-06 20:00:15 +00:00
(st.st_gid != buildUser.getGID()))
* Don't use FIFOs to make Nix create the output path on behalf of the builder. Instead, require that the Nix store has sticky permission (S_ISVTX); everyone can created files in the Nix store, but they cannot delete, rename or modify files created by others.
2005-10-17 16:52:29 +00:00
throw Error(format(
* Move killUser() to libutil so that the setuid helper can use it.
2006-12-07 00:16:07 +00:00
"builder does not have write permission to `%2%'; "
* Removed `build-allow-root'. * Added `build-users-group', the group under which builds are to be performed. * Check that /nix/store has 1775 permission and is owner by the build-users-group.
2006-12-03 15:32:38 +00:00
"try `chgrp %1% %2%; chmod 1775 %2%'")
* Get rid of `build-users'. We'll just take all the members of `build-users-group'. This makes configuration easier: you can just add users in /etc/group.
2006-12-06 20:00:15 +00:00
% buildUser.getGID() % nixStore);
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
}
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 00:46:59 +00:00
* Disable chroot builds for fixed-output derivations so that we don't need /etc in the chroot (in particular, /etc/resolv.conf for fetchurl). Not having /etc/resolv.conf in the chroot is a good thing, since we don't want normal derivations to download files.
2008-08-25 15:49:22 +00:00
/* Are we doing a chroot build? Note that fixed-output
derivations are never done in a chroot, mainly so that
functions like fetchurl (which needs a proper /etc/resolv.conf)
work properly. Purity checking for fixed-output derivations
is somewhat pointless anyway. */
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 00:46:59 +00:00
useChroot = queryBoolSetting("build-use-chroot", false);
* Detect whether chroot / bind-mount support is available.
2007-10-27 16:51:55 +00:00
* Disable chroot builds for fixed-output derivations so that we don't need /etc in the chroot (in particular, /etc/resolv.conf for fetchurl). Not having /etc/resolv.conf in the chroot is a good thing, since we don't want normal derivations to download files.
2008-08-25 15:49:22 +00:00
if (fixedOutput) useChroot = false;
* Hack to allow derivations to disable chroot builds by setting the attribute "__noChroot = true" (requested by Rob).
2010-06-01 10:01:14 +00:00
/* Hack to allow derivations to disable chroot builds. */
if (drv.env["__noChroot"] == "1") useChroot = false;
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 00:46:59 +00:00
if (useChroot) {
* Detect whether chroot / bind-mount support is available.
2007-10-27 16:51:55 +00:00
#if CHROOT_ENABLED
* Delete the chroot directory automatically. * Removed some debug messages.
2007-10-27 16:06:38 +00:00
/* Create a temporary directory in which we set up the chroot
* Put chroots in the Nix store. This ensures that we can create hard links to the inputs.
2008-12-12 17:14:57 +00:00
environment using bind-mounts. We put it in the Nix store
to ensure that we can create hard-links to non-directory
inputs in the fake Nix store in the chroot (see below). */
chrootRootDir = drvPath + ".chroot";
* Delete the chroot if it already exists (e.g. left over from an interrupted build).
2008-12-15 23:55:11 +00:00
if (pathExists(chrootRootDir)) deletePath(chrootRootDir);
* Delete the chroot directory automatically. * Removed some debug messages.
2007-10-27 16:06:38 +00:00
* Do chroot builds in a private namespace. This means that all the bind-mounts we do are only visible to the builder process and its children. So accidentally doing "rm -rf" on the chroot directory won't wipe out /nix/store and other bind-mounted directories anymore. Also, the bind-mounts in the private namespace disappear automatically when the builder exits.
2008-12-11 17:00:12 +00:00
/* Clean up the chroot directory automatically. */
autoDelChroot = boost::shared_ptr<AutoDelete>(new AutoDelete(chrootRootDir));
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 00:46:59 +00:00
* Create a /tmp with 1777 permission in the chroot. Some builders need a writable /tmp (they don't respect $TMPDIR).
2008-08-27 16:03:03 +00:00
printMsg(lvlChatty, format("setting up chroot environment in `%1%'") % chrootRootDir);
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 00:46:59 +00:00
* Create a /tmp with 1777 permission in the chroot. Some builders need a writable /tmp (they don't respect $TMPDIR).
2008-08-27 16:03:03 +00:00
/* Create a writable /tmp in the chroot. Many builders need
this. (Of course they should really respect $TMPDIR
instead.) */
Path chrootTmpDir = chrootRootDir + "/tmp";
createDirs(chrootTmpDir);
* Don't provide the whole Nix store in the chroot, but only the closure of the inputs. This really enforces that there can't be any undeclared dependencies on paths in the store. This is done by creating a fake Nix store and creating bind-mounts or hard-links in the fake store for all paths in the closure. After the build, the build output is moved from the fake store to the real store. TODO: the chroot has to be on the same filesystem as the Nix store for this to work, but this isn't enforced yet. (I.e. it only works currently if /tmp is on the same FS as /nix/store.)
2008-12-11 18:57:10 +00:00
chmod(chrootTmpDir, 01777);
* Create a /tmp with 1777 permission in the chroot. Some builders need a writable /tmp (they don't respect $TMPDIR).
2008-08-27 16:03:03 +00:00
Clear supplementary groups of `nixbld' in /etc/passwd in chroots. * src/libstore/build.cc (nix::DerivationGoal::startBuilder): Don't display any supplementary groups for `nixbld' in /etc/passwd.
2010-03-11 10:21:23 +00:00
/* Create a /etc/passwd with entries for the build user and the
nobody account. The latter is kind of a hack to support
* Sync with the trunk.
2010-03-11 15:45:05 +00:00
Samba-in-QEMU. */
* Provide a minimal /etc/passwd in the chroot to keep some builders happy.
2008-12-11 17:44:02 +00:00
createDirs(chrootRootDir + "/etc");
* Added an option "fsync-metadata" to fsync() changes to /nix/var/nix/db. * Removed the function writeStringToFile since it does (almost) the same thing as writeFile.
2010-01-29 12:22:58 +00:00
writeFile(chrootRootDir + "/etc/passwd",
* Provide a minimal /etc/passwd in the chroot to keep some builders happy.
2008-12-11 17:44:02 +00:00
(format(
Fix thinko in r20547. * src/libstore/build.cc (nix::DerivationGoal::startBuilder): Fix the GID of the build user in /etc/passwd.
2010-03-11 10:33:04 +00:00
"nixbld:x:%1%:%2%:Nix build user:/:/noshell\n"
* Provide a minimal /etc/passwd in the chroot to keep some builders happy.
2008-12-11 17:44:02 +00:00
"nobody:x:65534:65534:Nobody:/:/noshell\n")
Fix thinko in r20547. * src/libstore/build.cc (nix::DerivationGoal::startBuilder): Fix the GID of the build user in /etc/passwd.
2010-03-11 10:33:04 +00:00
% (buildUser.enabled() ? buildUser.getUID() : getuid())
* Sync with the trunk.
2010-03-11 10:52:52 +00:00
% (buildUser.enabled() ? buildUser.getGID() : getgid())).str());
* Provide a minimal /etc/passwd in the chroot to keep some builders happy.
2008-12-11 17:44:02 +00:00
Show the build user's group in /etc/group in chroots. * src/libstore/build.cc (nix::DerivationGoal::startBuilder): Create /etc/group showing the build user's group.
2010-03-11 14:47:04 +00:00
/* Declare the build user's group so that programs get a consistent
* Sync with the trunk.
2010-03-11 15:45:05 +00:00
view of the system (e.g., "id -gn"). */
writeFile(chrootRootDir + "/etc/group",
(format("nixbld:!:%1%:\n")
% (buildUser.enabled() ? buildUser.getGID() : getgid())).str());
Show the build user's group in /etc/group in chroots. * src/libstore/build.cc (nix::DerivationGoal::startBuilder): Create /etc/group showing the build user's group.
2010-03-11 14:47:04 +00:00
nix: add /etc/hosts with localhost entry to chroot builds.
2011-11-21 15:19:51 +00:00
/* Create /etc/hosts with localhost entry. */
writeFile(chrootRootDir + "/etc/hosts", "127.0.0.1 localhost\n");
* Create a /tmp with 1777 permission in the chroot. Some builders need a writable /tmp (they don't respect $TMPDIR).
2008-08-27 16:03:03 +00:00
/* Bind-mount a user-configurable set of directories from the
* Add /dev/pts to the default nix.conf.
2008-11-11 14:59:20 +00:00
host file system. The `/dev/pts' directory must be mounted
separately so that newly-created pseudo-terminals show
up. */
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 00:46:59 +00:00
Paths defaultDirs;
defaultDirs.push_back("/dev");
libstore: Always mount `/dev/pts' individually. This fixes problems such as Tcl's PTY handling: ERROR: The system has no more ptys. Ask your system administrator to create more.
2008-10-16 21:04:32 +00:00
defaultDirs.push_back("/dev/pts");
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 00:46:59 +00:00
* Use a PathSet for the chroot directories so that we don't accidentally bind-mount a directory twice.
2008-12-12 11:49:42 +00:00
Paths dirsInChroot_ = querySetting("build-chroot-dirs", defaultDirs);
dirsInChroot.insert(dirsInChroot_.begin(), dirsInChroot_.end());
dirsInChroot.insert(tmpDir);
* Don't provide the whole Nix store in the chroot, but only the closure of the inputs. This really enforces that there can't be any undeclared dependencies on paths in the store. This is done by creating a fake Nix store and creating bind-mounts or hard-links in the fake store for all paths in the closure. After the build, the build output is moved from the fake store to the real store. TODO: the chroot has to be on the same filesystem as the Nix store for this to work, but this isn't enforced yet. (I.e. it only works currently if /tmp is on the same FS as /nix/store.)
2008-12-11 18:57:10 +00:00
/* Make the closure of the inputs available in the chroot,
rather than the whole Nix store. This prevents any access
to undeclared dependencies. Directories are bind-mounted,
while other inputs are hard-linked (since only directories
can be bind-mounted). !!! As an extra security
precaution, make the fake Nix store only writable by the
build user. */
createDirs(chrootRootDir + nixStore);
chmod(chrootRootDir + nixStore, 01777);
foreach (PathSet::iterator, i, inputPaths) {
struct stat st;
if (lstat(i->c_str(), &st))
throw SysError(format("getting attributes of path `%1%'") % *i);
if (S_ISDIR(st.st_mode))
* Use a PathSet for the chroot directories so that we don't accidentally bind-mount a directory twice.
2008-12-12 11:49:42 +00:00
dirsInChroot.insert(*i);
* Don't provide the whole Nix store in the chroot, but only the closure of the inputs. This really enforces that there can't be any undeclared dependencies on paths in the store. This is done by creating a fake Nix store and creating bind-mounts or hard-links in the fake store for all paths in the closure. After the build, the build output is moved from the fake store to the real store. TODO: the chroot has to be on the same filesystem as the Nix store for this to work, but this isn't enforced yet. (I.e. it only works currently if /tmp is on the same FS as /nix/store.)
2008-12-11 18:57:10 +00:00
else {
Fix chroots builds Chroots are initialised by hard-linking inputs from the Nix store to the chroot. This doesn't work if the input has its immutable bit set, because it's forbidden to create hard links to immutable files. So temporarily clear the immutable bit when creating and destroying the chroot. Note that making regular files in the Nix store immutable isn't very reliable, since the bit can easily become cleared: for instance, if we run the garbage collector after running ‘nix-store --optimise’. So maybe we should only make directories immutable.
2012-02-18 00:23:52 +00:00
/* Creating a hard link to *i is impossible if its
immutable bit is set. So clear it first. */
makeMutable(*i);
* Don't provide the whole Nix store in the chroot, but only the closure of the inputs. This really enforces that there can't be any undeclared dependencies on paths in the store. This is done by creating a fake Nix store and creating bind-mounts or hard-links in the fake store for all paths in the closure. After the build, the build output is moved from the fake store to the real store. TODO: the chroot has to be on the same filesystem as the Nix store for this to work, but this isn't enforced yet. (I.e. it only works currently if /tmp is on the same FS as /nix/store.)
2008-12-11 18:57:10 +00:00
Path p = chrootRootDir + *i;
* When building in a chroot, make a copy of a file if hard-linking fails. This is likely to happen after a `nix-store --optimise', because some files may have 32000 links (NIX-111).
2009-10-22 08:28:33 +00:00
if (link(i->c_str(), p.c_str()) == -1) {
/* Hard-linking fails if we exceed the maximum
link count on a file (e.g. 32000 of ext3),
which is quite possible after a `nix-store
Handle EPERM when creating a hard link for the chroot There is a race condition when doing parallel builds with chroots and the immutable bit enabled. One process may call makeImmutable() before the other has called link(), in which case link() will fail with EPERM. We could retry or wrap the operation in a lock, but since this condition is rare and I'm lazy, we just use the existing copy fallback. Fixes #9.
2012-04-30 14:58:04 +00:00
--optimise'. It can also fail if another
process called makeImmutable() on *i after we
did makeMutable(). In those cases, make a copy
instead. */
if (errno != EMLINK && errno != EPERM)
* When building in a chroot, make a copy of a file if hard-linking fails. This is likely to happen after a `nix-store --optimise', because some files may have 32000 links (NIX-111).
2009-10-22 08:28:33 +00:00
throw SysError(format("linking `%1%' to `%2%'") % p % *i);
StringSink sink;
dumpPath(*i, sink);
StringSource source(sink.s);
restorePath(p, source);
}
Fix chroots builds Chroots are initialised by hard-linking inputs from the Nix store to the chroot. This doesn't work if the input has its immutable bit set, because it's forbidden to create hard links to immutable files. So temporarily clear the immutable bit when creating and destroying the chroot. Note that making regular files in the Nix store immutable isn't very reliable, since the bit can easily become cleared: for instance, if we run the garbage collector after running ‘nix-store --optimise’. So maybe we should only make directories immutable.
2012-02-18 00:23:52 +00:00
makeImmutable(*i);
regularInputPaths.insert(*i);
* Don't provide the whole Nix store in the chroot, but only the closure of the inputs. This really enforces that there can't be any undeclared dependencies on paths in the store. This is done by creating a fake Nix store and creating bind-mounts or hard-links in the fake store for all paths in the closure. After the build, the build output is moved from the fake store to the real store. TODO: the chroot has to be on the same filesystem as the Nix store for this to work, but this isn't enforced yet. (I.e. it only works currently if /tmp is on the same FS as /nix/store.)
2008-12-11 18:57:10 +00:00
}
}
* Detect whether chroot / bind-mount support is available.
2007-10-27 16:51:55 +00:00
#else
throw Error("chroot builds are not supported on this platform");
#endif
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 00:46:59 +00:00
}
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
/* Run the builder. */
printMsg(lvlChatty, format("executing builder `%1%'") %
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
drv.builder);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
/* Create the log file. */
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
Path logFile = openLogFile();
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
* When using the build hook, distinguish between the stderr of the hook script proper, and the stdout/stderr of the builder. Only the latter should be saved in /nix/var/log/nix/drvs. * Allow the verbosity to be set through an option. * Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
/* Create a pipe to get the output of the builder. */
builderOut.create();
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
/* Fork a child to build the package. Note that while we
currently use forks to run and wait for the children, it
shouldn't be hard to use threads for this on systems where
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
fork() is unavailable or inefficient.
If we're building in a chroot, then also set up private
namespaces for the build:
- The PID namespace causes the build to start as PID 1.
Processes outside of the chroot are not visible to those on
the inside, but processes inside the chroot are visible from
the outside (though with different PIDs).
- The private mount namespace ensures that all the bind mounts
we do will only show up in this process and its children, and
will disappear automatically when we're done.
- The private network namespace ensures that the builder cannot
talk to the outside world (or vice versa). It only has a
private loopback interface.
- The IPC namespace prevents the builder from communicating
with outside processes using SysV IPC mechanisms (shared
memory, message queues, semaphores). It also ensures that
all IPC objects are destroyed when the builder exits.
*/
#if CHROOT_ENABLED
if (useChroot) {
char stack[32 * 1024];
pid = clone(childEntry, stack + sizeof(stack) - 8,
CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | CLONE_NEWIPC | CLONE_NEWUTS | SIGCHLD, this);
} else
#endif
{
pid = fork();
if (pid == 0) initChild();
}
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
Check the return code of the clone() call
2012-06-27 13:52:06 +00:00
if (pid == -1) throw SysError("unable to fork");
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
/* parent */
pid.setSeparatePG(true);
builderOut.writeSide.close();
worker.childStarted(shared_from_this(), pid,
singleton<set<int> >(builderOut.readSide), true, true);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
if (printBuildTrace) {
printMsg(lvlError, format("@ build-started %1% %2% %3% %4%")
% drvPath % drv.outputs["out"].path % drv.platform % logFile);
}
}
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
void DerivationGoal::initChild()
{
/* Warning: in the child we should absolutely not make any SQLite
calls! */
try { /* child */
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
* Detect whether chroot / bind-mount support is available.
2007-10-27 16:51:55 +00:00
#if CHROOT_ENABLED
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
if (useChroot) {
/* Initialise the loopback interface. */
AutoCloseFD fd(socket(PF_INET, SOCK_DGRAM, IPPROTO_IP));
if (fd == -1) throw SysError("cannot open IP socket");
In chroot builds, use a private network namespace On Linux it's possible to run a process in its own network namespace, meaning that it gets its own set of network interfaces, disjunct from the rest of the system. We use this to completely remove network access to chroot builds, except that they get a private loopback interface. This means that: - Builders cannot connect to the outside network or to other processes on the same machine, except processes within the same build. - Vice versa, other processes cannot connect to processes in a chroot build, and open ports/connections do not show up in "netstat". - If two concurrent builders try to listen on the same port (e.g. as part of a test), they no longer conflict with each other. This was inspired by the "PrivateNetwork" flag in systemd.
2012-06-23 04:28:35 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
struct ifreq ifr;
strcpy(ifr.ifr_name, "lo");
ifr.ifr_flags = IFF_UP | IFF_LOOPBACK | IFF_RUNNING;
if (ioctl(fd, SIOCSIFFLAGS, &ifr) == -1)
throw SysError("cannot set loopback interface flags");
fd.close();
/* Set the hostname etc. to fixed values. */
char hostname[] = "localhost";
sethostname(hostname, sizeof(hostname));
char domainname[] = "(none)"; // kernel default
setdomainname(domainname, sizeof(domainname));
/* Bind-mount all the directories from the "host"
filesystem that we want in the chroot
environment. */
foreach (PathSet::iterator, i, dirsInChroot) {
Path source = *i;
Path target = chrootRootDir + source;
debug(format("bind mounting `%1%' to `%2%'") % source % target);
* Do chroot builds in a private namespace. This means that all the bind-mounts we do are only visible to the builder process and its children. So accidentally doing "rm -rf" on the chroot directory won't wipe out /nix/store and other bind-mounted directories anymore. Also, the bind-mounts in the private namespace disappear automatically when the builder exits.
2008-12-11 17:00:12 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
createDirs(target);
* Do chroot builds in a private namespace. This means that all the bind-mounts we do are only visible to the builder process and its children. So accidentally doing "rm -rf" on the chroot directory won't wipe out /nix/store and other bind-mounted directories anymore. Also, the bind-mounts in the private namespace disappear automatically when the builder exits.
2008-12-11 17:00:12 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
if (mount(source.c_str(), target.c_str(), "", MS_BIND, 0) == -1)
throw SysError(format("bind mount from `%1%' to `%2%' failed") % source % target);
* Do chroot builds in a private namespace. This means that all the bind-mounts we do are only visible to the builder process and its children. So accidentally doing "rm -rf" on the chroot directory won't wipe out /nix/store and other bind-mounted directories anymore. Also, the bind-mounts in the private namespace disappear automatically when the builder exits.
2008-12-11 17:00:12 +00:00
}
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
/* Bind a new instance of procfs on /proc to reflect our
private PID namespace. */
if (mount("none", (chrootRootDir + "/proc").c_str(), "proc", 0, 0) == -1)
throw SysError("mounting /proc");
Mount an empty /dev/shm tmpfs in the chroot This ensures that whatever the builder writes in /dev/shm is automatically cleaned up.
2012-06-27 13:52:27 +00:00
/* Mount a new tmpfs on /dev/shm to ensure that whatever
the builder puts in /dev/shm is cleaned up automatically. */
if (pathExists("/dev/shm"))
if (mount("none", (chrootRootDir + "/dev/shm").c_str(), "tmpfs", 0, 0) == -1)
throw SysError("mounting /dev/shm");
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
/* Do the chroot(). Below we do a chdir() to the
temporary build directory to make sure the current
directory is in the chroot. (Actually the order
doesn't matter, since due to the bind mount tmpDir and
tmpRootDit/tmpDir are the same directories.) */
if (chroot(chrootRootDir.c_str()) == -1)
throw SysError(format("cannot change root directory to `%1%'") % chrootRootDir);
}
* Detect whether chroot / bind-mount support is available.
2007-10-27 16:51:55 +00:00
#endif
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 00:46:59 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
commonChildInit(builderOut);
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
if (chdir(tmpDir.c_str()) == -1)
throw SysError(format("changing into `%1%'") % tmpDir);
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
/* Close all other file descriptors. */
closeMostFDs(set<int>());
Close almost all file descriptors in the builder This regression was accidentally introduced in 35355fc1fcffbe859395e360c0a6a1463f137d63.
2012-04-14 22:20:32 +00:00
* Support i686-linux builds directly on x86_64-linux Nix installations. This is implemented using the personality() syscall, which causes uname to return "i686" in child processes.
2009-01-12 16:30:32 +00:00
#ifdef CAN_DO_LINUX32_BUILDS
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
/* Change the personality to 32-bit if we're doing an
i686-linux build on an x86_64-linux machine. */
if (drv.platform == "i686-linux" && thisSystem == "x86_64-linux") {
if (personality(0x0008 | 0x8000000 /* == PER_LINUX32_3GB */) == -1)
throw SysError("cannot set i686-linux personality");
}
On Linux, pretend we're building on Linux 2.6 Setting the UNAME26 personality causes "uname" to return "2.6.x", regardless of the kernel version. This improves determinism in a few misbehaved packages.
2012-04-05 11:03:19 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
/* Impersonate a Linux 2.6 machine to get some determinism in
builds that depend on the kernel version. */
if ((drv.platform == "i686-linux" || drv.platform == "x86_64-linux") &&
queryBoolSetting("build-impersonate-linux-26", true))
{
int cur = personality(0xffffffff);
if (cur != -1) personality(cur | 0x0020000 /* == UNAME26 */);
}
* Support i686-linux builds directly on x86_64-linux Nix installations. This is implemented using the personality() syscall, which causes uname to return "i686" in child processes.
2009-01-12 16:30:32 +00:00
#endif
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
/* Fill in the environment. */
Strings envStrs;
foreach (Environment::const_iterator, i, env)
envStrs.push_back(i->first + "=" + i->second);
const char * * envArr = strings2CharPtrs(envStrs);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
Path program = drv.builder.c_str();
std::vector<const char *> args; /* careful with c_str()! */
string user; /* must be here for its c_str()! */
* If Nix is not running as root, call the setuid helper to start the builder under the desired build user.
2006-12-07 00:42:30 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
/* If we are running in `build-users' mode, then switch to the
user we allocated above. Make sure that we drop all root
privileges. Note that above we have closed all file
descriptors except std*, so that's safe. Also note that
setuid() when run as root sets the real, effective and
saved UIDs. */
if (buildUser.enabled()) {
printMsg(lvlChatty, format("switching to user `%1%'") % buildUser.getUser());
if (amPrivileged()) {
* If Nix is not running as root, call the setuid helper to start the builder under the desired build user.
2006-12-07 00:42:30 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
if (setgroups(0, 0) == -1)
throw SysError("cannot clear the set of supplementary groups");
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
if (setgid(buildUser.getGID()) == -1 ||
getgid() != buildUser.getGID() ||
getegid() != buildUser.getGID())
throw SysError("setgid failed");
if (setuid(buildUser.getUID()) == -1 ||
getuid() != buildUser.getUID() ||
geteuid() != buildUser.getUID())
throw SysError("setuid failed");
* If Nix is not running as root, call the setuid helper to start the builder under the desired build user.
2006-12-07 00:42:30 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
} else {
/* Let the setuid helper take care of it. */
program = nixLibexecDir + "/nix-setuid-helper";
args.push_back(program.c_str());
args.push_back("run-builder");
user = buildUser.getUser().c_str();
args.push_back(user.c_str());
args.push_back(drv.builder.c_str());
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
}
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
}
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
/* Fill in the arguments. */
string builderBasename = baseNameOf(drv.builder);
args.push_back(builderBasename.c_str());
foreach (Strings::iterator, i, drv.args)
args.push_back(i->c_str());
args.push_back(0);
* If Nix is not running as root, call the setuid helper to start the builder under the desired build user.
2006-12-07 00:42:30 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
restoreSIGPIPE();
* Restore SIGPIPE to SIG_DFL when running the builder. This prevents subtle and often hard-to-reproduce bugs where programs in pipes either barf with a "Broken pipe" message or not, depending on the exact timing conditions. This particularly happened in GNU M4 (and Bison, which uses M4).
2008-11-14 15:46:45 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
/* Execute the program. This should not return. */
execve(program.c_str(), (char * *) &args[0], (char * *) envArr);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
throw SysError(format("executing `%1%'")
% drv.builder);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
} catch (std::exception & e) {
std::cerr << format("build error: %1%") % e.what() << std::endl;
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
}
When using chroots, use a private PID namespace In a private PID namespace, processes have PIDs that are separate from the rest of the system. The initial child gets PID 1. Processes in the chroot cannot see processes outside of the chroot. This improves isolation between builds. However, processes on the outside can see processes in the chroot and send signals to them (if they have appropriate rights). Since the builder gets PID 1, it serves as the reaper for zombies in the chroot. This might turn out to be a problem. In that case we'll need to have a small PID 1 process that sits in a loop calling wait().
2012-06-25 19:45:16 +00:00
quickExit(1);
* Distributed builds and load balancing now seem to work pretty well. (Though the `build-remote.pl' script has a gigantic race condition).
2004-05-13 22:52:37 +00:00
}
* Special derivation attribute `allowedReferences' that causes Nix to check that the references of the output of a derivation are in the specified set. For instance, allowedReferences = []; specifies that the output cannot have any references. (This is useful, for instance, for the generation of bootstrap binaries for stdenv-linux, which must not have any references for purity). It could also be used to guard against undesired runtime dependencies, e.g., {gcc, dynlib}: derivation { ... allowedReferences = [dynlib]; } says that the output can refer to the path of `dynlib' but not `gcc'. A `forbiddedReferences' attribute would be more useful for this, though.
2006-10-19 16:09:24 +00:00
/* Parse a list of reference specifiers. Each element must either be
a store path, or the symbolic name of the output of the derivation
(such as `out'). */
PathSet parseReferenceSpecifiers(const Derivation & drv, string attr)
{
PathSet result;
Paths paths = tokenizeString(attr);
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (Strings::iterator, i, paths) {
* Special derivation attribute `allowedReferences' that causes Nix to check that the references of the output of a derivation are in the specified set. For instance, allowedReferences = []; specifies that the output cannot have any references. (This is useful, for instance, for the generation of bootstrap binaries for stdenv-linux, which must not have any references for purity). It could also be used to guard against undesired runtime dependencies, e.g., {gcc, dynlib}: derivation { ... allowedReferences = [dynlib]; } says that the output can refer to the path of `dynlib' but not `gcc'. A `forbiddedReferences' attribute would be more useful for this, though.
2006-10-19 16:09:24 +00:00
if (isStorePath(*i))
result.insert(*i);
else if (drv.outputs.find(*i) != drv.outputs.end())
result.insert(drv.outputs.find(*i)->second.path);
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
else throw BuildError(
* Special derivation attribute `allowedReferences' that causes Nix to check that the references of the output of a derivation are in the specified set. For instance, allowedReferences = []; specifies that the output cannot have any references. (This is useful, for instance, for the generation of bootstrap binaries for stdenv-linux, which must not have any references for purity). It could also be used to guard against undesired runtime dependencies, e.g., {gcc, dynlib}: derivation { ... allowedReferences = [dynlib]; } says that the output can refer to the path of `dynlib' but not `gcc'. A `forbiddedReferences' attribute would be more useful for this, though.
2006-10-19 16:09:24 +00:00
format("derivation contains an illegal reference specifier `%1%'")
% *i);
}
return result;
}
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
void DerivationGoal::computeClosure()
* Drain the output of the build hook to show error messages. Ugly hack.
2004-05-18 14:52:35 +00:00
{
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
map<Path, PathSet> allReferences;
* Store the size of a store path in the database (to be precise, the size of the NAR serialisation of the path, i.e., `nix-store --dump PATH'). This is useful for Hydra.
2010-11-16 17:11:46 +00:00
map<Path, HashResult> contentHashes;
* Build hooks: use nix-store --import. This prevents a redundant scan for runtime dependencies (i.e. the local machine shouldn't do a scan that the remote machine has already done). Also pipe directly into `nix-store --import': don't use a temporary file.
2009-02-02 17:24:10 +00:00
/* When using a build hook, the build hook can register the output
as valid (by doing `nix-store --import'). If so we don't have
to do anything here. */
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
if (hook) {
* Build hooks: use nix-store --import. This prevents a redundant scan for runtime dependencies (i.e. the local machine shouldn't do a scan that the remote machine has already done). Also pipe directly into `nix-store --import': don't use a temporary file.
2009-02-02 17:24:10 +00:00
bool allValid = true;
foreach (DerivationOutputs::iterator, i, drv.outputs)
if (!worker.store.isValidPath(i->second.path)) allValid = false;
if (allValid) return;
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Check whether the output paths were created, and grep each
output path to determine what other paths it references. Also make all
output paths read-only. */
* Build hooks: use nix-store --import. This prevents a redundant scan for runtime dependencies (i.e. the local machine shouldn't do a scan that the remote machine has already done). Also pipe directly into `nix-store --import': don't use a temporary file.
2009-02-02 17:24:10 +00:00
foreach (DerivationOutputs::iterator, i, drv.outputs) {
* Removed the `id' attribute hack. * Formalise the notion of fixed-output derivations, i.e., derivations for which a cryptographic hash of the output is known in advance. Changes to such derivations should not propagate upwards through the dependency graph. Previously this was done by specifying the hash component of the output path through the `id' attribute, but this is insecure since you can lie about it (i.e., you can specify any hash and then produce a completely different output). Now the responsibility for checking the output is moved from the builder to Nix itself. A fixed-output derivation can be created by specifying the `outputHash' and `outputHashAlgo' attributes, the latter taking values `md5', `sha1', and `sha256', and the former specifying the actual hash in hexadecimal or in base-32 (auto-detected by looking at the length of the attribute value). MD5 is included for compatibility but should be considered deprecated. * Removed the `drvPath' pseudo-attribute in derivation results. It's no longer necessary. * Cleaned up the support for multiple output paths in derivation store expressions. Each output now has a unique identifier (e.g., `out', `devel', `docs'). Previously there was no way to tell output paths apart at the store expression level. * `nix-hash' now has a flag `--base32' to specify that the hash should be printed in base-32 notation. * `fetchurl' accepts parameters `sha256' and `sha1' in addition to `md5'. * `nix-prefetch-url' now prints out a SHA-1 hash in base-32. (TODO: a flag to specify the hash.)
2005-01-17 16:55:19 +00:00
Path path = i->second.path;
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
if (!pathExists(path)) {
throw BuildError(
* Fixed format string error.
2004-07-06 11:21:34 +00:00
format("builder for `%1%' failed to produce output path `%2%'")
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
% drvPath % path);
* Don't throw an exception when a build fails. Just terminate the goal and allow the problem to be handled elsewhere (e.g., at top-level).
2004-06-25 10:21:44 +00:00
}
* Drain the output of the build hook to show error messages. Ugly hack.
2004-05-18 14:52:35 +00:00
* Check that the build result is owned by the build user, and that nobody else has write permission to the build result. This catches most hack attempts.
2005-10-17 16:59:25 +00:00
struct stat st;
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
if (lstat(path.c_str(), &st) == -1)
* Check that the build result is owned by the build user, and that nobody else has write permission to the build result. This catches most hack attempts.
2005-10-17 16:59:25 +00:00
throw SysError(format("getting attributes of path `%1%'") % path);
* Slightly nicer message.
2005-04-11 08:07:41 +00:00
startNest(nest, lvlTalkative,
format("scanning for references inside `%1%'") % path);
* Actually check that the result of fixed-output derivations matches the specified hash.
2005-01-17 19:01:48 +00:00
/* Check that fixed-output derivations produced the right
outputs (i.e., the content hash should match the specified
hash). */
if (i->second.hash != "") {
* Support for fixed-output hashes over directory trees (i.e., over the NAR dump of the path).
2005-02-22 21:14:41 +00:00
* Fix a huuuuge security hole in the Nix daemon. It didn't check that derivations added to the store by clients have "correct" output paths (meaning that the output paths are computed by hashing the derivation according to a certain algorithm). This means that a malicious user could craft a special .drv file to build *any* desired path in the store with any desired contents (so long as the path doesn't already exist). Then the attacker just needs to wait for a victim to come along and install the compromised path. For instance, if Alice (the attacker) knows that the latest Firefox derivation in Nixpkgs produces the path /nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1 then (provided this path doesn't already exist) she can craft a .drv file that creates that path (i.e., has it as one of its outputs), add it to the store using "nix-store --add", and build it with "nix-store -r". So the fake .drv could write a Trojan to the Firefox path. Then, if user Bob (the victim) comes along and does $ nix-env -i firefox $ firefox he executes the Trojan injected by Alice. The fix is to have the Nix daemon verify that derivation outputs are correct (in addValidPath()). This required some refactoring to move the hash computation code to libstore.
2011-07-20 18:10:47 +00:00
bool recursive; HashType ht; Hash h;
i->second.parseHashInfo(recursive, ht, h);
* Support for fixed-output hashes over directory trees (i.e., over the NAR dump of the path).
2005-02-22 21:14:41 +00:00
if (!recursive) {
/* The output path should be a regular file without
execute permission. */
if (!S_ISREG(st.st_mode) || (st.st_mode & S_IXUSR) != 0)
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
throw BuildError(
* Support for fixed-output hashes over directory trees (i.e., over the NAR dump of the path).
2005-02-22 21:14:41 +00:00
format("output path `%1% should be a non-executable regular file")
% path);
}
/* Check the hash. */
* Store the size of a store path in the database (to be precise, the size of the NAR serialisation of the path, i.e., `nix-store --dump PATH'). This is useful for Hydra.
2010-11-16 17:11:46 +00:00
Hash h2 = recursive ? hashPath(ht, path).first : hashFile(ht, path);
* Actually check that the result of fixed-output derivations matches the specified hash.
2005-01-17 19:01:48 +00:00
if (h != h2)
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
throw BuildError(
* Typo.
2005-05-10 14:21:46 +00:00
format("output path `%1%' should have %2% hash `%3%', instead has `%4%'")
* Fix a huuuuge security hole in the Nix daemon. It didn't check that derivations added to the store by clients have "correct" output paths (meaning that the output paths are computed by hashing the derivation according to a certain algorithm). This means that a malicious user could craft a special .drv file to build *any* desired path in the store with any desired contents (so long as the path doesn't already exist). Then the attacker just needs to wait for a victim to come along and install the compromised path. For instance, if Alice (the attacker) knows that the latest Firefox derivation in Nixpkgs produces the path /nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1 then (provided this path doesn't already exist) she can craft a .drv file that creates that path (i.e., has it as one of its outputs), add it to the store using "nix-store --add", and build it with "nix-store -r". So the fake .drv could write a Trojan to the Firefox path. Then, if user Bob (the victim) comes along and does $ nix-env -i firefox $ firefox he executes the Trojan injected by Alice. The fix is to have the Nix daemon verify that derivation outputs are correct (in addValidPath()). This required some refactoring to move the hash computation code to libstore.
2011-07-20 18:10:47 +00:00
% path % i->second.hashAlgo % printHash(h) % printHash(h2));
* Actually check that the result of fixed-output derivations matches the specified hash.
2005-01-17 19:01:48 +00:00
}
* Check that the build result is owned by the build user, and that nobody else has write permission to the build result. This catches most hack attempts.
2005-10-17 16:59:25 +00:00
/* Get rid of all weird permissions. */
* Canonicalise metadata of all files in store paths (set the mtime to 0, set the mode to either 444 or 555, set the group to the default).
2004-11-29 15:09:29 +00:00
canonicalisePathMetaData(path);
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
* Scan for references and compute the SHA-256 hash of the output in one pass. This halves the amount of I/O.
2009-03-28 20:51:33 +00:00
/* For this output path, find the references to other paths
contained in it. Compute the SHA-256 NAR hash at the same
time. The hash is stored in the database so that we can
verify later on whether nobody has messed with the store. */
* Store the size of a store path in the database (to be precise, the size of the NAR serialisation of the path, i.e., `nix-store --dump PATH'). This is useful for Hydra.
2010-11-16 17:11:46 +00:00
HashResult hash;
* Scan for references and compute the SHA-256 hash of the output in one pass. This halves the amount of I/O.
2009-03-28 20:51:33 +00:00
PathSet references = scanForReferences(path, allPaths, hash);
contentHashes[path] = hash;
* Remove the undocumented `noscan' feature. It's no longer necessary now that reference scanning is sufficiently streamy.
2006-11-13 18:19:05 +00:00
/* For debugging, print out the referenced and unreferenced
paths. */
* Scan for references and compute the SHA-256 hash of the output in one pass. This halves the amount of I/O.
2009-03-28 20:51:33 +00:00
foreach (PathSet::iterator, i, inputPaths) {
* Remove the undocumented `noscan' feature. It's no longer necessary now that reference scanning is sufficiently streamy.
2006-11-13 18:19:05 +00:00
PathSet::iterator j = references.find(*i);
if (j == references.end())
debug(format("unreferenced input: `%1%'") % *i);
else
debug(format("referenced input: `%1%'") % *i);
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
}
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
allReferences[path] = references;
* Renamed `normalise.cc' -> `build.cc', `storeexprs.cc' -> `derivations.cc', etc. * Store the SHA-256 content hash of store paths in the database after they have been built/added. This is so that we can check whether the store has been messed with (a la `rpm --verify'). * When registering path validity, verify that the closure property holds.
2005-01-19 16:39:47 +00:00
* Special derivation attribute `allowedReferences' that causes Nix to check that the references of the output of a derivation are in the specified set. For instance, allowedReferences = []; specifies that the output cannot have any references. (This is useful, for instance, for the generation of bootstrap binaries for stdenv-linux, which must not have any references for purity). It could also be used to guard against undesired runtime dependencies, e.g., {gcc, dynlib}: derivation { ... allowedReferences = [dynlib]; } says that the output can refer to the path of `dynlib' but not `gcc'. A `forbiddedReferences' attribute would be more useful for this, though.
2006-10-19 16:09:24 +00:00
/* If the derivation specifies an `allowedReferences'
attribute (containing a list of paths that the output may
refer to), check that all references are in that list. !!!
allowedReferences should really be per-output. */
if (drv.env.find("allowedReferences") != drv.env.end()) {
PathSet allowed = parseReferenceSpecifiers(drv, drv.env["allowedReferences"]);
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (PathSet::iterator, i, references)
* Special derivation attribute `allowedReferences' that causes Nix to check that the references of the output of a derivation are in the specified set. For instance, allowedReferences = []; specifies that the output cannot have any references. (This is useful, for instance, for the generation of bootstrap binaries for stdenv-linux, which must not have any references for purity). It could also be used to guard against undesired runtime dependencies, e.g., {gcc, dynlib}: derivation { ... allowedReferences = [dynlib]; } says that the output can refer to the path of `dynlib' but not `gcc'. A `forbiddedReferences' attribute would be more useful for this, though.
2006-10-19 16:09:24 +00:00
if (allowed.find(*i) == allowed.end())
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
throw BuildError(format("output is not allowed to refer to path `%1%'") % *i);
* Special derivation attribute `allowedReferences' that causes Nix to check that the references of the output of a derivation are in the specified set. For instance, allowedReferences = []; specifies that the output cannot have any references. (This is useful, for instance, for the generation of bootstrap binaries for stdenv-linux, which must not have any references for purity). It could also be used to guard against undesired runtime dependencies, e.g., {gcc, dynlib}: derivation { ... allowedReferences = [dynlib]; } says that the output can refer to the path of `dynlib' but not `gcc'. A `forbiddedReferences' attribute would be more useful for this, though.
2006-10-19 16:09:24 +00:00
}
* Drain the output of the build hook to show error messages. Ugly hack.
2004-05-18 14:52:35 +00:00
}
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
/* Register each output path as valid, and register the sets of
* Reject a build if there is a cycle among the outputs. This is necessary because existing code assumes that the references graph is acyclic.
2011-12-30 14:47:14 +00:00
paths referenced by each of them. If there are cycles in the
outputs, this will fail. */
* Store the size of a store path in the database (to be precise, the size of the NAR serialisation of the path, i.e., `nix-store --dump PATH'). This is useful for Hydra.
2010-11-16 17:11:46 +00:00
ValidPathInfos infos;
foreach (DerivationOutputs::iterator, i, drv.outputs) {
ValidPathInfo info;
info.path = i->second.path;
info.hash = contentHashes[i->second.path].first;
info.narSize = contentHashes[i->second.path].second;
info.references = allReferences[i->second.path];
info.deriver = drvPath;
infos.push_back(info);
}
worker.store.registerValidPaths(infos);
* Distributed builds and load balancing now seem to work pretty well. (Though the `build-remote.pl' script has a gigantic race condition).
2004-05-13 22:52:37 +00:00
}
* `nix-store --read-log / -l PATH' shows the build log of PATH, if available. For instance, $ nix-store -l $(which svn) | less lets you read the build log of the Subversion instance in your profile. * `nix-store -qb': if applied to a non-derivation, take the deriver.
2006-10-28 16:33:54 +00:00
string drvsLogDir = "drvs";
* Create missing log and temproots directories automatically (reported by Rob).
2005-03-24 17:46:38 +00:00
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
Path DerivationGoal::openLogFile()
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
{
Add option ‘build-keep-log’ to enable/disable writing of build logs Fixes #26.
2012-05-29 20:42:05 +00:00
if (!queryBoolSetting("build-keep-log", true)) return "";
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
/* Create a log file. */
* Create missing log and temproots directories automatically (reported by Rob).
2005-03-24 17:46:38 +00:00
Path dir = (format("%1%/%2%") % nixLogDir % drvsLogDir).str();
createDirs(dir);
Compress build logs on the fly using bzip2
2012-05-30 14:12:29 +00:00
Path logFileName = (format("%1%/%2%.bz2") % dir % baseNameOf(drvPath)).str();
AutoCloseFD fd = open(logFileName.c_str(), O_CREAT | O_WRONLY | O_TRUNC, 0666);
if (fd == -1)
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
throw SysError(format("creating log file `%1%'") % logFileName);
Compress build logs on the fly using bzip2
2012-05-30 14:12:29 +00:00
closeOnExec(fd);
if (!(fLogFile = fdopen(fd.borrow(), "w")))
throw SysError(format("opening file `%1%'") % logFileName);
int err;
if (!(bzLogFile = BZ2_bzWriteOpen(&err, fLogFile, 9, 0, 0)))
throw Error(format("cannot open compressed log file `%1%'") % logFileName);
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
return logFileName;
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
}
Compress build logs on the fly using bzip2
2012-05-30 14:12:29 +00:00
void DerivationGoal::closeLogFile()
{
if (bzLogFile) {
int err;
BZ2_bzWriteClose(&err, bzLogFile, 0, 0, 0);
bzLogFile = 0;
if (err != BZ_OK) throw Error(format("cannot close compressed log file (BZip2 error = %1%)") % err);
}
if (fLogFile) {
fclose(fLogFile);
fLogFile = 0;
}
}
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
void DerivationGoal::deleteTmpDir(bool force)
* The build hooks used to implement distributed builds can now be run in parallel. Hooks are more efficient: locks on output paths are only acquired when the hook says that it is willing to accept a build job. Hooks now work in two phases. First, they should first tell Nix whether they are willing to accept a job. Nix guarantuees that no two hooks will ever be in the first phase at the same time (this simplifies the implementation of hooks, since they don't have to perform locking (?)). Second, if they accept a job, they are then responsible for building it (on the remote system), and copying the result back. These can be run in parallel with other hooks and locally executed jobs. The implementation is a bit messy right now, though. * The directory `distributed' shows a (hacky) example of a hook that distributes build jobs over a set of machines listed in a configuration file.
2004-05-13 19:14:49 +00:00
{
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
if (tmpDir != "") {
* When keeping the temporary build directory (-K), change the owner back to the Nix account.
2006-12-07 23:27:40 +00:00
if (keepFailed && !force) {
* Started making Nix's diagnostic messages a bit more useful.
2004-06-22 14:48:59 +00:00
printMsg(lvlError,
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
format("builder for `%1%' failed; keeping build directory `%2%'")
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
% drvPath % tmpDir);
* When keeping the temporary build directory (-K), change the owner back to the Nix account.
2006-12-07 23:27:40 +00:00
if (buildUser.enabled() && !amPrivileged())
getOwnership(tmpDir);
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
else
* Change the ownership of store paths to the Nix account before deleting them using the setuid helper.
2006-12-07 14:14:35 +00:00
deletePathWrapped(tmpDir);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
tmpDir = "";
}
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
}
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
void DerivationGoal::handleChildOutput(int fd, const string & data)
* Write build logs to disk again.
2004-06-29 09:41:50 +00:00
{
* When using the build hook, distinguish between the stderr of the hook script proper, and the stdout/stderr of the builder. Only the latter should be saved in /nix/var/log/nix/drvs. * Allow the verbosity to be set through an option. * Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
if ((hook && fd == hook->builderOut.readSide) ||
(!hook && fd == builderOut.readSide))
* Always print hook output on stderr, even if --no-build-output is set. * In the build hook, print a trace message to allow Hydra to pick up the name of the remote machine used for the build.
2010-08-31 12:36:24 +00:00
{
if (verbosity >= buildVerbosity)
Use data() instead of c_str() where appropriate
2012-02-09 17:27:45 +00:00
writeToStderr((unsigned char *) data.data(), data.size());
Compress build logs on the fly using bzip2
2012-05-30 14:12:29 +00:00
if (bzLogFile) {
int err;
BZ2_bzWrite(&err, bzLogFile, (unsigned char *) data.data(), data.size());
if (err != BZ_OK) throw Error(format("cannot write to compressed log file (BZip2 error = %1%)") % err);
}
* Always print hook output on stderr, even if --no-build-output is set. * In the build hook, print a trace message to allow Hydra to pick up the name of the remote machine used for the build.
2010-08-31 12:36:24 +00:00
}
if (hook && fd == hook->fromHook.readSide)
Use data() instead of c_str() where appropriate
2012-02-09 17:27:45 +00:00
writeToStderr((unsigned char *) data.data(), data.size());
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
}
void DerivationGoal::handleEOF(int fd)
{
* Made the build hook mechanism more efficient. Rather than starting the hook every time we want to ask whether we can run a remote build (which can be very often), we now reuse a hook process for answering those queries until it accepts a build. So if there are N derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
worker.wakeUp(shared_from_this());
* Write build logs to disk again.
2004-06-29 09:41:50 +00:00
}
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 10:55:33 +00:00
PathSet DerivationGoal::checkPathValidity(bool returnValid)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 10:55:33 +00:00
PathSet result;
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (DerivationOutputs::iterator, i, drv.outputs)
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
if (worker.store.isValidPath(i->second.path)) {
* In nix-store: change `--build' back to `--realise'. Also brought back the query flag `--force-realise'. * Fixed some of the tests.
2005-01-25 10:55:33 +00:00
if (returnValid) result.insert(i->second.path);
} else {
if (!returnValid) result.insert(i->second.path);
}
return result;
* Added a switch `--fallback'. From the manual: Whenever Nix attempts to realise a derivation for which a closure is already known, but this closure cannot be realised, fall back on normalising the derivation. The most common scenario in which this is useful is when we have registered substitutes in order to perform binary distribution from, say, a network repository. If the repository is down, the realisation of the derivation will fail. When this option is specified, Nix will build the derivation instead. Thus, binary installation falls back on a source installation. This option is not the default since it is generally not desirable for a transient failure in obtaining the substitutes to lead to a full build from source (with the related consumption of resources).
2004-06-28 10:42:57 +00:00
}
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
bool DerivationGoal::pathFailed(const Path & path)
{
if (!worker.cacheFailure) return false;
if (!worker.store.hasPathFailed(path)) return false;
printMsg(lvlError, format("builder for `%1%' failed previously (cached)") % path);
if (printBuildTrace)
printMsg(lvlError, format("@ build-failed %1% %2% cached") % drvPath % path);
amDone(ecFailed);
return true;
}
* Refactorings.
2003-07-20 19:29:38 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
//////////////////////////////////////////////////////////////////////
* Refactorings.
2003-07-20 19:29:38 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
class SubstitutionGoal : public Goal
{
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
friend class Worker;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
private:
/* The store path that should be realised through a substitute. */
Path storePath;
* Fixed a serious bug in the computation of slices. Sometimes the slices would not be properly closed under the path reference relation.
2003-08-20 11:30:45 +00:00
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
/* The remaining substituters. */
Paths subs;
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
/* The current substituter. */
Path sub;
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
/* Whether any substituter can realise this path */
bool hasSubstitute;
* Make nix-env --dry-run print the paths to be substituted correctly again. (After the previous substituter mechanism refactoring I didn't update the code that obtains the references of substitutable paths.) This required some refactoring: the substituter programs are now kept running and receive/respond to info requests via stdin/stdout.
2008-08-02 12:54:35 +00:00
/* Path info returned by the substituter's query info operation. */
SubstitutablePathInfo info;
* Really fix the substitute mechanism, i.e., ensure the closure invariant by registering references through the manifest. * Added a test for nix-pull.
2005-01-25 17:08:52 +00:00
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
/* Pipe for the substitute's standard output/error. */
Pipe logPipe;
/* The process ID of the builder. */
* Wrapper class around pids.
2004-06-22 09:51:44 +00:00
Pid pid;
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
* Acquire a lock on the output path when running a substitute. Also delete obstructing invalid paths.
2004-06-21 09:35:50 +00:00
/* Lock on the store path. */
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 21:06:23 +00:00
boost::shared_ptr<PathLocks> outputLock;
* Acquire a lock on the output path when running a substitute. Also delete obstructing invalid paths.
2004-06-21 09:35:50 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
typedef void (SubstitutionGoal::*GoalState)();
GoalState state;
* Fixed a serious bug in the computation of slices. Sometimes the slices would not be properly closed under the path reference relation.
2003-08-20 11:30:45 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
public:
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
SubstitutionGoal(const Path & storePath, Worker & worker);
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
~SubstitutionGoal();
* Change the abstract syntax of slices. It used to be that ids were used as keys to reference slice elements, e.g., Slice(["1ef7..."], [("/nix/store/1ef7...-foo", "1ef7", ["8c99..."]), ...]) This was wrong, since ids represent contents, not locations. Therefore we now have: Slice(["/nix/store/1ef7..."], [("/nix/store/1ef7...-foo", "1ef7", ["/nix/store/8c99-..."]), ...]) * Fix a bug in the computation of slice closures that could cause slice elements to be duplicated.
2003-08-20 12:39:56 +00:00
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
void cancel();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
void work();
* Fixed a serious bug in the computation of slices. Sometimes the slices would not be properly closed under the path reference relation.
2003-08-20 11:30:45 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* The states. */
void init();
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
void tryNext();
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
void gotInfo();
void referencesValid();
* Substitutes should occupy a build slot.
2004-06-22 09:00:31 +00:00
void tryToRun();
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
void finished();
* Some more diagnostics changes.
2004-06-22 17:04:10 +00:00
* Write build logs to disk again.
2004-06-29 09:41:50 +00:00
/* Callback used by the worker to write to the log. */
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
void handleChildOutput(int fd, const string & data);
void handleEOF(int fd);
nix-store -r: do substitutions in parallel I.e. when multiple non-derivation arguments are passed to ‘nix-store -r’ to be substituted, do them in parallel.
2012-06-27 20:58:15 +00:00
Path getStorePath() { return storePath; }
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
};
* Fixed a serious bug in the computation of slices. Sometimes the slices would not be properly closed under the path reference relation.
2003-08-20 11:30:45 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
SubstitutionGoal::SubstitutionGoal(const Path & storePath, Worker & worker)
: Goal(worker)
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
, hasSubstitute(false)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
this->storePath = storePath;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
state = &SubstitutionGoal::init;
* Subtle bug in the builder: if a subgoal that is instantiated multiple times is also a top-level goal, then the second and later instantiations would never be created because there would be a stable pointer to the first one that would keep it alive in the WeakGoalMap. * Some tracing code for debugging this kind of problem.
2005-02-18 09:50:20 +00:00
name = (format("substitution of `%1%'") % storePath).str();
trace("created");
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
SubstitutionGoal::~SubstitutionGoal()
{
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
/* !!! Once we let substitution goals run under a build user, we
* Unify exportReferencesGraph and exportBuildReferencesGraph, and make sure that it works as expected when you pass it a derivation. That is, we have to make sure that all build-time dependencies are built, and that they are all in the input closure (otherwise remote builds might fail, for example). This is ensured at instantiation time by adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 17:36:42 +00:00
need to use the setuid helper just as in ~DerivationGoal().
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
Idem for cancel. */
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
if (pid != -1) worker.childTerminated(pid);
}
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
void SubstitutionGoal::cancel()
{
if (pid != -1) {
pid_t savedPid = pid;
pid.kill();
worker.childTerminated(savedPid);
}
amDone(ecFailed);
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
void SubstitutionGoal::work()
{
(this->*state)();
}
void SubstitutionGoal::init()
{
* Some more diagnostics changes.
2004-06-22 17:04:10 +00:00
trace("init");
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
worker.store.addTempRoot(storePath);
* Start of concurrent garbage collection. Processes write temporary roots to a per-process temporary file in /nix/var/nix/temproots while holding a write lock on that file. The garbage collector acquires read locks on all those files, thus blocking further progress in other Nix processes, and reads the sets of temporary roots.
2005-01-31 10:27:25 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* If the path already exists we're done. */
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
if (worker.store.isValidPath(storePath)) {
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
amDone(ecSuccess);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
return;
* Fixed a serious bug in the computation of slices. Sometimes the slices would not be properly closed under the path reference relation.
2003-08-20 11:30:45 +00:00
}
* Give a better error message when trying to build something and readOnlyMode is set.
2009-12-09 17:45:22 +00:00
if (readOnlyMode)
throw Error(format("cannot substitute path `%1%' - no write access to the Nix store") % storePath);
* build.cc: only use a substituter if it returns info for a path.
2008-08-04 13:15:35 +00:00
subs = substituters;
tryNext();
}
void SubstitutionGoal::tryNext()
{
trace("trying next substituter");
if (subs.size() == 0) {
/* None left. Terminate this goal and let someone else deal
with it. */
* Get rid of some superfluous error messages if a substituter fails. * Say "fetch" instead of "substitute".
2011-11-29 13:00:41 +00:00
debug(format("path `%1%' is required, but there is no substituter that can build it") % storePath);
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
/* Hack: don't indicate failure if there were no substituters.
In that case the calling derivation should just do a
build. */
amDone(hasSubstitute ? ecFailed : ecNoSubstituters);
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
return;
}
* build.cc: only use a substituter if it returns info for a path.
2008-08-04 13:15:35 +00:00
sub = subs.front();
subs.pop_front();
download-from-binary-cache: parallelise fetching of NAR info files Getting substitute information using the binary cache substituter has non-trivial latency overhead. A package or NixOS system configuration can have hundreds of dependencies, and in the worst case (when the local info cache is empty) we have to do a separate HTTP request for each of these. If the ping time to the server is t, getting N info files will take tN seconds; e.g., with a ping time of 0.1s to nixos.org, sequentially downloading 1000 info files (a typical NixOS config) will take at least 100 seconds. To fix this problem, the binary cache substituter can now perform requests in parallel. This required changing the substituter interface to support a function querySubstitutablePathInfos() that queries multiple paths at the same time, and rewriting queryMissing() to take advantage of parallelism. (Due to local caching, parallelising queryMissing() is sufficient for most use cases, since it's almost always called before building a derivation and thus fills the local info cache.) For example, parallelism speeds up querying all 1056 paths in a particular NixOS system configuration from 116s to 2.6s. It works so well because the eccentricity of the top-level derivation in the dependency graph is only 9. So we only need 10 round-trips (when using an unlimited number of parallel connections) to get everything. Currently we do a maximum of 150 parallel connections to the server. Thus it's important that the binary cache server (e.g. nixos.org) has a high connection limit. Alternatively we could use HTTP pipelining, but WWW::Curl doesn't support it and libcurl has a hard-coded limit of 5 requests per pipeline.
2012-07-06 23:08:20 +00:00
SubstitutablePathInfos infos;
PathSet dummy(singleton<PathSet>(storePath));
worker.store.querySubstitutablePathInfos(sub, dummy, infos);
SubstitutablePathInfos::iterator k = infos.find(storePath);
if (k == infos.end()) { tryNext(); return; }
info = k->second;
build.cc: Don't use hasSubstitute() Instead make a single call to querySubstitutablePathInfo() per derivation output. This is faster and prevents having to implement the "have" function in the binary cache substituter.
2012-07-08 22:39:24 +00:00
hasSubstitute = true;
* build.cc: only use a substituter if it returns info for a path.
2008-08-04 13:15:35 +00:00
* Typo.
2005-09-20 16:14:00 +00:00
/* To maintain the closure invariant, we first have to realise the
* Really fix the substitute mechanism, i.e., ensure the closure invariant by registering references through the manifest. * Added a test for nix-pull.
2005-01-25 17:08:52 +00:00
paths referenced by this one. */
* Make nix-env --dry-run print the paths to be substituted correctly again. (After the previous substituter mechanism refactoring I didn't update the code that obtains the references of substitutable paths.) This required some refactoring: the substituter programs are now kept running and receive/respond to info requests via stdin/stdout.
2008-08-02 12:54:35 +00:00
foreach (PathSet::iterator, i, info.references)
* Add missing files to dist. * Fix GC and substitute bugs related to self-references. Add a regression test.
2005-02-01 09:23:38 +00:00
if (*i != storePath) /* ignore self-references */
addWaitee(worker.makeSubstitutionGoal(*i));
* Really fix the substitute mechanism, i.e., ensure the closure invariant by registering references through the manifest. * Added a test for nix-pull.
2005-01-25 17:08:52 +00:00
if (waitees.empty()) /* to prevent hang (no wake-up event) */
referencesValid();
else
state = &SubstitutionGoal::referencesValid;
}
void SubstitutionGoal::referencesValid()
{
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
trace("all references realised");
* Really fix the substitute mechanism, i.e., ensure the closure invariant by registering references through the manifest. * Added a test for nix-pull.
2005-01-25 17:08:52 +00:00
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
if (nrFailed > 0) {
* Get rid of some superfluous error messages if a substituter fails. * Say "fetch" instead of "substitute".
2011-11-29 13:00:41 +00:00
debug(format("some references of path `%1%' could not be realised") % storePath);
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
amDone(ecFailed);
return;
}
* Really fix the substitute mechanism, i.e., ensure the closure invariant by registering references through the manifest. * Added a test for nix-pull.
2005-01-25 17:08:52 +00:00
* Make nix-env --dry-run print the paths to be substituted correctly again. (After the previous substituter mechanism refactoring I didn't update the code that obtains the references of substitutable paths.) This required some refactoring: the substituter programs are now kept running and receive/respond to info requests via stdin/stdout.
2008-08-02 12:54:35 +00:00
foreach (PathSet::iterator, i, info.references)
* Add missing files to dist. * Fix GC and substitute bugs related to self-references. Add a regression test.
2005-02-01 09:23:38 +00:00
if (*i != storePath) /* ignore self-references */
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
assert(worker.store.isValidPath(*i));
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
* Substitutes should occupy a build slot.
2004-06-22 09:00:31 +00:00
state = &SubstitutionGoal::tryToRun;
* Do a substitution even if --max-jobs == 0.
2009-03-31 21:14:07 +00:00
worker.wakeUp(shared_from_this());
* Substitutes should occupy a build slot.
2004-06-22 09:00:31 +00:00
}
void SubstitutionGoal::tryToRun()
{
* Some more diagnostics changes.
2004-06-22 17:04:10 +00:00
trace("trying to run");
* Do a substitution even if --max-jobs == 0.
2009-03-31 21:14:07 +00:00
/* Make sure that we are allowed to start a build. Note that even
is maxBuildJobs == 0 (no local builds allowed), we still allow
a substituter to run. This is because substitutions cannot be
distributed to another machine via the build hook. */
if (worker.getNrLocalBuilds() >= (maxBuildJobs == 0 ? 1 : maxBuildJobs)) {
* Substitutes should occupy a build slot.
2004-06-22 09:00:31 +00:00
worker.waitForBuildSlot(shared_from_this());
return;
}
* Fix a race condition with parallel builds where multiple fixed-output derivations or substitutions try to build the same store path at the same time. Locking generally catches this, but not between multiple goals in the same process. This happened especially often (actually, only) in the build farm with fetchurl downloads of the same file being executed on multiple machines and then copied back to the main machine where they would clobber each other (NIXBF-13). Solution: if a goal notices that the output path is already locked, then go to sleep until another goal finishes (hopefully the one locking the path) and try again.
2007-08-28 11:36:17 +00:00
/* Maybe a derivation goal has already locked this path
(exceedingly unlikely, since it should have used a substitute
first, but let's be defensive). */
* When there are multiple substituters, make sure to release the lock on the output path after trying each. Otherwise the pathIsLockedByMe() test gets confused.
2007-08-28 16:22:08 +00:00
outputLock.reset(); // make sure this goal's lock is gone
* Fix a race condition with parallel builds where multiple fixed-output derivations or substitutions try to build the same store path at the same time. Locking generally catches this, but not between multiple goals in the same process. This happened especially often (actually, only) in the build farm with fetchurl downloads of the same file being executed on multiple machines and then copied back to the main machine where they would clobber each other (NIXBF-13). Solution: if a goal notices that the output path is already locked, then go to sleep until another goal finishes (hopefully the one locking the path) and try again.
2007-08-28 11:36:17 +00:00
if (pathIsLockedByMe(storePath)) {
debug(format("restarting substitution of `%1%' because it's locked by another goal")
% storePath);
worker.waitForAnyGoal(shared_from_this());
return; /* restart in the tryToRun() state when another goal finishes */
}
* Acquire a lock on the output path when running a substitute. Also delete obstructing invalid paths.
2004-06-21 09:35:50 +00:00
/* Acquire a lock on the output path. */
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 21:06:23 +00:00
outputLock = boost::shared_ptr<PathLocks>(new PathLocks);
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
if (!outputLock->lockPaths(singleton<PathSet>(storePath), "", false)) {
worker.waitForAWhile(shared_from_this());
return;
}
* Acquire a lock on the output path when running a substitute. Also delete obstructing invalid paths.
2004-06-21 09:35:50 +00:00
/* Check again whether the path is invalid. */
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
if (worker.store.isValidPath(storePath)) {
* Acquire a lock on the output path when running a substitute. Also delete obstructing invalid paths.
2004-06-21 09:35:50 +00:00
debug(format("store path `%1%' has become valid") % storePath);
* Multiple and/or failing substitutes now work.
2004-06-24 13:40:38 +00:00
outputLock->setDeletion(true);
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
amDone(ecSuccess);
* Acquire a lock on the output path when running a substitute. Also delete obstructing invalid paths.
2004-06-21 09:35:50 +00:00
return;
}
* Get rid of some superfluous error messages if a substituter fails. * Say "fetch" instead of "substitute".
2011-11-29 13:00:41 +00:00
printMsg(lvlInfo, format("fetching path `%1%'...") % storePath);
* Some more diagnostics changes.
2004-06-22 17:04:10 +00:00
logPipe.create();
* Acquire a lock on the output path when running a substitute. Also delete obstructing invalid paths.
2004-06-21 09:35:50 +00:00
/* Remove the (stale) output path if it exists. */
if (pathExists(storePath))
* Change the ownership of store paths to the Nix account before deleting them using the setuid helper.
2006-12-07 14:14:35 +00:00
deletePathWrapped(storePath);
* Acquire a lock on the output path when running a substitute. Also delete obstructing invalid paths.
2004-06-21 09:35:50 +00:00
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
/* Fork the substitute program. */
* Wrapper class around pids.
2004-06-22 09:51:44 +00:00
pid = fork();
switch (pid) {
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
case -1:
throw SysError("unable to fork");
case 0:
try { /* child */
logPipe.readSide.close();
* Reduce gratuitous cut & pasting.
2004-06-22 10:21:44 +00:00
commonChildInit(logPipe);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
* Reduce gratuitous cut & pasting.
2004-06-22 10:21:44 +00:00
/* Fill in the arguments. */
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
Strings args;
args.push_back(baseNameOf(sub));
args.push_back("--substitute");
args.push_back(storePath);
* Reduce gratuitous cut & pasting.
2004-06-22 10:21:44 +00:00
const char * * argArr = strings2CharPtrs(args);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
execv(sub.c_str(), (char * *) argArr);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
throw SysError(format("executing `%1%'") % sub);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 21:06:23 +00:00
} catch (std::exception & e) {
* Support for doing builds in a chroot under Linux. The builder is executed in a chroot that contains just the Nix store, the temporary build directory, and a configurable set of additional directories (/dev and /proc by default). This allows a bit more purity enforcement: hidden build-time dependencies on directories such as /usr or /nix/var/nix/profiles are no longer possible. As an added benefit, accidental network downloads (cf. NIXPKGS-52) are prevented as well (because files such as /etc/resolv.conf are not available in the chroot). However the usefulness of chroots is diminished by the fact that many builders depend on /bin/sh, so you need /bin in the list of additional directories. (And then on non-NixOS you need /lib as well...)
2007-10-27 00:46:59 +00:00
std::cerr << format("substitute error: %1%") % e.what() << std::endl;
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
}
* Call find-runtime-roots.pl from the garbage collector to prevent running applications etc. from being garbage collected.
2006-07-20 12:17:25 +00:00
quickExit(1);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
}
/* parent */
* Wrapper class around pids.
2004-06-22 09:51:44 +00:00
pid.setSeparatePG(true);
* Terminate build hooks and substitutes with a TERM signal, not a KILL signal. This is necessary because those processes may have joined the BDB environment, so they have to be given a chance to clean up. (NIX-85)
2007-03-19 12:48:45 +00:00
pid.setKillSignal(SIGTERM);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
logPipe.writeSide.close();
worker.childStarted(shared_from_this(),
* Revert r19797, and use a simpler solution: just don't monitor build hooks for silence. It's unnecessary because the remote nix-store command is already monitoring the real build.
2010-02-03 21:38:41 +00:00
pid, singleton<set<int> >(logPipe.readSide), true, true);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
state = &SubstitutionGoal::finished;
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
if (printBuildTrace) {
printMsg(lvlError, format("@ substituter-started %1% %2%")
% storePath % sub);
}
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
}
void SubstitutionGoal::finished()
{
* Some more diagnostics changes.
2004-06-22 17:04:10 +00:00
trace("substitute finished");
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
/* Since we got an EOF on the logger pipe, the substitute is
presumed to have terminated. */
* Wrapper class around pids.
2004-06-22 09:51:44 +00:00
pid_t savedPid = pid;
int status = pid.wait(true);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
/* So the child is gone now. */
* Wrapper class around pids.
2004-06-22 09:51:44 +00:00
worker.childTerminated(savedPid);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
/* Close the read side of the logger pipe. */
logPipe.readSide.close();
debug(format("substitute for `%1%' finished") % storePath);
* Multiple and/or failing substitutes now work.
2004-06-24 13:40:38 +00:00
/* Check the exit status and the build result. */
try {
if (!statusOk(status))
* Get rid of some superfluous error messages if a substituter fails. * Say "fetch" instead of "substitute".
2011-11-29 13:00:41 +00:00
throw SubstError(format("fetching path `%1%' %2%")
* Multiple and/or failing substitutes now work.
2004-06-24 13:40:38 +00:00
% storePath % statusToString(status));
if (!pathExists(storePath))
* Get rid of some superfluous error messages if a substituter fails. * Say "fetch" instead of "substitute".
2011-11-29 13:00:41 +00:00
throw SubstError(format("substitute did not produce path `%1%'") % storePath);
* Multiple and/or failing substitutes now work.
2004-06-24 13:40:38 +00:00
} catch (SubstError & e) {
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
* Get rid of some superfluous error messages if a substituter fails. * Say "fetch" instead of "substitute".
2011-11-29 13:00:41 +00:00
printMsg(lvlInfo, e.msg());
* Multiple and/or failing substitutes now work.
2004-06-24 13:40:38 +00:00
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
if (printBuildTrace) {
printMsg(lvlError, format("@ substituter-failed %1% %2% %3%")
% storePath % status % e.msg());
}
* Multiple and/or failing substitutes now work.
2004-06-24 13:40:38 +00:00
/* Try the next substitute. */
state = &SubstitutionGoal::tryNext;
worker.wakeUp(shared_from_this());
return;
}
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
* Canonicalise metadata of all files in store paths (set the mtime to 0, set the mode to either 444 or 555, set the group to the default).
2004-11-29 15:09:29 +00:00
canonicalisePathMetaData(storePath);
* Make store objects created by substitutes read-only.
2004-09-22 12:15:04 +00:00
* Store the size of a store path in the database (to be precise, the size of the NAR serialisation of the path, i.e., `nix-store --dump PATH'). This is useful for Hydra.
2010-11-16 17:11:46 +00:00
HashResult hash = hashPath(htSHA256, storePath);
ValidPathInfo info2;
info2.path = storePath;
info2.hash = hash.first;
info2.narSize = hash.second;
info2.references = info.references;
info2.deriver = info.deriver;
worker.store.registerValidPath(info2);
* Re-enable support for substitutes in the normaliser. * A better substitute mechanism. Instead of generating a store expression for each store path for which we have a substitute, we can have a single store expression that builds a generic program that is invoked to build the desired store path, which is passed as an argument. This means that operations like `nix-pull' only produce O(1) files instead of O(N) files in the store when registering N substitutes. (It consumes O(N) database storage, of course, but that's not a performance problem). * Added a test for the substitute mechanism. * `nix-store --substitute' reads the substitutes from standard input, instead of from the command line. This prevents us from running into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
* Multiple and/or failing substitutes now work.
2004-06-24 13:40:38 +00:00
outputLock->setDeletion(true);
* Acquire a lock on the output path when running a substitute. Also delete obstructing invalid paths.
2004-06-21 09:35:50 +00:00
* Multiple and/or failing substitutes now work.
2004-06-24 13:40:38 +00:00
printMsg(lvlChatty,
format("substitution of path `%1%' succeeded") % storePath);
* Some somewhat ad hoc mechanism to allow the build farm to monitor build progress.
2008-11-12 11:08:27 +00:00
if (printBuildTrace) {
printMsg(lvlError, format("@ substituter-succeeded %1%") % storePath);
}
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
amDone(ecSuccess);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
void SubstitutionGoal::handleChildOutput(int fd, const string & data)
* Write build logs to disk again.
2004-06-29 09:41:50 +00:00
{
assert(fd == logPipe.readSide);
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
if (verbosity >= buildVerbosity)
Use data() instead of c_str() where appropriate
2012-02-09 17:27:45 +00:00
writeToStderr((unsigned char *) data.data(), data.size());
* Write build logs to disk again.
2004-06-29 09:41:50 +00:00
/* Don't write substitution output to a log file for now. We
probably should, though. */
}
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
void SubstitutionGoal::handleEOF(int fd)
{
if (fd == logPipe.readSide) worker.wakeUp(shared_from_this());
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
//////////////////////////////////////////////////////////////////////
static bool working = false;
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
Worker::Worker(LocalStore & store)
: store(store)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
/* Debugging: prevent recursive workers. */
if (working) abort();
working = true;
* Do a substitution even if --max-jobs == 0.
2009-03-31 21:14:07 +00:00
nrLocalBuilds = 0;
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
lastWokenUp = 0;
* Negative caching, i.e. caching of build failures. Disabled by default. This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
cacheFailure = queryBoolSetting("build-cache-failure", false);
* nix-instantiate: return exit status 100 to denote a permanent build failure. The build hook can use this to distinguish between transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
permanentFailure = false;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
Worker::~Worker()
{
working = false;
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
/* Explicitly get rid of all strong pointers now. After this all
goals that refer to this worker should be gone. (Otherwise we
are in trouble, since goals may call childTerminated() etc. in
their destructors). */
topGoals.clear();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
template<class T>
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
static GoalPtr addGoal(const Path & path,
Worker & worker, WeakGoalMap & goalMap)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
GoalPtr goal = goalMap[path].lock();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
if (!goal) {
goal = GoalPtr(new T(path, worker));
goalMap[path] = goal;
worker.wakeUp(goal);
* Refactorings.
2003-07-20 19:29:38 +00:00
}
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
return goal;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
* Refactorings.
2003-07-20 19:29:38 +00:00
* In normaliseFState(), wrap registration of the output paths and the normal form in a single transaction to ensure that if we crash, either everything is registered or nothing is. This is for recoverability: unregistered paths in the store can be deleted arbitrarily, while registered paths can only be deleted by running the garbage collector.
2003-08-01 15:41:47 +00:00
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
GoalPtr Worker::makeDerivationGoal(const Path & nePath)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
* Started removing closure store expressions, i.e., the explicit representation of closures as ATerms in the Nix store. Instead, the file system pointer graph is now stored in the Nix database. This has many advantages: - It greatly simplifies the implementation (we can drop the notion of `successors', and so on). - It makes registering roots for the garbage collector much easier. Instead of specifying the closure expression as a root, you can simply specify the store path that must be retained as a root. This could not be done previously, since there was no way to find the closure store expression containing a given store path. - Better traceability: it is now possible to query what paths are referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
return addGoal<DerivationGoal>(nePath, *this, derivationGoals);
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
}
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
GoalPtr Worker::makeSubstitutionGoal(const Path & storePath)
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
{
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
return addGoal<SubstitutionGoal>(storePath, *this, substitutionGoals);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
static void removeGoal(GoalPtr goal, WeakGoalMap & goalMap)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
* Subtle bug in the builder: if a subgoal that is instantiated multiple times is also a top-level goal, then the second and later instantiations would never be created because there would be a stable pointer to the first one that would keep it alive in the WeakGoalMap. * Some tracing code for debugging this kind of problem.
2005-02-18 09:50:20 +00:00
/* !!! inefficient */
for (WeakGoalMap::iterator i = goalMap.begin();
i != goalMap.end(); )
if (i->second.lock() == goal) {
WeakGoalMap::iterator j = i; ++j;
goalMap.erase(i);
i = j;
}
else ++i;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
void Worker::removeGoal(GoalPtr goal)
{
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 21:06:23 +00:00
nix::removeGoal(goal, derivationGoals);
nix::removeGoal(goal, substitutionGoals);
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
if (topGoals.find(goal) != topGoals.end()) {
topGoals.erase(goal);
/* If a top-level goal failed, then kill all other goals
(unless keepGoing was set). */
if (goal->getExitCode() == Goal::ecFailed && !keepGoing)
topGoals.clear();
}
* Fix a race condition with parallel builds where multiple fixed-output derivations or substitutions try to build the same store path at the same time. Locking generally catches this, but not between multiple goals in the same process. This happened especially often (actually, only) in the build farm with fetchurl downloads of the same file being executed on multiple machines and then copied back to the main machine where they would clobber each other (NIXBF-13). Solution: if a goal notices that the output path is already locked, then go to sleep until another goal finishes (hopefully the one locking the path) and try again.
2007-08-28 11:36:17 +00:00
/* Wake up goals waiting for any goal to finish. */
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
foreach (WeakGoals::iterator, i, waitingForAnyGoal) {
* Fix a race condition with parallel builds where multiple fixed-output derivations or substitutions try to build the same store path at the same time. Locking generally catches this, but not between multiple goals in the same process. This happened especially often (actually, only) in the build farm with fetchurl downloads of the same file being executed on multiple machines and then copied back to the main machine where they would clobber each other (NIXBF-13). Solution: if a goal notices that the output path is already locked, then go to sleep until another goal finishes (hopefully the one locking the path) and try again.
2007-08-28 11:36:17 +00:00
GoalPtr goal = i->lock();
if (goal) wakeUp(goal);
}
waitingForAnyGoal.clear();
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
}
* Remove lock files after building.
2003-11-21 16:05:19 +00:00
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
void Worker::wakeUp(GoalPtr goal)
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
{
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
goal->trace("woken up");
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
awake.insert(goal);
}
* True parallel builds. Nix can now run as many build jobs in parallel as possible (similar to GNU Make's `-j' switch). This is useful on SMP systems, but it is especially useful for doing builds on multiple machines. The idea is that a large derivation is initiated on one master machine, which then distributes sub-derivations to any number of slave machines. This should not happen synchronously or in lock-step, so the master must be capable of dealing with multiple parallel build jobs. We now have the infrastructure to support this. TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Do a substitution even if --max-jobs == 0.
2009-03-31 21:14:07 +00:00
unsigned Worker::getNrLocalBuilds()
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
* Do a substitution even if --max-jobs == 0.
2009-03-31 21:14:07 +00:00
return nrLocalBuilds;
* Refactorings.
2003-07-20 19:29:38 +00:00
}
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
void Worker::childStarted(GoalPtr goal,
* Revert r19797, and use a simpler solution: just don't monitor build hooks for silence. It's unnecessary because the remote nix-store command is already monitoring the real build.
2010-02-03 21:38:41 +00:00
pid_t pid, const set<int> & fds, bool inBuildSlot,
bool monitorForSilence)
* Refactorings.
2003-07-20 19:29:38 +00:00
{
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
Child child;
child.goal = goal;
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
child.fds = fds;
* Kill a build if it has gone for more than a certain number of seconds without producing output on stdout or stderr (NIX-65). This timeout can be specified using the `--max-silent-time' option or the `build-max-silent-time' configuration setting. The default is infinity (0). * Fix a tricky race condition: if we kill the build user before the child has done its setuid() to the build user uid, then it won't be killed, and we'll potentially lock up in pid.wait(). So also send a conventional kill to the child.
2006-12-08 15:44:00 +00:00
child.lastOutput = time(0);
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
child.inBuildSlot = inBuildSlot;
* Revert r19797, and use a simpler solution: just don't monitor build hooks for silence. It's unnecessary because the remote nix-store command is already monitoring the real build.
2010-02-03 21:38:41 +00:00
child.monitorForSilence = monitorForSilence;
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
children[pid] = child;
* Do a substitution even if --max-jobs == 0.
2009-03-31 21:14:07 +00:00
if (inBuildSlot) nrLocalBuilds++;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
* Refactorings.
2003-07-20 19:29:38 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Don't go into a (sometimes infinite) loop calling the build hook.
2004-07-01 16:24:35 +00:00
void Worker::childTerminated(pid_t pid, bool wakeSleepers)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
assert(pid != -1); /* common mistake */
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
Children::iterator i = children.find(pid);
assert(i != children.end());
if (i->second.inBuildSlot) {
* Do a substitution even if --max-jobs == 0.
2009-03-31 21:14:07 +00:00
assert(nrLocalBuilds > 0);
nrLocalBuilds--;
* Re-enable build hooks.
2004-06-19 21:45:04 +00:00
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
children.erase(pid);
* Don't go into a (sometimes infinite) loop calling the build hook.
2004-07-01 16:24:35 +00:00
if (wakeSleepers) {
/* Wake up goals waiting for a build slot. */
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (WeakGoals::iterator, i, wantingToBuild) {
* Don't go into a (sometimes infinite) loop calling the build hook.
2004-07-01 16:24:35 +00:00
GoalPtr goal = i->lock();
if (goal) wakeUp(goal);
}
wantingToBuild.clear();
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
}
* Substitutes and nix-pull now work again. * Fixed a segfault caused by the buffering of stderr. * Fix now allows the specification of the full output path. This should be used with great care, since it by-passes the normal hash generation. * Incremented the version number to 0.4 (prerelease).
2003-10-16 16:29:57 +00:00
}
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
void Worker::waitForBuildSlot(GoalPtr goal)
* Substitutes and nix-pull now work again. * Fixed a segfault caused by the buffering of stderr. * Fix now allows the specification of the full output path. This should be used with great care, since it by-passes the normal hash generation. * Incremented the version number to 0.4 (prerelease).
2003-10-16 16:29:57 +00:00
{
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
debug("wait for build slot");
* Do a substitution even if --max-jobs == 0.
2009-03-31 21:14:07 +00:00
if (getNrLocalBuilds() < maxBuildJobs)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
wakeUp(goal); /* we can do it right away */
else
wantingToBuild.insert(goal);
}
* Regression fix: realise substitutes and detect cycles.
2004-02-13 10:45:09 +00:00
* Substitutes and nix-pull now work again. * Fixed a segfault caused by the buffering of stderr. * Fix now allows the specification of the full output path. This should be used with great care, since it by-passes the normal hash generation. * Incremented the version number to 0.4 (prerelease).
2003-10-16 16:29:57 +00:00
* Fix a race condition with parallel builds where multiple fixed-output derivations or substitutions try to build the same store path at the same time. Locking generally catches this, but not between multiple goals in the same process. This happened especially often (actually, only) in the build farm with fetchurl downloads of the same file being executed on multiple machines and then copied back to the main machine where they would clobber each other (NIXBF-13). Solution: if a goal notices that the output path is already locked, then go to sleep until another goal finishes (hopefully the one locking the path) and try again.
2007-08-28 11:36:17 +00:00
void Worker::waitForAnyGoal(GoalPtr goal)
{
debug("wait for any goal");
waitingForAnyGoal.insert(goal);
}
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
void Worker::waitForAWhile(GoalPtr goal)
{
debug("wait for a while");
waitingForAWhile.insert(goal);
}
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
void Worker::run(const Goals & _topGoals)
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
{
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (Goals::iterator, i, _topGoals) topGoals.insert(*i);
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
* Started making Nix's diagnostic messages a bit more useful.
2004-06-22 14:48:59 +00:00
startNest(nest, lvlDebug, format("entered goal loop"));
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
while (1) {
* Catch SIGINT to terminate cleanly when the user tries to interrupt Nix. This is to prevent Berkeley DB from becoming wedged. Unfortunately it is not possible to throw C++ exceptions from a signal handler. In fact, you can't do much of anything except change variables of type `volatile sig_atomic_t'. So we set an interrupt flag in the signal handler and check it at various strategic locations in the code (by calling checkInterrupt()). Since this is unlikely to cover all cases (e.g., (semi-)infinite loops), sometimes SIGTERM may now be required to kill Nix.
2004-01-15 20:23:55 +00:00
checkInterrupt();
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Call every wake goal. */
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
while (!awake.empty() && !topGoals.empty()) {
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
WeakGoals awake2(awake);
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
awake.clear();
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (WeakGoals::iterator, i, awake2) {
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
checkInterrupt();
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
GoalPtr goal = i->lock();
if (goal) goal->work();
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
if (topGoals.empty()) break;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
* Substitutes and nix-pull now work again. * Fixed a segfault caused by the buffering of stderr. * Fix now allows the specification of the full output path. This should be used with great care, since it by-passes the normal hash generation. * Incremented the version number to 0.4 (prerelease).
2003-10-16 16:29:57 +00:00
}
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
if (topGoals.empty()) break;
* Substitutes and nix-pull now work again. * Fixed a segfault caused by the buffering of stderr. * Fix now allows the specification of the full output path. This should be used with great care, since it by-passes the normal hash generation. * Incremented the version number to 0.4 (prerelease).
2003-10-16 16:29:57 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Wait for input. */
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
if (!children.empty() || !waitingForAWhile.empty())
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
waitForInput();
* With `--max-jobs 0', print a nicer error message than "Assertion `!awake.empty()' failed."
2009-03-29 18:06:00 +00:00
else {
* Doh!
2009-03-31 15:50:03 +00:00
if (awake.empty() && maxBuildJobs == 0) throw Error(
* With `--max-jobs 0', print a nicer error message than "Assertion `!awake.empty()' failed."
2009-03-29 18:06:00 +00:00
"unable to start any build; either increase `--max-jobs' "
"or enable distributed builds");
* Get rid of the substitutes database table (NIX-47). Instead, if we need any info on substitutable paths, we just call the substituters (such as download-using-manifests.pl) directly. This means that it's no longer necessary for nix-pull to register substitutes or for nix-channel to clear them, which makes those operations much faster (NIX-95). Also, we don't have to worry about keeping nix-pull manifests (in /nix/var/nix/manifests) and the database in sync with each other. The downside is that there is some overhead in calling an external program to get the substitutes info. For instance, "nix-env -qas" takes a bit longer. Abolishing the substitutes table also makes the logic in local-store.cc simpler, as we don't need to store info for invalid paths. On the downside, you cannot do things like "nix-store -qR" on a substitutable but invalid path (but nobody did that anyway). * Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
assert(!awake.empty());
* With `--max-jobs 0', print a nicer error message than "Assertion `!awake.empty()' failed."
2009-03-29 18:06:00 +00:00
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
* Substitutes and nix-pull now work again. * Fixed a segfault caused by the buffering of stderr. * Fix now allows the specification of the full output path. This should be used with great care, since it by-passes the normal hash generation. * Incremented the version number to 0.4 (prerelease).
2003-10-16 16:29:57 +00:00
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
/* If --keep-going is not set, it's possible that the main goal
exited while some of its subgoals were still active. But if
--keep-going *is* set, then they must all be finished now. */
assert(!keepGoing || awake.empty());
assert(!keepGoing || wantingToBuild.empty());
assert(!keepGoing || children.empty());
* Refactorings.
2003-07-20 19:29:38 +00:00
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
void Worker::waitForInput()
* Refactorings.
2003-07-20 19:29:38 +00:00
{
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
printMsg(lvlVomit, "waiting for children");
* Refactorings.
2003-07-20 19:29:38 +00:00
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
/* Process output from the file descriptors attached to the
children, namely log output and output path creation commands.
We also use this to detect child termination: if we get EOF on
the logger pipe of a build, we assume that the builder has
terminated. */
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
bool useTimeout = false;
struct timeval timeout;
timeout.tv_usec = 0;
time_t before = time(0);
Add support for the `build-timeout' and `--timeout' options.
2011-06-30 15:19:13 +00:00
/* If a global timeout has been set, sleep until it's done. */
if (buildTimeout != 0) {
useTimeout = true;
if (lastWait == 0 || lastWait > before) lastWait = before;
timeout.tv_sec = std::max((time_t) 0, lastWait + buildTimeout - before);
}
* Kill a build if it has gone for more than a certain number of seconds without producing output on stdout or stderr (NIX-65). This timeout can be specified using the `--max-silent-time' option or the `build-max-silent-time' configuration setting. The default is infinity (0). * Fix a tricky race condition: if we kill the build user before the child has done its setuid() to the build user uid, then it won't be killed, and we'll potentially lock up in pid.wait(). So also send a conventional kill to the child.
2006-12-08 15:44:00 +00:00
/* If we're monitoring for silence on stdout/stderr, sleep until
the first deadline for any child. */
if (maxSilentTime != 0) {
time_t oldest = 0;
* Cleanup.
2009-03-17 11:42:55 +00:00
foreach (Children::iterator, i, children) {
* Revert r19797, and use a simpler solution: just don't monitor build hooks for silence. It's unnecessary because the remote nix-store command is already monitoring the real build.
2010-02-03 21:38:41 +00:00
if (i->second.monitorForSilence) {
oldest = oldest == 0 || i->second.lastOutput < oldest
? i->second.lastOutput : oldest;
}
}
if (oldest) {
Add support for the `build-timeout' and `--timeout' options.
2011-06-30 15:19:13 +00:00
time_t silenceTimeout = std::max((time_t) 0, oldest + maxSilentTime - before);
timeout.tv_sec = useTimeout
? std::min(silenceTimeout, timeout.tv_sec)
: silenceTimeout;
* Revert r19797, and use a simpler solution: just don't monitor build hooks for silence. It's unnecessary because the remote nix-store command is already monitoring the real build.
2010-02-03 21:38:41 +00:00
useTimeout = true;
printMsg(lvlVomit, format("sleeping %1% seconds") % timeout.tv_sec);
* Kill a build if it has gone for more than a certain number of seconds without producing output on stdout or stderr (NIX-65). This timeout can be specified using the `--max-silent-time' option or the `build-max-silent-time' configuration setting. The default is infinity (0). * Fix a tricky race condition: if we kill the build user before the child has done its setuid() to the build user uid, then it won't be killed, and we'll potentially lock up in pid.wait(). So also send a conventional kill to the child.
2006-12-08 15:44:00 +00:00
}
}
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
/* If we are polling goals that are waiting for a lock, then wake
up after a few seconds at most. */
* Make the poll interval configurable.
2009-03-30 11:34:03 +00:00
int wakeUpInterval = queryIntSetting("build-poll-interval", 5);
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
if (!waitingForAWhile.empty()) {
useTimeout = true;
2009-03-24 14:07:37 +00:00
if (lastWokenUp == 0)
* Make the poll interval configurable.
2009-03-30 11:34:03 +00:00
printMsg(lvlError, "waiting for locks or build slots...");
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
if (lastWokenUp == 0 || lastWokenUp > before) lastWokenUp = before;
timeout.tv_sec = std::max((time_t) 0, lastWokenUp + wakeUpInterval - before);
} else lastWokenUp = 0;
Include <cstring> to ensure that strcpy(), strlen(), and memset() are declared. An "using namespace std" was added locally in those functions that refer to names from <cstring>. That is not pretty, but it's a very portable solution, because strcpy() and friends will be found in both the 'std' and in the global namespace.
2010-06-24 17:51:04 +00:00
using namespace std;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Use select() to wait for the input side of any logger pipe to
become `available'. Note that `available' (i.e., non-blocking)
includes EOF. */
fd_set fds;
FD_ZERO(&fds);
int fdMax = 0;
* Cleanup.
2009-03-17 11:42:55 +00:00
foreach (Children::iterator, i, children) {
foreach (set<int>::iterator, j, i->second.fds) {
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
FD_SET(*j, &fds);
if (*j >= fdMax) fdMax = *j + 1;
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
* Refactorings.
2003-07-20 19:29:38 +00:00
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
if (select(fdMax, &fds, 0, 0, useTimeout ? &timeout : 0) == -1) {
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
if (errno == EINTR) return;
throw SysError("waiting for input");
}
* Refactorings.
2003-07-20 19:29:38 +00:00
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
time_t after = time(0);
* Kill a build if it has gone for more than a certain number of seconds without producing output on stdout or stderr (NIX-65). This timeout can be specified using the `--max-silent-time' option or the `build-max-silent-time' configuration setting. The default is infinity (0). * Fix a tricky race condition: if we kill the build user before the child has done its setuid() to the build user uid, then it won't be killed, and we'll potentially lock up in pid.wait(). So also send a conventional kill to the child.
2006-12-08 15:44:00 +00:00
Add support for the `build-timeout' and `--timeout' options.
2011-06-30 15:19:13 +00:00
/* Keep track of when we were last called. */
lastWait = after;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* Process all available file descriptors. */
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
/* Since goals may be canceled from inside the loop below (causing
them go be erased from the `children' map), we have to be
careful that we don't keep iterators alive across calls to
cancel(). */
set<pid_t> pids;
* Cleanup.
2009-03-17 11:42:55 +00:00
foreach (Children::iterator, i, children) pids.insert(i->first);
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
* Cleanup.
2009-03-17 11:42:55 +00:00
foreach (set<pid_t>::iterator, i, pids) {
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
checkInterrupt();
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
Children::iterator j = children.find(*i);
if (j == children.end()) continue; // child destroyed
GoalPtr goal = j->second.goal.lock();
* A flag `--keep-going / -k' to keep building goals if one fails, as much as possible. (This is similar to GNU Make's `-k' flag.) * Refactoring to implement this: previously we just bombed out when a build failed, but now we have to clean up. In particular this means that goals must be freed quickly --- they shouldn't hang around until the worker exits. So the worker now maintains weak pointers in order not to prevent garbage collection. * Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
assert(goal);
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
set<int> fds2(j->second.fds);
* Cleanup.
2009-03-17 11:42:55 +00:00
foreach (set<int>::iterator, k, fds2) {
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
if (FD_ISSET(*k, &fds)) {
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
unsigned char buffer[4096];
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
ssize_t rd = read(*k, buffer, sizeof(buffer));
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
if (rd == -1) {
if (errno != EINTR)
throw SysError(format("reading from %1%")
% goal->getName());
} else if (rd == 0) {
debug(format("%1%: got EOF") % goal->getName());
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
goal->handleEOF(*k);
j->second.fds.erase(*k);
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
} else {
printMsg(lvlVomit, format("%1%: read %2% bytes")
% goal->getName() % rd);
string data((char *) buffer, rd);
* Goal cancellation inside the waitForInput() loop needs to be handled very carefully, since it can invalidate iterators into the `children' map.
2006-12-08 18:41:48 +00:00
goal->handleChildOutput(*k, data);
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
j->second.lastOutput = after;
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
}
* Kill a build if it has gone for more than a certain number of seconds without producing output on stdout or stderr (NIX-65). This timeout can be specified using the `--max-silent-time' option or the `build-max-silent-time' configuration setting. The default is infinity (0). * Fix a tricky race condition: if we kill the build user before the child has done its setuid() to the build user uid, then it won't be killed, and we'll potentially lock up in pid.wait(). So also send a conventional kill to the child.
2006-12-08 15:44:00 +00:00
if (maxSilentTime != 0 &&
* Revert r19797, and use a simpler solution: just don't monitor build hooks for silence. It's unnecessary because the remote nix-store command is already monitoring the real build.
2010-02-03 21:38:41 +00:00
j->second.monitorForSilence &&
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
after - j->second.lastOutput >= (time_t) maxSilentTime)
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
{
printMsg(lvlError,
format("%1% timed out after %2% seconds of silence")
* Kill a build if it has gone for more than a certain number of seconds without producing output on stdout or stderr (NIX-65). This timeout can be specified using the `--max-silent-time' option or the `build-max-silent-time' configuration setting. The default is infinity (0). * Fix a tricky race condition: if we kill the build user before the child has done its setuid() to the build user uid, then it won't be killed, and we'll potentially lock up in pid.wait(). So also send a conventional kill to the child.
2006-12-08 15:44:00 +00:00
% goal->getName() % maxSilentTime);
* Some refactoring. * Throw more exceptions as BuildErrors instead of Errors. This matters when --keep-going is turned on. (A BuildError is caught and terminates the goal in question, an Error terminates the program.)
2006-12-08 17:26:21 +00:00
goal->cancel();
}
Add support for the `build-timeout' and `--timeout' options.
2011-06-30 15:19:13 +00:00
if (buildTimeout != 0 &&
after - before >= (time_t) buildTimeout)
{
printMsg(lvlError,
format("%1% timed out after %2% seconds of activity")
% goal->getName() % buildTimeout);
goal->cancel();
}
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
}
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
* Doh.
2009-03-24 23:40:24 +00:00
if (!waitingForAWhile.empty() && lastWokenUp + wakeUpInterval <= after) {
* No longer block while waiting for a lock on a store path. Instead poll for it (i.e. if we can't acquire the lock, then let the main select() loop wait for at most a few seconds and then try again). This improves parallelism: if two nix-store processes are both trying to build a path at the same time, the second one shouldn't block; it should first see if it can build other goals. Also, it prevents the deadlocks that have been occuring in Hydra lately, where a process waits for a lock held by another process that's waiting for a lock held by the first. The downside is that polling isn't really elegant, but POSIX doesn't provide a way to wait for locks in a select() loop. The only solution would be to spawn a thread for each lock to do a blocking fcntl() and then signal the main thread, but that would require pthreads.
2009-03-23 01:05:54 +00:00
lastWokenUp = after;
foreach (WeakGoals::iterator, i, waitingForAWhile) {
GoalPtr goal = i->lock();
if (goal) wakeUp(goal);
}
waitingForAWhile.clear();
}
* Refactorings.
2003-07-20 19:29:38 +00:00
}
* nix-instantiate: return exit status 100 to denote a permanent build failure. The build hook can use this to distinguish between transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
unsigned int Worker::exitStatus()
{
return permanentFailure ? 100 : 1;
}
* Beginning of secure multi-user Nix stores. If Nix is started as root (or setuid root), then builds will be performed under one of the users listed in the `build-users' configuration variables. This is to make it impossible to influence build results externally, allowing locally built derivations to be shared safely between users (see ASE-2005 paper). To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
//////////////////////////////////////////////////////////////////////
nix-store -r: do substitutions in parallel I.e. when multiple non-derivation arguments are passed to ‘nix-store -r’ to be substituted, do them in parallel.
2012-06-27 20:58:15 +00:00
void LocalStore::buildPaths(const PathSet & drvPaths)
* `fstateRefs' now works on derive expressions as well. TODO: make this more efficient. * A flag `-n' in 'nix --query' to normalise the argument. Default is not to normalise.
2003-07-29 10:43:12 +00:00
{
* `nix-store --build' now builds its arguments in parallel instead of sequentially (within the limits set by `--jobs'). This should greatly improve the utilisation of the build farm when doing Nixpkgs builds.
2005-01-19 15:02:02 +00:00
startNest(nest, lvlDebug,
format("building %1%") % showPaths(drvPaths));
* Fix the atrocious (exponential? factorial?) time complexity in `nix --query --requisites'.
2003-08-25 14:56:11 +00:00
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
Worker worker(*this);
* `nix-store --build' now builds its arguments in parallel instead of sequentially (within the limits set by `--jobs'). This should greatly improve the utilisation of the build farm when doing Nixpkgs builds.
2005-01-19 15:02:02 +00:00
Goals goals;
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (PathSet::const_iterator, i, drvPaths)
nix-store -r: do substitutions in parallel I.e. when multiple non-derivation arguments are passed to ‘nix-store -r’ to be substituted, do them in parallel.
2012-06-27 20:58:15 +00:00
if (isDerivation(*i))
goals.insert(worker.makeDerivationGoal(*i));
else
goals.insert(worker.makeSubstitutionGoal(*i));
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
worker.run(goals);
PathSet failed;
* Use foreach in a lot of places.
2009-04-21 11:52:16 +00:00
foreach (Goals::iterator, i, goals)
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
if ((*i)->getExitCode() == Goal::ecFailed) {
DerivationGoal * i2 = dynamic_cast<DerivationGoal *>(i->get());
nix-store -r: do substitutions in parallel I.e. when multiple non-derivation arguments are passed to ‘nix-store -r’ to be substituted, do them in parallel.
2012-06-27 20:58:15 +00:00
if (i2) failed.insert(i2->getDrvPath());
else failed.insert(dynamic_cast<SubstitutionGoal *>(i->get())->getStorePath());
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
}
if (!failed.empty())
* nix-instantiate: return exit status 100 to denote a permanent build failure. The build hook can use this to distinguish between transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
throw Error(format("build of %1% failed") % showPaths(failed), worker.exitStatus());
* `fstateRefs' now works on derive expressions as well. TODO: make this more efficient. * A flag `-n' in 'nix --query' to normalise the argument. Default is not to normalise.
2003-07-29 10:43:12 +00:00
}
* Put building in the store API.
2006-11-30 18:02:04 +00:00
void LocalStore::ensurePath(const Path & path)
* Refactorings.
2003-07-20 19:29:38 +00:00
{
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
/* If the path is already valid, we're done. */
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
if (isValidPath(path)) return;
* Big refactoring. Move to a much more explicitly state machine based approach. This makes it much easier to add extra complexity in the normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
* Merged the no-bdb branch (-r10900:HEAD https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
Worker worker(*this);
* When multiple derivations are specified in `nix-store -r', don't continue building when one fails unless `--keep-going' is specified. * When `--keep-going' is specified, print out the set of failing derivations at the end (otherwise it can be hard to find out which failed).
2005-02-23 11:19:27 +00:00
GoalPtr goal = worker.makeSubstitutionGoal(path);
Goals goals = singleton<Goals>(goal);
worker.run(goals);
if (goal->getExitCode() != Goal::ecSuccess)
* nix-instantiate: return exit status 100 to denote a permanent build failure. The build hook can use this to distinguish between transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
throw Error(format("path `%1%' does not exist and cannot be created") % path, worker.exitStatus());
* Refactorings.
2003-07-20 19:29:38 +00:00
}
* Use a proper namespace. * Optimise header file usage a bit. * Compile the parser as C++.
2006-09-04 21:06:23 +00:00
}
Copy permalink