kloenk/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity

Merge remote-tracking branch 'origin/master' into flakes

Browse source
This commit is contained in:
Eelco Dolstra 2019-08-08 15:49:13 +02:00
parent 336afe4d5f 05a10dd835
commit 1d750e0587
No known key found for this signature in database
GPG key ID: 8170B4726D7198DE
28 changed files with 659 additions and 166 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
.github/FUNDING.yml vendored
View file

@ -1,3 +0,0 @@
# These are supported funding model platforms
custom: https://nixos.org/nixos/foundation.html

2
README.md
View file

@ -1,3 +1,5 @@
[![Open Collective supporters](https://opencollective.com/nixos/tiers/supporter/badge.svg?label=Supporters&color=brightgreen)](https://opencollective.com/nixos)
Nix, the purely functional package manager
------------------------------------------

2
doc/manual/advanced-topics/advanced-topics.xml
View file

@ -7,6 +7,8 @@
<title>Advanced Topics</title>
<xi:include href="distributed-builds.xml" />
<xi:include href="cores-vs-jobs.xml" />
<xi:include href="diff-hook.xml" />
<xi:include href="post-build-hook.xml" />
</part>

121
doc/manual/advanced-topics/cores-vs-jobs.xml Normal file
View file

@ -0,0 +1,121 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="chap-tuning-cores-and-jobs">
<title>Tuning Cores and Jobs</title>
<para>Nix has two relevant settings with regards to how your CPU cores
will be utilized: <xref linkend="conf-cores" /> and
<xref linkend="conf-max-jobs" />. This chapter will talk about what
they are, how they interact, and their configuration trade-offs.</para>
<variablelist>
<varlistentry>
<term><xref linkend="conf-max-jobs" /></term>
<listitem><para>
Dictates how many separate derivations will be built at the same
time. If you set this to zero, the local machine will do no
builds. Nix will still substitute from binary caches, and build
remotely if remote builders are configured.
</para></listitem>
</varlistentry>
<varlistentry>
<term><xref linkend="conf-cores" /></term>
<listitem><para>
Suggests how many cores each derivation should use. Similar to
<command>make -j</command>.
</para></listitem>
</varlistentry>
</variablelist>
<para>The <xref linkend="conf-cores" /> setting determines the value of
<envar>NIX_BUILD_CORES</envar>. <envar>NIX_BUILD_CORES</envar> is equal
to <xref linkend="conf-cores" />, unless <xref linkend="conf-cores" />
equals <literal>0</literal>, in which case <envar>NIX_BUILD_CORES</envar>
will be the total number of cores in the system.</para>
<para>The total number of consumed cores is a simple multiplication,
<xref linkend="conf-cores" /> * <envar>NIX_BUILD_CORES</envar>.</para>
<para>The balance on how to set these two independent variables depends
upon each builder's workload and hardware. Here are a few example
scenarios on a machine with 24 cores:</para>
<table>
<caption>Balancing 24 Build Cores</caption>
<thead>
<tr>
<th><xref linkend="conf-max-jobs" /></th>
<th><xref linkend="conf-cores" /></th>
<th><envar>NIX_BUILD_CORES</envar></th>
<th>Maximum Processes</th>
<th>Result</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>24</td>
<td>24</td>
<td>24</td>
<td>
One derivation will be built at a time, each one can use 24
cores. Undersold if a job can’t use 24 cores.
</td>
</tr>
<tr>
<td>4</td>
<td>6</td>
<td>6</td>
<td>24</td>
<td>
Four derivations will be built at once, each given access to
six cores.
</td>
</tr>
<tr>
<td>12</td>
<td>6</td>
<td>6</td>
<td>72</td>
<td>
12 derivations will be built at once, each given access to six
cores. This configuration is over-sold. If all 12 derivations
being built simultaneously try to use all six cores, the
machine's performance will be degraded due to extensive context
switching between the 12 builds.
</td>
</tr>
<tr>
<td>24</td>
<td>1</td>
<td>1</td>
<td>24</td>
<td>
24 derivations can build at the same time, each using a single
core. Never oversold, but derivations which require many cores
will be very slow to compile.
</td>
</tr>
<tr>
<td>24</td>
<td>0</td>
<td>24</td>
<td>576</td>
<td>
24 derivations can build at the same time, each using all the
available cores of the machine. Very likely to be oversold,
and very likely to suffer context switches.
</td>
</tr>
</tbody>
</table>
<para>It is up to the derivations' build script to respect
host's requested cores-per-build by following the value of the
<envar>NIX_BUILD_CORES</envar> environment variable.</para>
</chapter>

160
doc/manual/advanced-topics/post-build-hook.xml Normal file
View file

@ -0,0 +1,160 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="chap-post-build-hook"
version="5.0"
>
<title>Using the <xref linkend="conf-post-build-hook" /></title>
<subtitle>Uploading to an S3-compatible binary cache after each build</subtitle>
<section xml:id="chap-post-build-hook-caveats">
<title>Implementation Caveats</title>
<para>Here we use the post-build hook to upload to a binary cache.
This is a simple and working example, but it is not suitable for all
use cases.</para>
<para>The post build hook program runs after each executed build,
and blocks the build loop. The build loop exits if the hook program
fails.</para>
<para>Concretely, this implementation will make Nix slow or unusable
when the internet is slow or unreliable.</para>
<para>A more advanced implementation might pass the store paths to a
user-supplied daemon or queue for processing the store paths outside
of the build loop.</para>
</section>
<section>
<title>Prerequisites</title>
<para>
This tutorial assumes you have configured an S3-compatible binary cache
according to the instructions at
<xref linkend="ssec-s3-substituter-authenticated-writes" />, and
that the <literal>root</literal> user's default AWS profile can
upload to the bucket.
</para>
</section>
<section>
<title>Set up a Signing Key</title>
<para>Use <command>nix-store --generate-binary-cache-key</command> to
create our public and private signing keys. We will sign paths
with the private key, and distribute the public key for verifying
the authenticity of the paths.</para>
<screen>
# nix-store --generate-binary-cache-key example-nix-cache-1 /etc/nix/key.private /etc/nix/key.public
# cat /etc/nix/key.public
example-nix-cache-1:1/cKDz3QCCOmwcztD2eV6Coggp6rqc9DGjWv7C0G+rM=
</screen>
<para>Then, add the public key and the cache URL to your
<filename>nix.conf</filename>'s <xref linkend="conf-trusted-public-keys" />
and <xref linkend="conf-substituters" /> like:</para>
<programlisting>
substituters = https://cache.nixos.org/ s3://example-nix-cache
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= example-nix-cache-1:1/cKDz3QCCOmwcztD2eV6Coggp6rqc9DGjWv7C0G+rM=
</programlisting>
<para>we will restart the Nix daemon a later step.</para>
</section>
<section>
<title>Implementing the build hook</title>
<para>Write the following script to
<filename>/etc/nix/upload-to-cache.sh</filename>:
</para>
<programlisting>
#!/bin/sh
set -eu
set -f # disable globbing
export IFS=' '
echo "Signing paths" $OUT_PATHS
nix sign-paths --key-file /etc/nix/key.private $OUT_PATHS
echo "Uploading paths" $OUT_PATHS
exec nix copy --to 's3://example-nix-cache' $OUT_PATHS
</programlisting>
<note>
<title>Should <literal>$OUT_PATHS</literal> be quoted?</title>
<para>
The <literal>$OUT_PATHS</literal> variable is a space-separated
list of Nix store paths. In this case, we expect and want the
shell to perform word splitting to make each output path its
own argument to <command>nix sign-paths</command>. Nix guarantees
the paths will not contain any spaces, however a store path
might contain glob characters. The <command>set -f</command>
disables globbing in the shell.
</para>
</note>
<para>
Then make sure the hook program is executable by the <literal>root</literal> user:
<screen>
# chmod +x /etc/nix/upload-to-cache.sh
</screen></para>
</section>
<section>
<title>Updating Nix Configuration</title>
<para>Edit <filename>/etc/nix/nix.conf</filename> to run our hook,
by adding the following configuration snippet at the end:</para>
<programlisting>
post-build-hook = /etc/nix/upload-to-cache.sh
</programlisting>
<para>Then, restart the <command>nix-daemon</command>.</para>
</section>
<section>
<title>Testing</title>
<para>Build any derivation, for example:</para>
<screen>
$ nix-build -E '(import &lt;nixpkgs&gt; {}).writeText "example" (builtins.toString builtins.currentTime)'
these derivations will be built:
/nix/store/s4pnfbkalzy5qz57qs6yybna8wylkig6-example.drv
building '/nix/store/s4pnfbkalzy5qz57qs6yybna8wylkig6-example.drv'...
running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
post-build-hook: Signing paths /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
post-build-hook: Uploading paths /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
/nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
</screen>
<para>Then delete the path from the store, and try substituting it from the binary cache:</para>
<screen>
$ rm ./result
$ nix-store --delete /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
</screen>
<para>Now, copy the path back from the cache:</para>
<screen>
$ nix store --realize /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
copying path '/nix/store/m8bmqwrch6l3h8s0k3d673xpmipcdpsa-example from 's3://example-nix-cache'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/m8bmqwrch6l3h8s0k3d673xpmipcdpsa-example
</screen>
</section>
<section>
<title>Conclusion</title>
<para>
We now have a Nix installation configured to automatically sign and
upload every local build to a remote binary cache.
</para>
<para>
Before deploying this to production, be sure to consider the
implementation caveats in <xref linkend="chap-post-build-hook-caveats" />.
</para>
</section>
</chapter>

78
doc/manual/command-ref/conf-file.xml
View file

@ -238,8 +238,9 @@ false</literal>.</para>
linkend='opt-cores'>--cores</option> command line switch and
defaults to <literal>1</literal>. The value <literal>0</literal>
means that the builder should use all available CPU cores in the
system.</para></listitem>
system.</para>
<para>See also <xref linkend="chap-tuning-cores-and-jobs" />.</para></listitem>
</varlistentry>
<varlistentry xml:id="conf-diff-hook"><term><literal>diff-hook</literal></term>
@ -482,8 +483,10 @@ builtins.fetchurl {
<varlistentry xml:id="conf-max-free"><term><literal>max-free</literal></term>
<listitem><para>This option defines after how many free bytes to stop collecting
garbage once the <literal>min-free</literal> condition gets triggered.</para></listitem>
<listitem><para>When a garbage collection is triggered by the
<literal>min-free</literal> option, it stops as soon as
<literal>max-free</literal> bytes are available. The default is
infinity (i.e. delete all garbage).</para></listitem>
</varlistentry>
@ -498,7 +501,10 @@ builtins.fetchurl {
regardless). It can be
overridden using the <option
linkend='opt-max-jobs'>--max-jobs</option> (<option>-j</option>)
command line switch.</para></listitem>
command line switch.</para>
<para>See also <xref linkend="chap-tuning-cores-and-jobs" />.</para>
</listitem>
</varlistentry>
<varlistentry xml:id="conf-max-silent-time"><term><literal>max-silent-time</literal></term>
@ -524,9 +530,11 @@ builtins.fetchurl {
<varlistentry xml:id="conf-min-free"><term><literal>min-free</literal></term>
<listitem>
<para>When the disk reaches <literal>min-free</literal> bytes of free disk space during a build, nix
will start to garbage-collection until <literal>max-free</literal> bytes are available on the disk.
A value of <literal>0</literal> (the default) means that this feature is disabled.</para>
<para>When free disk space in <filename>/nix/store</filename>
drops below <literal>min-free</literal> during a build, Nix
performs a garbage-collection until <literal>max-free</literal>
bytes are available or there is no more garbage. A value of
<literal>0</literal> (the default) disables this feature.</para>
</listitem>
</varlistentry>
@ -656,6 +664,62 @@ password <replaceable>my-password</replaceable>
</varlistentry>
<varlistentry xml:id="conf-post-build-hook">
<term><literal>post-build-hook</literal></term>
<listitem>
<para>Optional. The path to a program to execute after each build.</para>
<para>This option is only settable in the global
<filename>nix.conf</filename>, or on the command line by trusted
users.</para>
<para>When using the nix-daemon, the daemon executes the hook as
<literal>root</literal>. If the nix-daemon is not involved, the
hook runs as the user executing the nix-build.</para>
<itemizedlist>
<listitem><para>The hook executes after an evaluation-time build.</para></listitem>
<listitem><para>The hook does not execute on substituted paths.</para></listitem>
<listitem><para>The hook's output always goes to the user's terminal.</para></listitem>
<listitem><para>If the hook fails, the build succeeds but no further builds execute.</para></listitem>
<listitem><para>The hook executes synchronously, and blocks other builds from progressing while it runs.</para></listitem>
</itemizedlist>
<para>The program executes with no arguments. The program's environment
contains the following environment variables:</para>
<variablelist>
<varlistentry>
<term><envar>DRV_PATH</envar></term>
<listitem>
<para>The derivation for the built paths.</para>
<para>Example:
<literal>/nix/store/5nihn1a7pa8b25l9zafqaqibznlvvp3f-bash-4.4-p23.drv</literal>
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><envar>OUT_PATHS</envar></term>
<listitem>
<para>Output paths of the built derivation, separated by a space character.</para>
<para>Example:
<literal>/nix/store/zf5lbh336mnzf1nlswdn11g4n2m8zh3g-bash-4.4-p23-dev
/nix/store/rjxwxwv1fpn9wa2x5ssk5phzwlcv4mna-bash-4.4-p23-doc
/nix/store/6bqvbzjkcp9695dq0dpl5y43nvy37pq1-bash-4.4-p23-info
/nix/store/r7fng3kk3vlpdlh2idnrbn37vh4imlj2-bash-4.4-p23-man
/nix/store/xfghy8ixrhz3kyy6p724iv3cxji088dx-bash-4.4-p23</literal>.
</para>
</listitem>
</varlistentry>
</variablelist>
<para>See <xref linkend="chap-post-build-hook" /> for an example
implementation.</para>
</listitem>
</varlistentry>
<varlistentry xml:id="conf-repeat"><term><literal>repeat</literal></term>
<listitem><para>How many times to repeat builds to check whether

2
scripts/install-nix-from-closure.sh
View file

@ -12,7 +12,7 @@ if ! [ -e "$self/.reginfo" ]; then
echo "$0: incomplete installer (.reginfo is missing)" >&2
fi
if [ -z "$USER" ]; then
if [ -z "$USER" ] && ! USER=$(id -u -n); then
echo "$0: \$USER is not set" >&2
exit 1
fi

10
src/libexpr/primops.cc
View file

@ -832,8 +832,14 @@ static void prim_pathExists(EvalState & state, const Pos & pos, Value * * args,
{
PathSet context;
Path path = state.coerceToPath(pos, *args[0], context);
if (!context.empty())
throw EvalError(format("string '%1%' cannot refer to other paths, at %2%") % path % pos);
try {
state.realiseContext(context);
} catch (InvalidPathError & e) {
throw EvalError(format(
"cannot check the existence of '%1%', since path '%2%' is not valid, at %3%")
% path % e.path % pos);
}
try {
mkBool(v, pathExists(state.checkSourcePath(path)));
} catch (SysError & e) {

80
src/libstore/build.cc
View file

@ -1629,6 +1629,61 @@ void DerivationGoal::buildDone()
being valid. */
registerOutputs();
if (settings.postBuildHook != "") {
Activity act(*logger, lvlInfo, actPostBuildHook,
fmt("running post-build-hook '%s'", settings.postBuildHook),
Logger::Fields{drvPath});
PushActivity pact(act.id);
auto outputPaths = drv->outputPaths();
std::map<std::string, std::string> hookEnvironment = getEnv();
hookEnvironment.emplace("DRV_PATH", drvPath);
hookEnvironment.emplace("OUT_PATHS", chomp(concatStringsSep(" ", outputPaths)));
RunOptions opts(settings.postBuildHook, {});
opts.environment = hookEnvironment;
struct LogSink : Sink {
Activity & act;
std::string currentLine;
LogSink(Activity & act) : act(act) { }
void operator() (const unsigned char * data, size_t len) override {
for (size_t i = 0; i < len; i++) {
auto c = data[i];
if (c == '\n') {
flushLine();
} else {
currentLine += c;
}
}
}
void flushLine() {
if (settings.verboseBuild) {
printError("post-build-hook: " + currentLine);
} else {
act.result(resPostBuildLogLine, currentLine);
}
currentLine.clear();
}
~LogSink() {
if (currentLine != "") {
currentLine += '\n';
flushLine();
}
}
};
LogSink sink(act);
opts.standardOut = &sink;
opts.mergeStderrToStdout = true;
runProgram2(opts);
}
if (buildMode == bmCheck) {
done(BuildResult::Built);
return;
@ -2734,7 +2789,13 @@ void DerivationGoal::runChild()
on. */
if (fixedOutput) {
ss.push_back("/etc/resolv.conf");
ss.push_back("/etc/nsswitch.conf");
// Only use nss functions to resolve hosts and
// services. Don’t use it for anything else that may
// be configured for this system. This limits the
// potential impurities introduced in fixed outputs.
writeFile(chrootRootDir + "/etc/nsswitch.conf", "hosts: files dns\nservices: files\n");
ss.push_back("/etc/services");
ss.push_back("/etc/hosts");
if (pathExists("/var/run/nscd/socket"))
@ -3978,17 +4039,6 @@ void SubstitutionGoal::tryToRun()
return;
}
/* If the store path is already locked (probably by a
DerivationGoal), then put this goal to sleep. Note: we don't
acquire a lock here since that breaks addToStore(), so below we
handle an AlreadyLocked exception from addToStore(). The check
here is just an optimisation to prevent having to redo a
download due to a locked path. */
if (pathIsLockedByMe(worker.store.toRealPath(storePath))) {
worker.waitForAWhile(shared_from_this());
return;
}
maintainRunningSubstitutions = std::make_unique<MaintainCount<uint64_t>>(worker.runningSubstitutions);
worker.updateProgress();
@ -4028,12 +4078,6 @@ void SubstitutionGoal::finished()
try {
promise.get_future().get();
} catch (AlreadyLocked & e) {
/* Probably a DerivationGoal is already building this store
path. Sleep for a while and try again. */
state = &SubstitutionGoal::init;
worker.waitForAWhile(shared_from_this());
return;
} catch (std::exception & e) {
printError(e.what());

70
src/libstore/gc.cc
View file

@ -29,7 +29,7 @@ static string gcRootsDir = "gcroots";
read. To be precise: when they try to create a new temporary root
file, they will block until the garbage collector has finished /
yielded the GC lock. */
int LocalStore::openGCLock(LockType lockType)
AutoCloseFD LocalStore::openGCLock(LockType lockType)
{
Path fnGCLock = (format("%1%/%2%")
% stateDir % gcLockName).str();
@ -49,7 +49,7 @@ int LocalStore::openGCLock(LockType lockType)
process that can open the file for reading can DoS the
collector. */
return fdGCLock.release();
return fdGCLock;
}
@ -221,26 +221,22 @@ void LocalStore::findTempRoots(FDs & fds, Roots & tempRoots, bool censor)
//FDPtr fd(new AutoCloseFD(openLockFile(path, false)));
//if (*fd == -1) continue;
if (path != fnTempRoots) {
/* Try to acquire a write lock without blocking. This can
only succeed if the owning process has died. In that case
we don't care about its temporary roots. */
if (lockFile(fd->get(), ltWrite, false)) {
printError(format("removing stale temporary roots file '%1%'") % path);
unlink(path.c_str());
writeFull(fd->get(), "d");
continue;
}
/* Acquire a read lock. This will prevent the owning process
from upgrading to a write lock, therefore it will block in
addTempRoot(). */
debug(format("waiting for read lock on '%1%'") % path);
lockFile(fd->get(), ltRead, true);
/* Try to acquire a write lock without blocking. This can
only succeed if the owning process has died. In that case
we don't care about its temporary roots. */
if (lockFile(fd->get(), ltWrite, false)) {
printError(format("removing stale temporary roots file '%1%'") % path);
unlink(path.c_str());
writeFull(fd->get(), "d");
continue;
}
/* Acquire a read lock. This will prevent the owning process
from upgrading to a write lock, therefore it will block in
addTempRoot(). */
debug(format("waiting for read lock on '%1%'") % path);
lockFile(fd->get(), ltRead, true);
/* Read the entire file. */
string contents = readFile(fd->get());
@ -444,17 +440,22 @@ void LocalStore::findRuntimeRoots(Roots & roots, bool censor)
}
#if !defined(__linux__)
try {
std::regex lsofRegex(R"(^n(/.*)$)");
auto lsofLines =
tokenizeString<std::vector<string>>(runProgram(LSOF, true, { "-n", "-w", "-F", "n" }), "\n");
for (const auto & line : lsofLines) {
std::smatch match;
if (std::regex_match(line, match, lsofRegex))
unchecked[match[1]].emplace("{lsof}");
// lsof is really slow on OS X. This actually causes the gc-concurrent.sh test to fail.
// See: https://github.com/NixOS/nix/issues/3011
// Because of this we disable lsof when running the tests.
if (getEnv("_NIX_TEST_NO_LSOF") == "") {
try {
std::regex lsofRegex(R"(^n(/.*)$)");
auto lsofLines =
tokenizeString<std::vector<string>>(runProgram(LSOF, true, { "-n", "-w", "-F", "n" }), "\n");
for (const auto & line : lsofLines) {
std::smatch match;
if (std::regex_match(line, match, lsofRegex))
unchecked[match[1]].emplace("{lsof}");
}
} catch (ExecError & e) {
/* lsof not installed, lsof failed */
}
} catch (ExecError & e) {
/* lsof not installed, lsof failed */
}
#endif
@ -866,7 +867,12 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
void LocalStore::autoGC(bool sync)
{
auto getAvail = [this]() {
static auto fakeFreeSpaceFile = getEnv("_NIX_TEST_FREE_SPACE_FILE", "");
auto getAvail = [this]() -> uint64_t {
if (!fakeFreeSpaceFile.empty())
return std::stoll(readFile(fakeFreeSpaceFile));
struct statvfs st;
if (statvfs(realStoreDir.c_str(), &st))
throw SysError("getting filesystem info about '%s'", realStoreDir);
@ -887,7 +893,7 @@ void LocalStore::autoGC(bool sync)
auto now = std::chrono::steady_clock::now();
if (now < state->lastGCCheck + std::chrono::seconds(5)) return;
if (now < state->lastGCCheck + std::chrono::seconds(settings.minFreeCheckInterval)) return;
auto avail = getAvail();

6
src/libstore/globals.hh
View file

@ -315,6 +315,9 @@ public:
"pre-build-hook",
"A program to run just before a build to set derivation-specific build settings."};
Setting<std::string> postBuildHook{this, "", "post-build-hook",
"A program to run just after each succesful build."};
Setting<std::string> netrcFile{this, fmt("%s/%s", nixConfDir, "netrc"), "netrc-file",
"Path to the netrc file used to obtain usernames/passwords for downloads."};
@ -342,6 +345,9 @@ public:
Setting<uint64_t> maxFree{this, std::numeric_limits<uint64_t>::max(), "max-free",
"Stop deleting garbage when free disk space is above the specified amount."};
Setting<uint64_t> minFreeCheckInterval{this, 5, "min-free-check-interval",
"Number of seconds between checking free disk space."};
Setting<Paths> pluginFiles{this, {}, "plugin-files",
"Plugins to dynamically load at nix initialization time."};

9
src/libstore/local-store.cc
View file

@ -1203,7 +1203,8 @@ bool LocalStore::verifyStore(bool checkContents, RepairFlag repair)
bool errors = false;
/* Acquire the global GC lock to prevent a garbage collection. */
/* Acquire the global GC lock to get a consistent snapshot of
existing and valid paths. */
AutoCloseFD fdGCLock = openGCLock(ltWrite);
PathSet store;
@ -1214,13 +1215,11 @@ bool LocalStore::verifyStore(bool checkContents, RepairFlag repair)
PathSet validPaths2 = queryAllValidPaths(), validPaths, done;
fdGCLock = -1;
for (auto & i : validPaths2)
verifyPath(i, store, done, validPaths, repair, errors);
/* Release the GC lock so that checking content hashes (which can
take ages) doesn't block the GC or builds. */
fdGCLock = -1;
/* Optionally, check the content hashes (slow). */
if (checkContents) {
printInfo("checking hashes...");

2
src/libstore/local-store.hh
View file

@ -263,7 +263,7 @@ private:
bool isActiveTempFile(const GCState & state,
const Path & path, const string & suffix);
int openGCLock(LockType lockType);
AutoCloseFD openGCLock(LockType lockType);
void findRoots(const Path & path, unsigned char type, Roots & roots);

126
src/libstore/pathlocks.cc
View file

@ -5,9 +5,10 @@
#include <cerrno>
#include <cstdlib>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/file.h>
namespace nix {
@ -40,17 +41,14 @@ void deleteLockFile(const Path & path, int fd)
bool lockFile(int fd, LockType lockType, bool wait)
{
struct flock lock;
if (lockType == ltRead) lock.l_type = F_RDLCK;
else if (lockType == ltWrite) lock.l_type = F_WRLCK;
else if (lockType == ltNone) lock.l_type = F_UNLCK;
int type;
if (lockType == ltRead) type = LOCK_SH;
else if (lockType == ltWrite) type = LOCK_EX;
else if (lockType == ltNone) type = LOCK_UN;
else abort();
lock.l_whence = SEEK_SET;
lock.l_start = 0;
lock.l_len = 0; /* entire file */
if (wait) {
while (fcntl(fd, F_SETLKW, &lock) != 0) {
while (flock(fd, type) != 0) {
checkInterrupt();
if (errno != EINTR)
throw SysError(format("acquiring/releasing lock"));
@ -58,9 +56,9 @@ bool lockFile(int fd, LockType lockType, bool wait)
return false;
}
} else {
while (fcntl(fd, F_SETLK, &lock) != 0) {
while (flock(fd, type | LOCK_NB) != 0) {
checkInterrupt();
if (errno == EACCES || errno == EAGAIN) return false;
if (errno == EWOULDBLOCK) return false;
if (errno != EINTR)
throw SysError(format("acquiring/releasing lock"));
}
@ -70,14 +68,6 @@ bool lockFile(int fd, LockType lockType, bool wait)
}
/* This enables us to check whether are not already holding a lock on
a file ourselves. POSIX locks (fcntl) suck in this respect: if we
close a descriptor, the previous lock will be closed as well. And
there is no way to query whether we already have a lock (F_GETLK
only works on locks held by other processes). */
static Sync<StringSet> lockedPaths_;
PathLocks::PathLocks()
: deletePaths(false)
{
@ -91,7 +81,7 @@ PathLocks::PathLocks(const PathSet & paths, const string & waitMsg)
}
bool PathLocks::lockPaths(const PathSet & _paths,
bool PathLocks::lockPaths(const PathSet & paths,
const string & waitMsg, bool wait)
{
assert(fds.empty());
@ -99,75 +89,54 @@ bool PathLocks::lockPaths(const PathSet & _paths,
/* Note that `fds' is built incrementally so that the destructor
will only release those locks that we have already acquired. */
/* Sort the paths. This assures that locks are always acquired in
the same order, thus preventing deadlocks. */
Paths paths(_paths.begin(), _paths.end());
paths.sort();
/* Acquire the lock for each path. */
/* Acquire the lock for each path in sorted order. This ensures
that locks are always acquired in the same order, thus
preventing deadlocks. */
for (auto & path : paths) {
checkInterrupt();
Path lockPath = path + ".lock";
debug(format("locking path '%1%'") % path);
{
auto lockedPaths(lockedPaths_.lock());
if (lockedPaths->count(lockPath)) {
if (!wait) return false;
throw AlreadyLocked("deadlock: trying to re-acquire self-held lock '%s'", lockPath);
}
lockedPaths->insert(lockPath);
}
AutoCloseFD fd;
try {
while (1) {
AutoCloseFD fd;
/* Open/create the lock file. */
fd = openLockFile(lockPath, true);
while (1) {
/* Open/create the lock file. */
fd = openLockFile(lockPath, true);
/* Acquire an exclusive lock. */
if (!lockFile(fd.get(), ltWrite, false)) {
if (wait) {
if (waitMsg != "") printError(waitMsg);
lockFile(fd.get(), ltWrite, true);
} else {
/* Failed to lock this path; release all other
locks. */
unlock();
lockedPaths_.lock()->erase(lockPath);
return false;
}
/* Acquire an exclusive lock. */
if (!lockFile(fd.get(), ltWrite, false)) {
if (wait) {
if (waitMsg != "") printError(waitMsg);
lockFile(fd.get(), ltWrite, true);
} else {
/* Failed to lock this path; release all other
locks. */
unlock();
return false;
}
debug(format("lock acquired on '%1%'") % lockPath);
/* Check that the lock file hasn't become stale (i.e.,
hasn't been unlinked). */
struct stat st;
if (fstat(fd.get(), &st) == -1)
throw SysError(format("statting lock file '%1%'") % lockPath);
if (st.st_size != 0)
/* This lock file has been unlinked, so we're holding
a lock on a deleted file. This means that other
processes may create and acquire a lock on
`lockPath', and proceed. So we must retry. */
debug(format("open lock file '%1%' has become stale") % lockPath);
else
break;
}
/* Use borrow so that the descriptor isn't closed. */
fds.push_back(FDPair(fd.release(), lockPath));
debug(format("lock acquired on '%1%'") % lockPath);
} catch (...) {
lockedPaths_.lock()->erase(lockPath);
throw;
/* Check that the lock file hasn't become stale (i.e.,
hasn't been unlinked). */
struct stat st;
if (fstat(fd.get(), &st) == -1)
throw SysError(format("statting lock file '%1%'") % lockPath);
if (st.st_size != 0)
/* This lock file has been unlinked, so we're holding
a lock on a deleted file. This means that other
processes may create and acquire a lock on
`lockPath', and proceed. So we must retry. */
debug(format("open lock file '%1%' has become stale") % lockPath);
else
break;
}
/* Use borrow so that the descriptor isn't closed. */
fds.push_back(FDPair(fd.release(), lockPath));
}
return true;
@ -189,8 +158,6 @@ void PathLocks::unlock()
for (auto & i : fds) {
if (deletePaths) deleteLockFile(i.second, i.first);
lockedPaths_.lock()->erase(i.second);
if (close(i.first) == -1)
printError(
format("error (ignored): cannot close lock file on '%1%'") % i.second);
@ -208,11 +175,4 @@ void PathLocks::setDeletion(bool deletePaths)
}
bool pathIsLockedByMe(const Path & path)
{
Path lockPath = path + ".lock";
return lockedPaths_.lock()->count(lockPath);
}
}

4
src/libstore/pathlocks.hh
View file

@ -16,8 +16,6 @@ enum LockType { ltRead, ltWrite, ltNone };
bool lockFile(int fd, LockType lockType, bool wait);
MakeError(AlreadyLocked, Error);
class PathLocks
{
private:
@ -37,6 +35,4 @@ public:
void setDeletion(bool deletePaths);
};
bool pathIsLockedByMe(const Path & path);
}

2
src/libutil/logging.hh
View file

@ -26,6 +26,7 @@ typedef enum {
actVerifyPaths = 107,
actSubstitute = 108,
actQueryPathInfo = 109,
actPostBuildHook = 110,
} ActivityType;
typedef enum {
@ -36,6 +37,7 @@ typedef enum {
resSetPhase = 104,
resProgress = 105,
resSetExpected = 106,
resPostBuildLogLine = 107,
} ResultType;
typedef uint64_t ActivityId;

23
src/libutil/util.cc
View file

@ -85,6 +85,15 @@ void clearEnv()
unsetenv(name.first.c_str());
}
void replaceEnv(std::map<std::string, std::string> newEnv)
{
clearEnv();
for (auto newEnvVar : newEnv)
{
setenv(newEnvVar.first.c_str(), newEnvVar.second.c_str(), 1);
}
}
Path absPath(Path path, Path dir)
{
@ -1044,10 +1053,22 @@ void runProgram2(const RunOptions & options)
if (options.standardOut) out.create();
if (source) in.create();
ProcessOptions processOptions;
// vfork implies that the environment of the main process and the fork will
// be shared (technically this is undefined, but in practice that's the
// case), so we can't use it if we alter the environment
if (options.environment)
processOptions.allowVfork = false;
/* Fork. */
Pid pid = startProcess([&]() {
if (options.environment)
replaceEnv(*options.environment);
if (options.standardOut && dup2(out.writeSide.get(), STDOUT_FILENO) == -1)
throw SysError("dupping stdout");
if (options.mergeStderrToStdout)
if (dup2(STDOUT_FILENO, STDERR_FILENO) == -1)
throw SysError("cannot dup stdout into stderr");
if (source && dup2(in.readSide.get(), STDIN_FILENO) == -1)
throw SysError("dupping stdin");
@ -1074,7 +1095,7 @@ void runProgram2(const RunOptions & options)
execv(options.program.c_str(), stringsToCharPtrs(args_).data());
throw SysError("executing '%1%'", options.program);
});
}, processOptions);
out.writeSide = -1;

2
src/libutil/util.hh
View file

@ -276,12 +276,14 @@ struct RunOptions
std::optional<uid_t> uid;
std::optional<uid_t> gid;
std::optional<Path> chdir;
std::optional<std::map<std::string, std::string>> environment;
Path program;
bool searchPath = true;
Strings args;
std::optional<std::string> input;
Source * standardIn = nullptr;
Sink * standardOut = nullptr;
bool mergeStderrToStdout = false;
bool _killStderr = false;
RunOptions(const Path & program, const Strings & args)

15
src/nix-env/nix-env.cc
View file

@ -860,7 +860,10 @@ static void queryJSON(Globals & globals, vector<DrvInfo> & elems)
for (auto & i : elems) {
JSONObject pkgObj = topObj.object(i.attrPath);
pkgObj.attr("name", i.queryName());
auto drvName = DrvName(i.queryName());
pkgObj.attr("name", drvName.fullName);
pkgObj.attr("pname", drvName.name);
pkgObj.attr("version", drvName.version);
pkgObj.attr("system", i.querySystem());
JSONObject metaObj = pkgObj.object("meta");
@ -1026,10 +1029,14 @@ static void opQuery(Globals & globals, Strings opFlags, Strings opArgs)
else if (printAttrPath)
columns.push_back(i.attrPath);
if (xmlOutput)
attrs["name"] = i.queryName();
else if (printName)
if (xmlOutput) {
auto drvName = DrvName(i.queryName());
attrs["name"] = drvName.fullName;
attrs["pname"] = drvName.name;
attrs["version"] = drvName.version;
} else if (printName) {
columns.push_back(i.queryName());
}
if (compareVersions) {
/* Compare this element against the versions of the

16
src/nix/progress-bar.cc
View file

@ -170,6 +170,14 @@ public:
name, sub);
}
if (type == actPostBuildHook) {
auto name = storePathToName(getS(fields, 0));
if (hasSuffix(name, ".drv"))
name.resize(name.size() - 4);
i->s = fmt("post-build " ANSI_BOLD "%s" ANSI_NORMAL, name);
i->name = DrvName(name).name;
}
if (type == actQueryPathInfo) {
auto name = storePathToName(getS(fields, 0));
i->s = fmt("querying " ANSI_BOLD "%s" ANSI_NORMAL " on %s", name, getS(fields, 1));
@ -228,14 +236,18 @@ public:
update(*state);
}
else if (type == resBuildLogLine) {
else if (type == resBuildLogLine || type == resPostBuildLogLine) {
auto lastLine = trim(getS(fields, 0));
if (!lastLine.empty()) {
auto i = state->its.find(act);
assert(i != state->its.end());
ActInfo info = *i->second;
if (printBuildLogs) {
log(*state, lvlInfo, ANSI_FAINT + info.name.value_or("unnamed") + "> " + ANSI_NORMAL + lastLine);
auto suffix = "> ";
if (type == resPostBuildLogLine) {
suffix = " (post)> ";
}
log(*state, lvlInfo, ANSI_FAINT + info.name.value_or("unnamed") + suffix + ANSI_NORMAL + lastLine);
} else {
state->activities.erase(i->second);
info.lastLine = lastLine;

1
tests/common.sh.in
View file

@ -16,6 +16,7 @@ if [[ -n $NIX_STORE ]]; then
export _NIX_TEST_NO_SANDBOX=1
fi
export _NIX_IN_TEST=$TEST_ROOT/shared
export _NIX_TEST_NO_LSOF=1
export NIX_REMOTE=$NIX_REMOTE_
unset NIX_PATH
export TEST_HOME=$TEST_ROOT/test-home

1
tests/dependencies.nix
View file

@ -17,6 +17,7 @@ let {
builder = ./dependencies.builder0.sh + "/FOOBAR/../.";
input1 = input1 + "/.";
input2 = "${input2}/.";
input1_drv = input1;
meta.description = "Random test package";
};

59
tests/gc-auto.sh Normal file
View file

@ -0,0 +1,59 @@
source common.sh
clearStore
garbage1=$(nix add-to-store --name garbage1 ./tarball.sh)
garbage2=$(nix add-to-store --name garbage2 ./tarball.sh)
garbage3=$(nix add-to-store --name garbage3 ./tarball.sh)
fake_free=$TEST_ROOT/fake-free
export _NIX_TEST_FREE_SPACE_FILE=$fake_free
echo 1100 > $fake_free
expr=$(cat <<EOF
with import ./config.nix; mkDerivation {
name = "gc-A";
buildCommand = ''
[[ \$(ls \$NIX_STORE/*-garbage? | wc -l) = 3 ]]
mkdir \$out
echo foo > \$out/bar
echo 1...
sleep 2
echo 100 > $fake_free
echo 2...
sleep 2
echo 3...
[[ \$(ls \$NIX_STORE/*-garbage? | wc -l) = 1 ]]
'';
}
EOF
)
nix build -o $TEST_ROOT/result-A -L "($expr)" \
--min-free 1000 --max-free 2000 --min-free-check-interval 1 &
pid=$!
expr2=$(cat <<EOF
with import ./config.nix; mkDerivation {
name = "gc-B";
buildCommand = ''
mkdir \$out
echo foo > \$out/bar
echo 1...
sleep 2
echo 100 > $fake_free
echo 2...
sleep 2
echo 3...
'';
}
EOF
)
nix build -o $TEST_ROOT/result-B -L "($expr2)" \
--min-free 1000 --max-free 2000 --min-free-check-interval 1
wait "$pid"
[[ foo = $(cat $TEST_ROOT/result-A/bar) ]]
[[ foo = $(cat $TEST_ROOT/result-B/bar) ]]

5
tests/import-derivation.nix
View file

@ -10,7 +10,10 @@ let
'';
};
value = import bar;
value =
# Test that pathExists can check the existence of /nix/store paths
assert builtins.pathExists bar;
import bar;
in

2
tests/install-darwin.sh
View file

@ -34,7 +34,7 @@ cleanup() {
sudo rm -rf /etc/nix \
/nix \
/var/root/.nix-profile /var/root/.nix-defexpr /var/root/.nix-channels \
"$USER/.nix-profile" "$USER/.nix-defexpr" "$USER/.nix-channels"
"$HOME/.nix-profile" "$HOME/.nix-defexpr" "$HOME/.nix-channels"
}
verify() {

5
tests/local.mk
View file

@ -3,7 +3,9 @@ check:
nix_tests = \
init.sh hash.sh lang.sh add.sh simple.sh dependencies.sh \
gc.sh gc-concurrent.sh \
gc.sh \
gc-concurrent.sh \
gc-auto.sh \
referrers.sh user-envs.sh logging.sh nix-build.sh misc.sh fixed.sh \
gc-runtime.sh check-refs.sh filter-source.sh \
remote-store.sh export.sh export-graph.sh \
@ -27,6 +29,7 @@ nix_tests = \
plugins.sh \
search.sh \
nix-copy-ssh.sh \
post-hook.sh \
flakes.sh
# parallel.sh

15
tests/post-hook.sh Normal file
View file

@ -0,0 +1,15 @@
source common.sh
clearStore
export REMOTE_STORE=$TEST_ROOT/remote_store
# Build the dependencies and push them to the remote store
nix-build -o $TEST_ROOT/result dependencies.nix --post-build-hook $PWD/push-to-store.sh
clearStore
# Ensure that we the remote store contains both the runtime and buildtime
# closure of what we've just built
nix copy --from "$REMOTE_STORE" --no-require-sigs -f dependencies.nix
nix copy --from "$REMOTE_STORE" --no-require-sigs -f dependencies.nix input1_drv

4
tests/push-to-store.sh Executable file
View file

@ -0,0 +1,4 @@
#!/bin/sh
echo Pushing "$@" to "$REMOTE_STORE"
echo -n "$OUT_PATHS" | xargs -d: nix copy --to "$REMOTE_STORE" --no-require-sigs