* Build hook documentation.

* nix-store options.
This commit is contained in:
Eelco Dolstra 2005-04-07 09:36:35 +00:00
parent 806b91f104
commit f1ae10b992
5 changed files with 320 additions and 261 deletions

View file

@ -65,7 +65,10 @@ will call whenever it wants to build a derivation. The build hook
will perform it in the usual way if possible, or it can accept it, in will perform it in the usual way if possible, or it can accept it, in
which case it is responsible for somehow getting the inputs of the which case it is responsible for somehow getting the inputs of the
build to another machine, doing the build there, and getting the build to another machine, doing the build there, and getting the
results back.</para> results back. The details of the build hook protocol are described in
the documentation of the <link
linkend="envar-build-hook"><envar>NIX_BUILD_HOOK</envar>
variable</link>.</para>
<example id='ex-remote-systems'><title>Remote machine configuration: <example id='ex-remote-systems'><title>Remote machine configuration:
<filename>remote-systems.conf</filename></title> <filename>remote-systems.conf</filename></title>

View file

@ -118,6 +118,158 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
</varlistentry> </varlistentry>
<varlistentry id="envar-build-hook"><term><envar>NIX_BUILD_HOOK</envar></term>
<listitem>
<para>Specifies the location of the <emphasis>build hook</emphasis>,
which is a program (typically some script) that Nix will call
whenever it wants to build a derivation. This is used to implement
distributed builds (see <xref linkend="sec-distributed-builds"
/>). The protocol by which the calling Nix process and the build
hook communicate is as follows.</para>
<para>The build hook is called with the following command-line
arguments:
<orderedlist>
<listitem><para>A boolean value <literal>0</literal> or
<literal>1</literal> specifying whether Nix can locally execute
more builds, as per the <link
linkend="opt-max-jobs"><option>--max-jobs</option> option</link>.
The purpose of this argument is to allow the hook to not have to
maintain bookkeeping for the local machine.</para></listitem>
<listitem><para>The Nix platform identifier for the local machine
(e.g., <literal>i686-linux</literal>).</para></listitem>
<listitem><para>The Nix platform identifier for the derivation,
i.e., its <link linkend="attr-system"><varname>system</varname>
attribute</link>.</para></listitem>
<listitem><para>The store path of the derivation.</para></listitem>
</orderedlist>
</para>
<para>On the basis of this information, and whatever persistent
state the build hook keeps about other machines and their current
load, it has to decide what to do with the build. It should print
out on file descriptor 3 one of the following responses (terminated
by a newline, <literal>"\n"</literal>):
<variablelist>
<varlistentry><term><literal>decline</literal></term>
<listitem><para>The build hook is not willing or able to perform
the build; the calling Nix process should do the build itself,
if possible.</para></listitem>
</varlistentry>
<varlistentry><term><literal>postpone</literal></term>
<listitem><para>The build hook cannot perform the build now, but
can do so in the future (e.g., because all available build slots
on remote machines are in use). The calling Nix process should
postpone this build until at least one currently running build
has terminated.</para></listitem>
</varlistentry>
<varlistentry><term><literal>accept</literal></term>
<listitem><para>The build hook has accepted the
build.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>If the build hook accepts the build, it is possible that it is
no longer necessary to do the build because some other process has
performed the build in the meantime. To prevent races, the hook
must read from file descriptor 4 a single line that tells it whether
to continue:
<variablelist>
<varlistentry><term><literal>cancel</literal></term>
<listitem><para>The build has already been done, so the hook
should exit.</para></listitem>
</varlistentry>
<varlistentry><term><literal>okay</literal></term>
<listitem><para>The hook should proceed with the build. At this
point, the calling Nix process has acquired locks on the output
path, so no other Nix process will perform the
build.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>If the hook has been told to proceed, Nix will store in the
hooks current directory a number of text files that contain
information about the derivation:
<variablelist>
<varlistentry><term><filename>inputs</filename></term>
<listitem><para>The set of store paths that are inputs to the
build process (one per line). These have to be copied
<emphasis>to</emphasis> the remote machine (in addition to the
store derivation itself).</para></listitem>
</varlistentry>
<varlistentry><term><filename>outputs</filename></term>
<listitem><para>The set of store paths that are outputs of the
derivation (one per line). These have to be copied
<emphasis>from</emphasis> the remote machine if the build
succeeds.</para></listitem>
</varlistentry>
<varlistentry><term><filename>references</filename></term>
<listitem><para>The reference graph of the inputs, in the format
accepted by the command <link
linkend="rsec-nix-store-reg-val"><command>nix-store
--register-validity</command></link>. It is necessary to run
this command on the remote machine after copying the inputs to
inform Nix on the remote machine that the inputs are valid
paths.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>The hook should copy the inputs to the remote machine,
register the validity of the inputs, perform the remote build, and
copy the outputs back to the local machine. An exit code other than
<literal>0</literal> indicates that the hook has failed.</para>
</listitem>
</varlistentry>
</variablelist> </variablelist>
</sect1> </sect1>

View file

@ -262,11 +262,10 @@ $ nix-store --gc</screen>
<!--######################################################################--> <!--######################################################################-->
<refsection> <refsection><title>Operation <option>--query</option></title>
<title>Operation <option>--query</option></title>
<refsection><title>Synopsis</title>
<refsection>
<title>Synopsis</title>
<cmdsynopsis> <cmdsynopsis>
<command>nix-store</command> <command>nix-store</command>
<group choice='req'> <group choice='req'>
@ -287,55 +286,38 @@ $ nix-store --gc</screen>
<arg><option>-f</option></arg> <arg><option>-f</option></arg>
<arg choice='plain' rep='repeat'><replaceable>args</replaceable></arg> <arg choice='plain' rep='repeat'><replaceable>args</replaceable></arg>
</cmdsynopsis> </cmdsynopsis>
</refsection>
<refsection>
<title>Description</title>
<para>
The operation <option>--query</option> displays various bits
of information about store expressions or store paths. The
queries are described below. At most one query can be
specified. The default query is <option>--list</option>.
</para>
</refsection> </refsection>
<refsection> <refsection><title>Description</title>
<title>Common query options</title>
<para>The operation <option>--query</option> displays various bits of
information about store paths. The queries are described below. At
most one query can be specified. The default query is
<option>--list</option>.</para>
</refsection>
<refsection><title>Common query options</title>
<variablelist> <variablelist>
<varlistentry> <varlistentry><term><option>--use-output</option> / <option>-u</option></term>
<term><option>--normalise</option> / <option>-n</option></term>
<listitem> <listitem><para>For each argument to the query that is a store
<para> derivation, apply the query to the output path of the derivation
For those queries that take a Nix store expression, this instead.</para></listitem>
option causes those expressions to be normalised first.
</para>
</listitem>
</varlistentry> </varlistentry>
<varlistentry> <varlistentry><term><option>--force-realise</option> / <option>-f</option></term>
<term><option>--force-realise</option> / <option>-f</option></term>
<listitem> <listitem><para>Realise each argument to the query first (see
<para> <link linkend="rsec-nix-store-realise"><command>nix-store
For those queries that take a Nix store expression, this --realise</command></link>).</para></listitem>
option causes those expressions to be realised first.
This is just a short-cut for the common idiom
</para>
<screen>
nix-store --realise /nix/store/bla.store
x=`nix-store --query --normalise /nix/store/bla.store`
<emphasis>(do something with the path $x</emphasis></screen>
<para>
which using this flag can be written as
</para>
<screen>
x=`nix-store --query --normalise --force-realise /nix/store/bla.store`
<emphasis>(do something with the path $x</emphasis></screen>
</listitem>
</varlistentry> </varlistentry>
</variablelist> </variablelist>
@ -343,173 +325,103 @@ x=`nix-store --query --normalise --force-realise /nix/store/bla.store`
</refsection> </refsection>
<refsection id='nixref-queries'> <refsection id='nixref-queries'><title>Queries</title>
<title>Queries</title>
<variablelist> <variablelist>
<varlistentry> <varlistentry><term><option>--list</option> / <option>-l</option></term>
<term><option>--list</option> / <option>-l</option></term>
<listitem> <listitem><para>Prints out the <emphasis>output paths</emphasis>
<para> of the store expressions indicated by the identifiers
Prints out the <emphasis>output paths</emphasis> of the <replaceable>args</replaceable>. In the case of a derivation
store expressions indicated by the identifiers expression, these are the paths that will be produced when the
<replaceable>args</replaceable>. In the case of a derivation is realised. In the case of a closure expression,
derivation expression, these are the paths that will be these are the paths that were produced the derivation expression
produced when the derivation is realised. In the case of which the closure expression is a successor.</para></listitem>
of a closure expression, these are the paths that were
produced the derivation expression of which the closure
expression is a successor.
</para>
</listitem>
</varlistentry> </varlistentry>
<varlistentry> <varlistentry><term><option>--requisites</option> / <option>-R</option></term>
<term><option>--requisites</option> / <option>-R</option></term>
<listitem>
<para>
Prints out the requisite paths of the store expressions
indicated by the identifiers
<replaceable>args</replaceable>. The requisite paths of
a Nix expression are the paths that need to be present
in the system to be able to realise the expression.
That is, they form the <emphasis>closure</emphasis> of
the expression in the file system (i.e., no path in the
set of requisite paths points to anything outside the
set of requisite paths).
</para>
<para> <listitem><para>Prints out the requisite paths of the store
The notion of requisite paths is very useful when one expressions indicated by the identifiers
wants to distribute store expressions. Since they form a <replaceable>args</replaceable>. The requisite paths of a Nix
closure, they are the only paths one needs to distribute expression are the paths that need to be present in the system to
to another system to be able to realise the expression be able to realise the expression. That is, they form the
on the other system. <emphasis>closure</emphasis> of the expression in the file system
</para> (i.e., no path in the set of requisite paths points to anything
outside the set of requisite paths).</para>
<para> <para>The notion of requisite paths is very useful when one wants
This query is generally used to implement various kinds to distribute store expressions. Since they form a closure, they
of deployment. A <emphasis>source deployment</emphasis> are the only paths one needs to distribute to another system to be
is obtained by distributing the requisite paths of a able to realise the expression on the other system.</para>
derivation expression. A <emphasis>binary
deployment</emphasis> is obtained by distributing the
requisite paths of a closure expression. A
<emphasis>cache deployment</emphasis> is obtained by
distributing the requisite paths of a derivation
expression and specifying the option
<option>--include-successors</option>. This will
include not just the paths of a source and binary
deployment, but also all expressions and paths of
subterms of the source. This is useful if one wants to
realise on the target system a Nix expression that is
similar but not quite the same as the one being
distributed, since any common subterms will be reused.
</para>
<para> <para>This query is generally used to implement various kinds of
This query has a number of options: deployment. A <emphasis>source deployment</emphasis> is obtained
</para> by distributing the requisite paths of a derivation expression. A
<emphasis>binary deployment</emphasis> is obtained by distributing
the requisite paths of a closure expression. A <emphasis>cache
deployment</emphasis> is obtained by distributing the requisite
paths of a derivation expression and specifying the option
<option>--include-successors</option>. This will include not just
the paths of a source and binary deployment, but also all
expressions and paths of subterms of the source. This is useful
if one wants to realise on the target system a Nix expression that
is similar but not quite the same as the one being distributed,
since any common subterms will be reused.</para>
<para>This query has one option:</para>
<variablelist> <variablelist>
<varlistentry> <varlistentry><term><option>--include-outputs</option></term>
<term><option>--exclude-exprs</option></term>
<listitem>
<para>
Excludes the paths of store expressions. This
causes the closure property to be lost, that is,
the resulting set of paths is not enough to ensure
realisibility.
</para>
</listitem>
</varlistentry>
<varlistentry> <listitem><para>Also include the output path of store
<term><option>--include-successors</option></term> derivations, and their closures.</para></listitem>
<listitem>
<para>
Also include the requisites of successors (normal forms).
Only the requisites of <emphasis>known</emphasis>
successors are included, i.e., the normal forms of
derivation expressions that have never been normalised will
not be included.
</para>
<para>
Note that not just the successor of a derivation expression
will be included, but also the successors of all input
expressions of that derivation expression. I.e., all
normal forms of subterms involved in the normalisation of
the top-level term are included.
</para>
</listitem>
</varlistentry> </varlistentry>
</variablelist> </variablelist>
</listitem> </listitem>
</varlistentry> </varlistentry>
<varlistentry>
<term><option>--predecessors</option></term>
<listitem>
<para>
For each store expression stored at paths
<replaceable>args</replaceable>, prints its
<emphasis>predecessors</emphasis>. A derivation
expression <varname>p</varname> is a predecessor of a
store expression <varname>q</varname> iff
<varname>q</varname> is a successor of
<varname>p</varname>.
</para>
</listitem>
</varlistentry>
<varlistentry> <varlistentry><term><option>--graph</option></term>
<term><option>--graph</option></term>
<listitem> <listitem><para>Prints a graph of the closure of the store
<para> expressions identified by <replaceable>args</replaceable> in the
Prints a graph of the closure of the store expressions format of the <command>dot</command> tool of AT&amp;T's GraphViz
identified by <replaceable>args</replaceable> in the package.</para></listitem>
format of the <command>dot</command> tool of AT&amp;T's
GraphViz package.
</para>
</listitem>
</varlistentry> </varlistentry>
</variablelist> </variablelist>
</refsection> </refsection>
</refsection> </refsection>
<!--######################################################################--> <!--######################################################################-->
<refsection> <refsection id="rsec-nix-store-reg-val"><title>Operation <option>--register-validity</option></title>
<title>Operation <option>--successor</option></title>
<refsection><title>Synopsis</title>
<refsection>
<title>Synopsis</title>
<cmdsynopsis> <cmdsynopsis>
<command>nix-store</command> <command>nix-store</command>
<arg choice='req'><option>--successor</option></arg> <arg choice='req'><option>--register-validity</option></arg>
<arg choice='plain'
rep='repeat'><replaceable>srcpath</replaceable> <replaceable>sucpath</replaceable></arg>
</cmdsynopsis> </cmdsynopsis>
</refsection> </refsection>
<refsection> <refsection><title>Description</title>
<title>Description</title>
<para> <para>TODO</para>
The operation <option>--successor</option> registers that the
closure expression in <replaceable>sucpath</replaceable> is a
successor of the derivation expression in
<replaceable>srcpath</replaceable>. This is used to implement
binary deployment.
</para>
</refsection> </refsection>
@ -519,11 +431,10 @@ x=`nix-store --query --normalise --force-realise /nix/store/bla.store`
<!--######################################################################--> <!--######################################################################-->
<refsection> <refsection><title>Operation <option>--substitute</option></title>
<title>Operation <option>--substitute</option></title>
<refsection><title>Synopsis</title>
<refsection>
<title>Synopsis</title>
<cmdsynopsis> <cmdsynopsis>
<command>nix-store</command> <command>nix-store</command>
<arg choice='req'><option>--substitute</option></arg> <arg choice='req'><option>--substitute</option></arg>
@ -532,16 +443,13 @@ x=`nix-store --query --normalise --force-realise /nix/store/bla.store`
</cmdsynopsis> </cmdsynopsis>
</refsection> </refsection>
<refsection> <refsection><title>Description</title>
<title>Description</title>
<para> <para>The operation <option>--substitute</option> registers that the
The operation <option>--substitute</option> registers that the
store path <replaceable>srcpath</replaceable> can be built by store path <replaceable>srcpath</replaceable> can be built by
realising the derivation expression in realising the derivation expression in
<replaceable>subpath</replaceable>. This is used to implement <replaceable>subpath</replaceable>. This is used to implement binary
binary deployment. deployment.</para>
</para>
</refsection> </refsection>
@ -551,8 +459,7 @@ x=`nix-store --query --normalise --force-realise /nix/store/bla.store`
<!--######################################################################--> <!--######################################################################-->
<refsection> <refsection><title>Operation <option>--verify</option></title>
<title>Operation <option>--verify</option></title>
<refsection> <refsection>
<title>Synopsis</title> <title>Synopsis</title>
@ -562,17 +469,14 @@ x=`nix-store --query --normalise --force-realise /nix/store/bla.store`
</cmdsynopsis> </cmdsynopsis>
</refsection> </refsection>
<refsection> <refsection><title>Description</title>
<title>Description</title>
<para> <para>The operation <option>--verify</option> verifies the internal
The operation <option>--verify</option> verifies the internal consistency of the Nix database, and the consistency between the Nix
consistency of the Nix database, and the consistency between database and the Nix store. Any inconsistencies encountered are
the Nix database and the Nix store. Any inconsistencies automatically repaired. Inconsistencies are generally the result of
encountered are automatically repaired. Inconsistencies are the Nix store or database being modified by non-Nix tools, or of bugs
generally the result of the Nix store or database being in Nix itself.</para>
modified by non-Nix tools, or of bugs in Nix itself.
</para>
</refsection> </refsection>

View file

@ -84,7 +84,7 @@
</varlistentry> </varlistentry>
<varlistentry><term><option>--max-jobs</option> / <option>-j</option></term> <varlistentry id="opt-max-jobs"><term><option>--max-jobs</option> / <option>-j</option></term>
<listitem><para>Sets the maximum number of build jobs that Nix will <listitem><para>Sets the maximum number of build jobs that Nix will
perform in parallel to the specified number. The default is 1. A perform in parallel to the specified number. The default is 1. A

View file

@ -1112,7 +1112,7 @@ set, the attributes of which specify the inputs of the build.</para>
<itemizedlist> <itemizedlist>
<listitem><para>There must be an attribute named <listitem id="attr-system"><para>There must be an attribute named
<varname>system</varname> whose value must be a string specifying a <varname>system</varname> whose value must be a string specifying a
Nix platform identifier, such as <literal>"i686-linux"</literal> or Nix platform identifier, such as <literal>"i686-linux"</literal> or
<literal>"powerpc-darwin"</literal><footnote><para>To figure out <literal>"powerpc-darwin"</literal><footnote><para>To figure out