forked from lix-project/lix
Update nix-push manpage and document the binary cache format
This commit is contained in:
Eelco Dolstra
parent
82951e5582
commit
3a95e1a17c
|
|
@ -160,6 +160,18 @@
|
||||||
</glossentry>
|
</glossentry>
|
||||||
|
|
||||||
|
|
||||||
|
<glossentry xml:id="gloss-nar"><glossterm>NAR</glossterm>
|
||||||
|
|
||||||
|
<glossdef><para>A <emphasis>N</emphasis>ix
|
||||||
|
<emphasis>AR</emphasis>chive. This is a serialisation of a path in
|
||||||
|
the Nix store. It can contain regular files, directories and
|
||||||
|
symbolic links. NARs are generated and unpacked using
|
||||||
|
<command>nix-store --dump</command> and <command>nix-store
|
||||||
|
--restore</command>.</para></glossdef>
|
||||||
|
|
||||||
|
</glossentry>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</glosslist>
|
</glosslist>
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -12,24 +12,19 @@
|
||||||
|
|
||||||
<refnamediv>
|
<refnamediv>
|
||||||
<refname>nix-push</refname>
|
<refname>nix-push</refname>
|
||||||
<refpurpose>push store paths onto a network cache</refpurpose>
|
<refpurpose>generate a binary cache</refpurpose>
|
||||||
</refnamediv>
|
</refnamediv>
|
||||||
|
|
||||||
<refsynopsisdiv>
|
<refsynopsisdiv>
|
||||||
<cmdsynopsis>
|
<cmdsynopsis>
|
||||||
<command>nix-push</command>
|
<command>nix-push</command>
|
||||||
<group choice='req'>
|
<arg choice='plain'><option>--dest</option> <replaceable>dest-dir</replaceable></arg>
|
||||||
<arg choice='req'>
|
<arg><option>--bzip2</option></arg>
|
||||||
<arg choice='plain'><replaceable>archivesPutURL</replaceable></arg>
|
<arg><option>--force</option></arg>
|
||||||
<arg choice='plain'><replaceable>archivesGetURL</replaceable></arg>
|
<arg><option>--link</option></arg>
|
||||||
<arg choice='plain'><replaceable>manifestPutURL</replaceable></arg>
|
<arg><option>--manifest</option></arg>
|
||||||
</arg>
|
<arg><option>--manifest-path</option> <replaceable>filename</replaceable></arg>
|
||||||
<arg choice='req'>
|
<arg><option>--url-prefix</option> <replaceable>url</replaceable></arg>
|
||||||
<arg choice='plain'><option>--copy</option></arg>
|
|
||||||
<arg choice='plain'><replaceable>archivesDir</replaceable></arg>
|
|
||||||
<arg choice='plain'><replaceable>manifestFile</replaceable></arg>
|
|
||||||
</arg>
|
|
||||||
</group>
|
|
||||||
<arg choice='plain' rep='repeat'><replaceable>paths</replaceable></arg>
|
<arg choice='plain' rep='repeat'><replaceable>paths</replaceable></arg>
|
||||||
</cmdsynopsis>
|
</cmdsynopsis>
|
||||||
</refsynopsisdiv>
|
</refsynopsisdiv>
|
||||||
|
|
@ -37,93 +32,350 @@
|
||||||
|
|
||||||
<refsection><title>Description</title>
|
<refsection><title>Description</title>
|
||||||
|
|
||||||
<para>The command <command>nix-push</command> builds a set of store
|
<para>The command <command>nix-push</command> produces a
|
||||||
paths (if necessary), and then packages and uploads all store paths in
|
<emphasis>binary cache</emphasis>, a directory containing compressed
|
||||||
the resulting closures to a server. A network cache thus populated
|
Nix archives (NARs) plus some metadata of the closure of the specified
|
||||||
can subsequently be used to speed up software deployment on other
|
store paths. This directory can then be made available through a web
|
||||||
machines using the <command>nix-pull</command> command.</para>
|
server to other Nix installations, allowing them to skip building from
|
||||||
|
source and instead download binaries from the cache
|
||||||
|
automatically.</para>
|
||||||
|
|
||||||
<para><command>nix-push</command> performs the following actions.
|
<para><command>nix-push</command> performs the following actions.
|
||||||
|
|
||||||
<orderedlist>
|
<orderedlist>
|
||||||
|
|
||||||
<listitem><para>Each path in <replaceable>paths</replaceable> is
|
<listitem><para>Each path in <replaceable>paths</replaceable> is
|
||||||
realised (using <link
|
built (using <link
|
||||||
linkend='rsec-nix-store-realise'><literal>nix-store
|
linkend='rsec-nix-store-realise'><command>nix-store
|
||||||
--realise</literal></link>).</para></listitem>
|
--realise</command></link>).</para></listitem>
|
||||||
|
|
||||||
<listitem><para>All paths in the closure of the store expressions
|
<listitem><para>All paths in the closure of
|
||||||
stored in <replaceable>paths</replaceable> are determined (using
|
<replaceable>paths</replaceable> are determined (using
|
||||||
<literal>nix-store --query --requisites
|
<command>nix-store --query --requisites
|
||||||
--include-outputs</literal>). It should be noted that since the
|
--include-outputs</command>). Note that since the
|
||||||
<option>--include-outputs</option> flag is used, you get a combined
|
<option>--include-outputs</option> flag is used, if
|
||||||
source/binary distribution.</para></listitem>
|
<replaceable>paths</replaceable> includes a store derivation, you
|
||||||
|
get a combined source/binary distribution (e.g., source tarballs
|
||||||
|
will be included).</para></listitem>
|
||||||
|
|
||||||
<listitem><para>All store paths determined in the previous step are
|
<listitem><para>All store paths determined in the previous step are
|
||||||
packaged and compressed into a <command>bzip</command>ped NAR
|
packaged into a NAR (using <command>nix-store --dump</command>) and
|
||||||
archive (extension <filename>.nar.bz2</filename>).</para></listitem>
|
compressed using <command>xz</command> or <command>bzip2</command>.
|
||||||
|
The resulting files have the extension <filename>.nar.xz</filename>
|
||||||
|
or <filename>.nar.bz2</filename>. Also for each store path, Nix
|
||||||
|
generates a file with extension <filename>.narinfo</filename>
|
||||||
|
containing metadata such as the references, cryptographic hash and
|
||||||
|
size of each path.</para></listitem>
|
||||||
|
|
||||||
<listitem><para>A <emphasis>manifest</emphasis> is created that
|
<listitem><para>Optionally, a single <emphasis>manifest</emphasis>
|
||||||
contains information on the store paths, their eventual URLs in the
|
file is created that contains the same metadata as the
|
||||||
cache, and cryptographic hashes of the contents of the NAR
|
<filename>.narinfo</filename> files. This is for compatibility with
|
||||||
archives.</para></listitem>
|
Nix versions prior to 1.2 (see <command>nix-pull</command> for
|
||||||
|
details).</para></listitem>
|
||||||
|
|
||||||
<listitem><para>Each store path is uploaded to the remote directory
|
<listitem><para>A file named <option>nix-cache-info</option> is
|
||||||
specified by <replaceable>archivesPutURL</replaceable>. HTTP PUT
|
placed in the destination directory. The existence of this file
|
||||||
requests are used to do this. However, before a file
|
marks the directory as a binary cache.</para></listitem>
|
||||||
<varname>x</varname> is uploaded to
|
|
||||||
<literal><replaceable>archivesPutURL</replaceable>/</literal><varname>x</varname>,
|
|
||||||
<command>nix-push</command> first determines whether this upload is
|
|
||||||
unnecessary by issuing a HTTP HEAD request on
|
|
||||||
<literal><replaceable>archivesGetURL</replaceable>/</literal><varname>x</varname>.
|
|
||||||
This allows a cache to be shared between many partially overlapping
|
|
||||||
<command>nix-push</command> invocations. (We use two URLs because
|
|
||||||
the upload URL typically refers to a CGI script, while the download
|
|
||||||
URL just refers to a file system directory on the
|
|
||||||
server.)</para></listitem>
|
|
||||||
|
|
||||||
<listitem><para>The manifest is uploaded using an HTTP PUT request
|
|
||||||
to <replaceable>manifestPutURL</replaceable>. The corresponding
|
|
||||||
URL to download the manifest can then be used by
|
|
||||||
<command>nix-pull</command>.</para></listitem>
|
|
||||||
|
|
||||||
</orderedlist>
|
</orderedlist>
|
||||||
|
|
||||||
</para>
|
</para>
|
||||||
|
|
||||||
<!--
|
</refsection>
|
||||||
<para>TODO: <option>- -copy</option></para>
|
|
||||||
-->
|
|
||||||
|
<refsection><title>Options</title>
|
||||||
|
|
||||||
|
<variablelist>
|
||||||
|
|
||||||
|
<varlistentry><term><option>--dest</option> <replaceable>dest-dir</replaceable></term>
|
||||||
|
|
||||||
|
<listitem><para>Set the destination directory to
|
||||||
|
<replaceable>dir</replaceable>, which is created if it does not
|
||||||
|
exist. This flag is required.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><option>--bzip2</option></term>
|
||||||
|
|
||||||
|
<listitem><para>Compress NARs using <command>bzip2</command>
|
||||||
|
instead of <command>xz -9</command>. The latter compresses about
|
||||||
|
30% better on typical archives, decompresses about twice as fast,
|
||||||
|
but compresses a lot slower and is not supported by Nix prior to
|
||||||
|
version 1.2.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><option>--force</option></term>
|
||||||
|
|
||||||
|
<listitem><para>Overwrite <filename>.narinfo</filename> files if
|
||||||
|
they already exist.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><option>--link</option></term>
|
||||||
|
|
||||||
|
<listitem><para>By default, NARs are generated in the Nix store
|
||||||
|
and then copied to <replaceable>dest-dir</replaceable>. If this
|
||||||
|
option is given, hard links are used instead. This only works if
|
||||||
|
<replaceable>dest-dir</replaceable> is on the same filesystem as
|
||||||
|
the Nix store.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><option>--manifest</option></term>
|
||||||
|
|
||||||
|
<listitem><para>Force the generation of a manifest suitable for
|
||||||
|
use by <command>nix-pull</command>. The manifest is stored as
|
||||||
|
<filename><replaceable>dest-dir</replaceable>/MANIFEST</filename>.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><option>--manifest-path</option> <replaceable>filename</replaceable></term>
|
||||||
|
|
||||||
|
<listitem><para>Like <option>--manifest</option>, but store the
|
||||||
|
manifest in <replaceable>filename</replaceable>.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><option>--url-prefix</option> <replaceable>url</replaceable></term>
|
||||||
|
|
||||||
|
<listitem><para>Manifests are expected to contain the absolute
|
||||||
|
URLs of NARs. For generating these URLs, the prefix
|
||||||
|
<replaceable>url</replaceable> is used. It defaults to
|
||||||
|
<uri>file://<replaceable>dest-dir</replaceable></uri>.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
</variablelist>
|
||||||
|
|
||||||
</refsection>
|
</refsection>
|
||||||
|
|
||||||
|
|
||||||
<refsection><title>Examples</title>
|
<refsection><title>Examples</title>
|
||||||
|
|
||||||
<para>To upload files there typically is some CGI script on the server
|
<para>To add the closure of Thunderbird to a binary cache:
|
||||||
side. This script should be be protected with a password. The
|
|
||||||
following example uploads the store paths resulting from building the
|
|
||||||
Nix expressions in <filename>foo.nix</filename>, passing appropriate
|
|
||||||
authentication information:
|
|
||||||
|
|
||||||
<screen>
|
|
||||||
$ nix-push \
|
|
||||||
http://foo@bar:server.domain/cgi-bin/upload.pl/cache \
|
|
||||||
http://server.domain/cache \
|
|
||||||
http://foo@bar:server.domain/cgi-bin/upload.pl/MANIFEST \
|
|
||||||
$(nix-instantiate foo.nix)</screen>
|
|
||||||
|
|
||||||
This will push both sources and binaries (and any build-time
|
|
||||||
dependencies used in the build, such as compilers).</para>
|
|
||||||
|
|
||||||
<para>If we just want to push binaries, not sources and build-time
|
|
||||||
dependencies, we can do:
|
|
||||||
|
|
||||||
<screen>
|
<screen>
|
||||||
$ nix-push <replaceable>urls</replaceable> $(nix-store -r $(nix-instantiate foo.nix))</screen>
|
$ nix-push --dest /tmp/cache $(nix-build -A thunderbird)
|
||||||
|
</screen>
|
||||||
|
|
||||||
|
Assuming that <filename>/tmp/cache</filename> is exported by a web
|
||||||
|
server as <uri>http://example.org/cache</uri>, you can then use this
|
||||||
|
cache on another machine to speed up the installation of Thunderbird:
|
||||||
|
|
||||||
|
<screen>
|
||||||
|
$ nix-build -A thunderbird --option binary-caches http://example.org/cache
|
||||||
|
</screen>
|
||||||
|
|
||||||
|
Alternatively, you could add <literal>binary-caches =
|
||||||
|
http://example.org/cache</literal> to
|
||||||
|
<filename>nix.conf</filename>.</para>
|
||||||
|
|
||||||
|
<para>To also include build-time dependencies (such as source
|
||||||
|
tarballs):
|
||||||
|
|
||||||
|
<screen>
|
||||||
|
$ nix-push --dest /tmp/cache $(nix-instantiate -A thunderbird)
|
||||||
|
</screen>
|
||||||
|
|
||||||
</para>
|
</para>
|
||||||
|
|
||||||
|
<para>To generate a manifest suitable for <command>nix-pull</command>:
|
||||||
|
|
||||||
|
<screen>
|
||||||
|
$ nix-push --dest /tmp/cache $(nix-build -A thunderbird) --manifest
|
||||||
|
</screen>
|
||||||
|
|
||||||
|
On another machine you can then do:
|
||||||
|
|
||||||
|
<screen>
|
||||||
|
$ nix-pull http://example.org/cache
|
||||||
|
</screen>
|
||||||
|
|
||||||
|
to cause the binaries to be used by subsequent Nix operations.</para>
|
||||||
|
|
||||||
</refsection>
|
</refsection>
|
||||||
|
|
||||||
|
|
||||||
|
<refsection><title>Binary cache format and operation</title>
|
||||||
|
|
||||||
|
<para>A binary cache with URL <replaceable>url</replaceable> only
|
||||||
|
denotes a valid binary cache if the file
|
||||||
|
<uri><replaceable>url</replaceable>/nix-cache-info</uri> exists. If
|
||||||
|
this file does not exist (or cannot be downloaded), the cache is
|
||||||
|
ignored. If it does exist, it must be a text file containing cache
|
||||||
|
properties. Here’s an example:
|
||||||
|
|
||||||
|
<screen>
|
||||||
|
StoreDir: /nix/store
|
||||||
|
WantMassQuery: 1
|
||||||
|
</screen>
|
||||||
|
|
||||||
|
The properties that are currently supported are:
|
||||||
|
|
||||||
|
<variablelist>
|
||||||
|
|
||||||
|
<varlistentry><term><literal>StoreDir</literal></term>
|
||||||
|
|
||||||
|
<listitem><para>The path of the Nix store to which this binary
|
||||||
|
cache applies. Binaries are not relocatable — a binary built for
|
||||||
|
<filename>/nix/store</filename> won’t generally work in
|
||||||
|
<filename>/home/alice/store</filename> — so to prevent binaries
|
||||||
|
from being used in a wrong store, a binary cache is only used if
|
||||||
|
its <literal>StoreDir</literal> matches the local Nix
|
||||||
|
configuration. The default is
|
||||||
|
<filename>/nix/store</filename>.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><literal>WantMassQuery</literal></term>
|
||||||
|
|
||||||
|
<listitem><para>Query operations such as <command>nix-env
|
||||||
|
-qas</command> can cause thousands of cache queries, and thus
|
||||||
|
thousands of HTTP requests, to determine which packages are
|
||||||
|
available in binary form. While these requests are small, not
|
||||||
|
every server may appreciate a potential onslaught of queries. If
|
||||||
|
<literal>WantMassQuery</literal> is set to <literal>0</literal>
|
||||||
|
(default), “mass queries” such as <command>nix-env -qas</command>
|
||||||
|
will skip this cache. Thus a package may appear not to have a
|
||||||
|
binary substitute. However, the binary will still be used when
|
||||||
|
you actually install the package. If
|
||||||
|
<literal>WantMassQuery</literal> is set to <literal>1</literal>,
|
||||||
|
mass queries will use this cache.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
</variablelist>
|
||||||
|
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>Every time Nix needs to build some store path
|
||||||
|
<replaceable>p</replaceable>, it will check each configured binary
|
||||||
|
cache to see if it has a NAR file for <replaceable>p</replaceable>,
|
||||||
|
until it finds one. If no cache has a NAR, Nix will fall back to
|
||||||
|
building the path from source (if applicable). To see if a cache with
|
||||||
|
URL <replaceable>url</replaceable> has a binary for
|
||||||
|
<replaceable>p</replaceable>, Nix fetches
|
||||||
|
<replaceable>url/h</replaceable>, where <replaceable>h</replaceable>
|
||||||
|
is the hash part of <replaceable>p</replaceable>. Thus, if we have a
|
||||||
|
cache <uri>http://nixos.org/binary-cache</uri> and we want to obtain
|
||||||
|
the store path
|
||||||
|
<screen>
|
||||||
|
/nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13
|
||||||
|
</screen>
|
||||||
|
then Nix will attempt to fetch
|
||||||
|
<screen>
|
||||||
|
http://nixos.org/binary-cache/cj7a81wsm1ijwwpkks3725661h3263p5.narinfo
|
||||||
|
</screen>
|
||||||
|
(Commands such as <command>nix-env -qas</command> will issue an HTTP
|
||||||
|
HEAD request, since it only needs to know if the
|
||||||
|
<filename>.narinfo</filename> file exists.) The
|
||||||
|
<filename>.narinfo</filename> file is a simple text file that looks
|
||||||
|
like this:
|
||||||
|
|
||||||
|
<screen>
|
||||||
|
StorePath: /nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13
|
||||||
|
URL: nar/0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21.nar.bz2
|
||||||
|
Compression: bzip2
|
||||||
|
FileHash: sha256:0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21
|
||||||
|
FileSize: 10119014
|
||||||
|
NarHash: sha256:120mhshkcgvn48xvxz0fjbd47k615v0viv7jy4gy7dwhmqapgd9d
|
||||||
|
NarSize: 33702192
|
||||||
|
References: cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 jfcs9xnfbmiwqs224sb0qqsybbfl3sab-linux-headers-2.6.35.14
|
||||||
|
Deriver: xs32zzf2d58f6l5iz5fgwxrds2q5pnvn-glibc-2.13.drv
|
||||||
|
System: x86_64-linux
|
||||||
|
</screen>
|
||||||
|
|
||||||
|
The fields are as follows:
|
||||||
|
|
||||||
|
<variablelist>
|
||||||
|
|
||||||
|
<varlistentry><term><literal>StorePath</literal></term>
|
||||||
|
|
||||||
|
<listitem><para>The full store path, including the name part
|
||||||
|
(e.g., <literal>glibc-2.13</literal>). It must match the
|
||||||
|
requested store path.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><literal>URL</literal></term>
|
||||||
|
|
||||||
|
<listitem><para>The URL of the NAR, relative to the binary cache
|
||||||
|
URL.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><literal>Compression</literal></term>
|
||||||
|
|
||||||
|
<listitem><para>The compression method; either
|
||||||
|
<literal>xz</literal> or
|
||||||
|
<literal>bzip2</literal>.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><literal>FileHash</literal></term>
|
||||||
|
|
||||||
|
<listitem><para>The SHA-256 hash of the compressed
|
||||||
|
NAR.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><literal>FileSize</literal></term>
|
||||||
|
|
||||||
|
<listitem><para>The size of the compressed NAR.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><literal>NarHash</literal></term>
|
||||||
|
|
||||||
|
<listitem><para>The SHA-256 hash of the uncompressed NAR. This is
|
||||||
|
equal to the hash of the store path as returned by
|
||||||
|
<command>nix-store -q --hash
|
||||||
|
<replaceable>p</replaceable></command>.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><literal>NarSize</literal></term>
|
||||||
|
|
||||||
|
<listitem><para>The size of the uncompressed NAR.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><literal>References</literal></term>
|
||||||
|
|
||||||
|
<listitem><para>The references of the store path, without the Nix
|
||||||
|
store prefix.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><literal>Deriver</literal></term>
|
||||||
|
|
||||||
|
<listitem><para>The deriver of the store path, without the Nix
|
||||||
|
store prefix. This field is optional.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
<varlistentry><term><literal>System</literal></term>
|
||||||
|
|
||||||
|
<listitem><para>The Nix platform type of this binary, if known.
|
||||||
|
This field is optional.</para></listitem>
|
||||||
|
|
||||||
|
</varlistentry>
|
||||||
|
|
||||||
|
</variablelist>
|
||||||
|
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>Thus, in our example, after recursively ensuring that the
|
||||||
|
references exist (e.g.,
|
||||||
|
<filename>/nix/store/jfcs9xnfbmiwqs224sb0qqsybbfl3sab-linux-headers-2.6.35.14</filename>),
|
||||||
|
Nix will fetch <screen>
|
||||||
|
http://nixos.org/binary-cache/nar/0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21.nar.bz2
|
||||||
|
</screen> and decompress and unpack it to
|
||||||
|
<filename>/nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13</filename>.</para>
|
||||||
|
|
||||||
|
</refsection>
|
||||||
|
|
||||||
|
|
||||||
</refentry>
|
</refentry>
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue