From 3a95e1a17cd2755f00c26eb16ffd066bc6aa1a8d Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Tue, 6 Nov 2012 15:49:53 +0100 Subject: [PATCH] Update nix-push manpage and document the binary cache format --- doc/manual/glossary.xml | 12 ++ doc/manual/nix-push.xml | 404 ++++++++++++++++++++++++++++++++-------- 2 files changed, 340 insertions(+), 76 deletions(-) diff --git a/doc/manual/glossary.xml b/doc/manual/glossary.xml index efbf96f0a..d74940c90 100644 --- a/doc/manual/glossary.xml +++ b/doc/manual/glossary.xml @@ -160,6 +160,18 @@ +NAR + + A Nix + ARchive. This is a serialisation of a path in + the Nix store. It can contain regular files, directories and + symbolic links. NARs are generated and unpacked using + nix-store --dump and nix-store + --restore. + + + + diff --git a/doc/manual/nix-push.xml b/doc/manual/nix-push.xml index d2a7e063e..170b6f55a 100644 --- a/doc/manual/nix-push.xml +++ b/doc/manual/nix-push.xml @@ -12,24 +12,19 @@ nix-push - push store paths onto a network cache + generate a binary cache nix-push - - - archivesPutURL - archivesGetURL - manifestPutURL - - - - archivesDir - manifestFile - - + dest-dir + + + + + filename + url paths @@ -37,93 +32,350 @@ Description -The command nix-push builds a set of store -paths (if necessary), and then packages and uploads all store paths in -the resulting closures to a server. A network cache thus populated -can subsequently be used to speed up software deployment on other -machines using the nix-pull command. +The command nix-push produces a +binary cache, a directory containing compressed +Nix archives (NARs) plus some metadata of the closure of the specified +store paths. This directory can then be made available through a web +server to other Nix installations, allowing them to skip building from +source and instead download binaries from the cache +automatically. nix-push performs the following actions. Each path in paths is - realised (using nix-store - --realise). + built (using nix-store + --realise). - All paths in the closure of the store expressions - stored in paths are determined (using - nix-store --query --requisites - --include-outputs). It should be noted that since the - flag is used, you get a combined - source/binary distribution. + All paths in the closure of + paths are determined (using + nix-store --query --requisites + --include-outputs). Note that since the + flag is used, if + paths includes a store derivation, you + get a combined source/binary distribution (e.g., source tarballs + will be included). All store paths determined in the previous step are - packaged and compressed into a bzipped NAR - archive (extension .nar.bz2). + packaged into a NAR (using nix-store --dump) and + compressed using xz or bzip2. + The resulting files have the extension .nar.xz + or .nar.bz2. Also for each store path, Nix + generates a file with extension .narinfo + containing metadata such as the references, cryptographic hash and + size of each path. - A manifest is created that - contains information on the store paths, their eventual URLs in the - cache, and cryptographic hashes of the contents of the NAR - archives. + Optionally, a single manifest + file is created that contains the same metadata as the + .narinfo files. This is for compatibility with + Nix versions prior to 1.2 (see nix-pull for + details). - Each store path is uploaded to the remote directory - specified by archivesPutURL. HTTP PUT - requests are used to do this. However, before a file - x is uploaded to - archivesPutURL/x, - nix-push first determines whether this upload is - unnecessary by issuing a HTTP HEAD request on - archivesGetURL/x. - This allows a cache to be shared between many partially overlapping - nix-push invocations. (We use two URLs because - the upload URL typically refers to a CGI script, while the download - URL just refers to a file system directory on the - server.) + A file named is + placed in the destination directory. The existence of this file + marks the directory as a binary cache. - The manifest is uploaded using an HTTP PUT request - to manifestPutURL. The corresponding - URL to download the manifest can then be used by - nix-pull. - - - + + + +Options + + + + dest-dir + + Set the destination directory to + dir, which is created if it does not + exist. This flag is required. + + + + + + Compress NARs using bzip2 + instead of xz -9. The latter compresses about + 30% better on typical archives, decompresses about twice as fast, + but compresses a lot slower and is not supported by Nix prior to + version 1.2. + + + + + + Overwrite .narinfo files if + they already exist. + + + + + + By default, NARs are generated in the Nix store + and then copied to dest-dir. If this + option is given, hard links are used instead. This only works if + dest-dir is on the same filesystem as + the Nix store. + + + + + + Force the generation of a manifest suitable for + use by nix-pull. The manifest is stored as + dest-dir/MANIFEST. + + + + filename + + Like , but store the + manifest in filename. + + + + url + + Manifests are expected to contain the absolute + URLs of NARs. For generating these URLs, the prefix + url is used. It defaults to + file://dest-dir. + + + + + Examples -To upload files there typically is some CGI script on the server -side. This script should be be protected with a password. The -following example uploads the store paths resulting from building the -Nix expressions in foo.nix, passing appropriate -authentication information: - - -$ nix-push \ - http://foo@bar:server.domain/cgi-bin/upload.pl/cache \ - http://server.domain/cache \ - http://foo@bar:server.domain/cgi-bin/upload.pl/MANIFEST \ - $(nix-instantiate foo.nix) +To add the closure of Thunderbird to a binary cache: -This will push both sources and binaries (and any build-time -dependencies used in the build, such as compilers). - -If we just want to push binaries, not sources and build-time -dependencies, we can do: - -$ nix-push urls $(nix-store -r $(nix-instantiate foo.nix)) - +$ nix-push --dest /tmp/cache $(nix-build -A thunderbird) + + +Assuming that /tmp/cache is exported by a web +server as http://example.org/cache, you can then use this +cache on another machine to speed up the installation of Thunderbird: + + +$ nix-build -A thunderbird --option binary-caches http://example.org/cache + + +Alternatively, you could add binary-caches = +http://example.org/cache to +nix.conf. + +To also include build-time dependencies (such as source +tarballs): + + +$ nix-push --dest /tmp/cache $(nix-instantiate -A thunderbird) + + +To generate a manifest suitable for nix-pull: + + +$ nix-push --dest /tmp/cache $(nix-build -A thunderbird) --manifest + + +On another machine you can then do: + + +$ nix-pull http://example.org/cache + + +to cause the binaries to be used by subsequent Nix operations. + - + + +Binary cache format and operation + +A binary cache with URL url only +denotes a valid binary cache if the file +url/nix-cache-info exists. If +this file does not exist (or cannot be downloaded), the cache is +ignored. If it does exist, it must be a text file containing cache +properties. Here’s an example: + + +StoreDir: /nix/store +WantMassQuery: 1 + + +The properties that are currently supported are: + + + + StoreDir + + The path of the Nix store to which this binary + cache applies. Binaries are not relocatable — a binary built for + /nix/store won’t generally work in + /home/alice/store — so to prevent binaries + from being used in a wrong store, a binary cache is only used if + its StoreDir matches the local Nix + configuration. The default is + /nix/store. + + + + WantMassQuery + + Query operations such as nix-env + -qas can cause thousands of cache queries, and thus + thousands of HTTP requests, to determine which packages are + available in binary form. While these requests are small, not + every server may appreciate a potential onslaught of queries. If + WantMassQuery is set to 0 + (default), “mass queries” such as nix-env -qas + will skip this cache. Thus a package may appear not to have a + binary substitute. However, the binary will still be used when + you actually install the package. If + WantMassQuery is set to 1, + mass queries will use this cache. + + + + + + + +Every time Nix needs to build some store path +p, it will check each configured binary +cache to see if it has a NAR file for p, +until it finds one. If no cache has a NAR, Nix will fall back to +building the path from source (if applicable). To see if a cache with +URL url has a binary for +p, Nix fetches +url/h, where h +is the hash part of p. Thus, if we have a +cache http://nixos.org/binary-cache and we want to obtain +the store path + +/nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 + +then Nix will attempt to fetch + +http://nixos.org/binary-cache/cj7a81wsm1ijwwpkks3725661h3263p5.narinfo + +(Commands such as nix-env -qas will issue an HTTP +HEAD request, since it only needs to know if the +.narinfo file exists.) The +.narinfo file is a simple text file that looks +like this: + + +StorePath: /nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 +URL: nar/0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21.nar.bz2 +Compression: bzip2 +FileHash: sha256:0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21 +FileSize: 10119014 +NarHash: sha256:120mhshkcgvn48xvxz0fjbd47k615v0viv7jy4gy7dwhmqapgd9d +NarSize: 33702192 +References: cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 jfcs9xnfbmiwqs224sb0qqsybbfl3sab-linux-headers-2.6.35.14 +Deriver: xs32zzf2d58f6l5iz5fgwxrds2q5pnvn-glibc-2.13.drv +System: x86_64-linux + + +The fields are as follows: + + + + StorePath + + The full store path, including the name part + (e.g., glibc-2.13). It must match the + requested store path. + + + + URL + + The URL of the NAR, relative to the binary cache + URL. + + + + Compression + + The compression method; either + xz or + bzip2. + + + + FileHash + + The SHA-256 hash of the compressed + NAR. + + + + FileSize + + The size of the compressed NAR. + + + + NarHash + + The SHA-256 hash of the uncompressed NAR. This is + equal to the hash of the store path as returned by + nix-store -q --hash + p. + + + + NarSize + + The size of the uncompressed NAR. + + + + References + + The references of the store path, without the Nix + store prefix. + + + + Deriver + + The deriver of the store path, without the Nix + store prefix. This field is optional. + + + + System + + The Nix platform type of this binary, if known. + This field is optional. + + + + + + + +Thus, in our example, after recursively ensuring that the +references exist (e.g., +/nix/store/jfcs9xnfbmiwqs224sb0qqsybbfl3sab-linux-headers-2.6.35.14), +Nix will fetch +http://nixos.org/binary-cache/nar/0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21.nar.bz2 + and decompress and unpack it to +/nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13. + + + +