From f8ac8d1ec835cf070afd62c0cef14a81e9a49302 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 29 Oct 2004 15:26:26 +0000 Subject: [PATCH] * Began adding build farm docs. --- doc/manual/Makefile.am | 1 + doc/manual/build-farm.xml | 129 ++++++++++++++++++++++++++++++++++++++ doc/manual/manual.xml | 2 + 3 files changed, 132 insertions(+) create mode 100644 doc/manual/build-farm.xml diff --git a/doc/manual/Makefile.am b/doc/manual/Makefile.am index 28ebb4d15..0d1e85cc8 100644 --- a/doc/manual/Makefile.am +++ b/doc/manual/Makefile.am @@ -12,6 +12,7 @@ man1_MANS = nix-env.1 nix-store.1 nix-instantiate.1 \ SOURCES = manual.xml introduction.xml installation.xml \ package-management.xml writing-nix-expressions.xml \ + build-farm.xml \ $(man1_MANS:.1=.xml) \ troubleshooting.xml bugs.xml opt-common.xml opt-common-syn.xml \ quick-start.xml nix-lang-ref.xml style.css images diff --git a/doc/manual/build-farm.xml b/doc/manual/build-farm.xml new file mode 100644 index 000000000..2dd0932fc --- /dev/null +++ b/doc/manual/build-farm.xml @@ -0,0 +1,129 @@ +Setting up a Build Farm + +This chapter provides some sketchy information on how to set up +a Nix-based build farm. Nix is particularly suited as a basis for a +build farm, since: + + + + Nix supports distributed builds: a local Nix + installation can forward Nix builds to other machines over the + network. This allows multiple builds to be performed in parallel + (thus improving performce), but more in importantly, it allows Nix + to perform multi-platform builds in a semi-transparent way. For + instance, if you perform a build for a + powerpc-darwin on an + i686-linux machine, Nix can automatically forward + to build to a powerpc-darwin machine, if + available. + + The Nix expression language is ideal for providing + build jobs, plus all their dependencies. For instance, if your + package has some dependency, you don't have to manually install it + on all the machines in the build farm; they will be built + automatically. + + Proper release management requires that builds (if + deployed) are traceable: it should be possible to figure out from + exactly what sources they were built, in what configuration, etc.; + and it should be possible to reproduce the build, if necessary. + Nix's hashing scheme uniquely identifies builds, and Nix expressions + are self-contained. + + Nix will only rebuild things that have actually + changed. For instance, if the sources of a component haven't + changed between runs of the build farm, the component won't be + rebuild (unless it was garbage-collected). Also, dependencies + typically don't change very often, so they only need to be built + once. + + The results of a Nix build farm can be made + available through a channel, so that users can use succesfull builds + immediately. + + + + + + +
Overview + +TODO + +The sources of the Nix build farm are at . + +
+ + +
Setting up distributed builds + +You can enable distributed builds by setting the environment +variable NIX_BUILD_HOOK to point to a program that Nix +will call whenever it wants to build a derivation. The build hook +(typically a shell or Perl script) can decline the build, in which Nix +will perform it in the usual way if possible, or it can accept it, in +which case it is responsible for somehow getting the inputs of the +build to another machine, doing the build there, and getting the +results back. + +Remote machine configuration: +<filename>remote-systems.conf</filename> + +nix@mcflurry.labs.cs.uu.nl powerpc-darwin /home/nix/.ssh/id_quarterpounder_auto 2 +nix@scratchy.labs.cs.uu.nl i686-linux /home/nix/.ssh/id_scratchy_auto 1 + + + +An example build hook can be found in the Nix build farm +sources: . It should be suitable for most purposes, with maybe some minor +adjustments. It uses ssh and +rsync to copy the build inputs and outputs and +perform the remote build. You should define a list of available build +machines and set the environment variable +REMOTE_SYSTEMS to point to it. An example +configuration is shown in . Each +line in the file specifies a machine, with the following bits of +information: + + + + The name of the remote machine, with optionally the + user under which the remote build should be performed. This is + actually passed as an argument to ssh, so it can + be an alias defined in your + ~/.ssh/config. + + The Nix platform type identifier, such as + powerpc-darwin. + + The SSH private key to be used to log in to the + remote machine. Since builds should be non-interactive, this key + should not have a passphrase! + + The maximum load of the remote + machine. This is just the maximum number of jobs that + build-remote.pl will execute in parallel on the + machine. Typically this should be equal to the number of + CPUs. + + + +You should also set up the environment variable +CURRENT_LOAD to point at a file that +build-remote.pl uses to remember how many jobs it +is currently executing remotely. It doesn't look at the actual load +on the remote machine, so if you have multiple instances of Nix +running, they should use the same CURRENT_LOAD +fileAlthough there are probably some race conditions +in the script right now.. Maybe in the future +build-remote.pl will look at the actual remote +load. The load file should exist, so you should just create it as an +empty file initially. + +
+ + + diff --git a/doc/manual/manual.xml b/doc/manual/manual.xml index d2f19f030..84357e7fd 100644 --- a/doc/manual/manual.xml +++ b/doc/manual/manual.xml @@ -11,6 +11,7 @@ + @@ -47,6 +48,7 @@ &installation; &package-management; &writing-nix-expressions; + &build-farm; Command Reference