hydra/doc/manual/projects.xml
2011-10-12 21:42:28 +00:00

524 lines
18 KiB
XML
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xml:id="chap-projects">
<title>Creating and Managing Projects</title>
<para>
Once Hydra is installed and running, the next step is to add
projects to the build farm. We follow the example of the <link
xlink:href="http://nixos.org/patchelf.html">Patchelf
project</link>, a software tool written in C and using the GNU
Build System (GNU Autoconf and GNU Automake).
</para>
<para>
Log in to the web interface of your Hydra installation using the
user name and password you inserted in the database (by default,
Hydra's web server listens on <link
xlink:href="http://localhost:3000/"><literal>localhost:3000</literal></link>).
Then follow the "Create Project" link to create a new project.
</para>
<section>
<title>Project Information</title>
<para>
A project definition consists of some general information and a
set of job sets. The general information identifies a project,
its owner, and current state of activity.
Here's what we fill in for the patchelf project:
<screen>
Identifier: patchelf
</screen>
The <emphasis>identifier</emphasis> is the identity of the
project. It is used in URLs and in the names of build results.
</para>
<para>
The identifier should be a unique name (it is the primary
database key for the project table in the database). If you try
to create a project with an already existing identifier you'd
get an error message such as:
<screen>
I'm very sorry, but an error occurred:
DBIx::Class::ResultSet::create(): DBI Exception: DBD::SQLite::st execute failed: column name is not unique(19) at dbdimp.c line 402
</screen>
So try to create the project after entering just the general
information to figure out if you have chosen a unique name.
Job sets can be added once the project has been created.
<screen>
Display name: Patchelf
</screen>
The <emphasis>display name</emphasis> is used in menus.
<screen>
Description: A tool for modifying ELF binaries
</screen>
The <emphasis>description</emphasis> is used as short
documentation of the nature of the project.
<screen>
Owner: eelco
</screen>
The <emphasis>owner</emphasis> of a project can create and edit
job sets.
<screen>
Enabled: Yes
</screen>
Only if the project is <emphasis>enabled</emphasis> are builds
performed.
</para>
<para>
Once created there should be an entry for the project in the
sidebar. Go to the project page for the <link
xlink:href="http://localhost:3000/project/patchelf">Patchelf</link>
project.
</para>
</section>
<section>
<title>Job Sets</title>
<para>
A project can consist of multiple <emphasis>job sets</emphasis>
(hereafter <emphasis>jobsets</emphasis>), separate tasks that
can be built separately, but may depend on each other (without
cyclic dependencies, of course). Go to the <link
xlink:href="http://localhost:3000/project/patchelf/edit">Edit</link>
page of the Patchelf project and "Add a new jobset" by providing
the following "Information":
<screen>
Identifier: trunk
Description: Trunk
Nix expression: release.nix in input patchelfSrc
</screen>
This states that in order to build the <literal>trunk</literal>
jobset, the Nix expression in the file
<filename>release.nix</filename>, which can be obtained from
input <literal>patchelfSrc</literal>, should be
evaluated. (We'll have a look at
<filename>release.nix</filename> later.)
</para>
<para>
To realize a job we probably need a number of inputs, which can
be declared in the table below. As many inputs as required can
be added. For patchelf we declare the following inputs.
<screen>
patchelfSrc
'Subversion checkout' https://svn.nixos.org/repos/nix/patchelf/trunk
nixpkgs 'Subversion checkout' https://svn.nixos.org/repos/nix/nixpkgs/trunk
officialRelease Boolean false
system String value "i686-linux"
</screen>
</para>
</section>
<section>
<title>Release Set</title>
<!-- TODO -->
there must be one primary job
check the radio button of exactly one job
https://svn.nixos.org/repos/nix/nixpkgs/trunk
</section>
<section>
<title>Building Jobs</title>
</section>
<section>
<title>Build Recipes</title>
<para>
Build jobs and <emphasis>build recipes</emphasis> for a jobset are
specified in a text file written in the <link
xlink:href="http://nixos.org/nix/">Nix language</link>. The
recipe is actually called a <emphasis>Nix expression</emphasis> in
Nix parlance. By convention this file is often called
<filename>release.nix</filename>.
</para>
<para>
The <filename>release.nix</filename> file is typically kept under
version control, and the repository that contains it one of the
build inputs of the correspondingoften called
<literal>hydraConfig</literal> by convention. The repository for
that file and the actual file name are specified on the web
interface of Hydra under the <literal>Setup</literal> tab of the
jobset's overview page, under the <literal>Nix
expression</literal> heading. See, for example, the <link
xlink:href="http://hydra.nixos.org/jobset/patchelf/trunk">jobset
overview page</link> of the PatchELF project, and <link
xlink:href="https://svn.nixos.org/repos/nix/patchelf/trunk/release.nix">
the corresponding Nix file</link>.
</para>
<para>
Knowledge of the Nix language is recommended, but the example
below should already give a good idea of how it works:
</para>
<example xml:id='ex-hello'>
<title><filename>release.nix</filename> file for GNU Hello</title>
<programlisting>
{ nixpkgs }: <co xml:id='ex-hello-co-nixpkgs' />
let
pkgs = import nixpkgs {}; <co xml:id='ex-hello-co-import-nixpkgs' />
jobs = rec { <co xml:id='ex-hello-co-jobs' />
tarball = <co xml:id='ex-hello-co-tarball' />
{ helloSrc }: <co xml:id='ex-hello-co-tarball-args' />
pkgs.releaseTools.sourceTarball { <co xml:id='ex-hello-co-source-tarball' />
name = "hello-tarball";
src = helloSrc;
buildInputs = (with pkgs; [ gettext texLive texinfo ]);
};
build = <co xml:id='ex-hello-co-build' />
{ tarball ? jobs.tarball {} <co xml:id='ex-hello-co-build-args' />
, system ? builtins.currentSystem
}:
let pkgs = import nixpkgs { inherit system; }; in
pkgs.releaseTools.nixBuild { <co xml:id='ex-hello-co-nix-build' />
name = "hello" ;
src = tarball;
configureFlags = [ "--disable-silent-rules" ];
};
};
in
jobs <co xml:id='ex-hello-co-body' />
</programlisting>
</example>
<para>
<xref linkend='ex-hello' /> shows what a
<filename>release.nix</filename> file for <link
xlink:href="http://www.gnu.org/software/hello/">GNU Hello</link>
would you like. GNU Hello is representative of many GNU
and non-GNU free software projects:
<itemizedlist>
<listitem>it uses the GNU Build System, namely GNU Autoconf,
and GNU Automake; for users, it means it can be installed
using the <link
xlink:href="http://www.gnu.org/prep/standards/html_node/Managing-Releases.html">usual
<literal>./configure &amp;&amp; make install</literal>
procedure</link>;
</listitem>
<listitem>it uses Gettext for internationalization;</listitem>
<listitem>it has a Texinfo manual, which can be rendered as PDF
with TeX.</listitem>
</itemizedlist>
The file defines a jobset consisting of two jobs:
<literal>tarball</literal>, and <literal>build</literal>. It
contains the following elements (referenced from the figure by
numbers):
<calloutlist>
<callout arearefs='ex-hello-co-nixpkgs'>
<para>
This specifies a function of one named arguments,
<varname>nixpkgs</varname>. This function and those
defined below is called by Hydra. Here the
<varname>nixpkgs</varname> argument is meant to be a
checkout of the <link
xlink:href="http://nixos.org/nixpkgs/">Nixpkgs</link>
software distribution.
</para>
<para>
Hydra inspects the formal argument list of the function
(here, the <varname>nixpkgs</varname> argument) and passes
it the corresponding parameter specified as a build input
on Hydra's web interface. In this case, the web interface
should show a <varname>nixpkgs</varname> build input,
which is a checkout of the Nixpkgs source code repository.
</para>
</callout>
<callout arearefs='ex-hello-co-import-nixpkgs'>
<para>
This defines a variable <varname>pkgs</varname> holding
the set of packages provided by Nixpkgs.
</para>
</callout>
<callout arearefs='ex-hello-co-jobs'>
<para>
This defines a variable holding the two Hydra
jobsan <emphasis>attribute set</emphasis> in Nix.
</para>
</callout>
<callout arearefs='ex-hello-co-tarball'>
<para>
This is the definition of the first job, named
<varname>tarball</varname>. The purpose of this job is to
produce a usable source code tarball.
</para>
</callout>
<callout arearefs='ex-hello-co-tarball-args'>
<para>
The <varname>tarball</varname> takes an additional
argument called <varname>helloSrc</varname>. Again, this
argument is passed by Hydra and is meant to be a checkout
of GNU Hello's source code repository.
</para>
</callout>
<callout arearefs='ex-hello-co-source-tarball'>
<para>
The <varname>tarball</varname> job calls the
<varname>sourceTarball</varname> function, which (roughly)
runs <command>autoreconf &amp;&amp; ./configure &amp;&amp;
make dist</command> on the checkout. The
<varname>buildInputs</varname> attribute specifies
additional software dependencies for the
job<footnote><para>The package names used in
<varname>buildInputs</varname>e.g.,
<varname>texLive</varname>are the names of the
<emphasis>attributes</emphasis> corresponding to these
packages in Nixpkgs, specifically in the <link
xlink:href="https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/top-level/all-packages.nix"><filename>all-packages.nix</filename></link>
file. See the section entitled “Package Naming” in the
Nixpkgs manual for more information.</para></footnote>.
</para>
</callout>
<callout arearefs='ex-hello-co-build'>
<para>
This is the definition of the <varname>build</varname>
job, whose purpose is to build Hello from the tarball
produced above.
</para>
</callout>
<callout arearefs='ex-hello-co-build-args'>
<para>
The <varname>build</varname> function takes two additional
parameter: <varname>tarball</varname>, which is meant to
be the result of the <varname>tarball</varname> job, and
<varname>system</varname>, which should be a string
defining the Nix system typee.g.,
<literal>"x86_64-linux"</literal>.
</para>
<para>
Again, these parameters are passed by Hydra when it calls
<varname>build</varname>. Thus, they must be defined as
build inputs in Hydra: <varname>tarball</varname> should
have type <literal>Build Output</literal>, its value being
the latest output of the <varname>tarball</varname> job,
and <varname>system</varname> should be a string.
</para>
<para>
The question mark after <literal>tarball</literal> and
<literal>system</literal> defines default values for these
arguments, and is only useful for debugging.
</para>
</callout>
<callout arearefs='ex-hello-co-nix-build'>
<para>
The <varname>build</varname> job calls the
<varname>nixBuild</varname> function, which unpacks the
tarball, then runs <command>./configure &amp;&amp; make
&amp;&amp; make check &amp;&amp; make install</command>.
</para>
</callout>
<callout arearefs='ex-hello-co-body'>
<para>
Finally, the set of jobs is returned to Hydra, as a Nix
attribute set.
</para>
</callout>
</calloutlist>
</para>
</section>
<section>
<title>Building from the Command Line</title>
<para>
It is often useful to test a build recipe, for instance before
it is actually used by Hydra, when testing changes, or when
debugging a build issue. Since build recipes for Hydra jobsets
are just plain Nix expressions, they can be evaluated using the
standard Nix tools.
</para>
<para>
To evaluate the <varname>tarball</varname> jobset of <xref
linkend='ex-hello' />, just run:
<screen>
$ nix-build release.nix -A tarball
</screen>
However, doing this with <xref linkend='ex-hello' /> as is will
probably yield an error like this:
<screen>
error: cannot auto-call a function that has an argument without a default value (`nixpkgs')
</screen>
This is because no value was specified for the
<varname>nixpkgs</varname> argument of the Nix expression.
</para>
<para>
This is fixed by providing a default value for that argument in
the Nix expression, which will allow <command>nix-build</command>
to auto-call the function: instfead of writing <literal>{ nixpkgs
}:</literal>, we now write <literal>{ nixpkgs ? &lt;nixpkgs&gt;
}:</literal>. What it means is that, by default, the
<varname>nixpkgs</varname> variable will be bound to the absolute
path of any <filename>nixpkgs</filename> file found in the Nix
search path. Similarly, a default value for
<varname>helloSrc</varname> needs to be provided.
</para>
<para>
Thus, assuming a checkout of Nixpkgs is available under
<filename>$HOME/src/nixpkgs</filename>, the
<varname>tarball</varname> jobset can now be evaluated by running:
<screen>
$ nix-build -I ~/src release.nix -A tarball
</screen>
Similarly, the <varname>build</varname> jobset can be evaluated:
<screen>
$ nix-build -I ~/src release.nix -A build
</screen>
The <varname>build</varname> job reuses the result of the
<varname>tarball</varname> job, rebuilding it only if it needs to.
</para>
</section>
<section>
<title>Adding More Jobs</title>
<para>
<xref linkend='ex-hello' /> illustrates how to write the most
basic jobs, <varname>tarball</varname> and
<varname>build</varname>. In practice, much more can be done by
using features readily provided by Nixpkgs or by creating new jobs
as customizations of existing jobs.
</para>
<para>
For instance, test coverage report for projects compiled with GCC
can be automatically generated using the
<varname>coverageAnalysis</varname> function provided by Nixpkgs
instead of <varname>nixBuild</varname>. Back to our GNU Hello
example, we can define a <varname>coverage</varname> job that
produces an HTML code coverage report directly readable from the
corresponding Hydra build page:
<programlisting>
coverage =
{ tarball ? jobs.tarball {}
, system ? builtins.currentSystem
}:
let pkgs = import nixpkgs { inherit system; }; in
pkgs.releaseTools.coverageAnalysis {
name = "hello" ;
src = tarball;
configureFlags = [ "--disable-silent-rules" ];
}; </programlisting>
As can be seen, the only difference compared to
<varname>build</varname> is the use of
<varname>coverageAnalysis</varname>.
</para>
<para>
Nixpkgs provides many more build tools, including the ability to
run build in virtual machines, which can themselves run another
GNU/Linux distribution, which allows for the creation of packages
for these distributions. Please see <link
xlink:href="https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/build-support/release/">the
<filename>pkgs/build-support/release</filename> directory</link>
of Nixpkgs for more. The NixOS manual also contains information
about whole-system testing in virtual machine.
</para>
<para>
Now, assume we want to build Hello with an old version of GCC, and
with different <command>configure</command> flags. A new
<varname>build_exotic</varname> job can be written that simply
<emphasis>overrides</emphasis> the relevant arguments passed to
<varname>nixBuild</varname>:
<programlisting>
build_exotic =
{ tarball ? jobs.tarball {}
, system ? builtins.currentSystem
}:
let
pkgs = import nixpkgs { inherit system; };
build = jobs.build { inherit tarball system; };
in
pkgs.lib.overrideDerivation build (attrs: {
buildInputs = [ pkgs.gcc33 ];
preConfigure = "gcc --version";
configureFlags =
attrs.configureFlags ++ [ "--disable-nls" ];
}); </programlisting>
The <varname>build_exotic</varname> job reuses
<varname>build</varname> and overrides some of its arguments: it
adds a dependency on GCC 3.3, a pre-configure phase that runs
<command>gcc --version</command>, and adds the
<literal>--disable-nls</literal> configure flags.
</para>
<para>
This customization mechanism is very powerful. For instance, it
can be used to change the way Hello and <emphasis>all</emphasis>
its dependenciesincluding the C library and compiler used to
build itare built. See the Nixpkgs manual for more.
</para>
</section>
</chapter>
<!--
Local Variables:
indent-tabs-mode: nil
ispell-local-dictionary: "american"
End:
-->