524 lines
18 KiB
XML
524 lines
18 KiB
XML
<chapter xmlns="http://docbook.org/ns/docbook"
|
||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||
xml:id="chap-projects">
|
||
|
||
<title>Creating and Managing Projects</title>
|
||
|
||
<para>
|
||
Once Hydra is installed and running, the next step is to add
|
||
projects to the build farm. We follow the example of the <link
|
||
xlink:href="http://nixos.org/patchelf.html">Patchelf
|
||
project</link>, a software tool written in C and using the GNU
|
||
Build System (GNU Autoconf and GNU Automake).
|
||
</para>
|
||
|
||
<para>
|
||
Log in to the web interface of your Hydra installation using the
|
||
user name and password you inserted in the database (by default,
|
||
Hydra's web server listens on <link
|
||
xlink:href="http://localhost:3000/"><literal>localhost:3000</literal></link>).
|
||
Then follow the "Create Project" link to create a new project.
|
||
</para>
|
||
|
||
<section>
|
||
<title>Project Information</title>
|
||
|
||
<para>
|
||
A project definition consists of some general information and a
|
||
set of job sets. The general information identifies a project,
|
||
its owner, and current state of activity.
|
||
|
||
Here's what we fill in for the patchelf project:
|
||
|
||
<screen>
|
||
Identifier: patchelf
|
||
</screen>
|
||
|
||
The <emphasis>identifier</emphasis> is the identity of the
|
||
project. It is used in URLs and in the names of build results.
|
||
</para>
|
||
|
||
<para>
|
||
The identifier should be a unique name (it is the primary
|
||
database key for the project table in the database). If you try
|
||
to create a project with an already existing identifier you'd
|
||
get an error message such as:
|
||
|
||
<screen>
|
||
I'm very sorry, but an error occurred:
|
||
DBIx::Class::ResultSet::create(): DBI Exception: DBD::SQLite::st execute failed: column name is not unique(19) at dbdimp.c line 402
|
||
</screen>
|
||
|
||
So try to create the project after entering just the general
|
||
information to figure out if you have chosen a unique name.
|
||
Job sets can be added once the project has been created.
|
||
|
||
<screen>
|
||
Display name: Patchelf
|
||
</screen>
|
||
|
||
The <emphasis>display name</emphasis> is used in menus.
|
||
|
||
<screen>
|
||
Description: A tool for modifying ELF binaries
|
||
</screen>
|
||
|
||
The <emphasis>description</emphasis> is used as short
|
||
documentation of the nature of the project.
|
||
|
||
<screen>
|
||
Owner: eelco
|
||
</screen>
|
||
|
||
The <emphasis>owner</emphasis> of a project can create and edit
|
||
job sets.
|
||
|
||
<screen>
|
||
Enabled: Yes
|
||
</screen>
|
||
|
||
Only if the project is <emphasis>enabled</emphasis> are builds
|
||
performed.
|
||
</para>
|
||
|
||
<para>
|
||
Once created there should be an entry for the project in the
|
||
sidebar. Go to the project page for the <link
|
||
xlink:href="http://localhost:3000/project/patchelf">Patchelf</link>
|
||
project.
|
||
</para>
|
||
</section>
|
||
|
||
<section>
|
||
<title>Job Sets</title>
|
||
|
||
<para>
|
||
A project can consist of multiple <emphasis>job sets</emphasis>
|
||
(hereafter <emphasis>jobsets</emphasis>), separate tasks that
|
||
can be built separately, but may depend on each other (without
|
||
cyclic dependencies, of course). Go to the <link
|
||
xlink:href="http://localhost:3000/project/patchelf/edit">Edit</link>
|
||
page of the Patchelf project and "Add a new jobset" by providing
|
||
the following "Information":
|
||
|
||
<screen>
|
||
Identifier: trunk
|
||
Description: Trunk
|
||
Nix expression: release.nix in input patchelfSrc
|
||
</screen>
|
||
|
||
This states that in order to build the <literal>trunk</literal>
|
||
jobset, the Nix expression in the file
|
||
<filename>release.nix</filename>, which can be obtained from
|
||
input <literal>patchelfSrc</literal>, should be
|
||
evaluated. (We'll have a look at
|
||
<filename>release.nix</filename> later.)
|
||
|
||
</para>
|
||
|
||
<para>
|
||
To realize a job we probably need a number of inputs, which can
|
||
be declared in the table below. As many inputs as required can
|
||
be added. For patchelf we declare the following inputs.
|
||
|
||
<screen>
|
||
patchelfSrc
|
||
'Subversion checkout' https://svn.nixos.org/repos/nix/patchelf/trunk
|
||
|
||
nixpkgs 'Subversion checkout' https://svn.nixos.org/repos/nix/nixpkgs/trunk
|
||
|
||
officialRelease Boolean false
|
||
|
||
system String value "i686-linux"
|
||
</screen>
|
||
</para>
|
||
</section>
|
||
|
||
<section>
|
||
<title>Release Set</title>
|
||
|
||
<!-- TODO -->
|
||
there must be one primary job
|
||
|
||
check the radio button of exactly one job
|
||
|
||
https://svn.nixos.org/repos/nix/nixpkgs/trunk
|
||
</section>
|
||
|
||
<section>
|
||
<title>Building Jobs</title>
|
||
</section>
|
||
|
||
<section>
|
||
<title>Build Recipes</title>
|
||
|
||
<para>
|
||
Build jobs and <emphasis>build recipes</emphasis> for a jobset are
|
||
specified in a text file written in the <link
|
||
xlink:href="http://nixos.org/nix/">Nix language</link>. The
|
||
recipe is actually called a <emphasis>Nix expression</emphasis> in
|
||
Nix parlance. By convention this file is often called
|
||
<filename>release.nix</filename>.
|
||
</para>
|
||
|
||
<para>
|
||
The <filename>release.nix</filename> file is typically kept under
|
||
version control, and the repository that contains it one of the
|
||
build inputs of the corresponding–often called
|
||
<literal>hydraConfig</literal> by convention. The repository for
|
||
that file and the actual file name are specified on the web
|
||
interface of Hydra under the <literal>Setup</literal> tab of the
|
||
jobset's overview page, under the <literal>Nix
|
||
expression</literal> heading. See, for example, the <link
|
||
xlink:href="http://hydra.nixos.org/jobset/patchelf/trunk">jobset
|
||
overview page</link> of the PatchELF project, and <link
|
||
xlink:href="https://svn.nixos.org/repos/nix/patchelf/trunk/release.nix">
|
||
the corresponding Nix file</link>.
|
||
</para>
|
||
|
||
<para>
|
||
Knowledge of the Nix language is recommended, but the example
|
||
below should already give a good idea of how it works:
|
||
</para>
|
||
|
||
<example xml:id='ex-hello'>
|
||
<title><filename>release.nix</filename> file for GNU Hello</title>
|
||
<programlisting>
|
||
{ nixpkgs }: <co xml:id='ex-hello-co-nixpkgs' />
|
||
|
||
let
|
||
pkgs = import nixpkgs {}; <co xml:id='ex-hello-co-import-nixpkgs' />
|
||
|
||
jobs = rec { <co xml:id='ex-hello-co-jobs' />
|
||
|
||
tarball = <co xml:id='ex-hello-co-tarball' />
|
||
{ helloSrc }: <co xml:id='ex-hello-co-tarball-args' />
|
||
|
||
pkgs.releaseTools.sourceTarball { <co xml:id='ex-hello-co-source-tarball' />
|
||
name = "hello-tarball";
|
||
src = helloSrc;
|
||
buildInputs = (with pkgs; [ gettext texLive texinfo ]);
|
||
};
|
||
|
||
build = <co xml:id='ex-hello-co-build' />
|
||
{ tarball ? jobs.tarball {} <co xml:id='ex-hello-co-build-args' />
|
||
, system ? builtins.currentSystem
|
||
}:
|
||
|
||
let pkgs = import nixpkgs { inherit system; }; in
|
||
pkgs.releaseTools.nixBuild { <co xml:id='ex-hello-co-nix-build' />
|
||
name = "hello" ;
|
||
src = tarball;
|
||
configureFlags = [ "--disable-silent-rules" ];
|
||
};
|
||
};
|
||
in
|
||
jobs <co xml:id='ex-hello-co-body' />
|
||
</programlisting>
|
||
</example>
|
||
|
||
<para>
|
||
<xref linkend='ex-hello' /> shows what a
|
||
<filename>release.nix</filename> file for <link
|
||
xlink:href="http://www.gnu.org/software/hello/">GNU Hello</link>
|
||
would you like. GNU Hello is representative of many GNU
|
||
and non-GNU free software projects:
|
||
|
||
<itemizedlist>
|
||
<listitem>it uses the GNU Build System, namely GNU Autoconf,
|
||
and GNU Automake; for users, it means it can be installed
|
||
using the <link
|
||
xlink:href="http://www.gnu.org/prep/standards/html_node/Managing-Releases.html">usual
|
||
<literal>./configure && make install</literal>
|
||
procedure</link>;
|
||
</listitem>
|
||
<listitem>it uses Gettext for internationalization;</listitem>
|
||
<listitem>it has a Texinfo manual, which can be rendered as PDF
|
||
with TeX.</listitem>
|
||
</itemizedlist>
|
||
|
||
The file defines a jobset consisting of two jobs:
|
||
<literal>tarball</literal>, and <literal>build</literal>. It
|
||
contains the following elements (referenced from the figure by
|
||
numbers):
|
||
|
||
<calloutlist>
|
||
|
||
<callout arearefs='ex-hello-co-nixpkgs'>
|
||
<para>
|
||
This specifies a function of one named arguments,
|
||
<varname>nixpkgs</varname>. This function and those
|
||
defined below is called by Hydra. Here the
|
||
<varname>nixpkgs</varname> argument is meant to be a
|
||
checkout of the <link
|
||
xlink:href="http://nixos.org/nixpkgs/">Nixpkgs</link>
|
||
software distribution.
|
||
</para>
|
||
<para>
|
||
Hydra inspects the formal argument list of the function
|
||
(here, the <varname>nixpkgs</varname> argument) and passes
|
||
it the corresponding parameter specified as a build input
|
||
on Hydra's web interface. In this case, the web interface
|
||
should show a <varname>nixpkgs</varname> build input,
|
||
which is a checkout of the Nixpkgs source code repository.
|
||
</para>
|
||
</callout>
|
||
<callout arearefs='ex-hello-co-import-nixpkgs'>
|
||
<para>
|
||
This defines a variable <varname>pkgs</varname> holding
|
||
the set of packages provided by Nixpkgs.
|
||
</para>
|
||
</callout>
|
||
|
||
<callout arearefs='ex-hello-co-jobs'>
|
||
<para>
|
||
This defines a variable holding the two Hydra
|
||
jobs–an <emphasis>attribute set</emphasis> in Nix.
|
||
</para>
|
||
</callout>
|
||
|
||
<callout arearefs='ex-hello-co-tarball'>
|
||
<para>
|
||
This is the definition of the first job, named
|
||
<varname>tarball</varname>. The purpose of this job is to
|
||
produce a usable source code tarball.
|
||
</para>
|
||
</callout>
|
||
<callout arearefs='ex-hello-co-tarball-args'>
|
||
<para>
|
||
The <varname>tarball</varname> takes an additional
|
||
argument called <varname>helloSrc</varname>. Again, this
|
||
argument is passed by Hydra and is meant to be a checkout
|
||
of GNU Hello's source code repository.
|
||
</para>
|
||
</callout>
|
||
<callout arearefs='ex-hello-co-source-tarball'>
|
||
<para>
|
||
The <varname>tarball</varname> job calls the
|
||
<varname>sourceTarball</varname> function, which (roughly)
|
||
runs <command>autoreconf && ./configure &&
|
||
make dist</command> on the checkout. The
|
||
<varname>buildInputs</varname> attribute specifies
|
||
additional software dependencies for the
|
||
job<footnote><para>The package names used in
|
||
<varname>buildInputs</varname>–e.g.,
|
||
<varname>texLive</varname>–are the names of the
|
||
<emphasis>attributes</emphasis> corresponding to these
|
||
packages in Nixpkgs, specifically in the <link
|
||
xlink:href="https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/top-level/all-packages.nix"><filename>all-packages.nix</filename></link>
|
||
file. See the section entitled “Package Naming” in the
|
||
Nixpkgs manual for more information.</para></footnote>.
|
||
</para>
|
||
</callout>
|
||
|
||
<callout arearefs='ex-hello-co-build'>
|
||
<para>
|
||
This is the definition of the <varname>build</varname>
|
||
job, whose purpose is to build Hello from the tarball
|
||
produced above.
|
||
</para>
|
||
</callout>
|
||
<callout arearefs='ex-hello-co-build-args'>
|
||
<para>
|
||
The <varname>build</varname> function takes two additional
|
||
parameter: <varname>tarball</varname>, which is meant to
|
||
be the result of the <varname>tarball</varname> job, and
|
||
<varname>system</varname>, which should be a string
|
||
defining the Nix system type–e.g.,
|
||
<literal>"x86_64-linux"</literal>.
|
||
</para>
|
||
<para>
|
||
Again, these parameters are passed by Hydra when it calls
|
||
<varname>build</varname>. Thus, they must be defined as
|
||
build inputs in Hydra: <varname>tarball</varname> should
|
||
have type <literal>Build Output</literal>, its value being
|
||
the latest output of the <varname>tarball</varname> job,
|
||
and <varname>system</varname> should be a string.
|
||
</para>
|
||
<para>
|
||
The question mark after <literal>tarball</literal> and
|
||
<literal>system</literal> defines default values for these
|
||
arguments, and is only useful for debugging.
|
||
</para>
|
||
</callout>
|
||
<callout arearefs='ex-hello-co-nix-build'>
|
||
<para>
|
||
The <varname>build</varname> job calls the
|
||
<varname>nixBuild</varname> function, which unpacks the
|
||
tarball, then runs <command>./configure && make
|
||
&& make check && make install</command>.
|
||
</para>
|
||
</callout>
|
||
|
||
<callout arearefs='ex-hello-co-body'>
|
||
<para>
|
||
Finally, the set of jobs is returned to Hydra, as a Nix
|
||
attribute set.
|
||
</para>
|
||
</callout>
|
||
</calloutlist>
|
||
</para>
|
||
</section>
|
||
|
||
<section>
|
||
<title>Building from the Command Line</title>
|
||
|
||
<para>
|
||
It is often useful to test a build recipe, for instance before
|
||
it is actually used by Hydra, when testing changes, or when
|
||
debugging a build issue. Since build recipes for Hydra jobsets
|
||
are just plain Nix expressions, they can be evaluated using the
|
||
standard Nix tools.
|
||
</para>
|
||
|
||
<para>
|
||
To evaluate the <varname>tarball</varname> jobset of <xref
|
||
linkend='ex-hello' />, just run:
|
||
|
||
<screen>
|
||
$ nix-build release.nix -A tarball
|
||
</screen>
|
||
|
||
However, doing this with <xref linkend='ex-hello' /> as is will
|
||
probably yield an error like this:
|
||
|
||
<screen>
|
||
error: cannot auto-call a function that has an argument without a default value (`nixpkgs')
|
||
</screen>
|
||
|
||
This is because no value was specified for the
|
||
<varname>nixpkgs</varname> argument of the Nix expression.
|
||
</para>
|
||
|
||
<para>
|
||
This is fixed by providing a default value for that argument in
|
||
the Nix expression, which will allow <command>nix-build</command>
|
||
to auto-call the function: instfead of writing <literal>{ nixpkgs
|
||
}:</literal>, we now write <literal>{ nixpkgs ? <nixpkgs>
|
||
}:</literal>. What it means is that, by default, the
|
||
<varname>nixpkgs</varname> variable will be bound to the absolute
|
||
path of any <filename>nixpkgs</filename> file found in the Nix
|
||
search path. Similarly, a default value for
|
||
<varname>helloSrc</varname> needs to be provided.
|
||
</para>
|
||
|
||
<para>
|
||
Thus, assuming a checkout of Nixpkgs is available under
|
||
<filename>$HOME/src/nixpkgs</filename>, the
|
||
<varname>tarball</varname> jobset can now be evaluated by running:
|
||
|
||
<screen>
|
||
$ nix-build -I ~/src release.nix -A tarball
|
||
</screen>
|
||
|
||
Similarly, the <varname>build</varname> jobset can be evaluated:
|
||
|
||
<screen>
|
||
$ nix-build -I ~/src release.nix -A build
|
||
</screen>
|
||
|
||
The <varname>build</varname> job reuses the result of the
|
||
<varname>tarball</varname> job, rebuilding it only if it needs to.
|
||
</para>
|
||
|
||
</section>
|
||
|
||
<section>
|
||
<title>Adding More Jobs</title>
|
||
|
||
<para>
|
||
<xref linkend='ex-hello' /> illustrates how to write the most
|
||
basic jobs, <varname>tarball</varname> and
|
||
<varname>build</varname>. In practice, much more can be done by
|
||
using features readily provided by Nixpkgs or by creating new jobs
|
||
as customizations of existing jobs.
|
||
</para>
|
||
|
||
<para>
|
||
For instance, test coverage report for projects compiled with GCC
|
||
can be automatically generated using the
|
||
<varname>coverageAnalysis</varname> function provided by Nixpkgs
|
||
instead of <varname>nixBuild</varname>. Back to our GNU Hello
|
||
example, we can define a <varname>coverage</varname> job that
|
||
produces an HTML code coverage report directly readable from the
|
||
corresponding Hydra build page:
|
||
|
||
<programlisting>
|
||
coverage =
|
||
{ tarball ? jobs.tarball {}
|
||
, system ? builtins.currentSystem
|
||
}:
|
||
|
||
let pkgs = import nixpkgs { inherit system; }; in
|
||
pkgs.releaseTools.coverageAnalysis {
|
||
name = "hello" ;
|
||
src = tarball;
|
||
configureFlags = [ "--disable-silent-rules" ];
|
||
}; </programlisting>
|
||
|
||
As can be seen, the only difference compared to
|
||
<varname>build</varname> is the use of
|
||
<varname>coverageAnalysis</varname>.
|
||
</para>
|
||
|
||
<para>
|
||
Nixpkgs provides many more build tools, including the ability to
|
||
run build in virtual machines, which can themselves run another
|
||
GNU/Linux distribution, which allows for the creation of packages
|
||
for these distributions. Please see <link
|
||
xlink:href="https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/build-support/release/">the
|
||
<filename>pkgs/build-support/release</filename> directory</link>
|
||
of Nixpkgs for more. The NixOS manual also contains information
|
||
about whole-system testing in virtual machine.
|
||
</para>
|
||
|
||
<para>
|
||
Now, assume we want to build Hello with an old version of GCC, and
|
||
with different <command>configure</command> flags. A new
|
||
<varname>build_exotic</varname> job can be written that simply
|
||
<emphasis>overrides</emphasis> the relevant arguments passed to
|
||
<varname>nixBuild</varname>:
|
||
|
||
<programlisting>
|
||
build_exotic =
|
||
{ tarball ? jobs.tarball {}
|
||
, system ? builtins.currentSystem
|
||
}:
|
||
|
||
let
|
||
pkgs = import nixpkgs { inherit system; };
|
||
build = jobs.build { inherit tarball system; };
|
||
in
|
||
pkgs.lib.overrideDerivation build (attrs: {
|
||
buildInputs = [ pkgs.gcc33 ];
|
||
preConfigure = "gcc --version";
|
||
configureFlags =
|
||
attrs.configureFlags ++ [ "--disable-nls" ];
|
||
}); </programlisting>
|
||
|
||
The <varname>build_exotic</varname> job reuses
|
||
<varname>build</varname> and overrides some of its arguments: it
|
||
adds a dependency on GCC 3.3, a pre-configure phase that runs
|
||
<command>gcc --version</command>, and adds the
|
||
<literal>--disable-nls</literal> configure flags.
|
||
</para>
|
||
|
||
<para>
|
||
This customization mechanism is very powerful. For instance, it
|
||
can be used to change the way Hello and <emphasis>all</emphasis>
|
||
its dependencies–including the C library and compiler used to
|
||
build it–are built. See the Nixpkgs manual for more.
|
||
</para>
|
||
|
||
</section>
|
||
|
||
|
||
</chapter>
|
||
|
||
<!--
|
||
Local Variables:
|
||
indent-tabs-mode: nil
|
||
ispell-local-dictionary: "american"
|
||
End:
|
||
-->
|