hydra/doc/manual/projects.xml

482 lines
17 KiB
XML
Raw Normal View History

<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="chap-projects">
<title>Creating and Managing Projects</title>
<para>
Once Hydra is installed and running, the next step is to add
projects to the build farm. We follow the example of the <link
xlink:href="http://nixos.org/patchelf.html">Patchelf
project</link>, a software tool written in C and using the GNU
Build System (GNU Autoconf and GNU Automake).
</para>
<para>
Log in to the web interface of your Hydra installation using the
user name and password you inserted in the database (by default,
Hydra's web server listens on <link
xlink:href="http://localhost:3000/"><literal>localhost:3000</literal></link>).
Then follow the "Create Project" link to create a new project.
</para>
<section>
<title>Project Information</title>
<para>
A project definition consists of some general information and a
set of job sets. The general information identifies a project,
its owner, and current state of activity.
Here's what we fill in for the patchelf project:
2012-05-11 22:25:56 +00:00
<screen>
Identifier: patchelf
</screen>
The <emphasis>identifier</emphasis> is the identity of the
project. It is used in URLs and in the names of build results.
</para>
<para>
The identifier should be a unique name (it is the primary
database key for the project table in the database). If you try
to create a project with an already existing identifier you'd
get an error message such as:
<screen>
2013-01-22 13:41:02 +00:00
I'm very sorry, but an error occurred:
DBIx::Class::ResultSet::create(): DBI Exception: DBD::SQLite::st execute failed: column name is not unique(19) at dbdimp.c line 402
</screen>
So try to create the project after entering just the general
information to figure out if you have chosen a unique name.
Job sets can be added once the project has been created.
<screen>
2012-05-11 22:25:56 +00:00
Display name: Patchelf
</screen>
The <emphasis>display name</emphasis> is used in menus.
<screen>
2012-05-11 22:25:56 +00:00
Description: A tool for modifying ELF binaries
</screen>
The <emphasis>description</emphasis> is used as short
documentation of the nature of the project.
<screen>
2012-05-11 22:25:56 +00:00
Owner: eelco
</screen>
The <emphasis>owner</emphasis> of a project can create and edit
job sets.
<screen>
2012-05-11 22:25:56 +00:00
Enabled: Yes
</screen>
Only if the project is <emphasis>enabled</emphasis> are builds
performed.
</para>
<para>
Once created there should be an entry for the project in the
sidebar. Go to the project page for the <link
xlink:href="http://localhost:3000/project/patchelf">Patchelf</link>
project.
</para>
</section>
<section>
<title>Job Sets</title>
<para>
A project can consist of multiple <emphasis>job sets</emphasis>
(hereafter <emphasis>jobsets</emphasis>), separate tasks that
can be built separately, but may depend on each other (without
cyclic dependencies, of course). Go to the <link
xlink:href="http://localhost:3000/project/patchelf/edit">Edit</link>
page of the Patchelf project and "Add a new jobset" by providing
the following "Information":
<screen>
2012-05-11 22:25:56 +00:00
Identifier: trunk
Description: Trunk
Nix expression: release.nix in input patchelfSrc
</screen>
This states that in order to build the <literal>trunk</literal>
jobset, the Nix expression in the file
<filename>release.nix</filename>, which can be obtained from
input <literal>patchelfSrc</literal>, should be
evaluated. (We'll have a look at
<filename>release.nix</filename> later.)
</para>
<para>
To realize a job we probably need a number of inputs, which can
be declared in the table below. As many inputs as required can
be added. For patchelf we declare the following inputs.
<screen>
2013-01-22 13:41:02 +00:00
patchelfSrc
2016-10-12 16:00:30 +00:00
'Git checkout' https://github.com/NixOS/patchelf
2016-10-12 16:00:30 +00:00
nixpkgs 'Git checkout' https://github.com/NixOS/nixpkgs
2012-05-11 22:25:56 +00:00
officialRelease Boolean false
2013-01-22 13:41:02 +00:00
system String value "i686-linux"
</screen>
</para>
</section>
<section>
<title>Building Jobs</title>
</section>
<section>
<title>Build Recipes</title>
<para>
Build jobs and <emphasis>build recipes</emphasis> for a jobset are
specified in a text file written in the <link
xlink:href="http://nixos.org/nix/">Nix language</link>. The
recipe is actually called a <emphasis>Nix expression</emphasis> in
Nix parlance. By convention this file is often called
<filename>release.nix</filename>.
</para>
<para>
The <filename>release.nix</filename> file is typically kept under
version control, and the repository that contains it one of the
build inputs of the correspondingoften called
<literal>hydraConfig</literal> by convention. The repository for
that file and the actual file name are specified on the web
interface of Hydra under the <literal>Setup</literal> tab of the
jobset's overview page, under the <literal>Nix
expression</literal> heading. See, for example, the <link
xlink:href="http://hydra.nixos.org/jobset/patchelf/trunk">jobset
overview page</link> of the PatchELF project, and <link
2016-10-12 16:00:30 +00:00
xlink:href="https://github.com/NixOS/patchelf/blob/master/release.nix">
the corresponding Nix file</link>.
</para>
<para>
Knowledge of the Nix language is recommended, but the example
below should already give a good idea of how it works:
</para>
<example xml:id='ex-hello'>
<title><filename>release.nix</filename> file for GNU Hello</title>
<programlisting>
let
pkgs = import &lt;nixpkgs&gt; {}; <co xml:id='ex-hello-co-import-nixpkgs' />
jobs = rec { <co xml:id='ex-hello-co-jobs' />
tarball = <co xml:id='ex-hello-co-tarball' />
pkgs.releaseTools.sourceTarball { <co xml:id='ex-hello-co-source-tarball' />
name = "hello-tarball";
src = &lt;hello&gt;; <co xml:id='ex-hello-co-tarball-args' />
buildInputs = (with pkgs; [ gettext texLive texinfo ]);
};
build = <co xml:id='ex-hello-co-build' />
{ system ? builtins.currentSystem }: <co xml:id='ex-hello-co-build-args' />
let pkgs = import &lt;nixpkgs&gt; { inherit system; }; in
pkgs.releaseTools.nixBuild { <co xml:id='ex-hello-co-nix-build' />
2012-05-11 22:25:56 +00:00
name = "hello";
src = jobs.tarball;
2011-10-12 14:19:57 +00:00
configureFlags = [ "--disable-silent-rules" ];
};
};
in
2011-10-12 14:20:06 +00:00
jobs <co xml:id='ex-hello-co-body' />
</programlisting>
</example>
<para>
<xref linkend='ex-hello' /> shows what a
<filename>release.nix</filename> file for <link
xlink:href="http://www.gnu.org/software/hello/">GNU Hello</link>
would you like. GNU Hello is representative of many GNU
and non-GNU free software projects:
<itemizedlist>
<listitem>it uses the GNU Build System, namely GNU Autoconf,
and GNU Automake; for users, it means it can be installed
using the <link
xlink:href="http://www.gnu.org/prep/standards/html_node/Managing-Releases.html">usual
<literal>./configure &amp;&amp; make install</literal>
procedure</link>;
</listitem>
<listitem>it uses Gettext for internationalization;</listitem>
<listitem>it has a Texinfo manual, which can be rendered as PDF
with TeX.</listitem>
</itemizedlist>
The file defines a jobset consisting of two jobs:
<literal>tarball</literal>, and <literal>build</literal>. It
contains the following elements (referenced from the figure by
numbers):
<calloutlist>
<callout arearefs='ex-hello-co-import-nixpkgs'>
<para>
This defines a variable <varname>pkgs</varname> holding
the set of packages provided by <link
xlink:href="http://nixos.org/nixpkgs/">Nixpkgs</link>.
</para>
<para>
Since <varname>nixpkgs</varname> appears in angle brackets,
there must be a build input of that name in the Nix search
path. In this case, the web interface should show a
<varname>nixpkgs</varname> build input, which is a checkout
of the Nixpkgs source code repository; Hydra then adds this
and other build inputs to the Nix search path when
evaluating <filename>release.nix</filename>.
</para>
</callout>
<callout arearefs='ex-hello-co-jobs'>
<para>
This defines a variable holding the two Hydra
jobsan <emphasis>attribute set</emphasis> in Nix.
</para>
</callout>
<callout arearefs='ex-hello-co-tarball'>
<para>
This is the definition of the first job, named
<varname>tarball</varname>. The purpose of this job is to
produce a usable source code tarball.
</para>
</callout>
<callout arearefs='ex-hello-co-tarball-args'>
<para>
The <varname>tarball</varname> jobs expects a
<varname>hello</varname> build input to be available in the
Nix search path. Again, this input is passed by Hydra and
is meant to be a checkout of GNU Hello's source code
repository.
</para>
</callout>
<callout arearefs='ex-hello-co-source-tarball'>
<para>
The <varname>tarball</varname> job calls the
<varname>sourceTarball</varname> function, which (roughly)
runs <command>autoreconf &amp;&amp; ./configure &amp;&amp;
make dist</command> on the checkout. The
<varname>buildInputs</varname> attribute specifies
2011-10-12 21:42:28 +00:00
additional software dependencies for the
job<footnote><para>The package names used in
<varname>buildInputs</varname>e.g.,
<varname>texLive</varname>are the names of the
<emphasis>attributes</emphasis> corresponding to these
packages in Nixpkgs, specifically in the <link
2016-10-12 16:00:30 +00:00
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix"><filename>all-packages.nix</filename></link>
2011-10-12 21:42:28 +00:00
file. See the section entitled “Package Naming” in the
Nixpkgs manual for more information.</para></footnote>.
</para>
</callout>
<callout arearefs='ex-hello-co-build'>
<para>
This is the definition of the <varname>build</varname>
job, whose purpose is to build Hello from the tarball
produced above.
</para>
</callout>
<callout arearefs='ex-hello-co-build-args'>
<para>
The <varname>build</varname> function takes one
parameter, <varname>system</varname>, which should be a string
defining the Nix system typee.g.,
<literal>"x86_64-linux"</literal>. Additionally, it refers
to <varname>jobs.tarball</varname>, seen above.
</para>
<para>
Hydra inspects the formal argument list of the function
(here, the <varname>system</varname> argument) and passes it
the corresponding parameter specified as a build input on
Hydra's web interface. Here, <varname>system</varname> is
passed by Hydra when it calls <varname>build</varname>.
Thus, it must be defined as a build input of type string in
Hydra, which could take one of several values.
</para>
<para>
The question mark after <literal>system</literal> defines
the default value for this argument, and is only useful when
debugging locally.
</para>
</callout>
<callout arearefs='ex-hello-co-nix-build'>
<para>
The <varname>build</varname> job calls the
<varname>nixBuild</varname> function, which unpacks the
tarball, then runs <command>./configure &amp;&amp; make
&amp;&amp; make check &amp;&amp; make install</command>.
</para>
</callout>
<callout arearefs='ex-hello-co-body'>
<para>
Finally, the set of jobs is returned to Hydra, as a Nix
attribute set.
</para>
</callout>
</calloutlist>
</para>
</section>
<section>
<title>Building from the Command Line</title>
<para>
It is often useful to test a build recipe, for instance before
it is actually used by Hydra, when testing changes, or when
debugging a build issue. Since build recipes for Hydra jobsets
are just plain Nix expressions, they can be evaluated using the
standard Nix tools.
</para>
<para>
To evaluate the <varname>tarball</varname> jobset of <xref
linkend='ex-hello' />, just run:
2012-05-11 22:25:56 +00:00
<screen>
$ nix-build release.nix -A tarball
</screen>
However, doing this with <xref linkend='ex-hello' /> as is will
probably yield an error like this:
2012-05-11 22:25:56 +00:00
<screen>
error: user-thrown exception: file `hello' was not found in the Nix search path (add it using $NIX_PATH or -I)
2012-05-11 22:25:56 +00:00
</screen>
The error is self-explanatory. Assuming
<filename>$HOME/src/hello</filename> points to a checkout of
Hello, this can be fixed this way:
2012-05-11 22:25:56 +00:00
<screen>
$ nix-build -I ~/src release.nix -A tarball
</screen>
Similarly, the <varname>build</varname> jobset can be evaluated:
2012-05-11 22:25:56 +00:00
<screen>
$ nix-build -I ~/src release.nix -A build
</screen>
The <varname>build</varname> job reuses the result of the
<varname>tarball</varname> job, rebuilding it only if it needs to.
</para>
2013-01-22 13:41:02 +00:00
2011-10-12 14:54:20 +00:00
</section>
<section>
<title>Adding More Jobs</title>
<para>
<xref linkend='ex-hello' /> illustrates how to write the most
basic jobs, <varname>tarball</varname> and
<varname>build</varname>. In practice, much more can be done by
using features readily provided by Nixpkgs or by creating new jobs
as customizations of existing jobs.
</para>
<para>
For instance, test coverage report for projects compiled with GCC
can be automatically generated using the
<varname>coverageAnalysis</varname> function provided by Nixpkgs
instead of <varname>nixBuild</varname>. Back to our GNU Hello
example, we can define a <varname>coverage</varname> job that
produces an HTML code coverage report directly readable from the
corresponding Hydra build page:
2012-05-11 22:25:56 +00:00
<programlisting>
coverage =
{ system ? builtins.currentSystem }:
2012-05-11 22:25:56 +00:00
let pkgs = import nixpkgs { inherit system; }; in
pkgs.releaseTools.coverageAnalysis {
name = "hello";
src = jobs.tarball;
2012-05-11 22:25:56 +00:00
configureFlags = [ "--disable-silent-rules" ];
};
</programlisting>
2011-10-12 14:54:20 +00:00
As can be seen, the only difference compared to
<varname>build</varname> is the use of
<varname>coverageAnalysis</varname>.
</para>
<para>
Nixpkgs provides many more build tools, including the ability to
run build in virtual machines, which can themselves run another
GNU/Linux distribution, which allows for the creation of packages
for these distributions. Please see <link
2016-10-12 16:00:30 +00:00
xlink:href="https://github.com/NixOS/nixpkgs/tree/master/pkgs/build-support/release">the
2011-10-12 14:54:20 +00:00
<filename>pkgs/build-support/release</filename> directory</link>
of Nixpkgs for more. The NixOS manual also contains information
about whole-system testing in virtual machine.
</para>
<para>
Now, assume we want to build Hello with an old version of GCC, and
with different <command>configure</command> flags. A new
<varname>build_exotic</varname> job can be written that simply
<emphasis>overrides</emphasis> the relevant arguments passed to
<varname>nixBuild</varname>:
2012-05-11 22:25:56 +00:00
<programlisting>
build_exotic =
{ system ? builtins.currentSystem }:
2012-05-11 22:25:56 +00:00
let
pkgs = import nixpkgs { inherit system; };
build = jobs.build { inherit system; };
2012-05-11 22:25:56 +00:00
in
pkgs.lib.overrideDerivation build (attrs: {
buildInputs = [ pkgs.gcc33 ];
preConfigure = "gcc --version";
configureFlags =
attrs.configureFlags ++ [ "--disable-nls" ];
});
</programlisting>
2011-10-12 14:54:20 +00:00
The <varname>build_exotic</varname> job reuses
<varname>build</varname> and overrides some of its arguments: it
adds a dependency on GCC 3.3, a pre-configure phase that runs
<command>gcc --version</command>, and adds the
<literal>--disable-nls</literal> configure flags.
</para>
<para>
This customization mechanism is very powerful. For instance, it
can be used to change the way Hello and <emphasis>all</emphasis>
its dependenciesincluding the C library and compiler used to
build itare built. See the Nixpkgs manual for more.
</para>
</section>
2011-10-12 14:54:20 +00:00
<xi:include href="declarative-projects.xml" />
2011-10-12 14:54:20 +00:00
</chapter>
<!--
Local Variables:
indent-tabs-mode: nil
ispell-local-dictionary: "american"
End:
-->