Creating and Managing Projects
Once Hydra is installed and running, the next step is to add
projects to the build farm. We follow the example of the Patchelf
project, a software tool written in C and using the GNU
Build System (GNU Autoconf and GNU Automake).
Log in to the web interface of your Hydra installation using the
user name and password you inserted in the database (by default,
Hydra's web server listens on localhost:3000).
Then follow the "Create Project" link to create a new project.
Project Information
A project definition consists of some general information and a
set of job sets. The general information identifies a project,
its owner, and current state of activity.
Here's what we fill in for the patchelf project:
Identifier: patchelf
The identifier is the identity of the
project. It is used in URLs and in the names of build results.
The identifier should be a unique name (it is the primary
database key for the project table in the database). If you try
to create a project with an already existing identifier you'd
get an error message such as:
I'm very sorry, but an error occurred:
DBIx::Class::ResultSet::create(): DBI Exception: DBD::SQLite::st execute failed: column name is not unique(19) at dbdimp.c line 402
So try to create the project after entering just the general
information to figure out if you have chosen a unique name.
Job sets can be added once the project has been created.
Display name: Patchelf
The display name is used in menus.
Description: A tool for modifying ELF binaries
The description is used as short
documentation of the nature of the project.
Owner: eelco
The owner of a project can create and edit
job sets.
Enabled: Yes
Only if the project is enabled are builds
performed.
Once created there should be an entry for the project in the
sidebar. Go to the project page for the Patchelf
project.
Job Sets
A project can consist of multiple job sets
(hereafter jobsets), separate tasks that
can be built separately, but may depend on each other (without
cyclic dependencies, of course). Go to the Edit
page of the Patchelf project and "Add a new jobset" by providing
the following "Information":
Identifier: trunk
Description: Trunk
Nix expression: release.nix in input patchelfSrc
This states that in order to build the trunk
jobset, the Nix expression in the file
release.nix, which can be obtained from
input patchelfSrc, should be
evaluated. (We'll have a look at
release.nix later.)
To realize a job we probably need a number of inputs, which can
be declared in the table below. As many inputs as required can
be added. For patchelf we declare the following inputs.
patchelfSrc
'Subversion checkout' https://svn.nixos.org/repos/nix/patchelf/trunk
nixpkgs 'Subversion checkout' https://svn.nixos.org/repos/nix/nixpkgs/trunk
officialRelease Boolean false
system String value "i686-linux"
Release Set
there must be one primary job
check the radio button of exactly one job
https://svn.nixos.org/repos/nix/nixpkgs/trunk
Building JobsBuild Recipes
Build jobs and build recipes for a jobset are
specified in a text file written in the Nix language. The
recipe is actually called a Nix expression in
Nix parlance. By convention this file is often called
release.nix.
The release.nix file is typically kept under
version control, and the repository that contains it one of the
build inputs of the corresponding–often called
hydraConfig by convention. The repository for
that file and the actual file name are specified on the web
interface of Hydra under the Setup tab of the
jobset's overview page, under the Nix
expression heading. See, for example, the jobset
overview page of the PatchELF project, and
the corresponding Nix file.
Knowledge of the Nix language is recommended, but the example
below should already give a good idea of how it works:
release.nix file for GNU Hello
let
pkgs = import <nixpkgs> {};
jobs = rec {
tarball =
pkgs.releaseTools.sourceTarball {
name = "hello-tarball";
src = <hello>;
buildInputs = (with pkgs; [ gettext texLive texinfo ]);
};
build =
{ system ? builtins.currentSystem }:
let pkgs = import <nixpkgs> { inherit system; }; in
pkgs.releaseTools.nixBuild {
name = "hello";
src = jobs.tarball;
configureFlags = [ "--disable-silent-rules" ];
};
};
in
jobs shows what a
release.nix file for GNU Hello
would you like. GNU Hello is representative of many GNU
and non-GNU free software projects:
it uses the GNU Build System, namely GNU Autoconf,
and GNU Automake; for users, it means it can be installed
using the usual
./configure && make install
procedure;
it uses Gettext for internationalization;it has a Texinfo manual, which can be rendered as PDF
with TeX.
The file defines a jobset consisting of two jobs:
tarball, and build. It
contains the following elements (referenced from the figure by
numbers):
This defines a variable pkgs holding
the set of packages provided by Nixpkgs.
Since nixpkgs appears in angle brackets,
there must be a build input of that name in the Nix search
path. In this case, the web interface should show a
nixpkgs build input, which is a checkout
of the Nixpkgs source code repository; Hydra then adds this
and other build inputs to the Nix search path when
evaluating release.nix.
This defines a variable holding the two Hydra
jobs–an attribute set in Nix.
This is the definition of the first job, named
tarball. The purpose of this job is to
produce a usable source code tarball.
The tarball jobs expects a
hello build input to be available in the
Nix search path. Again, this input is passed by Hydra and
is meant to be a checkout of GNU Hello's source code
repository.
The tarball job calls the
sourceTarball function, which (roughly)
runs autoreconf && ./configure &&
make dist on the checkout. The
buildInputs attribute specifies
additional software dependencies for the
jobThe package names used in
buildInputs–e.g.,
texLive–are the names of the
attributes corresponding to these
packages in Nixpkgs, specifically in the all-packages.nix
file. See the section entitled “Package Naming” in the
Nixpkgs manual for more information..
This is the definition of the build
job, whose purpose is to build Hello from the tarball
produced above.
The build function takes one
parameter, system, which should be a string
defining the Nix system type–e.g.,
"x86_64-linux". Additionally, it refers
to jobs.tarball, seen above.
Hydra inspects the formal argument list of the function
(here, the system argument) and passes it
the corresponding parameter specified as a build input on
Hydra's web interface. Here, system is
passed by Hydra when it calls build.
Thus, it must be defined as a build input of type string in
Hydra, which could take one of several values.
The question mark after system defines
the default value for this argument, and is only useful when
debugging locally.
The build job calls the
nixBuild function, which unpacks the
tarball, then runs ./configure && make
&& make check && make install.
Finally, the set of jobs is returned to Hydra, as a Nix
attribute set.
Building from the Command Line
It is often useful to test a build recipe, for instance before
it is actually used by Hydra, when testing changes, or when
debugging a build issue. Since build recipes for Hydra jobsets
are just plain Nix expressions, they can be evaluated using the
standard Nix tools.
To evaluate the tarball jobset of , just run:
$ nix-build release.nix -A tarball
However, doing this with as is will
probably yield an error like this:
error: user-thrown exception: file `hello' was not found in the Nix search path (add it using $NIX_PATH or -I)
The error is self-explanatory. Assuming
$HOME/src/hello points to a checkout of
Hello, this can be fixed this way:
$ nix-build -I ~/src release.nix -A tarball
Similarly, the build jobset can be evaluated:
$ nix-build -I ~/src release.nix -A build
The build job reuses the result of the
tarball job, rebuilding it only if it needs to.
Adding More Jobs illustrates how to write the most
basic jobs, tarball and
build. In practice, much more can be done by
using features readily provided by Nixpkgs or by creating new jobs
as customizations of existing jobs.
For instance, test coverage report for projects compiled with GCC
can be automatically generated using the
coverageAnalysis function provided by Nixpkgs
instead of nixBuild. Back to our GNU Hello
example, we can define a coverage job that
produces an HTML code coverage report directly readable from the
corresponding Hydra build page:
coverage =
{ system ? builtins.currentSystem }:
let pkgs = import nixpkgs { inherit system; }; in
pkgs.releaseTools.coverageAnalysis {
name = "hello";
src = jobs.tarball;
configureFlags = [ "--disable-silent-rules" ];
};
As can be seen, the only difference compared to
build is the use of
coverageAnalysis.
Nixpkgs provides many more build tools, including the ability to
run build in virtual machines, which can themselves run another
GNU/Linux distribution, which allows for the creation of packages
for these distributions. Please see the
pkgs/build-support/release directory
of Nixpkgs for more. The NixOS manual also contains information
about whole-system testing in virtual machine.
Now, assume we want to build Hello with an old version of GCC, and
with different configure flags. A new
build_exotic job can be written that simply
overrides the relevant arguments passed to
nixBuild:
build_exotic =
{ system ? builtins.currentSystem }:
let
pkgs = import nixpkgs { inherit system; };
build = jobs.build { inherit system; };
in
pkgs.lib.overrideDerivation build (attrs: {
buildInputs = [ pkgs.gcc33 ];
preConfigure = "gcc --version";
configureFlags =
attrs.configureFlags ++ [ "--disable-nls" ];
});
The build_exotic job reuses
build and overrides some of its arguments: it
adds a dependency on GCC 3.3, a pre-configure phase that runs
gcc --version, and adds the
--disable-nls configure flags.
This customization mechanism is very powerful. For instance, it
can be used to change the way Hello and all
its dependencies–including the C library and compiler used to
build it–are built. See the Nixpkgs manual for more.