The Hydra Buildfarm User Manual

Draft (Version 0.1)

Eelco Dolstra and Eelco Visser

Delft University of Technology

Department of Software Technology

Chapter 1. Introduction

1.1. About Hydra

Hydra is a buildfarm system based on the Nix software deployment system.

... advantages ...

1.2. About Us

Hydra is the successor of the Nix Buildfarm, which was developed in tandem with the Nix software deployment system. Nix was originally developed at the Department of Information and Computing Sciences, Utrecht University by the TraCE project (2003-2008). The project was funded by the Software Engineering Research Program Jacquard to improve the support for variability in software systems. Funding for the development of Nix and Hydra is now provided by the NIRICT LaQuSo Build Farm project.

1.3. About this Manual

This manual tells you how to install the Hydra buildfarm software on your own server and how to operate that server using its web interface.

1.4. License

Hydra is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. Hydra is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

1.5. Hydra at nixos.org

The nixos.org installation of Hydra runs at

http://hydra.nixos.org

That installation is used to build software components from the Nix, NixOS, Stratego/XT, and related projects. If you are one of the developers on those projects, it is likely that you will be using the NixOS Hydra server in some way. If you need to administer automatic builds for your project, you should pull the right strings to get an account on the server. This manual will tell you how to set up new projects and build jobs within those projects and write a release.nix file to describe the build process of your project to Hydra. You can skip Chapter 2.

If your project does not yet have automatic builds within the NixOS Hydra server, it may actually be eligible. We are in the process of setting up a large buildfarm that should be able to support open source and academic software projects. Get in touch.

1.6. Hydra on your own buildfarm

If you need to run your own Hydra installation, Chapter 2 explains how to download and install the system on your own server.

Chapter 2. Installation and Configuration

This chapter explains how to install Hydra on your own buildfarm server.

2.1. Platform Requirements

To run Hydra you need a Linux server. For small projects, Hydra can be run on any reasonably modern machine. For individual projects you can even run Hydra on a laptop. However, the charm of a buildfarm server is usually that it operates without disturbing the developer's working environment and can serve releases over the internet. In conjunction you should typically have your source code administered in a version management system, such as subversion. Therefore, you will probably want to install a server that is connected to the internet. To scale up to large and/or many projects, you will need at least a considerable amount of diskspace to store builds. Since Hydra can schedule multiple simultaneous build jobs, it can be useful to have a multi-core machine, and/or attach multiple build machines in a network to the central Hydra server.

Of course we think it is a good idea to use the NixOS Linux distribution for your buildfarm server. But this is not a requirement. The Nix software deployment system can be installed on any Linux distribution in parallel to the regular package management system. Thus, you can use Hydra on a Suse, Fedora, or Ubuntu system.

Hydra on Windows??

2.2. Getting Nix

If your server runs NixOS you are all set to continue with installation of Hydra. Otherwise you first need to install Nix. The latest stable release is Nix 0.12. Installation instructions can be found in the Nix User's Guide.

Why Nix ...

2.3. Installation

To install Hydra, get the most recent 'closure' available from

http://hydra.nixos.org/releases/hydra/unstable

And follow the instructions that are revealed by clicking [help].

$ gunzip < hydra-build.closure.gz | nix-store --import

This unpacks the closure and imports its components into the Nix store.

$ nix-env -i /nix/store/...-hydra-build

This makes the tools in the Hydra package available in your Nix user environment.

Command completion should then reveal a number of tools related to hydra installed:

$ hydra_<tab>
hydra_build.pl         hydra_fastcgi.pl       hydra_scheduler.pl
hydra_cgi.pl           hydra_init.pl          hydra_server.pl
hydra_create.pl        hydra_queue_runner.pl  hydra_test.pl

2.4. Configuration

The Hydra software is installed in the Nix store, but to run it needs a directory for storing the database, logs, and session data. In your .bashrc or similar configuration file define:

 export HYDRA_DATA=/usr/local/hydra

and make sure that you actually create that directory. (Of course, you can use another directory, but then remember to also substitute that name in the commands below.)

Run hydra_init.pl to initialize the database

$ hydra_init.pl

Run hydra_server.pl to start the webserver at http://localhost:3000

$ hydra_server.pl

Also start the scheduler, which monitors the source repositories and adds builds to the queue, and the runner, which executes jobs in the queue.

$ hydra_scheduler.pl
$ hydra_queue_runner.pl

Now your Hydra server should be up and running and the web interface operational.

2.5. User Administration

To be able to add jobs and create projects you need to register users in the Hydra database. In the current version, the web interface does not yet support user administration. Use the following command to add a new user to the database.

$ sqlite3 /usr/local/hydra/hydra.sqlite "insert into Users(userName, emailAddress, password) values('eelco', 'blablah@example.org', '$(echo -n foobar | sha1sum | cut -c1-40)');"

where eelco is the username, and foobar the password. (Make sure to use other values!)

To give this user administrator privileges, follow this up by:

$ sqlite3 /usr/local/hydra/hydra.sqlite "insert into UserRoles(userName, role) values('eelco', 'admin');"

Now you should be able to create a project using the Hydra web interface.

Chapter 3. Creating Projects

The next step is to add projects to the buildfarm. We follow the example of the patchelf project at hydra.nixos.org. Note that the error messages provided as feedback by the webinterface can be somewhat unfriendly in the current version.

Login to the webinterface of your Hydra installation using the username and password you inserted in the database. Then follow the 'Create Project' link to create a new project.

3.1. General information

A project definition consists of some general information and a set of jobsets. We start with the general information. The general information of a project are mainly its name and owner. Here's what we fill in for the patchelf project:

  Identifier:    patchelf
  Display name:  Patchelf
  Description:   A tool for modifying ELF binaries
  Owner:         eelco
  Enabled:       Yes

The Identifier should be a unique name (it is the primary database key for the project table in the database). If you try to create a project with an already existing identifier you'd get an error message such as:

I'm very sorry, but an error occurred: 
DBIx::Class::ResultSet::create(): DBI Exception: DBD::SQLite::st execute failed: column name is not unique(19) at dbdimp.c line 402

So try to create the project after entering just the general information to figure out if you have chosen a unique name. Jobsets can be added once the project has been created.

Once create there should be an entry for the project in the sidebar. Go to the project page for the Patchelf project.

Jobsets

A project can consist of multiple `jobsets', separate tasks that can be built separately, but may depend on each other (without cyclic dependencies, of course). Go to the Edit page of the Patchelf project and 'Add a new jobset' by providing the following 'Information':

  Identifier:     trunk
  Description:    Trunk
  Nix expression: release.nix in input patchelfSrc

This states that in order to build the 'Trunk' jobset, the Nix expression in the file 'release.nix', which can be obtained from input 'patchelfSrc', should be evaluated. (We'll have a look at release.nix later.)

To realize a job we probably need a number of inputs, which can be declared in the table below. As many inputs as required can be added. For patchelf we declare the following inputs.

  nixpkgs 'CVS checkout' https://svn.nixos.org/repos/nix/nixpkgs/trunk

  officialRelease   Boolean false

  patchelfSrc Subversion checkout https://svn.nixos.org/repos/nix/patchelf/trunk

  system String value "i686-linux"

nixpkgs officialRelease patchelfSrc system

Release Set

there must be one primary job check the radio button of exactly one job https://svn.nixos.org/repos/nix/nixpkgs/trunk

Building Jobs

release.nix

- Voorbeelden van Nix expressies voor Hydra: https://svn.nixos.org/repos/nix/patchelf/trunk/release.nix https://svn.nixos.org/repos/nix/nix/trunk/release.nix https://svn.nixos.org/repos/nix/hydra/trunk/release.nix

Building on the command line

Overigens zijn die helemaal niet Hydra-specifiek, je kunt ze gewoon vanaf de command line bouwen, bijv. als je een patchelf checkout hebt (met een nixpkgs checkout in ../nixpkgs): $ nix-build release.nix -A rpm_fedora10i386