-
-
-Delft University of Technology
-
-Department of Software Technology
-
-
-Copyright 2008 Eelco Dolstra
-
-
-
-
Chapter 1. Introduction
-
-
-
1.1. About Hydra
-
-Hydra is a tool for continuous integration testing and software
-release that uses a purely functional language to describe build jobs
-and their dependencies. Continuous integration is a simple technique
-to improve the quality of the software development process. An
-automated system continuously or periodically checks out the source
-code of a project, builds it, runs tests, and produces reports for the
-developers. Thus, various errors that might accidentally be committed
-into the code base are automatically caught. Such a system allows
-more in-depth testing than what developers could feasibly do manually:
-
-
-
-
-
-
Portability testing: The software may need to be built
- and tested on many different platforms. It is infeasible for each
- developer to do this before every commit.
-
-
-
Likewise, many projects have very large test sets (e.g.,
- regression tests in a compiler, or stress tests in a DBMS) that can
- take hours or days to run to completion.
-
-
-
Many kinds of static and dynamic analyses can be performed as
- part of the tests, such as code coverage runs and static analyses.
-
-
-
It may also be necessary to build many different variants
- of the software. For instance, it may be necessary to verify that
- the component builds with various versions of a compiler.
-
-
-
Developers typically use incremental building to test their
- changes (since a full build may take too long), but this is
- unreliable with many build management tools (such as Make), i.e.,
- the result of the incremental build might differ from a full build.
-
-
-
It ensures that the software can be built from the sources under
- revision control. Users of version management systems such as CVS
- and Subversion often forget to place source files under revision
- control.
-
-
-
The machines on which the continuous integration system runs
- ideally provides a clean, well-defined build environment. If this
- environment is administered through proper SCM techniques, then
- builds produced by the system can be reproduced. In contrast,
- developer work environments are typically not under any kind of SCM
- control.
-
-
-
In large projects, developers often work on a particular
- component of the project, and do not build and test the composition
- of those components (again since this is likely to take too long).
- To prevent the phenomenon of ``big bang integration'', where
- components are only tested together near the end of the development
- process, it is important to test components together as soon as
- possible (hence continuous integration).
-
-
-
It allows software to be released by automatically
- creating packages that users can download and install. To do this
- manually represents an often prohibitive amount of work, as one may
- want to produce releases for many different platforms: e.g.,
- installers for Windows and Mac OS X, RPM or Debian packages for
- certain Linux distributions, and so on.
-
-
-
-
-
-
-In its simplest form, a continuous integration tool sits in a loop
-building and releasing software components from a version management
-system. For each component, it performs the following tasks:
-
-
-
-
It obtains the latest version of the component's source code
- from the version management system.
-
-
It runs the component's build process (which presumably includes
- the execution of the component's test set).
-
-
It presents the results of the build (such as error logs and
- releases) to the developers, e.g., by producing a web page.
-
-
-
-
-
-Examples of continuous integration tools include CruiseControl
-Tinderbox, Sisyphus, Anthill and BuildBot. These tools have various
-limitations.
-
-
-
-
They do not manage the build environment. The build
-environment consists of the dependencies necessary to perform a build
-action, e.g., compilers, libraries, etc. Setting up the environment
-is typically done manually, and without proper SCM control (so it may
-be hard to reproduce a build at a later time). Manual management of
-the environment scales poorly in the number of configurations that
-must be supported. For instance, suppose that we want to build a
-component that requires a certain compiler X. We then have to go to
-each machine and install X. If we later need a newer version of X,
-the process must be repeated all over again. An ever worse problem
-occurs if there are conflicting, mutually exclusive versions of the
-dependencies. Thus, simply installing the latest version is not an
-option. Of course, we can install these components in different
-directories and manually pass the appropriate paths to the build
-processes of the various components. But this is a rather tiresome
-and error-prone process.
-
-
They do not easily support variability in software
-systems. A system may have a great deal of build-time
-variability: optional functionality, whether to build a debug or
-production version, different versions of dependencies, and so on.
-(For instance, the Linux kernel now has over 2,600 build-time
-configuration switches.) It is therefore important that a continuous
-integration tool can easily select and test different instances from
-the configuration space of the system to reveal problems, such as
-erroneous interactions between features. In a continuous integration
-setting, it is also useful to test different combinations of versions
-of subsystems, e.g., the head revision of a component against stable
-releases of its dependencies, and vice versa, as this can reveal
-various integration problems.
-
-
-
-
-
-Hydra, is a continuous integration tool that solves these
-problems. It is built on top of the Nix
-package manager, which has a purely functional language for
-describing package build actions and their dependencies. This allows
-the build environment for projects to be produced automatically and
-deterministically, and variability in components to be expressed
-naturally using functions; and as such is an ideal fit for a
-continuous build system.
-
-
-
-
1.2. About Us
-
-Hydra is the successor of the Nix Buildfarm, which was developed in
-tandem with the Nix software deployment system. Nix was originally
-developed at the Department of Information and Computing Sciences,
-Utrecht University by the TraCE project (2003-2008). The project was
-funded by the Software Engineering Research Program Jacquard to
-improve the support for variability in software systems. Funding for
-the development of Nix and Hydra is now provided by the NIRICT LaQuSo
-Build Farm project.
-
-
1.3. About this Manual
-
-This manual tells you how to install the Hydra buildfarm software on
-your own server and how to operate that server using its web
-interface.
-
-
1.4. License
-
-Hydra is free software; you can redistribute it and/or modify it under
-the terms of the GNU Lesser General Public License as published by the
-Free Software Foundation; either version 2.1 of the License, or (at
-your option) any later version. Hydra is distributed in the hope that
-it will be useful, but WITHOUT ANY WARRANTY; without even the implied
-warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
-the GNU Lesser General Public License for more details.
-
-
1.5. Hydra at nixos.org
-
-The nixos.org installation of Hydra runs at
-
-
-
-That installation is used to build software components from the
-Nix,
-NixOS,
-Stratego/XT,
-and related projects.
-
-If you are one of the developers on those projects, it is likely that
-you will be using the NixOS Hydra server in some way. If you need to
-administer automatic builds for your project, you should pull the
-right strings to get an account on the server. This manual will tell
-you how to set up new projects and build jobs within those projects
-and write a release.nix file to describe the build process of your
-project to Hydra. You can skip Chapter 2.
-
-
-
-If your project does not yet have automatic builds within the NixOS
-Hydra server, it may actually be eligible. We are in the process of
-setting up a large buildfarm that should be able to support open
-source and academic software projects. Get in touch.
-
-
1.6. Hydra on your own buildfarm
-
-If you need to run your own Hydra installation, Chapter 2 explains
-how to download and install the system on your own server.
-
-
-
Chapter 2. Installation and Configuration
-
-This chapter explains how to install Hydra on your own buildfarm server.
-
-
2.1. Platform Requirements
-
-To run Hydra you need a Linux server. For small projects, Hydra can
-be run on any reasonably modern machine. For individual projects you
-can even run Hydra on a laptop. However, the charm of a buildfarm
-server is usually that it operates without disturbing the developer's
-working environment and can serve releases over the internet. In
-conjunction you should typically have your source code administered in
-a version management system, such as subversion. Therefore, you will
-probably want to install a server that is connected to the
-internet. To scale up to large and/or many projects, you will need at
-least a considerable amount of diskspace to store builds. Since Hydra
-can schedule multiple simultaneous build jobs, it can be useful to
-have a multi-core machine, and/or attach multiple build machines in a
-network to the central Hydra server.
-
-
-
-Of course we think it is a good idea to use the NixOS Linux distribution for your
-buildfarm server. But this is not a requirement. The Nix software
-deployment system can be installed on any Linux distribution in
-parallel to the regular package management system. Thus, you can use
-Hydra on a Suse, Fedora, or Ubuntu system.
-
-
2.2. Getting Nix
-
-If your server runs NixOS you are all set to continue with
-installation of Hydra. Otherwise you first need to install Nix.
-The latest stable release is Nix 0.12.
-Installation instructions can be found in the
-Nix User's Guide.
-
-
2.3. Installation
-
-To install Hydra, get the most recent 'closure' available from
-
-
-
-The Hydra software is installed in the Nix store, but to run it needs
-a directory for storing the database, logs, and session data. In your
-.bashrc or similar configuration file define:
-
-
- export HYDRA_DATA=/usr/local/hydra
-
-
-and make sure that you actually create that directory. (Of course, you
-can use another directory, but then remember to also substitute that
-name in the commands below.)
-
-
-
-Run hydra_init.pl to initialize the database
-
-
-
-Also start the scheduler, which monitors the source repositories and
-adds builds to the queue, and the runner, which executes jobs in the
-queue.
-
-
-$ hydra_scheduler.pl
-$ hydra_queue_runner.pl
-
-
-Now your Hydra server should be up and running and the web interface
-operational.
-
-
2.5. User Administration
-
-To be able to add jobs and create projects you need to register users
-in the Hydra database. In the current version, the web interface does
-not yet support user administration. Use the following command to add
-a new user to the database.
-
-
-
-where eelco is the username, and foobar the
-password. (Make sure to use other values!)
-
-
-
-To give this user administrator privileges, follow this up by:
-
-
-$ sqlite3 /usr/local/hydra/hydra.sqlite "insert into UserRoles(userName, role) values('eelco', 'admin');"
-
-
-Now you should be able to create a project using the Hydra web interface.
-
-
Chapter 3. Creating Projects
-
-The next step is to add projects to the buildfarm. We follow the
-example of the patchelf project at hydra.nixos.org. Note that the
-error messages provided as feedback by the webinterface can be
-somewhat unfriendly in the current version.
-
-
-
-Login
-to the webinterface of your Hydra installation using
-the username and password you inserted in the database.
-Then follow the
-'Create Project'
-link to create a new project.
-
-
3.1. General information
-
-A project definition consists of some general information and a set of
-jobsets. The general information identifies a project, its owner, and
-current state of activity.
-
-Here's what we fill in for the patchelf project:
-
-
- Identifier: patchelf
-
-
-The identifier is the identity of the project. It is
-used in URLs and in the names of build results.
-
-
-
-The identifier should be a unique name (it is the
-primary database key for the project table in the database). If you
-try to create a project with an already existing identifier you'd get
-an error message such as:
-
-
-I'm very sorry, but an error occurred:
-DBIx::Class::ResultSet::create(): DBI Exception: DBD::SQLite::st execute failed: column name is not unique(19) at dbdimp.c line 402
-
-
-So try to create the project after entering just the general
-information to figure out if you have chosen a unique name.
-Jobsets can be added once the project has been created.
-
-
-
- Display name: Patchelf
-
-
-The display name is used in menus.
-
-
- Description: A tool for modifying ELF binaries
-
-
-The description is used as short documentation of the
-nature of the project.
-
-
- Owner: eelco
-
-
-The owner of a project can create and edit jobsets.
-
-
- Enabled: Yes
-
-
-Only if the project is enabled are builds performed.
-
-
-
-Once created there should be an entry for the project in the
-sidebar. Go to the project page for the Patchelf project.
-
-
3.2. Jobsets
-
-A project can consist of multiple `jobsets', separate tasks that can
-be built separately, but may depend on each other (without cyclic
-dependencies, of course). Go to the
-Edit
-page of the Patchelf project and 'Add a new jobset'
-by providing the following 'Information':
-
-
-
-This states that in order to build the 'Trunk' jobset, the Nix
-expression in the file 'release.nix', which can be obtained from input
-'patchelfSrc', should be evaluated. (We'll have a look at release.nix
-later.)
-
-
-
-To realize a job we probably need a number of inputs, which can be
-declared in the table below. As many inputs as required can be added.
-For patchelf we declare the following inputs.
-
-
-
-there must be one primary job
-
-check the radio button of exactly one job
-
-https://svn.nixos.org/repos/nix/nixpkgs/trunk
-
-
3.3. Building Jobs
-
-
-
3.4. release.nix
-
-
-
-- Voorbeelden van Nix expressies voor Hydra:
-
-
-
-https://svn.nixos.org/repos/nix/patchelf/trunk/release.nix
-https://svn.nixos.org/repos/nix/nix/trunk/release.nix
-https://svn.nixos.org/repos/nix/hydra/trunk/release.nix
-
-
3.5. Building on the command line
-
-Overigens zijn die helemaal niet Hydra-specifiek, je kunt ze gewoon vanaf de
-command line bouwen, bijv. als je een patchelf checkout hebt (met een nixpkgs
-checkout in ../nixpkgs):
-
-$ nix-build release.nix -A rpm_fedora10i386
-
-
-
-
\ No newline at end of file
diff --git a/src/script/hydra_scheduler.pl b/src/script/hydra_evaluator.pl
similarity index 100%
rename from src/script/hydra_scheduler.pl
rename to src/script/hydra_evaluator.pl