From 6415d25b9322ee4a77f28672a19c654371f80347 Mon Sep 17 00:00:00 2001 From: Eelco Visser Date: Mon, 1 Dec 2008 23:20:42 +0000 Subject: [PATCH] ripped intro from the SCP paper and manualized it --- doc/manual/manual.html | 155 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 138 insertions(+), 17 deletions(-) diff --git a/doc/manual/manual.html b/doc/manual/manual.html index 7ca55eb3..4929cdbf 100644 --- a/doc/manual/manual.html +++ b/doc/manual/manual.html @@ -30,18 +30,148 @@ Copyright 2008 Eelco Dolstra

1.1. About Hydra

-Hydra is a buildfarm system based on the Nix software deployment -system. +Hydra is a tool for continuous integration testing and software +release that uses a purely functional language to describe build jobs +and their dependencies. Continuous integration is a simple technique +to improve the quality of the software development process. An +automated system continuously or periodically checks out the source +code of a project, builds it, runs tests, and produces reports for the +developers. Thus, various errors that might accidentally be committed +into the code base are automatically caught. Such a system allows +more in-depth testing than what developers could feasibly do manually:

-... advantages ... +

    +
  1. Portability testing: The software may need to be built + and tested on many different platforms. It is infeasible for each + developer to do this before every commit. +

    + +

  2. Likewise, many projects have very large test sets (e.g., + regression tests in a compiler, or stress tests in a DBMS) that can + take hours or days to run to completion. +

    + +

  3. Many kinds of static and dynamic analyses can be performed as + part of the tests, such as code coverage runs and static analyses. +

    + +

  4. It may also be necessary to build many different variants + of the software. For instance, it may be necessary to verify that + the component builds with various versions of a compiler. +

    + +

  5. Developers typically use incremental building to test their + changes (since a full build may take too long), but this is + unreliable with many build management tools (such as Make), i.e., + the result of the incremental build might differ from a full build. +

    + +

  6. It ensures that the software can be built from the sources under + revision control. Users of version management systems such as CVS + and Subversion often forget to place source files under revision + control. +

    + +

  7. The machines on which the continuous integration system runs + ideally provides a clean, well-defined build environment. If this + environment is administered through proper SCM techniques, then + builds produced by the system can be reproduced. In contrast, + developer work environments are typically not under any kind of SCM + control. +

    + +

  8. In large projects, developers often work on a particular + component of the project, and do not build and test the composition + of those components (again since this is likely to take too long). + To prevent the phenomenon of ``big bang integration'', where + components are only tested together near the end of the development + process, it is important to test components together as soon as + possible (hence continuous integration). +

    + +

  9. It allows software to be released by automatically + creating packages that users can download and install. To do this + manually represents an often prohibitive amount of work, as one may + want to produce releases for many different platforms: e.g., + installers for Windows and Mac OS X, RPM or Debian packages for + certain Linux distributions, and so on. +

    + +

+In its simplest form, a continuous integration tool sits in a loop +building and releasing software components from a version management +system. For each component, it performs the following tasks: + +

    + +
  1. It obtains the latest version of the component's source code + from the version management system. + +
  2. It runs the component's build process (which presumably includes + the execution of the component's test set). + +
  3. It presents the results of the build (such as error logs and + releases) to the developers, e.g., by producing a web page. + +
+

+Examples of continuous integration tools include CruiseControl +Tinderbox, Sisyphus, Anthill and BuildBot. These tools have various +limitations. + +

    + +
  1. They do not manage the build environment. The build +environment consists of the dependencies necessary to perform a build +action, e.g., compilers, libraries, etc. Setting up the environment +is typically done manually, and without proper SCM control (so it may +be hard to reproduce a build at a later time). Manual management of +the environment scales poorly in the number of configurations that +must be supported. For instance, suppose that we want to build a +component that requires a certain compiler X. We then have to go to +each machine and install X. If we later need a newer version of X, +the process must be repeated all over again. An ever worse problem +occurs if there are conflicting, mutually exclusive versions of the +dependencies. Thus, simply installing the latest version is not an +option. Of course, we can install these components in different +directories and manually pass the appropriate paths to the build +processes of the various components. But this is a rather tiresome +and error-prone process.

    + +

  2. They do not easily support variability in software +systems. A system may have a great deal of build-time +variability: optional functionality, whether to build a debug or +production version, different versions of dependencies, and so on. +(For instance, the Linux kernel now has over 2,600 build-time +configuration switches.) It is therefore important that a continuous +integration tool can easily select and test different instances from +the configuration space of the system to reveal problems, such as +erroneous interactions between features. In a continuous integration +setting, it is also useful to test different combinations of versions +of subsystems, e.g., the head revision of a component against stable +releases of its dependencies, and vice versa, as this can reveal +various integration problems. + +
+ +

+ +Hydra, is a continuous integration tool that solves these +problems. It is built on top of the Nix +package manager, which has a purely functional language for +describing package build actions and their dependencies. This allows +the build environment for projects to be produced automatically and +deterministically, and variability in components to be expressed +naturally using functions; and as such is an ideal fit for a +continuous build system.

@@ -136,11 +266,6 @@ deployment system can be installed on any Linux distribution in parallel to the regular package management system. Thus, you can use Hydra on a Suse, Fedora, or Ubuntu system. -

- -Hydra on Windows?? - -

2.2. Getting Nix

If your server runs NixOS you are all set to continue with @@ -332,7 +457,7 @@ Once created there should be an entry for the project in the sidebar. Go to the project page for the Patchelf project. -

Jobsets

+

3.2. Jobsets

A project can consist of multiple `jobsets', separate tasks that can be built separately, but may depend on each other (without cyclic @@ -385,7 +510,7 @@ officialRelease system -

Release Set

+

3.2. Release Set

there must be one primary job @@ -393,10 +518,10 @@ check the radio button of exactly one job https://svn.nixos.org/repos/nix/nixpkgs/trunk -

Building Jobs

+

3.3. Building Jobs

-

release.nix

+

3.4. release.nix

@@ -408,7 +533,7 @@ https://svn.nixos.org/repos/nix/patchelf/trunk/release.nix https://svn.nixos.org/repos/nix/nix/trunk/release.nix https://svn.nixos.org/repos/nix/hydra/trunk/release.nix -

Building on the command line

+

3.5. Building on the command line

Overigens zijn die helemaal niet Hydra-specifiek, je kunt ze gewoon vanaf de command line bouwen, bijv. als je een patchelf checkout hebt (met een nixpkgs @@ -417,9 +542,5 @@ checkout in ../nixpkgs): $ nix-build release.nix -A rpm_fedora10i386 -

Release Set

- - - \ No newline at end of file