hydra/doc/manual/introduction.xml

<chapter xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xml:id="chap-introduction">

  <title>Introduction</title>

  <section>
    <title>About Hydra</title>

    <para>
      Hydra is a tool for continuous integration testing and software
      release that uses a purely functional language to describe build jobs
      and their dependencies.  Continuous integration is a simple technique
      to improve the quality of the software development process.  An
      automated system continuously or periodically checks out the source
      code of a project, builds it, runs tests, and produces reports for the
      developers.  Thus, various errors that might accidentally be committed
      into the code base are automatically caught.  Such a system allows
      more in-depth testing than what developers could feasibly do manually:

      <itemizedlist>
        <listitem> <emphasis>Portability testing</emphasis>: The
        software may need to be built and tested on many different
        platforms.  It is infeasible for each developer to do this
        before every commit.
        </listitem>

        <listitem> Likewise, many projects have very large test sets
        (e.g., regression tests in a compiler, or stress tests in a
        DBMS) that can take hours or days to run to completion.
        </listitem>

        <listitem> Many kinds of static and dynamic analyses can be
        performed as part of the tests, such as code coverage runs and
        static analyses.
        </listitem>

        <listitem> It may also be necessary to build many different
        <emphasis>variants</emphasis> of the software.  For instance,
        it may be necessary to verify that the component builds with
        various versions of a compiler.
        </listitem>

        <listitem> Developers typically use incremental building to
        test their changes (since a full build may take too long), but
        this is unreliable with many build management tools (such as
        Make), i.e., the result of the incremental build might differ
        from a full build.
        </listitem>

        <listitem> It ensures that the  software can be built from the
        sources under  revision control.  Users  of version management
        systems  such as  CVS  and Subversion  often  forget to  place
        source files under revision control.
        </listitem>

        <listitem> The machines on which the continuous integration
        system runs ideally provides a clean, well-defined build
        environment.  If this environment is administered through
        proper SCM techniques, then builds produced by the system can
        be reproduced.  In contrast, developer work environments are
        typically not under any kind of SCM control.
        </listitem>

        <listitem> In large projects, developers often work on a
        particular component of the project, and do not build and test
        the composition of those components (again since this is
        likely to take too long).  To prevent the phenomenon of ``big
        bang integration'', where components are only tested together
        near the end of the development process, it is important to
        test components together as soon as possible (hence
        <emphasis>continuous integration</emphasis>).
        </listitem>

        <listitem> It allows software to be
        <emphasis>released</emphasis> by automatically creating
        packages that users can download and install.  To do this
        manually represents an often prohibitive amount of work, as
        one may want to produce releases for many different platforms:
        e.g., installers for Windows and Mac OS X, RPM or Debian
        packages for certain Linux distributions, and so on.
        </listitem>

      </itemizedlist>
    </para>

    <para>
      In its simplest form, a continuous integration tool sits in a
      loop building and releasing software components from a version
      management system.  For each component, it performs the
      following tasks:

      <itemizedlist>

        <listitem>It obtains the latest version of the component's
        source code from the version management system.
        </listitem>

        <listitem> It runs the component's build process (which
        presumably includes the execution of the component's test
        set).
        </listitem>

        <listitem> It presents the results of the build (such as error
        logs and releases) to the developers, e.g., by producing a web
        page.
        </listitem>

      </itemizedlist>

      Examples of continuous integration tools include Jenkins,
      CruiseControl Tinderbox, Sisyphus, Anthill and BuildBot. These
      tools have various limitations.

      <itemizedlist>

        <listitem> They do not manage the <emphasis>build
        environment</emphasis>.  The build environment consists of the
        dependencies necessary to perform a build action, e.g.,
        compilers, libraries, etc.  Setting up the environment is
        typically done manually, and without proper SCM control (so it
        may be hard to reproduce a build at a later time).  Manual
        management of the environment scales poorly in the number of
        configurations that must be supported.  For instance, suppose
        that we want to build a component that requires a certain
        compiler X.  We then have to go to each machine and install X.
        If we later need a newer version of X, the process must be
        repeated all over again.  An ever worse problem occurs if
        there are conflicting, mutually exclusive versions of the
        dependencies.  Thus, simply installing the latest version is
        not an option.  Of course, we can install these components in
        different directories and manually pass the appropriate paths
        to the build processes of the various components.  But this is
        a rather tiresome and error-prone process.
        </listitem>

        <listitem> They do not easily support <emphasis>variability in software
        systems</emphasis>.  A system may have a great deal of build-time
        variability: optional functionality, whether to build a debug or
        production version, different versions of dependencies, and so on.
        (For instance, the Linux kernel now has over 2,600 build-time
        configuration switches.)  It is therefore important that a continuous
        integration tool can easily select and test different instances from
        the configuration space of the system to reveal problems, such as
        erroneous interactions between features.  In a continuous integration
        setting, it is also useful to test different combinations of versions
        of subsystems, e.g., the head revision of a component against stable
        releases of its dependencies, and vice versa, as this can reveal
        various integration problems.
        </listitem>

      </itemizedlist>
    </para>

    <para>
      <emphasis>Hydra</emphasis>, is a continuous integration tool
      that solves these problems.  It is built on top of the <link
      xlink:href="http://nixos.org/nix/">Nix package manager</link>,
      which has a purely functional language for describing package
      build actions and their dependencies.  This allows the build
      environment for projects to be produced automatically and
      deterministically, and variability in components to be expressed
      naturally using functions; and as such is an ideal fit for a
      continuous build system.
    </para>

  </section>

  <section>
    <title>About Us</title>

    <para>
      Hydra is the successor of the Nix Buildfarm, which was developed
      in tandem with the Nix software deployment system. Nix was
      originally developed at the Department of Information and
      Computing Sciences, Utrecht University by the TraCE project
      (2003-2008). The project was funded by the Software Engineering
      Research Program Jacquard to improve the support for variability
      in software systems.  Funding for the development of Nix and
      Hydra is now provided by the NIRICT LaQuSo Build Farm project.
    </para>
  </section>

  <section>
    <title>About this Manual</title>

    <para>
      This manual tells you how to install the Hydra buildfarm
      software on your own server and how to operate that server using
      its web interface.
    </para>
  </section>


  <section>
    <title>License</title>

    <para>
      Hydra is free software: you can redistribute it and/or
      modify it under the terms of the GNU General Public License as
      published by the Free Software Foundation, either version 3 of
      the License, or (at your option) any later version.
    </para>

    <para>
      Hydra is distributed in the hope that it will be useful,
      but WITHOUT ANY WARRANTY; without even the implied warranty of
      MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
      <link xlink:href="http://www.gnu.org/licenses/">GNU General
      Public License</link> for more details.
    </para>
  </section>

  <section>
    <title>Hydra at <literal>nixos.org</literal></title>

    <para>
      The <literal>nixos.org</literal> installation of Hydra runs at
      <link
      xlink:href="http://hydra.nixos.org/"><literal>http://hydra.nixos.org/</literal></link>.

      That installation is used to build software components from the
      <link xlink:href="http://nixos.org">Nix</link>,
      <link xlink:href="http://nixos.org/nixos">NixOS</link>,
      <link xlink:href="http://www.gnu.org/">GNU</link>,
      <link xlink:href="http://strategoxt.org">Stratego/XT</link>,
      and related projects.
    </para>

    <para>
      If you are one of the developers on those projects, it is likely
      that you will be using the NixOS Hydra server in some way. If
      you need to administer automatic builds for your project, you
      should pull the right strings to get an account on the
      server. This manual will tell you how to set up new projects and
      build jobs within those projects and write a release.nix file to
      describe the build process of your project to Hydra. You can
      skip the next chapter.
    </para>

    <para>
      If your project does not yet have automatic builds within the
      NixOS Hydra server, it may actually be eligible. We are in the
      process of setting up a large buildfarm that should be able to
      support open source and academic software projects. Get in
      touch.
    </para>
  </section>

  <section>
    <title>Hydra on your own buildfarm</title>

    <para>
      If you need to run your own Hydra installation, <xref
      linkend="chap-installation" /> explains how to download and
      install the system on your own server.
    </para>
  </section>

</chapter>

<!--
 Local Variables:
 indent-tabs-mode: nil
 ispell-local-dictionary: "american"
 End:
 -->