hydra/doc/manual/introduction.xml

<chapter xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xml:id="chap-introduction">

  <title>Introduction</title>

  <section>
    <title>About Hydra</title>

    <para>
      Hydra is a tool for continuous integration testing and software
      release that uses a purely functional language to describe build jobs
      and their dependencies.  Continuous integration is a simple technique
      to improve the quality of the software development process.  An
      automated system continuously or periodically checks out the source
      code of a project, builds it, runs tests, and produces reports for the
      developers.  Thus, various errors that might accidentally be committed
      into the code base are automatically caught.  Such a system allows
      more in-depth testing than what developers could feasibly do manually:

      <itemizedlist>
	<listitem> <emphasis>Portability testing</emphasis>: The
	software may need to be built and tested on many different
	platforms.  It is infeasible for each developer to do this
	before every commit.
	</listitem>

	<listitem> Likewise, many projects have very large test sets
	(e.g., regression tests in a compiler, or stress tests in a
	DBMS) that can take hours or days to run to completion.
	</listitem>

	<listitem> Many kinds of static and dynamic analyses can be
	performed as part of the tests, such as code coverage runs and
	static analyses.
	</listitem>

	<listitem> It may also be necessary to build many different
	<emphasis>variants</emphasis> of the software.  For instance,
	it may be necessary to verify that the component builds with
	various versions of a compiler.
	</listitem>

	<listitem> Developers typically use incremental building to
	test their changes (since a full build may take too long), but
	this is unreliable with many build management tools (such as
	Make), i.e., the result of the incremental build might differ
	from a full build.
	</listitem>

	<listitem> It ensures that the  software can be built from the
	sources under  revision control.  Users  of version management
	systems  such as  CVS  and Subversion  often  forget to  place
	source files under revision control.
	</listitem>

	<listitem> The machines on which the continuous integration
	system runs ideally provides a clean, well-defined build
	environment.  If this environment is administered through
	proper SCM techniques, then builds produced by the system can
	be reproduced.  In contrast, developer work environments are
	typically not under any kind of SCM control.
	</listitem>

	<listitem> In large projects, developers often work on a
	particular component of the project, and do not build and test
	the composition of those components (again since this is
	likely to take too long).  To prevent the phenomenon of ``big
	bang integration'', where components are only tested together
	near the end of the development process, it is important to
	test components together as soon as possible (hence
	<emphasis>continuous integration</emphasis>).
	</listitem>

	<listitem> It allows software to be
	<emphasis>released</emphasis> by automatically creating
	packages that users can download and install.  To do this
	manually represents an often prohibitive amount of work, as
	one may want to produce releases for many different platforms:
	e.g., installers for Windows and Mac OS X, RPM or Debian
	packages for certain Linux distributions, and so on.
	</listitem>

      </itemizedlist>
    </para>

    <para>
      In its simplest form, a continuous integration tool sits in a
      loop building and releasing software components from a version
      management system.  For each component, it performs the
      following tasks:

      <itemizedlist>

	<listitem>It obtains the latest version of the component's
	source code from the version management system.
	</listitem>

	<listitem> It runs the component's build process (which
	presumably includes the execution of the component's test
	set).
	</listitem>

	<listitem> It presents the results of the build (such as error
	logs and releases) to the developers, e.g., by producing a web
	page.
	</listitem>

      </itemizedlist>

      Examples of continuous integration tools include Jenkins,
      CruiseControl Tinderbox, Sisyphus, Anthill and BuildBot. These
      tools have various limitations.

      <itemizedlist>

	<listitem> They do not manage the <emphasis>build
	environment</emphasis>.  The build environment consists of the
	dependencies necessary to perform a build action, e.g.,
	compilers, libraries, etc.  Setting up the environment is
	typically done manually, and without proper SCM control (so it
	may be hard to reproduce a build at a later time).  Manual
	management of the environment scales poorly in the number of
	configurations that must be supported.  For instance, suppose
	that we want to build a component that requires a certain
	compiler X.  We then have to go to each machine and install X.
	If we later need a newer version of X, the process must be
	repeated all over again.  An ever worse problem occurs if
	there are conflicting, mutually exclusive versions of the
	dependencies.  Thus, simply installing the latest version is
	not an option.  Of course, we can install these components in
	different directories and manually pass the appropriate paths
	to the build processes of the various components.  But this is
	a rather tiresome and error-prone process.
	</listitem>

	<listitem> They do not easily support <emphasis>variability in software
	systems</emphasis>.  A system may have a great deal of build-time
	variability: optional functionality, whether to build a debug or
	production version, different versions of dependencies, and so on.
	(For instance, the Linux kernel now has over 2,600 build-time
	configuration switches.)  It is therefore important that a continuous
	integration tool can easily select and test different instances from
	the configuration space of the system to reveal problems, such as
	erroneous interactions between features.  In a continuous integration
	setting, it is also useful to test different combinations of versions
	of subsystems, e.g., the head revision of a component against stable
	releases of its dependencies, and vice versa, as this can reveal
	various integration problems.
	</listitem>

      </itemizedlist>
    </para>

    <para>
      <emphasis>Hydra</emphasis>, is a continuous integration tool
      that solves these problems.  It is built on top of the <link
      xlink:href="http://nixos.org/nix/">Nix package manager</link>,
      which has a purely functional language for describing package
      build actions and their dependencies.  This allows the build
      environment for projects to be produced automatically and
      deterministically, and variability in components to be expressed
      naturally using functions; and as such is an ideal fit for a
      continuous build system.
    </para>

  </section>

  <section>
    <title>About Us</title>

    <para>
      Hydra is the successor of the Nix Buildfarm, which was developed
      in tandem with the Nix software deployment system. Nix was
      originally developed at the Department of Information and
      Computing Sciences, Utrecht University by the TraCE project
      (2003-2008). The project was funded by the Software Engineering
      Research Program Jacquard to improve the support for variability
      in software systems.  Funding for the development of Nix and
      Hydra is now provided by the NIRICT LaQuSo Build Farm project.
    </para>
  </section>

  <section>
    <title>About this Manual</title>

    <para>
      This manual tells you how to install the Hydra buildfarm
      software on your own server and how to operate that server using
      its web interface.
    </para>
  </section>


  <section>
    <title>License</title>

    <para>
      Hydra is free software: you can redistribute it and/or
      modify it under the terms of the GNU General Public License as
      published by the Free Software Foundation, either version 3 of
      the License, or (at your option) any later version.
    </para>

    <para>
      Hydra is distributed in the hope that it will be useful,
      but WITHOUT ANY WARRANTY; without even the implied warranty of
      MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
      <link xlink:href="http://www.gnu.org/licenses/">GNU General
      Public License</link> for more details.
    </para>
  </section>

  <section>
    <title>Hydra at <literal>nixos.org</literal></title>

    <para>
      The <literal>nixos.org</literal> installation of Hydra runs at
      <link
      xlink:href="http://hydra.nixos.org/"><literal>http://hydra.nixos.org/</literal></link>.

      That installation is used to build software components from the 
      <link xlink:href="http://nixos.org">Nix</link>, 
      <link xlink:href="http://nixos.org/nixos">NixOS</link>,
      <link xlink:href="http://www.gnu.org/">GNU</link>,
      <link xlink:href="http://strategoxt.org">Stratego/XT</link>,
      and related projects.
    </para>

    <para>
      If you are one of the developers on those projects, it is likely
      that you will be using the NixOS Hydra server in some way. If
      you need to administer automatic builds for your project, you
      should pull the right strings to get an account on the
      server. This manual will tell you how to set up new projects and
      build jobs within those projects and write a release.nix file to
      describe the build process of your project to Hydra. You can
      skip the next chapter.
    </para>

    <para>
      If your project does not yet have automatic builds within the
      NixOS Hydra server, it may actually be eligible. We are in the
      process of setting up a large buildfarm that should be able to
      support open source and academic software projects. Get in
      touch.
    </para>
  </section>

  <section>
    <title>Hydra on your own buildfarm</title>

    <para>
      If you need to run your own Hydra installation, <xref
      linkend="chap-installation" /> explains how to download and
      install the system on your own server.
    </para>
  </section>

</chapter>