ripped intro from the SCP paper and manualized it

doc/manual/manual.html

@@ -30,18 +30,148 @@ Copyright 2008 Eelco Dolstra
 
 <h3>1.1. About Hydra</h3>
 
-Hydra is a buildfarm system based on the Nix software deployment
+Hydra is a tool for continuous integration testing and software
-system.
+release that uses a purely functional language to describe build jobs
+and their dependencies.  Continuous integration is a simple technique
+to improve the quality of the software development process.  An
+automated system continuously or periodically checks out the source
+code of a project, builds it, runs tests, and produces reports for the
+developers.  Thus, various errors that might accidentally be committed
+into the code base are automatically caught.  Such a system allows
+more in-depth testing than what developers could feasibly do manually:
 
 <p/>
 
-... advantages ...
+<ol>
 
+<li> <em>Portability testing</em>: The software may need to be built
+  and tested on many different platforms.  It is infeasible for each
+  developer to do this before every commit. 
+  <p/>
+
+<li> Likewise, many projects have very large test sets (e.g.,
+  regression tests in a compiler, or stress tests in a DBMS) that can
+  take hours or days to run to completion. 
+  <p/>
+
+<li> Many kinds of static and dynamic analyses can be performed as
+  part of the tests, such as code coverage runs and static analyses.
+  <p/>
+
+<li> It may also be necessary to build many different <em>variants</em>
+  of the software.  For instance, it may be necessary to verify that
+  the component builds with various versions of a compiler.
+  <p/>
+
+<li> Developers typically use incremental building to test their
+  changes (since a full build may take too long), but this is
+  unreliable with many build management tools (such as Make), i.e.,
+  the result of the incremental build might differ from a full build.
+  <p/>
+
+<li> It ensures that the software can be built from the sources under
+  revision control.  Users of version management systems such as CVS
+  and Subversion often forget to place source files under revision
+  control.
+  <p/>
+
+<li> The machines on which the continuous integration system runs
+  ideally provides a clean, well-defined build environment.  If this
+  environment is administered through proper SCM techniques, then
+  builds produced by the system can be reproduced.  In contrast,
+  developer work environments are typically not under any kind of SCM
+  control.
+  <p/>
+
+<li> In large projects, developers often work on a particular
+  component of the project, and do not build and test the composition
+  of those components (again since this is likely to take too long).
+  To prevent the phenomenon of ``big bang integration'', where
+  components are only tested together near the end of the development
+  process, it is important to test components together as soon as
+  possible (hence <em>continuous integration</em>).
+  <p/>
+
+<li> It allows software to be <em>released</em> by automatically
+  creating packages that users can download and install.  To do this
+  manually represents an often prohibitive amount of work, as one may
+  want to produce releases for many different platforms: e.g.,
+  installers for Windows and Mac OS X, RPM or Debian packages for
+  certain Linux distributions, and so on.
+  <p/>
+
+</ol>
 
 <p/>
 
+In its simplest form, a continuous integration tool sits in a loop
+building and releasing software components from a version management
+system.  For each component, it performs the following tasks:
+
+<ol>
+
+<li>It obtains the latest version of the component's source code
+  from the version management system.
+
+<li> It runs the component's build process (which presumably includes
+  the execution of the component's test set).
+
+<li> It presents the results of the build (such as error logs and
+  releases) to the developers, e.g., by producing a web page.
+
+</ol>
+
 <p/>
 
+Examples of continuous integration tools include CruiseControl
+Tinderbox, Sisyphus, Anthill and BuildBot. These tools have various
+limitations.
+
+<ol>
+
+<li> They do not manage the <em>build environment</em>.  The build
+environment consists of the dependencies necessary to perform a build
+action, e.g., compilers, libraries, etc.  Setting up the environment
+is typically done manually, and without proper SCM control (so it may
+be hard to reproduce a build at a later time).  Manual management of
+the environment scales poorly in the number of configurations that
+must be supported.  For instance, suppose that we want to build a
+component that requires a certain compiler X.  We then have to go to
+each machine and install X.  If we later need a newer version of X,
+the process must be repeated all over again.  An ever worse problem
+occurs if there are conflicting, mutually exclusive versions of the
+dependencies.  Thus, simply installing the latest version is not an
+option.  Of course, we can install these components in different
+directories and manually pass the appropriate paths to the build
+processes of the various components.  But this is a rather tiresome
+and error-prone process.  <p/>
+
+<li> They do not easily support <em>variability in software
+systems</em>.  A system may have a great deal of build-time
+variability: optional functionality, whether to build a debug or
+production version, different versions of dependencies, and so on.
+(For instance, the Linux kernel now has over 2,600 build-time
+configuration switches.)  It is therefore important that a continuous
+integration tool can easily select and test different instances from
+the configuration space of the system to reveal problems, such as
+erroneous interactions between features.  In a continuous integration
+setting, it is also useful to test different combinations of versions
+of subsystems, e.g., the head revision of a component against stable
+releases of its dependencies, and vice versa, as this can reveal
+various integration problems.
+
+</ol>
+
+<p/>
+
+<em>Hydra</em>, is a continuous integration tool that solves these
+problems.  It is built on top of the <a href="http://nixos.org">Nix
+package manager</a>, which has a purely functional language for
+describing package build actions and their dependencies.  This allows
+the build environment for projects to be produced automatically and
+deterministically, and variability in components to be expressed
+naturally using functions; and as such is an ideal fit for a
+continuous build system.
 
 <p/>
 
@@ -136,11 +266,6 @@ deployment system can be installed on any Linux distribution in
 parallel to the regular package management system. Thus, you can use
 Hydra on a Suse, Fedora, or Ubuntu system.
 
-<p/>
-
-Hydra on Windows??
-
-
 <h3>2.2. Getting Nix</h3>
 
 If your server runs NixOS you are all set to continue with
@@ -332,7 +457,7 @@ Once created there should be an entry for the project in the
 sidebar. Go to the project page for the <a
 href="http://localhost:3000/project/patchelf">Patchelf</a> project.
 
-<h3>Jobsets</h3>
+<h3>3.2. Jobsets</h3>
 
 A project can consist of multiple `jobsets', separate tasks that can
 be built separately, but may depend on each other (without cyclic
@@ -385,7 +510,7 @@ officialRelease
 
 system 
 
-<h3>Release Set</h3>
+<h3>3.2. Release Set</h3>
 
 there must be one primary job
 
@@ -393,10 +518,10 @@ check the radio button of exactly one job
 
 https://svn.nixos.org/repos/nix/nixpkgs/trunk
 
-<h3>Building Jobs</h3>
+<h3>3.3. Building Jobs</h3>
 
 
-<h3>release.nix</h3>
+<h3>3.4. release.nix</h3>
 
 
 
@@ -408,7 +533,7 @@ https://svn.nixos.org/repos/nix/patchelf/trunk/release.nix
 https://svn.nixos.org/repos/nix/nix/trunk/release.nix
 https://svn.nixos.org/repos/nix/hydra/trunk/release.nix
 
-<h3>Building on the command line</h3>
+<h3>3.5. Building on the command line</h3>
 
 Overigens zijn die helemaal niet Hydra-specifiek, je kunt ze gewoon vanaf de
 command line bouwen, bijv. als je een patchelf checkout hebt (met een nixpkgs
@@ -417,9 +542,5 @@ checkout in ../nixpkgs):
 $ nix-build release.nix -A rpm_fedora10i386
 
 
-<h3>Release Set</h3>
-
-
-
 </body>
 </html>