hydra/doc/manual/projects.xml

<chapter xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xml:id="chap-projects">

  <title>Creating and Managing Projects</title>

  <para>
    Once Hydra is installed and running, the next step is to add
    projects to the build farm. We follow the example of the <link
    xlink:href="http://nixos.org/patchelf.html">Patchelf
    project</link>, a software tool written in C and using the GNU
    Build System (GNU Autoconf and GNU Automake).
  </para>

  <para>
    Log in to the web interface of your Hydra installation using the
    user name and password you inserted in the database (by default,
    Hydra's web server listens on <link
    xlink:href="http://localhost:3000/"><literal>localhost:3000</literal></link>).
    Then follow the "Create Project" link to create a new project.
  </para>

  <section>
    <title>Project Information</title>

    <para>
      A project definition consists of some general information and a
      set of job sets.  The general information identifies a project,
      its owner, and current state of activity.

      Here's what we fill in for the patchelf project:

      <screen>
  Identifier: patchelf
      </screen>

      The <emphasis>identifier</emphasis> is the identity of the
      project. It is used in URLs and in the names of build results.
    </para>

    <para>
      The identifier should be a unique name (it is the primary
      database key for the project table in the database). If you try
      to create a project with an already existing identifier you'd
      get an error message such as:

<screen>
I'm very sorry, but an error occurred: 
DBIx::Class::ResultSet::create(): DBI Exception: DBD::SQLite::st execute failed: column name is not unique(19) at dbdimp.c line 402
</screen>

       So try to create the project after entering just the general
       information to figure out if you have chosen a unique name.
       Job sets can be added once the project has been created.

<screen>
  Display name: Patchelf
</screen>

       The <emphasis>display name</emphasis> is used in menus.

<screen>
  Description: A tool for modifying ELF binaries
</screen>

       The <emphasis>description</emphasis> is used as short
       documentation of the nature of the project.

<screen>
  Owner: eelco
</screen>

       The <emphasis>owner</emphasis> of a project can create and edit
       job sets.

<screen>
  Enabled: Yes
</screen>

       Only if the project is <emphasis>enabled</emphasis> are builds
       performed.
    </para>

    <para>
      Once created there should be an entry for the project in the
      sidebar.  Go to the project page for the <link
      xlink:href="http://localhost:3000/project/patchelf">Patchelf</link>
      project.
    </para>
  </section>

  <section>
    <title>Job Sets</title>

    <para>
      A project can consist of multiple <emphasis>job sets</emphasis>
      (hereafter <emphasis>jobsets</emphasis>), separate tasks that
      can be built separately, but may depend on each other (without
      cyclic dependencies, of course). Go to the <link
      xlink:href="http://localhost:3000/project/patchelf/edit">Edit</link>
      page of the Patchelf project and "Add a new jobset" by providing
      the following "Information":

<screen>
  Identifier:     trunk
  Description:    Trunk
  Nix expression: release.nix in input patchelfSrc
</screen>

      This states that in order to build the <literal>trunk</literal>
      jobset, the Nix expression in the file
      <filename>release.nix</filename>, which can be obtained from
      input <literal>patchelfSrc</literal>, should be
      evaluated. (We'll have a look at
      <filename>release.nix</filename> later.)

    </para>

    <para>
      To realize a job we probably need a number of inputs, which can
      be declared in the table below. As many inputs as required can
      be added.  For patchelf we declare the following inputs.

<screen>
  patchelfSrc 
    'Subversion checkout' https://svn.nixos.org/repos/nix/patchelf/trunk

  nixpkgs 'Subversion checkout' https://svn.nixos.org/repos/nix/nixpkgs/trunk

  officialRelease   Boolean false

  system   String value "i686-linux" 
</screen>
    </para>
  </section>

  <section>
    <title>Release Set</title>

    <!-- TODO -->
    there must be one primary job

    check the radio button of exactly one job

    https://svn.nixos.org/repos/nix/nixpkgs/trunk
  </section>

  <section>
    <title>Building Jobs</title>
  </section>

  <section>
    <title>Build Recipes</title>

    <para>
      Build jobs and <emphasis>build recipes</emphasis> for a jobset are
      specified in a text file written in the <link
      xlink:href="http://nixos.org/nix/">Nix language</link>.  The
      recipe is actually called a <emphasis>Nix expression</emphasis> in
      Nix parlance.  By convention this file is often called
      <filename>release.nix</filename>.
    </para>

    <para>
      The <filename>release.nix</filename> file is typically kept under
      version control, and the repository that contains it one of the
      build inputs of the corresponding–often called
      <literal>hydraConfig</literal> by convention.  The repository for
      that file and the actual file name are specified on the web
      interface of Hydra under the <literal>Setup</literal> tab of the
      jobset's overview page, under the <literal>Nix
      expression</literal> heading.  See, for example, the <link
      xlink:href="http://hydra.nixos.org/jobset/patchelf/trunk">jobset
      overview page</link> of the PatchELF project, and <link
      xlink:href="https://svn.nixos.org/repos/nix/patchelf/trunk/release.nix">
      the corresponding Nix file</link>.
    </para>

    <para>
      Knowledge of the Nix language is recommended, but the example
      below should already give a good idea of how it works:
    </para>

    <example xml:id='ex-hello'>
      <title><filename>release.nix</filename> file for GNU Hello</title>
      <programlisting>
{ nixpkgs }: <co xml:id='ex-hello-co-nixpkgs' />

let
  pkgs = import nixpkgs {}; <co xml:id='ex-hello-co-import-nixpkgs' />

  jobs = rec { <co xml:id='ex-hello-co-jobs' />

    tarball = <co xml:id='ex-hello-co-tarball' />
      { helloSrc }: <co xml:id='ex-hello-co-tarball-args' />

      pkgs.releaseTools.sourceTarball { <co xml:id='ex-hello-co-source-tarball' />
        name = "hello-tarball";
        src = helloSrc;
        buildInputs = (with pkgs; [ gettext texLive texinfo ]);
      };

    build = <co xml:id='ex-hello-co-build' />
      { tarball ? jobs.tarball {}        <co xml:id='ex-hello-co-build-args' />
      , system ? builtins.currentSystem
      }:

      let pkgs = import nixpkgs { inherit system; }; in
      pkgs.releaseTools.nixBuild { <co xml:id='ex-hello-co-nix-build' />
        name = "hello" ;
        src = tarball;
        configureFlags = [ "--disable-silent-rules" ];
      };
  };
in
  jobs <co xml:id='ex-hello-co-body' />
      </programlisting>
    </example>

    <para>
      <xref linkend='ex-hello' /> shows what a
      <filename>release.nix</filename> file for <link
      xlink:href="http://www.gnu.org/software/hello/">GNU Hello</link>
      would you like.  GNU Hello is representative of many GNU
      and non-GNU free software projects:

      <itemizedlist>
	<listitem>it uses the GNU Build System, namely GNU Autoconf,
	and GNU Automake; for users, it means it can be installed
	using the <link
	xlink:href="http://www.gnu.org/prep/standards/html_node/Managing-Releases.html">usual
	<literal>./configure &amp;&amp; make install</literal>
	procedure</link>;
	</listitem>
	<listitem>it uses Gettext for internationalization;</listitem>
	<listitem>it has a Texinfo manual, which can be rendered as PDF
	with TeX.</listitem>
      </itemizedlist>

      The file defines a jobset consisting of two jobs:
      <literal>tarball</literal>, and <literal>build</literal>.  It
      contains the following elements (referenced from the figure by
      numbers):

      <calloutlist>

	<callout arearefs='ex-hello-co-nixpkgs'>
	  <para>
	    This specifies a function of one named arguments,
	    <varname>nixpkgs</varname>.  This function and those
	    defined below is called by Hydra.  Here the
	    <varname>nixpkgs</varname> argument is meant to be a
	    checkout of the <link
	    xlink:href="http://nixos.org/nixpkgs/">Nixpkgs</link>
	    software distribution.
	  </para>
	  <para>
	    Hydra inspects the formal argument list of the function
	    (here, the <varname>nixpkgs</varname> argument) and passes
	    it the corresponding parameter specified as a build input
	    on Hydra's web interface.  In this case, the web interface
	    should show a <varname>nixpkgs</varname> build input,
	    which is a checkout of the Nixpkgs source code repository.
	  </para>
	</callout>
	<callout arearefs='ex-hello-co-import-nixpkgs'>
	  <para>
	    This defines a variable <varname>pkgs</varname> holding
	    the set of packages provided by Nixpkgs.
	  </para>
	</callout>

	<callout arearefs='ex-hello-co-jobs'>
	  <para>
	    This defines a variable holding the two Hydra
	    jobs–an <emphasis>attribute set</emphasis> in Nix.
	  </para>
	</callout>

	<callout arearefs='ex-hello-co-tarball'>
	  <para>
	    This is the definition of the first job, named
	    <varname>tarball</varname>.  The purpose of this job is to
	    produce a usable source code tarball.
	  </para>
	</callout>
	<callout arearefs='ex-hello-co-tarball-args'>
	  <para>
	    The <varname>tarball</varname> takes an additional
	    argument called <varname>helloSrc</varname>.  Again, this
	    argument is passed by Hydra and is meant to be a checkout
	    of GNU Hello's source code repository.
	  </para>
	</callout>
	<callout arearefs='ex-hello-co-source-tarball'>
	  <para>
	    The <varname>tarball</varname> job calls the
	    <varname>sourceTarball</varname> function, which (roughly)
	    runs <command>autoreconf &amp;&amp; ./configure &amp;&amp;
	    make dist</command> on the checkout.  The
	    <varname>buildInputs</varname> attribute specifies
	    additional software dependencies for the job.
	  </para>
	</callout>

	<callout arearefs='ex-hello-co-build'>
	  <para>
	    This is the definition of the <varname>build</varname>
	    job, whose purpose is to build Hello from the tarball
	    produced above.
	  </para>
	</callout>
	<callout arearefs='ex-hello-co-build-args'>
	  <para>
	    The <varname>build</varname> function takes two additional
	    parameter: <varname>tarball</varname>, which is meant to
	    be the result of the <varname>tarball</varname> job, and
	    <varname>system</varname>, which should be a string
	    defining the Nix system type–e.g.,
	    <literal>"x86_64-linux"</literal>.
	  </para>
	  <para>
	    Again, these parameters are passed by Hydra when it calls
	    <varname>build</varname>.  Thus, they must be defined as
	    build inputs in Hydra: <varname>tarball</varname> should
	    have type <literal>Build Output</literal>, its value being
	    the latest output of the <varname>tarball</varname> job,
	    and <varname>system</varname> should be a string.
	  </para>
	  <para>
	    The question mark after <literal>tarball</literal> and
	    <literal>system</literal> defines default values for these
	    arguments, and is only useful for debugging.
	  </para>
	</callout>
	<callout arearefs='ex-hello-co-nix-build'>
	  <para>
	    The <varname>build</varname> job calls the
	    <varname>nixBuild</varname> function, which unpacks the
	    tarball, then runs <command>./configure &amp;&amp; make
	    &amp;&amp; make check &amp;&amp; make install</command>.
	  </para>
	</callout>

	<callout arearefs='ex-hello-co-body'>
	  <para>
	    Finally, the set of jobs is returned to Hydra, as a Nix
	    attribute set.
	  </para>
	</callout>
      </calloutlist>
    </para>
  </section>

  <section>
    <title>Building from the Command Line</title>

    <para>
      It is often useful to test a build recipe, for instance before
      it is actually used by Hydra, when testing changes, or when
      debugging a build issue.  Since build recipes for Hydra jobsets
      are just plain Nix expressions, they can be evaluated using the
      standard Nix tools.
    </para>

    <para>
      To evaluate the <varname>tarball</varname> jobset of <xref
      linkend='ex-hello' />, just run:

      <screen>
	$ nix-build release.nix -A tarball
      </screen>

      However, doing this with <xref linkend='ex-hello' /> as is will
      probably yield an error like this:

      <screen>
	error: cannot auto-call a function that has an argument without a default value (`nixpkgs')
      </screen>

      This is because no value was specified for the
      <varname>nixpkgs</varname> argument of the Nix expression.
    </para>

    <para>
      This is fixed by providing a default value for that argument in
      the Nix expression, which will allow <command>nix-build</command>
      to auto-call the function: instfead of writing <literal>{ nixpkgs
      }:</literal>, we now write <literal>{ nixpkgs ? &lt;nixpkgs&gt;
      }:</literal>.  What it means is that, by default, the
      <varname>nixpkgs</varname> variable will be bound to the absolute
      path of any <filename>nixpkgs</filename> file found in the Nix
      search path.  Similarly, a default value for
      <varname>helloSrc</varname> needs to be provided.
    </para>

    <para>
      Thus, assuming a checkout of Nixpkgs is available under
      <filename>$HOME/src/nixpkgs</filename>, the
      <varname>tarball</varname> jobset can now be evaluated by running:

      <screen>
	$ nix-build -I ~/src release.nix -A tarball
      </screen>

      Similarly, the <varname>build</varname> jobset can be evaluated:

      <screen>
	$ nix-build -I ~/src release.nix -A build
      </screen>

      The <varname>build</varname> job reuses the result of the
      <varname>tarball</varname> job, rebuilding it only if it needs to.
    </para>
      
  </section>

  <section>
    <title>Adding More Jobs</title>

    <para>
      <xref linkend='ex-hello' /> illustrates how to write the most
      basic jobs, <varname>tarball</varname> and
      <varname>build</varname>.  In practice, much more can be done by
      using features readily provided by Nixpkgs or by creating new jobs
      as customizations of existing jobs.
    </para>

    <para>
      For instance, test coverage report for projects compiled with GCC
      can be automatically generated using the
      <varname>coverageAnalysis</varname> function provided by Nixpkgs
      instead of <varname>nixBuild</varname>.  Back to our GNU Hello
      example, we can define a <varname>coverage</varname> job that
      produces an HTML code coverage report directly readable from the
      corresponding Hydra build page:

      <programlisting>
    coverage =
      { tarball ? jobs.tarball {}
      , system ? builtins.currentSystem
      }:

      let pkgs = import nixpkgs { inherit system; }; in
      pkgs.releaseTools.coverageAnalysis {
        name = "hello" ;
        src = tarball;
        configureFlags = [ "--disable-silent-rules" ];
      }; </programlisting>

      As can be seen, the only difference compared to
      <varname>build</varname> is the use of
      <varname>coverageAnalysis</varname>.
    </para>

    <para>
      Nixpkgs provides many more build tools, including the ability to
      run build in virtual machines, which can themselves run another
      GNU/Linux distribution, which allows for the creation of packages
      for these distributions.  Please see <link
      xlink:href="https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/build-support/release/">the
      <filename>pkgs/build-support/release</filename> directory</link>
      of Nixpkgs for more.  The NixOS manual also contains information
      about whole-system testing in virtual machine.
    </para>

    <para>
      Now, assume we want to build Hello with an old version of GCC, and
      with different <command>configure</command> flags.  A new
      <varname>build_exotic</varname> job can be written that simply
      <emphasis>overrides</emphasis> the relevant arguments passed to
      <varname>nixBuild</varname>:

      <programlisting>
    build_exotic =
      { tarball ? jobs.tarball {}
      , system ? builtins.currentSystem
      }:

      let
        pkgs = import nixpkgs { inherit system; };
        build = jobs.build { inherit tarball system; };
      in
        pkgs.lib.overrideDerivation build (attrs: {
          buildInputs = [ pkgs.gcc33 ];
	  preConfigure = "gcc --version";
	  configureFlags =
	    attrs.configureFlags ++ [ "--disable-nls" ];
        }); </programlisting>

      The <varname>build_exotic</varname> job reuses
      <varname>build</varname> and overrides some of its arguments: it
      adds a dependency on GCC 3.3, a pre-configure phase that runs
      <command>gcc --version</command>, and adds the
      <literal>--disable-nls</literal> configure flags.
    </para>

    <para>
      This customization mechanism is very powerful.  For instance, it
      can be used to change the way Hello and <emphasis>all</emphasis>
      its dependencies–including the C library and compiler used to
      build it–are built.  See the Nixpkgs manual for more.
    </para>

  </section>


</chapter>