hydra/doc/manual/projects.xml

<chapter xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xmlns:xi="http://www.w3.org/2001/XInclude"
         xml:id="chap-projects">

  <title>Creating and Managing Projects</title>

  <para>
    Once Hydra is installed and running, the next step is to add
    projects to the build farm. We follow the example of the <link
    xlink:href="http://nixos.org/patchelf.html">Patchelf
    project</link>, a software tool written in C and using the GNU
    Build System (GNU Autoconf and GNU Automake).
  </para>

  <para>
    Log in to the web interface of your Hydra installation using the
    user name and password you inserted in the database (by default,
    Hydra's web server listens on <link
    xlink:href="http://localhost:3000/"><literal>localhost:3000</literal></link>).
    Then follow the "Create Project" link to create a new project.
  </para>

  <section>
    <title>Project Information</title>

    <para>
      A project definition consists of some general information and a
      set of job sets.  The general information identifies a project,
      its owner, and current state of activity.

      Here's what we fill in for the patchelf project:

<screen>
Identifier: patchelf
</screen>

      The <emphasis>identifier</emphasis> is the identity of the
      project. It is used in URLs and in the names of build results.
    </para>

    <para>
      The identifier should be a unique name (it is the primary
      database key for the project table in the database). If you try
      to create a project with an already existing identifier you'd
      get an error message such as:

<screen>
I'm very sorry, but an error occurred:
DBIx::Class::ResultSet::create(): DBI Exception: DBD::SQLite::st execute failed: column name is not unique(19) at dbdimp.c line 402
</screen>

       So try to create the project after entering just the general
       information to figure out if you have chosen a unique name.
       Job sets can be added once the project has been created.

<screen>
Display name: Patchelf
</screen>

       The <emphasis>display name</emphasis> is used in menus.

<screen>
Description: A tool for modifying ELF binaries
</screen>

       The <emphasis>description</emphasis> is used as short
       documentation of the nature of the project.

<screen>
Owner: eelco
</screen>

       The <emphasis>owner</emphasis> of a project can create and edit
       job sets.

<screen>
Enabled: Yes
</screen>

       Only if the project is <emphasis>enabled</emphasis> are builds
       performed.
    </para>

    <para>
      Once created there should be an entry for the project in the
      sidebar.  Go to the project page for the <link
      xlink:href="http://localhost:3000/project/patchelf">Patchelf</link>
      project.
    </para>
  </section>

  <section>
    <title>Job Sets</title>

    <para>
      A project can consist of multiple <emphasis>job sets</emphasis>
      (hereafter <emphasis>jobsets</emphasis>), separate tasks that
      can be built separately, but may depend on each other (without
      cyclic dependencies, of course). Go to the <link
      xlink:href="http://localhost:3000/project/patchelf/edit">Edit</link>
      page of the Patchelf project and "Add a new jobset" by providing
      the following "Information":

<screen>
Identifier:     trunk
Description:    Trunk
Nix expression: release.nix in input patchelfSrc
</screen>

      This states that in order to build the <literal>trunk</literal>
      jobset, the Nix expression in the file
      <filename>release.nix</filename>, which can be obtained from
      input <literal>patchelfSrc</literal>, should be
      evaluated. (We'll have a look at
      <filename>release.nix</filename> later.)

    </para>

    <para>
      To realize a job we probably need a number of inputs, which can
      be declared in the table below. As many inputs as required can
      be added.  For patchelf we declare the following inputs.

<screen>
patchelfSrc
'Git checkout' https://github.com/NixOS/patchelf

nixpkgs 'Git checkout' https://github.com/NixOS/nixpkgs

officialRelease   Boolean false

system   String value "i686-linux"
</screen>
    </para>
  </section>

  <section>
    <title>Building Jobs</title>
  </section>

  <section>
    <title>Build Recipes</title>

    <para>
      Build jobs and <emphasis>build recipes</emphasis> for a jobset are
      specified in a text file written in the <link
      xlink:href="http://nixos.org/nix/">Nix language</link>.  The
      recipe is actually called a <emphasis>Nix expression</emphasis> in
      Nix parlance.  By convention this file is often called
      <filename>release.nix</filename>.
    </para>

    <para>
      The <filename>release.nix</filename> file is typically kept under
      version control, and the repository that contains it one of the
      build inputs of the corresponding–often called
      <literal>hydraConfig</literal> by convention.  The repository for
      that file and the actual file name are specified on the web
      interface of Hydra under the <literal>Setup</literal> tab of the
      jobset's overview page, under the <literal>Nix
      expression</literal> heading.  See, for example, the <link
      xlink:href="http://hydra.nixos.org/jobset/patchelf/trunk">jobset
      overview page</link> of the PatchELF project, and <link
      xlink:href="https://github.com/NixOS/patchelf/blob/master/release.nix">
      the corresponding Nix file</link>.
    </para>

    <para>
      Knowledge of the Nix language is recommended, but the example
      below should already give a good idea of how it works:
    </para>

    <example xml:id='ex-hello'>
      <title><filename>release.nix</filename> file for GNU Hello</title>
      <programlisting>
let
  pkgs = import &lt;nixpkgs&gt; {}; <co xml:id='ex-hello-co-import-nixpkgs' />

  jobs = rec { <co xml:id='ex-hello-co-jobs' />

    tarball = <co xml:id='ex-hello-co-tarball' />
      pkgs.releaseTools.sourceTarball { <co xml:id='ex-hello-co-source-tarball' />
        name = "hello-tarball";
        src = &lt;hello&gt;; <co xml:id='ex-hello-co-tarball-args' />
        buildInputs = (with pkgs; [ gettext texLive texinfo ]);
      };

    build = <co xml:id='ex-hello-co-build' />
      { system ? builtins.currentSystem }:  <co xml:id='ex-hello-co-build-args' />

      let pkgs = import &lt;nixpkgs&gt; { inherit system; }; in
      pkgs.releaseTools.nixBuild { <co xml:id='ex-hello-co-nix-build' />
        name = "hello";
        src = jobs.tarball;
        configureFlags = [ "--disable-silent-rules" ];
      };
  };
in
  jobs <co xml:id='ex-hello-co-body' />
      </programlisting>
    </example>

    <para>
      <xref linkend='ex-hello' /> shows what a
      <filename>release.nix</filename> file for <link
      xlink:href="http://www.gnu.org/software/hello/">GNU Hello</link>
      would look like.  GNU Hello is representative of many GNU
      and non-GNU free software projects:

      <itemizedlist>
        <listitem>it uses the GNU Build System, namely GNU Autoconf,
        and GNU Automake; for users, it means it can be installed
        using the <link
        xlink:href="http://www.gnu.org/prep/standards/html_node/Managing-Releases.html">usual
        <literal>./configure &amp;&amp; make install</literal>
        procedure</link>;
        </listitem>
        <listitem>it uses Gettext for internationalization;</listitem>
        <listitem>it has a Texinfo manual, which can be rendered as PDF
        with TeX.</listitem>
      </itemizedlist>

      The file defines a jobset consisting of two jobs:
      <literal>tarball</literal>, and <literal>build</literal>.  It
      contains the following elements (referenced from the figure by
      numbers):

      <calloutlist>

        <callout arearefs='ex-hello-co-import-nixpkgs'>
          <para>
            This defines a variable <varname>pkgs</varname> holding
            the set of packages provided by <link
            xlink:href="http://nixos.org/nixpkgs/">Nixpkgs</link>.
          </para>
          <para>
            Since <varname>nixpkgs</varname> appears in angle brackets,
            there must be a build input of that name in the Nix search
            path.  In this case, the web interface should show a
            <varname>nixpkgs</varname> build input, which is a checkout
            of the Nixpkgs source code repository; Hydra then adds this
            and other build inputs to the Nix search path when
            evaluating <filename>release.nix</filename>.
          </para>
        </callout>

        <callout arearefs='ex-hello-co-jobs'>
          <para>
            This defines a variable holding the two Hydra
            jobs–an <emphasis>attribute set</emphasis> in Nix.
          </para>
        </callout>

        <callout arearefs='ex-hello-co-tarball'>
          <para>
            This is the definition of the first job, named
            <varname>tarball</varname>.  The purpose of this job is to
            produce a usable source code tarball.
          </para>
        </callout>
        <callout arearefs='ex-hello-co-source-tarball'>
          <para>
            The <varname>tarball</varname> job calls the
            <varname>sourceTarball</varname> function, which (roughly)
            runs <command>autoreconf &amp;&amp; ./configure &amp;&amp;
            make dist</command> on the checkout.  The
            <varname>buildInputs</varname> attribute specifies
            additional software dependencies for the
            job<footnote><para>The package names used in
            <varname>buildInputs</varname>–e.g.,
            <varname>texLive</varname>–are the names of the
            <emphasis>attributes</emphasis> corresponding to these
            packages in Nixpkgs, specifically in the <link
             xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix"><filename>all-packages.nix</filename></link>
            file.  See the section entitled “Package Naming” in the
            Nixpkgs manual for more information.</para></footnote>.
          </para>
        </callout>
        <callout arearefs='ex-hello-co-tarball-args'>
          <para>
            The <varname>tarball</varname> jobs expects a
            <varname>hello</varname> build input to be available in the
            Nix search path.  Again, this input is passed by Hydra and
            is meant to be a checkout of GNU Hello's source code
            repository.
          </para>
        </callout>

        <callout arearefs='ex-hello-co-build'>
          <para>
            This is the definition of the <varname>build</varname>
            job, whose purpose is to build Hello from the tarball
            produced above.
          </para>
        </callout>
        <callout arearefs='ex-hello-co-build-args'>
          <para>
            The <varname>build</varname> function takes one
            parameter, <varname>system</varname>, which should be a string
            defining the Nix system type–e.g.,
            <literal>"x86_64-linux"</literal>.  Additionally, it refers
            to <varname>jobs.tarball</varname>, seen above.
          </para>
          <para>
            Hydra inspects the formal argument list of the function
            (here, the <varname>system</varname> argument) and passes it
            the corresponding parameter specified as a build input on
            Hydra's web interface.  Here, <varname>system</varname> is
            passed by Hydra when it calls <varname>build</varname>.
            Thus, it must be defined as a build input of type string in
            Hydra, which could take one of several values.
          </para>
          <para>
            The question mark after <literal>system</literal> defines
            the default value for this argument, and is only useful when
            debugging locally.
          </para>
        </callout>
        <callout arearefs='ex-hello-co-nix-build'>
          <para>
            The <varname>build</varname> job calls the
            <varname>nixBuild</varname> function, which unpacks the
            tarball, then runs <command>./configure &amp;&amp; make
            &amp;&amp; make check &amp;&amp; make install</command>.
          </para>
        </callout>

        <callout arearefs='ex-hello-co-body'>
          <para>
            Finally, the set of jobs is returned to Hydra, as a Nix
            attribute set.
          </para>
        </callout>
      </calloutlist>
    </para>
  </section>

  <section>
    <title>Building from the Command Line</title>

    <para>
      It is often useful to test a build recipe, for instance before
      it is actually used by Hydra, when testing changes, or when
      debugging a build issue.  Since build recipes for Hydra jobsets
      are just plain Nix expressions, they can be evaluated using the
      standard Nix tools.
    </para>

    <para>
      To evaluate the <varname>tarball</varname> jobset of <xref
      linkend='ex-hello' />, just run:

<screen>
$ nix-build release.nix -A tarball
</screen>

      However, doing this with <xref linkend='ex-hello' /> as is will
      probably yield an error like this:

<screen>
error: user-thrown exception: file `hello' was not found in the Nix search path (add it using $NIX_PATH or -I)
</screen>

       The error is self-explanatory.  Assuming
       <filename>$HOME/src/hello</filename> points to a checkout of
       Hello, this can be fixed this way:

<screen>
$ nix-build -I ~/src release.nix -A tarball
</screen>

      Similarly, the <varname>build</varname> jobset can be evaluated:

<screen>
$ nix-build -I ~/src release.nix -A build
</screen>

      The <varname>build</varname> job reuses the result of the
      <varname>tarball</varname> job, rebuilding it only if it needs to.
    </para>

  </section>

  <section>
    <title>Adding More Jobs</title>

    <para>
      <xref linkend='ex-hello' /> illustrates how to write the most
      basic jobs, <varname>tarball</varname> and
      <varname>build</varname>.  In practice, much more can be done by
      using features readily provided by Nixpkgs or by creating new jobs
      as customizations of existing jobs.
    </para>

    <para>
      For instance, test coverage report for projects compiled with GCC
      can be automatically generated using the
      <varname>coverageAnalysis</varname> function provided by Nixpkgs
      instead of <varname>nixBuild</varname>.  Back to our GNU Hello
      example, we can define a <varname>coverage</varname> job that
      produces an HTML code coverage report directly readable from the
      corresponding Hydra build page:

<programlisting>
coverage =
  { system ? builtins.currentSystem }:

  let pkgs = import nixpkgs { inherit system; }; in
  pkgs.releaseTools.coverageAnalysis {
    name = "hello";
    src = jobs.tarball;
    configureFlags = [ "--disable-silent-rules" ];
  };
</programlisting>

      As can be seen, the only difference compared to
      <varname>build</varname> is the use of
      <varname>coverageAnalysis</varname>.
    </para>

    <para>
      Nixpkgs provides many more build tools, including the ability to
      run build in virtual machines, which can themselves run another
      GNU/Linux distribution, which allows for the creation of packages
      for these distributions.  Please see <link
       xlink:href="https://github.com/NixOS/nixpkgs/tree/master/pkgs/build-support/release">the
      <filename>pkgs/build-support/release</filename> directory</link>
      of Nixpkgs for more.  The NixOS manual also contains information
      about whole-system testing in virtual machine.
    </para>

    <para>
      Now, assume we want to build Hello with an old version of GCC, and
      with different <command>configure</command> flags.  A new
      <varname>build_exotic</varname> job can be written that simply
      <emphasis>overrides</emphasis> the relevant arguments passed to
      <varname>nixBuild</varname>:

<programlisting>
build_exotic =
  { system ? builtins.currentSystem }:

  let
    pkgs = import nixpkgs { inherit system; };
    build = jobs.build { inherit system; };
  in
    pkgs.lib.overrideDerivation build (attrs: {
      buildInputs = [ pkgs.gcc33 ];
      preConfigure = "gcc --version";
      configureFlags =
        attrs.configureFlags ++ [ "--disable-nls" ];
    });
</programlisting>

      The <varname>build_exotic</varname> job reuses
      <varname>build</varname> and overrides some of its arguments: it
      adds a dependency on GCC 3.3, a pre-configure phase that runs
      <command>gcc --version</command>, and adds the
      <literal>--disable-nls</literal> configure flags.
    </para>

    <para>
      This customization mechanism is very powerful.  For instance, it
      can be used to change the way Hello and <emphasis>all</emphasis>
      its dependencies–including the C library and compiler used to
      build it–are built.  See the Nixpkgs manual for more.
    </para>

  </section>

  <xi:include href="declarative-projects.xml" />

  <section>
    <title>Email Notifications</title>
    <para>
      Hydra can send email notifications when the status of a build changes. This provides
      immediate feedback to maintainers or committers when a change causes build failures.
    </para>

    <para>
      The simplest approach to enable Email Notifications is to use the ssmtp package, which
      simply hands off the emails to another SMTP server. For details on how to configure ssmtp,
      see the documentation for the <varname>networking.defaultMailServer</varname> option.
      To use ssmtp for the Hydra email notifications, add it to the path option of the Hydra services
      in your <filename>/etc/nixos/configuration.nix</filename> file:
<programlisting>
systemd.services.hydra-queue-runner.path = [ pkgs.ssmtp ];
systemd.services.hydra-server.path = [ pkgs.ssmtp ];
</programlisting>
    </para>
  </section>

</chapter>

<!--
 Local Variables:
 indent-tabs-mode: nil
 ispell-local-dictionary: "american"
 End:
 -->