hydra/doc/manual/projects.xml

<chapter xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xmlns:xi="http://www.w3.org/2001/XInclude"
         xml:id="chap-projects">

  <title>Creating and Managing Projects</title>

  <para>
    Once Hydra is installed and running, the next step is to add
    projects to the build farm. We follow the example of the <link
    xlink:href="http://nixos.org/patchelf.html">Patchelf
    project</link>, a software tool written in C and using the GNU
    Build System (GNU Autoconf and GNU Automake).
  </para>

  <para>
    Log in to the web interface of your Hydra installation using the
    user name and password you inserted in the database (by default,
    Hydra's web server listens on <link
    xlink:href="http://localhost:3000/"><literal>localhost:3000</literal></link>).
    Then follow the "Create Project" link to create a new project.
  </para>

  <section>
    <title>Project Information</title>

    <para>
      A project definition consists of some general information and a
      set of job sets.  The general information identifies a project,
      its owner, and current state of activity.

      Here's what we fill in for the patchelf project:

<screen>
Identifier: patchelf
</screen>

      The <emphasis>identifier</emphasis> is the identity of the
      project. It is used in URLs and in the names of build results.
    </para>

    <para>
      The identifier should be a unique name (it is the primary
      database key for the project table in the database). If you try
      to create a project with an already existing identifier you'd
      get an error message such as:

<screen>
I'm very sorry, but an error occurred:
DBIx::Class::ResultSet::create(): DBI Exception: DBD::SQLite::st execute failed: column name is not unique(19) at dbdimp.c line 402
</screen>

       So try to create the project after entering just the general
       information to figure out if you have chosen a unique name.
       Job sets can be added once the project has been created.

<screen>
Display name: Patchelf
</screen>

       The <emphasis>display name</emphasis> is used in menus.

<screen>
Description: A tool for modifying ELF binaries
</screen>

       The <emphasis>description</emphasis> is used as short
       documentation of the nature of the project.

<screen>
Owner: eelco
</screen>

       The <emphasis>owner</emphasis> of a project can create and edit
       job sets.

<screen>
Enabled: Yes
</screen>

       Only if the project is <emphasis>enabled</emphasis> are builds
       performed.
    </para>

    <para>
      Once created there should be an entry for the project in the
      sidebar.  Go to the project page for the <link
      xlink:href="http://localhost:3000/project/patchelf">Patchelf</link>
      project.
    </para>
  </section>

  <section>
    <title>Job Sets</title>

    <para>
      A project can consist of multiple <emphasis>job sets</emphasis>
      (hereafter <emphasis>jobsets</emphasis>), separate tasks that
      can be built separately, but may depend on each other (without
      cyclic dependencies, of course). Go to the <link
      xlink:href="http://localhost:3000/project/patchelf/edit">Edit</link>
      page of the Patchelf project and "Add a new jobset" by providing
      the following "Information":

<screen>
Identifier:     trunk
Description:    Trunk
Nix expression: release.nix in input patchelfSrc
</screen>

      This states that in order to build the <literal>trunk</literal>
      jobset, the Nix expression in the file
      <filename>release.nix</filename>, which can be obtained from
      input <literal>patchelfSrc</literal>, should be
      evaluated. (We'll have a look at
      <filename>release.nix</filename> later.)

    </para>

    <para>
      To realize a job we probably need a number of inputs, which can
      be declared in the table below. As many inputs as required can
      be added.  For patchelf we declare the following inputs.

<screen>
patchelfSrc
'Git checkout' https://github.com/NixOS/patchelf

nixpkgs 'Git checkout' https://github.com/NixOS/nixpkgs

officialRelease   Boolean false

system   String value "i686-linux"
</screen>
    </para>
  </section>

  <section>
    <title>Building Jobs</title>
  </section>

  <section>
    <title>Build Recipes</title>

    <para>
      Build jobs and <emphasis>build recipes</emphasis> for a jobset are
      specified in a text file written in the <link
      xlink:href="http://nixos.org/nix/">Nix language</link>.  The
      recipe is actually called a <emphasis>Nix expression</emphasis> in
      Nix parlance.  By convention this file is often called
      <filename>release.nix</filename>.
    </para>

    <para>
      The <filename>release.nix</filename> file is typically kept under
      version control, and the repository that contains it one of the
      build inputs of the corresponding–often called
      <literal>hydraConfig</literal> by convention.  The repository for
      that file and the actual file name are specified on the web
      interface of Hydra under the <literal>Setup</literal> tab of the
      jobset's overview page, under the <literal>Nix
      expression</literal> heading.  See, for example, the <link
      xlink:href="http://hydra.nixos.org/jobset/patchelf/trunk">jobset
      overview page</link> of the PatchELF project, and <link
      xlink:href="https://github.com/NixOS/patchelf/blob/master/release.nix">
      the corresponding Nix file</link>.
    </para>

    <para>
      Knowledge of the Nix language is recommended, but the example
      below should already give a good idea of how it works:
    </para>

    <example xml:id='ex-hello'>
      <title><filename>release.nix</filename> file for GNU Hello</title>
      <programlisting>
let
  pkgs = import &lt;nixpkgs&gt; {}; <co xml:id='ex-hello-co-import-nixpkgs' />

  jobs = rec { <co xml:id='ex-hello-co-jobs' />

    tarball = <co xml:id='ex-hello-co-tarball' />
      pkgs.releaseTools.sourceTarball { <co xml:id='ex-hello-co-source-tarball' />
        name = "hello-tarball";
        src = &lt;hello&gt;; <co xml:id='ex-hello-co-tarball-args' />
        buildInputs = (with pkgs; [ gettext texLive texinfo ]);
      };

    build = <co xml:id='ex-hello-co-build' />
      { system ? builtins.currentSystem }:  <co xml:id='ex-hello-co-build-args' />

      let pkgs = import &lt;nixpkgs&gt; { inherit system; }; in
      pkgs.releaseTools.nixBuild { <co xml:id='ex-hello-co-nix-build' />
        name = "hello";
        src = jobs.tarball;
        configureFlags = [ "--disable-silent-rules" ];
      };
  };
in
  jobs <co xml:id='ex-hello-co-body' />
      </programlisting>
    </example>

    <para>
      <xref linkend='ex-hello' /> shows what a
      <filename>release.nix</filename> file for <link
      xlink:href="http://www.gnu.org/software/hello/">GNU Hello</link>
      would you like.  GNU Hello is representative of many GNU
      and non-GNU free software projects:

      <itemizedlist>
        <listitem>it uses the GNU Build System, namely GNU Autoconf,
        and GNU Automake; for users, it means it can be installed
        using the <link
        xlink:href="http://www.gnu.org/prep/standards/html_node/Managing-Releases.html">usual
        <literal>./configure &amp;&amp; make install</literal>
        procedure</link>;
        </listitem>
        <listitem>it uses Gettext for internationalization;</listitem>
        <listitem>it has a Texinfo manual, which can be rendered as PDF
        with TeX.</listitem>
      </itemizedlist>

      The file defines a jobset consisting of two jobs:
      <literal>tarball</literal>, and <literal>build</literal>.  It
      contains the following elements (referenced from the figure by
      numbers):

      <calloutlist>

        <callout arearefs='ex-hello-co-import-nixpkgs'>
          <para>
            This defines a variable <varname>pkgs</varname> holding
            the set of packages provided by <link
            xlink:href="http://nixos.org/nixpkgs/">Nixpkgs</link>.
          </para>
          <para>
            Since <varname>nixpkgs</varname> appears in angle brackets,
            there must be a build input of that name in the Nix search
            path.  In this case, the web interface should show a
            <varname>nixpkgs</varname> build input, which is a checkout
            of the Nixpkgs source code repository; Hydra then adds this
            and other build inputs to the Nix search path when
            evaluating <filename>release.nix</filename>.
          </para>
        </callout>

        <callout arearefs='ex-hello-co-jobs'>
          <para>
            This defines a variable holding the two Hydra
            jobs–an <emphasis>attribute set</emphasis> in Nix.
          </para>
        </callout>

        <callout arearefs='ex-hello-co-tarball'>
          <para>
            This is the definition of the first job, named
            <varname>tarball</varname>.  The purpose of this job is to
            produce a usable source code tarball.
          </para>
        </callout>
        <callout arearefs='ex-hello-co-tarball-args'>
          <para>
            The <varname>tarball</varname> jobs expects a
            <varname>hello</varname> build input to be available in the
            Nix search path.  Again, this input is passed by Hydra and
            is meant to be a checkout of GNU Hello's source code
            repository.
          </para>
        </callout>
        <callout arearefs='ex-hello-co-source-tarball'>
          <para>
            The <varname>tarball</varname> job calls the
            <varname>sourceTarball</varname> function, which (roughly)
            runs <command>autoreconf &amp;&amp; ./configure &amp;&amp;
            make dist</command> on the checkout.  The
            <varname>buildInputs</varname> attribute specifies
            additional software dependencies for the
            job<footnote><para>The package names used in
            <varname>buildInputs</varname>–e.g.,
            <varname>texLive</varname>–are the names of the
            <emphasis>attributes</emphasis> corresponding to these
            packages in Nixpkgs, specifically in the <link
             xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix"><filename>all-packages.nix</filename></link>
            file.  See the section entitled “Package Naming” in the
            Nixpkgs manual for more information.</para></footnote>.
          </para>
        </callout>

        <callout arearefs='ex-hello-co-build'>
          <para>
            This is the definition of the <varname>build</varname>
            job, whose purpose is to build Hello from the tarball
            produced above.
          </para>
        </callout>
        <callout arearefs='ex-hello-co-build-args'>
          <para>
            The <varname>build</varname> function takes one
            parameter, <varname>system</varname>, which should be a string
            defining the Nix system type–e.g.,
            <literal>"x86_64-linux"</literal>.  Additionally, it refers
            to <varname>jobs.tarball</varname>, seen above.
          </para>
          <para>
            Hydra inspects the formal argument list of the function
            (here, the <varname>system</varname> argument) and passes it
            the corresponding parameter specified as a build input on
            Hydra's web interface.  Here, <varname>system</varname> is
            passed by Hydra when it calls <varname>build</varname>.
            Thus, it must be defined as a build input of type string in
            Hydra, which could take one of several values.
          </para>
          <para>
            The question mark after <literal>system</literal> defines
            the default value for this argument, and is only useful when
            debugging locally.
          </para>
        </callout>
        <callout arearefs='ex-hello-co-nix-build'>
          <para>
            The <varname>build</varname> job calls the
            <varname>nixBuild</varname> function, which unpacks the
            tarball, then runs <command>./configure &amp;&amp; make
            &amp;&amp; make check &amp;&amp; make install</command>.
          </para>
        </callout>

        <callout arearefs='ex-hello-co-body'>
          <para>
            Finally, the set of jobs is returned to Hydra, as a Nix
            attribute set.
          </para>
        </callout>
      </calloutlist>
    </para>
  </section>

  <section>
    <title>Building from the Command Line</title>

    <para>
      It is often useful to test a build recipe, for instance before
      it is actually used by Hydra, when testing changes, or when
      debugging a build issue.  Since build recipes for Hydra jobsets
      are just plain Nix expressions, they can be evaluated using the
      standard Nix tools.
    </para>

    <para>
      To evaluate the <varname>tarball</varname> jobset of <xref
      linkend='ex-hello' />, just run:

<screen>
$ nix-build release.nix -A tarball
</screen>

      However, doing this with <xref linkend='ex-hello' /> as is will
      probably yield an error like this:

<screen>
error: user-thrown exception: file `hello' was not found in the Nix search path (add it using $NIX_PATH or -I)
</screen>

       The error is self-explanatory.  Assuming
       <filename>$HOME/src/hello</filename> points to a checkout of
       Hello, this can be fixed this way:

<screen>
$ nix-build -I ~/src release.nix -A tarball
</screen>

      Similarly, the <varname>build</varname> jobset can be evaluated:

<screen>
$ nix-build -I ~/src release.nix -A build
</screen>

      The <varname>build</varname> job reuses the result of the
      <varname>tarball</varname> job, rebuilding it only if it needs to.
    </para>

  </section>

  <section>
    <title>Adding More Jobs</title>

    <para>
      <xref linkend='ex-hello' /> illustrates how to write the most
      basic jobs, <varname>tarball</varname> and
      <varname>build</varname>.  In practice, much more can be done by
      using features readily provided by Nixpkgs or by creating new jobs
      as customizations of existing jobs.
    </para>

    <para>
      For instance, test coverage report for projects compiled with GCC
      can be automatically generated using the
      <varname>coverageAnalysis</varname> function provided by Nixpkgs
      instead of <varname>nixBuild</varname>.  Back to our GNU Hello
      example, we can define a <varname>coverage</varname> job that
      produces an HTML code coverage report directly readable from the
      corresponding Hydra build page:

<programlisting>
coverage =
  { system ? builtins.currentSystem }:

  let pkgs = import nixpkgs { inherit system; }; in
  pkgs.releaseTools.coverageAnalysis {
    name = "hello";
    src = jobs.tarball;
    configureFlags = [ "--disable-silent-rules" ];
  };
</programlisting>

      As can be seen, the only difference compared to
      <varname>build</varname> is the use of
      <varname>coverageAnalysis</varname>.
    </para>

    <para>
      Nixpkgs provides many more build tools, including the ability to
      run build in virtual machines, which can themselves run another
      GNU/Linux distribution, which allows for the creation of packages
      for these distributions.  Please see <link
       xlink:href="https://github.com/NixOS/nixpkgs/tree/master/pkgs/build-support/release">the
      <filename>pkgs/build-support/release</filename> directory</link>
      of Nixpkgs for more.  The NixOS manual also contains information
      about whole-system testing in virtual machine.
    </para>

    <para>
      Now, assume we want to build Hello with an old version of GCC, and
      with different <command>configure</command> flags.  A new
      <varname>build_exotic</varname> job can be written that simply
      <emphasis>overrides</emphasis> the relevant arguments passed to
      <varname>nixBuild</varname>:

<programlisting>
build_exotic =
  { system ? builtins.currentSystem }:

  let
    pkgs = import nixpkgs { inherit system; };
    build = jobs.build { inherit system; };
  in
    pkgs.lib.overrideDerivation build (attrs: {
      buildInputs = [ pkgs.gcc33 ];
      preConfigure = "gcc --version";
      configureFlags =
        attrs.configureFlags ++ [ "--disable-nls" ];
    });
</programlisting>

      The <varname>build_exotic</varname> job reuses
      <varname>build</varname> and overrides some of its arguments: it
      adds a dependency on GCC 3.3, a pre-configure phase that runs
      <command>gcc --version</command>, and adds the
      <literal>--disable-nls</literal> configure flags.
    </para>

    <para>
      This customization mechanism is very powerful.  For instance, it
      can be used to change the way Hello and <emphasis>all</emphasis>
      its dependencies–including the C library and compiler used to
      build it–are built.  See the Nixpkgs manual for more.
    </para>

  </section>

  <xi:include href="declarative-projects.xml" />

</chapter>

<!--
 Local Variables:
 indent-tabs-mode: nil
 ispell-local-dictionary: "american"
 End:
 -->