ma27/hydra
0
0
Fork 0
forked from lix-project/hydra
Code Pull requests Activity

Manual: Remove tabs, indent consistently

Browse source
This commit is contained in:
Eelco Dolstra 2012-02-28 15:38:47 +01:00
parent da26294fdb
commit 918fc5e6df
2 changed files with 280 additions and 269 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

348
doc/manual/installation.xml
View file

@ -2,213 +2,217 @@
xmlns:xlink="http://www.w3.org/1999/xlink"
xml:id="chap-installation">
<title>Installation</title>
<para>
This chapter explains how to install Hydra on your own build farm server.
</para>
<section>
<title>Prerequisites</title>
<para>
To install and use Hydra you need to have installed the following dependencies:
<itemizedlist>
<listitem>Nix</listitem>
<listitem>either PostgreSQL or SQLite</listitem>
<listitem>many Perl packages, notably Catalyst, EmailSender,
and NixPerl (see the <link
xlink:href="https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/development/tools/misc/hydra/default.nix">Hydra
expression in Nixpkgs</link> for the complete
list).</listitem>
</itemizedlist>
At the moment, Hydra runs only on GNU/Linux
(<emphasis>i686-linux</emphasis> and
<emphasis>x86_64_linux</emphasis>).
</para>
<para>
For small projects, Hydra can be run on any reasonably modern
machine. For individual projects you can even run Hydra on a
laptop. However, the charm of a buildfarm server is usually that
it operates without disturbing the developer's working
environment and can serve releases over the internet. In
conjunction you should typically have your source code
administered in a version management system, such as
subversion. Therefore, you will probably want to install a
server that is connected to the internet. To scale up to large
and/or many projects, you will need at least a considerable
amount of diskspace to store builds. Since Hydra can schedule
multiple simultaneous build jobs, it can be useful to have a
multi-core machine, and/or attach multiple build machines in a
network to the central Hydra server.
</para>
<para>
Of course we think it is a good idea to use the <a
href="http://nixos.org/nixos">NixOS</a> GNU/Linux distribution
for your buildfarm server. But this is not a requirement. The
Nix software deployment system can be installed on any GNU/Linux
distribution in parallel to the regular package management
system. Thus, you can use Hydra on a Debian, Fedora, SuSE, or
Ubuntu system.
</para>
</section>
<section>
<title>Getting Nix</title>
<para>
If your server runs NixOS you are all set to continue with
installation of Hydra. Otherwise you first need to install Nix.
The latest stable version can be found one <link
xlink:href="http://nixos.org/nix/download.html">the Nix web
site</link>, along with a manual, which includes installation
instructions.
</para>
</section>
<section>
<title>Installation</title>
<para>
This chapter explains how to install Hydra on your own build farm server.
Hydra can be installed using Nixpkgs:
<screen>
nix-env -Ai hydra -f /path/to/nixpkgs</screen>
This makes the tools available in your Nix user environment,
<literal>$HOME/.nix-profile</literal> by default.
</para>
<section>
<title>Prerequisites</title>
<para>
To install and use Hydra you need to have installed the following dependencies:
<itemizedlist>
<listitem>Nix</listitem>
<listitem>either PostgreSQL or SQLite</listitem>
<listitem>many Perl packages, notably Catalyst,
EmailSender, and NixPerl (see the <link
xlink:href="https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/development/tools/misc/hydra/default.nix">Hydra
expression in Nixpkgs</link> for the complete
list).</listitem>
</itemizedlist>
At the moment, Hydra runs only on GNU/Linux
(<emphasis>i686-linux</emphasis> and
<emphasis>x86_64_linux</emphasis>).
</para>
<para>
For small projects, Hydra can be run on any reasonably
modern machine. For individual projects you can even run
Hydra on a laptop. However, the charm of a buildfarm server
is usually that it operates without disturbing the
developer's working environment and can serve releases over
the internet. In conjunction you should typically have your
source code administered in a version management system,
such as subversion. Therefore, you will probably want to
install a server that is connected to the internet. To scale
up to large and/or many projects, you will need at least a
considerable amount of diskspace to store builds. Since
Hydra can schedule multiple simultaneous build jobs, it can
be useful to have a multi-core machine, and/or attach
multiple build machines in a network to the central Hydra
server.
</para>
<para>
Of course we think it is a good idea to use the <a
href="http://nixos.org/nixos">NixOS</a> GNU/Linux
distribution for your buildfarm server. But this is not a
requirement. The Nix software deployment system can be
installed on any GNU/Linux distribution in parallel to the
regular package management system. Thus, you can use Hydra
on a Debian, Fedora, SuSE, or Ubuntu system.
</para>
</section>
<section>
<title>Getting Nix</title>
<para>
If your server runs NixOS you are all set to continue with
installation of Hydra. Otherwise you first need to install
Nix. The latest stable version can be found one <link
xlink:href="http://nixos.org/nix/download.html">the Nix web
site</link>, along with a manual, which includes installation
instructions.
</para>
</section>
<section>
<title>Installation</title>
<para>
Hydra can be installed using Nixpkgs:
<screen>
nix-env -Ai hydra -f /path/to/nixpkgs</screen>
This makes the tools available in your Nix user environment,
<literal>$HOME/.nix-profile</literal> by default.
</para>
<para>
Alternatively, the latest development snapshot can be
installed by visiting the URL
<link xlink:href="http://hydra.nixos.org/view/hydra/unstable"><literal>http://hydra.nixos.org/view/hydra/unstable</literal></link>
and use the one-click install available at one of the build pages. You can also
install Hydra through the channel by performing the following commands:
<para>
Alternatively, the latest development snapshot can be installed
by visiting the URL <link
xlink:href="http://hydra.nixos.org/view/hydra/unstable"><literal>http://hydra.nixos.org/view/hydra/unstable</literal></link>
and use the one-click install available at one of the build
pages. You can also install Hydra through the channel by
performing the following commands:
<screen>
nix-channel --add http://hydra.nixos.org/jobset/hydra/trunk/channel/latest
nix-channel --update
nix-env -i hydra</screen>
</para>
</para>
<para>
Command completion should reveal a number of command-line tools from Hydra:
<para>
Command completion should reveal a number of command-line tools from Hydra:
<screen>
<screen>
hydra-build hydra-evaluator hydra-server
hydra-eval-jobs hydra-queue-runner hydra-update-gc-roots</screen>
</para>
</section>
</para>
</section>
<section>
<title>Creating the database</title>
<para>
Hydra stores its results in a database, which can be a
PostgreSQL or SQLite database. The latter is easier to
setup, but the former scales better.
</para>
<section>
<title>Creating the database</title>
<para>
Hydra stores its results in a database, which can be a
PostgreSQL or SQLite database. The latter is easier to setup,
but the former scales better.
</para>
<para>To setup a PostgreSQL
database with <emphasis>hydra</emphasis> as database name
and user name, issue the following commands:
<screen>
<para>
To setup a PostgreSQL database with <emphasis>hydra</emphasis>
as database name and user name, issue the following commands:
<screen>
createdb hydra
echo "CREATE USER hydra WITH PASSWORD '&lt;your-password&gt;' ;" | psql hydra
cat $prefix/share/hydra/sql/hydra-postgresql.sql | psql hydra
echo "GRANT ALL ON DATABASE hydra TO hydra;" | psql hydra</screen>
Note that <emphasis>$prefix</emphasis> is the location of
Hydra in the nix store.
</para>
<para>
For SQLite, the following command is all it takes to
create the database:
Note that <emphasis>$prefix</emphasis> is the location of Hydra
in the nix store.
</para>
<screen>
cat $prefix/share/hydra/sql/hydra-sqlite.sql | sqlite3 /path/to/hydra.sqlite
</screen>
</para>
<para>
For SQLite, the following command is all it takes to create the
database:
<para>
To add a user <emphasis>root</emphasis> with <emphasis>admin</emphasis> privileges, execute:
<screen>
<screen>
cat $prefix/share/hydra/sql/hydra-sqlite.sql | sqlite3 /path/to/hydra.sqlite</screen>
</para>
<para>
To add a user <emphasis>root</emphasis> with
<emphasis>admin</emphasis> privileges, execute:
<screen>
echo "INSERT INTO Users(userName, emailAddress, password) VALUES ('root', 'some@email.adress.com', '$(echo -n foobar | sha1sum | cut -c1-40)');" | psql hydra
echo "INSERT INTO UserRoles(userName, role) values('root', 'admin');" | psql hydra
</screen>
For SQLite the same commands can be used, with
<command>psql hydra</command> replaced by
<command>sqlite3 /path/to/hydra.sqlite</command>.
</para>
echo "INSERT INTO UserRoles(userName, role) values('root', 'admin');" | psql hydra</screen>
<para>
Hydra uses an environment variable to know which database
should be used, and a variable which point to a location
that holds some state. To set these variables for a
PostgreSQL database, add the following to the
<filename>.profile</filename> of the user running the
Hydra services.
For SQLite the same commands can be used, with <command>psql
hydra</command> replaced by <command>sqlite3
/path/to/hydra.sqlite</command>.
</para>
<screen>
<para>
Hydra uses an environment variable to know which database should
be used, and a variable which point to a location that holds
some state. To set these variables for a PostgreSQL database,
add the following to the <filename>.profile</filename> of the
user running the Hydra services.
<screen>
export HYDRA_DBI="dbi:Pg:dbname=hydra;host=localhost;"
export HYDRA_DATA=/var/lib/hydra</screen>
Make sure that the <emphasis>HYDRA_DATA</emphasis>
directory exists and is writable for the user which will
run the Hydra services. For a SQLite database, the
<varname>HYDRA_DBI</varname> should be set to something
like <literal>dbi:SQLite:/path/to/hydra.sqlite</literal>
Make sure that the <emphasis>HYDRA_DATA</emphasis> directory
exists and is writable for the user which will run the Hydra
services. For a SQLite database, the
<varname>HYDRA_DBI</varname> should be set to something like
<literal>dbi:SQLite:/path/to/hydra.sqlite</literal>
</para>
</section>
</para>
</section>
<section>
<title>Getting Started</title>
<section>
<title>Getting Started</title>
<para>
To start the Hydra web server, execute:
<screen>
<para>
To start the Hydra web server, execute:
<screen>
hydra-server</screen>
When the server is started, you can browse to
<ulink>http://localhost:3000/</ulink> to start configuring
your Hydra instance.
</para>
When the server is started, you can browse to
<ulink>http://localhost:3000/</ulink> to start configuring
your Hydra instance.
</para>
<para>
The <command>hydra-server</command> command launches the
web server. There are two other processes that come into
play:
<para>
The <command>hydra-server</command> command launches the web
server. There are two other processes that come into play:
<itemizedlist>
<listitem>
The <emphasis>evaluator</emphasis> is responsible for
peridically evaluating job sets, checking out their
dependencies off their version control systems (VCS),
and queueing new builds if the result of the evaluation
changed. It is launched by the
<command>hydra-evaluator</command> command.
</listitem>
<listitem>
The <emphasis>queue runner</emphasis> launches builds
(using Nix) as they are queued by the evaluator,
scheduling them onto the configured Nix hosts. It is
launched using the
<command>hydra-queue-runner</command> command.
</listitem>
</itemizedlist>
All three processes must be running for Hydra to be fully
functional, though it's possible to temporarily stop any one
of them for maintenance purposes, for instance.
</para>
</section>
<itemizedlist>
<listitem>
The <emphasis>evaluator</emphasis> is responsible for
peridically evaluating job sets, checking out their
dependencies off their version control systems (VCS), and
queueing new builds if the result of the evaluation changed.
It is launched by the <command>hydra-evaluator</command>
command.
</listitem>
<listitem>
The <emphasis>queue runner</emphasis> launches builds (using
Nix) as they are queued by the evaluator, scheduling them
onto the configured Nix hosts. It is launched using the
<command>hydra-queue-runner</command> command.
</listitem>
</itemizedlist>
All three processes must be running for Hydra to be fully
functional, though it's possible to temporarily stop any one of
them for maintenance purposes, for instance.
</para>
</section>
</chapter>
<!--
Local Variables:
indent-tabs-mode: nil
ispell-local-dictionary: "american"
End:
-->

201
doc/manual/introduction.xml
View file

@ -19,67 +19,67 @@
more in-depth testing than what developers could feasibly do manually:
<itemizedlist>
<listitem> <emphasis>Portability testing</emphasis>: The
software may need to be built and tested on many different
platforms. It is infeasible for each developer to do this
before every commit.
</listitem>
<listitem> <emphasis>Portability testing</emphasis>: The
software may need to be built and tested on many different
platforms. It is infeasible for each developer to do this
before every commit.
</listitem>
<listitem> Likewise, many projects have very large test sets
(e.g., regression tests in a compiler, or stress tests in a
DBMS) that can take hours or days to run to completion.
</listitem>
<listitem> Likewise, many projects have very large test sets
(e.g., regression tests in a compiler, or stress tests in a
DBMS) that can take hours or days to run to completion.
</listitem>
<listitem> Many kinds of static and dynamic analyses can be
performed as part of the tests, such as code coverage runs and
static analyses.
</listitem>
<listitem> Many kinds of static and dynamic analyses can be
performed as part of the tests, such as code coverage runs and
static analyses.
</listitem>
<listitem> It may also be necessary to build many different
<emphasis>variants</emphasis> of the software. For instance,
it may be necessary to verify that the component builds with
various versions of a compiler.
</listitem>
<listitem> It may also be necessary to build many different
<emphasis>variants</emphasis> of the software. For instance,
it may be necessary to verify that the component builds with
various versions of a compiler.
</listitem>
<listitem> Developers typically use incremental building to
test their changes (since a full build may take too long), but
this is unreliable with many build management tools (such as
Make), i.e., the result of the incremental build might differ
from a full build.
</listitem>
<listitem> Developers typically use incremental building to
test their changes (since a full build may take too long), but
this is unreliable with many build management tools (such as
Make), i.e., the result of the incremental build might differ
from a full build.
</listitem>
<listitem> It ensures that the software can be built from the
sources under revision control. Users of version management
systems such as CVS and Subversion often forget to place
source files under revision control.
</listitem>
<listitem> It ensures that the software can be built from the
sources under revision control. Users of version management
systems such as CVS and Subversion often forget to place
source files under revision control.
</listitem>
<listitem> The machines on which the continuous integration
system runs ideally provides a clean, well-defined build
environment. If this environment is administered through
proper SCM techniques, then builds produced by the system can
be reproduced. In contrast, developer work environments are
typically not under any kind of SCM control.
</listitem>
<listitem> The machines on which the continuous integration
system runs ideally provides a clean, well-defined build
environment. If this environment is administered through
proper SCM techniques, then builds produced by the system can
be reproduced. In contrast, developer work environments are
typically not under any kind of SCM control.
</listitem>
<listitem> In large projects, developers often work on a
particular component of the project, and do not build and test
the composition of those components (again since this is
likely to take too long). To prevent the phenomenon of ``big
bang integration'', where components are only tested together
near the end of the development process, it is important to
test components together as soon as possible (hence
<emphasis>continuous integration</emphasis>).
</listitem>
<listitem> In large projects, developers often work on a
particular component of the project, and do not build and test
the composition of those components (again since this is
likely to take too long). To prevent the phenomenon of ``big
bang integration'', where components are only tested together
near the end of the development process, it is important to
test components together as soon as possible (hence
<emphasis>continuous integration</emphasis>).
</listitem>
<listitem> It allows software to be
<emphasis>released</emphasis> by automatically creating
packages that users can download and install. To do this
manually represents an often prohibitive amount of work, as
one may want to produce releases for many different platforms:
e.g., installers for Windows and Mac OS X, RPM or Debian
packages for certain Linux distributions, and so on.
</listitem>
<listitem> It allows software to be
<emphasis>released</emphasis> by automatically creating
packages that users can download and install. To do this
manually represents an often prohibitive amount of work, as
one may want to produce releases for many different platforms:
e.g., installers for Windows and Mac OS X, RPM or Debian
packages for certain Linux distributions, and so on.
</listitem>
</itemizedlist>
</para>
@ -92,19 +92,19 @@
<itemizedlist>
<listitem>It obtains the latest version of the component's
source code from the version management system.
</listitem>
<listitem>It obtains the latest version of the component's
source code from the version management system.
</listitem>
<listitem> It runs the component's build process (which
presumably includes the execution of the component's test
set).
</listitem>
<listitem> It runs the component's build process (which
presumably includes the execution of the component's test
set).
</listitem>
<listitem> It presents the results of the build (such as error
logs and releases) to the developers, e.g., by producing a web
page.
</listitem>
<listitem> It presents the results of the build (such as error
logs and releases) to the developers, e.g., by producing a web
page.
</listitem>
</itemizedlist>
@ -114,40 +114,40 @@
<itemizedlist>
<listitem> They do not manage the <emphasis>build
environment</emphasis>. The build environment consists of the
dependencies necessary to perform a build action, e.g.,
compilers, libraries, etc. Setting up the environment is
typically done manually, and without proper SCM control (so it
may be hard to reproduce a build at a later time). Manual
management of the environment scales poorly in the number of
configurations that must be supported. For instance, suppose
that we want to build a component that requires a certain
compiler X. We then have to go to each machine and install X.
If we later need a newer version of X, the process must be
repeated all over again. An ever worse problem occurs if
there are conflicting, mutually exclusive versions of the
dependencies. Thus, simply installing the latest version is
not an option. Of course, we can install these components in
different directories and manually pass the appropriate paths
to the build processes of the various components. But this is
a rather tiresome and error-prone process.
</listitem>
<listitem> They do not manage the <emphasis>build
environment</emphasis>. The build environment consists of the
dependencies necessary to perform a build action, e.g.,
compilers, libraries, etc. Setting up the environment is
typically done manually, and without proper SCM control (so it
may be hard to reproduce a build at a later time). Manual
management of the environment scales poorly in the number of
configurations that must be supported. For instance, suppose
that we want to build a component that requires a certain
compiler X. We then have to go to each machine and install X.
If we later need a newer version of X, the process must be
repeated all over again. An ever worse problem occurs if
there are conflicting, mutually exclusive versions of the
dependencies. Thus, simply installing the latest version is
not an option. Of course, we can install these components in
different directories and manually pass the appropriate paths
to the build processes of the various components. But this is
a rather tiresome and error-prone process.
</listitem>
<listitem> They do not easily support <emphasis>variability in software
systems</emphasis>. A system may have a great deal of build-time
variability: optional functionality, whether to build a debug or
production version, different versions of dependencies, and so on.
(For instance, the Linux kernel now has over 2,600 build-time
configuration switches.) It is therefore important that a continuous
integration tool can easily select and test different instances from
the configuration space of the system to reveal problems, such as
erroneous interactions between features. In a continuous integration
setting, it is also useful to test different combinations of versions
of subsystems, e.g., the head revision of a component against stable
releases of its dependencies, and vice versa, as this can reveal
various integration problems.
</listitem>
<listitem> They do not easily support <emphasis>variability in software
systems</emphasis>. A system may have a great deal of build-time
variability: optional functionality, whether to build a debug or
production version, different versions of dependencies, and so on.
(For instance, the Linux kernel now has over 2,600 build-time
configuration switches.) It is therefore important that a continuous
integration tool can easily select and test different instances from
the configuration space of the system to reveal problems, such as
erroneous interactions between features. In a continuous integration
setting, it is also useful to test different combinations of versions
of subsystems, e.g., the head revision of a component against stable
releases of its dependencies, and vice versa, as this can reveal
various integration problems.
</listitem>
</itemizedlist>
</para>
@ -258,3 +258,10 @@
</section>
</chapter>
<!--
Local Variables:
indent-tabs-mode: nil
ispell-local-dictionary: "american"
End:
-->