Commit graph

2232 commits

Author SHA1 Message Date
Eelco Dolstra d959afebe1 Store unset descriptions etc. as nulls 2015-10-08 12:37:56 +02:00
Eelco Dolstra 4b31b23c04 Handle attrsets in meta.license (i.e. lib.licenses) 2015-10-08 11:56:30 +02:00
Eelco Dolstra b8ff29f0db Periodically compress logs 2015-10-07 13:12:24 +02:00
Eelco Dolstra 8e8e31ce86 Re-implement log size limits
The old queue runner already had this. However, we now store "log
limit exceeded" as a separate status code in the database.
2015-10-06 17:35:08 +02:00
Eelco Dolstra 82504fe010 hydra-queue-runner: Use substitutes
This allows Hydra to use binaries from available binary caches. It
makes the queue monitor thread quite a bit slower, so if you don't
want to use binary caches, it's better to add "--option
build-use-substitutes false" to the hydra-queue-runner invocation.

Fixed #243.
2015-10-05 14:57:44 +02:00
Eelco Dolstra ae2cc61be6 Fix hydra.queue.machines.total stat
This was broken by the fact that "machines" now also contains
removed/disabled machines.
2015-09-22 12:59:19 +02:00
Eelco Dolstra f8141fdc98 Set propagatedFrom for cached failed build steps 2015-09-11 15:55:26 +02:00
Eelco Dolstra 2dc9b9e32a Don't show "localhost" as machine for cached failed build steps 2015-09-11 14:12:07 +02:00
Eelco Dolstra aea1ed7445 hydra-evaluator: Fix error handling in fetching inputs
This caused bogus "cannot find the input containing the job
expression" errors.
2015-09-11 13:49:46 +02:00
aszlig 99cbe06b0d
channel-contents: Fix display of last paragraph.
The last paragraph states about package installation of the "following"
jobs, but it only applies to generic channels, so let's only display it
there.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:39:55 +02:00
aszlig 8b4f51f657
Channel.pm: Actually deliver channel expressions.
So this is the final part which is needed in order to be able to deliver
custom channels, everything else is now just polishing.

We do this by simply redirecting to the build product download URL and
we use binary_cache_url the same way as in NixChannel.pm.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:39:55 +02:00
aszlig db0ef9e49b
Tie custom channels into channel-contents.tt.
We should now get an overview and help text on how to add a particular
channel and also a bit of information about the builds that are required
for a channel to get upgraded.

Right now we only select the latest successful build in the latest
successful evaluation, so if someone wants to have more information about
which channel has failed, (s)he still has to look at the "Channels" tab
of the jobset.

We can make this more fancy at some later point if this is really
needed, because right now we're only interested in the latest build,
because it's the only thing necessary to deliver the channel contents.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:39:55 +02:00
aszlig f1dd08afc8
build.tt: Fix reference to build.ischannel.
It's actually lower-case _despite_ the spelling in the SQL file(s),
because the schema auto-generator from DBIx::Class doesn't take it into
account because it's working on SQLite and the latter seems to ignore
case.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:39:55 +02:00
aszlig d1de12cac1
product-list: Handle channel along with "file".
We want to have contents and detauls of channel expressions as well and
we already have that in product.type == file, so why not reuse the same
for the channel expression?

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:38:45 +02:00
aszlig c9aadd8b7d
Jobset.pm: Factor out build->eval mapping.
We now have a searchBuildsAndEvalsForJobset, which creates such a
mapping for us, so we don't need to duplicate code in jobs_tab and
channels_tab.

Also, we're going to use this for the overview of a particular channel
as well, so it makes sense to put it in CatalystUtils instead of
directly in Jobset.pm.

Instead of eval->jobs, it's now eval->builds, because it's really an
aggregate over the builds schema, rather than the job schema.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:38:45 +02:00
aszlig 1fdc258de0
ListBuilds: Simplify nix channel dispatcher.
We only allow channel/latest anyway, so it really doesn't make sense to
explicitly specify this in the PathPart and provide other dispatcher
once we have more than just "latest", which greatly simplifies the
dispatch tree.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:38:44 +02:00
aszlig ed361a0145
Build.pm: Don't get isChannel using buildproducts.
We now have a column for that, so no need for counting rows which was a
bit inefficient anyway, because we only would have needed the first row
in the result.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:38:44 +02:00
aszlig ba7402ed6c
Jobset.pm: Don't show channels in "Jobs" tab.
Now that we have our dedicated "Channels" tab, there is no need anymore
to show redundant information.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:36:38 +02:00
aszlig 8f7486cd9c
Jobset.pm: Detect channels by isChannel attribute.
We now no longer need that additional join of the build outputs and can
solely use the isChannel column of the Builds table to determine whether
it's a channel build.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:36:38 +02:00
aszlig 06b76ab275
Add isChannel column and meta attribute.
This is to properly separate channels from regular jobs and also make
sure that we can always iterate on them, no matter whether the build has
failed. The reason why we were not able to do this until now was because
we were iterating on the build products, and whenever some constituent
of a channel job has failed, we didn't get a build output.

So whenever there is a meta.isHydraChannel, we can now properly
distinguish it from the other jobs.

I still don't have any clue, why "make -C src/sql update-dbix" without
*any* modifications tries to create additional schema definitions. But
I've checked the md5sums of the existing schema definitions and they
don't seem to match, so it seems that they already have been tampered
with.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:36:38 +02:00
aszlig a235de3ad7
Make channels a product type instead of subtype.
Now we can provide different channel expressions for one particular
channel build. Not sure yet how this would be useful, but I found it
more appropriate to use a type instead of a subtype of "file".

This should get us consistent with the provious commit.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:12:13 +02:00
aszlig f6bb8afc30
Show build products on aggregates with channels.
This is to get a bit more consistency among channel builds but doesn't
do a radical change on the display. Ideally we may want to have a
channel overview with all the constituents and a small help showing how
the user can add the channel.

Unfortunately, this also introduces an inconsistency: We previously used
the *subtype* "channel", but now we're expecting "channel" as the type
of the product, so we need to change this for the channels overview as
well.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:12:12 +02:00
aszlig 7b60aed5ed
jobset: Add a new "channels" tab.
It's very similar to "jobs" and the code is pretty much the same, except
that we don't do filtering on it. At least it doesn't waste space for a
filter option when there are usually WAY less channel jobs than ordinary
jobs.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:08:24 +02:00
aszlig 19e7114f0e
Add channel as a product type.
Currently I'm using a (not very well) downscaled version of the NixOS
logo, so we want to replace it by a proper image ASAP.

Other than that, the idea is to have something like this in
hydra-build-products:

file channel $out/channel.tar.bz2

Right now of course, it's only displayed at the corresponding builds, so
we might want to have aggregates on all channels for a project, jobset
or maybe even single jobs?

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-09-10 17:08:24 +02:00
Eelco Dolstra c1dd3fe4be Machine status page: Show disabled machines and some machine stats 2015-09-09 16:51:43 +02:00
Eelco Dolstra 6075ac6fed Remove localhost hack 2015-09-09 16:50:59 +02:00
Eelco Dolstra ee9bf7ace7 Account steps with preferLocalBuild as a separate system type
They will show up in machineTypes as (e.g.) x86_64-linux:local instead
of x86_64-linux. This is to prevent the Hydra provisioner from
creating machines for steps that are supposed to be executed locally.
2015-09-02 13:42:25 +02:00
Eelco Dolstra 7e954aff03 Keep machine stats even when a machine is removed from the machines file
This is important for the Hydra provisioner, since it needs to be able
to see whether a disabled machine still has jobs running on it.
2015-09-02 13:31:47 +02:00
Eelco Dolstra 2d116d758a Return JSON for /eval 2015-09-02 13:13:55 +02:00
Eelco Dolstra 4f018b0612 Include the eval IDs in the JSON /build result 2015-09-02 12:57:58 +02:00
Eelco Dolstra 2b9bf7526d hydra-send-stats: Keep track of the number of machines 2015-08-27 12:05:06 +02:00
Eelco Dolstra 68796b2762 Fix email notification 2015-08-26 17:25:00 +02:00
Eelco Dolstra 2a7fbd57cc Allow the machines file to specify host public keys
It's easier for the Hydra provisioner to put host public keys in the
machines file than to separately manage the known_hosts file
(especially when the provisioner runs on a different machine).
2015-08-26 13:43:02 +02:00
Eelco Dolstra 88d7eb5247 Remove double entry for localhost on machines status page 2015-08-26 13:01:45 +02:00
Eelco Dolstra 7aa52517e9 Support multiple machines files
This is primarily useful for the Hydra provisioner, which can write
its machines to another file than /etc/nix/machines.
2015-08-25 15:34:53 +02:00
Rob Vermaas 442bed3b59 Force bash usage in dev-shell. Should fix #236 2015-08-17 15:47:58 +00:00
Eelco Dolstra 7a654259ff Wake the dispatcher when the machines file has changed 2015-08-17 15:48:10 +02:00
Eelco Dolstra 092d60735b Keep track of wait time per system type
I.e., how much time the currently runnable steps per system type have
been waiting. This is useful for deciding whether to provision more
machines.
2015-08-17 15:45:44 +02:00
Eelco Dolstra 99bfc37764 Don't abort steps that have an unsupported system type
This is necessary because the required system type can become
available later (e.g. by being provisioned by the
auto-scaler). However, in the future, we may want to fail steps if
they have been unsupported for more than a certain amount of time.
2015-08-17 15:10:41 +02:00
Eelco Dolstra ea1eb2e3fb Keep track of requiredSystemFeatures in the machine stats
For example, steps that require the "kvm" feature may require a
different kind of machine to be provisioned. This can also be used to
require performance-sensitive tests to run on a particular kind of
machine, e.g., by setting requiredSystemFeatures to something like
"ec2-i2.8xlarge".
2015-08-17 14:37:57 +02:00
Eelco Dolstra a6e3cb53b9 Use /usr/bin/env to find perl
This is nicer in nix-shell.
2015-08-17 14:18:20 +02:00
Eelco Dolstra d571e44b86 Keep stats for the Hydra auto scaler
"hydra-queue-runner --status" now prints how many runnable and running
build steps exist for each machine type. This allows additional
machines to be provisioned based on the Hydra load.
2015-08-17 13:50:41 +02:00
Eelco Dolstra 69e9f73cf6 Use nixUnstable for the tarball
Not needed for building, only for dev-shell.
2015-08-17 11:17:54 +02:00
Eelco Dolstra b0c8eecd37 Merge branch 'build-ng' 2015-08-12 20:32:48 +02:00
Eelco Dolstra 48d6f0de2a Fix the API test 2015-08-12 16:07:19 +02:00
Eelco Dolstra 62cd9d18b6 Check for a valid number of shares 2015-08-12 15:36:55 +02:00
Eelco Dolstra 006d105f93 Fix syntax error 2015-08-12 14:53:34 +02:00
Eelco Dolstra d4759c1da2 hydra-queue-runner: Detect changes to the scheduling shares 2015-08-12 13:17:56 +02:00
Eelco Dolstra 2e3899ed27 Don't set Expires header for logs of unfinished builds/steps 2015-08-12 12:22:14 +02:00
Eelco Dolstra 576dc0c120 For completeness, re-implement meta.schedulingPriority 2015-08-12 12:05:43 +02:00