vriska/hydra
0
0
Fork 0
forked from lix-project/hydra
Code Pull requests Activity
4135 commits 1 branch 0 tags 17 MiB
Commit graph

2961 commits

Author SHA1 Message Date
Cole Helbling
5d3912962b RunCommand: use helper functions to ensure filenames and paths are the same 
Otherwise, it's possible someone updates the format in one place but not
the others, leading to broken or incorrect functionality.
 2022-01-28 13:03:15 -08:00
Cole Helbling
14090fbb86 runcommand-log.tt: init 2022-01-28 13:03:15 -08:00
Janne Heß
796ce165d4 RunCommand: Allow displaying command output 2022-01-28 13:03:15 -08:00
Janne Heß
4cb5e6cd94 RunCommand: Capture the output of the commands 2022-01-28 13:00:17 -08:00
Graham Christensen
ef362e92d1 GithubStatus: try pushing statuses for cached buildqueued/buildfinished events 2022-01-25 12:42:28 -05:00
Graham Christensen
f6e86efc9f
Merge pull request #1091 from Ma27/ssh-remote-store-location 
hydra-queue-runner: support store URIs declaring an alternate store location
 2022-01-24 14:10:54 -05:00
Graham Christensen
3a4ea6e563
Merge pull request #1124 from obsidiansystems/simplify--closure-of-path-set 
simplify, `computeFSClosure` can take a set now
 2022-01-24 14:09:35 -05:00
Graham Christensen
c280692f91
Merge pull request #1126 from DeterminateSystems/build-localhost-paths 
build-remote: copy missing paths from the binary cache to localhost
 2022-01-21 16:16:33 -05:00
Graham Christensen
44cd890ae3
Merge pull request #1130 from DeterminateSystems/prompt-password 
hydra-create-user: support prompting for password
 2022-01-21 15:38:39 -05:00
Graham Christensen
ba96a13407 Record metrics when getting the closure to localhost 2022-01-21 15:38:05 -05:00
Graham Christensen
7e9e82398d build-remote: copy missing paths from the binary cache to localhost 
In a Hydra instance I saw:

    possibly transient failure building ‘/nix/store/X.drv’ on ‘localhost’:
      dependency '/nix/store/Y' of '/nix/store/Y.drv' does not exist,
      and substitution is disabled

This is confusing because the Hydra in question does have substitution enabled.

This instance uses:

  keep-outputs = true
  keep-derivations = true

and an S3 binary cache which is not configured as a substituter in the nix.conf.

It appears this instance encountered a situation where store path Y was built
and present in the binary cache, and Y.drv was GC rooted on the instance,
however Y was not on the host.

When Hydra would try to build this path locally, it would look in the binary
cache to see if it was cached:

    (nix)
    439      bool valid = isValidPathUncached(storePath);
    440
    441      if (diskCache && !valid)
    442          // FIXME: handle valid = true case.
    443          diskCache->upsertNarInfo(getUri(), hashPart, 0);
    444
    445      return valid;

Since it was cached, the store path was considered Valid.

The queue monitor would then not put this input in for substitution, because
the path is valid:

    (hydra)
    470          if (!destStore->isValidPath(*i.second.path(*localStore, step->drv->name, i.first))) {
    471              valid = false;
    472              missing.insert_or_assign(i.first, i.second);
    473          }

Hydra appears to correctly handle the case of missing paths that need
to be substituted from the binary cache already, but since most
Hydra instances use `keep-outputs` *and* all paths in the binary cache
originate from that machine, it is not common for a path to be cached
and not GC rooted locally.

I'll run Hydra with this patch for a while and see if we run in to the
problem again.

A big thanks to John Ericson who helped debug this particular issue.
 2022-01-21 15:26:45 -05:00
Graham Christensen
e351054f61
Merge pull request #1129 from DeterminateSystems/fixup-argon2 
Fixup argon2 instructions in hydra-create-user
 2022-01-21 13:01:37 -05:00
Graham Christensen
0eeced7f08 hydra-create-user: Warn that creating users with a plaintext password is deprecated 2022-01-21 12:56:15 -05:00
Graham Christensen
98928a4125 fixups 2022-01-21 12:52:06 -05:00
Graham Christensen
76fbde6d6b Set noecho when reading passwords 2022-01-21 11:11:09 -05:00
Graham Christensen
b8f72d7ff2 LDAP support: require the prefix 'hydra_' to match documentation 2022-01-21 10:48:04 -05:00
Graham Christensen
bb893d0bd5 hydra-create-user: support prompting for passwords 
I'm not sure this is a good implementation as-is. It does work,
but the password gets echo'd to the screen. I tried to use IO::Prompt
but IO::Prompt really seems to want to read the password from ARGV.
 2022-01-21 10:40:56 -05:00
Graham Christensen
3a6c25489c Hydra::Helper::Nix: expose a captureStdoutStderrWithStdin, make it available in tests 2022-01-21 10:40:06 -05:00
Graham Christensen
d4fe7e55dd Hydra::Helper::Nix: sort exported functions 2022-01-21 10:40:06 -05:00
Graham Christensen
4945306a2b hydra-create-user: make docs about using --password-hash better 2022-01-21 10:39:22 -05:00
John Ericson
e7a1ae87aa simplify, computeFSClosure can take a set now 2022-01-20 14:53:01 -05:00
Graham Christensen
8c50cd06e4 machines: ensure the jobset name is present 2022-01-15 17:11:08 -05:00
Graham Christensen
c8dc6a9419 Plugins: get project and jobset information from the project and jobset tables 2022-01-15 15:58:02 -05:00
Graham Christensen
9dc40e0816 evaluator: don't save project, jobset on builds 2022-01-15 15:58:02 -05:00
Graham Christensen
c539deea99 builds: add a build->project func to get the project via the jobset 2022-01-15 15:58:02 -05:00
Graham Christensen
f120909547 builds: drop project, jobset columns 
Indexes were haphazardly dropped.
 2022-01-15 15:58:02 -05:00
Graham Christensen
1caff3a250
Merge pull request #1117 from DeterminateSystems/project-jobset/queue-runner 
queue-runner: track jobsets by ID
 2022-01-15 15:57:14 -05:00
Graham Christensen
9671d4d135
Merge pull request #1119 from DeterminateSystems/project-jobset/update-gc-roots 
Project jobset: update-gc-roots
 2022-01-15 15:57:06 -05:00
Graham Christensen
7544d4ff47 hydra-update-gc-roots: get project and jobset information from the project and jobset tables 2022-01-15 14:26:45 -05:00
Graham Christensen
72c3110002 queue-runner: track jobsets by ID 2022-01-15 14:06:00 -05:00
Graham Christensen
17c6bd4fd8 DeclarativeJobsets: get the jobset name from the jobset table 2022-01-15 13:46:32 -05:00
Graham Christensen
b2cdde0901 DeclarativeJobsets: test basic functionality 2022-01-15 13:46:32 -05:00
Graham Christensen
8c3c573953 hydra-eval-jobset: fixup old reference to project / jobset columns 2022-01-15 12:32:16 -05:00
Graham Christensen
2abcd84931
Merge pull request #1115 from DeterminateSystems/project-jobset/builds-json-repr 
Project jobset: update builds json repr
 2022-01-15 12:09:45 -05:00
Graham Christensen
6bb9adc1a5 Builds: get the project and jobset names from the their tables 2022-01-14 22:45:26 -05:00
Graham Christensen
f4c4b496d8 Projects: delete: delete all builds first 
Deleting jobsets first would fail because buildmetrics has an FK
to the jobset. However, the jobset / project relationship is not
marked as CASCADE.

Deleting all the builds automatically cascades to delete
buildmetrics, so deleting the relevant builds first, then deleting
the jobset solves it.
 2022-01-14 20:37:55 -05:00
Graham Christensen
31c6c26121 common.tt: fixup refs 2022-01-14 16:49:58 -05:00
Graham Christensen
c7c4759600 search: fix references to jobset / project info 2022-01-14 16:38:25 -05:00
Graham Christensen
0b33550871 search.tt: fixup project and jobset reference 2022-01-14 16:38:25 -05:00
Graham Christensen
42a871e413
Merge pull request #1111 from DeterminateSystems/project-jobset/queue-summary-machines 
Project jobset columns: fixup /queue-summary and /machines
 2022-01-14 15:34:43 -05:00
Graham Christensen
8a663f2cf8 machines: fixup refs 2022-01-14 15:23:19 -05:00
Graham Christensen
c945529f05 queue summary: fix refs 2022-01-14 15:23:19 -05:00
Graham Christensen
a81e358016 API: test api/push-github 2022-01-14 14:57:32 -05:00
Graham Christensen
20db82b001 API test /api/push 2022-01-14 14:57:24 -05:00
Graham Christensen
fe095a56c5 API: test /nrbuilds and fix jobset / project references 2022-01-14 14:57:15 -05:00
Graham Christensen
86473f4b3c API: fixup filtering latestbuilds by project and jobset 2022-01-14 14:57:10 -05:00
Graham Christensen
1f49a5af66 Fixup project name, jobset name, in buildToHash 2022-01-14 14:57:00 -05:00
Graham Christensen
97596c3e93 buildListColumns: remove project, jobset. 2022-01-14 12:48:48 -05:00
Graham Christensen
2803f06e6e CatalystUtils::buildListColumns: add jobset_id 2022-01-14 11:28:39 -05:00
Graham Christensen
12ff981d24 Builds: LatestSucceeded*: remove project, jobset references 2022-01-14 11:28:39 -05:00
Graham Christensen
5c5ecdb9fc Helper::Nix::isLocalStore: a file store is local 2022-01-14 11:28:39 -05:00
Graham Christensen
e019028ce9 common.tt: access project and jobset names from respective tables 2022-01-14 11:28:39 -05:00
Graham Christensen
4086f3bafd Result::Projects: give a builds method to access all of a project's builds 2022-01-14 11:28:39 -05:00
Graham Christensen
29f9e30b10
Merge pull request #1107 from DeterminateSystems/runcommand/project-jobset-name 
RunCommand: update the source of project and jobset names
 2022-01-14 09:26:07 -05:00
Graham Christensen
42edd3a9d8 hydra-notify: respond to cached_build_queued 2022-01-14 09:13:17 -05:00
Graham Christensen
6b7f1da11e hydra-notify: operate on cached_build_finished events 2022-01-14 09:13:17 -05:00
Graham Christensen
5b35a597b4 RunCommand: update project / jobset name refs 2022-01-13 20:44:51 -05:00
Maximilian Bosch
a18b487403
hydra-queue-runner: support store URIs declaring an alternate store location 
When having a builder like this in `/etc/nix/machines`

    ssh://mfbuild?remote-store=/home/bosch/store

Hydra cannot build there since it tries to pass the entire value to
`ssh(1)` which doesn't work. Also, an alternate store-location is e.g.
used if the user isn't a trusted user on the remote system and thus
cannot use `/nix/store`.

If such a URI is given, Hydra will now add a `--store /home/bosch/store`
to the `ssh`-command to select the appropriate location remotely.
 2022-01-12 15:56:05 +01:00
Graham Christensen
96b2200895 hydra-notify: sort subscriptions 2022-01-11 13:28:04 -05:00
Graham Christensen
4da80e736e hydra-eval-jobset: send notifications when cached queued / finished builds are submitted 2022-01-11 13:28:04 -05:00
Graham Christensen
f30ca3c423 showJobName: use jobset/project columns 2022-01-11 11:51:31 -05:00
Graham Christensen
ceb6ea252a CatalystUtils.pm: don't use Hydra::Helper::Nix 
Using it causes database information to get fixated early, before tests can set a
new database. We only used it in one case, and that is an absolute reference anyway. The
tests for channel generation are passing, and that uses
[requireLocalStore, so this should be fine.
 2022-01-11 11:47:42 -05:00
Cole Helbling
9b12a88d81 BuildDiff: ensure buildstatus is defined 
buildstatus won't be defined if e.g. a build is queued but none have
failed.
 2022-01-10 12:11:06 -08:00
Cole Helbling
b98dbe01c3 BuildDiff: slight cleanup 
Remove debugging Data::Dumper import, make comment describing function
inputs more readable.
 2022-01-10 12:11:06 -08:00
Cole Helbling
ef6a2c96e7 JobsetEval: add the result of buildDiff to $c->stash 2022-01-10 12:11:06 -08:00
Cole Helbling
98375e9086 BuildDiff: bring back the sort 2022-01-10 12:11:05 -08:00
Cole Helbling
ceb81e7d78 BuildDiff: actual list reference to $builds2 2022-01-10 12:11:05 -08:00
Cole Helbling
458214381d BuildDiff: document what the inputs mean 
They were tripping me up, so it helped to know exactly what they were at
the ~point they're used.
 2022-01-10 12:11:05 -08:00
Cole Helbling
ff12218d02 BuildDiff: cleanup imports and make slightly more readable 2022-01-10 12:11:05 -08:00
Cole Helbling
ec98bdaaaa BuildDiff: move cmdBuilds from Controller/JobsetEval 2022-01-10 12:11:05 -08:00
Cole Helbling
0c51f3ca7d BuildDiff: make actual hash ref 2022-01-10 12:11:05 -08:00
Cole Helbling
1349887974 BuildDiff: split out from JobsetEval GET impl 2022-01-10 12:11:05 -08:00
Cole Helbling
aba0422865 JobsetEval: allow restarting failed builds even with no eval to compare to 2022-01-10 12:10:56 -08:00
Graham Christensen
98c88a4dbf
Merge pull request #1097 from DeterminateSystems/api-test-warnings 
Api test warnings: clean up
 2022-01-10 14:03:38 -05:00
Graham Christensen
265908c1cd
Merge pull request #1099 from helsinki-systems/feat/dark-theme 
Provide a dark theme when the user prefers it
 2022-01-10 14:01:36 -05:00
Janne Heß
680cd43e40
Provide a dark theme when the user prefers it 2022-01-10 19:33:51 +01:00
Graham Christensen
e254b62f13
Merge pull request #1096 from DeterminateSystems/pathinput-revision-hash 
PathInput: make the hash part of the revision
 2022-01-10 13:32:58 -05:00
Janne Heß
a69693a832
hydra-eval-jobset: Fix two minor bugs 
I'm honestly too lazy to create two commits for fixing these one-line
issues so here's one.

The first hunk fixes the name of the projectName input. This is relevant
now because it gets logged and the log message looks stupid when there
is an input without a name.

The second hunk fixes a warning when using declarative non-flake
jobsets. The implementation may look weird but it's just the same as the
logical implication operator of nix.
 2022-01-10 18:28:41 +01:00
Graham Christensen
3874db4fb7 PathInput: make the hash part of the revision 2022-01-10 11:23:22 -05:00
Graham Christensen
832fa8c73d Controller/Jobset: make some warnings about undef's go away 2022-01-10 10:33:33 -05:00
Graham Christensen
e72d2225e9
Merge pull request #1094 from helsinki-systems/feat/print-whats-evaluated 
hydra-eval-jobset: Print the jobset that is evaluated
 2022-01-10 09:29:46 -05:00
Janne Heß
22f9045a03
Merge branch 'master' into feat/print-whats-evaluated 2022-01-10 13:45:57 +01:00
Janne Heß
f3d77c3e6b
hydra-eval-jobset: Print the jobset that is evaluated 
This is useful for systems that use concurrent evals
 2022-01-10 13:44:07 +01:00
Graham Christensen
1505a56a70 build.tt: only show RunCommandLogs tab if there are any to show 2022-01-07 15:14:02 -05:00
Graham Christensen
2f382ba067 Add migration 79: RunCommand logs 2022-01-07 15:05:33 -05:00
Graham Christensen
fe149613b3 Builds page: show RunCommand logs 2022-01-07 15:05:33 -05:00
Graham Christensen
a7aeb766aa RunCommandLogs: add a helper for if it failed with an exec error 2022-01-07 15:05:33 -05:00
Graham Christensen
d849856dcd RunCommandLogs: add helpers for if it is running/failed/signaled 2022-01-07 15:05:33 -05:00
Graham Christensen
c81acbf93e RunCommandLogs: test recording a signaled process without a core dump 2022-01-07 15:05:33 -05:00
Graham Christensen
d003fec8a5 RunCommandLogs: add a did_succeed helper 2022-01-07 15:05:33 -05:00
Graham Christensen
5bb3e2be78 Record the errno if exec fails 2022-01-07 15:05:33 -05:00
Graham Christensen
4ce8239cea RunCommand: create run logs for each execution 2022-01-07 15:05:33 -05:00
Graham Christensen
52843195db RunCommandLogs: init table 2022-01-07 15:05:33 -05:00
Graham Christensen
9708f6bf6f Fetch the tracker HTML from the config file, remove HYDRA_TRACKER 
The indentation in the hydra.conf makes it possible to include multi-line
strings without it being likely that the contents of the tracker
is mis-parsed or interrupts tho config parser.

It isn't impossible / foolproof probably, but it shouldn't be likely.
 2022-01-05 15:27:21 -05:00
Graham Christensen
2db422f7b0 Event: interested -> interestedIn 2021-12-21 20:57:53 -05:00
Graham Christensen
e84bbc7f90 hydra-eval-jobset: notify build_queued 2021-12-21 20:57:53 -05:00
Graham Christensen
a14501c616 Tasks: only execute the event if the plugin is interested in it 2021-12-21 20:57:53 -05:00
Graham Christensen
633fc36d6a TaskDispatcher: pre-declare the notify_no_such_plugin metric 2021-12-21 20:57:53 -05:00
Graham Christensen
d195e545f5 hydra-notify: listen for build_queued events 2021-12-21 20:57:53 -05:00
Graham Christensen
cf514baaf2 hydra-notify: track the duration of handling an event 
Currently we only track how long individual plugins take.
With #1083 we stop executing a lot of plugins, but we
don't have a way to measure its practical impact on the
execution time of handling events.
 2021-12-21 19:46:19 -05:00
Graham Christensen
f2f8d571f3 queue-runner-status: fixup from perlcritic-level-2 
>
> from_json was legacy / deprecated. This was the only use.
 2021-12-15 17:24:23 -05:00
Graham Christensen
e43f487157 perlcritic: return undef in list context too 2021-12-14 10:24:53 -05:00
Graham Christensen
aa7b37838a PerlCritic: MoreUtils -> SomeUtils 2021-12-14 10:16:25 -05:00
Graham Christensen
a658b80863 perlcritic: Readonly -> ReadonlyX 2021-12-14 10:16:25 -05:00
Graham Christensen
30a7f068b7 AttributeSet helper: skip wantarray 2021-12-14 10:16:25 -05:00
Graham Christensen
d2f03ca050 fixup foreach 2021-12-14 10:16:25 -05:00
Graham Christensen
4dfe787bc2 perlcritic: each() called at line 752, column 35. The each function may cause undefined behavior when operating on the hash while iterating. Use a foreach loop over the hash's keys or values instead. 2021-12-14 10:16:25 -05:00
Graham Christensen
7dcf6a01c6 JSON -> JSON::MaybeXS 2021-12-13 15:37:56 -05:00
Graham Christensen
26b197ea62 RunCommand: calculate all the commands to run against before starting 2021-12-08 20:29:32 -05:00
Graham Christensen
2ce0ab9f51 RunCommand: move JSON generation to its own function 2021-12-08 20:09:48 -05:00
Graham Christensen
7a5b8a38d9
builds page: capitalize tabs (#1072) 
Make tabs Title Case
 2021-12-08 20:02:14 -05:00
Graham Christensen
2cbeca5c44
Merge pull request #1071 from DeterminateSystems/log-fetches-evals 
hydra-eval-jobset: log fetches and evaluations
 2021-12-08 16:00:29 -05:00
Graham Christensen
264092169c hydra-eval-jobset: log fetches and evaluations 2021-12-08 09:25:27 -05:00
Cole Helbling
47cabac4bf hydra-eval-jobset: fix use of uninitialized value 
This happens with flake jobsets for obvious reasons (namely, that nixexprinput
and nixexprpath may be undefined for a flake jobset).

12:38:59 hydra-evaluator.1    | Use of uninitialized value $args[0] in join or string at /home/vin/workspace/vcs/hydra/src/script/hydra-eval-jobset line 648.
12:38:59 hydra-evaluator.1    | Use of uninitialized value $args[1] in join or string at /home/vin/workspace/vcs/hydra/src/script/hydra-eval-jobset line 648.
 2021-12-03 12:12:53 -08:00
Cole Helbling
5659d2982c Hydra/Controller/Root: remove deprecated LocalRegex 
11:38:20 hydra-server.1       | DEPRECATION WARNING: The Regex dispatch type is deprecated.
11:38:20 hydra-server.1       |   It is recommended that you convert Regex and LocalRegex
11:38:20 hydra-server.1       |   methods to Chained methods. at /nix/store/aa6gw57fnahd4824pbhmvcs0jlypmynq-hydra-perl-deps/lib/perl5/site_perl/5.32.1/Catalyst/DispatchType/Regex.pm line 210.
 2021-12-03 12:12:53 -08:00
Cole Helbling
d56815cae6 Hydra/Helper/CatalystUtils: default trim arg to empty string 
12:34:12 hydra-server.1       | Use of uninitialized value $s in substitution (s///) at /home/vin/workspace/vcs/hydra/src/script/../lib/Hydra/Helper/CatalystUtils.pm line 283, <$fh> line 1.
 2021-11-30 13:10:31 -08:00
Cole Helbling
78233b8083 Hydra: Unicode::Encoding is auto-applied 
11:28:20 hydra-server.1       | [warn] Unicode::Encoding plugin is auto-applied, please remove this from your appclass and make sure to define "encoding" config
 2021-11-30 12:42:05 -08:00
Cole Helbling
f26b3e1779 Hydra/Event: fix scalar context warning 
12:10:15 hydra-notify.1       | %channels_to_events{...} in scalar context better written as $channels_to_events{...} at /home/vin/workspace/vcs/hydra/src/lib/Hydra/Event.pm line 20.
 2021-11-30 12:42:01 -08:00
Graham Christensen
b06457c75c Titles of pages: make project:jobset:job names clickable 
But don't make the final element clickable when we're looking at that
thing.
 2021-11-22 20:20:06 -05:00
Graham Christensen
f1343b3a4c layout.tt: support a title with HTML in it 
By default, title is escaped. To support links in titles, support
providing the title with HTML in it.
 2021-11-22 13:38:53 -05:00
Graham Christensen
03323f6ef1 TT: add helpers for linking to jobs, jobsets, and projects, and for generating colon separated names. 2021-11-22 13:38:53 -05:00
Graham Christensen
ecb4697930 update-dbix: overwrite modifications 
Prevents authors from mistakenly corrupting the hashes
 2021-11-19 15:02:07 -05:00
Eelco Dolstra
9ae676072c
Merge pull request #1051 from DeterminateSystems/restart-cancel-resultset-p2 
Validate and fix canceling builds
 2021-10-27 18:22:20 +02:00
Graham Christensen
d19890a4e1 JobsetEval/cancel: test & fix 2021-10-27 11:43:36 -04:00
Graham Christensen
ef9a9fa481
Merge pull request #1025 from lukegb/hydra-better-errors 
Produce better errors for failing jobsets
 2021-10-26 12:35:45 -04:00
Luke Granger-Brown
67ebce8493 Output evaluation errors without crashing if aggregate job is broken. 
At the moment, aggregate jobs can easily break and cause the entire
evaluation to fail, which is not ideal. For Nixpkgs, we do have some
important aggregate jobs (like `tested`), but for debugging and building
purposes it's still useful to get a partial result even if the channel
won't actually advance.

This commit changes the behaviour of hydra-eval-jobs such that it
aggregates any errors found during the construction of an aggregate, and
will instead annotate the job with the evaluation failure such that it
shows up in a "cleaner" way.

There are really two types of failure that we care about: one is where
the attribute just ends up missing altogether in the final output, and
also where the attribute is in the output but fails to evaluate. Both
are handled here.

Note that this does mean that the same error message may be output
multiple times, but this aids debuggability because it'll be much
clearer what's blocking the job from being created.
 2021-10-26 10:14:34 +01:00
Graham Christensen
ffedbe5996 restart/cancelBuilds: always pass resultsets explicitly 2021-10-25 22:21:48 -04:00
Graham Christensen
592fef9d82
Merge pull request #1049 from DeterminateSystems/drop-systemtypes 
SystemTypes: drop database table.
 2021-10-24 21:46:57 -04:00
Graham Christensen
802a5dfe68
Merge pull request #1048 from DeterminateSystems/fixup-1003 
Hydra::Helper::Nix::getMachines: add a test
 2021-10-24 21:38:38 -04:00
Graham Christensen
ff888032eb SystemTypes: drop database table. It was originally removed in #65, but put back in fcd511c4de, and now totally unused. 2021-10-24 21:38:04 -04:00
Graham Christensen
5fbf1470bd (perl) machines file: support machine lines with multiple spaces between fields 2021-10-24 21:30:53 -04:00
Graham Christensen
b817124337 Hydra::Helper::Nix::getMachines: add a test 
Fix parsing breakage from #1003: assigning the lines to $lines broke chomp and the filters.

This test validates the parsing works as expected, and also fixes
a minor bug where '-' in features isn't pruned, like in the C++
repo.
 2021-10-23 22:53:55 -04:00
Graham Christensen
8cf158b7a9
Revert "Fix unhelpful error messages in aggregate jobs." 2021-10-23 22:23:29 -04:00
Graham Christensen
3516950d3c
Merge pull request #825 from samueldr/fix/unhelpful-errors-in-aggregates 
Fix unhelpful error messages in aggregate jobs.
 2021-10-23 20:34:50 -04:00
Graham Christensen
d98d74fa4f perlcritic: doesn't like prototypes 2021-10-20 13:09:39 -04:00
Graham Christensen
7095d00608 perlcritic: make all open() calls three-argument 
Two-argument calls end up parsing the second argument to guess what
should happen, three-arg parses ... less?
 2021-10-20 13:09:39 -04:00
Graham Christensen
f0d0358ee4
Merge pull request #1043 from DeterminateSystems/perl-warnings 
Fixup Perl warnings around undefined variables
 2021-10-20 10:48:05 -04:00
Graham Christensen
a887b3d346 fixup! EmailNotification: address Use of uninitialized value in numeric eq (==) 2021-10-20 10:40:08 -04:00
Graham Christensen
a36d23c1dd fixup! BitBucketStatus: address Use of uninitialized value in numeric eq (==) 2021-10-20 10:40:08 -04:00
Graham Christensen
3c38629fad GitHubPulls: output sorted json without using jq 2021-10-20 10:05:31 -04:00
Graham Christensen
30d36da057 CoverityScan: fixup tarballshandle introduced in 21e1ff0da1 2021-10-19 22:04:57 -04:00
Graham Christensen
093e235bf0 GithubRefs: fixup "my" variable $type masks earlier declaration in same scope 2021-10-19 22:02:41 -04:00
Graham Christensen
0f8d02894a hydra-eval-jobset: Scalar value @declInputs[0] better written as $declInputs[0] at hydra-eval-jobset line 570. 2021-10-19 21:52:39 -04:00
Graham Christensen
31cf249aed S3Backup: check for bzip2 and xz Nix Config support (Use of uninitialized value ::Config::xz in concatenation) 2021-10-19 21:52:39 -04:00
Graham Christensen
bf692c9e8c BitBucketStatus: address Use of uninitialized value in numeric eq (==) 2021-10-19 21:52:39 -04:00
Graham Christensen
4fd90ec784 EmailNotification: address Use of uninitialized value in numeric eq (==) 2021-10-19 21:52:39 -04:00
Graham Christensen
01112e9bd3 GitInput: deal with undefined deepClone 2021-10-19 21:52:39 -04:00
Eelco Dolstra
888ba467fd /eval: Return more columns 
Fixes #1038.
 2021-10-12 12:41:37 +02:00
Graham Christensen
4c8279044b
Merge pull request #1026 from DeterminateSystems/constituents 
add api endpoint: /build/<id>/constituents
 2021-09-28 09:21:21 -04:00
Eelco Dolstra
19e668b7cd
Merge pull request #981 from jonringer/pure_eval 
Allow for evaluator_pure_eval to be configured
 2021-09-27 20:25:01 +02:00
Eelco Dolstra
2745226ada
Merge pull request #1003 from DeterminateSystems/perlcritic-level-4 
perlcritic: level 4
 2021-09-27 20:23:55 +02:00
Eelco Dolstra
66375fd2d9 Fix 'Error loading tab: 0 error' with uBlock Origin 
/metrics-tab matches a rule in one of uBlock Origin's default
blocklists, so let's name it something else.
 2021-09-27 17:42:56 +02:00
Eelco Dolstra
9bb3e30a46 Fix building $(srcdir)/static 
Fixes

  mv: cannot move './static/bootstrap-4.3.1-dist' to './static/bootstrap/bootstrap-4.3.1-dist': Directory not empty

when 'make' is called more than once.
 2021-09-27 17:01:25 +02:00
Graham Christensen
c60c8d10ea add api endpoint: /build/<id>/constituents 
Returns a list of constituent builds
 2021-09-24 16:30:44 -04:00
Maximilian Bosch
255d4a91bc
jobset-eval: pass full parameter to "Compare to..." 
When I take a look at *all* failing builds (by clicking at `[...] more
jobs omitted`) and I try to compare the failures to another jobset, I'd
like to still view *all* failing builds in the compare-view.

This wasn't the case before since the `full=`-param was ignored by the
compare-buttons.
 2021-09-08 23:23:29 +02:00
Graham Christensen
741590c923 perlcritic: explicitly assign the result of readdir/file reads 2021-09-07 21:55:55 -04:00
Graham Christensen
85bb1c7ef9 (perlcritic) JSON::Any -> JSON::MaybeXS 2021-09-07 21:55:55 -04:00
Graham Christensen
c603ae35f0 perlcritic: Don't conditionally declare variables 2021-09-07 21:35:01 -04:00
Graham Christensen
b1879132af ResultSet::TaskRetries: add missing method, get_retryable_task 
Yet again, manual testing is proving to be insufficient. I'm pretty
sure I wrote this code but lost it in a rebase, or perhaps the switch
to result classes.

At any rate, this implements the actual "fetch a retry row and run it"
for the hydra-notify daemon.

Tested by hand.
 2021-09-07 11:12:10 -04:00
Graham Christensen
d6aa3f8d15
Merge pull request #1011 from DeterminateSystems/retryable-notifications 
Retryable notifications
 2021-09-07 10:04:32 -04:00
Graham Christensen
21e1ff0da1 perlcritic: don't open files as bare words 2021-09-06 22:20:17 -04:00
Graham Christensen
efd1d78b97 Hydra::Helper::Nix: getHydraNotifyPrometheusConfig fixup strict warning around double-declaring cfg 2021-09-06 22:13:33 -04:00
Graham Christensen
c880888f1e File::Slurp -> File::Slurper 2021-09-06 22:13:33 -04:00
Your Name
c54629b732 Helper/Nix.pm: perlcritic statsd in scalar context 2021-09-06 22:13:33 -04:00
Your Name
4677a7c894 perlcritic: use strict, use warnings 2021-09-06 22:13:33 -04:00
Graham Christensen
dd06ab7a99 Merge pull request #1004 from DeterminateSystems/avoid-a-b 
Avoid $a, $b
 2021-09-06 21:35:17 -04:00
Graham Christensen
ca4b8d449f
BitBucketStatus: tbuild -> build (typo) 2021-09-06 21:12:13 -04:00
Graham Christensen
d9df26ac5a
POD: improve wording, punctuation, caps 
Co-authored-by: Cole Helbling <cole.helbling@determinate.systems>
 2021-09-03 09:13:34 -04:00
Graham Christensen
ea3ae0693e Hook up the retryable tasks with hydra-notify 2021-09-02 10:06:26 -04:00
Graham Christensen
c0e86faa78 hydra-notify: call TaskRetries->save_task if a task fails 2021-09-02 10:06:26 -04:00
Graham Christensen
b0055a23df TaskDispatcher: Support re-queueing tasks, and dropping tasks after 100 failures. 2021-09-02 10:06:26 -04:00
Graham Christensen
d0b0fc21b3 ResultSet::TaskRetries: teach about saving tasks 2021-09-02 10:06:26 -04:00
Graham Christensen
147fa4d029 Result::TaskRetries: Teach about requeue 2021-09-02 10:06:26 -04:00
Graham Christensen
42c2d2f387 Hydra::Math: add an exponential_backoff function 2021-09-02 10:06:26 -04:00
Graham Christensen
29738364fb ResultSet::TaskRetries: add get_seconds_to_next_retry 
Get the number of seconds before the next retriable task is ready.

This number is specifically intended to be used as a timeout, where
`undef` means never time out.
 2021-09-02 10:06:26 -04:00
Graham Christensen
c4134c8e84 TaskRetries: init table 2021-09-02 10:06:26 -04:00
Graham Christensen
1f2adf61aa hydra-notify: extract runPluginsForEvent to a TaskDispatcher 2021-09-02 10:06:26 -04:00
Graham Christensen
fa57fb8f25 hydra.sql: explain update-dbix.pl map 2021-08-26 22:10:19 -04:00
Graham Christensen
397d13a300 DBIx::Class: migrate to use_namespaces 
This gives us a place to put helper functions that act on entire
tables, not just individual records.

This should be a backwards compatible change, except in places we're
manually using result class names.
 2021-08-26 12:37:19 -04:00
Your Name
24467a7bde Nix::getHydraNotifyPrometheusConfig: print errors if the configuration provided is invalid. 2021-08-24 11:35:38 -04:00
Your Name
4ebdcc290e fixup! hydra-notify: pre-declare metrics 2021-08-24 10:57:23 -04:00
Your Name
45e8fa5319 hydra-notify: support sending diagnostic dumps to STDERR on request 2021-08-24 10:56:13 -04:00
Your Name
de2282bcf4 hydra-notify: print out log lines indicating it is or is not launching the exporter 2021-08-24 10:56:13 -04:00
Your Name
5c1228e141 hydra-notify: pre-declare metrics 2021-08-24 10:56:13 -04:00
Your Name
6d7ee27d25 hydra-notify: make the prometheus endpoint configurable, default-off 2021-08-24 10:56:13 -04:00
Your Name
5d0ad5f649 hydra-notify: initial scratch take of prometheus events 2021-08-24 10:56:12 -04:00
Graham Christensen
e9ba0b56f6 BuildFinished : $b -> $build (perlcritic) 2021-08-20 21:52:32 -04:00
Graham Christensen
5e10648a98 CoverityScan: $b -> $build (perlcritic) 2021-08-20 21:52:32 -04:00
Graham Christensen
c929f5cac0 SlackNotification: $build -> $topbuild, $b -> $build (perlcritic) 2021-08-20 21:52:32 -04:00
Graham Christensen
b4f9807290 GithubStatus: $build -> $topbuild, $b -> $build (perlcritic) 2021-08-20 21:52:32 -04:00
Graham Christensen
5c9d8794b1 GiteaStatus: $build -> $topbuild, $b -> $build (perlcritic) 2021-08-20 21:52:32 -04:00
Graham Christensen
3920bf1df3 BitBucketStatus: $build -> $topbuild, $b -> $build (perlcritic) 2021-08-20 21:52:32 -04:00
Graham Christensen
bcf37ab705 GitlabStatus: $build -> $topbuild, $b -> $build (perlcritic) 2021-08-20 21:52:32 -04:00
Graham Christensen
b4a19fd3c8 HIpChatNotification: $build -> $topbuild, $b -> $build (perlcritic) 2021-08-20 21:52:32 -04:00
Graham Christensen
0d4ce15c62 EmailNotification: $build -> $topbuild, $b -> $build (perlcritic) 2021-08-20 21:52:32 -04:00
Graham Christensen
0f88014631 CircleCiNotification: $b -> $build (perlcritic) 2021-08-20 21:52:32 -04:00
Graham Christensen
b6714cdce3 JobsetEval: $a, $b to $left, $right (perlcritic) 2021-08-20 21:52:32 -04:00
Graham Christensen
d0fb3cbf8f Job: $b -> $build (perlcritic) 2021-08-20 21:52:32 -04:00
Graham Christensen
1cda08351f CatalystUtils: rename $b to $builds (perlcritic) 2021-08-20 21:52:32 -04:00
Your Name
4632132d02 fixup perlcritic: indirect new 
> Subroutine new called using indirect syntax at line 14, column 14.  See page 349 of PBP.  (Severity: 5)
 2021-08-20 11:39:19 -04:00
Your Name
6e65c3b320 hydra-notify: fixup printing of build IDs 
Used to print:

    sending notifications for build Hydra::Model::DB::Builds=HASH(0x124cf960)->id...

Now it prints:

    sending notifications for build 123...
 2021-08-16 16:09:05 -04:00
Your Name
2c50227082 hydra-notify: properly call new_event 2021-08-16 15:52:25 -04:00
Your Name
e572a5e576 hydra-notify: use Hydra::Event 2021-08-16 15:52:14 -04:00
Graham Christensen
fda3ed1d4e
Merge pull request #999 from DeterminateSystems/fixup/pr-998 
lib/makefile: include Event/*.pm
 2021-08-16 14:13:18 -04:00
Your Name
cfd3034af8 lib/makefile: include Event/*.pm 2021-08-16 14:04:48 -04:00
Graham Christensen
fa6d7abc13 hydra-notify: move BuildFinished processing to an Event 2021-08-13 16:51:29 -04:00
Graham Christensen
4a1389e36e hydra-notify: move StepFinished processing to an Event 2021-08-13 16:51:29 -04:00
Graham Christensen
4fdb20d3bd hydra-notify: move BuildStarted processing to an Event 2021-08-13 16:51:29 -04:00
Graham Christensen
10e85e3422 hydra-notify: Create a helper for running each plugin on an event 2021-08-13 16:51:29 -04:00
Graham Christensen
d02c6794f4 Event: teach how to execute a plugin 2021-08-13 16:51:29 -04:00
Graham Christensen
4e86e55008 Event.pm: add a new_event helper to parse and construct an Event 
An Event will be part of many Tasks
 2021-08-13 16:51:29 -04:00
Graham Christensen
64a3e75c10 Event: init structure and parse existing messages 2021-08-13 16:51:29 -04:00
Graham Christensen
a14c8ad5f8
Merge pull request #995 from DeterminateSystems/declarative-jobsets-plugin 
Declarative jobsets: move event handling to a plugin
 2021-08-12 15:56:13 -04:00
Graham Christensen
0f958f3425
Merge pull request #997 from DeterminateSystems/abstract-listener 
Abstract over postgres' LISTEN/NOTIFY
 2021-08-12 14:00:34 -04:00
Graham Christensen
5027003285 Abstract over postgres' LISTEN/NOTIFY 
This lets us test the event loop if we wanted, and lets us
test the listening behavior in isolation.
 2021-08-12 13:54:05 -04:00
Graham Christensen
593af41808 Declarative jobsets: move event handling to a plugin 
Declarative jobsets were sort of tucked in to the event hanlder
itself. It turned out that it could have been implemented as a
plugin without much trouble.
 2021-08-12 12:48:18 -04:00
Graham Christensen
9c5f317453 hydra-notify: move buildFinished query in to the function impl 
This is more consistent with the other event handlers, of dealing
with IDs and not objects.
 2021-08-12 12:30:35 -04:00
Eelco Dolstra
5edb58b314 Fix build 2021-08-10 13:47:16 +02:00
Graham Christensen
c7c322545d
Merge pull request #992 from DeterminateSystems/sql/fixup-comment 
hydra.sql: Update comment on regeneration
 2021-08-06 14:54:12 -04:00
Graham Christensen
4169f22231 update-dbix.pl: correct indentation 2021-08-06 14:40:57 -04:00
Graham Christensen
5bd8dc171b hydra.sql: Update comment on regeneration 
We no longer need to generate the hydra-postgres.sql document,
that is a relic from when sqlite was also supported.
 2021-08-06 14:40:34 -04:00
Shay Bergmann
503b0e0b6f
Remove unnecessary comment 2021-07-26 17:16:40 +00:00
Shay Bergmann
173ecb5d92
Apply requested changes to Config.pm 2021-07-26 17:16:40 +00:00
Shay Bergmann
4f95774ca3
Factor out common options 2021-07-26 03:43:09 +00:00
Shay Bergmann
e98bd0ec8d
Comment around dupe to avoid drift 2021-07-22 23:27:21 -04:00
Shay Bergmann
54c8028ec0
Make hydra-server honor config file includes 2021-07-22 17:25:30 -04:00
Nathan van Doorn
5c47343b40 Fix deprecation warnings with libpqxx usage 2021-07-05 19:45:25 +01:00
Graham Christensen
e02f34722f
Merge pull request #887 from helsinki-systems/feat/eval-overview-duration 
Show evaluation time in the evaluations overview
 2021-06-29 10:59:02 -04:00
Janne Heß
94b88b34ae
Show evaluation time in the evaluations overview 
Parts of this code have been authored by @grahamc
 2021-06-29 16:53:10 +02:00
Jonathan Ringer
24959a3ca6 Allow for evaluator_pure_eval to be configured 2021-06-28 16:24:40 -07:00
Damien Diederen
df7dab1291 GitInput: Include deepClone option in the cache key 
Without this commit, two jobsets using the same repository as input,
but different `deepClone` options, end up incorrectly sharing the same
"checkout" for a given (`uri`, `branch`, `revision`) tuple.  The
presence or absence of `.git` is determined by the jobset execution
order.

This patch adds the missing `isDeepClone` boolean to the cache key.

The database upgrade script empties the `CachedGitInputs` table, as we
don't know if existing checkouts are deep clones.  Unfortunately, this
generally forces rebuilds even for correct `deepClone` checkouts, as
the binary contents of `.git` are not deterministic.

Fixes #510
 2021-06-19 17:37:40 +02:00
Graham Christensen
09ad52ab60 Document getEvals a bit 
Also drop the $self parameter.

Co-authored-by: gustavderdrache
 2021-06-16 13:15:01 -04:00
Graham Christensen
5d95abf540 getBuilds: clarify the names of evals vs. the query builder 2021-06-16 11:53:04 -04:00
Graham Christensen
bf5c76feb6 getEvals: order by the eval table's ID 
I broke this when I added `me.` in f1e75c8bff

I added me. to disambiguate `id`, but:

* eval.id works on the per-build page
* me.id works on the other pages
* Just id works everywhere if I drop:

    , prefetch => { evaluationerror => [ ] },

  but this causes a query per row to collect the evaluationerror
  records later, this becomes significantly slow on non-trivial
  datasets.

Using evals->current_source_alias will use the correct alias
whether it is me or eval or something else.
 2021-06-16 11:24:30 -04:00
Graham Christensen
71453dd341 Expose Prometheus metrics at /metrics 
Exposes metrics:

* http_request_duration_seconds_bucket
* http_request_size_bytes_bucket
* http_response_size_bytes_bucket
* http_requests_total

with labels of action and controller to help identify popular
endpoints and their performance characteristics.
 2021-06-10 20:02:42 -04:00
Graham Christensen
382ba590ad Drop unused query from GET Build. 2021-06-01 11:47:05 -04:00
Graham Christensen
a9e4ede006 SQL: create better indexes for builds based on the jobset id 
These are primarily used by the jobsetOverview renders.
 2021-06-01 11:23:22 -04:00
Graham Christensen
719d0a6134 jobset overview: join jobsets to builds by jobset id 2021-06-01 11:19:33 -04:00
Graham Christensen
508d99d611 Join to builds via jobset_id when easy 2021-06-01 11:16:47 -04:00
Cole Helbling
45ced33ab6
Projects: don't respond with unnecessary declarative field 
If the project isn't declarative, who cares about it in the response? After
setting the `declfile` to an empty string, everything related to declarative-
ness is wiped out, anyways.
 2021-05-24 15:26:34 -07:00
Graham Christensen
3dc745de68
Merge pull request #964 from DeterminateSystems/fix-cascade 
Jobsets: remove defunct Jobs relationship
 2021-05-09 21:17:12 -04:00
Cole Helbling
7ba58ca3ea
Jobsets: remove defunct Jobs relationship 
It appears the Jobs table was removed in
8adb433e3b, but the Jobsets schema was never
updated to reflect this. This relationship was added in
efa1f1d4fb, roughly 3 months prior.

Previously, one would see a message similar to the following logged when
deleting a jobset:

    17:38:23 hydra-server.1       | DBIx::Class::Relationship::CascadeActions::delete(): Skipping cascade delete on relationship 'jobs' - related resultsource 'Hydra::Schema::Jobs' is not registered with this schema at /home/vin/workspace/vcs/hydra/src/script/../lib/Hydra/Controller/Jobset.pm line 106
 2021-05-09 17:52:48 -07:00
Graham Christensen
67b6f0d7ed
Merge pull request #962 from DeterminateSystems/del-.jobsets 
Project: delete the `.jobsets` jobset if project is no longer declarative
 2021-05-06 11:57:34 -04:00
Cole Helbling
6107040bf5
Project: clear decltype and declvalue when project is no longer declarative 2021-05-06 08:47:18 -07:00
Cole Helbling
bd9c52dbd0
Project: delete the .jobsets jobset if project is no longer declarative 
"No longer declarative" as defined by the "Edit project" page is an empty spec
file.
 2021-05-05 14:03:51 -07:00
Graham Christensen
6c1151b14a lazy tabs: trigger the load event 
Something in the upgrade of Bootstrap and JQuery broke lazy tab loading.
I don't understand what is providing the tab behavior, how it should
work, or what the correct fix is.

I can tell you that this patch fixes the issue: when loading a tab
with a URL fragment deep-linking to a lazily loaded tab... it now
loads.

Close #959
 2021-05-05 14:42:12 -04:00
Eelco Dolstra
886e6f85e4
Merge pull request #924 from hackworthltd/fix-flake-github-status 
Fix GitHub status update for private flakes.
 2021-05-03 16:24:51 +02:00
Cole Helbling
f3947acfc4
Projects: use JSON in Schema 2021-05-02 18:25:37 -07:00
Cole Helbling
ad13d13436
Project: add declfile, decltype, declvalue to API 
This makes it possible to create declarative projects via a PUT request, and
also exposes the currently-configured values to GET requests.
 2021-05-02 17:45:14 -07:00
Cole Helbling
7c9ed6b919
Project: fix DELETE route 
This appears to have been broken in ac3e8a4a59,
which removed the `jobsetevals` column from the Projects schema, but didn't
update the Controller accordingly.

Fixes the test added in the previous commit.
 2021-04-30 10:43:21 -07:00