raito/hydra
0
0
Fork 0
forked from lix-project/hydra
Code Pull requests Activity
%!d() commits 2 branches 0 tags 17 MiB
Commit graph

2986 commits

Author SHA1 Message Date
Graham Christensen f353a7ac41 update-gc-roots: try subselecting the jobset table 2022-03-19 23:56:47 -04:00
Graham Christensen 145667cb53 hydra-update-gc-roots: allow cached refs to the build's jobset 
Re-executing this search_related on every access turned out to
create very problematic performance. If a jobset had a lot of
error output stored in the jobset, and there were many hundreds
or thousands of active jobs, this could easily cause >1Gbps of
network traffic.
 2022-03-19 23:56:47 -04:00
Graham Christensen a582e4c485 HydraTestContext: add \n's to various dies 2022-03-19 14:46:53 -04:00
Graham Christensen 074a2f96bf hydra-eval-jobset: emit a useful error if constituents errored 2022-03-19 14:37:12 -04:00
Cole Helbling c0f826b92d hydra-queue-runner: get the listening port from the exposer itself 
Otherwise, when the port is randomly chosen (e.g. by specifying no port,
or a port of 0), it will just show that the port is 0 and not the port
that is actually serving the metrics.
 2022-03-14 08:41:45 -07:00
Cole Helbling 52a29d43e6 hydra-queue-runner: make registry member of State, configurable metrics port 
Thanks to the updated prometheus-cpp library, specifying a port of 0
will cause it to pick a random (available) port -- ideal for tests.
 2022-03-11 11:58:10 -08:00
Cole Helbling 3bf31bd6a6 hydra-queue-runner: add simple "up" exporter 
There are probably better ways to achieve this (and will likely need to
be refactored a bit to support further metrics).
 2022-03-10 12:36:58 -08:00
Graham Christensen 9316544abf
src/hydra-eval-jobs/hydra-eval-jobs.cc: .get<std::string> for drvPath 
Co-authored-by: Kayla Fire <firestack@users.noreply.github.com>
 2022-02-21 12:41:21 -05:00
Graham Christensen 290e0653ad hydra-eval-jobs: GC root aggregate jobs 2022-02-20 12:28:40 -05:00
John Ericson 445bba337b Make copyClosureTo take a regular C++ ref to the store 
This is syntactically lighter wait, and demonstates there are no weird
dynamic lifetimes involved, just regular passing reference to callee
which it only borrows for the duration of the call.
 2022-02-20 17:22:43 +00:00
John Ericson f14c583ce5 Use copyClosure instead of computeFSClosure + copyPaths 
It is more terse, and in the future it is possible `copyClosure` will
become more sophisticated.
 2022-02-19 11:59:17 -05:00
Graham Christensen 4c41ca08e1
Merge pull request #1155 from helsinki-systems/fix/graph-readability 
build-graphs: Fix readability in dark mode
 2022-02-14 11:27:37 -05:00
ajs124 1c84676527
Fit more content on screen 2022-02-13 18:33:37 +01:00
Janne Heß 6d146deaf0
build-graphs: Fix readability in dark mode 2022-02-13 14:00:17 +01:00
Graham Christensen 27ddde1e9e dynamic runcommand: print a notice on the build page if it is disabled 2022-02-11 15:04:54 -05:00
Cole Helbling a22a8fa62d AddBuilds: reject declarative jobsets with dynamic runcommand enabled if disabled elsewhere 2022-02-11 14:35:52 -05:00
Cole Helbling 928ba9e854 Controller/{Jobset,Project}: error when enabling dynamic runcommand but it's disabled elsewhere 2022-02-11 14:35:52 -05:00
Cole Helbling d680c209fe edit-project.tt: disable when disabled by server 
Also add a tooltip describing why it's disabled, to make it easier to
chase down.
 2022-02-11 14:35:52 -05:00
Cole Helbling 6053e5fd4b edit-jobset.tt: disable when disabled by project and server 
Also add a tooltip describing why it's disabled, to make it easier to
chase down.
 2022-02-11 14:35:52 -05:00
Cole Helbling dfd3a67424 project.tt: more info on why Dynamic RunCommand is disabled 2022-02-11 14:35:52 -05:00
Cole Helbling 3f4f183792 jobset.tt: more info on why Dynamic RunCommand is disabled 2022-02-11 14:35:52 -05:00
Graham Christensen 71c06f2ce7 LDAP normalization errors: note that the error came while normalizing the roles. 2022-02-11 10:55:27 -05:00
Graham Christensen f07fb7d279 LDAP support: include BC support for the YAML based loading 
Includes a refactoring of the configuration loader.
 2022-02-11 10:49:38 -05:00
Janne Heß 61d74a7194 Redo LDAP config in the main configuration and add role mappings 2022-02-11 10:49:38 -05:00
Graham Christensen d0bc0d0eda
Merge pull request #1152 from DeterminateSystems/parallel-tests 
Parallel tests, fix a hydra-queue-runner race condition
 2022-02-10 12:11:20 -05:00
Graham Christensen 4acaf9c8b0 hydra-queue-runner: don't dispatch until the machines parser has completed one run 
Periodically, I have seen tests fail because of out of order queue runner behavior:

    checking the queue for builds > 0...
    loading build 1 (tests:basic:empty_dir)
    aborting unsupported build step '...-empty-dir.drv' (type 'x86_64-linux')
    marking build 1 as failed
    adding new machine ‘localhost’

This patch should prevent the dispatcher from running before any machines are
made available.
 2022-02-10 10:54:30 -05:00
Graham Christensen 9ae7c8bddc Hydra::Helper::Exec add an expectOkay which dies with stdout / stderr on exit 2022-02-09 20:56:10 -05:00
Graham Christensen 845e6d4760 captureStdoutStderr*: move to Hydra::Helper::Exec which helps avoid some environment variable fixation problems 2022-02-09 14:28:50 -05:00
Graham Christensen 517dce285a eval_added event: change interface to traceID\tjobsetID\tevaluationID 
I was not going to break the interface until I noticed
the current implementation uses the string literal \t.
 2022-02-08 09:51:35 -05:00
Graham Christensen d512e6220f eval_failed event: change interface to traceID\tjobsetID 
I was not going to break the interface until I noticed the other eval_* events used literal \ts
 2022-02-08 09:51:35 -05:00
Graham Christensen 2597fa8c11 eval_cached event: change interface to traceID\tjobsetID\tevaluationID 
I was not going to break the interface until I noticed
the current implementation uses the string literal \t.
 2022-02-08 09:51:35 -05:00
Graham Christensen c30f084f32 eval_started event: change interface to traceID\tjobsetID 
I was not going to break the interface until I noticed
the current implementation uses the string literal \t.
 2022-02-08 09:51:35 -05:00
Graham Christensen 8a18326f2b Sort notification classes / events 2022-02-07 16:08:27 -05:00
Graham Christensen d8b56f022d RunCommand: print a warning if the hook isn't run because the project / jobset doens't have it enabled 2022-02-01 10:58:54 -05:00
Graham Christensen 3aa2393091 Jobsets: add a supportsDynamicRunCommand which also checks the project's dynamic runcommand support 2022-02-01 10:58:54 -05:00
Graham Christensen daa6864a58 Project result: add a supportsDynamicRunCommand helper 2022-02-01 10:58:54 -05:00
Graham Christensen bc1630bd27 fixup! RunCommand: Add a WIP execution of dynamic commands 2022-02-01 10:58:54 -05:00
Graham Christensen 8a96f07f58 Project: enable enabling dynamic runcommand per project 2022-02-01 10:58:54 -05:00
Graham Christensen 1affb1cfb1 jobset API: expose and check the enable_dynamic_run_command 2022-02-01 10:58:54 -05:00
Graham Christensen 726ea80e99 HTTP/Jobset: support setting / reading enable_dynamic_run_command 2022-02-01 10:58:54 -05:00
Graham Christensen 1802bd0113 Declarative Jobs: add support for the enable_dynamic_run_command flag 2022-02-01 10:58:54 -05:00
Graham Christensen 0810f5debc finish making the dynamic hooks only run on project & jobset agreement 2022-02-01 10:58:54 -05:00
Graham Christensen aef11685a0 regenerate schema files after adding the flag to the projects 2022-02-01 10:58:54 -05:00
Graham Christensen 85a53694c8 sql: add enable_dynamic_run_command to the Project as well 2022-02-01 10:58:54 -05:00
Graham Christensen a9bfabd672 sql: add a migration for enable_dynamic_run_command 2022-02-01 10:58:23 -05:00
Graham Christensen 3cce0c5ef6 Only run dynamic runcommand hooks if the jobset enables them 2022-02-01 10:57:30 -05:00
Graham Christensen 97a1d2d1d4 Jobsets: add enable_dynamic_run_command 2022-02-01 10:57:30 -05:00
Graham Christensen 216d8bee35 DynamicRunCommand: don't run if the build failed 2022-02-01 10:57:30 -05:00
Graham Christensen 1a30a0c2f1 Dynamic RunCommand: validate that the job's out exists, is a file (or points to a file) which is executable. 2022-02-01 10:57:30 -05:00
Graham Christensen e7f68045f4 DynamicRunCommand: pull out the function determining if a build is 
eligible for execution under dynamic run commands.
 2022-02-01 10:57:30 -05:00
Graham Christensen e56c49333f RunCommand: Add a WIP execution of dynamic commands 
This in-progress feature will run a dynamically generated set of
buildFinished hooks, which must be nested under the `runCommandHook.*`
attribute set. This implementation is not very good, with some to-dos:

1. Only run if the build succeeded
2. Verify the output is named $out and that it is an executable file
   (or a symlink to a file)
3. Require the jobset itself have a flag enabling the feature, since
   this feature can be a bit dangerous if various people of different
   trust levels can create the jobs.
 2022-02-01 10:57:30 -05:00
Graham Christensen ea311a0eb4 RunCommand: enable the plugin if dynamicruncommand is set 2022-02-01 10:57:30 -05:00
Graham Christensen 85b842e0ac
Merge pull request #1137 from DeterminateSystems/runcommand-logs 
Store and display the output of RunCommands
 2022-01-31 16:26:31 -05:00
Cole Helbling b57345ba1f hydra.sql: add IndexRunCommandLogsOnBuildID index 2022-01-31 12:56:34 -08:00
Cole Helbling d0b6329aa8 sql/upgrade-81: remove unnecessary comment 2022-01-31 12:55:36 -08:00
Cole Helbling 8c67e32480 RunCommand: ensure we reset the umask 2022-01-31 12:55:36 -08:00
Cole Helbling 34e4c119f4 build.tt: don't duplicate RunCommandLog buttons 2022-01-31 11:40:16 -08:00
Cole Helbling 61189ecca9 Helper/Nix: constructRunCommandLogPath: verify uuid is valid 
This shouldn't be possible normally, but it is possible to:

    $db->resultset('RunCommandLogs')->new({ uuid => "../etc/passwd" });

if you have access to the `$db`.
 2022-01-31 08:58:33 -08:00
Cole Helbling e381751564 Helper/Nix: constructRunCommandLogPath: return undef in case of an error 
This allows us to give a web request to an invalid UUID a 404.
 2022-01-31 08:58:33 -08:00
Cole Helbling 8eab7b8543 Helper/Nix: constructRunCommandLogPath: take RunCommandLog as input 
This way we ensure that it actually exists in the database, rather than
blindly trusting user-generated input.
 2022-01-31 08:58:33 -08:00
Cole Helbling 61914d56c6 runcommand-log.tt: escape the command 2022-01-31 08:58:33 -08:00
Cole Helbling 71bbb042db build.tt: link to the pretty, raw, and tail versions of the log 
Also split it out to a new div -- there are now 3 lines per
RunCommandLog -- the first saying when it started, the second saying how
long it ran for (or has been running), and the third with the buttons
for the pretty, raw, and tail versions of the log.
 2022-01-31 08:58:33 -08:00
Cole Helbling 3594ba942a Controller/Build: use showLog in view_runcommandlog 
This also adds the `runcommandlog` object to the stash so that we can
access its uuid as well as command run in order to display more useful
and specific information on the webpage.
 2022-01-31 08:58:33 -08:00
Cole Helbling 1d0076408b Controller/Build: pass log_uri to showLog in place of drvPath 
This way, we can reuse the `showLog` sub for other things, such as
`view_runcommandlog` (which doesn't have a drvPath attached).
 2022-01-31 08:58:33 -08:00
Cole Helbling ff390e89a6 Controller/Build: remove unused parameter from showLog 2022-01-31 08:58:33 -08:00
Cole Helbling fc3cf4ecb2 RunCommandLogs: identify and access via uuid 
Using a sha1 of the command combined with the build ID is not a
particularly good or unique identifier:

* A build could fail, be restarted, and then succeed -- assuming no
configuration changes, the sha1 hash of the command as well as the build
ID will be the same. This would lead to an overwritten log file.

* Allowing user input to influence filenames is not the best of ideas.
 2022-01-31 08:58:33 -08:00
Graham Christensen dcb0c1425c RunCommandLogs: set a UUID automatically 2022-01-31 08:58:33 -08:00
Graham Christensen cf49a05ff5 RunCommandLogs: add a uuid to each log entry 2022-01-31 08:58:33 -08:00
Graham Christensen 94ed9ed7ff
Merge pull request #1136 from DeterminateSystems/github-status-cached-evals 
GithubStatus: try pushing statuses for cached buildqueued/buildfinished events
 2022-01-31 09:11:37 -05:00
Cole Helbling 244300c1ad RunCommand: remove unused and problematic imports 
Since breaking the filename construction out to a helper function,
Hydra::Model::DB is no longer used. Importing Hydra::Helper::Nix,
however, has the potential to break tests, so just use the functions we
need without importing the entire module.
 2022-01-28 13:07:11 -08:00
Cole Helbling fdf6f4d3da RunCommand: use IPC::Run3::run3 instead 
run3 just seems to do better handling for what we want to do, and
requires less deep-reaching changes to this plugin to get it to play
nice, as IPC::Run::run would.
 2022-01-28 13:07:11 -08:00
Cole Helbling 3432cd7636 build.tt: split runcommand logic across multiple lines 
Helps with readability.
 2022-01-28 13:07:11 -08:00
Cole Helbling 1554750acc RunCommand: use make_path over mkdir 
This will make all necessary parent directories a la `mkdir -p`.
 2022-01-28 13:03:15 -08:00
Cole Helbling bf3c46ed43 RunCommand: use IPC::Run to spawn the command 
This allows `logPath`s with spaces and other characters that might
otherwise cause problems inside a `system()` call.
 2022-01-28 13:03:15 -08:00
Cole Helbling bb16f4fb10 RunCommand: set umask when creating log paths 
This uses the somewhat restrictive umask of 0027 so that people outside
the user or group cannot read the files. This also helps to inhibit
TOCTOU where someone else has a handle to our file before we chmod it
and after we close it.
 2022-01-28 13:03:15 -08:00
Cole Helbling 5d3912962b RunCommand: use helper functions to ensure filenames and paths are the same 
Otherwise, it's possible someone updates the format in one place but not
the others, leading to broken or incorrect functionality.
 2022-01-28 13:03:15 -08:00
Cole Helbling 14090fbb86 runcommand-log.tt: init 2022-01-28 13:03:15 -08:00
Janne Heß 796ce165d4 RunCommand: Allow displaying command output 2022-01-28 13:03:15 -08:00
Janne Heß 4cb5e6cd94 RunCommand: Capture the output of the commands 2022-01-28 13:00:17 -08:00
Graham Christensen ef362e92d1 GithubStatus: try pushing statuses for cached buildqueued/buildfinished events 2022-01-25 12:42:28 -05:00
Graham Christensen f6e86efc9f
Merge pull request #1091 from Ma27/ssh-remote-store-location 
hydra-queue-runner: support store URIs declaring an alternate store location
 2022-01-24 14:10:54 -05:00
Graham Christensen 3a4ea6e563
Merge pull request #1124 from obsidiansystems/simplify--closure-of-path-set 
simplify, `computeFSClosure` can take a set now
 2022-01-24 14:09:35 -05:00
Graham Christensen c280692f91
Merge pull request #1126 from DeterminateSystems/build-localhost-paths 
build-remote: copy missing paths from the binary cache to localhost
 2022-01-21 16:16:33 -05:00
Graham Christensen 44cd890ae3
Merge pull request #1130 from DeterminateSystems/prompt-password 
hydra-create-user: support prompting for password
 2022-01-21 15:38:39 -05:00
Graham Christensen ba96a13407 Record metrics when getting the closure to localhost 2022-01-21 15:38:05 -05:00
Graham Christensen 7e9e82398d build-remote: copy missing paths from the binary cache to localhost 
In a Hydra instance I saw:

    possibly transient failure building ‘/nix/store/X.drv’ on ‘localhost’:
      dependency '/nix/store/Y' of '/nix/store/Y.drv' does not exist,
      and substitution is disabled

This is confusing because the Hydra in question does have substitution enabled.

This instance uses:

  keep-outputs = true
  keep-derivations = true

and an S3 binary cache which is not configured as a substituter in the nix.conf.

It appears this instance encountered a situation where store path Y was built
and present in the binary cache, and Y.drv was GC rooted on the instance,
however Y was not on the host.

When Hydra would try to build this path locally, it would look in the binary
cache to see if it was cached:

    (nix)
    439      bool valid = isValidPathUncached(storePath);
    440
    441      if (diskCache && !valid)
    442          // FIXME: handle valid = true case.
    443          diskCache->upsertNarInfo(getUri(), hashPart, 0);
    444
    445      return valid;

Since it was cached, the store path was considered Valid.

The queue monitor would then not put this input in for substitution, because
the path is valid:

    (hydra)
    470          if (!destStore->isValidPath(*i.second.path(*localStore, step->drv->name, i.first))) {
    471              valid = false;
    472              missing.insert_or_assign(i.first, i.second);
    473          }

Hydra appears to correctly handle the case of missing paths that need
to be substituted from the binary cache already, but since most
Hydra instances use `keep-outputs` *and* all paths in the binary cache
originate from that machine, it is not common for a path to be cached
and not GC rooted locally.

I'll run Hydra with this patch for a while and see if we run in to the
problem again.

A big thanks to John Ericson who helped debug this particular issue.
 2022-01-21 15:26:45 -05:00
Graham Christensen e351054f61
Merge pull request #1129 from DeterminateSystems/fixup-argon2 
Fixup argon2 instructions in hydra-create-user
 2022-01-21 13:01:37 -05:00
Graham Christensen 0eeced7f08 hydra-create-user: Warn that creating users with a plaintext password is deprecated 2022-01-21 12:56:15 -05:00
Graham Christensen 98928a4125 fixups 2022-01-21 12:52:06 -05:00
Graham Christensen 76fbde6d6b Set noecho when reading passwords 2022-01-21 11:11:09 -05:00
Graham Christensen b8f72d7ff2 LDAP support: require the prefix 'hydra_' to match documentation 2022-01-21 10:48:04 -05:00
Graham Christensen bb893d0bd5 hydra-create-user: support prompting for passwords 
I'm not sure this is a good implementation as-is. It does work,
but the password gets echo'd to the screen. I tried to use IO::Prompt
but IO::Prompt really seems to want to read the password from ARGV.
 2022-01-21 10:40:56 -05:00
Graham Christensen 3a6c25489c Hydra::Helper::Nix: expose a captureStdoutStderrWithStdin, make it available in tests 2022-01-21 10:40:06 -05:00
Graham Christensen d4fe7e55dd Hydra::Helper::Nix: sort exported functions 2022-01-21 10:40:06 -05:00
Graham Christensen 4945306a2b hydra-create-user: make docs about using --password-hash better 2022-01-21 10:39:22 -05:00
John Ericson e7a1ae87aa simplify, computeFSClosure can take a set now 2022-01-20 14:53:01 -05:00
Graham Christensen 8c50cd06e4 machines: ensure the jobset name is present 2022-01-15 17:11:08 -05:00
Graham Christensen c8dc6a9419 Plugins: get project and jobset information from the project and jobset tables 2022-01-15 15:58:02 -05:00
Graham Christensen 9dc40e0816 evaluator: don't save project, jobset on builds 2022-01-15 15:58:02 -05:00
Graham Christensen c539deea99 builds: add a build->project func to get the project via the jobset 2022-01-15 15:58:02 -05:00
Graham Christensen f120909547 builds: drop project, jobset columns 
Indexes were haphazardly dropped.
 2022-01-15 15:58:02 -05:00
Graham Christensen 1caff3a250
Merge pull request #1117 from DeterminateSystems/project-jobset/queue-runner 
queue-runner: track jobsets by ID
 2022-01-15 15:57:14 -05:00
Graham Christensen 9671d4d135
Merge pull request #1119 from DeterminateSystems/project-jobset/update-gc-roots 
Project jobset: update-gc-roots
 2022-01-15 15:57:06 -05:00
Graham Christensen 7544d4ff47 hydra-update-gc-roots: get project and jobset information from the project and jobset tables 2022-01-15 14:26:45 -05:00
Graham Christensen 72c3110002 queue-runner: track jobsets by ID 2022-01-15 14:06:00 -05:00
Graham Christensen 17c6bd4fd8 DeclarativeJobsets: get the jobset name from the jobset table 2022-01-15 13:46:32 -05:00
Graham Christensen b2cdde0901 DeclarativeJobsets: test basic functionality 2022-01-15 13:46:32 -05:00
Graham Christensen 8c3c573953 hydra-eval-jobset: fixup old reference to project / jobset columns 2022-01-15 12:32:16 -05:00
Graham Christensen 2abcd84931
Merge pull request #1115 from DeterminateSystems/project-jobset/builds-json-repr 
Project jobset: update builds json repr
 2022-01-15 12:09:45 -05:00
Graham Christensen 6bb9adc1a5 Builds: get the project and jobset names from the their tables 2022-01-14 22:45:26 -05:00
Graham Christensen f4c4b496d8 Projects: delete: delete all builds first 
Deleting jobsets first would fail because buildmetrics has an FK
to the jobset. However, the jobset / project relationship is not
marked as CASCADE.

Deleting all the builds automatically cascades to delete
buildmetrics, so deleting the relevant builds first, then deleting
the jobset solves it.
 2022-01-14 20:37:55 -05:00
Graham Christensen 31c6c26121 common.tt: fixup refs 2022-01-14 16:49:58 -05:00
Graham Christensen c7c4759600 search: fix references to jobset / project info 2022-01-14 16:38:25 -05:00
Graham Christensen 0b33550871 search.tt: fixup project and jobset reference 2022-01-14 16:38:25 -05:00
Graham Christensen 42a871e413
Merge pull request #1111 from DeterminateSystems/project-jobset/queue-summary-machines 
Project jobset columns: fixup /queue-summary and /machines
 2022-01-14 15:34:43 -05:00
Graham Christensen 8a663f2cf8 machines: fixup refs 2022-01-14 15:23:19 -05:00
Graham Christensen c945529f05 queue summary: fix refs 2022-01-14 15:23:19 -05:00
Graham Christensen a81e358016 API: test api/push-github 2022-01-14 14:57:32 -05:00
Graham Christensen 20db82b001 API test /api/push 2022-01-14 14:57:24 -05:00
Graham Christensen fe095a56c5 API: test /nrbuilds and fix jobset / project references 2022-01-14 14:57:15 -05:00
Graham Christensen 86473f4b3c API: fixup filtering latestbuilds by project and jobset 2022-01-14 14:57:10 -05:00
Graham Christensen 1f49a5af66 Fixup project name, jobset name, in buildToHash 2022-01-14 14:57:00 -05:00
Graham Christensen 97596c3e93 buildListColumns: remove project, jobset. 2022-01-14 12:48:48 -05:00
Graham Christensen 2803f06e6e CatalystUtils::buildListColumns: add jobset_id 2022-01-14 11:28:39 -05:00
Graham Christensen 12ff981d24 Builds: LatestSucceeded*: remove project, jobset references 2022-01-14 11:28:39 -05:00
Graham Christensen 5c5ecdb9fc Helper::Nix::isLocalStore: a file store is local 2022-01-14 11:28:39 -05:00
Graham Christensen e019028ce9 common.tt: access project and jobset names from respective tables 2022-01-14 11:28:39 -05:00
Graham Christensen 4086f3bafd Result::Projects: give a builds method to access all of a project's builds 2022-01-14 11:28:39 -05:00
Graham Christensen 29f9e30b10
Merge pull request #1107 from DeterminateSystems/runcommand/project-jobset-name 
RunCommand: update the source of project and jobset names
 2022-01-14 09:26:07 -05:00
Graham Christensen 42edd3a9d8 hydra-notify: respond to cached_build_queued 2022-01-14 09:13:17 -05:00
Graham Christensen 6b7f1da11e hydra-notify: operate on cached_build_finished events 2022-01-14 09:13:17 -05:00
Graham Christensen 5b35a597b4 RunCommand: update project / jobset name refs 2022-01-13 20:44:51 -05:00
Maximilian Bosch a18b487403
hydra-queue-runner: support store URIs declaring an alternate store location 
When having a builder like this in `/etc/nix/machines`

    ssh://mfbuild?remote-store=/home/bosch/store

Hydra cannot build there since it tries to pass the entire value to
`ssh(1)` which doesn't work. Also, an alternate store-location is e.g.
used if the user isn't a trusted user on the remote system and thus
cannot use `/nix/store`.

If such a URI is given, Hydra will now add a `--store /home/bosch/store`
to the `ssh`-command to select the appropriate location remotely.
 2022-01-12 15:56:05 +01:00
Graham Christensen 96b2200895 hydra-notify: sort subscriptions 2022-01-11 13:28:04 -05:00
Graham Christensen 4da80e736e hydra-eval-jobset: send notifications when cached queued / finished builds are submitted 2022-01-11 13:28:04 -05:00
Graham Christensen f30ca3c423 showJobName: use jobset/project columns 2022-01-11 11:51:31 -05:00
Graham Christensen ceb6ea252a CatalystUtils.pm: don't use Hydra::Helper::Nix 
Using it causes database information to get fixated early, before tests can set a
new database. We only used it in one case, and that is an absolute reference anyway. The
tests for channel generation are passing, and that uses
[requireLocalStore, so this should be fine.
 2022-01-11 11:47:42 -05:00
Cole Helbling 9b12a88d81 BuildDiff: ensure buildstatus is defined 
buildstatus won't be defined if e.g. a build is queued but none have
failed.
 2022-01-10 12:11:06 -08:00
Cole Helbling b98dbe01c3 BuildDiff: slight cleanup 
Remove debugging Data::Dumper import, make comment describing function
inputs more readable.
 2022-01-10 12:11:06 -08:00
Cole Helbling ef6a2c96e7 JobsetEval: add the result of buildDiff to $c->stash 2022-01-10 12:11:06 -08:00
Cole Helbling 98375e9086 BuildDiff: bring back the sort 2022-01-10 12:11:05 -08:00
Cole Helbling ceb81e7d78 BuildDiff: actual list reference to $builds2 2022-01-10 12:11:05 -08:00
Cole Helbling 458214381d BuildDiff: document what the inputs mean 
They were tripping me up, so it helped to know exactly what they were at
the ~point they're used.
 2022-01-10 12:11:05 -08:00
Cole Helbling ff12218d02 BuildDiff: cleanup imports and make slightly more readable 2022-01-10 12:11:05 -08:00
Cole Helbling ec98bdaaaa BuildDiff: move cmdBuilds from Controller/JobsetEval 2022-01-10 12:11:05 -08:00
Cole Helbling 0c51f3ca7d BuildDiff: make actual hash ref 2022-01-10 12:11:05 -08:00
Cole Helbling 1349887974 BuildDiff: split out from JobsetEval GET impl 2022-01-10 12:11:05 -08:00
Cole Helbling aba0422865 JobsetEval: allow restarting failed builds even with no eval to compare to 2022-01-10 12:10:56 -08:00
Graham Christensen 98c88a4dbf
Merge pull request #1097 from DeterminateSystems/api-test-warnings 
Api test warnings: clean up
 2022-01-10 14:03:38 -05:00
Graham Christensen 265908c1cd
Merge pull request #1099 from helsinki-systems/feat/dark-theme 
Provide a dark theme when the user prefers it
 2022-01-10 14:01:36 -05:00
Janne Heß 680cd43e40
Provide a dark theme when the user prefers it 2022-01-10 19:33:51 +01:00
Graham Christensen e254b62f13
Merge pull request #1096 from DeterminateSystems/pathinput-revision-hash 
PathInput: make the hash part of the revision
 2022-01-10 13:32:58 -05:00
Janne Heß a69693a832
hydra-eval-jobset: Fix two minor bugs 
I'm honestly too lazy to create two commits for fixing these one-line
issues so here's one.

The first hunk fixes the name of the projectName input. This is relevant
now because it gets logged and the log message looks stupid when there
is an input without a name.

The second hunk fixes a warning when using declarative non-flake
jobsets. The implementation may look weird but it's just the same as the
logical implication operator of nix.
 2022-01-10 18:28:41 +01:00
Graham Christensen 3874db4fb7 PathInput: make the hash part of the revision 2022-01-10 11:23:22 -05:00
Graham Christensen 832fa8c73d Controller/Jobset: make some warnings about undef's go away 2022-01-10 10:33:33 -05:00
Graham Christensen e72d2225e9
Merge pull request #1094 from helsinki-systems/feat/print-whats-evaluated 
hydra-eval-jobset: Print the jobset that is evaluated
 2022-01-10 09:29:46 -05:00
Janne Heß 22f9045a03
Merge branch 'master' into feat/print-whats-evaluated 2022-01-10 13:45:57 +01:00
Janne Heß f3d77c3e6b
hydra-eval-jobset: Print the jobset that is evaluated 
This is useful for systems that use concurrent evals
 2022-01-10 13:44:07 +01:00
Graham Christensen 1505a56a70 build.tt: only show RunCommandLogs tab if there are any to show 2022-01-07 15:14:02 -05:00
Graham Christensen 2f382ba067 Add migration 79: RunCommand logs 2022-01-07 15:05:33 -05:00
Graham Christensen fe149613b3 Builds page: show RunCommand logs 2022-01-07 15:05:33 -05:00
Graham Christensen a7aeb766aa RunCommandLogs: add a helper for if it failed with an exec error 2022-01-07 15:05:33 -05:00
Graham Christensen d849856dcd RunCommandLogs: add helpers for if it is running/failed/signaled 2022-01-07 15:05:33 -05:00
Graham Christensen c81acbf93e RunCommandLogs: test recording a signaled process without a core dump 2022-01-07 15:05:33 -05:00
Graham Christensen d003fec8a5 RunCommandLogs: add a did_succeed helper 2022-01-07 15:05:33 -05:00
Graham Christensen 5bb3e2be78 Record the errno if exec fails 2022-01-07 15:05:33 -05:00
Graham Christensen 4ce8239cea RunCommand: create run logs for each execution 2022-01-07 15:05:33 -05:00
Graham Christensen 52843195db RunCommandLogs: init table 2022-01-07 15:05:33 -05:00
Graham Christensen 9708f6bf6f Fetch the tracker HTML from the config file, remove HYDRA_TRACKER 
The indentation in the hydra.conf makes it possible to include multi-line
strings without it being likely that the contents of the tracker
is mis-parsed or interrupts tho config parser.

It isn't impossible / foolproof probably, but it shouldn't be likely.
 2022-01-05 15:27:21 -05:00
Graham Christensen 2db422f7b0 Event: interested -> interestedIn 2021-12-21 20:57:53 -05:00
Graham Christensen e84bbc7f90 hydra-eval-jobset: notify build_queued 2021-12-21 20:57:53 -05:00
Graham Christensen a14501c616 Tasks: only execute the event if the plugin is interested in it 2021-12-21 20:57:53 -05:00
Graham Christensen 633fc36d6a TaskDispatcher: pre-declare the notify_no_such_plugin metric 2021-12-21 20:57:53 -05:00
Graham Christensen d195e545f5 hydra-notify: listen for build_queued events 2021-12-21 20:57:53 -05:00
Graham Christensen cf514baaf2 hydra-notify: track the duration of handling an event 
Currently we only track how long individual plugins take.
With #1083 we stop executing a lot of plugins, but we
don't have a way to measure its practical impact on the
execution time of handling events.
 2021-12-21 19:46:19 -05:00
Graham Christensen f2f8d571f3 queue-runner-status: fixup from perlcritic-level-2 
>
> from_json was legacy / deprecated. This was the only use.
 2021-12-15 17:24:23 -05:00
Graham Christensen e43f487157 perlcritic: return undef in list context too 2021-12-14 10:24:53 -05:00
Graham Christensen aa7b37838a PerlCritic: MoreUtils -> SomeUtils 2021-12-14 10:16:25 -05:00
Graham Christensen a658b80863 perlcritic: Readonly -> ReadonlyX 2021-12-14 10:16:25 -05:00
Graham Christensen 30a7f068b7 AttributeSet helper: skip wantarray 2021-12-14 10:16:25 -05:00
Graham Christensen d2f03ca050 fixup foreach 2021-12-14 10:16:25 -05:00
Graham Christensen 4dfe787bc2 perlcritic: each() called at line 752, column 35. The each function may cause undefined behavior when operating on the hash while iterating. Use a foreach loop over the hash's keys or values instead. 2021-12-14 10:16:25 -05:00
Graham Christensen 7dcf6a01c6 JSON -> JSON::MaybeXS 2021-12-13 15:37:56 -05:00
Graham Christensen 26b197ea62 RunCommand: calculate all the commands to run against before starting 2021-12-08 20:29:32 -05:00
Graham Christensen 2ce0ab9f51 RunCommand: move JSON generation to its own function 2021-12-08 20:09:48 -05:00
Graham Christensen 7a5b8a38d9
builds page: capitalize tabs (#1072) 
Make tabs Title Case
 2021-12-08 20:02:14 -05:00
Graham Christensen 2cbeca5c44
Merge pull request #1071 from DeterminateSystems/log-fetches-evals 
hydra-eval-jobset: log fetches and evaluations
 2021-12-08 16:00:29 -05:00
Graham Christensen 264092169c hydra-eval-jobset: log fetches and evaluations 2021-12-08 09:25:27 -05:00
Cole Helbling 47cabac4bf hydra-eval-jobset: fix use of uninitialized value 
This happens with flake jobsets for obvious reasons (namely, that nixexprinput
and nixexprpath may be undefined for a flake jobset).

12:38:59 hydra-evaluator.1    | Use of uninitialized value $args[0] in join or string at /home/vin/workspace/vcs/hydra/src/script/hydra-eval-jobset line 648.
12:38:59 hydra-evaluator.1    | Use of uninitialized value $args[1] in join or string at /home/vin/workspace/vcs/hydra/src/script/hydra-eval-jobset line 648.
 2021-12-03 12:12:53 -08:00
Cole Helbling 5659d2982c Hydra/Controller/Root: remove deprecated LocalRegex 
11:38:20 hydra-server.1       | DEPRECATION WARNING: The Regex dispatch type is deprecated.
11:38:20 hydra-server.1       |   It is recommended that you convert Regex and LocalRegex
11:38:20 hydra-server.1       |   methods to Chained methods. at /nix/store/aa6gw57fnahd4824pbhmvcs0jlypmynq-hydra-perl-deps/lib/perl5/site_perl/5.32.1/Catalyst/DispatchType/Regex.pm line 210.
 2021-12-03 12:12:53 -08:00
Cole Helbling d56815cae6 Hydra/Helper/CatalystUtils: default trim arg to empty string 
12:34:12 hydra-server.1       | Use of uninitialized value $s in substitution (s///) at /home/vin/workspace/vcs/hydra/src/script/../lib/Hydra/Helper/CatalystUtils.pm line 283, <$fh> line 1.
 2021-11-30 13:10:31 -08:00
Cole Helbling 78233b8083 Hydra: Unicode::Encoding is auto-applied 
11:28:20 hydra-server.1       | [warn] Unicode::Encoding plugin is auto-applied, please remove this from your appclass and make sure to define "encoding" config
 2021-11-30 12:42:05 -08:00
Cole Helbling f26b3e1779 Hydra/Event: fix scalar context warning 
12:10:15 hydra-notify.1       | %channels_to_events{...} in scalar context better written as $channels_to_events{...} at /home/vin/workspace/vcs/hydra/src/lib/Hydra/Event.pm line 20.
 2021-11-30 12:42:01 -08:00
Graham Christensen b06457c75c Titles of pages: make project:jobset:job names clickable 
But don't make the final element clickable when we're looking at that
thing.
 2021-11-22 20:20:06 -05:00
Graham Christensen f1343b3a4c layout.tt: support a title with HTML in it 
By default, title is escaped. To support links in titles, support
providing the title with HTML in it.
 2021-11-22 13:38:53 -05:00
Graham Christensen 03323f6ef1 TT: add helpers for linking to jobs, jobsets, and projects, and for generating colon separated names. 2021-11-22 13:38:53 -05:00
Graham Christensen ecb4697930 update-dbix: overwrite modifications 
Prevents authors from mistakenly corrupting the hashes
 2021-11-19 15:02:07 -05:00
Eelco Dolstra 9ae676072c
Merge pull request #1051 from DeterminateSystems/restart-cancel-resultset-p2 
Validate and fix canceling builds
 2021-10-27 18:22:20 +02:00
Graham Christensen d19890a4e1 JobsetEval/cancel: test & fix 2021-10-27 11:43:36 -04:00
Graham Christensen ef9a9fa481
Merge pull request #1025 from lukegb/hydra-better-errors 
Produce better errors for failing jobsets
 2021-10-26 12:35:45 -04:00