Commit graph

2957 commits

Author SHA1 Message Date
Maximilian Bosch c7716817a9
Update Nix to 2.13 2023-02-04 20:11:53 +01:00
Linus Heckemann 5b35e13898
hydra-queue-runner: use initializer lists for constructing JSON
And also fix the parts that were broken
2023-02-04 20:08:27 +01:00
Linus Heckemann 96e36201eb
hydra-queue-runner: adapt to nlohmann::json 2023-02-04 20:08:09 +01:00
Josef Kemetmüller ad99d3366f Fix MIME types when serving .js and .css
To correctly render HTML reports we make sure to return the following MIME
types instead of "text/plain"

- *.css: "text/css"
- *.js: "application/javascript"

Fixes: #1267
2022-12-29 22:26:59 +01:00
Graham Christensen f48f00ee6d
Merge pull request #1256 from cransom/hydra-evaluator-broken-connection
exit with error if database connectivity lost
2022-12-22 19:28:51 -05:00
Graham Christensen 01802efc17
Merge pull request #1263 from Ma27/fix-my-jobs-tab
Fix "My Jobs" tab in user dashboard
2022-12-05 01:55:49 +01:00
Eelco Dolstra 14d4624dc2
Merge pull request #1215 from knedlsepp/fix-hydra-with-latest-git
scmdiff: Hardcode `--git-dir`
2022-11-24 12:04:23 +01:00
Maximilian Bosch fd765bc97a
Fix "My Jobs" tab in user dashboard
Nowadays `Builds` doesn't reference `Project` directly anymore. This
means that simply resolving both `jobset` and `project` with a single
JOIN from `Builds` doesn't work anymore. Instead we need to resolve the
relation to `jobset` first and then the relation to `project`.

For similar fixes see e.g. c7c4759600.
2022-11-22 20:54:51 +01:00
Niklas Hambüchen d1d171ee90
renderInputDiff: Increase git hash length 6 -> 8
Nixpkgs has so many commits that length 6 is often ambiguous,
making the use of the shown values with git difficult.
2022-11-02 17:30:32 +01:00
Casey Ransom 70ad3a924a exit with error if database connectivity lost
There's currently no automatic recovery for disconnected databases in
the evaluator. This means if the database is ever temporarily
unavailable, hydra-evaluator will sit and spin with no work
accomplished.

If this condition is caught, the daemon will exit and systemd will be
responsible for resuming the service.
2022-10-26 16:13:40 -04:00
John Ericson 3526d61ff2 Merge remote-tracking branch 'upstream/master' into split-buildRemote 2022-10-25 11:24:54 -04:00
Théophane Hufschmitt 143c31734f Move all the build remote utils to their namespace
Just don't pollute the global one
2022-10-25 10:04:29 +02:00
Janne Heß 312cb42275
Merge pull request #1234 from helsinki-systems/feat/squiggly-line
Add a squiggly line to the Hydra link on hover
2022-10-21 22:57:20 +02:00
Jörg Thalheim 94d19e1972 hydra: fix localhost detection when protocol prefix are used 2022-09-29 20:46:13 +02:00
Maximilian Bosch d3fe4ffbf6
Job: expose closuresize and size (output size in the UI) as prometheus metrics 2022-09-22 10:47:22 +02:00
Graham Christensen d6cbf227cb
Merge pull request #1243 from K900/maybe-fix-trees
Run the JS code to make build trees collapsible at the right time
2022-09-08 09:55:14 -04:00
Eelco Dolstra 44e1efff7f Send the right nix-serve client version
We were using protocol version 6 but requesting version 4. The only
reason that this worked was because of a broken version check in
'nix-store --serve'. That was fixed in
c2d7456926,
which had the side-effect of breaking hydra-queue-runner.
2022-09-08 11:51:13 +02:00
Eelco Dolstra b16470c544 nix flake info -> nix flake metadata
This gets rid of a deprecation warning.
2022-09-06 19:23:20 +02:00
Janne Heß 371402c3c1
Drop the HipChat plugin
https://en.wikipedia.org/wiki/HipChat says:
> Following this, HipChat and Stride customers were migrated to the
> Slack group collaboration platform in a transition that was completed by
> February 2019.
2022-08-20 19:16:43 +02:00
Janne Heß 9addaeb17f
Add a squiggly line to the Hydra link on hover
The effect is the same as the one on links in mail bodys on
https://lists.apache.org/
2022-08-17 11:44:19 +02:00
Ilya K 2b8a8fdd9c Make the tree a little less dense 2022-08-12 09:46:32 +03:00
Ilya K 93bbd6925b Also restore the "expand all" and "collapse all" buttons 2022-08-12 09:46:17 +03:00
Ilya K 74caaa696e Run the JS code to make build trees collapsible at the right time 2022-08-11 13:30:19 +03:00
Marco Rebhan a58e2f1a64
Use libmagic for better output MIME detection 2022-08-04 22:34:58 +02:00
Janne Heß e2756042b8
Merge pull request #965 from helsinki-systems/css_more_content
Fit more content on screen
2022-07-13 23:47:04 +02:00
Janne Heß e05118171b
Merge pull request #1229 from helsinki-systems/fix/nix-cat-store
replace nix cat-store with nix store cat
2022-07-01 13:28:27 +02:00
ajs124 bb1f04ed86 AddBuilds: fix declarative jobsets with dynamic runcommand enabled
$project->{enable_dynamic_run_command} is undefined
2022-06-30 01:49:30 +02:00
ajs124 bab671124d replace nix cat-store with nix store cat
the former was deprecated in favor of the latter
2022-06-30 00:24:09 +02:00
Maximilian Bosch 5c01800fbe
flake: Update Nix to 2.9.1
NOTE: I'm well-aware that we have to be careful with this to avoid new
regressions on hydra.nixos.org, so this should only be merged after
extensive testing from more people.

Motivation: I updated Nix in my deployment to 2.9.1 and decided to also
update Hydra in one go (and compile it against the newer Nix). Given
that this also updates the C++ code in `hydra-{queue-runner,eval-jobs}`
this patch might become useful in the future though.
2022-06-16 14:54:57 +02:00
Josef Kemetmüller b6ea85a601 scmdiff: Hardcode --git-dir
The newest version of git refuses to work on repositories not owned by
the current user. This leads to issues with the /api/scmdiff endpoint:

May 27 11:16:05 myhydra hydra-server[923698]: fatal: unsafe repository ('/var/lib/hydra/scm/git/57ea036ec7ecd85c8dd085e02ecc6f12dd5c079a6203d16aea49f586cadfb2be' is owned by someone else)
May 27 11:16:05 myhydra hydra-server[923698]: To add an exception for this directory, call:
May 27 11:16:05 myhydra hydra-server[923698]:         git config --global --add safe.directory /var/lib/hydra/scm/git/57ea036ec7ecd85c8dd085e02ecc6f12dd5c079a6203d16aea49f586cadfb2be
May 27 11:16:05 myhydra hydra-server[923701]: warning: Not a git repository. Use --no-index to compare two paths outside a working tree
May 27 11:16:05 myhydra hydra-server[923701]: usage: git diff --no-index [<options>] <path> <path>

I used the same solution that was used in NixOS/nix#6440.

Fixes #1214
2022-05-27 12:12:44 +02:00
Maximilian Bosch a8b590014b
Fix email notifications for jobsets w/git-inputs
I started to wonder quite recently why Hydra doesn't send email
notifications anymore to me. I saw the following issue in the log of
`hydra-notify.service`:

    May 22 11:57:29 hydra 9bik0bxyxbrklhx6lqwifd6af8kj84va-hydra-notify[1887289]: fatal: unsafe repository ('/var/lib/hydra/scm/git/3e70c16c266ef70dc4198705a688acccf71e932878f178277c9ac47d133cc663' is owned by someone else)
    May 22 11:57:29 hydra 9bik0bxyxbrklhx6lqwifd6af8kj84va-hydra-notify[1887289]: To add an exception for this directory, call:
    May 22 11:57:29 hydra 9bik0bxyxbrklhx6lqwifd6af8kj84va-hydra-notify[1887289]:         git config --global --add safe.directory /var/lib/hydra/scm/git/3e70c16c266ef70dc4198705a688acccf71e932878f178277c9ac47d133cc663
    May 22 11:57:29 hydra 9bik0bxyxbrklhx6lqwifd6af8kj84va-hydra-notify[1886654]: error running build_finished hooks: command `git log --pretty=format:%H%x09%an%x09%ae%x09%at b0c30a7557685d25a8ab3f34fdb775e66db0bc4c..eaf28389fcebc2beca13a802f79b2cca6e9ca309 --git-dir=.git' failed with e>

This is also a problem because of Git's fix for CVE-2022-24765[1], so I
applied the same fix as for Nix[2], by using `--git-dir` which skips the
code-path for the ownership-check[3].

[1] https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/
[2] https://github.com/NixOS/nix/pull/6440
[3] To quote `git(1)`:
    > Specifying the location of the ".git" directory using this option
    > (or GIT_DIR environment variable) turns off the repository
    > discovery that tries to find a directory with ".git" subdirectory
2022-05-22 14:14:14 +02:00
Ulrik Strid 3c71be5b5b GithubPulls: Don't fail on missing Link 2022-05-18 08:14:00 +02:00
Kayla Firestack 2cdd7974de fix(hydra-eval-jobs): fix typo 2022-04-29 13:06:16 -04:00
Kayla Firestack 62cdbc4138 feat(hydra-eval-jobs.cc): add check_pid_status_nonblocking to catch handler 2022-04-21 10:55:51 -04:00
Kayla Firestack cb4fa0000f fix(hydra-eval-jobs.cc): add function to report pid status 2022-04-21 10:55:51 -04:00
Graham Christensen 5c90edd19f
Merge pull request #1103 from DeterminateSystems/runcommand/dynamic
Dynamic RunCommand
2022-04-19 10:09:47 -04:00
Graham Christensen e1965250b5
Merge pull request #1173 from DeterminateSystems/queue-runner-exporter
hydra-queue-runner metrics
2022-04-07 12:27:33 -04:00
Cole Helbling f8dc48f171
hydra-queue-runner: fixup: remove extraneous newline 2022-04-06 17:53:11 -07:00
Graham Christensen 59ac96a99c Track the number of steps created 2022-04-06 20:23:02 -04:00
Graham Christensen 1c12c5882f hydra queue runner: instrument the process of loading new builds with prom 2022-04-06 20:18:29 -04:00
Graham Christensen 5de08d412e queue metrics: refactor the metrics into a struct 2022-04-06 20:00:30 -04:00
Graham Christensen 46f52b4c4e bring back the working version Cole made 2022-04-06 15:49:38 -04:00
Cole Helbling 5bff730f2c WIP: I love it when they delete the assignment operator :) 2022-04-06 11:41:40 -07:00
Cole Helbling edf3c348f2 hydra-queue-runner: make entire address configurable 2022-04-06 10:59:45 -07:00
Cole Helbling 33bc60b83c hydra-queue-runner: move exporter back to State::run
It's (arguably) better than risking pinning the thread at 100% due to
the busy `while` loop.
2022-04-06 10:49:14 -07:00
Eelco Dolstra 71a036ed00 Update to Nix master
Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/ec90fc4d1f42db3c5e3c74dc186487d10a28c221' (2022-04-05)
  → 'github:NixOS/nix/5fe4fe823c193cbb7bfa05a468de91eeab09058d' (2022-04-05)
• Updated input 'nix/nixpkgs':
    'github:NixOS/nixpkgs/82891b5e2c2359d7e58d08849e4c89511ab94234' (2021-09-28)
  → 'github:NixOS/nixpkgs/530a53dcbc9437363471167a5e4762c5fcfa34a1' (2022-02-19)
2022-04-05 17:31:30 +02:00
Cole Helbling 8c5636fe18
hydra-queue-runner: use port 9198 by default
Co-authored-by: Graham Christensen <graham@grahamc.com>
2022-04-02 17:32:14 -07:00
Eelco Dolstra bcaad1c934 openConnection(): Don't throw exceptions in forked child
On hydra.nixos.org the queue runner had child processes that were
stuck handling an exception:

  Thread 1 (Thread 0x7f501f7fe640 (LWP 1413473) "bld~v54h5zkhmb3"):
  #0  futex_wait (private=0, expected=2, futex_word=0x7f50c27969b0 <_rtld_local+2480>) at ../sysdeps/nptl/futex-internal.h:146
  #1  __lll_lock_wait (futex=0x7f50c27969b0 <_rtld_local+2480>, private=0) at lowlevellock.c:52
  #2  0x00007f50c21eaee4 in __GI___pthread_mutex_lock (mutex=0x7f50c27969b0 <_rtld_local+2480>) at ../nptl/pthread_mutex_lock.c:115
  #3  0x00007f50c1854bef in __GI___dl_iterate_phdr (callback=0x7f50c190c020 <_Unwind_IteratePhdrCallback>, data=0x7f501f7fb040) at dl-iteratephdr.c:40
  #4  0x00007f50c190d2d1 in _Unwind_Find_FDE () from /nix/store/65hafbsx91127farbmyyv4r5ifgjdg43-glibc-2.33-117/lib/libgcc_s.so.1
  #5  0x00007f50c19099b3 in uw_frame_state_for () from /nix/store/65hafbsx91127farbmyyv4r5ifgjdg43-glibc-2.33-117/lib/libgcc_s.so.1
  #6  0x00007f50c190ab90 in uw_init_context_1 () from /nix/store/65hafbsx91127farbmyyv4r5ifgjdg43-glibc-2.33-117/lib/libgcc_s.so.1
  #7  0x00007f50c190b08e in _Unwind_RaiseException () from /nix/store/65hafbsx91127farbmyyv4r5ifgjdg43-glibc-2.33-117/lib/libgcc_s.so.1
  #8  0x00007f50c1b02ab7 in __cxa_throw () from /nix/store/dd8swlwhpdhn6bv219562vyxhi8278hs-gcc-10.3.0-lib/lib/libstdc++.so.6
  #9  0x00007f50c1d01abe in nix::parseURL (url="root@cb893012.packethost.net") at src/libutil/url.cc:53
  #10 0x0000000000484f55 in extraStoreArgs (machine="root@cb893012.packethost.net") at build-remote.cc:35
  #11 operator() (__closure=0x7f4fe9fe0420) at build-remote.cc:79
  ...

Maybe the fork happened while another thread was holding some global
stack unwinding lock
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71744). Anyway, since
the hanging child inherits all file descriptors to SSH clients,
shutting down remote builds (via 'child.to = -1' in
State::buildRemote()) doesn't work and 'child.pid.wait()' hangs
forever.

So let's not do any significant work between fork and exec.
2022-03-30 22:39:48 +02:00
ajs124 089da272c7 fix build against nix 2.7.0
fix build after such commits as df552ff53e68dff8ca360adbdbea214ece1d08ee
and e862833ec662c1bffbe31b9a229147de391e801a
2022-03-29 15:38:24 -04:00
ajs124 c64c5f0a7e hydra-queue-runner: rename build-result.hh to hydra-build-result.hh 2022-03-29 15:34:29 -04:00
Graham Christensen 3b048ed136 Revert "Revert "Use copyClosure instead of computeFSClosure + copyPaths""
This reverts commit 8e3ada2afc.
2022-03-29 15:28:47 -04:00
Cole Helbling 4789eba92c hydra-queue-runer: split metrics functionality into its own function 2022-03-29 10:55:28 -07:00
Cole Helbling 928b3b8268 hydra-queue-runner: fix priority of flag over config file 2022-03-29 10:42:07 -07:00
Cole Helbling 5ddb9a98ca fixup! hydra-queue-runner: log message before and after exporter is started 2022-03-29 08:47:41 -07:00
Cole Helbling 905a7a7beb hydra-queue-runner: read metrics port from queue_runner_metrics_port config 2022-03-29 08:46:43 -07:00
Cole Helbling 9cdc5aceed hydra-queue-runner: log message before and after exporter is started
This way, if something goes wrong between the two, it's easier to narrow
down where the issue lies.
2022-03-29 08:41:19 -07:00
Théophane Hufschmitt 6e571e26ff Build the resolved derivation and not the original one 2022-03-29 17:05:30 +02:00
Théophane Hufschmitt 92b627ac1b Remove an accidental re-indenting of a comment
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-03-29 17:04:19 +02:00
Théophane Hufschmitt b430d41afd Use the BuildOptions more eagerly 2022-03-29 17:04:19 +02:00
Théophane Hufschmitt fd0ae78eba Factor out the copying from the build store 2022-03-29 17:04:19 +02:00
Théophane Hufschmitt a778a89f04 Factor out the queryPathInfos part of the build 2022-03-29 17:04:19 +02:00
Théophane Hufschmitt 365776f5d7 Factor out the building part 2022-03-29 17:04:19 +02:00
Théophane Hufschmitt 9f1b911625 Factor more stuff out 2022-03-29 17:04:17 +02:00
Théophane Hufschmitt 2f494b7834 Factor out the creation of the log file 2022-03-29 16:52:59 +02:00
Théophane Hufschmitt 5db8642224 Factor out a struct representing a connection to a machine 2022-03-29 16:52:59 +02:00
Cole Helbling 8e3ada2afc Revert "Use copyClosure instead of computeFSClosure + copyPaths"
This reverts commit f14c583ce5.
2022-03-28 09:54:02 -07:00
Eelco Dolstra 962bf36939
Merge pull request #1162 from obsidiansystems/less-ref
Make `copyClosureTo` take a regular C++ ref to the store
2022-03-23 16:25:59 +01:00
Eelco Dolstra 3390415905
Merge pull request #1125 from obsidiansystems/simplify--copyClosure
Use `copyClosure` instead of `computeFSClosure` + `copyPaths`
2022-03-23 12:49:22 +01:00
Cole Helbling 8503a7917b fixup! hydra-queue-runner: make registry member of State, configurable metrics port 2022-03-22 13:38:13 -07:00
Graham Christensen e5393c2cf8 fixup: make id non-ambiguous 2022-03-19 23:56:47 -04:00
Graham Christensen 137be3452e Reduce the jobset cols on the remaining two queries 2022-03-19 23:56:47 -04:00
Graham Christensen f353a7ac41 update-gc-roots: try subselecting the jobset table 2022-03-19 23:56:47 -04:00
Graham Christensen 145667cb53 hydra-update-gc-roots: allow cached refs to the build's jobset
Re-executing this search_related on every access turned out to
create very problematic performance. If a jobset had a lot of
error output stored in the jobset, and there were many hundreds
or thousands of active jobs, this could easily cause >1Gbps of
network traffic.
2022-03-19 23:56:47 -04:00
Graham Christensen a582e4c485 HydraTestContext: add \n's to various dies 2022-03-19 14:46:53 -04:00
Graham Christensen 074a2f96bf hydra-eval-jobset: emit a useful error if constituents errored 2022-03-19 14:37:12 -04:00
Cole Helbling c0f826b92d hydra-queue-runner: get the listening port from the exposer itself
Otherwise, when the port is randomly chosen (e.g. by specifying no port,
or a port of 0), it will just show that the port is 0 and not the port
that is actually serving the metrics.
2022-03-14 08:41:45 -07:00
Cole Helbling 52a29d43e6 hydra-queue-runner: make registry member of State, configurable metrics port
Thanks to the updated prometheus-cpp library, specifying a port of 0
will cause it to pick a random (available) port -- ideal for tests.
2022-03-11 11:58:10 -08:00
Cole Helbling 3bf31bd6a6 hydra-queue-runner: add simple "up" exporter
There are probably better ways to achieve this (and will likely need to
be refactored a bit to support further metrics).
2022-03-10 12:36:58 -08:00
Graham Christensen 9316544abf
src/hydra-eval-jobs/hydra-eval-jobs.cc: .get<std::string> for drvPath
Co-authored-by: Kayla Fire <firestack@users.noreply.github.com>
2022-02-21 12:41:21 -05:00
Graham Christensen 290e0653ad hydra-eval-jobs: GC root aggregate jobs 2022-02-20 12:28:40 -05:00
John Ericson 445bba337b Make copyClosureTo take a regular C++ ref to the store
This is syntactically lighter wait, and demonstates there are no weird
dynamic lifetimes involved, just regular passing reference to callee
which it only borrows for the duration of the call.
2022-02-20 17:22:43 +00:00
John Ericson f14c583ce5 Use copyClosure instead of computeFSClosure + copyPaths
It is more terse, and in the future it is possible `copyClosure` will
become more sophisticated.
2022-02-19 11:59:17 -05:00
Graham Christensen 4c41ca08e1
Merge pull request #1155 from helsinki-systems/fix/graph-readability
build-graphs: Fix readability in dark mode
2022-02-14 11:27:37 -05:00
ajs124 1c84676527
Fit more content on screen 2022-02-13 18:33:37 +01:00
Janne Heß 6d146deaf0
build-graphs: Fix readability in dark mode 2022-02-13 14:00:17 +01:00
Graham Christensen 27ddde1e9e dynamic runcommand: print a notice on the build page if it is disabled 2022-02-11 15:04:54 -05:00
Cole Helbling a22a8fa62d AddBuilds: reject declarative jobsets with dynamic runcommand enabled if disabled elsewhere 2022-02-11 14:35:52 -05:00
Cole Helbling 928ba9e854 Controller/{Jobset,Project}: error when enabling dynamic runcommand but it's disabled elsewhere 2022-02-11 14:35:52 -05:00
Cole Helbling d680c209fe edit-project.tt: disable when disabled by server
Also add a tooltip describing why it's disabled, to make it easier to
chase down.
2022-02-11 14:35:52 -05:00
Cole Helbling 6053e5fd4b edit-jobset.tt: disable when disabled by project and server
Also add a tooltip describing why it's disabled, to make it easier to
chase down.
2022-02-11 14:35:52 -05:00
Cole Helbling dfd3a67424 project.tt: more info on why Dynamic RunCommand is disabled 2022-02-11 14:35:52 -05:00
Cole Helbling 3f4f183792 jobset.tt: more info on why Dynamic RunCommand is disabled 2022-02-11 14:35:52 -05:00
Graham Christensen 71c06f2ce7 LDAP normalization errors: note that the error came while normalizing the roles. 2022-02-11 10:55:27 -05:00
Graham Christensen f07fb7d279 LDAP support: include BC support for the YAML based loading
Includes a refactoring of the configuration loader.
2022-02-11 10:49:38 -05:00
Janne Heß 61d74a7194 Redo LDAP config in the main configuration and add role mappings 2022-02-11 10:49:38 -05:00
Graham Christensen d0bc0d0eda
Merge pull request #1152 from DeterminateSystems/parallel-tests
Parallel tests, fix a hydra-queue-runner race condition
2022-02-10 12:11:20 -05:00
Graham Christensen 4acaf9c8b0 hydra-queue-runner: don't dispatch until the machines parser has completed one run
Periodically, I have seen tests fail because of out of order queue runner behavior:

    checking the queue for builds > 0...
    loading build 1 (tests:basic:empty_dir)
    aborting unsupported build step '...-empty-dir.drv' (type 'x86_64-linux')
    marking build 1 as failed
    adding new machine ‘localhost’

This patch should prevent the dispatcher from running before any machines are
made available.
2022-02-10 10:54:30 -05:00
Graham Christensen 9ae7c8bddc Hydra::Helper::Exec add an expectOkay which dies with stdout / stderr on exit 2022-02-09 20:56:10 -05:00
Graham Christensen 845e6d4760 captureStdoutStderr*: move to Hydra::Helper::Exec which helps avoid some environment variable fixation problems 2022-02-09 14:28:50 -05:00
Graham Christensen 517dce285a eval_added event: change interface to traceID\tjobsetID\tevaluationID
I was not going to break the interface until I noticed
the current implementation uses the string literal \t.
2022-02-08 09:51:35 -05:00