Commit graph

2938 commits

Author SHA1 Message Date
John Ericson 75f26f1fc4 Clean up std::optional dereferencing in the queue runner
Instead of doing this partial operation a number of times, assert (with
a comment, get a reference to the thing inside, and use that just once.
(This refactor was done twice, "just once" for each time.)
2023-12-23 19:10:58 -05:00
John Ericson 831021808c
Merge pull request #1318 from obsidiansystems/use-build-result-serialiser
Use factored-out `BuildResult` serializer
2023-12-08 11:25:05 -05:00
John Ericson 6a54ab24e2 Use factored-out BuildResult serializer
For the record, here is the Nix 2.19 version:
https://github.com/NixOS/nix/blob/2.19-maintenance/src/libstore/serve-protocol.cc,
which is what we would initially use.

It is a more complete version of what Hydra has today except for one
thing: it always unconditionally sets the start/stop times.

I think that is correct at the other end seems to unconditionally
measure them, but just to be extra careful, I reproduced the old
behavior of falling back on Hydra's own measurements if `startTime` is
0.

The only difference is that the fallback `stopTime` is now measured from
after the entire `BuildResult` is transferred over the wire, but I think
that should be negligible if it is measurable at all. (And remember,
this is fallback case I already suspect is dead code.)
2023-12-07 02:00:22 -05:00
John Ericson 86cd5e9076 copyClosureTo: Use SubstituteFlag instead of bool
This matches Nix (in the same serialization logic in
`src/libstore/legacy-ssh-store.cc`) and adds clarity.
2023-12-07 00:18:50 -05:00
John Ericson a5d44b60ea
Merge pull request #1313 from obsidiansystems/split-buildRemote
Split the `buildRemote` function, take 2
2023-12-04 11:37:36 -05:00
John Ericson 363604846a Again, use const in for loop
As requested by @teh. Was lost in merge with master, now added back.
2023-12-04 11:31:05 -05:00
John Ericson 162b538912 Remove unused thisArrow variable 2023-12-04 11:27:39 -05:00
John Ericson 104baef503 Document the connection initialization process 2023-12-04 09:42:04 -05:00
Janne Heß 874fcae1e8
Merge pull request #1301 from delroth/queue-runner-perf
queue-runner: only re-sort runnables by prio once per dispatch cycle
2023-12-04 15:27:14 +01:00
John Ericson 67eeabd518 Merge remote-tracking branch 'upstream/master' into split-buildRemote 2023-12-04 09:12:58 -05:00
John Ericson 622c25e3c4 Sedding prior to merge 2023-12-04 08:56:06 -05:00
John Ericson c922e73c11 Update to Nix 2.19
Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/f5f4de6a550327b4b1a06123c2e450f1b92c73b6' (2023-10-02)
  → 'github:NixOS/nix/50f8f1c8bc019a4c0fd098b9ac674b94cfc6af0d' (2023-11-27)
2023-11-30 15:26:46 -05:00
John Ericson e172461e55 Use const in for loop
As requested by @teh
2023-11-30 12:19:20 -05:00
John Ericson 0917145622 Make new functions not in header static 2023-11-30 12:19:05 -05:00
John Ericson 2bda7ca642 Further use Machine::Connection to deduplicate 2023-11-30 11:31:58 -05:00
John Ericson 831a2d9bd5 Merge remote-tracking branch 'upstream/master' into split-buildRemote 2023-11-30 11:27:40 -05:00
chayleaf e9da80fff6
support nix 2.18 2023-11-21 18:41:52 +07:00
Janne Heß 8f48e4ddec
Merge pull request #1268 from knedlsepp/fix-mime
Fix MIME types when serving .js and .css to fix rendering of HTML reports
2023-11-17 22:16:27 +01:00
Stig Palmquist 6a5fb9efae Set output length of C::P::Argon2 hashes to 16
Since the default lengths in Crypt::Passphrase::Argon2 changed from 16
to 32 in in 0.009, some tests that expected the passphrase to be
unchanged started failing.
2023-10-20 00:09:28 +02:00
Janne Heß c1a5ff3959
Merge pull request #1258 from nh2/patch-1
`renderInputDiff`: Increase git hash length 6 -> 8
2023-09-09 17:17:43 +02:00
Pierre Bourdon b7c864c515
queue-runner: only re-sort runnables by prio once per dispatch cycle
The previous implementation was O(N²lg(N)) due to sorting the full
runnables priority list once per runnable being scheduled. While not
confirmed, this is suspected to cause performance issues and
bottlenecking with the queue runner when the runnable list gets large
enough.

This commit changes the dispatcher to instead only sort runnables per
priority once per dispatch cycle. This has the drawback of being less
reactive to runnable priority changes: the previous code would react
immediately, while this might end up using "old" priorities until the
next dispatch cycle. However, dispatch cycles are not supposed to take
very long (seconds, not minutes/hours), so this is not expected to have
much or any practical impact.

Ideally runnables would be maintained in a sorted data structure instead
of the current approach of copying + sorting in the scheduler. This
would however be a much more invasive change to implement, and might
have to wait until we can confirm where the queue runner bottlenecks
actually lie.
2023-09-08 23:38:30 +02:00
Eelco Dolstra 35ccc9ebb2
Fix indentation
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-08-23 17:04:45 +02:00
Linus Heckemann 9f0427385f Apply LTO fix suggested by Ericson2314 2023-08-20 14:55:56 +02:00
Linus Heckemann b23431a657 Support Nix 2.17 2023-08-04 15:53:48 +02:00
Janne Heß d135b123cd
Merge pull request #1292 from Ma27/fix-queue-runner-stats
hydra-queue-runner: fix stats
2023-07-17 09:56:05 +02:00
Maximilian Bosch 5c35d1be20
hydra-queue-runner: fix stats 2023-06-25 17:28:15 +02:00
Eelco Dolstra 9f69bb5c2c Fix compilation against Nix 2.16 2023-06-23 15:06:55 +02:00
Julien Malka b4099df91e
hydra-eval-jobs: fix jobs containing a dot being dropped 2023-04-25 10:37:41 +02:00
Eelco Dolstra 082495e34e
Merge pull request #1275 from Ma27/nix-2.13
Nix 2.13 + nixpkgs input update
2023-03-27 13:30:13 +02:00
Rob Vermaas f88bef15ed Use new Google for Web signin, the old way will be deprecated Mar 31st 2023 2023-03-14 09:04:12 +01:00
Cole Helbling 810d2e6b51 Drop unused IndexBuildOutputsOnPath index
Also it's larger than the actual table it's indexing lol.

    -[ RECORD 30 ]----------+-----------------------------------------
    table_name              | buildoutputs
    index_name              | indexbuildoutputsonpath
    index_scans_count       | 0
    index_size              | 31 GB
    table_reads_index_count | 2128699937
    table_reads_seq_count   | 0
    table_reads_count       | 2128699937
    table_writes_count      | 22442976
    table_size              | 28 GB
2023-03-06 07:56:05 -08:00
Maximilian Bosch c7716817a9
Update Nix to 2.13 2023-02-04 20:11:53 +01:00
Linus Heckemann 5b35e13898
hydra-queue-runner: use initializer lists for constructing JSON
And also fix the parts that were broken
2023-02-04 20:08:27 +01:00
Linus Heckemann 96e36201eb
hydra-queue-runner: adapt to nlohmann::json 2023-02-04 20:08:09 +01:00
Josef Kemetmüller ad99d3366f Fix MIME types when serving .js and .css
To correctly render HTML reports we make sure to return the following MIME
types instead of "text/plain"

- *.css: "text/css"
- *.js: "application/javascript"

Fixes: #1267
2022-12-29 22:26:59 +01:00
Graham Christensen f48f00ee6d
Merge pull request #1256 from cransom/hydra-evaluator-broken-connection
exit with error if database connectivity lost
2022-12-22 19:28:51 -05:00
Graham Christensen 01802efc17
Merge pull request #1263 from Ma27/fix-my-jobs-tab
Fix "My Jobs" tab in user dashboard
2022-12-05 01:55:49 +01:00
Eelco Dolstra 14d4624dc2
Merge pull request #1215 from knedlsepp/fix-hydra-with-latest-git
scmdiff: Hardcode `--git-dir`
2022-11-24 12:04:23 +01:00
Maximilian Bosch fd765bc97a
Fix "My Jobs" tab in user dashboard
Nowadays `Builds` doesn't reference `Project` directly anymore. This
means that simply resolving both `jobset` and `project` with a single
JOIN from `Builds` doesn't work anymore. Instead we need to resolve the
relation to `jobset` first and then the relation to `project`.

For similar fixes see e.g. c7c4759600.
2022-11-22 20:54:51 +01:00
Niklas Hambüchen d1d171ee90
renderInputDiff: Increase git hash length 6 -> 8
Nixpkgs has so many commits that length 6 is often ambiguous,
making the use of the shown values with git difficult.
2022-11-02 17:30:32 +01:00
Casey Ransom 70ad3a924a exit with error if database connectivity lost
There's currently no automatic recovery for disconnected databases in
the evaluator. This means if the database is ever temporarily
unavailable, hydra-evaluator will sit and spin with no work
accomplished.

If this condition is caught, the daemon will exit and systemd will be
responsible for resuming the service.
2022-10-26 16:13:40 -04:00
John Ericson 3526d61ff2 Merge remote-tracking branch 'upstream/master' into split-buildRemote 2022-10-25 11:24:54 -04:00
Théophane Hufschmitt 143c31734f Move all the build remote utils to their namespace
Just don't pollute the global one
2022-10-25 10:04:29 +02:00
Janne Heß 312cb42275
Merge pull request #1234 from helsinki-systems/feat/squiggly-line
Add a squiggly line to the Hydra link on hover
2022-10-21 22:57:20 +02:00
Jörg Thalheim 94d19e1972 hydra: fix localhost detection when protocol prefix are used 2022-09-29 20:46:13 +02:00
Maximilian Bosch d3fe4ffbf6
Job: expose closuresize and size (output size in the UI) as prometheus metrics 2022-09-22 10:47:22 +02:00
Graham Christensen d6cbf227cb
Merge pull request #1243 from K900/maybe-fix-trees
Run the JS code to make build trees collapsible at the right time
2022-09-08 09:55:14 -04:00
Eelco Dolstra 44e1efff7f Send the right nix-serve client version
We were using protocol version 6 but requesting version 4. The only
reason that this worked was because of a broken version check in
'nix-store --serve'. That was fixed in
c2d7456926,
which had the side-effect of breaking hydra-queue-runner.
2022-09-08 11:51:13 +02:00
Eelco Dolstra b16470c544 nix flake info -> nix flake metadata
This gets rid of a deprecation warning.
2022-09-06 19:23:20 +02:00
Janne Heß 371402c3c1
Drop the HipChat plugin
https://en.wikipedia.org/wiki/HipChat says:
> Following this, HipChat and Stride customers were migrated to the
> Slack group collaboration platform in a transition that was completed by
> February 2019.
2022-08-20 19:16:43 +02:00