Future of constituents in Hydra?

ma27 commented

2024-10-13 21:46:02 +00:00

Member

370a4bf138 started removing those giving the following rationale:

The feature cannot easily be ported to nix-eval-jobs since it requires
deep integration into the evaluator, and h.n.o doesn't use it

However, this is used for the unstable job[1][2].

Anyways, I hacked the constituents feature into my fork of nix-eval-jobs[3] (in a way that doesn't give up the streaming of results[4]) and used that to bring it back to Hydra[5] because I use this on my private Hydra, I just didn't notice for a while because jobs with existing constituents kept these constituents (leading to interesting side-effects when these child jobs were removed).

My question is now, do we plan to keep using this kind of feature? If not, what's the ideal solution here?

Sure, my code needs some polishing (it's mostly the old hydra-eval-jobs part stitched together), but I'd be happy to make this ready to integrate here.

cc @delroth @raitobezarius @yu-re-ka @vriska

[1] 564451a037/pkgs/top-level/release.nix (L141)
[2] https://hydra.nixos.org/job/nixpkgs/trunk/unstable#tabs-constituents
[3] 5ce92e2c4d
[4] well, aggregate jobs are only printed at the very end. But this is IMHO OK since there's usually very few aggregate jobs in contrast to very many non-aggregate jobs.
[5] cdfc5c81e8

https://git.lix.systems/lix-project/hydra/commit/370a4bf138a830c4de8a05e248ace043adbc8f4f started removing those giving the following rationale: > The feature cannot easily be ported to nix-eval-jobs since it requires > deep integration into the evaluator, and h.n.o doesn't use it However, this is used for the `unstable` job[1][2]. Anyways, I hacked the constituents feature into my fork of `nix-eval-jobs`[3] (in a way that doesn't give up the streaming of results[4]) and used that to bring it back to Hydra[5] because I use this on my private Hydra, I just didn't notice for a while because jobs with existing constituents kept these constituents (leading to interesting side-effects when these child jobs were removed). My question is now, do we plan to keep using this kind of feature? If not, what's the ideal solution here? Sure, my code needs some polishing (it's mostly the old hydra-eval-jobs part stitched together), but I'd be happy to make this ready to integrate here. cc @delroth @raitobezarius @yu-re-ka @vriska [1] https://github.com/NixOS/nixpkgs/blob/564451a037500d67b2c798ef7bddf43772ab6096/pkgs/top-level/release.nix#L141 [2] https://hydra.nixos.org/job/nixpkgs/trunk/unstable#tabs-constituents [3] https://git.lix.systems/ma27/nix-eval-jobs/commit/5ce92e2c4d318b30edaafa4d8664b9e15e941f32 [4] well, aggregate jobs are only printed at the very end. But this is IMHO OK since there's usually very few aggregate jobs in contrast to very many non-aggregate jobs. [5] https://git.lix.systems/ma27/hydra/commit/cdfc5c81e8037d3e4818a3e459d0804b2c157ea9

vriska commented

2024-10-13 22:35:53 +00:00

Member

I do think continuing to support aggregate jobs would be ideal, I just didn't really have a reason to reimplement them myself.

delroth commented

2024-10-14 05:15:21 +00:00

Member

If you want to port it feel free - but I think this would possibly be better served by channel scripts instead? In practice the only real way h.n.o uses that is to expose a "is channel broken?" bit, as far as I understand, and channel scripts could be doing this check instead. No strong opinion either way.

ma27 commented

2024-10-27 15:01:32 +00:00

Author

Member

In practice the only real way h.n.o uses that is to expose a "is channel broken?" bit, as far as I understand, and channel scripts could be doing this check instead

Sure, but I think it's pretty useful to have a UI integrated into CI that shows you what's still missing (also over time).

But I guess I'll consider this a positive reaction :D
Will reach out to the nix-eval-jobs maintainers soonish to see if they'd be open to accept a (polished) variant of my current patch.

Oh, also cc @raito for opinions, just noticed your handle here isn't the same as on GitHub 🙃

> In practice the only real way h.n.o uses that is to expose a "is channel broken?" bit, as far as I understand, and channel scripts could be doing this check instead Sure, but I think it's pretty useful to have a UI integrated into CI that shows you what's still missing (also over time). But I guess I'll consider this a positive reaction :D Will reach out to the nix-eval-jobs maintainers soonish to see if they'd be open to accept a (polished) variant of my current patch. Oh, also cc @raito for opinions, just noticed your handle here isn't the same as on GitHub 🙃

raito commented

2024-10-27 18:17:44 +00:00

Owner

I'm also in agreement with @delroth on this one. I would prefer carving this feature in the channel-scripts and decreasing the amount of things in Hydra so that it's easier to do things in there, as a general principle.

But this really boils down to: if there's someone willing to port this and convinced this is useful, I wouldn't block anyone from doing that.

I'm also in agreement with @delroth on this one. I would prefer carving this feature in the channel-scripts and decreasing the amount of things in Hydra so that it's easier to do things in there, as a general principle. But this really boils down to: if there's someone willing to port this and convinced this is useful, I wouldn't block anyone from doing that.

ma27 referenced this issue

2024-11-23 09:44:21 +00:00

compilation failure against latest lix release-2.91 #12

ma27 commented

2024-11-23 10:12:56 +00:00

Author

Member

if there's someone willing to port this and convinced this is useful, I wouldn't block anyone from doing that.

So, for me there are two things about constituents that I find useful:

you have an actual UI where you see what's missing / failing to build.
this can be used for any Hydra project.

If the threshold is "somebody must maintain this", then - well - here I am ;-)

If you have concerns about this slowing down other development efforts, then we may also discuss a feature toggle for that - your call.

I guess I'll file a PR against nix-eval-jobs now to get some opinions on how to proceed on that end because that's the biggest question mark for me.

Thank you for your input.

> if there's someone willing to port this and convinced this is useful, I wouldn't block anyone from doing that. So, for me there are two things about constituents that I find useful: * you have an actual UI where you see what's missing / failing to build. * this can be used for _any_ Hydra project. If the threshold is "somebody must maintain this", then - well - here I am ;-) If you have concerns about this slowing down other development efforts, then we may also discuss a feature toggle for that - your call. I guess I'll file a PR against nix-eval-jobs now to get some opinions on how to proceed on that end because that's the biggest question mark for me. Thank you for your input.

ma27 referenced this issue from lix-project/nix-eval-jobs

2024-11-23 10:29:51 +00:00

WIP: Bring back constituents to Hydra #17

Future of constituents in Hydra? #11