ma27/hydra
0
0
Fork 0
forked from lix-project/hydra
Code Pull requests Activity
3860 commits 4 branches 0 tags 25 MiB
Commit graph

3860 commits

This branch
This branch
All branches
Author SHA1 Message Date
fricklerhandwerk 0803634a41 add architecture notes 
meeting notes from @edolstra giving a one-hour tour of the code
 2022-04-06 09:01:10 +02:00
Graham Christensen 3c181463f0
Merge pull request #1191 from ncfavier/nixos-search 
Prepare for nixos-search integration
 2022-03-31 16:27:43 -04:00
Naïm Favier 5e3374cb86
Prepare for nixos-search integration 2022-03-31 12:55:15 +02:00
Graham Christensen 84f54cb011
Merge pull request #1189 from NixOS/fix-hang 
openConnection(): Don't throw exceptions in forked child
 2022-03-30 17:16:02 -04:00
Eelco Dolstra bcaad1c934 openConnection(): Don't throw exceptions in forked child 
On hydra.nixos.org the queue runner had child processes that were
stuck handling an exception:

  Thread 1 (Thread 0x7f501f7fe640 (LWP 1413473) "bld~v54h5zkhmb3"):
  #0  futex_wait (private=0, expected=2, futex_word=0x7f50c27969b0 <_rtld_local+2480>) at ../sysdeps/nptl/futex-internal.h:146
  #1  __lll_lock_wait (futex=0x7f50c27969b0 <_rtld_local+2480>, private=0) at lowlevellock.c:52
  #2  0x00007f50c21eaee4 in __GI___pthread_mutex_lock (mutex=0x7f50c27969b0 <_rtld_local+2480>) at ../nptl/pthread_mutex_lock.c:115
  #3  0x00007f50c1854bef in __GI___dl_iterate_phdr (callback=0x7f50c190c020 <_Unwind_IteratePhdrCallback>, data=0x7f501f7fb040) at dl-iteratephdr.c:40
  #4  0x00007f50c190d2d1 in _Unwind_Find_FDE () from /nix/store/65hafbsx91127farbmyyv4r5ifgjdg43-glibc-2.33-117/lib/libgcc_s.so.1
  #5  0x00007f50c19099b3 in uw_frame_state_for () from /nix/store/65hafbsx91127farbmyyv4r5ifgjdg43-glibc-2.33-117/lib/libgcc_s.so.1
  #6  0x00007f50c190ab90 in uw_init_context_1 () from /nix/store/65hafbsx91127farbmyyv4r5ifgjdg43-glibc-2.33-117/lib/libgcc_s.so.1
  #7  0x00007f50c190b08e in _Unwind_RaiseException () from /nix/store/65hafbsx91127farbmyyv4r5ifgjdg43-glibc-2.33-117/lib/libgcc_s.so.1
  #8  0x00007f50c1b02ab7 in __cxa_throw () from /nix/store/dd8swlwhpdhn6bv219562vyxhi8278hs-gcc-10.3.0-lib/lib/libstdc++.so.6
  #9  0x00007f50c1d01abe in nix::parseURL (url="root@cb893012.packethost.net") at src/libutil/url.cc:53
  #10 0x0000000000484f55 in extraStoreArgs (machine="root@cb893012.packethost.net") at build-remote.cc:35
  #11 operator() (__closure=0x7f4fe9fe0420) at build-remote.cc:79
  ...

Maybe the fork happened while another thread was holding some global
stack unwinding lock
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71744). Anyway, since
the hanging child inherits all file descriptors to SSH clients,
shutting down remote builds (via 'child.to = -1' in
State::buildRemote()) doesn't work and 'child.pid.wait()' hangs
forever.

So let's not do any significant work between fork and exec.
 2022-03-30 22:39:48 +02:00
Graham Christensen bbb0998699
Merge pull request #1188 from DeterminateSystems/nix-2.7 
Nix 2.7
 2022-03-29 15:49:37 -04:00
ajs124 089da272c7 fix build against nix 2.7.0 
fix build after such commits as df552ff53e68dff8ca360adbdbea214ece1d08ee
and e862833ec662c1bffbe31b9a229147de391e801a
 2022-03-29 15:38:24 -04:00
ajs124 c64c5f0a7e hydra-queue-runner: rename build-result.hh to hydra-build-result.hh 2022-03-29 15:34:29 -04:00
Graham Christensen 4368ff5d5b flake.lock: Add 
Flake lock file updates:

• Added input 'nix':
    'github:NixOS/nix/ffe155abd36366a870482625543f9bf924a58281' (2022-03-07)
• Added input 'nix/lowdown-src':
    'github:kristapsdz/lowdown/d2c2b44ff6c27b936ec27358a2653caaef8f73b8' (2021-10-06)
• Added input 'nix/nixpkgs':
    'github:NixOS/nixpkgs/82891b5e2c2359d7e58d08849e4c89511ab94234' (2021-09-28)
• Added input 'nix/nixpkgs-regression':
    'github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2' (2022-01-24)
• Added input 'nixpkgs':
    follows 'nix/nixpkgs'
 2022-03-29 15:33:08 -04:00
Graham Christensen 98da457e16 nix: 2.7.0 2022-03-29 15:31:11 -04:00
Graham Christensen 20a8437094 flake.nix: set nix to 2.6.0 2022-03-29 15:29:33 -04:00
Graham Christensen fd3690a0c1 flake.lock: Update 
Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/a6ba313a0aac3b6e2fef434cb42d190a0849238e' (2021-08-10)
  → 'github:NixOS/nix/a1cd7e58606a41fcf62bf8637804cf8306f17f62' (2022-01-24)
• Updated input 'nix/lowdown-src':
    'github:kristapsdz/lowdown/148f9b2f586c41b7e36e73009db43ea68c7a1a4d' (2021-04-03)
  → 'github:kristapsdz/lowdown/d2c2b44ff6c27b936ec27358a2653caaef8f73b8' (2021-10-06)
• Updated input 'nix/nixpkgs':
    'github:NixOS/nixpkgs/f77036342e2b690c61c97202bf48f2ce13acc022' (2021-06-28)
  → 'github:NixOS/nixpkgs/82891b5e2c2359d7e58d08849e4c89511ab94234' (2021-09-28)
• Added input 'nix/nixpkgs-regression':
    'github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2' (2022-01-24)
 2022-03-29 15:29:23 -04:00
Graham Christensen 3b048ed136 Revert "Revert "Use copyClosure instead of computeFSClosure + copyPaths"" 
This reverts commit 8e3ada2afc.
 2022-03-29 15:28:47 -04:00
Graham Christensen 78ef4ae9a5
Merge pull request #1187 from DeterminateSystems/revert-to-nix-2.4pre20210810_a6ba313 
Revert "Build against Nix 2.5.1" - build against nix-2.4pre20210810_a…
 2022-03-29 09:35:15 -04:00
Graham Christensen dc709422a6 Revert "Build against Nix 2.5.1" - build against nix-2.4pre20210810_a6ba313 
This reverts commit 921e27d6c0.
 2022-03-29 09:24:51 -04:00
Graham Christensen 47c7170c52
Merge pull request #1185 from DeterminateSystems/revert-from-nix-2.6 
Revert to Nix 2.5.1
 2022-03-28 14:45:15 -04:00
Cole Helbling 921e27d6c0 Build against Nix 2.5.1 2022-03-28 11:36:14 -07:00
Cole Helbling 2ba83a5cba t/jobs/empty-dir-builder: provide output for nix log 2022-03-28 09:54:02 -07:00
Cole Helbling 127a644595 Revert "Update Nix to 2.6" 
This reverts commit 5ae26aa760.
 2022-03-28 09:54:02 -07:00
Cole Helbling 8e3ada2afc Revert "Use copyClosure instead of computeFSClosure + copyPaths" 
This reverts commit f14c583ce5.
 2022-03-28 09:54:02 -07:00
Eelco Dolstra 962bf36939
Merge pull request #1162 from obsidiansystems/less-ref 
Make `copyClosureTo` take a regular C++ ref to the store
 2022-03-23 16:25:59 +01:00
Eelco Dolstra 3390415905
Merge pull request #1125 from obsidiansystems/simplify--copyClosure 
Use `copyClosure` instead of `computeFSClosure` + `copyPaths`
 2022-03-23 12:49:22 +01:00
Graham Christensen 01fb23ddf6
Merge pull request #1178 from DeterminateSystems/hydra-update-gc-roots/network-traffic 
hydra-update-gc-roots: allow cached refs to the build's jobset
 2022-03-21 09:05:54 -04:00
Graham Christensen e5393c2cf8 fixup: make id non-ambiguous 2022-03-19 23:56:47 -04:00
Graham Christensen 137be3452e Reduce the jobset cols on the remaining two queries 2022-03-19 23:56:47 -04:00
Graham Christensen f353a7ac41 update-gc-roots: try subselecting the jobset table 2022-03-19 23:56:47 -04:00
Graham Christensen 145667cb53 hydra-update-gc-roots: allow cached refs to the build's jobset 
Re-executing this search_related on every access turned out to
create very problematic performance. If a jobset had a lot of
error output stored in the jobset, and there were many hundreds
or thousands of active jobs, this could easily cause >1Gbps of
network traffic.
 2022-03-19 23:56:47 -04:00
Graham Christensen 22026da4f8
Merge pull request #1176 from DeterminateSystems/broken-constituent 
Broken constituents: emit useful log messages on evaluation errors on constituents
 2022-03-19 14:55:17 -04:00
Graham Christensen a582e4c485 HydraTestContext: add \n's to various dies 2022-03-19 14:46:53 -04:00
Graham Christensen 074a2f96bf hydra-eval-jobset: emit a useful error if constituents errored 2022-03-19 14:37:12 -04:00
Graham Christensen 0c51de6334 hydra-evaluate-jobset: assert it logs errored constituents properly 2022-03-19 14:35:30 -04:00
Graham Christensen 25f6bae847 HydraTestContext: make it easy to create a jobset without evaluating 2022-03-19 14:34:43 -04:00
Graham Christensen 28b682b85b
Merge pull request #1170 from NixOS/dependabot/github_actions/actions/checkout-3 
build(deps): bump actions/checkout from 2 to 3
 2022-03-09 21:13:47 -05:00
dependabot[bot] b9ec3a41d6
build(deps): bump actions/checkout from 2 to 3 
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-07 15:01:23 +00:00
Graham Christensen a2546121f0
Merge pull request #1163 from DeterminateSystems/constituents 
Constituents: Add a GC root for indirect aggregates
 2022-02-21 20:42:42 -05:00
Graham Christensen 9316544abf
src/hydra-eval-jobs/hydra-eval-jobs.cc: .get<std::string> for drvPath 
Co-authored-by: Kayla Fire <firestack@users.noreply.github.com>
 2022-02-21 12:41:21 -05:00
Graham Christensen 290e0653ad hydra-eval-jobs: GC root aggregate jobs 2022-02-20 12:28:40 -05:00
Graham Christensen e0921eba0a Create a basic test which verifies we can't delete the derivation of aggregate jobs 2022-02-20 12:28:40 -05:00
Graham Christensen be46f02164 tests: relocate evaluator tests 2022-02-20 12:28:40 -05:00
Graham Christensen 5d169e3a2e Add a test validating direct and indirect constituents 2022-02-20 12:28:40 -05:00
John Ericson 445bba337b Make copyClosureTo take a regular C++ ref to the store 
This is syntactically lighter wait, and demonstates there are no weird
dynamic lifetimes involved, just regular passing reference to callee
which it only borrows for the duration of the call.
 2022-02-20 17:22:43 +00:00
John Ericson f14c583ce5 Use copyClosure instead of computeFSClosure + copyPaths 
It is more terse, and in the future it is possible `copyClosure` will
become more sophisticated.
 2022-02-19 11:59:17 -05:00
Graham Christensen dfb3eccfaa
Merge pull request #1140 from Ma27/nix-update 
Update Nix to 2.6
 2022-02-19 08:38:34 -05:00
Graham Christensen 4c41ca08e1
Merge pull request #1155 from helsinki-systems/fix/graph-readability 
build-graphs: Fix readability in dark mode
 2022-02-14 11:27:37 -05:00
Graham Christensen 1124230d9f
Merge pull request #1151 from DeterminateSystems/ldap-tests-inc-mapping 
ldap: support configurable roles with backwards compatibility
 2022-02-14 09:30:54 -05:00
Janne Heß 6d146deaf0
build-graphs: Fix readability in dark mode 2022-02-13 14:00:17 +01:00
Graham Christensen 78e9872251 ldap.t: write the password to an external .conf file 2022-02-11 11:27:10 -05:00
Graham Christensen 848fb3b265 ldap-legacy.t: specify the root password manually 2022-02-11 11:26:56 -05:00
Graham Christensen 0bd4a75918 HydraTestContext: support running a sub before running hydra-init. 2022-02-11 11:26:27 -05:00
Graham Christensen 05ca71069f ldap config: document putting the password in a separate file 2022-02-11 11:24:45 -05:00