ma27/hydra
0
0
Fork 0
forked from lix-project/hydra
Code Pull requests Activity
3725 commits 5 branches 0 tags 26 MiB
Commit graph

3725 commits

This branch
This branch
All branches
Author SHA1 Message Date
Graham Christensen f6e86efc9f
Merge pull request #1091 from Ma27/ssh-remote-store-location 
hydra-queue-runner: support store URIs declaring an alternate store location
 2022-01-24 14:10:54 -05:00
Graham Christensen 3a4ea6e563
Merge pull request #1124 from obsidiansystems/simplify--closure-of-path-set 
simplify, `computeFSClosure` can take a set now
 2022-01-24 14:09:35 -05:00
Graham Christensen bb68b56f61
Merge pull request #1133 from helsinki-systems/doc/config-format 
doc: Document the file format of the config
 2022-01-21 20:49:18 -05:00
Graham Christensen c280692f91
Merge pull request #1126 from DeterminateSystems/build-localhost-paths 
build-remote: copy missing paths from the binary cache to localhost
 2022-01-21 16:16:33 -05:00
Graham Christensen 44cd890ae3
Merge pull request #1130 from DeterminateSystems/prompt-password 
hydra-create-user: support prompting for password
 2022-01-21 15:38:39 -05:00
Graham Christensen ba96a13407 Record metrics when getting the closure to localhost 2022-01-21 15:38:05 -05:00
Graham Christensen 7e9e82398d build-remote: copy missing paths from the binary cache to localhost 
In a Hydra instance I saw:

    possibly transient failure building ‘/nix/store/X.drv’ on ‘localhost’:
      dependency '/nix/store/Y' of '/nix/store/Y.drv' does not exist,
      and substitution is disabled

This is confusing because the Hydra in question does have substitution enabled.

This instance uses:

  keep-outputs = true
  keep-derivations = true

and an S3 binary cache which is not configured as a substituter in the nix.conf.

It appears this instance encountered a situation where store path Y was built
and present in the binary cache, and Y.drv was GC rooted on the instance,
however Y was not on the host.

When Hydra would try to build this path locally, it would look in the binary
cache to see if it was cached:

    (nix)
    439      bool valid = isValidPathUncached(storePath);
    440
    441      if (diskCache && !valid)
    442          // FIXME: handle valid = true case.
    443          diskCache->upsertNarInfo(getUri(), hashPart, 0);
    444
    445      return valid;

Since it was cached, the store path was considered Valid.

The queue monitor would then not put this input in for substitution, because
the path is valid:

    (hydra)
    470          if (!destStore->isValidPath(*i.second.path(*localStore, step->drv->name, i.first))) {
    471              valid = false;
    472              missing.insert_or_assign(i.first, i.second);
    473          }

Hydra appears to correctly handle the case of missing paths that need
to be substituted from the binary cache already, but since most
Hydra instances use `keep-outputs` *and* all paths in the binary cache
originate from that machine, it is not common for a path to be cached
and not GC rooted locally.

I'll run Hydra with this patch for a while and see if we run in to the
problem again.

A big thanks to John Ericson who helped debug this particular issue.
 2022-01-21 15:26:45 -05:00
Graham Christensen 952f629b7c Test the queue runner in the scenario where a dependency is available in the cache but GC'd locally, where we're building locally 2022-01-21 15:26:45 -05:00
Graham Christensen 5c3e48fd0d CliRunners: decode UTF8 before printing stderr/stdout 
Fixes yath output. It used to say:

    step â is now runnable

Now it says:

    step ‘/run...2ipqz6hbc41m4c5w5bkq-dependent-job.drv’ is now runnable
 2022-01-21 15:24:33 -05:00
Graham Christensen e4407f8c93 HydraTestContext: expose the nix state dir 2022-01-21 15:12:10 -05:00
Janne Heß 56308dbb05
doc: Document the file format of the config 2022-01-21 20:48:50 +01:00
Graham Christensen da1af1ce68 Docs: use hydra-create-user --password-prompt 2022-01-21 13:05:12 -05:00
Graham Christensen e351054f61
Merge pull request #1129 from DeterminateSystems/fixup-argon2 
Fixup argon2 instructions in hydra-create-user
 2022-01-21 13:01:37 -05:00
Graham Christensen 61325853a6
Merge pull request #1132 from DeterminateSystems/ldap-role-match 
LDAP support: require the prefix 'hydra_' to match documentation
 2022-01-21 12:58:35 -05:00
Graham Christensen 0eeced7f08 hydra-create-user: Warn that creating users with a plaintext password is deprecated 2022-01-21 12:56:15 -05:00
Graham Christensen 98928a4125 fixups 2022-01-21 12:52:06 -05:00
Graham Christensen a888a57baf tests.ldap: verify the hydra_ prefix is required 2022-01-21 12:46:02 -05:00
Graham Christensen 76fbde6d6b Set noecho when reading passwords 2022-01-21 11:11:09 -05:00
Graham Christensen 8ba4ae461e
Merge pull request #1131 from kenranunderscore/fix-invalid-yaml 
Fix invalid YAML in documentation
 2022-01-21 11:03:28 -05:00
Graham Christensen b8f72d7ff2 LDAP support: require the prefix 'hydra_' to match documentation 2022-01-21 10:48:04 -05:00
Graham Christensen bb893d0bd5 hydra-create-user: support prompting for passwords 
I'm not sure this is a good implementation as-is. It does work,
but the password gets echo'd to the screen. I tried to use IO::Prompt
but IO::Prompt really seems to want to read the password from ARGV.
 2022-01-21 10:40:56 -05:00
Graham Christensen 3a6c25489c Hydra::Helper::Nix: expose a captureStdoutStderrWithStdin, make it available in tests 2022-01-21 10:40:06 -05:00
Graham Christensen d4fe7e55dd Hydra::Helper::Nix: sort exported functions 2022-01-21 10:40:06 -05:00
Graham Christensen 4945306a2b hydra-create-user: make docs about using --password-hash better 2022-01-21 10:39:22 -05:00
Johannes Maier 4476aba5f7 Fix invalid YAML in documentation 2022-01-21 16:38:59 +01:00
Graham Christensen 12d0d0c176 hydra-create-user: use test_context over test_init 2022-01-21 10:37:48 -05:00
Graham Christensen bb9864bad7 hydra-create-user.t: rename the various users based on their password type 2022-01-21 09:12:47 -05:00
John Ericson e7a1ae87aa simplify, computeFSClosure can take a set now 2022-01-20 14:53:01 -05:00
Graham Christensen ed1b532b74
Merge pull request #1093 from DeterminateSystems/builds-jobset-project 
Builds: drop the jobset and project columns
 2022-01-17 10:11:55 -05:00
Graham Christensen 8c50cd06e4 machines: ensure the jobset name is present 2022-01-15 17:11:08 -05:00
Graham Christensen c8dc6a9419 Plugins: get project and jobset information from the project and jobset tables 2022-01-15 15:58:02 -05:00
Graham Christensen 9dc40e0816 evaluator: don't save project, jobset on builds 2022-01-15 15:58:02 -05:00
Graham Christensen c539deea99 builds: add a build->project func to get the project via the jobset 2022-01-15 15:58:02 -05:00
Graham Christensen f120909547 builds: drop project, jobset columns 
Indexes were haphazardly dropped.
 2022-01-15 15:58:02 -05:00
Graham Christensen 1caff3a250
Merge pull request #1117 from DeterminateSystems/project-jobset/queue-runner 
queue-runner: track jobsets by ID
 2022-01-15 15:57:14 -05:00
Graham Christensen 9671d4d135
Merge pull request #1119 from DeterminateSystems/project-jobset/update-gc-roots 
Project jobset: update-gc-roots
 2022-01-15 15:57:06 -05:00
Graham Christensen f1a608ac35
Merge pull request #1118 from DeterminateSystems/project-jobset/declarativejobsets 
Project jobset: Update DeclarativeJobsets to not use the project or jobsets columns on Bulids
 2022-01-15 15:56:48 -05:00
Graham Christensen 7544d4ff47 hydra-update-gc-roots: get project and jobset information from the project and jobset tables 2022-01-15 14:26:45 -05:00
Graham Christensen 52dda56b99 hydra-update-gc-roots: add a very basic test validating successful execution 2022-01-15 14:20:11 -05:00
Graham Christensen 72c3110002 queue-runner: track jobsets by ID 2022-01-15 14:06:00 -05:00
Graham Christensen 17c6bd4fd8 DeclarativeJobsets: get the jobset name from the jobset table 2022-01-15 13:46:32 -05:00
Graham Christensen b2cdde0901 DeclarativeJobsets: test basic functionality 2022-01-15 13:46:32 -05:00
Graham Christensen eef633c1cc tests: create a declarative project spec and the autoconfig machinery 2022-01-15 13:46:32 -05:00
Graham Christensen 9d3b14dd43
Merge pull request #1116 from DeterminateSystems/project-jobset/input-type-build 
Project jobset: update input type "build"
 2022-01-15 12:48:20 -05:00
Graham Christensen 8c3c573953 hydra-eval-jobset: fixup old reference to project / jobset columns 2022-01-15 12:32:16 -05:00
Graham Christensen 2abcd84931
Merge pull request #1115 from DeterminateSystems/project-jobset/builds-json-repr 
Project jobset: update builds json repr
 2022-01-15 12:09:45 -05:00
Graham Christensen d91593e93a
Merge pull request #1114 from DeterminateSystems/project-jobset/delete-project 
Projects: delete: delete all builds first
 2022-01-14 23:04:55 -05:00
Graham Christensen 6bb9adc1a5 Builds: get the project and jobset names from the their tables 2022-01-14 22:45:26 -05:00
Graham Christensen cb68629417 test a Build's json representation 2022-01-14 21:19:48 -05:00
Graham Christensen f4c4b496d8 Projects: delete: delete all builds first 
Deleting jobsets first would fail because buildmetrics has an FK
to the jobset. However, the jobset / project relationship is not
marked as CASCADE.

Deleting all the builds automatically cascades to delete
buildmetrics, so deleting the relevant builds first, then deleting
the jobset solves it.
 2022-01-14 20:37:55 -05:00