Cole Helbling
5d3912962b
RunCommand: use helper functions to ensure filenames and paths are the same
...
Otherwise, it's possible someone updates the format in one place but not
the others, leading to broken or incorrect functionality.
2022-01-28 13:03:15 -08:00
Cole Helbling
4a441b54ce
hydra-module: /var/lib/hydra -> ${baseDir}
2022-01-28 13:03:15 -08:00
Cole Helbling
14090fbb86
runcommand-log.tt: init
2022-01-28 13:03:15 -08:00
Janne Heß
796ce165d4
RunCommand: Allow displaying command output
2022-01-28 13:03:15 -08:00
Janne Heß
4cb5e6cd94
RunCommand: Capture the output of the commands
2022-01-28 13:00:17 -08:00
Graham Christensen
ef362e92d1
GithubStatus: try pushing statuses for cached buildqueued/buildfinished events
2022-01-25 12:42:28 -05:00
Graham Christensen
001539c3d2
Merge pull request #1127 from NixOS/grahamc-patch-1
...
Create a bug report issue template
2022-01-25 09:57:17 -05:00
Graham Christensen
0a5f625746
Update bug_report.md
2022-01-24 20:36:08 -05:00
Graham Christensen
f6e86efc9f
Merge pull request #1091 from Ma27/ssh-remote-store-location
...
hydra-queue-runner: support store URIs declaring an alternate store location
2022-01-24 14:10:54 -05:00
Graham Christensen
3a4ea6e563
Merge pull request #1124 from obsidiansystems/simplify--closure-of-path-set
...
simplify, `computeFSClosure` can take a set now
2022-01-24 14:09:35 -05:00
Graham Christensen
bb68b56f61
Merge pull request #1133 from helsinki-systems/doc/config-format
...
doc: Document the file format of the config
2022-01-21 20:49:18 -05:00
Graham Christensen
c280692f91
Merge pull request #1126 from DeterminateSystems/build-localhost-paths
...
build-remote: copy missing paths from the binary cache to localhost
2022-01-21 16:16:33 -05:00
Graham Christensen
44cd890ae3
Merge pull request #1130 from DeterminateSystems/prompt-password
...
hydra-create-user: support prompting for password
2022-01-21 15:38:39 -05:00
Graham Christensen
ba96a13407
Record metrics when getting the closure to localhost
2022-01-21 15:38:05 -05:00
Graham Christensen
7e9e82398d
build-remote: copy missing paths from the binary cache to localhost
...
In a Hydra instance I saw:
possibly transient failure building ‘/nix/store/X.drv’ on ‘localhost’:
dependency '/nix/store/Y' of '/nix/store/Y.drv' does not exist,
and substitution is disabled
This is confusing because the Hydra in question does have substitution enabled.
This instance uses:
keep-outputs = true
keep-derivations = true
and an S3 binary cache which is not configured as a substituter in the nix.conf.
It appears this instance encountered a situation where store path Y was built
and present in the binary cache, and Y.drv was GC rooted on the instance,
however Y was not on the host.
When Hydra would try to build this path locally, it would look in the binary
cache to see if it was cached:
(nix)
439 bool valid = isValidPathUncached(storePath);
440
441 if (diskCache && !valid)
442 // FIXME: handle valid = true case.
443 diskCache->upsertNarInfo(getUri(), hashPart, 0);
444
445 return valid;
Since it was cached, the store path was considered Valid.
The queue monitor would then not put this input in for substitution, because
the path is valid:
(hydra)
470 if (!destStore->isValidPath(*i.second.path(*localStore, step->drv->name, i.first))) {
471 valid = false;
472 missing.insert_or_assign(i.first, i.second);
473 }
Hydra appears to correctly handle the case of missing paths that need
to be substituted from the binary cache already, but since most
Hydra instances use `keep-outputs` *and* all paths in the binary cache
originate from that machine, it is not common for a path to be cached
and not GC rooted locally.
I'll run Hydra with this patch for a while and see if we run in to the
problem again.
A big thanks to John Ericson who helped debug this particular issue.
2022-01-21 15:26:45 -05:00
Graham Christensen
952f629b7c
Test the queue runner in the scenario where a dependency is available in the cache but GC'd locally, where we're building locally
2022-01-21 15:26:45 -05:00
Graham Christensen
5c3e48fd0d
CliRunners: decode UTF8 before printing stderr/stdout
...
Fixes yath output. It used to say:
step â is now runnable
Now it says:
step ‘/run...2ipqz6hbc41m4c5w5bkq-dependent-job.drv’ is now runnable
2022-01-21 15:24:33 -05:00
Graham Christensen
e4407f8c93
HydraTestContext: expose the nix state dir
2022-01-21 15:12:10 -05:00
Janne Heß
56308dbb05
doc: Document the file format of the config
2022-01-21 20:48:50 +01:00
Graham Christensen
da1af1ce68
Docs: use hydra-create-user --password-prompt
2022-01-21 13:05:12 -05:00
Graham Christensen
e351054f61
Merge pull request #1129 from DeterminateSystems/fixup-argon2
...
Fixup argon2 instructions in hydra-create-user
2022-01-21 13:01:37 -05:00
Graham Christensen
61325853a6
Merge pull request #1132 from DeterminateSystems/ldap-role-match
...
LDAP support: require the prefix 'hydra_' to match documentation
2022-01-21 12:58:35 -05:00
Graham Christensen
0eeced7f08
hydra-create-user: Warn that creating users with a plaintext password is deprecated
2022-01-21 12:56:15 -05:00
Graham Christensen
98928a4125
fixups
2022-01-21 12:52:06 -05:00
Graham Christensen
a888a57baf
tests.ldap: verify the hydra_ prefix is required
2022-01-21 12:46:02 -05:00
Graham Christensen
76fbde6d6b
Set noecho when reading passwords
2022-01-21 11:11:09 -05:00
Graham Christensen
8ba4ae461e
Merge pull request #1131 from kenranunderscore/fix-invalid-yaml
...
Fix invalid YAML in documentation
2022-01-21 11:03:28 -05:00
Graham Christensen
b8f72d7ff2
LDAP support: require the prefix 'hydra_' to match documentation
2022-01-21 10:48:04 -05:00
Graham Christensen
bb893d0bd5
hydra-create-user: support prompting for passwords
...
I'm not sure this is a good implementation as-is. It does work,
but the password gets echo'd to the screen. I tried to use IO::Prompt
but IO::Prompt really seems to want to read the password from ARGV.
2022-01-21 10:40:56 -05:00
Graham Christensen
3a6c25489c
Hydra::Helper::Nix: expose a captureStdoutStderrWithStdin, make it available in tests
2022-01-21 10:40:06 -05:00
Graham Christensen
d4fe7e55dd
Hydra::Helper::Nix: sort exported functions
2022-01-21 10:40:06 -05:00
Graham Christensen
4945306a2b
hydra-create-user: make docs about using --password-hash better
2022-01-21 10:39:22 -05:00
Johannes Maier
4476aba5f7
Fix invalid YAML in documentation
2022-01-21 16:38:59 +01:00
Graham Christensen
12d0d0c176
hydra-create-user: use test_context over test_init
2022-01-21 10:37:48 -05:00
Graham Christensen
bb9864bad7
hydra-create-user.t: rename the various users based on their password type
2022-01-21 09:12:47 -05:00
Graham Christensen
46b8f7cce8
Create a bug report issue template
2022-01-20 20:10:28 -05:00
John Ericson
e7a1ae87aa
simplify, computeFSClosure
can take a set now
2022-01-20 14:53:01 -05:00
Graham Christensen
ed1b532b74
Merge pull request #1093 from DeterminateSystems/builds-jobset-project
...
Builds: drop the jobset and project columns
2022-01-17 10:11:55 -05:00
Graham Christensen
8c50cd06e4
machines: ensure the jobset name is present
2022-01-15 17:11:08 -05:00
Graham Christensen
c8dc6a9419
Plugins: get project and jobset information from the project and jobset tables
2022-01-15 15:58:02 -05:00
Graham Christensen
9dc40e0816
evaluator: don't save project, jobset on builds
2022-01-15 15:58:02 -05:00
Graham Christensen
c539deea99
builds: add a build->project func to get the project via the jobset
2022-01-15 15:58:02 -05:00
Graham Christensen
f120909547
builds: drop project, jobset columns
...
Indexes were haphazardly dropped.
2022-01-15 15:58:02 -05:00
Graham Christensen
1caff3a250
Merge pull request #1117 from DeterminateSystems/project-jobset/queue-runner
...
queue-runner: track jobsets by ID
2022-01-15 15:57:14 -05:00
Graham Christensen
9671d4d135
Merge pull request #1119 from DeterminateSystems/project-jobset/update-gc-roots
...
Project jobset: update-gc-roots
2022-01-15 15:57:06 -05:00
Graham Christensen
f1a608ac35
Merge pull request #1118 from DeterminateSystems/project-jobset/declarativejobsets
...
Project jobset: Update DeclarativeJobsets to not use the project or jobsets columns on Bulids
2022-01-15 15:56:48 -05:00
Graham Christensen
7544d4ff47
hydra-update-gc-roots: get project and jobset information from the project and jobset tables
2022-01-15 14:26:45 -05:00
Graham Christensen
52dda56b99
hydra-update-gc-roots: add a very basic test validating successful execution
2022-01-15 14:20:11 -05:00
Graham Christensen
72c3110002
queue-runner: track jobsets by ID
2022-01-15 14:06:00 -05:00
Graham Christensen
17c6bd4fd8
DeclarativeJobsets: get the jobset name from the jobset table
2022-01-15 13:46:32 -05:00