Commit graph

3950 commits

Author SHA1 Message Date
Cole Helbling 47c1f89d5a t/RunCommand: fixup basic.t to use uuid 2022-01-31 08:58:33 -08:00
Cole Helbling fc3cf4ecb2 RunCommandLogs: identify and access via uuid
Using a sha1 of the command combined with the build ID is not a
particularly good or unique identifier:

* A build could fail, be restarted, and then succeed -- assuming no
configuration changes, the sha1 hash of the command as well as the build
ID will be the same. This would lead to an overwritten log file.

* Allowing user input to influence filenames is not the best of ideas.
2022-01-31 08:58:33 -08:00
Graham Christensen dcb0c1425c RunCommandLogs: set a UUID automatically 2022-01-31 08:58:33 -08:00
Graham Christensen 89e8676d80 UUID4Tiny: init 2022-01-31 08:58:33 -08:00
Graham Christensen cf49a05ff5 RunCommandLogs: add a uuid to each log entry 2022-01-31 08:58:33 -08:00
Graham Christensen 94ed9ed7ff
Merge pull request #1136 from DeterminateSystems/github-status-cached-evals
GithubStatus: try pushing statuses for cached buildqueued/buildfinished events
2022-01-31 09:11:37 -05:00
Graham Christensen 3ae4b19d12 flake: update to postgres 13 for UUID commands 2022-01-28 13:07:11 -08:00
Cole Helbling 244300c1ad RunCommand: remove unused and problematic imports
Since breaking the filename construction out to a helper function,
Hydra::Model::DB is no longer used. Importing Hydra::Helper::Nix,
however, has the potential to break tests, so just use the functions we
need without importing the entire module.
2022-01-28 13:07:11 -08:00
Cole Helbling fdf6f4d3da RunCommand: use IPC::Run3::run3 instead
run3 just seems to do better handling for what we want to do, and
requires less deep-reaching changes to this plugin to get it to play
nice, as IPC::Run::run would.
2022-01-28 13:07:11 -08:00
Cole Helbling 3e722f1d0a t/RunCommand: remove duplicate use 2022-01-28 13:07:11 -08:00
Cole Helbling c8f3943d03 hydra-module: log files don't have a .drv extension 2022-01-28 13:07:11 -08:00
Cole Helbling 3432cd7636 build.tt: split runcommand logic across multiple lines
Helps with readability.
2022-01-28 13:07:11 -08:00
Cole Helbling 988e79c6e5 t/RunCommand: test that the log file exists on the filesystem 2022-01-28 13:07:11 -08:00
Cole Helbling 1554750acc RunCommand: use make_path over mkdir
This will make all necessary parent directories a la `mkdir -p`.
2022-01-28 13:03:15 -08:00
Cole Helbling bf3c46ed43 RunCommand: use IPC::Run to spawn the command
This allows `logPath`s with spaces and other characters that might
otherwise cause problems inside a `system()` call.
2022-01-28 13:03:15 -08:00
Cole Helbling bb16f4fb10 RunCommand: set umask when creating log paths
This uses the somewhat restrictive umask of 0027 so that people outside
the user or group cannot read the files. This also helps to inhibit
TOCTOU where someone else has a handle to our file before we chmod it
and after we close it.
2022-01-28 13:03:15 -08:00
Cole Helbling 5d3912962b RunCommand: use helper functions to ensure filenames and paths are the same
Otherwise, it's possible someone updates the format in one place but not
the others, leading to broken or incorrect functionality.
2022-01-28 13:03:15 -08:00
Cole Helbling 4a441b54ce hydra-module: /var/lib/hydra -> ${baseDir} 2022-01-28 13:03:15 -08:00
Cole Helbling 14090fbb86 runcommand-log.tt: init 2022-01-28 13:03:15 -08:00
Janne Heß 796ce165d4 RunCommand: Allow displaying command output 2022-01-28 13:03:15 -08:00
Janne Heß 4cb5e6cd94 RunCommand: Capture the output of the commands 2022-01-28 13:00:17 -08:00
Graham Christensen ef362e92d1 GithubStatus: try pushing statuses for cached buildqueued/buildfinished events 2022-01-25 12:42:28 -05:00
Graham Christensen 001539c3d2
Merge pull request #1127 from NixOS/grahamc-patch-1
Create a bug report issue template
2022-01-25 09:57:17 -05:00
Graham Christensen 0a5f625746
Update bug_report.md 2022-01-24 20:36:08 -05:00
Graham Christensen f6e86efc9f
Merge pull request #1091 from Ma27/ssh-remote-store-location
hydra-queue-runner: support store URIs declaring an alternate store location
2022-01-24 14:10:54 -05:00
Graham Christensen 3a4ea6e563
Merge pull request #1124 from obsidiansystems/simplify--closure-of-path-set
simplify, `computeFSClosure` can take a set now
2022-01-24 14:09:35 -05:00
Graham Christensen bb68b56f61
Merge pull request #1133 from helsinki-systems/doc/config-format
doc: Document the file format of the config
2022-01-21 20:49:18 -05:00
Graham Christensen c280692f91
Merge pull request #1126 from DeterminateSystems/build-localhost-paths
build-remote: copy missing paths from the binary cache to localhost
2022-01-21 16:16:33 -05:00
Graham Christensen 44cd890ae3
Merge pull request #1130 from DeterminateSystems/prompt-password
hydra-create-user: support prompting for password
2022-01-21 15:38:39 -05:00
Graham Christensen ba96a13407 Record metrics when getting the closure to localhost 2022-01-21 15:38:05 -05:00
Graham Christensen 7e9e82398d build-remote: copy missing paths from the binary cache to localhost
In a Hydra instance I saw:

    possibly transient failure building ‘/nix/store/X.drv’ on ‘localhost’:
      dependency '/nix/store/Y' of '/nix/store/Y.drv' does not exist,
      and substitution is disabled

This is confusing because the Hydra in question does have substitution enabled.

This instance uses:

  keep-outputs = true
  keep-derivations = true

and an S3 binary cache which is not configured as a substituter in the nix.conf.

It appears this instance encountered a situation where store path Y was built
and present in the binary cache, and Y.drv was GC rooted on the instance,
however Y was not on the host.

When Hydra would try to build this path locally, it would look in the binary
cache to see if it was cached:

    (nix)
    439      bool valid = isValidPathUncached(storePath);
    440
    441      if (diskCache && !valid)
    442          // FIXME: handle valid = true case.
    443          diskCache->upsertNarInfo(getUri(), hashPart, 0);
    444
    445      return valid;

Since it was cached, the store path was considered Valid.

The queue monitor would then not put this input in for substitution, because
the path is valid:

    (hydra)
    470          if (!destStore->isValidPath(*i.second.path(*localStore, step->drv->name, i.first))) {
    471              valid = false;
    472              missing.insert_or_assign(i.first, i.second);
    473          }

Hydra appears to correctly handle the case of missing paths that need
to be substituted from the binary cache already, but since most
Hydra instances use `keep-outputs` *and* all paths in the binary cache
originate from that machine, it is not common for a path to be cached
and not GC rooted locally.

I'll run Hydra with this patch for a while and see if we run in to the
problem again.

A big thanks to John Ericson who helped debug this particular issue.
2022-01-21 15:26:45 -05:00
Graham Christensen 952f629b7c Test the queue runner in the scenario where a dependency is available in the cache but GC'd locally, where we're building locally 2022-01-21 15:26:45 -05:00
Graham Christensen 5c3e48fd0d CliRunners: decode UTF8 before printing stderr/stdout
Fixes yath output. It used to say:

    step â is now runnable

Now it says:

    step ‘/run...2ipqz6hbc41m4c5w5bkq-dependent-job.drv’ is now runnable
2022-01-21 15:24:33 -05:00
Graham Christensen e4407f8c93 HydraTestContext: expose the nix state dir 2022-01-21 15:12:10 -05:00
Janne Heß 56308dbb05
doc: Document the file format of the config 2022-01-21 20:48:50 +01:00
Graham Christensen da1af1ce68 Docs: use hydra-create-user --password-prompt 2022-01-21 13:05:12 -05:00
Graham Christensen e351054f61
Merge pull request #1129 from DeterminateSystems/fixup-argon2
Fixup argon2 instructions in hydra-create-user
2022-01-21 13:01:37 -05:00
Graham Christensen 61325853a6
Merge pull request #1132 from DeterminateSystems/ldap-role-match
LDAP support: require the prefix 'hydra_' to match documentation
2022-01-21 12:58:35 -05:00
Graham Christensen 0eeced7f08 hydra-create-user: Warn that creating users with a plaintext password is deprecated 2022-01-21 12:56:15 -05:00
Graham Christensen 98928a4125 fixups 2022-01-21 12:52:06 -05:00
Graham Christensen a888a57baf tests.ldap: verify the hydra_ prefix is required 2022-01-21 12:46:02 -05:00
Graham Christensen 76fbde6d6b Set noecho when reading passwords 2022-01-21 11:11:09 -05:00
Graham Christensen 8ba4ae461e
Merge pull request #1131 from kenranunderscore/fix-invalid-yaml
Fix invalid YAML in documentation
2022-01-21 11:03:28 -05:00
Graham Christensen b8f72d7ff2 LDAP support: require the prefix 'hydra_' to match documentation 2022-01-21 10:48:04 -05:00
Graham Christensen bb893d0bd5 hydra-create-user: support prompting for passwords
I'm not sure this is a good implementation as-is. It does work,
but the password gets echo'd to the screen. I tried to use IO::Prompt
but IO::Prompt really seems to want to read the password from ARGV.
2022-01-21 10:40:56 -05:00
Graham Christensen 3a6c25489c Hydra::Helper::Nix: expose a captureStdoutStderrWithStdin, make it available in tests 2022-01-21 10:40:06 -05:00
Graham Christensen d4fe7e55dd Hydra::Helper::Nix: sort exported functions 2022-01-21 10:40:06 -05:00
Graham Christensen 4945306a2b hydra-create-user: make docs about using --password-hash better 2022-01-21 10:39:22 -05:00
Johannes Maier 4476aba5f7 Fix invalid YAML in documentation 2022-01-21 16:38:59 +01:00
Graham Christensen 12d0d0c176 hydra-create-user: use test_context over test_init 2022-01-21 10:37:48 -05:00