Commit graph

161 commits

Author SHA1 Message Date
Cole Helbling 6e6475d860 flake: replace aliases with their proper names
Newer Nixpkgs have added a throw for these aliases.
2022-03-11 11:58:10 -08:00
Cole Helbling a0cb73579d flake: update newNixpkgs for newer prometheus-cpp 2022-03-11 11:58:10 -08:00
Cole Helbling 3bf31bd6a6 hydra-queue-runner: add simple "up" exporter
There are probably better ways to achieve this (and will likely need to
be refactored a bit to support further metrics).
2022-03-10 12:36:58 -08:00
Cole Helbling 5bbaa18a8f flake: fix foreman execution
[vin@scadrial:~/workspace/vcs/hydra]$ foreman -h
    Warning: the running version of Bundler (2.1.4) is older than the version that created the lockfile (2.2.20). We suggest you to upgrade to the version that created the lockfile by running `gem install bundler:2.2.20`.
    Traceback (most recent call last):
    	2: from /nix/store/ycshcdssxcj9sjf6yzb1ydw4fcglf66y-foreman-0.87.2/bin/foreman:20:in `<main>'
    	1: from /nix/store/ggqacj06n6qfm1iww0bih9ph0j89wcna-bundler-2.1.4/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/rubygems_integration.rb:413:in `block in replace_bin_path'
    /nix/store/ggqacj06n6qfm1iww0bih9ph0j89wcna-bundler-2.1.4/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/lib/bundler/rubygems_integration.rb:374:in `block in replace_bin_path': can't find executable foreman for gem foreman. foreman is not currently included in the bundle, perhaps you meant to add it to your Gemfile? (Gem::Exception)
2022-03-10 12:26:13 -08:00
Graham Christensen dfb3eccfaa
Merge pull request #1140 from Ma27/nix-update
Update Nix to 2.6
2022-02-19 08:38:34 -05:00
Graham Christensen f07fb7d279 LDAP support: include BC support for the YAML based loading
Includes a refactoring of the configuration loader.
2022-02-11 10:49:38 -05:00
Janne Heß 61d74a7194 Redo LDAP config in the main configuration and add role mappings 2022-02-11 10:49:38 -05:00
Graham Christensen d0bc0d0eda
Merge pull request #1152 from DeterminateSystems/parallel-tests
Parallel tests, fix a hydra-queue-runner race condition
2022-02-10 12:11:20 -05:00
Graham Christensen 5506962537 Test::Postgresql: update to a tagged fork to improve paralellism 2022-02-10 09:37:48 -05:00
Graham Christensen 8bce8109e3 flake.nix: drop the LDAP server test 2022-02-09 20:56:10 -05:00
Graham Christensen e4adec36f3 Set an env var in the build environment to point to the root of openldap 2022-02-09 20:56:10 -05:00
Graham Christensen 60adee7f94 hydra check inputs: add openldap 2022-02-09 20:56:10 -05:00
Graham Christensen 68c6c3d556 flake.nix: format with nixpkgs-fmt 2022-02-09 10:43:40 -05:00
Maximilian Bosch 5ae26aa760
Update Nix to 2.6 2022-02-06 15:05:15 +01:00
Marco Turchetto 73f2cc184f missing develop dependencies 2022-02-05 20:28:44 +01:00
Graham Christensen 89e8676d80 UUID4Tiny: init 2022-01-31 08:58:33 -08:00
Graham Christensen 3ae4b19d12 flake: update to postgres 13 for UUID commands 2022-01-28 13:07:11 -08:00
Cole Helbling fdf6f4d3da RunCommand: use IPC::Run3::run3 instead
run3 just seems to do better handling for what we want to do, and
requires less deep-reaching changes to this plugin to get it to play
nice, as IPC::Run::run would.
2022-01-28 13:07:11 -08:00
Graham Christensen 44cd890ae3
Merge pull request #1130 from DeterminateSystems/prompt-password
hydra-create-user: support prompting for password
2022-01-21 15:38:39 -05:00
Graham Christensen a888a57baf tests.ldap: verify the hydra_ prefix is required 2022-01-21 12:46:02 -05:00
Graham Christensen 76fbde6d6b Set noecho when reading passwords 2022-01-21 11:11:09 -05:00
Graham Christensen aa7b37838a PerlCritic: MoreUtils -> SomeUtils 2021-12-14 10:16:25 -05:00
Graham Christensen a658b80863 perlcritic: Readonly -> ReadonlyX 2021-12-14 10:16:25 -05:00
Cole Helbling 5659d2982c Hydra/Controller/Root: remove deprecated LocalRegex
11:38:20 hydra-server.1       | DEPRECATION WARNING: The Regex dispatch type is deprecated.
11:38:20 hydra-server.1       |   It is recommended that you convert Regex and LocalRegex
11:38:20 hydra-server.1       |   methods to Chained methods. at /nix/store/aa6gw57fnahd4824pbhmvcs0jlypmynq-hydra-perl-deps/lib/perl5/site_perl/5.32.1/Catalyst/DispatchType/Regex.pm line 210.
2021-12-03 12:12:53 -08:00
Marco Turchetto 68a9aac251 add missing develop dependency 2021-11-10 17:51:01 +01:00
Graham Christensen c317ab0dbe shell: add pixz so NAR serving works from foreman 2021-10-20 13:09:39 -04:00
Eelco Dolstra 2745226ada
Merge pull request #1003 from DeterminateSystems/perlcritic-level-4
perlcritic: level 4
2021-09-27 20:23:55 +02:00
Graham Christensen 99161c7c53
Merge pull request #1012 from helsinki-systems/fix/ldap-test
fix ldap nixos test
2021-09-24 14:27:26 -04:00
Jörg Thalheim e985a277b3 migrate to new openldap module syntax 2021-09-23 20:32:09 +02:00
Graham Christensen 85bb1c7ef9 (perlcritic) JSON::Any -> JSON::MaybeXS 2021-09-07 21:55:55 -04:00
Graham Christensen c880888f1e File::Slurp -> File::Slurper 2021-09-06 22:13:33 -04:00
ajs124 59cb44261d fix ldap nixos test
passwords were replaced with salted sha1 instead of sha256, because I
don't want to have to figure out how to make slapd load this module

We could also just do {CLEARTEXT} for the purpose of this test
2021-08-29 01:47:34 +02:00
Maximilian Bosch e36353532c
gitea-plugin: fix test
The test seems to be broken for a while[1]. The cause for this is that
in gitea 1.14 the `create-user` command got renamed to `user create`.

[1] https://hydra.nixos.org/build/151092299
2021-08-29 00:08:59 +02:00
Your Name 5d0ad5f649 hydra-notify: initial scratch take of prometheus events 2021-08-24 10:56:12 -04:00
Your Name 098e48d055 Flake: add perlcritic deps 2021-08-19 16:17:49 -04:00
Your Name 194eda28b7 flake.nix sort hydra-perl-deps inputs 2021-08-18 15:10:55 -04:00
Your Name d311a0d002 hydraJobs.tests.notifications: fixup file reference 2021-08-16 15:51:51 -04:00
Nathan van Doorn b63dfb7ad4 Rename version to version.txt
<version> is a standard header with C++20 which could cause issues if a library checks it exists then imports it

Because we have the root of this repo in the include path, it'd see that <version> exists (with, e.g., __has_include), and then try to include it as a header

But because it's just a file that says 0.1, this would fail

This happens with libpqxx 7
2021-07-05 19:47:58 +01:00
Amanda Cameron b0f8a207fe
flake: stdenv.lib -> lib
Nixpkgs on unstable has removed `stdenv.lib` as they've been warning for a while now. This removes the extra references to it in the flake.nix

I'm not entirely sure if this is right, but I figured it was trivial enough to give a quick try using the GH Editor while I was waiting for a job to finish
2021-06-17 16:50:38 -04:00
Graham Christensen 71453dd341 Expose Prometheus metrics at /metrics
Exposes metrics:

* http_request_duration_seconds_bucket
* http_request_size_bytes_bucket
* http_response_size_bytes_bucket
* http_requests_total

with labels of action and controller to help identify popular
endpoints and their performance characteristics.
2021-06-10 20:02:42 -04:00
Graham Christensen db6bad108b Use PrometheusTinyShared 2021-06-10 17:47:11 -04:00
Graham Christensen 1652631b21 Prometheus: init Tiny, TinyShared 2021-06-10 17:47:11 -04:00
Graham Christensen c6670b805f init: HashSharedMem for Prometheus 2021-06-10 17:47:11 -04:00
Graham Christensen b1fcc0c25b init DataRandom perl package for Prometheus stats 2021-06-10 17:47:11 -04:00
Graham Christensen 015d8298cb flake.nix: drop the API test 2021-04-28 18:32:23 -04:00
Maximilian Bosch 963085a04a
Add AuthenSASL to perl deps; improve email docs
The addition of AuthenSASL seems to be necessary to properly
authenticate against an SMTP server. Without this I got errors
such as

    error with Hydra::Plugin::EmailNotification=HASH(0x6ad0128)->buildFinished: SMTP auth requires MIME::Base64 and Authen::SASL
2021-04-26 18:33:57 +02:00
Graham Christensen 1da70030b7 Users: transparently upgrade passwords to Argon2
Passwords that are sha1 will be transparently upgraded to argon2,
and future comparisons will use Argon2

Co-authored-by: Graham Christensen <graham@grahamc.com>
2021-04-15 11:35:11 -04:00
Graham Christensen 29620df85e Passwords: check in constant time
The default password comparison logic does not use
constant time validation. Switching to constant time
offers a meager improvement by removing a timing
oracle.

A prepatory step in moving to Argon2id password storage, since we'll need this change anyway after
for validating existing passwords.

Co-authored-by: Graham Christensen <graham@grahamc.com>
2021-04-15 11:34:56 -04:00
Maximilian Bosch 3bc5252c80
Get rid of now obsolte LD_PRELOAD hack 2021-04-03 09:56:15 +02:00
Maximilian Bosch 2beb1f5405
Replace TestHTTPMockServer with python script
This seems to work fine in a `nix build`-sandbox as it doesn't depend on
`getprotobyname`.
2021-04-03 01:07:07 +02:00
Maximilian Bosch d16bf5b8cd
Implement yath-test for the new Gitea plugin 2021-04-02 19:11:25 +02:00
Maximilian Bosch f9f5ab2fb1
Make gitea public URL configurable
Otherwise, it will be obtained from the jobset input that contains the
URL to the git repo to build.
2021-03-30 23:01:36 +02:00
Maximilian Bosch eecea56131
Implement VM-test for gitea plugin 2021-03-30 22:35:39 +02:00
Cole Helbling 025be052b7
tests: move to t, allow yath test from root
By moving the tests subdirectory to t, we gain the ability to run `yath
test` with no arguments from inside `nix develop` in the root of the
the repo.

(`nix develop` is necessary in order to set the proper env vars for
`yath` to find our test libraries.)
2021-03-05 09:49:06 -08:00
Graham Christensen 9ddc6e355f
flake: add TestPostgreSQL for per-test DBs 2021-02-23 21:12:02 -05:00
Graham Christensen bfcc50f978
Merge pull request #877 from immae/mdbook
Convert documentation to markdown / mdbook
2021-02-23 20:16:33 -05:00
Ismaël Bouya 9d916877fb
Add markdown files for documentation
projects.xml and declarative-projects.xml were merged with xmllint, and
then I ran that to convert files
for i in *.xml; do pandoc -s -f docbook -t markdown $i -o ${i/xml/md}; done
2021-02-24 01:07:00 +01:00
Graham Christensen 45d9a22d73
flake.nix: add perlPackages until they're available from nixpkgs
These packages were added to Nixpkgs in https://github.com/NixOS/nixpkgs/pull/113702.
2021-02-19 17:06:49 -05:00
Graham Christensen 2240035e20
Run tests with yath
This will let us run tests in parallel, and creates a more Perl-standard
test development experience.
2021-02-19 17:04:19 -05:00
Eelco Dolstra 525a229dac Convert validate-openapi to a Hydra job 2021-01-03 18:47:05 +01:00
Eelco Dolstra 79d34ed7c9 Follow nix/nixpkgs 2020-10-20 12:21:06 +02:00
Eelco Dolstra 90a2f8eb2b Merge branch 'master' of github.com:NixOS/hydra into master 2020-10-20 12:19:53 +02:00
Eelco Dolstra 338d946620 Remove nix from hydra-perl-deps
Also add nix to passthru. This makes it easier to override nix in 'nix
develop', e.g.

  $ nix develop \
    --redirect .#hydraJobs.build.x86_64-linux.nix ~/Dev/nix/outputs/out \
    --redirect .#hydraJobs.build.x86_64-linux.nix.dev ~/Dev/nix/outputs/dev
2020-10-20 12:15:18 +02:00
Andreas Rammhold 458b0ceb85
ldap: Use hashed passwords in the test
We should lead with good example and when users copy code from this repo
it should not incentivize them to use plain text passwords.
2020-10-12 17:18:37 +02:00
Andreas Rammhold 656b0c27b3
Update to NixOS 20.09
This allows us to make use of the openldap modules for hashed passwords.
In 20.03 the openldap expression didn't yet build those modules.
2020-10-12 17:18:01 +02:00
Andreas Rammhold ee60f4ccd9 tests: fix invalid YAML
Somwhere while polishing the initial PR I forgot to add that colo.
2020-09-12 19:57:24 +02:00
Andreas Rammhold f229da352e
LDAP add test for roles and multiple users 2020-09-10 17:09:32 +02:00
Andreas Rammhold cfc01e2518
LDAP: add VM test to flake.nix
In this newly added test an OpenLDAP server will provide one user
(called `user`) and it will be attempted to login as that said user.
Also logging in with any other password must fail.
2020-09-10 17:09:32 +02:00
Andreas Rammhold b8c19337b6
LDAP: add the required packages to the perlPackage via the overlay
Nixpkgs doesn't currently provide these required packages. In order to
use this feature without waiting for a newer release of NixOS/Nixpkgs
thes have been packages inline.
2020-09-10 16:47:33 +02:00
edef b9ff7b2671 include perlPackages.YAML in buildInputs 2020-09-09 13:02:49 +02:00
Nikola Knezevic fceaed2b24 Make PathInput plugin cache validity configurable
PathInput plugin keeps a cache of path evaluations. This cache is simple, and
path is not checked more than once every N seconds, where N=30. The caching is
there to avoid expensive calls to `nix-store --add`.

This change makes the validity period configurable. The main use case is
`api-test.pl` which was implemented wrong for a while, as the invocation of
`hydra-eval-jobset` would return the previous evaluation, claiming there are no
changes. The test has been fixed to check better for a new evaluation.
2020-06-04 12:26:47 +02:00
Eelco Dolstra af9f635a5a
Use PostgreSQL 11
This is what we use for hydra.nixos.org.
2020-05-27 17:34:14 +02:00
Tobias Pflug 31262f14fb Address PR comments:
- scripts -> foreman
- drop runHydra
- drop devShell
- move postgresql to buildInputs
2020-05-15 13:48:21 +02:00
Tobias Pflug d9d58b7055 Use pg_ctl for readiness check
Using `pg_ctl status` is more reliable than relying checking an
open port via netcat.
2020-05-15 12:37:16 +02:00
Tobias Pflug 07a4465260 Use custom ports for postgres/hydra
Use custom ports so hydra and postgres can run in environments where
the default ports are in use already.
2020-05-15 12:37:16 +02:00
Andreas Rammhold eb06a435ab Add devShell for faster feedback
This adds a `devShell` which unlike `runHydra` doesn't start hydra
automatically and doesn't receive hydra as build input. It is better
suited for interactive development cycles:

```
$ nix-shell -A devShell
$ ./bootstrap
$ configurePhase
$ make
$ # hack hack hack
$ foreman start
  # test test test
  <C-c>
$ # hack hack hack
```
2020-05-15 12:37:16 +02:00
Andreas Rammhold d1237c315d Add runHydra shell
runHyda automatically starts hydra and postgres:

```
$ nix-shell -A runHydra
```

The shell receives hydra from the working copy as buildInput.
Running hydra, queue-runner, evaluator and postgres is managed
by foreman (https://github.com/ddollar/foreman) and configured
in `Procfile`.
2020-05-15 12:37:16 +02:00
Eelco Dolstra cb06f0af47
flake.nix: Remove edition field 2020-05-12 14:03:06 +02:00
Nikola Knezevic 3efa8223c9 Convert all tests to the Python DSL
Since Perl-based NixOS tests will be deprecated in nixpkgs 20.09, this
change migrates all tests to the new Python-based NixOS tests.
2020-05-11 22:05:30 +02:00
Maximilian Bosch 16a8b4d086
Re-add DBDSQLite since it's needed by the Perl-bindings of Nix 2020-04-16 00:42:40 +02:00
Maximilian Bosch efcbc08686
Get rid of dependency to SQLite
SQLite isn't properly supported by Hydra for a few years now[1], but
Hydra still depends on it. Apart from a slightly bigger closure this can
cause confusion by users since Hydra picks up SQLite rather than
PostgreSQL by default if HYDRA_DBI isn't configured properly[2]

[1] 78974abb69
[2] https://logs.nix.samueldr.com/nixos-dev/2020-04-10#3297342;
2020-04-16 00:42:40 +02:00
Eelco Dolstra 400a0fbd14
lastModified -> lastModifiedDate 2020-04-09 15:44:53 +02:00
Eelco Dolstra f6119f2b19
perlDeps: Use lib.closePropagation
Apparentely, buildEnv in 20.03 no longer respects
propagated-build-inputs.

Note that the use of a library function (closePropagation) seems
fundamentally wrong to me - propagated-build-inputs should be used at
runtime, not at evaluation time.
2020-03-31 13:21:01 +02:00
Eelco Dolstra 810075cd3d
Update to 20.03 2020-03-31 12:55:35 +02:00
Eelco Dolstra 101a9b3797
Run 'make check'
This got lost when I converted from releaseTools.nixBuild to
stdenv.mkDerivation. Doh!
2020-03-04 15:42:51 +01:00
Eelco Dolstra 6f1d68bda4
Revert "hydra-eval-jobs -> nix eval-hydra-jobs"
This reverts commit 345512a6d0.
2020-02-19 20:36:52 +01:00
Eelco Dolstra 4b9cdc4933 Remove boehmgc dependency
This is now propagated by Nix.
2020-02-15 21:56:35 +01:00
Eelco Dolstra 345512a6d0
hydra-eval-jobs -> nix eval-hydra-jobs 2020-02-15 15:59:34 +01:00
Eelco Dolstra 23c9ca3e94
Remove hydra-eval-guile-jobs
This hasn't been used in a long time (Guix uses its own CI system),
and it probably doesn't work anymore.
2020-02-15 15:59:34 +01:00
Eelco Dolstra f0e8e56f89 Update flake.lock 2020-02-03 18:14:06 +01:00
Eelco Dolstra 4c8f600eaa
Update to latest Nix and fix compilation 2019-11-07 23:16:05 +01:00
Eelco Dolstra 1784c362ed
Simplify flake.nix 2019-11-07 18:46:32 +01:00
Eelco Dolstra 55b0afa08f
Merge remote-tracking branch 'origin/master' into flake 2019-11-07 18:42:15 +01:00
Eelco Dolstra 7ba06c13ac Bump flake dependencies 2019-10-22 20:19:40 +02:00
Eelco Dolstra 551e936fac Update flake.nix
Currently Hydra is only supported on x86_64-linux.
2019-10-15 20:21:42 +02:00
Eelco Dolstra 80738fadb1
Add hydraProxy module 2019-10-04 21:31:27 +02:00
Eelco Dolstra d0b972bdd0
Use the overlay from the nix flake 2019-10-04 17:32:29 +02:00
Eelco Dolstra 8d2896b36b
epoch -> edition 2019-09-26 18:35:00 +02:00
Eelco Dolstra e110d5a38f
Build the container in Hydra 2019-09-24 21:48:28 +02:00
Eelco Dolstra 40e7b6e884
Add a NixOS container to demo Hydra
It contains Hydra, PostgreSQL and an frontend proxy. So you can get a
running Hydra instance by doing

  $ nixos-container create hydra --flake hydra
  $ nixos-container start hydra

The web interface is available on port 80.
2019-09-24 19:58:10 +02:00