Compare commits

..

7 commits

Author SHA1 Message Date
Maximilian Bosch b1db31d026
Allow specifying jobset inputs for flake builds for Hydra plugins
It seems to be a common pattern to configure Hydra plugins on a
per-jobset basis with jobset inputs. Since these are not needed anymore
for flakes, it's also not possible anymore to use plugins for flake
jobsets.

This patch changes this and adds a warning to avoid confusion. In the
future, we may want to restrict jobset inputs for flakes to string
values only.
2024-08-20 09:24:53 +02:00
Maximilian Bosch eb32395fea
Add 'private' flag to projects to display project & all associated things for authenticated users only 2024-08-20 09:24:53 +02:00
Maximilian Bosch 3ee51dbe58 readIntoSocket: fix with store URIs containing an &
The third argument to `open()` in `-|` mode is passed to a shell if it's
a string. In my case the store URI contains
`?secret-key=${signingKey.directory}/secret&compression=zstd`

For the `nix store cat` case this means that

* until `&` the process will be started in the background. This fails
  immediately because no path to cat is specified.
* `compression=zstd` is a variable assignment
* the `$path` argument to `store cat` is attempted to be executed as
  another command

Passing just the list solves the problem.
2024-08-18 21:41:54 +00:00
Maximilian Bosch e987f74954
doc: drop dev-notes & make update-dbix more discoverable
`dev-notes` are severely outdated. I dropped everything except one note
that I moved to hacking.md. The parts about creating users are also
covered elsewhere.

The `update-dbix` part got a just command to make it discoverable again.
2024-08-18 14:47:09 +02:00
Maximilian Bosch 1f802c008c
flake.lock: Update
Flake lock file updates:

• Updated input 'lix':
    'git+https://git.lix.systems/lix-project/lix?ref=refs/heads/main&rev=5137cea99044d54337e439510a647743110b2d7d' (2024-08-10)
  → 'git+https://git.lix.systems/lix-project/lix?ref=refs/heads/main&rev=278fddc317cf0cf4d3602d0ec0f24d1dd281fadb' (2024-08-17)
• Updated input 'nix-eval-jobs':
    'git+https://git.lix.systems/lix-project/nix-eval-jobs?ref=refs/heads/main&rev=c057494450f2d1420726ddb0bab145a5ff4ddfdd' (2024-07-17)
  → 'git+https://git.lix.systems/lix-project/nix-eval-jobs?ref=refs/heads/main&rev=42a160bce2fd9ffebc3809746bc80cc7208f9b08' (2024-08-13)
• Updated input 'nix-eval-jobs/flake-parts':
    'github:hercules-ci/flake-parts/9227223f6d922fee3c7b190b2cc238a99527bbb7' (2024-07-03)
  → 'github:hercules-ci/flake-parts/8471fe90ad337a8074e957b69ca4d0089218391d' (2024-08-01)
• Updated input 'nix-eval-jobs/treefmt-nix':
    'github:numtide/treefmt-nix/0fb28f237f83295b4dd05e342f333b447c097398' (2024-07-15)
  → 'github:numtide/treefmt-nix/349de7bc435bdff37785c2466f054ed1766173be' (2024-08-12)
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/a781ff33ae258bbcfd4ed6e673860c3e923bf2cc' (2024-08-10)
  → 'github:NixOS/nixpkgs/c3d4ac725177c030b1e289015989da2ad9d56af0' (2024-08-15)
2024-08-18 14:18:36 +02:00
Maximilian Bosch 3a4e0d4917
Get dev environment working again
* justfile, inspired from Lix.
* let foreman use the stuff from outputs, similar to what Lix does>
* mess around with PERL5LIB[1] and PATH to get tests running locally.

[1] I don't really know how `Setup` was found before tbh.
2024-08-18 14:18:36 +02:00
Maximilian Bosch 3517acc5ba
Add direnv & PLS to the dev setup 2024-08-18 14:18:35 +02:00
5 changed files with 33 additions and 139 deletions

View file

@ -78,11 +78,11 @@ $ nix-build
### Development Environment
You can use the provided shell.nix to get a working development environment:
```
$ nix-shell
$ autoreconfPhase
$ configurePhase # NOTE: not ./configure
$ make
$ nix develop
[nix-shell]$ just setup
[nix-shell]$ just install
```
### Executing Hydra During Development
@ -91,10 +91,9 @@ When working on new features or bug fixes you need to be able to run Hydra from
can be done using [foreman](https://github.com/ddollar/foreman):
```
$ nix-shell
$ # hack hack
$ make
$ foreman start
$ nix develop
[nix-shell]$ just install
[nix-shell]$ foreman start
```
Have a look at the [Procfile](./Procfile) if you want to see how the processes are being started. In order to avoid
@ -115,22 +114,22 @@ Start by following the steps in [Development Environment](#development-environme
Then, you can run the tests and the perlcritic linter together with:
```console
$ nix-shell
$ make check
$ nix develop
[nix-shell]$ just test
```
You can run a single test with:
```
$ nix-shell
$ yath test ./t/foo/bar.t
$ nix develop
[nix-shell]$ yath test ./t/foo/bar.t
```
And you can run just perlcritic with:
```
$ nix-shell
$ make perlcritic
$ nix develop
[nix-shell]$ just perlcritic
```
### JSON API

View file

@ -1,122 +0,0 @@
* Recreating the schema bindings:
$ make -C src/sql update-dbix
* Running the test server:
$ DBIC_TRACE=1 ./script/hydra_server.pl
* Setting the maximum number of concurrent builds per system type:
$ psql -d hydra <<< "insert into SystemTypes(system, maxConcurrent) values('i686-linux', 3);"
* Creating a user:
$ hydra-create-user root --email-address 'e.dolstra@tudelft.nl' \
--password-prompt
(Replace "foobar" with the desired password.)
To make the user an admin:
$ hydra-create-user root --role admin
To enable a non-admin user to create projects:
$ hydra-create-user root --role create-projects
* Changing the priority of a scheduled build:
update buildschedulinginfo set priority = 200 where id = <ID>;
* Changing the priority of all builds for a jobset:
update buildschedulinginfo set priority = 20 where id in (select id from builds where finished = 0 and project = 'nixpkgs' and jobset = 'trunk');
* Steps to install:
- Install the Hydra closure.
- Set HYDRA_DATA to /somewhere.
- Run hydra_init.pl
- Start hydra_server
- Visit http://localhost:3000/
- Create a user (see above)
- Create a project, jobset etc.
- Start hydra_evaluator and hydra_queue_runner
* Job selection:
php-sat:build [system = "i686-linux"]
php-sat:build [same system]
tarball [same patchelfSrc]
--if system i686-linux --arg build {...}
* Restart all aborted builds in a given evaluation (e.g. 820909):
> update builds set finished = 0 where id in (select id from builds where finished = 1 and buildstatus = 3 and exists (select 1 from jobsetevalmembers where eval = 820909 and build = id));
* Restart all builds in a given evaluation that had a build step time out:
> update builds set finished = 0 where id in (select id from builds where finished = 1 and buildstatus != 0 and exists (select 1 from jobsetevalmembers where eval = 926992 and build = id) and exists (select 1 from buildsteps where build = id and status = 7));
* select * from (select project, jobset, job, system, max(timestamp) timestamp from builds where finished = 1 group by project, jobset, job, system) x join builds y on x.timestamp = y.timestamp and x.project = y.project and x.jobset = y.jobset and x.job = y.job and x.system = y.system;
select * from (select project, jobset, job, system, max(timestamp) timestamp from builds where finished = 1 group by project, jobset, job, system) natural join builds;
* Delete all scheduled builds that are not already building:
delete from builds where finished = 0 and not exists (select 1 from buildschedulinginfo s where s.id = builds.id and busy != 0);
* select x.project, x.jobset, x.job, x.system, x.id, x.timestamp, r.buildstatus, b.id, b.timestamp
from (select project, jobset, job, system, max(id) as id from Builds where finished = 1 group by project, jobset, job, system) as a_
natural join Builds x
natural join BuildResultInfo r
left join Builds b on b.id =
(select max(id) from builds c
natural join buildresultinfo r2
where x.project = c.project and x.jobset = c.jobset and x.job = c.job and x.system = c.system
and x.id > c.id and r.buildstatus != r2.buildstatus);
* Using PostgreSQL (version 9.2 or newer is required):
$ HYDRA_DBI="dbi:Pg:dbname=hydra;" hydra-server
* Find the builds with the highest number of build steps:
select id, (select count(*) from buildsteps where build = x.id) as n from builds x order by n desc;
* Evaluating the NixOS Hydra jobs:
$ ./hydra_eval_jobs ~/Dev/nixos-wc/release.nix --arg nixpkgs '{outPath = /home/eelco/Dev/nixpkgs-wc;}' --arg nixosSrc '{outPath = /home/eelco/Dev/nixos-wc; rev = 1234;}' --arg services '{outhPath = /home/eelco/services-wc;}' --argstr system i686-linux --argstr system x86_64-linux --arg officialRelease false
* Show all the failing jobs/systems in the nixpkgs:stdenv jobset that
succeed in the nixpkgs:trunk jobset:
select job, system from builds b natural join buildresultinfo where project = 'nixpkgs' and jobset = 'stdenv' and iscurrent = 1 and finished = 1 and buildstatus != 0 and exists (select 1 from builds natural join buildresultinfo where project = 'nixpkgs' and jobset = 'trunk' and job = b.job and system = b.system and iscurrent = 1 and finished = 1 and buildstatus = 0) order by job, system;
* Get all Nixpkgs jobs that have never built succesfully:
select project, jobset, job from builds b1
where project = 'nixpkgs' and jobset = 'trunk' and iscurrent = 1
group by project, jobset, job
having not exists
(select 1 from builds b2 where b1.project = b2.project and b1.jobset = b2.jobset and b1.job = b2.job and finished = 1 and buildstatus = 0)
order by project, jobset, job;

View file

@ -97,3 +97,14 @@ Off NixOS, change `/etc/nix/nix.conf`:
```conf
trusted-users = root YOURUSERNAME
```
### Updating schema bindings
```
just update-dbix
```
### Find the builds with the highest number of build steps:
select id, (select count(*) from buildsteps where build = x.id) as n from builds x order by n desc;

View file

@ -1,11 +1,17 @@
setup *OPTIONS:
meson setup build --prefix="$PWD/outputs/out" $mesonFlags {{ OPTIONS }}
meson setup build --prefix="$PWD/outputs/out" $mesonFlags {{ OPTIONS }}
build *OPTIONS:
meson compile -C build {{ OPTIONS }}
meson compile -C build {{ OPTIONS }}
install *OPTIONS: (build OPTIONS)
meson install -C build
test *OPTIONS:
meson test -C build --print-errorlogs {{ OPTIONS }}
update-dbix:
cd src/sql && ./update-dbix-harness.sh
perlcritic:
perlcritic .

View file

@ -6,7 +6,7 @@
-- add a map of the lowercase name of your table to the CamelCase
-- version of your table.
--
-- 3. Run `make -C src/sql update-dbix` in the root
-- 3. Run `just update-dbix` in the root
-- of the project directory, and git add / git commit the changed,
-- generated files.
--