Commit graph

3128 commits

Author SHA1 Message Date
Eelco Dolstra 6bb876cb35
Merge pull request #846 from grahamc/buildoutputs-index-hash-path
BuildOutputs: index path with HASH
2021-01-18 20:01:44 +01:00
Graham Christensen bc4b96d053
BuildOutputs: index path with HASH
Looking at AWS' Performance Insights for a Hydra instance, I found
the hydra-queue-runner's query:

    select id, buildStatus, releaseName, closureSize, size
    from Builds b
    join BuildOutputs o on b.id = o.build
    where
      finished = ?
      and (buildStatus = ? or buildStatus = ?)
      and path = $1

was the slowest query by at least 10x. Running an explain on this
showed why:

hydra=> explain select id, buildStatus, releaseName, closureSize, size
    from Builds b join BuildOutputs o on b.id = o.build where
    finished = 1 and (buildStatus = 0 or buildStatus = 6) and
    path = '/nix/store/s93khs2dncf2cy273mbyr4fb4ns3db20-MIDIVisualizer-5.1';

                                                     QUERY PLAN
    ------------------------------------------------------------------------
     Gather  (cost=1000.43..33718.98 rows=2 width=56)
       Workers Planned: 2
       ->  Nested Loop  (cost=0.43..32718.78 rows=1 width=56)
             ->  Parallel Seq Scan on buildoutputs o  (cost=0.00..32710.32
                                                       rows=1
                                                        width=4)
                   Filter: (path = '/nix/store/s93kh...snip...'::text)
             ->  Index Scan using indexbuildsonjobsetidfinishedid on builds b
                                            (cost=0.43..8.45 rows=1 width=56)
                   Index Cond: ((id = o.build) AND (finished = 1))
                   Filter: ((buildstatus = 0) OR (buildstatus = 6))
    (8 rows)

A paralell sequential scan is definitely better than a sequential scan, but the
cost ranging from 0 to 32710 is not great. Looking at the table, I saw the `path`
column is completely unindex:

    hydra=> \d buildoutputs
                Table "public.buildoutputs"
    Column |  Type   | Collation | Nullable | Default
    --------+---------+-----------+----------+---------
    build  | integer |           | not null |
    name   | text    |           | not null |
    path   | text    |           | not null |
    Indexes:
        "buildoutputs_pkey" PRIMARY KEY, btree (build, name)
    Foreign-key constraints:
        "buildoutputs_build_fkey" FOREIGN KEY (build) REFERENCES builds(id)
            ON DELETE CASCADE

Since we always do exact matches on the path and don't care about ordering,
and since the path column is very high cardinality a `hash` index is a
good candidate. Note that I did test a btree index and it performed
similarly well, but slightly worse.

After creating the index (this took about 10 seconds) on a test database:

    create index IndexBuildOutputsPath on BuildOutputs using hash(path);

We get a *significantly* reduced cost:

    hydra=> explain select id, buildStatus, releaseName, closureSize, size
    hydra->     from Builds b join BuildOutputs o on b.id = o.build where
    hydra->     finished = 1 and (buildStatus = 0 or buildStatus = 6) and
    hydra->     path = '/nix/store/s93khs2dncf2cy273mbyr4fb4ns3db20-MIDIVisualizer-5.1';
                                                QUERY PLAN
    -------------------------------------------------------------------------------------------------------
    Nested Loop  (cost=0.43..41.41 rows=2 width=56)
    ->  Index Scan using buildoutputs_path_hash on buildoutputs o  (cost=0.00..16.05 rows=3 width=4)
            Index Cond: (path = '/nix/store/s93khs2dncf2cy273mbyr4fb4ns3db20-MIDIVisualizer-5.1'::text)
    ->  Index Scan using indexbuildsonjobsetidfinishedid on builds b  (cost=0.43..8.45 rows=1 width=56)
            Index Cond: ((id = o.build) AND (finished = 1))
            Filter: ((buildstatus = 0) OR (buildstatus = 6))
    (6 rows)

For direct comparison, the overall query plan was changed:

    From: Gather      (cost=1000.43..33718.98 rows=2 width=56)
    To:   Nested Loop (cost=   0.43.....41.41 rows=2 width=56)

and the query plan for buildoutputs changed from a maximum cost of
32,710 down to 16.

In practical terms, the query's planning and execution time was reduced:

Before (ms) | Try 1   | Try 2   | Try 3
------------+---------+---------+--------
Planning    |   0.898 |   0.416 |   0.383
Execution   | 138.644 | 172.331 | 375.585

After (ms)  | Try 1   | Try 2   | Try 3
------------+---------+---------+--------
Planning    |   0.298 |   0.290 |   0.296
Execution   | 219.625 |   0.035 |   0.034
2021-01-18 11:28:05 -05:00
Eelco Dolstra be0aa7eb85
Merge pull request #841 from pingiun/github-login
Implement GitHub logins
2021-01-05 14:51:51 +01:00
Jelle Besseling 43d662f63a
Don't use enable_github_login option after all
Instead the github_client_id option is used to detect if github logins
should be enabled.
2021-01-04 18:09:49 +01:00
Jelle Besseling c49ca66689
Die when no email is found 2021-01-04 18:09:05 +01:00
Jelle Besseling 20d8134936
Update src/lib/Hydra/Controller/User.pm
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-04 17:48:43 +01:00
Jelle Besseling 19f9d8249f
Update src/lib/Hydra/Controller/User.pm
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-04 17:48:37 +01:00
Eelco Dolstra 20d09518f8
Merge pull request #839 from pingiun/shield-io
Add endpoint to generate a shields.io badge
2021-01-04 14:09:09 +01:00
Eelco Dolstra 525a229dac Convert validate-openapi to a Hydra job 2021-01-03 18:47:05 +01:00
Eelco Dolstra ce7b23ae09 Disable broken validate-openapi test 2021-01-03 18:40:08 +01:00
Eelco Dolstra c4062c2772
Merge pull request #842 from pingiun/also-trigger-flakes
Also trigger flake based jobsets with push-github endpoint
2021-01-03 18:28:29 +01:00
Eelco Dolstra b59a5850a8 Merge branch 'receiveContents' of https://github.com/orivej/hydra 2021-01-03 18:26:04 +01:00
Eelco Dolstra 2a695a621d Merge branch 'update-for-nix2020' of https://github.com/matthewbauer/hydra 2021-01-03 18:24:35 +01:00
Eelco Dolstra 896e72169f flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/79aa7d95183cbe6c0d786965f0dbff414fd1aa67' -> 'github:NixOS/nix/8a2ce0f455da32bc20978e68c0aad9efb4560abc'
2021-01-03 18:17:05 +01:00
Eelco Dolstra bc10fd9d75
Merge pull request #831 from NixOS/dependabot/github_actions/cachix/install-nix-action-v12
Bump cachix/install-nix-action from v10 to v12
2021-01-03 18:12:07 +01:00
Jelle Besseling 5f4eddbe57
Use email scope 2020-12-31 13:40:33 +01:00
Jelle Besseling e88355b3d4
Use email api call 2020-12-31 13:40:32 +01:00
Matthew Bauer d38ef4492e Update for Nix Value::type API changes
This updates hydra to be compatible with Nix https://github.com/NixOS/nix/pull/4355.

Along with https://github.com/NixOS/hydra/pull/840 needed for https://github.com/NixOS/nixpkgs/pull/107909

/cc @edolstra
2020-12-29 17:25:51 -06:00
Jelle Besseling 1b3000e132
Allow push-github endpoint to also trigger flakes 2020-12-28 15:27:09 +01:00
Jelle Besseling bbd4891133
Implement GitHub logins
Requires the following configuration options
enable_github_login = 1
github_client_id
github_client_secret
Or github_client_secret_file which points to a file with the secret
2020-12-28 14:37:03 +01:00
Orivej Desh 34a856c7ab Update for receiveContents taking string_view
nix change: https://github.com/NixOS/nix/commit/faa31f40
2020-12-26 11:23:26 +00:00
Jelle Besseling f64230b45e
Add endpoint to generate a shields.io badge 2020-12-25 15:05:34 +01:00
Eelco Dolstra bde8d81876
Merge pull request #811 from helsinki-systems/fix/override-constraint
Stop violating not null constraint
2020-11-22 00:02:26 +01:00
Janne Heß bd0ab9a5fb
Stop violating not null constraint
Fixes this error:

ERROR: failed to process declarative jobset test:inputs,
DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st
execute failed: ERROR:  null value in column "emailoverride" violates
not-null constraint
2020-11-21 22:04:40 +01:00
Eelco Dolstra b40db4c4f8 flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/5e6eabe1551f3fa2a463fced39a4a48b504ed1ab' -> 'github:NixOS/nix/79aa7d95183cbe6c0d786965f0dbff414fd1aa67'
2020-11-19 17:23:39 +01:00
Ricardo M. Correia f47749a62d Fix persistent hash mismatch errors when importing
This would start happening if the network connection between the Hydra
server and the remote build server breaks after sucessfully importing
at least one output of a derivation, but before having finished
importing all outputs.

Fixes #816.
2020-11-10 04:50:35 +01:00
dependabot[bot] 2dcdc06e3a
Bump cachix/install-nix-action from v10 to v12
Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from v10 to v12.
- [Release notes](https://github.com/cachix/install-nix-action/releases)
- [Commits](https://github.com/cachix/install-nix-action/compare/v10...07da2520eebede906fbeefa9dd0a2b635323909d)

Signed-off-by: dependabot[bot] <support@github.com>
2020-11-09 05:34:56 +00:00
Eelco Dolstra 0bdeb63a15 flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/e0ca98c2071b815578470e280df8fdb750c7e23b' -> 'github:NixOS/nix/5e6eabe1551f3fa2a463fced39a4a48b504ed1ab'
2020-11-03 14:57:06 +01:00
Eelco Dolstra 73dfef364b Copy deriver field to the binary cache
Fixes https://github.com/NixOS/nixos-org-configurations/issues/129.
2020-11-02 17:08:02 +01:00
Eelco Dolstra d50f9a242f
Merge pull request #826 from Taneb/patch-2
Remove Debug prints from GitLabStatus.pm
2020-11-02 12:13:42 +01:00
Nathan van Doorn 2742fde8c2
Remove Debug prints from GitLabStatus.pm
These make the hydra-queue-runner logs very noisy even when not using the GitlabStatus plugin.
Also, they shouldn't be necessary except when developing the plugin itself and should have been removed before release.
2020-11-02 10:14:54 +00:00
Eelco Dolstra 87317812a8 Fix some broken indices
These indices basically did nothing since they put "id" first.

In particular this makes /job/.../all much faster.
2020-10-28 14:30:44 +01:00
Eelco Dolstra 2922f45b4f Ignore 'nix develop' outputs directory 2020-10-28 13:41:34 +01:00
Eelco Dolstra 8bb23905c3 Build: Remove unused prevBuild
This speeds up loading the page a lot in the case where there is no
previous evaluation (for some reason).
2020-10-28 13:29:31 +01:00
Eelco Dolstra d9dc7ca18b getPreviousBuild: Get previous build in the job, not jobset
Broken since 8adb433e3.
2020-10-28 13:29:02 +01:00
Eelco Dolstra a1b5418114
Remove dead link 2020-10-27 12:35:36 +01:00
Eelco Dolstra be709d450b Fix sysbuild
596f4cf4b9
2020-10-22 13:27:52 +02:00
Eelco Dolstra 79d34ed7c9 Follow nix/nixpkgs 2020-10-20 12:21:06 +02:00
Eelco Dolstra 94d696df0e flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/20a7d8d23a19530207378fc74ddd558232240f43' -> 'github:NixOS/nix/e0ca98c2071b815578470e280df8fdb750c7e23b'
* Updated 'nix/nixpkgs': 'github:NixOS/nixpkgs/3a10a004bb5802d5f23c58886722e4239705e733' -> 'github:NixOS/nixpkgs/ad0d20345219790533ebe06571f82ed6b034db31'
* Updated 'nixpkgs': 'github:NixOS/nixpkgs/5659cb448e9b615d642c5fe52779c2223e72f7eb' -> follows 'nix/nixpkgs'
2020-10-20 12:20:33 +02:00
Eelco Dolstra 90a2f8eb2b Merge branch 'master' of github.com:NixOS/hydra into master 2020-10-20 12:19:53 +02:00
Eelco Dolstra 338d946620 Remove nix from hydra-perl-deps
Also add nix to passthru. This makes it easier to override nix in 'nix
develop', e.g.

  $ nix develop \
    --redirect .#hydraJobs.build.x86_64-linux.nix ~/Dev/nix/outputs/out \
    --redirect .#hydraJobs.build.x86_64-linux.nix.dev ~/Dev/nix/outputs/dev
2020-10-20 12:15:18 +02:00
Eelco Dolstra 4e05acc471 Fix localhost builds 2020-10-20 12:11:46 +02:00
Eelco Dolstra 6cd2bb6954 Fix build 2020-10-18 21:01:06 +02:00
Eelco Dolstra deb9b68afd flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/8b4a542d1767e0df7b3c0902b766f34352cb0958' -> 'github:NixOS/nix/20a7d8d23a19530207378fc74ddd558232240f43'
* Updated 'nix/nixpkgs': 'github:NixOS/nixpkgs/70717a337f7ae4e486ba71a500367cad697e5f09' -> 'github:NixOS/nixpkgs/3a10a004bb5802d5f23c58886722e4239705e733'
2020-10-18 20:55:30 +02:00
Eelco Dolstra bea7c2581f Merge branch 'update-nix-unstable' of https://github.com/Ma27/hydra 2020-10-18 20:40:57 +02:00
Eelco Dolstra 869671e556
Merge pull request #823 from andir/ldap
ldap: Use hashed passwords in the test
2020-10-13 12:37:16 +02:00
Andreas Rammhold 458b0ceb85
ldap: Use hashed passwords in the test
We should lead with good example and when users copy code from this repo
it should not incentivize them to use plain text passwords.
2020-10-12 17:18:37 +02:00
Andreas Rammhold 656b0c27b3
Update to NixOS 20.09
This allows us to make use of the openldap modules for hashed passwords.
In 20.03 the openldap expression didn't yet build those modules.
2020-10-12 17:18:01 +02:00
Maximilian Bosch 9cc76f6d69
Fix build with latest Nix
Recently a few internal APIs have changed[1]. The `outputPaths` function
has been removed and a lot of data structures are modeled with
`std::optional` which broke compilation.

This patch updates the code in `hydra-queue-runner` accordingly to make
sure that Hydra compiles again.

[1] https://github.com/NixOS/nix/pull/3883
2020-09-26 23:37:39 +02:00
Maximilian Bosch 2394140843
flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/2a8017092025a7c108ca1a829a8b4f0bcf7ee2c0' -> 'github:NixOS/nix/8b4a542d1767e0df7b3c0902b766f34352cb0958'
* Updated 'nix/lowdown-src': 'github:edolstra/lowdown/c7a4e715af1e233080842db82d15b261cb74cb28' -> 'github:kristapsdz/lowdown/1705b4a26fbf065d9574dce47a94e8c7c79e052f'
* Updated 'nixpkgs': 'github:NixOS/nixpkgs/252bfe0107587d40092057f338e9ffcf7bbd90cb' -> 'github:NixOS/nixpkgs/5659cb448e9b615d642c5fe52779c2223e72f7eb'
2020-09-26 23:36:37 +02:00