lix-project/hydra
3
0
Fork 6
Code Issues 2 Pull requests 1 Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/master' into ca-derivations-prep

Browse source
This commit is contained in:
John Ericson 2023-12-23 19:19:54 -05:00
parent 642156372f 02e453fc8c
commit 588a0c5269
2 changed files with 32 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

24
doc/manual/src/configuration.md
View file

@ -74,6 +74,30 @@ following:
} }
} }
Populating a Cache
------------------
A common use for Hydra is to pre-build and cache derivations which
take a long time to build. While it is possible to direcly access the
Hydra server's store over SSH, a more scalable option is to upload
built derivations to a remote store like an [S3-compatible object
store](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-help-stores.html#s3-binary-cache-store). Setting
the `store_uri` parameter will cause Hydra to sign and upload
derivations as they are built:
```
store_uri = s3://cache-bucket-name?compression=zstd&parallel-compression=true&write-nar-listing=1&ls-compression=br&log-compression=br&secret-key=/path/to/cache/private/key
```
This example uses [Zstandard](https://github.com/facebook/zstd)
compression on derivations to reduce CPU usage on the server, but
[Brotli](https://brotli.org/) compression for derivation listings and
build logs because it has better browser support.
See [`nix help
stores`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-help-stores.html)
for a description of the store URI format.
Statsd Configuration Statsd Configuration
-------------------- --------------------

16
src/hydra-queue-runner/queue-monitor.cc
View file

@ -495,19 +495,19 @@ Step::ptr State::createStep(ref<Store> destStore,
if (!missing.empty()) { if (!missing.empty()) {
size_t avail = 0; size_t avail = 0;
for (auto & [i, maybePath] : missing) { for (auto & [i, pathOpt] : missing) {
// If we don't know the output path from the destination // If we don't know the output path from the destination
// store, see if the local store can tell us. // store, see if the local store can tell us.
if (/* localStore != destStore && */ !maybePath && experimentalFeatureSettings.isEnabled(Xp::CaDerivations)) if (/* localStore != destStore && */ !pathOpt && experimentalFeatureSettings.isEnabled(Xp::CaDerivations))
if (auto maybeRealisation = localStore->queryRealisation(i)) if (auto maybeRealisation = localStore->queryRealisation(i))
maybePath = maybeRealisation->outPath; pathOpt = maybeRealisation->outPath;
if (!maybePath) { if (!pathOpt) {
// No hope of getting the store object if we don't know // No hope of getting the store object if we don't know
// the path. // the path.
continue; continue;
} }
auto & path = *maybePath; auto & path = *pathOpt;
if (/* localStore != destStore && */ localStore->isValidPath(path)) if (/* localStore != destStore && */ localStore->isValidPath(path))
avail++; avail++;
@ -521,11 +521,11 @@ Step::ptr State::createStep(ref<Store> destStore,
if (missing.size() == avail) { if (missing.size() == avail) {
valid = true; valid = true;
for (auto & [i, maybePath] : missing) { for (auto & [i, pathOpt] : missing) {
// If we found everything, then we should know the path // If we found everything, then we should know the path
// to every missing store object now. // to every missing store object now.
assert(maybePath); assert(pathOpt);
auto & path = *maybePath; auto & path = *pathOpt;
try { try {
time_t startTime = time(0); time_t startTime = time(0);