Eelco Dolstra
232ca8fea2
Fix build
2016-03-02 17:05:07 +01:00
Eelco Dolstra
b98a061c24
Add some instrumentation to keep track of dispatcher cost
2016-03-02 14:18:39 +01:00
Eelco Dolstra
6beee0ab49
Fix segfault sorting runnable steps
...
Same problem as d744362e4a
2016-03-02 13:59:24 +01:00
Eelco Dolstra
7cd08c7c46
Warn if PostgreSQL appears stalled
2016-02-29 15:10:30 +01:00
Eelco Dolstra
922dc541c2
Add log message
2016-02-29 11:58:06 +01:00
Eelco Dolstra
610a8d67ae
Better AWS error messages
2016-02-26 22:40:27 +01:00
Eelco Dolstra
1a055e7e9e
Reduce severity level of some message
2016-02-26 21:31:08 +01:00
Eelco Dolstra
6bb860fd6e
Add FIXME
2016-02-26 21:15:05 +01:00
Eelco Dolstra
53ca41ef9f
Use US standard S3 region
2016-02-26 20:57:47 +01:00
Eelco Dolstra
c635f5d0ea
Fix Makefile.am
2016-02-26 19:54:55 +01:00
Eelco Dolstra
b9afaadfb3
Keep better bytesReceived/bytesSent stats
2016-02-26 16:17:05 +01:00
Eelco Dolstra
6d741d2ffa
Prevent download of NARs we just uploaded
2016-02-26 15:21:44 +01:00
Eelco Dolstra
02190b0fef
Support hydra-build-products on binary cache stores
2016-02-26 14:45:03 +01:00
Eelco Dolstra
8e24ad6f0d
Sync with Nix
2016-02-25 10:58:31 +01:00
Eelco Dolstra
8321a3eb27
Sync with Nix
2016-02-24 14:04:31 +01:00
Eelco Dolstra
7b509237cd
Bleh Automake
2016-02-22 18:05:15 +01:00
Eelco Dolstra
6c3ae36648
hydra-queue-runner: Get store mode configuration from hydra.conf
...
To use the local Nix store (default):
store_mode = direct
To use a local binary cache:
store_mode = local-binary-cache
binary_cache_dir = /var/lib/hydra/binary-cache
To use an S3 bucket:
store_mode = s3-binary-cache
binary_cache_s3_bucket = my-nix-bucket
Also, respect binary_cache_{secret,public}_key_file for signing the
binary cache.
2016-02-22 17:23:06 +01:00
Eelco Dolstra
94817d77d9
BinaryCacheStore: Respect build-use-substitutes
2016-02-22 17:21:39 +01:00
Eelco Dolstra
5668aa5f71
After uploading a .narinfo, add it to the LRU cache
2016-02-20 10:35:16 +01:00
Eelco Dolstra
88a05763cc
Pool local store connections
2016-02-20 00:04:08 +01:00
Eelco Dolstra
1cefd6cac8
Fix log message
2016-02-20 00:02:37 +01:00
Eelco Dolstra
2b76094a23
S3BinaryCacheStore::isValidPath(): Do a GET instead of HEAD
2016-02-19 17:41:11 +01:00
Eelco Dolstra
bd76f9120a
Cache .narinfo lookups
2016-02-19 16:19:40 +01:00
Eelco Dolstra
a0f74047da
Keep some statistics for the binary cache stores
2016-02-19 14:24:23 +01:00
Eelco Dolstra
dc4a00347d
Use a single BinaryCacheStore for all threads
...
This will make it easier to do caching / keep stats. Also, we won't
have S3Client's connection pooling if we create multiple S3Client
instances.
2016-02-18 17:31:19 +01:00
Eelco Dolstra
00a7be13a2
Make queue runner internal status available under /queue-runner-status
2016-02-18 17:11:46 +01:00
Eelco Dolstra
8c9fc677c1
Typo
2016-02-18 16:43:24 +01:00
Eelco Dolstra
db3fcc0f5e
Enable substitution on the build machines
...
If properly configured, this allows them to get store paths directly
from S3, rather than having to receive them from the queue runner.
2016-02-18 16:42:05 +01:00
Eelco Dolstra
2d40888e2e
Add an S3-backed binary cache store
2016-02-18 16:18:50 +01:00
Eelco Dolstra
0e254ca66d
Refactor local binary cache code into a subclass
2016-02-18 14:06:17 +01:00
Eelco Dolstra
a992f688d1
Rename class
2016-02-18 13:02:20 +01:00
Eelco Dolstra
de77cc2910
Rename file
2016-02-18 13:02:20 +01:00
Eelco Dolstra
ce5790285a
Merge remote-tracking branch 'origin/master' into binary-cache
2016-02-17 11:54:59 +01:00
Eelco Dolstra
d7a123fcd4
Keep track of the time we spend copying to/from build machines
2016-02-17 10:30:23 +01:00
Eelco Dolstra
25022bf5fd
hydra-queue-runner: Support generating a signed binary cache
2016-02-16 16:41:42 +01:00
Eelco Dolstra
744cee134e
hydra-queue-runner: Compress binary cache NARs using xz
2016-02-15 21:56:53 +01:00
Eelco Dolstra
2d0dd7fb49
hydra-queue-runner: Write directly to a binary cache
2016-02-15 21:10:29 +01:00
Eelco Dolstra
92d8b59361
Process Nix API changes
2016-02-11 15:59:47 +01:00
Eelco Dolstra
97f8c61928
Fix hydra-queue-runner --build-one
2015-12-29 17:53:33 +01:00
Eelco Dolstra
c087472c71
Remove superfluous "has" function
2015-11-02 14:29:12 +01:00
Eelco Dolstra
2d128d2a6f
Don't show redundant "removing machine..." messages
2015-10-30 18:22:43 +01:00
Eelco Dolstra
d8d188301d
Fix division-by-zero crash
...
Not clear why step_->jobsets was empty...
2015-10-30 18:01:48 +01:00
Eelco Dolstra
1ff48da3d3
int2String -> std::to_string
2015-10-30 18:01:38 +01:00
Eelco Dolstra
4d1816b152
Remove obsolete Builds columns and provide accurate "Running builds"
...
This removes the "busy", "locker" and "logfile" columns, which are no
longer used by the queue runner. The "Running builds" page now only
shows builds that have an active build step.
2015-10-27 15:37:17 +01:00
Eelco Dolstra
53c80d9526
getQueuedBuilds(): Periodically stop to handle priority bumps
...
Previously, priority bumps could take a long time to get noticed if
getQueuedBuilds() was busy processing zillions of queue
additions. (This was made worse by the reintroduction of substitute
checking.)
2015-10-22 17:00:46 +02:00
Eelco Dolstra
71bf7e02d5
Use nix::willBuildLocally()
2015-10-21 15:44:29 +02:00
Eelco Dolstra
8e8e31ce86
Re-implement log size limits
...
The old queue runner already had this. However, we now store "log
limit exceeded" as a separate status code in the database.
2015-10-06 17:35:08 +02:00
Eelco Dolstra
82504fe010
hydra-queue-runner: Use substitutes
...
This allows Hydra to use binaries from available binary caches. It
makes the queue monitor thread quite a bit slower, so if you don't
want to use binary caches, it's better to add "--option
build-use-substitutes false" to the hydra-queue-runner invocation.
Fixed #243 .
2015-10-05 14:57:44 +02:00
Eelco Dolstra
f8141fdc98
Set propagatedFrom for cached failed build steps
2015-09-11 15:55:26 +02:00
Eelco Dolstra
6075ac6fed
Remove localhost hack
2015-09-09 16:50:59 +02:00
