hydra/src
Eelco Dolstra 9127f5bbc3 hydra-queue-runner: Limit memory usage
When using a binary cache store, the queue runner receives NARs from
the build machines, compresses them, and uploads them to the
cache. However, keeping multiple large NARs in memory can cause the
queue runner to run out of memory. This can happen for instance when
it's processing multiple ISO images concurrently.

The fix is to use a TokenServer to prevent the builder threads to
store more than a certain total size of NARs concurrently (at the
moment, this is hard-coded at 4 GiB). Builder threads that cause the
limit to be exceeded will block until other threads have finished.

The 4 GiB limit does not include certain other allocations, such as
for xz compression or for FSAccessor::readFile(). But since these are
unlikely to be more than the size of the NARs and hydra.nixos.org has
32 GiB RAM, it should be fine.
2016-03-09 14:30:13 +01:00
..
hydra-eval-jobs Process Nix API changes 2016-02-11 15:59:47 +01:00
hydra-queue-runner hydra-queue-runner: Limit memory usage 2016-03-09 14:30:13 +01:00
lib Add a more concise queue page 2016-03-08 19:44:51 +01:00
root Add a more concise queue page 2016-03-08 19:44:51 +01:00
script Fix hydra.queue.machines.total stat 2015-09-22 12:59:19 +02:00
sql Keep track of the time we spend copying to/from build machines 2016-02-17 10:30:23 +01:00
ttf Add font for the captcha 2013-03-04 12:16:13 +01:00
xsl Fix UTF-8 handling of log files 2014-08-13 18:53:29 +02:00
Makefile.am Start of single-process hydra-queue-runner 2015-05-28 17:39:29 +02:00
Makefile.PL * Move everything up one directory. 2009-03-05 13:41:57 +00:00