lix/doc/manual/src/advanced-topics/cores-vs-jobs.md

# Tuning Cores and Jobs

Nix has two relevant settings with regards to how your CPU cores will
be utilized: `cores` and `max-jobs`. This chapter will talk about what
they are, how they interact, and their configuration trade-offs.

  - `max-jobs`\
    Dictates how many separate derivations will be built at the same
    time. If you set this to zero, the local machine will do no
    builds.  Nix will still substitute from binary caches, and build
    remotely if remote builders are configured.

  - `cores`\
    Suggests how many cores each derivation should use. Similar to
    `make -j`.

The `cores` setting determines the value of
`NIX_BUILD_CORES`. `NIX_BUILD_CORES` is equal to `cores`, unless
`cores` equals `0`, in which case `NIX_BUILD_CORES` will be the total
number of cores in the system.

The maximum number of consumed cores is a simple multiplication,
`max-jobs` \* `NIX_BUILD_CORES`.

The balance on how to set these two independent variables depends upon
each builder's workload and hardware. Here are a few example scenarios
on a machine with 24 cores:

| `max-jobs` | `cores` | `NIX_BUILD_CORES` | Maximum Processes | Result                                                                                                                                                                                                                                                                                 |
| --------------------- | ------------------ | ----------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1                     | 24                 | 24                | 24                | One derivation will be built at a time, each one can use 24 cores. Undersold if a job can’t use 24 cores.                                                                                                                                                                              |
| 4                     | 6                  | 6                 | 24                | Four derivations will be built at once, each given access to six cores.                                                                                                                                                                                                                |
| 12                    | 6                  | 6                 | 72                | 12 derivations will be built at once, each given access to six cores. This configuration is over-sold. If all 12 derivations being built simultaneously try to use all six cores, the machine's performance will be degraded due to extensive context switching between the 12 builds. |
| 24                    | 1                  | 1                 | 24                | 24 derivations can build at the same time, each using a single core. Never oversold, but derivations which require many cores will be very slow to compile.                                                                                                                            |
| 24                    | 0                  | 24                | 576               | 24 derivations can build at the same time, each using all the available cores of the machine. Very likely to be oversold, and very likely to suffer context switches.                                                                                                                  |

It is up to the derivations' build script to respect host's requested
cores-per-build by following the value of the `NIX_BUILD_CORES`
environment variable.
-												Pandoc conversion

											
										
										
											2020-07-22 21:17:48 +00:00
+								# Tuning Cores and Jobs
-												Fix some dangling references

											
										
										
											2020-07-24 12:31:33 +00:00
+								Nix has two relevant settings with regards to how your CPU cores will
 								be utilized: `cores` and `max-jobs`. This chapter will talk about what
 								they are, how they interact, and their configuration trade-offs.
-												Pandoc conversion

											
										
										
											2020-07-22 21:17:48 +00:00
-												Replace the trailing markdown spaces by a backslash

They are equivalent according to
<https://spec.commonmark.org/0.29/#hard-line-breaks>,
and the trailing spaces tend to be a pain (because the make git
complain, editors tend to want to remove them − the `.editorconfig`
actually specifies that − etc..).

											
										
										
											2021-04-23 12:30:42 +00:00
+								  - `max-jobs`\
-												Pandoc conversion

											
										
										
											2020-07-22 21:17:48 +00:00
+								    Dictates how many separate derivations will be built at the same
-												Fix some dangling references

											
										
										
											2020-07-24 12:31:33 +00:00
+								    time. If you set this to zero, the local machine will do no
 								    builds.  Nix will still substitute from binary caches, and build
 								    remotely if remote builders are configured.
-												Pandoc conversion

											
										
										
											2020-07-22 21:17:48 +00:00
-												Replace the trailing markdown spaces by a backslash

They are equivalent according to
<https://spec.commonmark.org/0.29/#hard-line-breaks>,
and the trailing spaces tend to be a pain (because the make git
complain, editors tend to want to remove them − the `.editorconfig`
actually specifies that − etc..).

											
										
										
											2021-04-23 12:30:42 +00:00
+								  - `cores`\
-												Fix some dangling references

											
										
										
											2020-07-24 12:31:33 +00:00
+								    Suggests how many cores each derivation should use. Similar to
 								    `make -j`.
-												Pandoc conversion

											
										
										
											2020-07-22 21:17:48 +00:00
-												Fix some dangling references

											
										
										
											2020-07-24 12:31:33 +00:00
+								The `cores` setting determines the value of
 								`NIX_BUILD_CORES`. `NIX_BUILD_CORES` is equal to `cores`, unless
 								`cores` equals `0`, in which case `NIX_BUILD_CORES` will be the total
 								number of cores in the system.
-												Pandoc conversion

											
										
										
											2020-07-22 21:17:48 +00:00
 								The maximum number of consumed cores is a simple multiplication,
-												Fix some dangling references

											
										
										
											2020-07-24 12:31:33 +00:00
+								`max-jobs` \* `NIX_BUILD_CORES`.
-												Pandoc conversion

											
										
										
											2020-07-22 21:17:48 +00:00
 								The balance on how to set these two independent variables depends upon
 								each builder's workload and hardware. Here are a few example scenarios
 								on a machine with 24 cores:
-												Fix some dangling references

											
										
										
											2020-07-24 12:31:33 +00:00
+								| `max-jobs` | `cores` | `NIX_BUILD_CORES` | Maximum Processes | Result                                                                                                                                                                                                                                                                                 |
-												Pandoc conversion

											
										
										
											2020-07-22 21:17:48 +00:00
+								| --------------------- | ------------------ | ----------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 								| 1                     | 24                 | 24                | 24                | One derivation will be built at a time, each one can use 24 cores. Undersold if a job can’t use 24 cores.                                                                                                                                                                              |
 								| 4                     | 6                  | 6                 | 24                | Four derivations will be built at once, each given access to six cores.                                                                                                                                                                                                                |
 								| 12                    | 6                  | 6                 | 72                | 12 derivations will be built at once, each given access to six cores. This configuration is over-sold. If all 12 derivations being built simultaneously try to use all six cores, the machine's performance will be degraded due to extensive context switching between the 12 builds. |
 								| 24                    | 1                  | 1                 | 24                | 24 derivations can build at the same time, each using a single core. Never oversold, but derivations which require many cores will be very slow to compile.                                                                                                                            |
 								| 24                    | 0                  | 24                | 576               | 24 derivations can build at the same time, each using all the available cores of the machine. Very likely to be oversold, and very likely to suffer context switches.                                                                                                                  |
 								It is up to the derivations' build script to respect host's requested
-												Reconvert

											
										
										
											2020-07-23 08:44:54 +00:00
+								cores-per-build by following the value of the `NIX_BUILD_CORES`
-												Pandoc conversion

											
										
										
											2020-07-22 21:17:48 +00:00
+								environment variable.