Commit graph

3464 commits

Author SHA1 Message Date
Taeer Bar-Yam
e5eb05c599 getBuildLog: factor out resolving derivations 2022-12-15 15:58:54 -05:00
Taeer Bar-Yam
3b27181ee5 fix missing function after rebase 2022-12-08 16:59:21 -05:00
regnat
04b113f6cb Fix nix log with CA derivations
Fix #6209

When trying to run `nix log <installable>`, try first to resolve the derivation pointed to
by `<installable>` as it is the resolved one that holds the build log.

This has a couple of shortcomings:
1. It’s expensive as it requires re-reading the derivation
2. It’s brittle because if the derivation doesn’t exist anymore or can’t
   be resolved (which is the case if any one of its build inputs is missing),
   then we can’t access the log anymore

However, I don’t think we can do better (at least not right now).
The alternatives I see are:
1. Copy the build log for the un-resolved derivation. But that means a
   lot of duplication
2. Store the results of the resolving in the db. Which might be the best
   long-term solution, but leads to a whole new class of potential
   issues.
2022-12-08 16:03:20 -05:00
8e0946e8df Remove repeat and enforce-determinism options
These only functioned if a very narrow combination of conditions held:

- The result path does not yet exist (--check did not result in
  repeated builds), AND
- The result path is not available from any configured substituters, AND
- No remote builders that can build the path are available.

If any of these do not hold, a derivation would be built 0 or 1 times
regardless of the repeat option. Thus, remove it to avoid confusion.
2022-12-07 11:36:48 +01:00
Eelco Dolstra
54906bc93c
Merge pull request #7382 from fricklerhandwerk/doc-automatic-uid
move documentation on `auto-allocate-uids` to options docs
2022-12-06 11:31:34 +01:00
Eelco Dolstra
484578d3f9
Tweak option descriptions 2022-12-06 10:30:36 +01:00
Taeer Bar-Yam
e4f9f3bf24 check the store for input before failing (hopefully fix #6700) 2022-12-05 11:27:47 -05:00
Eelco Dolstra
1e6a5d1ff6 Clean up cgroup handling in getMaxCPU()
Also, don't assume in LocalDerivationGoal that cgroups are mounted on
/sys/fs/cgroup.
2022-12-02 12:59:13 +01:00
Eelco Dolstra
1211e59a03 Move cgroup.{cc,hh} to libutil 2022-12-02 12:38:03 +01:00
Valentin Gagarin
0ea62670ed move documentation on auto-allocate-uids to options docs
this is where it belongs and can be found together with the other
options.
2022-12-01 04:40:02 +01:00
Eelco Dolstra
fbc53e97ed
Merge pull request #3600 from NixOS/auto-uid-allocation
Automatic UID allocation
2022-11-29 14:01:42 +01:00
Eelco Dolstra
4f762e2b02 Restore ownership of / for non-uid-range builds 2022-11-29 13:10:53 +01:00
Eelco Dolstra
67bcb99700 Add a setting for enabling cgroups 2022-11-28 21:54:02 +01:00
Eelco Dolstra
ff12d1c1a1 Check that auto-allocated UIDs don't clash with existing accounts 2022-11-28 20:49:17 +01:00
Eelco Dolstra
dbf78a7ada
Merge pull request #7313 from yorickvP/nlohmann-everywhere
Replace src/libutil/json.cc with nlohmann
2022-11-28 15:03:48 +01:00
Eelco Dolstra
5b798f6cae Fix random client failures during GC server shutdown
We need to close the GC server socket before shutting down the active
GC client connections, otherwise a client may (re)connect and get
ECONNRESET. But also handle ECONNRESET for resilience.

Fixes random failures like

  GC socket disconnected
  connecting to '/tmp/nix-shell.y07M0H/nix-test/default/var/nix/gc-socket/socket'
  sending GC root '/tmp/nix-shell.y07M0H/nix-test/default/store/kb5yzija0f1x5xkqkgclrdzldxj6nnc6-non-blocking'
  reading GC root from client: error: unexpected EOF reading a line
  1 store paths deleted, 0.00 MiB freed
  error: reading from file: Connection reset by peer

in gc-non-blocking.sh.
2022-11-27 12:57:18 +01:00
Eelco Dolstra
6292d5616e Merge remote-tracking branch 'origin/master' into auto-uid-allocation 2022-11-23 11:16:09 +01:00
Eelco Dolstra
05d0892443
Merge pull request #7328 from edolstra/nix-build-stats
nix build --json: Include build statistics
2022-11-22 14:41:15 +01:00
Eelco Dolstra
3d23b9d032 SimpleUserLock::getSupplementaryGIDs(): Filter out main gid
This avoids having the user's gid in the supplementary group list as
well.
2022-11-22 10:26:17 +01:00
Eelco Dolstra
b37c2d84b6 Always call setgroups()
We shouldn't skip this if the supplementary group list is empty,
because then the sandbox won't drop the supplementary groups of the
parent (like "root").
2022-11-22 10:26:17 +01:00
Eelco Dolstra
02c02ee7c3
Merge pull request #6456 from amjoseph-nixpkgs/seccomp-mips
local-derivation-goal.cc: enable seccomp filters for mips{32,64}
2022-11-21 23:03:00 +01:00
Eelco Dolstra
c776dfbb35
Use hex for startId
Co-authored-by: Linus Heckemann <git@sphalerite.org>
2022-11-21 18:46:55 +01:00
Eelco Dolstra
9d17ce07e8 AutoUserLock: If sandboxing is disabled, use the build users group
We have to use a gid that has write access to the Nix store.
2022-11-21 12:55:49 +01:00
Eelco Dolstra
f0baa5c128 nix build --json: Include build statistics
Example:

  # nix build -L --extra-experimental-features cgroups --impure --expr 'with import <nixpkgs> {}; runCommand "foo" {} "dd if=/dev/urandom bs=1M count=1024 | md5sum; mkdir $out"' --json
  [
    {
      "cpuSystem": 1.911431,
      "cpuUser": 1.214249,
      "drvPath": "/nix/store/xzdqz67xba18hljhycp0hwfigzrs2z69-foo.drv",
      "outputs": {
        "out": "/nix/store/rh9mc9l2gkpq8kn2sgzndr6ll7ffjh6l-foo"
      },
      "startTime": 1669024076,
      "stopTime": 1669024079
    }
  ]
2022-11-21 12:06:01 +01:00
Eelco Dolstra
e7a5b76844 Rename derivedPathsWithHintsToJSON -> builtPathsToJSON 2022-11-21 11:56:20 +01:00
Eelco Dolstra
82d5cf2a76 Fix macOS build 2022-11-21 11:45:41 +01:00
Eelco Dolstra
653b32a78f Merge remote-tracking branch 'origin/master' into auto-uid-allocation 2022-11-21 11:33:23 +01:00
Eelco Dolstra
ec45f4b82e Fix indentation 2022-11-21 11:12:45 +01:00
Eelco Dolstra
300753d594 nix build --json: Include build statistics
Example:

  # nix build -L --extra-experimental-features cgroups --impure --expr 'with import <nixpkgs> {}; runCommand "foo" {} "dd if=/dev/urandom bs=1M count=1024 | md5sum; mkdir $out"' --json
  [
    {
      "cpuSystem": 1.911431,
      "cpuUser": 1.214249,
      "drvPath": "/nix/store/xzdqz67xba18hljhycp0hwfigzrs2z69-foo.drv",
      "outputs": {
        "out": "/nix/store/rh9mc9l2gkpq8kn2sgzndr6ll7ffjh6l-foo"
      },
      "startTime": 1669024076,
      "stopTime": 1669024079
    }
  ]
2022-11-21 10:49:01 +01:00
Eelco Dolstra
f538ee4342 Rename derivedPathsWithHintsToJSON -> builtPathsToJSON 2022-11-21 09:38:08 +01:00
Eelco Dolstra
e6b71f84a0 Use cgroup.kill to quickly kill cgroups 2022-11-18 16:59:36 +01:00
Eelco Dolstra
fa68eb367e Get CPU stats from the cgroup 2022-11-18 13:40:59 +01:00
Eelco Dolstra
128910ba23 Separate cgroup support from auto-uid-allocation
The new experimental feature 'cgroups' enables the use of cgroups for
all builds. This allows better containment and enables setting
resource limits and getting some build stats.
2022-11-18 10:39:28 +01:00
Eelco Dolstra
f423d4425f Fix segfault in unprivileged mode 2022-11-17 11:56:45 +01:00
09f00dd4d0
Replace src/libutil/json.cc with nlohmann json generation 2022-11-16 16:50:50 +01:00
Théophane Hufschmitt
4bf70b74a7
Merge pull request #7294 from tobim/support-aws-sdk-1.10
libstore: link to aws-crt-cpp
2022-11-15 16:51:09 +01:00
Théophane Hufschmitt
3ade5f5d60
Merge pull request #7283 from hercules-ci/issue-6572
Fix #6572 `requires non-existent output`
2022-11-15 16:24:24 +01:00
7e162c69fe derivation-goal: Fix requires non-existing output error
It occurred when a output of the dependency was already available,
so it didn't need rebuilding and didn't get added to the
inputDrvOutputs.
This process-related info wasn't suitable for the purpose of finding
the actual input paths for the builder. It is better to do this in
absolute terms by querying the store.
2022-11-14 17:52:55 +01:00
Théophane Hufschmitt
8b4352d79b Merge remote-tracking branch 'nixos/master' into readFile-scan-references 2022-11-14 15:00:05 +01:00
Tobias Mayer
07f2cb1e8f
libstore: link to aws-crt-cpp
This change is needed to support aws-sdk-cpp 1.10 and newer.

I opted not to make this dependent on the sdk version because
the crt dependency has been in the interface of the older
sdk as well, and it was only coincidence that libstore didn't
make use of any privately defined symbols directly.
2022-11-12 14:34:23 +01:00
Naïm Favier
e7ed9ae0c7
Restrict readFile context to references that appear in the string
When calling `builtins.readFile` on a store path, the references of that
path are currently added to the resulting string's context.

This change makes those references the *possible* context of the string,
but filters them to keep only the references whose hash actually appears
in the string, similarly to what is done for determining the runtime
references of a path.
2022-11-11 13:04:34 +01:00
Eelco Dolstra
6c6eff8ac4 Remove the SystemdCgroup feature 2022-11-10 17:24:12 +01:00
Eelco Dolstra
05d258667d Fix build on macOS 2022-11-08 08:00:29 -08:00
Eelco Dolstra
2fde7e0108 Split auto UID allocation from cgroups
Cgroups are now only used for derivations that require the uid-range
range feature. This allows auto UID allocation even on systems that
don't have cgroups (like macOS).

Also, make things work on modern systems that use cgroups v2 (where
there is a single hierarchy and no "systemd" controller).
2022-11-08 16:03:42 +01:00
Eelco Dolstra
b95faccf03 Merge remote-tracking branch 'origin/master' into auto-uid-allocation 2022-11-03 17:43:40 +01:00
Théophane Hufschmitt
f8d0193383 Pass the right argv when calling the build hook
Call it as `['nix', '__build-remote', ... ]` rather than the previous
`["__build-remote", "nix __build-remote", ... ]` which seemed to have
been most likely unintended
2022-10-27 11:53:04 +02:00
Austin Kiekintveld
8e7804273c Defer to SSH config files for ForwardAgent option
Currently, Nix passes `-a` when it runs commands on a remote machine via
SSH, which disables agent forwarding. This causes issues when the
`ForwardAgent` option is set in SSH config files, as the command line
operation always overrides those.

In particular, this causes issues if the command being run is `sudo`
and the remote machine is configured with the equivalent of NixOS's
`security.pam.enableSSHAgentAuth` option. Not allowing SSH agent
forwarding can cause authentication to fail unexpectedly.

This can currently be worked around by setting `NIX_SSHOPTS="-A"`, but
we should defer to the options in the SSH config files to be least
surprising for users.
2022-10-22 19:51:22 -05:00
Eelco Dolstra
e136d57f26
Implement BinaryCacheStore::queryPathFromHashPart() 2022-10-18 17:48:09 +02:00
Eelco Dolstra
3093bd3a85
Merge pull request #7168 from NixOS/rosetta-test
Improve Rosetta detection
2022-10-14 17:35:57 +02:00
Eelco Dolstra
285277a61a Remove useless debug statements
We haven't parsed the '-v' command line flags yet when this code executes,
so we can't actually get debug output here.
2022-10-14 00:35:33 -07:00