Commit graph

16535 commits

Author SHA1 Message Date
eldritch horrors d265dd5993 libstore: count all substitutions toward the same limit
limiting CA substitutions was a rather recent addition, and it used a
dedicated counter to not interfere with regular substitutions. though
this works fine it somewhat contradicts the documentation; job limits
should apply to all kinds of substitutions, or be one limit for each.

Change-Id: I1505105b14260ecc1784039b2cc4b7afcf9115c8
2024-07-30 15:37:27 +00:00
eldritch horrors d9af753a7f libstore: always wake up goals on EOF
all goals do this. it makes no sense to not notify a goal of EOF
conditions because this is the universal signal for "child done"

Change-Id: Ic3980de312547e616739c57c6248a8e81308b5ee
2024-07-30 15:37:27 +00:00
eldritch horrors 6c0dcd1220 libstore: simplify substitution handleEOF
both substitution goals add only this single fd to their wait set.

Change-Id: Ibf921f5bb3919106208a0871523b32c8f67fb3d3
2024-07-30 15:37:27 +00:00
eldritch horrors 548c973e82 libstore: remove Worker::updateProgress
just update progress every time a goal has returned from work(). there
seem to be no performance penalties, and the code is much simpler now.

Change-Id: I288ee568b764ee61f40a498d986afda49987cb50
2024-07-29 22:16:11 +00:00
Artemis Tosini 3058029fba
libutil: Add bindPath function from libstore
bindPath/doBind is a useful function in build that is used in several
parts of LocalDerivationGoal. Moving this function makes it easier to
split LocalDerivationGoal implementation between several files.

Change-Id: Ic5a0768479c153c1aa3ed425f12604b20bbf0f42
2024-07-27 19:40:40 +00:00
jade 6abad7cb23 Merge "releng: Remove workaround for skopeo feature we didn't know about" into main 2024-07-26 11:09:17 +00:00
V. a98dce2a1f devendor pegtl
Change-Id: I609a58985fc5210806d0959049a48976ae079c30
2024-07-26 11:22:34 +04:00
V. 393794ad92 update flake.lock
Change-Id: I7afb53c929d297061dba6ec4a3ec7c6e3c6a553e
2024-07-26 11:22:34 +04:00
alois31 d945e89e19 Merge changes I45d3895f,I541be3ea,Ibe51416d into main
* changes:
  libstore/build: block io_uring
  libstore/build: use an allowlist approach to syscall filtering
  libstore/build: always treat seccomp setup failures as fatal
2024-07-26 07:08:35 +00:00
Artemis Tosini 60a48311e8 Merge "libutil: Support getSelfExe on FreeBSD" into main 2024-07-25 23:10:30 +00:00
jade c4c7cb7613 Merge changes Ic0dfcfe2,Ibe73851f,Ia7a8df1c,I400b2031 into main
* changes:
  package.nix: remove dead code
  diff-closures: remove gratuitous copy
  tree-wide: NULL -> nullptr
  libutil: rip out GNU Hurd support code
2024-07-25 18:05:41 +00:00
alois31 e7188e211a
libstore/build: block io_uring
Unfortunately, io_uring is totally opaque to seccomp, and while currently there
are no dangerous operations implemented, there is no guarantee that it remains
this way. This means that io_uring should be blocked entirely to ensure that
the sandbox is future-proof. This has not been observed to cause issues in
practice.

Change-Id: I45d3895f95abe1bc103a63969f444c334dbbf50d
2024-07-25 18:24:45 +02:00
alois31 127ee1a101
libstore/build: use an allowlist approach to syscall filtering
Previously, system call filtering (to prevent builders from storing files with
setuid/setgid permission bits or extended attributes) was performed using a
blocklist. While this looks simple at first, it actually carries significant
security and maintainability risks: after all, the kernel may add new syscalls
to achieve the same functionality one is trying to block, and it can even be
hard to actually add the syscall to the blocklist when building against a C
library that doesn't know about it yet. For a recent demonstration of this
happening in practice to Nix, see the introduction of fchmodat2 [0] [1].

The allowlist approach does not share the same drawback. While it does require
a rather large list of harmless syscalls to be maintained in the codebase,
failing to update this list (and roll out the update to all users) in time has
rather benign effects; at worst, very recent programs that already rely on new
syscalls will fail with an error the same way they would on a slightly older
kernel that doesn't support them yet. Most importantly, no unintended new ways
of performing dangerous operations will be silently allowed.

Another possible drawback is reduced system call performance due to the larger
filter created by the allowlist requiring more computation [2]. However, this
issue has not convincingly been demonstrated yet in practice, for example in
systemd or various browsers. To the contrary, it has been measured that the the
actual filter constructed here has approximately the same overhead as a very
simple filter blocking only one system call.

This commit tries to keep the behavior as close to unchanged as possible. The
system call list is in line with libseccomp 2.5.5 and glibc 2.39, which are the
latest versions at the point of writing. Since libseccomp 2.5.5 is already a
requirement and the distributions shipping this together with older versions of
glibc are mostly not a thing any more, this should not lead to more build
failures any more.

[0] https://github.com/NixOS/nixpkgs/issues/300635
[1] https://github.com/NixOS/nix/issues/10424
[2] https://github.com/flatpak/flatpak/pull/4462#issuecomment-1061690607

Change-Id: I541be3ea9b249bcceddfed6a5a13ac10b11e16ad
2024-07-25 18:24:40 +02:00
alois31 233408f677
libstore/build: always treat seccomp setup failures as fatal
In f047e4357b, I missed the behavior that if
building without a dedicated build user (i.e. in single-user setups), seccomp
setup failures are silently ignored. This was introduced without explanation 7
years ago (ff6becafa8). Hopefully the only
use-case nowadays is causing spurious test suite successes when messing up the
seccomp filter during development. Let's try removing it.

Change-Id: Ibe51416d9c7a6dd635c2282990224861adf1ceab
2024-07-25 18:21:26 +02:00
Qyriad 8d12e0fbb7 fix building with Musl, fixing static builds
Musl stdout macro expands¹ to something that isn't a valid identifier,
so we get syntax errors when compiling usage of a method called stdout
with Musl's stdio.h.

[1]: https://git.musl-libc.org/cgit/musl/tree/include/stdio.h?id=ab31e9d6a0fa7c5c408856c89df2dfb12c344039#n67

Change-Id: I10e6f6a49504399bf8edd59c5d9e4e62449469e8
2024-07-24 17:21:40 +00:00
Artemis Tosini 3b96b51cf4
libutil: Support getSelfExe on FreeBSD
getSelfExe is used in a few places re-execute nix.
Current code in this file uses ifdefs to support several
platforms, just keep doing that

Change-Id: Iecc2ada0101aea0c30524e3a1218594f919d74bf
2024-07-24 01:28:03 +00:00
jade 98e8cf9c63 package.nix: remove dead code
Change-Id: Ic0dfcfe27dbf13da4f7f74f5fab8ce6fa718d28f
2024-07-23 21:53:43 +02:00
jade 12a5838d11 diff-closures: remove gratuitous copy
This was done originally because std::smatch does not accept `const char
*` as iterators. However, this was because we should have been using
std::cmatch instead.

Change-Id: Ibe73851fd39755e883df2d33d22fed72ac0a04ae
2024-07-23 21:45:30 +02:00
jade eecc4ff1c0 releng: Remove workaround for skopeo feature we didn't know about
It turns out skopeo *does* support not saying the tag, but I couldn't
find it in the docs.

Asking the author in https://github.com/containers/skopeo/issues/2354
yielded that this can be requested as `@@unknown-digest@@`.

So now we have a perfectly cromulent docker upload chain, yay!!

Change-Id: I256f3cbeef4fe28b3d68d0dda57f02cdaee3996b
2024-07-23 12:11:15 -07:00
jade 2436f2110a tree-wide: NULL -> nullptr
This is slightly more type safe and is more in line with modern C++.

Change-Id: Ia7a8df1c7788085020d1bdc941d6f9cee356144e
2024-07-23 21:06:55 +02:00
jade 916b5c68fb libutil: rip out GNU Hurd support code
Nobody has stepped up to add further support for Hurd since this code
appeared in 2010 or 2014. We don't need it.

Change-Id: I400b2031a225551ea3c71a3ef3ea9fdb599dfba3
2024-07-23 20:52:04 +02:00
Artemis Tosini 53f3e39815
libstore: Add FreeBSD findPlatformRoots
Use libprocstat to find garbage collector roots on FreeBSD.
Tested working on a FreeBSD machine, although there is no CI yet

Change-Id: Id36bac8c3de6cc4de94e2d76e9663dd4b76068a9
2024-07-23 17:49:33 +00:00
Isabel d2422771eb
nix flake show: add the description if it exists
(cherry picked from commit 8cd1d02f90eb9915e640c5d370d919fad9833c65)

nix flake show: Only print up to the first new line if it exists.

(cherry picked from commit 5281a44927bdb51bfe6e5de12262d815c98f6fe7)

add tests

(cherry picked from commit 74ae0fbdc70a5079a527fe143c4832d1357011f7)

Handle long strings, embedded new lines and empty descriptions

(cherry picked from commit 2ca7b3afdbbd983173a17fa0a822cf7623601367)

Account for total length of 80

(cherry picked from commit 1cc808c18cbaaf26aaae42bb1d7f7223f25dd364)

docs: add nix flake show description release note

fix: remove white space

nix flake show: trim length based on terminal size

test: account for terminal size

docs(flake-description): before and after commands; add myself to credits

Upstream-PR: https://github.com/NixOS/nix/pull/10980
Change-Id: Ie1c667dc816b3dd81e65a1f5395e57ea48ee0362
2024-07-23 13:21:15 +01:00
Pierre Bourdon 73c013a5df Merge "libexpr/gc-alloc: fix compilation with !HAVE_BOEHMGC" into main 2024-07-22 23:14:59 +00:00
Pierre Bourdon e76245f8e9
libexpr/gc-alloc: fix compilation with !HAVE_BOEHMGC
Fixes: 72ee25b402
Change-Id: Ib59386af1415a8ed4b53af24ec22a4ffa5e5877d
2024-07-23 00:50:09 +02:00
eldritch horrors 472ff1b833 libstore: keep Goal errors as unique_ptrs
Error is pretty large, and most goals do not fail. this alone more than
halves the size of Goal on x86_64-linux, from 720 bytes down to 344. in
derived classes the difference is not as dramatic, but even the largest
derived class (`LocalDerivationGoal`) loses almost 20% of its footprint

Change-Id: Ifda8f94c81b6566eeb3e52d55d9796ec40c7bce8
2024-07-22 19:01:40 +00:00
eldritch horrors 7bf1aff44a libstore: remove an always-defaulted argument
Change-Id: I3c7f17d5492a16bb54480fa1aa384b96fba72d61
2024-07-22 19:01:40 +00:00
eldritch horrors 58a91d70c9 libstore: use std::async instead of Goal threads
the goals are either already using std::async and merely forgot to
remove std::thread vestiges or they emulate async with threads and
promises. we can simply use async directly everywhere for clarity.

Change-Id: I3f05098310a25984f10fff1e68c573329002b500
2024-07-22 19:01:40 +00:00
eldritch horrors ad36fb43ad libstore: remove addToWeakGoals
under owner_less it's equivalent to insert(), only sometimes a little
bit faster because it does not construct a weak_ptr if the goal is in
the set already. this small difference in performance does not matter
here and c++23 will make insert transparent anyway, so we can drop it

Change-Id: I7cbd7d6e0daa95d67145ec58183162f6c4743b15
2024-07-22 19:01:40 +00:00
eldritch horrors d70e045f90 libstore: remove Goal::ecBusy
this should be an optional. "busy" is not an *exit* code!

Change-Id: Ic231cb27b022312b1a7a7b9602f32845b7a9c934
2024-07-22 19:01:40 +00:00
eldritch horrors 20f53346df libstore: remove unused Worker::waitForAnyGoal
Change-Id: Ia3ebd434b17052b6760ce74d8e20025a72148613
2024-07-22 19:01:40 +00:00
V. 85e3b9b871 De-vendor nixfmt
Change-Id: I1a051be495318a507d07f6d0a6b157616e26774c
2024-07-22 21:09:58 +04:00
eldritch horrors c74eb81356 enable -Werror=suggest-override
*accidentally* overriding a function is almost guaranteed to be an
error. overriding a function without labeling it as such is merely
bad style, but bad style that makes the code harder to understand.

Change-Id: Ic0594f3d1604ab6b3c1a75cb5facc246effe45f0
2024-07-22 16:26:55 +00:00
eldritch horrors 0463cf2aef libexpr: fix -Wunused-const-variable warning
Change-Id: Ib986ece0ab2eff83e7abd7f1f915cd8f761827ad
2024-07-22 16:26:55 +00:00
alois31 2d4aca2546
libutil/logging: fix build without precompiled header
Commit 0109368c3f missed to include a required
header, which is not noticed when the precompiled header is enabled because
it's included in that. Also include it in the file so that the build without
precompiled header works too.

Change-Id: Id7a7979684b64f937f7f8191612952d73c113015
2024-07-21 12:54:31 +02:00
alois31 94a8e5fe0d Merge "libstore/binary-cache-store: use correct buffer size for NAR decompression" into main 2024-07-21 10:42:33 +00:00
jade 4fa6961aa2 Merge "gc: refactor the gc server thread out into a class without changing it" into main 2024-07-21 10:36:10 +00:00
alois31 391088900e
libstore/binary-cache-store: use correct buffer size for NAR decompression
Due to a leftover from a previous version where the buffer was allocated on the
stack, the change introduced in commit 4ec87742a1
accidentally passes the size of a pointer as the size of the buffer to the
decompressor. Since the former is much smaller (usually 8 bytes instead of 64
kilobytes), this is safe, but leads to considerable overhead; most notably, due
to excessive progress reports, which happen for each chunk. Pass the proper
buffer size instead.

Change-Id: If4bf472d33e21587acb5235a2d99e3cb10914633
2024-07-21 11:28:23 +02:00
Winter Cute 1917e6c765 Merge "Fix namespace warning being emitted if sandbox is disabled" into main 2024-07-20 22:14:33 +00:00
Qyriad 72ee25b402 libexpr: add a strongly typed version of gcAllocBytes()
This commit adds a new helper template function to gc-alloc.hh (which is
probably where you want to look at first, O great reviewer [custom file
ordering in review diffs when]), which uses a type argument to determine
the size to allocate, rather than making the caller use sizeof().

Change-Id: Ib5d138d91a28bdda304a80db24ea9fb08669ad22
2024-07-20 20:20:01 +00:00
Qyriad e67dac1d74 libexpr: rename confusing makeImmutableString -> gcCopyStringIfNeeded
The purpose of this function has little to do with immutability. Value's
strings are never mutated, and the point of this function is to
singleton empty strings.

Change-Id: Ifd41dd952409d54e4d3de9ab59064e6928b0e480
2024-07-20 20:20:01 +00:00
Qyriad a3361557e3 libexpr: refactor gc-agnostic helpers into one place
Change-Id: Icc4b367e4f670d47256f62a3a002cd248a5c2d3b
2024-07-20 20:20:01 +00:00
eldritch horrors 0109368c3f libutil: make basic loggers thread-safe
SimpleLogger is not fully thread-safe, and all loggers that wrap it are
also not safe accordingly. this does not affect much, but in rare cases
it can cause interleaving of messages on stderr when used with the json
or raw log formats. the fix applied here is a bit of a hack, but fixing
this properly requires rearchitecting the logger infrastructure. nested
loggers are not the most natural abstraction here, and it is biting us.

Change-Id: Ifbf34fe1e85c60e73b59faee50e7411c7b5e7c12
2024-07-20 12:33:49 +00:00
eldritch horrors d8c09b5836 libutil: remove warnOnce macro
it's only used once, and even that one use is highly questionable. more
instances of warnOnce should be much more principled than this has been

Change-Id: I5856570c99cb44462e700d753d0c706a5db03c4b
2024-07-20 12:33:49 +00:00
Winter Cute 3da41fdb82 Fix namespace warning being emitted if sandbox is disabled
If useChroot = false, and user namespaces aren't available for some
reason (e.g. within a Docker container), this fixes a pointless warning
being emitted, as we would never attempt to use them even if they were
available.

Change-Id: Ibcee91c088edd2cd19e70218d5a5802bff8f537b
2024-07-19 19:14:54 -04:00
jade 77ff799cc8 gc: refactor the gc server thread out into a class without changing it
This removes a *whole load* of variables from scope and enforces thread
boundaries with the type system.

There is not much change of significance in here, so the things to watch
out for while reviewing it are primarily that the destructor ordering
may have changed inadvertently, I think.

Change-Id: I3cd87e6d5a08dfcf368637407251db22a8906316
2024-07-19 20:55:55 +00:00
jade 22252825c4 Merge changes Id8b3d289,Ib75ab5b8,I3792eeb3 into main
* changes:
  Fixup a bunch of references to nixos.org manuals
  Add release notes for removing overflow from Nix language
  expr: fix a compiler warning about different signs in comparison
2024-07-19 18:52:46 +00:00
alois31 aba5f19680 Merge changes I829581a3,I0016970d,I5dac8e77,Ib7560fe5 into main
* changes:
  doc/release-notes: add for pretty printing improvements
  libexpr/print: do not show elided nested items when there are none
  libexpr/print: never show empty attrsets or derivations as «repeated»
  libexpr/print: pretty-print idempotently
2024-07-19 06:40:13 +00:00
jade 26e56780ca Fixup a bunch of references to nixos.org manuals
(plus one reference to CppNix github)

Change-Id: Id8b3d2897f3b54e286861805cfd421adc4d5de47
2024-07-18 19:27:33 +00:00
jade 10cc3b288d Add release notes for removing overflow from Nix language
Change-Id: Ib75ab5b8b4d879035d7ee7678f9cd0c491a39c0a
2024-07-18 19:27:33 +00:00