Commit graph

9578 commits

Author SHA1 Message Date
eldritch horrors ceccac835c libutil: remove callback.hh
it's no longer used. it really shouldn't have existed this long since it
was just a mashup of both std::promise and std::packaged_task in a shape
that makes composition unnecessarily difficult. all but a single case of
Callback pattern calls were fully synchronous anyway, and even this sole
outlier was by far not important enough to justify the extra complexity.

Change-Id: I208aec4572bf2501cdbd0f331f27d505fca3a62f
2024-05-10 02:21:11 +02:00
eldritch horrors b66451ae7f libstore: de-callback-ify FileTransfer
also add a few more tests for exception propagation behavior. using
packaged_tasks and futures (which only allow a single call to a few
of their methods) introduces error paths that weren't there before.

Change-Id: I42ca5236f156fefec17df972f6e9be45989cf805
2024-05-10 02:21:11 +02:00
eldritch horrors 28a98d152c libstore: de-callback-ify Store::queryRealisation
Change-Id: I8d74745c519518f163f51dfaa39063836f17599e
2024-05-09 23:18:05 +02:00
eldritch horrors 17965bf11c libstore: un-callback-ify Store::queryRealisationUncached
Change-Id: I4a328f46eaac3bb8b19ddc091306de83348be9cf
2024-05-09 23:18:05 +02:00
eldritch horrors 2f4a1dd6e0 libstore: de-callback-ify Store::queryPathInfoUncached
Change-Id: I23a156aaff5328f67ca16ccd85c0ea1711b21e35
2024-05-09 23:18:05 +02:00
eldritch horrors c77bd88259 libstore: de-callback-ify BinaryCacheStore::getFile
Change-Id: I36b3eb9f645aa04058151e7b2353e15e6f29057b
2024-05-09 23:18:05 +02:00
eldritch horrors 1a002d1a11 libstore: de-callback-ify CA realisation substitution
this is the *only* real user of file transfer download completion
callbacks, and a pretty spurious user at that (seeing how nothing
here is even turned on by default and indeed a dependency of path
substitution which *isn't* async, and concurrency-limited). it'll
be a real pain to keep this around, and realistically it would be
a lot better to overhaul substitution in general to be *actually*
async. that requires a proper async framework footing though, and
we don't have anything of the sort, but it's also blocking *that*

Change-Id: I1bf671f217c654a67377087607bf608728cbfc83
2024-05-09 23:18:05 +02:00
Qyriad 1f9b0fba23 add and fix -Wignored-qualifiers
Change-Id: I4bffa766ae04dd80355f9b8c10e59700e4b406da
2024-05-09 07:08:44 -06:00
Qyriad 2bbe3efd16 add and fix -Wdeprecated-copy
*so* many warnings, from only two definitions

Change-Id: If2561cd500c05a1e33cce984faf9f3e42a8a95ac
2024-05-09 13:02:45 +00:00
Qyriad 010bbd1b0e Merge "nix3-eval: don't elide top-level errors" into main 2024-05-09 11:46:13 +00:00
Qyriad 54322f09d3 nix3-eval: don't elide top-level errors
Fixes #276.

Change-Id: I83e71beb5c35d6f3b10a4186caa5e52a2f95b510
2024-05-08 13:37:20 -06:00
Théophane Hufschmitt adea821d87
libstore: Fix sandbox=relaxed
The fix for the Darwin vulnerability in ecdbc3b207
also broke setting `__sandboxProfile` when `sandbox=relaxed` or
`sandbox=false`. This cppnix change fixes `sandbox=relaxed` and
adds a suitable test.

Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
Change-Id: I40190f44f3e1d61846df1c7b89677c20a1488522
2024-05-08 19:31:43 +00:00
alois31 243c0f18da
repl: show a progress bar while performing builds
In commit 946fc12e4e, the progress bar in the
repl was disabled again because it was observed to erase incremental output
from attrset evaluations from the terminal. Let's try adding the progress bar
again, this time showing up only when a build is initiated, which does not have
incremental output that could be destroyed to begin with. While this does mean
that we won't have a progress bar for eval-time fetching or IFD, it's still
better than nothing.

Change-Id: If4eb1035cd0c876f5b4ff1e2434b9baf99f150ac
2024-05-08 20:54:41 +02:00
Qyriad b9be46fb31 remove the autoconf+Make buildsystem
We're not using it anymore. Any leftover bugs in the Meson buildsystem
are now just bugs.

Closes #249.

Change-Id: I0465a0c37ae819f94d40e7829f5bff046aa63d73
2024-05-07 17:04:30 -06:00
Nikodem Rabuliński e8a603fb2f Merge changes Icf26010a,Ib6161567 into main
* changes:
  Always initialize curl in parent process on darwin
  Fix failing darwin tests
2024-05-07 21:26:24 +00:00
Nikodem Rabuliński f894cce79b
Always initialize curl in parent process on darwin
Because of an objc quirk[1], calling curl_global_init for the first time
after fork() will always result in a crash.
Up until now the solution has been to set
OBJC_DISABLE_INITIALIZE_FORK_SAFETY for every nix process to ignore
that error.
This is less than ideal because we were setting it in package.nix,
which meant that running nix tests locally would fail because
that variable was not set.
Instead of working around that error we address it at the core -
by calling curl_global_init inside initLibStore, which should mean
curl will already have been initialized by the time we try to do so in
a forked process.

[1] 01edf1705f/runtime/objc-initialize.mm (L614-L636)

Change-Id: Icf26010a8be655127cc130efb9c77b603a6660d0
2024-05-07 20:43:17 +02:00
eldritch horrors 964ac8b0e8 libutil: de-callback-ify computeClosure
only two users of this function exist. only one used it in a way that
even bears resemblance to asynchronicity, and even that one didn't do
it right. fully async and parallel computation would have only worked
if any getEdgesAsync never calls the continuation it receives itself,
only from more derived callbacks running on other threads. calling it
directly would cause the decoupling promise to be awaited immediately
*on the original thread*, completely negating all nice async effects.

Change-Id: I0aa640950cf327533a32dee410105efdabb448df
2024-05-07 14:35:20 +00:00
eldritch horrors 230860dbb8 libstore: limit CA realisation info substitution concurrency
this seems to be an oversight, considering that regular substitutions
are concurrency-limited. while not particularly necessary at present,
once we've removed the `Callback` based interfaces it will be needed.

Change-Id: Ide2d08169fcc24752cbd07a1d33fb8482f7034f5
2024-05-07 14:35:20 +00:00
eldritch horrors 29f93e1e0d libutil: throw EndOfFile at sourceToSink end
... how has this never broken anything before

Change-Id: If3789c02028e8f929481514f63d76b0b46bfc182
2024-05-07 14:35:20 +00:00
Qyriad 005b2b61e6 fix fallback chroot store creation after b247ef72d
When /nix/var (or, more precisely, NIX_STATE_DIR) does not exist at all,
Lix falls back to creating an adhoc chroot store in XDG_DATA_HOME.

b247ef72d[1] changed the way Store classes are initialized, and in the
migration, a `params2` was accidentally changed to `params`. This commit
restores the correct behavior, and in lieu of a single *character* fix,
this commit also changes the variable name to something more reasonable.

Fixes #274.

[1]: b247ef72dc

n.b., this code might deserve some more looking at anyway. this fallback
store creation throws away *all* Store params passed to
openFromNonUri() in favor of an entirely new set which only contains
the `root` param, which may or may not be the correct behavior

Change-Id: Ibea559b88a50e6d6e75a1f87d9d7816cabb2a8f3
2024-05-06 20:54:21 -06:00
eldritch horrors f75d0752ce filetransfer: correctly abort empty transfers
returning 0 from the callback for errors signals successful transfer if
the source returned no data even though the exception we've just caught
clearly disagrees. while this is not all that important (since the only
viable cause of such errors will be dataCallback, and the sole instance
of it being used already takes care of exceptions) we can just do this.

Change-Id: I2bb150eff447121d82e8e3aa4e00057c40523ac6
2024-05-06 21:18:23 +02:00
eldritch horrors 121edecf65 filetransfer: extract decompressor creation
this will be necessary if we want download() to return a source instead
of consuming a sink, which will in turn be needed to remove coroutines.

Change-Id: I34ec241e9bbc5d32fbcd243b244e29c3757533aa
2024-05-06 20:19:03 +02:00
jade 748d8310fa Fix the pages in the manual for Lix
This doesn't comprehensively fix everything outdated in the manual, or
make the manual greatly better, but it does note down where at least
jade noticed it was wrong, and it does fix all the instances of
referencing Nix to conform to the style guide to the best of our
ability.

A lot of things have been commented out for being wrong, and there are
three types of FIXME introduced:

- FIXME(Lix): generically Lix needs to fix it
- FIXME(Qyriad): re #215
- FIXME(meson): docs got outdated by meson changes and need rewriting

I did fix a bunch of it that I could, but there could certainly be
mistakes and this is definitely just an incremental improvement.

Fixes: #266
Change-Id: I5993c4603d7f026a887089fce77db08394362135
2024-05-05 16:11:01 -07:00
Qyriad 99903f24eb fix integer overflow on i686 with high phys memory
sizeof(long) is 4 bytes on i686 GCC.
With ~32 GiB of memory and a page size of 4096, there are 7988420 pages.
(7988420 * 4096) is bigger than INT32_MAX folks.

This has gone unnoticed for 9 years, and only came up thanks to
94ea517db[1] adding integer overflow sensitization checks, which caused
this broken code to emit an illegal instruction, crashing Lix the
instant the buildsystem ran Lix to generate the docs files.

[1]: 94ea517dbe729765b69638190f4bea3f6a632b40

Change-Id: I50bb9ea072aac11b449d79e5d55525887a6e5a99
2024-05-05 19:47:46 +00:00
Qyriad 4998699e1a Merge "point nix3-upgrade-nix to releases.lix.systems/manifest.nix" into main 2024-05-05 19:47:36 +00:00
jade 4fd32351f3 Merge "Warn on untrusted client settings being ignored" into main 2024-05-05 19:35:35 +00:00
eldritch horrors 6b08138929 filetransfer: abort transfer on receiver exception
not doing this will cause transfers that had their readers disappear to
linger. with lingering transfers the curl thread can't shut down, which
will cause nix itself to not shut down until the transfer finishes some
other way (most likely network timeouts). also add a new test for this.

Change-Id: Id2401b3ac85731c824db05918d4079125be25b57
2024-05-05 18:09:31 +00:00
Qyriad 7ab076f21c point nix3-upgrade-nix to releases.lix.systems/manifest.nix
This file is currently manually managed, but will be automated along
with the rest of the release process.

Change-Id: I77839919549aaac73de582b2e563ce3ef914a8cb
2024-05-05 17:45:50 +00:00
jade 2e8f9ac944 Warn on untrusted client settings being ignored
These are such a footgun and trip people up a lot. Let's make Lix louder
about this.

Related: #261
Change-Id: I6a8d57c9817caaa6b0cbf886c615dda51038f628
2024-05-04 21:30:26 -07:00
jade 47fb494676 Merge "Actually try making a userns before assuming they don't work" into main 2024-05-05 03:58:44 +00:00
jade fb5d6f325b Merge "Fix /etc/group having desynced IDs from the actual UID in the sandbox" into main 2024-05-05 03:57:29 +00:00
jade e3b702fa22 Actually try making a userns before assuming they don't work
If unprivileged userns are *believed* to be disabled (such as with
"kernel.unprivileged_userns_clone = 0"), Lix would previously *give up*
on trying to use a user namespace before actually trying it, even if, in
cases such as unprivileged_userns_clone, it would actually be allowed
since Nix has CAP_SYS_ADMIN when running as daemon.

(see, e.g. 25d4709a4f)

We changed it to actually try it first, and then diagnose possible
causes, and also to be more loud about the whole thing, using warnings
instead of debugs. These warnings will only print on the first build run
by the daemon, which is, tbh, eh, shrug.

This is what led to us realizing that no-userns was a poorly exercised
condition.

Change-Id: I8e4f21afc89c574020dc7e89a560cc740ce6573a
2024-05-05 00:37:24 +00:00
jade 9909a175bf Fix /etc/group having desynced IDs from the actual UID in the sandbox
This was found when `logrotate.conf` failed to build in a NixOS system
with:

    /nix/store/26zdl4pyw5qazppj8if5lm8bjzxlc07l-coreutils-9.3/bin/id: cannot find name for group ID 30000

This was surprising because it seemed to mean that /etc/group was busted
in the sandbox. Indeed it was:

    root0:
    nixbld:!💯
    nogroup65534:

We diagnosed this to sandboxUid() being called before
usingUserNamespace() was called, in setting up /etc/group inside the
sandbox. This code desperately needs refactoring.

We also moved the /etc/group code to be with the /etc/passwd code, but
honestly this code is all spaghetti'd all over the place and needs some
more serious tidying than we did here.

We also moved some checks to be earlier to improve locality with where
the things they are checking come from.

Change-Id: Ie29798771f3593c46ec313a32960fa955054aceb
2024-05-04 17:36:50 -07:00
jade 4886d4592b Remove a URL literal from fetchTarball docs
Change-Id: I254b793b42f77ffe9f357f3b376683e5758f23b5
2024-05-04 16:55:27 -07:00
Maximilian Bosch 799d0132f0 Merge "Revert "Revert "Merge pull request #6621 from Kha/nested-follows""" into main 2024-05-04 08:52:29 +00:00
Maximilian Bosch 79d0ae6670 Merge "libstore/local-derivation-goal: prohibit creating setuid/setgid binaries" into main 2024-05-04 07:26:15 +00:00
Maximilian Bosch 0e38720502 Revert "Revert "Merge pull request #6621 from Kha/nested-follows""
This reverts commit a8b3d777fb.

This undoes the revert of PR#6621, which allows nested `follows`, i.e.

    {
      inputs = {
        foo.url = "github:bar/foo";
        foo.inputs.bar.inputs.nixpkgs = "nixpkgs";
      };
    }

does the expected thing now. This is useful to avoid the 1000 instances
of nixpkgs problem without having each flake in the dependency tree to
expose all of its transitive dependencies for modification.

This was in fact part of Nix before and the C++ changes applied w/o
conflicts. However, it got reverted then because people didn't want to
merge lazy-trees against it which was supposed to be merged soon back in
October 2022.

Fixes: #201

Change-Id: I5ddef914135b695717b2ef88862d57ced5e7aa3c
2024-05-03 22:54:38 +00:00
Maximilian Bosch f8617f9dc6 Merge "Rename nix show-config to nix config show" into main 2024-05-03 22:07:33 +00:00
Maximilian Bosch 045ee37438 libstore/local-derivation-goal: prohibit creating setuid/setgid binaries
With Linux kernel >=6.6 & glibc 2.39 a `fchmodat2(2)` is available that
isn't filtered away by the libseccomp sandbox.

Being able to use this to bypass that restriction has surprising results
for some builds such as lxc[1]:

> With kernel ≥6.6 and glibc 2.39, lxc's install phase uses fchmodat2,
> which slips through 9b88e52846/src/libstore/build/local-derivation-goal.cc (L1650-L1663).
> The fixupPhase then uses fchmodat, which fails.
> With older kernel or glibc, setting the suid bit fails in the
> install phase, which is not treated as fatal, and then the
> fixup phase does not try to set it again.

Please note that there are still ways to bypass this sandbox[2] and this is
mostly a fix for the breaking builds.

This change works by creating a syscall filter for the `fchmodat2`
syscall (number 452 on most systems). The problem is that glibc 2.39
is needed to have the correct syscall number available via
`__NR_fchmodat2` / `__SNR_fchmodat2`, but this flake is still on
nixpkgs 23.11. To have this change everywhere and not dependent on the
glibc this package is built against, I added a header
"fchmodat2-compat.hh" that sets the syscall number based on the
architecture. On most platforms its 452 according to glibc with a few
exceptions:

    $ rg --pcre2 'define __NR_fchmodat2 (?!452)'
    sysdeps/unix/sysv/linux/x86_64/x32/arch-syscall.h
    58:#define __NR_fchmodat2 1073742276

    sysdeps/unix/sysv/linux/mips/mips64/n32/arch-syscall.h
    67:#define __NR_fchmodat2 6452

    sysdeps/unix/sysv/linux/mips/mips64/n64/arch-syscall.h
    62:#define __NR_fchmodat2 5452

    sysdeps/unix/sysv/linux/mips/mips32/arch-syscall.h
    70:#define __NR_fchmodat2 4452

    sysdeps/unix/sysv/linux/alpha/arch-syscall.h
    59:#define __NR_fchmodat2 562

I added a small regression-test to the setuid integration-test that
attempts to set the suid bit on a file using the fchmodat2 syscall.
I confirmed that the test fails without the change in
local-derivation-goal.

Additionally, we require libseccomp 2.5.5 or greater now: as it turns
out, libseccomp maintains an internal syscall table and
validates each rule against it. This means that when using libseccomp
2.5.4 or older, one may pass `452` as syscall number against it, but
since it doesn't exist in the internal structure, `libseccomp` will refuse
to create a filter for that. This happens with nixpkgs-23.11, i.e. on
stable NixOS and when building Lix against the project's flake.

To work around that

* a backport of libseccomp 2.5.5 on upstream nixpkgs has been
  scheduled[3].

* the package now uses libseccomp 2.5.5 on its own already. This is to
  provide a quick fix since the correct fix for 23.11 is still a staging cycle
  away.

We still need the compat header though since `SCMP_SYS(fchmodat2)`
internally transforms this into `__SNR_fchmodat2` which points to
`__NR_fchmodat2` from glibc 2.39, so it wouldn't build on glibc 2.38.
The updated syscall table from libseccomp 2.5.5 is NOT used for that
step, but used later, so we need both, our compat header and their
syscall table 🤷

Relevant PRs in CppNix:

* https://github.com/NixOS/nix/pull/10591
* https://github.com/NixOS/nix/pull/10501

[1] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2031073804
[2] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2030844251
[3] https://github.com/NixOS/nixpkgs/pull/306070

(cherry picked from commit ba6804518772e6afb403dd55478365d4b863c854)
Change-Id: I6921ab5a363188c6bff617750d00bb517276b7fe
2024-05-03 16:29:06 +02:00
Théophane Hufschmitt 8458d98b27 Rename nix show-config to nix config show
Part of #7672

My main motivation is to be able to use `nix.checkConfig`[1]. This
doesn't work with Lix currently since the module uses `nix show-config`
if the Nix version is <2.20pre and `nix config show` otherwise. I think
this is the only instance where nixpkgs checks for which Nix commands
exist that affects us now, so I figured we could just perform the rename
here as well[2] and still provide the current version number[3].

I don't have a strong opinion on whether to deprecate `nix show-config`,
the warning is added there automatically.

(cherry picked from commit f300e11b056dea414d7d77bbc6e5a7dc5d9ddd41)

[1] https://nixos.org/manual/nixos/stable/options.html#opt-nix.checkConfig
[2] I should add that I don't use the "official" ways of installing Lix
    because using the flake directly and callPackaging it seemed to fit
    better into my workflow: I already have a little mess to make
    sure Hydra from the flake uses the correct pkgs.nix and I didn't
    want to complicate it further while keeping a single package-set I
    can build in CI. Don't get me wrong, I think such a module for a
    quick-start is very important, just giving context on why I bother
    in the first place :)
[3] When we go public, I think it's worth considering to add support in
    nixpkgs itself for Lix.

Change-Id: I47b4239b05cbeda3c370d2fa56ea768b768768ac
2024-05-03 16:26:16 +02:00
Qyriad 19645a4a64 Merge changes Id1a67156,I03f4c7c1,I146736bb,I3b1453cb into main
* changes:
  docs: clarify how ^ works for -E/-f installables
  docs: give translation examples from nix-build -E/-A to installables
  docs: clarify how the different kinds of installables are selected
  docs: guide to installables docs in installable commands' docs
2024-05-03 13:39:49 +00:00
Artemis Tosini 6f0636a7ed Merge "libstore: check additionalSandboxProfile" into main 2024-05-03 03:35:02 +00:00
Artemis Tosini ecdbc3b207 libstore: check additionalSandboxProfile
Currently LocalDerivationGoal allows setting `__sandboxProfile`
to add sandbox parameters on Darwin when `sandbox=true`.
This was only supposed to have an effect when `sandbox=relaxed`

Change-Id: Ide44ee82d7e4d6b545285eab26547e7014817d3f
2024-05-03 00:59:18 +00:00
eldritch horrors d55b158e24 libutil: make rewriteStrings sound
this is used in CA rewriting, replacement of placeholders in
derivations, generating scripts for devShells, and some more
places. in all of these transitive replacements are unsound,
and overlapping replacements would be as well. there even is
a test that transitive replacements do not happen (in the CA
RewriteSink suite), but none for overlapping replacements. a
minimally surprising binary rewriter surely would not do any
of these replacements, the only reason we have not seen this
break yet is probably that rewriteStrings is only called for
store paths and things that look like store paths (and those
should never overlap nor admit such transitive replacements)

Change-Id: I6fc29f939d5061d9f56c752624a823ece8437c07
2024-05-03 00:50:31 +00:00
Qyriad 076dfd30c6 Merge changes from topic "profile-v3" into main
* changes:
  nix3-profile: remove check "name" attr in manifests
  Add profile migration test
  nix3-profile: make element names stable
  getNameFromURL(): Support uppercase characters in attribute names
  nix3-profile: remove indices
  nix3-profile: allow using human-readable names to select packages
  implement parsing human-readable names from URLs
2024-05-02 20:15:48 +00:00
Qyriad 6a8b379628 nix3-profile: remove check "name" attr in manifests
It doesn't seem to have ever been used.

Based off of commit a748e88bf4cca0fdc6ce75188e88017a7899d16b

Upstream-PR: https://github.com/NixOS/nix/pull/9656
Change-Id: Idcf250a645fa43f2ef11fb15b503b070a62a917e
2024-05-02 12:59:15 -06:00
Qyriad e0911eef73 nix3-profile: make element names stable
Based off of commit 6268a45b650f563bae2360e0540920a2959bdd40

Upstream-PR: https://github.com/NixOS/nix/pull/9656
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Change-Id: I0fcf069a8537c61ad6fc4eee1f3c193a708ea1c4
2024-05-02 12:59:15 -06:00
Eelco Dolstra ce70f02aff getNameFromURL(): Support uppercase characters in attribute names
In particular, this makes it handle 'legacyPackages' correctly.

(cherry picked from commit 936a3642264ac159f3f9093710be3465b70e0e89)

Upstream-PR: https://github.com/NixOS/nix/pull/9657
Change-Id: Icc4efe02f7f8e90a2970589f72fd3d3cd4418d95
2024-05-02 12:02:28 -06:00
Qyriad e98fc952a8 nix3-profile: remove indices
Based off of commit 3187bc9ac3dd193b9329ef68c73ac3cca794ed78

Upstream-PR: https://github.com/NixOS/nix/pull/9656
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Change-Id: I8ac4a33314cd1cf9de95404c20f58e883460acc7
2024-05-02 12:02:28 -06:00
Qyriad f88423813f nix3-profile: allow using human-readable names to select packages
These names are parsed from the URL provided for that package

Based off of commit 257b768436a0e8ab7887f9b790c5b92a7fe51ef5

Upstream-PR: https://github.com/NixOS/nix/pull/8678
Co-authored-by: Felix Uhl <felix.uhl@outlook.com>
Change-Id: I76d5f9cfb11d3d2915b3dd1db21d7bb49e91f4fb
2024-05-02 12:02:28 -06:00
Robert Hensing b7ce11c97d Disallow store path names that are . or .. (plus opt. -)
As discussed in the maintainer meeting on 2024-01-29.

Mainly this is to avoid a situation where the name is parsed and
treated as a file name, mostly to protect users.
.-* and ..-* are also considered invalid because they might strip
on that separator to remove versions. Doesn't really work, but that's
what we decided, and I won't argue with it, because .-* probably
doesn't seem to have a real world application anyway.
We do still permit a 1-character name that's just "-", which still
poses a similar risk in such a situation. We can't start disallowing
trailing -, because a non-zero number of users will need it and we've
seen how annoying and painful such a change is.

What matters most is preventing a situation where . or .. can be
injected, and to just get this done.

(cherry picked from commit f1b4663805a9dbcb1ace64ec110092d17c9155e0)
Change-Id: I900a8509933cee662f888c3c76fa8986b0058839
2024-05-02 19:34:38 +02:00
Robert Hensing 3a058dc4b3 Revert "StorePath: reject names starting with '.'"
This reverts commit 24bda0c7b381e1a017023c6f7cb9661fae8560bd.

(cherry picked from commit 9ddd0f2af8fd95e1380027a70d0aa650ea2fd5e4)
Change-Id: Ideb547e2a8ac911cf39d58d3e0c1553867bdd776
2024-05-02 18:53:51 +02:00
Qyriad 1425aa0b7c implement parsing human-readable names from URLs
Based off of commit 257b768436a0e8ab7887f9b790c5b92a7fe51ef5

Upstream-PR: https://github.com/NixOS/nix/pull/8678
Co-authored-by: Felix Uhl <felix.uhl@outlook.com>
Change-Id: Idcb7f6191ca3310ef9dc854197f7798260c3f71d
2024-04-30 18:11:14 -06:00
Qyriad 4942e4e2d2 docs: clarify how ^ works for -E/-f installables
We didn't even realize you *could* use this syntax with -E and -f, much
less that the attribute path could be *empty*.

Change-Id: Id1a6715609f3a76a5ce477bd43a7832effbbe07b
2024-04-29 08:53:43 -06:00
Qyriad 6abeea70e9 docs: give translation examples from nix-build -E/-A to installables
Change-Id: I03f4c7c1049063539a35ba500a07bb8f866d4cb7
2024-04-29 08:18:00 -06:00
Qyriad bd2619868c docs: clarify how the different kinds of installables are selected
Change-Id: I146736bb97ebe035e04be69ce9fb60a557e38c6c
2024-04-29 08:18:00 -06:00
Qyriad 194ba8a02f docs: guide to installables docs in installable commands' docs
The installables syntax is not documented in any of the man pages or
docbook pages for any of those individual commands. And while these
commands really should at least peripherally individually document how
installables work, in the meantime we can at least direct people to the
right place.

This commit also clarifies the unexpected fact that `nix profile remove`
and `nix profile upgrade` do *not* take installables.

Change-Id: I3b1453cb197a613bbab639c66a466365c3592c6d
2024-04-29 08:06:05 -06:00
Qyriad ee5a1b5a4c nix3-upgrade-nix: allow manually specifying new nix
This allows manually specifying a store path for the new Nix that
gets linked into Nix's profile.

Change-Id: Ib71711ffb466febf4a6892e3fdbda644e053770d
2024-04-29 01:19:21 +00:00
Qyriad aae12f5848 fix nix upgrade-nix on new-style profiles
nix3-profile automatically migrates any profile its used on to its style
of profile -- the ones with manifest.json instead of manifest.nix. On
non-NixOS systems, Nix is conventionally installed to the profile at
/nix/var/nix/profiles/default, so if a user passed that to `--profile`
of `nix profile`, then it would break upgrade-nix from ever working
again, without recreating the profile.

This commit fixes that, and allows upgrade-nix to work on either kind of
profile.

Fixes #16.

Change-Id: I4c49b1beba93bb50e8f8a107edc451affe08c3f7
2024-04-29 01:19:21 +00:00
Qyriad 2bd57d4d36 refactor some nix-env and profile code to libcmd
Notably, ProfileManifest and ProfileElement are useful generic
profile management code, and nix profile is not the only place in the
codebase where profiles are relevant.

This commit is in preparation for fixing upgrade-nix's interaction with
new-style profiles.

Change-Id: Iefc8bbd34b4bc6012175cb3d6e6a8207973bc792
2024-04-29 01:19:21 +00:00
puck 0c831765bd Run all derivation builders inside the sandbox on macOS
This replaces the external sandbox-exec call with direct calls into
libsandbox. This API is technically deprecated and is missing some
prototypes, but all major browsers depend on it, so it is unlikely to
materially change without warning.

This commit also ensures the netrc file is only written if the
derivation is in fact meant to be able to access the internet.

This change commits a sin of not actually actively declaring its
dependency on macOS's libsandbox.dylib; this is due to the dylib
cache in macOS making that explicit dependency unnecessary. In the
future this might become a problem, so this commit marks our sins.

Co-authored-by: Artemis Tosini <lix@artem.ist>
Co-authored-by: Lunaphied <lunaphied@lunaphied.me>
Change-Id: Ia302141a53ce7b0327c1aad86a117b6645fe1189
2024-04-27 14:44:15 -06:00
puck 9229e87347 Fix progress bar on copyPaths
This variable should not be shared between activities.

Change-Id: I4eee89bc7acb320a3972dc3a55bfb087d3a9eb3a
2024-04-27 18:03:15 +00:00
Ilya K 9462c01c3e libstore/ssh: shut
This is just logspam, and we have NIX_SSHOPTS for people that want the logspam.

Change-Id: Ieff71473686f0661f9c53c212f8952dd2c9565c3
2024-04-27 12:05:17 +03:00
Maximilian Bosch 8773439a85 Merge "ssh-ng: Set log-fd for ssh to 4 by default" into main 2024-04-26 18:30:33 +00:00
Artemis Tosini 789aa39576 Merge "gc: Find roots using libproc on Darwin" into main 2024-04-26 17:26:45 +00:00
Maximilian Bosch 104448e75d ssh-ng: Set log-fd for ssh to 4 by default
That's expected by `build-remote` and makes sure that errors are
correctly forwarded to the user. For instance, let's say that the
host-key of `example.org` is unknown and

    nix-build ../nixpkgs -A hello -j0 --builders 'ssh-ng://example.org'

is issued, then you get the following output:

    cannot build on 'ssh-ng://example.org?&': error: failed to start SSH connection to 'example.org'
    Failed to find a machine for remote build!
    derivation: yh46gakxq3kchrbihwxvpn5bmadcw90b-hello-2.12.1.drv
    required (system, features): (x86_64-linux, [])
    2 available machines:
    [...]

The relevant information (`Host key verification failed`) ends up in the
daemon's log, but that's not very obvious considering that the daemon
isn't very chatty normally.

This can be fixed - the same way as its done for legacy-ssh - by passing
fd 4 to the SSH wrapper. Now you'd get the following error:

    cannot build on 'ssh-ng://example.org': error: failed to start SSH connection to 'example.org': Host key verification failed.
    Failed to find a machine for remote build!
    [...]

...and now it's clear what's wrong.

Please note that this is won't end up in the derivation's log.

For previous discussion about this change see
https://github.com/NixOS/nix/pull/7659.

Change-Id: I5790856dbf58e53ea3e63238b015ea06c347cf92
2024-04-26 19:04:06 +02:00
eldritch horrors a1ad4e52a6 filetransfer: don't decompress in curl wrapper itself
only decompress the response once all data has been received (in the
fully buffered case), or at least outside of the curl wrapper itself
(in the receive-to-sink case). unfortunately this means we will have
to duplicate decompression logic for these two cases for time being,
but once the curl wrapper has been rewritten to return a real future
or Source we can deduplicate this logic again. the curl wrapper will
have to turn into a proper Source first and use decompression source
logic which also does not currently exist—only decompression *sinks*

Change-Id: I66bc692f07d9b9e69fe10689ee73a2de8d65e35c
2024-04-26 15:26:37 +00:00
eldritch horrors fb0996aaa8 filetransfer: remove dataCallback from interface
this is highly questionable. single-arg download calls will misbehave
with it set, and two-arg download calls will just overwrite it. being
an implementation detail this should not have been in the API at all.

Change-Id: I613772951ee03d8302366085f06a53601d13f132
2024-04-26 15:26:37 +00:00
eldritch horrors dfe3baea12 filetransfer: make two-arg download abstract
this lets each implementation of FileTransfer (of which currently only
the one exists at all) implement appropriate handling for its internal
behaviours that are not otherwise exposed. in curl this lets us switch
the buffer-full handling method from "block the entire curl thread" to
"pause just the one transfer", move the non-libcurl body decompression
out of the actual curl wrapper (which will let us eventually morph the
curl wrapper intto an actual source of Sources), and some other things

Change-Id: Id6d3593cde6b4915aab3e90a43b175c103cc3f18
2024-04-26 15:26:37 +00:00
Artemis Tosini c03de0df62 gc: Find roots using libproc on Darwin
Previously, the garbage collector found runtime roots on Darwin by
shelling out to `lsof -n -w -F n` then parsing the result.
However, this requires an lsof binary and can be extremely slow.

The official Apple lsof returns in a reasonable amount of time,
about 250ms in my tests, but the lsof packaged in nixpkgs is quite slow,
taking about 40 seconds to run the command.

Using libproc directly is about the same speed as Apple lsof,
and allows us to reënable several tests that were disabled on Darwin.

Change-Id: Ifa0adda7984e13c15535693baba835aae79a3577
2024-04-25 23:24:21 -04:00
eldritch horrors 5420b3afd6 filetransfer: drop errorSink
just accumulate error data into result.data as we would for successful
transfers without a dataCallback. errorSink and data would contain the
same data in error cases anyway, so splitting them is not very useful.

Change-Id: I00e449866454389ac6a564ab411c903fd357dabf
2024-04-25 01:33:22 +02:00
eldritch horrors 5e69f8aa3d filetransfer: restore http status line reporting
this was broken in 75b62e5260.

Change-Id: If8583e802afbcde822623036bf41a9708fbc7c8d
2024-04-25 01:33:08 +02:00
eldritch horrors 38442e3123 filetransfer: remove decompress request parameter
this is never read.

Change-Id: I4c46f140519843a21e452958900e81edd2f78be2
2024-04-25 01:33:08 +02:00
Artemis Tosini 7114b0465a Merge "libstore: Create platform LocalStore subclasses" into main 2024-04-24 15:35:32 +00:00
Artemis Tosini b247ef72dc libstore: Create platform LocalStore subclasses
This creates new subclasses of LocalStore for each OS to include
platform-specific functionality. Currently this just includes garbage
collector roots but it could be extended to sandboxing as well.

In order to make sure that the generic LocalStore is not accidentally
constructed, its constructor is protected. A Fallback is provided which
implements no functionality except constructors.

Change-Id: I836a28e90b68309873f75afb83e0f1b2e2c89fb3
2024-04-23 16:17:05 +00:00
Qyriad be4a3168c9 Merge changes Ia3e7b1e6,If09be814 into main
* changes:
  meson: flip the switch!!
  meson: fix cross compilation
2024-04-23 11:12:09 +00:00
Qyriad 05e3b1d39e meson: fix cross compilation
This should fix cross compilation in the base case, but this is
difficult to test as cross compilation is broken in many different
places right now. This should bring Meson back up to cross parity with
the Make buildsystem though.

Change-Id: If09be8142d1fc975a82b994143ff35be1297dad8
2024-04-22 21:41:58 -06:00
eldritch horrors 86bfede948 libstore: use curl functions for reading headers
don't reimplement header parsing. this was only really needed due to the
ancient github bug we no longer care about, everything else we have done
in custom code can also be done using curl itself. doing this also fixes
possible sources of header smuggling (because the header function didn't
unfold headers and we'd trim them before parsing, which would've made us
read contents of one header as a fully formed header in itself). this is
a slight behavior change because we now honor only the first instance of
a given header where previous behavior was to honor either the last or a
combination of all of them (accept-ranges was logical-or'd by accident).

Change-Id: I93cb93ddb91ab98c8991f846014926f6ef039fdb
2024-04-23 01:04:56 +00:00
eldritch horrors 257d7ffa7b libstore: remove github etag workaround
this was a workaround for a *github* bug that happend *in 2015*.
not only is github no longer buggy, it shouldn't have been nix's
responsibility to work around these bugs like this to begin with

while we're at it we'll also remove another workaround—again for
github specifically and again for etag handling—from 2021 that's
also not needed any more. future workarounds for serverside bugs
should probably come with an expiration date that mutates into a
build warning after a while, otherwise this *will* happen again.

Change-Id: I74f739ae3e36d40350f78bebcb5869aa8cc9adcd
2024-04-23 01:04:56 +00:00
eldritch horrors ff9a4fc336 libstore: use curl_multi_{poll,wakeup}
the previous solution to the wakeup problem (adding a pipe and passing
it as an additional fd to curl_multi_wait) worked, but there have been
builtin alternatives for this since 2020. not only do these save code,
they're also a lot more likely to work natively on windows when needed

Change-Id: Iab751b900997110a8d15de45ea3ab0c42f7e5973
2024-04-22 21:37:20 +00:00
eldritch horrors e5903aab65 libstore: remove ancient libcurl feature checks
the oldest version checked for here is 7.47, which was released in
2016. it's probably safe to say that we do not need these any more

Change-Id: I003411f6b2ce6d56f7ca337390df3ea86bd59a99
2024-04-22 19:45:22 +00:00
puck c8c838381d Merge "Fix exportReferencesGraph when given store subpath" into main 2024-04-21 15:37:59 +00:00
puck 272c2ff15f remove extraneous cache entry from github fetcher
This isn't necessary, as it's already covered by the tarball fetcher's
cache.

Change-Id: I85e35f5a61594f27b8f30d82145f92c5d6559e1f
2024-04-21 10:46:05 +00:00
Alyssa Ross c1319831fb Fix exportReferencesGraph when given store subpath
With Nix 2.3, it was possible to pass a subpath of a store path to
exportReferencesGraph:

	with import <nixpkgs> {};

	let
	  hello = writeShellScriptBin "hello" ''
	    echo ${toString builtins.currentTime}
	  '';
	in

	writeClosure [ "${hello}/bin/hello" ]

This regressed with Nix 2.4, with a very confusing error message, that
presumably indicates it was unintentional:

	error: path '/nix/store/3gl7kgjr4pwf03f0x70dgx9ln3bhl7zc-hello/bin/hello' is not in the Nix store

(cherry picked from commit 0774e8ba33c060f56bad3ff696796028249e915a)
Change-Id: I00920fb33077b831a1bb4a1b68d515ba8c3c2a69
2024-04-21 10:27:32 +00:00
Qyriad 111db8b38f meson: correctly embed sandbox shell when asked
Change-Id: I2f6c0d42245204a516d2e424eea26a6391e975ad
2024-04-18 16:15:58 -06:00
Qyriad 077f45ee38 meson: correctly set -DSANDBOX_SHELL if we have it
The statically embedded busybox is not required for Lix to work, but
package.nix explicitly sets this, which was accidentally being ignored.

Change-Id: Ieeff830ac7d1f5fabe84d1a6cfd82f13d79035bf
2024-04-18 10:45:27 -06:00
Ilya K 6d79aa3d70 Merge "libstore/build: set NO_NEW_PRIVS for the sandbox" into main 2024-04-16 05:33:41 +00:00
Qyriad a41abb4594 fix probable format bug in DerivationGoal::buildDone
Either the contents of `line` could cause format errors, or this usage
is Technically safe. However, I trust nothing, especially with
boost::format.

Change-Id: I07933b20bde3b305a6e5d61c2a7bab6ecb042ad9
2024-04-15 23:09:40 +00:00
Qyriad 4e68deef80 abort with a descriptive message on bad HintFmt usage
Change-Id: Ic2f05572042343a8160fd971394372f5f2706fc4
2024-04-15 23:09:16 +00:00
Ilya K effc28f6f5 libstore/build: set NO_NEW_PRIVS for the sandbox
Change-Id: I711f64e2b68495ed9c85c1a4bd5025405805e43a
2024-04-15 10:25:29 +03:00
Qyriad 80bbfe2034 don't throw an exception for the trivial case of isStorePath()...
Previously if isStorePath() was called on anything other than a
top-level /nix/store/some-path, it would throw a BadStorePath exception.
This commit duplicates the absolutely trivial check, into
maybeParseStorePath(), and leaves exception throwing to
parseStorePath(), the function that assumes you're already giving a
valid path instead of the one whose purpose is to check if its valid or
not...

Change-Id: I8dda548f0f88d14ca8c3ee927d64e0ec0681fc7b
2024-04-14 21:08:07 +00:00
Qyriad ddb4d3fa4c Merge "don't boost::to_few_args when an eval cached string type errors" into main 2024-04-14 21:07:47 +00:00
Ilya K b469c6509b libstore/build: just copy the magic /etc files into the sandbox
Saves us a bunch of thinking about how to handle symlinks, and prevents
the DNS config from changing on the fly under the build, which may or may
not be a good thing?

Change-Id: I071e6ae7e220884690b788d94f480866f428db71
2024-04-13 12:43:19 +03:00
Ilya K d363bc2f12 Merge "Merge pull request #10456 from NixOS/fixpermdeniedbind" into main 2024-04-11 19:08:33 +00:00
eldritch horrors e4a8c01bdf Merge changes Iedf46484,I76b51eac,I6a084827,I60193f9f into main
* changes:
  meson: fix log-dir
  manual: build docs with dummy envs
  libcmd: install generated headers as well
  docs: redo content generation for mdbook and manual
2024-04-11 14:33:16 +00:00
Ilya K d106bb553b Merge "Merge pull request #10362 from obsidiansystems/maybeLstat" into main 2024-04-11 13:45:46 +00:00
eldritch horrors f42678802c libcmd: install generated headers as well
these seem to have been forgotten.

Change-Id: I6a084827d087f8098c19b62f2060a874d87202a1
2024-04-11 13:36:04 +00:00
eldritch horrors 725f5cd358 docs: redo content generation for mdbook and manual
manpages can be rendered using the markdown output of mdbook, the rest
of the manual can generated out of the main doc/manual source tree. we
still use lowdown to actually render manpages instead of eg mdbook-man
because lowdown does generate reasonably good manpages (though that is
also somewhat debatable, but they're a lot better than mdbook-man).

doing this not only lets us drastically simplify the lowdown pipeline,
but also remove all custom {{#include}} handling since now mdbook does
all of it, even for the manpage builds. even the lowdown wrapper isn't
entirely necessary because lowdown can take all wrapper arguments with
command line flags rather than bits of input file content.

This also implements running mdbook in Meson, in order to generate the
manpages. The mdbook outputs are also installed in the usual location.

Co-authored-by: Qyriad <qyriad@qyriad.me>

Change-Id: I60193f9fd0f15d48872f071af35855cda2a0f40b
2024-04-11 13:32:06 +00:00
Théophane Hufschmitt 07b627cc6d Merge pull request #10456 from NixOS/fixpermdeniedbind
Fix adding symlink to the sandbox paths

(cherry-picked from commit da1e977bf48cff2a635034c85e7c13878e38efc2)

Change-Id: I221c85a38180800ec6552d2e86a88df48398fad8
2024-04-11 15:43:58 +03:00
John Ericson aeee22e5a1 Merge pull request #10362 from obsidiansystems/maybeLstat
Factor out `nix::maybeLstat`

(cherry-picked from commit 9b88e5284608116b7db0dbd3d5dd7a33b90d52d7)

Change-Id: Id890525e847c890fad6593c594772826ac4d1d50
2024-04-11 15:43:41 +03:00