Fixes:
- Identifiers starting with _ are prohibited
- Some driveby header dependency cleaning which wound up with doing some
extra fixups.
- Fucking C style casts, man. C++ made these 1000% worse by letting you
also do memory corruption with them with references.
- Remove casts to Expr * where ExprBlackHole is an incomplete type by
introducing an explicitly-cast eBlackHoleAddr as Expr *.
- An incredibly illegal cast of the text bytes of the StorePath hash
into a size_t directly. You can't DO THAT.
Replaced with actually parsing the hash so we get 100% of the bits
being entropy, then memcpying the start of the hash. If this shows
up in a profile we should just make the hash parser faster with a
lookup table or something sensible like that.
- This horrendous bit of UB which I thankfully slapped a deprecation
warning on, built, and it didn't trigger anywhere so it was dead
code and I just deleted it. But holy crap you *cannot* do that.
inline void mkString(const Symbol & s)
{
mkString(((const std::string &) s).c_str());
}
- Some wrong lints. Lots of wrong macro lints, one wrong
suspicious-sizeof lint triggered by the template being instantiated
with only pointers, but the calculation being correct for both
pointers and not-pointers.
- Exceptions in destructors strike again. I tried to catch the
exceptions that might actually happen rather than all the exceptions
imaginable. We can let the runtime hard-kill it on other exceptions
imo.
Change-Id: I71761620846cba64d66ee7ca231b20c061e69710
It's nice for this to be a separate function and not just inline in
`absPath`.
Prepared as part of cl/1865, though I don't think I actually ended up
using it there.
Change-Id: I24d9d4a984cee0af587010baf04b3939a1c147ec
this makes WorkResult copyable, and just all around easier to deal with.
in the future we'll need this to let Goal::work() return a promise for a
WorkResult (or even just a Finished) that can be awaited by other goals.
Change-Id: Ic5a1ce04c5a0f8e683bd00a2ed2b77a2e28989c1
this should be done where we're actually trying to build something, not
in the main worker loop that shouldn't have to be aware of such details
Change-Id: I07276740c0e2e5591a8ce4828a4bfc705396527e
This caused an absolute saga which I would not like anyone else to have
to experience. Let's put in a laser targeted error message that
diagnoses this exact problem.
Fixes: #484
Change-Id: I2a79f04aeb4a1b67c10115e5e39501d958836298
I don't know why the AWS sdk disabled it by default. It would be nice
to have test coverage of the s3 store or proxies, but neither currently
exist.
Fixes: #433
Change-Id: If1e76169a3d66dbec2e926af0d0d0eccf983b97b
This avoids C++'s standard library regexes, which aren't the same
across platforms, and have many other issues, like using stack
so much that they stack overflow when processing a lot of data.
To avoid backwards and forward compatibility issues, regexes are
processed using a function converting libstdc++ regexes into Boost
regexes, escaping characters that Boost needs to have escaped, and
rejecting features that Boost has and libstdc++ doesn't.
Related context:
- Original failed attempt to use `boost::regex` in CppNix, failed due to
boost icu dependency being large (disabling ICU is no longer necessary
because linking ICU requires using a different header file,
`boost/regex/icu.hpp`): https://github.com/NixOS/nix/pull/3826
- An attempt to use PCRE, rejected due to providing less backwards
compatibility with `std::regex` than `boost::regex`:
https://github.com/NixOS/nix/pull/7336
- Second attempt to use `boost::regex`, failed due to `}` regex failing
to compile (dealt with by writing a wrapper that parses a regular
expression and escapes `}` characters):
https://github.com/NixOS/nix/pull/7762Closes#34. Closes#476.
Change-Id: Ieb0eb9e270a93e4c7eed412ba4f9f96cb00a5fa4
nixpkgs delivered us the untimely gift of a meson 1.5 upgrade, which
*does* make our lives easier by allowing us to delete wrap generation
code, but it does so at the cost of renaming all rust crates in such a
way that the wrap logic cannot tolerate the new names on the old meson
version 😭.
It also means that support burden for this is going to be atrocious
until we either give in and vendor meson 1.5 or we make a CI target for
it. Neither seems appealing, though the latter is not super absurd for
ensuring we don't break nixpkgs unstable.
This commit causes meson 1.5 to ignore the .wrap files in subprojects/
entirely (since they have the wrong names lol) and instead use
Cargo.lock, so it now hard-depends on our workspace reshuffling
improvement.
It also deletes the hack that we were using to get the sources of Cargo
deps into meson by using a feature that went unnoticed when this code
was originally written: MESON_PACKAGE_CACHE_DIR:
8a202de6ec/mesonbuild/wrap/wrap.py (L490-L502)
Change-Id: I7a28f12fc2812c6ed7537b60bc3025c141a05874
This is purely to let Cargo's dependency resolver do stuff for us, we do
not actually intend to build this stuff with Cargo to begin with.
Change-Id: I4c08d55595c7c27b7096375022581e1e34308a87
There have been multiple setting types for paths that are supposed to be
canonicalised, depending on whether zero or one, one, or any number of paths is
to be specified. Naturally, they behaved in slightly different ways in the
code. Simplify things by unifying them and removing special behaviour (mainly
the "multiple paths type can coerce to boolean" thing).
Change-Id: I7c1ce95e9c8e1829a866fb37d679e167811e9705
Commit 0dd1d8ca1c included an accidental revert
of 1461e6cdda (actually slightly worse), leading
to the progress bar not being stopped properly when a legacy command was
invoked with `--log-format bar` (or similar options that show a progress bar).
Move the progress bar stopping code to its proper place again to fix this
regression.
Change-Id: I676333da096d5990b717a387924bb988c9b73fab
lix-doc is now built with Meson, with lix-doc's dependencies built as
Meson subprojects, either fetched on demand with .wrap files, or fetched
in advance by Nix with importCargoLock. It even builds statically.
Fixes#256.
Co-authored-by: Lunaphied <lunaphied@lunaphied.me>
Co-authored-by: Jade Lovelace <lix@jade.fyi>
Change-Id: I3a4731ff13278e7117e0316bc0d7169e85f5eb0c
Closes#460
I managed to trigger the issue by having the following inputs (shortened):
authentik-nix.url = "github:nix-community/authentik-nix";
authentik-nix.inputs.poetry2nix.inputs.nixpkgs.follows = "nixpkgs";
When evaluating this using
nix-eval-jobs --flake .#hydraJobs
I got the following error:
error: cannot update unlocked flake input 'authentik-nix/poetry2nix' in pure mode
The issue we have here is that `authentik-nix/poetry2nix` was written
into the `overrideMap` which caused Nix to assume it's a new input and
tried to refetch it (#460) or errored out in pure mode
(nix-eval-jobs / Hydra).
The testcase unfortunately only involves checking for the output log
and makes sure that something *is* logged on the first fetch so that
the test doesn't rot when the logging changes since I didn't
manage to trigger the error above with the reproducer from #460. In
fact, I only managed to trigger the `cannot update unlocked flake input`
error in this context with `nix-eval-jobs`.
Change-Id: Ifd00091eec9a0067ed4bb3e5765a15d027328807
this can be a proper WorkResult now. childTerminated is unfortunately a
lot more stubborn and won't be made private for quite a while yet. once
we can get rid of the Worker poll loop that *should* be possible though
Change-Id: I2218df202da5cb84e852f6a37e4c20367495b617
we'll need this once we want to pass extra information out of accepting
replies, such as fd sets or possibly even async output reader promises.
Change-Id: I5e2f18cdb80b0d2faf3067703cc18bd263329b3f
don't keep fds open we're not using. currently this does not cause any
problems, but it does increase the size of our fd table needlessly and
in the future, when we have proper async processing, having builderOut
open in the daemon once the hook has been fully started is problematic
Change-Id: I6e7fb773b280b042873103638d3e04272ca1e4fc
this is useless to do on the face of it, but it'll make it easier to
convert the entire output handling to use async io and promises soon
Change-Id: I2d1eb62c4bbf8f57bd558b9599c08710a389b1a8
only DerivationGoal can set the hook to anything at all. it always sets
buildOutFD to something that is not related to fromHook in any way, and
mixing the two would have rather dire consequences for log consistency.
Change-Id: Ida86727fd1cd5e1ecd78f07f3bde330a346658a8
all derivation goals need a log fd of some description. let's save this
single fd in a dedicated pointer field for all subclasses so that later
we have just the one spot to change if we turn this into async promises
Change-Id: If223adf90909247363fb823d751cae34d25d0c0b
we don't need to expose information about how busy a Worker is if the
worker can instead tell its work items whether they are in a slot. in
the future we might use this to not start items waiting for a slot if
no slots are currently available, but that requires more preparation.
Change-Id: Ibe01ac536da7e6d6f80520164117c43e772f9bd9
They are like experimental features, but opt-in instead of opt-out. They
will allow us to gracefully remove language features. See #437
Change-Id: I9ca04cc48e6926750c4d622c2b229b25cc142c42
Seems a little bit Rich that musl does not implement close_range because
they suspect that the system call itself is a bad idea, so they uhhhh
are considering not implementing a wrapper. Let's just fix the problem
at hand by writing our own wrapper.
Change-Id: I1f8e5858e4561d58a5450503d9c4585aded2b216
Turns out strings do not like being resized to -4.
This was discovered while messing with the tests to remove unbuffer and
trying stdbuf instead. Turns out that was not the right approach.
This basically rewrites the handling of this case to be much more
correct, and fixes a bug where with small window sizes where it would
ALSO truncate the attr names in addition to the optional descriptions.
Change-Id: Ifd1beeaffdb47cbb5f4a462b183fcb6c0ff6c524
This will stop printing stuff to dumb terminals that they don't support.
I've overall audited usage of isatty and replaced the ones with intent
to mean "is a Real terminal" with checking for that. I've also caught a
case of carelessly assuming "is a tty" means "should be colour" in
nix-env.
Change-Id: I6d83725d9a2d932ac94ff2294f92c0a1100d23c9
this is only used to close non-stdio files in derivation sandboxes. we
may as well encode that in its name, drop the unnecessary integer set,
and use close_range to deal with the actual closing of files. not only
is this clearer, it also makes sandbox setup on linux fast by 1ms each
Change-Id: Id90e259a49c7bc896189e76bfbbf6ef2c0bcd3b2
implementing a build hook is pretty much impossible without either being
a nix, or blindly forwarding the important bits of all build requests to
some kind of nix. we've found no uses of build-hook in the wild, and the
build-hook protocol (apart from being entirely undocumented) is not able
to convey any kind of versioning information between hook and daemon. if
we want to upgrade this infrastructure (which we do), this must not stay
Change-Id: I1ec4976a35adf8105b8ca9240b7984f8b91e147e
this is a bit of a hack, but it's apparently the cleanest way of doing
this in the absence of any kind of priority/provenance information for
values of some given setting. we'll need this to deprecate build-hook.
Change-Id: I03644a9c3f17681c052ecdc610b4f1301266ab9e
sure, linux has been providing argv[0] by default for a while now. other
OSes may not be as forthcoming though, and relying on the OS to create a
world in which we can just make assumptions we could test for instead is
unnecessarily lazy. we *could* default argv0, but that's a little silly.
notably we abort instead of returning normally to avoid confusions where
a caller interprets our exit status like a Worker build results bitmask.
Change-Id: Id73f8cd0a630293b789c59a8c4b0c4a2b936b505
* changes:
sqlite: add a Use::fromStrNullable
util: implement charptr_cast
tree-wide: fix a pile of lints
refactor: make HashType and Base enum classes for type safety
build: integrate clang-tidy into CI
There were several usages of the raw sqlite primitives along with C
style casts, seemingly because nobody thought to use an optional for
getting a string or NULL.
Let's fix this API given we already *have* a wrapper.
Change-Id: I526cceedc2e356209d8fb62e11b3572282c314e8
I don't like having so many reinterpret_cast statements that have to
actually be looked at to determine if they are UB. A huge number of the
reinterpret_cast instances in Lix are actually casting to some pointer
of some character type, which is always valid no matter the source type.
However, it is also worth looking at if it is not casting both *from* a
character type and also *to* a character type, since IMO splatting a
struct into a character array should be a very deliberate action instead
of just being about dealing with bad APIs.
So let's write a template that encapsulates this invariant so we can
not worry about the trivially safe reinterpret_cast invocations.
Change-Id: Ia4e2f1fa0c567123a96604ddadb3bdd7449660a4
This:
- Converts a bunch of C style casts into C++ casts.
- Removes some very silly pointer subtraction code (which is no more or
less busted on i686 than it began)
- Fixes some "technically UB" that never had to be UB in the first
place.
- Makes finally follow the noexcept status of the inner function. Maybe
in the future we should ban the function from not being noexcept, but
that is not today.
- Makes various locally-used exceptions inherit from std::exception.
Change-Id: I22e66972602604989b5e494fd940b93e0e6e9297
This still has utterly unacceptably bad output format design that I
would not inflict on anyone I like, but it *does* now exist, and you
*can* find the errors in the log.
Future work would obviously be to fix that and integrate the actual
errors into Gerrit using codechecker or so.
Followup issue: #457
Fixes: #147
Change-Id: Ifca22e443d357762125f4ad6bc4f568af3a26c62
The |> operator is a reverse function operator with low binding strength
to replace lib.pipe. Implements RFC 148, see the RFC text for more
details. Closes#438.
Change-Id: I21df66e8014e0d4dd9753dd038560a2b0b7fd805
Currently, the parser relies on the global experimental feature flags.
In order to properly test conditional language features, we instead need
to pass it around in the parser::State.
This means that the parser cannot cache the result of isEnabled anymore,
which wouldn't necessarily hurt performance if the function didn't
perform a linear search on the list of enabled features on every single
call. While we could simply evaluate once at the start of parsing and
cache the result in the parser state, the more sustainable solution
would be to fix `isEnabled` such that all callers may profit from the
performance improvement.
Change-Id: Ic9b9c5d882b6270e1114988b63e6064d36c25cf2
This adds a second form to the `:log` command: it now can accept a
derivation path in addition to a derivation expression. As derivation
store paths start with `/nix/store`, this is not ambiguous.
Resolves: #51
Change-Id: Iebc7b011537e7012fae8faed4024ea1b8fdc81c3
This has been causing various seemingly spurious CI failures as well as
some failures on people running tests on beta builds.
lix> ++(nix-collect-garbage-dry-run.sh:20) nix-store --gc --print-dead
lix> ++(nix-collect-garbage-dry-run.sh:20) wc -l
lix> finding garbage collector roots...
lix> error: Listing pid 87261 file descriptors: Undefined error: 0
There is no real way to write a proper test for this, other than to
start a process like the following:
int main(void) {
for (int i = 0; i < 1000; ++i) {
close(i);
}
sleep(10000);
}
and then let Lix's gc look at it.
I have a relatively high confidence this *will* fix the problem since I
have manually confirmed the behaviour of the libproc call is
as-unexpected, and it would perfectly explain the observed symptom.
Fixes: #446
Change-Id: I67669b98377af17895644b3bafdf42fc33abd076
* changes:
tree-wide: fix various lint warnings
flake & doxygen: update tagline
nix flake metadata: print modified dates for input flakes
cli: eat terminal codes from stdout also
Implement forcing CLI colour on, and document it better
manual: fix a syntax error in redirects.js that made it not do anything
misc docs/meson tidying
build: implement clang-tidy using our plugin
The growth of the seccomp filter in 127ee1a101
made its compilation time significant (roughly 10 milliseconds have been
measured on one machine). For this reason, it is now precompiled and cached in
the parent process so that this overhead is not hit for every single build. It
is still not optimal when going through the daemon, because compilation still
happens once per client, but it's better than before and doing it only once for
the entire daemon requires excessive crimes with the current architecture.
Fixes: #461
Change-Id: I2277eaaf6bab9bd74bbbfd9861e52392a54b61a3
This is a preparation for precompiling the filter, which is done separately.
The behaviour should be unchanged for now.
Change-Id: I899aa7242962615949208597aca88913feba1cb8
The seccomp setup code was a huge chunk of conditionally compiled
platform-specific code. For this reason, it is appropriate to move it to the
platform-specific implementation file. Ideally its setup could be moved a bit
to make it happen at the same place as the Darwin restrictions, but that change
is going to be less mechanical.
Change-Id: I496aa3c4fabf34656aba1e32b0089044ab5b99f8
This was always in the lock file and we can simply actually print it.
The test for this is a little bit silly but it should correctly
control for my daring to exercise timezone code *and* locale code in a
test, which I strongly suspect nobody dared do before.
Sample (abridged):
```
Path: /nix/store/gaxb42z68bcr8lch467shvmnhjjzgd8b-source
Last modified: 1970-01-01 00:16:40
Inputs:
├───flake-compat: github:edolstra/flake-compat/0f9255e01c2351cc7d116c072cb317785dd33b33
│ Last modified: 2023-10-04 13:37:54
├───flake-utils: github:numtide/flake-utils/b1d9ab70662946ef0850d488da1c9019f3a9752a
│ Last modified: 2024-03-11 08:33:50
│ └───systems: github:nix-systems/default/da67096a3b9bf56a91d16901293e51ba5b49a27e
│ Last modified: 2023-04-09 08:27:08
```
Change-Id: I355f82cb4b633974295375ebad646fb6e2107f9b
This *should* be sound, plus or minus the amount that the terminal code
eating code is messed up already.
This is useful for testing CLI output because it will strip the escapes
enough to just shove the expected output in a file.
Change-Id: I8a9b58fafb918466ac76e9ab585fc32fb9294819
The docs page has an incorrect escape that leads to a backslash
appearing in output. Meson stuff is self-explanatory, just shortens and
simplifies a bit.
Change-Id: Ib63adf934efd3caeb82ca82988f230e8858a79f9
Expose an option for disabling the BDW-GC build dependency entirely. Fix the
place where one of its headers was included (unnecessarily) without proper
guarding. Finally, use this machinery to exclude BDW-GC from the ASAN builds
entirely (its usage has already been disabled due to compatibility issues
anyway), to ensure this configuration is not regressed again.
Change-Id: I2ebe8094abf67e7d1e99eed971de3e99d071c10b
this begins a long and arduous journey to remove all result state from
Goal, to eventually drop the std::enable_shared_from_this base, and to
completely eliminate all unsynchronized modification of states of both
Goal and Worker. by the end of this we will hopefully be able to start
and reap multiple derivation builds in parallel, which should speed up
the process quite a bit (at least for short local builds, others might
not notice a large difference. the build hooks will remain a problem.)
Change-Id: I57dcd9b2cab4636ed4aa24cdec67124fef883345
In the SSH code, the logger was conditionally paused, but unconditionally
resumed. This was fine as long as resuming the logger was idempotent. Starting
with 0dd1d8ca1c, it isn't any more, and the
behaviour of the code in question was missed. Consequently, an assertion
failure is triggered for example when performing builds against an "SSH" store
on localhost. Fix the issue by only resuming the logger when it has actually
been paused.
Fixes: #458
Change-Id: Ib1e4d047744a129f15730b7216f9c9368c2f4211
we still mutate goal state to store the results of any given goal run,
but now we also have that information in Worker and could in theory do
something else with it. we could return a map of goal to goal results,
which would also let us better diagnose failures of subgoals (at all).
Change-Id: I1df956bbd9fa8cc9485fb6df32918d68dda3ff48
this is the first step towards removing all result-related mutation of
Goal state from goal implementations themselves, and into Worker state
instead. once that is done we can treat all non-const Goal fields like
private state of the goal itself, and make threading of goals possible
Change-Id: I69ff7d02a6fd91a65887c6640bfc4f5fb785b45c
once goals run on multiple threads these fields must by synchronized as
one, or we try to run build hooks to often (or worse, not often enough)
Change-Id: I47860e46fe5c6db41755b2a3a1d9dbb5701c4ca4
this will usually be used either directly (which is always fine) or in
Finally blocks (where it must never throw execptions). make sure that,
exceptions being handled or not, the calling wait() in Finally doesn't
cause crashes due to the Finally no-nested-exceptions-thrown assertion
Change-Id: Ib83a5d9483b1fe83b9a957dcefeefce5d088f06d
The original attempt at this introduced a regression; this commit
reverts the revert and fixes the regression.
This reverts commit 3e151d4d77.
Fix to the regression:
flakeref: fix handling of `?dir=` param for flakes in subdirs
As reported in #419[1], accessing a flake in a subdir of a Git
repository fails with the previous commit[2] applied with the error
error: unsupported Git input attribute 'dir'
The problem is that the `dir`-param is inserted into the parsed URL if a
flake is fetched from the subdir of a Git repository. However, for the
fetching part this isn't even needed. The fix is to just pass `subdir`
as second argument to `FlakeRef` (which needs a `basedir` that can be
empty) and leave the parsedURL as-is.
Added a regression test to make sure we don't run into this again.
[1] #419
[2] e22172aaf6b6a366cecd3c025590e68fa2b91bcc,
originally 3e151d4d77
Change-Id: I2c72d5a32e406a7ca308e271730bd0af01c5d18b
* changes:
remove unused headers in installable-attr-path
libexpr: include the type of the non-derivation value in the type error
libexpr: mild cleanup to getDerivations
libexpr: DrvInfo: remove unused bad-citizen constructor
cleanup and slightly refactor DrvInfo::queryOutputs
* changes:
releng: move officialRelease to version.json
Add -Werror CI job
ci: add a asan+ubsan test run on x86_64-linux
tree-wide: add support for asan!
Shuffled the logic around a bit so the shorter code paths are early
returns, added comments, etc.
Should be NFC.
Change-Id: Ie3ddb3d0eddd614d6f8c37bf9a4d5a50282084ea
DrvInfo's constructor that only takes `EvalState` leaves everything else
empty; a DrvInfo which has no iota of information about the derivation
it represents is not useful, and was not used anywhere.
Change-Id: Ic4d93a08cb2748b8cef9a61e41e70404834b23f9
What if you could find memory bugs in Lix without really trying very
hard? I've had variously scuffed patches to do this, but this is
blocked on boost coroutines removal at this point tbh.
Change-Id: Id762af076aa06ad51e77a6c17ed10275929ed578
there are no other uses for this yet, but asking for just a subset of
outputs does seem at least somewhat useful to have as a generic thing
Change-Id: I30ff5055a666c351b1b086b8d05b9d7c9fb1c77a
If `:edit`ing a store path, don't reload repl afterwards
to avoid losing local variables: store is immutable,
so "editing" a store path is always just viewing it.
Resolves: #341
Change-Id: I3747f75ce26e0595e953069c39ddc3ee80699718
limiting CA substitutions was a rather recent addition, and it used a
dedicated counter to not interfere with regular substitutions. though
this works fine it somewhat contradicts the documentation; job limits
should apply to all kinds of substitutions, or be one limit for each.
Change-Id: I1505105b14260ecc1784039b2cc4b7afcf9115c8
all goals do this. it makes no sense to not notify a goal of EOF
conditions because this is the universal signal for "child done"
Change-Id: Ic3980de312547e616739c57c6248a8e81308b5ee
just update progress every time a goal has returned from work(). there
seem to be no performance penalties, and the code is much simpler now.
Change-Id: I288ee568b764ee61f40a498d986afda49987cb50
bindPath/doBind is a useful function in build that is used in several
parts of LocalDerivationGoal. Moving this function makes it easier to
split LocalDerivationGoal implementation between several files.
Change-Id: Ic5a0768479c153c1aa3ed425f12604b20bbf0f42
Unfortunately, io_uring is totally opaque to seccomp, and while currently there
are no dangerous operations implemented, there is no guarantee that it remains
this way. This means that io_uring should be blocked entirely to ensure that
the sandbox is future-proof. This has not been observed to cause issues in
practice.
Change-Id: I45d3895f95abe1bc103a63969f444c334dbbf50d
Previously, system call filtering (to prevent builders from storing files with
setuid/setgid permission bits or extended attributes) was performed using a
blocklist. While this looks simple at first, it actually carries significant
security and maintainability risks: after all, the kernel may add new syscalls
to achieve the same functionality one is trying to block, and it can even be
hard to actually add the syscall to the blocklist when building against a C
library that doesn't know about it yet. For a recent demonstration of this
happening in practice to Nix, see the introduction of fchmodat2 [0] [1].
The allowlist approach does not share the same drawback. While it does require
a rather large list of harmless syscalls to be maintained in the codebase,
failing to update this list (and roll out the update to all users) in time has
rather benign effects; at worst, very recent programs that already rely on new
syscalls will fail with an error the same way they would on a slightly older
kernel that doesn't support them yet. Most importantly, no unintended new ways
of performing dangerous operations will be silently allowed.
Another possible drawback is reduced system call performance due to the larger
filter created by the allowlist requiring more computation [2]. However, this
issue has not convincingly been demonstrated yet in practice, for example in
systemd or various browsers. To the contrary, it has been measured that the the
actual filter constructed here has approximately the same overhead as a very
simple filter blocking only one system call.
This commit tries to keep the behavior as close to unchanged as possible. The
system call list is in line with libseccomp 2.5.5 and glibc 2.39, which are the
latest versions at the point of writing. Since libseccomp 2.5.5 is already a
requirement and the distributions shipping this together with older versions of
glibc are mostly not a thing any more, this should not lead to more build
failures any more.
[0] https://github.com/NixOS/nixpkgs/issues/300635
[1] https://github.com/NixOS/nix/issues/10424
[2] https://github.com/flatpak/flatpak/pull/4462#issuecomment-1061690607
Change-Id: I541be3ea9b249bcceddfed6a5a13ac10b11e16ad
In f047e4357b, I missed the behavior that if
building without a dedicated build user (i.e. in single-user setups), seccomp
setup failures are silently ignored. This was introduced without explanation 7
years ago (ff6becafa8). Hopefully the only
use-case nowadays is causing spurious test suite successes when messing up the
seccomp filter during development. Let's try removing it.
Change-Id: Ibe51416d9c7a6dd635c2282990224861adf1ceab
getSelfExe is used in a few places re-execute nix.
Current code in this file uses ifdefs to support several
platforms, just keep doing that
Change-Id: Iecc2ada0101aea0c30524e3a1218594f919d74bf
This was done originally because std::smatch does not accept `const char
*` as iterators. However, this was because we should have been using
std::cmatch instead.
Change-Id: Ibe73851fd39755e883df2d33d22fed72ac0a04ae
Nobody has stepped up to add further support for Hurd since this code
appeared in 2010 or 2014. We don't need it.
Change-Id: I400b2031a225551ea3c71a3ef3ea9fdb599dfba3
Use libprocstat to find garbage collector roots on FreeBSD.
Tested working on a FreeBSD machine, although there is no CI yet
Change-Id: Id36bac8c3de6cc4de94e2d76e9663dd4b76068a9
(cherry picked from commit 8cd1d02f90eb9915e640c5d370d919fad9833c65)
nix flake show: Only print up to the first new line if it exists.
(cherry picked from commit 5281a44927bdb51bfe6e5de12262d815c98f6fe7)
add tests
(cherry picked from commit 74ae0fbdc70a5079a527fe143c4832d1357011f7)
Handle long strings, embedded new lines and empty descriptions
(cherry picked from commit 2ca7b3afdbbd983173a17fa0a822cf7623601367)
Account for total length of 80
(cherry picked from commit 1cc808c18cbaaf26aaae42bb1d7f7223f25dd364)
docs: add nix flake show description release note
fix: remove white space
nix flake show: trim length based on terminal size
test: account for terminal size
docs(flake-description): before and after commands; add myself to credits
Upstream-PR: https://github.com/NixOS/nix/pull/10980
Change-Id: Ie1c667dc816b3dd81e65a1f5395e57ea48ee0362
Error is pretty large, and most goals do not fail. this alone more than
halves the size of Goal on x86_64-linux, from 720 bytes down to 344. in
derived classes the difference is not as dramatic, but even the largest
derived class (`LocalDerivationGoal`) loses almost 20% of its footprint
Change-Id: Ifda8f94c81b6566eeb3e52d55d9796ec40c7bce8
the goals are either already using std::async and merely forgot to
remove std::thread vestiges or they emulate async with threads and
promises. we can simply use async directly everywhere for clarity.
Change-Id: I3f05098310a25984f10fff1e68c573329002b500
under owner_less it's equivalent to insert(), only sometimes a little
bit faster because it does not construct a weak_ptr if the goal is in
the set already. this small difference in performance does not matter
here and c++23 will make insert transparent anyway, so we can drop it
Change-Id: I7cbd7d6e0daa95d67145ec58183162f6c4743b15
*accidentally* overriding a function is almost guaranteed to be an
error. overriding a function without labeling it as such is merely
bad style, but bad style that makes the code harder to understand.
Change-Id: Ic0594f3d1604ab6b3c1a75cb5facc246effe45f0
Commit 0109368c3f missed to include a required
header, which is not noticed when the precompiled header is enabled because
it's included in that. Also include it in the file so that the build without
precompiled header works too.
Change-Id: Id7a7979684b64f937f7f8191612952d73c113015
Due to a leftover from a previous version where the buffer was allocated on the
stack, the change introduced in commit 4ec87742a1
accidentally passes the size of a pointer as the size of the buffer to the
decompressor. Since the former is much smaller (usually 8 bytes instead of 64
kilobytes), this is safe, but leads to considerable overhead; most notably, due
to excessive progress reports, which happen for each chunk. Pass the proper
buffer size instead.
Change-Id: If4bf472d33e21587acb5235a2d99e3cb10914633
This commit adds a new helper template function to gc-alloc.hh (which is
probably where you want to look at first, O great reviewer [custom file
ordering in review diffs when]), which uses a type argument to determine
the size to allocate, rather than making the caller use sizeof().
Change-Id: Ib5d138d91a28bdda304a80db24ea9fb08669ad22
The purpose of this function has little to do with immutability. Value's
strings are never mutated, and the point of this function is to
singleton empty strings.
Change-Id: Ifd41dd952409d54e4d3de9ab59064e6928b0e480
SimpleLogger is not fully thread-safe, and all loggers that wrap it are
also not safe accordingly. this does not affect much, but in rare cases
it can cause interleaving of messages on stderr when used with the json
or raw log formats. the fix applied here is a bit of a hack, but fixing
this properly requires rearchitecting the logger infrastructure. nested
loggers are not the most natural abstraction here, and it is biting us.
Change-Id: Ifbf34fe1e85c60e73b59faee50e7411c7b5e7c12
it's only used once, and even that one use is highly questionable. more
instances of warnOnce should be much more principled than this has been
Change-Id: I5856570c99cb44462e700d753d0c706a5db03c4b
If useChroot = false, and user namespaces aren't available for some
reason (e.g. within a Docker container), this fixes a pointless warning
being emitted, as we would never attempt to use them even if they were
available.
Change-Id: Ibcee91c088edd2cd19e70218d5a5802bff8f537b
This removes a *whole load* of variables from scope and enforces thread
boundaries with the type system.
There is not much change of significance in here, so the things to watch
out for while reviewing it are primarily that the destructor ordering
may have changed inadvertently, I think.
Change-Id: I3cd87e6d5a08dfcf368637407251db22a8906316
* changes:
Fixup a bunch of references to nixos.org manuals
Add release notes for removing overflow from Nix language
expr: fix a compiler warning about different signs in comparison
* changes:
doc/release-notes: add for pretty printing improvements
libexpr/print: do not show elided nested items when there are none
libexpr/print: never show empty attrsets or derivations as «repeated»
libexpr/print: pretty-print idempotently
* changes:
docs: document the actual comparison rules instead of lies
daemon: remove workaround for macOS kernel bug that seems fixed
daemon: fix a crash bug "FATAL: exception not rethrown"
When the configured maximum depth has been reached, attribute sets and lists
are printed with ellipsis to indicate the elision of nested items. Previously,
this happened even in case the structure being printed is empty, so that such
items do not in fact exist. This is confusing, so stop doing it.
Change-Id: I0016970dad3e42625e085dc896e6f476b21226c9
The repeated value detection logic exists so that the occurrence of large
common substructures does not fill up the screen or the computer's memory.
However, empty attribute sets and derivations (when their detection is enabled)
are always cheap to print, and in practice I have observed them to make up a
significant majority of the cases where I was annoyed by the repeated value
detection kicking in. Furthermore, `nix-instantiate --eval` already disables
this logic for empty attribute sets, and empty lists are already exempted
everywhere. For these reasons, always print empty attribute sets and
derivations as what they are.
Change-Id: I5dac8e7739f9d726b76fd0521ec46f38af94463f
When pretty-printing is enabled, previously an unforced thunk would trigger
indentation, even when it subsequently does not evaluate to a nested structure.
The resulting output looked inconsistent, and furthermore pretty-printing was
not idempotent (since pretty-printing the same value again, which is now fully
evaluated, will not trigger indentation).
When strict evaluation is enabled, force the item before inspecting its type,
so that it is properly known whether it contains a nested structure.
Furthermore, there is no need to cause indentation for unforced thunks, since
the very next operation will be printing them as `«thunk»`.
This is mostly a port of https://github.com/NixOS/nix/pull/11100 , but we only
force the item when it's going to be forced anyway due to strict
pretty-printing, and a new test was written since the REPL testing framework in
Lix is different.
Co-Authored-By: Robert Hensing <robert@roberthensing.nl>
Change-Id: Ib7560fe531d09e05ca6b2037a523fe21a26d9d58
this is cursed. deeply and profoundly cursed. under NO CIRCUMSTANCES
must protocol serializer helpers be applied to temporaries! doing so
will inevitably cause dangling references and cause the entire thing
to crash. we need to do this even so to get rid of boost coroutines,
and likewise to encapsulate the serializers we suffer today at least
a little bit to allow a gradual migration to an actual IPC protocol.
(this isn't a problem that's unique to generators. c++ coroutines in
general cannot safely take references to arbitrary temporaries since
c++ does not have a lifetime system that can make this safe. -sigh-)
Change-Id: I2921ba451e04d86798752d140885d3c5cc08e146
this doesn't have a test because this code path is only reached by
clients that predate 2.4, and we really should not be caring about
those any more right now. even the test suite doesn't, and the few
tests that might care are disabled because they will not even work
Change-Id: Id9eb190065138fedb2c7d90c328ff9eb9d97385b
this is not completely necessary at this point because the parser right
now already returns a generator to pass through all input data it read,
but the nar parser *was* very lax and would accept nars that weren't in
canonical form (defined as the form dumpPath would return). nar hashing
depends on these things, and as such rewriting the parser now allows us
to reject non-canonical nars that extract to the same store contents as
their canonical counterpart but have different nar hashes despite that.
Change-Id: Iccd319e3bd5912d8297014c84c495edc59019bb7
This was filed as https://github.com/nixos/nix/issues/7584, but as far
as I can tell, the previous solution of POLLHUP works just fine on macOS
14. I've also tested on an ancient machine with macOS 10.15.7, which
also has POLLHUP work correctly.
It's possible this might regress some older versions of macOS that have
a kernel bug, but I went looking through the history on the sources and
didn't find anything that looked terribly convincingly like a bug fix
between 2020 and today. If such a broken version exists, it seems pretty
reasonable to suggest simply updating the OS.
Change-Id: I178a038baa000f927ea2cbc4587d69d8ab786843
This is caused by pthread_cancel effectively throwing a
not-specifically-identifiable C++ exception into the targeted thread,
which, if it is not rethrown, terminates the process entirely.
This is rather "impolite" behaviour, we would say. But thread
cancellation is *always* busted, and we should simply not use it where
unnecessary. It's particularly unnecessary when what we *actually* need
it for is, err, interrupting a poll(2).
That can in turn be achieved by simply listening to more stuff in the
poll, namely, a pipe, which we send a character to when needing to
stop the thread.
While looking at this code, we also investigated whether any of the
poll() madness is required, or was even *ever* required. Curiously we
found in the XNU kernel source code that the thing about needing to
listen to POLLHUP is probably *correct*, but switching it to POLLRDNORM
should not have made any difference at all. We've left a FIXME to look
into that further because what's written here is super janky.
94d3b45284/bsd/kern/sys_generic.c (L1751-L1758)
This is the crash on some Hydra machines:
Thread 1 (Thread 0x7f56b77776c0 (LWP 955542) (Exiting)):
0 0x00007f56b8e9b7dc in __pthread_kill_implementation () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
1 0x00007f56b8e49516 in raise () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
2 0x00007f56b8e31935 in abort () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
3 0x00007f56b8e327f3 in __libc_message_impl.cold () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
4 0x00007f56b8e8e8e9 in __libc_fatal () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
5 0x00007f56b8ea23c4 in unwind_cleanup () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
6 0x00007f56b9d2a1b8 in nix::triggerInterrupt() [clone .cold] () from /nix/store/sahgw550p621m9dy1pd7whl9c5g1g0p7-lix-2.90.0-rc1/lib/liblixutil.so
7 0x00007f56b990ac9d in std:🧵:_State_impl<std:🧵:_Invoker<std::tuple<nix::MonitorFdHup::MonitorFdHup(int)::{lambda()#1}> > >::_M_run() () from /nix/store/sahgw550p621m9dy1pd7whl9c5g1g0p7-lix-2.90.0-rc1/lib/liblixstore.so
8 0x00007f56b90e86d3 in execute_native_thread_routine () from /nix/store/c6r62m84hywf4i6qq1h28f13zv38yqyp-gcc-13.3.0-lib/lib/libstdc++.so.6
9 0x00007f56b8e99a42 in start_thread () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
10 0x00007f56b8f1905c in clone3 () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
As for testing, we've started a daemon with this change and verified it
deals with HUPs correctly on x86_64-linux, but I don't think we can
easily test the destructor behaviour without whatever Hydra was
doing that broke.
Change-Id: I29c7de0425674494b6e43c075810126c3ff77363
This also bans various sneaking of negative numbers from the language
into unsuspecting builtins as was exposed while auditing the
consequences of changing the Nix language integer type to a newtype.
It's unlikely that this change comprehensively ensures correctness when
passing integers out of the Nix language and we should probably add a
checked-narrowing function or something similar, but that's out of scope
for the immediate change.
During the development of this I found a few fun facts about the
language:
- You could overflow integers by converting from unsigned JSON values.
- You could overflow unsigned integers by converting negative numbers
into them when going into Nix config, into fetchTree, and into flake
inputs.
The flake inputs and Nix config cannot actually be tested properly
since they both ban thunks, however, we put in checks anyway because
it's possible these could somehow be used to do such shenanigans some
other way.
Note that Lix has banned Nix language integer overflows since the very
first public beta, but threw a SIGILL about them because we run with
-fsanitize=signed-overflow -fsanitize-undefined-trap-on-error in
production builds. Since the Nix language uses signed integers, overflow
was simply undefined behaviour, and since we defined that to trap, it
did.
Trapping on it was a bad UX, but we didn't even entirely notice
that we had done this at all until it was reported as a bug a couple of
months later (which is, to be fair, that flag working as intended), and
it's got enough production time that, aside from code that is IMHO buggy
(and which is, in any case, not in nixpkgs) such as
#445, we don't think
anyone doing anything reasonable actually depends on wrapping overflow.
Even for weird use cases such as doing funny bit crimes, it doesn't make
sense IMO to have wrapping behaviour, since two's complement arithmetic
overflow behaviour is so *aggressively* not what you want for *any* kind
of mathematics/algorithms. The Nix language exists for package
management, a domain where bit crimes are already only dubiously in
scope to begin with, and it makes a lot more sense for that domain for
the integers to never lose precision, either by throwing errors if they
would, or by being arbitrary-precision.
This change will be ported to CppNix as well, to maintain language
consistency.
Fixes: #423
Change-Id: I51f253840c4af2ea5422b8a420aa5fafbf8fae75
The actual motive here is the avoidance of integer overflow if we were
to make these use checked NixInts and retain the subtraction.
However, the actual *intent* of this code is a three-way comparison,
which can be done with operator<=>, so we should just do *that* instead.
Change-Id: I7f9a7da1f3176424b528af6d1b4f1591e4ab26bf
upcast_goal was only ever needed to break circular includes, but the
same solution that gave us upcast_goal also lets us fully remove it:
just upcast goals without a wrapper function, but only in .cc files.
Change-Id: I9c71654b2535121459ba7dcfd6c5da5606904032
this will let us turn copyNAR into a generator as well, which in turn is
necessary to turn the users of copyNAR into generators without resorting
to sinkToSource coroutines. currently this uses the SerializingTransform
in all cases, even for copyNAR where it is not necessary. should this be
a performance problem we can easily swap out the transform for one which
does not produce any bytes of its own, but that should not be necessary.
Change-Id: I7e685879318fcbb78d8b88abfddd7752360eb0ce
the sole remaining user of this function can use makeDecompressionSource
instead, while making the sinkToSource in the caller unnecessary as well
Change-Id: I4258227b5dbbb735a75b477d8a57007bfca305e9
the rewriting sink was just broken. when given a rewrite set that
contained a key that is also a proper infix of another key it was
possible to produce an incorrectly rewritten result if the writer
used the wrong block size. fixing this duplicates rewriteStrings,
to avoid this we'll rewrite rewriteStrings to use RewritingSource
in a new mode that'll allow rewrites we had previously forbidden.
Change-Id: I57fa0a9a994e654e11d07172b8e31d15f0b7e8c0
This updates the version of rnix used and refactors the code generally
to be more precise and capable in it's identification of both lambdas
and determining which documentation comments are attached.
Change-Id: Ib0dddabd71f772c95077f9d7654023b37a7a1fd2
`nix-collect-garbage --dry-run` previously elided the entire garbage
collection check, meaning that it would just exit the script without
printing anything.
This change makes the dry run flag instead set the GC action to
`gcReturnDead` rather than `gcDeleteDead`, and then continue with the
script. So if you set `--dry-run`, it will print the paths it *would*
have garbage collected, but not actually delete them.
I filed a bug for this: #432 but then realised I could give fixing it a go myself.
Change-Id: I062dbf1a80bbab192b5fd0b3a453a0b555ad16f2
DrvInfo's query methods all use mutable fields to cache, but like.
that's basically the entire interface for DrvInfo. Not only that, but
these formerly-const-marked functions can even throw due to eval errors!
Changing this only required removing some `const` markers in nix-env,
and changing a single inline `queryInstalled()` call to be an lvalue
instead.
Change-Id: I796807118f3b35b0e93668b5e28210d9e521b2ae
Activities can set display attributes in their log output using the "Select
Graphics Rendition" functionality. To prevent interfering with subsequent text
displayed, these should be reset after writing the log line. The multiline
progress bar neglected to do this, resulting for example in a colorised
"building …" header in the next line. Reset the attributes properly, like the
standard progress bar already does.
Change-Id: I1dc69f4a1d747a76b83e8721a72d9bb0e5554488
This rather simple function existed just to check some flags,
but the response varies by platform. This is a perfect case for
our subclasses.
Change-Id: Ieb1732a8d024019236e0d0028ad843a24ec3dc59
size tracking can be done with a LengthSink and a tee. match tracking
was defeated by never having done any match tracking, all users would
see the same (empty) set of matches at all times. match tracking with
bytes offsets alone would not be sufficient in the general case, only
because computeHashModulo uses a single rewrite could it have worked.
Change-Id: Idb214b5222e0ea24f450f5505712a342b63d7570
this much more closely mimics what is actually happening: we're reading
data from somewhere else, actively, rather than passively waiting. with
the data flow matching the underlying system interactions better we can
remove a few sinkToSource calls that merely exists to undo the mismatch
caused by not treating subprocess output as a data source to begin with
Change-Id: If4abfc2f8398fb5e88c9b91a8bdefd5504bb2d11
this will let us also return a source for the program output later,
which will in turn make sinkToSource unnecessary for program output
processing. this may also reopen a path for provigin program input,
but that still needs a proper async io framework to avoid problems.
Change-Id: Iaf93f47db99c38cfaf134bd60ed6a804d7ddf688
Turns errors like this:
let
throwMsg = a: throw (a + " invalid bar");
in throwMsg "bullshit"
error:
… from call site
at «string»:3:4:
2| throwMsg = a: throw (a + " invalid bar");
3| in throwMsg "bullshit"
| ^
… while calling 'throwMsg'
at «string»:2:14:
1| let
2| throwMsg = a: throw (a + " invalid bar");
| ^
3| in throwMsg "bullshit"
… while calling the 'throw' builtin
at «string»:2:17:
1| let
2| throwMsg = a: throw (a + " invalid bar");
| ^
3| in throwMsg "bullshit"
error: bullshit invalid bar
into errors like this:
let
throwMsg = a: throw (a + " invalid bar");
in throwMsg "bullshit"
error:
… from call site
at «string»:3:4:
2| throwMsg = a: throw (a + " invalid bar");
3| in throwMsg "bullshit"
| ^
… while calling 'throwMsg'
at «string»:2:14:
1| let
2| throwMsg = a: throw (a + " invalid bar");
| ^
3| in throwMsg "bullshit"
… caused by explicit throw
at «string»:2:17:
1| let
2| throwMsg = a: throw (a + " invalid bar");
| ^
3| in throwMsg "bullshit"
error: bullshit invalid bar
Change-Id: I593688928ece20f97999d1bf03b2b46d9ac338cb
Turns errors like:
let
somepkg.src = throw "invalid foobar";
in somepkg.src.meta
error:
… while evaluating the attribute 'src.meta'
at «string»:2:3:
1| let
2| somepkg.src = throw "invalid foobar";
| ^
3| in somepkg.src.meta
… while calling the 'throw' builtin
at «string»:2:17:
1| let
2| somepkg.src = throw "invalid foobar";
| ^
3| in somepkg.src.meta
error: invalid foobar
into errors like:
let
somepkg.src = throw "invalid foobar";
in somepkg.src.meta
error:
… while evaluating the attribute 'src.meta'
at «string»:2:3:
1| let
2| somepkg.src = throw "invalid foobar";
| ^
3| in somepkg.src.meta
… while evaluating 'somepkg.src' to select 'meta' on it
at «string»:3:4:
2| somepkg.src = throw "invalid foobar";
3| in somepkg.src.meta
| ^
… while calling the 'throw' builtin
at «string»:2:17:
1| let
2| somepkg.src = throw "invalid foobar";
| ^
3| in somepkg.src.meta
error: invalid foobar
And for type errors, from:
let
somepkg.src = "I'm not an attrset";
in somepkg.src.meta
error:
… while evaluating the attribute 'src.meta'
at «string»:2:3:
1| let
2| somepkg.src = "I'm not an attrset";
| ^
3| in somepkg.src.meta
… while selecting an attribute
at «string»:3:4:
2| somepkg.src = "I'm not an attrset";
3| in somepkg.src.meta
| ^
error: expected a set but found a string: "I'm not an attrset"
into:
let
somepkg.src = "I'm not an attrset";
in somepkg.src.meta
error:
… while evaluating the attribute 'src.meta'
at «string»:2:3:
1| let
2| somepkg.src = "I'm not an attrset";
| ^
3| in somepkg.src.meta
… while selecting 'meta' on 'somepkg.src'
at «string»:3:4:
2| somepkg.src = "I'm not an attrset";
3| in somepkg.src.meta
| ^
error: expected a set but found a string: "I'm not an attrset"
For the low price of an enumerate() and a lambda you too can have the
incorrect line of code actually show up in the trace!
Change-Id: Ic1491c86e33c167891bdac9adad6224784760bd6
Turns errors like:
let
errpkg = throw "invalid foobar";
in errpkg.meta
error:
… while calling the 'throw' builtin
at «string»:2:12:
1| let
2| errpkg = throw "invalid foobar";
| ^
3| in errpkg.meta
error: invalid foobar
into errors like:
let
errpkg = throw "invalid foobar";
in errpkg.meta
error:
… while evaluating 'errpkg' to select 'meta' on it
at «string»:3:4:
2| errpkg = throw "invalid foobar";
3| in errpkg.meta
| ^
… while calling the 'throw' builtin
at «string»:2:12:
1| let
2| errpkg = throw "invalid foobar";
| ^
3| in errpkg.meta
error: invalid foobar
For the low price of one try/catch, you too can have the incorrect line
of code actually show up in the trace!
Change-Id: If8d6200ec1567706669d405c34adcd7e2d2cd29d
Add a platform-specific function for starting sandboxed child.
Generally this just means startProcess, but on Linux we use flags
for clone to start a new namespace
Change-Id: I41c8aba62676a162388bbe5ab8a7518904c7b058
Add a new OS-specific hook called `prepareSandbox`, run before forking
On Darwin this is empty as nothing is required,
on Linux this creates the chroot directory and adds basic files,
and on platforms using a fallback this throws an exception
Change-Id: Ie30c38c387f2e0e5844b2afa32fd4d33b1180dae
generators are a better basis for serializers than streaming into sinks
as we do currently for many reasons, such as being usable as sources if
one wishes to (without requiring an intermediate sink to serialize full
data sets into memory, or boost coroutines to turn sinks into sources),
composing more naturally (as one can just yield a sub-generator instead
of being forced to wrap entire substreams into clunky functions or even
more clunky custom types to implement operator<< on), allowing wrappers
to transform data with clear ownership semantics (removing the need for
explicit memory allocations and Source wrappers), and many other things
Change-Id: I361d89ff556354f6930d9204f55117565f2f7f20
the `*Source` name is a slight misnomer since we do also have a
Source type, but we can probably live with this for time being.
Change-Id: I54eb2e59a4009014e324797f16b80b962759c7d3
not used anywhere yet, but we'll use this a lot soon for generators that
return file contents, wire protocol fragments, or indeed any byte stream
Change-Id: I01a46f9bf9d75aaf4a5d7662773b99f498862a28
this will be the basis of non-boost coroutines in lix. anything that is
a boost coroutine *should* be representable with a Generator coroutine,
and many things that are not currently boost coroutines but behave much
like one (such as, notably, serializers) should be as well. this allows
us to greatly simplify many things that look like iteration but aren't.
Change-Id: I2cebcefa0148b631fb30df4c8cfa92167a407e34
not printing activities at all when no progress information is available
hides *all* progress information from e.g. flake show. this is not ideal
and needs to be fixed, but the fix *still* has problems with flake show:
in multiline mode we will overwrite all useful flake show output as soon
as the progress bar is redrawn. flake show output is also mangled in any
number of other situations (like -v being set), so we should probably be
not too worried about it and fix progress reporting properly another day
Change-Id: I6d39d670e261bbae00560b6a8e15dec8e16b35c4
Previously, the progress bar had two subtly different states in which the bar
would not actually render, both with their own shortcomings: inactive (which
was irreversible) and paused (reversible, but swallowing logs). Furthermore,
there was no way of resetting the statistics, so a very bad solution was
implemented (243c0f18da) that would create a new
logger for each line of the repl, leaking the previous one and discarding the
value of printBuildLogs. Finally, if stderr was not attached to a TTY, the
update thread was started even though the logger was not active, violating the
invariant required by the destructor (which is not observed because the logger
is leaked).
In this commit, the two aforementioned states are unified into a single one,
which can be exited again, correctly upholds the invariant that the update
thread is only running while the progress bar is active, and does not swallow
logs. The latter change in behavior is not expected to be a problems in the
rare cases where the paused state was used before, since other loggers (like
the simple one) don't exhibit it anyway. The startProgressBar/stopProgressBar
API is removed due to being a footgun, and a new method for properly resetting
the progress is added.
Co-Authored-By: Qyriad <qyriad@qyriad.me>
Change-Id: I2b7c3eb17d439cd0c16f7b896cfb61239ac7ff3a
The `allow-flake-configuration` option allows the user to control whether to
accept configuration options supplied by flakes. Unfortunately, setting this
to false really meant "ask each time" (with an option to remember the choice
for each specific option encountered). Let no mean no, and introduce (and
default to) a separate value for the "ask each time" behaviour.
Co-Authored-By: Jade Lovelace <lix@jade.fyi>
Change-Id: I7ccd67a95bfc92cffc1ebdc972d243f5191cc1b4
We previously allowed you to map any flake URL to any other flake URL,
including shorthand flakerefs, indirect flake URLs like `flake:nixpkgs`,
direct flake URLs like `github:NixOS/nixpkgs`, or local paths.
But flake registry entries mapping from direct flake URLs often come
from swapping the 'from' and 'to' arguments by accident, and even when
created intentionally, they may not actually work correctly.
This patch rejects those URLs (and fully-qualified flake: URLs), making
it harder to swap the arguments by accident.
Fixes#181.
Change-Id: I24713643a534166c052719b8770a4edfcfdb8cf3
This is a shameless layering violation in favour of UX. It falls back
trivially to "unknown", so it's purely a UX feature.
Diagnostic sample:
```
error: hash mismatch in fixed-output derivation '/nix/store/sjfw324j4533lwnpmr5z4icpb85r63ai-x1.drv':
likely URL: https://meow.puppy.forge/puppy.tar.gz
specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
got: sha256-a1Qvp3FOOkWpL9kFHgugU1ok5UtRPSu+NwCZKbbaEro=
```
Change-Id: I873eedcf7984ab23f57a6754be00232b5cb5b02c
this most notably affects `nix eval`: if there is no progress bar to be
shown and no activities going on we should not print anything at all. a
progress bar with no activities would print a bunch of terminal escapes
*and a space*, which is not helpful in simple cases like nix eval -E 1.
notably this does *not* affect nix eval called on non-terminal outputs,
but it is slightly confusing nevertheless (and not difficult to avoid).
fixes #424
Change-Id: Iee793c79ba5a485d6606e0d292ed2eae6dfb7216
this gives about 20% performance improvements on pure parsing. obviously
it will be less on full eval, but depending on how much parsing is to be
done (e.g. including hackage-packages.nix or not) it's more like 4%-10%.
this has been tested (with thousands of core hours of fuzzing) to ensure
that the ASTs produced by the new parser are exactly the same as the old
one would have produced. error messages will change (sometimes by a lot)
and are not yet perfect, but we would rather leave this as is for later.
test results for running only the parser (excluding the variable binding
code) in a tight loop with inputs and parameters as given are promising:
- 40% faster on lix's package.nix at 10000 iterations
- 1.3% faster on nixpkgs all-packages.nix at 1000 iterations
- equivalent on all of nixpkgs concatenated at 100 iterations
(excluding invalid files, each file surrounded with parens)
more realistic benchmarks are somewhere in between the extremes, parsing
once again getting the largest uplift. other realistic workloads improve
by a few percentage points as well, notably system builds are 4% faster.
Benchmarks summary (from ./bench/summarize.jq bench/bench-*.json)
old/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
mean: 0.408s ± 0.025s
user: 0.355s | system: 0.033s
median: 0.389s
range: 0.388s ... 0.442s
relative: 1
new/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
mean: 0.332s ± 0.024s
user: 0.279s | system: 0.033s
median: 0.314s
range: 0.313s ... 0.361s
relative: 0.814
---
old/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 6.133s ± 0.022s
user: 5.395s | system: 0.437s
median: 6.128s
range: 6.099s ... 6.183s
relative: 1
new/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 5.925s ± 0.025s
user: 5.176s | system: 0.456s
median: 5.934s
range: 5.861s ... 5.943s
relative: 0.966
---
GC_INITIAL_HEAP_SIZE=10g old/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 4.503s ± 0.027s
user: 3.731s | system: 0.547s
median: 4.499s
range: 4.478s ... 4.541s
relative: 1
GC_INITIAL_HEAP_SIZE=10g new/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 4.285s ± 0.031s
user: 3.504s | system: 0.571s
median: 4.281s
range: 4.221s ... 4.328s
relative: 0.951
---
old/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
mean: 16.475s ± 0.07s
user: 14.088s | system: 1.572s
median: 16.495s
range: 16.351s ... 16.536s
relative: 1
new/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
mean: 15.973s ± 0.013s
user: 13.558s | system: 1.615s
median: 15.973s
range: 15.946s ... 15.99s
relative: 0.97
---
Change-Id: Ie66ec2d045dec964632c6541e25f8f0797319ee2
This reverts commit 35eec921af.
Reason for revert: Regressed nix-eval-jobs, and it appears to be this change is buggy/missing a case. It just needs another pass.
Code causing the problem in n-e-j, when invoked with `nix-eval-jobs --flake '.#hydraJobs'`:
```
n-e-j/tests/assets » ../../build/src/nix-eval-jobs --meta --workers 1 --flake .#hydraJobs
warning: unknown setting 'trusted-users'
warning: `--gc-roots-dir' not specified
error: unsupported Git input attribute 'dir'
error: worker error: error: unsupported Git input attribute 'dir'
```
```
nix::Value *vRoot = [&]() {
if (args.flake) {
auto [flakeRef, fragment, outputSpec] =
nix::parseFlakeRefWithFragmentAndExtendedOutputsSpec(
args.releaseExpr, nix::absPath("."));
nix::InstallableFlake flake{
{}, state, std::move(flakeRef), fragment, outputSpec,
{}, {}, args.lockFlags};
return flake.toValue(*state).first;
} else {
return releaseExprTopLevelValue(*state, autoArgs, args);
}
}();
```
Inspecting the program behaviour reveals that `dir` was in fact set in the URL going into the fetcher. This is in turn because unlike in the case changed in this commit, it was not erased before handing it to libfetchers, which is probably just a mistake.
```
(rr) up
3 0x00007ffff60262ae in nix::fetchers::Input::fromURL (url=..., requireTree=requireTree@entry=true) at src/libfetchers/fetchers.cc:39
warning: Source file is more recent than executable.
39 auto res = inputScheme->inputFromURL(url, requireTree);
(rr) p url
$1 = (const nix::ParsedURL &) @0x7fffdc874190: {url = "git+file:///home/jade/lix/nix-eval-jobs",
base = "git+file:///home/jade/lix/nix-eval-jobs", scheme = "git+file", authority = std::optional<std::string> = {[contained value] = ""},
path = "/home/jade/lix/nix-eval-jobs", query = std::map with 1 element = {["dir"] = "tests/assets"}, fragment = ""}
(rr) up
4 0x00007ffff789d904 in nix::parseFlakeRefWithFragment (url=".#hydraJobs", baseDir=std::optional<std::string> = {...},
allowMissing=allowMissing@entry=false, isFlake=isFlake@entry=true) at src/libexpr/flake/flakeref.cc:179
warning: Source file is more recent than executable.
179 FlakeRef(Input::fromURL(parsedURL, isFlake), getOr(parsedURL.query, "dir", "")),
(rr) p parsedURL
$2 = {url = "git+file:///home/jade/lix/nix-eval-jobs", base = "git+file:///home/jade/lix/nix-eval-jobs", scheme = "git+file",
authority = std::optional<std::string> = {[contained value] = ""}, path = "/home/jade/lix/nix-eval-jobs", query = std::map with 1 element = {
["dir"] = "tests/assets"}, fragment = ""}
(rr) list
174
175 if (pathExists(flakeRoot + "/.git/shallow"))
176 parsedURL.query.insert_or_assign("shallow", "1");
177
178 return std::make_pair(
179 FlakeRef(Input::fromURL(parsedURL, isFlake), getOr(parsedURL.query, "dir", "")),
180 fragment);
181 }
```
Change-Id: Ib55a882eaeb3e59228857761dc1e3b2e366b0f5e
This is a squash of upstream PRs #10303, #10312 and #10883.
fix: Treat empty TMPDIR as unset
Fixes an instance of
nix: src/libutil/util.cc:139: nix::Path nix::canonPath(PathView, bool): Assertion `path != ""' failed.
... which I've been getting in one of my shells for some reason.
I have yet to find out why TMPDIR was empty, but it's no reason for
Nix to break.
(cherry picked from commit c3fb2aa1f9d1fa756dac38d3588c836c5a5395dc)
fix: Treat empty XDG_RUNTIME_DIR as unset
See preceding commit. Not observed in the wild, but is sensible
and consistent with TMPDIR behavior.
(cherry picked from commit b9e7f5aa2df3f0e223f5c44b8089cbf9b81be691)
local-derivation-goal.cc: Reuse defaultTempDir()
(cherry picked from commit fd31945742710984de22805ee8d97fbd83c3f8eb)
fix: remove usage of XDG_RUNTIME_DIR for TMP
(cherry picked from commit 1363f51bcb24ab9948b7b5093490a009947f7453)
tests/functional: Add count()
(cherry picked from commit 6221770c9de4d28137206bdcd1a67eea12e1e499)
Remove uncalled for message
(cherry picked from commit b1fe388d33530f0157dcf9f461348b61eda13228)
Add build-dir setting
(cherry picked from commit 8b16cced18925aa612049d08d5e78eccbf0530e4)
Change-Id: Ic7b75ff0b6a3b19e50a4ac8ff2d70f15c683c16a
Following the latest hacking.md currently fails because of a missing
include in upstream editline. This patch fixes the build by adding
that missing include.
Fixes#410.
Change-Id: Iefd4cb687ed3da71ccda9fe9624f38e6ef4623e5
this was only used in one place, and that place has been rewritten to
use a temporary file instead. keeping this around is not very helpful
at this time, and in any case we'd be better off rewriting subprocess
handling in rust where we not only have a much safer library for such
things but also async frameworks necessary for this easily available.
Change-Id: I6f8641b756857c84ae2602cdf41f74ee7a1fda02
we want to remove runProgram's ability to provide stdin to a process
because the concurrency issues of handling both stdin and stdout are
much more pronounced once runProgram returns not is collected output
but a source. this is possible in the current c++ framework, however
it isn't necessary in almost all cases (as demonstrated by only this
single user existing) and in much better handled with a proper async
concurrency model that lets the caller handle both at the same time.
Change-Id: I29da1e1ad898d45e2e33a7320b246d5003e7712b
This is primarily for readability, but iwrc chaining std::string's
operator+ is also pretty scuffed performance-wise, and this was doing a
lot of operator+ chainging.
Change-Id: I9f25235df153eb2bbb491f1ff7ebbeed9a8ec985
jade
Pierre Bourdon
eldritch horrors
sugar🍬🍭🏳️⚧️
alois31
piegames
Qyriad
Maximilian Bosch
Artemis Tosini
Max “Goldstein” Siling
Isabel
Winter Cute
nan-git
Lunaphied
Quantum Jump
Delan Azabani
Robert Hensing