Commit graph

9388 commits

Author SHA1 Message Date
eldritch horrors 620de98d0c libutil: drop Pool resources on exceptional free
if a scope owning a resource does not gracefully drop that resource
while handling exceptions from deeper down the call stack we should
assume the resource is invalid state and drop it. currently it *is*
true that such cases do not cause resources to be freed, but thanks
to validator misuses this has so far not caused any larger problem.

Change-Id: Ie4f91bcd60a64d05c5ff9d22cc97954816d13b97
2024-03-30 23:40:45 +00:00
Rebecca Turner 4b730f328e Merge "Add pre-commit checks" into main 2024-03-30 22:26:59 +00:00
eldritch horrors 5956f509b6 meson: install libexec binaries
Change-Id: I149892bf081e1569d7786f085e890bc3d2eb50e5
2024-03-30 14:34:22 +00:00
Rebecca Turner 2a98ba8b97 Add pre-commit checks
The big ones here are `trim-trailing-whitespace` and `end-of-file-fixer`
(which makes sure that every file ends with exactly one newline
character).

Change-Id: Idca73b640883188f068f9903e013cf0d82aa1123
2024-03-29 22:57:40 -07:00
jade a4f5bb951d Merge "Document ExprLambda fields" into main 2024-03-30 03:22:53 +00:00
Jade Lovelace 817322fbfb Document ExprLambda fields
We got confused what formals did and had to briefly figure it out. We
should just have docs, so these are some.

Change-Id: If3e794a401e69d022785cbfa0b0c2e2284f41f58
2024-03-29 18:29:59 -07:00
Rebecca Turner a5a25894c1 Move escapeString to its own file
Change-Id: Ie5c954ec73c46c9d3c679ef99a83a29cc7a08352
2024-03-29 16:26:29 -07:00
eldritch horrors 1da1f501fc meson: fix state-dir default value
the autoconf build system defaults to /nix/var, not /nix/var/nix. the
latter is only used in libstore, so we'll move the extra segment there.

Change-Id: Idfbc988ee302355982abdcd51d6d7b5d5d661c0d
2024-03-29 19:14:23 +00:00
Winter Cute 6646b80396 meson: add missing explicit dependency on nlohmann_json
Without this, the Meson setup won't bail out if nlohmann_json is
missing, leading to subpar DX (and maybe worse, but I'm not entirely
sure).

Change-Id: I5913111060226b540dcf003257c99a08e84da0de
2024-03-29 14:16:58 -04:00
Rebecca Turner 877750b7c5 Merge "Move DebugChar into its own file" into main 2024-03-29 16:20:14 +00:00
eldritch horrors 6e5db5e4a2 meson: install missing/generated headers
one headers (args/root.hh) was simply missing, and the generated headers
were not installed. not all of them *should* be installed either, only a
select few (and sadly this needs a custom target for each one, it seems)

Change-Id: I37b25517895d0e5e521abc1202fa65624de57ed1
2024-03-29 02:45:48 +00:00
eldritch horrors 69bfd21e20 meson: install pkg-config files for libraries
Change-Id: I14b9d81d09f188eacfb9c68bcfb84751c18e3779
2024-03-29 02:45:48 +00:00
Rebecca Turner 236bc046ba Merge "Remove HintFmt::operator%" into main 2024-03-29 01:13:45 +00:00
raito 55350bd68d Merge "feat: unprivileged read-only open of SQLite DB" into main 2024-03-29 00:49:17 +00:00
Rebecca Turner 5ec2efb686 Move DebugChar into its own file
Change-Id: Ia40549e5d0b78ece8dd0722c3a5a032b9915f24b
2024-03-28 15:54:12 -07:00
Rebecca Turner 62332c1250 Merge "Move shell_words into its own file" into main 2024-03-28 22:49:00 +00:00
Qyriad 81e50fef70 Merge "meson: implement functional tests" into main 2024-03-28 20:38:05 +00:00
jade ae065a992d Merge "progress-bar.cc: fix signed overflow" into main 2024-03-28 15:21:11 +00:00
jade ffbad9b762 progress-bar.cc: fix signed overflow
this was caused by the use of std::chrono::duration::max() which gets
multiplied by some ratio to calculate nanoseconds to wait. then, it
explodes because that is a signed integer overflow. this was definitely
a bug.

error below:

/nix/store/fdiknsmnnczx6brsbppyljcs9hqckawk-gcc-12.3.0/include/c++/12.3.0/bits/chrono.h:225:38: runtime error: signed integer overflow: 9223372036854775807 * 1000000 cannot be represented in type 'long'
    #0 0x736d376b2b69 in std::chrono::duration<long, std::ratio<1l, 1000000000l>> std::chrono::__duration_cast_impl<std::chrono:
:duration<long, std::ratio<1l, 1000000000l>>, std::ratio<1000000l, 1l>, long, false, true>::__cast<long, std::ratio<1l, 1000l>>(
std::chrono::duration<long, std::ratio<1l, 1000l>> const&) /nix/store/fdiknsmnnczx6brsbppyljcs9hqckawk-gcc-12.3.0/include/c++/12
.3.0/bits/chrono.h:225:38
    #1 0x736d376b2b69 in std::enable_if<__is_duration<std::chrono::duration<long, std::ratio<1l, 1000000000l>>>::value, std::chr
ono::duration<long, std::ratio<1l, 1000000000l>>>::type std::chrono::duration_cast<std::chrono::duration<long, std::ratio<1l, 10
00000000l>>, long, std::ratio<1l, 1000l>>(std::chrono::duration<long, std::ratio<1l, 1000l>> const&) /nix/store/fdiknsmnnczx6brs
bppyljcs9hqckawk-gcc-12.3.0/include/c++/12.3.0/bits/chrono.h:270:9
    #2 0x736d376b2b69 in std::enable_if<__is_duration<std::chrono::duration<long, std::ratio<1l, 1000000000l>>>::value, std::chr
ono::duration<long, std::ratio<1l, 1000000000l>>>::type std::chrono::ceil<std::chrono::duration<long, std::ratio<1l, 1000000000l
>>, long, std::ratio<1l, 1000l>>(std::chrono::duration<long, std::ratio<1l, 1000l>> const&) /nix/store/fdiknsmnnczx6brsbppyljcs9
hqckawk-gcc-12.3.0/include/c++/12.3.0/bits/chrono.h:386:14
    #3 0x736d376b2b69 in std::cv_status std::condition_variable::wait_for<long, std::ratio<1l, 1000l>>(std::unique_lock<std::mut
ex>&, std::chrono::duration<long, std::ratio<1l, 1000l>> const&) /nix/store/fdiknsmnnczx6brsbppyljcs9hqckawk-gcc-12.3.0/include/
c++/12.3.0/condition_variable:164:6
    #4 0x736d376b1ee9 in std::cv_status nix::Sync<nix::ProgressBar::State, std::mutex>::Lock::wait_for<long, std::ratio<1l, 1000
l>>(std::condition_variable&, std::chrono::duration<long, std::ratio<1l, 1000l>> const&) /home/jade/lix/lix/src/libutil/sync.hh:
65:23
    #5 0x736d376b1ee9 in nix::ProgressBar::ProgressBar(bool)::'lambda'()::operator()() const /home/jade/lix/lix/src/libmain/prog
ress-bar.cc:99:27
    #6 0x736d36de25c2 in execute_native_thread_routine (/nix/store/a3zlvnswi1p8cg7i9w4lpnvaankc7dxx-gcc-12.3.0-lib/lib/libstdc++
.so.6+0xe05c2)
    #7 0x736d36b6b0e3 in start_thread (/nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libc.so.6+0x8b0e3) (BuildId
: 287831bffdbdde0ec25dbd021d12bdfc0ab9f5ff)
    #8 0x736d36bed5e3 in __clone (/nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libc.so.6+0x10d5e3) (BuildId: 28
7831bffdbdde0ec25dbd021d12bdfc0ab9f5ff)

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /nix/store/fdiknsmnnczx6brsbppyljcs9hqckawk-gcc-12.3.0/include/c++/12.3.
0/bits/chrono.h:225:38 in

Change-Id: Ia0303242cdfd5d49385ae9e99718d709625a4633
2024-03-27 22:56:04 -07:00
Winter Cute 80405d0626 Stop vendoring toml11
We don't apply any patches to it, and vendoring it locks users into
bugs (it hasn't been updated since its introduction in late 2021).

Closes lix-project/lix#164

Change-Id: Ied071c841fc30b0dfb575151afd1e7f66970fdb9
2024-03-27 21:04:00 -04:00
Qyriad 038daad218 meson: implement functional tests
Functional tests can be run with
`meson test -C build --suite installcheck`.

Notably, functional tests must be run *after* running `meson install`
(Lix's derivation runs the installcheck suite in installCheckPhase so it
does this correctly), due to some quirks between Meson and the testing
system.

As far as I can tell the functional tests are meant to be run after
installing anyway, but unfortunately I can't transparently make
`meson test --suite installcheck` depend on the install targets.

The script that runs the functional tests, meson/run-test.py, checks
that `meson install` has happened and fails fast with a (hopefully)
helpful error message if any of the functional tests are run before
installing.

TODO: this change needs reflection in developer documentation

Change-Id: I8dcb5fdfc0b6cb17580973d24ad930abd57018f6
2024-03-27 18:37:50 -06:00
jade edba570664 HOT SALE: 15% off your build times!
This was achieved by running maintainers/buildtime_report.sh on the
build directory of a meson build, then asking "why the heck is json
eating our build times", and strategically moving the json using bits
out of widely included headers.

It turns out that putting literally any metrics whatsoever into the
build had immediate and predictable results.

Results are 1382.5s frontend time -> 1175.4s frontend time, back end
time approximately invariant.

Related: lix-project/lix#159

Change-Id: I7edea95c8536203325c8bb4dae5f32d727a21b2d
2024-03-27 03:52:57 +00:00
Rebecca Turner aee3d639b5 Move shell_words into its own file
Change-Id: I34c0ebfb6dcea49bf632d8880e04075335a132bf
2024-03-26 16:44:04 -07:00
Rebecca Turner 8e63eca912 Remove HintFmt::operator%
Change-Id: Ibcf1a7848b4b18ec9b0807628ff229079ae7a0fe
2024-03-26 15:40:05 -07:00
raito 80b66b5065 libstore/filetransfer: use Lix UA and unnix error message
Once this commit lands, we are even more visible in analytics FWIW.

Change-Id: Id7e0c162315d0f191edbea9cb5fb82ce363704b9
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-03-26 16:06:27 +00:00
jade 531b8d0ab8 Merge "libmain: version printer uses Lix instead of Nix" into main 2024-03-26 16:06:15 +00:00
Ilya K a69f6e185a build-remote: fix format string shenanigans
HintFmt(string) invokes the HintFmt("%s", literal) constructor,
which is not what we want here. Add a constructor with a proper name
and call that.

Next step: rename all the other ones to HintFmt::literal(string).

Fixes: lix-project/lix#178

Change-Id: If52d2eb8864ceb8663e05992e9d1fffef573d6b8
2024-03-26 07:58:24 +00:00
Théophane Hufschmitt 86881226b0 Merge pull request #8817 from iFreilicht/flake-update-lock-overhaul
Overhaul `nix flake update` and `nix flake lock` UX

(cherry picked from commit 12a0ae73dbb37becefa5a442eb4532ff0de9ce65)
Change-Id: Iff3b4f4235ebb1948ec612036b39ab29e4ca22b2
2024-03-25 17:36:24 -06:00
Eelco Dolstra aa7653608d Minor cleanup in libexpr/flake/flake.cc
(cherry picked from commit 05316d401fa509557c71140e17bb19814412fcb8)
Change-Id: I6ba0b55709f5fe21beb4e9f3bf72ee28715d15f3
2024-03-25 15:30:36 +00:00
Eelco Dolstra b525d0f20c Input: Replace markFileChanged() by putFile()
Committing a lock file using markFileChanged() required the input to
be writable by the caller in the local filesystem (using the path
returned by getSourcePath()). putFile() abstracts over this.

(cherry picked from commit 95d657c8b3ae4282e24628ba7426edb90c8f3942)
Change-Id: Ie081c5d9eb4e923b229191c5e23ece85145557ff
2024-03-25 15:30:36 +00:00
John Ericson 3d065192c0 Overhaul completions, redo #6693 (#8131)
As I complained in
https://github.com/NixOS/nix/pull/6784#issuecomment-1421777030 (a
comment on the wrong PR, sorry again!), #6693 introduced a second
completions mechanism to fix a bug. Having two completion mechanisms
isn't so nice.

As @thufschmitt also pointed out, it was a bummer to go from `FlakeRef`
to `std::string` when collecting flake refs. Now it is `FlakeRefs`
again.

The underlying issue that sought to work around was that completion of
arguments not at the end can still benefit from the information from
latter arguments.

To fix this better, we rip out that change and simply defer all
completion processing until after all the (regular, already-complete)
arguments have been passed.

In addition, I noticed the original completion logic used some global
variables. I do not like global variables, because even if they save
lines of code, they also obfuscate the architecture of the code.

I got rid of them  moved them to a new `RootArgs` class, which now has
`parseCmdline` instead of `Args`. The idea is that we have many argument
parsers from subcommands and what-not, but only one root args that owns
the other per actual parsing invocation. The state that was global is
now part of the root args instead.

This did, admittedly, add a bunch of new code. And I do feel bad about
that. So I went and added a lot of API docs to try to at least make the
current state of things clear to the next person.

--

This is needed for RFC 134 (tracking issue #7868). It was very hard to
modularize `Installable` parsing when there were two completion
arguments. I wouldn't go as far as to say it is *easy* now, but at least
it is less hard (and the completions test finally passed).

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Change-Id: If18cd5be78da4a70635e3fdcac6326dbfeea71a5
(cherry picked from commit 67eb37c1d0de28160cd25376e51d1ec1b1c8305b)
2024-03-25 15:30:36 +00:00
Tom Bereknyei 4494f9097f feat: notation to refer to no attribute search prefix
An attrPath prefix of "." indicates no need to try default attrPath prefixes. For example `nixpkgs#legacyPackages.x86_64-linux.ERROR` searches through

```
trying flake output attribute 'packages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute ''
trying flake output attribute 'legacyPackages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
trying flake output attribute 'legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
```

And there is no way to specify that one does not want the automatic
search behavior. Now one can specify
`nixpkgs#.legacyPackages.x86_64-linux.ERROR` to only refer to the rooted
attribute path without any default injection of attribute search path or
system.

Change-Id: Iac1334e1470137b7ce11dcf845513810230638ec
(cherry picked from commit d4aed18883b361133607296fb6cd789c47427a38)
2024-03-25 15:30:36 +00:00
Lunaphied d3d7489571 Merge "Improve new CLI UX by supporting short -E flag for --expr" into main 2024-03-25 14:13:44 +00:00
raito ad8a4b380e libmain: version printer uses Lix instead of Nix
Change-Id: I014ff24b900c0b9a48b7a63c8bb8b86cde3ebe54
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-03-25 08:04:31 +00:00
Lunaphied 185ecf1f45 Improve new CLI UX by supporting short -E flag for --expr
Change-Id: I55881c846da8416a92a14deedfa5bbbf09a122fb
2024-03-24 21:17:51 -06:00
eldritch horrors c856b82c2e libstore: despecialcase protocol version check
protocol versions are sent as u64. on the peer we read them as uint64,
check that the upper half is 0, and throw an exception if not. we then
read an arbitrary amount of data from the peer and dump it to the user
terminal. this is a little bit ridiculous, can never happen in correct
implementation, and is severly untested. let us just drop it entirely.

Change-Id: Ibd2f53a765341ed6439d40d9d1eac11e79c6b5e3
2024-03-24 18:45:22 +00:00
eldritch horrors 3e428f2289 libstore: un-inline copyNAR expansions
these are copies of copyNAR with only some variables renamed.

Change-Id: I98ddd7a98250fa5d304e18e1debf417e9f7768dd
2024-03-24 15:24:02 +01:00
jade 946fc12e4e Revert "Merge pull request #9476 from alois31/restore-progress-bar"
Observed to regress nix repl attrset printing with narrow windows.

This reverts commit a2d5e803cf.

Fixes: lix-project/lix#168

Change-Id: I8e0031475b4ec26d6a71014357d973578b70815c
2024-03-23 18:04:29 -07:00
eldritch horrors 652f52f071 libutil: don't memset 64k in drainFD
this is not needed and introduces a bunch of memset calls, making up for
3% of valgrind cycle estimation *alone*. real-world impact is a lot
lower on our test machine, but we suspect that less powerful machines
would see an impact from dropping this.

Change-Id: Iad10e9d556e64fdeb0bee0059a4e52520058d11e
2024-03-23 22:17:46 +00:00
raito 8044540c42 feat: unprivileged read-only open of SQLite DB
If the state SQLite database is configured to use a write-ahead-log, it
creates WAL files in the state directory.

When the state SQLite database is closed by the `nix-daemon` after
builds, those files are removed.

When an unprivileged user would like to open _in read only_ that
database, they cannot do so because they would need to create those WAL
files and they do not have the permission to do so.

For this, SQLite offers a "persistent WAL" feature [1] to leave the WAL
files around, even after closing the database.

This CL enable the persistent WAL mode.

Fixes: https://github.com/NixOS/nix/issues/10300
[1]: https://www.sqlite.org/wal.html

Change-Id: Id8ae534d7d2290457af28782e5215222ae051fe5
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-03-23 15:07:48 +01:00
Qyriad b4d07656ff build: optionally build and install with meson
This commit adds several meson.build, which successfully build and
install Lix executables, libraries, and headers. Meson does not yet
build docs, Perl bindings, or run tests, which will be added in
following commits. As such, this commit does not remove the existing
build system, or make it the default, and also as such, this commit has
several FIXMEs and TODOs as notes for what should be done before the
existing autoconf + make buildsystem can be removed and Meson made the
default. This commit does not modify any source files.

A Meson-enabled build is also added as a Hydra job, and to
`nix flake check`.

Change-Id: I667c8685b13b7bab91e281053f807a11616ae3d4
2024-03-22 08:36:50 -06:00
eldritch horrors 22e3f0e987 libexpr: unbreak PosTable performance
this was mostly an inconvenience for error reporting, but fully broke
the debugger (because the debugger does *a lot* of eager position
resolution). copying the line offsets into a local and filling that
local when empty without also storing the calculated offsets back does
kind of ... not cache anything.

fixes lix-project/lix#165

Change-Id: Iccb0ba193ce2f15c832978daecf7b9bebbbe8585
2024-03-20 13:45:36 +01:00
eldritch horrors d9a83886f9 libutil: remove exception handling workingness check
within lix itself this problem is caught by the test suite. outside of
lix itself three cases can be had: either the problem is fully inside
lix libs, fully inside user code, or it exists at the boundary. the
first is caught by the test suite, the second isn't caught at all, and
the third is something lix should not be responsible for.

Change-Id: I95aa35d8cb6f0ef5816a2941c467bc0c15916063
2024-03-19 06:09:42 -06:00
jade 4050245faa Merge changes I72c945ca,I2138bb4d,Ib96749f3 into main
* changes:
  Release notes for builtins.nixVersion change
  un-nixes ur lix, a little
  issue importer: list issues that are *not* closed when finding existing issues
2024-03-18 20:19:53 -06:00
jade 30233d87f9 un-nixes ur lix, a little
I didn't really go attack the docs because we need to pull a bunch of
PRs. I went looking for strings in the code that called lix nix.

Change-Id: I2138bb4dd239096bc530946b281db7f875195b39
2024-03-18 18:20:24 -07:00
eldritch horrors f38ae92a38 libutil: make AutoCloseFD a better resource
add a reset() method to close the wrapped fd instead of assigning magic
constants. also make the from-fd constructor explicit so you can't
accidentally assign the *wrong* magic constant, or even an unrelated
integer that also just happens to be an fd by pure chance.

Change-Id: I51311b0f6e040240886b5103d39d1794a6acc325
2024-03-18 15:42:52 -06:00
eldritch horrors afb839a0c9 libexpr: associate let exprs with the correct StaticEnv
static env association is from expr to its enclosing scope, but let
exprs set their association to their *inner* scope. this skips one level
of envs and will cause segfaults if the parent is a with expr.

fixes #145

Change-Id: I1d22146110f071ede21b4eed7ed34b5850ef2ef3
2024-03-18 14:15:22 -07:00
eldritch horrors b3599166ad libexpr: sort binding name in debugger
not doing this exposes the binding name order to the annoying
interference of parse order on symbol order, which wouldn't be so bad if
it didn't make the tests less reliable and, importantly, dependent on
linker behavior (due to primop initialization being done in static
initializer, and the order of static initializers being defined only
within a single translation unit).

fixes #143

Change-Id: I3cf417893fbcf19e9ad3ff8986deb7cbcf3ca511
2024-03-18 20:03:31 +01:00
jade 47a237f7ec Merge "Delete hasPrefix and hasSuffix from the codebase" into main 2024-03-18 12:01:39 -06:00
eldritch horrors 86a1121d16 use byte indexed locations for PosIdx
we now keep not a table of all positions, but a table of all origins and
their sizes. position indices are now direct pointers into the virtual
concatenation of all parsed contents. this slightly reduces memory usage
and time spent in the parser, at the cost of not being able to report
positions if the total input size exceeds 4GiB. this limit is not unique
to nix though, rustc and clang also limit their input to 4GiB (although
at least clang refuses to process inputs that are larger, we will not).

this new 4GiB limit probably will not cause any problems for quite a
while, all of nixpkgs together is less than 100MiB in size and already
needs over 700MiB of memory and multiple seconds just to parse. 4GiB
worth of input will easily take multiple minutes and over 30GiB of
memory without even evaluating anything. if problems *do* arise we can
probably recover the old table-based system by adding some tracking to
Pos::Origin (or increasing the size of PosIdx outright), but for time
being this looks like more complexity than it's worth.

since we now need to read the entire input again to determine the
line/column of a position we'll make unsafeGetAttrPos slightly lazy:
mostly the set it returns is only used to determine the file of origin
of an attribute, not its exact location. the thunks do not add
measurable runtime overhead.

notably this change is necessary to allow changing the parser since
apparently nothing supports nix's very idiosyncratic line ending choice
of "anything goes", making it very hard to calculate line/column
positions in the parser (while byte offsets are very easy).

(cherry picked from commit 5d9fdab3de0ee17c71369ad05806b9ea06dfceda)
Change-Id: Ie0b2430cb120c09097afa8c0101884d94f4bbf34
2024-03-18 16:12:46 +01:00