Compare commits

...

1230 commits

Author SHA1 Message Date
Linus Heckemann c95b73d8a1 Merge "libstore: report all differing outputs rather than just the first" into main 2024-10-30 19:04:57 +00:00
V. 56ead73fda Merge "chore: remove monolithic coreutils requirement" into main 2024-10-30 16:53:43 +00:00
Dusk Banks 22eb47f0fd tests/functional/flakes: test with UTF-8 bullets
using UTF-8 bullets in the sample avoids locale confusion where Bash
doesn't know to treat `•` as a single character.

Signed-off-by: Dusk Banks <me@bb010g.com>
Change-Id: I829019b66e93e6d33ac3a6641df07d0dd2332a5a
2024-10-30 08:21:58 -07:00
Dusk Banks 8b2f8d538b Merge "libstore: restore mode after changing xattrs" into main 2024-10-30 14:56:43 +00:00
V. fb1b211037 chore: remove monolithic coreutils requirement
It's only used in a couple of tests, and only in such a way that
replacing it with a random command suffices.
I also removed a few pointless uses of the variable.

Fixes: lix-project/lix#376
Change-Id: I90aedb61d64b02f7c9b007e72f9d614cc1b37a2e
2024-10-30 15:12:35 +04:00
Linus Heckemann 8b0ac51f12 libstore: report all differing outputs rather than just the first
Before:

error: derivation '/nix/store/4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'

After:

error: derivation '4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: outputs differ
         output differs: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'
         output differs: output '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0' differs from '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0.check'

Change-Id: Ib2871fa602bf1fa9c00e2565b3a2e1b26f908152
2024-10-29 18:34:54 +01:00
eldritch horrors 9c22a4d31b libstore: don't use curl for file:// downloads
curl can't pause downloads of file:// urls, which is very much in the
way of making the curl wrapper fully asynchronous. we can emulate all
the things curl does, but unfortunately curl is *rather extensive* in
its support of frankly weird shit. hopefully this subset of features,
which notably does not include curl readdir support, is enough for us

Change-Id: I5f67768c4b512565655b94b0421270c7dbbd8d11
2024-10-28 18:52:49 +00:00
eldritch horrors c83b13eafd libstore: reunify all file transfer methods again
with the api cleaned up we can suddenly reunify uploads, downloads, and
existence checks through curl in the same wrapper function. uploads and
existence checks simply don't use the result source, and given that all
transfers (or at least *most* transfers to date) go through the network
the few extra allocations do not hurt us at all. even for file:// calls
the overhead won't much matter as going to disk and back *is* expensive

Change-Id: I4f9ca6681a8fc303377b4cf4c63e3363ae32c18b
2024-10-28 18:52:49 +00:00
eldritch horrors d65838a900 libstore: remove FileTransfer::enqueueDownload
it's no longer needed. `download` can do everything `enqueueDownload`
did, and a lot more. e.g. not block the calling thread, for instance.

Change-Id: I4b36235ed707c92d117b4c33efa3db50d26f9a84
2024-10-28 18:52:49 +00:00
eldritch horrors c68f0cdf00 libstore: return transfer metadata from download
as promised earlier. nothing uses it yet, but just you wait.

Change-Id: I77d185578d96c2134b756d20f2fcf1c02de0da6f
2024-10-28 18:52:49 +00:00
eldritch horrors 14eff10fe4 libstore: split callback into metadata and finished parts
this will let us return metadata from FileTransfer::download, which in
turn is necessary to remove enqueueDownload. it also opens avenues for
streaming downloads that keep download metadata instead of dropping it

Change-Id: If0fc6af5eb2aeb689fc866c345c9d7bce4d59f2d
2024-10-28 18:52:49 +00:00
eldritch horrors 923abe347c libstore: use data callback for simple downloads too
this is an intermediate step towards removing enqueueDownload entirely.

Change-Id: I05ec0c7f4a234fdc966e5005308b37f6f905d433
2024-10-28 18:52:49 +00:00
eldritch horrors 64864c3730 libstore: pass only data to TransferItem data callback
with encoding being handled by curl the reference is no longer needed.

Change-Id: Ibfaf5f55e5314e81ce45ba4523b960c401dd2e1c
2024-10-28 18:52:49 +00:00
eldritch horrors 10488f7431 libstore: use curl content-encoding support, not our own
let's use the automatic decoding functions curl provides instead of
implementing them ourselves for the dubious ability to support both
xz and bzip2 encodings as well, neither of which anything will send

Change-Id: I3edfebeb596a0e9d5c986efca9270501c996f2dd
2024-10-28 18:52:49 +00:00
Justin ! 8c567c0424 Merge "libutil: implement PathsSetting<PathSet>" into main 2024-10-28 14:22:59 +00:00
eldritch horrors 61146c73ce Merge changes I0220cedd,Ide0c0512,I6fcd920e,I85ec62ff,I35853a91, ... into main
* changes:
  libstore: check that transfer headers don't change during retries
  libstore: use effective transfer url for retries
  libstore: collect effective url and cachedness earlier
  libstore: remove TransferItem::active
  libstore: always allocate TransferItem::req
  libstore: remove FileTransferResult::data
  libstore: de-future-ize FileTransfer::enqueueUpload
  libstore: remove FileTransferRequest
  libstore: remove FileTransferRequest::expectedETag
  libstore: remove FileTransferResult::bodySize
  libstore: remove FileTransferRequest::verifyTLS
  libstore: remove FiletransferRequest::head
2024-10-28 01:36:45 +00:00
Linus Heckemann f55ed83991 Merge "libutil: handle json builder log messages with unexpected format" into main 2024-10-27 22:54:11 +00:00
eldritch horrors 212a14bb1f libstore: check that transfer headers don't change during retries
etag changing implies with high probability that the content of the
resource changed. immutable url changing implies that the immutable
url we got previously was wrong, which is probably a server bug. if
the encoding changes our decoding will break completely, so that is
also very illegal. one notable change we still allow is etags going
away completely, mostly since this does not imply any data changes.

Change-Id: I0220ceddc3fd732cd1b3bb39b40021cc631baadc
2024-10-27 21:44:38 +00:00
eldritch horrors 7c716b9716 libstore: use effective transfer url for retries
do not retread the entire redirection path if we've seen the end of the
road. this avoids silently downloading wrong data, and notifies us when
a url we've received data from turns into a redirect when retrying. for
reasons of simplicity we don't turn of libcurl redirects on retries. if
we did that we'd have to conditionally process http status codes, which
sounds annoying and would make the header callback even more of a mess.

Change-Id: Ide0c0512ef9b2579350101246d654a2375541a39
2024-10-27 21:44:38 +00:00
eldritch horrors 2b3bdda027 libstore: collect effective url and cachedness earlier
this will let us return metadata for a transfer without having to wait
for the entire transfer to complete. more importantly for current uses
though is that we could now send retries to the effective url directly
instead of retreading the entire redirect path. this improves latency,
and in such cases where redirects change while we're downloading it'll
also improve correctness (previously we'd silently download bad data).

Change-Id: I6fcd920eb96fbdb2e960b73773c0b854e0300e99
2024-10-27 21:44:38 +00:00
eldritch horrors 97c76c4655 libstore: remove TransferItem::active
it's always legal to call curl_multi_remove_handle on a valid pair of
multi and easy handles. removing an easy handle that is not currently
attached to a multi handle is a no-op, and removing an easy handle of
a different multi handle is something we can't reasonably trigger. if
we *did* ever manage it would result in an error we'd ignore, and the
handles in question would not be changed at all. this is just simpler

Change-Id: I85ec62ff89385981ca49d243376b9c32586bd128
2024-10-27 21:44:38 +00:00
eldritch horrors c82407fc1e libstore: always allocate TransferItem::req
there's no reason not to do this. also improve error handling a bit.

Change-Id: I35853a919fa58a9a34ad47ffab6de77ba6f7fb86
2024-10-27 21:44:38 +00:00
eldritch horrors 982d049d3b libstore: remove FileTransferResult::data
return it as a separate item in a pair instead. this will let us remove
enqueueDownload() in favor of returning metadata from download() itself

Change-Id: I74fad2ca15f920da1eefabc950c2baa2c360f2ba
2024-10-27 21:44:38 +00:00
eldritch horrors 5cd7055044 libstore: de-future-ize FileTransfer::enqueueUpload
it's only used once, and synchronously at that.

Change-Id: Ife9db15dd97bc0de8de59a25d27f3f7afeb8791b
2024-10-27 21:44:38 +00:00
eldritch horrors 6f18e1ebde libstore: remove FileTransferRequest
it's just a uri and some headers now. those can be function arguments
with no loss of clarity. *actual* additional arguments, for example a
TLS context with additional certificates, could be added on a new and
improved FileTransfer class that carries not just a backend reference
but some real, visible context for its transfers. curl not being very
multi-threading-friendly when using multi handles will make sharing a
bit hard anyway once we drop the single global download worker thread

Change-Id: Id2112c95cbd118c6d920488f38d272d7da926460
2024-10-27 21:44:38 +00:00
eldritch horrors a839c31e6c libstore: remove FileTransferRequest::expectedETag
just another http specific used in only one place.

Change-Id: I99361a7226f4e6cd8f18170d3683c0025657bcb3
2024-10-27 21:44:38 +00:00
eldritch horrors 30bec83fa4 libstore: remove FileTransferResult::bodySize
it's only used for internal bookkeeping in TransferItem.

Change-Id: I467c5be023488be4a8a76e5f98a4ef25762df6f3
2024-10-27 21:44:38 +00:00
eldritch horrors d82b212d33 libstore: remove FileTransferRequest::verifyTLS
it's never set to false.

Change-Id: I1e436c82f1097091a08faa1dfada75e51bd5edf9
2024-10-27 21:44:38 +00:00
eldritch horrors 220251ba51 libstore: remove FiletransferRequest::head
add a method to FileTransfer that provides this functionality instead.

Change-Id: Ic1933a5df76a109c248c9c5efea065356b20a6f9
2024-10-27 21:44:38 +00:00
raito 9f682204b5 Merge changes I85b6075a,Iee41b055 into main
* changes:
  libstore: ban unpacking case hacked filenames from NARs
  testsuite: add a NAR generator with some evil NARs
2024-10-27 18:12:18 +00:00
raito f9e7df01f3 Merge "daemon: stop eating SIGINTs" into main 2024-10-27 18:11:20 +00:00
raito f7edee7c14 Merge changes I8e11ddbe,Idb8d9a00 into main
* changes:
  nix-shell: stop using dynamic format strings!!
  tests: move nix-shell related tests to subdir
2024-10-27 18:10:17 +00:00
eldritch horrors 6c2609c5f9 libstore: remove FileTransferRequest::tries
it's never set, and then only used internally. *once*.

Change-Id: I32585b1821e979f3ebb53b794ba0d1f576126b92
2024-10-26 21:42:35 +00:00
eldritch horrors af27d1ecd8 libstore: make baseRetryTimeMs a FileTransfer property
we don't even need this outside of tests. maybe we should not do
automatic retries at this level at all and use retrying wrappers
instead? at some point we may have to do this, but not just yet.

Change-Id: If0088aa55215be81f1770c25b3bb1b5268c65cf8
2024-10-26 21:42:35 +00:00
eldritch horrors 1e3b45546c libstore: remove FileTransferRequest::parentAct
never set explicitly, and transfers are never instantiated with one
current activity but submitted with a *different* current activity.

Change-Id: I1a3ec57c02013565aeb9e9398ea42d0c4279095e
2024-10-26 21:42:35 +00:00
eldritch horrors ce3e1d1e7a libstore: remove FileTransferRequests::data
use separate upload and download methods instead.

Change-Id: I5baa2177c8ddd70268c75ff074e361b2f17dddbd
2024-10-26 21:42:35 +00:00
eldritch horrors 2d49efaa2e libstore: remove Filetransfer::transfer
just use enqueueFileTransfer().get() insteaad.

Change-Id: I67a43c9d3d5f68ac3f9e8ba7973c243dd78b86a3
2024-10-26 21:42:35 +00:00
eldritch horrors 98b55c3a1d libstore: move FileTransferRequest::verb to TransferItem
this function is only used internally by curl wrapper.

Change-Id: I71d4c430cb069e2c949be769c17fede8dd04d480
2024-10-26 21:42:35 +00:00
eldritch horrors a83bf24281 libstore: remove FileTransferRequest::mimeType
it's only used by HttpBinaryCacheStore, and even there used in only on
place. this one place can set the header explicitly, which it now does

Change-Id: Id89228150669e25e7f59a3d6bd939e46059ce29e
2024-10-26 21:42:35 +00:00
eldritch horrors a8d6577bf0 libstore: HttpBinaryCacheStore::{makeRequest -> makeURI}
it only sets the one field anyway (and the parent activity as a side
effect that does not depend on the exact location of the constructor
call). when FileTransferRequest goes away we would need this anyway.

Change-Id: I35cf2ed3533239181449a62cf34cd282b395e5db
2024-10-26 20:35:16 +00:00
eldritch horrors 59e364c2a8 tests: stop using OCR for nix copy tests
this is fragile, slow as fuck, breaks constantly under high concurrency,
and completely unnecessary since ssh bypasses the stdio file descriptors
*anyway*. we do still check that we see ssh messages to ensure that none
of our subprocess handling messes with ssh's /dev/tty, but that's it now

Change-Id: Ib8e31e1999f813d07a27efc63a9d3454a9e4fcdd
2024-10-26 17:50:54 +00:00
Yureka b020d1fc27 Merge "fix build for 32-bit platforms" into main 2024-10-26 15:46:05 +00:00
alois31 2734a9cf94 Merge changes I29e66ad8,I77ea62cd,I7cd58d92 into main
* changes:
  treewide: make more settings conditionally available
  libstore/build: only send overridden settings to the build hook
  treewide: consistently mark overridden settings as such
2024-10-23 15:20:51 +00:00
eldritch horrors 5f1344dd8a libstore: turn Worker::run into a promise
a first little step into pushing the event loops up, up and away.
eventually we will want them to be instantiated only at the roots
of every thread (since kj binds loops to threads), but not today.

Change-Id: Ic97f1debba382a5a3f46daeaf2d6d434ee42569f
2024-10-23 11:55:12 +00:00
eldritch horrors faee771b30 libstore: hide Worker and goals where possible
goals should be considered internal to the worker architecture due to
the tight coupling of the two, and we can finally do that. doing this
is also a prerequisite for turning Worker::run() into a real promise.

Change-Id: I7cf273d4a6fdb75b8d192fce1af07c6265ff6980
2024-10-23 11:55:12 +00:00
eldritch horrors b8cc54df0a libstore: return relevant store path in goal result
we now do not need the goal pointers any more to process worker results.

Change-Id: I1a021b862ca666bcd23fee9f38973e90e6f94a72
2024-10-23 11:55:12 +00:00
eldritch horrors 67f1aafd61 libstore: restrict curl protocols
previously it was possible to fetchurl a dict server, or an ldap server,
or an imap server. this is a bit of a problem, both because rare schemes
may not be available on all systems, and because some schemes (e.g. scp)
are inherently insecure in potentially surprising ways we needn't allow.

Change-Id: I18fc567c6f58c3221b5ea8ce927f4da780057828
2024-10-23 11:32:14 +00:00
eldritch horrors 1d9d40b2a6 libstore: move failingExitStatus into worker results
we already have a results type for the entire worker call, we may as
well put this bit of info in there. we do keep the assumption of the
old code that the field will only be read if some goals have failed;
fixing that is a very different mess, and not immediately necessary.

Change-Id: If3fc32649dcd88e1987cdd1758c6c5743e3b35ac
2024-10-22 15:32:36 +00:00
eldritch horrors 343aca3a27 libstore: clear derivation build activities when done
without this derivations do not show as completely processed in the
internal-json logs (or the newer multiline output). the former also
breaks external tools like nix-output-monitor which, like multiline
output, grow vertically until at least some goals are finally freed

Change-Id: I55758daf526ba29ae15fb82e0d88da8afb45bf5c
2024-10-22 14:55:55 +00:00
Lily Ballard 5ce1d8463a Merge "libstore: add missing #include on darwin" into main 2024-10-22 02:05:44 +00:00
Yureka b77687945e fix build for 32-bit platforms
Change-Id: I113e131eb5c66c42c0bbc60181a7faafc02ca02e
2024-10-21 08:52:29 +02:00
Lily Ballard 4308ec1ae4 gitignore: ignore *.pyc files
Running our installcheck test suite creates these files.

Change-Id: I97ac8f1aa165a491c55dff6b48486db17b75443b
2024-10-20 22:08:35 -07:00
Lily Ballard 65551175e3 libstore: add missing #include on darwin
optimise-store.cc used std::regex on darwin, but forgot to include the
header. This probably compiled due to the precompiled headers file, but
it caused errors in the editor.

Change-Id: I23297c08cb66d44e4d4f303560f46e4adc7d5a43
2024-10-20 22:08:35 -07:00
Lily Ballard 068f4b147d libstore: fix sign comparison warnings in darwin platform
It's not clear to me if `proc_pidinfo()` or `proc_pidfdinfo()` can
actually return negative values, the syscall wrappers convert `-1` into
zero and the semantics suggest that negative values don't make sense,
but just to be safe we'll preserve the int type until we've checked that
it's a positive value.

Fixes: lix-project/lix#548
Change-Id: If575aec6b1e27dba63091c7a0316c7b3788747cd
2024-10-20 13:13:11 -07:00
eldritch horrors 0ff8f91325 libutil: disallow AsyncCollect relocations
some promises capture `this`.  we could also allocate a shared state,
but this thing doesn't really need to ever be moved anyway. so there.

Change-Id: I50b5c44684a8ab4e984b1323de21f97ace4a864a
2024-10-19 19:47:46 +00:00
eldritch horrors b0e619b8bd Merge "libstore: always release build/substitution slot tokens" into main 2024-10-19 18:32:30 +00:00
eldritch horrors 564d931134 libstore: always release build/substitution slot tokens
not doing this can freeze slots until the goal that occupied them is
freed (rather than simply complete), and then can freeze the system.

fixes #549

Change-Id: I042df04222f8ffbaa18ef4a4eae6cbd6f89b679e
2024-10-19 16:17:58 +02:00
jade dccde94369 daemon: stop eating SIGINTs
Daemon client handler processes are forked off of the main nix process
and thus will not have a signal handler thread anymore. This leads to a
high likelihood of bustage, since the Worker infrastructure expects the
interrupt infrastructure to actually, you know, work, to be able to get
interrupted.

The expected behaviour after fork is either:
- Start a signal handler thread if you expect to do complicated things
  that need ReceiveInterrupts.
- Call restoreProcessContext and don't handle signals specially
  otherwise.

Change-Id: I73d36b5bbf96dddd21d5e1c3bd0484d715c00e8b
2024-10-18 21:34:18 -07:00
Dusk Banks 60b89c63db libstore: restore mode after changing xattrs
this was missed in 3f07c65510. if mode is
not restored, the tree will have mutability where it shouldn't.

fixes test `functional-repair`, which fails from the output path
directory being unnecessarily writable:

```
++(repair.sh:12) nix-hash /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2
+(repair.sh:12) hash=d790f49fc89cb6f384b6dbe450790d07
+(repair.sh:15) chmod u+w /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2
+(repair.sh:16) touch /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2/bad
+(repair.sh:18) nix-store --verify --check-contents -v
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/20z19rjkwmwpb2ba4x29kac6xnslai38-dependencies-top.drv'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/3mmawi9lj33xz96cf6kw9989xc8v5i96-fod-input'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/5fp4r2kh1fcv6mv9dv6rywhhr1am9hhm-builder-dependencies-input-0.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/9kryn4ihv6b7bjswv2rsjq4533n2w5zk-fod-input.drv'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/ap8s0fim8s3ilzj8aqwlwk7gv1crjp4j-dependencies-top'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/i2ipd9p282zkqr1zb4glqiqigv8ybsyk-dependencies.builder0.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/i3xyw46h0lsx7ad6bczwi9sqjjx5f0j0-dependencies-input-0'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2'
path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2' was modified! expected hash 'sha256:1rhaylnjs5lbp089lnk7qvsypqmbm5vvyvxvv7i68b4x33pncqgs', got 'sha256:0nkjrmdc6ixf935chj3zhpqph5i15p306ffdsa850qh8mncpnsmc'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/pfaxqiw8zf3bw0w8w8gazswh76d729yy-builder-dependencies-input-2.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/pzzms3k5wl8d3wszv3maw29zylfkiiw0-dependencies-input-1'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/riwx84p57ckfvgli9nwhx88z6zh1c8ss-builder-fod-input.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/rjsphx7xvnqh2qafdrr7fiyxqc1rljhw-dependencies-input-2.drv'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/scywzq87dvm0c5f1h16hww6mvzhcsx3f-builder-dependencies-input-1.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/v65j4w033i05nyfd2h0g8h41ijmfglp7-dependencies-input-0.drv'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/ykwibh62xyp81k75vdw9c7y21kg4ibzf-dependencies-input-1.drv'
warning: not all store errors were fixed
+(repair.sh:21) nix-store --verify --check-contents --repair
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...
path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2' was modified! expected hash 'sha256:1rhaylnjs5lbp089lnk7qvsypqmbm5vvyvxvv7i68b4x33pncqgs', got 'sha256:0nkjrmdc6ixf935chj3zhpqph5i15p306ffdsa850qh8mncpnsmc'
checking path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2'...
path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2' is corrupted or missing!
checking path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/i3xyw46h0lsx7ad6bczwi9sqjjx5f0j0-dependencies-input-0'...
repairing outputs of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/rjsphx7xvnqh2qafdrr7fiyxqc1rljhw-dependencies-input-2.drv'...
+(repair.sh:23) '[' -e /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2/bad ']'
+(repair.sh:24) '[' -w /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2 ']'
++(repair.sh:24) onError
++(/var/home/bb010g/Sources/Nix/lix/build/tests/functional/common/vars-and-functions.sh:244) set +x
repair.sh: test failed at:
  main in repair.sh:24
```

fixes test `functional-simple`, which fails from the output path file
being unnecessarily writable:

```
+(simple.sh:11) echo 'output path is /tmp/nix-shell.IIUJAq/nix-test/simple/store/9fqn0rs99ymn1r09yip7bsifcdh3ra0y-simple'
+(simple.sh:13) '[' -w /tmp/nix-shell.IIUJAq/nix-test/simple/store/9fqn0rs99ymn1r09yip7bsifcdh3ra0y-simple ']'
++(simple.sh:13) onError
++(/var/home/bb010g/Sources/Nix/lix/build/tests/functional/common/vars-and-functions.sh:244) set +x
simple.sh: test failed at:
  main in simple.sh:13
```

fixes test `functional-optimise-store`, which fails from the unnecessary
mutability making Lix avoid hard linking:

```
++(optimise-store.sh:5) nix-build - --no-out-link --auto-optimise-store
this derivation will be built:
  /tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/2ql4kjxhnzdard8d6n3h9hc1m3lawr2b-foo1.drv
building '/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/2ql4kjxhnzdard8d6n3h9hc1m3lawr2b-foo1.drv'...
warning: skipping suspicious writable file '/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/h7i1pp848f9a8452y0s18kgsnis77vjn-foo1/foo'
+(optimise-store.sh:5) outPath1=/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/h7i1pp848f9a8452y0s18kgsnis77vjn-foo1
++(optimise-store.sh:6) echo 'with import ./config.nix; mkDerivation { name = "foo2"; builder = builtins.toFile "builder" "mkdir $out; echo hello > $out/foo"; }'
++(optimise-store.sh:6) nix-build - --no-out-link --auto-optimise-store
this derivation will be built:
  /tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/wjgvfhfsp14w06im8bbp1kqzz7smdkcy-foo2.drv
building '/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/wjgvfhfsp14w06im8bbp1kqzz7smdkcy-foo2.drv'...
warning: skipping suspicious writable file '/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/gjg6gayj2f6x3h53sp5i21nbnsd7b4i3-foo2/foo'
+(optimise-store.sh:6) outPath2=/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/gjg6gayj2f6x3h53sp5i21nbnsd7b4i3-foo2
++(optimise-store.sh:8) stat --format=%i /tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/h7i1pp848f9a8452y0s18kgsnis77vjn-foo1/foo
+(optimise-store.sh:8) inode1=328316
++(optimise-store.sh:9) stat --format=%i /tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/gjg6gayj2f6x3h53sp5i21nbnsd7b4i3-foo2/foo
+(optimise-store.sh:9) inode2=328404
+(optimise-store.sh:10) '[' 328316 '!=' 328404 ']'
+(optimise-store.sh:11) echo 'inodes do not match'
+(optimise-store.sh:12) exit 1
```

Signed-off-by: Dusk Banks <me@bb010g.com>
Change-Id: I87eeb74e718746a587be2ac52bcc9b5b1e5529db
2024-10-18 20:57:22 -07:00
jade 60578b4d7d Merge "flake.lock: update everything" into main 2024-10-19 00:03:42 +00:00
jade eadce58a90 Merge "testsuite: Fix tests on systems with a non-master defaultBranch" into main 2024-10-19 00:00:30 +00:00
jade ca55060ac6 packaging: use in-tree capnproto derivation
This is done because the one in nixpkgs has several problems and we
don't want to conflict with them fixing those problems:
- not building shared libs
- not building debuginfo (more critical for us due to
  lix-project/lix#549)
- not setting the correct cmake build type
- not setting the correct cxxflags to build the coroutine library
- not building with clang (required for the coroutine library since gcc
  is known to miscompile coroutines *in kj*).

CC: lix-project/lix#551

Fixes: lix-project/lix#550
Change-Id: Ia5b78dc1809963fdd1a8203b127a216cb575d751
2024-10-18 15:40:22 -07:00
jade 77b6f6734f packaging: remove workaround for clang stdenv asserts
This workaround was for the stdenv not being set when callPackage'ing
package.nix for some of the stranger CI outputs.

Change-Id: I2acdd6efa721b90dd3cb04358544d25d591ff084
2024-10-18 15:40:22 -07:00
piegames e2d00ac3a8 libexpr: Fix typo in error message
Closes #523

Change-Id: Ib5705e405b74d07a8fcf0163847405e9c791c3e3
2024-10-18 19:37:23 +02:00
piegames 3ba5ef91bc benchmarks: Fix purity
Some stuff wasn't working on my machine because it depended on the host
environment in some ways, fixed those.

Change-Id: Iff4931a9a26c6827978f1ee6434710f406d67a96
2024-10-18 19:37:23 +02:00
piegames e5de1d13c4 libexpr: Optimize complex indented strings
The old behavior results in lots of concatenations happening for no good
reason and is an artifact of the technical limitations of the old parser
(combined with some lack of care for such details).

Change-Id: I0d78d6220ca6aeaa10bc437e48e08bf7922e0bb3
2024-10-18 19:37:23 +02:00
piegames 878e181882 libexpr: Print interpolations more accurately in show
This is only a minor semantical distinction, but we should be able to
properly test it, and the parser tests rely on show for that.

Change-Id: I25e868cf9544e30cdff17deb5fd50a434e0f367e
2024-10-18 11:40:04 +00:00
piegames c852ae60da libexpr: Rewrite stripIndentation for indented strings
This commit should faithfully reproduce the old behavior down to the
bugs. The new code is a lot more readable, all quirks are well
documented, and it is overall much more maintainable.

Change-Id: I629585918e4f2b7d296b6b8330235cdc90b7bade
2024-10-18 11:40:04 +00:00
piegames 765771a355 tests/functional-lang: Add test cases for indented strings
Accidentally came across them elsewhere, wanted them covered in the
ind-strings test as well.

Change-Id: Iba418a687388ba85516d13d4c9d815744297dc5c
2024-10-18 11:40:04 +00:00
piegames 14291856e4 tests/functional-lang: Test indented string parsing
There is already an eval test for it, but it doesn't test the parser's
output

Change-Id: I94809080f90ace8aa3f610bcd315e9df0cf5d12f
2024-10-18 11:40:04 +00:00
piegames e7d6212f77 libexpr: move parser semantics into separate file
Another preparation for forking off and versioning the parser

Change-Id: I7b1225a44a3b81486414c1d37bd3e76a3ab307f9
2024-10-18 11:40:04 +00:00
piegames f98ee07573 libexpr: rename grammar to grammar::v1
Let's make some space in the namespace for a v2

https://wiki.lix.systems/books/lix-contributors/page/nix-lang-v2
Change-Id: If56e6dbf680d931233aa822ef91c8832464471e4
2024-10-18 11:40:04 +00:00
piegames 580df9bdb3 tests/functional-lang: Small cleanup of flagfiles handling
Flag files now properly work for all tests.

Change-Id: If652235960e8cf4120270a1b974c17586a31a157
2024-10-18 11:40:04 +00:00
jade 52b64662a4 Merge "fix: macOS build broken by fatal lowdown CLI sandbox setup" into main 2024-10-17 21:17:49 +00:00
alois31 689eb45630
treewide: make more settings conditionally available
Some settings only make sense on particular platforms, or only when a certain
experimental feature is enabled. Several of those were already conditionally
available. Do the same for a bunch more instead of silently ignoring them.
Exceptionally, the use-case-hack setting is not made conditional because it is
included in the test suite.

Change-Id: I29e66ad8ee6178a7c0eff9efb55c3410fae32514
2024-10-15 19:55:50 +02:00
alois31 ece99fee23
libstore/build: only send overridden settings to the build hook
The build hook is still running locally, so it will run with the same default
settings. Hence, just as with the daemon, it is enough to send it only the
overridden settings. This will prevent warnings like

   warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled

when the user didn't actually set those settings.

This is inspired by and an alternative to [0].
[0] https://github.com/NixOS/nix/pull/10049

Change-Id: I77ea62cd017614b16b55979dd30e75f09f860d21
2024-10-15 19:55:50 +02:00
alois31 4dbbd721eb
treewide: consistently mark overridden settings as such
Only overridden settings are sent to the daemon, and we're going to do the same
for the build hook to. It needs to be ensured that overridden settings are in
fact consistently marked as such, so that they actually get sent.

Change-Id: I7cd58d925702f86cf2c35ad121eb191ceb62a355
2024-10-15 19:55:50 +02:00
Linus Heckemann e55cd3beea libutil: handle json builder log messages with unexpected format
Before this change, expressions like:

with import <nixpkgs> {};
runCommand "foo" {} ''
  echo '@nix {}' >&$NIX_LOG_FD
''

would result in Lix crashing, because accessing nonexistent fields of
a JSON object throws an exception.

Rather than handling each field individually, we just catch JSON
exceptions wholesale. Since these log messages are an unusual
circumstance, log a warning when this happens.

Fixes #544.

Change-Id: Idc2d8acf6e37046b3ec212f42e29269163dca893
2024-10-15 11:15:42 +02:00
Justin ! 5a06b17b91
libutil: implement PathsSetting<PathSet>
By implementing the `PathSet` specialization for `PathsSetting`, we'll
be able to use `PathsSetting` for the `sandboxPaths` setting in
`src/libstore/globals.hh`.

Fixes: lix-project/lix#498

Change-Id: I8bf7dfff98609d1774fdb36d63e57d787bcc829f
2024-10-14 20:31:04 -04:00
jade f6077314fa Merge "Fix std::terminate call in thread pool" into main 2024-10-15 00:11:59 +00:00
jade a020f5f6cb fix: macOS build broken by fatal lowdown CLI sandbox setup
This failed due to https://github.com/NixOS/nixpkgs/pull/346945, which
makes a second lowdown-unsandboxed that works in nix builds, and the
regular lowdown has executables that fail closed when the sandbox setup
fails.

The actual failure here is only visible on nixos-unstable at the moment,
not 24.05, but this commit should fix it up for all versions.

Fixes: lix-project/lix#547
Change-Id: I50c0ecb59518ef01a7c0181114c1b4c5a7c6b78b
2024-10-14 16:49:16 -07:00
V. fbf7a8b440 Merge "Remove ancient let from 2 test files" into main 2024-10-14 22:06:48 +00:00
V. 31ff77b3f9 Remove ancient let from 2 test files
Change-Id: I992bc7f9e1cfcb1e4038fbe6ee04178bbf938556
2024-10-14 17:19:22 +04:00
jade c1f4c60bc2 nix-shell: stop using dynamic format strings!!
This was always a terrible idea independently of whether it crashes.
Stop doing it!

This commit was verified by running nix-shell on a trivial derivation
with --debug --verbose to get the vomit-level output of the shell rc
file and then diffing it before/after this change. I have reasonable
confidence it did not regress anything, though this code is genuinely
really hard to follow (which is a second reason that I split it into two
fmt calls).

Fixes: lix-project/lix#533
Change-Id: I8e11ddbece2b12749fda13efe0b587a71b00bfe5
2024-10-13 23:12:45 -07:00
jade 8497f0fe19 tests: move nix-shell related tests to subdir
This change feels kind of gross and reveals a fair bit about the
disorganization of our tests, but I think it makes parts of it a bit
better.

Change-Id: Idb8d9a00cbd75d5c156678c6b408b42b59d5e4d7
2024-10-13 23:12:45 -07:00
Maximilian Bosch 4682e40183 ssh-ng: better way to keep SSH errors visible
A better fix than in 104448e75d, hence a
revert + the fix.

It turns out that this commit has the side-effect that when having e.g.
`StrictHostKeyChecking=accept-new` for a remote builder, the warnings à la

    Warning: Permanently added 'builder' (ED25519) to the list of known hosts.

actually end up in the derivation's log whereas hostkey verification
errors don't, but only in the stderr of the `nix-build` invocation
(which was the motivation for the patch).

This change writes the stderr from the build-hook to

* the daemon's stderr, so that the SSH errors appear in the journal
  (which was the case before 104448e75d)
* the client's stderr, as a log message
* NOT to the drv log (this is handled via `handleJSONLogMessage`)

I tried to fix the issue for legacy-ssh as well, but failed and
ultimately decided to not bother.

I know that we'll sooner or later replace the entire component, however
this is the part of the patch I have working for a while, so I figured I
might still submit it for the time being.

Change-Id: I21ca1aa0d8ae281d2eacddf26e0aa825272707e5
2024-10-14 06:01:18 +00:00
Zebreus d726236e27 testsuite: Fix tests on systems with a non-master defaultBranch
When the git default branch is not set to master the installcheck
test suite fails. This patch adjusts the test setup scripts to
ignore the system and user git config files.

GIT_CONFIG_SYSTEM is set to /dev/null to ignore /etc/gitconfig

GIT_CONFIG_GLOBAL is not set because the global config files
are loaded from $HOME or $XDG_CONFIG_HOME which we already
reset.

git documentation: https://git-scm.com/docs/git#Documentation/git.txt-codeGITCONFIGGLOBALcode

Change-Id: Ie73bbed1db9419c9885b9d57e4edb7a4047d5cce
2024-10-14 05:56:02 +00:00
jade 326cbecb61 Merge changes I327db40f,If762efce into main
* changes:
  testsuite: use xdist for parallel test running
  testsuite: add a functional2 test suite based on pytest
2024-10-14 05:53:44 +00:00
Maximilian Bosch a322fcea4a
worker: respect C-c on sudo nix-build
While debugging something else I observed that latest `main` ignores
`Control-C` on `sudo nix-build`.

After reading through the capnproto docs, it seems as if the promise
must be fulfilled to actually terminate the `promise.wait()` below.

This also applies to scenarios such as stopping the client
(`nix-build`), but the builders on the daemon-side are still running,
i.e. closes #540

Co-authored-by: eldritch horrors <pennae@lix.systems>

Change-Id: I9634d14df4909fc1b65d05654aad0309bcca8a0a
2024-10-12 21:16:30 +02:00
jade a0fb52c0af Fix std::terminate call in thread pool
So we received a report that the thread pool crashed due to an
Interrupted exception.

Relevant log tail:

copying path '/nix/store/0kal2k73inviikxv9f1ciaj39lkl9a87-etc-os-release' to 'ssh://192.168.0.27'...
Lix crashed. This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included:

error (ignored): error: interrupted by the user
Exception: nix::Interrupted: error: interrupted by the user

Relevant stack trace:

 4# __cxa_rethrow in /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib/lib/libstdc++.so.6
 5# nix::ignoreExceptionExceptInterrupt(nix::Verbosity) in /nix/store/ghxr2ykqc3rrfcy8rzdys0rzx9ah5fqj-lix-2.92.0-dev-pre20241005-ed9b7f4/lib/liblixutil.so
 6# nix::ThreadPool::doWork(bool) in /nix/store/ghxr2ykqc3rrfcy8rzdys0rzx9ah5fqj-lix-2.92.0-dev-pre20241005-ed9b7f4/lib/liblixutil.so
 7# 0x00007FA7A00E86D3 in /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib/lib/libstdc++.so.6
 8# 0x00007FA79FE99A42 in /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6
 9# 0x00007FA79FF1905C in /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6

Notably, this is *not* in the main thread, so this implies that the
thread didn't get joined properly before their destructors got called.
That, in turn, should have only possibly happened because join() threw
on a previous iteration of the loop joining threads, I think. Or if it
threw while in the ThreadPool destructor. Either way we had better stop
letting Interrupted fall out of our child threads!

If:
- Interrupted was thrown inside the action in the main thread: it would
  have fallen out of doWork if state->exception was already set and got
  caught by ThreadPool::process, calling shutdown() and the join loop
  which would crash the process entirely.
- Interrupted was thrown inside the action on a secondary thread: it
  would have been caught and put into the exception field and then
  possibly rethrown to fall out of the thread (since it was previously
  ignoreExceptionExceptInterrupt).

The one possible hole in this hypothesis is that there is an "error
(ignored)" line in there implying that at least one Interrupted got
eaten by an ignoreExceptionInDestructor. It's also unclear whether this
got reordered because of stderr buffering.

Fixes: lix-project/lix#542
Change-Id: I322cf050da660af78f5cb0e08ec6e6d27d09ac76
2024-10-09 15:38:40 -07:00
jade 822997bd34 libstore: ban unpacking case hacked filenames from NARs
There is absolutely no good reason these should show up in NARs besides
misconfigured systems and as long as the case hack exists, unpacking
such a NAR will cause its repacking to be wrong on systems with case
hack enabled.

This should not have any security impact on Lix to fix, but it was one
of the vectors for CVE-2024-45593:
https://github.com/NixOS/nix/security/advisories/GHSA-h4vv-h3jq-v493

Change-Id: I85b6075aacc069ee7039240b0f525804a2d8edcb
2024-10-09 14:47:39 -07:00
jade 4180b84a67 testsuite: use xdist for parallel test running
This is capped at 12 because 3.7 seconds of startup is painful enough
and 5.5 seconds with 24 was more annoying.

Change-Id: I327db40fd98deaa5330cd9cf6de99fb07b2c1cb0
2024-10-09 14:47:39 -07:00
jade 3571817e3a testsuite: add a NAR generator with some evil NARs
This also rewrites a lot of the command handling in the fixtures
library, since we want to more precisely control which way that the nix
store is set up in the tests, rather than the previous method of
renaming /nix/store to some temp dir (which allows builds but does not
allow any /nix/store paths or stability across runs, which is a
significant issue for snapshot testing).

It uses a builder to reduce the amount of state carelessly thrown
around.

The evil NARs are inspired by CVE-2024-45593
(https://github.com/NixOS/nix/security/advisories/GHSA-h4vv-h3jq-v493).

No bugs were found in this endeavor.

Change-Id: Iee41b055fa96529c5a3c761f680ed1d0667ba5da
2024-10-09 14:47:39 -07:00
jade 3caf3e1e08 testsuite: add a functional2 test suite based on pytest
I am tired of bad shell scripts, let me write bad python quickly
instead. It's definitely, $100%, better.

This is not planned as an immediate replacement of the old test suite,
but we::jade would not oppose tests getting ported.

What is here is a mere starting point and there is a lot more
functionality that we need.

Fixes: lix-project/lix#488

Change-Id: If762efce69030bb667491b263b874c36024bf7b6
2024-10-09 14:47:39 -07:00
jade 9865ebaaa6 Merge "Remove static initializers for RegisterLegacyCommand" into main 2024-10-09 20:37:58 +00:00
jade 7f7a38f278 Merge changes Ib27cb43d,I03687b8b into main
* changes:
  testsuite: override NIX_CONF_DIR and NIX_USER_CONF_FILES
  Remove some outdated `make test` invocation suggestions
2024-10-09 20:37:16 +00:00
Rebecca Turner 0012887310 Merge "Add release note for CTRL-C improvements" into main 2024-10-08 22:15:56 +00:00
Lulu 4ea8c9d643 Set c++ version to c++23
I followed @pennae's advice and moved the constructor definition of
`AttrName` from the header file `nixexpr.hh` to `nixexpr.cc`.

Change-Id: I733f56c25635b366b11ba332ccec38dd7444e793
2024-10-08 20:05:28 +02:00
Lulu 43e79f4434 Fix gcc warning -Wmissing-field-initializers
The approach that was taken here was to add default values to the type
definitions rather than specify them whenever they are missing.

Now the only remaining warning is '-Wunused-parameter' which @jade said
is usually counterproductive and that we can just disable it:
lix-project/lix#456 (comment)

So this change adds the flags '-Wall', '-Wextra' and
'-Wno-unused-parameter', so that all warnings are enabled except for
'-Wunused-parameter'.

Change-Id: Ic223a964d67ab429e8da804c0721ba5e25d53012
2024-10-08 01:44:38 +00:00
Lulu 299813f324 Merge "Avoid calling memcpy when len == 0 in filetransfer.cc" into main 2024-10-08 01:41:41 +00:00
Lulu d6e1b11d3e Fix gcc warning -Wsign-compare
Add the compile flag '-Wsign-compare' and adapt the code to fix all
cases of this warning.

Change-Id: I26b08fa5a03e4ac294daf697d32cf9140d84350d
2024-10-08 01:32:12 +02:00
Lulu 51a5025913 Avoid calling memcpy when len == 0 in filetransfer.cc
There was a bug report about a potential call to `memcpy` with a null
pointer which is not reproducible:
lix-project/lix#492

This occurred in `src/libstore/filetransfer.cc` in `InnerSource::read`.

To ensure that this doesn't happen, an early return is added before
calling `memcpy` if the length of the data to be copied is 0.

This change also adds a test that ensures that when `InnerSource::read`
is called with an empty file, it throws an `EndOfFile` exception.

Change-Id: Ia18149bee9a3488576c864f28475a3a0c9eadfbb
2024-10-08 01:26:30 +02:00
eldritch horrors ed9b7f4f84 libstore: remove Worker::{childStarted, goalFinished}
these two functions are now nearly trivial and much better inline into
makeGoalCommon. keeping them separate also separates information about
goal completion flows and how failure information ends up in `Worker`.

Change-Id: I6af86996e4a2346583371186595e3013c88fb082
2024-10-05 21:19:51 +00:00
eldritch horrors 649d8cd08f libstore: remove Worker::removeGoal
we can use our newfound powers of Goal::work Is A Real Promise to remove
completed goals from continuation promises. apart from being much easier
to follow it's also a lot more efficient because we have the iterator to
the item we are trying to remove, skipping a linear search of the cache.

Change-Id: Ie0190d051c5f4b81304d98db478348b20c209df5
2024-10-05 21:19:51 +00:00
eldritch horrors 9adf6f4568 libstore: remove Goal::notify
Goal::work() is a fully usable promise that does not rely on the worker
to report completion conditions. as such we no longer need the `notify`
field that enabled this interplay. we do have to clear goal caches when
destroying the worker though, otherwise goal promises may (incorrectly)
keep goals alive due to strong shared pointers created by childStarted.

Change-Id: Ie607209aafec064dbdf3464fe207d70ba9ee158a
2024-10-05 21:19:51 +00:00
eldritch horrors 03cbc0ecb9 libstore: move Goal::ex to WorkResult
yet another duplicated field. it's the last one though.

Change-Id: I352df8d306794d262d8c9066f3be78acd40e82cf
2024-10-05 21:19:51 +00:00
eldritch horrors 1caf2afb1d libstore: move Goal::buildResult to WorkResult
derivation goals still hold a BuildResult member variable since parts of
these results of accumulated in different places, but the Goal class now
no longer has such a field. substitution goals don't need it at all, and
derivation goals should also be refactored to not drop their buildResult

Change-Id: Ic6d3d471cdbe790a6e09a43445e25bedec6ed446
2024-10-05 20:53:39 +00:00
eldritch horrors 7ff60b7445 libstore: move Goal::exitCode to WorkResult
the field is simply duplicated between the two, and now that we can
return WorkResults from Worker::run we no longer need both of them.

Change-Id: I82fc47d050b39b7bb7d1656445630d271f6c9830
2024-10-05 20:17:20 +00:00
eldritch horrors fc6291e46d libstore: return goal results from Worker::run()
this will be needed to move all interesting result fields out of Goal
proper and into WorkResult. once that is done we can treat goals as a
totally internal construct of the worker mechanism, which also allows
us to fully stop exposing unclear intermediate state to Worker users.

Change-Id: I98d7778a4b5b2590b7b070bdfc164a22a0ef7190
2024-10-05 20:12:13 +00:00
eldritch horrors 40f154c0ed libstore: remove Worker::topGoals
since we now propagate goal exceptions properly we no longer need to
check topGoals for a reason to abort early. any early abort reasons,
whether by exception or a clean top goal failure, can now be handled
by inspecting the goal result in the main loop. this greatly reduces
goal-to-goal interactions that do not happen at the main loop level.

since the underscore-free name is now available for use as variables
we'll migrate to that where we currently use `_topGoals` for locals.

Change-Id: I5727c5ea7799647c0a69ab76975b1a03a6558aa6
2024-10-05 19:53:30 +00:00
eldritch horrors f389a54079 libstore: propagate goal exceptions using promises
drop childException since it's no longer needed. also makes
waitForInput, childFinished, and childTerminated redundant.

Change-Id: I05d88ffd323c5b5c909ac21056162f69ffb0eb9f
2024-10-05 19:44:47 +00:00
eldritch horrors 7ef4466018 libstore: have goals promise WorkResults, not void
Change-Id: Idd218ec1572eda84dc47accc0dcd8a954d36f098
2024-10-05 19:06:59 +00:00
eldritch horrors a9f2aab226 libstore: extract Worker::goalFinished specifics
there's no reason to have the worker set information on goals that the
goals themselves return from their entry point. doing this in the goal
`work()` function is much cleaner, and a prerequisite to removing more
implicit strong shared references to goals that are currently running.

Change-Id: Ibb3e953ab8482a6a21ce2ed659d5023a991e7923
2024-10-05 19:06:59 +00:00
eldritch horrors 99edc2ae38 libstore: check for interrupts in parallel promise
this simplifies the worker loop, and lets us remove it entirely later.
note that ideally only one promise waiting for interrupts should exist
in the entire system. not one per event loop, one per *process*. extra
interrupt waiters make interrupt response nondeterministic and as such
aren't great for user experience. if anything wants to react to aborts
caused by explicit interruptions, or anything else, those things would
be much better served using RAII guards such as Finally (or KJ_DEFER).

Change-Id: I41d035ff40172d536e098153c7375b0972110d51
2024-10-05 19:06:59 +00:00
eldritch horrors 896a123605 libstore: remove Goal::StillAlive
this was a triumph. i'm making a note here: huge success. it's hard to
overstate my satisfaction! i'm not even angry. i'm being so sincere ri

actually, no. we *are* angry. this was one dumbass odyssey. nobody has
asked for this. but not doing it would have locked us into old, broken
protocols forever or (possibly worse) forced us to write our own async
framework building on the old did-you-mean-continuations in Worker. if
we had done that we'd be locked into ever more, and ever more complex,
manual state management all over the place. this just could not stand.

Change-Id: I43a6de1035febff59d2eff83be9ad52af4659871
2024-10-05 18:21:02 +00:00
Rebecca Turner 0d484aa498
Add release note for CTRL-C improvements
I'm very excited for cl/2016, so others will probably be excited also!
Let's add a release note.

Change-Id: Ic84a4444241aafce4cb6d5a6d1dddb47e7a7dd7b
2024-10-05 10:40:51 -07:00
Rebecca Turner 86b213e632 Merge "Split ignoreException to avoid suppressing CTRL-C" into main 2024-10-05 17:33:00 +00:00
eldritch horrors a3dd07535c fix build test error count checks
with async runtime scheduling we can no longer guarantee exact error
counts for builds that do not set keepGoing. the old behavior can be
recovered with a number of hacks that affect scheduling, but none of
those are very easy to follow now advisable. exact error counts will
like not be needed for almost all uses except tests, and *those* had
better check the actual messages rather than how many they got. more
messages can even help to avoid unnecessary rebuilds for most users.

Change-Id: I1c9aa7a401227dcaf2e19975b8cb83c5d4f85d64
2024-10-05 16:21:19 +00:00
alois31 5df2cccc49
doc: install the HTML manual again
In 0e6b3435a1, installation of the HTML manual
was accidentally dropped: setting install_dir on a custom_target only sets the
directory where something is going to be installed if it is installed at all,
but does not itself trigger installation. The latter has to be explicitly
requested, which is just what we do here to get the manual back.

Change-Id: Iff8b791de7e7cb4c8d747c2a9b1154b5fcc32fe0
2024-10-05 10:49:34 +02:00
jade 345e3d068a testsuite: override NIX_CONF_DIR and NIX_USER_CONF_FILES
The test suite can load the global configuration files under certain
circumstances, and, though we would really rather it didn't ever do that
at all, we should at least break the mechanism.

Fixes: lix-project/lix#474
Change-Id: Ib27cb43dd5dfaa70ac491c395b5ba308fd7bd289
2024-10-04 19:17:08 -07:00
jade 19edaed81b Remove some outdated make test invocation suggestions
These should be meson.

Change-Id: I03687b8b03f50fb1684e7ffcd487be855052d6c2
2024-10-04 18:55:52 -07:00
eldritch horrors 5b1715e633 libstore: forbid addWantedGoals when finished
due to event loop scheduling behavior it's possible for a derivation
goal to fully finish (having seen all paths it was asked to create),
but to not notify the worker of this in time to prevent another goal
asking the recently-finished goal for more outputs. if this happened
the finished goal would ignore the request for more outputs since it
considered itself fully done, and the delayed result reporting would
cause the requesting goal to assume its request had been honored. if
the requested goal had finished *properly* the worker would recreate
it instead of asking for more outputs, and this would succeed. it is
thus safe to always recreate goals once they are done, so we now do.

Change-Id: Ifedd69ca153372c623abe9a9b49cd1523588814f
2024-10-04 17:49:57 +00:00
Rebecca Turner 0b29859cfe Merge "editorconfig: Add meson.build" into main 2024-10-04 16:36:20 +00:00
Olivia Crain 1bfc37fea5 Merge "internal-api-docs: allow Doxygen to build regardless of workdir" into main 2024-10-04 09:59:01 +00:00
Olivia Crain 8f300fbd82 Merge "build: let meson add compiler flags for libstdc++ assertions" into main 2024-10-04 09:58:32 +00:00
Rebecca Turner 36073781fb
editorconfig: Add meson.build
Change-Id: Ibb59ddc21f5d3ef7fb4c900e3413e426c201334d
2024-10-01 16:09:47 -07:00
Rebecca Turner b63d4a0c62
Remove static initializers for RegisterLegacyCommand
This moves the "legacy"/"nix2" commands under a new `src/legacy/`
directory, instead of being scattered around in a bunch of different
directories.

A new `liblegacy` build target is defined, and the `nix` binary is
linked against it.

Then, `RegisterLegacyCommand` is replaced with `LegacyCommand::add`
calls in functions like `registerNixCollectGarbage()`. These
registration functions are called explicitly in `src/nix/main.cc`.

See: lix-project/lix#359

Change-Id: Id450ffc3f793374907599cfcc121863b792aac1a
2024-10-01 16:08:58 -07:00
Robert Hensing ee0c195eba
Split ignoreException to avoid suppressing CTRL-C
This splits `ignoreException` into `ignoreExceptionExceptInterrupt`
(which ignores all exceptions except `Interrupt`, which indicates a
SIGINT/CTRL-C) and `ignoreExceptionInDestructor` (which ignores all
exceptions, so that destructors do not throw exceptions).

This prevents many cases where Nix ignores CTRL-C entirely.
See: https://github.com/NixOS/nix/issues/7245

Upstream-PR: https://github.com/NixOS/nix/pull/11618
Change-Id: Ie7d2467eedbe840d1b9fa2e88a4e88e4ab26a87b
2024-10-01 15:49:56 -07:00
eldritch horrors 7752927660 libstore: turn DerivationGoal::work into *one* promise
Change-Id: Ic2f7bc2bd6a1879ad614e4be81a7214f64eb0e85
2024-10-01 11:55:47 +00:00
eldritch horrors 3edc272341 libstore: turn DrvOutputSubstitutionGoal::work into *one* promise
Change-Id: I2d4dcedff0a278d2d8f3d264a9186dfb399275e2
2024-10-01 11:55:42 +00:00
eldritch horrors 9b05636937 libstore: make PathSubstitutionGoal::work *one* promise
Change-Id: I38cfe8c7059251b581f1013c4213804f36b985ea
2024-10-01 11:55:36 +00:00
eldritch horrors 9889c79fe3 libstore: turn Worker::updateStatistics into a promise
we'll now loop to update displayed statistics, and use this loop to
limit the update rate to 50 times per second. we could have updated
much more frequently before this (once per iteration of `runImpl`),
much faster than would ever be useful in practice. aggressive stats
updates can even impede progress due to terminal or network delays.

Change-Id: Ifba755a2569f73c919b1fbb06a142c0951395d6d
2024-10-01 11:55:29 +00:00
eldritch horrors 732de75f67 libstore: remove Worker::wakeUp()
Worker::run() is now entirely based on the kj event loop and promises,
so we need not handle awakeness of goals manually any more. every goal
can instead, once it has finished a partial work call, defer itself to
being called again in the next iteration of the loop. same end effect.

Change-Id: I320eee2fa60bcebaabd74d1323fa96d1402c1d15
2024-10-01 13:55:03 +02:00
eldritch horrors d5db0b1abc libstore: turn periodic gc attempt into a promise
notably we will check whether we want to do GC at all only once during
startup, and we'll only attempt GC every ten seconds rather than every
time a goal has finished a partial work call. this shouldn't cause any
problems in practice since relying on auto-gc is not deterministic and
stores in which builds can fill all remaining free space in merely ten
seconds are severely troubled even when gargage collection runs a lot.

Change-Id: I1175a56bf7f4e531f8be90157ad88750ff2ddec4
2024-10-01 11:36:45 +00:00
eldritch horrors b0c7c1ec66 libstore: turn Worker::run() main loop into a promise
Change-Id: Ib112ea9a3e67d5cb3d7d0ded30bbd25c96262470
2024-10-01 11:36:45 +00:00
eldritch horrors d31310bf59 libstore: turn waitForInput into a promise
Change-Id: I8355d8d3f6c43a812990c1912b048e5735b07f7b
2024-10-01 11:36:45 +00:00
raito 8e05cc1e6c Revert "libstore: remove worker removeGoal"
Revert submission 1946

Reason for revert: regression in building (found via bisection)

Reported by users:
> error: path '/nix/store/04ca5xwvasz6s3jg0k7njz6rzi0d225w-jq-1.7.1-dev' does not exist in the store

Reverted changes: /q/submissionid:1946

Change-Id: I6f1a4b2f7d7ef5ca430e477fc32bca62fd97036b
2024-10-01 11:07:57 +00:00
Jonas Chevalier a16ceb9411 Merge "fix(nix fmt): remove the default "." argument" into main 2024-09-30 16:10:32 +00:00
eldritch horrors aa33c34c9b libstore: merge ContinueImmediately and StillAlive
nothing needs to signal being still active but not actively pollable,
only that immediate polling for the next goal work phase is in order.

Change-Id: Ia43c1015e94ba4f5f6b9cb92943da608c4a01555
2024-09-29 15:29:56 +00:00
eldritch horrors ccd2862666 libstore: remove worker removeGoal
this was immensely inefficient on large caches, as can exist when many
derivations are buildable simultaneously. since we have smart pointers
to goals we can do cache maintenance in goal deleters instead, and use
the exact iterators instead of doing a linear search. this *does* rely
on goals being deleted to remove them from the cache, which isn't true
for toplevel goals. those would have previously been removed when done
in all cases, removing the cache entry when keep-going is set. this is
arguably incorrect since it might result in those goals being retried,
although that could only happen with dynamic derivations or the likes.
(luckily dynamic derivations not complete enough to allow this at all)

Change-Id: I8e750b868393588c33e4829333d370f2c509ce99
2024-09-29 15:29:56 +00:00
eldritch horrors 47ddd11933 libstore: extract a real makeGoalCommon
makeDerivationGoalCommon had the right idea, but it didn't quite go far
enough. let's do the rest and remove the remaining factory duplication.

Change-Id: I1fe32446bdfb501e81df56226fd962f85720725b
2024-09-29 15:07:30 +00:00
eldritch horrors 7f4f86795c libstore: remove Goal::key
this was a debugging aid from day one that should not have any impact on
build semantics, and if it *does* have an impact on build semantics then
build semantics are seriously broken. keeping the order imposed by these
keys will be impossible once we let a real event loop schedule our jobs.

Change-Id: I5c313324e1f213ab6453d82f41ae5e59de809a5b
2024-09-29 14:29:14 +00:00
eldritch horrors a5240b23ab libstore: make non-cache goal pointers strong
without circular references we do not need weak goal pointers except for
caches, which should not prevent goal destructors running. caches though
cannot create circular references even when they keep strong references.
if we removed goals from caches when their work() is fully finished, not
when their destructors are run, we could keep strong pointers in caches.
since we do not gain much from this we keep those pointers weak for now.

Change-Id: I1d4a6850ff5e264443c90eb4531da89f5e97a3a0
2024-09-29 14:29:14 +00:00
eldritch horrors 8fb642b6e0 libstore: remove Goal::WaitForWorld
have DerivationGoal and its subclasses produce a wrapper promise for
their intermediate results instead, and return this wrapper promise.
Worker already handles promises that do not complete immediately, so
we do not have to duplicate this into an entire result type variant.

Change-Id: Iae8dbf63cfc742afda4d415922a29ac5a3f39348
2024-09-29 14:29:14 +00:00
eldritch horrors 1a52e4f755 libstore: fix build tests
the new event loop could very occasionally notice that a dependency of
some goal has failed, process the failure, cause the depending goal to
fail accordingly, and in the doing of the latter two steps let further
dependencies that previously have not been reported as failed do their
reporting anyway. in such cases a goal could fail with "1 dependencies
failed", but more than one dependency failure message was shown. we'll
now report the correct number of failed dependency goals in all cases.

Change-Id: I5aa95dcb2db4de4fd5fee8acbf5db833531d81a8
2024-09-29 13:17:15 +00:00
eldritch horrors 3f7519526f libstore: have makeLocalDerivationGoal return unique_ptrs
these can be unique rather than shared because shared_ptr has a
converting constructor. preparatory refactor for something else
and not necessary on its own, and the extra allocations we must
do for shared_ptr control blocks isn't usually relevant anyway.

Change-Id: I5391715545240c6ec8e83a031206edafdfc6462f
2024-09-29 12:09:24 +00:00
Maximilian Bosch 289e7a6b5a Merge "libfetchers/git: restore compat with builtins.fetchGit from 2.3" into main 2024-09-29 08:56:16 +00:00
Olivia Crain f12b60273b Merge changes I5c640824,I09ffc92e,I259583b7 into main
* changes:
  build: require meson 1.4.0 or newer
  build: fix deprecated uses of configure_file
  build: install html manual without using install_subdir
2024-09-28 23:41:30 +00:00
Maximilian Bosch 04daff94e3
libfetchers/git: restore compat with builtins.fetchGit from 2.3
Since fb38459d6e, each `ref` is appended
with `refs/heads` unless it starts with `refs/` already. This regressed
two use-cases that worked fine before:

* Specifying a commit hash as `ref`: now, if `ref` looks like a commit
  hash it will be directly passed to `git fetch`.

* Specifying a tag without `refs/tags` as prefix: now, the fetcher prepends
  `refs/*` to a ref that doesn't start with `refs/` and doesn't look
  like a commit hash. That way, both a branch and a tag specified in
  `ref` can be fetched.

  The order of preference in git is

  * file in `refs/` (e.g. `HEAD`)
  * file in `refs/tags/`
  * file in `refs/heads` (i.e. a branch)

  After fetching `refs/*`, ref is resolved the same way as git does.

Change-Id: Idd49b97cbdc8c6fdc8faa5a48bef3dec25e4ccc3
2024-09-28 14:52:06 +02:00
Olivia Crain 4780dd6bc4
build: let meson add compiler flags for libstdc++ assertions
We have manually enabled libstdc++ assertions since cl/797. Meson 1.4.0
(the minimum version we mandate) enables this by default, so we can
remove the enabling compiler flag from the list of project arguments.

With this patch, `-D_GLIBCXX_ASSERTIONS=1` is still present in the
compile command logs when building with both gccStdenv and clangStdenv.

See: https://gerrit.lix.systems/c/lix/+/797
See: https://mesonbuild.com/Release-notes-for-1-4-0.html#ndebug-setting-now-controls-c-stdlib-assertions
Change-Id: I53483fadfe5cbd11ba35544b437d3a9ee8031631
2024-09-27 12:26:07 -05:00
Olivia Crain b86863d935
build: require meson 1.4.0 or newer
This was already the de facto requirement, we use the method `full_path`
on a file object (introduced in Meson 1.4.0) in the functional test
suite's build.

This version of Meson is in NixOS 24.05, so there should be no
compatibility issues should this make it into a backported release of
Lix.

CC: lix-project/lix#247
Change-Id: I5c640824807353b6eb4287e7ed09c4e89a4bdde2
2024-09-27 11:57:53 -05:00
Olivia Crain 624f44bf25
build: fix deprecated uses of configure_file
Using `configure_file` to copy files has been deprecated since Meson 0.64.0.
The intended replacement is the `fs.copyfile` method.

This removes the following deprecation warning that arises when a minimum
Meson version is specified:

``
Project [...] uses feature deprecated since '0.64.0': copy arg in configure_file. Use fs.copyfile instead
``

Change-Id: I09ffc92e96311ef9ed594343a0a16d51e74b114a
2024-09-27 11:55:32 -05:00
Olivia Crain 0e6b3435a1
build: install html manual without using install_subdir
In Meson, `install_subdir` is meant to be used with directories in the source
directory. When using it to install the HTML manual, we provide it with a path
under the build directory.

We should instead specify an install directory for the HTML manual as part of
the custom target that builds it.

What we do currently isn't broken, just semantically incorrect. Changing it does
get rid of the following deprecation warning, though:

``
Project [...] uses feature deprecated since '0.60.0': install_subdir with empty directory. It worked by accident and is buggy. Use install_emptydir instead.
``

Change-Id: I259583b7bdff8ecbb3b342653d70dc5f034c7fad
2024-09-27 11:55:28 -05:00
eldritch horrors ae5d8dae1b libstore: turn Goal::WaitForGoals into a promise
also gets rid of explicit strong references to dependencies of any goal,
and weak references to dependers as well. those are now only held within
promises representing goal completion and thus independent of the goal's
relation to each other. the weak references to dependers was only needed
for notifications, and that's much better handled entirely by kj itself.

Change-Id: I00d06df9090f8d6336ee4bb0c1313a7052fb016b
2024-09-27 16:40:27 +02:00
eldritch horrors 852da07b67 libstore: replace Goal::WaitForSlot with semaphores
now that we have an event loop in the worker we can use it and its
magical execution suspending properties to replace the slot counts
we managed explicitly with semaphores and raii tokens. technically
this would not have needed an event loop base to be doable, but it
is a whole lot easier to wait for a token to be available if there
is a callback mechanism ready for use that doesn't require a whole
damn dedicated abstract method in Goal to work, and specific calls
to that dedicated method strewn all over the worker implementation

Change-Id: I1da7cf386d94e2bbf2dba9b53ff51dbce6a0cff7
2024-09-27 16:40:27 +02:00
eldritch horrors bf32085d63 libstore: simplify Worker::waitForInput
with waitForAWhile turned into promised the core functionality of
waitForInput is now merely to let gc run every so often if needed

Change-Id: I68da342bbc1d67653901cf4502dabfa5bc947628
2024-09-27 16:40:26 +02:00
eldritch horrors cd1ceffb0e libstore: make waiting for a while a promise
this simplifies waitForInput quite a lot, and at the same time makes
polling less thundering-herd-y. it even fixes early polling wakeups!

Change-Id: I6dfa62ce91729b8880342117d71af5ae33366414
2024-09-27 16:39:33 +02:00
eldritch horrors 0478949c72 libstore: turn builder output processing into event loop
this removes the rather janky did-you-mean-async poll loop we had so
far. sadly kj does not play well with pty file descriptors, so we do
have to add our own async input stream that does not eat pty EIO and
turns it into an exception. that's still a *lot* better than the old
code, and using a real even loop makes everything else easier later.

Change-Id: Idd7e0428c59758602cc530bcad224cd2fed4c15e
2024-09-27 16:38:16 +02:00
Jonas Chevalier 2265536e85 fix(nix fmt): remove the default "." argument
When `nix fmt` is called without an argument, Nix appends the "." argument before calling the formatter. The comment in the code is:
> Format the current flake out of the box

This also happens when formatting sub-folders.

This means that the formatter is now unable to distinguish, as an interface, whether the "." argument is coming from the flake or the user's intent to format the current folder. This decision should be up to the formatter.

Treefmt, for example, will automatically look up the project's root and format all the files. This is the desired behaviour. But because the "." argument is passed, it cannot function as expected.

Upstream-PR: https://github.com/nixos/nix/pull/11438

Change-Id: I60fb6b3ed4ec1b24f81b5f0d76c0be98470817ce
2024-09-26 14:32:29 -07:00
jade 14dc84ed03 Merge changes Iaa2e0e9d,Ia973420f into main
* changes:
  Fix passing custom CA files into the builtin:fetchurl sandbox
  [security] builtin:fetchurl: Enable TLS verification
2024-09-26 20:53:46 +00:00
eldritch horrors 619a93bd54 Merge "libutil: add async collection mechanism" into main 2024-09-26 17:23:52 +00:00
jade 5dc7671d81 Merge "fmt: fail hard on bad format strings going into nix::fmt too" into main 2024-09-26 17:07:29 +00:00
jade b6038e988d Merge "main: log stack traces for std::terminate" into main 2024-09-26 17:06:01 +00:00
eldritch horrors 531d040e8c libutil: add async collection mechanism
like kj::joinPromisesFailFast this allows waiting for the results of
multiple promises at once, but unlike it not all input promises must
be complete (or any of them failed) for results to become available.

Change-Id: I0e4a37e7bd90651d56b33d0bc5afbadc56cde70c
2024-09-26 16:56:08 +00:00
eldritch horrors ca9256a789 libutil: add an async semaphore implementation
like a normal semaphore, but with awaitable acquire actions. this is
primarily intended as an intermediate concurrency limiting device in
the Worker code, but it may find other uses over time. we do not use
std::counting_semaphore as a base because the counter of that is not
inspectable as will be needed for Worker. we also do not need atomic
operations for cross-thread consistency since we don't have multiple
threads (thanks to kj event loops being confined to a single thread)

Change-Id: Ie2bcb107f3a2c0185138330f7cbba4cec6cbdd95
2024-09-26 16:32:02 +00:00
eldritch horrors 4b66e1e24f fix internal-api-docs build
this one is also run from a gcc stdenv.

Change-Id: I91ff6915c6689ece15224f348f54367cff5d2b5a
2024-09-26 16:11:43 +00:00
puck 37b22dae04 Fix passing custom CA files into the builtin:fetchurl sandbox
Without this, verifying TLS certificates would fail on macOS, as well
as any system that doesn't have a certificate file at /etc/ssl/certs/ca-certificates.crt,
which includes e.g. Fedora.

Change-Id: Iaa2e0e9db3747645b5482c82e3e0e4e8f229f5f9
2024-09-26 15:25:28 +00:00
Maximilian Bosch 31954b5136 Merge "flake: use clangStdenv for overlays.default" into main 2024-09-26 07:13:41 +00:00
jade acf963468f flake.lock: update everything
Periodic updating of everything.

Change-Id: Ie006fd7c2dc0725309dea831685ea7b24b569df5
2024-09-25 18:42:16 -07:00
Eelco Dolstra c1631b0a39 [security] builtin:fetchurl: Enable TLS verification
This is better for privacy and to avoid leaking netrc credentials in a
MITM attack, but also the assumption that we check the hash no longer
holds in some cases (in particular for impure derivations).

Partially reverts 5db358d4d7.

(cherry picked from commit c04bc17a5a0fdcb725a11ef6541f94730112e7b6)
(cherry picked from commit f2f47fa725fc87bfb536de171a2ea81f2789c9fb)
(cherry picked from commit 7b39cd631e0d3c3d238015c6f450c59bbc9cbc5b)

Upstream-PR: https://github.com/NixOS/nix/pull/11585

Change-Id: Ia973420f6098113da05a594d48394ce1fe41fbb9
2024-09-25 18:40:58 -07:00
jade aca19187d0 fmt: fail hard on bad format strings going into nix::fmt too
Previously we would only crash the program for bad HintFmt calls.
nix::fmt should also crash.

Change-Id: I4ba0abeb8557b208bd9c0be624c022a60446ef7e
2024-09-25 15:20:48 -07:00
jade 19e0ce2c03 main: log stack traces for std::terminate
These stack traces kind of suck for the reasons mentioned on the
CppTrace page here (no symbols for inline functions is a major one):
https://github.com/jeremy-rifkin/cpptrace

I would consider using CppTrace if it were packaged, but to be honest, I
think that the more reasonable option is actually to move entirely to
out-of-process crash handling and symbolization.

The reason for this is that if you want to generate anything of
substance on SIGSEGV or really any deadly signal, you are stuck in
async-signal-safe land, which is not a place to be trying to run a
symbolizer. LLVM does it anyway, probably carefully, and chromium *can*
do it on debug builds but in general uses crashpad:
https://source.chromium.org/chromium/chromium/src/+/main:base/debug/stack_trace_posix.cc;l=974;drc=82dff63dbf9db05e9274e11d9128af7b9f51ceaa;bpv=1;bpt=1

However, some stack traces are better than *no* stack traces when we get
mystery exceptions falling out the bottom of the program. I've also
promoted the path for "mystery exceptions falling out the bottom of the
program" to hard crash and generate a core dump because although there's
been some months since the last one of these, these are nonetheless
always *atrociously* diagnosed.

We can't improve the crash handling further until either we use Crashpad
(which involves more C++ deps, no thanks) or we put in the ostensibly
work in progress Rust minidump infrastructure, in which case we need to
finish full support for Rust in libutil first.

Sample report:

Lix crashed. This is a bug. We would appreciate if you report it at https://git.lix.systems/lix-project/lix/issues with the following information included:

Exception: std::runtime_error: lol
Stack trace:
 0# nix::printStackTrace() in /home/jade/lix/lix3/build/src/nix/../libutil/liblixutil.so
 1# 0x000073C9862331F2 in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
 2# 0x000073C985F2E21A in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
 3# 0x000073C985F2E285 in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
 4# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
 5# 0x00005CF65B6B048B in /home/jade/lix/lix3/build/src/nix/nix
 6# 0x000073C985C8810E in /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
 7# __libc_start_main in /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
 8# 0x00005CF65B610335 in /home/jade/lix/lix3/build/src/nix/nix

Change-Id: I1a9f6d349b617fd7145a37159b78ecb9382cb4e9
2024-09-25 14:03:45 -07:00
jade 8a6b84df14 Merge "package.nix: fix cross for editline" into main 2024-09-25 20:23:39 +00:00
Maximilian Bosch eccbe9586a
flake: use clangStdenv for overlays.default
We don't support GCC anymore for building, so the overlay currently
fails to evaluate with

    error: assertion '((stdenv).cc.isClang || lintInsteadOfBuild)' failed

`clangStdenv` seems like a reasonable default now.

Noticed while upgrading Lix for our Hydra fork.

Change-Id: I948a7c03b3e5648fc7c596f96e1b8053a9e7f92f
2024-09-25 18:31:34 +02:00
Olivia Crain 2f794733b2
internal-api-docs: allow Doxygen to build regardless of workdir
Previously, Doxygen needed to be ran from the project's source root dir
due to the relative paths in the config's `INPUT` tag. We now preprocess
the relative paths by prefixing them with the absolute path of the
project's source root dir. The HTML output remains unchanged.

Fixes: lix-project/lix#240
Change-Id: I85f099c22bfc5fdbf26be27c2db7dcbc8155c8b2
2024-09-24 13:26:22 -05:00
alois31 5f298f74c9 Merge "local-store: make extended attribute handling more robust" into main 2024-09-21 07:55:13 +00:00
jade 79246a3733 Merge "util: fix brotli decompression of empty input" into main 2024-09-18 23:36:25 +00:00
jade 789b19a0cf util: fix brotli decompression of empty input
This caused an infinite loop before since it would just keep asking the
underlying source for more data.

In practice this happened because an HTTP server served a
response to a HEAD request (for which curl will not retrieve any body or
call our write callback function) with Content-Encoding: br, leading to
decompressing nothing at all and going into an infinite loop.

This adds a test to make sure none of our compression methods do that
again, as well as just patching the HTTP client to never feed empty data
into a compression algorithm (since they absolutely have the right to
throw CompressionError on unexpectedly-short streams!).

Reported on Matrix: https://matrix.to/#/!lymvtcwDJ7ZA9Npq:lix.systems/$8BWQR_zKxCQDJ40C5NnDo4bQPId3pZ_aoDj2ANP7Itc?via=lix.systems&via=matrix.org&via=tchncs.de

Change-Id: I027566e280f0f569fdb8df40e5ecbf46c211dad1
2024-09-18 15:37:29 -07:00
Rebecca Turner 0943b214c9 Merge "tests/compression: rewrite" into main 2024-09-18 20:38:18 +00:00
alois31 2afdf1ed66
path-info: wipe the progress bar before printing
The legitimate output of `nix path-info` may visually interfere with the
progress bar, by appending to stale progress output before the latter has been
erased. Conveniently, all expensive operations (evaluation or building) have
already been performed before, so we can simply wipe the progress bar at this
point to fix the issue.

Fixes: lix-project/lix#343
Change-Id: Id9a807a5c882295b3e6fbf841f9c15dc96f67f6e
2024-09-18 19:26:40 +02:00
jade ed381cd58a package.nix: fix cross for editline
editline's dep on ncurses is a runtime one, so it should be in
buildInputs, not nativeBuildInputs.

CC: lix-project/lix#527
Change-Id: I631c192a55677b0cc77faa7511986f1fa2205e91
2024-09-17 20:43:21 -07:00
jade 4046e019ca tests/compression: rewrite
This test suite was in desperate need of using the parameterization
available with gtest, and was a bunch of useless duplicated code. At
least now it's not duplicated code, though it still probably should be
more full of property tests.

Change-Id: Ia8ccee7ef4f02b2fa40417b79aa8c8f0626ea479
2024-09-17 19:07:48 -07:00
Rebecca Turner 8ab5743904 Merge "Remove readline support" into main 2024-09-17 16:36:34 +00:00
Rebecca Turner 7ae0409989
Remove readline support
Lix cannot be built with GNU readline, and we would "rather not" be GPL.

Change-Id: I0e86f0f10dab966ab1d1d467fb61fd2de50c00de
2024-09-16 10:48:20 -07:00
Maximilian Bosch 80202e3ca3
common-eval-args: raise warning if --arg isn't a valid Nix identifier
See lix-project/lix#496.

The core idea is to be able to do e.g.

    nix-instantiate -A some-nonfree-thing --arg config.allowUnfree true

which is currently not possible since `config.allowUnfree` is
interpreted as attribute name with a dot in it.

In order to change that (probably), Jade suggested to find out if there
are any folks out there relying on this behavior.

For such a use-case, it may still be possible to accept strings, i.e.
`--arg '"config.allowUnfree"'.

Change-Id: I986c73619fbd87a95b55e2f0ac03feaed3de2d2d
2024-09-15 16:52:30 +02:00
jade 727258241f fix: docs issue template was busted
Apparently forgejo has a more creative interpretation of \(\) than I was
hoping in their markdown parser and thought it was maths. I have no idea
then how you put a link in parens next to another square-bracket link,
but I am not going to worry about it.

There were several more typos, which I also fixed.

Fixes: lix-project/lix#517
Change-Id: I6b144c6881f92ca60ba72a304ce7a0bcb9c6659a
2024-09-14 19:28:46 +00:00
jade 5246cea6c8 Merge "store: add a hint on how to fix Lix installs broken by macOS Sequoia" into main 2024-09-14 19:28:24 +00:00
jade 8f88590d13 Merge changes Ia1481da4,Ifca1d74d into main
* changes:
  archive: refactor bad mutable-state API in the NAR parse listener
  archive: rename ParseSink to NARParseVisitor
2024-09-14 19:26:08 +00:00
alois31 3f07c65510
local-store: make extended attribute handling more robust
* Move the extended attribute deletion after the hardlink sanity check. We
  shouldn't be removing extended attributes on random files.
* Make the entity owner-writable before attempting to remove extended
  attributes, since this operation usually requires write access on the file,
  and we shouldn't fail xattr deletion on a file that has been made unwritable
  by the builder or a previous canonicalisation pass.

Fixes: lix-project/lix#507
Change-Id: I7e6ccb71649185764cd5210f4a4794ee174afea6
2024-09-14 10:36:22 +02:00
jade b7fc37b015 store: add a hint on how to fix Lix installs broken by macOS Sequoia
This is not a detailed diagnosis, and it's not worth writing one, tbh.
This error basically never happens in normal operation, so diagnosing it
by changing the error on macOS is good enough.

Relevant: lix-project/lix-installer#24
Relevant: lix-project/lix-installer#18
Relevant: lix-project/lix#521

Change-Id: I03701f917d116575c72a97502b8e1617679447f2
2024-09-14 07:31:30 +00:00
jade ca1dc3f70b archive: refactor bad mutable-state API in the NAR parse listener
Remove the mutable state stuff that assumes that one file is being
written a time. It's true that we don't write multiple files
interleaved, but that mutable state is evil.

Change-Id: Ia1481da48255d901e4b09a9b783e7af44fae8cff
2024-09-13 17:11:43 -07:00
alois31 b2fc007811 Merge "fish-completion: leave the shell prompt intact" into main 2024-09-12 06:14:22 +00:00
alois31 82aa1ccab4
fish-completion: leave the shell prompt intact
When generating shell completions, no logging output should be visible because
it would destroy the shell prompt. Originally this was attempted to be done by
simply disabling the progress bar (ca946860ce),
since the situation is particularly bad there (the screen clearing required for
the rendering ends up erasing the shell prompt). Due to overlooking the
implementation of this hack, it was accidentally undone during a later change
(0dd1d8ca1c).
Since even with the hack correctly in place, it is still possible to mess up
the prompt by logging output (for example warnings for disabled experimental
features, or messages generated by `builtins.trace`), simply send it to the bit
bucket where it belongs. This was already done for bash and zsh
(9d840758a8), and it seems that fish was simply
missed at that time. The last trace of the no-longer-working and obsolete hack
is deleted too.

Fixes: lix-project/lix#513
Change-Id: I59f1ebf90903034e2059298fa8d76bf970bc3315
2024-09-11 19:03:11 +02:00
Rebecca Turner df0137226d
editline: Vendor cl/1883 patch to recognize Alt+Left/Alt+Right
This vendors the patch added in cl/1883 to avoid GitHub
garbage-collecting the commits we're referring to.

As @emilazy pointed out on GitHub:

> GitHub can garbage‐collect unmerged PR commits if they are later
> force‐pushed, which means that code review in upstreams can cause
> Nixpkgs builds to fail to reproduce in future.

See: https://github.com/NixOS/nixpkgs/pull/341131#discussion_r1753046220
See: https://github.com/troglobit/editline/pull/70
See: https://gerrit.lix.systems/c/lix/+/1883

Change-Id: Ifff522f7f23310d6dbe9efc72fd40be5500ae872
2024-09-11 09:35:00 -07:00
jade 81c2e0ac8e archive: rename ParseSink to NARParseVisitor
- Rename the listener to not be called a "sink". If it were a "sink" it
  would be eating bytes and conform with any of the Nix sink stuff
  (maybe FileHandle should be a Sink itself! but that's a later CL's
  problem). This is a parser listener.
- Move the RetrieveRegularNARSink thing into store-api.cc, which is its
  only usage, and fix it to actually do what it is stated to do: crash
  if its invariants are violated.

  It's, of course, used to erm, unpack single-file NAR files, generated
  via a horrible contraption of sources and sinks that looks like a
  plumbing blueprint. Refactoring that is a future task.
- Add a description of the invariants of NARParseVisitor in preparation
  of refactoring it.

Change-Id: Ifca1d74d2947204a1f66349772e54dad0743e944
2024-09-11 01:10:49 -07:00
Rebecca Turner 24db81eaf2 Merge "repl: Patch editline to recognize Meta-Left & Meta-Right" into main 2024-09-11 01:02:27 +00:00
Rebecca Turner cc183fdbc1 Merge "repl-overlays: Provide an elaborate example" into main 2024-09-10 00:17:01 +00:00
Rebecca Turner f5ae72d445 Merge "Add getCwd" into main 2024-09-10 00:10:40 +00:00
Rebecca Turner 6de6cae3e7
repl: Patch editline to recognize Meta-Left & Meta-Right
This applies https://github.com/troglobit/editline/pull/70 to our build
of editline, which translates `meta-left` and `meta-right` into
`fd_word` and `bk_word`. This makes `nix repl` soooo much nicer to use!

Note: My terminal renders `meta-left` as `\e\e[C` and `meta-right` as
`\e\e[D`.

Closes lix-project/lix#501

Change-Id: I048b10cf17231bbf4e6bf38e1d1d8572cedaa194
2024-09-09 15:34:50 -07:00
alois31 8f7ab26f96 Merge changes If8ec210f,I6e2851b2 into main
* changes:
  libfetchers: serialise accept-flake-config properly
  libstore: declare SandboxMode JSON serialisation in the header
2024-09-09 16:14:23 +00:00
eldritch horrors c14486ae8d forbid gcc for compilation, only allow clang
while gcc 12 and older miscompile our generators, gcc 13 and older
outright crash on kj coroutines. (newer gcc versions may fix this)

Change-Id: I19f12c8c147239680eb0fa5a84ef5c7de38c9263
2024-09-09 01:48:20 +00:00
alois31 e9505dcc5a Merge "libmain/progress-bar: erase all lines of the multi-line format" into main 2024-09-08 13:42:17 +00:00
eldritch horrors f2a49032a6 libstore: turn Worker in a kj event loop user
using a proper event loop basis we no longer have to worry about most of
the intricacies of poll(), or platform-dependent replacements for it. we
may even be able to use the event loop and its promise system for all of
our scheduling in the future. we don't do any real async processing yet,
this is just preparation to separate the first such change from the huge
api design difference with the async framework we chose (kj from capnp):

kj::Promise, unlike std::future, doesn't return exceptions unmangled. it
instead wraps any non-kj exception into a kj exception, erasing all type
information and preserving mostly the what() string in the process. this
makes sense in the capnp rpc use case where unrestricted exception types
can't be transferred, and since it moves error handling styles closer to
a world we'd actually like there's no harm in doing it only here for now

Change-Id: I20f888de74d525fb2db36ca30ebba4bcfe9cc838
2024-09-08 01:57:48 +00:00
eldritch horrors 92eccfbd68 libutil: add a result type using boost outcome
we're using boost::outcome rather than leaf or stl types because stl
types are not available everywhere and leaf does not provide its own
storage for error values, relying on thread-locals and the stack. if
we want to use promises we won't have a stack and would have to wrap
everything into leaf-specific allocating wrappers, so outcome it is.

Change-Id: I35111a1f9ed517e7f12a839e2162b1ba6a993f8f
2024-09-08 01:57:48 +00:00
alois31 4715d557ef
libmain/progress-bar: erase all lines of the multi-line format
When the multi-line log format is enabled, the progress bar usually occupies
multiple lines on the screen. When stopping the progress bar, only the last
line was wiped, leaving all others visible on the screen. Erase all lines
belonging to the progress bar to prevent these leftovers.
Asking the user for input is theoretically affected by a similar issue, but
this is not observed in practice since the only place where the user is asked
(whether configuration options coming from flakes should be accepted) does not
actually have multiple lines on the progress bar. However, there is no real
reason to not fix this either, so let's do it anyway.

Change-Id: Iaa5a701874fca32e6f06d85912835d86b8fa7a16
2024-09-07 10:37:12 +02:00
alois31 991d8ce275 Merge "Stop the logger in legacy commands again" into main 2024-09-06 17:07:16 +00:00
Rebecca Turner 72589e7032 Merge "Test including relative paths in configuration" into main 2024-09-03 18:48:47 +00:00
Rebecca Turner 644176a631 Merge "Expand comment on std::string operator+" into main 2024-09-03 18:48:37 +00:00
alois31 63ee2cdda3
libfetchers: serialise accept-flake-config properly
The AcceptFlakeConfig type used was missing its JSON serialisation definition,
so it was incorrectly serialised as an integer, ending up that way for example
in the nix.conf manual page. Declare a proper serialisation.

Change-Id: If8ec210f9d4dd42fe480c4e97d0a4920eb66a01e
2024-09-02 18:50:15 +02:00
alois31 d7c37324bb
libstore: declare SandboxMode JSON serialisation in the header
The JSON serialisation should be declared in the header so that all translation
units can see it when needed, even though it seems that it has not been used
anywhere else so far. Unfortunately, this means we cannot use the
NLOHMANN_JSON_SERIALIZE_ENUM convenience macro, since it uses a slightly
different signature, but the code is not too bad either.

Change-Id: I6e2851b250e0b53114d2fecb8011ff1ea9379d0f
2024-09-02 18:50:14 +02:00
Rebecca Turner 75c0de3e3c
Test including relative paths in configuration
Change-Id: If6c69a5e16d1ccd223fba392890f08f0032fb754
2024-09-01 15:52:48 -07:00
Rebecca Turner fc4a160878
repl-overlays: Provide an elaborate example
This is the repl overlay from my dotfiles, which I think provides a
reasonable and ergonomic set of variables. We can iterate on this over
time, or (perhaps?) provide a sentinel value like `repl-overlays =
<DEFAULT>` to include a "suggested default" overlay like this one.

Change-Id: I8eba3934c50fbac8367111103e66c7375b8d134e
2024-09-01 15:30:58 -07:00
Rebecca Turner b7b1b9723f
Clarify that diff-hook no longer needs to be an absolute path
See: https://gerrit.lix.systems/c/lix/+/1864
Change-Id: Ic70bfe42b261a83f2cb68b8f102833b739b8e03a
2024-09-01 15:20:09 -07:00
Rebecca Turner 9d8f433246
Expand comment on std::string operator+
Nuts!

Change-Id: Ib5bc0606d7c86e57ef76dd7bcc89dce91bd3d50a
2024-09-01 15:10:31 -07:00
Rebecca Turner 02eb07cfd5 Merge changes I5566a985,I88cf53d3 into main
* changes:
  Support relative and `~/` paths in config settings
  Thread `ApplyConfigOptions` through config parsing
2024-09-01 22:06:36 +00:00
eldritch horrors d75df91f74 libstore: add build result to Goal::Finished
it just makes sense to have it too, rather than just the pass/fail
information we keep so far. once we turn goals into something more
promise-shaped it'll also help detangle the current data flow mess

Change-Id: I915cf04d177cad849ea7a5833215d795326f1946
2024-08-30 19:01:30 +02:00
eldritch horrors a385c5935a libstore: rename Goal::Finished::result to exitCode
the more useful type for `result` is BuildResult.

Change-Id: If93d9384e8d686eb63b33320f1d565f9b9afbf3a
2024-08-30 19:01:30 +02:00
eldritch horrors dc0cace604 libstore: remove queryMissing call from Worker
it doesn't have a purpose except cache priming, which is largely
irrelevant by default (since another code path already runs this
exact query). our store implementations do not benefit that much
from this either, and the more bursty load may indeed harm them.

Change-Id: I1cc12f8c21cede42524317736d5987f1e43fc9c9
2024-08-30 19:01:30 +02:00
eldritch horrors e0fd0ba211 libstore: use notifications for stats counters
updating statistics *immediately* when any counter changes declutters
things somewhat and makes useful status reports less dependent on the
current worker main loop. using callbacks will make it easier to move
the worker loop into kj entirely, using only promises for scheduling.

Change-Id: I695dfa83111b1ec09b1a54cff268f3c1d7743ed6
2024-08-30 19:01:30 +02:00
eldritch horrors c2b90d235f libstore: don't ContinueImmediately where we can tail call
there's no reason to go through the event loop in these cases. returning
ContinueImmediately here is just a very convoluted way of jumping to the
state we've just set after unwinding one frame of the stack, which never
matters in the cases changed here because there are no live RAII guards.

Change-Id: I7c00948c22e3caf35e934c1a14ffd2d40efc5547
2024-08-30 19:01:30 +02:00
eldritch horrors e55ec75619 libstore: print dependency errors from DerivationGoal
this is not ideal, but it's better than having this stuck in the worker
loop itself. setting ex on all failing goals is not problematic because
only toplevel goals can ever be observable, all the others are ignored.
notably only derivation goals ever set `ex`, substitution goals do not.

Change-Id: I02e2164487b2955df053fef3c8e774d557aa638a
2024-08-30 11:13:07 +00:00
eldritch horrors 869666cb65 libstore: hide Worker goal factory methods
this doesn't serve a great purpose yet except to confine construction of
goals to the stack frame of Worker::run() and its child frames. we don't
need this yet (and the goal constructors remain fully visible), but in a
future change that fully removes the current worker loop we'll need some
way of knowing which goals are top-level goals without passing the goals
themselves around. once that's possible we can remove visible goals as a
concept and rely on build result futures and a scheduler built upon them

Change-Id: Ia73cdeffcfb9ba1ce9d69b702dc0bc637a4c4ce6
2024-08-30 10:18:28 +00:00
eldritch horrors a5c1e73fa8 libstore: add "is dependency" info to goal
whether goal errors are reported via the `ex` member or just printed to
the log depends on whether the goal is a toplevel goal or a dependency.
if goals are aware of this themselves we can move error printing out of
the worker loop, and since a running worker can only be used by running
goals it's totally sufficient to keep a `Worker::running` flag for this

Change-Id: I6b5cbe6eccee1afa5fde80653c4b968554ddd16f
2024-08-30 10:18:28 +00:00
jade bb161a96cf manual: note that __sandboxProfile allows bypassing the darwin sandbox
(but only if it is set to relaxed. no security hole here.)

Thanks to lilyball for pointing out this omission in the docs.

Change-Id: I2408a943bfe817fe660fe1c8fefef898aaf5f7e9
2024-08-29 17:06:39 -07:00
jade a510d17484
build-time: hide boost stacktrace in a .cc file
Saves about 16s of CPU time. Not a lot but not nothing. Feels more like
the principle of the thing.

Change-Id: I0992d4024317c20d6985a7977d5649edfb9f46bb
2024-08-28 09:55:09 -07:00
jade 04f8a14833
tree-wide: shuffle headers around for about 30s compile time
This didn't really feel so worth it afterwards, but I did untangle a
bunch of stuff that should not have been tangled.

The general gist of this change is that variant bullshit was causing a
bunch of compile time, and it seems like the only way to deal with
variant induced compile time is to keep variant types out of headers.
Explicit template instantiation seems to do nothing for them.

I also seem to have gotten some back-end time improvement from
explicitly instantiating regex, but I don't know why. There is no
corresponding front-end time improvement from it: regex is still at the
top of the sinners list.

**** Templates that took longest to instantiate:
 15231 ms: std::basic_regex<char>::_M_compile (28 times, avg 543 ms)
 15066 ms: std::__detail::_Compiler<std::regex_traits<char>>::_Compiler (28 times, avg 538 ms)
 12571 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_disjunction (28 times, avg 448 ms)
 12454 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_alternative (28 times, avg 444 ms)
 12225 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_term (28 times, avg 436 ms)
 11363 ms: nlohmann::basic_json<>::parse<const char *> (21 times, avg 541 ms)
 10628 ms: nlohmann::basic_json<>::basic_json (109 times, avg 97 ms)
 10134 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_atom (28 times, avg 361 ms)

Back-end time before messing with the regex:
**** Function sets that took longest to compile / optimize:
  8076 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms)
  4382 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1247 times, avg 3 ms)
  3137 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 22 ms)
  2896 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 16 ms)
  2304 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (210 times, avg 10 ms)
  2116 ms: bool std::__detail::_Compiler<$>::_M_expression_term<$>(std::__detai... (112 times, avg 18 ms)
  2051 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (244 times, avg 8 ms)
  2037 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 21 ms)
  1928 ms: std::__detail::_Compiler<$>::_M_quantifier() (28 times, avg 68 ms)
  1859 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (41 times, avg 45 ms)
  1824 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (973 times, avg 1 ms)
  1810 ms: std::__detail::_BracketMatcher<$>::_BracketMatcher(std::__detail::_B... (112 times, avg 16 ms)
  1793 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1793 ms)
  1759 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (281 times, avg 6 ms)
  1722 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 90 ms)
  1677 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms)
  1674 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (249 times, avg 6 ms)
  1660 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (304 times, avg 5 ms)
  1599 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 84 ms)
  1568 ms: void std::__detail::_Compiler<$>::_M_insert_bracket_matcher<$>(bool) (112 times, avg 14 ms)
  1541 ms: std::__shared_ptr<$>::~__shared_ptr() (531 times, avg 2 ms)
  1539 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (41 times, avg 37 ms)
  1471 ms: void std::__detail::_Compiler<$>::_M_insert_character_class_matcher<... (112 times, avg 13 ms)

After messing with the regex (notice std::__detail::_Compiler vanishes
here, but I don't know why):

**** Function sets that took longest to compile / optimize:
  8054 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms)
  4313 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1217 times, avg 3 ms)
  3259 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 23 ms)
  3045 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 17 ms)
  2314 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (207 times, avg 11 ms)
  1923 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (216 times, avg 8 ms)
  1817 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 100 ms)
  1816 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 19 ms)
  1788 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (40 times, avg 44 ms)
  1749 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (278 times, avg 6 ms)
  1724 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (248 times, avg 6 ms)
  1697 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms)
  1684 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1684 ms)
  1680 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (303 times, avg 5 ms)
  1589 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 88 ms)
  1483 ms: non-virtual thunk to boost::wrapexcept<$>::~wrapexcept() (181 times, avg 8 ms)
  1447 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (40 times, avg 36 ms)
  1441 ms: std::__shared_ptr<$>::~__shared_ptr() (496 times, avg 2 ms)
  1420 ms: boost::stacktrace::basic_stacktrace<$>::init(unsigned long, unsigned... (137 times, avg 10 ms)
  1396 ms: boost::basic_format<$>::~basic_format() (194 times, avg 7 ms)
  1290 ms: std::__cxx11::basic_string<$>::_M_replace_cold(char*, unsigned long,... (231 times, avg 5 ms)
  1258 ms: std::vector<$>::~vector() (354 times, avg 3 ms)
  1222 ms: std::__cxx11::basic_string<$>::_M_replace(unsigned long, unsigned lo... (231 times, avg 5 ms)
  1194 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (49 times, avg 24 ms)
  1186 ms: bool tao::pegtl::internal::sor<$>::match<$>(std::integer_sequence<$>... (1 times, avg 1186 ms)
  1149 ms: std::__detail::_Executor<$>::_M_dfs(std::__detail::_Executor<$>::_Ma... (70 times, avg 16 ms)
  1123 ms: toml::detail::sequence<$>::invoke(toml::detail::location&) (69 times, avg 16 ms)
  1110 ms: nlohmann::json_abi_v3_11_3::basic_json<$>::json_value::destroy(nlohm... (55 times, avg 20 ms)
  1079 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (541 times, avg 1 ms)
  1033 ms: nlohmann::json_abi_v3_11_3::detail::lexer<$>::scan_number() (20 times, avg 51 ms)

Change-Id: I10af282bcd4fc39c2d3caae3453e599e4639c70b
2024-08-28 09:55:05 -07:00
jade e6f2af06e6
clang-tidy: fix the fact that we are not linting headers properly
This, however, took fixing a pile of lints that we predictably missed
because of this bug.

Change-Id: I92c36feb4a03f62bc594c2051c7bd7418d13fb08
2024-08-28 09:52:08 -07:00
jade 4d89844207
build: remove about 30 cpu-sec of compile time by explicit instantiation
Apparently the fmt contraption has some extremely popular overloads, and
the boost stuff in there gets built approximately infinite times in
every compilation unit.

Change-Id: Ideba2db7d6bf8559e4d91974bab636f5ed106198
2024-08-28 09:52:05 -07:00
Rebecca Turner 422550fd68 Merge "libstore: remove static initializers for Store registrations" into main 2024-08-28 16:43:22 +00:00
jade 5d31e889d7 Merge "treewide: fix a bunch of lints" into main 2024-08-28 03:40:27 +00:00
Pierre Bourdon 4f02255c20
libstore: remove static initializers for Store registrations
Ref #359.

Change-Id: Ia45530ddee25fa9fc399ff10738bb0d8bbc8b221
2024-08-26 16:27:31 -07:00
Rebecca Turner 0dc486a5bf Merge "Fix comment in getHome" into main 2024-08-26 23:17:04 +00:00
jade 0cc285f87b
treewide: fix a bunch of lints
Fixes:
- Identifiers starting with _ are prohibited
- Some driveby header dependency cleaning which wound up with doing some
  extra fixups.
- Fucking C style casts, man. C++ made these 1000% worse by letting you
  also do memory corruption with them with references.
  - Remove casts to Expr * where ExprBlackHole is an incomplete type by
    introducing an explicitly-cast eBlackHoleAddr as Expr *.
  - An incredibly illegal cast of the text bytes of the StorePath hash
    into a size_t directly. You can't DO THAT.

    Replaced with actually parsing the hash so we get 100% of the bits
    being entropy, then memcpying the start of the hash. If this shows
    up in a profile we should just make the hash parser faster with a
    lookup table or something sensible like that.
  - This horrendous bit of UB which I thankfully slapped a deprecation
    warning on, built, and it didn't trigger anywhere so it was dead
    code and I just deleted it. But holy crap you *cannot* do that.

    inline void mkString(const Symbol & s)
    {
        mkString(((const std::string &) s).c_str());
    }
- Some wrong lints. Lots of wrong macro lints, one wrong
  suspicious-sizeof lint triggered by the template being instantiated
  with only pointers, but the calculation being correct for both
  pointers and not-pointers.
- Exceptions in destructors strike again. I tried to catch the
  exceptions that might actually happen rather than all the exceptions
  imaginable. We can let the runtime hard-kill it on other exceptions
  imo.

Change-Id: I71761620846cba64d66ee7ca231b20c061e69710
2024-08-26 16:13:03 -07:00
Rebecca Turner ca08f1217d
rowan: 0.15.15 -> 0.15.16
This fixes an ambiguous pointer comparison warning.

See: https://github.com/rust-analyzer/rowan/pull/162
Change-Id: Iaac2c8cab0051eb97211893ad547d8dfa8fda560
2024-08-26 11:34:43 -07:00
Rebecca Turner 742303dc3a
Add getCwd
It's nice for this to be a separate function and not just inline in
`absPath`.

Prepared as part of cl/1865, though I don't think I actually ended up
using it there.

Change-Id: I24d9d4a984cee0af587010baf04b3939a1c147ec
2024-08-26 11:22:07 -07:00
Rebecca Turner 690f07272e
Support relative and ~/ paths in config settings
Change-Id: I5566a9858ba255f4ac5051d1368c7dfb24460f0a
2024-08-25 15:54:22 -07:00
Rebecca Turner 5fc6fcb310
Thread ApplyConfigOptions through config parsing
This makes no changes to logic but makes the `ApplyConfigOptions` value
available to consumers.

Change-Id: I88cf53d38faac8472c556aee55c13d0acbd1e5db
2024-08-25 15:54:22 -07:00
eldritch horrors b6884388a1 add dedicated test for hash mismatch url reporting
the current test relies on derivation build order being deterministic,
which will not be a reasonable expectation for all that long any more.

Change-Id: I9be44a7725185f614a9a4c724045b8b1e6962c03
2024-08-25 22:21:32 +00:00
Rebecca Turner 0582999bd1 Merge "Add ApplyConfigOptions" into main 2024-08-25 22:06:45 +00:00
eldritch horrors 398894b856 libstore: make Goal::ex a shared_ptr
this makes WorkResult copyable, and just all around easier to deal with.
in the future we'll need this to let Goal::work() return a promise for a
WorkResult (or even just a Finished) that can be awaited by other goals.

Change-Id: Ic5a1ce04c5a0f8e683bd00a2ed2b77a2e28989c1
2024-08-25 21:21:55 +00:00
eldritch horrors 30a87b4cd5 libstore: remove unused Goal ctor parameter
Change-Id: I9345fe272d6df5bd592621ce2da369fc1cd36d6d
2024-08-25 20:40:19 +00:00
jade 72f91767a8 Merge "fix: good errors for failures caused by allowSubstitutes" into main 2024-08-25 20:00:58 +00:00
jade 3bf8819fa2 Merge changes Ief8e8ebc,Id3135db0,If1e76169 into main
* changes:
  libutil: delete unused boost context cruft
  build: remove approximately 400 seconds of CPU time (30%)
  fix: use http proxy for s3 access
2024-08-25 19:59:46 +00:00
Rebecca Turner c300efc0e1
Add ApplyConfigOptions
Change-Id: Ic876bcabd0b68e579bbd30ca1755919df43d4813
2024-08-25 12:18:20 -07:00
eldritch horrors cae260a158 libstore: diagnose local build failure in goal
this should be done where we're actually trying to build something, not
in the main worker loop that shouldn't have to be aware of such details

Change-Id: I07276740c0e2e5591a8ce4828a4bfc705396527e
2024-08-25 19:55:47 +02:00
eldritch horrors 04b591dc1d devShell: can we have debuggers?
macos: we have debuggers at home

(the debuggers at home: entitled little brats)

Change-Id: Iefd4b5880da97846a81d601db05d2b46530a2b58
2024-08-24 21:34:13 +02:00
jade 686120ee4a fix: good errors for failures caused by allowSubstitutes
This caused an absolute saga which I would not like anyone else to have
to experience. Let's put in a laser targeted error message that
diagnoses this exact problem.

Fixes: lix-project/lix#484
Change-Id: I2a79f04aeb4a1b67c10115e5e39501d958836298
2024-08-23 17:49:15 -07:00
Rebecca Turner fabc9f29b8
Fix comment in getHome
The logic in the comment is the opposite of the truth.

Change-Id: I64add84539209782ffa46431f3db1fb306d90b3f
2024-08-23 15:15:21 -07:00
Rebecca Turner c5949bfe31 Merge "libutil/config: unify path setting types" into main 2024-08-23 22:09:11 +00:00
jade 7e677d15a4 libutil: delete unused boost context cruft
This was from before we got rid of the boost coroutines. Now we don't
need any of this code.

Change-Id: Ief8e8ebc184f02f48e30cb253a66b540faa56329
2024-08-23 13:23:33 -07:00
jade af546be205 build: remove approximately 400 seconds of CPU time (30%)
This took parsing time from 1421s or so to 1060s or so. The reason is
entirely nlohmann. All of the stuff below is just Obliterated because it's
built in the PCH instead:

**** Templates that took longest to instantiate:
219051 ms: nlohmann::basic_json<>::parse<const char *> (276 times, avg 793 ms)
169675 ms: nlohmann::basic_json<>::basic_json (1127 times, avg 150 ms)
129416 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (276 times, avg 468 ms)
 98155 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (276 times, avg 355 ms)
 81322 ms: nlohmann::basic_json<>::json_value::json_value (1405 times, avg 57 ms)
 53531 ms: nlohmann::detail::json_sax_dom_callback_parser<nlohmann::basic_json<... (276 times, avg 193 ms)

clang-only. This brings the clang build time to not far from *half* of
the gcc build time.

Also, clang does not enjoy so much to miscompile coroutines. Maybe we
should just be clang-only.

Change-Id: Id3135db0094e4560830674090e32e6da2c22fcc6
2024-08-23 13:23:33 -07:00
jade 9aacf425dc fix: use http proxy for s3 access
I don't know why the AWS sdk disabled it by default. It would be nice
to have test coverage of the s3 store or proxies, but neither currently
exist.

Fixes: lix-project/lix#433
Change-Id: If1e76169a3d66dbec2e926af0d0d0eccf983b97b
2024-08-23 13:23:33 -07:00
Rebecca Turner 9845637359
lix-clang-tidy: Require Clang >= 16
Nixpkgs Clang on macOS is 16, not 17.

nix-repl> packages.aarch64-darwin.nix-clangStdenv.stdenv.cc
«derivation /nix/store/ycych9qpim4r42hjkznl8f6zmj0jns45-clang-wrapper-16.0.6.drv»

nix-repl> packages.x86_64-linux.nix-clangStdenv.stdenv.cc
«derivation /nix/store/y48dhgidb2vs230r9ayim14q61xwcdg9-clang-wrapper-17.0.6.drv»

Change-Id: Ib267b8882f80eef4db665fb9df50ae285ea68b2b
2024-08-23 12:17:01 -07:00
jade 87fd6e0095 Merge "Revert "libexpr: Replace regex engine with boost::regex"" into main 2024-08-22 22:34:10 +00:00
jade 9896d309cb Revert "libexpr: Replace regex engine with boost::regex"
This reverts commit 447212fa65.

Reason for revert: Regression in eval behaviour bug-compatibility.

Expected behaviour (Nix 2.18.5, macOS and Linux [libstdc++/libc++]):

```
nix-repl> builtins.match "\\.*(.*)" ".keep"
[ "keep" ]

nix-repl> builtins.match "(\\.*)(.*)" ".keep"
[ "." "keep" ]
```

Actual behaviour (boost::regex):

```
nix-repl> builtins.match "\\.*(.*)" ".keep"
[ ".keep" ]

nix-repl> builtins.match "(\\.*)(.*)" ".keep"
[
  "."
  "keep"
]
```

Bug: lix-project/lix#483
Change-Id: Id462eb8586dcd54856cf095f09b3e3a216955b60
2024-08-22 18:35:11 +00:00
sugar🍬🍭🏳️‍⚧️ f2e7f8bab8 Merge "libexpr: Replace regex engine with boost::regex" into main 2024-08-22 07:20:00 +00:00
sugar🍬🍭🏳️‍⚧️ 447212fa65 libexpr: Replace regex engine with boost::regex
This avoids C++'s standard library regexes, which aren't the same
across platforms, and have many other issues, like using stack
so much that they stack overflow when processing a lot of data.

To avoid backwards and forward compatibility issues, regexes are
processed using a function converting libstdc++ regexes into Boost
regexes, escaping characters that Boost needs to have escaped, and
rejecting features that Boost has and libstdc++ doesn't.

Related context:

- Original failed attempt to use `boost::regex` in CppNix, failed due to
  boost icu dependency being large (disabling ICU is no longer necessary
  because linking ICU requires using a different header file,
  `boost/regex/icu.hpp`): https://github.com/NixOS/nix/pull/3826

- An attempt to use PCRE, rejected due to providing less backwards
  compatibility with `std::regex` than `boost::regex`:
  https://github.com/NixOS/nix/pull/7336

- Second attempt to use `boost::regex`, failed due to `}` regex failing
  to compile (dealt with by writing a wrapper that parses a regular
  expression and escapes `}` characters):
  https://github.com/NixOS/nix/pull/7762

Closes #34. Closes #476.

Change-Id: Ieb0eb9e270a93e4c7eed412ba4f9f96cb00a5fa4
2024-08-22 03:17:55 +02:00
jade 651cc0e5b4 fix: build with meson 1.5 also
nixpkgs delivered us the untimely gift of a meson 1.5 upgrade, which
*does* make our lives easier by allowing us to delete wrap generation
code, but it does so at the cost of renaming all rust crates in such a
way that the wrap logic cannot tolerate the new names on the old meson
version 😭.

It also means that support burden for this is going to be atrocious
until we either give in and vendor meson 1.5 or we make a CI target for
it. Neither seems appealing, though the latter is not super absurd for
ensuring we don't break nixpkgs unstable.

This commit causes meson 1.5 to ignore the .wrap files in subprojects/
entirely (since they have the wrong names lol) and instead use
Cargo.lock, so it now hard-depends on our workspace reshuffling
improvement.

It also deletes the hack that we were using to get the sources of Cargo
deps into meson by using a feature that went unnoticed when this code
was originally written: MESON_PACKAGE_CACHE_DIR:
8a202de6ec/mesonbuild/wrap/wrap.py (L490-L502)

Change-Id: I7a28f12fc2812c6ed7537b60bc3025c141a05874
2024-08-21 17:09:10 +00:00
jade dba615098d build: move to a Cargo workspace
This is purely to let Cargo's dependency resolver do stuff for us, we do
not actually intend to build this stuff with Cargo to begin with.

Change-Id: I4c08d55595c7c27b7096375022581e1e34308a87
2024-08-21 17:09:10 +00:00
alois31 e3c289dbe9
libutil/config: unify path setting types
There have been multiple setting types for paths that are supposed to be
canonicalised, depending on whether zero or one, one, or any number of paths is
to be specified. Naturally, they behaved in slightly different ways in the
code. Simplify things by unifying them and removing special behaviour (mainly
the "multiple paths type can coerce to boolean" thing).

Change-Id: I7c1ce95e9c8e1829a866fb37d679e167811e9705
2024-08-21 17:57:23 +02:00
alois31 de552c42cb
Stop the logger in legacy commands again
Commit 0dd1d8ca1c included an accidental revert
of 1461e6cdda (actually slightly worse), leading
to the progress bar not being stopped properly when a legacy command was
invoked with `--log-format bar` (or similar options that show a progress bar).
Move the progress bar stopping code to its proper place again to fix this
regression.

Change-Id: I676333da096d5990b717a387924bb988c9b73fab
2024-08-21 17:28:42 +02:00
piegames e38410799b Merge "libexpr: Soft-deprecate ancient let syntax" into main 2024-08-21 11:22:48 +00:00
piegames 0edfea450b libexpr: Soft-deprecate ancient let syntax
Change-Id: I6802b26f038578870ea1fa1ed298f0c4b1f29c4a
2024-08-21 12:59:03 +02:00
jade 3cbbe22fab Merge "flake: fix compiler warning" into main 2024-08-21 10:44:01 +00:00
piegames 0a8888d1c7 treewide: Stop using ancient let syntax
Shows for how long these tests have not been touched by anyone …

Change-Id: I3d0c1209a86283ddb012db4e7d45073264fdd0eb
2024-08-21 06:55:52 +00:00
piegames 7210ed1b87 libexpr: Soft-deprecate __overrides
Change-Id: I787e69e1dad6edc5ccdb747b74a9ccd6e8e13bb3
2024-08-21 06:55:52 +00:00
jade c25c43d8c8 flake: fix compiler warning
GCC was complaining, rightfully, about mixed-sign comparisons in there.
I removed some extra sign mixing too.

Change-Id: I949a618c7405c23d4dc3fd17440ea2d7b5c22c9d
2024-08-20 16:13:17 -07:00
Audrey Dutcher ac6974777e Merge "tests/functional/restricted: Don't use a process substitution" into main 2024-08-20 22:51:38 +00:00
jade 736b5d5913 lix-doc: move under src/
This is required to make more meson stuff easier/possible, and honestly
it *is* now Lix sources anyhow.

Change-Id: Ia6c38fabce9aa5c53768745ee38c5cf344f5c226
2024-08-20 13:38:46 -06:00
Qyriad 95863b258b build: build lix-doc with Meson! 🎉
lix-doc is now built with Meson, with lix-doc's dependencies built as
Meson subprojects, either fetched on demand with .wrap files, or fetched
in advance by Nix with importCargoLock. It even builds statically.

Fixes #256.

Co-authored-by: Lunaphied <lunaphied@lunaphied.me>
Co-authored-by: Jade Lovelace <lix@jade.fyi>

Change-Id: I3a4731ff13278e7117e0316bc0d7169e85f5eb0c
2024-08-20 17:21:13 +00:00
Yureka f1533160aa Merge "libutil: fix conditional for close_range availability" into main 2024-08-20 07:55:01 +00:00
Yureka df49d37b71 libutil: fix conditional for close_range availability
This check is wrong and would cause the close_range() function being called even when it's not available

Change-Id: Ide65b36830e705fe772196c37349873353622761
2024-08-20 08:58:25 +02:00
Audrey Dutcher ae628d4af2 tests/functional/restricted: Don't use a process substitution
The <() process substitution syntax doesn't work for this one testcase
in bash for FreeBSD. The exact reason for this is unknown, possibly to
do with pipe vs file vs fifo EOF behavior. The prior behavior was this
test hanging forever, with no children of the bash process.

Change-Id: I71822a4b9dea6059b34300568256c5b7848109ac
2024-08-19 20:37:51 -07:00
Maximilian Bosch 040e783232 flake: don't refetch unmodified inputs by recursive follows
Closes #460

I managed to trigger the issue by having the following inputs (shortened):

    authentik-nix.url = "github:nix-community/authentik-nix";
    authentik-nix.inputs.poetry2nix.inputs.nixpkgs.follows = "nixpkgs";

When evaluating this using

    nix-eval-jobs --flake .#hydraJobs

I got the following error:

    error: cannot update unlocked flake input 'authentik-nix/poetry2nix' in pure mode

The issue we have here is that `authentik-nix/poetry2nix` was written
into the `overrideMap` which caused Nix to assume it's a new input and
tried to refetch it (#460) or errored out in pure mode
(nix-eval-jobs / Hydra).

The testcase unfortunately only involves checking for the output log
and makes sure that something *is* logged on the first fetch so that
the test doesn't rot when the logging changes since I didn't
manage to trigger the error above with the reproducer from #460. In
fact, I only managed to trigger the `cannot update unlocked flake input`
error in this context with `nix-eval-jobs`.

Change-Id: Ifd00091eec9a0067ed4bb3e5765a15d027328807
2024-08-19 19:57:12 +00:00
eldritch horrors e727dbc3a3 libstore: un-enable_shared_from_this Goal
it's no longer needed for anything, and not even a great idea.

Change-Id: Ia7a59e1e3f9d8f4ad2ac3b054e38485157c210a6
2024-08-19 09:13:44 +00:00
eldritch horrors b40369942c libstore: make Worker::childStarted private
this can be a proper WorkResult now. childTerminated is unfortunately a
lot more stubborn and won't be made private for quite a while yet. once
we can get rid of the Worker poll loop that *should* be possible though

Change-Id: I2218df202da5cb84e852f6a37e4c20367495b617
2024-08-19 09:13:44 +00:00
eldritch horrors fca523d661 libstore: turn HookReply into a variant type
we'll need this once we want to pass extra information out of accepting
replies, such as fd sets or possibly even async output reader promises.

Change-Id: I5e2f18cdb80b0d2faf3067703cc18bd263329b3f
2024-08-19 09:13:44 +00:00
eldritch horrors 5e9db09761 libstore: downsize hook pipes
don't keep fds open we're not using. currently this does not cause any
problems, but it does increase the size of our fd table needlessly and
in the future, when we have proper async processing, having builderOut
open in the daemon once the hook has been fully started is problematic

Change-Id: I6e7fb773b280b042873103638d3e04272ca1e4fc
2024-08-19 09:13:44 +00:00
eldritch horrors e513cd2beb libstore: run childStarted as late as possible
otherwise we *technically* give away the output fds before we've read them.

Change-Id: I6ad0d6a1bb553ecfcdd7708f50d34142a425374d
2024-08-19 09:13:44 +00:00
eldritch horrors fb8eb539fc libstore: move respect-timeoutiness to goal method
this is useless to do on the face of it, but it'll make it easier to
convert the entire output handling to use async io and promises soon

Change-Id: I2d1eb62c4bbf8f57bd558b9599c08710a389b1a8
2024-08-19 09:13:44 +00:00
jade 3d14567d0b Merge "doc: fix broken meson deps for various manuals outputs" into main 2024-08-19 04:30:39 +00:00
jade 925e08b858 Merge "build: limit clang-tidy concurrency and respect NIX_BUILD_CORES" into main 2024-08-19 02:55:48 +00:00
eldritch horrors 5cbca85535 libstore: clarify that build log fd and hook log fd are different
only DerivationGoal can set the hook to anything at all. it always sets
buildOutFD to something that is not related to fromHook in any way, and
mixing the two would have rather dire consequences for log consistency.

Change-Id: Ida86727fd1cd5e1ecd78f07f3bde330a346658a8
2024-08-18 22:44:11 +00:00
jade ecfe9345cf build: limit clang-tidy concurrency and respect NIX_BUILD_CORES
Apparently it was impolite to lint with 128 jobs on our CI machine with
128 threads. Let's fix it.

Change-Id: I9ca7306294c6773c6f233690ba49d45a1da6bf7a
2024-08-18 15:39:05 -07:00
jade 84543b459c doc: fix broken meson deps for various manuals outputs
This is incredibly haunted, but it can happen that you change libutil,
breaking the generation of the .json files, which then does not rebuild
the files. I don't expect they are slow to build, so it does not seem so
bad to just rebuild them every time instead of extracting a list of all
the possible deps.

We want to delete this nonsense anyway and replace it with generated
code.

Change-Id: Ia576d1a3bdee48fbaefbb5ac194354428d179a84
2024-08-18 15:19:15 -07:00
eldritch horrors e2d330aeed libstore: remove DerivationGoal::isReadDesc
all derivation goals need a log fd of some description. let's save this
single fd in a dedicated pointer field for all subclasses so that later
we have just the one spot to change if we turn this into async promises

Change-Id: If223adf90909247363fb823d751cae34d25d0c0b
2024-08-18 22:04:06 +00:00
piegames 007211e7a2 libutil: Optimize feature checks
Instead of doing a linear search on an std::set, we use a bitset enum.

Change-Id: Ide537f6cffdd16d06e59aaeb2e4ac0acb6493421
2024-08-18 16:56:49 +00:00
eldritch horrors 7506d680ac libstore: don't ignore max-build-log-size for ssh-ng
Change-Id: Ieab14662bea6e6f5533325f0e945147be998f9a2
2024-08-18 09:10:05 +00:00
eldritch horrors 38f550708d libstore: add explicit in-build-slot-ness to goals
we don't need to expose information about how busy a Worker is if the
worker can instead tell its work items whether they are in a slot. in
the future we might use this to not start items waiting for a slot if
no slots are currently available, but that requires more preparation.

Change-Id: Ibe01ac536da7e6d6f80520164117c43e772f9bd9
2024-08-18 09:10:05 +00:00
eldritch horrors 176e1058f1 libstore: remove method without definition
Change-Id: I676411752a4b1777045d7211ac1176693f1a3d7d
2024-08-18 09:10:05 +00:00
eldritch horrors 91a74ba82a libstore: remove unused includes in worker code
Change-Id: I6c7fccc4e710e23a22faae2669cb75f2f6da27b4
2024-08-18 09:10:05 +00:00
eldritch horrors b66fd9ff4b libstore: make Worker::removeGoal private
Change-Id: I8583d9ff752f702a10ec52b0330b0d4d4d2614fa
2024-08-18 09:10:05 +00:00
piegames 278fddc317 libexpr: Deprecate URL literals
Closes #437.

Change-Id: I9f67fc965bb4a7e7fd849e5067ac1cb3bab064cd
2024-08-17 20:31:57 +02:00
piegames 49d61b2e4b libexpr: Introduce Deprecated features
They are like experimental features, but opt-in instead of opt-out. They
will allow us to gracefully remove language features. See #437

Change-Id: I9ca04cc48e6926750c4d622c2b229b25cc142c42
2024-08-17 19:47:51 +02:00
piegames 1c080a8239 treewide: Stop using URL literals
They must die

Change-Id: Ibe2b1818b21d98ec1a68836d01d5dad729b8c501
2024-08-17 15:48:10 +00:00
Artemis Tosini 41a0b08e64 meson: Don't use target_machine
The target_machine variable is meant for the target
of cross compilers. We are not a cross compiler, so
instead reuse our host_machine based checks.

Fixes Linux→FreeBSD cross, since Meson can't figure
out `target_machine.kernel()` in that case.

Fixes: lix-project/lix#469

Change-Id: Ia46a64c8d507c3b08987a1de1eda171ff5e50df4
2024-08-16 14:24:03 +00:00
Artemis Tosini b016eb0895 Merge "libutil: Add bindPath function from libstore" into main 2024-08-13 19:39:10 +00:00
jade f9a3bf6ccc Update version to 2.92
Change-Id: Ib64d695c50a733e0e739ff193f1ea65ed7cb0a57
2024-08-12 18:06:08 -07:00
jade 4d04adf6ba release: merge release 2.91.0 back to mainline
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh

Change-Id: I8fc975f856631dec7fb3314abd436675adabb59c
2024-08-12 16:05:27 -07:00
jade 10ac99a79c release: 2.91.0 "Dragon's Breath"
Release produced with releng/create_release.xsh

Change-Id: I2fa79b268c44b5f024dd833ee366d3e83f054af1
2024-08-12 16:05:26 -07:00
jade 7e0fee5309 release: release notes for 2.91.0
Release created with releng/create_release.xsh

Change-Id: Ieb6ca02d3cf986b28440fce3792e8c38ce80a33e
2024-08-12 16:04:22 -07:00
jade 5137cea990 README: clarify license to match documentation
For years both the documentation and nixpkgs have said that CppNix is
LGPL-2.1-or-later, not LGPL-2.1-only as is somewhat implied by the
README. We are choosing to update the README to match the rest of the
references.

Related: https://github.com/NixOS/nix/pull/5218
Change-Id: I6a765ae7857a2f84872f80a25983c4c4b2b3b1c1
2024-08-10 16:11:58 -07:00
jade b9ed79c99a libutil: deal with Linux systems that do not implement close_range
Seems a little bit Rich that musl does not implement close_range because
they suspect that the system call itself is a bad idea, so they uhhhh
are considering not implementing a wrapper. Let's just fix the problem
at hand by writing our own wrapper.

Change-Id: I1f8e5858e4561d58a5450503d9c4585aded2b216
2024-08-10 16:11:58 -07:00
jade b15d5cc6ee nix: remove explosions if you have a window size less than four
Turns out strings do not like being resized to -4.

This was discovered while messing with the tests to remove unbuffer and
trying stdbuf instead. Turns out that was not the right approach.

This basically rewrites the handling of this case to be much more
correct, and fixes a bug where with small window sizes where it would
ALSO truncate the attr names in addition to the optional descriptions.

Change-Id: Ifd1beeaffdb47cbb5f4a462b183fcb6c0ff6c524
2024-08-10 16:11:58 -07:00
jade 0c76195351 build: remove expect as a dependency
I was packaging Lix 2.91 for nixpkgs and was annoyed at the expect
dependency. Turns out that you can replace unbuffer with a pretty-short
Python script.

It became less short after I found out that Linux was converting \n to
\r\n in the terminal subsystem, which was not very funny, but is at
least solved by twiddling termios bits.

Change-Id: I8a2700abcbbf6a9902e01b05b40fa9340c0ab90c
2024-08-10 16:10:16 -07:00
jade 292567e0b0 fix: check if it is a Real terminal, not just if it is a terminal
This will stop printing stuff to dumb terminals that they don't support.

I've overall audited usage of isatty and replaced the ones with intent
to mean "is a Real terminal" with checking for that. I've also caught a
case of carelessly assuming "is a tty" means "should be colour" in
nix-env.

Change-Id: I6d83725d9a2d932ac94ff2294f92c0a1100d23c9
2024-08-10 16:07:21 -07:00
jade 3775b6ac88 package: remove unused autotools code, empty file
I noticed there was some stuff setting configureFlags that definitely do
not do anything with meson, so let's rip them out.

As for the empty file, it was added when I was thinking I needed a fake
C++ target to convince meson to create the necessary dependencies. That
was not in fact possible so it should have never been committed.

Change-Id: Ied4723d8a5d21aed85f352c48b080ab2c977a496
2024-08-09 23:22:11 -07:00
jade 9851be99b9 version: update, and add codename
We're going for Dragon's Breath because horrors called dibs on it.

It's fine to merge this a little before the final release, since all the
dev versions have -pre in them anyway.

Change-Id: I763acb2fc1bf76030f7feaed983addf6ae2fdd53
2024-08-09 23:22:11 -07:00
jade 7ca47a0e69 rl-next: add extra context to a few release notes
This was found while writing the release blog post.

Change-Id: Ifd55f308d4d4c831273cbe6ea35d29a38e134783
2024-08-09 23:22:11 -07:00
jade 35c9069c66 rl-next: fix incorrect CL list syntax
This also fixes the script to not pass pre-commit by failing to parse an
int if this mistake is made again.

Change-Id: I714369f515dc9987cf0c600d54a2ac745ba56830
2024-08-09 19:03:08 -07:00
eldritch horrors c7d97802e4 libutil: rename and optimize closeMostFDs
this is only used to close non-stdio files in derivation sandboxes. we
may as well encode that in its name, drop the unnecessary integer set,
and use close_range to deal with the actual closing of files. not only
is this clearer, it also makes sandbox setup on linux fast by 1ms each

Change-Id: Id90e259a49c7bc896189e76bfbbf6ef2c0bcd3b2
2024-08-09 19:59:17 +00:00
eldritch horrors 35a2f28a46 libstore: deprecate the build-hook setting
implementing a build hook is pretty much impossible without either being
a nix, or blindly forwarding the important bits of all build requests to
some kind of nix. we've found no uses of build-hook in the wild, and the
build-hook protocol (apart from being entirely undocumented) is not able
to convey any kind of versioning information between hook and daemon. if
we want to upgrade this infrastructure (which we do), this must not stay

Change-Id: I1ec4976a35adf8105b8ca9240b7984f8b91e147e
2024-08-09 19:30:45 +00:00
jade 790d1079e1 Merge changes Ib7c80826,I636f8a71,I67669b98 into main
* changes:
  perl: un-autos your conf
  build: declare all the deps as -isystem
  darwin: workaround PROC_PIDLISTFDS on processes with no fds
2024-08-09 19:24:29 +00:00
Qyriad 346e340cbf Merge "libexpr: move Value implementations out of eval.cc" into main 2024-08-09 14:25:13 +00:00
eldritch horrors 5d4686bcd5 libutil: allow marking settings as deprecated
this is a bit of a hack, but it's apparently the cleanest way of doing
this in the absence of any kind of priority/provenance information for
values of some given setting. we'll need this to deprecate build-hook.

Change-Id: I03644a9c3f17681c052ecdc610b4f1301266ab9e
2024-08-09 11:33:09 +00:00
eldritch horrors baa4fda340 main: require argv[0]
sure, linux has been providing argv[0] by default for a while now. other
OSes may not be as forthcoming though, and relying on the OS to create a
world in which we can just make assumptions we could test for instead is
unnecessarily lazy. we *could* default argv0, but that's a little silly.

notably we abort instead of returning normally to avoid confusions where
a caller interprets our exit status like a Worker build results bitmask.

Change-Id: Id73f8cd0a630293b789c59a8c4b0c4a2b936b505
2024-08-09 11:33:09 +00:00
eldritch horrors 6491cde997 resolve-system-dependencies: remove entirely
this hasn't been used since 2020, and hasn't been compiled since may.

Change-Id: I865550966630eee6ba18d742ba36f0a90901279d
2024-08-09 11:33:09 +00:00
Qyriad 0787dcf5f6 libexpr: move Value implementations out of eval.cc
Change-Id: I2ce8a9713533888b3d109a56947156eb3a5ab492
2024-08-08 22:01:12 -06:00
jade 3b902683e9 Merge changes I0373ac01,I7b543967,I537103eb into main
* changes:
  releng: fix the git push
  releng: clarify/update docs, add instructions after tag
  Fix is_maintenance_branch heuristic
2024-08-08 23:12:11 +00:00
jade 9682ab4f38 Merge changes I6358a393,I2d9f276b,Idd096dc9 into main
* changes:
  clang-tidy: write a lint for charptr_cast
  tree-wide: automated migration to charptr_cast
  clang-tidy: enforce the new rules
2024-08-08 23:09:30 +00:00
jade 757041c3e7 Merge changes I526cceed,Ia4e2f1fa,I22e66972,I9fbd55a9,Ifca22e44 into main
* changes:
  sqlite: add a Use::fromStrNullable
  util: implement charptr_cast
  tree-wide: fix a pile of lints
  refactor: make HashType and Base enum classes for type safety
  build: integrate clang-tidy into CI
2024-08-08 22:43:10 +00:00
jade a5f0954c29 clang-tidy: write a lint for charptr_cast
This lets us ensure that nobody is putting in new reinterpret_cast
instances where they could safely use charptr_cast instead.

Change-Id: I6358a3934c8133c7150042635843bdbb6b9218d4
2024-08-08 14:53:17 -07:00
jade 4ed8461cac sqlite: add a Use::fromStrNullable
There were several usages of the raw sqlite primitives along with C
style casts, seemingly because nobody thought to use an optional for
getting a string or NULL.

Let's fix this API given we already *have* a wrapper.

Change-Id: I526cceedc2e356209d8fb62e11b3572282c314e8
2024-08-08 14:53:17 -07:00
jade a85c4ce535 tree-wide: automated migration to charptr_cast
The lint did it :3

Change-Id: I2d9f276b01ebbf14101de4257ea13e44ff6fe0a0
2024-08-08 14:53:17 -07:00
jade a318c96851 util: implement charptr_cast
I don't like having so many reinterpret_cast statements that have to
actually be looked at to determine if they are UB. A huge number of the
reinterpret_cast instances in Lix are actually casting to some pointer
of some character type, which is always valid no matter the source type.

However, it is also worth looking at if it is not casting both *from* a
character type and also *to* a character type, since IMO splatting a
struct into a character array should be a very deliberate action instead
of just being about dealing with bad APIs.

So let's write a template that encapsulates this invariant so we can
not worry about the trivially safe reinterpret_cast invocations.

Change-Id: Ia4e2f1fa0c567123a96604ddadb3bdd7449660a4
2024-08-08 14:53:17 -07:00
jade c1291fd102 clang-tidy: enforce the new rules
Fixes: lix-project/lix#241

Change-Id: Idd096dc9ca92ffd4be8c22d293ba5bf2ec48a85f
2024-08-08 14:53:17 -07:00
jade e34833c025 tree-wide: fix a pile of lints
This:
- Converts a bunch of C style casts into C++ casts.
- Removes some very silly pointer subtraction code (which is no more or
  less busted on i686 than it began)
- Fixes some "technically UB" that never had to be UB in the first
  place.
- Makes finally follow the noexcept status of the inner function. Maybe
  in the future we should ban the function from not being noexcept, but
  that is not today.
- Makes various locally-used exceptions inherit from std::exception.

Change-Id: I22e66972602604989b5e494fd940b93e0e6e9297
2024-08-08 14:53:17 -07:00
jade 370ac940dd refactor: make HashType and Base enum classes for type safety
Change-Id: I9fbd55a9d50464a56fe11cb42a06a206914150d8
2024-08-08 14:53:17 -07:00
jade f3ef0899c7 build: integrate clang-tidy into CI
This still has utterly unacceptably bad output format design that I
would not inflict on anyone I like, but it *does* now exist, and you
*can* find the errors in the log.

Future work would obviously be to fix that and integrate the actual
errors into Gerrit using codechecker or so.

Followup issue: lix-project/lix#457

Fixes: lix-project/lix#147
Change-Id: Ifca22e443d357762125f4ad6bc4f568af3a26c62
2024-08-08 14:53:17 -07:00
piegames e03cd8b3a6 Merge "libexpr: Add experimental pipe operator" into main 2024-08-08 18:15:21 +00:00
eldritch horrors a957219df2 libstore: make Worker::waitForInput private
Change-Id: I71a42acd5a4a9a18b55cf754cdf9896614134398
2024-08-08 12:02:17 +00:00
eldritch horrors ba85e501ce libstore: make Worker status flags private
Change-Id: I16ec8994c6448d70b686a2e4c10f19d4e240750d
2024-08-08 12:02:17 +00:00
eldritch horrors fc987b4123 libstore: remove Goal::addWaitee
Change-Id: I1b00d1a537d84790878cb0e81aaa1cbaa143d62d
2024-08-08 12:02:17 +00:00
eldritch horrors 4c3010a1be libstore: make Worker::wakeUp private
Change-Id: Iffa55272fe6ef4adaf3e9d4d25e5339792c2e460
2024-08-08 12:02:17 +00:00
eldritch horrors 3ecb46e3e7 libstore: make Worker::waitForAWhile private
Change-Id: I0cdcd436ee71124ca992b4f4fe307624a25f11e9
2024-08-08 12:02:17 +00:00
eldritch horrors b33c969519 libstore: make Worker::waitForBuildSlot private
Change-Id: I02a54846cd65622edbd7a1d6c24a623b4a59e5b3
2024-08-08 12:02:17 +00:00
piegames 28ae24f3f7 libexpr: Add experimental pipe operator
The |> operator is a reverse function operator with low binding strength
to replace lib.pipe. Implements RFC 148, see the RFC text for more
details. Closes #438.

Change-Id: I21df66e8014e0d4dd9753dd038560a2b0b7fd805
2024-08-08 11:13:53 +02:00
jade 7246c2d104 releng: fix the git push
This was broken because gerrit requires that the revision actually
is known before it is pushed as a tag.

Also, arguably this fixes the original problem mentioned in
lix-project/lix#439

Change-Id: I0373ac01584440f18d32b8da5699bb359cc2c89a
2024-08-07 21:46:44 -07:00
jade 83247b1c38 releng: clarify/update docs, add instructions after tag
This is not a proper fix for the confusion that can happen about how the
tags are supposed to be used.

For a proper fix, we need to do
lix-project/lix#439 and implement
worktrees such that the user never sees the git state anymore.

Change-Id: I7b543967f522cede486e42684b48cad47da95429
2024-08-07 20:52:09 -07:00
jade 8a86f38bca Fix is_maintenance_branch heuristic
This was broken because Nix language's version comparison does not know
how to deal with versions like -rc1 and considers them newer, which is
in this case not desirable.

That in turn led to not tagging 2.90.0 docker images as "latest" since
the heuristic was wrong.

This commit also adds some more cross-checking and failsafes in case the
person running releng does not have a local main branch that is up to
date.

Fixes: lix-project/lix#443
Change-Id: I537103ebab58ae978c00e06972abe14432dd9c80
2024-08-07 20:14:45 -07:00
Max “Goldstein” Siling 6fdb47f0b2 Merge "src/libcmd/repl.cc: allow :log /path/to/store.drv" into main 2024-08-07 21:48:01 +00:00
jade 0800a81a95 Merge "oops: fix warning about catching polymorphic exception" into main 2024-08-07 19:06:54 +00:00
piegames ec7552ff74 libexpr/parser: Test experimental features
Currently, the parser relies on the global experimental feature flags.
In order to properly test conditional language features, we instead need
to pass it around in the parser::State.

This means that the parser cannot cache the result of isEnabled anymore,
which wouldn't necessarily hurt performance if the function didn't
perform a linear search on the list of enabled features on every single
call. While we could simply evaluate once at the start of parsing and
cache the result in the parser state, the more sustainable solution
would be to fix `isEnabled` such that all callers may profit from the
performance improvement.

Change-Id: Ic9b9c5d882b6270e1114988b63e6064d36c25cf2
2024-08-07 13:07:50 +00:00
Max “Goldstein” Siling 9adfd9b8ad
src/libcmd/repl.cc: allow :log /path/to/store.drv
This adds a second form to the `:log` command: it now can accept a
derivation path in addition to a derivation expression. As derivation
store paths start with `/nix/store`, this is not ambiguous.

Resolves: lix-project/lix#51
Change-Id: Iebc7b011537e7012fae8faed4024ea1b8fdc81c3
2024-08-07 15:58:44 +03:00
Maximilian Bosch 27a63db710 Merge "fix: warn and document when advanced attributes will have no impact due to __structuredAttrs" into main 2024-08-07 10:38:39 +00:00
jade d1fd1dc8ac perl: un-autos your conf
I definitely don't think we were using this, and it is probably an
omission in the original autoconf deletion more than anything.

Change-Id: Ib7c8082685e550575bca5af06f0e93adf982bd7c
2024-08-07 02:52:00 -07:00
jade f8fb335eb7 build: declare all the deps as -isystem
I don't know why but I was getting a spurious -Werror=switch-enum inside
toml11. It does not make sense why it did not occur before, but it
should be stopped.

This was not done at an earlier stage to better match the legacy make
build system, but we don't use it anyway.

Change-Id: I636f8a71e8a0ba5e0feb80b435ae24c3af995c5d
2024-08-07 02:52:00 -07:00
jade 1437d3df15 darwin: workaround PROC_PIDLISTFDS on processes with no fds
This has been causing various seemingly spurious CI failures as well as
some failures on people running tests on beta builds.

lix> ++(nix-collect-garbage-dry-run.sh:20) nix-store --gc --print-dead
lix> ++(nix-collect-garbage-dry-run.sh:20) wc -l
lix> finding garbage collector roots...
lix> error: Listing pid 87261 file descriptors: Undefined error: 0

There is no real way to write a proper test for this, other than to
start a process like the following:

int main(void) {
    for (int i = 0; i < 1000; ++i) {
        close(i);
    }
    sleep(10000);
}

and then let Lix's gc look at it.

I have a relatively high confidence this *will* fix the problem since I
have manually confirmed the behaviour of the libproc call is
as-unexpected, and it would perfectly explain the observed symptom.

Fixes: lix-project/lix#446
Change-Id: I67669b98377af17895644b3bafdf42fc33abd076
2024-08-07 02:52:00 -07:00
alois31 780998f4ea Merge "package: improve support for building without BDW-GC" into main 2024-08-07 07:07:28 +00:00
jade d280e4990c oops: fix warning about catching polymorphic exception
This was introduced in I0fc80718eb7e02d84cc4b5d5deec4c0f41116134 and
unnoticed since it only appears in gcc builds.

Change-Id: I1de80ce2a8fab63efdca7ca0de2a302ceb118267
2024-08-06 22:45:19 -07:00
jade 529eed74c4 Merge changes I0fc80718,Ia182b86f,I355f82cb,I8a9b58fa,Id89f8a1f, ... into main
* changes:
  tree-wide: fix various lint warnings
  flake & doxygen: update tagline
  nix flake metadata: print modified dates for input flakes
  cli: eat terminal codes from stdout also
  Implement forcing CLI colour on, and document it better
  manual: fix a syntax error in redirects.js that made it not do anything
  misc docs/meson tidying
  build: implement clang-tidy using our plugin
2024-08-07 00:50:30 +00:00
alois31 2c48460850
libstore/linux: precompile and cache the seccomp BPF
The growth of the seccomp filter in 127ee1a101
made its compilation time significant (roughly 10 milliseconds have been
measured on one machine). For this reason, it is now precompiled and cached in
the parent process so that this overhead is not hit for every single build. It
is still not optimal when going through the daemon, because compilation still
happens once per client, but it's better than before and doing it only once for
the entire daemon requires excessive crimes with the current architecture.

Fixes: lix-project/lix#461
Change-Id: I2277eaaf6bab9bd74bbbfd9861e52392a54b61a3
2024-08-06 19:10:33 +02:00
alois31 403fa9e2b6
libstore/linux: compile the seccomp BPF explicitly
This is a preparation for precompiling the filter, which is done separately.
The behaviour should be unchanged for now.

Change-Id: I899aa7242962615949208597aca88913feba1cb8
2024-08-06 18:31:40 +02:00
alois31 741d3b441c
libstore: add LocalDerivationGoal setupSyscallFilter hook
The seccomp setup code was a huge chunk of conditionally compiled
platform-specific code. For this reason, it is appropriate to move it to the
platform-specific implementation file. Ideally its setup could be moved a bit
to make it happen at the same place as the Darwin restrictions, but that change
is going to be less mechanical.

Change-Id: I496aa3c4fabf34656aba1e32b0089044ab5b99f8
2024-08-06 18:27:09 +02:00
alois31 f84997cbef
package: don't hide system-wide manual pages
When MANPATH is unset or contains an empty component, a reasonable default is
used. Previously (after 3dced96741), when MANPATH
was unset, the shell hook would only place a location containing the Lix manual
pages there, and system-wide manual pages would become unavailable in the
development shell, which is undesired. Fix the issue by including an empty
component in this case.

Change-Id: Ib3c67a831d709fe2a87520e15917eebb59397bd1
2024-08-06 17:18:05 +02:00
jade ca9d3e6e00 tree-wide: fix various lint warnings
Change-Id: I0fc80718eb7e02d84cc4b5d5deec4c0f41116134
2024-08-04 20:55:45 -07:00
jade 9238e62ae6 flake & doxygen: update tagline
This tagline was left over from CppNix and we should make it tastier.

Change-Id: Ia182b86f6e751591be71a50521992ad73c7b38b5
2024-08-04 20:41:19 -07:00
jade bd1344ec54 nix flake metadata: print modified dates for input flakes
This was always in the lock file and we can simply actually print it.

The test for this is a little bit silly but it should correctly
control for my daring to exercise timezone code *and* locale code in a
test, which I strongly suspect nobody dared do before.

Sample (abridged):
```
Path:          /nix/store/gaxb42z68bcr8lch467shvmnhjjzgd8b-source
Last modified: 1970-01-01 00:16:40
Inputs:
├───flake-compat: github:edolstra/flake-compat/0f9255e01c2351cc7d116c072cb317785dd33b33
│   Last modified: 2023-10-04 13:37:54
├───flake-utils: github:numtide/flake-utils/b1d9ab70662946ef0850d488da1c9019f3a9752a
│   Last modified: 2024-03-11 08:33:50
│   └───systems: github:nix-systems/default/da67096a3b9bf56a91d16901293e51ba5b49a27e
│       Last modified: 2023-04-09 08:27:08
```

Change-Id: I355f82cb4b633974295375ebad646fb6e2107f9b
2024-08-04 20:41:19 -07:00
jade 5f0ef50077 cli: eat terminal codes from stdout also
This *should* be sound, plus or minus the amount that the terminal code
eating code is messed up already.

This is useful for testing CLI output because it will strip the escapes
enough to just shove the expected output in a file.

Change-Id: I8a9b58fafb918466ac76e9ab585fc32fb9294819
2024-08-04 20:41:19 -07:00
jade 378ec5fb06 Implement forcing CLI colour on, and document it better
This is necessary to make some old tests work when testing colour
against non-interactive outputs.

Change-Id: Id89f8a1f45c587fede35a69db85f7a52f2c0a981
2024-08-04 20:41:19 -07:00
jade 700762d8b2 manual: fix a syntax error in redirects.js that made it not do anything
lol lmao

Let's put in a syntax checker in CI so we do not have to deal with this
nonsense ever again.

Change-Id: I0fe875e0cfc59ab1783087762e5bb07e09ded105
2024-08-04 20:41:19 -07:00
jade 0f998056fa misc docs/meson tidying
The docs page has an incorrect escape that leads to a backslash
appearing in output. Meson stuff is self-explanatory, just shortens and
simplifies a bit.

Change-Id: Ib63adf934efd3caeb82ca82988f230e8858a79f9
2024-08-04 20:41:19 -07:00
jade 3daeeaefb1 build: implement clang-tidy using our plugin
The principle of this is that you can either externally build it with
Nix (actual implementation will be in a future commit), or it can be
built with meson if the Nix one is not passed in.

The idea I have is that dev shells don't receive the one from Nix to
avoid having to build it, but CI can use the one from Nix and save some
gratuitous rebuilds.

The design of this is that you can run `ninja -C build clang-tidy` and
it will simply correctly clang-tidy the codebase in spite of PCH
bullshit caused by the cc-wrapper.

This is a truly horrendous number of hacks in a ball, caused by bugs in
several pieces of software, and I am not even getting started.

I don't consider this to fix the clang-tidy issue filing, since we still
have a fair number of issues to fix even on the existing minimal
configuration, and I have not yet implemented it in CI. Realistically we
will need to do something like https://github.com/Ericsson/codechecker
to be able to silence warnings without physically touching the code, or
at least *diff* reports between versions.

Also, the run-clang-tidy output design is rather atrocious and must
not be inflicted upon anyone I have respect for, since it buries the
diagnostics in a pile of invocation logs. We would do really well to
integrate with the Gerrit SARIF stuff so we can dump the reports on
people in a user-friendly manner.

Related: lix-project/lix#147

Change-Id: Ifefe533f3b56874795de231667046b2da6ff2461
2024-08-04 20:41:19 -07:00
Tom Bereknyei 7fc481396c
fix: warn and document when advanced attributes will have no impact due to __structuredAttrs
Backport of https://github.com/NixOS/nix/pull/10884.

Change-Id: I82cc2794730ae9f4a9b7df0185ed0aea83efb65a
2024-08-03 13:32:51 +02:00
alois31 58758c0f87
package: improve support for building without BDW-GC
Expose an option for disabling the BDW-GC build dependency entirely. Fix the
place where one of its headers was included (unnecessarily) without proper
guarding. Finally, use this machinery to exclude BDW-GC from the ASAN builds
entirely (its usage has already been disabled due to compatibility issues
anyway), to ensure this configuration is not regressed again.

Change-Id: I2ebe8094abf67e7d1e99eed971de3e99d071c10b
2024-08-03 06:14:41 +02:00
eldritch horrors 66469fc281 libstore: move Goal::waiteeDone into Worker::goalFinished
this begins a long and arduous journey to remove all result state from
Goal, to eventually drop the std::enable_shared_from_this base, and to
completely eliminate all unsynchronized modification of states of both
Goal and Worker. by the end of this we will hopefully be able to start
and reap multiple derivation builds in parallel, which should speed up
the process quite a bit (at least for short local builds, others might
not notice a large difference. the build hooks will remain a problem.)

Change-Id: I57dcd9b2cab4636ed4aa24cdec67124fef883345
2024-08-03 00:08:44 +00:00
alois31 32ca194ebf Merge "libstore/ssh: only resume the logger when we paused it" into main 2024-08-02 16:59:44 +00:00
alois31 a93dade821
libstore/ssh: only resume the logger when we paused it
In the SSH code, the logger was conditionally paused, but unconditionally
resumed. This was fine as long as resuming the logger was idempotent. Starting
with 0dd1d8ca1c, it isn't any more, and the
behaviour of the code in question was missed. Consequently, an assertion
failure is triggered for example when performing builds against an "SSH" store
on localhost. Fix the issue by only resuming the logger when it has actually
been paused.

Fixes: lix-project/lix#458
Change-Id: Ib1e4d047744a129f15730b7216f9c9368c2f4211
2024-08-02 18:38:14 +02:00
eldritch horrors e5177dddff libstore: move Goal::amDone to Worker
we still mutate goal state to store the results of any given goal run,
but now we also have that information in Worker and could in theory do
something else with it. we could return a map of goal to goal results,
which would also let us better diagnose failures of subgoals (at all).

Change-Id: I1df956bbd9fa8cc9485fb6df32918d68dda3ff48
2024-08-02 13:52:15 +00:00
eldritch horrors dfcab1c3f0 libstore: return finishedness from Goal methods
this is the first step towards removing all result-related mutation of
Goal state from goal implementations themselves, and into Worker state
instead. once that is done we can treat all non-const Goal fields like
private state of the goal itself, and make threading of goals possible

Change-Id: I69ff7d02a6fd91a65887c6640bfc4f5fb785b45c
2024-08-02 13:52:15 +00:00
eldritch horrors 724b345eb9 libstore: encapsulate worker build hook state
once goals run on multiple threads these fields must by synchronized as
one, or we try to run build hooks to often (or worse, not often enough)

Change-Id: I47860e46fe5c6db41755b2a3a1d9dbb5701c4ca4
2024-08-02 13:52:15 +00:00
eldritch horrors 868eb5ecde libutil: make RunningProgram::wait more resilient
this will usually be used either directly (which is always fine) or in
Finally blocks (where it must never throw execptions). make sure that,
exceptions being handled or not, the calling wait() in Finally doesn't
cause crashes due to the Finally no-nested-exceptions-thrown assertion

Change-Id: Ib83a5d9483b1fe83b9a957dcefeefce5d088f06d
2024-08-02 13:12:44 +00:00
Jeremy List c907d805bf Merge "package: make aws-sdk-cpp build input optional" into main 2024-08-02 11:42:13 +00:00
Isabel 9eb374dc6d Merge "nix flake show: add the description if it exists" into main 2024-08-02 07:56:06 +00:00
Maximilian Bosch 3bb8c627ae Merge "Reapply "libfetchers: make attribute / URL query handling consistent"" into main 2024-08-02 04:50:25 +00:00
Maximilian Bosch 87fd1f024c Reapply "libfetchers: make attribute / URL query handling consistent"
The original attempt at this introduced a regression; this commit
reverts the revert and fixes the regression.

This reverts commit 3e151d4d77.

Fix to the regression:

flakeref: fix handling of `?dir=` param for flakes in subdirs

As reported in #419[1], accessing a flake in a subdir of a Git
repository fails with the previous commit[2] applied with the error

    error: unsupported Git input attribute 'dir'

The problem is that the `dir`-param is inserted into the parsed URL if a
flake is fetched from the subdir of a Git repository. However, for the
fetching part this isn't even needed. The fix is to just pass `subdir`
as second argument to `FlakeRef` (which needs a `basedir` that can be
empty) and leave the parsedURL as-is.

Added a regression test to make sure we don't run into this again.

[1] lix-project/lix#419
[2] e22172aaf6b6a366cecd3c025590e68fa2b91bcc,
    originally 3e151d4d77

Change-Id: I2c72d5a32e406a7ca308e271730bd0af01c5d18b
2024-08-01 15:41:30 -07:00
jade 8b69d13368 Merge "flake: remove control character from file by using fromJSON" into main 2024-08-01 22:25:36 +00:00
Jeremy List f41190552f
package: make aws-sdk-cpp build input optional
I have added an option to turn off this build input because I'm much
more comfortable when I don't have that type of thing on my computer.
Its default value is true in order to avoid impacting anyone who depends
on AWS features.

Change-Id: Ic57f3c9b9468f422e9fbdcf3ba0fe96177631067
2024-08-02 09:14:48 +12:00
Qyriad 61a93d5308 Merge changes Icc4747ae,Id4717b5b,Ie3ddb3d0,Ic4d93a08,I00d9ff70 into main
* changes:
  remove unused headers in installable-attr-path
  libexpr: include the type of the non-derivation value in the type error
  libexpr: mild cleanup to getDerivations
  libexpr: DrvInfo: remove unused bad-citizen constructor
  cleanup and slightly refactor DrvInfo::queryOutputs
2024-08-01 16:25:43 +00:00
jade e6fc3e9227 flake: remove control character from file by using fromJSON
I was reminded by various evil things puck did to the evaluator
involving null bytes that you can get funny bytes by abusing JSON
parsing. It's neater than putting binary in the source file, so let's do
it.

Change-Id: I1ff2e0d829eb303fbed81fa2ebb3a39412e89ff1
2024-07-31 23:23:42 -07:00
jade a3ab2cc78a Merge changes from topic "undefined-behaviour" into main
* changes:
  releng: move officialRelease to version.json
  Add -Werror CI job
  ci: add a asan+ubsan test run on x86_64-linux
  tree-wide: add support for asan!
2024-08-01 04:01:34 +00:00
Qyriad 17d7e88707 remove unused headers in installable-attr-path
Change-Id: Icc4747aed195e3855b128c73df82e202405af6a8
2024-08-01 00:37:13 +00:00
Qyriad 4f6a3d7e9e libexpr: include the type of the non-derivation value in the type error
Change-Id: Id4717b5b0df7c09b0dbf17e642d8713a0a3efbae
2024-08-01 00:37:03 +00:00
Qyriad 5ffed6d06a libexpr: mild cleanup to getDerivations
Shuffled the logic around a bit so the shorter code paths are early
returns, added comments, etc.

Should be NFC.

Change-Id: Ie3ddb3d0eddd614d6f8c37bf9a4d5a50282084ea
2024-08-01 00:36:55 +00:00
Qyriad 6a30ea0cc4 libexpr: DrvInfo: remove unused bad-citizen constructor
DrvInfo's constructor that only takes `EvalState` leaves everything else
empty; a DrvInfo which has no iota of information about the derivation
it represents is not useful, and was not used anywhere.

Change-Id: Ic4d93a08cb2748b8cef9a61e41e70404834b23f9
2024-08-01 00:36:41 +00:00
Qyriad eb18dcb0ea cleanup and slightly refactor DrvInfo::queryOutputs
Change-Id: I00d9ff707fe61995737b86af6d2eaa1e4d8116ff
2024-08-01 00:36:27 +00:00
jade 5eecdd3ae9 releng: move officialRelease to version.json
This was causing a few bits of suffering downstream, in particular, in
the NixOS module, which, after this change, can have the
`officialRelease` stuff in *it* completely deleted since we now have
correct defaulting in package.nix for it.

It also eliminates some automated editing of Nix files, which is
certainly always welcome to eliminate.

Fixes: lix-project/lix#406
Change-Id: Id12f3018cff4633e379dbfcbe26b7bc84922bdaf
2024-07-31 14:13:39 -07:00
jade b5c6ce7a53 Add -Werror CI job
We should cause CLs that introduce compiler warnings to fail CI. Sadly
this will only cover Clang, but it will cover Clang for free, so it's
truly impossible to say if it's bad or not.

Change-Id: I45ca20d77251af9671d5cbe0d29cb08c5f1d03c2
2024-07-31 14:13:39 -07:00
jade e51263057f ci: add a asan+ubsan test run on x86_64-linux
This should at least catch out blatantly bad patches that don't pass the
test suite with ASan. We don't do this to the integration tests since
they run on relatively limited-memory VMs and so it may not be super
safe to run an evaluator with leak driven garbage collection for them.

Fixes: lix-project/lix#403
Fixes: lix-project/lix#319
Change-Id: I5267b02626866fd33e8b4d8794344531af679f78
2024-07-31 14:13:39 -07:00
jade 19ae87e5ce tree-wide: add support for asan!
What if you could find memory bugs in Lix without really trying very
hard? I've had variously scuffed patches to do this, but this is
blocked on boost coroutines removal at this point tbh.

Change-Id: Id762af076aa06ad51e77a6c17ed10275929ed578
2024-07-31 14:13:39 -07:00
Qyriad ddfca6e81b libexpr: implement actual constructors for nix::Value
Change-Id: Iebc2bb4e4ea5e93045afe47677df756de4ec4d05
2024-07-31 15:38:37 +02:00
V. c347d3df8f Merge changes I609a5898,I7afb53c9 into main
* changes:
  devendor pegtl
  update flake.lock
2024-07-30 20:32:52 +00:00
eldritch horrors 97a389b0be libstore: move Goal::getBuildResult to BuildResult
there are no other uses for this yet, but asking for just a subset of
outputs does seem at least somewhat useful to have as a generic thing

Change-Id: I30ff5055a666c351b1b086b8d05b9d7c9fb1c77a
2024-07-30 16:37:13 +00:00
Max “Goldstein” Siling 53bfcf2586 Merge "src/libcmd/repl.cc: avoid unneeded reload after :e" into main 2024-07-30 16:32:56 +00:00
Max “Goldstein” Siling 1a6d7a3af4 src/libcmd/repl.cc: avoid unneeded reload after :e
If `:edit`ing a store path, don't reload repl afterwards
to avoid losing local variables: store is immutable,
so "editing" a store path is always just viewing it.

Resolves: lix-project/lix#341
Change-Id: I3747f75ce26e0595e953069c39ddc3ee80699718
2024-07-30 16:08:26 +00:00
eldritch horrors d265dd5993 libstore: count all substitutions toward the same limit
limiting CA substitutions was a rather recent addition, and it used a
dedicated counter to not interfere with regular substitutions. though
this works fine it somewhat contradicts the documentation; job limits
should apply to all kinds of substitutions, or be one limit for each.

Change-Id: I1505105b14260ecc1784039b2cc4b7afcf9115c8
2024-07-30 15:37:27 +00:00
eldritch horrors d9af753a7f libstore: always wake up goals on EOF
all goals do this. it makes no sense to not notify a goal of EOF
conditions because this is the universal signal for "child done"

Change-Id: Ic3980de312547e616739c57c6248a8e81308b5ee
2024-07-30 15:37:27 +00:00
eldritch horrors 6c0dcd1220 libstore: simplify substitution handleEOF
both substitution goals add only this single fd to their wait set.

Change-Id: Ibf921f5bb3919106208a0871523b32c8f67fb3d3
2024-07-30 15:37:27 +00:00
eldritch horrors 548c973e82 libstore: remove Worker::updateProgress
just update progress every time a goal has returned from work(). there
seem to be no performance penalties, and the code is much simpler now.

Change-Id: I288ee568b764ee61f40a498d986afda49987cb50
2024-07-29 22:16:11 +00:00
Artemis Tosini 3058029fba
libutil: Add bindPath function from libstore
bindPath/doBind is a useful function in build that is used in several
parts of LocalDerivationGoal. Moving this function makes it easier to
split LocalDerivationGoal implementation between several files.

Change-Id: Ic5a0768479c153c1aa3ed425f12604b20bbf0f42
2024-07-27 19:40:40 +00:00
jade 6abad7cb23 Merge "releng: Remove workaround for skopeo feature we didn't know about" into main 2024-07-26 11:09:17 +00:00
V. a98dce2a1f devendor pegtl
Change-Id: I609a58985fc5210806d0959049a48976ae079c30
2024-07-26 11:22:34 +04:00
V. 393794ad92 update flake.lock
Change-Id: I7afb53c929d297061dba6ec4a3ec7c6e3c6a553e
2024-07-26 11:22:34 +04:00
alois31 d945e89e19 Merge changes I45d3895f,I541be3ea,Ibe51416d into main
* changes:
  libstore/build: block io_uring
  libstore/build: use an allowlist approach to syscall filtering
  libstore/build: always treat seccomp setup failures as fatal
2024-07-26 07:08:35 +00:00
Artemis Tosini 60a48311e8 Merge "libutil: Support getSelfExe on FreeBSD" into main 2024-07-25 23:10:30 +00:00
jade c4c7cb7613 Merge changes Ic0dfcfe2,Ibe73851f,Ia7a8df1c,I400b2031 into main
* changes:
  package.nix: remove dead code
  diff-closures: remove gratuitous copy
  tree-wide: NULL -> nullptr
  libutil: rip out GNU Hurd support code
2024-07-25 18:05:41 +00:00
alois31 e7188e211a
libstore/build: block io_uring
Unfortunately, io_uring is totally opaque to seccomp, and while currently there
are no dangerous operations implemented, there is no guarantee that it remains
this way. This means that io_uring should be blocked entirely to ensure that
the sandbox is future-proof. This has not been observed to cause issues in
practice.

Change-Id: I45d3895f95abe1bc103a63969f444c334dbbf50d
2024-07-25 18:24:45 +02:00
alois31 127ee1a101
libstore/build: use an allowlist approach to syscall filtering
Previously, system call filtering (to prevent builders from storing files with
setuid/setgid permission bits or extended attributes) was performed using a
blocklist. While this looks simple at first, it actually carries significant
security and maintainability risks: after all, the kernel may add new syscalls
to achieve the same functionality one is trying to block, and it can even be
hard to actually add the syscall to the blocklist when building against a C
library that doesn't know about it yet. For a recent demonstration of this
happening in practice to Nix, see the introduction of fchmodat2 [0] [1].

The allowlist approach does not share the same drawback. While it does require
a rather large list of harmless syscalls to be maintained in the codebase,
failing to update this list (and roll out the update to all users) in time has
rather benign effects; at worst, very recent programs that already rely on new
syscalls will fail with an error the same way they would on a slightly older
kernel that doesn't support them yet. Most importantly, no unintended new ways
of performing dangerous operations will be silently allowed.

Another possible drawback is reduced system call performance due to the larger
filter created by the allowlist requiring more computation [2]. However, this
issue has not convincingly been demonstrated yet in practice, for example in
systemd or various browsers. To the contrary, it has been measured that the the
actual filter constructed here has approximately the same overhead as a very
simple filter blocking only one system call.

This commit tries to keep the behavior as close to unchanged as possible. The
system call list is in line with libseccomp 2.5.5 and glibc 2.39, which are the
latest versions at the point of writing. Since libseccomp 2.5.5 is already a
requirement and the distributions shipping this together with older versions of
glibc are mostly not a thing any more, this should not lead to more build
failures any more.

[0] https://github.com/NixOS/nixpkgs/issues/300635
[1] https://github.com/NixOS/nix/issues/10424
[2] https://github.com/flatpak/flatpak/pull/4462#issuecomment-1061690607

Change-Id: I541be3ea9b249bcceddfed6a5a13ac10b11e16ad
2024-07-25 18:24:40 +02:00
alois31 233408f677
libstore/build: always treat seccomp setup failures as fatal
In f047e4357b, I missed the behavior that if
building without a dedicated build user (i.e. in single-user setups), seccomp
setup failures are silently ignored. This was introduced without explanation 7
years ago (ff6becafa8). Hopefully the only
use-case nowadays is causing spurious test suite successes when messing up the
seccomp filter during development. Let's try removing it.

Change-Id: Ibe51416d9c7a6dd635c2282990224861adf1ceab
2024-07-25 18:21:26 +02:00
Qyriad 8d12e0fbb7 fix building with Musl, fixing static builds
Musl stdout macro expands¹ to something that isn't a valid identifier,
so we get syntax errors when compiling usage of a method called stdout
with Musl's stdio.h.

[1]: https://git.musl-libc.org/cgit/musl/tree/include/stdio.h?id=ab31e9d6a0fa7c5c408856c89df2dfb12c344039#n67

Change-Id: I10e6f6a49504399bf8edd59c5d9e4e62449469e8
2024-07-24 17:21:40 +00:00
Artemis Tosini 3b96b51cf4
libutil: Support getSelfExe on FreeBSD
getSelfExe is used in a few places re-execute nix.
Current code in this file uses ifdefs to support several
platforms, just keep doing that

Change-Id: Iecc2ada0101aea0c30524e3a1218594f919d74bf
2024-07-24 01:28:03 +00:00
jade 98e8cf9c63 package.nix: remove dead code
Change-Id: Ic0dfcfe27dbf13da4f7f74f5fab8ce6fa718d28f
2024-07-23 21:53:43 +02:00
jade 12a5838d11 diff-closures: remove gratuitous copy
This was done originally because std::smatch does not accept `const char
*` as iterators. However, this was because we should have been using
std::cmatch instead.

Change-Id: Ibe73851fd39755e883df2d33d22fed72ac0a04ae
2024-07-23 21:45:30 +02:00
jade eecc4ff1c0 releng: Remove workaround for skopeo feature we didn't know about
It turns out skopeo *does* support not saying the tag, but I couldn't
find it in the docs.

Asking the author in https://github.com/containers/skopeo/issues/2354
yielded that this can be requested as `@@unknown-digest@@`.

So now we have a perfectly cromulent docker upload chain, yay!!

Change-Id: I256f3cbeef4fe28b3d68d0dda57f02cdaee3996b
2024-07-23 12:11:15 -07:00
jade 2436f2110a tree-wide: NULL -> nullptr
This is slightly more type safe and is more in line with modern C++.

Change-Id: Ia7a8df1c7788085020d1bdc941d6f9cee356144e
2024-07-23 21:06:55 +02:00
jade 916b5c68fb libutil: rip out GNU Hurd support code
Nobody has stepped up to add further support for Hurd since this code
appeared in 2010 or 2014. We don't need it.

Change-Id: I400b2031a225551ea3c71a3ef3ea9fdb599dfba3
2024-07-23 20:52:04 +02:00
Artemis Tosini 53f3e39815
libstore: Add FreeBSD findPlatformRoots
Use libprocstat to find garbage collector roots on FreeBSD.
Tested working on a FreeBSD machine, although there is no CI yet

Change-Id: Id36bac8c3de6cc4de94e2d76e9663dd4b76068a9
2024-07-23 17:49:33 +00:00
Isabel d2422771eb
nix flake show: add the description if it exists
(cherry picked from commit 8cd1d02f90eb9915e640c5d370d919fad9833c65)

nix flake show: Only print up to the first new line if it exists.

(cherry picked from commit 5281a44927bdb51bfe6e5de12262d815c98f6fe7)

add tests

(cherry picked from commit 74ae0fbdc70a5079a527fe143c4832d1357011f7)

Handle long strings, embedded new lines and empty descriptions

(cherry picked from commit 2ca7b3afdbbd983173a17fa0a822cf7623601367)

Account for total length of 80

(cherry picked from commit 1cc808c18cbaaf26aaae42bb1d7f7223f25dd364)

docs: add nix flake show description release note

fix: remove white space

nix flake show: trim length based on terminal size

test: account for terminal size

docs(flake-description): before and after commands; add myself to credits

Upstream-PR: https://github.com/NixOS/nix/pull/10980
Change-Id: Ie1c667dc816b3dd81e65a1f5395e57ea48ee0362
2024-07-23 13:21:15 +01:00
Pierre Bourdon 73c013a5df Merge "libexpr/gc-alloc: fix compilation with !HAVE_BOEHMGC" into main 2024-07-22 23:14:59 +00:00
Pierre Bourdon e76245f8e9
libexpr/gc-alloc: fix compilation with !HAVE_BOEHMGC
Fixes: 72ee25b402
Change-Id: Ib59386af1415a8ed4b53af24ec22a4ffa5e5877d
2024-07-23 00:50:09 +02:00
eldritch horrors 472ff1b833 libstore: keep Goal errors as unique_ptrs
Error is pretty large, and most goals do not fail. this alone more than
halves the size of Goal on x86_64-linux, from 720 bytes down to 344. in
derived classes the difference is not as dramatic, but even the largest
derived class (`LocalDerivationGoal`) loses almost 20% of its footprint

Change-Id: Ifda8f94c81b6566eeb3e52d55d9796ec40c7bce8
2024-07-22 19:01:40 +00:00
eldritch horrors 7bf1aff44a libstore: remove an always-defaulted argument
Change-Id: I3c7f17d5492a16bb54480fa1aa384b96fba72d61
2024-07-22 19:01:40 +00:00
eldritch horrors 58a91d70c9 libstore: use std::async instead of Goal threads
the goals are either already using std::async and merely forgot to
remove std::thread vestiges or they emulate async with threads and
promises. we can simply use async directly everywhere for clarity.

Change-Id: I3f05098310a25984f10fff1e68c573329002b500
2024-07-22 19:01:40 +00:00
eldritch horrors ad36fb43ad libstore: remove addToWeakGoals
under owner_less it's equivalent to insert(), only sometimes a little
bit faster because it does not construct a weak_ptr if the goal is in
the set already. this small difference in performance does not matter
here and c++23 will make insert transparent anyway, so we can drop it

Change-Id: I7cbd7d6e0daa95d67145ec58183162f6c4743b15
2024-07-22 19:01:40 +00:00
eldritch horrors d70e045f90 libstore: remove Goal::ecBusy
this should be an optional. "busy" is not an *exit* code!

Change-Id: Ic231cb27b022312b1a7a7b9602f32845b7a9c934
2024-07-22 19:01:40 +00:00
eldritch horrors 20f53346df libstore: remove unused Worker::waitForAnyGoal
Change-Id: Ia3ebd434b17052b6760ce74d8e20025a72148613
2024-07-22 19:01:40 +00:00
V. 85e3b9b871 De-vendor nixfmt
Change-Id: I1a051be495318a507d07f6d0a6b157616e26774c
2024-07-22 21:09:58 +04:00
eldritch horrors c74eb81356 enable -Werror=suggest-override
*accidentally* overriding a function is almost guaranteed to be an
error. overriding a function without labeling it as such is merely
bad style, but bad style that makes the code harder to understand.

Change-Id: Ic0594f3d1604ab6b3c1a75cb5facc246effe45f0
2024-07-22 16:26:55 +00:00
eldritch horrors 0463cf2aef libexpr: fix -Wunused-const-variable warning
Change-Id: Ib986ece0ab2eff83e7abd7f1f915cd8f761827ad
2024-07-22 16:26:55 +00:00
alois31 2d4aca2546
libutil/logging: fix build without precompiled header
Commit 0109368c3f missed to include a required
header, which is not noticed when the precompiled header is enabled because
it's included in that. Also include it in the file so that the build without
precompiled header works too.

Change-Id: Id7a7979684b64f937f7f8191612952d73c113015
2024-07-21 12:54:31 +02:00
alois31 94a8e5fe0d Merge "libstore/binary-cache-store: use correct buffer size for NAR decompression" into main 2024-07-21 10:42:33 +00:00
jade 4fa6961aa2 Merge "gc: refactor the gc server thread out into a class without changing it" into main 2024-07-21 10:36:10 +00:00
alois31 391088900e
libstore/binary-cache-store: use correct buffer size for NAR decompression
Due to a leftover from a previous version where the buffer was allocated on the
stack, the change introduced in commit 4ec87742a1
accidentally passes the size of a pointer as the size of the buffer to the
decompressor. Since the former is much smaller (usually 8 bytes instead of 64
kilobytes), this is safe, but leads to considerable overhead; most notably, due
to excessive progress reports, which happen for each chunk. Pass the proper
buffer size instead.

Change-Id: If4bf472d33e21587acb5235a2d99e3cb10914633
2024-07-21 11:28:23 +02:00
Winter Cute 1917e6c765 Merge "Fix namespace warning being emitted if sandbox is disabled" into main 2024-07-20 22:14:33 +00:00
Qyriad 72ee25b402 libexpr: add a strongly typed version of gcAllocBytes()
This commit adds a new helper template function to gc-alloc.hh (which is
probably where you want to look at first, O great reviewer [custom file
ordering in review diffs when]), which uses a type argument to determine
the size to allocate, rather than making the caller use sizeof().

Change-Id: Ib5d138d91a28bdda304a80db24ea9fb08669ad22
2024-07-20 20:20:01 +00:00
Qyriad e67dac1d74 libexpr: rename confusing makeImmutableString -> gcCopyStringIfNeeded
The purpose of this function has little to do with immutability. Value's
strings are never mutated, and the point of this function is to
singleton empty strings.

Change-Id: Ifd41dd952409d54e4d3de9ab59064e6928b0e480
2024-07-20 20:20:01 +00:00
Qyriad a3361557e3 libexpr: refactor gc-agnostic helpers into one place
Change-Id: Icc4b367e4f670d47256f62a3a002cd248a5c2d3b
2024-07-20 20:20:01 +00:00
eldritch horrors 0109368c3f libutil: make basic loggers thread-safe
SimpleLogger is not fully thread-safe, and all loggers that wrap it are
also not safe accordingly. this does not affect much, but in rare cases
it can cause interleaving of messages on stderr when used with the json
or raw log formats. the fix applied here is a bit of a hack, but fixing
this properly requires rearchitecting the logger infrastructure. nested
loggers are not the most natural abstraction here, and it is biting us.

Change-Id: Ifbf34fe1e85c60e73b59faee50e7411c7b5e7c12
2024-07-20 12:33:49 +00:00
eldritch horrors d8c09b5836 libutil: remove warnOnce macro
it's only used once, and even that one use is highly questionable. more
instances of warnOnce should be much more principled than this has been

Change-Id: I5856570c99cb44462e700d753d0c706a5db03c4b
2024-07-20 12:33:49 +00:00
Winter Cute 3da41fdb82 Fix namespace warning being emitted if sandbox is disabled
If useChroot = false, and user namespaces aren't available for some
reason (e.g. within a Docker container), this fixes a pointless warning
being emitted, as we would never attempt to use them even if they were
available.

Change-Id: Ibcee91c088edd2cd19e70218d5a5802bff8f537b
2024-07-19 19:14:54 -04:00
jade 77ff799cc8 gc: refactor the gc server thread out into a class without changing it
This removes a *whole load* of variables from scope and enforces thread
boundaries with the type system.

There is not much change of significance in here, so the things to watch
out for while reviewing it are primarily that the destructor ordering
may have changed inadvertently, I think.

Change-Id: I3cd87e6d5a08dfcf368637407251db22a8906316
2024-07-19 20:55:55 +00:00
jade 22252825c4 Merge changes Id8b3d289,Ib75ab5b8,I3792eeb3 into main
* changes:
  Fixup a bunch of references to nixos.org manuals
  Add release notes for removing overflow from Nix language
  expr: fix a compiler warning about different signs in comparison
2024-07-19 18:52:46 +00:00
alois31 aba5f19680 Merge changes I829581a3,I0016970d,I5dac8e77,Ib7560fe5 into main
* changes:
  doc/release-notes: add for pretty printing improvements
  libexpr/print: do not show elided nested items when there are none
  libexpr/print: never show empty attrsets or derivations as «repeated»
  libexpr/print: pretty-print idempotently
2024-07-19 06:40:13 +00:00
jade 26e56780ca Fixup a bunch of references to nixos.org manuals
(plus one reference to CppNix github)

Change-Id: Id8b3d2897f3b54e286861805cfd421adc4d5de47
2024-07-18 19:27:33 +00:00
jade 10cc3b288d Add release notes for removing overflow from Nix language
Change-Id: Ib75ab5b8b4d879035d7ee7678f9cd0c491a39c0a
2024-07-18 19:27:33 +00:00
jade 50a63f8435 expr: fix a compiler warning about different signs in comparison
We know that variable is >=0, so we can just cast it to unsigned.

Change-Id: I3792eeb3ca43e6a507cc44c1a70584d42b2acd7b
2024-07-18 19:27:33 +00:00
jade 5ee1e6ea98 Merge changes Ib20e9aa0,I178a038b,I29c7de04 into main
* changes:
  docs: document the actual comparison rules instead of lies
  daemon: remove workaround for macOS kernel bug that seems fixed
  daemon: fix a crash bug "FATAL: exception not rethrown"
2024-07-18 17:40:32 +00:00
alois31 768d1f29a2
doc/release-notes: add for pretty printing improvements
Change-Id: I829581a3f5b8b742e6c866dcdbbc635f91afceb5
2024-07-18 19:08:20 +02:00
alois31 40c39aa5d2
libexpr/print: do not show elided nested items when there are none
When the configured maximum depth has been reached, attribute sets and lists
are printed with ellipsis to indicate the elision of nested items. Previously,
this happened even in case the structure being printed is empty, so that such
items do not in fact exist. This is confusing, so stop doing it.

Change-Id: I0016970dad3e42625e085dc896e6f476b21226c9
2024-07-18 18:41:34 +02:00
alois31 b5da823138
libexpr/print: never show empty attrsets or derivations as «repeated»
The repeated value detection logic exists so that the occurrence of large
common substructures does not fill up the screen or the computer's memory.
However, empty attribute sets and derivations (when their detection is enabled)
are always cheap to print, and in practice I have observed them to make up a
significant majority of the cases where I was annoyed by the repeated value
detection kicking in. Furthermore, `nix-instantiate --eval` already disables
this logic for empty attribute sets, and empty lists are already exempted
everywhere. For these reasons, always print empty attribute sets and
derivations as what they are.

Change-Id: I5dac8e7739f9d726b76fd0521ec46f38af94463f
2024-07-18 18:41:34 +02:00
alois31 81a0624d76
libexpr/print: pretty-print idempotently
When pretty-printing is enabled, previously an unforced thunk would trigger
indentation, even when it subsequently does not evaluate to a nested structure.
The resulting output looked inconsistent, and furthermore pretty-printing was
not idempotent (since pretty-printing the same value again, which is now fully
evaluated, will not trigger indentation).
When strict evaluation is enabled, force the item before inspecting its type,
so that it is properly known whether it contains a nested structure.
Furthermore, there is no need to cause indentation for unforced thunks, since
the very next operation will be printing them as `«thunk»`.

This is mostly a port of https://github.com/NixOS/nix/pull/11100 , but we only
force the item when it's going to be forced anyway due to strict
pretty-printing, and a new test was written since the REPL testing framework in
Lix is different.

Co-Authored-By: Robert Hensing <robert@roberthensing.nl>
Change-Id: Ib7560fe531d09e05ca6b2037a523fe21a26d9d58
2024-07-18 18:41:28 +02:00
alois31 7b1abf8107 Merge "doc/manual: clarify documentation related to the $$ parser bug" into main 2024-07-18 15:01:20 +00:00
alois31 72db9cd67b doc/release-notes: link the upcoming release notes again
The insertion marker comment broke the list into two parts, the first
containing only the link to the upcoming release notes and the second the
past releases. This confused the generator, leading to the first part being
discarded. Indent the marker comment so that it's syntactically part of the
preceding item, and in particular doesn't split the list any more.

Change-Id: I357c51bb03e4e0d79a76d30158615fd9eda95ea8
2024-07-17 22:12:41 +00:00
raito 67f62bcdb4 doc/release-notes: add date for major release
Change-Id: I93aab93c069bb3989c3f8d17e0862899e6f76865
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-07-17 22:12:41 +00:00
alois31 beb231784e doc/manual: clarify documentation related to the $$ parser bug
Due to a mistake in the grammar, a dollar character implicitly escapes a second
dollar character that immediately follows, so that it cannot start an
interpolation. Unfortunately, this behaviour has since come to be relied upon,
so it cannot be fixed. Furthermore, the documentation on regular strings did
not mention this behaviour at all, while in the case of indented strings it was
rather implicit.
Mention it explicitly in both cases, and describe how an interpolation can
follow a dollar character (namely, by escaping that). Since we have to touch
that section anyway, state that any character (other than n, r, and t; but
notably including `$` even if not succeeded by `{`) can be escaped using a
backslash in regular strings.

Change-Id: I7e5d68a9a4130eec98ce8218b485168f4b31a677
2024-07-17 22:01:48 +00:00
Max “Goldstein” Siling 68567206f2 Merge "tests/functional/repl.sh: actually fail test on wrong stdout" into main 2024-07-17 21:50:59 +00:00
Max “Goldstein” Siling 3a36c8bb90 tests/functional/repl.sh: actually fail test on wrong stdout
Previous test implementation assumed that grep supports newlines
in patterns. It doesn't, so tests spuriously passed, even though
some tests outputs were broken.

This patches output (and expected output) before grepping,
so there're no newlines in pattern.

Change-Id: Ie6561f9f2e18b83d976f162269d20136e2595141
2024-07-17 21:48:13 +00:00
eldritch horrors ef0de7c79f remove boost coroutine references
we no longer need these since sinkToSource and sourceToSink are gone.

Change-Id: Ibbf440e2cf71bf3e9f3b833af2d78a21fb1b3193
2024-07-16 01:50:16 +00:00
eldritch horrors dfedbc154f remove sourceToSink, sinkToSource, and boehm patch
Change-Id: I1379841299713175d0225b82a67f50660f9eb5e2
2024-07-16 01:50:16 +00:00
eldritch horrors d094dd0396 libstore: remove remaining sinkToSource uses
Change-Id: Id1ee0d2ad4a3774f4bbb960d76f0f76ac4f3eff9
2024-07-16 01:50:16 +00:00
eldritch horrors 6b4d46e9e0 libstore: remove WriteConn::sink fields
we no longer need these since we're no longer using sinks to serialize things.

Change-Id: Iffb1a3eab33c83f611c88fa4e8beaa8d5ffa079b
2024-07-16 00:57:42 +00:00
eldritch horrors a5d1f69841 libstore: generatorize protocol serializers
this is cursed. deeply and profoundly cursed. under NO CIRCUMSTANCES
must protocol serializer helpers be applied to temporaries! doing so
will inevitably cause dangling references and cause the entire thing
to crash. we need to do this even so to get rid of boost coroutines,
and likewise to encapsulate the serializers we suffer today at least
a little bit to allow a gradual migration to an actual IPC protocol.

(this isn't a problem that's unique to generators. c++ coroutines in
general cannot safely take references to arbitrary temporaries since
c++ does not have a lifetime system that can make this safe. -sigh-)

Change-Id: I2921ba451e04d86798752d140885d3c5cc08e146
2024-07-16 00:57:42 +00:00
eldritch horrors 5271424d14 libstore: remove a sinkToSouce from old daemon protocol
this doesn't have a test because this code path is only reached by
clients that predate 2.4, and we really should not be caring about
those any more right now. even the test suite doesn't, and the few
tests that might care are disabled because they will not even work

Change-Id: Id9eb190065138fedb2c7d90c328ff9eb9d97385b
2024-07-16 00:57:42 +00:00
eldritch horrors 4ec87742a1 libstore: rewrite the nar parser as a contents generator
this is not completely necessary at this point because the parser right
now already returns a generator to pass through all input data it read,
but the nar parser *was* very lax and would accept nars that weren't in
canonical form (defined as the form dumpPath would return). nar hashing
depends on these things, and as such rewriting the parser now allows us
to reject non-canonical nars that extract to the same store contents as
their canonical counterpart but have different nar hashes despite that.

Change-Id: Iccd319e3bd5912d8297014c84c495edc59019bb7
2024-07-16 00:57:42 +00:00
Qyriad c052716edd Merge changes I8d87c0e9,I25937702 into main
* changes:
  nix3-upgrade-nix: always use the /new/ nix-env to perform the installation
  libutil: implement a realPath() utility
2024-07-15 23:18:03 +00:00
eldritch horrors 3447dbfb2c libstore: rewrite narFromPath as generator
Change-Id: Ifa783c2c65c06ddd1d0212016d5bfd07666ea91c
2024-07-15 21:50:25 +00:00
Lunaphied 5e16b10cb1 Merge "use clangStdenv for the default devShell, so we get clangd by default" into main 2024-07-15 21:47:23 +00:00
Qyriad ae7eab49b9 nix3-upgrade-nix: always use the /new/ nix-env to perform the installation
Fixes #411.

Change-Id: I8d87c0e9295deea26ff33234e15ee33cc68ab303
2024-07-15 15:26:53 -06:00
Qyriad d9c51ec4e5 libutil: implement a realPath() utility
Just a wrapper around POSIX realpath().

Change-Id: I2593770285dbae573eace490efce5b272b00b001
2024-07-15 15:26:53 -06:00
nan-git 1eb5d22132 Merge "libexpr/eval.cc: remove unnecessary C string conversion" into main 2024-07-14 16:53:54 +00:00
nan-git 505640baec libexpr/eval.cc: remove unnecessary C string conversion
Change-Id: I5b7c21df84ff8ff64cf6a1e261fc3729a06bd4f6
2024-07-14 02:13:58 +00:00
Artemis Tosini 201e8b6994
nix-support/binary-tarball.nix: Pass through root paths
Passing through root paths allows external programs to see
which nix and cacert are in a binary tarball,
e.g. to recreate it from substituters

Change-Id: I27431134df53bbc6623484f8a0822004b51f7c87
2024-07-13 23:22:43 +00:00
jade 702f02c31f docs: document the actual comparison rules instead of lies
Although the comparison rules are ugly and we do not like various parts
of them, we must not hide them away for only catgirls to know about, so
the documentation should actually say how they work.

Change-Id: Ib20e9aa0e7b6486ade4f401035aafd85fbb08c91
2024-07-13 01:17:14 +02:00
jade 69e2ee5b25 daemon: remove workaround for macOS kernel bug that seems fixed
This was filed as https://github.com/nixos/nix/issues/7584, but as far
as I can tell, the previous solution of POLLHUP works just fine on macOS
14. I've also tested on an ancient machine with macOS 10.15.7, which
also has POLLHUP work correctly.

It's possible this might regress some older versions of macOS that have
a kernel bug, but I went looking through the history on the sources and
didn't find anything that looked terribly convincingly like a bug fix
between 2020 and today. If such a broken version exists, it seems pretty
reasonable to suggest simply updating the OS.

Change-Id: I178a038baa000f927ea2cbc4587d69d8ab786843
2024-07-13 01:17:14 +02:00
jade a8f443d960 docs: update to define integer overflow
Change-Id: Ie8a1b31035f2d27a220e5df2e9e178ec3b39ee68
2024-07-13 00:59:33 +02:00
jade b3fb8d9822 daemon: fix a crash bug "FATAL: exception not rethrown"
This is caused by pthread_cancel effectively throwing a
not-specifically-identifiable C++ exception into the targeted thread,
which, if it is not rethrown, terminates the process entirely.

This is rather "impolite" behaviour, we would say. But thread
cancellation is *always* busted, and we should simply not use it where
unnecessary. It's particularly unnecessary when what we *actually* need
it for is, err, interrupting a poll(2).

That can in turn be achieved by simply listening to more stuff in the
poll, namely, a pipe, which we send a character to when needing to
stop the thread.

While looking at this code, we also investigated whether any of the
poll() madness is required, or was even *ever* required. Curiously we
found in the XNU kernel source code that the thing about needing to
listen to POLLHUP is probably *correct*, but switching it to POLLRDNORM
should not have made any difference at all. We've left a FIXME to look
into that further because what's written here is super janky.

94d3b45284/bsd/kern/sys_generic.c (L1751-L1758)

This is the crash on some Hydra machines:

Thread 1 (Thread 0x7f56b77776c0 (LWP 955542) (Exiting)):
0  0x00007f56b8e9b7dc in __pthread_kill_implementation () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
1  0x00007f56b8e49516 in raise () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
2  0x00007f56b8e31935 in abort () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
3  0x00007f56b8e327f3 in __libc_message_impl.cold () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
4  0x00007f56b8e8e8e9 in __libc_fatal () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
5  0x00007f56b8ea23c4 in unwind_cleanup () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
6  0x00007f56b9d2a1b8 in nix::triggerInterrupt() [clone .cold] () from /nix/store/sahgw550p621m9dy1pd7whl9c5g1g0p7-lix-2.90.0-rc1/lib/liblixutil.so
7  0x00007f56b990ac9d in std:🧵:_State_impl<std:🧵:_Invoker<std::tuple<nix::MonitorFdHup::MonitorFdHup(int)::{lambda()#1}> > >::_M_run() () from /nix/store/sahgw550p621m9dy1pd7whl9c5g1g0p7-lix-2.90.0-rc1/lib/liblixstore.so
8  0x00007f56b90e86d3 in execute_native_thread_routine () from /nix/store/c6r62m84hywf4i6qq1h28f13zv38yqyp-gcc-13.3.0-lib/lib/libstdc++.so.6
9  0x00007f56b8e99a42 in start_thread () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
10 0x00007f56b8f1905c in clone3 () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6

As for testing, we've started a daemon with this change and verified it
deals with HUPs correctly on x86_64-linux, but I don't think we can
easily test the destructor behaviour without whatever Hydra was
doing that broke.

Change-Id: I29c7de0425674494b6e43c075810126c3ff77363
2024-07-13 00:59:33 +02:00
jade 917c9bdee7 language: cleanly ban integer overflows
This also bans various sneaking of negative numbers from the language
into unsuspecting builtins as was exposed while auditing the
consequences of changing the Nix language integer type to a newtype.

It's unlikely that this change comprehensively ensures correctness when
passing integers out of the Nix language and we should probably add a
checked-narrowing function or something similar, but that's out of scope
for the immediate change.

During the development of this I found a few fun facts about the
language:
- You could overflow integers by converting from unsigned JSON values.
- You could overflow unsigned integers by converting negative numbers
  into them when going into Nix config, into fetchTree, and into flake
  inputs.

  The flake inputs and Nix config cannot actually be tested properly
  since they both ban thunks, however, we put in checks anyway because
  it's possible these could somehow be used to do such shenanigans some
  other way.

Note that Lix has banned Nix language integer overflows since the very
first public beta, but threw a SIGILL about them because we run with
-fsanitize=signed-overflow -fsanitize-undefined-trap-on-error in
production builds. Since the Nix language uses signed integers, overflow
was simply undefined behaviour, and since we defined that to trap, it
did.

Trapping on it was a bad UX, but we didn't even entirely notice
that we had done this at all until it was reported as a bug a couple of
months later (which is, to be fair, that flag working as intended), and
it's got enough production time that, aside from code that is IMHO buggy
(and which is, in any case, not in nixpkgs) such as
lix-project/lix#445, we don't think
anyone doing anything reasonable actually depends on wrapping overflow.

Even for weird use cases such as doing funny bit crimes, it doesn't make
sense IMO to have wrapping behaviour, since two's complement arithmetic
overflow behaviour is so *aggressively* not what you want for *any* kind
of mathematics/algorithms. The Nix language exists for package
management, a domain where bit crimes are already only dubiously in
scope to begin with, and it makes a lot more sense for that domain for
the integers to never lose precision, either by throwing errors if they
would, or by being arbitrary-precision.

This change will be ported to CppNix as well, to maintain language
consistency.

Fixes: lix-project/lix#423

Change-Id: I51f253840c4af2ea5422b8a420aa5fafbf8fae75
2024-07-13 00:59:33 +02:00
jade f9641b9efd libutil: add checked arithmetic tools
This is in preparation for adding checked arithmetic to the evaluator.

Change-Id: I6e115ce8f5411feda1706624977a4dcd5efd4d13
2024-07-13 00:56:37 +02:00
Lunaphied 0339b2fbd2 use clangStdenv for the default devShell, so we get clangd by default
The default-stdenv-devShell can always be used with `.#native-stdenvPackages`.

Change-Id: I9b3e72210ba5219b6b65c71a2818110769623904
2024-07-12 20:52:33 +00:00
jade dde51af97d Use std::strong_ordering for version comparison
The actual motive here is the avoidance of integer overflow if we were
to make these use checked NixInts and retain the subtraction.

However, the actual *intent* of this code is a three-way comparison,
which can be done with operator<=>, so we should just do *that* instead.

Change-Id: I7f9a7da1f3176424b528af6d1b4f1591e4ab26bf
2024-07-12 16:48:28 +02:00
eldritch horrors 4b109ec1a8 libstore: remove upcast_goal
upcast_goal was only ever needed to break circular includes, but the
same solution that gave us upcast_goal also lets us fully remove it:
just upcast goals without a wrapper function, but only in .cc files.

Change-Id: I9c71654b2535121459ba7dcfd6c5da5606904032
2024-07-11 21:31:52 +00:00
eldritch horrors a5d431a911 libstore: turn copyNAR into a generator
Change-Id: Id452f6a03faa1037ff13af0f63e32883966ff40d
2024-07-11 20:37:27 +00:00
eldritch horrors 03db4efab9 libstore: turn the NAR parser into a passthrough generator
this will let us turn copyNAR into a generator as well, which in turn is
necessary to turn the users of copyNAR into generators without resorting
to sinkToSource coroutines. currently this uses the SerializingTransform
in all cases, even for copyNAR where it is not necessary. should this be
a performance problem we can easily swap out the transform for one which
does not produce any bytes of its own, but that should not be necessary.

Change-Id: I7e685879318fcbb78d8b88abfddd7752360eb0ce
2024-07-11 20:37:27 +00:00
eldritch horrors 31478c810a libutil: remove makeDecompressionSink
the sole remaining user of this function can use makeDecompressionSource
instead, while making the sinkToSource in the caller unnecessary as well

Change-Id: I4258227b5dbbb735a75b477d8a57007bfca305e9
2024-07-11 11:39:18 +00:00
eldritch horrors 5587dbdcf0 libstore: make BinaryCacheStore::getFile return a source
this lets us remove the last true remaining uses of
makeDecompressionSink.

Change-Id: I146ca2bbe1a9ae9a367117a7b8a304b23a63e5e2
2024-07-11 11:39:18 +00:00
eldritch horrors df8851f286 libutil: rewrite RewritingSink as source
the rewriting sink was just broken. when given a rewrite set that
contained a key that is also a proper infix of another key it was
possible to produce an incorrectly rewritten result if the writer
used the wrong block size. fixing this duplicates rewriteStrings,
to avoid this we'll rewrite rewriteStrings to use RewritingSource
in a new mode that'll allow rewrites we had previously forbidden.

Change-Id: I57fa0a9a994e654e11d07172b8e31d15f0b7e8c0
2024-07-11 11:39:18 +00:00
Lunaphied 014410cbf0 Merge "lix-doc: update dependencies and refactor" into main 2024-07-10 17:47:27 +00:00
eldritch horrors f0c751d4d6 Merge changes I81552018,Ieb65c133 into main
* changes:
  releng: add releaseTests flake output, test script
  add aarch64-linux as a cross-build target
2024-07-10 17:45:17 +00:00
piegames 3dced96741 devShell: Fix errors with environment variables
Previous code assumed that $MANPATH always exists

Change-Id: I5a4d012045ba6ff9086373b3f46a75d82285d393
2024-07-10 14:22:47 +02:00
alois31 fa92f41a18 Merge "libmain: clear display attributes in the multiline progress bar" into main 2024-07-10 03:43:07 +00:00
Lunaphied 41963df4a5 lix-doc: update dependencies and refactor
This updates the version of rnix used and refactors the code generally
to be more precise and capable in it's identification of both lambdas
and determining which documentation comments are attached.

Change-Id: Ib0dddabd71f772c95077f9d7654023b37a7a1fd2
2024-07-10 02:25:48 +00:00
eldritch horrors f5aa5b6815 releng: add releaseTests flake output, test script
this is supposed to be a set of outputs we want to always succeed for
releases. sadly we can't add nixos installer tests using lix to these
because the nixos test framework does not allow overriding nix in the
installer test suites due to unfortunate oversights in the framework.

Change-Id: I815520181ccca70a47205d38ba27e73529347f04
2024-07-09 22:50:51 +02:00
eldritch horrors 4d8c66ec6f add aarch64-linux as a cross-build target
we want to be sure we can cross-build to aarch64 for releases, add a
target to our crossSystems list to make those cheacks easier to run.

Change-Id: Ieb65c1333a5232641ace0ba4d122fc7d528ebc04
2024-07-09 22:49:10 +02:00
Quantum Jump 6e0ca02425 Fix dry-run flag for nix-collect-garbage
`nix-collect-garbage --dry-run` previously elided the entire garbage
collection check, meaning that it would just exit the script without
printing anything.

This change makes the dry run flag instead set the GC action to
`gcReturnDead` rather than `gcDeleteDead`, and then continue with the
script. So if you set `--dry-run`, it will print the paths it *would*
have garbage collected, but not actually delete them.

I filed a bug for this: lix-project/lix#432 but then realised I could give fixing it a go myself.

Change-Id: I062dbf1a80bbab192b5fd0b3a453a0b555ad16f2
2024-07-09 13:55:05 +00:00
Qyriad accfd8aa9d libexpr: stop lying about DrvInfo's constness
DrvInfo's query methods all use mutable fields to cache, but like.
that's basically the entire interface for DrvInfo. Not only that, but
these formerly-const-marked functions can even throw due to eval errors!
Changing this only required removing some `const` markers in nix-env,
and changing a single inline `queryInstalled()` call to be an lvalue
instead.

Change-Id: I796807118f3b35b0e93668b5e28210d9e521b2ae
2024-07-08 17:52:02 +00:00
alois31 f5ff70d7f3
libmain: clear display attributes in the multiline progress bar
Activities can set display attributes in their log output using the "Select
Graphics Rendition" functionality. To prevent interfering with subsequent text
displayed, these should be reset after writing the log line. The multiline
progress bar neglected to do this, resulting for example in a colorised
"building …" header in the next line. Reset the attributes properly, like the
standard progress bar already does.

Change-Id: I1dc69f4a1d747a76b83e8721a72d9bb0e5554488
2024-07-08 19:08:23 +02:00
Artemis Tosini d461cc1d7b
libstore: make LocalDerivationGoal::needsHashRewrite virtual
This rather simple function existed just to check some flags,
but the response varies by platform. This is a perfect case for
our subclasses.

Change-Id: Ieb1732a8d024019236e0d0028ad843a24ec3dc59
2024-07-07 18:06:08 +00:00
eldritch horrors 55a32f24d3 libutil: remove RewritingSink match/size tracking
size tracking can be done with a LengthSink and a tee. match tracking
was defeated by never having done any match tracking, all users would
see the same (empty) set of matches at all times. match tracking with
bytes offsets alone would not be sufficient in the general case, only
because computeHashModulo uses a single rewrite could it have worked.

Change-Id: Idb214b5222e0ea24f450f5505712a342b63d7570
2024-07-06 12:36:37 +02:00
eldritch horrors 5af76dee37 libutil: turn HashModuloSink into a free function
Change-Id: I5878007502fa68c2816a0f4c61f7d0e60bdde702
2024-07-06 12:36:37 +02:00
eldritch horrors 4162a66cee libutil: return sources from runProgram2
this much more closely mimics what is actually happening: we're reading
data from somewhere else, actively, rather than passively waiting. with
the data flow matching the underlying system interactions better we can
remove a few sinkToSource calls that merely exists to undo the mismatch
caused by not treating subprocess output as a data source to begin with

Change-Id: If4abfc2f8398fb5e88c9b91a8bdefd5504bb2d11
2024-07-06 12:36:36 +02:00
eldritch horrors b6a08a2fed libutil: return a program handle from runProgram2
this will let us also return a source for the program output later,
which will in turn make sinkToSource unnecessary for program output
processing. this may also reopen a path for provigin program input,
but that still needs a proper async io framework to avoid problems.

Change-Id: Iaf93f47db99c38cfaf134bd60ed6a804d7ddf688
2024-07-05 22:28:16 +00:00
eldritch horrors f4f6d1d8e2 libutil: convert readFileSource to a generator
Change-Id: I5f92b15fd367d46eb047d74ab6e317b4f51a46d3
2024-07-05 22:28:16 +00:00
eldritch horrors 06220a71c1 libstore: convert dumpPath to a generator
Change-Id: Ic4cf5562504aa29130304469936f958c0426e5ef
2024-07-05 22:28:16 +00:00
eldritch horrors b51ea465de libutil: allow construction of sources from generators
Change-Id: I78ff8d0720f06bce731e26d5e1c53b1382bbd589
2024-07-05 22:28:16 +00:00
Qyriad b9f91ec3c5 mildly cleanup libexpr/eval.hh
Change-Id: I40d01a8f8b7fb101279c6f88ebdf1f0969d9d7f0
2024-07-04 17:43:03 -06:00
Qyriad 4c7165be86 distinguish between throws & errors during throw
Turns errors like this:

let
  throwMsg = a: throw (a + " invalid bar");
in throwMsg "bullshit"

error:
       … from call site
         at «string»:3:4:
            2|   throwMsg = a: throw (a + " invalid bar");
            3| in throwMsg "bullshit"
             |    ^

       … while calling 'throwMsg'
         at «string»:2:14:
            1| let
            2|   throwMsg = a: throw (a + " invalid bar");
             |              ^
            3| in throwMsg "bullshit"

       … while calling the 'throw' builtin
         at «string»:2:17:
            1| let
            2|   throwMsg = a: throw (a + " invalid bar");
             |                 ^
            3| in throwMsg "bullshit"

       error: bullshit invalid bar

into errors like this:

let
  throwMsg = a: throw (a + " invalid bar");
in throwMsg "bullshit"

error:
       … from call site
         at «string»:3:4:
            2|   throwMsg = a: throw (a + " invalid bar");
            3| in throwMsg "bullshit"
             |    ^

       … while calling 'throwMsg'
         at «string»:2:14:
            1| let
            2|   throwMsg = a: throw (a + " invalid bar");
             |              ^
            3| in throwMsg "bullshit"

       … caused by explicit throw
         at «string»:2:17:
            1| let
            2|   throwMsg = a: throw (a + " invalid bar");
             |                 ^
            3| in throwMsg "bullshit"

       error: bullshit invalid bar

Change-Id: I593688928ece20f97999d1bf03b2b46d9ac338cb
2024-07-04 17:43:03 -06:00
Qyriad 14bf54bd39 trace which part of foo.bar.baz errors
Turns errors like:

let
  somepkg.src = throw "invalid foobar";
in somepkg.src.meta

error:
       … while evaluating the attribute 'src.meta'
         at «string»:2:3:
            1| let
            2|   somepkg.src = throw "invalid foobar";
             |   ^
            3| in somepkg.src.meta

       … while calling the 'throw' builtin
         at «string»:2:17:
            1| let
            2|   somepkg.src = throw "invalid foobar";
             |                 ^
            3| in somepkg.src.meta

       error: invalid foobar

into errors like:

let
  somepkg.src = throw "invalid foobar";
in somepkg.src.meta

error:
       … while evaluating the attribute 'src.meta'
         at «string»:2:3:
            1| let
            2|   somepkg.src = throw "invalid foobar";
             |   ^
            3| in somepkg.src.meta

       … while evaluating 'somepkg.src' to select 'meta' on it
         at «string»:3:4:
            2|   somepkg.src = throw "invalid foobar";
            3| in somepkg.src.meta
             |    ^

       … while calling the 'throw' builtin
         at «string»:2:17:
            1| let
            2|   somepkg.src = throw "invalid foobar";
             |                 ^
            3| in somepkg.src.meta

       error: invalid foobar

And for type errors, from:

let
  somepkg.src = "I'm not an attrset";
in somepkg.src.meta

error:
       … while evaluating the attribute 'src.meta'
         at «string»:2:3:
            1| let
            2|   somepkg.src = "I'm not an attrset";
             |   ^
            3| in somepkg.src.meta

       … while selecting an attribute
         at «string»:3:4:
            2|   somepkg.src = "I'm not an attrset";
            3| in somepkg.src.meta
             |    ^

       error: expected a set but found a string: "I'm not an attrset"

into:

let
  somepkg.src = "I'm not an attrset";
in somepkg.src.meta

error:
       … while evaluating the attribute 'src.meta'
         at «string»:2:3:
            1| let
            2|   somepkg.src = "I'm not an attrset";
             |   ^
            3| in somepkg.src.meta

       … while selecting 'meta' on 'somepkg.src'
         at «string»:3:4:
            2|   somepkg.src = "I'm not an attrset";
            3| in somepkg.src.meta
             |    ^

       error: expected a set but found a string: "I'm not an attrset"

For the low price of an enumerate() and a lambda you too can have the
incorrect line of code actually show up in the trace!

Change-Id: Ic1491c86e33c167891bdac9adad6224784760bd6
2024-07-04 17:42:35 -06:00
Qyriad d00edfb28d trace when the foo part of foo.bar.baz errors
Turns errors like:

let
  errpkg = throw "invalid foobar";
in errpkg.meta

error:
       … while calling the 'throw' builtin
         at «string»:2:12:
            1| let
            2|   errpkg = throw "invalid foobar";
             |            ^
            3| in errpkg.meta

       error: invalid foobar

into errors like:

let
  errpkg = throw "invalid foobar";
in errpkg.meta

error:
       … while evaluating 'errpkg' to select 'meta' on it
         at «string»:3:4:
            2|   errpkg = throw "invalid foobar";
            3| in errpkg.meta
             |    ^

       … while calling the 'throw' builtin
         at «string»:2:12:
            1| let
            2|   errpkg = throw "invalid foobar";
             |            ^
            3| in errpkg.meta

       error: invalid foobar

For the low price of one try/catch, you too can have the incorrect line
of code actually show up in the trace!

Change-Id: If8d6200ec1567706669d405c34adcd7e2d2cd29d
2024-07-04 16:33:02 -06:00
Qyriad 139cfdfb53 add an ExprPrinter class, like ValuePrinter
To be used Shortly

Change-Id: I9def7975aa55f251eb8486391677771f7352d7ce
2024-07-04 15:55:38 -06:00
Qyriad 59bf6825ef add an impl of Expr::show for ExprInheritFrom that doesn't crash
ExprVar::show() assumes it has a name. dynamic inherits do not
necessarily (ever?) have a name.

Change-Id: If10893188e307431da17f0c1bd0787adc74f7141
2024-07-04 15:55:38 -06:00
Qyriad 4f0c27abe1 give ExprInheritFrom a handle to what its standing in for
Change-Id: I12088e0b618407e5432523bbc97be63c8d6fce62
2024-07-04 15:55:38 -06:00
Artemis Tosini e040b762a4
libstore: add LocalDerivationGoal startChild hook
Add a platform-specific function for starting sandboxed child.
Generally this just means startProcess, but on Linux we use flags
for clone to start a new namespace

Change-Id: I41c8aba62676a162388bbe5ab8a7518904c7b058
2024-07-03 22:37:41 +00:00
Artemis Tosini af1dcc2d5e
libstore: Add LocalDerivationGoal prepareSandbox hook
Add a new OS-specific hook called `prepareSandbox`, run before forking
On Darwin this is empty as nothing is required,
on Linux this creates the chroot directory and adds basic files,
and on platforms using a fallback this throws an exception

Change-Id: Ie30c38c387f2e0e5844b2afa32fd4d33b1180dae
2024-07-03 22:16:03 +00:00
eldritch horrors 5eec6418de libutil: begin porting serialization to generators
generators are a better basis for serializers than streaming into sinks
as we do currently for many reasons, such as being usable as sources if
one wishes to (without requiring an intermediate sink to serialize full
data sets into memory, or boost coroutines to turn sinks into sources),
composing more naturally (as one can just yield a sub-generator instead
of being forced to wrap entire substreams into clunky functions or even
more clunky custom types to implement operator<< on), allowing wrappers
to transform data with clear ownership semantics (removing the need for
explicit memory allocations and Source wrappers), and many other things

Change-Id: I361d89ff556354f6930d9204f55117565f2f7f20
2024-07-03 11:46:53 +00:00
eldritch horrors c65f5dd18e libutil: convert drainFD to a Bytes generator
the `*Source` name is a slight misnomer since we do also have a
Source type, but we can probably live with this for time being.

Change-Id: I54eb2e59a4009014e324797f16b80b962759c7d3
2024-07-03 11:46:53 +00:00
eldritch horrors b252b3c6e3 libutil: allow draining Generator<Bytes> into sinks
Change-Id: I442d03a5399096d4baca9a2618b4c4b64db36c4b
2024-07-03 11:46:53 +00:00
eldritch horrors 4857feb910 libutil: add Bytes type
not used anywhere yet, but we'll use this a lot soon for generators that
return file contents, wire protocol fragments, or indeed any byte stream

Change-Id: I01a46f9bf9d75aaf4a5d7662773b99f498862a28
2024-07-03 11:46:53 +00:00
eldritch horrors 73ddc4540f libutil: generator type with on-yield value mapping
this will be the basis of non-boost coroutines in lix. anything that is
a boost coroutine *should* be representable with a Generator coroutine,
and many things that are not currently boost coroutines but behave much
like one (such as, notably, serializers) should be as well. this allows
us to greatly simplify many things that look like iteration but aren't.

Change-Id: I2cebcefa0148b631fb30df4c8cfa92167a407e34
2024-07-03 11:46:53 +00:00
Qyriad 45ac449d39 Merge "Revert "Revert "bump the extra --version info from info to notice, -vv -> -v""" into main 2024-07-02 22:26:34 +00:00
Qyriad 18a06aad52 Revert "Revert "bump the extra --version info from info to notice, -vv -> -v""
This reverts commit 9f16a20f3d.

Since c55e93ca2, the original reason for reverting d003dcd7f
no longer applies.

Change-Id: If88f8555a6060d0530dcfecdf55afaa40afd1ad9
2024-07-02 22:03:17 +00:00
eldritch horrors e7517419a6 libmain: better fix for #424, #425
not printing activities at all when no progress information is available
hides *all* progress information from e.g. flake show. this is not ideal
and needs to be fixed, but the fix *still* has problems with flake show:
in multiline mode we will overwrite all useful flake show output as soon
as the progress bar is redrawn. flake show output is also mangled in any
number of other situations (like -v being set), so we should probably be
not too worried about it and fix progress reporting properly another day

Change-Id: I6d39d670e261bbae00560b6a8e15dec8e16b35c4
2024-07-02 17:16:30 +02:00
alois31 24852355d8 Merge "tree-wide: unify progress bar inactive and paused states" into main 2024-07-02 14:12:07 +00:00
Delan Azabani 865a3732fa Merge "Reject fully-qualified URLs in 'from' argument of nix registry add" into main 2024-07-02 07:20:01 +00:00
alois31 0dd1d8ca1c
tree-wide: unify progress bar inactive and paused states
Previously, the progress bar had two subtly different states in which the bar
would not actually render, both with their own shortcomings: inactive (which
was irreversible) and paused (reversible, but swallowing logs). Furthermore,
there was no way of resetting the statistics, so a very bad solution was
implemented (243c0f18da) that would create a new
logger for each line of the repl, leaking the previous one and discarding the
value of printBuildLogs. Finally, if stderr was not attached to a TTY, the
update thread was started even though the logger was not active, violating the
invariant required by the destructor (which is not observed because the logger
is leaked).

In this commit, the two aforementioned states are unified into a single one,
which can be exited again, correctly upholds the invariant that the update
thread is only running while the progress bar is active, and does not swallow
logs. The latter change in behavior is not expected to be a problems in the
rare cases where the paused state was used before, since other loggers (like
the simple one) don't exhibit it anyway. The startProgressBar/stopProgressBar
API is removed due to being a footgun, and a new method for properly resetting
the progress is added.

Co-Authored-By: Qyriad <qyriad@qyriad.me>
Change-Id: I2b7c3eb17d439cd0c16f7b896cfb61239ac7ff3a
2024-07-01 18:19:34 +02:00
jade d3286d0990 Merge changes Ie29a8a89,I873eedcf into main
* changes:
  store: delete obsolete lsof-disabling code
  store: guess the URL of failing fixed-output derivations
2024-07-01 16:11:32 +00:00
eldritch horrors 010ff57ebb enable -Werror=unused-result
we do not have any of these warnings appearing at the moment, but it
seems like a good idea to enable [[nodiscard]] checking anyway. once
we start introducing more functions with must-use conditions we will
need such checking, and the rust stdlib has proven them very useful.

Change-Id: Ibb6b042ae1ec5f527f8dc2809a7816a4c1548ae2
2024-06-30 21:30:43 +00:00
alois31 a55112898e
libexpr/flake: allow automatic rejection of configuration options from flakes
The `allow-flake-configuration` option allows the user to control whether to
accept configuration options supplied by flakes. Unfortunately, setting this
to false really meant "ask each time" (with an option to remember the choice
for each specific option encountered). Let no mean no, and introduce (and
default to) a separate value for the "ask each time" behaviour.

Co-Authored-By: Jade Lovelace <lix@jade.fyi>
Change-Id: I7ccd67a95bfc92cffc1ebdc972d243f5191cc1b4
2024-06-30 19:28:14 +02:00
Delan Azabani b2944d93a6 Reject fully-qualified URLs in 'from' argument of nix registry add
We previously allowed you to map any flake URL to any other flake URL,
including shorthand flakerefs, indirect flake URLs like `flake:nixpkgs`,
direct flake URLs like `github:NixOS/nixpkgs`, or local paths.

But flake registry entries mapping from direct flake URLs often come
from swapping the 'from' and 'to' arguments by accident, and even when
created intentionally, they may not actually work correctly.

This patch rejects those URLs (and fully-qualified flake: URLs), making
it harder to swap the arguments by accident.

Fixes #181.

Change-Id: I24713643a534166c052719b8770a4edfcfdb8cf3
2024-06-29 05:11:31 +00:00
jade d85309f7ca store: delete obsolete lsof-disabling code
Since Ifa0adda7984e, we don't use this code anymore on macOS, so we have
no reason to have a knob to disable it anymore.

Change-Id: Ie29a8a8978d9aefd4551895f4f9b3cc0827496df
2024-06-27 22:53:36 -07:00
jade d92712673b store: guess the URL of failing fixed-output derivations
This is a shameless layering violation in favour of UX. It falls back
trivially to "unknown", so it's purely a UX feature.

Diagnostic sample:

```
error: hash mismatch in fixed-output derivation '/nix/store/sjfw324j4533lwnpmr5z4icpb85r63ai-x1.drv':
        likely URL: https://meow.puppy.forge/puppy.tar.gz
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-a1Qvp3FOOkWpL9kFHgugU1ok5UtRPSu+NwCZKbbaEro=
```

Change-Id: I873eedcf7984ab23f57a6754be00232b5cb5b02c
2024-06-27 22:44:16 -07:00
jade 5dc85e8b72 Merge "packaging: make pegtl use the __forDefaults mechanism" into main 2024-06-26 22:11:52 +00:00
jade 77c5364596 Merge "doc/hacking: fix up some outdated info about cross, hydra links" into main 2024-06-26 22:11:36 +00:00
eldritch horrors 3dd7d023f4 libmain: don't print empty lines
this most notably affects `nix eval`: if there is no progress bar to be
shown and no activities going on we should not print anything at all. a
progress bar with no activities would print a bunch of terminal escapes
*and a space*, which is not helpful in simple cases like nix eval -E 1.
notably this does *not* affect nix eval called on non-terminal outputs,
but it is slightly confusing nevertheless (and not difficult to avoid).

fixes lix-project/lix#424

Change-Id: Iee793c79ba5a485d6606e0d292ed2eae6dfb7216
2024-06-26 17:44:04 +00:00
jade 9afb0fe41c Merge changes I476a2516,I8a274227 into main
* changes:
  doc/hacking: fix internal api docs section to say to enable it
  doc: Add more about the release note generator
2024-06-26 17:34:45 +00:00
alois31 30da1b17d9 Merge "libmain/progress-bar: move implementation out of the header" into main 2024-06-26 16:05:44 +00:00
jade f7d54cb6b1 packaging: make pegtl use the __forDefaults mechanism
This avoids needing to pass it in when callPackage'ing Lix from external
code.

Change-Id: Ie07e84a151e38614064609a2f6dbff165e193be7
2024-06-26 00:44:46 -07:00
jade 85c1241201 doc/hacking: fix up some outdated info about cross, hydra links
We would like to build these with Hydra but we do not currently have a
Hydra to build them with conveniently.

Change-Id: I0832a33881138dd1caab3805df7ad097db347e62
2024-06-25 21:46:26 -07:00
jade 33d53c4983 doc/hacking: fix internal api docs section to say to enable it
I filed a bug to build these in releng in the future:
lix-project/lix#422

Change-Id: I476a2516cc2be382d4b7c8529a02f9212a78fdb2
2024-06-25 21:26:18 -07:00
jade e537678f1e doc: Add more about the release note generator
Change-Id: I8a274227cb1b05d442d3f644603dd2844ecc9d05
2024-06-25 21:22:37 -07:00
jade 4ac2c496d4 Merge "change shebangs of all .sh scripts to bash" into main 2024-06-25 22:18:26 +00:00
jade aceef13682 Merge changes If0ddec6b,Iaa63ed18 into main
* changes:
  Add some release notes for things we did
  packaging: don't build internal api docs by default in dev shells
2024-06-25 22:16:04 +00:00
Lunaphied f170870ae7 Merge ".envrc: remove MAKEFLAGS and use clang environment by default" into main 2024-06-25 20:42:46 +00:00
Lunaphied 97c86908a4 .envrc: remove MAKEFLAGS and use clang environment by default
MAKEFLAGS hasn't been relevant since we switched off the Make
buildsystem and using the clang environment by default gives you clangd
by default which most developers will want.

Change-Id: I9c11d0613577047e6c908f049c1ffaca5fb5ff67
2024-06-25 12:36:18 -06:00
eldritch horrors e6cd67591b libexpr: rewrite the parser with pegtl instead of flex/bison
this gives about 20% performance improvements on pure parsing. obviously
it will be less on full eval, but depending on how much parsing is to be
done (e.g. including hackage-packages.nix or not) it's more like 4%-10%.

this has been tested (with thousands of core hours of fuzzing) to ensure
that the ASTs produced by the new parser are exactly the same as the old
one would have produced. error messages will change (sometimes by a lot)
and are not yet perfect, but we would rather leave this as is for later.

test results for running only the parser (excluding the variable binding
code) in a tight loop with inputs and parameters as given are promising:

  - 40% faster on lix's package.nix at 10000 iterations
  - 1.3% faster on nixpkgs all-packages.nix at 1000 iterations
  - equivalent on all of nixpkgs concatenated at 100 iterations
    (excluding invalid files, each file surrounded with parens)

more realistic benchmarks are somewhere in between the extremes, parsing
once again getting the largest uplift. other realistic workloads improve
by a few percentage points as well, notably system builds are 4% faster.

Benchmarks summary (from ./bench/summarize.jq bench/bench-*.json)
old/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  mean:     0.408s ± 0.025s
            user: 0.355s | system: 0.033s
  median:   0.389s
  range:    0.388s ... 0.442s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  mean:     0.332s ± 0.024s
            user: 0.279s | system: 0.033s
  median:   0.314s
  range:    0.313s ... 0.361s
  relative: 0.814

---

old/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     6.133s ± 0.022s
            user: 5.395s | system: 0.437s
  median:   6.128s
  range:    6.099s ... 6.183s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     5.925s ± 0.025s
            user: 5.176s | system: 0.456s
  median:   5.934s
  range:    5.861s ... 5.943s
  relative: 0.966

---

GC_INITIAL_HEAP_SIZE=10g old/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     4.503s ± 0.027s
            user: 3.731s | system: 0.547s
  median:   4.499s
  range:    4.478s ... 4.541s
  relative: 1

GC_INITIAL_HEAP_SIZE=10g new/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     4.285s ± 0.031s
            user: 3.504s | system: 0.571s
  median:   4.281s
  range:    4.221s ... 4.328s
  relative: 0.951

---

old/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
  mean:     16.475s ± 0.07s
            user: 14.088s | system: 1.572s
  median:   16.495s
  range:    16.351s ... 16.536s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
  mean:     15.973s ± 0.013s
            user: 13.558s | system: 1.615s
  median:   15.973s
  range:    15.946s ... 15.99s
  relative: 0.97

---

Change-Id: Ie66ec2d045dec964632c6541e25f8f0797319ee2
2024-06-25 12:24:58 +00:00
jade c097ebe66b Merge "Revert "libfetchers: make attribute / URL query handling consistent"" into main 2024-06-25 10:19:52 +00:00
jade e19f27917c Add some release notes for things we did
Change-Id: If0ddec6b64a43c3d8f6cae39e0292863f3b49401
2024-06-24 16:26:12 -07:00
jade 1245340e44 packaging: don't build internal api docs by default in dev shells
These are totally available and you can just turn them on, but they have
very bad dependency tracking and thus bloat incremental change times,
which is not really ok.

Change-Id: Iaa63ed18a789e74fcb757248cd24c3b194afcc80
2024-06-24 15:57:38 -07:00
jade 3e151d4d77 Revert "libfetchers: make attribute / URL query handling consistent"
This reverts commit 35eec921af.

Reason for revert: Regressed nix-eval-jobs, and it appears to be this change is buggy/missing a case. It just needs another pass.

Code causing the problem in n-e-j, when invoked with `nix-eval-jobs --flake '.#hydraJobs'`:

```
n-e-j/tests/assets » ../../build/src/nix-eval-jobs --meta --workers 1 --flake .#hydraJobs
warning: unknown setting 'trusted-users'
warning: `--gc-roots-dir' not specified
error: unsupported Git input attribute 'dir'
error: worker error: error: unsupported Git input attribute 'dir'
```

```
  nix::Value *vRoot = [&]() {
        if (args.flake) {
            auto [flakeRef, fragment, outputSpec] =
                nix::parseFlakeRefWithFragmentAndExtendedOutputsSpec(
                    args.releaseExpr, nix::absPath("."));
            nix::InstallableFlake flake{
                {}, state, std::move(flakeRef), fragment, outputSpec,
                {}, {},    args.lockFlags};

            return flake.toValue(*state).first;
        } else {
            return releaseExprTopLevelValue(*state, autoArgs, args);
        }
    }();
```

Inspecting the program behaviour reveals that `dir` was in fact set in the URL going into the fetcher. This is in turn because unlike in the case changed in this commit, it was not erased before handing it to libfetchers, which is probably just a mistake.

```
(rr) up
3  0x00007ffff60262ae in nix::fetchers::Input::fromURL (url=..., requireTree=requireTree@entry=true) at src/libfetchers/fetchers.cc:39
warning: Source file is more recent than executable.
39              auto res = inputScheme->inputFromURL(url, requireTree);
(rr) p url
$1 = (const nix::ParsedURL &) @0x7fffdc874190: {url = "git+file:///home/jade/lix/nix-eval-jobs", 
  base = "git+file:///home/jade/lix/nix-eval-jobs", scheme = "git+file", authority = std::optional<std::string> = {[contained value] = ""}, 
  path = "/home/jade/lix/nix-eval-jobs", query = std::map with 1 element = {["dir"] = "tests/assets"}, fragment = ""}
(rr) up
4  0x00007ffff789d904 in nix::parseFlakeRefWithFragment (url=".#hydraJobs", baseDir=std::optional<std::string> = {...}, 
    allowMissing=allowMissing@entry=false, isFlake=isFlake@entry=true) at src/libexpr/flake/flakeref.cc:179
warning: Source file is more recent than executable.
179                                 FlakeRef(Input::fromURL(parsedURL, isFlake), getOr(parsedURL.query, "dir", "")),
(rr) p parsedURL
$2 = {url = "git+file:///home/jade/lix/nix-eval-jobs", base = "git+file:///home/jade/lix/nix-eval-jobs", scheme = "git+file", 
  authority = std::optional<std::string> = {[contained value] = ""}, path = "/home/jade/lix/nix-eval-jobs", query = std::map with 1 element = {
    ["dir"] = "tests/assets"}, fragment = ""}
(rr) list
174
175                             if (pathExists(flakeRoot + "/.git/shallow"))
176                                 parsedURL.query.insert_or_assign("shallow", "1");
177
178                             return std::make_pair(
179                                 FlakeRef(Input::fromURL(parsedURL, isFlake), getOr(parsedURL.query, "dir", "")),
180                                 fragment);
181                         }
```

Change-Id: Ib55a882eaeb3e59228857761dc1e3b2e366b0f5e
2024-06-24 22:49:17 +00:00
vigress8 c7af89c797 change shebangs of all .sh scripts to bash
On operating systems where /bin/sh is not Bash, some scripts are invalid
because of bashisms, and building Lix fails with errors like this:
`render-manpage.sh: 3: set: Illegal option -o pipefail`
This modifies all scripts that use a `/bin/sh` shebang to `/usr/bin/env
bash`, including currently POSIX-compliant ones, to prevent any future
confusion.

Change-Id: Ia074cc6db42d40fc59a63726f6194ea0149ea5e0
2024-06-24 14:00:43 -07:00
jade d5637ee790 devShell: guard against running from another directory
I was working on nix-eval-jobs with a dev shell with some shenanigans to
run against a locally built Lix and it was getting really annoying when
`nix develop ../lix#` was messing up my other git repo's hooks.

This is a fix via blunt force, but it is at least obvious how it works.

Change-Id: Ia29eeb5be57ab6a2c88451c00ea18a51e4dfe65e
2024-06-24 13:41:38 -07:00
jade eb5de71adc justfile: accept extra options to just setup and pass them to meson
This lets you get the default options and still be able to add more.

Change-Id: Ife32c348b1498ff2ccdddf051a5bba520cfa36f0
2024-06-24 13:37:01 -07:00
alois31 206a5dbb8f
libmain/progress-bar: move implementation out of the header
Change-Id: Ib4b42ebea290ee575294df6b2f17a38a5d850b80
2024-06-24 20:39:50 +02:00
Robert Hensing d86009bd76 Add build-dir setting, clean up default TMPDIR handling
This is a squash of upstream PRs #10303, #10312 and #10883.

fix: Treat empty TMPDIR as unset

Fixes an instance of

    nix: src/libutil/util.cc:139: nix::Path nix::canonPath(PathView, bool): Assertion `path != ""' failed.

... which I've been getting in one of my shells for some reason.
I have yet to find out why TMPDIR was empty, but it's no reason for
Nix to break.

(cherry picked from commit c3fb2aa1f9d1fa756dac38d3588c836c5a5395dc)

fix: Treat empty XDG_RUNTIME_DIR as unset

See preceding commit. Not observed in the wild, but is sensible
and consistent with TMPDIR behavior.

(cherry picked from commit b9e7f5aa2df3f0e223f5c44b8089cbf9b81be691)

local-derivation-goal.cc: Reuse defaultTempDir()

(cherry picked from commit fd31945742710984de22805ee8d97fbd83c3f8eb)

fix: remove usage of XDG_RUNTIME_DIR for TMP

(cherry picked from commit 1363f51bcb24ab9948b7b5093490a009947f7453)

tests/functional: Add count()

(cherry picked from commit 6221770c9de4d28137206bdcd1a67eea12e1e499)

Remove uncalled for message

(cherry picked from commit b1fe388d33530f0157dcf9f461348b61eda13228)

Add build-dir setting

(cherry picked from commit 8b16cced18925aa612049d08d5e78eccbf0530e4)
Change-Id: Ic7b75ff0b6a3b19e50a4ac8ff2d70f15c683c16a
2024-06-24 11:30:32 +03:00
Delan Azabani 4c3d93611f Merge "Fix build instructions in hacking.md and justfile" into main 2024-06-24 06:46:08 +00:00
Delan Azabani 37f276ba9d Merge "Fix compile error under gcc with -Denable-pch-std=false" into main 2024-06-24 06:45:40 +00:00
Delan Azabani 5d33e4cd59 Fix build instructions in hacking.md and justfile
The stdenv phases don’t actually do anything (at least not anymore),
and our justfile doesn’t behave the same as our docs.

This patch removes the stdenv phases from the docs, documents our
usage of just, and makes `just setup` heed `$mesonFlags`.

Fixes #413.
Fixes #414.

Change-Id: Ieb0b2a8ae420526238b5f9a73d7849ec6919995d
2024-06-24 12:55:00 +08:00
Delan Azabani 3baffbdcc5 Fix compile error under gcc with -Denable-pch-std=false
Following the latest hacking.md currently fails because of a missing
include in upstream editline. This patch fixes the build by adding
that missing include.

Fixes #410.

Change-Id: Iefd4cb687ed3da71ccda9fe9624f38e6ef4623e5
2024-06-24 12:47:04 +08:00
eldritch horrors d477b34d1d libutil: remove runProgram2 stdin functionality
this was only used in one place, and that place has been rewritten to
use a temporary file instead. keeping this around is not very helpful
at this time, and in any case we'd be better off rewriting subprocess
handling in rust where we not only have a much safer library for such
things but also async frameworks necessary for this easily available.

Change-Id: I6f8641b756857c84ae2602cdf41f74ee7a1fda02
2024-06-23 17:29:40 +00:00
eldritch horrors 2bbdaf0b19 libfetchers: write git commit message to tempfile
we want to remove runProgram's ability to provide stdin to a process
because the concurrency issues of handling both stdin and stdout are
much more pronounced once runProgram returns not is collected output
but a source. this is possible in the current c++ framework, however
it isn't necessary in almost all cases (as demonstrated by only this
single user existing) and in much better handled with a proper async
concurrency model that lets the caller handle both at the same time.

Change-Id: I29da1e1ad898d45e2e33a7320b246d5003e7712b
2024-06-23 17:29:40 +00:00
eldritch horrors 2fe9157808 flakes: add --commit-lock-file message test
we had no test to ensure that we generated a commit message at all?

Change-Id: Ic9aa8fde92b83e1ea6f61cd2a21867aa73d4e885
2024-06-23 17:29:40 +00:00
Qyriad 521e08cbde Merge changes I697f4f39,I9f25235d into main
* changes:
  doc-comment Fields for Activity and Result types
  mildly refactor the renderActivity if hell-chain
2024-06-23 16:55:36 +00:00
Maximilian Bosch 7c3b8229cc Merge "libstore: reduce loglevel of waiting for a machine to build" into main 2024-06-23 16:33:44 +00:00
Qyriad 701eb332bd doc-comment Fields for Activity and Result types
The ones we were able to figure out, at least.

Change-Id: I697f4f3942e35a7adf1b2a6cc28b3168d1de111c
2024-06-23 16:30:48 +00:00
Qyriad ce4dee0fa5 mildly refactor the renderActivity if hell-chain
This is primarily for readability, but iwrc chaining std::string's
operator+ is also pretty scuffed performance-wise, and this was doing a
lot of operator+ chainging.

Change-Id: I9f25235df153eb2bbb491f1ff7ebbeed9a8ec985
2024-06-23 16:30:38 +00:00
Maximilian Bosch 5f0062285c Merge "libfetchers: make attribute / URL query handling consistent" into main 2024-06-23 15:51:34 +00:00
eldritch horrors ce6cb14995 libutil: return Pid from startProcess, not pid_t
Change-Id: Icc8a15090c77f54ea7d9220aadedcd4a19922814
2024-06-23 11:52:49 +00:00
eldritch horrors 3d155fc509 libutil: give Pid proper resource semantics
copy-constructing or assigning from pid_t can easily lead to duplicate
Pid instances for the same process if a pid_t was used carelessly, and
Pid itself was copy-constructible. both could cause surprising results
such as killing processes twice (which could become very problemantic,
but luckily modern systems don't reuse PIDs all that quickly), or more
than one piece of the code believing it owns a process when neither do

Change-Id: Ifea7445f84200b34c1a1d0acc2cdffe0f01e20c6
2024-06-23 11:52:49 +00:00
eldritch horrors b43a2e84c4 libutil: make Pid -> pid_t operations explicit
Change-Id: I3137cc140590001fe7ba542844e735944a0a9255
2024-06-23 11:52:49 +00:00
eldritch horrors a9949f4760 libutil: add some serialize.hh serializer tests
Change-Id: I0116265a18bc44bba16c07bf419af70d5195f07d
2024-06-23 11:52:49 +00:00
eldritch horrors 39a1e248c9 libutil: remove sinkToSource eof callback
this is only used in one place, and only to set a nicer error message on
EndOfFile. the only caller that actually *catches* this exception should
provide an error message in that catch block rather than forcing support
for setting error message so deep into the stack. copyStorePath is never
called outside of PathSubstitutionGoal anyway, which catches everything.

Change-Id: Ifbae8706d781c388737706faf4c8a8b7917ca278
2024-06-23 11:52:49 +00:00
Artemis Tosini f80d95e36d Merge "libstore: Start creating LocalDerivationGoal subclasses" into main 2024-06-23 08:11:51 +00:00
Artemis Tosini 12f5d27363 libstore: Start creating LocalDerivationGoal subclasses
LocalDerivationGoal includes a large number of low-level sandboxing
primitives for Darwin and Linux, intermingled with ifdefs.
Start creating platform-specific classes to make it easier to add new
platforms and review platform-specific code.

This change only creates support infrastructure and moves two function,
more functions will be moved in future changes.

Change-Id: I9fc29fa2a7345107d4fc96c46fa90b4eabf6bb89
2024-06-23 03:33:07 +00:00
Qyriad fb8553f63c mildly cleanup ExprSelect::eval
Better variable names, some comments, and a slight logic rearrange.

Change-Id: I9685ae252f83217aa85f06432234159c9ad19d1c
2024-06-22 18:52:57 -06:00
Qyriad e09cc60df9 doc-comment ExprSelect's fields
Change-Id: I63e79991a4bab93421266785e9258e0f5bb89b8f
2024-06-22 18:52:57 -06:00
Maximilian Bosch fc6a1451af
libstore: reduce loglevel of waiting for a machine to build
This comes quite often when the available job slots on all remote
builders are exhausted and this is pretty spammy.

This isn't really an issue, but expected behavior.

A better way to display this is a nom-like approach where all scheduled
builds are shown in a tree and pending builds are being marked as such
IMHO.

Change-Id: I6bc14e6054f84e3eb0768127b490e263d8cdcf89
2024-06-22 17:38:25 +02:00
Maximilian Bosch 35eec921af
libfetchers: make attribute / URL query handling consistent
The original idea was to fix lix#174, but for a user friendly solution,
I figured that we'd need more consistency:

* Invalid query params will cause an error, just like invalid
  attributes. This has the following two consequences:

  * The `?dir=`-param from flakes will be removed before the URL to be
    fetched is passed to libfetchers.

  * The tarball fetcher doesn't allow URLs with custom query params
    anymore. I think this was questionable anyways given that an
    arbitrary set of query params was silently removed from the URL you
    wanted to fetch. The correct way is to use an attribute-set
    with a key `url` that contains the tarball URL to fetch.

  * Same for the git & mercurial fetchers: in that case it doesn't even
    matter though: both fetchers added unused query params to the URL
    that's passed from the input scheme to the fetcher (`url2` in the code).
    It turns out that this was never used since the query parameters were
    erased again in `getActualUrl`.

* Validation happens for both attributes and URLs. Previously, a lot of
  fetchers validated e.g. refs/revs only when specified in a URL and
  the validity of attribute names only in `inputFromAttrs`.

  Now, all the validation is done in `inputFromAttrs` and `inputFromURL`
  constructs attributes that will be passed to `inputFromAttrs`.

* Accept all attributes as URL query parameters. That also includes
  lesser used ones such as `narHash`.

  And "output" attributes like `lastModified`: these could be declared
  already when declaring inputs as attribute rather than URL. Now the
  behavior is at least consistent.

  Personally, I think we should differentiate in the future between
  "fetched input" (basically the attr-set that ends up in the lock-file)
  and "unfetched input" earlier: both inputFrom{Attrs,URL} entrypoints
  are probably OK for unfetched inputs, but for locked/fetched inputs
  a custom entrypoint should be used. Then, the current entrypoints
  wouldn't have to allow these attributes anymore.

Change-Id: I1be1992249f7af8287cfc37891ab505ddaa2e8cd
2024-06-22 14:42:43 +02:00
kloenk da4e46dd1f libmain: add progress bar with multiple status lines
Add the log-formats `multiline` and `multiline-with-logs` which offer
multiple current active building status lines.

Change-Id: Idd8afe62f8591b5d8b70e258c5cefa09be4cab03
2024-06-22 14:20:27 +02:00
Qyriad 21865ccce0 Merge "add a basic libmain test for the progress bar rendering" into main 2024-06-22 02:04:14 +00:00
jade 375f4c0337 Merge "libstore: remove operations that are never called by supported clients" into main 2024-06-21 20:46:18 +00:00
Qyriad fd250c51ed add a basic libmain test for the progress bar rendering
Hooray for leaky abstraction allowing us to test this particular part of
the render pipeline.

Change-Id: Ie0f251ff874f63324e6a9c6388b84ec6507eeae2
2024-06-20 13:56:53 -06:00
Qyriad e44dcd63c4 remove InstallableValueCommand class
Change-Id: Id12383f4741cba07159712700ebcfbe37e61560c
2024-06-20 17:00:06 +00:00
Qyriad 6515b1a495 de-inheritance CmdSearch for InstallableValueCommand
Change-Id: I125c8cac05c8e924b55e4eb1060496e35ea4e941
2024-06-20 17:00:06 +00:00
Qyriad 50be55ffca de-inheritance CmdEdit for InstallableValueCommand
Change-Id: If85ea78954a45470b0b25c08dc7d40bfebd53610
2024-06-20 17:00:06 +00:00
Qyriad 079eeb1de7 de-inheritance CmdRun for InstallableValueCommand
Change-Id: Ief858c1488197211e2ee8b70aa40ed6c65743558
2024-06-20 17:00:06 +00:00
Qyriad b9e9235ac0 de-inheritance CmdBundle for InstallableValueCommand
Change-Id: Icbac4ef927ddcaf0d2a74b376e5a77299529cd34
2024-06-20 17:00:06 +00:00
Qyriad 1e5f134560 de-inheritance CmdEval for InstallableValueCommand
Change-Id: I08b1702310e863d15de26dc838eb0bcb62417c10
2024-06-20 17:00:06 +00:00
Qyriad 8ba1939540 use a type alias for ProgressBar's chosen time point type
Change-Id: I621a455b1daba806fc498958aee7931fbfc55445
2024-06-20 15:24:27 +00:00
Qyriad f9594b592b extract ProgressBar declaration into its header file
Change-Id: Ica9e2ec41d99eaa196a0d535501edf45c589b2b6
2024-06-20 15:24:27 +00:00
Qyriad 3a4c21fc9e slight cleanup to ProgressBar::getStatus()
Binaries were identical before and after this commit on our machine

Change-Id: I6f8bfbe3298d6c5f43d5702c7a1e05cb180226cc
2024-06-20 15:24:27 +00:00
Ilya K 697ef65c14 Merge "BrotliDecompressionSource: don't bail out too early" into main 2024-06-20 07:06:52 +00:00
Ilya K 7d52d74bbe BrotliDecompressionSource: don't bail out too early
If we've consumed the entire input, that doesn't actually mean we're
done decompressing - there might be more output left. This worked (?)
in most cases because the input and output sizes are pretty comparable,
but sometimes they're not and then things get very funny.

Change-Id: I73435a654a911b8ce25119f713b80706c5783c1b
2024-06-20 09:21:13 +03:00
jade 6c29a2a6fc Merge "libstore: fix queryValidPaths concurrency" into main 2024-06-20 05:55:08 +00:00
jade 50472aa5be libstore: remove operations that are never called by supported clients
I did a whole bunch of `git log -S` to find out exactly when all these
things were obsoleted and found the commit in which their usage was
removed, which I have added in either the error message or a comment.

I've also made *some* of the version checks into static asserts for when
we update the minimum supported protocol version.

In the end this is not a lot of code we are deleting, but it's code that
we will never have to support into the future when we build a protocol
bridge, which is why I did it. It is not in the support baseline.

Change-Id: Iea3c80795c75ea74f328cf7ede7cbedf8c41926b
2024-06-19 19:41:04 -07:00
Eelco Dolstra fb7d315411 Merge pull request #10570 from layus/shared_caches
Share evaluation caches across installables

Before:

$ rm -rf ~/.cache/nix && time -f '%E' nix build --dry-run \
  'nixpkgs#hello' \
  'nixpkgs#clang' \
  'nixpkgs#cargo' \
  'nixpkgs#rustup' \
  'nixpkgs#bear' \
  'nixpkgs#firefox' \
  'nixpkgs#git-revise' \
  'nixpkgs#hyperfine' \
  'nixpkgs#curlie' \
  'nixpkgs#xz' \
  'nixpkgs#ripgrep'
0:03.61

After:

$ rm -rf ~/.cache/nix && time -f '%E' nix build --dry-run \
  'nixpkgs#hello' \
  'nixpkgs#clang' \
  'nixpkgs#cargo' \
  'nixpkgs#rustup' \
  'nixpkgs#bear' \
  'nixpkgs#firefox' \
  'nixpkgs#git-revise' \
  'nixpkgs#hyperfine' \
  'nixpkgs#curlie' \
  'nixpkgs#xz' \
  'nixpkgs#ripgrep'
0:01.46

This could probably use a more proper benchmark...

Fixes #313

(cherry picked from commit de51e5c335865e3e0a8cccd283fec1a52cce243f)
Change-Id: I9350bebd462b6af12c51db5bf432321abfe84a16
2024-06-19 18:39:11 +00:00
eldritch horrors c55dcc6c13 filetransfer: return a Source from download()
without this we will not be able to get rid of makeDecompressionSink,
which in turn will be necessary to get rid of sourceToSink (since the
libarchive archive wrapper *must* be a Source due to api limitations)

Change-Id: Iccd3d333ba2cbcab49cb5a1d3125624de16bce27
2024-06-19 10:50:12 +00:00
eldritch horrors 11f4a5bc7e libutil: return a source from readFile
don't consume a sink, return a source instead. the only reason to not do
this is a very slight reduction in dynamic allocations, but since we are
going to *at least* do disk io that will not be a lot of overhead anyway

Change-Id: Iae2f879ec64c3c3ac1d5310eeb6a85e696d4614a
2024-06-19 10:50:12 +00:00
eldritch horrors 67f778670c libutil: add makeDecompressionSource
Change-Id: Iac7f24d79e24417436b9b5cbefd6af051aeea0a6
2024-06-19 10:50:12 +00:00
eldritch horrors 3425e90d76 libstore: BinaryCacheStore::getFile{ -> Contents}
if we want have getFile return a source instead of consuming a sink
we'll have to disambiguate this overload another way, eg like this.

Change-Id: Ia26de2020c309a37e7ccc3775c1ad1f32e0a778b
2024-06-19 10:50:12 +00:00
alois31 fed34594d8 Merge "libfetchers: represent unfetched submodules consistently" into main 2024-06-19 07:08:19 +00:00
jade 85f282ef57 Merge changes Id0e651e4,I0ed20da8,I76bd6d22,I5d8ffb7b into main
* changes:
  store: fix null reference from DerivationGoal::waiteeDone
  libmain: fix UB in verbosity assignment
  build: make UBSan work :)
  libexpr: fix accessing uninitialized values and fix pure-eval docs
2024-06-19 03:31:47 +00:00
Qyriad b338435b75 Merge "refactor lambda formals handling" into main 2024-06-19 03:13:46 +00:00
jade c897fba787 store: fix null reference from DerivationGoal::waiteeDone
This happened during a PathSubstitutionGoal of a .drv file:

substitution of '/tmp/jade/nix-test/ca/eval-store/store/1lj7lsq5y0f25mfbnq6d3zd0bw5ay33n-dependencies-input-2.drv'

What happened here is that since PathSubstitutionGoal is not a
DerivationGoal, in production builds, the UB was not caught, since it
would early-exit from failing a dynamic_cast to DerivationGoal * on the
very next line, but before the null reference was ever used.

This was nonetheless UB. The fix should be to just rearrange the two
lines; I don't think there is a further bug there, since *substituting a
.drv* **necessarily** means you cannot have the representation of
the derivation as would be necessary for drv to not be null there.

Test failure:

++(eval-store.sh:12) _RR_TRACE_DIR=/home/jade/.local/share/rr rr record -- nix build -f dependencies.nix --eval-store /tmp/jade/nix-test/ca/eval-store/eval-store -o /tmp/jade/nix-test/ca/eval-store/result
don't know how to build these paths:
  /tmp/jade/nix-test/ca/eval-store/store/6y51mf0p57ggipgab6hdjabbvplzsicq-dependencies-top.drv
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/8027afyvqb87y1sf5xhdkqsflqn1ziy8-dependencies.builder0.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/7r5pqyncvfgrryf9gzy1z56z3xigi61x-builder-dependencies-input-0.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/nhmgm87zlqy3ks96dxrn7l37b72azi99-builder-dependencies-input-1.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/nq4qa2j6y8ajqazlfq6h46ck637my1n6-builder-dependencies-input-2.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/6vh0vna9l5afck01y7iaks3hm9ikwqyj-builder-fod-input.sh' to 'local'...
building '/tmp/jade/nix-test/ca/eval-store/store/gy91pqymf2nc5v7ld1bad94xpwxdi25s-dependencies-input-0.drv'...
building '/tmp/jade/nix-test/ca/eval-store/store/w7wlkjx97ivmnrymkac5av3nyp94hzvq-dependencies-input-1.drv'...
../src/libstore/build/derivation-goal.cc:1556:22: runtime error: reference binding to null pointer of type 'Derivation'
    0 0x734ba59a6886 in nix::DerivationGoal::waiteeDone(std::shared_ptr<nix::Goal>, nix::Goal::ExitCode) /home/jade/lix/lix2/build/src/libstore/build/derivation-goal.cc:1556:12
    1 0x734ba59c0962 in nix::Goal::amDone(nix::Goal::ExitCode, std::optional<nix::Error>) /home/jade/lix/lix2/build/src/libstore/build/goal.cc:95:25
    2 0x734ba5a1c44a in nix::PathSubstitutionGoal::done(nix::Goal::ExitCode, nix::BuildResult::Status, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>) /home/jade/lix/lix2/build/src/libstore/build/substitution-goal.cc:38:5
    3 0x734ba5a1b454 in nix::PathSubstitutionGoal::init() /home/jade/lix/lix2/build/src/libstore/build/substitution-goal.cc:56:9
    4 0x734ba5a2a6c6 in nix::Worker::run(std::set<std::shared_ptr<nix::Goal>, nix::CompareGoalPtrs, std::allocator<std::shared_ptr<nix::Goal>>> const&) /home/jade/lix/lix2/build/src/libstore/build/worker.cc:320:23
    5 0x734ba59b93d8 in nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath>> const&, nix::BuildMode, std::shared_ptr<nix::Store>) /home/jade/lix/lix2/build/src/libstore/build/entry-points.cc:60:12
    6 0x734ba663c107 in nix::Installable::build2(nix::ref<nix::Store>, nix::ref<nix::Store>, nix::Realise, std::vector<nix::ref<nix::Installable>, std::allocator<nix::ref<nix::Installable>>> const&, nix::BuildMode) /home/jade/lix/lix2/build/src/libcmd/installables.cc:637:36

Change-Id: Id0e651e480bebf6356733b01bc639e9bb59c7bd0
2024-06-18 19:26:54 -07:00
jade f2fff1faa4 libmain: fix UB in verbosity assignment
This was generating an out-of-range verbosity value. We should just
process it as an int and then convert to verbosity with a clamping
function, which trivially avoids any domain type violations.

Change-Id: I0ed20da8e1496a1225ff3008b76827d99265d404
2024-06-18 19:26:54 -07:00
jade e0a3a5f226 build: make UBSan work :)
This is really just a question of turning off the production sanitizer
configuration so we get nice diagnostics. Not much else to say.

Change-Id: I76bd6d225320056ed95bd89955f00beff2db0d2f
2024-06-18 19:25:58 -07:00
jade 1eef1927b6 libexpr: fix accessing uninitialized values and fix pure-eval docs
We got UBSan working on Lix, so we of course immediately found a bug and
some definitely nonsense behaviour.

Accessing `pureEval` or `restrictEval` from a default setting value is
nonsense, since they would never be actually set by the time that value
is set so they are not going to do anything. The configuration is not
applied in an initializer (and even if it were, it's not going to be in
the right order).

After looking into *that*, we hunted down what actually was applying
these, since clearly this code did not do anything. The EvalState
constructor should have a "search path added and removed here :)" sign
on it, because that's where it is done. We added an explicit
initialization of the optional in there because it was otherwise unclear
why pureEval also has the search path to allowed paths setup code run.

We then realized that the `pureEval` documentation was *also* bogus, and
we rewrote it. In so doing, we realized that we forgot to file a bug to
make `builtins.storePath` work in pure eval mode, so we filed one of
those: lix-project/lix#402

Yaks have been thoroughly shorn.

UBSan report:

    ../src/libexpr/eval-settings.cc:66:10: runtime error: member call on address 0x752fa9a13060 which does not point to an object of type 'nix::BaseSetting<b
    ool>'
    0x752fa9a13060: note: object has invalid vptr
     00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00
                  ^~~~~~~~~~~~~~~~~~~~~~~
                  invalid vptr
        0 0x752fa95106a6 in nix::EvalSettings::getDefaultNixPath[abi:cxx11]() /home/jade/lix/lix2/build/src/libexpr/eval-settings.cc:66:10
        1 0x752fa950e420 in nix::EvalSettings::EvalSettings() /home/jade/lix/lix2/build/src/libexpr/eval-settings.hh:36:15
        2 0x752fa9469f1f in __cxx_global_var_init.50 /home/jade/lix/lix2/build/src/libexpr/eval-settings.cc:98:14
        3 0x752fa9469f1f in _GLOBAL__sub_I_eval_settings.cc /home/jade/lix/lix2/build/src/libexpr/eval-settings.cc
        4 0x752fabbd308d in call_init (/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2+0x508d) (BuildId: a5b8228edc9f16078ac3c894af964eeb990ecb4c)
        5 0x752fabbd317b in _dl_init (/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2+0x517b) (BuildId: a5b8228edc9f16078ac3c894af964eeb990ecb4c)
        6 0x752fabbe9c2f in _dl_start_user (/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2+0x1bc2f) (BuildId: a5b8228edc9f16078ac3c894af964eeb990ecb4c)

Change-Id: I5d8ffb7bfbe24b6584020ac74eed93d9f2e6d111
2024-06-18 19:25:35 -07:00
Qyriad 54d2c189ae refactor lambda formals handling
Change-Id: Iebffd5436109da270ee870670a20f5ee7db9a204
2024-06-19 00:57:56 +00:00
eldritch horrors a960576f58 libutil: BrotliDecompression{Sink -> Source}
Change-Id: I9579dd08f7bd0f927bde9d3128515b0cee15f320
2024-06-19 00:54:06 +00:00
eldritch horrors 0b9a72524a filetransfer: {up,down}load -> transfer
even the transfer function is not all that necessary since there aren't
that many users, but we'll keep it for now. we could've kept both names
but we also kind of want to use `download` for something else very soon

Change-Id: I005e403ee59de433e139e37aa2045c26a523ccbf
2024-06-18 23:58:25 +00:00
jade 66a9fbb7ff libstore: fix queryValidPaths concurrency
The lock usage was obviously wrong so it was entirely serialized. This
has the predicted speedups, the only question is whether it is sound
because it's exposing a bunch of new code to actual concurrency.

I did audit all the stores' queryPathInfoUncached implementations and
they all look *intended* to be thread safe, but whether that is actually
sound or not: lol lmao. I am highly confident in the s3 one because it
is calling s3 sdk methods that are thread safe and has no actual state.

Others are using Pool and look to be *supposed* to be thread safe, but
unsure if they actually are.

Change-Id: I0369152a510e878b5ac56c9ac956a98d48cd5fef
2024-06-18 23:29:08 +00:00
jade b9b1bbd22f diff-closures: fix a use after free
Found by looking for interesting asan reports from the test suite.

What happened here is that name got overwritten, but it was what
actually held the backing memory for the thing it got overwritten by,
which was a by-reference value coming out of std::regex.

Due to absurd reasons I cannot seem to use a string_view iterator here,
so I just copy the string with a longer lifetime instead. idk lol

==3796364==ERROR: AddressSanitizer: heap-use-after-free on address 0x503000014c61 at pc 0x74843523bf1d bp 0x7ffc68351330 sp 0x7ffc68350af0
READ of size 3 at 0x503000014c61 thread T0
    0 0x74843523bf1c in __asan_memcpy (/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/resource-root/lib/linux/libclang_rt.asan-x86_64.so+0x159f1c)
    1 0x6403cf6cbff4 in std::char_traits<char>::copy(char*, char const*, unsigned long) /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/char_traits.h:445:33
    <...>
    7 0x6403cf6cbff4 in std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::str() const /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/regex.h:966:6
    8 0x6403cf6cbff4 in std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>() const /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/regex.h:955:16
    9 0x6403cf6cbff4 in nix::getClosureInfo[abi:cxx11](nix::ref<nix::Store>, nix::StorePath const&) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:37:26
    10 0x6403cf6cd70c in nix::printClosureDiff(nix::ref<nix::Store>, nix::StorePath const&, nix::StorePath const&, std::basic_string_view<char, std::char_traits<char>>) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:54:25
    11 0x6403cf873331 in CmdProfileDiffClosures::run(nix::ref<nix::Store>) /home/jade/lix/lix2/build/src/nix/profile.cc:479:17
    <...>

0x503000014c61 is located 17 bytes inside of 21-byte region [0x503000014c50,0x503000014c65)
freed by thread T0 here:
    0 0x748435250470 in operator delete(void*) (/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/resource-root/lib/linux/libclang_rt.asan-x86_64.so+0x16e470)
    <...>
    6 0x6403cf6cbda2 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::~basic_string() /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/basic_string.h:792:9
    7 0x6403cf6cbda2 in nix::getClosureInfo[abi:cxx11](nix::ref<nix::Store>, nix::StorePath const&) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:36:13
    8 0x6403cf6cd70c in nix::printClosureDiff(nix::ref<nix::Store>, nix::StorePath const&, nix::StorePath const&, std::basic_string_view<char, std::char_traits<char>>) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:54:25
    <...>

previously allocated by thread T0 here:
    0 0x74843524fa38 in operator new(unsigned long) (/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/resource-root/lib/linux/libclang_rt.asan-x86_64.so+0x16da38)
    <...>
    9 0x6403cf6cb68c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::basic_string<std::basic_string_view<char, std::char_traits<char>>, void>(std::basic_string_view<char, std::char_traits<char>> const&, std::allocator<char> const&) /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/basic_string.h:784:4
    10 0x6403cf6cb68c in nix::getClosureInfo[abi:cxx11](nix::ref<nix::Store>, nix::StorePath const&) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:33:21
    11 0x6403cf6cd70c in nix::printClosureDiff(nix::ref<nix::Store>, nix::StorePath const&, nix::StorePath const&, std::basic_string_view<char, std::char_traits<char>>) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:54:25
    12 0x6403cf873331 in CmdProfileDiffClosures::run(nix::ref<nix::Store>) /home/jade/lix/lix2/build/src/nix/profile.cc:479:17
    <...>

Change-Id: I9c408cf2a3d3155f9f7b2ad4848ee6c741331db0
2024-06-18 15:11:49 -07:00
jade 8e6661cce7 store-api: fix/clarify capture lifetimes in copyPaths
This seems to fix a use of stack after return.

Change-Id: If690a6defb9a3225684685132cf78b227e271447
2024-06-18 15:11:49 -07:00
jade 9185ab7bf0 libstore: work around aws sdk log spam at debug level
aws-sdk-cpp spams logs about sending TLS data in the otherwise rather
helpful debug logs. I've filed a PR upstream to stop it, but for now we
can just fix their verbosity ourselves.

Upstream-PR: https://github.com/aws/aws-sdk-cpp/pull/3003
Change-Id: I0c41a50d5f5958106836d6345843f4b05b9c8981
2024-06-18 15:11:49 -07:00
jade 02ca60809d s3: delete obsolete ifdefs
The versions checked for are so old that we can just drop support.

Change-Id: Ib9cf136d1cb9a4a91a6613102c4fd15e1190363b
2024-06-18 15:11:49 -07:00
jade 3626738b9b libutil: tidy Sync and fix its move constructor
There was a previously-unused move constructor that just called abort,
which makes no sense since it ought to just be `= delete` if you don't
want one (commit history says it was Eelco doing an optimistic
performance optimisation in 2016, so it probably would not pass review
today).

However, a Lock has some great reasons to be moved! You might need to
unlock it early, for instance, or give it to someone else. So we change
the move constructor to instead hollow out the moved-from object and
make it unusable.

Change-Id: Iff2a4c2f7ebd0a558c4866d4dfe526bc8558bed7
2024-06-18 15:11:31 -07:00
alois31 aa00a5a8c9 libfetchers: represent unfetched submodules consistently
Unfetched submodules are included as empty directories in archives, so they end
up as such in the store when fetched in clean mode. Make sure the same happens
in dirty mode too. Fortunately, they are already correctly represented in the
ls-files output, so we just need to make sure to include the empty directory in
our filter.

Fixes: https://github.com/NixOS/nix/issues/6247
Change-Id: I60d06ff360cfa305d081b920838c893c06da801c
2024-06-18 00:54:51 +00:00
jade ce2b48aa41 Merge changes from topic "protocol" into main
* changes:
  libstore client: remove remaining dead code
  libstore: refuse to serialise ancient protocols
  libstore client: remove support for <2.3 clients
  libstore daemon: remove very old protocol support (<2.3)
  Delete old ValidPathInfo test, fix UnkeyedValidPathInfo
  Set up minimum protocol version
2024-06-17 22:08:48 +00:00
eldritch horrors bcb774688f libexpr: add expr memory management
with the prepatory work done this mostly means turning plain pointers
into unique_ptrs, with all the associated churn that necessitates. we
might want to change some of these to box_ptrs at some point as well,
but that would be a semantic change that isn't fully appropriate yet.

Change-Id: I0c238c118617420650432f4ed45569baa3e3f413
2024-06-17 19:46:44 +00:00
eldritch horrors ad5366c2ad libexpr: pass Exprs as references, not pointers
almost all places where Exprs are passed as pointers expect the pointers
to be non-null. pass them as references to encode this constraint in the
type system as well (and also communicate that Exprs must not be freed).

Change-Id: Ia98f166fec3c23151f906e13acb4a0954a5980a2
2024-06-17 19:46:44 +00:00
eldritch horrors b8f49a8eaf libexpr: store ExprConcatStrings elements as direct vector
storing a pointer only adds an unnecessary indirection at runtime.

Change-Id: If06dd05effdf1ccb0df0873580f50c775608925d
2024-06-17 19:46:44 +00:00
eldritch horrors dad8bc679e libexpr: don't immediately throw parser errors
now that destructors are hooked up we want to give the C skeleton every
real chance to actually run them. since bison does not call destructors
on values that have been passed to semantic actions even when an action
causes an abort we will also have to delete some things manually still.

Change-Id: Ia22bdaa9e969b74e17a6c496e35e6c2d86b7d750
2024-06-17 19:46:44 +00:00
eldritch horrors 9592a9fd57 libexpr: hook up bison destructors for state objects
this doesn't help much yet since the state objects themselves also leak
all memory they are given, but it is a first necessary step to properly
managing parser memory. notably we have to clear $$ when returning from
the parser since even the start symbol is subject to automatic deletion
by the bison-generated parser before returning control to the call site

Change-Id: I80245b0c747308e80923e7f18ce4e1a4898f93b0
2024-06-17 19:46:44 +00:00
Qyriad 19a93dd025 mini-refactor "lambda.name or anonymous lambda" logic
Change-Id: I08d39c4ad5b967de526c0d5c5e6299256c7967f3
2024-06-17 15:13:18 +00:00
Qyriad 010d93393e repl: implement tab completing :colon commands
This uses a minor hack in which we check the rl_line_buffer global
variable to workaround editline not including the colon in its
completion callback.

Fixes #361

Change-Id: Id159d209c537443ef5e37a975982e8e12ce1f486
2024-06-17 13:08:02 +00:00
jade c1f2733dd6 libstore client: remove remaining dead code
Change-Id: I1764b3878439ff7b20ff64bd4efcf03070bb0e5e
2024-06-16 19:15:08 -07:00
jade c22a7f50cb libstore: refuse to serialise ancient protocols
We don't want to deal with these at all, let's stop doing so.

(marking this one as the fix commit since its immediate predecessors
aren't the complete fix)
Fixes: lix-project/lix#325

Change-Id: Ieea1b0b8ac0f903d1e24e5b3e63cfe12eeec119d
2024-06-16 19:15:08 -07:00
jade 985ce8a865 libstore client: remove support for <2.3 clients
Change-Id: I71c2e8ca644b6187e0084f35e82f3316c9d425b0
2024-06-16 19:15:06 -07:00
jade 7b1d38bc4f libstore daemon: remove very old protocol support (<2.3)
Change-Id: Ic05f478a659c199a66fe78ae05d357d317ac41b0
2024-06-16 19:13:51 -07:00
jade 24255748b4 Delete old ValidPathInfo test, fix UnkeyedValidPathInfo
The UnkeyedValidPathInfo test was testing an ancient version but not the
current version. Doesn't make much sense to me.

Change-Id: Ib476a4297d9075f2dcd31a073b3e7b149b2189af
2024-06-16 19:13:51 -07:00
jade a17282fc66 Set up minimum protocol version
Change-Id: Ibb931109a8328cfb22964542ab53644cc4181f9e
2024-06-16 19:13:51 -07:00
jade 6aead00a01 Merge "Merge pull request #10799 from hercules-ci/safer-tab-completion" into main 2024-06-17 02:12:15 +00:00
jade 6c541e0bef Merge changes I9cf007c8,I9b9ba058 into main
* changes:
  releng: fix broken manifest from 2.90-rc1
  Change the sqlite missing valid path message to say it's the db
2024-06-17 02:06:04 +00:00
Qyriad 0ba37dc00d Merge "build: correctly propagate changes to Lix source to manual" into main 2024-06-16 22:10:59 +00:00
jade 4e02951335 releng: fix broken manifest from 2.90-rc1
Well that is embarrassing. I think the proper thing is to just quickly
ship a -rc2, so I will open a backport of this.

    $ curl https://releases.lix.systems/manifest.nix
    # This file was generated by releng/create_release.xsh in Lix
    {
        aarch64-linux = "/nix/store/mrbknq000af7iaqhk53bnpk1fvfrc1xp-lix-2.90.0-rc1";
      aarch64-darwin = "/nix/store/z1bdccwsk34iv491aygh0mm1lgpf7yy1-lix-2.90.0-rc1";
      x86_64-darwin = "/nix/store/xqvfpdhzck44v6kyhgi9f8v0xybksb6a-lix-2.90.0-rc1";
      x86_64-linux = "/nix/store/h2ml0nx4477r84y82jgm8y80jpr72gqw-lix-release-tarballs";
    }

Change-Id: I9cf007c850c2faf995a3a9d92457517b8501d1a1
2024-06-16 13:34:04 -07:00
Artemis Tosini ce2070139c Merge changes I81e76796,Iba319126 into main
* changes:
  libutil: Set boost defines for FreeBSD
  meson.build: Allow undefined symbols on FreeBSD
2024-06-16 14:42:54 +00:00
jade 4004d12483 Change the sqlite missing valid path message to say it's the db
I meant to edit https://gerrit.lix.systems/c/lix/+/1161 but apparently
clicked the wrong button somehow. Oops.

Change-Id: I9b9ba058ec9206d3c8abe125d91dc554cced52fe
2024-06-15 22:31:23 -07:00
julia dd70044cde Merge changes I07d2da41,I864d7340,I86612c64 into main
* changes:
  Change error messages about 'invalid paths' to 'path does not exist'.
  Add a clearer error message for InvalidPathError during evaluation
  Harmonise the Store::queryPathInfoUncached interface
2024-06-16 04:29:13 +00:00
jade b4035ed1d1 Merge "docs: expand importNative/exec example (#10803)" into main 2024-06-16 04:04:20 +00:00
Robert Hensing b588a761fe Merge pull request #10799 from hercules-ci/safer-tab-completion
Add repl completion test

(cherry picked from commit 1e2b26734b4da101247678aec405c9dcfdc33f98)

Change-Id: Ic3de39e71960a05a8676190b1ec9a7f0bb6057f5
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
2024-06-16 04:03:32 +00:00
julia 89c782b0c0 Change error messages about 'invalid paths' to 'path does not exist'.
Fixes #270.

Change-Id: I07d2da41498cfdf324a03af40533044d58c97c7e
2024-06-16 03:55:39 +00:00
julia 6c311a4afa Add a clearer error message for InvalidPathError during evaluation
Part of #270, #271

Change-Id: I864d7340f26d3c0f9c45db7b6b545face38d8294
2024-06-16 03:53:00 +00:00
julia 0fa289f559 Harmonise the Store::queryPathInfoUncached interface
This:
 - Consistently returns `nullptr` for a non-existent
   store path, instead of a mix of `nullptr` and
   throwing exceptions.

 - If a store returns "bad" store paths in response
   to a request (e.g. incorrect hash or name), don't
   cache this result. This removes some duplication
   of code at the cache-access layer of queryPathInfo()
   checking this, and ­allows us to provide more
   specific errors.

Part of #270.

Change-Id: I86612c6499b1a37ab872c712c2304d6a3ff19edb
2024-06-16 03:53:00 +00:00
Qyriad ff99f4a882 build: correctly propagate changes to Lix source to manual
This adds the nix executable as an input argument to the custom targets
that build the manual (HTML, and the man pages), so that changes to
things like src/nix/nix.md cause the correct rebuilds

Change-Id: Iffaa2c14acbfc721caef7b8cae9b53ecd365f26d
2024-06-16 03:43:17 +00:00
Artemis Tosini f70b4258cd libutil: Set boost defines for FreeBSD
FreeBSD uses libunwind unwind.h, which does not require
`_GNU_SOURCE` to expose `_Unwind_Backtrace`.

Tell Boost that.

Change-Id: I81e767967b1458118b86d212b5552d4d0a1200d9
2024-06-16 03:41:15 +00:00
Artemis Tosini e680b0913a meson.build: Allow undefined symbols on FreeBSD
The linker cannot find environ while linking libutil.
Fix build by allowing undefined symbols

Change-Id: Iba319126284d48454b50db40d3aadf37e0339668
2024-06-16 03:41:15 +00:00
jade 4734ce7831 version: update to 2.91-dev for main
This commit constitutes the branch-off for 2.91. The parent of this
commit will be the branch point for release-2.90.

Change-Id: I7f047545df29a9cff93346137c865dcbf1415488
2024-06-15 18:46:18 -07:00
jade 79404f7ffc releng: automatically add to the summary page
Also delete the obsolescent maintainers/release-notes script that is
unmaintained.

Change-Id: I3f4a75d790e8e00e970358ca8f32e8295c91aac3
2024-06-15 18:46:18 -07:00
jade f95a47e8c4 release: release notes for 2.90.0
For now we just need to put the release notes in the final spot. We will
have to fix the date on both 2.90 and 2.91 branches, but such as it is.

Release created with releng/create_release.xsh

Closes: lix-project/lix#318
Change-Id: I38e79b40e7f632c8a286f2f09865a84dc93eca90
2024-06-15 18:46:18 -07:00
jade 9923fb6dd9 version: update to 2.90.0-rc2
Change-Id: I7cfa8414fa937de940b1598cc300497ce2d47635
2024-06-14 20:29:59 -07:00
jade a9c610fe37 release: merge release 2.90.0-rc1 back to mainline
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh

Change-Id: I92296a1746b54a081004fe2bb23e9e37fd33b3e5
2024-06-14 19:40:14 -07:00
jade f82a2a9aaa release: 2.90.0-rc1 "Vanilla Ice Cream"
Release produced with releng/create_release.xsh

Change-Id: Ib8df5ea5096d47b25f74988447146c1b5072b869
2024-06-14 19:40:13 -07:00
jade e1059bfa34 Merge changes from topic "releng" into main
* changes:
  releng: add sha256 for the manual tarball
  releng: fix upload of multiarch images to forgejo
  releng: fix git checking
  releng: fix logging inside interactive xonsh
  releng: support multiple systems
  version: update to 2.90.0-rc1
2024-06-15 02:38:09 +00:00
jade d5c670ad01 releng: add sha256 for the manual tarball
Whoops.

Change-Id: Ic6f8cdcb074d679e9b1fc3323c106cc853328dcc
2024-06-13 17:14:06 -07:00
jade 16ea19ced8 releng: fix upload of multiarch images to forgejo
Forgejo appears to immediately delete registry content that is
overwritten. This means that we are forced to delete our previous
workaround of making a temporary tag and use a new, more absurd
workaround of making an entire temporary image that we basically only
need to create to get its hash.

However, on the plus side, the new workaround doesn't create garbage
tags to begin with, which means that we don't have to deal with GitHub
not implementing the standardized tag delete endpoint and instead
only implementing a proprietary one.

Upstream-Bug: https://github.com/containers/skopeo/issues/2354
Change-Id: I220e7ce9a17fd230c38882f12c009a166dcc9336
2024-06-13 17:12:45 -07:00
jade 7be0d237e0 releng: fix git checking
Change-Id: I82ddd918311b48e596adb807b81221973113fe7a
2024-06-13 15:24:21 -07:00
jade e715e5fd31 releng: fix logging inside interactive xonsh
I don't know when this broke, it seems like it happened since the 24.05
upgrade, so xonsh 0.15.

What happened is that xonsh was trying to intercept log output, which
explodes if you have the logger survive past one command input. This is,
however, impossible to avoid if you are trying to use logging when you
import releng from inside xonsh for interactive use!

The error below is because the memory handler backing the stdout/stderr
of the one command that's just been run was closed after the command
completed.

Change-Id: I2be642aebf93da9818d08ff8b97c2e72ba5ac581

--- Logging error ---
Traceback (most recent call last):
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/logging/__init__.py", line 1113, in emit
    stream.write(msg + self.terminator)
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/base_shell.py", line 183, in write
    self.mem.write(s)
ValueError: I/O operation on closed file.
Call stack:
  File "/nix/store/xgdp1p1gv8ni1awnkzyqasnn6gz5wlvx-xonsh-0.15.1/bin/xonsh", line 8, in <module>
    sys.exit(main())
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/main.py", line 470, in main
    sys.exit(main_xonsh(args))
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/main.py", line 514, in main_xonsh
    shell.shell.cmdloop()
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/ptk_shell/shell.py", line 406, in cmd
loop
    line = self.singleline(auto_suggest=auto_suggest)
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/ptk_shell/shell.py", line 374, in sin
gleline
    line = self.prompter.prompt(**prompt_args)
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/prompt_toolkit/shortcuts/prompt.py", line 1
026, in prompt
    return self.app.run(
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/prompt_toolkit/application/application.py",
 line 1002, in run
    return asyncio.run(coro)
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/runners.py", line 189, in run
    with Runner(debug=debug) as runner:
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/runners.py", line 59, in __enter__
    self._lazy_init()
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/runners.py", line 137, in _lazy_init
    self._loop = events.new_event_loop()
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/events.py", line 810, in new_event_loop
    return get_event_loop_policy().new_event_loop()
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/events.py", line 699, in new_event_loop
    return self._loop_factory()
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/unix_events.py", line 64, in __init__
    super().__init__(selector)
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/selector_events.py", line 54, in __init__
    logger.debug('Using selector: %s', selector.__class__.__name__)
Message: 'Using selector: %s'
Arguments: ('EpollSelector',)

Change-Id: I90959809129aaf96aad4577599031688599ed85e
2024-06-13 15:17:44 -07:00
jade 068576042b packaging: Move binaryTarball to a passthru attr in package.nix
This is motivated by flakes being bad and all the stuff that calls
things by "system" being utterly unable to cope with cross compilation.
So if we go shove it in package.nix it is suddenly usable from cross
contexts.

Usage:

```
nix build -L .#nix-riscv64-linux.binaryTarball
```

Change-Id: I702ebf2ac5bd9d1c57662f968b000073134df336
2024-06-13 15:14:22 -07:00
jade d194939ff5 flake.nix: add riscv64 cross target
It builds. I have not tested the binaries since I don't have hardware,
but I would be rather surprised if it were broken, given that nix *runs*
on this platform.

Change-Id: I0b474ffcd4a431bf117a303d0b65fa6532113f48
2024-06-13 15:04:06 -07:00
jade 74fb2e8c47 releng: support multiple systems
I guess this is kind of important to being able to "release it".

Change-Id: Id6f295d0b4944fa1203783a400a246727dbd94b6
2024-06-13 14:36:03 -07:00
jade ac28cff28f version: update to 2.90.0-rc1
Change-Id: I913852cfb88b3b300ffb1050a91784be659ae66a
2024-06-13 13:03:25 -07:00
alois31 1d6fd94cf9 Merge "tests/libcmd: set HOME to a temporary directory" into main 2024-06-13 05:27:55 +00:00
jade d0b28f0e74 releng: fix docs upload
There were two bugs I found:
1. If the build isn't already done in the store, nix-store --realise
   does not know how to build it. You have to just give it the
   derivation and I guess it will realise all outputs, which is fine.
2. cp without -T will not overwrite an existing manual directory,
   creating a path manual/manual.

Change-Id: Ibebfd136a266da5330944a985e636ebb776f1909
2024-06-12 15:34:23 -07:00
jade 38e4e69633 .github: clean up stale things and wrong references
We do not need a stale bot. We do not need dependabot.

Fixes: lix-project/lix#391
Change-Id: I983fae4dc4cd9022b12f70e330b5c984c5fc1b9d
2024-06-12 15:34:23 -07:00
Pierre Bourdon 248ecb11af dev shell: Add bashInteractive
This was bothersome to me as a zsh+direnv user.

Change-Id: Ia5b54cc63647a5c6ced2b5412e972dac1abf8184
2024-06-12 15:34:23 -07:00
jade 73898cad0e tests/flake-registry: Fix occasional deadlocks
This seems to have been caused by having the wrong PID. I don't know why
it worked before in the sandbox, but the code was definitely wrong
before, so let's just fix it.

Change-Id: I556580bdf614c716566310e975a36daa6d6c9a91
2024-06-12 15:34:23 -07:00
jade 59b5965bbf doc: Write an index of environment variables used in testsuite
This was originally going to be just the testsuite but I kinda just
documented all of them.

I am tired of us not documenting these. This is a starting point to
producing an actually good index. I would like to enforce it in a
pre-commit hook eventually that we document all environment variables
used in Lix itself, even if it is terse dev facing docs.

This is full of a bunch of TODOs caused by auditing code. They should
probably be done at some point.

Change-Id: I7c0d3b257e19bae23d47d1efbd7361d203bccb0e
2024-06-12 15:34:23 -07:00
jade 211f79d4a2 doc/testing.md: Rewrite some outdated sections for meson and current source layout
Change-Id: Ia23f82c9a564b55bd799afbda59c28c9b0a65c13
2024-06-12 15:34:23 -07:00
jade 5f6eb6eb44 doc: rewrite the multi-user documentation to actually talk about security
It's in the security section, and it was totally outdated anyway.

I took the opportunity to write down the stuff we already believed.

Change-Id: I73e62ae85a82dad13ef846e31f377c3efce13cb0
2024-06-12 15:34:23 -07:00
jade d9345d8836 tests: verify that NIX_DAEMON_PACKAGE's version looks like a version
Followup to https://gerrit.lix.systems/c/lix/+/1417 to ensure that this
parser will never take something that doesn't look like a version.

It turns out this problem is less alarming than initially thought
because it only applies to the testsuite in a non-default mode.

Change-Id: I26aba24aaf0215f2b782966314b94784db766266
2024-06-12 15:34:23 -07:00
jade 6939ffc9f9 Check devShells in CI
We should not let these regress in CI by having broken dependencies or
similar. Still need to fix the evaluation error checking in
buildbot-nix, but this is a useful step regardless.

Fixes: lix-project/lix#383

Change-Id: I3883184165440e66256c989117f2ab2e54c3aafd
2024-06-12 15:34:23 -07:00
jade 479055aee8 Misc workaround removals since 24.05 upgrade
Change-Id: I9491b103333cb0e25c245199e88365ded7800d2e
2024-06-12 15:34:23 -07:00
Pierre Bourdon f7b6552699 [resubmit] flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)
-- message from cl/1418 --

The boehmgc changes are bundled into this commit because doing otherwise
would require an annoying dance of "adding compatibility for < 8.2.6 and
>= 8.2.6" then updating the pin then removing the (now unneeded)
compatibility. It doesn't seem worth the trouble to me given the low
complexity of said changes.

Rebased coroutine-sp-fallback.diff patch taken from https://github.com/NixOS/nixpkgs/pull/317227

-- jade resubmit changes --

This is a resubmission of https://gerrit.lix.systems/c/lix/+/1418, which
was reverted in https://gerrit.lix.systems/c/lix/+/1432 for breaking CI
evaluation without being detected.

I have run `nix flake check -Lv` on this one before submission and it
passes on my machine and crucially without eval errors, so the CI result
should be accurate.

It seems like someone renamed forbiddenDependenciesRegex to
forbiddenDependenciesRegexes in nixpkgs and also changed the type
incompatibly. That's pretty silly, but at least it's just an eval error.

Also, `xonsh` regressed the availability of `xonsh-unwrapped`, but it
was fixed by us in https://github.com/NixOS/nixpkgs/pull/317636, which
is now in our channel, so we update nixpkgs compared to the original
iteration of this to simply get that.

We originally had a regression related to some reorganization of the
nixpkgs lib test suite in which there was broken parameter passing.
This, too, we got quickfixed in nixpkgs, so we don't need any changes
for it: https://github.com/NixOS/nixpkgs/pull/317772

Related: https://gerrit.lix.systems/c/lix/+/1428
Fixes: lix-project/lix#385

Change-Id: I26d41ea826fec900ebcad0f82a727feb6bcd28f3
2024-06-12 15:34:22 -07:00
alois31 3c0434999e tests/libcmd: set HOME to a temporary directory
The libcmd unit test creates files (more specifically, the fetcher cache) in
its home directory. In the single-user sandbox, this leads to the creation of
/homeless-shelter, since this is the default HOME and the root is writable.
Unfortunately, this conflicts with the assumption of the functional tests that
this directory does not exist. Use a different home directory to prevent these
test failures, and thus restore the ability to build inside the single-user
sandbox.

Fixes: lix-project/lix#365
Change-Id: I4df8c53d043234b95a7c0ac45fc5ee89e8d46aff
2024-06-12 22:13:55 +00:00
Qyriad f46194faa2 build: remove unused 'deps' variable
This never actually got used

Change-Id: I8f3f1d413124b27913d59a75cff42319cbaac385
2024-06-12 21:40:57 +00:00
jade 8a3d063a49 Merge changes from topic "releng" into main
* changes:
  releng: add prod environment, ready for release
  releng: automatically figure out if we should tag latest for docker
  releng: support multiarch docker images
  manual: rewrite the docker guide now that we have images
  Rewrite docker to be sensible and smaller
  Implement docker upload in the releng tools
2024-06-11 04:45:12 +00:00
jade f432e464dd Merge "tests: fix daemon version in isDaemonNewer function" into main 2024-06-10 23:22:05 +00:00
jade a986a8dfa1 Merge "Revert "flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)"" into main 2024-06-10 04:48:12 +00:00
jade 8a09465c3a Revert "flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)"
This reverts commit 28a079f841.

Reason for revert: This caused a pile of regressions in CI, and does not pass nix flake check. Some number of them are fixed in CL: https://gerrit.lix.systems/c/lix/+/1429 but there's more to be fixed.

We should defer this after 2.90.

Change-Id: Ib839d0fcb08eb52094af2b521145e3c1b4e0556f
2024-06-10 04:29:13 +00:00
jade 82dc712d93 releng: add prod environment, ready for release
I am *reasonably* confident that this releng infrastructure can actually
build a Lix 2.90 and release it successfully. Let's make it possible to
do, and add some cute colours to the confirmation message.

Change-Id: I85e498b6fb49ffc5e75c0a72c5e45fb1f69030d3
2024-06-09 20:33:24 -07:00
jade ce71d0e9ab releng: automatically figure out if we should tag latest for docker
For example, when releasing from release-2.90, if `main` has a 2.91 tag
ancestor, we know that 2.91 was released, so we should *not* tag latest.

Change-Id: Ia56b17a2ee03bbec74b7c271c742858c690d450d
2024-06-09 20:33:24 -07:00
jade 9aeb314e6a releng: support multiarch docker images
If we don't want to have separate registry tags by architecture (EWWWW),
we need to be able to build multiarch docker images. This is pretty
simple, and just requires making a manifest pointing to each of the
component images.

I was *going* to just do this API prodding with manifest-tool, but it
doesn't support putting metadata on the outer manifest, which is
actually kind of a problem because it then doesn't render the metadata
on github. So I guess we get a simple little containers API
implementation that is 90% auth code.

Change-Id: I8bdd118d4cbc13b23224f2fb174b232432686bea
2024-06-09 20:33:24 -07:00
jade 4392d89eea manual: rewrite the docker guide now that we have images
Change-Id: I5bdf47e67059ae4099552750a47ae070dbe094df
2024-06-09 20:33:24 -07:00
jade 9bb7fb8f69 Rewrite docker to be sensible and smaller
I have checked the image can build things and inspected `diff -ru`
compared to the old image. As far as I can tell it is more or less
the same besides the later git change.

Layers are now 65MB or less, and we aren't against the maxLayers limit
for the broken automatic layering to do anything but shove one store
path in a layer (which is good behaviour, actually).

This uses nix2container which streams images, so the build time is much
shorter.

I have also taken the opportunity to, in addition to fixing the 400MB
single layer (terrible, and what motivated this in the first place),
delete about 200MB of closure size inflicted by git vs gitMinimal
causing both perl and python to get into closure.

People mostly use this thing for CI, so I don't really think you need
advanced git operations, and large git can be added at the user side if
really motivated.

With love for whichever container developer somewhat ironically assumed
that one would not run skopeo in a minimal container that doesn't have a
/var/tmp.

Fixes: lix-project/lix#378

Change-Id: Icc3aa20e64446276716fbbb87535fd5b50628010
2024-06-09 20:33:24 -07:00
jade 7dfa2a761e Merge changes from topic "releng" into main
* changes:
  releng: support pushing the manual to docs also
  Expose officialRelease from the flake
  Put into place initial release engineering
2024-06-09 08:28:52 +00:00
jade ff95b980d4 Implement docker upload in the releng tools
This uses skopeo to not think about docker daemons. I, however, noticed
that the docker image we had would have totally terrible cache hits, so
I rewrote it.

Fixes: lix-project/lix#252

Change-Id: I3c5b6c1f3ba0b9dfcac212b2148f390e0cd542b7
2024-06-09 00:30:12 -07:00
Pierre Bourdon 28a079f841
flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)
The boehmgc changes are bundled into this commit because doing otherwise
would require an annoying dance of "adding compatibility for < 8.2.6 and
>= 8.2.6" then updating the pin then removing the (now unneeded)
compatibility. It doesn't seem worth the trouble to me given the low
complexity of said changes.

Rebased coroutine-sp-fallback.diff patch taken from https://github.com/NixOS/nixpkgs/pull/317227

Change-Id: I8c590e9fe25c0f566d0cfeacb96d8cf50abf12e8
2024-06-09 01:25:53 +02:00
Pierre Bourdon 9281a12532
tests/nixos/nix-copy: fix NixOS >= 24.05 compatibility
4b128008c5d9fde881ce1b0a25e60ae0415a14d5 in nixpkgs introduced a default
hashedPasswordFile for root in NixOS tests, which takes precedence over
the password option set in the nix-copy test.

Change-Id: Iffaebec5992e50614b854033f0d14312c8d275b5
2024-06-08 17:59:08 +02:00
Mario Rodas a05de58ebd tests: fix daemon version in isDaemonNewer function
Since ad8a4b380e, the version printer returns "nix (Lix, like Nix) 2.x",
hence the `daemonVersion` was being set to the string "like".

Using `compareVersions` with a letter compares them lexicographically:

   builtins.compareVersions "like" "2.12pre20230103"  // => -1
   builtins.compareVersions "like" "2.16.0"           // => -1

This caused that `isDaemonNewer` always returned 1, falsy in Bash terms.
Therefore, the test suite skipped those tests where they use it.

Fixes lix-project/lix#324

Change-Id: If6682515bf0bf8b8add641af9a4e98b50a9acb51
2024-06-08 04:20:00 +00:00
jade 4f94531209 Merge changes from topic "releng" into main
* changes:
  Add meson release note
  Move version to a JSON file so we can have release names
  Remove rl-next-dev
2024-06-07 03:53:31 +00:00
jade 98e8475147 releng: support pushing the manual to docs also
Change-Id: Ifd0b51425ee4955e0230fb2804a6f54ef0fe16e9
2024-06-06 20:53:08 -07:00
jade bdf1b264ad Expose officialRelease from the flake
Change-Id: If87beb3f31dfb5d59862294ac2e1c821ea864277
2024-06-06 20:53:08 -07:00
jade c32a01f9eb Put into place initial release engineering
This can release x86_64-linux binaries to staging, with ephemeral keys.
I think it's good enough to review at least at this point, so we don't
keep adding more stuff to it to make it harder to review.

Change-Id: Ie95e8f35d1252f5d014e819566f170b30eda152e
2024-06-06 20:53:08 -07:00
Qyriad ec768df004 Merge changes Ic4be41eb,I48db2385 into main
* changes:
  devshells: only enable pch for clang
  build: expose option to enable or disable precompiled std headers
2024-06-06 22:21:52 +00:00
jade 611b1de441 Add meson release note
Change-Id: I4d2d08dc77a3ab4dce9fbb129c1487aa8c9f1722
2024-06-06 15:08:12 -07:00
jade 9c77c62e73 Move version to a JSON file so we can have release names
Change-Id: I5ff3396a302565ee5ee6c2db97e048e403779076
2024-06-06 15:08:12 -07:00
jade 24057dcb6a Remove rl-next-dev
We realized that there's really no good place to put these dev facing
bulletins, and the user-facing release notes aren't really the worst
place to put them, I guess, and we do kind of hope that it converts
users to devs.

Change-Id: Id9387b2964fe291cb5a3f74ad6344157f19b540c
2024-06-06 15:08:12 -07:00
jade 1659404626 Add xonsh to the shell
Change-Id: If8f3825d2bdcc3f1d00583a11d890c1c8ab37b9f
2024-06-06 14:50:27 -07:00
jade e0748377dc pname: nix -> lix
This had a regression last time: https://gerrit.lix.systems/c/lix/+/1196

But f3f68fcfa fixed upgrade-nix to not be broken, so this should be ok tbh.

Change-Id: I48ea1359790878bb8ead5d8a4b3f61caa4aabfb5
2024-06-06 20:42:29 +00:00
Qyriad 766e718f67 devshells: only enable pch for clang
clangd seems to break if GCC is using precompiled headers for C++'s
standard library, so this sets -Denable-pch-std=${stdenv.cc.isClang}

Fixes #374.

Change-Id: Ic4be41ebe7576ebcb9c208275596f953c2003109
2024-06-06 12:48:13 -06:00
Qyriad 06e65e537b build: expose option to enable or disable precompiled std headers
They are enabled by default, and Meson will also prints whether or not
they're enabled at the bottom at the end of configuration.

Change-Id: I48db238510bf9e74340b86f243f4bbe360794281
2024-06-06 12:46:26 -06:00
jade 8f9bcd20eb Merge "libstore/filetransfer: fix no-s3 build" into main 2024-06-06 03:08:14 +00:00
Linus Heckemann 609b721425 libstore/filetransfer: fix no-s3 build
Fixes a compiler error that looks like:

error: could not convert '[...]' from 'future<void>' to 'future<nix::FileTransferResult>'
Change-Id: I4aeadfeba0dadfdf133f25e6abce90ede7a86ca6
2024-06-05 15:50:57 -07:00
Pierre Bourdon 6e59b4b407 Merge "tests/nixos: make the tarball-flakes test better reflect real use cases" into main 2024-06-04 15:14:02 +00:00
Pierre Bourdon a3256a9375
tests/nixos: make the tarball-flakes test better reflect real use cases
In most real world cases, the Link header is set on the redirect, not on
the final file. This regressed in Lix earlier and while new unit tests
were added to cover it, this integration test should probably have also
caught it.

Change-Id: I2a9d8d952fff36f2c22cfd751451c2b523f7045c
2024-06-04 08:12:59 +02:00
Olmo Kramer 72d85acba4 nix flake update: add test for multiple inputs from nix#10073
Upstream-PR: https://github.com/NixOS/nix/pull/10073
Change-Id: I53fcb43b387e55439e062e208877afeb88493bb4
2024-06-03 21:50:33 +00:00
Nikodem Rabuliński 5d3910330d Show message about --update-input being replaced by nix flake update
Fixes: lix-project/lix#283

Change-Id: I6ee23874cb09f51d788521273076a25ba8764859
2024-06-03 21:50:33 +00:00
Nikodem Rabuliński cc3674ea93 Accept multiple arguments to nix flake update
Fixes: lix-project/lix#194

Change-Id: Ia7bd4f7640384be9827dbb7e2c594f0aa5f1aff8
2024-06-03 21:50:33 +00:00
Qyriad c55e93ca23 Revert "nix3: always use the same verbosity default (info)"
This reverts commit d0390b5cf2.

Other parts of the codebase will need to be adjusted in response to a
default verbosity change. Let's just push this to after 2.90.

Fixes #362.
Fixes #367.

Change-Id: I04648473579146851bda41d764adc1ef954c355d
2024-06-01 18:29:19 -06:00
Qyriad d374a9908f Merge "build: fix static linking with a hack" into main 2024-06-01 19:17:13 +00:00
raito b8cb7abcf0 chore: rebrand Nix to Lix when it makes sense
Here's my guide so far:

$ rg '((?!(recursive).*) Nix
(?!(daemon|store|expression|Rocks!|Packages|language|derivation|archive|account|user|sandbox|flake).*))'
-g '!doc/' --pcre2

All items from this query have been tackled. For the documentation side:
that's for lix-project/lix#162.

Additionally, all remaining references to github.com/NixOS/nix which
were not relevant were also replaced.

Fixes: lix-project/lix#148.
Fixes: lix-project/lix#162.
Change-Id: Ib3451fae5cb8ab8cd9ac9e4e4551284ee6794545
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-06-01 20:31:24 +02:00
Linus Heckemann 5312e60be6 Merge "libfetchers: allow fetching gitlab refs with >1 commit" into main 2024-06-01 09:54:11 +00:00
Qyriad e54d4c9381 build: fix static linking with a hack
This causes libstore, libexpr, libfetchers, and libutil to be linked
with -Wl,--whole-archive to executables, when building statically.

libstore for the store backends, libexpr for the primops, libfetchers
for the fetcher backends I assume(?), and libutil for the nix::logger
initializer (which notably shows in pre-main constructors when HOME is
not owned by the user. cursed.).

This workaround should be removed when #359 is fixed.

Fixes #306.

Change-Id: Ie9ef0154e09a6ed97920ee8ab23810ca5e2de84c
2024-05-31 21:47:16 -06:00
jade c7ca87461d Merge "build-remote: truncate+hash store URI used in lockfile paths" into main 2024-05-31 19:22:32 +00:00
jade 7081889faa Merge "truncate WAL files on exit" into main 2024-05-31 19:21:30 +00:00
jade adedac70fa Merge changes Ifcb0d310,I664366b8,Ibe7cf546 into main
* changes:
  gitignore: delete 90% of it
  build-time: remove 20% more by PCH'ing C++ stdlib
  shellHook: make it actually run
2024-05-31 19:19:29 +00:00
Linus Heckemann 82de36f77a libfetchers: allow fetching gitlab refs with >1 commit
Change-Id: I945c4c5512def9eff728bb67fe3c03ae17f99d6d
2024-05-31 21:12:04 +02:00
jade a75d7a5777 Merge "libutil: fix args assert being thrown on Darwin in nix-eval-jobs" into main 2024-05-31 18:57:13 +00:00
alois31 ff08d95420 Merge "libstore/build: copy ca-certificates too" into main 2024-05-31 16:44:18 +00:00
annalee 713cd7e9e7 truncate WAL files on exit
Fix for https://github.com/NixOS/nix/issues/10300

18a2620273  enabled persistent WAL files that will never get truncated. to fix this, journal_size_limit is set to 2^40, which results in the WAL files being truncated to 0 on exit, as well as limiting the WAL files to 2^40 bytes following a checkpoint.

this aligns lix with the nix change: https://github.com/NixOS/nix/pull/10301

https://www.sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpersistwal
https://www.sqlite.org/pragma.html#pragma_journal_size_limit
ed517a7082/src/wal.c (L2518)

PR-Link: https://github.com/lix-project/lix/pull/9

Co-Authored-By: paparodeo <170618376+paparodeo@users.noreply.github.com>
Change-Id: I90ec1a467c92c582ff8c07dd363a4cf789782214
2024-05-31 12:22:15 +00:00
Lunaphied d4b7e6baca build-remote: truncate+hash store URI used in lockfile paths
Fixes: lix-project/lix#157
Fixes: lix-project/lix#221

Previously the entire escaped store URI was included. This would cause
build failures if a very long or deeply nested path was being used in
the store.

Now, we use the first 48 characters of the URL (escaped), then 16 bytes
of hash of the entire URL. This should never collide and limits the
length of the file name to a bit over 64, which is fine.

Change-Id: Ic1ba690a94e83749567c2c29460b8d1bcf2ac413
2024-05-31 12:18:24 +00:00
jade ac78c1dcd5 libutil: fix args assert being thrown on Darwin in nix-eval-jobs
This is because a dynamic_cast<nix::RootArgs *> of a (n-e-j) MyArgs
returns nullptr even though MyArgs has virtual nix::RootArgs as a
parent.

class MyArgs : virtual public nix::MixEvalArgs,
               virtual public nix::MixCommonArgs,
               virtual nix::RootArgs { ... };

So this should work right?? But it does not. We found out that it's
caused by -fvisibility=hidden in n-e-j, but honestly this code was bad
anyway.

The trivial solution is to simply stop relying on RTTI working properly
here, which is probably better OO architecture anyway. However, I am not
100% confident *this* is sound, since we have this horrible hierarchy:

      Args (defines getRoot)
     /        |           \
RootArgs  MixCommonArgs  MixEvalArgs
(overrides)

I am not confident that this is guaranteed to resolve from Args always
in the case of this override.

Assertion failed: (res), function getRoot, file src/libutil/args.cc, line 67.
6MyArgsProcess 60503 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #4: 0x0000000100b1a41c liblixutil.dylib`nix::Args::processArgs(std::__1::list<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, bool) [inlined] nix::Args::getRoot(this=0x00000001000d0688) at args.cc:67:5 [opt]
   64       std::cout << typeid(*p).name();
   65
   66       auto * res = dynamic_cast<RootArgs *>(p);
-> 67       assert(res);
   68       return *res;
   69   }
   70
Target 0: (nix-eval-jobs) stopped.

(lldb) p this
(MyArgs *) 0x00000001000d0688
(lldb) p *this
(nix::Args) {
  longFlags = size=180  { ... }
  shortFlags = size=4  { ... }
  expectedArgs = size=1  { ... }
  processedArgs = size=0 {}
  hiddenCategories = size=1 {
    [0] = "Options to override configuration settings"
  }
  parent = nullptr
}

We also found that if we did this:
class [[gnu::visibility("default")]] RootArgs : virtual public Args

it would work properly (???!). This is of course, very strange, because
objdump -Ct output on liblixexpr.dylib is identical both with and
without it.

Possibly related: https://www.qt.io/blog/quality-assurance/one-way-dynamic_cast-across-library-boundaries-can-fail-and-how-to-fix-it

Fixes: lix-project/nix-eval-jobs#2
Change-Id: I6b9ed968ed56420a9c4d2dffd18999d78c2761bd
2024-05-31 12:17:06 +00:00
terru - 0c6cb34de6 Merge "document context-dependent keywords" into main 2024-05-31 10:07:50 +00:00
alois31 cf756fdf3c libstore/build: copy ca-certificates too
In b469c6509b, the ca-certificates file was
missed. It should be copied too so that we don't end up bind-mounting a broken
symlink.

Change-Id: Ic9b292d602eb94b0e78f77f2a27a19d24665783c
2024-05-31 07:54:18 +00:00
jade a6b33cb3b2 gitignore: delete 90% of it
*laughs in meson putting it all in build/*

Change-Id: Ifcb0d3104cf9e64c4de91c3a92828899a209d00d
2024-05-30 22:24:55 +00:00
jade 0f99ed43f1 build-time: remove 20% more by PCH'ing C++ stdlib
It seems like someone implemented precompiled headers a long time ago
and then it never got ported to meson or maybe didn't work at all.

This is, however, blessedly easy to simply implement. I went looking for
`#define` that could affect the result of precompiling the headers, and
as far as I can tell we aren't doing any of that, so this should truly
just be free build time savings.

Previous state:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

New state:
**** Time summary:
Compilation (567 times):
  Parsing (frontend):         1123.0 s
  Codegen & opts (backend):   1078.1 s

I wonder if the "regression" in codegen time is just doing the PCH
operation a few times, because meson does it per-target.

Change-Id: I664366b8069bab4851308b3a7571bea97ac64022
2024-05-30 21:54:21 +00:00
jade e6e5cacabe shellHook: make it actually run
When we changed this in I91cb6eb6668f3a8eace36ecbdb01eb367861d77b to
not run in nested shells, we didn't predict that `nix develop` would do
something ridiculous and append -env to things silently. `nix-shell` of
course does not do this, so we need to tolerate both.

Change-Id: Ibe7cf546823d7358ebc0414ecbe154e3e3194f45
2024-05-30 21:54:21 +00:00
Linus Heckemann 3df013597d libfetchers: handle nonexistent refs in GitLab repos more gracefully
Before:

$ nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: [json.exception.type_error.302] type must be string, but is null

After:

$ outputs/out/bin/nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error:
       … while updating the lock file of flake 'git+file:///home/linus/projects/lix?ref=refs/heads/fix-gitlab-nonexistent&rev=915f16a619a36237a099b9aa9afed6d14ff613b4'

       … while updating the flake input 'nixpkgs'

       … while fetching the input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'

       error: No commits returned by GitLab API -- does the ref really exist?

Change-Id: Id9bc79d98348500e152ed519bb3ac79a3d15c38d
2024-05-30 21:53:51 +00:00
jade 2f104bbe3b Merge "Revert "tests/filetransfer: reënable on Darwin"" into main 2024-05-30 21:53:38 +00:00
jade 533d469875 Revert "tests/filetransfer: reënable on Darwin"
This reverts commit 285bc67318.

Reason for revert: lix-project/lix#364

For some reason this broke `main` even though the change we are reverting passed CI! Mysterious, haunted, etc. Needs more debugging, let's turn it off for now.

Change-Id: Ica4819d61cd35b83eb52985bfcb657e858f025a9
2024-05-30 21:38:32 +00:00
Qyriad 260db1ea64 Merge "build: fix static aws-cpp-sdk" into main 2024-05-30 20:53:49 +00:00
jade c161687b5f Merge "tests/filetransfer: reënable on Darwin" into main 2024-05-30 14:58:23 +00:00
jade 18aa3e1d57 Merge "Remove 100s of CPU time (10%) from build times (1465s -> 1302s)" into main 2024-05-30 14:57:37 +00:00
jade 53d40888ff Merge "unix-domain-socket.cc: add comment explaining why bindConnectProcHelper" into main 2024-05-30 14:56:57 +00:00
Maximilian Bosch 3d78b4847e Merge "release-notes: add missing credits/category to consistent-nix-build entry" into main 2024-05-30 14:38:47 +00:00
Maximilian Bosch 6abac7aacc release-notes: add missing credits/category to consistent-nix-build entry
Change-Id: I737422a2ff9d66be30cc432f8c1ddba9b1e71f4f
2024-05-30 14:05:12 +00:00
Ilya K da95bf8c82 libstore/filetransfer: remove debug print
foo.

Change-Id: I7d7db22f68046d2ecf3b594b4ee6fd9c9dac4be1
2024-05-30 16:42:45 +03:00
Maximilian Bosch ce82067566 Merge "libutil/args: warn on unknown settings after parsing all flags" into main 2024-05-30 09:24:14 +00:00
Qyriad 6475793678 build: fix static aws-cpp-sdk
Change-Id: I310830951106f194f6960a6b2d52b5081a7f6156
2024-05-30 00:40:25 -06:00
Jörg Thalheim 194b6cc611 docs: expand importNative/exec example (#10803)
Co-authored-by: Qyriad <qyriad@qyriad.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
(cherry picked from commit 5786e1ae7c300b3c7434e7df99b41f180dc42e37)
Change-Id: I16b408ba7c70dca985c05c71bf6195fe9f0b5841
2024-05-30 08:12:03 +02:00
jade 7575db522e Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.

Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1465.3 s
  Codegen & opts (backend):   1110.9 s

<snip>

**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: serialise.hh strings.hh
  6x: <direct include>
  6x: archive.hh serialise.hh strings.hh
  ...

173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
  14x: store-api.hh nar-info.hh hash.hh
  11x: <direct include>
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh
  6x: fetchers.hh hash.hh
  ...

169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-derived-path.hh
  1x: installable-value.hh
  ...

159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
  19x: <direct include>
  14x: store-api.hh nar-info.hh hash.hh serialise.hh
  11x: serialise.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
  ...

156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
  19x: file-system.hh
  11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  ...

132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

  done in 0.6s.
```

After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
  done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

<snip>

**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: <direct include>
  6x: serialise.hh strings.hh
  6x: fetchers.hh hash.hh serialise.hh strings.hh
  ...

154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-value.hh
  1x: installable-derived-path.hh
  ...

136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
  37x: command.hh installable-value.hh installables.hh
  25x: store-api.hh realisation.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: libexpr.hh value.hh context.hh
  6x: shared.hh
  ...

87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
  35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
  12x: regex sstream istream
  10x: file-system.hh types.hh ref.hh memory unique_ptr.h
  10x: gtest.h memory unique_ptr.h
  10x: globals.hh types.hh ref.hh memory unique_ptr.h
  6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
  ...

85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
  37x: command.hh installable-value.hh installables.hh derived-path.hh
  20x: globals.hh
  20x: logging.hh
  16x: store-api.hh logging.hh
  6x: <direct include>
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
  ...

  done in 0.5s.
```

Change-Id: I27f0a2d566db17832cd9be935f12efe7f95b92d0
2024-05-29 22:16:15 -07:00
Qyriad 2760818f06 package: fix derivation correctness when static
Change-Id: I394bb72d9f378cd78acc6cf67a9bb15e342d57c4
2024-05-30 04:21:57 +00:00
Qyriad eac3546d50 package: return from shellHook correctly
If our shellHook is being run from a nested nix-shell (see 7a12bc200¹),
then (I think) it is run from a bash function due to the nesting, then
`return` is correct. If its `eval`'d though, then there isn't really a
correct way to early exit. So we can just unconditionally be executed in
a function.

Basically, we have IIFE at home.

[1]: 7a12bc2007

Change-Id: Iacad25cbbf66cde2911604e6061e56ad6212af7e
2024-05-30 04:21:57 +00:00
Qyriad 68937f2b64 package: dequalify {host,build}Platform for convenience
Change-Id: I5ffeac894a5bff101683cf3d566c63b478779962
2024-05-30 04:21:47 +00:00
jade 285bc67318 tests/filetransfer: reënable on Darwin
Since we put __darwinAllowLocalNetworking in our derivation in
I752b81c85ebeaab4e582ac01c239d69d65580f37, this stuff will just work
fine. I checked our derivation works on the darwin community builder.

Change-Id: I40e3a801d6bb38efede79af4aded65c1e1f57cec
2024-05-29 20:41:22 -07:00
Cole Helbling 031d924116 libutil/args: warn on unknown settings after parsing all flags
Upstream change: https://github.com/NixOS/nix/pull/10701

Change-Id: Icf271df57ec529dd8c64667d1ef9f6dbf02d33d3
2024-05-30 03:07:21 +00:00
jade 26b3a1b9ce unix-domain-socket.cc: add comment explaining why bindConnectProcHelper
We reviewed this code a while ago, and we neglected to get a comment in
saying why it's Like This at the time. Let's fix that, since it is code
that looks very absurd at first glance.

Change-Id: Ib67b49605ef9ef1c84ecda1db16be74fc9105398
2024-05-29 19:50:04 -07:00
Qyriad 218630a241 Merge "docs: enable non-default TOC folding" into main 2024-05-30 02:45:44 +00:00
jade 562ff516ab Merge changes from topic "libutil-split" into main
* changes:
  util.hh: Delete remaining file and clean up headers
  util.hh: Move nativeSystem to local-derivation-goal.cc
  util.hh: Move stuff to types.hh
  util.cc: Delete remaining file
  util.{hh,cc}: Move ignoreException to error.{hh,cc}
  util.{hh,cc}: Split out namespaces.{hh,cc}
  util.{hh,cc}: Split out users.{hh,cc}
  util.{hh,cc}: Split out strings.{hh,cc}
  util.{hh,cc}: Split out unix-domain-socket.{hh,cc}
  util.{hh,cc}: Split out child.{hh,cc}
  util.{hh,cc}: Split out current-process.{hh,cc}
  util.{hh,cc}: Split out processes.{hh,cc}
  util.{hh,cc}: Split out file-descriptor.{hh,cc}
  util.{hh,cc}: Split out file-system.{hh,cc}
  util.{hh,cc}: Split out terminal.{hh,cc}
  util.{hh,cc}: Split out environment-variables.{hh,cc}
2024-05-30 02:33:05 +00:00
Qyriad afeaa2371c docs: enable non-default TOC folding
Folding by default would prevent things like "Ctrl+F for nix-env" from
working trivially, but the user should be able to fold if they want to.

Change-Id: I5273272289f0f24e1f040c691580acfe33f66bd4
2024-05-30 00:23:01 +00:00
Mario Rodas c71f21da3a Merge "tests: fix functional-timeout" into main 2024-05-29 23:57:49 +00:00
eldritch horrors dd4a2c1759 libstore: fix http abuses no longer working
while refactoring the curl wrapper we inadvertently broken the immutable
flake protocol, because the immutable flake protocol accumulates headers
across the entire redirect chain instead of using only the headers given
in the final response of the chain. this is a problem because Some Known
Providers Of Flake Infrastructure set rel=immutable link headers only in
the penultimate entry of the redirect chain, and curl does not regard it
as worth returning to us via its response header enumeration mechanisms.

fixes lix-project/lix#358

Change-Id: I645c3932b465cde848bd6a3565925a1e3cbcdda0
2024-05-29 22:35:29 +00:00
Tom Hubrecht a39ba22ff7 util.hh: Delete remaining file and clean up headers
Change-Id: Ic1f68e6af658e94ef7922841dd3ad4c69551ef56
2024-05-29 12:38:51 +02:00
Tom Hubrecht f0eb650ee8 util.hh: Move nativeSystem to local-derivation-goal.cc
Change-Id: I74565fbfd3aeedef8f50465808fac712b84e47ad
2024-05-29 11:42:42 +02:00
Tom Hubrecht d73c40ff3d util.hh: Move stuff to types.hh
Change-Id: Ia852306a4b8aac6856dc42bc69e4b58b53a0d67c
2024-05-29 11:42:42 +02:00
Tom Hubrecht 74513483bc util.cc: Delete remaining file
Change-Id: I2b47848904f2ce7bd78b83738e99a4c9da627751
2024-05-29 11:42:42 +02:00
Tom Hubrecht 93ebb3e7df util.{hh,cc}: Move ignoreException to error.{hh,cc}
Change-Id: Iae6464217a55c313a983e5c651b26a4a1e446706
2024-05-29 11:42:42 +02:00
Tom Hubrecht 8b6d2d3915 util.{hh,cc}: Split out namespaces.{hh,cc}
Change-Id: I8fd3f3b50c15ede29d489066b4e8d99c2c4636a6
2024-05-29 11:41:16 +02:00
Tom Hubrecht f79ee66646 util.{hh,cc}: Split out users.{hh,cc}
Change-Id: I1bd92479a2cb7e5c2c2e1541b80474adb05ea0df
2024-05-29 11:01:34 +02:00
Tom Hubrecht b910551120 util.{hh,cc}: Split out strings.{hh,cc}
Change-Id: I4f642d1046d56b5db26f1b0296ee16a0e02d444a
2024-05-29 11:01:34 +02:00
Tom Hubrecht 5b5a75979a util.{hh,cc}: Split out unix-domain-socket.{hh,cc}
Change-Id: I3f9a628e0f8998b6146f5caa8ae9842361a66b8b
2024-05-29 11:01:34 +02:00
Tom Hubrecht e81ed5f12d util.{hh,cc}: Split out child.{hh,cc}
Change-Id: Iec4824e071f537b17dd62dbb8c01b8eec14e9783
2024-05-29 11:01:34 +02:00
Tom Hubrecht 2473e1253d util.{hh,cc}: Split out current-process.{hh,cc}
Change-Id: I77095b9d37e85310075bada7a076ccd482c28e47
2024-05-29 11:01:34 +02:00
Tom Hubrecht 9a52e4688c util.{hh,cc}: Split out processes.{hh,cc}
Change-Id: I39280dc40ca3f7f9007bc6c898ffcf760e2238b7
2024-05-29 11:01:34 +02:00
Tom Hubrecht 8cd9aa24a8 util.{hh,cc}: Split out file-descriptor.{hh,cc}
Change-Id: I0dd0f9a9c2003fb887e076127e7f825fd3289c76
2024-05-29 09:54:47 +02:00
Tom Hubrecht 6b5078c815 util.{hh,cc}: Split out file-system.{hh,cc}
Change-Id: Ifa89a529e7e34e7291eca87d802d2f569cf2493e
2024-05-29 09:54:47 +02:00
Tom Hubrecht 81bdf8d2d6 util.{hh,cc}: Split out terminal.{hh,cc}
Change-Id: I9de2296b4012d50f540124001d54d6ca3be4c6da
2024-05-29 09:54:47 +02:00
terru - d8e452a91b document context-dependent keywords
Documents some of the weirdness of __curPos and the or keyword.
This does not fit well into any existing section for either of
them, though the use of or as a quasi-operator is mentioned in
the section on operators.

Addresses lix-project/lix#353

Change-Id: I7c906c8368843dca6944e8b22573b6d201cd9a76
2024-05-28 23:44:55 +02:00
terru - 71b32bb87c Merge "package: don't run shellHook in nested nix-shells" into main 2024-05-28 10:01:37 +00:00
Tom Hubrecht 6fd6795bc4 util.{hh,cc}: Split out environment-variables.{hh,cc}
Change-Id: Icff0aa33fda5147bd5dbe256a0b9d6a6c8a2c3f6
2024-05-28 11:29:29 +02:00
Mario Rodas ec5f025ec2 tests: fix functional-timeout
grepQuietInvert is a typo introduced by c11836126b.

The test functional-timeout was failing silently because Bash considered
the command-not-found error as truthy.

Change-Id: Ic13829d02ec55d6ecd63a0f4d34ec0d32379609f
2024-05-28 04:20:00 +00:00
terru - 7a12bc2007 package: don't run shellHook in nested nix-shells
If a nested nix-shell is run inside a nix-shell, then the outer shell's
shellHook will be passed through and run again, unless the nested shell
defines its own.

With lix's hook, this can be annoying: forgetting to exit its nix-shell,
cd'ing to another repository & entering a nested nix-shell will happily
install lix's pre-commit hook in it.

This change makes lix's hook return early in such cases.

Change-Id: I91cb6eb6668f3a8eace36ecbdb01eb367861d77b
2024-05-27 22:09:57 +02:00
terru - 0b91a4b0ec shell.nix: conform to documentation
According to doc/manual/src/contributing/hacking.md, all development
shells should also be available in classic nix using the -A argument
to nix-shell, e.g.

  nix-shell -A native-clangStdenvPackages

This was not actually the case; flake-compat generates attrsets like

  nix-shell -A devShells.x86_64-linux.native-clangStdenvPackages

instead. These are unwieldy to use, so rather than changing the docs,
this changes the shell.nix file to result in the documented behavior.

Change-Id: I0920ccbdfddacf371f0aeaae7e290db65cf76ee7
2024-05-27 16:47:40 +02:00
Qyriad 2cd1ef2201 Merge "libfetchers: fix URL logging" into main 2024-05-27 14:20:15 +00:00
Qyriad ec5039653d Merge "Revert "bump the extra --version info from info to notice, -vv -> -v"" into main 2024-05-26 21:51:05 +00:00
eldritch horrors ed6b3165ea worker: process timeouts first, and exclusively
check goals for timeouts first, and their activity fds only if no
timeout has occurred. checking for timeouts *after* activity sets
us up for assertion failures by running multiple build completion
notifiers, the first of which will kill/reap the the goal process
and consuming the Pid instance. when the second notifier attempts
to do the same it will core dump with an assertion failure in Pid
and take down not only the single goal, but the entire daemon and
all goals it was building. luckily this is rare in practice since
it requires a build to both finish and time out at the same time.

writing a test for this is not feasible due to how much it relies
on scheduling to actually trigger the underlying bug, but on idle
machines it can usually be triggered by running multiple sleeping
builds with timeout set to the sleep duration and `--keep-going`:

    nix-build --timeout 10 --builders '' --keep-going -E '
      with import <nixpkgs> {};
      builtins.genList
        (i: runCommand "foo-${toString i}" {} "sleep 10")
        100
    '

Change-Id: I394d36b2e5ffb909cf8a19977d569bbdb71cb67b
2024-05-26 20:57:09 +00:00
Qyriad 742c62a6eb docstrings: NixRepl::getDerivationPath: exceptions directly thrown
getDerivationPath() directly throws nix::Error for invalid derivations

Change-Id: I81ead950060b789794fa683b61c6349fece1690d
2024-05-26 17:19:26 +00:00
Qyriad 9f16a20f3d Revert "bump the extra --version info from info to notice, -vv -> -v"
This reverts commit d003dcd7f4.

d0390b5cf¹ bumped the default verbosity of nix3 commands, making this
change obsolete.

[1]: d0390b5cf2

Change-Id: I1ddc36ea4514cd8d2855ff334accf6e98c210879
2024-05-26 11:15:10 -06:00
Maximilian Bosch 5986a720d4 Merge "git putFile: support flake maximalists" into main 2024-05-26 10:16:40 +00:00
Qyriad ebd00b2d0b libfetchers: fix URL logging
8c06b7b43¹ made libfetchers log the URL being fetched just before the
actual fetch, particularly in case something freezes. This used the base
URL, to not include query parameters, as the Nixpkgs lib tests assume
that stderr logs will be equal across shallow and non-shallow git
fetches (and shallow fetches have the ?shallow=1 query parameter).

8c06b7b43 assumed that the `base` field of ParsedURL would be populated,
as the comment simply says "URL without query/fragment"... but
apparently it is not populated when the URL being fetched is *already*
fetched, which caused libfetchers to log things like

fetching gitlab input ''

which is. silly. but you know, busted lix be busted.

Anyway, with this commit we just remove the query params before printing
instead, which seems to do the right thing

[1]: 8c06b7b431

Change-Id: I9b9988992029aa6abef786f20b66e68c2ebb97d4
2024-05-25 19:37:38 -06:00
Artemis Tosini 53e2b0740c
libstore: remove duplicate builder variable
The `builder` local variable and duplicate `args.push_back` are no
longer required since the Darwin sandbox stopped using `sandbox-exec`.
The `drv->isBuiltin` check is not required either, as args are not
accessed when the builder is builtin.

Change-Id: I80b939bbd6f727b01793809921810ff09b579d54
2024-05-25 22:14:55 +00:00
jade dd53bce476 Merge "libstore: parse the buildMode instead of unchecked cast" into main 2024-05-25 17:42:09 +00:00
alois31 ddfe379a6b Merge "libstore/build: always enable seccomp filtering and no-new-privileges" into main 2024-05-25 04:21:53 +00:00
jade 2a7a824d83 libstore: parse the buildMode instead of unchecked cast
Change-Id: Icf6af7935e8f139bef36b40ad475e973aa48855c
2024-05-24 20:45:05 -06:00
Qyriad 5b4b216fac Merge "nix3-upgrade-nix: fix when there are differing pnames" into main 2024-05-25 02:20:24 +00:00
Qyriad 1c0f3c540e Merge changes I3ab84cc5,Iba34ad42 into main
* changes:
  nix3: always use the same verbosity default (info)
  libfetchers: log fetches by URL just before they happen
2024-05-25 01:52:44 +00:00
Qyriad f3f68fcfac nix3-upgrade-nix: fix when there are differing pnames
Change-Id: I19c7e24a4d46137127e76b7bb133e0184d73d1b6
2024-05-25 01:50:32 +00:00
Qyriad 076c19e0d1 change "evaluating file" logs to debug
I can't imagine wanting this unless you are debugging something (in
which case it's very useful)

Change-Id: I90c6f182c18486e9f6b15a59379bbb8e88fb8e7f
2024-05-24 23:03:08 +00:00
alois31 f047e4357b libstore/build: always enable seccomp filtering and no-new-privileges
Seccomp filtering and the no-new-privileges functionality improve the security
of the sandbox, and have been enabled by default for a long time. In
lix-project/lix#265 it was decided that they
should be enabled unconditionally. Accordingly, remove the allow-new-privileges
(which had weird behavior anyway) and filter-syscall settings, and force the
security features on. Syscall filtering can still be enabled at build time to
support building on architectures libseccomp doesn't support.

Change-Id: Iedbfa18d720ae557dee07a24f69b2520f30119cb
2024-05-24 21:19:29 +00:00
Qyriad d0390b5cf2 nix3: always use the same verbosity default (info)
Change-Id: I3ab84cc583e3e8b1c05a8ae1a7a087204f513d03
2024-05-24 15:15:42 -06:00
Qyriad 8c06b7b431 libfetchers: log fetches by URL just before they happen
Addresses but does not close #305, as we still need an indicator for
frozen fetches.

Change-Id: Iba34ad42dc1c8772f7da249b90fe794b041bbf73
2024-05-24 15:15:42 -06:00
jade 19ea351642 Merge "clang-tidy: work with angle brackets and external projects" into main 2024-05-24 19:11:26 +00:00
Qyriad adfc22c3e3 Merge "make CTRL+Z work in the REPL" into main 2024-05-24 17:37:33 +00:00
Qyriad 933f1f48a2 Merge "justfile: remove --quiet from just test" into main 2024-05-24 17:37:25 +00:00
Qyriad 65da3e7199 make CTRL+Z work in the REPL
Editline just wasn't being built with --enable-sigstop lol

Change-Id: I35a78f74ea100d97f26b2b41990deb373fd9cd9a
2024-05-24 03:10:12 +00:00
Qyriad 2b397c6629 Merge "add editorconfig for markdown" into main 2024-05-24 03:08:25 +00:00
jade 745b5d3d4f clang-tidy: work with angle brackets and external projects
Also fix the readme

Change-Id: I422dff5536bf01d43983621aa01035bd77ac0252
2024-05-24 02:22:58 +00:00
Qyriad e1d2fb4a65 add editorconfig for markdown
Change-Id: I493fc37fde425fc5c5c24f9b077bdc235271233c
2024-05-24 02:22:44 +00:00
Qyriad bb6d43b63b justfile: remove --quiet from just test
`meson test` refuses to let `--verbose` (which shows the entire
invocation and stdio) override `--quiet`, but if neither are specified
in the justfile then you can use either `just test -q` or `just test -v`

Change-Id: I449e13084ce64666b7ee2ab4280818782fb8185a
2024-05-24 02:22:25 +00:00
Pierre Bourdon 6922d67eb3 package.nix: add meta.mainProgram
nix-repl> lib.getExe (builtins.getFlake ".").packages.x86_64-linux.nix
"/nix/store/ajps2zn5hlap0l2abvnfdaphg8k0789r-lix-2.90.0pre20240524_dirty/bin/nix"

Fixes #316.

Change-Id: Iec3125dc2dc99d100beb6357f7d1555456924ddc
2024-05-24 02:15:45 +00:00
jade d1fa446454 [emergency revert]: make pname nix once more
Embarrassingly, I submitted a CL overriding submit requirements since
I thought it was spurious failures. However, the CI failure was in fact
real, and I have hopefully learned my lesson. The CI failure is that:

```
vm-test-run-nix-upgrade-nix> machine # installing 'nix-2.18.1'
vm-test-run-nix-upgrade-nix> machine # building '/nix/store/2b6fdf7wvahd00bg2ff0393bhd597a0h-user-environment.drv'...
vm-test-run-nix-upgrade-nix> machine # error: Unable to build profile. There is a conflict for the following files:
vm-test-run-nix-upgrade-nix> machine #
vm-test-run-nix-upgrade-nix> machine #          /nix/store/dn6mhhr92bh3ad0n4pd1538ww88khjii-nix-2.18.1/lib/libboost_context.so
vm-test-run-nix-upgrade-nix> machine #          /nix/store/w4vffn9iq0znk8bcg5i2giij90xy6db6-lix-2.90.0pre20240523_c97e171/lib/libboost_context.so
vm-test-run-nix-upgrade-nix> machine # error: builder for '/nix/store/2b6fdf7wvahd00bg2ff0393bhd597a0h-user-environment.drv' failed with exit code 1
vm-test-run-nix-upgrade-nix> machine # error: program '/nix/store/w4vffn9iq0znk8bcg5i2giij90xy6db6-lix-2.90.0pre20240523_c97e171/bin/nix-env' failed with exit code 100
```

This is definitely caused by the pname not being the same, so we had
better revert that part of the change until we know we won't regress
anything by doing this.

Fixes: https://gerrit.lix.systems/c/lix/+/1152/5
Change-Id: I0e9d573987f2819c106fb7cea87410fa75152274
2024-05-24 01:33:42 +00:00
Pierre Bourdon 6ade981476 Merge "nixos/tests: enable remoteBuilds tests against Nix 2.18" into main 2024-05-24 00:55:25 +00:00
Qyriad 00bf2b105d Merge changes I462a8cf0,I3b0bcea3,I2acd56e7,Ifc149764,I9e2ef170 into main
* changes:
  docs: linkify nix3-build mention in nix-build.md
  build: make internal-api-docs PHONY
  cleanup lookupFileArg
  add docstring to lookupFileArg
  add libcmd test for lookupFileArg
2024-05-23 23:28:42 +00:00
jade 9530b7f2b2 Merge "packaging: rename nixexpr -> lixexpr and so on" into main 2024-05-23 23:19:28 +00:00
jade c97e17144e packaging: rename nixexpr -> lixexpr and so on
This breaks downstreams linking to us on purpose to make sure that if
someone is linking to Lix they're doing it on purpose and crucially not
mixing up Nix and Lix versions in compatibility code.

We still need to fix the internal includes to follow the same schema so
we can drop the single-level include system entirely. However, this
requires a little more effort.

This adds pkg-config for libfetchers and config.h.

Migration path:
expr.hh      -> lix/libexpr/expr.hh
nix/config.h -> lix/config.h

To apply this migration automatically, remove all `<nix/>` from
includes, so: `#include <nix/expr.hh>` -> `#include <expr.hh>`. Then,
the correct paths will be resolved from the tangled mess, and the
clang-tidy automated fix will work.

Then run the following for out of tree projects:

```
lix_root=$HOME/lix
(cd $lix_root/clang-tidy && nix develop -c 'meson setup build && ninja -C build')
run-clang-tidy -checks='-*,lix-fixincludes' -load=$lix_root/clang-tidy/build/liblix-clang-tidy.so -p build/ -fix src
```

Related: lix-project/nix-eval-jobs#5
Fixes: lix-project/lix#279
Change-Id: I7498e903afa6850a731ef8ce77a70da6b2b46966
2024-05-23 16:45:23 -06:00
Qyriad a0172dc81b docs: linkify nix3-build mention in nix-build.md
Change-Id: I462a8cf0da42b5045ce84b48dc1841ecdccbb89e
2024-05-23 14:11:10 -06:00
Qyriad a8b2fc6d41 build: make internal-api-docs PHONY
Since we're skipping Meson's dependency tracking, for the
internal-api-docs custom target, we should just consider it a phony
target and build it on every request.

Change-Id: I3b0bcea30ee9a4830023ccc5bededf995e96cccc
2024-05-23 14:11:10 -06:00
Qyriad a0dcfbb084 cleanup lookupFileArg
Change-Id: I2acd56e7a542b12138f43c95af78fdd50e944619
2024-05-23 14:11:10 -06:00
Qyriad 3a597f1d0b add docstring to lookupFileArg
Change-Id: Ifc149764f5a15725d3d630677c6da29def4b0f3e
2024-05-23 14:11:10 -06:00
Qyriad 0565f97e78 add libcmd test for lookupFileArg
Change-Id: I9e2ef170ffe916f902daec8b5630d29434c5d5f2
2024-05-23 14:11:10 -06:00
Pierre Bourdon 6260563bed
nixos/tests: enable remoteBuilds tests against Nix 2.18
Fixes #321.

Change-Id: I60812aec9f9b68ab742413835c581d3b53432b9b
2024-05-23 21:16:40 +02:00
Qyriad 677cf75473 repl: log errors writing to history file
These errors are now logged and explicitly ignored, rather than
implicitly ignored.

Change-Id: Ia26015466a17f2b11952df5317a4d150d79dc184
2024-05-23 18:10:23 +00:00
Pierre Bourdon 79121e0c44 Merge "repl: do not crash when tab-completing import errors" into main 2024-05-23 10:48:38 +00:00
eldritch horrors 829521b91a Merge "doc: sort change-authors.yml" into main 2024-05-23 02:35:53 +00:00
eldritch horrors f281727e70 Merge "libstore: remove unused copyPath function" into main 2024-05-23 02:31:50 +00:00
eldritch horrors 9a75150d19 doc: sort change-authors.yml
this should make it easier to spot future instances of entries being
duplicated by accident. also add a pre-commit check to remain sorted

Change-Id: I500caf862e93480b38c9d51144273bb2dcab1af0
2024-05-23 02:09:05 +00:00
Pierre Bourdon d8bc3bfb6d
repl: do not crash when tab-completing import errors
File not found while importing causes a SysError, not an EvalError,
which is not currently caught by the tab-completion handler. Ignoring
all SysErrors might seem "dangerous" but this is the tab-completion
handler, any exception being bubbled up from there causes unexpected
behavior (causes the whole repl to exit).

Fixes #340.

Change-Id: I643048a47935e77f582decc539d9e51bdb96c890
2024-05-23 02:52:54 +02:00
jade d05e0b9f1f fix: enlarge envSize by several times to not be close to nixpkgs size
nixpkgs has 23000 attributes, and our previous limit would be hit if you
have more than one nixpkgs in the environment, for example, because
`repl-overlays` will load the new stuff from the environment on top of
the existing environment.

This is not really testable since if we did write such a test, it would
just be testing this constant tbh...

Fixes: lix-project/lix#337
Change-Id: I49197bfb4db55b082f914f0d70e84f5f5f110954
2024-05-22 17:08:37 -07:00
Qyriad 06c1375e52 Merge "libfetchers: fallback to memory SQLite if fs IO fails" into main 2024-05-22 21:17:46 +00:00
jade dcc7ea5498 release notes: add a bunch of them
Also fix typos introduced by the commits I read.

I have run the addDrvOutputDependencies release note past Ericson since
I was confused by what the heck it was doing, and he was saying it was
reasonable.

Change-Id: Id015353b00938682f7faae7de43df7f991a5237e
2024-05-22 21:13:56 +02:00
alois31 0bf4c2971f Merge "doc: fix repl-interrupt release note entry" into main 2024-05-22 18:59:54 +00:00
eldritch horrors 47523944c5 libstore: remove unused copyPath function
Change-Id: Ibfb0848d8b337bd38947a745c240838865cea401
2024-05-22 18:26:01 +02:00
Qyriad 6881476232 libfetchers: fallback to memory SQLite if fs IO fails
nix::fetchers::CacheImpl uses $XDG_CACHE_HOME, or its default based on
$HOME, to store its SQLite database. If the current process can't write
to that directory for whatever reason, though, any eval-time fetching
would fail just initializing the cache.

With this change, IO errors initializing the fetcher cache are logged
but ignored, and nix::fetchers::CacheImpl falls back to an in-memory¹
database instead.

Notably, this will fix any uses eval fetching while Lix itself is being
run in a derivation builder (such as during tests), as the derivation
builder does not set $XDG_CACHE_HOME, and sets $HOME to the non-existent
directory /homeless-shelter.

Before:

$ env -u XDG_CACHE_HOME HOME=/homeless-shelter nix -Lv eval --impure -E 'fetchTarball "https://git.lix.systems/lix-project/lix/archive/main.tar.gz"'
error:
       … while calling the 'fetchTarball' builtin
         at «string»:1:1:
            1| fetchTarball "https://git.lix.systems/lix-project/lix/archive/main.tar.gz"
             | ^

       error: creating directory '/homeless-shelter': Permission denied

After:

$ env -u XDG_CACHE_HOME HOME=/homeless-shelter nix -Lv eval --impure -E 'fetchTarball "https://git.lix.systems/lix-project/lix/archive/main.tar.gz"'
warning: ignoring error initializing Lix fetcher cache: error: creating directory '/homeless-shelter': Permission denied
"/nix/store/s9lxdnn0awp37n560bg4fgr497ah4hvw-source"

¹: https://www.sqlite.org/inmemorydb.html

Change-Id: I15c38c9baaf215fc6e192b8a4c70b9692a69bc22
2024-05-21 09:30:25 -06:00
alois31 d5fdb995d3
doc: fix repl-interrupt release note entry
The timing of the merge resulted in the newly introduced metadata not being
present.

Change-Id: I07f28cf37703ec05c3e1b96301797a42d913264b
2024-05-21 16:34:04 +02:00
Qyriad 20981461d4 print type and value in "flake attr is not a derivation" errors
This turns errors like:

error: flake output attribute 'hydraJobs' is not a derivation or path

into errors like:

error: expected flake output attribute 'hydraJobs' to be a derivation or
path but found a set: { binaryTarball = «thunk»; build = «thunk»; etc> }

This change affects all InstallableFlake commands.

Change-Id: I899757af418b6f98201006ec6ee13a448c07077c
2024-05-21 05:55:13 -06:00
Artemis Tosini 3de77e6dbd Merge "libutil: Create chmodPath function" into main 2024-05-20 15:13:53 +00:00
raito 8e1a883186 Merge "chore: remove incorrect maintainers/*.md documentation" into main 2024-05-20 12:35:20 +00:00
Graham Christensen 8220da8a53
git putFile: support flake maximalists
Passing the commit message as an argument causes update failures on repositories with
lots of flake inputs. In some cases, the commit message is over 250,000 bytes.

Upstream PR: https://github.com/NixOS/nix/pull/10686

(cherry picked from commit 8b5e8f4fba5728f2b3e90fcd1ab15df77e3ea0e8)
Change-Id: I2c196a21cc9bedc24d57a828a0c5b9467e072f76
2024-05-20 13:51:29 +02:00
jade 992c63fc0b Merge "Remove upload-release.pl" into main 2024-05-20 00:38:12 +00:00
Qyriad 589953e832 Merge "fix -Wdeprecated-copy on clang (BaseError copy assignment)" into main 2024-05-20 00:11:12 +00:00
puck bfb91db4f6 repl-interacter: save history after entering every line
Fixes: lix-project/lix#328
Change-Id: Iedd79ff5f72e84766ebd234c63856170afc624f0
2024-05-19 22:47:45 +00:00
puck 40311973a8 change-authors: add puck
Change-Id: I04b8cd04a168b3adea7790f816e774d5d90fcea2
2024-05-19 22:47:45 +00:00
Artemis Tosini 5411fbf204
libutil: Create chmodPath function
Move the identical static `chmod_` functions in libstore to
libutil. the function is called `chmodPath` instead of `chmod`
as otherwise it will shadow the standard library chmod in the nix
namespace, which is somewhat confusing.

Change-Id: I7b5ce379c6c602e3d3a1bbc49dbb70b1ae8f7bad
2024-05-19 22:07:58 +00:00
jade a354779d78 Remove upload-release.pl
We are doing releases totally differently than Nix so this will need
rewriting anyway.

Change-Id: Iba4ad160b9d215fcbf20a14243fd87cfbb527760
2024-05-19 13:53:39 -07:00
Qyriad 4eb6779ea8 fix -Wdeprecated-copy on clang (BaseError copy assignment)
2bbe3efd1¹ added the -Wdeprecated-copy warning, and fixed the instances
of it which GCC warned about, in HintFmt and ref<T>. However, when
building with Clang, there is an additional deprecated-copy warning in
BaseError. This commit explicitly defaults the copy assignment operator
for BaseError and silences this warning.

1: 2bbe3efd16
Change-Id: I50aa4a7ab1a7aae5d7b31f765994abd3db06379d
2024-05-19 12:32:13 -06:00
raito 93dbb698b3 chore: remove incorrect maintainers/*.md documentation
Fate has something different in store for the release process,
backporting process and the general maintainer documentation.

See lix-project/lix#260.

Change-Id: I626686ff4059aee22a3ab1664b52581b2dbf6ed7
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-05-19 16:58:52 +02:00
eldritch horrors 774c56094f libstore: fix old RemoteStore::addToStore serializer
having the serializer write into `*conn` is not legal because we are
in a sinkToSource that will be drained by the remote we're connected
to. writing into `*conn` directly can break the framing protocol. it
is unlikely this code was ever run: to protocol it caters to is from
2016(!) and thoroughly untested in-tree, and since it's been present
since nix 2.17 and the 1.18 protocol broken here is nix 2.0 we might
safely assume that daemons older than nix 2.1 are no longer used now

see also #325 (though that wants <2.3 gone, this is sadly only <2.1)

Change-Id: I9d674c18f6d802f61c5d85dfd9608587b73e70a5
2024-05-19 11:57:55 +00:00
Alyssa Ross 139d31f876 Improve nix-store --delete failure message
On several occasions I've found myself confused when trying to delete
a store path, because I am told it's still alive, but
nix-store --query --roots doesn't show anything.  Let's save future
users this confusion by mentioning that a path might be alive due to
having referrers, not just roots.

(cherry picked from commit 979a019014569eee7d0071605f6ff500b544f6ac)

Upstream-PR: https://github.com/NixOS/nix/pull/10733
Change-Id: I54ae839a85f3de3393493fba27fd40d7d3af0516
2024-05-18 14:49:40 -06:00
puck 62b1adf8c1 Merge "nix cat/dump-path/key: stop progress bar before writeFull" into main 2024-05-18 20:13:48 +00:00
jade d7d1547a41 Merge "lix-doc: don't chomp bold headings off" into main 2024-05-18 18:24:49 +00:00
puck 1fe58bd8a7 nix cat/dump-path/key: stop progress bar before writeFull
These commands outputs data that may not end with a newline. This
causes problems when the progress bar redraws, as that completely
wipes the last line of output. As nix key generate-secret outputs
a single line of text with no output, it shows up entirely blank,
making it look like nothing happened.

Fixes: lix-project/lix#320
Change-Id: I5ac706d71d839b6dfa760b60a351414cd96297cf
2024-05-18 17:51:16 +00:00
Pierre Bourdon d1c8fd3b09 Merge "derived-path: refuse built derived path with a non-derivation base" into main 2024-05-18 07:26:26 +00:00
julia 7a3745b076
Deprecate the online flake registries and vendor the default registry
Fixes #183, #110, #116.

The default flake-registry option becomes 'vendored', and refers
to a vendored flake-registry.json file in the install path.

Vendored copy of the flake-registry is from github:NixOS/flake-registry
at commit 9c69f7bd2363e71fe5cd7f608113290c7614dcdd.

Change-Id: I752b81c85ebeaab4e582ac01c239d69d65580f37
2024-05-18 12:27:23 +10:00
Qyriad 236466faf3 package: add --print-errorlogs to meson's tests
This should have been in there originally, which is our mistake,
considering that debugging CI failures is basically impossible without
it.

Change-Id: I4ab8799e6e0abca1984ed9801fe10c58200861a3
2024-05-17 21:42:33 +00:00
puck 23c92f0815 Merge "primops: change to std::function, allowing the passing of user data" into main 2024-05-17 21:37:41 +00:00
puck 92e1df23b3 Merge "Loosen constness on listElems() result" into main 2024-05-17 21:37:35 +00:00
jade 0d2cc81956 Merge "make lix dev shells un-bear-able since we un-make them now" into main 2024-05-17 20:44:02 +00:00
Qyriad 93b7edfd07 Merge "docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation" into main 2024-05-17 18:17:05 +00:00
jade e1119f4378 make lix dev shells un-bear-able since we un-make them now
We don't need bear anymore, since we don't have any more bad build
systems that lack compile commands generation inside Lix.

Change-Id: I7809ddfd993180468f846e8cd862bdd54d5b31ec
2024-05-16 23:43:44 -07:00
julia 005ee33a9a Merge "Allow enabling core dumps from builds for nix & child processes" into main 2024-05-17 01:10:44 +00:00
Qyriad 5ff076d8ad docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation
Both of these still needs their own actual documentation, but they are
at least now mentioned that they exist and what they're enabled by.

Change-Id: I235b9e8e627e04ed06611423c8e67a8eca233120
2024-05-17 00:41:35 +00:00
Pierre Bourdon 5a1824ebe1
derived-path: refuse built derived path with a non-derivation base
Example: /nix/store/dr53sp25hyfsnzjpm8mh3r3y36vrw3ng-neovim-0.9.5^out

This is nonsensical since selecting outputs can only be done for a
buildable derivation, not for a realised store path. The build worker
side of things ends up crashing with an assertion when trying to handle
such malformed paths.

Change-Id: Ia3587c71fe3da5bea45d4e506e1be4dd62291ddf
2024-05-17 02:16:15 +02:00
julia 5b7dcb3005 Allow enabling core dumps from builds for nix & child processes
Fixes lix-project/lix#268

Change-Id: I3f1b0ddf064f891cca8b53229c5c31c74cea3d9f
2024-05-16 17:11:21 -07:00
Qyriad 5756be19f6 Merge "add clarifying doc-comments on {get,create}NixStateDir()" into main 2024-05-17 00:10:38 +00:00
jade 9322a1cbe7 Merge "builtins: fix builtins.langVersion docs to state it's deprecated" into main 2024-05-16 21:19:15 +00:00
Yorick 194654c96f primops: change to std::function, allowing the passing of user data
(cherry picked from commit 48aa57549d514432d6621c1e29f051951eca2d7f)
Change-Id: Ib7d5c6514031ceb6c42ac44588be6b0c1c3c225b
2024-05-16 13:01:40 +00:00
puck c6bb377c91 Loosen constness on listElems() result
Change-Id: I1caff000362c83e5172413a036c22a2e9ed3ede8
2024-05-16 13:01:40 +00:00
jade 6d84d6bdc1 builtins: fix builtins.langVersion docs to state it's deprecated
This builtin was always a problem and nixpkgs uses it in exactly one
place, to give up if the Nix version is absurdly old. It has no other
use cases, and doesn't work in a multi-implementation world anyway.

Change-Id: I03c36e118591029e2ef14b091fe14a311c66a08a
2024-05-15 21:54:12 -07:00
Mel Zuser 9249c89dc6 build: fix build when gc is disabled
Change-Id: I8d3eb8874a4138668011b525c3b400a55a1f4866
2024-05-15 18:48:00 -07:00
FireFly eca8bce081 lix-doc: don't chomp bold headings off
There are a few places in nixpkgs lib where `**Foo**:` is used as a heading instead of the usual markdown `# Foo` ones. I think this is intentional with how it gets rendered in the manual, e.g. [`lib.lists.sortOn`][1].

[1]: https://nixos.org/manual/nixpkgs/stable/#function-library-lib.lists.sortOn

`nix-doc` prints this as
```
   *Laws**:
       ```nix
       sortOn f == sort (p: q: f p < f q)
       ```
```
chomping off the first asterisk as part of `cleanup_single_line` that's meant to deal with `/** \n * \n * \n */` style doc comments. This also means the usage in lix ends up funny-looking with a trailing asterisk as if there's a footnote to pay attention to (which is how I first noticed it, heh)

The fix:

When cleaning up a single line and removing a prefix comment character,
ensure it's followed by whitespace (or the last character of the line).

Upstream-PR: https://github.com/lf-/nix-doc/pull/26
Change-Id: If2870c53a632f6bbbcca98a4bfbd72f5bef37879
2024-05-15 15:24:03 -07:00
jade 7cfaf057e3 release-notes: build unreleased release notes by default
Surely if you have unreleased changes you want them on a page right?

`officialRelease` means "this is a *release version*", which is a
reasonable case to not want it, but we are not that here.

I understand wanting to be able to turn it off for deps reasons or
something, but other than that, uhh, seems better to just turn it on
always; it is basically free compute-wise to the point we run it on
pre-commit.

Part two of fixing lix#297.

Fixes: lix-project/lix#297
Change-Id: I0f8dd1ae42458df371aef529c456e47a7ac04ae0
2024-05-15 15:01:38 -07:00
jade 2a7b3d7c94 build-release-notes: add change author metadata and use it
Change-Id: I6f5fb54f70b02a467bbdee4c526f59da1193f7db
2024-05-15 14:33:35 -07:00
jade c8329e5667 build-release-notes: add change author info DB
This allows us to have links to peoples' GitHub and Forgejo profiles.

I used YAML because I don't want to introduce a dependency on having a
working Nix evaluator to be able to build release notes, and we already
have a YAML parser in this script.

Change-Id: Idf2813f79e0407460c796cba6c383496465e152d
2024-05-15 14:33:35 -07:00
jade 272db8af1b doc: add release note credits and categories for all the changes in Lix
This does not add missing release notes, and it doesn't do anything
about the profiles feature we would really like to have so we can have
consistent credit.

Change-Id: I72a6f7acfcff85f380be17dac76501a6f4693776
2024-05-15 14:33:35 -07:00
jade 79ee9355ae build-release-notes: support categories
Change-Id: Icdcbd2cf5bf075e7006ce1f8bc0c9d6c35cfd072
2024-05-15 13:27:31 -07:00
jade 03655c310d build-release-notes: fail if the directory does not exist
This was a combination of two problems: the python didn't throw an error
because apparently glob on a nonexistent directory doesn't crash, and
secondarily, bash ignores bad exit codes without `set -e` if they are
not in the final/only command.

Change-Id: I812bde7a4daee5c77ffe9d7c73a25fd14969f548
2024-05-15 20:18:50 +00:00
alois31 f6397cc286 doc: fix rl-next build
Use the correct directory for the rl-next build, so that the release notes
actually get built and the page doesn't end up empty. I don't know why the
exception didn't cause a build failure before.

Fixes: lix-project/lix#297
Change-Id: Ic72b9bb4c0d2d1f633f2af90cce4a3a2796d7f9b
2024-05-15 20:18:50 +00:00
jade f5828f2ee0 Merge "feat: add credits field to release note generator" into main 2024-05-15 20:18:21 +00:00
Maximilian Bosch aabc902040 Merge "nix3-build: show all FOD errors with --keep-going" into main 2024-05-15 18:46:06 +00:00
alois31 0903a99bad Merge changes I8456c47b,I48253f5f into main
* changes:
  repl: clear the interrupt before reading the next line
  libutil: remove the interrupt-blocking code
2024-05-15 15:46:11 +00:00
Maximilian Bosch e146393183
nix3-build: show all FOD errors with --keep-going
Basically I'd expect the same behavior as with `nix-build`, i.e.
with `--keep-going` the hash-mismatch error of each failing
fixed-output derivation is shown.

The approach is derived from `Store::buildPaths` (`entry-point.cc`):
instead of throwing the first build-result, check if there are any build
errors and if so, display all of them and throw after that.

Unfortunately, the BuildResult struct doesn't have an `ErrorInfo`
(there's a FIXME for that at least), so I have to construct my own here.
This is a rather cheap bugfix and I decided against touching too many
parts of libstore for that (also I don't know if that's in line with the
ongoing refactoring work).

Closes lix-project/lix#302

Change-Id: I378ab984fa271e6808c6897c45e0f070eb4c6fac
2024-05-15 15:35:18 +02:00
raito c52eba4424 feat: add credits field to release note generator
Now, we can credit folks for their work.

The credit generator is very basic, we probably want a database of
profiles and link to their preferred page or something.

Change-Id: Ida81905750371e5e125d0ce7e554d0526265cf8e
Co-Authored-By: Jade Lovelace <lix@jade.fyi>
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-05-14 19:26:09 -07:00
jade 312f66b307 Merge "doc: add a script to upload the nightly manual manually" into main 2024-05-15 00:19:05 +00:00
jade 58ff8960cd doc: add a script to upload the nightly manual manually
This is not like, perfect, since it is a manual operation, but we can
automate it in the future. rclone is used, since it seems like awscli is
not (obviously at least?) able to sync directories such that old things
are deleted, and rclone does this thing properly.

Fixes: lix-project/meta#2
Change-Id: Ia6a46d861342a6d29b22f981ba4e35e79f79e60e
2024-05-14 14:28:01 -07:00
alois31 eeb7e71810
repl: clear the interrupt before reading the next line
Otherwise, it will be thrown again during exit when the repl is terminated by
end-of-input after the last command was interrupted.

Change-Id: I8456c47bc36cfb0892efdad5420f318f7e6526d5
2024-05-13 09:04:05 +02:00
alois31 914b0febf7
libutil: remove the interrupt-blocking code
The interrupt-blocking code was originally introduced 20 years ago so that
trying to log an error message does not result in an interrupt exception being
thrown and then going unhandled (c8d3882cdc).
However, the logging code does not check for interrupts any more
(054be50257), so this reasoning is no longer
applicable. Delete this code so that later interrupts are unblocked again, for
example in the next line entered into the repl.

Closes: lix-project/lix#296
Change-Id: I48253f5f4272e75001148c13046e709ef5427fbd
2024-05-13 09:03:53 +02:00
Pierre Bourdon e9ca5c92d2
tests: don't build test plugin shared libs on static builds
This changes the way plugins.sh is excluded to remove the need for
BUILD_SHARED_LIBS along the way.

Change-Id: I19fe55b4a37c49a11fffa61c8a3be7e8d1a51b4d
2024-05-12 23:04:21 +02:00
Artemis Tosini 4b35e6a75e Merge "libstore: Fix sandbox=relaxed" into main 2024-05-12 03:51:19 +00:00
Qyriad 88d9b70f67 add clarifying doc-comments on {get,create}NixStateDir()
NixStateDir() != NIX_STATE_DIR. These functions should honestly probably
be renamed.

Change-Id: I00f54b742bba6188bbc7f2410956d956780b99d3
2024-05-11 15:40:00 -06:00
Pierre Bourdon a30c567336
filetransfer: unit test content-encoding handling
Very basic behavior test to ensure that gzip data gets internally
decompressed by the file transfer pipeline.

Change a std::string_view return value in the test harness to
std::string. I wouldn't call myself a C++ beginner and I still managed
to shoot myself in the foot like three times with the lifetime
managements there (e.g. [&] { return an_std_string; } ends up with a
dangling string_view!).

Change-Id: I1360750d4181ce1ca2a3aa4dc0e97e131351c469
2024-05-11 15:58:32 +02:00
Pierre Bourdon 38d825b21e
filetransfer: fix decompression regression from 121edecf
121edecf65 added a new state field to
carry over content encoding settings from transfer to sink creation, but
never actually set that field.

Change-Id: I714b2efe745561e851b78a4791479b3501db8c72
2024-05-11 14:49:23 +02:00
Qyriad f6dc40cd1c Merge "bump the extra --version info from info to notice, -vv -> -v" into main 2024-05-10 21:56:09 +00:00
raito 4ebbd4362f Merge "feat(libstore): print the first line of stdout of SSH in case of failure" into main 2024-05-10 19:33:36 +00:00
raito 8404a1f66d feat(libstore): print the first line of stdout of SSH in case of failure
In case of failure to connect as can be seen in
https://buildbot.lix.systems/#/builders/39/builds/1386/steps/1/logs/stdio

It is difficult to understand what happened, if we enabled the talkative
verbose level, we could learn about the first line SSH sent us.

In practice, this is not workable, we can just make it warn all the
time.

Change-Id: Iaaf56894060a58f2dfc78254bb60b1c43482f9bb
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-05-10 20:22:47 +02:00
Qyriad d003dcd7f4 bump the extra --version info from info to notice, -vv -> -v
requiring *two* --verbose to print extra information on --version is a
weird flex

Change-Id: I05d043da1bf583f34e9d1fc206144ea4ca9a859d
2024-05-10 11:55:17 -06:00
eldritch horrors ceccac835c libutil: remove callback.hh
it's no longer used. it really shouldn't have existed this long since it
was just a mashup of both std::promise and std::packaged_task in a shape
that makes composition unnecessarily difficult. all but a single case of
Callback pattern calls were fully synchronous anyway, and even this sole
outlier was by far not important enough to justify the extra complexity.

Change-Id: I208aec4572bf2501cdbd0f331f27d505fca3a62f
2024-05-10 02:21:11 +02:00
eldritch horrors b66451ae7f libstore: de-callback-ify FileTransfer
also add a few more tests for exception propagation behavior. using
packaged_tasks and futures (which only allow a single call to a few
of their methods) introduces error paths that weren't there before.

Change-Id: I42ca5236f156fefec17df972f6e9be45989cf805
2024-05-10 02:21:11 +02:00
eldritch horrors 28a98d152c libstore: de-callback-ify Store::queryRealisation
Change-Id: I8d74745c519518f163f51dfaa39063836f17599e
2024-05-09 23:18:05 +02:00
eldritch horrors 17965bf11c libstore: un-callback-ify Store::queryRealisationUncached
Change-Id: I4a328f46eaac3bb8b19ddc091306de83348be9cf
2024-05-09 23:18:05 +02:00
eldritch horrors 2f4a1dd6e0 libstore: de-callback-ify Store::queryPathInfoUncached
Change-Id: I23a156aaff5328f67ca16ccd85c0ea1711b21e35
2024-05-09 23:18:05 +02:00
eldritch horrors c77bd88259 libstore: de-callback-ify BinaryCacheStore::getFile
Change-Id: I36b3eb9f645aa04058151e7b2353e15e6f29057b
2024-05-09 23:18:05 +02:00
eldritch horrors 1a002d1a11 libstore: de-callback-ify CA realisation substitution
this is the *only* real user of file transfer download completion
callbacks, and a pretty spurious user at that (seeing how nothing
here is even turned on by default and indeed a dependency of path
substitution which *isn't* async, and concurrency-limited). it'll
be a real pain to keep this around, and realistically it would be
a lot better to overhaul substitution in general to be *actually*
async. that requires a proper async framework footing though, and
we don't have anything of the sort, but it's also blocking *that*

Change-Id: I1bf671f217c654a67377087607bf608728cbfc83
2024-05-09 23:18:05 +02:00
Qyriad 9ae90612a7 Merge changes I4bffa766,If2561cd5 into main
* changes:
  add and fix -Wignored-qualifiers
  add and fix -Wdeprecated-copy
2024-05-09 17:52:11 +00:00
jade d5804d0c6d Merge "fix: eval error of .#devShells.x86_64-linux.x86_64-freebsd13" into main 2024-05-09 15:22:03 +00:00
Maximilian Bosch b962a266c4 Merge "flake: update nixpkgs input to latest nixos-23.11" into main 2024-05-09 13:17:21 +00:00
Qyriad 1f9b0fba23 add and fix -Wignored-qualifiers
Change-Id: I4bffa766ae04dd80355f9b8c10e59700e4b406da
2024-05-09 07:08:44 -06:00
jade d1dacad708 fix: eval error of .#devShells.x86_64-linux.x86_64-freebsd13
This is broken and our resident nixbsd maintainers say it should
probably just be temporarily removed till we switch to 24.05 instead of
diagnosing it.

Originally introduced in: https://github.com/nixos/nix/pull/8887

Fixes: lix-project/lix#277
Change-Id: I1e7db8859620024a7b37dbd0cc1c5ec139b9e5cb
2024-05-09 13:07:30 +00:00
Qyriad 2bbe3efd16 add and fix -Wdeprecated-copy
*so* many warnings, from only two definitions

Change-Id: If2561cd500c05a1e33cce984faf9f3e42a8a95ac
2024-05-09 13:02:45 +00:00
Qyriad 010bbd1b0e Merge "nix3-eval: don't elide top-level errors" into main 2024-05-09 11:46:13 +00:00
alois31 da0e1f5716 Merge "repl: show a progress bar while performing builds" into main 2024-05-09 04:32:55 +00:00
Maximilian Bosch a4c943403f
flake: update nixpkgs input to latest nixos-23.11
This includes the update to libseccomp 2.5.5[1], so we don't need to
override it on our own.

[1] https://nixpk.gs/pr-tracker.html?pr=306070

Change-Id: I1fa9c7fcc23e501d75f774745107c6bb086ced70
2024-05-08 23:03:28 +02:00
Patrick Jackson 8552519bb8 Merge "Improve the justfile" into main 2024-05-08 20:54:48 +00:00
Qyriad 54322f09d3 nix3-eval: don't elide top-level errors
Fixes #276.

Change-Id: I83e71beb5c35d6f3b10a4186caa5e52a2f95b510
2024-05-08 13:37:20 -06:00
Théophane Hufschmitt adea821d87
libstore: Fix sandbox=relaxed
The fix for the Darwin vulnerability in ecdbc3b207
also broke setting `__sandboxProfile` when `sandbox=relaxed` or
`sandbox=false`. This cppnix change fixes `sandbox=relaxed` and
adds a suitable test.

Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
Change-Id: I40190f44f3e1d61846df1c7b89677c20a1488522
2024-05-08 19:31:43 +00:00
alois31 243c0f18da
repl: show a progress bar while performing builds
In commit 946fc12e4e, the progress bar in the
repl was disabled again because it was observed to erase incremental output
from attrset evaluations from the terminal. Let's try adding the progress bar
again, this time showing up only when a build is initiated, which does not have
incremental output that could be destroyed to begin with. While this does mean
that we won't have a progress bar for eval-time fetching or IFD, it's still
better than nothing.

Change-Id: If4eb1035cd0c876f5b4ff1e2434b9baf99f150ac
2024-05-08 20:54:41 +02:00
Patrick Jackson 85f51fc10b Improve the justfile
Adds descriptions and a 'list' function that runs as the default

Change-Id: Ifee2c8ccd2694af0ca8bd94744f8be99f91b254a
2024-05-08 08:19:37 -07:00
Qyriad f782c8a60a flake: refactor devShell creation
Now instead of a derivation overridden from Lix, we use a mkShell
derivation parameterized on an already called package.nix. This also
lets callPackage take care of the buildPackages distinction for the
devShell.

Change-Id: I5ddfec40d83fa6136032da7606fe6d3d5014ef42
2024-05-07 18:09:51 -06:00
Qyriad 8822fd7dd5 package: default the build-release-notes arg like we do with lix-doc
Change-Id: I0e2df55efc1cd6ea0a3252b9f26676e84612fdb6
2024-05-07 17:07:53 -06:00
Qyriad b9be46fb31 remove the autoconf+Make buildsystem
We're not using it anymore. Any leftover bugs in the Meson buildsystem
are now just bugs.

Closes #249.

Change-Id: I0465a0c37ae819f94d40e7829f5bff046aa63d73
2024-05-07 17:04:30 -06:00
Patrick Jackson d184981af0 Merge "feat: setup gerrit commit-msg hook with nix develop" into main 2024-05-07 22:51:53 +00:00
Qyriad 8715a0ac4e Merge changes If1077a7b,I62da3161,Iebb4667b into main
* changes:
  flake: fix devShell on i686-linux by disabling ClangBuildAnalyzer on it
  flake: fix eval of checks & devshell on i686-linux
  flake: move the pre-commit definition to its own file
2024-05-07 22:14:22 +00:00
Qyriad 7e940cc170 flake: fix devShell on i686-linux by disabling ClangBuildAnalyzer on it
ClangBuildAnalyzer doesn't build on i686-linux due to
`long long int`/`size_t` conversion errors, so let's just exclude it
from the devshell on that platform

Change-Id: If1077a7b3860db4381999c8e304f6d4b2bc96a05
2024-05-07 15:31:25 -06:00
Nikodem Rabuliński e8a603fb2f Merge changes Icf26010a,Ib6161567 into main
* changes:
  Always initialize curl in parent process on darwin
  Fix failing darwin tests
2024-05-07 21:26:24 +00:00
Patrick Jackson 9af8694367 feat: setup gerrit commit-msg hook with nix develop
Closes #273

Change-Id: Id883d2cda06adbcae53b8c360ad015330f0af81b
2024-05-07 14:20:09 -07:00
Nikodem Rabuliński f894cce79b
Always initialize curl in parent process on darwin
Because of an objc quirk[1], calling curl_global_init for the first time
after fork() will always result in a crash.
Up until now the solution has been to set
OBJC_DISABLE_INITIALIZE_FORK_SAFETY for every nix process to ignore
that error.
This is less than ideal because we were setting it in package.nix,
which meant that running nix tests locally would fail because
that variable was not set.
Instead of working around that error we address it at the core -
by calling curl_global_init inside initLibStore, which should mean
curl will already have been initialized by the time we try to do so in
a forked process.

[1] 01edf1705f/runtime/objc-initialize.mm (L614-L636)

Change-Id: Icf26010a8be655127cc130efb9c77b603a6660d0
2024-05-07 20:43:17 +02:00
Qyriad aac32327d5 flake: fix eval of checks & devshell on i686-linux
Change-Id: I62da3161327051005e3f48f83974140efef4417e
2024-05-07 12:38:01 -06:00
Qyriad 4f98d21b71 flake: move the pre-commit definition to its own file
It's a good hundred LOC, and wasn't coupled to the actual flake logic at
all.

Change-Id: Iebb4667b3197dbd8cb2b019014e99fa651848832
2024-05-07 12:38:01 -06:00
eldritch horrors 964ac8b0e8 libutil: de-callback-ify computeClosure
only two users of this function exist. only one used it in a way that
even bears resemblance to asynchronicity, and even that one didn't do
it right. fully async and parallel computation would have only worked
if any getEdgesAsync never calls the continuation it receives itself,
only from more derived callbacks running on other threads. calling it
directly would cause the decoupling promise to be awaited immediately
*on the original thread*, completely negating all nice async effects.

Change-Id: I0aa640950cf327533a32dee410105efdabb448df
2024-05-07 14:35:20 +00:00
eldritch horrors 230860dbb8 libstore: limit CA realisation info substitution concurrency
this seems to be an oversight, considering that regular substitutions
are concurrency-limited. while not particularly necessary at present,
once we've removed the `Callback` based interfaces it will be needed.

Change-Id: Ide2d08169fcc24752cbd07a1d33fb8482f7034f5
2024-05-07 14:35:20 +00:00
eldritch horrors 29f93e1e0d libutil: throw EndOfFile at sourceToSink end
... how has this never broken anything before

Change-Id: If3789c02028e8f929481514f63d76b0b46bfc182
2024-05-07 14:35:20 +00:00
Qyriad 005b2b61e6 fix fallback chroot store creation after b247ef72d
When /nix/var (or, more precisely, NIX_STATE_DIR) does not exist at all,
Lix falls back to creating an adhoc chroot store in XDG_DATA_HOME.

b247ef72d[1] changed the way Store classes are initialized, and in the
migration, a `params2` was accidentally changed to `params`. This commit
restores the correct behavior, and in lieu of a single *character* fix,
this commit also changes the variable name to something more reasonable.

Fixes #274.

[1]: b247ef72dc

n.b., this code might deserve some more looking at anyway. this fallback
store creation throws away *all* Store params passed to
openFromNonUri() in favor of an entirely new set which only contains
the `root` param, which may or may not be the correct behavior

Change-Id: Ibea559b88a50e6d6e75a1f87d9d7816cabb2a8f3
2024-05-06 20:54:21 -06:00
jade 241b9992fd Merge "fix: readme typo" into main 2024-05-07 01:49:34 +00:00
jade f5ea4cd045 Merge "Remove README.md from build dependencies" into main 2024-05-07 01:49:21 +00:00
jade 4e327b8c60 Merge "chore: re-work the contribution guide" into main 2024-05-07 01:49:04 +00:00
raito 36d69864f3 chore: re-work the contribution guide
As per our bootstrap governance discussions, here's a very simple
proposal which links as much as possible to our wiki.

Change-Id: I88b1c43f933ff7e529151b1e933fad40283383c4
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-05-07 01:28:35 +00:00
jade bf1a22aa8e Remove README.md from build dependencies
Now we will simply hit cache every time anyone changes readme, yay!

Change-Id: I5906f589d319ff6d43cbd2b467887e08f7474283
2024-05-06 18:23:57 -07:00
Abhiram 864ffff2cc fix: readme typo
The text on the matrix link was wrong.

Change-Id: I5d15b27eef0d7d03505920ebf1c03fecbb7faabe
2024-05-06 18:14:48 -07:00
eldritch horrors f75d0752ce filetransfer: correctly abort empty transfers
returning 0 from the callback for errors signals successful transfer if
the source returned no data even though the exception we've just caught
clearly disagrees. while this is not all that important (since the only
viable cause of such errors will be dataCallback, and the sole instance
of it being used already takes care of exceptions) we can just do this.

Change-Id: I2bb150eff447121d82e8e3aa4e00057c40523ac6
2024-05-06 21:18:23 +02:00
eldritch horrors 121edecf65 filetransfer: extract decompressor creation
this will be necessary if we want download() to return a source instead
of consuming a sink, which will in turn be needed to remove coroutines.

Change-Id: I34ec241e9bbc5d32fbcd243b244e29c3757533aa
2024-05-06 20:19:03 +02:00
Nikodem Rabuliński 83a2cd0c46
Fix failing darwin tests
Some tests were failing on darwin,
if the auto-allocate-uids featrure was enabled.
This was because AAU on darwin works by setuid-ing as a non-existent
user, so the tests that were relying on `whoami` were failing.

In the case of trusted-users we fall back to printing the user id,
which is already handled gracefully in the daemon code - i.e. when
a user does not exist or for some other reason looking up their
username is not possible, the daemon falls back to searching for their
uid inside the trusted-users list.

When whoami is used to print the username for other purpose,
we default to printing nixbld.

Change-Id: Ib61615677565098cb5fbf5e26a946ef427c58caf
2024-05-06 18:56:40 +02:00
jade 106b959043 Merge "Fix the pages in the manual for Lix" into main 2024-05-06 04:23:55 +00:00
Qyriad 3f64344bc8 meson: install org.nixos.nix-daemon.plist on macOS
Change-Id: I70079a553ec355be944f9940258f9abf861759fc
2024-05-05 18:33:35 -06:00
jade 748d8310fa Fix the pages in the manual for Lix
This doesn't comprehensively fix everything outdated in the manual, or
make the manual greatly better, but it does note down where at least
jade noticed it was wrong, and it does fix all the instances of
referencing Nix to conform to the style guide to the best of our
ability.

A lot of things have been commented out for being wrong, and there are
three types of FIXME introduced:

- FIXME(Lix): generically Lix needs to fix it
- FIXME(Qyriad): re lix-project/lix#215
- FIXME(meson): docs got outdated by meson changes and need rewriting

I did fix a bunch of it that I could, but there could certainly be
mistakes and this is definitely just an incremental improvement.

Fixes: lix-project/lix#266
Change-Id: I5993c4603d7f026a887089fce77db08394362135
2024-05-05 16:11:01 -07:00
jade 7cffd7a3b5 Merge "gitignore nocontribmsg" into main 2024-05-05 20:40:14 +00:00
jade 4b3e8a68d4 gitignore nocontribmsg
follow-on to https://gerrit.lix.systems/c/lix/+/996

Change-Id: I5b88697d5c802d2788e07387de406e4a4bd351e3
2024-05-05 20:39:54 +00:00
Qyriad a3d4aca836 Merge "README: update to be Lix" into main 2024-05-05 20:18:28 +00:00
Qyriad 7e2a1b438b README: update to be Lix
Change-Id: I15b2513de61cffa2002799c4d12d251ef0970b9f
2024-05-05 19:47:56 +00:00
Qyriad 99903f24eb fix integer overflow on i686 with high phys memory
sizeof(long) is 4 bytes on i686 GCC.
With ~32 GiB of memory and a page size of 4096, there are 7988420 pages.
(7988420 * 4096) is bigger than INT32_MAX folks.

This has gone unnoticed for 9 years, and only came up thanks to
94ea517db[1] adding integer overflow sensitization checks, which caused
this broken code to emit an illegal instruction, crashing Lix the
instant the buildsystem ran Lix to generate the docs files.

[1]: 94ea517dbe729765b69638190f4bea3f6a632b40

Change-Id: I50bb9ea072aac11b449d79e5d55525887a6e5a99
2024-05-05 19:47:46 +00:00
Qyriad 4998699e1a Merge "point nix3-upgrade-nix to releases.lix.systems/manifest.nix" into main 2024-05-05 19:47:36 +00:00
jade 4fd32351f3 Merge "Warn on untrusted client settings being ignored" into main 2024-05-05 19:35:35 +00:00
eldritch horrors 6b08138929 filetransfer: abort transfer on receiver exception
not doing this will cause transfers that had their readers disappear to
linger. with lingering transfers the curl thread can't shut down, which
will cause nix itself to not shut down until the transfer finishes some
other way (most likely network timeouts). also add a new test for this.

Change-Id: Id2401b3ac85731c824db05918d4079125be25b57
2024-05-05 18:09:31 +00:00
Qyriad 7ab076f21c point nix3-upgrade-nix to releases.lix.systems/manifest.nix
This file is currently manually managed, but will be automated along
with the rest of the release process.

Change-Id: I77839919549aaac73de582b2e563ce3ef914a8cb
2024-05-05 17:45:50 +00:00
Qyriad 03a20ef1ff Merge "add a contributor notice message to the dev shell hook" into main 2024-05-05 17:38:25 +00:00
Qyriad 10c1081b88 add a contributor notice message to the dev shell hook
It can be turned off by creating a file `.nocontribmsg` in the root
of the repo.

Change-Id: Iecc5c647c824a0416e527550226447780b94c08e
2024-05-05 11:22:38 -06:00
Maximilian Bosch 80dd6ab229 Merge "tests/flakes/follow-paths: test that warning about non-existent input works recursively" into main 2024-05-05 16:37:59 +00:00
eldritch horrors 6a5f100b8b doc: fix build littering doc/
mdbook has the unfortunate habit of creating stub files for chapters it
can't find on disk. turn off this helpful feature as it masks errors in
the summary file, and fix a recently introduced instance of this error.

Change-Id: I10d86aac0489c9c494bd5c8a50047415f4d4b18d
2024-05-05 16:13:27 +00:00
jade 2e8f9ac944 Warn on untrusted client settings being ignored
These are such a footgun and trip people up a lot. Let's make Lix louder
about this.

Related: lix-project/lix#261
Change-Id: I6a8d57c9817caaa6b0cbf886c615dda51038f628
2024-05-04 21:30:26 -07:00
jade 47fb494676 Merge "Actually try making a userns before assuming they don't work" into main 2024-05-05 03:58:44 +00:00
jade fb5d6f325b Merge "Fix /etc/group having desynced IDs from the actual UID in the sandbox" into main 2024-05-05 03:57:29 +00:00
jade e3b702fa22 Actually try making a userns before assuming they don't work
If unprivileged userns are *believed* to be disabled (such as with
"kernel.unprivileged_userns_clone = 0"), Lix would previously *give up*
on trying to use a user namespace before actually trying it, even if, in
cases such as unprivileged_userns_clone, it would actually be allowed
since Nix has CAP_SYS_ADMIN when running as daemon.

(see, e.g. 25d4709a4f)

We changed it to actually try it first, and then diagnose possible
causes, and also to be more loud about the whole thing, using warnings
instead of debugs. These warnings will only print on the first build run
by the daemon, which is, tbh, eh, shrug.

This is what led to us realizing that no-userns was a poorly exercised
condition.

Change-Id: I8e4f21afc89c574020dc7e89a560cc740ce6573a
2024-05-05 00:37:24 +00:00
jade 9909a175bf Fix /etc/group having desynced IDs from the actual UID in the sandbox
This was found when `logrotate.conf` failed to build in a NixOS system
with:

    /nix/store/26zdl4pyw5qazppj8if5lm8bjzxlc07l-coreutils-9.3/bin/id: cannot find name for group ID 30000

This was surprising because it seemed to mean that /etc/group was busted
in the sandbox. Indeed it was:

    root0:
    nixbld:!💯
    nogroup65534:

We diagnosed this to sandboxUid() being called before
usingUserNamespace() was called, in setting up /etc/group inside the
sandbox. This code desperately needs refactoring.

We also moved the /etc/group code to be with the /etc/passwd code, but
honestly this code is all spaghetti'd all over the place and needs some
more serious tidying than we did here.

We also moved some checks to be earlier to improve locality with where
the things they are checking come from.

Change-Id: Ie29798771f3593c46ec313a32960fa955054aceb
2024-05-04 17:36:50 -07:00
jade 4886d4592b Remove a URL literal from fetchTarball docs
Change-Id: I254b793b42f77ffe9f357f3b376683e5758f23b5
2024-05-04 16:55:27 -07:00
Maximilian Bosch 8dfb30a235 Merge "package: remove assert for libseccomp version" into main 2024-05-04 20:33:59 +00:00
Maximilian Bosch fa8009257a Merge "tests: actually run mercurial tests" into main 2024-05-04 19:37:38 +00:00
Maximilian Bosch d3b41f3a7d tests: actually run mercurial tests
The binary to check for is called hg not hq.

Change-Id: I812a30f9347d5bf0573cdacc3fc887960887ee92
2024-05-04 16:48:07 +02:00
Maximilian Bosch 3580a4b7bf package: remove assert for libseccomp version
This has the following downsides:

* you cannot build Lix against nixos-unstable.
* this will immediately break as soon as libseccomp will hit
  nixos-23.11 (given that people will probably use the package.nix via
  our overlay or override nixpkgs via `follows`).

Hence, removing the assert again and add a better FIXME comment.

Change-Id: I284e10cf08e1873fef70ed869a1638aa89792422
2024-05-04 11:25:29 +02:00
Maximilian Bosch 799d0132f0 Merge "Revert "Revert "Merge pull request #6621 from Kha/nested-follows""" into main 2024-05-04 08:52:29 +00:00
Maximilian Bosch 79d0ae6670 Merge "libstore/local-derivation-goal: prohibit creating setuid/setgid binaries" into main 2024-05-04 07:26:15 +00:00
Maximilian Bosch e91be79d8e tests/flakes/follow-paths: test that warning about non-existent input works recursively
When I added the warning that an input X has an override for a
non-existent input, the recursive flake input override fix wasn't
implemented yet[1].

This patch tests that both work together.

[1] https://github.com/NixOS/nix/pull/6663

Change-Id: I90dc032029b7160ab4a97d28c480c59d3a6f0150
2024-05-03 22:54:38 +00:00
Maximilian Bosch 0e38720502 Revert "Revert "Merge pull request #6621 from Kha/nested-follows""
This reverts commit a8b3d777fb.

This undoes the revert of PR#6621, which allows nested `follows`, i.e.

    {
      inputs = {
        foo.url = "github:bar/foo";
        foo.inputs.bar.inputs.nixpkgs = "nixpkgs";
      };
    }

does the expected thing now. This is useful to avoid the 1000 instances
of nixpkgs problem without having each flake in the dependency tree to
expose all of its transitive dependencies for modification.

This was in fact part of Nix before and the C++ changes applied w/o
conflicts. However, it got reverted then because people didn't want to
merge lazy-trees against it which was supposed to be merged soon back in
October 2022.

Fixes: lix-project/lix#201

Change-Id: I5ddef914135b695717b2ef88862d57ced5e7aa3c
2024-05-03 22:54:38 +00:00
Maximilian Bosch f8617f9dc6 Merge "Rename nix show-config to nix config show" into main 2024-05-03 22:07:33 +00:00
Maximilian Bosch 045ee37438 libstore/local-derivation-goal: prohibit creating setuid/setgid binaries
With Linux kernel >=6.6 & glibc 2.39 a `fchmodat2(2)` is available that
isn't filtered away by the libseccomp sandbox.

Being able to use this to bypass that restriction has surprising results
for some builds such as lxc[1]:

> With kernel ≥6.6 and glibc 2.39, lxc's install phase uses fchmodat2,
> which slips through 9b88e52846/src/libstore/build/local-derivation-goal.cc (L1650-L1663).
> The fixupPhase then uses fchmodat, which fails.
> With older kernel or glibc, setting the suid bit fails in the
> install phase, which is not treated as fatal, and then the
> fixup phase does not try to set it again.

Please note that there are still ways to bypass this sandbox[2] and this is
mostly a fix for the breaking builds.

This change works by creating a syscall filter for the `fchmodat2`
syscall (number 452 on most systems). The problem is that glibc 2.39
is needed to have the correct syscall number available via
`__NR_fchmodat2` / `__SNR_fchmodat2`, but this flake is still on
nixpkgs 23.11. To have this change everywhere and not dependent on the
glibc this package is built against, I added a header
"fchmodat2-compat.hh" that sets the syscall number based on the
architecture. On most platforms its 452 according to glibc with a few
exceptions:

    $ rg --pcre2 'define __NR_fchmodat2 (?!452)'
    sysdeps/unix/sysv/linux/x86_64/x32/arch-syscall.h
    58:#define __NR_fchmodat2 1073742276

    sysdeps/unix/sysv/linux/mips/mips64/n32/arch-syscall.h
    67:#define __NR_fchmodat2 6452

    sysdeps/unix/sysv/linux/mips/mips64/n64/arch-syscall.h
    62:#define __NR_fchmodat2 5452

    sysdeps/unix/sysv/linux/mips/mips32/arch-syscall.h
    70:#define __NR_fchmodat2 4452

    sysdeps/unix/sysv/linux/alpha/arch-syscall.h
    59:#define __NR_fchmodat2 562

I added a small regression-test to the setuid integration-test that
attempts to set the suid bit on a file using the fchmodat2 syscall.
I confirmed that the test fails without the change in
local-derivation-goal.

Additionally, we require libseccomp 2.5.5 or greater now: as it turns
out, libseccomp maintains an internal syscall table and
validates each rule against it. This means that when using libseccomp
2.5.4 or older, one may pass `452` as syscall number against it, but
since it doesn't exist in the internal structure, `libseccomp` will refuse
to create a filter for that. This happens with nixpkgs-23.11, i.e. on
stable NixOS and when building Lix against the project's flake.

To work around that

* a backport of libseccomp 2.5.5 on upstream nixpkgs has been
  scheduled[3].

* the package now uses libseccomp 2.5.5 on its own already. This is to
  provide a quick fix since the correct fix for 23.11 is still a staging cycle
  away.

We still need the compat header though since `SCMP_SYS(fchmodat2)`
internally transforms this into `__SNR_fchmodat2` which points to
`__NR_fchmodat2` from glibc 2.39, so it wouldn't build on glibc 2.38.
The updated syscall table from libseccomp 2.5.5 is NOT used for that
step, but used later, so we need both, our compat header and their
syscall table 🤷

Relevant PRs in CppNix:

* https://github.com/NixOS/nix/pull/10591
* https://github.com/NixOS/nix/pull/10501

[1] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2031073804
[2] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2030844251
[3] https://github.com/NixOS/nixpkgs/pull/306070

(cherry picked from commit ba6804518772e6afb403dd55478365d4b863c854)
Change-Id: I6921ab5a363188c6bff617750d00bb517276b7fe
2024-05-03 16:29:06 +02:00
Théophane Hufschmitt 8458d98b27 Rename nix show-config to nix config show
Part of #7672

My main motivation is to be able to use `nix.checkConfig`[1]. This
doesn't work with Lix currently since the module uses `nix show-config`
if the Nix version is <2.20pre and `nix config show` otherwise. I think
this is the only instance where nixpkgs checks for which Nix commands
exist that affects us now, so I figured we could just perform the rename
here as well[2] and still provide the current version number[3].

I don't have a strong opinion on whether to deprecate `nix show-config`,
the warning is added there automatically.

(cherry picked from commit f300e11b056dea414d7d77bbc6e5a7dc5d9ddd41)

[1] https://nixos.org/manual/nixos/stable/options.html#opt-nix.checkConfig
[2] I should add that I don't use the "official" ways of installing Lix
    because using the flake directly and callPackaging it seemed to fit
    better into my workflow: I already have a little mess to make
    sure Hydra from the flake uses the correct pkgs.nix and I didn't
    want to complicate it further while keeping a single package-set I
    can build in CI. Don't get me wrong, I think such a module for a
    quick-start is very important, just giving context on why I bother
    in the first place :)
[3] When we go public, I think it's worth considering to add support in
    nixpkgs itself for Lix.

Change-Id: I47b4239b05cbeda3c370d2fa56ea768b768768ac
2024-05-03 16:26:16 +02:00
Qyriad 19645a4a64 Merge changes Id1a67156,I03f4c7c1,I146736bb,I3b1453cb into main
* changes:
  docs: clarify how ^ works for -E/-f installables
  docs: give translation examples from nix-build -E/-A to installables
  docs: clarify how the different kinds of installables are selected
  docs: guide to installables docs in installable commands' docs
2024-05-03 13:39:49 +00:00
Artemis Tosini 6f0636a7ed Merge "libstore: check additionalSandboxProfile" into main 2024-05-03 03:35:02 +00:00
Artemis Tosini ecdbc3b207 libstore: check additionalSandboxProfile
Currently LocalDerivationGoal allows setting `__sandboxProfile`
to add sandbox parameters on Darwin when `sandbox=true`.
This was only supposed to have an effect when `sandbox=relaxed`

Change-Id: Ide44ee82d7e4d6b545285eab26547e7014817d3f
2024-05-03 00:59:18 +00:00
eldritch horrors d55b158e24 libutil: make rewriteStrings sound
this is used in CA rewriting, replacement of placeholders in
derivations, generating scripts for devShells, and some more
places. in all of these transitive replacements are unsound,
and overlapping replacements would be as well. there even is
a test that transitive replacements do not happen (in the CA
RewriteSink suite), but none for overlapping replacements. a
minimally surprising binary rewriter surely would not do any
of these replacements, the only reason we have not seen this
break yet is probably that rewriteStrings is only called for
store paths and things that look like store paths (and those
should never overlap nor admit such transitive replacements)

Change-Id: I6fc29f939d5061d9f56c752624a823ece8437c07
2024-05-03 00:50:31 +00:00
Qyriad 076dfd30c6 Merge changes from topic "profile-v3" into main
* changes:
  nix3-profile: remove check "name" attr in manifests
  Add profile migration test
  nix3-profile: make element names stable
  getNameFromURL(): Support uppercase characters in attribute names
  nix3-profile: remove indices
  nix3-profile: allow using human-readable names to select packages
  implement parsing human-readable names from URLs
2024-05-02 20:15:48 +00:00
Qyriad 6a8b379628 nix3-profile: remove check "name" attr in manifests
It doesn't seem to have ever been used.

Based off of commit a748e88bf4cca0fdc6ce75188e88017a7899d16b

Upstream-PR: https://github.com/NixOS/nix/pull/9656
Change-Id: Idcf250a645fa43f2ef11fb15b503b070a62a917e
2024-05-02 12:59:15 -06:00
Eelco Dolstra 5d2031f92d Add profile migration test
(cherry picked from commit 72560f7bbef2ab3c02b8ca040fe084328bdd5fbe)

Upstream-PR: https://github.com/NixOS/nix/pull/9656
Change-Id: I405e5848e2627a76940220fb6aebadfb8f094afb
2024-05-02 12:59:15 -06:00
Qyriad e0911eef73 nix3-profile: make element names stable
Based off of commit 6268a45b650f563bae2360e0540920a2959bdd40

Upstream-PR: https://github.com/NixOS/nix/pull/9656
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Change-Id: I0fcf069a8537c61ad6fc4eee1f3c193a708ea1c4
2024-05-02 12:59:15 -06:00
Eelco Dolstra ce70f02aff getNameFromURL(): Support uppercase characters in attribute names
In particular, this makes it handle 'legacyPackages' correctly.

(cherry picked from commit 936a3642264ac159f3f9093710be3465b70e0e89)

Upstream-PR: https://github.com/NixOS/nix/pull/9657
Change-Id: Icc4efe02f7f8e90a2970589f72fd3d3cd4418d95
2024-05-02 12:02:28 -06:00
Qyriad e98fc952a8 nix3-profile: remove indices
Based off of commit 3187bc9ac3dd193b9329ef68c73ac3cca794ed78

Upstream-PR: https://github.com/NixOS/nix/pull/9656
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Change-Id: I8ac4a33314cd1cf9de95404c20f58e883460acc7
2024-05-02 12:02:28 -06:00
Qyriad f88423813f nix3-profile: allow using human-readable names to select packages
These names are parsed from the URL provided for that package

Based off of commit 257b768436a0e8ab7887f9b790c5b92a7fe51ef5

Upstream-PR: https://github.com/NixOS/nix/pull/8678
Co-authored-by: Felix Uhl <felix.uhl@outlook.com>
Change-Id: I76d5f9cfb11d3d2915b3dd1db21d7bb49e91f4fb
2024-05-02 12:02:28 -06:00
Robert Hensing b7ce11c97d Disallow store path names that are . or .. (plus opt. -)
As discussed in the maintainer meeting on 2024-01-29.

Mainly this is to avoid a situation where the name is parsed and
treated as a file name, mostly to protect users.
.-* and ..-* are also considered invalid because they might strip
on that separator to remove versions. Doesn't really work, but that's
what we decided, and I won't argue with it, because .-* probably
doesn't seem to have a real world application anyway.
We do still permit a 1-character name that's just "-", which still
poses a similar risk in such a situation. We can't start disallowing
trailing -, because a non-zero number of users will need it and we've
seen how annoying and painful such a change is.

What matters most is preventing a situation where . or .. can be
injected, and to just get this done.

(cherry picked from commit f1b4663805a9dbcb1ace64ec110092d17c9155e0)
Change-Id: I900a8509933cee662f888c3c76fa8986b0058839
2024-05-02 19:34:38 +02:00
Robert Hensing 4b3dc66386 test: Generate distinct hashes
Gen::just is the constant generator. Don't just return that!

(cherry picked from commit 8406da28773f050e00a006e4812e3ecbf919a2a9)
Change-Id: Ibfd0bd40f90942077a4720086ce0cd3bfabef79d
2024-05-02 19:34:21 +02:00
Robert Hensing 2eec547d7d test: Generate distinct path names
Gen: :just is the constant generator. Don't just return that!

(cherry picked from commit 69bbd5852af9b2f0b794162bd1debcdf64fc6648)
Change-Id: Id6e58141f5a42a1f67bd11d48c87b32a3ebd0500
2024-05-02 19:34:04 +02:00
Robert Hensing fae6ae2122 parseStorePath: Support leading period
(cherry picked from commit b13e6a76b4f289c6db69ffaa7bd35b7e44f2a391)
Change-Id: Ie14be437d87d17248f80e1f009aa2a4311ddede6
2024-05-02 19:34:03 +02:00
Robert Hensing 3a058dc4b3 Revert "StorePath: reject names starting with '.'"
This reverts commit 24bda0c7b381e1a017023c6f7cb9661fae8560bd.

(cherry picked from commit 9ddd0f2af8fd95e1380027a70d0aa650ea2fd5e4)
Change-Id: Ideb547e2a8ac911cf39d58d3e0c1553867bdd776
2024-05-02 18:53:51 +02:00
Qyriad 1425aa0b7c implement parsing human-readable names from URLs
Based off of commit 257b768436a0e8ab7887f9b790c5b92a7fe51ef5

Upstream-PR: https://github.com/NixOS/nix/pull/8678
Co-authored-by: Felix Uhl <felix.uhl@outlook.com>
Change-Id: Idcb7f6191ca3310ef9dc854197f7798260c3f71d
2024-04-30 18:11:14 -06:00
Qyriad 4942e4e2d2 docs: clarify how ^ works for -E/-f installables
We didn't even realize you *could* use this syntax with -E and -f, much
less that the attribute path could be *empty*.

Change-Id: Id1a6715609f3a76a5ce477bd43a7832effbbe07b
2024-04-29 08:53:43 -06:00
Qyriad 6abeea70e9 docs: give translation examples from nix-build -E/-A to installables
Change-Id: I03f4c7c1049063539a35ba500a07bb8f866d4cb7
2024-04-29 08:18:00 -06:00
Qyriad bd2619868c docs: clarify how the different kinds of installables are selected
Change-Id: I146736bb97ebe035e04be69ce9fb60a557e38c6c
2024-04-29 08:18:00 -06:00
Qyriad 194ba8a02f docs: guide to installables docs in installable commands' docs
The installables syntax is not documented in any of the man pages or
docbook pages for any of those individual commands. And while these
commands really should at least peripherally individually document how
installables work, in the meantime we can at least direct people to the
right place.

This commit also clarifies the unexpected fact that `nix profile remove`
and `nix profile upgrade` do *not* take installables.

Change-Id: I3b1453cb197a613bbab639c66a466365c3592c6d
2024-04-29 08:06:05 -06:00
Qyriad e2ab89a74b add VM test for nix upgrade-nix
This commit adds a new NixOS VM test, which tests that `nix upgrade-nix`
works on both kinds of profiles (manifest.nix and manifest.json).

Done as a separate commit from 831d18a13, since it relies on the
--store-path argument from 026c90e5f as well.

Change-Id: I5fc94b751d252862cb6cffb541a4c072faad9f3b
2024-04-29 01:19:21 +00:00
Qyriad ee5a1b5a4c nix3-upgrade-nix: allow manually specifying new nix
This allows manually specifying a store path for the new Nix that
gets linked into Nix's profile.

Change-Id: Ib71711ffb466febf4a6892e3fdbda644e053770d
2024-04-29 01:19:21 +00:00
Qyriad aae12f5848 fix nix upgrade-nix on new-style profiles
nix3-profile automatically migrates any profile its used on to its style
of profile -- the ones with manifest.json instead of manifest.nix. On
non-NixOS systems, Nix is conventionally installed to the profile at
/nix/var/nix/profiles/default, so if a user passed that to `--profile`
of `nix profile`, then it would break upgrade-nix from ever working
again, without recreating the profile.

This commit fixes that, and allows upgrade-nix to work on either kind of
profile.

Fixes #16.

Change-Id: I4c49b1beba93bb50e8f8a107edc451affe08c3f7
2024-04-29 01:19:21 +00:00
Qyriad 2bd57d4d36 refactor some nix-env and profile code to libcmd
Notably, ProfileManifest and ProfileElement are useful generic
profile management code, and nix profile is not the only place in the
codebase where profiles are relevant.

This commit is in preparation for fixing upgrade-nix's interaction with
new-style profiles.

Change-Id: Iefc8bbd34b4bc6012175cb3d6e6a8207973bc792
2024-04-29 01:19:21 +00:00
Qyriad da677fce39 Merge "remove Github workflow files" into main 2024-04-28 23:31:59 +00:00
Qyriad 6bc394c6d2 remove Github workflow files
Fixes #251. We aren't using them.

Change-Id: Ib24da6e0277bdd6afa9103c553c743a610f7b4e1
2024-04-28 02:56:19 -06:00
puck 0c831765bd Run all derivation builders inside the sandbox on macOS
This replaces the external sandbox-exec call with direct calls into
libsandbox. This API is technically deprecated and is missing some
prototypes, but all major browsers depend on it, so it is unlikely to
materially change without warning.

This commit also ensures the netrc file is only written if the
derivation is in fact meant to be able to access the internet.

This change commits a sin of not actually actively declaring its
dependency on macOS's libsandbox.dylib; this is due to the dylib
cache in macOS making that explicit dependency unnecessary. In the
future this might become a problem, so this commit marks our sins.

Co-authored-by: Artemis Tosini <lix@artem.ist>
Co-authored-by: Lunaphied <lunaphied@lunaphied.me>
Change-Id: Ia302141a53ce7b0327c1aad86a117b6645fe1189
2024-04-27 14:44:15 -06:00
Qyriad 76b45b4861 Merge "docs(nix-env): summarize of each subcommand in --help" into main 2024-04-27 18:42:46 +00:00
puck 9229e87347 Fix progress bar on copyPaths
This variable should not be shared between activities.

Change-Id: I4eee89bc7acb320a3972dc3a55bfb087d3a9eb3a
2024-04-27 18:03:15 +00:00
Ilya K 9462c01c3e libstore/ssh: shut
This is just logspam, and we have NIX_SSHOPTS for people that want the logspam.

Change-Id: Ieff71473686f0661f9c53c212f8952dd2c9565c3
2024-04-27 12:05:17 +03:00
Qyriad 78ce710722 docs(nix-env): summarize of each subcommand in --help
This should have been there from the beginning. As much as nix-env is a
pile of problems we don't need trivial docs papercuts like this adding
to it.

Change-Id: I0c53e4b146af2fefdd0e4743d850672729cb2194
2024-04-26 21:56:08 -06:00
Maximilian Bosch 8773439a85 Merge "ssh-ng: Set log-fd for ssh to 4 by default" into main 2024-04-26 18:30:33 +00:00
Artemis Tosini 789aa39576 Merge "gc: Find roots using libproc on Darwin" into main 2024-04-26 17:26:45 +00:00
Maximilian Bosch 104448e75d ssh-ng: Set log-fd for ssh to 4 by default
That's expected by `build-remote` and makes sure that errors are
correctly forwarded to the user. For instance, let's say that the
host-key of `example.org` is unknown and

    nix-build ../nixpkgs -A hello -j0 --builders 'ssh-ng://example.org'

is issued, then you get the following output:

    cannot build on 'ssh-ng://example.org?&': error: failed to start SSH connection to 'example.org'
    Failed to find a machine for remote build!
    derivation: yh46gakxq3kchrbihwxvpn5bmadcw90b-hello-2.12.1.drv
    required (system, features): (x86_64-linux, [])
    2 available machines:
    [...]

The relevant information (`Host key verification failed`) ends up in the
daemon's log, but that's not very obvious considering that the daemon
isn't very chatty normally.

This can be fixed - the same way as its done for legacy-ssh - by passing
fd 4 to the SSH wrapper. Now you'd get the following error:

    cannot build on 'ssh-ng://example.org': error: failed to start SSH connection to 'example.org': Host key verification failed.
    Failed to find a machine for remote build!
    [...]

...and now it's clear what's wrong.

Please note that this is won't end up in the derivation's log.

For previous discussion about this change see
https://github.com/NixOS/nix/pull/7659.

Change-Id: I5790856dbf58e53ea3e63238b015ea06c347cf92
2024-04-26 19:04:06 +02:00
eldritch horrors a1ad4e52a6 filetransfer: don't decompress in curl wrapper itself
only decompress the response once all data has been received (in the
fully buffered case), or at least outside of the curl wrapper itself
(in the receive-to-sink case). unfortunately this means we will have
to duplicate decompression logic for these two cases for time being,
but once the curl wrapper has been rewritten to return a real future
or Source we can deduplicate this logic again. the curl wrapper will
have to turn into a proper Source first and use decompression source
logic which also does not currently exist—only decompression *sinks*

Change-Id: I66bc692f07d9b9e69fe10689ee73a2de8d65e35c
2024-04-26 15:26:37 +00:00
eldritch horrors fb0996aaa8 filetransfer: remove dataCallback from interface
this is highly questionable. single-arg download calls will misbehave
with it set, and two-arg download calls will just overwrite it. being
an implementation detail this should not have been in the API at all.

Change-Id: I613772951ee03d8302366085f06a53601d13f132
2024-04-26 15:26:37 +00:00
eldritch horrors dfe3baea12 filetransfer: make two-arg download abstract
this lets each implementation of FileTransfer (of which currently only
the one exists at all) implement appropriate handling for its internal
behaviours that are not otherwise exposed. in curl this lets us switch
the buffer-full handling method from "block the entire curl thread" to
"pause just the one transfer", move the non-libcurl body decompression
out of the actual curl wrapper (which will let us eventually morph the
curl wrapper intto an actual source of Sources), and some other things

Change-Id: Id6d3593cde6b4915aab3e90a43b175c103cc3f18
2024-04-26 15:26:37 +00:00
Maximilian Bosch ce76d3eab2 Merge "justfile: allow passing args to meson compile" into main 2024-04-26 07:45:22 +00:00
Artemis Tosini c03de0df62 gc: Find roots using libproc on Darwin
Previously, the garbage collector found runtime roots on Darwin by
shelling out to `lsof -n -w -F n` then parsing the result.
However, this requires an lsof binary and can be extremely slow.

The official Apple lsof returns in a reasonable amount of time,
about 250ms in my tests, but the lsof packaged in nixpkgs is quite slow,
taking about 40 seconds to run the command.

Using libproc directly is about the same speed as Apple lsof,
and allows us to reënable several tests that were disabled on Darwin.

Change-Id: Ifa0adda7984e13c15535693baba835aae79a3577
2024-04-25 23:24:21 -04:00
Maximilian Bosch ecad3632cc justfile: allow passing args to meson compile
My main motivation for this change is to limit the amount of compile
jobs to make sure my machine is still usable for something else when
building a fresh Lix locally.

Also made `build` a dependency of `install`: this is analogous to
`make install` in CppNix where this both recompiles changed files and
installs the artifacts into `outputs/out`. May be a little more pleasant
to work with that, especially when you're used to contributing to
CppNix.

Change-Id: I321e2b0daf1c5e20f82c04e2dd158056c80ed86c
2024-04-25 14:26:38 +02:00
eldritch horrors 5420b3afd6 filetransfer: drop errorSink
just accumulate error data into result.data as we would for successful
transfers without a dataCallback. errorSink and data would contain the
same data in error cases anyway, so splitting them is not very useful.

Change-Id: I00e449866454389ac6a564ab411c903fd357dabf
2024-04-25 01:33:22 +02:00
eldritch horrors 5e69f8aa3d filetransfer: restore http status line reporting
this was broken in 75b62e5260.

Change-Id: If8583e802afbcde822623036bf41a9708fbc7c8d
2024-04-25 01:33:08 +02:00
eldritch horrors 38442e3123 filetransfer: remove decompress request parameter
this is never read.

Change-Id: I4c46f140519843a21e452958900e81edd2f78be2
2024-04-25 01:33:08 +02:00
Artemis Tosini 7114b0465a Merge "libstore: Create platform LocalStore subclasses" into main 2024-04-24 15:35:32 +00:00
Qyriad f24223931d meson: remove unnecessary parts of cross file
Meson cross files layer, the last value of each key takes effect.

https: //mesonbuild.com/Machine-files.html#loading-multiple-machine-files
Change-Id: I22d886f71cd51f0ce520d3fc22aed4bcf074bb91
2024-04-23 10:20:20 -06:00
Artemis Tosini b247ef72dc libstore: Create platform LocalStore subclasses
This creates new subclasses of LocalStore for each OS to include
platform-specific functionality. Currently this just includes garbage
collector roots but it could be extended to sandboxing as well.

In order to make sure that the generic LocalStore is not accidentally
constructed, its constructor is protected. A Fallback is provided which
implements no functionality except constructors.

Change-Id: I836a28e90b68309873f75afb83e0f1b2e2c89fb3
2024-04-23 16:17:05 +00:00
Qyriad be4a3168c9 Merge changes Ia3e7b1e6,If09be814 into main
* changes:
  meson: flip the switch!!
  meson: fix cross compilation
2024-04-23 11:12:09 +00:00
Qyriad b913a939b0 meson: flip the switch!!
This commit makes Meson the default buildsystem for Lix.
The Make buildsystem is now deprecated and will be removed soon, but has
not yet, which will be done in a later commit when all seems good. The
mesonBuild jobs have been removed, and have not been replaced with
equivalent jobs to ensure the Make buildsystem still works.

The full, new commands in a development shell are:

$ meson setup ./build "--prefix=$out" $mesonFlags

(A simple `meson setup ./build` will also build, but will do a different
thing, not having the settings from package.nix applied.)

$ meson compile -C build
$ meson test -C build --suite=check
$ meson install -C build
$ meson test -C build --suite=installcheck

(Check and installcheck may both be done after install, allowing you to
omit the --suite argument entirely, but this is the order package.nix
runs them in.)

If tests fail and Meson helpfully has no output for why, use the
`--print-error-logs` option to `meson test`. Why this is not the default
I cannot explain.

If you change a setting in the buildsystem, most cases will
automatically regenerate the Meson configuration, but some cases, like
trying to build a specific target whose name is new to the buildsystem
(e.g. `meson compile -C build src/libmelt/libmelt.dylib`, when
`libmelt.dylib` did not exist as a target the last time the buildsystem
was generated), then you can reconfigure using new settings but
existing options, and only recompiling stuff affected by the changes:

$ meson setup --reconfigure build

Note that changes to the default values in `meson.options` or in the
`default_options :` argument to project() are NOT propagated with
`--reconfigure`.

If you want a totally clean build, you can use:

$ meson setup --wipe build

That will work regardless of if `./build` exists or not.

Specific, named targets may be addressed in
`meson build -C build <target>` with the "target ID" if there is one,
which is the first string argument passed to target functions that
have one, and unrelated to the variable name, e.g.:

libexpr_dylib = library('nixexpr', …)

can be addressed with:

$ meson compile -C build nixexpr

All targets may be addressed as their output, relative to the build
directory, e.g.:

$ meson compile -C build src/libexpr/libnixexpr.so

But Meson does not consider intermediate files like object files
targets. To build a specific object file, use Ninja directly and
specify the output file relative to the build directory:

$ ninja -C build src/libexpr/libnixexpr.so.p/nixexpr.cc.o

To inspect the canonical source of truth on what the state of the
buildsystem configuration is, use:

$ meson introspect

Have fun!

Change-Id: Ia3e7b1e6fae26daf3162e655b4ded611a5cd57ad
2024-04-22 21:41:58 -06:00
Qyriad 05e3b1d39e meson: fix cross compilation
This should fix cross compilation in the base case, but this is
difficult to test as cross compilation is broken in many different
places right now. This should bring Meson back up to cross parity with
the Make buildsystem though.

Change-Id: If09be8142d1fc975a82b994143ff35be1297dad8
2024-04-22 21:41:58 -06:00
eldritch horrors 86bfede948 libstore: use curl functions for reading headers
don't reimplement header parsing. this was only really needed due to the
ancient github bug we no longer care about, everything else we have done
in custom code can also be done using curl itself. doing this also fixes
possible sources of header smuggling (because the header function didn't
unfold headers and we'd trim them before parsing, which would've made us
read contents of one header as a fully formed header in itself). this is
a slight behavior change because we now honor only the first instance of
a given header where previous behavior was to honor either the last or a
combination of all of them (accept-ranges was logical-or'd by accident).

Change-Id: I93cb93ddb91ab98c8991f846014926f6ef039fdb
2024-04-23 01:04:56 +00:00
eldritch horrors 257d7ffa7b libstore: remove github etag workaround
this was a workaround for a *github* bug that happend *in 2015*.
not only is github no longer buggy, it shouldn't have been nix's
responsibility to work around these bugs like this to begin with

while we're at it we'll also remove another workaround—again for
github specifically and again for etag handling—from 2021 that's
also not needed any more. future workarounds for serverside bugs
should probably come with an expiration date that mutates into a
build warning after a while, otherwise this *will* happen again.

Change-Id: I74f739ae3e36d40350f78bebcb5869aa8cc9adcd
2024-04-23 01:04:56 +00:00
Qyriad 7063170d5f tests: add error messages to the asserts in tarball flakes test
In hopes of avoiding opaque error messages like the one in
https://buildbot.lix.systems/#/builders/49/builds/1054/steps/1/logs/stdio

Traceback (most recent call last):
  File "/nix/store/wj6wh89jhd2492r781qsr09r9wydfs6m-nixos-test-driver-1.1/bin/.nixos-test-driver-wrapped", line 9, in <module>
    sys.exit(main())
             ^^^^^^
  File "/nix/store/wj6wh89jhd2492r781qsr09r9wydfs6m-nixos-test-driver-1.1/lib/python3.11/site-packages/test_driver/__init__.py", line 126, in main
    driver.run_tests()
  File "/nix/store/wj6wh89jhd2492r781qsr09r9wydfs6m-nixos-test-driver-1.1/lib/python3.11/site-packages/test_driver/driver.py", line 159, in run_tests
    self.test_script()
  File "/nix/store/wj6wh89jhd2492r781qsr09r9wydfs6m-nixos-test-driver-1.1/lib/python3.11/site-packages/test_driver/driver.py", line 151, in test_script
    exec(self.tests, symbols, None)
  File "<string>", line 13, in <module>
AssertionError

Change-Id: Idd2212a1c3714ce58c7c3a9f34c2ca4313eb6d55
2024-04-22 16:13:36 -06:00
eldritch horrors ff9a4fc336 libstore: use curl_multi_{poll,wakeup}
the previous solution to the wakeup problem (adding a pipe and passing
it as an additional fd to curl_multi_wait) worked, but there have been
builtin alternatives for this since 2020. not only do these save code,
they're also a lot more likely to work natively on windows when needed

Change-Id: Iab751b900997110a8d15de45ea3ab0c42f7e5973
2024-04-22 21:37:20 +00:00
eldritch horrors e5903aab65 libstore: remove ancient libcurl feature checks
the oldest version checked for here is 7.47, which was released in
2016. it's probably safe to say that we do not need these any more

Change-Id: I003411f6b2ce6d56f7ca337390df3ea86bd59a99
2024-04-22 19:45:22 +00:00
puck c8c838381d Merge "Fix exportReferencesGraph when given store subpath" into main 2024-04-21 15:37:59 +00:00
puck 272c2ff15f remove extraneous cache entry from github fetcher
This isn't necessary, as it's already covered by the tarball fetcher's
cache.

Change-Id: I85e35f5a61594f27b8f30d82145f92c5d6559e1f
2024-04-21 10:46:05 +00:00
Alyssa Ross c1319831fb Fix exportReferencesGraph when given store subpath
With Nix 2.3, it was possible to pass a subpath of a store path to
exportReferencesGraph:

	with import <nixpkgs> {};

	let
	  hello = writeShellScriptBin "hello" ''
	    echo ${toString builtins.currentTime}
	  '';
	in

	writeClosure [ "${hello}/bin/hello" ]

This regressed with Nix 2.4, with a very confusing error message, that
presumably indicates it was unintentional:

	error: path '/nix/store/3gl7kgjr4pwf03f0x70dgx9ln3bhl7zc-hello/bin/hello' is not in the Nix store

(cherry picked from commit 0774e8ba33c060f56bad3ff696796028249e915a)
Change-Id: I00920fb33077b831a1bb4a1b68d515ba8c3c2a69
2024-04-21 10:27:32 +00:00
Qyriad 111db8b38f meson: correctly embed sandbox shell when asked
Change-Id: I2f6c0d42245204a516d2e424eea26a6391e975ad
2024-04-18 16:15:58 -06:00
eldritch horrors a326344253 tests: unhaunt the flakes nixos tests
these should really wait for networks to come up, otherwise they can fail.

fixes #235

Change-Id: I08989e8bdb0de280df74660ac43983de5c34fa9d
2024-04-18 20:09:19 +00:00
Qyriad f9d08cc44c meson: embed source paths as relative to the source root and avoid ../src
Change-Id: Ifab83cb7a3bfde717a4d6032ede8be75dc61f2b1
2024-04-18 10:45:27 -06:00
Qyriad 077f45ee38 meson: correctly set -DSANDBOX_SHELL if we have it
The statically embedded busybox is not required for Lix to work, but
package.nix explicitly sets this, which was accidentally being ignored.

Change-Id: Ieeff830ac7d1f5fabe84d1a6cfd82f13d79035bf
2024-04-18 10:45:27 -06:00
Qyriad cf0744ceed Merge "build internal API docs with Meson" into main 2024-04-17 21:48:25 +00:00
Ilya K 6d79aa3d70 Merge "libstore/build: set NO_NEW_PRIVS for the sandbox" into main 2024-04-16 05:33:41 +00:00
Qyriad b81eec6ed5 build internal API docs with Meson
This commit adds the capability for building the Doxygen internal API
docs in the Meson buildsystem, and also makes doing so the default for
the internal-api-docs hydra job. Aside from the /nix-support directory,
which differed only by the hash part of a store path, the outputs of
hydraJobs.internal-api-docs before and after this commit were
bit-for-bit identical on my machine.

Change-Id: I98f0017891c25b06866c15f7652fe74f706ec8e1
2024-04-15 19:05:07 -06:00
Qyriad a41abb4594 fix probable format bug in DerivationGoal::buildDone
Either the contents of `line` could cause format errors, or this usage
is Technically safe. However, I trust nothing, especially with
boost::format.

Change-Id: I07933b20bde3b305a6e5d61c2a7bab6ecb042ad9
2024-04-15 23:09:40 +00:00
Qyriad 4e68deef80 abort with a descriptive message on bad HintFmt usage
Change-Id: Ic2f05572042343a8160fd971394372f5f2706fc4
2024-04-15 23:09:16 +00:00
Ilya K effc28f6f5 libstore/build: set NO_NEW_PRIVS for the sandbox
Change-Id: I711f64e2b68495ed9c85c1a4bd5025405805e43a
2024-04-15 10:25:29 +03:00
Qyriad 80bbfe2034 don't throw an exception for the trivial case of isStorePath()...
Previously if isStorePath() was called on anything other than a
top-level /nix/store/some-path, it would throw a BadStorePath exception.
This commit duplicates the absolutely trivial check, into
maybeParseStorePath(), and leaves exception throwing to
parseStorePath(), the function that assumes you're already giving a
valid path instead of the one whose purpose is to check if its valid or
not...

Change-Id: I8dda548f0f88d14ca8c3ee927d64e0ec0681fc7b
2024-04-14 21:08:07 +00:00
Qyriad ddb4d3fa4c Merge "don't boost::to_few_args when an eval cached string type errors" into main 2024-04-14 21:07:47 +00:00
Ilya K 8d15e6af4b Merge "libstore/build: just copy the magic /etc files into the sandbox" into main 2024-04-13 12:15:20 +00:00
Ilya K b469c6509b libstore/build: just copy the magic /etc files into the sandbox
Saves us a bunch of thinking about how to handle symlinks, and prevents
the DNS config from changing on the fly under the build, which may or may
not be a good thing?

Change-Id: I071e6ae7e220884690b788d94f480866f428db71
2024-04-13 12:43:19 +03:00
Qyriad ded64e2822 Merge changes I60d8e6f7,Ic635687b into main
* changes:
  binary tarball: include cacert in root paths
  flake: factor out binary tarball into its own file
2024-04-12 13:24:47 +00:00
Qyriad a3be742bda binary tarball: include cacert in root paths
93cc06334 removed nss-cacert from the binary tarball, but they're
necessary for global compatibility (and for our installer). This is what
results in cacerts being in the default profile, so e.g. the daemon has
TLS certs without having to use the system ones.

There's a fallback behavior in the daemon script in case these wind up
missing from the profile, but we don't want to have to rely on that,
since the fallback fails if it doesn't recognize one of a handful of
distros.

Change-Id: I60d8e6f734469548e80d5f38113ef168f67cbf7d
2024-04-12 07:04:37 -06:00
Qyriad 629351163d flake: factor out binary tarball into its own file
Bit-for-bit identical, and this one is callPackage-able

Change-Id: Ic635687b0054e107271a9c24ae69101f5e0fba9e
2024-04-12 06:35:54 -06:00
Ilya K d363bc2f12 Merge "Merge pull request #10456 from NixOS/fixpermdeniedbind" into main 2024-04-11 19:08:33 +00:00
eldritch horrors e4a8c01bdf Merge changes Iedf46484,I76b51eac,I6a084827,I60193f9f into main
* changes:
  meson: fix log-dir
  manual: build docs with dummy envs
  libcmd: install generated headers as well
  docs: redo content generation for mdbook and manual
2024-04-11 14:33:16 +00:00
Ilya K d106bb553b Merge "Merge pull request #10362 from obsidiansystems/maybeLstat" into main 2024-04-11 13:45:46 +00:00
eldritch horrors cd79b8d65a meson: fix log-dir
the make build system sets this with an extra /nix segment.

Change-Id: Iedf464843196faeae5b59698837faca3a4f23586
2024-04-11 13:36:04 +00:00
eldritch horrors adab839c98 manual: build docs with dummy envs
this was previously used because the macOS docs build would otherwise
pull files out of the host nix store. or something. not sure about it

Change-Id: I76b51eac1ebc5de5f00e2e4be086dd8db3eeb8e6
2024-04-11 13:36:04 +00:00
eldritch horrors f42678802c libcmd: install generated headers as well
these seem to have been forgotten.

Change-Id: I6a084827d087f8098c19b62f2060a874d87202a1
2024-04-11 13:36:04 +00:00
eldritch horrors 725f5cd358 docs: redo content generation for mdbook and manual
manpages can be rendered using the markdown output of mdbook, the rest
of the manual can generated out of the main doc/manual source tree. we
still use lowdown to actually render manpages instead of eg mdbook-man
because lowdown does generate reasonably good manpages (though that is
also somewhat debatable, but they're a lot better than mdbook-man).

doing this not only lets us drastically simplify the lowdown pipeline,
but also remove all custom {{#include}} handling since now mdbook does
all of it, even for the manpage builds. even the lowdown wrapper isn't
entirely necessary because lowdown can take all wrapper arguments with
command line flags rather than bits of input file content.

This also implements running mdbook in Meson, in order to generate the
manpages. The mdbook outputs are also installed in the usual location.

Co-authored-by: Qyriad <qyriad@qyriad.me>

Change-Id: I60193f9fd0f15d48872f071af35855cda2a0f40b
2024-04-11 13:32:06 +00:00
Théophane Hufschmitt 07b627cc6d Merge pull request #10456 from NixOS/fixpermdeniedbind
Fix adding symlink to the sandbox paths

(cherry-picked from commit da1e977bf48cff2a635034c85e7c13878e38efc2)

Change-Id: I221c85a38180800ec6552d2e86a88df48398fad8
2024-04-11 15:43:58 +03:00
John Ericson aeee22e5a1 Merge pull request #10362 from obsidiansystems/maybeLstat
Factor out `nix::maybeLstat`

(cherry-picked from commit 9b88e5284608116b7db0dbd3d5dd7a33b90d52d7)

Change-Id: Id890525e847c890fad6593c594772826ac4d1d50
2024-04-11 15:43:41 +03:00
eldritch horrors a0875f6adf libstore: fix glossary link in documentation
this should be a link, not an anchor. it should also point to the
`gloss-store` element, not the `#gloss-store` element.

Change-Id: I1f2803093179549637e10f917ad73399a419131b
2024-04-11 02:34:45 +02:00
Qyriad 70af056de8 don't boost::to_few_args when an eval cached string type errors
Change-Id: Id3cb762622e156ceaf9d5bb95c2c704ffe474d0e
2024-04-10 18:30:12 -06:00
Rebecca Turner 99845e0e01 Merge "Print top-level errors normally in nix repl" into main 2024-04-10 15:40:03 +00:00
Qyriad 784a46654c Merge "docs: generalize manpage generation script as json-to-tree.py" into main 2024-04-10 13:40:47 +00:00
Qyriad 723ddadf92 docs: generalize manpage generation script as json-to-tree.py
This should be capable of replacing every invocation of
nix eval --write-to.

Change-Id: I60387bc9b0fc54a91244eddb639beaa64d705878
2024-04-10 15:04:34 +02:00
Rebecca Turner 9e7e927837 Print top-level errors normally in nix repl
Previously, errors while printing values in `nix repl` would be printed
in `«error: ...»` brackets rather than displayed normally:

```
nix-repl> legacyPackages.aarch64-darwin.pythonPackages.APScheduler
«error: Package ‘python-2.7.18.7’ in /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:335 is marked as insecure, refusing to evaluate.»
```

Now, errors will be displayed normally if they're emitted at the
top-level of an expression:

```
nix-repl> legacyPackages.aarch64-darwin.pythonPackages.APScheduler
error:
       … in the condition of the assert statement
         at /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/lib/customisation.nix:268:17:
          267|     in commonAttrs // {
          268|       drvPath = assert condition; drv.drvPath;
             |                 ^
          269|       outPath = assert condition; drv.outPath;

       … in the left operand of the OR (||) operator
         at /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/passthrufun.nix:28:45:
           27|         if lib.isDerivation value then
           28|           lib.extendDerivation (valid value || throw "${name} should use `buildPythonPackage` or `toPythonModule` if it is to be part of the Python packages set.") {} value
             |                                             ^
           29|         else

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: Package ‘python-2.7.18.7’ in /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:335 is marked as insecure, refusing to evaluate.
```

Errors emitted in nested structures (like e.g. when printing `nixpkgs`)
will still be printed in brackets.

Change-Id: I25aeddf08c017582718cb9772a677bf51b9fc2ad
2024-04-09 08:34:40 -07:00
Qyriad 4b99f43aeb meson: correctly differentiate $profiledir and $sysconfdir/profile.d
The configured sysconfdir is used to look for nix.conf, so it needs
to be /etc, and not $out/etc, so we separate out the place where shell
profile files are installed, which is the only other place sysconfdir is
at all used.

See lix-project/lix#231 (comment)
for more info.

Change-Id: Idbed8ba82e711b8a9d6b6127904befa27d58e279
2024-04-09 02:25:58 -06:00
Qyriad 2a98fe28cb meson: correctly set NIX_CONF_DIR to $sysconfdir/nix
Instead of $sysconfdir.

Fixes #231, but there's more to do in following commits to make
Meson-built Lix actually look in /etc/nix.

Change-Id: Ia8d627070f405843add46e05cff5134b76b8eb48
2024-04-09 02:25:58 -06:00
Qyriad 7e139c52dd format repl-overlays.nix to fix pre-commit checks
Change-Id: I9487f5832160c1093d8752f883c1be85f8310e0f
2024-04-09 02:21:26 -06:00
Rebecca Turner a95c62673b Merge "Add repl-overlays" into main 2024-04-09 05:29:48 +00:00
jade b0293f2de1 Merge "Add benchmarking scripts" into main 2024-04-09 03:01:00 +00:00
jade f4fc2b5d10 Add benchmarking scripts
These scripts were originally written by horrors, and have since been
hacked up a lot by jade. We are putting them up as a CL since it is
better to have checked in benchmarking scripts than to not have
benchmarking scripts.

cc: lix-project/lix#23

Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I95c2f9d24753ac468944c5781deec9508fd5cb8c
2024-04-08 19:50:24 -07:00
eldritch horrors e6aab36d60 docs: put the manual build into a known tmpdir
this isn't strictly necessary, but it'll make it a lot easier to put the
generated files used by the autoconf build system in this directory too.
doing this now already will make the meson transition a lot easier later

Change-Id: I5fb39eade2ff88b6093c9ee436c9e8db793e9448
2024-04-09 02:09:36 +00:00
eldritch horrors c1ae35d34a docs: don't compute SUMMARY.md during build
this would make meson build compatibility unnecessarily hard and
the cli does not change often enough to justify this complexity.

Change-Id: I17b1870cdf8538feeaa01a9945db97af2175a642
2024-04-09 02:09:36 +00:00
eldritch horrors d9bc197ff4 docs: don't compute rl-next.md during build
not sure why this was done the way it was considering that includes are
a feature the doc toolchain had previously. let's just always have some
kind of entry for the upcoming release in the dev manual builds even if
that means having a completely empty release notes chapter.

the release notes generation script isn't entirely functional right now
due to pre-commit hooks, but it's good enough for time being. we need a
better release process for notes anyway.

Change-Id: Ifda6912cf5233db013f72a30247a62d6f22b1565

Change-Id: I9eb347ec4aabc5be2b816ff0fd3e4be45f93b934
2024-04-09 02:09:36 +00:00
eldritch horrors 07ef9f9f5d docs: skip do custom include processing for mdbook
mdbook already does include processing of its own, and the custom
processing code has always admitted as much. we don't need it for
the mdbook build at this point if we run our preprocessors in the
right order, and maybe we can even have mdbook to return complete
pages to us that we only have to pass to lowdown without any more
preprocessing of our own.

Change-Id: Icd978acbc3b1e215fee8f062c53ab2cb2a222ab1
2024-04-09 02:09:36 +00:00
eldritch horrors 14088e1ec5 docs: remove many link-to-self
for some reason these three were anchors, not links, but had they been
links they wouldn't've worked because they're not defined anywhere but
here. in the print version of the manual they're duplicated many times
over (creating id collisions), so we should better remove them anyway.

Change-Id: I8988a7c32c812dee0f0b6d4953faa7cd1255228d
2024-04-09 02:09:36 +00:00
Rebecca Turner 727b43478c Add repl-overlays
Adds a `repl-overlays` option, which specifies files that can overlay
and modify the top-level bindings in `nix repl`. For example, with the
following contents in `~/.config/nix/repl.nix`:

    info: final: prev: let
      optionalAttrs = predicate: attrs:
        if predicate
        then attrs
        else {};
    in
      optionalAttrs (prev ? legacyPackages && prev.legacyPackages ? ${info.currentSystem})
      {
        pkgs = prev.legacyPackages.${info.currentSystem};
      }

We can run `nix repl` and use `pkgs` to refer to `legacyPackages.${currentSystem}`:

    $ nix repl --repl-overlays ~/.config/nix/repl.nix nixpkgs
    Lix 2.90.0
    Type :? for help.
    Loading installable 'flake:nixpkgs#'...
    Added 5 variables.
    Loading 'repl-overlays'...
    Added 6 variables.
    nix-repl> pkgs.bash
    «derivation /nix/store/g08b5vkwwh0j8ic9rkmd8mpj878rk62z-bash-5.2p26.drv»

Change-Id: Ic12e0f2f210b2f46e920c33088dfe1083f42391a
2024-04-08 17:11:47 -07:00
jade 1e74bffd5c pre-commit check for pragma once and ///@file
This is in our style guide, we can cheaply enforce it, let's do it.

```
$ pre-commit
check-case-conflicts.....................................................Passed
check-executables-have-shebangs..........................................Passed
check-headers............................................................Failed
- hook id: check-headers
- exit code: 1

Missing pattern @file in file src/libexpr/value.hh

We found some header files that don't conform to the style guide.

The Lix style guide requests that header files:
- Begin with `#pragma once` so they only get parsed once
- Contain a doxygen comment (`/**` or `///`) containing `@file`, for
  example, `///@file`, which will make doxygen generate docs for them.

  When adding that, consider also adding a `@brief` with a sentence
  explaining what the header is for.

For more details: https://wiki.lix.systems/link/3#bkmrk-header-files

check-merge-conflicts....................................................Passed
check-shebang-scripts-are-executable.....................................Passed
check-symlinks.......................................(no files to check)Skipped
end-of-file-fixer........................................................Passed
mixed-line-endings.......................................................Passed
no-commit-to-branch......................................................Passed
release-notes........................................(no files to check)Skipped
treefmt..................................................................Passed
trim-trailing-whitespace.................................................Passed
```

Fixes: lix-project/lix#233
Change-Id: I77150b9298c844ffedd0f85cc5250ae9208502e3
2024-04-08 16:10:57 -07:00
jade c58e3f826e pragma once and ///@file everything missing it
Change-Id: Ia1a72348336794b5fb9f2694dd750266089b904e
2024-04-08 15:40:12 -07:00
jade 54bc1f1b98 "but doctor, I AM the untrusted store": nix doctor had wrong trustedness
This probably snuck in in a refactor using truthiness or so. The
trustedness flag was having the optional fullness checked, rather than
the actual contained trust level.

Also adds some tests.

```
m1@6876551b-255d-4cb0-af02-8a4f17b27e2e ~ % nix store ping
warning: 'nix store ping' is a deprecated alias for 'nix store info'
Store URL: daemon
Version: 2.20.4
Trusted: 0
m1@6876551b-255d-4cb0-af02-8a4f17b27e2e ~ % nix doctor
warning: 'doctor' is a deprecated alias for 'config check'
[PASS] PATH contains only one nix version.
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: daemon
```

Fixes: lix-project/lix#232
Change-Id: I21576e2a0a755036edf8814133345987617ba3d0
2024-04-08 15:40:12 -07:00
jade 0d37b99a15 build: enable libstdc++ assertions
Benchmarks say that it does not regress performance by more than 1%
(which is where it gets really hard to measure accurately anyhow).

Meson appears to be planning to do this for us without asking us in a
release we will get in the future, and it seems good enough to ship
today:
https://mesonbuild.com/Release-notes-for-1-4-0.html#ndebug-setting-now-controls-c-stdlib-assertions

Benchmarks:

| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix` | 418.4 ± 25.0 | 396.9 | 451.2 | 1.01 ± 0.08 |
| `result/bin/nix --extra-experimental-features 'nix-command flakes' eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix` | 416.1 ± 23.9 | 397.1 | 445.4 | 1.00 |

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `GC_INITIAL_HEAP_SIZE=10g result-asserts/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 4.147 ± 0.021 | 4.123 | 4.195 | 1.00 |
| `GC_INITIAL_HEAP_SIZE=10g result/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 4.149 ± 0.027 | 4.126 | 4.215 | 1.00 ± 0.01 |

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 5.838 ± 0.023 | 5.799 | 5.867 | 1.01 ± 0.01 |
| `result/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 5.788 ± 0.044 | 5.715 | 5.876 | 1.00 |

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello` | 15.993 ± 0.081 | 15.829 | 16.096 | 1.01 ± 0.01 |
| `result/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello` | 15.897 ± 0.075 | 15.807 | 16.047 | 1.00 |

Fixes: lix-project/lix#4

Change-Id: Id3a6f38274ba94d5d10b09edd19dfd96bc3e7d5f
2024-04-08 15:40:12 -07:00
jade 06f17a5c78 release-notes: check with pre-commit
This required making the build-release-notes script understand how to
check multiple directories.

Change-Id: I057f5f636155ab6c6fb5755da5217b7e72249ece
2024-04-08 15:40:12 -07:00
jade 6fcab7ee95 pre-commit: stop using the flake
The flake for pre-commit-checks is rather questionable. We ignored
it so it uses our own nixpkgs and doesn't reimport nixpkgs. This should
save a couple of seconds of eval time!

Change-Id: I4584982beb32e0122f791fa29f6a544bdbb9e201
2024-04-08 15:29:23 -07:00
jade 860b34cf07 build-release-notes: support multiple PRs/CLs
Change-Id: I923e4af099d02d2324777cc9f7e6d38b130aac10
2024-04-08 15:29:19 -07:00
Rebecca Turner 5ec256625e Merge "Don't run libstore unit tests in project root" into main 2024-04-08 22:08:34 +00:00
Rebecca Turner 3910d4aa68 Merge "Fix REPL test parser nits" into main 2024-04-08 21:52:06 +00:00
Rebecca Turner cfbcf12276 Format Nix code with nixfmt
Change-Id: I61efeb666ff7481c05fcb247168290e86a250151
2024-04-08 13:00:00 -07:00
Rebecca Turner 12f2565f91 Don't run libstore unit tests in project root
This keeps the libstore unit tests from writing
`libstore-unit-tests.xml` to the project root.

Change-Id: I0d9909aabf9f3574cc1e72a5ae81daefba9a394b
2024-04-08 12:43:35 -07:00
Rebecca Turner 4a49a497eb Fix REPL test parser nits
Follow-up to https://gerrit.lix.systems/c/lix/+/546

Change-Id: Ie603f01e5520329bf879e061cea9e3fba45213fd
2024-04-08 11:14:34 -07:00
Rebecca Turner b323340538 Add nixfmt
Change-Id: I7f21695e3971cfd02b2cce0dd016ff6eb3389905
2024-04-08 09:42:34 -07:00
jade e55fc5af71 Merge "nix-doc -> lix-doc, make self-contained in package.nix" into main 2024-04-08 07:45:00 +00:00
Rebecca Turner 4600eabdae Merge "Rename some parse failure tests to parse-fail-*" into main 2024-04-08 06:33:51 +00:00
Rebecca Turner 8ed8b9aef0 Merge "Add PathsSetting" into main 2024-04-08 06:31:11 +00:00
jade 0145d45806 nix-doc -> lix-doc, make self-contained in package.nix
package.nix previously needed this callPackage'd externally, which
didn't make a lot of sense to us since this is an internal dependency.
Thus we changed it to make it more self contained.

Change-Id: I4935bc0bc80e1a132bc9b1519e917791da95037c
2024-04-08 04:05:13 +00:00
Robert Hensing b995c17f0e Merge pull request #10208 from 9999years/print-strings-directly
`:print` strings directly in `nix repl`

(cherry picked from commit 3539172fd2f7cee639ce46423c58beca4231f2db)
Change-Id: I1972f3bf3b56312851f38288509d371d37f21677
Upstream-PR: https://github.com/NixOS/nix/pull/10208
2024-04-07 19:10:43 -07:00
Rebecca Turner 6f863e8ccd Add PathsSetting
Change-Id: I1165f6ef033a5f757ca3716d3f8008ba36b01fd0
2024-04-07 18:22:15 -07:00
jade de20392c37 flake: remove dead code
Some of this code existed for installer tests, and indeed its removal is
an indication that our daemon cross-compatibility tests were removed.
Although these are not like, super critical tests, we would like to
restore them.

See: lix-project/lix#33
Change-Id: I75c733b25c00eca3a9676d498703bbfc1d6ec21b
2024-04-07 17:30:22 -07:00
Qyriad 84599abd4c Merge "flake: fix rl-next and rl-next-dev checks after 32eaa8a29 (fix nix flake check)" into main 2024-04-07 23:26:19 +00:00
Qyriad 0c5e2cfb33 package: just use fileset from lib
The following command is now sufficient to build Lix from outside of the
flake:

nix-build -E 'let pkgs = import <nixpkgs> { }; in pkgs.callPackage
./package.nix { build-release-notes = false; nix-doc = pkgs.callPackage
./nix-doc/package.nix { }; }'

Change-Id: Ie6b14b446480ac07c7266d4fba20042b04cc35b9
2024-04-07 23:10:05 +00:00
Qyriad 55e0c73695 flake: fix rl-next and rl-next-dev checks after 32eaa8a29 (fix nix flake check)
follow-up to 32eaa8a29[1] "flake: move release note checks to hydraJobs",
this commit fixes a load-bearing typo for`checks.rl-next` and
`checks.rl-next-dev`.

[1]: 32eaa8a2910793538deab31f85534faf7e722ef7

Change-Id: I9383ed21f7eccc337c0c2f65525418b735a94a1d
2024-04-07 17:05:51 -06:00
Qyriad 3ac2dd0613 package: put boehmgc patch logic in package.nix
In our view it really doesn't make sense to not have this in in
package.nix in some way. These patches aren't just for performance or
something -- Lix flat out doesn't build without these patches.

(Arguably that makes them a buildsystem responsibility as well, but that
can wait for when we're ready to start adding subproject fallback
dependency resolution to Meson.)

This is a step towards making `package.nix` more self-sufficient and
`callPackage`able without excessive external logic.

With this change the following command is enough to build Lix from out
of the flake:

nix-build -E 'let pkgs = import <nixpkgs> { }; in pkgs.callPackage
./package.nix { build-release-notes = false; inherit (pkgs.lib) fileset;
nix-doc = pkgs.callPackage ./nix-doc/package.nix { }; }'

Change-Id: Ia37fe8171f87d3293033de8be07d9bab12716f1d
2024-04-07 22:44:04 +00:00
Rebecca Turner 06e11778b5 Merge "Rewrite REPL test parser" into main 2024-04-06 21:36:26 +00:00
Qyriad 5bac308c7c meson: fix warm nix3 CLI manual generation
`nix eval --write-to` refuses to write to a directory that exists at
all, so now we generate in a temporary directory, and copy the generated
tree to the build directory. This is equivalent to what the Make
buildsystem did, actually, but hopefully more robust.

Future work: documenting the doc generation architecture in the
top-level meson.build outline comment.

Change-Id: Ic3eb6d26e3cc249a1c042fd3ced22d637ac66a69
2024-04-06 14:43:14 -06:00
eldritch horrors 1b5b9de04e flake: move release note checks to hydraJobs
having them in checks only does not run them in CI, which can cause
broken release notes entries to pass.

fixes #228

Change-Id: If0ba7b1be0b6525fc884a27e941cbc84b5a160f9
2024-04-06 15:51:52 +00:00
terru - a33c95be5b avoid markdown which the repl's :doc cannot handle
code blocks, if not surrounded by empty lines, have the language
tags (in these cases, always `nix`) show up in the output of :doc.

for example:

  nix-repl> :doc builtins.parseFlakeRef
  Synopsis: builtins.parseFlakeRef flake-ref

    Parse a flake reference, and return its exploded form.

    For example: nix builtins.parseFlakeRef
    "github:NixOS/nixpkgs/23.05?dir=lib"  evaluates to: nix { dir =
    "lib"; owner = "NixOS"; ref = "23.05"; repo = "nixpkgs"; type =
    "github"; }

is now instead:
  nix-repl> :doc builtins.parseFlakeRef
  Synopsis: builtins.parseFlakeRef flake-ref

    Parse a flake reference, and return its exploded form.

    For example:

      | builtins.parseFlakeRef "github:NixOS/nixpkgs/23.05?dir=lib"

    evaluates to:

      | { dir = "lib"; owner = "NixOS"; ref = "23.05"; repo = "nixpkgs"; type = "github"; }

(closes #225)

Change-Id: I0741aeb1006a5376bb2f663d202c7a4da7e38cce
2024-04-06 13:14:14 +00:00
Qyriad fae48884c5 Merge "package: add python to nativeBuildInputs" into main 2024-04-06 05:29:34 +00:00
Qyriad 9bd52cb952 Merge "always re-eval cached failures" into main 2024-04-06 05:20:14 +00:00
jade 9e166f3456 Merge "Create clang-tidy check to rename all our includes" into main 2024-04-06 05:01:21 +00:00
jade 43cf487c25 Create clang-tidy check to rename all our includes
It is a little bit scuffed, but it seems to produce correct results. We
can run it at a later date when we want to explode every in-flight
commit in existence and then need to filter-branch them.

Fixes: lix-project/lix#188

Change-Id: Id97e4651f78804a941d941df02c7c1b21ce453b6
2024-04-06 04:40:19 +00:00
Qyriad a4ef195a9f always re-eval cached failures
This is terrible UX, and frankly an eval failure should be a cache
invalidation anyway.

This removes the CachedEvalError type entirely.

Fixes #223.

Change-Id: I91f8003eabd0ea45003024e96d1de3c7ae8e49d8
2024-04-06 04:35:25 +00:00
Qyriad 11ad383327 package: add python to nativeBuildInputs
Commit c21d11ac0 "docs: replace sed invocation with an mdbook
preprocessor for @docroot@" added a direct build dependency on Python.
This has been accidentally working so far because Python is already a
*transitive* dependency of Lix's derivation.

Change-Id: I32d6b4f2665dbbfad7014613457dd58aa4ec73da
2024-04-05 22:28:27 -06:00
Qyriad b47c159aec Merge "docs: replace sed invocation with an mdbook preprocessor for @docroot@" into main 2024-04-06 01:05:32 +00:00
Qyriad 4e11b0d04d Merge "meson: add missing tests: ca, dyn-drv, plugins, libstoreconsumer" into main 2024-04-06 01:05:09 +00:00
eldritch horrors e9e1b6963c Merge changes I1fa30114,I3ca208b6,Ide4c6e00,I74c46b9f,I05fa6a9d, ... into main
* changes:
  Revert "libutil: drop Pool resources on exceptional free"
  Revert "libutil: remove Pool::Handle::bad"
  Revert "libstore: remove one Resource::good flag"
  Revert "libstore: using throwing finally in withFramedSink"
  Revert "libutil: allow graceful dropping of Pool::Handle"
  Revert "libutil: drop Fs{Source,Sink}::good"
  libutil: guard Finally against invalid exception throws
2024-04-05 23:17:18 +00:00
Qyriad 405e41e288 Merge "meson: run repl characterization test" into main 2024-04-05 20:57:42 +00:00
Rebecca Turner ee423f391d Rewrite REPL test parser
- Use a recursive descent parser so that it's easy to extend.
- Add `@args` to enable customizing command-line arguments
- Add `@should-start` to enable `nix repl` tests that error before
  entering the REPL
- Make sure to read all stdout output before comparing. This catches
  some extra output we were tossing out before!

Change-Id: I5522555df4c313024ab15cd10f9f04e7293bda3a
2024-04-05 13:14:21 -07:00
eldritch horrors 38dc6f5b69 Revert "libutil: drop Pool resources on exceptional free"
This reverts commit de2884b82b376d10de5c400d8e73bc7d98f195d2.

Change-Id: I1fa301149d7c2ed3d266a40c15b2d010e12e44e6
2024-04-05 20:13:02 +00:00
eldritch horrors 52f741c23a Revert "libutil: remove Pool::Handle::bad"
This reverts commit 792844fb861ea7367ac2316c78fec055363f2f9e.

Change-Id: I3ca208b62edfd5cd1199478f75cd2edf19a364f6
2024-04-05 20:13:02 +00:00
eldritch horrors 0b8a17cab6 Revert "libstore: remove one Resource::good flag"
This reverts commit 87249eb579bf57f4f09e9fca100588a4d6b90b4c.

Change-Id: Ide4c6e00c4155216a17e46671ff47151d7bb85b4
2024-04-05 20:13:02 +00:00
eldritch horrors ad30972867 Revert "libstore: using throwing finally in withFramedSink"
This reverts commit 491caad6f62c21ffbcdebe662e63ec0f72e6f3a2.

this is not actually legal for nix! throwing exceptions in destructors
is fine, but the way nix is set up we'll end up throwing the exception
we received from the remote *twice* in some cases, and such cases will
cause an immediate terminate without active exception.

Change-Id: I74c46b9f26fd791086e4193ec60eb1deb9a5bb2a
2024-04-05 20:13:02 +00:00
eldritch horrors c77b6e1fdd Revert "libutil: allow graceful dropping of Pool::Handle"
This reverts commit 8075541d82d05347321d35b9934ccee5f82142f4.

Change-Id: I05fa6a9de1308a4827a6557cf2807eb47ca64da6
2024-04-05 20:13:02 +00:00
eldritch horrors 821ad98beb Revert "libutil: drop Fs{Source,Sink}::good"
This reverts commit 1340807e30dba4b3972c31f02861bbaeaeb60e61.

Change-Id: I34d2a80eb3c3e9d79cb02b92cd1189da32d18cb6
2024-04-05 20:13:02 +00:00
eldritch horrors 6c777476c9 libutil: guard Finally against invalid exception throws
throwing exceptions is fine, but throwing exceptions during exception
handling is hard enough to do correctly that we should just forbid it
entirely out of an overabundance of caution. in cases where terminate
is the correct answer the users of Finally must call it manually now.

Change-Id: Ia51a2cb4a0638500550bfabc89cf01a6d8098983
2024-04-05 20:13:02 +00:00
Rebecca Turner 5081109592 Fix Boost with make build
This prevents the autotools build from discovering Boost in Homebrew
installation directories on macOS.

Change-Id: I624309165c9371c391fd657424ba4c4f3182b385
2024-04-05 11:09:59 -07:00
Qyriad 07e1ddc23d meson: add missing tests: ca, dyn-drv, plugins, libstoreconsumer
Change-Id: I6a74ebaf93697cb99aadd6b51538c2766b0a808a
2024-04-04 20:43:38 -06:00
Rebecca Turner 83d720b730 Merge "Make repl_characterization.cc constants string_views" into main 2024-04-04 23:08:09 +00:00
Rebecca Turner 0fec7d2db6 Rename some parse failure tests to parse-fail-*
These were mistakenly labeled `eval-fail-*`.

Note that the `lang.sh` runner passes `parse-fail-*` tests on stdin, so
filenames are removed from error messages.

Change-Id: I7f3a0d78b6cfa87af29aaa1b7af19d5a57fd4ade
2024-04-04 15:49:30 -07:00
Qyriad c355354772 docs: replace sed invocation with an mdbook preprocessor for @docroot@
We're not entirely clear on why the links preprocessor has to be done
*before* rather than after, but we assume it is probably that as a
builtin preprocessor it does some processing on the raw book source,
and not just the JSON data.

Also a real use for Python pattern matching? I know I was surprised too.

Change-Id: Ibe8b59e7b5bd5f357a655d8b4c5f0b0f58a67d6b
2024-04-04 21:43:19 +00:00
Qyriad 9166babbaf Revert "meson: move nix3 manpage generation into command-ref/"
This reverts commit 70954233743a233744787103d3211237a28ddbca.

This seems to have broken running ninja on warm build directories, which
is not what we want. Reverted until we figure out something better

Change-Id: I9623ae078917e7c59a930bf8044a216501d4bb20
2024-04-04 11:59:08 -06:00
Rebecca Turner b44514819b Merge "Add EscapeStringOptions and escapeString tests" into main 2024-04-04 17:41:13 +00:00
Qyriad 3f4c7a68f4 meson: run repl characterization test
This was mistakenly left out of 69c3363f2.

Fixes #205.

Change-Id: Ib9a2088c9eee3a192623bee107ba32221d048640
2024-04-03 17:05:30 -06:00
Qyriad c263554ec6 Merge "meson: move nix3 manpage generation into command-ref/" into main 2024-04-03 21:34:27 +00:00
Qyriad c96c5db030 meson: move nix3 manpage generation into command-ref/
This puts the generated files where they are for the make system.
This is in preparation for further meson-mdbook stuff.

Change-Id: I934df6854a80af5ccf381cf1da0bda0187a8bcfc
2024-04-03 13:52:56 -06:00
Lunaphied 7a1054fa5f repl: improve :doc builtin repl command to support lambdas.
For a long time `nix repl` has supported displaying documentation set on
builtins, however, it has long been convention to use Markdown comments
on Nix functions themselves for documentation. This exposes that
information to `nix repl` users in a nice and formatted way.

NixOS/rfcs#145 doc-comments are primarily what this feature is intended
to consume, however, support for lambda documentation in the repl is
experimental. We do our best effort to support the RFC here.

These changes are based on [the nix-doc library](https://github.com/lf-/nix-doc) and
are licensed under the terms described in the relevant source files.

Change-Id: Ic6fe947d39a22540705d890737e336c4720b0a22
2024-04-03 13:47:22 -06:00
Rebecca Turner 84018e1264 Make repl_characterization.cc constants string_views
Change-Id: I96455ee365799ae87d9ab433cf102c691616a45e
2024-04-03 16:25:54 +00:00
Rebecca Turner ea10fe7ab0 Add EscapeStringOptions and escapeString tests
Change-Id: I86ead2f969c9e03c9edfa51bbc92ee06393fd7d6
2024-04-03 16:25:54 +00:00
Rebecca Turner 56c7dfd652 Merge "Add treefmt pre-commit hook" into main 2024-04-03 00:09:09 +00:00
Qyriad fa4a0fc6a4 Merge "meson: implement perl bindings" into main 2024-04-02 19:08:24 +00:00
Qyriad 4f5e532b8b meson: implement perl bindings
Change-Id: Ie1bfb0aa784e6136a82d518a652d0ae60c4b047a
2024-04-02 11:22:42 -06:00
Rebecca Turner 71ef7fd428 Merge "Use upstream cachix/git-hooks.nix for pre-commit" into main 2024-04-02 17:20:33 +00:00
eldritch horrors d2aca15f31 doc: remove the -- unescaping hack
lowdown has an option to disable this behavior. use it.

Change-Id: I06d4201b7f528021ddb8e94ac6baff385402e080
2024-04-01 20:15:28 +00:00
eldritch horrors ed2bb919e7 meson: add switch to disable docs
the make build system can do this too.

Change-Id: I8c07d159cab54a8749c50dc33615f60bd251a86d
2024-04-01 20:15:28 +00:00
Rebecca Turner ca9acf8184 Use upstream cachix/git-hooks.nix for pre-commit
Change-Id: I6bbc7d6da9accd7d2daffa9d780384df7226670e
2024-04-01 10:38:50 -07:00
jade efbf5c81bb Merge "Delete autoconf generated code from tree" into main 2024-04-01 17:34:29 +00:00
Ilya K e9960e147b libexpr/flake: don't purple the flake.lock diff
Change-Id: I6f9471fb0ddd51fadb209ad970abd215238ba5c4
2024-04-01 08:57:50 +03:00
Artemis Tosini 9adefa9de4 flake: Add cmake to devShell
Meson fails to setup when cmake is not found.
Add cmake to the default devShell so meson build works.

Change-Id: I4d933efac9540c564f3171e43c23e7a645722ef7
2024-03-31 21:02:50 +00:00
Théophane Hufschmitt 6c29016a09 Merge pull request #9920 from 9999years/forbid-nested-debuggers
Forbid nested debuggers

(cherry picked from commit e164b39ee90fd655dbb7f479fdd4fbe38cc883bd)
Change-Id: Iff62f40fd251116516a63e2d3f9fb5b21480b16d
2024-03-31 17:28:25 +00:00
eldritch horrors 45623f077f libutil: drop Fs{Source,Sink}::good
setting this only on exceptions caused by actual fd access is not
sufficient to diagnose all errors (such as SerialisationError) in
some cases. this usually does not have any negative effects since
those errors will end up killing the process in another way. this
is not a reliable assumption though and we should be using proper
error handling (and closing connections more often, preferring to
close over keeping something open that might be in a weird state)

Change-Id: I1b792cd7ad8ba9ff0f6bd174945ab2575ff2208e
2024-03-31 16:42:40 +00:00
eldritch horrors f402c45cfa libutil: allow graceful dropping of Pool::Handle
not needed yet, but returning a resource from the exception handling
path that has ownership of a handle is currently not well-supported.
we could also add a default constructor to Handle, but then we would
also need to change the pool reference to a pointer. eventually that
should be done since now resources can be swapped between pools with
clever moves, but since that's not a problem yet we won't do it now.

Change-Id: I26eb06581f7be34569e9e67a33da736128d167af
2024-03-31 15:46:01 +00:00
jade 73507a7167 Merge changes Ib62d3d68,Ic3e7affe into main
* changes:
  Make things that can throw not noexcept anymore
  Fix various clang-tidy lints
2024-03-31 15:38:48 +00:00
jade 6165f21074 Merge "Add basic clang-tidy config" into main 2024-03-31 15:38:29 +00:00
eldritch horrors 2d3fd6b9ab libstore: using throwing finally in withFramedSink
the duplication of exception handling was added without justification,
so we can only assume that it was done like this because Finally could
not throw exceptions safely. since this has now been rectified we will
deduplicate this handler code again.

Change-Id: I40721f3378c0fd9f34e2914a16d383f6e2713b40
2024-03-31 14:23:15 +00:00
eldritch horrors dd06f9b792 libutil: make ~Finally noexcept(false)
this is supposed to act like a finally block does in other languages. a
finally block should be able to throw exceptions of its own rather than
just crashing the entire program when it throws it own exceptions. even
in the rare case of a finally throwing an unexpected exception it might
be better to report the exception from Finally instead of the original,
at least that can keep our program running instead of letting it crash.

Change-Id: Id42011e46b1df369152b4564938c0e93fa1acf32
2024-03-31 13:42:01 +00:00
eldritch horrors c777dcd1ae libstore: remove one Resource::good flag
usage of this flag previously kept connections open much longer than
necessary, and at the same time obscured that a connection was being
dropped when it *was* set. new variable names clarify this somewhat.

Change-Id: I11f6f08f37a5e4dc04ea6c6036ea589154b121c6
2024-03-31 00:52:41 +00:00
eldritch horrors 0b6d353474 libutil: remove Pool::Handle::bad
it was used incorrectly (not swapped on handle move), only used in one
place (that is now handled with exception handling detection in Handle
itself), and if ever reintroduced should be replaced with a different,
more understandable mechanism (like an explicit dropAsInvalid method).

Change-Id: Ie3e5d5cfa81d335429cb2ee5c3ad85c74a9df17b
2024-03-31 00:30:43 +00:00
eldritch horrors 862f20a4ba libutil: remove Pool::flushBad
this was never actually used, and bad design in the first place—why
should a bad resource be put back into the idle pool? just drop it.

Change-Id: Idab8774bee19dadae0209d404c4fb86dd4aeba1e
2024-03-31 00:07:09 +00:00
eldritch horrors 620de98d0c libutil: drop Pool resources on exceptional free
if a scope owning a resource does not gracefully drop that resource
while handling exceptions from deeper down the call stack we should
assume the resource is invalid state and drop it. currently it *is*
true that such cases do not cause resources to be freed, but thanks
to validator misuses this has so far not caused any larger problem.

Change-Id: Ie4f91bcd60a64d05c5ff9d22cc97954816d13b97
2024-03-30 23:40:45 +00:00
Rebecca Turner 4b730f328e Merge "Add pre-commit checks" into main 2024-03-30 22:26:59 +00:00
Qyriad 2041bdac2d perl: put perl dependencies in buildInputs instead of manually passing them to configure
Perl has an env hook[1]. Passing the paths manually without putting them
in buildInputs is harder to understand, plays less nicely with dev
shells, and is less build-generic.

Produced identical output on my x86_64-linux machine, and on my
aarch64-darwin machine was identical save for the derivation output path
which gets embedded into the .dylib Mach-O.

Change-Id: Ib313caa5a6f0b0e3154ce6f05379033920d0d290
2024-03-30 14:44:32 -06:00
eldritch horrors 5956f509b6 meson: install libexec binaries
Change-Id: I149892bf081e1569d7786f085e890bc3d2eb50e5
2024-03-30 14:34:22 +00:00
eldritch horrors 351c1a9c52 meson: build and install nix3 manpages
this notably does *not* install the `nix3-manpages` manpage the old
system generated, mostly because that page was empty and just a bug
with a coat of documentation paint.

Change-Id: I7a4248a72e7bb5e0cc925a6311a33b6b72589569
2024-03-30 14:26:47 +00:00
eldritch horrors bfbcc3352b meson: add manpages for old cli and config files
Change-Id: I93384ec774d1945a649f6aaf7cd967c3fb7197f9
2024-03-30 14:26:47 +00:00
eldritch horrors 65d544b1a6 build: extract make "functions" into scripts
we'll want to use these for the meson builds, and probably eventually
rewrite them in something that isn't plain shell. diffoscope confirms
that out/share and doc/share are equal before and after these changes

Change-Id: I49aa418fc8615cad86d67328e08c28a7405ec952
2024-03-30 14:26:47 +00:00
Rebecca Turner 655dae03de Add treefmt pre-commit hook
This lets us use different formatters for different filetypes.

Change-Id: Ib52383dd5097c8919a65e299aca2b5a55412223c
2024-03-29 22:57:43 -07:00
Rebecca Turner 2a98ba8b97 Add pre-commit checks
The big ones here are `trim-trailing-whitespace` and `end-of-file-fixer`
(which makes sure that every file ends with exactly one newline
character).

Change-Id: Idca73b640883188f068f9903e013cf0d82aa1123
2024-03-29 22:57:40 -07:00
jade 194a1b91af Make things that can throw not noexcept anymore
This does involve making a large number of destructors able to throw,
because we had to change it high in the class hierarchy. Oh well.

Change-Id: Ib62d3d6895b755f20322bb8acc9bf43daf0174b2
2024-03-29 20:26:38 -07:00
jade 99f159c536 Add basic clang-tidy config
This has not yet had all the warnings Obliterated, but it is a start and
is not *super* far away from being able to run the current configuration
in CI, which will catch some limited number of mistakes.

I tried the meson clang-tidy target and it seems to fail to find flags
for several files, which seems broken. Unsure what is up with that, but
we can use run-clang-tidy or other tooling instead.

We have an extremely annoying situation with the lexer table, which
means that the lexer probably must be moved to another directory with
its own .clang-tidy file to disable the lints in it, *or* write scuffed
code that prepends a disable comment to the top of the generated file.
None of the comment-based lint disabling features work since yacc dumps
a bunch of non compliant code at the top of the file before anything the
user can control.

Change-Id: I1d2aa6ec32deb1db1fbd581127334db1b972323c
2024-03-29 20:26:38 -07:00
jade 1fa6a3e335 Fix various clang-tidy lints
* some things that can throw are marked noexcept
  yet the linter seems to think not. Maybe they can't throw in practice.
  I would rather not have the UB possibility in pretty obvious cold
  paths.
* various default-case-missing complaints
* a fair pile of casts from integer to character, which are in fact
  deliberate.
* an instance of <https://clang.llvm.org/extra/clang-tidy/checks/bugprone/move-forwarding-reference.html>
* bugprone-not-null-terminated-result on handing a string to curl in
  chunks of bytes. our usage is fine.
* reassigning a unique_ptr by CRIMES instead of using release(), then
  using release() and ignoring the result. wild. let's use release() for
  its intended purpose.

Change-Id: Ic3e7affef12383576213a8a7c8145c27e662513d
2024-03-29 20:26:38 -07:00
jade a4f5bb951d Merge "Document ExprLambda fields" into main 2024-03-30 03:22:53 +00:00
Rebecca Turner 0b4e738944 Add a justfile to make meson friendly :)
Change-Id: Id6e4528392266c6f2444e030b67293abe297ed17
2024-03-30 03:03:46 +00:00
Jade Lovelace 817322fbfb Document ExprLambda fields
We got confused what formals did and had to briefly figure it out. We
should just have docs, so these are some.

Change-Id: If3e794a401e69d022785cbfa0b0c2e2284f41f58
2024-03-29 18:29:59 -07:00
Rebecca Turner a5a25894c1 Move escapeString to its own file
Change-Id: Ie5c954ec73c46c9d3c679ef99a83a29cc7a08352
2024-03-29 16:26:29 -07:00
eldritch horrors 5a54b0a20c meson: install systemd files
Change-Id: Idacf602fd379c82a051f00df2293cb02c8b286d4
2024-03-29 20:10:33 +00:00
eldritch horrors e28dc26084 meson: install shell files
Change-Id: I7c30690e5763d095cf7444333f7b687509051c5f
2024-03-29 20:10:33 +00:00
jade 7ccf49c021 Delete autoconf generated code from tree
Since we don't have a ./configure checked in, it makes no sense to keep
any of these scripts since people are already going to be running
autoreconf anyway.

Plus they will be completely deletable when meson shows up.

This fixes `autoreconfPhase` causing git diffs.

Change-Id: Ibb2aee422c562a23faadfdedb55b5c18c41a9420
2024-03-29 12:33:47 -07:00
eldritch horrors 1da1f501fc meson: fix state-dir default value
the autoconf build system defaults to /nix/var, not /nix/var/nix. the
latter is only used in libstore, so we'll move the extra segment there.

Change-Id: Idfbc988ee302355982abdcd51d6d7b5d5d661c0d
2024-03-29 19:14:23 +00:00
Winter Cute 6646b80396 meson: add missing explicit dependency on nlohmann_json
Without this, the Meson setup won't bail out if nlohmann_json is
missing, leading to subpar DX (and maybe worse, but I'm not entirely
sure).

Change-Id: I5913111060226b540dcf003257c99a08e84da0de
2024-03-29 14:16:58 -04:00
Rebecca Turner 9d97d1cb68 Merge "Set MAKEFLAGS=-j and GTEST_BRIEF in .envrc" into main 2024-03-29 17:09:13 +00:00
Rebecca Turner 877750b7c5 Merge "Move DebugChar into its own file" into main 2024-03-29 16:20:14 +00:00
eldritch horrors 6e5db5e4a2 meson: install missing/generated headers
one headers (args/root.hh) was simply missing, and the generated headers
were not installed. not all of them *should* be installed either, only a
select few (and sadly this needs a custom target for each one, it seems)

Change-Id: I37b25517895d0e5e521abc1202fa65624de57ed1
2024-03-29 02:45:48 +00:00
eldritch horrors 69bfd21e20 meson: install pkg-config files for libraries
Change-Id: I14b9d81d09f188eacfb9c68bcfb84751c18e3779
2024-03-29 02:45:48 +00:00
eldritch horrors 86b954a7af meson: increase functional test timeout
sometimes these fail with timeouts on loaded machines. let's up the
timeouts until we can pull the tests apart to more reasonable sizes

Change-Id: I2dfff2183cc1f3ff5e6107f43748ac046fe00d05
2024-03-29 02:19:36 +00:00
Rebecca Turner f9d5079c69 Set MAKEFLAGS=-j and GTEST_BRIEF in .envrc
- Enable parallel builds by default (and allow using environment
  variables to override `make` variables)
  - Hopefully we can get rid of this once we have Meson
- Set `GTEST_BRIEF=1`
  - This only shows failed tests, instead of listing every test on its
    own line.

```
$ GTEST_BRIEF=1 make check
[==========] 328 tests from 15 test suites ran. (37 ms total)
[  PASSED  ] 328 tests.
```

Change-Id: Id8103a8f24a9681be2be87e1b4df6fd5fdd7e4fd
2024-03-28 18:17:28 -07:00
Rebecca Turner 236bc046ba Merge "Remove HintFmt::operator%" into main 2024-03-29 01:13:45 +00:00
raito 55350bd68d Merge "feat: unprivileged read-only open of SQLite DB" into main 2024-03-29 00:49:17 +00:00
Rebecca Turner 5ec2efb686 Move DebugChar into its own file
Change-Id: Ia40549e5d0b78ece8dd0722c3a5a032b9915f24b
2024-03-28 15:54:12 -07:00
Rebecca Turner 62332c1250 Merge "Move shell_words into its own file" into main 2024-03-28 22:49:00 +00:00
Qyriad 81e50fef70 Merge "meson: implement functional tests" into main 2024-03-28 20:38:05 +00:00
jade 61d394e344 Merge "Build with traps on signed overflow" into main 2024-03-28 20:27:32 +00:00
jade ae065a992d Merge "progress-bar.cc: fix signed overflow" into main 2024-03-28 15:21:11 +00:00
jade 14207e1cf8 Build with traps on signed overflow
This is UB, we should not be doing it, and we can cheaply turn it into
crashes reliably. We would much rather have crashes than the program
doing something silly.

Benchmarks, but i wonder if they are nonsense because they get identical
times across compilers?!

| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `result-clang/bin/nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix` | 375.5 ± 24.0 | 353.8 | 408.8 | 1.00 |
| `result-gcc/bin/nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix` | 407.9 ± 26.0 | 385.1 | 449.5 | 1.09 ± 0.10 |
| `result-clangsan/bin/nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix` | 382.2 ± 26.6 | 354.9 | 419.0 | 1.02 ± 0.10 |
| `result-gccsan/bin/nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix` | 408.6 ± 24.6 | 384.5 | 441.9 | 1.09 ± 0.10 |

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `result-clang/bin/nix search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello` | 17.199 ± 0.167 | 16.930 | 17.499 | 1.01 ± 0.01 |
| `result-gcc/bin/nix search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello` | 17.409 ± 0.126 | 17.242 | 17.633 | 1.02 ± 0.01 |
| `result-clangsan/bin/nix search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello` | 17.080 ± 0.137 | 16.879 | 17.350 | 1.00 |
| `result-gccsan/bin/nix search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello` | 17.396 ± 0.160 | 17.131 | 17.660 | 1.02 ± 0.01 |

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `result-clang/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 6.267 ± 0.069 | 6.197 | 6.415 | 1.02 ± 0.01 |
| `result-gcc/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 6.232 ± 0.045 | 6.180 | 6.311 | 1.01 ± 0.01 |
| `result-clangsan/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 6.162 ± 0.020 | 6.133 | 6.196 | 1.00 |
| `result-gccsan/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 6.229 ± 0.031 | 6.199 | 6.289 | 1.01 ± 0.01 |

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `GC_INITIAL_HEAP_SIZE=10g result-clang/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 4.683 ± 0.044 | 4.630 | 4.761 | 1.00 |
| `GC_INITIAL_HEAP_SIZE=10g result-gcc/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 4.750 ± 0.041 | 4.680 | 4.812 | 1.01 ± 0.01 |
| `GC_INITIAL_HEAP_SIZE=10g result-clangsan/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 4.703 ± 0.040 | 4.640 | 4.760 | 1.00 ± 0.01 |
| `GC_INITIAL_HEAP_SIZE=10g result-gccsan/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 4.766 ± 0.037 | 4.727 | 4.844 | 1.02 ± 0.01 |

Change-Id: I616ca3eab670317587d47b41870d8ac963c019ae
2024-03-27 23:54:04 -07:00
jade ffbad9b762 progress-bar.cc: fix signed overflow
this was caused by the use of std::chrono::duration::max() which gets
multiplied by some ratio to calculate nanoseconds to wait. then, it
explodes because that is a signed integer overflow. this was definitely
a bug.

error below:

/nix/store/fdiknsmnnczx6brsbppyljcs9hqckawk-gcc-12.3.0/include/c++/12.3.0/bits/chrono.h:225:38: runtime error: signed integer overflow: 9223372036854775807 * 1000000 cannot be represented in type 'long'
    #0 0x736d376b2b69 in std::chrono::duration<long, std::ratio<1l, 1000000000l>> std::chrono::__duration_cast_impl<std::chrono:
:duration<long, std::ratio<1l, 1000000000l>>, std::ratio<1000000l, 1l>, long, false, true>::__cast<long, std::ratio<1l, 1000l>>(
std::chrono::duration<long, std::ratio<1l, 1000l>> const&) /nix/store/fdiknsmnnczx6brsbppyljcs9hqckawk-gcc-12.3.0/include/c++/12
.3.0/bits/chrono.h:225:38
    #1 0x736d376b2b69 in std::enable_if<__is_duration<std::chrono::duration<long, std::ratio<1l, 1000000000l>>>::value, std::chr
ono::duration<long, std::ratio<1l, 1000000000l>>>::type std::chrono::duration_cast<std::chrono::duration<long, std::ratio<1l, 10
00000000l>>, long, std::ratio<1l, 1000l>>(std::chrono::duration<long, std::ratio<1l, 1000l>> const&) /nix/store/fdiknsmnnczx6brs
bppyljcs9hqckawk-gcc-12.3.0/include/c++/12.3.0/bits/chrono.h:270:9
    #2 0x736d376b2b69 in std::enable_if<__is_duration<std::chrono::duration<long, std::ratio<1l, 1000000000l>>>::value, std::chr
ono::duration<long, std::ratio<1l, 1000000000l>>>::type std::chrono::ceil<std::chrono::duration<long, std::ratio<1l, 1000000000l
>>, long, std::ratio<1l, 1000l>>(std::chrono::duration<long, std::ratio<1l, 1000l>> const&) /nix/store/fdiknsmnnczx6brsbppyljcs9
hqckawk-gcc-12.3.0/include/c++/12.3.0/bits/chrono.h:386:14
    #3 0x736d376b2b69 in std::cv_status std::condition_variable::wait_for<long, std::ratio<1l, 1000l>>(std::unique_lock<std::mut
ex>&, std::chrono::duration<long, std::ratio<1l, 1000l>> const&) /nix/store/fdiknsmnnczx6brsbppyljcs9hqckawk-gcc-12.3.0/include/
c++/12.3.0/condition_variable:164:6
    #4 0x736d376b1ee9 in std::cv_status nix::Sync<nix::ProgressBar::State, std::mutex>::Lock::wait_for<long, std::ratio<1l, 1000
l>>(std::condition_variable&, std::chrono::duration<long, std::ratio<1l, 1000l>> const&) /home/jade/lix/lix/src/libutil/sync.hh:
65:23
    #5 0x736d376b1ee9 in nix::ProgressBar::ProgressBar(bool)::'lambda'()::operator()() const /home/jade/lix/lix/src/libmain/prog
ress-bar.cc:99:27
    #6 0x736d36de25c2 in execute_native_thread_routine (/nix/store/a3zlvnswi1p8cg7i9w4lpnvaankc7dxx-gcc-12.3.0-lib/lib/libstdc++
.so.6+0xe05c2)
    #7 0x736d36b6b0e3 in start_thread (/nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libc.so.6+0x8b0e3) (BuildId
: 287831bffdbdde0ec25dbd021d12bdfc0ab9f5ff)
    #8 0x736d36bed5e3 in __clone (/nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libc.so.6+0x10d5e3) (BuildId: 28
7831bffdbdde0ec25dbd021d12bdfc0ab9f5ff)

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /nix/store/fdiknsmnnczx6brsbppyljcs9hqckawk-gcc-12.3.0/include/c++/12.3.
0/bits/chrono.h:225:38 in

Change-Id: Ia0303242cdfd5d49385ae9e99718d709625a4633
2024-03-27 22:56:04 -07:00
Winter Cute 80405d0626 Stop vendoring toml11
We don't apply any patches to it, and vendoring it locks users into
bugs (it hasn't been updated since its introduction in late 2021).

Closes lix-project/lix#164

Change-Id: Ied071c841fc30b0dfb575151afd1e7f66970fdb9
2024-03-27 21:04:00 -04:00
Qyriad 038daad218 meson: implement functional tests
Functional tests can be run with
`meson test -C build --suite installcheck`.

Notably, functional tests must be run *after* running `meson install`
(Lix's derivation runs the installcheck suite in installCheckPhase so it
does this correctly), due to some quirks between Meson and the testing
system.

As far as I can tell the functional tests are meant to be run after
installing anyway, but unfortunately I can't transparently make
`meson test --suite installcheck` depend on the install targets.

The script that runs the functional tests, meson/run-test.py, checks
that `meson install` has happened and fails fast with a (hopefully)
helpful error message if any of the functional tests are run before
installing.

TODO: this change needs reflection in developer documentation

Change-Id: I8dcb5fdfc0b6cb17580973d24ad930abd57018f6
2024-03-27 18:37:50 -06:00
jade edba570664 HOT SALE: 15% off your build times!
This was achieved by running maintainers/buildtime_report.sh on the
build directory of a meson build, then asking "why the heck is json
eating our build times", and strategically moving the json using bits
out of widely included headers.

It turns out that putting literally any metrics whatsoever into the
build had immediate and predictable results.

Results are 1382.5s frontend time -> 1175.4s frontend time, back end
time approximately invariant.

Related: lix-project/lix#159

Change-Id: I7edea95c8536203325c8bb4dae5f32d727a21b2d
2024-03-27 03:52:57 +00:00
jade 412a9c9f67 Enable clang build timing analysis
I didn't enable this by default for clang due to making the build time
10% worse or so. Unfortunate, but tbh devs for whom 10% of build time is
not *that* bad should probably simply enable this.

Change-Id: I8d1e5b6f3f76c649a4e2f115f534f7f97cee46e6
2024-03-27 03:52:57 +00:00
jade 50c6feeb77 Add release notes system for dev facing release notes
We keep changing dev stuff and we probably should keep the news up to
date?

Change-Id: I819da6a29f1c56c8ab8d758c159a9c96164cb04e
2024-03-27 03:52:57 +00:00
eldritch horrors 0436f4cfa6 flake: always build release notes in devshell
Change-Id: I0e02567fe8f102a8a8f1558aa094eefacdac9393
2024-03-27 03:09:14 +00:00
eldritch horrors 279e30e7ef build: replace changelog-d with local script
hacking changelog-d to support not just github but also forgejo and
gerrit is a lot more complicated than it's worth, even moreso since
the entire thing can just as well be done with ~60 lines of python.
this new script is also much cheaper to instantiate (being python),
so having it enabled in all shells is far less of a hassle.

we've also adjusted existing release notes that referenced a gerrit
cl to auto-link to the cl in question, making the diff a bit bigger

closes lix-project/lix#176

Change-Id: I8ba7dd0070aad9ba4474401731215fcf5d9d2130
2024-03-27 03:09:14 +00:00
eldritch horrors 8fd02df90d manual: fix release notes
fix key spelling errors, type errors, things-should-not-be-comments errors

Change-Id: I3ce12873aa78002bca686bd88404771895b05d30
2024-03-27 03:09:14 +00:00
Rebecca Turner aee3d639b5 Move shell_words into its own file
Change-Id: I34c0ebfb6dcea49bf632d8880e04075335a132bf
2024-03-26 16:44:04 -07:00
Rebecca Turner 8e63eca912 Remove HintFmt::operator%
Change-Id: Ibcf1a7848b4b18ec9b0807628ff229079ae7a0fe
2024-03-26 15:40:05 -07:00
jade da22dbc333 Merge "Issue importer: do not notify" into main 2024-03-26 22:04:48 +00:00
Ilya K fa3088a878 Merge "envrc: improve" into main 2024-03-26 20:49:40 +00:00
jade a5254186fd Merge "libstore/filetransfer: use Lix UA and unnix error message" into main 2024-03-26 20:46:27 +00:00
Qyriad 207d24da4e Merge "meson: implement unit tests" into main 2024-03-26 20:06:09 +00:00
Ilya K 312cee142a envrc: improve
- add shellcheck hint
- load .envrc.local if one exists
- use a variable to allow choosing the shell variant

Change-Id: Iea34e5a800f5d463e5792020c5c293b8b3071ca5
2024-03-26 19:40:43 +03:00
raito 80b66b5065 libstore/filetransfer: use Lix UA and unnix error message
Once this commit lands, we are even more visible in analytics FWIW.

Change-Id: Id7e0c162315d0f191edbea9cb5fb82ce363704b9
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-03-26 16:06:27 +00:00
jade 531b8d0ab8 Merge "libmain: version printer uses Lix instead of Nix" into main 2024-03-26 16:06:15 +00:00
Ilya K 7170e9f947 envrc: add
direnv good.

Change-Id: I81bd0086b847c6ee642a201fbc991a1f63fc7d7a
2024-03-26 15:19:37 +03:00
Ilya K a69f6e185a build-remote: fix format string shenanigans
HintFmt(string) invokes the HintFmt("%s", literal) constructor,
which is not what we want here. Add a constructor with a proper name
and call that.

Next step: rename all the other ones to HintFmt::literal(string).

Fixes: lix-project/lix#178

Change-Id: If52d2eb8864ceb8663e05992e9d1fffef573d6b8
2024-03-26 07:58:24 +00:00
Lunaphied 73624f7d9c Merge "Merge pull request #8817 from iFreilicht/flake-update-lock-overhaul" into main 2024-03-26 00:45:49 +00:00
Qyriad e1ffe56793 meson: implement unit tests
Unit tests can be run with `meson test -C build --suite check`.
`--suite check` is optional, as right now that's the only test suite,
but when functional tests are added those will be in a separate suite.

Change-Id: I7f22f1cde4b489b3cdb5f9a36a544f0c409fcc1f
2024-03-26 00:43:33 +01:00
Théophane Hufschmitt 86881226b0 Merge pull request #8817 from iFreilicht/flake-update-lock-overhaul
Overhaul `nix flake update` and `nix flake lock` UX

(cherry picked from commit 12a0ae73dbb37becefa5a442eb4532ff0de9ce65)
Change-Id: Iff3b4f4235ebb1948ec612036b39ab29e4ca22b2
2024-03-25 17:36:24 -06:00
eldritch horrors c3a5f937d3 flake.nix: linearize meson builds
parallel meson builds need too much ram. linearize them for now, and
hopefully we can remove the make build system and this hack soonish.

Change-Id: I42c092db8b0c63680e77da2263cdfe9e7f6575be
2024-03-25 21:48:55 +00:00
Qyriad 787c4397f1 Merge "issue template: use nix --version instead of nix-env --version" into main 2024-03-25 21:47:53 +00:00
Qyriad 1da1aa5045 issue template: use nix --version instead of nix-env --version
`nix --version` doesn't require `nix-command` experimental feature to
run and we could all do with less nix-env

Change-Id: I90748d591c574d96eda46591e9f9ce828311da29
2024-03-25 12:59:30 -06:00
Eelco Dolstra aa7653608d Minor cleanup in libexpr/flake/flake.cc
(cherry picked from commit 05316d401fa509557c71140e17bb19814412fcb8)
Change-Id: I6ba0b55709f5fe21beb4e9f3bf72ee28715d15f3
2024-03-25 15:30:36 +00:00
Eelco Dolstra b525d0f20c Input: Replace markFileChanged() by putFile()
Committing a lock file using markFileChanged() required the input to
be writable by the caller in the local filesystem (using the path
returned by getSourcePath()). putFile() abstracts over this.

(cherry picked from commit 95d657c8b3ae4282e24628ba7426edb90c8f3942)
Change-Id: Ie081c5d9eb4e923b229191c5e23ece85145557ff
2024-03-25 15:30:36 +00:00
John Ericson 3d065192c0 Overhaul completions, redo #6693 (#8131)
As I complained in
https://github.com/NixOS/nix/pull/6784#issuecomment-1421777030 (a
comment on the wrong PR, sorry again!), #6693 introduced a second
completions mechanism to fix a bug. Having two completion mechanisms
isn't so nice.

As @thufschmitt also pointed out, it was a bummer to go from `FlakeRef`
to `std::string` when collecting flake refs. Now it is `FlakeRefs`
again.

The underlying issue that sought to work around was that completion of
arguments not at the end can still benefit from the information from
latter arguments.

To fix this better, we rip out that change and simply defer all
completion processing until after all the (regular, already-complete)
arguments have been passed.

In addition, I noticed the original completion logic used some global
variables. I do not like global variables, because even if they save
lines of code, they also obfuscate the architecture of the code.

I got rid of them  moved them to a new `RootArgs` class, which now has
`parseCmdline` instead of `Args`. The idea is that we have many argument
parsers from subcommands and what-not, but only one root args that owns
the other per actual parsing invocation. The state that was global is
now part of the root args instead.

This did, admittedly, add a bunch of new code. And I do feel bad about
that. So I went and added a lot of API docs to try to at least make the
current state of things clear to the next person.

--

This is needed for RFC 134 (tracking issue #7868). It was very hard to
modularize `Installable` parsing when there were two completion
arguments. I wouldn't go as far as to say it is *easy* now, but at least
it is less hard (and the completions test finally passed).

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Change-Id: If18cd5be78da4a70635e3fdcac6326dbfeea71a5
(cherry picked from commit 67eb37c1d0de28160cd25376e51d1ec1b1c8305b)
2024-03-25 15:30:36 +00:00
Tom Bereknyei 4494f9097f feat: notation to refer to no attribute search prefix
An attrPath prefix of "." indicates no need to try default attrPath prefixes. For example `nixpkgs#legacyPackages.x86_64-linux.ERROR` searches through

```
trying flake output attribute 'packages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute ''
trying flake output attribute 'legacyPackages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
trying flake output attribute 'legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
```

And there is no way to specify that one does not want the automatic
search behavior. Now one can specify
`nixpkgs#.legacyPackages.x86_64-linux.ERROR` to only refer to the rooted
attribute path without any default injection of attribute search path or
system.

Change-Id: Iac1334e1470137b7ce11dcf845513810230638ec
(cherry picked from commit d4aed18883b361133607296fb6cd789c47427a38)
2024-03-25 15:30:36 +00:00
Lunaphied d3d7489571 Merge "Improve new CLI UX by supporting short -E flag for --expr" into main 2024-03-25 14:13:44 +00:00
raito ad8a4b380e libmain: version printer uses Lix instead of Nix
Change-Id: I014ff24b900c0b9a48b7a63c8bb8b86cde3ebe54
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-03-25 08:04:31 +00:00
jade 5a1c35f907 Merge "Restore system-install profile files from the previous installer" into main 2024-03-25 04:17:36 +00:00
jade dee6b75702 Restore system-install profile files from the previous installer
These files are required to get Nix in PATH in existing multi-user installs using
the legacy installer. We really could use some tests.

Cc: lix-project/lix#33

This partially reverts commit 93cc063344.

Fixes: lix-project/lix#173

Change-Id: Iafb55280596732670a432f604b897f48562868e4
2024-03-25 04:17:14 +00:00
Lunaphied 185ecf1f45 Improve new CLI UX by supporting short -E flag for --expr
Change-Id: I55881c846da8416a92a14deedfa5bbbf09a122fb
2024-03-24 21:17:51 -06:00
eldritch horrors c856b82c2e libstore: despecialcase protocol version check
protocol versions are sent as u64. on the peer we read them as uint64,
check that the upper half is 0, and throw an exception if not. we then
read an arbitrary amount of data from the peer and dump it to the user
terminal. this is a little bit ridiculous, can never happen in correct
implementation, and is severly untested. let us just drop it entirely.

Change-Id: Ibd2f53a765341ed6439d40d9d1eac11e79c6b5e3
2024-03-24 18:45:22 +00:00
eldritch horrors 3e428f2289 libstore: un-inline copyNAR expansions
these are copies of copyNAR with only some variables renamed.

Change-Id: I98ddd7a98250fa5d304e18e1debf417e9f7768dd
2024-03-24 15:24:02 +01:00
jade 33da9c09c8 Issue importer: do not notify
This uses the forgejo patch we have for dont_notify on issue creation on
the api, and indeed does not notify, so we can simply run the script
safely :D

Fixes: lix-project/web-services#38

Change-Id: I86bcbf9b4499b439b79b82af84ee7df0f8eb3298
2024-03-23 19:03:34 -07:00
jade 946fc12e4e Revert "Merge pull request #9476 from alois31/restore-progress-bar"
Observed to regress nix repl attrset printing with narrow windows.

This reverts commit a2d5e803cf.

Fixes: lix-project/lix#168

Change-Id: I8e0031475b4ec26d6a71014357d973578b70815c
2024-03-23 18:04:29 -07:00
eldritch horrors 652f52f071 libutil: don't memset 64k in drainFD
this is not needed and introduces a bunch of memset calls, making up for
3% of valgrind cycle estimation *alone*. real-world impact is a lot
lower on our test machine, but we suspect that less powerful machines
would see an impact from dropping this.

Change-Id: Iad10e9d556e64fdeb0bee0059a4e52520058d11e
2024-03-23 22:17:46 +00:00
raito 8044540c42 feat: unprivileged read-only open of SQLite DB
If the state SQLite database is configured to use a write-ahead-log, it
creates WAL files in the state directory.

When the state SQLite database is closed by the `nix-daemon` after
builds, those files are removed.

When an unprivileged user would like to open _in read only_ that
database, they cannot do so because they would need to create those WAL
files and they do not have the permission to do so.

For this, SQLite offers a "persistent WAL" feature [1] to leave the WAL
files around, even after closing the database.

This CL enable the persistent WAL mode.

Fixes: https://github.com/NixOS/nix/issues/10300
[1]: https://www.sqlite.org/wal.html

Change-Id: Id8ae534d7d2290457af28782e5215222ae051fe5
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-03-23 15:07:48 +01:00
Qyriad b4d07656ff build: optionally build and install with meson
This commit adds several meson.build, which successfully build and
install Lix executables, libraries, and headers. Meson does not yet
build docs, Perl bindings, or run tests, which will be added in
following commits. As such, this commit does not remove the existing
build system, or make it the default, and also as such, this commit has
several FIXMEs and TODOs as notes for what should be done before the
existing autoconf + make buildsystem can be removed and Meson made the
default. This commit does not modify any source files.

A Meson-enabled build is also added as a Hydra job, and to
`nix flake check`.

Change-Id: I667c8685b13b7bab91e281053f807a11616ae3d4
2024-03-22 08:36:50 -06:00
jade a7161b6c0f Merge "clang-tidy check infrastructure" into main 2024-03-21 12:28:13 -06:00
Qyriad fab55aff0e flake: fix arm32 Linux cross devShell on macOS (fix nix flake check)
Change-Id: Iacac97de0b3d5f2df52c7bc985148624a351f45d
2024-03-21 08:22:38 -06:00
eldritch horrors 22e3f0e987 libexpr: unbreak PosTable performance
this was mostly an inconvenience for error reporting, but fully broke
the debugger (because the debugger does *a lot* of eager position
resolution). copying the line offsets into a local and filling that
local when empty without also storing the calculated offsets back does
kind of ... not cache anything.

fixes lix-project/lix#165

Change-Id: Iccb0ba193ce2f15c832978daecf7b9bebbbe8585
2024-03-20 13:45:36 +01:00
jade 5a28d70d1e Merge "un-ups your start" into main 2024-03-19 14:29:57 -06:00
eldritch horrors d9a83886f9 libutil: remove exception handling workingness check
within lix itself this problem is caught by the test suite. outside of
lix itself three cases can be had: either the problem is fully inside
lix libs, fully inside user code, or it exists at the boundary. the
first is caught by the test suite, the second isn't caught at all, and
the third is something lix should not be responsible for.

Change-Id: I95aa35d8cb6f0ef5816a2941c467bc0c15916063
2024-03-19 06:09:42 -06:00
jade 4050245faa Merge changes I72c945ca,I2138bb4d,Ib96749f3 into main
* changes:
  Release notes for builtins.nixVersion change
  un-nixes ur lix, a little
  issue importer: list issues that are *not* closed when finding existing issues
2024-03-18 20:19:53 -06:00
jade 985bd5eb9f un-ups your start
We do not need upstart, it is so thoroughly obsolete that we should not
care about supporting it.

Change-Id: Ie0ca084740845555fddffacc899cd129c9a4c1fe
2024-03-18 18:28:08 -07:00
jade 20b4a97af3 Release notes for builtins.nixVersion change
Change-Id: I72c945cab464d26d73f5594ef0a4bb2184545da4
2024-03-18 18:26:52 -07:00
jade 30233d87f9 un-nixes ur lix, a little
I didn't really go attack the docs because we need to pull a bunch of
PRs. I went looking for strings in the code that called lix nix.

Change-Id: I2138bb4dd239096bc530946b281db7f875195b39
2024-03-18 18:20:24 -07:00
jade 81be5eb7c6 issue importer: list issues that are *not* closed when finding existing issues
Turns out also, you cannot set the queue to 0 with any success. So we
really should just like, prevent notifications in forgejo itself.

Filed a bug for that:
lix-project/web-services#38

Change-Id: Ib96749f3159659182904963cab7b2ef88fc64442
2024-03-18 18:14:31 -07:00
jade 6b0020749d clang-tidy check infrastructure
This brings in infrastructure for developing new custom clang-tidy lints
and refactors for Lix.

Change-Id: I3df5f5855712ab4f97d4e84d771e5e818f81f881
2024-03-18 16:10:29 -07:00
1223 changed files with 41417 additions and 38736 deletions

View file

@ -48,4 +48,3 @@ PointerAlignment: Middle
SortIncludes: Never
SpaceAfterCStyleCast: true
SpaceAfterTemplateKeyword: false

35
.clang-tidy Normal file
View file

@ -0,0 +1,35 @@
UseColor: true
Checks:
- -*
- bugprone-*
# too many warnings
- -bugprone-assignment-in-if-condition
# too many warnings
- -bugprone-narrowing-conversions
# kind of nonsense
- -bugprone-easily-swappable-parameters
# too many warnings for now
- -bugprone-implicit-widening-of-multiplication-result
# Lix's exception handling is Questionable
- -bugprone-empty-catch
# many warnings
- -bugprone-unchecked-optional-access
# many warnings, seems like a questionable lint
- -bugprone-branch-clone
# all thrown exceptions must derive from std::exception
- hicpp-exception-baseclass
# capturing async lambdas are dangerous
- cppcoreguidelines-avoid-capturing-lambda-coroutines
# crimes must be appropriately declared as crimes
- cppcoreguidelines-pro-type-cstyle-cast
- lix-*
# This can not yet be applied to Lix itself since we need to do source
# reorganization so that lix/ include paths work.
- -lix-fixincludes
# This lint is included as an example, but the lib function it replaces is
# already gone.
- -lix-hasprefixsuffix
CheckOptions:
bugprone-reserved-identifier.AllowedIdentifiers: '__asan_default_options'

View file

@ -24,3 +24,12 @@ indent_size = 4
# Match diffs, avoid to trim trailing whitespace
[*.{diff,patch}]
trim_trailing_whitespace = false
[*.md]
indent_style = space
indent_size = 2
max_line_length = 0
[meson.build]
indent_style = space
indent_size = 2

5
.envrc Normal file
View file

@ -0,0 +1,5 @@
# shellcheck shell=bash
source_env_if_exists .envrc.local
# Use native-clangStdenvPackages to get clangd by default.
use flake ".#${LIX_SHELL_VARIANT:-native-clangStdenvPackages}" "${LIX_SHELL_EXTRA_ARGS[@]}"
export GTEST_BRIEF=1

18
.github/CODEOWNERS vendored
View file

@ -1,18 +0,0 @@
# Pull requests concerning the listed files will automatically invite the respective maintainers as reviewers.
# This file is not used for denoting any kind of ownership, but is merely a tool for handling notifications.
#
# Merge permissions are required for maintaining an entry in this file.
# For documentation on this mechanism, see https://help.github.com/articles/about-codeowners/
# Default reviewers if nothing else matches
* @edolstra
# This file
.github/CODEOWNERS @edolstra
# Public documentation
/doc @fricklerhandwerk
*.md @fricklerhandwerk
# Libstore layer
/src/libstore @thufschmitt

View file

@ -25,7 +25,7 @@ you probably want to file an issue at https://github.com/NixOS/nixpkgs/issues.
A clear and concise description of what you expected to happen.
## `nix-env --version` output
## `nix --version` output
## Additional context

View file

@ -9,7 +9,7 @@ assignees: ''
## Platform
<!-- select the platform on which you tried to install Nix -->
<!-- select the platform on which you tried to install Lix -->
- [ ] Linux: <!-- state your distribution, e.g. Arch Linux, Ubuntu, ... -->
- [ ] macOS

View file

@ -2,7 +2,7 @@
name: Missing or incorrect documentation
about: Help us improve the reference manual
title: ''
labels: documentation
labels: docs
assignees: ''
---
@ -19,9 +19,10 @@ assignees: ''
<!-- make sure this issue is not redundant or obsolete -->
- [ ] checked [latest Nix manual] \([source])
- [ ] checked [open documentation issues and pull requests] for possible duplicates
- [ ] checked [latest Lix manual] or its [source code]
- [ ] checked [documentation issues] and [recent documentation changes] for possible duplicates
[latest Nix manual]: https://nixos.org/manual/nix/unstable/
[source]: https://github.com/NixOS/nix/tree/master/doc/manual/src
[open documentation issues and pull requests]: https://github.com/NixOS/nix/labels/documentation
[latest Lix manual]: https://docs.lix.systems/manual/lix/nightly
[source code]: https://git.lix.systems/lix-project/lix/src/main/doc/manual/src
[documentation issues]: https://git.lix.systems/lix-project/lix/issues?labels=151&state=all
[recent documentation changes]: https://gerrit.lix.systems/q/p:lix+path:%22%5Edoc/manual/.*%22

35
.github/STALE-BOT.md vendored
View file

@ -1,35 +0,0 @@
# Stale bot information
- Thanks for your contribution!
- To remove the stale label, just leave a new comment.
- _How to find the right people to ping?_ &rarr; [`git blame`](https://git-scm.com/docs/git-blame) to the rescue! (or GitHub's history and blame buttons.)
- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on [Matrix - #nix:nixos.org](https://matrix.to/#/#nix:nixos.org).
## Suggestions for PRs
1. GitHub sometimes doesn't notify people who commented / reviewed a PR previously, when you (force) push commits. If you have addressed the reviews you can [officially ask for a review](https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from those who commented to you or anyone else.
2. If it is unfinished but you plan to finish it, please mark it as a draft.
3. If you don't expect to work on it any time soon, closing it with a short comment may encourage someone else to pick up your work.
4. To get things rolling again, rebase the PR against the target branch and address valid comments.
5. If you need a review to move forward, ask in [the Discourse thread for PRs that need help](https://discourse.nixos.org/t/prs-in-distress/3604).
6. If all you need is a merge, check the git history to find and [request reviews](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from people who usually merge related contributions.
## Suggestions for issues
1. If it is resolved (either for you personally, or in general), please consider closing it.
2. If this might still be an issue, but you are not interested in promoting its resolution, please consider closing it while encouraging others to take over and reopen an issue if they care enough.
3. If you still have interest in resolving it, try to ping somebody who you believe might have an interest in the topic. Consider discussing the problem in [our Discourse Forum](https://discourse.nixos.org/).
4. As with all open source projects, your best option is to submit a Pull Request that addresses this issue. We :heart: this attitude!
**Memorandum on closing issues**
Don't be afraid to close an issue that holds valuable information. Closed issues stay in the system for people to search, read, cross-reference, or even reopen--nothing is lost! Closing obsolete issues is an important way to help maintainers focus their time and effort.
## Useful GitHub search queries
- [Open PRs with any stale-bot interaction](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+)
- [Open PRs with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22)
- [Open PRs with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+)
- [Open Issues with any stale-bot interaction](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+)
- [Open Issues with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22+)
- [Open Issues with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+)

View file

@ -1,6 +0,0 @@
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"

23
.github/labeler.yml vendored
View file

@ -1,23 +0,0 @@
"documentation":
- doc/manual/*
- src/nix/**/*.md
"store":
- src/libstore/store-api.*
- src/libstore/*-store.*
"fetching":
- src/libfetchers/**/*
"repl":
- src/libcmd/repl.*
- src/nix/repl.*
"new-cli":
- src/nix/**/*
"with-tests":
# Unit tests
- src/*/tests/**/*
# Functional and integration tests
- tests/functional/**/*

9
.github/stale.yml vendored
View file

@ -1,9 +0,0 @@
# Configuration for probot-stale - https://github.com/probot/stale
daysUntilStale: 180
daysUntilClose: false
exemptLabels:
- "critical"
- "never-stale"
staleLabel: "stale"
markComment: false
closeComment: false

View file

@ -1,32 +0,0 @@
name: Backport
on:
pull_request_target:
types: [closed, labeled]
permissions:
contents: read
jobs:
backport:
name: Backport Pull Request
permissions:
# for zeebe-io/backport-action
contents: write
pull-requests: write
if: github.repository_owner == 'NixOS' && github.event.pull_request.merged == true && (github.event_name != 'labeled' || startsWith('backport', github.event.label.name))
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
# required to find all branches
fetch-depth: 0
- name: Create backport PRs
# should be kept in sync with `version`
uses: zeebe-io/backport-action@v1.4.0
with:
# Config README: https://github.com/zeebe-io/backport-action#backport-action
github_token: ${{ secrets.GITHUB_TOKEN }}
github_workspace: ${{ github.workspace }}
pull_description: |-
Automatic backport to `${target_branch}`, triggered by a label in #${pull_number}.
# should be kept in sync with `uses`
version: v0.0.5

View file

@ -1,135 +0,0 @@
name: "CI"
on:
pull_request:
push:
permissions: read-all
jobs:
tests:
needs: [check_secrets]
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v23
with:
# The sandbox would otherwise be disabled by default on Darwin
extra_nix_config: "sandbox = true"
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- uses: cachix/cachix-action@v12
if: needs.check_secrets.outputs.cachix == 'true'
with:
name: '${{ env.CACHIX_NAME }}'
signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
- run: nix --experimental-features 'nix-command flakes' flake check -L
check_secrets:
permissions:
contents: none
name: Check Cachix and Docker secrets present for installer tests
runs-on: ubuntu-latest
outputs:
cachix: ${{ steps.secret.outputs.cachix }}
docker: ${{ steps.secret.outputs.docker }}
steps:
- name: Check for secrets
id: secret
env:
_CACHIX_SECRETS: ${{ secrets.CACHIX_SIGNING_KEY }}${{ secrets.CACHIX_AUTH_TOKEN }}
_DOCKER_SECRETS: ${{ secrets.DOCKERHUB_USERNAME }}${{ secrets.DOCKERHUB_TOKEN }}
run: |
echo "::set-output name=cachix::${{ env._CACHIX_SECRETS != '' }}"
echo "::set-output name=docker::${{ env._DOCKER_SECRETS != '' }}"
installer:
needs: [tests, check_secrets]
if: github.event_name == 'push' && needs.check_secrets.outputs.cachix == 'true'
runs-on: ubuntu-latest
outputs:
installerURL: ${{ steps.prepare-installer.outputs.installerURL }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- uses: cachix/install-nix-action@v23
with:
install_url: https://releases.nixos.org/nix/nix-2.13.3/install
- uses: cachix/cachix-action@v12
with:
name: '${{ env.CACHIX_NAME }}'
signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
- id: prepare-installer
run: scripts/prepare-installer-for-github-actions
installer_test:
needs: [installer, check_secrets]
if: github.event_name == 'push' && needs.check_secrets.outputs.cachix == 'true'
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- uses: cachix/install-nix-action@v23
with:
install_url: '${{needs.installer.outputs.installerURL}}'
install_options: "--tarball-url-prefix https://${{ env.CACHIX_NAME }}.cachix.org/serve"
- run: sudo apt install fish zsh
if: matrix.os == 'ubuntu-latest'
- run: brew install fish
if: matrix.os == 'macos-latest'
- run: exec bash -c "nix-instantiate -E 'builtins.currentTime' --eval"
- run: exec sh -c "nix-instantiate -E 'builtins.currentTime' --eval"
- run: exec zsh -c "nix-instantiate -E 'builtins.currentTime' --eval"
- run: exec fish -c "nix-instantiate -E 'builtins.currentTime' --eval"
- run: exec bash -c "nix-channel --add https://releases.nixos.org/nixos/unstable/nixos-23.05pre466020.60c1d71f2ba nixpkgs"
- run: exec bash -c "nix-channel --update && nix-env -iA nixpkgs.hello && hello"
docker_push_image:
needs: [check_secrets, tests]
if: >-
github.event_name == 'push' &&
github.ref_name == 'master' &&
needs.check_secrets.outputs.cachix == 'true' &&
needs.check_secrets.outputs.docker == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v23
with:
install_url: https://releases.nixos.org/nix/nix-2.13.3/install
- run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
- run: echo NIX_VERSION="$(nix --experimental-features 'nix-command flakes' eval .\#default.version | tr -d \")" >> $GITHUB_ENV
- uses: cachix/cachix-action@v12
if: needs.check_secrets.outputs.cachix == 'true'
with:
name: '${{ env.CACHIX_NAME }}'
signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
- run: nix --experimental-features 'nix-command flakes' build .#dockerImage -L
- run: docker load -i ./result/image.tar.gz
- run: docker tag nix:$NIX_VERSION nixos/nix:$NIX_VERSION
- run: docker tag nix:$NIX_VERSION nixos/nix:master
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- run: docker push nixos/nix:$NIX_VERSION
- run: docker push nixos/nix:master

View file

@ -1,20 +0,0 @@
name: Hydra status
permissions: read-all
on:
schedule:
- cron: "12,42 * * * *"
workflow_dispatch:
jobs:
check_hydra_status:
name: Check Hydra status
if: github.repository_owner == 'NixOS'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- run: bash scripts/check-hydra-status.sh

View file

@ -1,24 +0,0 @@
name: "Label PR"
on:
pull_request_target:
types: [edited, opened, synchronize, reopened]
# WARNING:
# When extending this action, be aware that $GITHUB_TOKEN allows some write
# access to the GitHub API. This means that it should not evaluate user input in
# a way that allows code injection.
permissions:
contents: read
pull-requests: write
jobs:
labels:
runs-on: ubuntu-latest
if: github.repository_owner == 'NixOS'
steps:
- uses: actions/labeler@v4
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
sync-labels: false

147
.gitignore vendored
View file

@ -1,133 +1,5 @@
Makefile.config
perl/Makefile.config
# /
/aclocal.m4
/autom4te.cache
/precompiled-headers.h.gch
/config.*
/configure
/stamp-h1
/svn-revision
/libtool
# /doc/manual/
/doc/manual/*.1
/doc/manual/*.5
/doc/manual/*.8
/doc/manual/generated/*
/doc/manual/nix.json
/doc/manual/conf-file.json
/doc/manual/language.json
/doc/manual/xp-features.json
/doc/manual/src/SUMMARY.md
/doc/manual/src/SUMMARY-rl-next.md
/doc/manual/src/command-ref/new-cli
/doc/manual/src/command-ref/conf-file.md
/doc/manual/src/command-ref/experimental-features-shortlist.md
/doc/manual/src/contributing/experimental-feature-descriptions.md
/doc/manual/src/language/builtins.md
/doc/manual/src/language/builtin-constants.md
/doc/manual/src/release-notes/rl-next.md
# /scripts/
/scripts/nix-profile.sh
/scripts/nix-profile-daemon.sh
/scripts/nix-profile.fish
/scripts/nix-profile-daemon.fish
# /src/libexpr/
/src/libexpr/lexer-tab.cc
/src/libexpr/lexer-tab.hh
/src/libexpr/parser-tab.cc
/src/libexpr/parser-tab.hh
/src/libexpr/parser-tab.output
/src/libexpr/nix.tbl
/src/libexpr/tests
/tests/unit/libexpr/libnixexpr-tests
# /src/libstore/
*.gen.*
/src/libstore/tests
/tests/unit/libstore/libnixstore-tests
# /src/libutil/
/src/libutil/tests
/tests/unit/libutil/libnixutil-tests
/src/nix/nix
/src/nix/doc
# /src/nix-env/
/src/nix-env/nix-env
# /src/nix-instantiate/
/src/nix-instantiate/nix-instantiate
# /src/nix-store/
/src/nix-store/nix-store
/src/nix-prefetch-url/nix-prefetch-url
/src/nix-collect-garbage/nix-collect-garbage
# /src/nix-channel/
/src/nix-channel/nix-channel
# /src/nix-build/
/src/nix-build/nix-build
/src/nix-copy-closure/nix-copy-closure
/src/error-demo/error-demo
/src/build-remote/build-remote
# /tests/functional/
/tests/functional/test-tmp
/tests/functional/common/vars-and-functions.sh
/tests/functional/result*
/tests/functional/restricted-innocent
/tests/functional/shell
/tests/functional/shell.drv
/tests/functional/config.nix
/tests/functional/ca/config.nix
/tests/functional/dyn-drv/config.nix
/tests/functional/repl-result-out
/tests/functional/debugger-test-out
/tests/functional/test-libstoreconsumer/test-libstoreconsumer
# /tests/functional/lang/
/tests/functional/lang/*.out
/tests/functional/lang/*.out.xml
/tests/functional/lang/*.err
/tests/functional/lang/*.ast
/perl/lib/Nix/Config.pm
/perl/lib/Nix/Store.cc
/misc/systemd/nix-daemon.service
/misc/systemd/nix-daemon.socket
/misc/systemd/nix-daemon.conf
/misc/upstart/nix-daemon.conf
/src/resolve-system-dependencies/resolve-system-dependencies
outputs/
*.a
*.o
*.o.tmp
*.so
*.dylib
*.dll
*.exe
*.dep
*~
*.pc
*.plist
# GNU Global
GPATH
GRTAGS
@ -139,16 +11,31 @@ GTAGS
# auto-generated compilation database
compile_commands.json
nix-rust/target
rust-project.json
result
result-*
.vscode/
.direnv/
# clangd and possibly more
.cache/
# Mac OS
.DS_Store
# ClangBuildAnalyzer output, see maintainers/buildtime_report.sh
buildtime.bin
.envrc.local
# We generate this with a Nix shell hook
/.pre-commit-config.yaml
/.nocontribmsg
/release
# Rust build files when using Cargo (not actually supported for building but it spews the files anyway)
/target/
# Python compiled files from the test suite
*.pyc

1
.this-is-lix Normal file
View file

@ -0,0 +1 @@
This is a file used by the dev shell shellHook in package.nix to check that this is actually a Lix repo before installing git hooks. Its contents have no meaning.

View file

@ -1 +0,0 @@
2.90.0

View file

@ -1,76 +1,53 @@
# Contributing to Nix
# Contributing to Lix
Welcome and thank you for your interest in contributing to Nix!
We appreciate your support.
Welcome and thank you for considering contributing to Lix! We're currently in a soft release phase, and your support means a lot to us.
Reading and following these guidelines will help us make the contribution process easy and effective for everyone involved.
To ensure a smooth and effective contribution process, here is a summary of our guidelines:
## Getting help?
If you have any question regarding getting started or reporting bugs, feel free
to reach out to us.
On Matrix, we have a space at `#space:lix.systems`, composed of:
- [`#discuss:lix.systems`](https://matrix.to/#/#discuss:lix.systems) for discussions on Lix.
- [`#dev:lix.systems`](https://matrix.to/#/#dev:lix.systems) for the development channel on Lix.
## Report a bug
1. Check on the [GitHub issue tracker](https://github.com/NixOS/nix/issues) if your bug was already reported.
- Check if your bug has already been reported in the [issue tracker](https://git.lix.systems/lix-project/lix/issues).
- If you can't find the bug or feature, please open a new issue.
2. If you were not able to find the bug or feature [open a new issue](https://github.com/NixOS/nix/issues/new/choose)
3. The issue templates will guide you in specifying your issue.
The more complete the information you provide, the more likely it can be found by others and the more useful it is in the future.
Make sure reported bugs can be reproduced easily.
4. Once submitted, do not expect issues to be picked up or solved right away.
The only way to ensure this, is to [work on the issue yourself](#making-changes-to-nix).
We maintain a copy of the upstream Nix bugs. Their organisation can be read about [here](https://wiki.lix.systems/books/lix-contributors/page/bug-tracker-organisation).
## Report a security vulnerability
Check out the [security policy](https://github.com/NixOS/nix/security/policy).
For security vulnerabilities, reach out by email at `security at lix dot systems`.
## Making changes to Nix
## Making changes to Lix
1. Check for [pull requests](https://github.com/NixOS/nix/pulls) that might already cover the contribution you are about to make.
There are many open pull requests that might already do what you intent to work on.
You can use [labels](https://github.com/NixOS/nix/labels) to filter for relevant topics.
Before diving into making changes, we want to engage with you and your ideas.
2. Search for related issues that cover what you're going to work on. It could help to mention there that you will work on the issue.
We have a few policies in effect; please take the time to familiarize yourself:
Issues labeled [good first issue](https://github.com/NixOS/nix/labels/good-first-issue) should be relatively easy to fix and are likely to get merged quickly.
Pull requests addressing issues labeled [idea approved](https://github.com/NixOS/nix/labels/idea%20approved) are especially welcomed by maintainers and will receive prioritised review.
- [Style guide on code](https://wiki.lix.systems/books/lix-contributors/page/code)
- [Freeze policy and recommended contributions](https://wiki.lix.systems/books/lix-contributors/page/freezes-and-recommended-contributions)
3. Check the [Nix reference manual](https://nixos.org/manual/nix/unstable/contributing/hacking.html) for information on building Nix and running its tests.
To avoid duplication of effort, it may be a good idea to check out the list of
[pending pull requests](https://gerrit.lix.systems/q/status:open+-is:wip) (or "change lists", as Gerrit calls them). Once you have
an idea of what you might want to do, we recommend dropping a message on our
Matrix to ensure your contribution fits with our current schedule and plans
For contributions to the command line interface, please check the [CLI guidelines](https://nixos.org/manual/nix/unstable/contributing/cli-guideline.html).
When you're ready and your changes are ready to go:
4. Make your changes!
- Submit your code.
- Submitting a GitHub PR [on our mirror](https://github.com/lix-project/lix) is totally ok if that's easier for you and your change is relatively small (300 lines or so).
5. [Create a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) for your changes.
* Link related issues in your pull request to inform interested parties and future contributors about your change.
* Make sure to have [a clean history of commits on your branch by using rebase](https://www.digitalocean.com/community/tutorials/how-to-rebase-and-update-a-pull-request).
If your pull request closes one or multiple issues, note that in the description using `Closes: #<number>`, as it will then happen automatically when your change is merged.
* [Mark the pull request as draft](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/changing-the-stage-of-a-pull-request) if you're not done with the changes.
We may ask you to resubmit it as a Gerrit CL if it is necessary for the change you're making.
- Our primary code review system is [our Gerrit instance](https://gerrit.lix.systems), where you can open a change list (CL).
If you're new to Gerrit, check out [our wiki page about Gerrit](https://wiki.lix.systems/books/lix-contributors/page/gerrit).
- Make sure to link any related issues.
- If needed, indicate that the change is 'work in progress'.
6. Do not expect your pull request to be reviewed immediately.
Nix maintainers follow a [structured process for reviews and design decisions](https://github.com/NixOS/nix/tree/master/maintainers#project-board-protocol), which may or may not prioritise your work.
Following this checklist will make the process smoother for everyone:
- [ ] Fixes an [idea approved](https://github.com/NixOS/nix/labels/idea%20approved) issue
- [ ] Tests, as appropriate:
- Functional tests [`tests/functional/**.sh`](./tests/functional)
- Unit tests [`src/*/tests`](./src/)
- Integration tests [`tests/nixos/*`](./tests/nixos)
- [ ] User documentation in the [manual](..doc/manual/src)
- [ ] API documentation in header files
- [ ] Code and comments are self-explanatory
- [ ] Commit message explains **why** the change was made
- [ ] New feature or incompatible change: updated [release notes](./doc/manual/src/release-notes/rl-next.md)
7. If you need additional feedback or help to getting pull request into shape, ask other contributors using [@mentions](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#mentioning-people-and-teams).
## Making changes to the Nix manual
The Nix reference manual is hosted on https://nixos.org/manual/nix.
The underlying source files are located in [`doc/manual/src`](./doc/manual/src).
For small changes you can [use GitHub to edit these files](https://docs.github.com/en/repositories/working-with-files/managing-files/editing-files)
For larger changes see the [Nix reference manual](https://nixos.org/manual/nix/unstable/contributing/hacking.html).
## Getting help
Whenever you're stuck or do not know how to proceed, you can always ask for help.
The appropriate channels to do so can be found on the [NixOS Community](https://nixos.org/community/) page.
You can obtain an account on our platforms by clicking "Sign In with GitHub" on the sign-in page.

22
COPYING
View file

@ -55,7 +55,7 @@ modified by someone else and passed on, the recipients should know
that what they have is not the original version, so that the original
author's reputation will not be affected by problems that might be
introduced by others.
Finally, software patents pose a constant threat to the existence of
any free program. We wish to make sure that a company cannot
effectively restrict the users of a free program by obtaining a
@ -111,7 +111,7 @@ modification follow. Pay close attention to the difference between a
"work based on the library" and a "work that uses the library". The
former contains code derived from the library, whereas the latter must
be combined with the library in order to run.
GNU LESSER GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
@ -146,7 +146,7 @@ such a program is covered only if its contents constitute a work based
on the Library (independent of the use of the Library in a tool for
writing it). Whether that is true depends on what the Library does
and what the program that uses the Library does.
1. You may copy and distribute verbatim copies of the Library's
complete source code as you receive it, in any medium, provided that
you conspicuously and appropriately publish on each copy an
@ -158,7 +158,7 @@ Library.
You may charge a fee for the physical act of transferring a copy,
and you may at your option offer warranty protection in exchange for a
fee.
2. You may modify your copy or copies of the Library or any portion
of it, thus forming a work based on the Library, and copy and
distribute such modifications or work under the terms of Section 1
@ -216,7 +216,7 @@ instead of to this License. (If a newer version than version 2 of the
ordinary GNU General Public License has appeared, then you can specify
that version instead if you wish.) Do not make any other change in
these notices.
Once this change is made in a given copy, it is irreversible for
that copy, so the ordinary GNU General Public License applies to all
subsequent copies and derivative works made from that copy.
@ -267,7 +267,7 @@ Library will still fall under Section 6.)
distribute the object code for the work under the terms of Section 6.
Any executables containing that work also fall under Section 6,
whether or not they are linked directly with the Library itself.
6. As an exception to the Sections above, you may also combine or
link a "work that uses the Library" with the Library to produce a
work containing portions of the Library, and distribute that work
@ -329,7 +329,7 @@ restrictions of other proprietary libraries that do not normally
accompany the operating system. Such a contradiction means you cannot
use both them and the Library together in an executable that you
distribute.
7. You may place library facilities that are a work based on the
Library side-by-side in a single library together with other library
facilities not covered by this License, and distribute such a combined
@ -370,7 +370,7 @@ subject to these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties with
this License.
11. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
@ -422,7 +422,7 @@ conditions either of that version or of any later version published by
the Free Software Foundation. If the Library does not specify a
license version number, you may choose any version ever published by
the Free Software Foundation.
14. If you wish to incorporate parts of the Library into other free
programs whose distribution conditions are incompatible with these,
write to the author to ask for permission. For software which is
@ -456,7 +456,7 @@ SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Libraries
If you develop a new library, and you want it to be of the greatest
@ -500,5 +500,3 @@ necessary. Here is a sample; alter the names:
Ty Coon, President of Vice
That's all there is to it!

79
Cargo.lock generated Normal file
View file

@ -0,0 +1,79 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3
[[package]]
name = "countme"
version = "3.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7704b5fdd17b18ae31c4c1da5a2e0305a2bf17b5249300a9ee9ed7b72114c636"
[[package]]
name = "dissimilar"
version = "1.0.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59f8e79d1fbf76bdfbde321e902714bf6c49df88a7dda6fc682fc2979226962d"
[[package]]
name = "expect-test"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9e0be0a561335815e06dab7c62e50353134c796e7a6155402a64bcff66b6a5e0"
dependencies = [
"dissimilar",
"once_cell",
]
[[package]]
name = "hashbrown"
version = "0.14.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
[[package]]
name = "lix-doc"
version = "0.0.1"
dependencies = [
"expect-test",
"rnix",
"rowan",
]
[[package]]
name = "once_cell"
version = "1.19.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3fdb12b2476b595f9358c5161aa467c2438859caa136dec86c26fdd2efe17b92"
[[package]]
name = "rnix"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bb35cedbeb70e0ccabef2a31bcff0aebd114f19566086300b8f42c725fc2cb5f"
dependencies = [
"rowan",
]
[[package]]
name = "rowan"
version = "0.15.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0a542b0253fa46e632d27a1dc5cf7b930de4df8659dc6e720b647fc72147ae3d"
dependencies = [
"countme",
"hashbrown",
"rustc-hash",
"text-size",
]
[[package]]
name = "rustc-hash"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2"
[[package]]
name = "text-size"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f18aa187839b2bdb1ad2fa35ead8c4c2976b64e4363c386d45ac0f7ee85c9233"

6
Cargo.toml Normal file
View file

@ -0,0 +1,6 @@
[workspace]
resolver = "2"
members = ["src/lix-doc"]
[workspace.package]
edition = "2021"

View file

@ -1,72 +0,0 @@
include mk/build-dir.mk
-include $(buildprefix)Makefile.config
clean-files += $(buildprefix)Makefile.config
ifeq ($(ENABLE_BUILD), yes)
makefiles = \
mk/precompiled-headers.mk \
local.mk \
src/libutil/local.mk \
src/libstore/local.mk \
src/libfetchers/local.mk \
src/libmain/local.mk \
src/libexpr/local.mk \
src/libcmd/local.mk \
src/nix/local.mk \
src/resolve-system-dependencies/local.mk \
scripts/local.mk \
misc/bash/local.mk \
misc/fish/local.mk \
misc/zsh/local.mk \
misc/systemd/local.mk \
misc/launchd/local.mk \
misc/upstart/local.mk
endif
ifeq ($(ENABLE_BUILD)_$(ENABLE_TESTS), yes_yes)
UNIT_TEST_ENV = _NIX_TEST_UNIT_DATA=unit-test-data
makefiles += \
tests/unit/libutil/local.mk \
tests/unit/libutil-support/local.mk \
tests/unit/libstore/local.mk
endif
ifeq ($(ENABLE_TESTS), yes)
makefiles += \
tests/unit/libstore-support/local.mk \
tests/unit/libexpr/local.mk \
tests/unit/libexpr-support/local.mk \
tests/functional/local.mk \
tests/functional/ca/local.mk \
tests/functional/dyn-drv/local.mk \
tests/functional/test-libstoreconsumer/local.mk \
tests/functional/repl_characterization/local.mk \
tests/functional/plugins/local.mk
else
makefiles += \
mk/disable-tests.mk
endif
# Some makefiles require access to built programs and must be included late.
makefiles-late =
ifeq ($(ENABLE_BUILD), yes)
makefiles-late += doc/manual/local.mk
makefiles-late += doc/internal-api/local.mk
endif
# Miscellaneous global Flags
OPTIMIZE = 1
ifeq ($(OPTIMIZE), 1)
GLOBAL_CXXFLAGS += -O2 $(CXXLTO)
GLOBAL_LDFLAGS += $(CXXLTO)
else
GLOBAL_CXXFLAGS += -O0 -U_FORTIFY_SOURCE
endif
include mk/lib.mk
GLOBAL_CXXFLAGS += -g -Wall -Wimplicit-fallthrough -include $(buildprefix)config.h -std=c++2a -I src

View file

@ -1,51 +0,0 @@
AR = @AR@
BDW_GC_LIBS = @BDW_GC_LIBS@
BOOST_LDFLAGS = @BOOST_LDFLAGS@
BUILD_SHARED_LIBS = @BUILD_SHARED_LIBS@
CC = @CC@
CFLAGS = @CFLAGS@
CXX = @CXX@
CXXFLAGS = @CXXFLAGS@
CXXLTO = @CXXLTO@
EDITLINE_LIBS = @EDITLINE_LIBS@
ENABLE_S3 = @ENABLE_S3@
GTEST_LIBS = @GTEST_LIBS@
HAVE_LIBCPUID = @HAVE_LIBCPUID@
HAVE_SECCOMP = @HAVE_SECCOMP@
HOST_OS = @host_os@
LDFLAGS = @LDFLAGS@
LIBARCHIVE_LIBS = @LIBARCHIVE_LIBS@
LIBBROTLI_LIBS = @LIBBROTLI_LIBS@
LIBCURL_LIBS = @LIBCURL_LIBS@
LIBSECCOMP_LIBS = @LIBSECCOMP_LIBS@
LOWDOWN_LIBS = @LOWDOWN_LIBS@
OPENSSL_LIBS = @OPENSSL_LIBS@
PACKAGE_NAME = @PACKAGE_NAME@
PACKAGE_VERSION = @PACKAGE_VERSION@
RAPIDCHECK_HEADERS = @RAPIDCHECK_HEADERS@
SHELL = @bash@
SODIUM_LIBS = @SODIUM_LIBS@
SQLITE3_LIBS = @SQLITE3_LIBS@
bash = @bash@
bindir = @bindir@
datadir = @datadir@
datarootdir = @datarootdir@
doc_generate = @doc_generate@
docdir = @docdir@
embedded_sandbox_shell = @embedded_sandbox_shell@
exec_prefix = @exec_prefix@
includedir = @includedir@
libdir = @libdir@
libexecdir = @libexecdir@
localstatedir = @localstatedir@
lsof = @lsof@
mandir = @mandir@
pkglibdir = $(libdir)/$(PACKAGE_NAME)
prefix = @prefix@
sandbox_shell = @sandbox_shell@
storedir = @storedir@
sysconfdir = @sysconfdir@
system = @system@
ENABLE_BUILD = @ENABLE_BUILD@
ENABLE_TESTS = @ENABLE_TESTS@
internal_api_docs = @internal_api_docs@

View file

@ -1,36 +1,29 @@
# Nix
# Lix
[![Open Collective supporters](https://opencollective.com/nixos/tiers/supporter/badge.svg?label=Supporters&color=brightgreen)](https://opencollective.com/nixos)
[![Test](https://github.com/NixOS/nix/workflows/Test/badge.svg)](https://github.com/NixOS/nix/actions)
**Lix** is an implementation of **Nix**, a powerful package management system for Linux and other Unix systems that makes package management reliable and reproducible.
Nix is a powerful package manager for Linux and other Unix systems that makes package
management reliable and reproducible. Please refer to the [Nix manual](https://nixos.org/nix/manual)
for more details.
Read more about us at https://lix.systems.
## Installation
On Linux and macOS the easiest way to install Nix is to run the following shell command
On Linux and macOS the easiest way to install Lix is to run the following shell command
(as a user other than root):
```console
$ curl -L https://nixos.org/nix/install | sh
$ curl -sSf -L https://install.lix.systems/lix | sh -s -- install
```
Information on additional installation methods is available on the [Nix download page](https://nixos.org/download.html).
For systems that **already have a Nix implementation installed**, such as NixOS systems, read our [install page](https://lix.systems/install)
## Building And Developing
See our [Hacking guide](https://nixos.org/manual/nix/unstable/contributing/hacking.html) in our manual for instruction on how to
to set up a development environment and build Nix from source.
See our [Hacking guide](https://git.lix.systems/lix-project/lix/src/branch/main/doc/manual/src/contributing/hacking.md) in our manual for instruction on how to to set up a development environment and build Lix from source.
## Additional Resources
- [Nix manual](https://nixos.org/nix/manual)
- [Nix jobsets on hydra.nixos.org](https://hydra.nixos.org/project/nix)
- [NixOS Discourse](https://discourse.nixos.org/)
- [Matrix - #nix:nixos.org](https://matrix.to/#/#nix:nixos.org)
- [IRC - #nixos on libera.chat](irc://irc.libera.chat/#nixos)
- [Our wiki](https://wiki.lix.systems)
- [Matrix - #space:lix.systems](https://matrix.to/#/#space:lix.systems)
## License
Nix is released under the [LGPL v2.1](./COPYING).
Lix is released under [LGPL-2.1-or-later](./COPYING).

3
bench/.gitignore vendored Normal file
View file

@ -0,0 +1,3 @@
bench-*.json
bench-*.md
nixpkgs

91
bench/README.md Normal file
View file

@ -0,0 +1,91 @@
# Benchmarking scripts for Lix
These are very much WIP, and have a few clumsy assumptions that we would
somewhat rather be fixed, but we have committed them to let others be able to
do benchmarking in the mean time.
## Benchmarking procedure
Build some Lixes you want to compare, by whichever means you wish.
Get a computer that is not busy and *strongly preferably* is bare-metal or at
least not a cloud VM (e.g. go make coffee when running benchmarks).
From the root of a Lix checkout, run `./bench/bench.sh resultlink-one
resultlink-two`, where `resultlink-one` and `resultlink-two` are the result
links from the builds you want to test (they can be any directory with bin/nix
in it, however).
To get the summary again, run `./bench/summarize.jq bench/bench-*.json`.
## Example results
(vim tip: `:r !bench/summarize.jq bench/bench-*.json` to dump it directly into
your editor)
```
result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6
c6fe19ccb59cac54b5b3f25e160870 hello
mean: 15.993s ± 0.081s
user: 13.321s | system: 1.865s
median: 15.994s
range: 15.829s ... 16.096s
relative: 1
result/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19cc
b59cac54b5b3f25e160870 hello
mean: 15.897s ± 0.075s
user: 13.248s | system: 1.843s
median: 15.88s
range: 15.807s ... 16.047s
relative: 0.994
---
result/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
mean: 0.4s ± 0.024s
user: 0.335s | system: 0.046s
median: 0.386s
range: 0.379s ... 0.43s
relative: 1
result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
mean: 0.404s ± 0.024s
user: 0.338s | system: 0.046s
median: 0.386s
range: 0.384s ... 0.436s
relative: 1.008
---
result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 5.838s ± 0.023s
user: 5.083s | system: 0.464s
median: 5.845s
range: 5.799s ... 5.867s
relative: 1
result/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 5.788s ± 0.044s
user: 5.056s | system: 0.439s
median: 5.79s
range: 5.715s ... 5.876s
relative: 0.991
---
GC_INITIAL_HEAP_SIZE=10g result-asserts/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 4.147s ± 0.021s
user: 3.457s | system: 0.487s
median: 4.147s
range: 4.123s ... 4.195s
relative: 1
GC_INITIAL_HEAP_SIZE=10g result/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 4.149s ± 0.027s
user: 3.483s | system: 0.456s
median: 4.142s
range: 4.126s ... 4.215s
relative: 1
---
```

68
bench/bench.sh Executable file
View file

@ -0,0 +1,68 @@
#!/usr/bin/env nix-shell
#!nix-shell -i bash -p bash -p hyperfine
set -euo pipefail
shopt -s inherit_errexit
scriptdir=$(cd "$(dirname -- "$0")" ; pwd -P)
cd "$scriptdir/.."
if [[ $# -lt 2 ]]; then
# FIXME(jade): it is a reasonable use case to want to run a benchmark run
# on just one build. However, since we are using hyperfine in comparison
# mode, we would have to combine the JSON ourselves to support that, which
# would probably be better done by writing a benchmarking script in
# not-bash.
echo "Fewer than two result dirs given, nothing to compare!" >&2
echo "Pass some directories (with names indicating which alternative they are) with bin/nix in them" >&2
echo "Usage: ./bench/bench.sh result-1 result-2 [result-3...]" >&2
exit 1
fi
_exit=""
trap "$_exit" EXIT
flake_args=("--extra-experimental-features" "nix-command flakes")
# XXX: yes this is very silly. flakes~!!
nix build "${flake_args[@]}" --impure --expr '(builtins.getFlake "git+file:.").inputs.nixpkgs.outPath' -o bench/nixpkgs
# We must ignore the global config, or else NIX_PATH won't work reliably.
# See https://github.com/NixOS/nix/issues/9574
export NIX_CONF_DIR='/var/empty'
export NIX_REMOTE="$(mktemp -d)"
_exit='rm -rfv "$NIX_REMOTE"; $_exit'
export NIX_PATH="nixpkgs=bench/nixpkgs:nixos-config=bench/configuration.nix"
builds=("$@")
flake_args="${flake_args[*]@Q}"
hyperfineArgs=(
--parameter-list BUILD "$(IFS=,; echo "${builds[*]}")"
--warmup 2 --runs 10
)
declare -A cases
cases=(
[search]="{BUILD}/bin/nix $flake_args search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello"
[rebuild]="{BUILD}/bin/nix $flake_args eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'"
[rebuild-lh]="GC_INITIAL_HEAP_SIZE=10g {BUILD}/bin/nix eval $flake_args --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'"
[parse]="{BUILD}/bin/nix $flake_args eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix"
)
benches=(
rebuild
rebuild-lh
search
parse
)
for k in "${benches[@]}"; do
taskset -c 2,3 \
chrt -f 50 \
hyperfine "${hyperfineArgs[@]}" --export-json="bench/bench-${k}.json" --export-markdown="bench/bench-${k}.md" "${cases[$k]}"
done
echo "Benchmarks summary (from ./bench/summarize.jq bench/bench-*.json)"
bench/summarize.jq bench/*.json

325
bench/configuration.nix Normal file
View file

@ -0,0 +1,325 @@
{
config,
pkgs,
lib,
...
}:
{
boot = {
initrd = {
availableKernelModules = [
"xhci_pci"
"ahci"
];
kernelModules = [ "dm-snapshot" ];
luks.devices = {
croot = {
device = "/dev/sdb";
allowDiscards = true;
};
};
};
kernelModules = [ "kvm-intel" ];
kernelPackages = pkgs.linuxPackages_latest;
loader = {
systemd-boot.enable = true;
efi.canTouchEfiVariables = true;
};
};
hardware = {
enableRedistributableFirmware = true;
cpu.intel.updateMicrocode = true;
opengl.driSupport32Bit = true;
opengl.extraPackages = with pkgs; [
vaapiIntel
intel-media-driver
intel-compute-runtime
];
};
fileSystems = {
"/" = {
device = "/dev/sda2";
fsType = "xfs";
options = [ "noatime" ];
};
"/boot" = {
device = "/dev/sda1";
fsType = "vfat";
};
"/nas" = {
device = "nas:/";
fsType = "nfs4";
options = [
"ro"
"x-systemd.automount"
];
};
};
swapDevices = [ { device = "/dev/swap"; } ];
networking = {
useDHCP = false;
hostName = "host";
wireless = {
enable = true;
interfaces = [ "eth1" ];
};
interfaces = {
eth0.useDHCP = true;
eth1.useDHCP = true;
};
wg-quick.interfaces = {
wg0 = {
address = [ "2001:db8::1" ];
privateKeyFile = "/etc/secrets/wg0.key";
peers = [
{
publicKey = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
endpoint = "[2001:db8::2]:61021";
allowedIPs = [ "2001::db8:1::/64" ];
}
];
};
};
firewall.allowedUDPPorts = [ 4567 ];
};
i18n = {
defaultLocale = "en_US.UTF-8";
inputMethod.enabled = "ibus";
};
services = {
xserver = {
enable = true;
layout = "us";
xkbVariant = "altgr-intl";
xkbOptions = "ctrl:nocaps";
libinput.enable = true;
wacom.enable = true;
videoDrivers = [ "modesetting" ];
modules = [ pkgs.xf86_input_wacom ];
displayManager.sx.enable = true;
windowManager.i3.enable = true;
};
udev.extraHwdb = ''
# not like this mattered at all
# we're not running udev from here
'';
udev.extraRules = ''
# ACTION=="add", SUBSYSTEM=="input", ...
'';
};
sound.enable = true;
hardware.pulseaudio = {
enable = true;
package = pkgs.pulseaudioFull;
daemon.config = {
lock-memory = "yes";
realtime-scheduling = "yes";
rlimit-rtprio = "-1";
};
};
programs = {
light.enable = true;
wireshark = {
enable = true;
package = pkgs.wireshark-qt;
};
gnupg.agent = {
enable = true;
};
};
fonts.packages = with pkgs; [
font-awesome
noto-fonts
noto-fonts-cjk
noto-fonts-emoji
noto-fonts-extra
dejavu_fonts
powerline-fonts
source-code-pro
cantarell-fonts
];
users = {
mutableUsers = false;
users = {
user = {
isNormalUser = true;
group = "user";
extraGroups = [
"wheel"
"video"
"audio"
"dialout"
"users"
"kvm"
"wireshark"
];
password = "unimportant";
};
};
groups = {
user = { };
};
};
security = {
pam.loginLimits = [
{
domain = "@audio";
item = "memlock";
type = "-";
value = "unlimited";
}
{
domain = "@audio";
item = "rtprio";
type = "-";
value = "99";
}
{
domain = "@audio";
item = "nofile";
type = "soft";
value = "99999";
}
{
domain = "@audio";
item = "nofile";
type = "hard";
value = "99999";
}
];
sudo.extraRules = [
{
users = [ "user" ];
commands = [
{
command = "${pkgs.linuxPackages.cpupower}/bin/cpupower";
options = [ "NOPASSWD" ];
}
];
}
];
};
environment.systemPackages = with pkgs; [
a2jmidid
age
ardour
bemenu
blender
breeze-icons
breeze-qt5
bubblewrap
calf
claws-mail
darktable
duperemove
emacs
feh
file
firefox
fluidsynth
gnome3.adwaita-icon-theme
gnuplot
graphviz
helm
i3status-rust
inkscape
jack2
jq
krita
ldns
libqalculate
libreoffice
man-pages
nheko
nix-diff
nix-index
nix-output-monitor
open-music-kontrollers.patchmatrix
pamixer
pavucontrol
pciutils
picom
pwgen
redshift
ripgrep
rlwrap
silver-searcher
soundfont-fluid
whois
wol
xclip
xdot
xdotool
xorg.xkbcomp
yt-dlp
zathura
borgbackup
linuxPackages.cpupower
mtr
kitty
xf86_input_wacom
];
environment.pathsToLink = [ "/share/soundfonts" ];
systemd.user.services.run-python = {
after = [ "network-online.target" ];
script = ''
exec ${pkgs.python3}/bin/python
'';
serviceConfig = {
CapabilityBoundingSet = [ "" ];
KeyringMode = "private";
LockPersonality = true;
MemoryDenyWriteExecute = true;
NoNewPrivileges = true;
PrivateDevices = true;
PrivateTmp = true;
PrivateUsers = true;
ProcSubset = "pid";
ProtectClock = true;
ProtectControlGroups = true;
ProtectHome = true;
ProtectHostname = true;
ProtectKernelLogs = true;
ProtectKernelModules = true;
ProtectKernelTunables = true;
ProtectProc = "invisible";
ProtectSystem = "strict";
RestrictAddressFamilies = "AF_INET AF_INET6";
RestrictNamespaces = true;
RestrictRealtime = true;
RestrictSUIDSGID = true;
SystemCallArchitectures = "native";
SystemCallFilter = [
"@system-service"
"~ @resources @privileged"
];
UMask = "077";
};
};
system.stateVersion = "23.11";
}

22
bench/summarize.jq Executable file
View file

@ -0,0 +1,22 @@
#!/usr/bin/env -S jq -Mrf
def round3:
. * 1000 | round | . / 1000
;
def stats($first):
[
" mean: \(.mean | round3)s ± \(.stddev | round3)s",
" user: \(.user | round3)s | system: \(.system | round3)s",
" median: \(.median | round3)s",
" range: \(.min | round3)s ... \(.max | round3)s",
" relative: \(.mean / $first.mean | round3)"
]
| join("\n")
;
def fmt($first):
"\(.command)\n" + (. | stats($first))
;
[.results | .[0] as $first | .[] | fmt($first)] | join("\n\n") | (. + "\n\n---\n")

View file

@ -1,99 +0,0 @@
diff --git a/darwin_stop_world.c b/darwin_stop_world.c
index 0468aaec..b348d869 100644
--- a/darwin_stop_world.c
+++ b/darwin_stop_world.c
@@ -356,6 +356,7 @@ GC_INNER void GC_push_all_stacks(void)
int nthreads = 0;
word total_size = 0;
mach_msg_type_number_t listcount = (mach_msg_type_number_t)THREAD_TABLE_SZ;
+ size_t stack_limit;
if (!EXPECT(GC_thr_initialized, TRUE))
GC_thr_init();
@@ -411,6 +412,19 @@ GC_INNER void GC_push_all_stacks(void)
GC_push_all_stack_sections(lo, hi, p->traced_stack_sect);
}
if (altstack_lo) {
+ // When a thread goes into a coroutine, we lose its original sp until
+ // control flow returns to the thread.
+ // While in the coroutine, the sp points outside the thread stack,
+ // so we can detect this and push the entire thread stack instead,
+ // as an approximation.
+ // We assume that the coroutine has similarly added its entire stack.
+ // This could be made accurate by cooperating with the application
+ // via new functions and/or callbacks.
+ stack_limit = pthread_get_stacksize_np(p->id);
+ if (altstack_lo >= altstack_hi || altstack_lo < altstack_hi - stack_limit) { // sp outside stack
+ altstack_lo = altstack_hi - stack_limit;
+ }
+
total_size += altstack_hi - altstack_lo;
GC_push_all_stack(altstack_lo, altstack_hi);
}
diff --git a/include/gc.h b/include/gc.h
index edab6c22..f2c61282 100644
--- a/include/gc.h
+++ b/include/gc.h
@@ -2172,6 +2172,11 @@ GC_API void GC_CALL GC_win32_free_heap(void);
(*GC_amiga_allocwrapper_do)(a,GC_malloc_atomic_ignore_off_page)
#endif /* _AMIGA && !GC_AMIGA_MAKINGLIB */
+#if !__APPLE__
+/* Patch doesn't work on apple */
+#define NIX_BOEHM_PATCH_VERSION 1
+#endif
+
#ifdef __cplusplus
} /* extern "C" */
#endif
diff --git a/pthread_stop_world.c b/pthread_stop_world.c
index b5d71e62..aed7b0bf 100644
--- a/pthread_stop_world.c
+++ b/pthread_stop_world.c
@@ -768,6 +768,8 @@ STATIC void GC_restart_handler(int sig)
/* world is stopped. Should not fail if it isn't. */
GC_INNER void GC_push_all_stacks(void)
{
+ size_t stack_limit;
+ pthread_attr_t pattr;
GC_bool found_me = FALSE;
size_t nthreads = 0;
int i;
@@ -851,6 +853,37 @@ GC_INNER void GC_push_all_stacks(void)
hi = p->altstack + p->altstack_size;
/* FIXME: Need to scan the normal stack too, but how ? */
/* FIXME: Assume stack grows down */
+ } else {
+#ifdef HAVE_PTHREAD_ATTR_GET_NP
+ if (!pthread_attr_init(&pattr)
+ || !pthread_attr_get_np(p->id, &pattr))
+#else /* HAVE_PTHREAD_GETATTR_NP */
+ if (pthread_getattr_np(p->id, &pattr))
+#endif
+ {
+ ABORT("GC_push_all_stacks: pthread_getattr_np failed!");
+ }
+ if (pthread_attr_getstacksize(&pattr, &stack_limit)) {
+ ABORT("GC_push_all_stacks: pthread_attr_getstacksize failed!");
+ }
+ if (pthread_attr_destroy(&pattr)) {
+ ABORT("GC_push_all_stacks: pthread_attr_destroy failed!");
+ }
+ // When a thread goes into a coroutine, we lose its original sp until
+ // control flow returns to the thread.
+ // While in the coroutine, the sp points outside the thread stack,
+ // so we can detect this and push the entire thread stack instead,
+ // as an approximation.
+ // We assume that the coroutine has similarly added its entire stack.
+ // This could be made accurate by cooperating with the application
+ // via new functions and/or callbacks.
+ #ifndef STACK_GROWS_UP
+ if (lo >= hi || lo < hi - stack_limit) { // sp outside stack
+ lo = hi - stack_limit;
+ }
+ #else
+ #error "STACK_GROWS_UP not supported in boost_coroutine2 (as of june 2021), so we don't support it in Nix."
+ #endif
}
GC_push_all_stack_sections(lo, hi, traced_stack_sect);
# ifdef STACK_GROWS_UP

View file

@ -1,12 +0,0 @@
diff --git a/include/gc_allocator.h b/include/gc_allocator.h
index 597c7f13..587286be 100644
--- a/include/gc_allocator.h
+++ b/include/gc_allocator.h
@@ -312,6 +312,7 @@ public:
template<>
class traceable_allocator<void> {
+public:
typedef size_t size_type;
typedef ptrdiff_t difference_type;
typedef void* pointer;

1700
config/config.guess vendored

File diff suppressed because it is too large Load diff

1860
config/config.sub vendored

File diff suppressed because it is too large Load diff

View file

@ -1,527 +0,0 @@
#!/bin/sh
# install - install a program, script, or datafile
scriptversion=2011-11-20.07; # UTC
# This originates from X11R5 (mit/util/scripts/install.sh), which was
# later released in X11R6 (xc/config/util/install.sh) with the
# following copyright and license.
#
# Copyright (C) 1994 X Consortium
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to
# deal in the Software without restriction, including without limitation the
# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
# sell copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# X CONSORTIUM BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
# AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNEC-
# TION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#
# Except as contained in this notice, the name of the X Consortium shall not
# be used in advertising or otherwise to promote the sale, use or other deal-
# ings in this Software without prior written authorization from the X Consor-
# tium.
#
#
# FSF changes to this file are in the public domain.
#
# Calling this script install-sh is preferred over install.sh, to prevent
# 'make' implicit rules from creating a file called install from it
# when there is no Makefile.
#
# This script is compatible with the BSD install script, but was written
# from scratch.
nl='
'
IFS=" "" $nl"
# set DOITPROG to echo to test this script
# Don't use :- since 4.3BSD and earlier shells don't like it.
doit=${DOITPROG-}
if test -z "$doit"; then
doit_exec=exec
else
doit_exec=$doit
fi
# Put in absolute file names if you don't have them in your path;
# or use environment vars.
chgrpprog=${CHGRPPROG-chgrp}
chmodprog=${CHMODPROG-chmod}
chownprog=${CHOWNPROG-chown}
cmpprog=${CMPPROG-cmp}
cpprog=${CPPROG-cp}
mkdirprog=${MKDIRPROG-mkdir}
mvprog=${MVPROG-mv}
rmprog=${RMPROG-rm}
stripprog=${STRIPPROG-strip}
posix_glob='?'
initialize_posix_glob='
test "$posix_glob" != "?" || {
if (set -f) 2>/dev/null; then
posix_glob=
else
posix_glob=:
fi
}
'
posix_mkdir=
# Desired mode of installed file.
mode=0755
chgrpcmd=
chmodcmd=$chmodprog
chowncmd=
mvcmd=$mvprog
rmcmd="$rmprog -f"
stripcmd=
src=
dst=
dir_arg=
dst_arg=
copy_on_change=false
no_target_directory=
usage="\
Usage: $0 [OPTION]... [-T] SRCFILE DSTFILE
or: $0 [OPTION]... SRCFILES... DIRECTORY
or: $0 [OPTION]... -t DIRECTORY SRCFILES...
or: $0 [OPTION]... -d DIRECTORIES...
In the 1st form, copy SRCFILE to DSTFILE.
In the 2nd and 3rd, copy all SRCFILES to DIRECTORY.
In the 4th, create DIRECTORIES.
Options:
--help display this help and exit.
--version display version info and exit.
-c (ignored)
-C install only if different (preserve the last data modification time)
-d create directories instead of installing files.
-g GROUP $chgrpprog installed files to GROUP.
-m MODE $chmodprog installed files to MODE.
-o USER $chownprog installed files to USER.
-s $stripprog installed files.
-t DIRECTORY install into DIRECTORY.
-T report an error if DSTFILE is a directory.
Environment variables override the default commands:
CHGRPPROG CHMODPROG CHOWNPROG CMPPROG CPPROG MKDIRPROG MVPROG
RMPROG STRIPPROG
"
while test $# -ne 0; do
case $1 in
-c) ;;
-C) copy_on_change=true;;
-d) dir_arg=true;;
-g) chgrpcmd="$chgrpprog $2"
shift;;
--help) echo "$usage"; exit $?;;
-m) mode=$2
case $mode in
*' '* | *' '* | *'
'* | *'*'* | *'?'* | *'['*)
echo "$0: invalid mode: $mode" >&2
exit 1;;
esac
shift;;
-o) chowncmd="$chownprog $2"
shift;;
-s) stripcmd=$stripprog;;
-t) dst_arg=$2
# Protect names problematic for 'test' and other utilities.
case $dst_arg in
-* | [=\(\)!]) dst_arg=./$dst_arg;;
esac
shift;;
-T) no_target_directory=true;;
--version) echo "$0 $scriptversion"; exit $?;;
--) shift
break;;
-*) echo "$0: invalid option: $1" >&2
exit 1;;
*) break;;
esac
shift
done
if test $# -ne 0 && test -z "$dir_arg$dst_arg"; then
# When -d is used, all remaining arguments are directories to create.
# When -t is used, the destination is already specified.
# Otherwise, the last argument is the destination. Remove it from $@.
for arg
do
if test -n "$dst_arg"; then
# $@ is not empty: it contains at least $arg.
set fnord "$@" "$dst_arg"
shift # fnord
fi
shift # arg
dst_arg=$arg
# Protect names problematic for 'test' and other utilities.
case $dst_arg in
-* | [=\(\)!]) dst_arg=./$dst_arg;;
esac
done
fi
if test $# -eq 0; then
if test -z "$dir_arg"; then
echo "$0: no input file specified." >&2
exit 1
fi
# It's OK to call 'install-sh -d' without argument.
# This can happen when creating conditional directories.
exit 0
fi
if test -z "$dir_arg"; then
do_exit='(exit $ret); exit $ret'
trap "ret=129; $do_exit" 1
trap "ret=130; $do_exit" 2
trap "ret=141; $do_exit" 13
trap "ret=143; $do_exit" 15
# Set umask so as not to create temps with too-generous modes.
# However, 'strip' requires both read and write access to temps.
case $mode in
# Optimize common cases.
*644) cp_umask=133;;
*755) cp_umask=22;;
*[0-7])
if test -z "$stripcmd"; then
u_plus_rw=
else
u_plus_rw='% 200'
fi
cp_umask=`expr '(' 777 - $mode % 1000 ')' $u_plus_rw`;;
*)
if test -z "$stripcmd"; then
u_plus_rw=
else
u_plus_rw=,u+rw
fi
cp_umask=$mode$u_plus_rw;;
esac
fi
for src
do
# Protect names problematic for 'test' and other utilities.
case $src in
-* | [=\(\)!]) src=./$src;;
esac
if test -n "$dir_arg"; then
dst=$src
dstdir=$dst
test -d "$dstdir"
dstdir_status=$?
else
# Waiting for this to be detected by the "$cpprog $src $dsttmp" command
# might cause directories to be created, which would be especially bad
# if $src (and thus $dsttmp) contains '*'.
if test ! -f "$src" && test ! -d "$src"; then
echo "$0: $src does not exist." >&2
exit 1
fi
if test -z "$dst_arg"; then
echo "$0: no destination specified." >&2
exit 1
fi
dst=$dst_arg
# If destination is a directory, append the input filename; won't work
# if double slashes aren't ignored.
if test -d "$dst"; then
if test -n "$no_target_directory"; then
echo "$0: $dst_arg: Is a directory" >&2
exit 1
fi
dstdir=$dst
dst=$dstdir/`basename "$src"`
dstdir_status=0
else
# Prefer dirname, but fall back on a substitute if dirname fails.
dstdir=`
(dirname "$dst") 2>/dev/null ||
expr X"$dst" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \
X"$dst" : 'X\(//\)[^/]' \| \
X"$dst" : 'X\(//\)$' \| \
X"$dst" : 'X\(/\)' \| . 2>/dev/null ||
echo X"$dst" |
sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{
s//\1/
q
}
/^X\(\/\/\)[^/].*/{
s//\1/
q
}
/^X\(\/\/\)$/{
s//\1/
q
}
/^X\(\/\).*/{
s//\1/
q
}
s/.*/./; q'
`
test -d "$dstdir"
dstdir_status=$?
fi
fi
obsolete_mkdir_used=false
if test $dstdir_status != 0; then
case $posix_mkdir in
'')
# Create intermediate dirs using mode 755 as modified by the umask.
# This is like FreeBSD 'install' as of 1997-10-28.
umask=`umask`
case $stripcmd.$umask in
# Optimize common cases.
*[2367][2367]) mkdir_umask=$umask;;
.*0[02][02] | .[02][02] | .[02]) mkdir_umask=22;;
*[0-7])
mkdir_umask=`expr $umask + 22 \
- $umask % 100 % 40 + $umask % 20 \
- $umask % 10 % 4 + $umask % 2
`;;
*) mkdir_umask=$umask,go-w;;
esac
# With -d, create the new directory with the user-specified mode.
# Otherwise, rely on $mkdir_umask.
if test -n "$dir_arg"; then
mkdir_mode=-m$mode
else
mkdir_mode=
fi
posix_mkdir=false
case $umask in
*[123567][0-7][0-7])
# POSIX mkdir -p sets u+wx bits regardless of umask, which
# is incompatible with FreeBSD 'install' when (umask & 300) != 0.
;;
*)
tmpdir=${TMPDIR-/tmp}/ins$RANDOM-$$
trap 'ret=$?; rmdir "$tmpdir/d" "$tmpdir" 2>/dev/null; exit $ret' 0
if (umask $mkdir_umask &&
exec $mkdirprog $mkdir_mode -p -- "$tmpdir/d") >/dev/null 2>&1
then
if test -z "$dir_arg" || {
# Check for POSIX incompatibilities with -m.
# HP-UX 11.23 and IRIX 6.5 mkdir -m -p sets group- or
# other-writable bit of parent directory when it shouldn't.
# FreeBSD 6.1 mkdir -m -p sets mode of existing directory.
ls_ld_tmpdir=`ls -ld "$tmpdir"`
case $ls_ld_tmpdir in
d????-?r-*) different_mode=700;;
d????-?--*) different_mode=755;;
*) false;;
esac &&
$mkdirprog -m$different_mode -p -- "$tmpdir" && {
ls_ld_tmpdir_1=`ls -ld "$tmpdir"`
test "$ls_ld_tmpdir" = "$ls_ld_tmpdir_1"
}
}
then posix_mkdir=:
fi
rmdir "$tmpdir/d" "$tmpdir"
else
# Remove any dirs left behind by ancient mkdir implementations.
rmdir ./$mkdir_mode ./-p ./-- 2>/dev/null
fi
trap '' 0;;
esac;;
esac
if
$posix_mkdir && (
umask $mkdir_umask &&
$doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir"
)
then :
else
# The umask is ridiculous, or mkdir does not conform to POSIX,
# or it failed possibly due to a race condition. Create the
# directory the slow way, step by step, checking for races as we go.
case $dstdir in
/*) prefix='/';;
[-=\(\)!]*) prefix='./';;
*) prefix='';;
esac
eval "$initialize_posix_glob"
oIFS=$IFS
IFS=/
$posix_glob set -f
set fnord $dstdir
shift
$posix_glob set +f
IFS=$oIFS
prefixes=
for d
do
test X"$d" = X && continue
prefix=$prefix$d
if test -d "$prefix"; then
prefixes=
else
if $posix_mkdir; then
(umask=$mkdir_umask &&
$doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir") && break
# Don't fail if two instances are running concurrently.
test -d "$prefix" || exit 1
else
case $prefix in
*\'*) qprefix=`echo "$prefix" | sed "s/'/'\\\\\\\\''/g"`;;
*) qprefix=$prefix;;
esac
prefixes="$prefixes '$qprefix'"
fi
fi
prefix=$prefix/
done
if test -n "$prefixes"; then
# Don't fail if two instances are running concurrently.
(umask $mkdir_umask &&
eval "\$doit_exec \$mkdirprog $prefixes") ||
test -d "$dstdir" || exit 1
obsolete_mkdir_used=true
fi
fi
fi
if test -n "$dir_arg"; then
{ test -z "$chowncmd" || $doit $chowncmd "$dst"; } &&
{ test -z "$chgrpcmd" || $doit $chgrpcmd "$dst"; } &&
{ test "$obsolete_mkdir_used$chowncmd$chgrpcmd" = false ||
test -z "$chmodcmd" || $doit $chmodcmd $mode "$dst"; } || exit 1
else
# Make a couple of temp file names in the proper directory.
dsttmp=$dstdir/_inst.$$_
rmtmp=$dstdir/_rm.$$_
# Trap to clean up those temp files at exit.
trap 'ret=$?; rm -f "$dsttmp" "$rmtmp" && exit $ret' 0
# Copy the file name to the temp name.
(umask $cp_umask && $doit_exec $cpprog "$src" "$dsttmp") &&
# and set any options; do chmod last to preserve setuid bits.
#
# If any of these fail, we abort the whole thing. If we want to
# ignore errors from any of these, just make sure not to ignore
# errors from the above "$doit $cpprog $src $dsttmp" command.
#
{ test -z "$chowncmd" || $doit $chowncmd "$dsttmp"; } &&
{ test -z "$chgrpcmd" || $doit $chgrpcmd "$dsttmp"; } &&
{ test -z "$stripcmd" || $doit $stripcmd "$dsttmp"; } &&
{ test -z "$chmodcmd" || $doit $chmodcmd $mode "$dsttmp"; } &&
# If -C, don't bother to copy if it wouldn't change the file.
if $copy_on_change &&
old=`LC_ALL=C ls -dlL "$dst" 2>/dev/null` &&
new=`LC_ALL=C ls -dlL "$dsttmp" 2>/dev/null` &&
eval "$initialize_posix_glob" &&
$posix_glob set -f &&
set X $old && old=:$2:$4:$5:$6 &&
set X $new && new=:$2:$4:$5:$6 &&
$posix_glob set +f &&
test "$old" = "$new" &&
$cmpprog "$dst" "$dsttmp" >/dev/null 2>&1
then
rm -f "$dsttmp"
else
# Rename the file to the real destination.
$doit $mvcmd -f "$dsttmp" "$dst" 2>/dev/null ||
# The rename failed, perhaps because mv can't rename something else
# to itself, or perhaps because mv is so ancient that it does not
# support -f.
{
# Now remove or move aside any old file at destination location.
# We try this two ways since rm can't unlink itself on some
# systems and the destination file might be busy for other
# reasons. In this case, the final cleanup might fail but the new
# file should still install successfully.
{
test ! -f "$dst" ||
$doit $rmcmd -f "$dst" 2>/dev/null ||
{ $doit $mvcmd -f "$dst" "$rmtmp" 2>/dev/null &&
{ $doit $rmcmd -f "$rmtmp" 2>/dev/null; :; }
} ||
{ echo "$0: cannot unlink or rename $dst" >&2
(exit 1); exit 1
}
} &&
# Now rename the file to the real destination.
$doit $mvcmd "$dsttmp" "$dst"
}
fi || exit 1
trap '' 0
fi
done
# Local variables:
# eval: (add-hook 'write-file-hooks 'time-stamp)
# time-stamp-start: "scriptversion="
# time-stamp-format: "%:y-%02m-%02d.%02H"
# time-stamp-time-zone: "UTC"
# time-stamp-end: "; # UTC"
# End:

View file

@ -1,389 +0,0 @@
AC_INIT([nix],[m4_esyscmd(bash -c "echo -n $(cat ./.version)$VERSION_SUFFIX")])
AC_CONFIG_MACRO_DIRS([m4])
AC_CONFIG_SRCDIR(README.md)
AC_CONFIG_AUX_DIR(config)
AC_PROG_SED
# Construct a Nix system name (like "i686-linux"):
# https://www.gnu.org/software/autoconf/manual/html_node/Canonicalizing.html#index-AC_005fCANONICAL_005fHOST-1
# The inital value is produced by the `config/config.guess` script:
# upstream: https://git.savannah.gnu.org/cgit/config.git/tree/config.guess
# It has the following form, which is not documented anywhere:
# <cpu>-<vendor>-<os>[<version>][-<abi>]
# If `./configure` is passed any of the `--host`, `--build`, `--target` options, the value comes from `config/config.sub` instead:
# upstream: https://git.savannah.gnu.org/cgit/config.git/tree/config.sub
AC_CANONICAL_HOST
AC_MSG_CHECKING([for the canonical Nix system name])
AC_ARG_WITH(system, AS_HELP_STRING([--with-system=SYSTEM],[Platform identifier (e.g., `i686-linux').]),
[system=$withval],
[case "$host_cpu" in
i*86)
machine_name="i686";;
amd64)
machine_name="x86_64";;
armv6|armv7)
machine_name="${host_cpu}l";;
*)
machine_name="$host_cpu";;
esac
case "$host_os" in
linux-gnu*|linux-musl*)
# For backward compatibility, strip the `-gnu' part.
system="$machine_name-linux";;
*)
# Strip the version number from names such as `gnu0.3',
# `darwin10.2.0', etc.
system="$machine_name-`echo $host_os | "$SED" -e's/@<:@0-9.@:>@*$//g'`";;
esac])
AC_MSG_RESULT($system)
AC_SUBST(system)
AC_DEFINE_UNQUOTED(SYSTEM, ["$system"], [platform identifier ('cpu-os')])
# State should be stored in /nix/var, unless the user overrides it explicitly.
test "$localstatedir" = '${prefix}/var' && localstatedir=/nix/var
# Assign a default value to C{,XX}FLAGS as the default configure script sets them
# to -O2 otherwise, which we don't want to have hardcoded
CFLAGS=${CFLAGS-""}
CXXFLAGS=${CXXFLAGS-""}
AC_PROG_CC
AC_PROG_CXX
AC_PROG_CPP
AC_CHECK_TOOL([AR], [ar])
# Use 64-bit file system calls so that we can support files > 2 GiB.
AC_SYS_LARGEFILE
# Solaris-specific stuff.
AC_STRUCT_DIRENT_D_TYPE
case "$host_os" in
solaris*)
# Solaris requires -lsocket -lnsl for network functions
LDFLAGS="-lsocket -lnsl $LDFLAGS"
;;
esac
ENSURE_NO_GCC_BUG_80431
# Check for pubsetbuf.
AC_MSG_CHECKING([for pubsetbuf])
AC_LANG_PUSH(C++)
AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <iostream>
using namespace std;
static char buf[1024];]],
[[cerr.rdbuf()->pubsetbuf(buf, sizeof(buf));]])],
[AC_MSG_RESULT(yes) AC_DEFINE(HAVE_PUBSETBUF, 1, [Whether pubsetbuf is available.])],
AC_MSG_RESULT(no))
AC_LANG_POP(C++)
AC_CHECK_FUNCS([statvfs pipe2])
# Check for lutimes, optionally used for changing the mtime of
# symlinks.
AC_CHECK_FUNCS([lutimes])
# Check whether the store optimiser can optimise symlinks.
AC_MSG_CHECKING([whether it is possible to create a link to a symlink])
ln -s bla tmp_link
if ln tmp_link tmp_link2 2> /dev/null; then
AC_MSG_RESULT(yes)
AC_DEFINE(CAN_LINK_SYMLINK, 1, [Whether link() works on symlinks.])
else
AC_MSG_RESULT(no)
fi
rm -f tmp_link tmp_link2
# Check for <locale>.
AC_LANG_PUSH(C++)
AC_CHECK_HEADERS([locale])
AC_LANG_POP(C++)
AC_DEFUN([NEED_PROG],
[
AC_PATH_PROG($1, $2)
if test -z "$$1"; then
AC_MSG_ERROR([$2 is required])
fi
])
NEED_PROG(bash, bash)
AC_PATH_PROG(flex, flex, false)
AC_PATH_PROG(bison, bison, false)
AC_PATH_PROG(dot, dot)
AC_PATH_PROG(lsof, lsof, lsof)
NEED_PROG(jq, jq)
AC_SUBST(coreutils, [$(dirname $(type -p cat))])
AC_ARG_WITH(store-dir, AS_HELP_STRING([--with-store-dir=PATH],[path of the Nix store (defaults to /nix/store)]),
storedir=$withval, storedir='/nix/store')
AC_SUBST(storedir)
# Look for boost, a required dependency.
# Note that AX_BOOST_BASE only exports *CPP* BOOST_CPPFLAGS, no CXX flags,
# and CPPFLAGS are not passed to the C++ compiler automatically.
# Thus we append the returned CPPFLAGS to the CXXFLAGS here.
AX_BOOST_BASE([1.66], [CXXFLAGS="$BOOST_CPPFLAGS $CXXFLAGS"], [AC_MSG_ERROR([Nix requires boost.])])
# For unknown reasons, setting this directly in the ACTION-IF-FOUND above
# ends up with LDFLAGS being empty, so we set it afterwards.
LDFLAGS="$BOOST_LDFLAGS $LDFLAGS"
# On some platforms, new-style atomics need a helper library
AC_MSG_CHECKING(whether -latomic is needed)
AC_LINK_IFELSE([AC_LANG_SOURCE([[
#include <stdint.h>
uint64_t v;
int main() {
return (int)__atomic_load_n(&v, __ATOMIC_ACQUIRE);
}]])], GCC_ATOMIC_BUILTINS_NEED_LIBATOMIC=no, GCC_ATOMIC_BUILTINS_NEED_LIBATOMIC=yes)
AC_MSG_RESULT($GCC_ATOMIC_BUILTINS_NEED_LIBATOMIC)
if test "x$GCC_ATOMIC_BUILTINS_NEED_LIBATOMIC" = xyes; then
LDFLAGS="-latomic $LDFLAGS"
fi
# Running the functional tests without building Nix is useful for testing
# different pre-built versions of Nix against each other.
AC_ARG_ENABLE(build, AS_HELP_STRING([--disable-build],[Do not build nix]),
ENABLE_BUILD=$enableval, ENABLE_BUILD=yes)
AC_SUBST(ENABLE_BUILD)
# Building without tests is useful for bootstrapping with a smaller footprint
# or running the tests in a separate derivation. Otherwise, we do compile and
# run them.
AC_ARG_ENABLE(tests, AS_HELP_STRING([--disable-tests],[Do not build the tests]),
ENABLE_TESTS=$enableval, ENABLE_TESTS=yes)
AC_SUBST(ENABLE_TESTS)
# Building without API docs is the default as Nix' C++ interfaces are internal and unstable.
AC_ARG_ENABLE(internal_api_docs, AS_HELP_STRING([--enable-internal-api-docs],[Build API docs for Nix's internal unstable C++ interfaces]),
internal_api_docs=$enableval, internal_api_docs=no)
AC_SUBST(internal_api_docs)
# LTO is currently broken with clang for unknown reasons; ld segfaults in the llvm plugin
AC_ARG_ENABLE(lto, AS_HELP_STRING([--enable-lto],[Enable LTO (only supported with GCC) [default=no]]),
lto=$enableval, lto=no)
if test "$lto" = yes; then
if $CXX --version | grep -q GCC; then
AC_SUBST(CXXLTO, [-flto=jobserver])
else
echo "error: LTO is only supported with GCC at the moment" >&2
exit 1
fi
else
AC_SUBST(CXXLTO, [""])
fi
PKG_PROG_PKG_CONFIG
AC_ARG_ENABLE(shared, AS_HELP_STRING([--enable-shared],[Build shared libraries for Nix [default=yes]]),
shared=$enableval, shared=yes)
if test "$shared" = yes; then
AC_SUBST(BUILD_SHARED_LIBS, 1, [Whether to build shared libraries.])
else
AC_SUBST(BUILD_SHARED_LIBS, 0, [Whether to build shared libraries.])
PKG_CONFIG="$PKG_CONFIG --static"
fi
# Look for OpenSSL, a required dependency. FIXME: this is only (maybe)
# used by S3BinaryCacheStore.
PKG_CHECK_MODULES([OPENSSL], [libcrypto >= 1.1.1], [CXXFLAGS="$OPENSSL_CFLAGS $CXXFLAGS"])
# Look for libarchive.
PKG_CHECK_MODULES([LIBARCHIVE], [libarchive >= 3.1.2], [CXXFLAGS="$LIBARCHIVE_CFLAGS $CXXFLAGS"])
# Workaround until https://github.com/libarchive/libarchive/issues/1446 is fixed
if test "$shared" != yes; then
LIBARCHIVE_LIBS+=' -lz'
fi
# Look for SQLite, a required dependency.
PKG_CHECK_MODULES([SQLITE3], [sqlite3 >= 3.6.19], [CXXFLAGS="$SQLITE3_CFLAGS $CXXFLAGS"])
# Look for libcurl, a required dependency.
PKG_CHECK_MODULES([LIBCURL], [libcurl], [CXXFLAGS="$LIBCURL_CFLAGS $CXXFLAGS"])
# Look for editline, a required dependency.
# The the libeditline.pc file was added only in libeditline >= 1.15.2,
# see https://github.com/troglobit/editline/commit/0a8f2ef4203c3a4a4726b9dd1336869cd0da8607,
# but e.g. Ubuntu 16.04 has an older version, so we fall back to searching for
# editline.h when the pkg-config approach fails.
PKG_CHECK_MODULES([EDITLINE], [libeditline], [CXXFLAGS="$EDITLINE_CFLAGS $CXXFLAGS"], [
AC_CHECK_HEADERS([editline.h], [true],
[AC_MSG_ERROR([Nix requires libeditline; it was found neither via pkg-config nor its normal header.])])
AC_SEARCH_LIBS([readline read_history], [editline], [],
[AC_MSG_ERROR([Nix requires libeditline; it was not found via pkg-config, but via its header, but required functions do not work. Maybe it is too old? >= 1.14 is required.])])
])
# Look for libsodium.
PKG_CHECK_MODULES([SODIUM], [libsodium], [CXXFLAGS="$SODIUM_CFLAGS $CXXFLAGS"])
# Look for libbrotli{enc,dec}.
PKG_CHECK_MODULES([LIBBROTLI], [libbrotlienc libbrotlidec], [CXXFLAGS="$LIBBROTLI_CFLAGS $CXXFLAGS"])
# Look for libcpuid.
have_libcpuid=
if test "$machine_name" = "x86_64"; then
AC_ARG_ENABLE([cpuid],
AS_HELP_STRING([--disable-cpuid], [Do not determine microarchitecture levels with libcpuid (relevant to x86_64 only)]))
if test "x$enable_cpuid" != "xno"; then
PKG_CHECK_MODULES([LIBCPUID], [libcpuid],
[CXXFLAGS="$LIBCPUID_CFLAGS $CXXFLAGS"
have_libcpuid=1
AC_DEFINE([HAVE_LIBCPUID], [1], [Use libcpuid])]
)
fi
fi
AC_SUBST(HAVE_LIBCPUID, [$have_libcpuid])
# Look for libseccomp, required for Linux sandboxing.
case "$host_os" in
linux*)
AC_ARG_ENABLE([seccomp-sandboxing],
AS_HELP_STRING([--disable-seccomp-sandboxing],[Don't build support for seccomp sandboxing (only recommended if your arch doesn't support libseccomp yet!)
]))
if test "x$enable_seccomp_sandboxing" != "xno"; then
PKG_CHECK_MODULES([LIBSECCOMP], [libseccomp],
[CXXFLAGS="$LIBSECCOMP_CFLAGS $CXXFLAGS"])
have_seccomp=1
AC_DEFINE([HAVE_SECCOMP], [1], [Whether seccomp is available and should be used for sandboxing.])
else
have_seccomp=
fi
;;
*)
have_seccomp=
;;
esac
AC_SUBST(HAVE_SECCOMP, [$have_seccomp])
# Look for aws-cpp-sdk-s3.
AC_LANG_PUSH(C++)
AC_CHECK_HEADERS([aws/s3/S3Client.h],
[AC_DEFINE([ENABLE_S3], [1], [Whether to enable S3 support via aws-sdk-cpp.]) enable_s3=1],
[AC_DEFINE([ENABLE_S3], [0], [Whether to enable S3 support via aws-sdk-cpp.]) enable_s3=])
AC_SUBST(ENABLE_S3, [$enable_s3])
AC_LANG_POP(C++)
if test -n "$enable_s3"; then
declare -a aws_version_tokens=($(printf '#include <aws/core/VersionConfig.h>\nAWS_SDK_VERSION_STRING' | $CPP $CPPFLAGS - | grep -v '^#.*' | sed 's/"//g' | tr '.' ' '))
AC_DEFINE_UNQUOTED([AWS_VERSION_MAJOR], ${aws_version_tokens@<:@0@:>@}, [Major version of aws-sdk-cpp.])
AC_DEFINE_UNQUOTED([AWS_VERSION_MINOR], ${aws_version_tokens@<:@1@:>@}, [Minor version of aws-sdk-cpp.])
AC_DEFINE_UNQUOTED([AWS_VERSION_PATCH], ${aws_version_tokens@<:@2@:>@}, [Patch version of aws-sdk-cpp.])
fi
# Whether to use the Boehm garbage collector.
AC_ARG_ENABLE(gc, AS_HELP_STRING([--enable-gc],[enable garbage collection in the Nix expression evaluator (requires Boehm GC) [default=yes]]),
gc=$enableval, gc=yes)
if test "$gc" = yes; then
PKG_CHECK_MODULES([BDW_GC], [bdw-gc])
CXXFLAGS="$BDW_GC_CFLAGS $CXXFLAGS"
AC_DEFINE(HAVE_BOEHMGC, 1, [Whether to use the Boehm garbage collector.])
fi
if test "$ENABLE_TESTS" = yes; then
# Look for gtest.
PKG_CHECK_MODULES([GTEST], [gtest_main gmock_main])
# Look for rapidcheck.
AC_ARG_VAR([RAPIDCHECK_HEADERS], [include path of gtest headers shipped by RAPIDCHECK])
# No pkg-config yet, https://github.com/emil-e/rapidcheck/issues/302
AC_LANG_PUSH(C++)
AC_SUBST(RAPIDCHECK_HEADERS)
[CXXFLAGS="-I $RAPIDCHECK_HEADERS $CXXFLAGS"]
[LIBS="-lrapidcheck -lgtest $LIBS"]
AC_CHECK_HEADERS([rapidcheck/gtest.h], [], [], [#include <gtest/gtest.h>])
dnl AC_CHECK_LIB doesn't work for C++ libs with mangled symbols
AC_LINK_IFELSE([
AC_LANG_PROGRAM([[
#include <gtest/gtest.h>
#include <rapidcheck/gtest.h>
]], [[
return RUN_ALL_TESTS();
]])
],
[],
[AC_MSG_ERROR([librapidcheck is not found.])])
AC_LANG_POP(C++)
fi
# Look for nlohmann/json.
PKG_CHECK_MODULES([NLOHMANN_JSON], [nlohmann_json >= 3.9])
# documentation generation switch
AC_ARG_ENABLE(doc-gen, AS_HELP_STRING([--disable-doc-gen],[disable documentation generation]),
doc_generate=$enableval, doc_generate=yes)
AC_SUBST(doc_generate)
# Look for lowdown library.
PKG_CHECK_MODULES([LOWDOWN], [lowdown >= 0.9.0], [CXXFLAGS="$LOWDOWN_CFLAGS $CXXFLAGS"])
# Setuid installations.
AC_CHECK_FUNCS([setresuid setreuid lchown])
# Nice to have, but not essential.
AC_CHECK_FUNCS([strsignal posix_fallocate sysconf])
AC_ARG_WITH(sandbox-shell, AS_HELP_STRING([--with-sandbox-shell=PATH],[path of a statically-linked shell to use as /bin/sh in sandboxes]),
sandbox_shell=$withval)
AC_SUBST(sandbox_shell)
if test ${cross_compiling:-no} = no && ! test -z ${sandbox_shell+x}; then
AC_MSG_CHECKING([whether sandbox-shell has the standalone feature])
# busybox shell sometimes allows executing other busybox applets,
# even if they are not in the path, breaking our sandbox
if PATH= $sandbox_shell -c "busybox" 2>&1 | grep -qv "not found"; then
AC_MSG_RESULT(enabled)
AC_MSG_ERROR([Please disable busybox FEATURE_SH_STANDALONE])
else
AC_MSG_RESULT(disabled)
fi
fi
AC_ARG_ENABLE(embedded-sandbox-shell, AS_HELP_STRING([--enable-embedded-sandbox-shell],[include the sandbox shell in the Nix binary [default=no]]),
embedded_sandbox_shell=$enableval, embedded_sandbox_shell=no)
AC_SUBST(embedded_sandbox_shell)
if test "$embedded_sandbox_shell" = yes; then
AC_DEFINE(HAVE_EMBEDDED_SANDBOX_SHELL, 1, [Include the sandbox shell in the Nix binary.])
fi
# Expand all variables in config.status.
test "$prefix" = NONE && prefix=$ac_default_prefix
test "$exec_prefix" = NONE && exec_prefix='${prefix}'
for name in $ac_subst_vars; do
declare $name="$(eval echo "${!name}")"
declare $name="$(eval echo "${!name}")"
declare $name="$(eval echo "${!name}")"
done
rm -f Makefile.config
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([])
AC_OUTPUT

View file

@ -1,10 +1,9 @@
(import
(
let lock = builtins.fromJSON (builtins.readFile ./flake.lock); in
fetchTarball {
url = "https://github.com/edolstra/flake-compat/archive/${lock.nodes.flake-compat.locked.rev}.tar.gz";
sha256 = lock.nodes.flake-compat.locked.narHash;
}
)
{ src = ./.; }
).defaultNix
(import (
let
lock = builtins.fromJSON (builtins.readFile ./flake.lock);
in
fetchTarball {
url = "https://github.com/edolstra/flake-compat/archive/${lock.nodes.flake-compat.locked.rev}.tar.gz";
sha256 = lock.nodes.flake-compat.locked.narHash;
}
) { src = ./.; }).defaultNix

View file

@ -14,11 +14,13 @@ PROJECT_NAME = "Nix"
PROJECT_NUMBER = @PACKAGE_VERSION@
OUTPUT_DIRECTORY = @docdir@
# Using the PROJECT_BRIEF tag one can provide an optional one line description
# for a project that appears at the top of each page and should give viewer a
# quick idea about the purpose of the project. Keep the description short.
PROJECT_BRIEF = "Nix, the purely functional package manager; unstable internal interfaces"
PROJECT_BRIEF = "Lix: A modern, delicious implementation of the Nix package manager; unstable internal interfaces"
# If the GENERATE_LATEX tag is set to YES, doxygen will generate LaTeX output.
# The default value is: YES.
@ -31,32 +33,7 @@ GENERATE_LATEX = NO
# spaces. See also FILE_PATTERNS and EXTENSION_MAPPING
# Note: If this tag is empty the current directory is searched.
# FIXME Make this list more maintainable somehow. We could maybe generate this
# in the Makefile, but we would need to change how `.in` files are preprocessed
# so they can expand variables despite configure variables.
INPUT = \
src/libcmd \
src/libexpr \
src/libexpr/flake \
tests/unit/libexpr \
tests/unit/libexpr/value \
tests/unit/libexpr/test \
tests/unit/libexpr/test/value \
src/libexpr/value \
src/libfetchers \
src/libmain \
src/libstore \
src/libstore/build \
src/libstore/builtins \
tests/unit/libstore \
tests/unit/libstore/test \
src/libutil \
tests/unit/libutil \
tests/unit/libutil/test \
src/nix \
src/nix-env \
src/nix-store
INPUT = @INPUT_PATHS@
# If the MACRO_EXPANSION tag is set to YES, doxygen will expand all macro names
# in the source code. If set to NO, only conditional compilation will be
@ -95,3 +72,15 @@ EXPAND_AS_DEFINED = \
DECLARE_WORKER_SERIALISER \
DECLARE_SERVE_SERIALISER \
LENGTH_PREFIXED_PROTO_HELPER
# The STRIP_FROM_PATH tag can be used to strip a user-defined part of the path.
# Stripping is only done if one of the specified strings matches the left-hand
# part of the path. The tag can be used to show relative paths in the file list.
# If left blank the directory from which doxygen is run is used as the path to
# strip.
#
# Note that you can specify absolute paths here, but also relative paths, which
# will be relative from the directory where doxygen is started.
# This tag requires that the tag FULL_PATH_NAMES is set to YES.
STRIP_FROM_PATH = "@PROJECT_SOURCE_ROOT@"

View file

@ -1,19 +0,0 @@
.PHONY: internal-api-html
ifeq ($(internal_api_docs), yes)
$(docdir)/internal-api/html/index.html $(docdir)/internal-api/latex: $(d)/doxygen.cfg
mkdir -p $(docdir)/internal-api
{ cat $< ; echo "OUTPUT_DIRECTORY=$(docdir)/internal-api" ; } | doxygen -
# Generate the HTML API docs for Nix's unstable internal interfaces.
internal-api-html: $(docdir)/internal-api/html/index.html
else
# Make a nicer error message
internal-api-html:
@echo "Internal API docs are disabled. Configure with '--enable-internal-api-docs', or avoid calling 'make internal-api-html'."
@exit 1
endif

View file

@ -0,0 +1,60 @@
internal_api_sources = [
'src/libcmd',
'src/libexpr',
'src/libexpr/flake',
'tests/unit/libexpr',
'tests/unit/libexpr/value',
'tests/unit/libexpr/test',
'tests/unit/libexpr/test/value',
'src/libexpr/value',
'src/libfetchers',
'src/libmain',
'src/libstore',
'src/libstore/build',
'src/libstore/builtins',
'tests/unit/libstore',
'tests/unit/libstore/test',
'src/libutil',
'tests/unit/libutil',
'tests/unit/libutil/test',
'src/nix',
'src/nix-env',
'src/nix-store',
]
# We feed Doxygen absolute paths so it can be invoked from any working directory.
internal_api_sources_absolute = []
foreach src : internal_api_sources
internal_api_sources_absolute += '"' + (meson.project_source_root() / src) + '"'
endforeach
internal_api_sources_oneline = ' \\\n '.join(internal_api_sources_absolute)
doxygen_cfg = configure_file(
input : 'doxygen.cfg.in',
output : 'doxygen.cfg',
configuration : {
'PACKAGE_VERSION': meson.project_version(),
'RAPIDCHECK_HEADERS': rapidcheck_meson.get_variable('includedir'),
'docdir' : meson.current_build_dir(),
'INPUT_PATHS' : internal_api_sources_oneline,
'PROJECT_SOURCE_ROOT' : meson.project_source_root(),
},
)
internal_api_docs = custom_target(
'internal-api-docs',
command : [
doxygen.full_path(),
'@INPUT0@',
],
input : [
doxygen_cfg,
],
output : 'html',
install : true,
install_dir : datadir / 'doc/nix/internal-api',
build_always_stale : true,
)
alias_target('internal-api-html', internal_api_docs)

0
doc/manual/anchors.jq Executable file → Normal file
View file

View file

@ -1,16 +1,37 @@
[book]
title = "Nix Reference Manual"
title = "Lix Reference Manual"
[build]
create-missing = false
[output.html]
additional-css = ["custom.css"]
additional-js = ["redirects.js"]
edit-url-template = "https://github.com/NixOS/nix/tree/master/doc/manual/{path}"
git-repository-url = "https://github.com/NixOS/nix"
# Using our GitHub mirror enables easier typo fixes since there is no easy way
# to just submit a Gerrit CL by the web for trivial stuff.
edit-url-template = "https://github.com/lix-project/lix/tree/main/doc/manual/{path}"
git-repository-url = "https://git.lix.systems/lix-project/lix"
# Folding by default would prevent things like "Ctrl+F for nix-env" from working
# trivially, but the user should be able to fold if they want to.
fold.enable = true
fold.level = 30
# Handles replacing @docroot@ with a path to ./src relative to that markdown file,
# {{#include handlebars}}, and the @generated@ syntax used within these. it mostly
# but not entirely replaces the links preprocessor (which we cannot simply use due
# to @generated@ files living in a different directory to make meson happy). we do
# not want to disable the links preprocessor entirely though because that requires
# disabling *all* built-in preprocessors and selectively reenabling those we want.
[preprocessor.substitute]
command = "python3 doc/manual/substitute.py"
before = ["anchors", "links"]
[preprocessor.anchors]
renderers = ["html"]
command = "jq --from-file doc/manual/anchors.jq"
[output.markdown]
[output.linkcheck]
# no Internet during the build (in the sandbox)
follow-web-links = false

View file

@ -0,0 +1,161 @@
# This file is a mapping of metadata for change authors, sort of like
# maintainer-list.nix in nixpkgs.
#
# It's used for crediting people accurately in release notes. The release notes
# script will link to forgejo, then to GitHub if forgejo is not present.
#
# When adding someone from outside the Lix project, you generally want to simply link their GitHub profile without adding a display name unless they are well-known in the community by that display name.
#
# See doc/manual/src/contributing/hacking.md for more documentation on this file's format and typical usage.
9999years:
display_name: wiggles
forgejo: rbt
github: 9999years
Artturin:
github: Artturin
DavHau:
github: DavHau
Kha:
github: Kha
Lunaphied:
forgejo: Lunaphied
github: Lunaphied
Qyriad:
forgejo: Qyriad
github: Qyriad
SharzyL:
github: SharzyL
alois31:
forgejo: alois31
github: alois31
artemist:
display_name: Artemis Tosini
forgejo: artemist
cole-h:
display_name: Cole Helbling
github: cole-h
delan:
display_name: delan
forgejo: delan
github: delan
edolstra:
display_name: Eelco Dolstra
github: edolstra
ericson:
display_name: John Ericson
github: ericson2314
goldstein:
display_name: goldstein
forgejo: goldstein
github: GoldsteinE
horrors:
display_name: eldritch horrors
forgejo: pennae
github: pennae
iFreilicht:
github: iFreilicht
isabelroses:
forgejo: isabelroses
github: isabelroses
jade:
forgejo: jade
github: lf-
kjeremy:
github: kjeremy
kloenk:
forgejo: kloenk
github: kloenk
lheckemann:
forgejo: lheckemann
github: lheckemann
lovesegfault:
github: lovesegfault
ma27:
forgejo: ma27
github: ma27
matthewbauer:
github: matthewbauer
midnightveil:
display_name: julia
forgejo: midnightveil
github: midnightveil
ncfavier:
github: ncfavier
piegames:
display_name: piegames
forgejo: piegames
github: piegamesde
puck:
display_name: puck
forgejo: puck
github: puckipedia
quantumjump:
display_name: Quantum Jump
github: QuantumBJump
r-vdp:
github: r-vdp
raito:
display_name: Raito Bezarius
forgejo: raito
github: RaitoBezarius
roberth:
display_name: Robert Hensing
github: roberth
thufschmitt:
display_name: Théophane Hufschmitt
github: thufschmitt
tomberek:
display_name: Tom Bereknyei
github: tomberek
valentin:
display_name: Valentin Gagarin
github: fricklerhandwerk
vigress8:
display_name: Vigress
forgejo: vigress8
github: vigress8
winter:
forgejo: winter
github: winterqt
yshui:
github: yshui
zimbatm:
github: zimbatm

View file

@ -5,7 +5,13 @@ in
builtinsInfo:
let
showBuiltin = name: { doc, type, impure-only }:
showBuiltin =
name:
{
doc,
type,
impure-only,
}:
let
type' = optionalString (type != null) " (${type})";

View file

@ -5,7 +5,14 @@ in
builtinsInfo:
let
showBuiltin = name: { doc, args, arity, experimental-feature }:
showBuiltin =
name:
{
doc,
args,
arity,
experimental-feature,
}:
let
experimentalNotice = optionalString (experimental-feature != null) ''
This function is only available if the [${experimental-feature}](@docroot@/contributing/experimental-features.md#xp-feature-${experimental-feature}) experimental feature is enabled.

22
doc/manual/generate-deps.py Executable file
View file

@ -0,0 +1,22 @@
#!/usr/bin/env python3
import glob
import sys
# meson expects makefile-style dependency declarations, i.e.
#
# target: dependency...
#
# meson seems to pass depfiles straight on to ninja even though
# it also parses the file itself (or at least has code to do so
# in its tree), so we must live by ninja's rules: only slashes,
# spaces and octothorpes can be escaped, anything else is taken
# literally. since the rules for these aren't even the same for
# all three we will just fail when we encounter any of them (if
# asserts are off for some reason the depfile will likely point
# to nonexistant paths, making everything phony and thus fine.)
for path in glob.glob(sys.argv[1] + '/**', recursive=True):
assert '\\' not in path
assert ' ' not in path
assert '#' not in path
print("ignored:", path)

View file

@ -0,0 +1,14 @@
# Usually "experimental" or "deprecated"
kind:
# "xp" or "dp"
kindShort:
with builtins;
with import ./utils.nix;
let
showExperimentalFeature = name: doc: ''
- [`${name}`](@docroot@/contributing/${kind}-features.md#${kindShort}-feature-${name})
'';
in
xps: indent " " (concatStrings (attrValues (mapAttrs showExperimentalFeature xps)))

View file

@ -0,0 +1,18 @@
# Usually "experimental" or "deprecated"
_kind:
# "xp" or "dp"
kindShort:
with builtins;
with import ./utils.nix;
let
showFeature =
name: doc:
squash ''
## [`${name}`]{#${kindShort}-feature-${name}}
${doc}
'';
in
xps: (concatStringsSep "\n" (attrValues (mapAttrs showFeature xps)))

View file

@ -1,8 +1,26 @@
let
inherit (builtins)
attrNames attrValues fromJSON listToAttrs mapAttrs
concatStringsSep concatMap length lessThan replaceStrings sort;
inherit (import ./utils.nix) concatStrings optionalString filterAttrs trim squash unique showSettings;
attrNames
attrValues
fromJSON
listToAttrs
mapAttrs
concatStringsSep
concatMap
length
lessThan
replaceStrings
sort
;
inherit (import ./utils.nix)
concatStrings
optionalString
filterAttrs
trim
squash
unique
showSettings
;
in
inlineHTML: commandDump:
@ -11,7 +29,13 @@ let
commandInfo = fromJSON commandDump;
showCommand = { command, details, filename, toplevel }:
showCommand =
{
command,
details,
filename,
toplevel,
}:
let
result = ''
@ -35,26 +59,27 @@ let
${maybeOptions}
'';
showSynopsis = command: args:
showSynopsis =
command: args:
let
showArgument = arg: "*${arg.label}*" + optionalString (! arg ? arity) "...";
showArgument = arg: "*${arg.label}*" + optionalString (!arg ? arity) "...";
arguments = concatStringsSep " " (map showArgument args);
in ''
`${command}` [*option*...] ${arguments}
in
''
`${command}` [*option*...] ${arguments}
'';
maybeSubcommands = optionalString (details ? commands && details.commands != {})
''
where *subcommand* is one of the following:
maybeSubcommands = optionalString (details ? commands && details.commands != { }) ''
where *subcommand* is one of the following:
${subcommands}
'';
${subcommands}
'';
subcommands = if length categories > 1
then listCategories
else listSubcommands details.commands;
subcommands = if length categories > 1 then listCategories else listSubcommands details.commands;
categories = sort (x: y: x.id < y.id) (unique (map (cmd: cmd.category) (attrValues details.commands)));
categories = sort (x: y: x.id < y.id) (
unique (map (cmd: cmd.category) (attrValues details.commands))
);
listCategories = concatStrings (map showCategory categories);
@ -71,11 +96,11 @@ let
'';
# TODO: move this confusing special case out of here when implementing #8496
maybeStoreDocs = optionalString
(details ? doc)
(replaceStrings ["@stores@"] [storeDocs] details.doc);
maybeStoreDocs = optionalString (details ? doc) (
replaceStrings [ "@stores@" ] [ storeDocs ] details.doc
);
maybeOptions = optionalString (details.flags != {}) ''
maybeOptions = optionalString (details.flags != { }) ''
# Options
${showOptions details.flags toplevel.flags}
@ -85,51 +110,70 @@ let
> See [`man nix.conf`](@docroot@/command-ref/conf-file.md#command-line-flags) for overriding configuration settings with command line flags.
'';
showOptions = options: commonOptions:
showOptions =
options: commonOptions:
let
allOptions = options // commonOptions;
showCategory = cat: ''
${optionalString (cat != "") "**${cat}:**"}
${listOptions (filterAttrs (n: v: v.category == cat) allOptions)}
'';
'';
listOptions = opts: concatStringsSep "\n" (attrValues (mapAttrs showOption opts));
showOption = name: option:
showOption =
name: option:
let
result = trim ''
- ${item}
${option.description}
'';
item = if inlineHTML
then ''<span id="opt-${name}">[`--${name}`](#opt-${name})</span> ${shortName} ${labels}''
else "`--${name}` ${shortName} ${labels}";
shortName = optionalString
(option ? shortName)
("/ `-${option.shortName}`");
labels = optionalString
(option ? labels)
(concatStringsSep " " (map (s: "*${s}*") option.labels));
in result;
item =
if inlineHTML then
''<span id="opt-${name}">[`--${name}`](#opt-${name})</span> ${shortName} ${labels}''
else
"`--${name}` ${shortName} ${labels}";
shortName = optionalString (option ? shortName) ("/ `-${option.shortName}`");
labels = optionalString (option ? labels) (concatStringsSep " " (map (s: "*${s}*") option.labels));
in
result;
categories = sort lessThan (unique (map (cmd: cmd.category) (attrValues allOptions)));
in concatStrings (map showCategory categories);
in squash result;
in
concatStrings (map showCategory categories);
in
squash result;
appendName = filename: name: (if filename == "nix" then "nix3" else filename) + "-" + name;
processCommand = { command, details, filename, toplevel }:
processCommand =
{
command,
details,
filename,
toplevel,
}:
let
cmd = {
inherit command;
name = filename + ".md";
value = showCommand { inherit command details filename toplevel; };
value = showCommand {
inherit
command
details
filename
toplevel
;
};
};
subcommand = subCmd: processCommand {
command = command + " " + subCmd;
details = details.commands.${subCmd};
filename = appendName filename subCmd;
inherit toplevel;
};
in [ cmd ] ++ concatMap subcommand (attrNames details.commands or {});
subcommand =
subCmd:
processCommand {
command = command + " " + subCmd;
details = details.commands.${subCmd};
filename = appendName filename subCmd;
inherit toplevel;
};
in
[ cmd ] ++ concatMap subcommand (attrNames details.commands or { });
manpages = processCommand {
command = "nix";
@ -138,14 +182,15 @@ let
toplevel = commandInfo.args;
};
tableOfContents = let
showEntry = page:
" - [${page.command}](command-ref/new-cli/${page.name})";
in concatStringsSep "\n" (map showEntry manpages) + "\n";
storeDocs =
let
showStore = name: { settings, doc, experimentalFeature }:
showStore =
name:
{
settings,
doc,
experimentalFeature,
}:
let
experimentalFeatureNote = optionalString (experimentalFeature != null) ''
> **Warning**
@ -161,7 +206,8 @@ let
extra-experimental-features = ${experimentalFeature}
```
'';
in ''
in
''
## ${name}
${doc}
@ -172,6 +218,7 @@ let
${showSettings { inherit inlineHTML; } settings}
'';
in concatStrings (attrValues (mapAttrs showStore commandInfo.stores));
in (listToAttrs manpages) // { "SUMMARY.md" = tableOfContents; }
in
concatStrings (attrValues (mapAttrs showStore commandInfo.stores));
in
listToAttrs manpages

View file

@ -1,9 +0,0 @@
with builtins;
with import ./utils.nix;
let
showExperimentalFeature = name: doc:
''
- [`${name}`](@docroot@/contributing/experimental-features.md#xp-feature-${name})
'';
in xps: indent " " (concatStrings (attrValues (mapAttrs showExperimentalFeature xps)))

View file

@ -1,11 +0,0 @@
with builtins;
with import ./utils.nix;
let
showExperimentalFeature = name: doc:
squash ''
## [`${name}`]{#xp-feature-${name}}
${doc}
'';
in xps: (concatStringsSep "\n" (attrValues (mapAttrs showExperimentalFeature xps)))

61
doc/manual/json-to-tree.py Executable file
View file

@ -0,0 +1,61 @@
#!/usr/bin/env python3
"""
This script is a helper for this project's Meson buildsystem, to replace its
usage of `nix eval --write-to`. Writing a JSON object as a nested directory
tree is more generic, easier to maintain, and far, far less cursed. Nix
has 'good' support for JSON output. Let's just use it.
"""
import argparse
from pathlib import Path
import json
import sys
name = 'json-to-tree.py'
def log(*args, **kwargs):
kwargs['file'] = sys.stderr
return print(f'{name}:', *args, **kwargs)
def write_dict_to_directory(current_directory: Path, data: dict, files_written=0):
current_directory.mkdir(parents=True, exist_ok=True)
for key, value in data.items():
nested_path = current_directory / key
match value:
case dict(nested_data):
files_written += write_dict_to_directory(nested_path, nested_data)
case str(content):
nested_path.write_text(content)
files_written += 1
case rest:
assert False, \
f'should have been called on a dict or string, not {type(rest)=}\n\t{rest=}'
return files_written
def main():
parser = argparse.ArgumentParser(name)
parser.add_argument('-i', '--input', type=argparse.FileType('r'), default='-',
help='The JSON input to operate on and output as a directory tree',
)
parser.add_argument('-o', '--output', type=Path, required=True,
help='The place to put the directory tree',
)
args = parser.parse_args()
json_string = args.input.read()
try:
data = json.loads(json_string)
except json.JSONDecodeError:
log(f'could not decode JSON from input: {json_string}')
raise
files_written = write_dict_to_directory(args.output, data)
log(f'wrote {files_written} files')
sys.exit(main())

View file

@ -1,218 +0,0 @@
ifeq ($(doc_generate),yes)
# The version of Nix used to generate the doc. Can also be
# `$(nix_INSTALL_PATH)` or just `nix` (to grap ambient from the `PATH`),
# if one prefers.
doc_nix = $(nix_PATH)
MANUAL_SRCS := \
$(call rwildcard, $(d)/src, *.md) \
$(call rwildcard, $(d)/src, */*.md)
man-pages := $(foreach n, \
nix-env.1 nix-store.1 \
nix-build.1 nix-shell.1 nix-instantiate.1 \
nix-collect-garbage.1 \
nix-prefetch-url.1 nix-channel.1 \
nix-hash.1 nix-copy-closure.1 \
nix.conf.5 nix-daemon.8 \
nix-profiles.5 \
, $(d)/$(n))
# man pages for subcommands
# convert from `$(d)/src/command-ref/nix-{1}/{2}.md` to `$(d)/nix-{1}-{2}.1`
# FIXME: unify with how nix3-cli man pages are generated
man-pages += $(foreach subcommand, \
$(filter-out %opt-common.md %env-common.md, $(wildcard $(d)/src/command-ref/nix-*/*.md)), \
$(d)/$(subst /,-,$(subst $(d)/src/command-ref/,,$(subst .md,.1,$(subcommand)))))
clean-files += $(d)/*.1 $(d)/*.5 $(d)/*.8
# Provide a dummy environment for nix, so that it will not access files outside the macOS sandbox.
# Set cores to 0 because otherwise nix show-config resolves the cores based on the current machine
dummy-env = env -i \
HOME=/dummy \
NIX_CONF_DIR=/dummy \
NIX_SSL_CERT_FILE=/dummy/no-ca-bundle.crt \
NIX_STATE_DIR=/dummy \
NIX_CONFIG='cores = 0'
nix-eval = $(dummy-env) $(doc_nix) eval --experimental-features nix-command -I nix/corepkgs=corepkgs --store dummy:// --impure --raw
# re-implement mdBook's include directive to make it usable for terminal output and for proper @docroot@ substitution
define process-includes
while read -r line; do \
set -euo pipefail; \
filename="$$(dirname $(1))/$$(sed 's/{{#include \(.*\)}}/\1/'<<< $$line)"; \
test -f "$$filename" || ( echo "#include-d file '$$filename' does not exist." >&2; exit 1; ); \
matchline="$$(sed 's|/|\\/|g' <<< $$line)"; \
sed -i "/$$matchline/r $$filename" $(2); \
sed -i "s/$$matchline//" $(2); \
done < <(grep '{{#include' $(1))
endef
$(d)/nix-env-%.1: $(d)/src/command-ref/nix-env/%.md
@printf "Title: %s\n\n" "$(subst nix-env-,nix-env --,$$(basename "$@" .1))" > $^.tmp
$(render-subcommand)
$(d)/nix-store-%.1: $(d)/src/command-ref/nix-store/%.md
@printf -- 'Title: %s\n\n' "$(subst nix-store-,nix-store --,$$(basename "$@" .1))" > $^.tmp
$(render-subcommand)
# FIXME: there surely is some more deduplication to be achieved here with even darker Make magic
define render-subcommand
@cat $^ >> $^.tmp
@$(call process-includes,$^,$^.tmp)
$(trace-gen) lowdown -sT man --nroff-nolinks -M section=1 $^.tmp -o $@
@# fix up `lowdown`'s automatic escaping of `--`
@# https://github.com/kristapsdz/lowdown/blob/edca6ce6d5336efb147321a43c47a698de41bb7c/entity.c#L202
@sed -i 's/\e\[u2013\]/--/' $@
@rm $^.tmp
endef
$(d)/%.1: $(d)/src/command-ref/%.md
@printf "Title: %s\n\n" "$$(basename $@ .1)" > $^.tmp
@cat $^ >> $^.tmp
@$(call process-includes,$^,$^.tmp)
$(trace-gen) lowdown -sT man --nroff-nolinks -M section=1 $^.tmp -o $@
@rm $^.tmp
$(d)/%.8: $(d)/src/command-ref/%.md
@printf "Title: %s\n\n" "$$(basename $@ .8)" > $^.tmp
@cat $^ >> $^.tmp
$(trace-gen) lowdown -sT man --nroff-nolinks -M section=8 $^.tmp -o $@
@rm $^.tmp
$(d)/nix.conf.5: $(d)/src/command-ref/conf-file.md
@printf "Title: %s\n\n" "$$(basename $@ .5)" > $^.tmp
@cat $^ >> $^.tmp
@$(call process-includes,$^,$^.tmp)
$(trace-gen) lowdown -sT man --nroff-nolinks -M section=5 $^.tmp -o $@
@rm $^.tmp
$(d)/nix-profiles.5: $(d)/src/command-ref/files/profiles.md
@printf "Title: %s\n\n" "$$(basename $@ .5)" > $^.tmp
@cat $^ >> $^.tmp
$(trace-gen) lowdown -sT man --nroff-nolinks -M section=5 $^.tmp -o $@
@rm $^.tmp
$(d)/src/SUMMARY.md: $(d)/src/SUMMARY.md.in $(d)/src/SUMMARY-rl-next.md $(d)/src/command-ref/new-cli $(d)/src/contributing/experimental-feature-descriptions.md
@cp $< $@
@$(call process-includes,$@,$@)
$(d)/src/command-ref/new-cli: $(d)/nix.json $(d)/utils.nix $(d)/generate-manpage.nix $(doc_nix)
@rm -rf $@ $@.tmp
$(trace-gen) $(nix-eval) --write-to $@.tmp --expr 'import doc/manual/generate-manpage.nix true (builtins.readFile $<)'
@mv $@.tmp $@
$(d)/src/command-ref/conf-file.md: $(d)/conf-file.json $(d)/utils.nix $(d)/src/command-ref/conf-file-prefix.md $(d)/src/command-ref/experimental-features-shortlist.md $(doc_nix)
@cat doc/manual/src/command-ref/conf-file-prefix.md > $@.tmp
$(trace-gen) $(nix-eval) --expr '(import doc/manual/utils.nix).showSettings { inlineHTML = true; } (builtins.fromJSON (builtins.readFile $<))' >> $@.tmp;
@mv $@.tmp $@
$(d)/nix.json: $(doc_nix)
$(trace-gen) $(dummy-env) $(doc_nix) __dump-cli > $@.tmp
@mv $@.tmp $@
$(d)/conf-file.json: $(doc_nix)
$(trace-gen) $(dummy-env) $(doc_nix) show-config --json --experimental-features nix-command > $@.tmp
@mv $@.tmp $@
$(d)/src/contributing/experimental-feature-descriptions.md: $(d)/xp-features.json $(d)/utils.nix $(d)/generate-xp-features.nix $(doc_nix)
@rm -rf $@ $@.tmp
$(trace-gen) $(nix-eval) --write-to $@.tmp --expr 'import doc/manual/generate-xp-features.nix (builtins.fromJSON (builtins.readFile $<))'
@mv $@.tmp $@
$(d)/src/command-ref/experimental-features-shortlist.md: $(d)/xp-features.json $(d)/utils.nix $(d)/generate-xp-features-shortlist.nix $(doc_nix)
@rm -rf $@ $@.tmp
$(trace-gen) $(nix-eval) --write-to $@.tmp --expr 'import doc/manual/generate-xp-features-shortlist.nix (builtins.fromJSON (builtins.readFile $<))'
@mv $@.tmp $@
$(d)/xp-features.json: $(doc_nix)
$(trace-gen) $(dummy-env) NIX_PATH=nix/corepkgs=corepkgs $(doc_nix) __dump-xp-features > $@.tmp
@mv $@.tmp $@
$(d)/src/language/builtins.md: $(d)/language.json $(d)/generate-builtins.nix $(d)/src/language/builtins-prefix.md $(doc_nix)
@cat doc/manual/src/language/builtins-prefix.md > $@.tmp
$(trace-gen) $(nix-eval) --expr 'import doc/manual/generate-builtins.nix (builtins.fromJSON (builtins.readFile $<)).builtins' >> $@.tmp;
@cat doc/manual/src/language/builtins-suffix.md >> $@.tmp
@mv $@.tmp $@
$(d)/src/language/builtin-constants.md: $(d)/language.json $(d)/generate-builtin-constants.nix $(d)/src/language/builtin-constants-prefix.md $(doc_nix)
@cat doc/manual/src/language/builtin-constants-prefix.md > $@.tmp
$(trace-gen) $(nix-eval) --expr 'import doc/manual/generate-builtin-constants.nix (builtins.fromJSON (builtins.readFile $<)).constants' >> $@.tmp;
@cat doc/manual/src/language/builtin-constants-suffix.md >> $@.tmp
@mv $@.tmp $@
$(d)/language.json: $(doc_nix)
$(trace-gen) $(dummy-env) NIX_PATH=nix/corepkgs=corepkgs $(doc_nix) __dump-language > $@.tmp
@mv $@.tmp $@
# Generate "Upcoming release" notes (or clear it and remove from menu)
$(d)/src/release-notes/rl-next.md: $(d)/rl-next $(d)/rl-next/*
@if type -p changelog-d > /dev/null; then \
echo " GEN " $@; \
changelog-d doc/manual/rl-next > $@; \
else \
echo " NULL " $@; \
true > $@; \
fi
$(d)/src/SUMMARY-rl-next.md: $(d)/src/release-notes/rl-next.md
$(trace-gen) true
@if [ -s $< ]; then \
echo ' - [Upcoming release](release-notes/rl-next.md)' > $@; \
else \
true > $@; \
fi
# Generate the HTML manual.
.PHONY: manual-html
manual-html: $(docdir)/manual/index.html
install: $(docdir)/manual/index.html
# Generate 'nix' manpages.
install: $(mandir)/man1/nix3-manpages
man: doc/manual/generated/man1/nix3-manpages
all: doc/manual/generated/man1/nix3-manpages
# FIXME: unify with how the other man pages are generated.
# this one works differently and does not use any of the amenities provided by `/mk/lib.mk`.
$(mandir)/man1/nix3-manpages: doc/manual/generated/man1/nix3-manpages
@mkdir -p $(DESTDIR)$$(dirname $@)
$(trace-install) install -m 0644 $$(dirname $<)/* $(DESTDIR)$$(dirname $@)
doc/manual/generated/man1/nix3-manpages: $(d)/src/command-ref/new-cli
@mkdir -p $(DESTDIR)$$(dirname $@)
$(trace-gen) for i in doc/manual/src/command-ref/new-cli/*.md; do \
name=$$(basename $$i .md); \
tmpFile=$$(mktemp); \
if [[ $$name = SUMMARY ]]; then continue; fi; \
printf "Title: %s\n\n" "$$name" > $$tmpFile; \
cat $$i >> $$tmpFile; \
lowdown -sT man --nroff-nolinks -M section=1 $$tmpFile -o $(DESTDIR)$$(dirname $@)/$$name.1; \
rm $$tmpFile; \
done
@touch $@
$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/anchors.jq $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/contributing/experimental-feature-descriptions.md $(d)/src/command-ref/conf-file.md $(d)/src/language/builtins.md $(d)/src/language/builtin-constants.md $(d)/src/release-notes/rl-next.md
$(trace-gen) \
tmp="$$(mktemp -d)"; \
cp -r doc/manual "$$tmp"; \
find "$$tmp" -name '*.md' | while read -r file; do \
$(call process-includes,$$file,$$file); \
done; \
find "$$tmp" -name '*.md' | while read -r file; do \
docroot="$$(realpath --relative-to="$$(dirname "$$file")" $$tmp/manual/src)"; \
sed -i "s,@docroot@,$$docroot,g" "$$file"; \
done; \
set -euo pipefail; \
RUST_LOG=warn mdbook build "$$tmp/manual" -d $(DESTDIR)$(docdir)/manual.tmp 2>&1 \
| { grep -Fv "because fragment resolution isn't implemented" || :; }; \
rm -rf "$$tmp/manual"
@rm -rf $(DESTDIR)$(docdir)/manual
@mv $(DESTDIR)$(docdir)/manual.tmp/html $(DESTDIR)$(docdir)/manual
@rm -rf $(DESTDIR)$(docdir)/manual.tmp
endif

362
doc/manual/meson.build Normal file
View file

@ -0,0 +1,362 @@
nix_env_for_docs = {
'HOME': '/dummy',
'NIX_CONF_DIR': '/dummy',
'NIX_SSL_CERT_FILE': '/dummy/no-ca-bundle.crt',
'NIX_STATE_DIR': '/dummy',
'NIX_CONFIG': 'cores = 0',
}
nix_for_docs = [ nix, '--experimental-features', 'nix-command' ]
nix_eval_for_docs_common = nix_for_docs + [
'eval',
'-I', 'nix/corepkgs=corepkgs',
'--store', 'dummy://',
'--impure',
]
nix_eval_for_docs = nix_eval_for_docs_common + '--raw'
conf_file_json = custom_target(
command : nix_for_docs + [ 'show-config', '--json' ],
capture : true,
output : 'conf-file.json',
env : nix_env_for_docs,
# FIXME: put the actual lib targets in here? meson have introspection challenge 2024 though.
build_always_stale : true,
)
nix_conf_file_md_body = custom_target(
command : nix_eval_for_docs + [
'--expr',
'(import @INPUT0@).showSettings { inlineHTML = true; } (builtins.fromJSON (builtins.readFile @INPUT1@))',
],
capture : true,
input : [
'utils.nix',
conf_file_json,
],
output : 'conf-file.md.body',
env : nix_env_for_docs,
)
nix_conf_file_md = custom_target(
command : [ 'cat', '@INPUT@' ],
capture : true,
input : [
'src/command-ref/conf-file.md',
nix_conf_file_md_body,
],
output : 'conf-file.md',
)
nix_exp_features_json = custom_target(
command : [ nix, '__dump-xp-features' ],
capture : true,
output : 'xp-features.json',
# FIXME: put the actual lib targets in here? meson have introspection challenge 2024 though.
build_always_stale : true,
)
language_json = custom_target(
command: [nix, '__dump-language'],
output : 'language.json',
capture : true,
env : nix_env_for_docs,
# FIXME: put the actual lib targets in here? meson have introspection challenge 2024 though.
build_always_stale : true,
)
nix3_cli_json = custom_target(
command : [ nix, '__dump-cli' ],
capture : true,
output : 'nix.json',
env : nix_env_for_docs,
# FIXME: put the actual lib targets in here? meson have introspection challenge 2024 though.
build_always_stale : true,
)
generate_manual_deps = files(
'generate-deps.py',
)
# Generates builtins.md and builtin-constants.md.
subdir('src/language')
# Generates new-cli pages, {experimental,deprecated}-features-shortlist.md, and conf-file.md.
subdir('src/command-ref')
# Generates {experimental,deprecated}-feature-descriptions.md.
subdir('src/contributing')
# Generates rl-next-generated.md.
subdir('src/release-notes')
manual = custom_target(
'manual',
command : [
bash,
'-euo', 'pipefail',
'-c',
'''
@0@ @INPUT0@ @CURRENT_SOURCE_DIR@ > @DEPFILE@
cd @SOURCE_ROOT@
@1@ build doc/manual -d @2@ | { grep -Fv "because fragment resolution isn't implemented" || :; }
rm -rf @2@/manual
mv @2@/html @2@/manual
find @2@/manual -iname meson.build -delete
'''.format(
python.full_path(),
mdbook.full_path(),
meson.current_build_dir(),
),
],
input : [
generate_manual_deps,
'book.toml',
'anchors.jq',
'custom.css',
nix3_cli_files,
experimental_features_shortlist_md,
experimental_feature_descriptions_md,
deprecated_features_shortlist_md,
deprecated_feature_descriptions_md,
conf_file_md,
builtins_md,
builtin_constants_md,
rl_next_generated,
nix,
],
output : [
'manual',
'markdown',
],
install : true,
install_dir : [
datadir / 'doc/nix',
false,
],
depfile : 'manual.d',
env : {
'RUST_LOG': 'info',
'MDBOOK_SUBSTITUTE_SEARCH': meson.current_build_dir() / 'src',
},
)
manual_md = manual[1]
nix_nested_manpages = [
[ 'nix-env',
[
'delete-generations',
'install',
'list-generations',
'query',
'rollback',
'set-flag',
'set',
'switch-generation',
'switch-profile',
'uninstall',
'upgrade',
],
],
[ 'nix-store',
[
'add-fixed',
'add',
'delete',
'dump-db',
'dump',
'export',
'gc',
'generate-binary-cache-key',
'import',
'load-db',
'optimise',
'print-env',
'query',
'read-log',
'realise',
'repair-path',
'restore',
'serve',
'verify',
'verify-path',
],
],
]
foreach command : nix_nested_manpages
foreach page : command[1]
title = command[0] + ' --' + page
section = '1'
custom_target(
command : [
'./render-manpage.sh',
'--out-no-smarty',
title,
section,
'@INPUT0@/command-ref' / command[0] / (page + '.md'),
'@OUTPUT0@',
],
input : [
manual_md,
nix,
],
output : command[0] + '-' + page + '.1',
install : true,
install_dir : mandir / 'man1',
)
endforeach
endforeach
nix3_manpages = [
'nix3-build',
'nix3-bundle',
'nix3-config',
'nix3-config-show',
'nix3-copy',
'nix3-daemon',
'nix3-derivation-add',
'nix3-derivation',
'nix3-derivation-show',
'nix3-develop',
'nix3-doctor',
'nix3-edit',
'nix3-eval',
'nix3-flake-archive',
'nix3-flake-check',
'nix3-flake-clone',
'nix3-flake-info',
'nix3-flake-init',
'nix3-flake-lock',
'nix3-flake',
'nix3-flake-metadata',
'nix3-flake-new',
'nix3-flake-prefetch',
'nix3-flake-show',
'nix3-flake-update',
'nix3-fmt',
'nix3-hash-file',
'nix3-hash',
'nix3-hash-path',
'nix3-hash-to-base16',
'nix3-hash-to-base32',
'nix3-hash-to-base64',
'nix3-hash-to-sri',
'nix3-help',
'nix3-help-stores',
'nix3-key-convert-secret-to-public',
'nix3-key-generate-secret',
'nix3-key',
'nix3-log',
'nix3-nar-cat',
'nix3-nar-dump-path',
'nix3-nar-ls',
'nix3-nar',
'nix3-path-info',
'nix3-print-dev-env',
'nix3-profile-diff-closures',
'nix3-profile-history',
'nix3-profile-install',
'nix3-profile-list',
'nix3-profile',
'nix3-profile-remove',
'nix3-profile-rollback',
'nix3-profile-upgrade',
'nix3-profile-wipe-history',
'nix3-realisation-info',
'nix3-realisation',
'nix3-registry-add',
'nix3-registry-list',
'nix3-registry',
'nix3-registry-pin',
'nix3-registry-remove',
'nix3-repl',
'nix3-run',
'nix3-search',
'nix3-shell',
'nix3-store-add-file',
'nix3-store-add-path',
'nix3-store-cat',
'nix3-store-copy-log',
'nix3-store-copy-sigs',
'nix3-store-delete',
'nix3-store-diff-closures',
'nix3-store-dump-path',
'nix3-store-gc',
'nix3-store-ls',
'nix3-store-make-content-addressed',
'nix3-store',
'nix3-store-optimise',
'nix3-store-path-from-hash-part',
'nix3-store-ping',
'nix3-store-prefetch-file',
'nix3-store-repair',
'nix3-store-sign',
'nix3-store-verify',
'nix3-upgrade-nix',
'nix3-why-depends',
'nix',
]
foreach page : nix3_manpages
section = '1'
custom_target(
command : [
bash,
'@INPUT0@',
page,
section,
'@INPUT1@/command-ref/new-cli/@0@.md'.format(page),
'@OUTPUT@',
],
input : [
'render-manpage.sh',
manual_md,
nix,
],
output : page + '.1',
install : true,
install_dir : mandir / 'man1',
)
endforeach
nix_manpages = [
[ 'nix-env', 1 ],
[ 'nix-store', 1 ],
[ 'nix-build', 1 ],
[ 'nix-shell', 1 ],
[ 'nix-instantiate', 1 ],
[ 'nix-collect-garbage', 1 ],
[ 'nix-prefetch-url', 1 ],
[ 'nix-channel', 1 ],
[ 'nix-hash', 1 ],
[ 'nix-copy-closure', 1 ],
[ 'nix.conf', 5, 'conf-file.md' ],
[ 'nix-daemon', 8 ],
[ 'nix-profiles', 5, 'files/profiles.md' ],
]
foreach entry : nix_manpages
title = entry[0]
# nix.conf.5 and nix-profiles.5 are based off of conf-file.md and files/profiles.md,
# rather than a stem identical to its mdbook source.
# Therefore we use an optional third element of this array to override the name pattern
md_file = entry.get(2, title + '.md')
section = entry[1].to_string()
custom_target(
command : [
bash,
'@INPUT0@',
title,
section,
'@INPUT1@/command-ref/@0@'.format(md_file),
'@OUTPUT@',
],
input : [
'render-manpage.sh',
manual_md,
entry.get(3, []),
nix,
],
output : '@0@.@1@'.format(entry[0], entry[1]),
install : true,
install_dir : mandir / 'man@0@'.format(entry[1]),
)
endforeach

32
doc/manual/process-includes.sh Executable file
View file

@ -0,0 +1,32 @@
#!/usr/bin/env bash
set -euo pipefail
infile="$1"
outfile="$2"
shift 2
# set a search path for includes. the old makefile-based system splorked
# everything into the source tree and was thus able to not have a search
# path, but the meson system generates intermediate files into dedicated
# directories separate from the source. we still retain the implicit old
# behavior for now as the base search path, once meson is the default we
# can revisit this and remove the implicit search path entry. it's fine.
set -- "$(dirname "$infile")" "$@"
# re-implement mdBook's include directive to make it usable for terminal output and for proper @docroot@ substitution
(grep '{{#include' "$infile" || true) | while read -r line; do
found=false
include="$(printf "$line" | sed 's/{{#include \(.*\)}}/\1/')"
for path in "$@"; do
filename="$path/$include"
if [ -e "$filename" ]; then
found=true
matchline="$(printf "$line" | sed 's|/|\\/|g')"
sed -i "/$matchline/r $filename" "$outfile"
sed -i "s/$matchline//" "$outfile"
break
fi
done
$found || ( echo "#include-d file '$filename' does not exist." >&2; exit 1; )
done

8
doc/manual/rclone.conf Normal file
View file

@ -0,0 +1,8 @@
[lix-docs]
type = s3
provider = Other
env_auth = true
endpoint = https://s3.lix.systems
location_constraint = garage
region = garage
acl = private

View file

@ -24,7 +24,6 @@ const redirects = {
"chap-writing-nix-expressions": "language/index.html",
"part-command-ref": "command-ref/command-ref.html",
"conf-allow-import-from-derivation": "command-ref/conf-file.html#conf-allow-import-from-derivation",
"conf-allow-new-privileges": "command-ref/conf-file.html#conf-allow-new-privileges",
"conf-allowed-uris": "command-ref/conf-file.html#conf-allowed-uris",
"conf-allowed-users": "command-ref/conf-file.html#conf-allowed-users",
"conf-auto-optimise-store": "command-ref/conf-file.html#conf-auto-optimise-store",
@ -346,7 +345,7 @@ const redirects = {
"linux": "uninstall.html#linux",
"macos": "uninstall.html#macos",
"uninstalling": "uninstall.html",
}
},
"contributing/hacking.html": {
"nix-with-flakes": "#building-nix-with-flakes",
"classic-nix": "#building-nix",

25
doc/manual/render-manpage.sh Executable file
View file

@ -0,0 +1,25 @@
#!/usr/bin/env bash
set -euo pipefail
lowdown_args=
if [ "$1" = --out-no-smarty ]; then
lowdown_args=--out-no-smarty
shift
fi
[ "$#" = 4 ] || {
echo "wrong number of args passed" >&2
exit 1
}
title="$1"
section="$2"
infile="$3"
outfile="$4"
(
printf "Title: %s\n\n" "$title"
cat "$infile"
) | lowdown -sT man --nroff-nolinks $lowdown_args -M section="$section" -o "$outfile"

View file

View file

@ -0,0 +1,21 @@
---
synopsis: "Build failures caused by `allowSubstitutes = false` while being the wrong system now produce a decent error"
issues: [fj#484]
cls: [1841]
category: Fixes
credits: jade
---
Nix allows derivations to set `allowSubstitutes = false` in order to force them to be built locally without querying substituters for them.
This is useful for derivations that are very fast to build (especially if they produce large output).
However, this can shoot you in the foot if the derivation *has* to be substituted such as if the derivation is for another architecture, which is what `--always-allow-substitutes` is for.
Perhaps such derivations that are known to be impossible to build locally should ignore `allowSubstitutes` (irrespective of remote builders) in the future, but this at least reports the failure and solution directly.
```
$ nix build -f fail.nix
error: a 'unicornsandrainbows-linux' with features {} is required to build '/nix/store/...-meow.drv', but I am a 'x86_64-linux' with features {...}
Hint: the failing derivation has allowSubstitutes set to false, forcing it to be built rather than substituted.
Passing --always-allow-substitutes to force substitution may resolve this failure if the path is available in a substituter.
```

View file

@ -0,0 +1,10 @@
---
synopsis: "`Alt+Left` and `Alt+Right` go back/forwards by words in `nix repl`"
issues: [fj#501]
cls: [1883]
category: Fixes
credits: 9999years
---
`nix repl` now recognizes `Alt+Left` and `Alt+Right` for navigating by words
when entering input in `nix repl` on more terminals/platforms.

View file

@ -1,40 +0,0 @@
---
synopsis: Concise error printing in `nix repl`
prs: 9928
---
Previously, if an element of a list or attribute set threw an error while
evaluating, `nix repl` would print the entire error (including source location
information) inline. This output was clumsy and difficult to parse:
```
nix-repl> { err = builtins.throw "uh oh!"; }
{ err = «error:
… while calling the 'throw' builtin
at «string»:1:9:
1| { err = builtins.throw "uh oh!"; }
| ^
error: uh oh!»; }
```
Now, only the error message is displayed, making the output much more readable.
```
nix-repl> { err = builtins.throw "uh oh!"; }
{ err = «error: uh oh!»; }
```
However, if the whole expression being evaluated throws an error, source
locations and (if applicable) a stack trace are printed, just like you'd expect:
```
nix-repl> builtins.throw "uh oh!"
error:
… while calling the 'throw' builtin
at «string»:1:1:
1| builtins.throw "uh oh!"
| ^
error: uh oh!
```

View file

@ -1,2 +0,0 @@
organization: NixOS
repository: nix

View file

@ -0,0 +1,15 @@
---
synopsis: "Drop support for `xz` and `bzip2` Content-Encoding"
category: Miscellany
cls: [2134]
credits: horrors
---
Lix no longer supports the non-standard HTTP Content-Encoding values `xz` and `bzip2`.
We do not expect this to cause any problems in practice since these encodings *aren't*
standard, and any server delivering them anyway without being asked to is already well
and truly set on the path of causing inexplicable client breakages.
Lix's ability to decompress files compressed with `xz` or `bzip2` is unaffected. We're
only bringing Lix more in line with the HTTP standard; all post-transfer data handling
remains as it was before.

View file

@ -0,0 +1,13 @@
---
synopsis: Ctrl-C stops Nix commands much more reliably and responsively
issues: [7245, fj#393]
cls: [2016]
prs: [11618]
category: Fixes
credits: [roberth, 9999years]
---
CTRL-C will now stop Nix commands much more reliably and responsively. While
there are still some cases where a Nix command can be slow or unresponsive
following a `SIGINT` (please report these as issues!), the vast majority of
signals will now cause the Nix command to quit quickly and consistently.

View file

@ -1,9 +0,0 @@
---
synopsis: "`--debugger` can now access bindings from `let` expressions"
prs: 9918
issues: 8827.
---
Breakpoints and errors in the bindings of a `let` expression can now access
those bindings in the debugger. Previously, only the body of `let` expressions
could access those bindings.

View file

@ -1,9 +0,0 @@
---
synopsis: Enter the `--debugger` when `builtins.trace` is called if `debugger-on-trace` is set
prs: 9914
---
If the `debugger-on-trace` option is set and `--debugger` is given,
`builtins.trace` calls will behave similarly to `builtins.break` and will enter
the debug REPL. This is useful for determining where warnings are being emitted
from.

View file

@ -0,0 +1,17 @@
---
synopsis: Deprecated language features
issues: [fj#437]
cls: [1785, 1736, 1735, 1744]
category: Breaking Changes
credits: [piegames, horrors]
---
A system for deprecation (and then the planned removal) of undesired language features has been put into place.
It is controlled via feature flags much like experimental features, except that the deprecations are enabled default,
and can be disabled via the flags for backwards compatibility (opt-out with `--extra-deprecated-features` or the Nix configuration file).
- `url-literals`: **URL literals** have long been obsolete and discouraged of use, and now they are officially deprecated.
This means that all URLs must be properly put within quotes like all other strings.
- `rec-set-overrides`: **__overrides** is an old arcane syntax which has not been in use for more than a decade.
It is soft-deprecated with a warning only, with the plan to turn that into an error in a future release.
- `ancient-let`: **The old `let` syntax** (`let { body = …; … }`) is soft-deprecated with a warning as well. Use the regular `let … in` instead.

View file

@ -0,0 +1,10 @@
---
synopsis: "transfers no longer allow arbitrary url schemas"
category: Breaking Changes
cls: [2106]
credits: horrors
---
Lix no longer allows transfers using arbitrary url schemas. Only `http://`, `https://`, `ftp://`, `ftps://`, and `file://` urls are supported going forward. This affects `builtins.fetchurl`, `<nix/fetchurl.nix>`, transfers to and from binary caches, and all other uses of the internal file transfer code. Flake inputs using multi-protocol schemas (e.g. `git+ssh`) are not affected as those use external utilities to transfer data.
The `s3://` scheme is not affected at all by this change and continues to work if S3 support is built into Lix.

View file

@ -1,6 +0,0 @@
---
synopsis: Fix handling of truncated `.drv` files.
prs: 9673
---
Previously a `.drv` that was truncated in the middle of a string would case nix to enter an infinite loop, eventually exhausting all memory and crashing.

View file

@ -1,22 +0,0 @@
---
synopsis: Duplicate attribute reports are more accurate
# prs: cl 557
---
Duplicate attribute errors are now more accurate, showing the path at which an error was detected rather than the full, possibly longer, path that caused the error.
Error reports are now
```ShellSession
$ nix eval --expr '{ a.b = 1; a.b.c.d = 1; }'
error: attribute 'a.b' already defined at «string»:1:3
at «string»:1:12:
1| { a.b = 1; a.b.c.d = 1;
| ^
```
instead of
```ShellSession
$ nix eval --expr '{ a.b = 1; a.b.c.d = 1; }'
error: attribute 'a.b.c.d' already defined at «string»:1:3
at «string»:1:12:
1| { a.b = 1; a.b.c.d = 1;
| ^
```

View file

@ -1,8 +0,0 @@
synopsis: Disallow empty search regex in `nix search`
prs: #9481
description: {
[`nix search`](@docroot@/command-ref/new-cli/nix3-search.md) now requires a search regex to be passed. To show all packages, use `^`.
}

View file

@ -1,25 +0,0 @@
---
synopsis: The `--debugger` will start more reliably in `let` expressions and function calls
prs: 9917
issues: 6649
---
Previously, if you attempted to evaluate this file with the debugger:
```nix
let
a = builtins.trace "before inner break" (
builtins.break "hello"
);
b = builtins.trace "before outer break" (
builtins.break a
);
in
b
```
Nix would correctly enter the debugger at `builtins.break a`, but if you asked
it to `:continue`, it would skip over the `builtins.break "hello"` expression
entirely.
Now, Nix will correctly enter the debugger at both breakpoints.

View file

@ -1,7 +0,0 @@
---
synopsis: Reduce eval memory usage and wall time
prs: 9658
---
Reduce the size of the `Env` struct used in the evaluator by a pointer, or 8 bytes on most modern machines.
This reduces memory usage during eval by around 2% and wall time by around 3%.

View file

@ -1,12 +0,0 @@
---
synopsis: Add new `eval-system` setting
prs: 4093
---
Add a new `eval-system` option.
Unlike `system`, it just overrides the value of `builtins.currentSystem`.
This is more useful than overriding `system`, because you can build these derivations on remote builders which can work on the given system.
In contrast, `system` also effects scheduling which will cause Nix to build those derivations locally even if that doesn't make sense.
`eval-system` only takes effect if it is non-empty.
If empty (the default) `system` is used as before, so there is no breakage.

View file

@ -0,0 +1,23 @@
---
synopsis: restore backwards-compatibility of `builtins.fetchGit` with Nix 2.3
issues: [5291, 5128]
credits: [ma27]
category: Fixes
---
Compatibility with `builtins.fetchGit` from Nix 2.3 has been restored as follows:
* Until now, each `ref` was prefixed with `refs/heads` unless it starts with `refs/` itself.
Now, this is not done if the `ref` looks like a commit hash.
* Specifying `builtins.fetchGit { ref = "a-tag"; /* … */ }` was broken because `refs/heads` was appended.
Now, the fetcher doesn't turn a ref into `refs/heads/ref`, but into `refs/*/ref`. That way,
the value in `ref` can be either a tag or a branch.
* The ref resolution happens the same way as in git:
* If `refs/ref` exists, it's used.
* If a tag `refs/tags/ref` exists, it's used.
* If a branch `refs/heads/ref` exists, it's used.

View file

@ -1,7 +0,0 @@
---
synopsis: consistent order of lambda formals in printed expressions
prs: 9874
---
Always print lambda formals in lexicographic order rather than the internal, creation-time based symbol order.
This makes printed formals independent of the context they appear in.

View file

@ -0,0 +1,10 @@
---
synopsis: HTTP proxy environment variables are now respected for S3 binary cache stores
issues: [fj#433]
cls: [1788]
category: Fixes
credits: jade
---
Due to "legacy reasons" (according to the AWS C++ SDK docs), the AWS SDK ignores system proxy configuration by default.
We turned it back on.

View file

@ -1,6 +0,0 @@
---
synopsis: fix duplicate attribute error positions for `inherit`
prs: 9874
---
When an inherit caused a duplicate attribute error the position of the error was not reported correctly, placing the error with the inherit itself or at the start of the bindings block instead of the offending attribute name.

View file

@ -1,7 +0,0 @@
---
synopsis: "`inherit (x) ...` evaluates `x` only once"
prs: 9847
---
`inherit (x) a b ...` now evaluates the expression `x` only once for all inherited attributes rather than once for each inherited attribute.
This does not usually have a measurable impact, but side-effects (such as `builtins.trace`) would be duplicated and expensive expressions (such as derivations) could cause a measurable slowdown.

View file

@ -1,6 +0,0 @@
---
synopsis: Fix `nix-env --query --drv-path --json`
prs: 9257
---
Fixed a bug where `nix-env --query` ignored `--drv-path` when `--json` was set.

View file

@ -1,33 +0,0 @@
---
synopsis: "`nix flake check` logs the checks"
issues: 8882
prs: 8893
---
`nix flake check` now logs the checks it runs and the derivations it evaluates:
```
$ nix flake check -v
evaluating flake...
checking flake output 'checks'...
checking derivation 'checks.aarch64-darwin.ghciwatch-tests'...
derivation evaluated to /nix/store/nh7dlvsrhds4cxl91mvgj4h5cbq6skmq-ghciwatch-test-0.3.0.drv
checking derivation 'checks.aarch64-darwin.ghciwatch-clippy'...
derivation evaluated to /nix/store/9cb5a6wmp6kf6hidqw9wphidvb8bshym-ghciwatch-clippy-0.3.0.drv
checking derivation 'checks.aarch64-darwin.ghciwatch-doc'...
derivation evaluated to /nix/store/8brdd3jbawfszpbs7vdpsrhy80as1il8-ghciwatch-doc-0.3.0.drv
checking derivation 'checks.aarch64-darwin.ghciwatch-fmt'...
derivation evaluated to /nix/store/wjhs0l1njl5pyji53xlmfjrlya0wmz8p-ghciwatch-fmt-0.3.0.drv
checking derivation 'checks.aarch64-darwin.ghciwatch-audit'...
derivation evaluated to /nix/store/z0mps8dyj2ds7c0fn0819y5h5611033z-ghciwatch-audit-0.3.0.drv
checking flake output 'packages'...
checking derivation 'packages.aarch64-darwin.default'...
derivation evaluated to /nix/store/41abbdyglw5x9vcsvd89xan3ydjf8d7r-ghciwatch-0.3.0.drv
checking flake output 'apps'...
checking flake output 'devShells'...
checking derivation 'devShells.aarch64-darwin.default'...
derivation evaluated to /nix/store/bc935gz7dylzmcpdb5cczr8gngv8pmdb-nix-shell.drv
running 5 flake checks...
warning: The check omitted these incompatible systems: aarch64-linux, x86_64-darwin, x86_64-linux
Use '--all-systems' to check all.
```

View file

@ -0,0 +1,38 @@
---
synopsis: Removing the `.` default argument passed to the `nix fmt` formatter
issues: []
prs: [11438]
cls: [1902]
category: Breaking Changes
credits: zimbatm
---
The underlying formatter no longer receives the ". " default argument when `nix fmt` is called with no arguments.
This change was necessary as the formatter wasn't able to distinguish between
a user wanting to format the current folder with `nix fmt .` or the generic
`nix fmt`.
The default behaviour is now the responsibility of the formatter itself, and
allows tools such as treefmt to format the whole tree instead of only the
current directory and below.
This may cause issues with some formatters: nixfmt, nixpkgs-fmt and alejandra currently format stdin when no arguments are passed.
Here is a small wrapper example that will restore the previous behaviour for such a formatter:
```nix
{
outputs = { self, nixpkgs, systems }:
let
eachSystem = nixpkgs.lib.genAttrs (import systems) (system: nixpkgs.legacyPackages.${system});
in
{
formatter = eachSystem (pkgs:
pkgs.writeShellScriptBin "formatter" ''
if [[ $# = 0 ]]; set -- .; fi
exec "${pkgs.nixfmt-rfc-style}/bin/nixfmt "$@"
'');
};
}
```

View file

@ -1,24 +0,0 @@
---
synopsis: Coercion errors include the failing value
issues: #561
prs: #9754
---
The `error: cannot coerce a <TYPE> to a string` message now includes the value
which caused the error.
Before:
```
error: cannot coerce a set to a string
```
After:
```
error: cannot coerce a set to a string: { aesSupport = «thunk»;
avx2Support = «thunk»; avx512Support = «thunk»; avxSupport = «thunk»;
canExecute = «thunk»; config = «thunk»; darwinArch = «thunk»; darwinMinVersion
= «thunk»; darwinMinVersionVariable = «thunk»; darwinPlatform = «thunk»; «84
attributes elided»}
```

View file

@ -1,23 +0,0 @@
---
synopsis: Type errors include the failing value
issues: #561
prs: #9753
---
In errors like `value is an integer while a list was expected`, the message now
includes the failing value.
Before:
```
error: value is a set while a string was expected
```
After:
```
error: expected a string but found a set: { ghc810 = «thunk»;
ghc8102Binary = «thunk»; ghc8107 = «thunk»; ghc8107Binary = «thunk»;
ghc865Binary = «thunk»; ghc90 = «thunk»; ghc902 = «thunk»; ghc92 = «thunk»;
ghc924Binary = «thunk»; ghc925 = «thunk»; «17 attributes elided»}
```

View file

@ -0,0 +1,10 @@
---
synopsis: "The beginnings of a new pytest-based functional test suite"
category: Development
cls: [2036, 2037]
credits: jade
---
The existing integration/functional test suite is based on a large volume of shell scripts.
This often makes it somewhat challenging to debug at the best of times.
The goal of the pytest test suite is to make tests have more obvious dependencies on files and to make tests more concise and easier to write, as well as making new testing methods like snapshot testing easy.

View file

@ -0,0 +1,17 @@
---
synopsis: readline support removed
cls: [1885]
category: Packaging
credits: [9999years]
---
Support for building Lix with [`readline`][readline] instead of
[`editline`][editline] has been removed. `readline` support hasn't worked for a
long time (attempting to use it would lead to build errors) and would make Lix
subject to the GPL if it did work. In the future, we're hoping to replace
`editline` with [`rustyline`][rustyline] for improved ergonomics in the `nix
repl`.
[readline]: https://en.wikipedia.org/wiki/GNU_Readline
[editline]: https://github.com/troglobit/editline
[rustyline]: https://github.com/kkawakam/rustyline

View file

@ -1,37 +0,0 @@
---
synopsis: "Visual clutter in `--debugger` is reduced"
prs: 9919
---
Before:
```
info: breakpoint reached
Starting REPL to allow you to inspect the current state of the evaluator.
Welcome to Nix 2.20.0pre20231222_dirty. Type :? for help.
nix-repl> :continue
error: uh oh
Starting REPL to allow you to inspect the current state of the evaluator.
Welcome to Nix 2.20.0pre20231222_dirty. Type :? for help.
nix-repl>
```
After:
```
info: breakpoint reached
Nix 2.20.0pre20231222_dirty debugger
Type :? for help.
nix-repl> :continue
error: uh oh
nix-repl>
```

View file

@ -0,0 +1,30 @@
---
synopsis: Relative and tilde paths in configuration
issues: [fj#482]
cls: [1851, 1863, 1864]
category: Features
credits: [9999years]
---
[Configuration settings](@docroot@/command-ref/conf-file.md) can now refer to
files with paths relative to the file they're written in or relative to your
home directory (with `~/`).
This makes settings like
[`repl-overlays`](@docroot@/command-ref/conf-file.md#conf-repl-overlays) and
[`secret-key-files`](@docroot@/command-ref/conf-file.md#conf-repl-overlays)
much easier to set, especially if you'd like to refer to files in an existing
dotfiles repo cloned into your home directory.
If you put `repl-overlays = repl.nix` in your `~/.config/nix/nix.conf`, it'll
load `~/.config/nix/repl.nix`. Similarly, you can set `repl-overlays =
~/.dotfiles/repl.nix` to load a file relative to your home directory.
Configuration files can also
[`include`](@docroot@/command-ref/conf-file.md#file-format) paths relative to
your home directory.
Only user configuration files (like `$XDG_CONFIG_HOME/nix/nix.conf` or the
files listed in `$NIX_USER_CONF_FILES`) can use tilde paths relative to your
home directory. Configuration listed in the `$NIX_CONFIG` environment variable
may not use relative paths.

View file

@ -0,0 +1,8 @@
---
synopsis: "Dependency on monolithic coreutils removed"
category: Development
cls: [2108]
credits: vigress8
---
Previously, the build erroneously depended on a `coreutils` binary, which requires `coreutils` to be built with a specific configuration. This was only used in one test and was not required to be a single binary. This dependency is removed now.

View file

@ -0,0 +1,22 @@
---
synopsis: "Reproducibility check builds now report all differing outputs"
cls: [2069]
category: Improvements
credits: [lheckemann]
---
`nix-build --check` allows rerunning the build of an already-built derivation to check that it produces the same output again.
If a multiple-output derivation with impure behaviour is built with `--check`, only the first output would be shown in the resulting error message (and kept for comparison):
```
error: derivation '/nix/store/4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test
-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'
```
Now, all differing outputs are kept and reported:
```
error: derivation '4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: outputs differ
output differs: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'
output differs: output '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0' differs from '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0.check'
```

View file

@ -1,23 +0,0 @@
---
synopsis: "In the debugger, `while evaluating the attribute` errors now include position information"
prs: 9915
---
Before:
```
0: while evaluating the attribute 'python311.pythonForBuild.pkgs'
0x600001522598
```
After:
```
0: while evaluating the attribute 'python311.pythonForBuild.pkgs'
/nix/store/hg65h51xnp74ikahns9hyf3py5mlbbqq-source/overrides/default.nix:132:27
131|
132| bootstrappingBase = pkgs.${self.python.pythonAttr}.pythonForBuild.pkgs;
| ^
133| in
```

View file

@ -1,42 +0,0 @@
---
synopsis: Source locations are printed more consistently in errors
issues: 561
prs: 9555
---
Source location information is now included in error messages more
consistently. Given this code:
```nix
let
attr = {foo = "bar";};
key = {};
in
attr.${key}
```
Previously, Nix would show this unhelpful message when attempting to evaluate
it:
```
error:
… while evaluating an attribute name
error: value is a set while a string was expected
```
Now, the error message displays where the problematic value was found:
```
error:
… while evaluating an attribute name
at bad.nix:4:11:
3| key = {};
4| in attr.${key}
| ^
5|
error: expected a string but found a set: { }
```

View file

@ -1,32 +0,0 @@
---
synopsis: Some stack overflow segfaults are fixed
issues: 9616
prs: 9617
---
The number of nested function calls has been restricted, to detect and report
infinite function call recursions. The default maximum call depth is 10,000 and
can be set with [the `max-call-depth`
option](@docroot@/command-ref/conf-file.md#conf-max-call-depth).
This fixes segfaults or the following unhelpful error message in many cases:
error: stack overflow (possible infinite recursion)
Before:
```
$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
Segmentation fault: 11
```
After:
```
$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
error: stack overflow
at «string»:1:14:
1| (x: x x) (x: x x)
| ^
```

View file

@ -0,0 +1,26 @@
---
synopsis: "Some Lix crashes now produce reporting instructions and a stack trace, then abort"
cls: [1854]
category: Improvements
credits: jade
---
Lix, being a C++ program, can crash in a few kinds of ways.
It can obviously do a memory access violation, which will generate a core dump and thus be relatively debuggable.
But, worse, it could throw an unhandled exception, and, in the past, we would just show the message but not where it comes from, in spite of this always being a bug, since we expect all such errors to be translated to a Lix specific error.
Now the latter kind of bug should print reporting instructions, a rudimentary stack trace and (depending on system configuration) generate a core dump.
Sample output:
```
Lix crashed. This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included:
Exception: std::runtime_error: test exception
Stack trace:
0# nix::printStackTrace() in /home/jade/lix/lix3/build/src/nix/../libutil/liblixutil.so
1# 0x000073C9862331F2 in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
2# 0x000073C985F2E21A in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
3# 0x000073C985F2E285 in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
4# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
...
```

View file

@ -0,0 +1,10 @@
---
synopsis: "`<nix/fetchurl.nix>` now uses TLS verification"
category: Fixes
prs: [11585]
credits: edolstra
---
Previously `<nix/fetchurl.nix>` did not do TLS verification. This was because the Nix sandbox in the past did not have access to TLS certificates, and Nix checks the hash of the fetched file anyway. However, this can expose authentication data from `netrc` and URLs to man-in-the-middle attackers. In addition, Nix now in some cases (such as when using impure derivations) does *not* check the hash. Therefore we have now enabled TLS verification. This means that downloads by `<nix/fetchurl.nix>` will now fail if you're fetching from a HTTPS server that does not have a valid certificate.
`<nix/fetchurl.nix>` is also known as the builtin derivation builder `builtin:fetchurl`. It's not to be confused with the evaluation-time function `builtins.fetchurl`, which was not affected by this issue.

View file

@ -1,31 +0,0 @@
---
synopsis: Better error reporting for `with` expressions
prs: 9658
---
`with` expressions using non-attrset values to resolve variables are now reported with proper positions.
Previously an incorrect `with` expression would report no position at all, making it hard to determine where the error originated:
```
nix-repl> with 1; a
error:
<borked>
at «none»:0: (source not available)
error: value is an integer while a set was expected
```
Now position information is preserved and reported as with most other errors:
```
nix-repl> with 1; a
error:
… while evaluating the first subexpression of a with expression
at «string»:1:1:
1| with 1; a
| ^
error: expected a set but found an integer: 1
```

252
doc/manual/src/SUMMARY.md Normal file
View file

@ -0,0 +1,252 @@
# Table of Contents
- [Introduction](introduction.md)
- [Quick Start](quick-start.md)
- [Installation](installation/installation.md)
- [Supported Platforms](installation/supported-platforms.md)
- [Installing a Binary Distribution](installation/installing-binary.md)
- [Installing Lix from Source](installation/installing-source.md)
- [Prerequisites](installation/prerequisites-source.md)
- [Obtaining a Source Distribution](installation/obtaining-source.md)
- [Building Lix from Source](installation/building-source.md)
- [Using Lix within Docker](installation/installing-docker.md)
- [Security](installation/nix-security.md)
- [Single-User Mode](installation/single-user.md)
- [Multi-User Mode](installation/multi-user.md)
- [Environment Variables](installation/env-variables.md)
- [Upgrading Lix](installation/upgrading.md)
- [Uninstalling Lix](installation/uninstall.md)
- [Package Management](package-management/package-management.md)
- [Basic Package Management](package-management/basic-package-mgmt.md)
- [Profiles](package-management/profiles.md)
- [Garbage Collection](package-management/garbage-collection.md)
- [Garbage Collector Roots](package-management/garbage-collector-roots.md)
- [Sharing Packages Between Machines](package-management/sharing-packages.md)
- [Serving a Nix store via HTTP](package-management/binary-cache-substituter.md)
- [Copying Closures via SSH](package-management/copy-closure.md)
- [Serving a Nix store via SSH](package-management/ssh-substituter.md)
- [Serving a Nix store via S3](package-management/s3-substituter.md)
- [Nix Language](language/index.md)
- [Data Types](language/values.md)
- [Language Constructs](language/constructs.md)
- [String interpolation](language/string-interpolation.md)
- [Operators](language/operators.md)
- [Derivations](language/derivations.md)
- [Advanced Attributes](language/advanced-attributes.md)
- [Built-in Constants](language/builtin-constants.md)
- [Built-in Functions](language/builtins.md)
- [Advanced Topics](advanced-topics/advanced-topics.md)
- [Remote Builds](advanced-topics/distributed-builds.md)
- [Tuning Cores and Jobs](advanced-topics/cores-vs-jobs.md)
- [Verifying Build Reproducibility](advanced-topics/diff-hook.md)
- [Using the `post-build-hook`](advanced-topics/post-build-hook.md)
- [Command Reference](command-ref/command-ref.md)
- [Common Options](command-ref/opt-common.md)
- [Common Environment Variables](command-ref/env-common.md)
- [Main Commands](command-ref/main-commands.md)
- [nix-build](command-ref/nix-build.md)
- [nix-shell](command-ref/nix-shell.md)
- [nix-store](command-ref/nix-store.md)
- [nix-store --add-fixed](command-ref/nix-store/add-fixed.md)
- [nix-store --add](command-ref/nix-store/add.md)
- [nix-store --delete](command-ref/nix-store/delete.md)
- [nix-store --dump-db](command-ref/nix-store/dump-db.md)
- [nix-store --dump](command-ref/nix-store/dump.md)
- [nix-store --export](command-ref/nix-store/export.md)
- [nix-store --gc](command-ref/nix-store/gc.md)
- [nix-store --generate-binary-cache-key](command-ref/nix-store/generate-binary-cache-key.md)
- [nix-store --import](command-ref/nix-store/import.md)
- [nix-store --load-db](command-ref/nix-store/load-db.md)
- [nix-store --optimise](command-ref/nix-store/optimise.md)
- [nix-store --print-env](command-ref/nix-store/print-env.md)
- [nix-store --query](command-ref/nix-store/query.md)
- [nix-store --read-log](command-ref/nix-store/read-log.md)
- [nix-store --realise](command-ref/nix-store/realise.md)
- [nix-store --repair-path](command-ref/nix-store/repair-path.md)
- [nix-store --restore](command-ref/nix-store/restore.md)
- [nix-store --serve](command-ref/nix-store/serve.md)
- [nix-store --verify-path](command-ref/nix-store/verify-path.md)
- [nix-store --verify](command-ref/nix-store/verify.md)
- [nix-env](command-ref/nix-env.md)
- [nix-env --delete-generations](command-ref/nix-env/delete-generations.md)
- [nix-env --install](command-ref/nix-env/install.md)
- [nix-env --list-generations](command-ref/nix-env/list-generations.md)
- [nix-env --query](command-ref/nix-env/query.md)
- [nix-env --rollback](command-ref/nix-env/rollback.md)
- [nix-env --set-flag](command-ref/nix-env/set-flag.md)
- [nix-env --set](command-ref/nix-env/set.md)
- [nix-env --switch-generation](command-ref/nix-env/switch-generation.md)
- [nix-env --switch-profile](command-ref/nix-env/switch-profile.md)
- [nix-env --uninstall](command-ref/nix-env/uninstall.md)
- [nix-env --upgrade](command-ref/nix-env/upgrade.md)
- [Utilities](command-ref/utilities.md)
- [nix-channel](command-ref/nix-channel.md)
- [nix-collect-garbage](command-ref/nix-collect-garbage.md)
- [nix-copy-closure](command-ref/nix-copy-closure.md)
- [nix-daemon](command-ref/nix-daemon.md)
- [nix-hash](command-ref/nix-hash.md)
- [nix-instantiate](command-ref/nix-instantiate.md)
- [nix-prefetch-url](command-ref/nix-prefetch-url.md)
- [Experimental Commands](command-ref/experimental-commands.md)
- [nix](command-ref/new-cli/nix.md)
- [nix build](command-ref/new-cli/nix3-build.md)
- [nix bundle](command-ref/new-cli/nix3-bundle.md)
- [nix config](command-ref/new-cli/nix3-config.md)
- [nix config show](command-ref/new-cli/nix3-config-show.md)
- [nix copy](command-ref/new-cli/nix3-copy.md)
- [nix daemon](command-ref/new-cli/nix3-daemon.md)
- [nix derivation](command-ref/new-cli/nix3-derivation.md)
- [nix derivation add](command-ref/new-cli/nix3-derivation-add.md)
- [nix derivation show](command-ref/new-cli/nix3-derivation-show.md)
- [nix develop](command-ref/new-cli/nix3-develop.md)
- [nix doctor](command-ref/new-cli/nix3-doctor.md)
- [nix edit](command-ref/new-cli/nix3-edit.md)
- [nix eval](command-ref/new-cli/nix3-eval.md)
- [nix flake](command-ref/new-cli/nix3-flake.md)
- [nix flake archive](command-ref/new-cli/nix3-flake-archive.md)
- [nix flake check](command-ref/new-cli/nix3-flake-check.md)
- [nix flake clone](command-ref/new-cli/nix3-flake-clone.md)
- [nix flake info](command-ref/new-cli/nix3-flake-info.md)
- [nix flake init](command-ref/new-cli/nix3-flake-init.md)
- [nix flake lock](command-ref/new-cli/nix3-flake-lock.md)
- [nix flake metadata](command-ref/new-cli/nix3-flake-metadata.md)
- [nix flake new](command-ref/new-cli/nix3-flake-new.md)
- [nix flake prefetch](command-ref/new-cli/nix3-flake-prefetch.md)
- [nix flake show](command-ref/new-cli/nix3-flake-show.md)
- [nix flake update](command-ref/new-cli/nix3-flake-update.md)
- [nix fmt](command-ref/new-cli/nix3-fmt.md)
- [nix hash](command-ref/new-cli/nix3-hash.md)
- [nix hash file](command-ref/new-cli/nix3-hash-file.md)
- [nix hash path](command-ref/new-cli/nix3-hash-path.md)
- [nix hash to-base16](command-ref/new-cli/nix3-hash-to-base16.md)
- [nix hash to-base32](command-ref/new-cli/nix3-hash-to-base32.md)
- [nix hash to-base64](command-ref/new-cli/nix3-hash-to-base64.md)
- [nix hash to-sri](command-ref/new-cli/nix3-hash-to-sri.md)
- [nix help](command-ref/new-cli/nix3-help.md)
- [nix help-stores](command-ref/new-cli/nix3-help-stores.md)
- [nix key](command-ref/new-cli/nix3-key.md)
- [nix key convert-secret-to-public](command-ref/new-cli/nix3-key-convert-secret-to-public.md)
- [nix key generate-secret](command-ref/new-cli/nix3-key-generate-secret.md)
- [nix log](command-ref/new-cli/nix3-log.md)
- [nix nar](command-ref/new-cli/nix3-nar.md)
- [nix nar cat](command-ref/new-cli/nix3-nar-cat.md)
- [nix nar dump-path](command-ref/new-cli/nix3-nar-dump-path.md)
- [nix nar ls](command-ref/new-cli/nix3-nar-ls.md)
- [nix path-info](command-ref/new-cli/nix3-path-info.md)
- [nix print-dev-env](command-ref/new-cli/nix3-print-dev-env.md)
- [nix profile](command-ref/new-cli/nix3-profile.md)
- [nix profile diff-closures](command-ref/new-cli/nix3-profile-diff-closures.md)
- [nix profile history](command-ref/new-cli/nix3-profile-history.md)
- [nix profile install](command-ref/new-cli/nix3-profile-install.md)
- [nix profile list](command-ref/new-cli/nix3-profile-list.md)
- [nix profile remove](command-ref/new-cli/nix3-profile-remove.md)
- [nix profile rollback](command-ref/new-cli/nix3-profile-rollback.md)
- [nix profile upgrade](command-ref/new-cli/nix3-profile-upgrade.md)
- [nix profile wipe-history](command-ref/new-cli/nix3-profile-wipe-history.md)
- [nix realisation](command-ref/new-cli/nix3-realisation.md)
- [nix realisation info](command-ref/new-cli/nix3-realisation-info.md)
- [nix registry](command-ref/new-cli/nix3-registry.md)
- [nix registry add](command-ref/new-cli/nix3-registry-add.md)
- [nix registry list](command-ref/new-cli/nix3-registry-list.md)
- [nix registry pin](command-ref/new-cli/nix3-registry-pin.md)
- [nix registry remove](command-ref/new-cli/nix3-registry-remove.md)
- [nix repl](command-ref/new-cli/nix3-repl.md)
- [nix run](command-ref/new-cli/nix3-run.md)
- [nix search](command-ref/new-cli/nix3-search.md)
- [nix shell](command-ref/new-cli/nix3-shell.md)
- [nix store](command-ref/new-cli/nix3-store.md)
- [nix store add-file](command-ref/new-cli/nix3-store-add-file.md)
- [nix store add-path](command-ref/new-cli/nix3-store-add-path.md)
- [nix store cat](command-ref/new-cli/nix3-store-cat.md)
- [nix store copy-log](command-ref/new-cli/nix3-store-copy-log.md)
- [nix store copy-sigs](command-ref/new-cli/nix3-store-copy-sigs.md)
- [nix store delete](command-ref/new-cli/nix3-store-delete.md)
- [nix store diff-closures](command-ref/new-cli/nix3-store-diff-closures.md)
- [nix store dump-path](command-ref/new-cli/nix3-store-dump-path.md)
- [nix store gc](command-ref/new-cli/nix3-store-gc.md)
- [nix store ls](command-ref/new-cli/nix3-store-ls.md)
- [nix store make-content-addressed](command-ref/new-cli/nix3-store-make-content-addressed.md)
- [nix store optimise](command-ref/new-cli/nix3-store-optimise.md)
- [nix store path-from-hash-part](command-ref/new-cli/nix3-store-path-from-hash-part.md)
- [nix store ping](command-ref/new-cli/nix3-store-ping.md)
- [nix store prefetch-file](command-ref/new-cli/nix3-store-prefetch-file.md)
- [nix store repair](command-ref/new-cli/nix3-store-repair.md)
- [nix store sign](command-ref/new-cli/nix3-store-sign.md)
- [nix store verify](command-ref/new-cli/nix3-store-verify.md)
- [nix upgrade-nix](command-ref/new-cli/nix3-upgrade-nix.md)
- [nix why-depends](command-ref/new-cli/nix3-why-depends.md)
- [Files](command-ref/files.md)
- [nix.conf](command-ref/conf-file.md)
- [Profiles](command-ref/files/profiles.md)
- [manifest.nix](command-ref/files/manifest.nix.md)
- [manifest.json](command-ref/files/manifest.json.md)
- [Channels](command-ref/files/channels.md)
- [Default Nix expression](command-ref/files/default-nix-expression.md)
- [Architecture and Design](architecture/architecture.md)
- [File System Object](architecture/file-system-object.md)
- [Protocols](protocols/protocols.md)
- [Serving Tarball Flakes](protocols/tarball-fetcher.md)
- [Derivation "ATerm" file format](protocols/derivation-aterm.md)
- [Glossary](glossary.md)
- [Contributing](contributing/contributing.md)
- [Hacking](contributing/hacking.md)
- [Testing](contributing/testing.md)
- [Experimental Features](contributing/experimental-features.md)
- [Deprecated Features](contributing/deprecated-features.md)
- [CLI guideline](contributing/cli-guideline.md)
- [C++ style guide](contributing/cxx.md)
- [Release Notes](release-notes/release-notes.md)
- [Upcoming release](release-notes/rl-next.md)
<!-- RELENG-AUTO-INSERTION-MARKER (see releng/release_notes.py) -->
- [Lix 2.91 (2024-08-12)](release-notes/rl-2.91.md)
- [Lix 2.90 (2024-07-10)](release-notes/rl-2.90.md)
- [Nix 2.18 (2023-09-20)](release-notes/rl-2.18.md)
- [Nix 2.17 (2023-07-24)](release-notes/rl-2.17.md)
- [Nix 2.16 (2023-05-31)](release-notes/rl-2.16.md)
- [Nix 2.15 (2023-04-11)](release-notes/rl-2.15.md)
- [Nix 2.14 (2023-02-28)](release-notes/rl-2.14.md)
- [Nix 2.13 (2023-01-17)](release-notes/rl-2.13.md)
- [Nix 2.12 (2022-12-06)](release-notes/rl-2.12.md)
- [Nix 2.11 (2022-08-25)](release-notes/rl-2.11.md)
- [Nix 2.10 (2022-07-11)](release-notes/rl-2.10.md)
- [Nix 2.9 (2022-05-30)](release-notes/rl-2.9.md)
- [Nix 2.8 (2022-04-19)](release-notes/rl-2.8.md)
- [Nix 2.7 (2022-03-07)](release-notes/rl-2.7.md)
- [Nix 2.6 (2022-01-24)](release-notes/rl-2.6.md)
- [Nix 2.5 (2021-12-13)](release-notes/rl-2.5.md)
- [Nix 2.4 (2021-11-01)](release-notes/rl-2.4.md)
- [Nix 2.3 (2019-09-04)](release-notes/rl-2.3.md)
- [Nix 2.2 (2019-01-11)](release-notes/rl-2.2.md)
- [Nix 2.1 (2018-09-02)](release-notes/rl-2.1.md)
- [Nix 2.0 (2018-02-22)](release-notes/rl-2.0.md)
- [Nix 1.11.10 (2017-06-12)](release-notes/rl-1.11.10.md)
- [Nix 1.11 (2016-01-19)](release-notes/rl-1.11.md)
- [Nix 1.10 (2015-09-03)](release-notes/rl-1.10.md)
- [Nix 1.9 (2015-06-12)](release-notes/rl-1.9.md)
- [Nix 1.8 (2014-12-14)](release-notes/rl-1.8.md)
- [Nix 1.7 (2014-04-11)](release-notes/rl-1.7.md)
- [Nix 1.6.1 (2013-10-28)](release-notes/rl-1.6.1.md)
- [Nix 1.6 (2013-09-10)](release-notes/rl-1.6.md)
- [Nix 1.5.2 (2013-05-13)](release-notes/rl-1.5.2.md)
- [Nix 1.5 (2013-02-27)](release-notes/rl-1.5.md)
- [Nix 1.4 (2013-02-26)](release-notes/rl-1.4.md)
- [Nix 1.3 (2013-01-04)](release-notes/rl-1.3.md)
- [Nix 1.2 (2012-12-06)](release-notes/rl-1.2.md)
- [Nix 1.1 (2012-07-18)](release-notes/rl-1.1.md)
- [Nix 1.0 (2012-05-11)](release-notes/rl-1.0.md)
- [Nix 0.16 (2010-08-17)](release-notes/rl-0.16.md)
- [Nix 0.15 (2010-03-17)](release-notes/rl-0.15.md)
- [Nix 0.14 (2010-02-04)](release-notes/rl-0.14.md)
- [Nix 0.13 (2009-11-05)](release-notes/rl-0.13.md)
- [Nix 0.12 (2008-11-20)](release-notes/rl-0.12.md)
- [Nix 0.11 (2007-12-31)](release-notes/rl-0.11.md)
- [Nix 0.10.1 (2006-10-11)](release-notes/rl-0.10.1.md)
- [Nix 0.10 (2006-10-06)](release-notes/rl-0.10.md)
- [Nix 0.9.2 (2005-09-21)](release-notes/rl-0.9.2.md)
- [Nix 0.9.1 (2005-09-20)](release-notes/rl-0.9.1.md)
- [Nix 0.9 (2005-09-16)](release-notes/rl-0.9.md)
- [Nix 0.8.1 (2005-04-13)](release-notes/rl-0.8.1.md)
- [Nix 0.8 (2005-04-11)](release-notes/rl-0.8.md)
- [Nix 0.7 (2005-01-12)](release-notes/rl-0.7.md)
- [Nix 0.6 (2004-11-14)](release-notes/rl-0.6.md)
- [Nix 0.5 and earlier](release-notes/rl-0.5.md)

View file

@ -1,162 +0,0 @@
# Table of Contents
- [Introduction](introduction.md)
- [Quick Start](quick-start.md)
- [Installation](installation/installation.md)
- [Supported Platforms](installation/supported-platforms.md)
- [Installing a Binary Distribution](installation/installing-binary.md)
- [Installing Nix from Source](installation/installing-source.md)
- [Prerequisites](installation/prerequisites-source.md)
- [Obtaining a Source Distribution](installation/obtaining-source.md)
- [Building Nix from Source](installation/building-source.md)
- [Using Nix within Docker](installation/installing-docker.md)
- [Security](installation/nix-security.md)
- [Single-User Mode](installation/single-user.md)
- [Multi-User Mode](installation/multi-user.md)
- [Environment Variables](installation/env-variables.md)
- [Upgrading Nix](installation/upgrading.md)
- [Uninstalling Nix](installation/uninstall.md)
- [Package Management](package-management/package-management.md)
- [Basic Package Management](package-management/basic-package-mgmt.md)
- [Profiles](package-management/profiles.md)
- [Garbage Collection](package-management/garbage-collection.md)
- [Garbage Collector Roots](package-management/garbage-collector-roots.md)
- [Sharing Packages Between Machines](package-management/sharing-packages.md)
- [Serving a Nix store via HTTP](package-management/binary-cache-substituter.md)
- [Copying Closures via SSH](package-management/copy-closure.md)
- [Serving a Nix store via SSH](package-management/ssh-substituter.md)
- [Serving a Nix store via S3](package-management/s3-substituter.md)
- [Nix Language](language/index.md)
- [Data Types](language/values.md)
- [Language Constructs](language/constructs.md)
- [String interpolation](language/string-interpolation.md)
- [Operators](language/operators.md)
- [Derivations](language/derivations.md)
- [Advanced Attributes](language/advanced-attributes.md)
- [Built-in Constants](language/builtin-constants.md)
- [Built-in Functions](language/builtins.md)
- [Advanced Topics](advanced-topics/advanced-topics.md)
- [Remote Builds](advanced-topics/distributed-builds.md)
- [Tuning Cores and Jobs](advanced-topics/cores-vs-jobs.md)
- [Verifying Build Reproducibility](advanced-topics/diff-hook.md)
- [Using the `post-build-hook`](advanced-topics/post-build-hook.md)
- [Command Reference](command-ref/command-ref.md)
- [Common Options](command-ref/opt-common.md)
- [Common Environment Variables](command-ref/env-common.md)
- [Main Commands](command-ref/main-commands.md)
- [nix-build](command-ref/nix-build.md)
- [nix-shell](command-ref/nix-shell.md)
- [nix-store](command-ref/nix-store.md)
- [nix-store --add-fixed](command-ref/nix-store/add-fixed.md)
- [nix-store --add](command-ref/nix-store/add.md)
- [nix-store --delete](command-ref/nix-store/delete.md)
- [nix-store --dump-db](command-ref/nix-store/dump-db.md)
- [nix-store --dump](command-ref/nix-store/dump.md)
- [nix-store --export](command-ref/nix-store/export.md)
- [nix-store --gc](command-ref/nix-store/gc.md)
- [nix-store --generate-binary-cache-key](command-ref/nix-store/generate-binary-cache-key.md)
- [nix-store --import](command-ref/nix-store/import.md)
- [nix-store --load-db](command-ref/nix-store/load-db.md)
- [nix-store --optimise](command-ref/nix-store/optimise.md)
- [nix-store --print-env](command-ref/nix-store/print-env.md)
- [nix-store --query](command-ref/nix-store/query.md)
- [nix-store --read-log](command-ref/nix-store/read-log.md)
- [nix-store --realise](command-ref/nix-store/realise.md)
- [nix-store --repair-path](command-ref/nix-store/repair-path.md)
- [nix-store --restore](command-ref/nix-store/restore.md)
- [nix-store --serve](command-ref/nix-store/serve.md)
- [nix-store --verify-path](command-ref/nix-store/verify-path.md)
- [nix-store --verify](command-ref/nix-store/verify.md)
- [nix-env](command-ref/nix-env.md)
- [nix-env --delete-generations](command-ref/nix-env/delete-generations.md)
- [nix-env --install](command-ref/nix-env/install.md)
- [nix-env --list-generations](command-ref/nix-env/list-generations.md)
- [nix-env --query](command-ref/nix-env/query.md)
- [nix-env --rollback](command-ref/nix-env/rollback.md)
- [nix-env --set-flag](command-ref/nix-env/set-flag.md)
- [nix-env --set](command-ref/nix-env/set.md)
- [nix-env --switch-generation](command-ref/nix-env/switch-generation.md)
- [nix-env --switch-profile](command-ref/nix-env/switch-profile.md)
- [nix-env --uninstall](command-ref/nix-env/uninstall.md)
- [nix-env --upgrade](command-ref/nix-env/upgrade.md)
- [Utilities](command-ref/utilities.md)
- [nix-channel](command-ref/nix-channel.md)
- [nix-collect-garbage](command-ref/nix-collect-garbage.md)
- [nix-copy-closure](command-ref/nix-copy-closure.md)
- [nix-daemon](command-ref/nix-daemon.md)
- [nix-hash](command-ref/nix-hash.md)
- [nix-instantiate](command-ref/nix-instantiate.md)
- [nix-prefetch-url](command-ref/nix-prefetch-url.md)
- [Experimental Commands](command-ref/experimental-commands.md)
{{#include ./command-ref/new-cli/SUMMARY.md}}
- [Files](command-ref/files.md)
- [nix.conf](command-ref/conf-file.md)
- [Profiles](command-ref/files/profiles.md)
- [manifest.nix](command-ref/files/manifest.nix.md)
- [manifest.json](command-ref/files/manifest.json.md)
- [Channels](command-ref/files/channels.md)
- [Default Nix expression](command-ref/files/default-nix-expression.md)
- [Architecture and Design](architecture/architecture.md)
- [File System Object](architecture/file-system-object.md)
- [Protocols](protocols/protocols.md)
- [Serving Tarball Flakes](protocols/tarball-fetcher.md)
- [Derivation "ATerm" file format](protocols/derivation-aterm.md)
- [Glossary](glossary.md)
- [Contributing](contributing/contributing.md)
- [Hacking](contributing/hacking.md)
- [Testing](contributing/testing.md)
- [Experimental Features](contributing/experimental-features.md)
- [CLI guideline](contributing/cli-guideline.md)
- [C++ style guide](contributing/cxx.md)
- [Release Notes](release-notes/release-notes.md)
{{#include ./SUMMARY-rl-next.md}}
- [Release 2.18 (2023-09-20)](release-notes/rl-2.18.md)
- [Release 2.17 (2023-07-24)](release-notes/rl-2.17.md)
- [Release 2.16 (2023-05-31)](release-notes/rl-2.16.md)
- [Release 2.15 (2023-04-11)](release-notes/rl-2.15.md)
- [Release 2.14 (2023-02-28)](release-notes/rl-2.14.md)
- [Release 2.13 (2023-01-17)](release-notes/rl-2.13.md)
- [Release 2.12 (2022-12-06)](release-notes/rl-2.12.md)
- [Release 2.11 (2022-08-25)](release-notes/rl-2.11.md)
- [Release 2.10 (2022-07-11)](release-notes/rl-2.10.md)
- [Release 2.9 (2022-05-30)](release-notes/rl-2.9.md)
- [Release 2.8 (2022-04-19)](release-notes/rl-2.8.md)
- [Release 2.7 (2022-03-07)](release-notes/rl-2.7.md)
- [Release 2.6 (2022-01-24)](release-notes/rl-2.6.md)
- [Release 2.5 (2021-12-13)](release-notes/rl-2.5.md)
- [Release 2.4 (2021-11-01)](release-notes/rl-2.4.md)
- [Release 2.3 (2019-09-04)](release-notes/rl-2.3.md)
- [Release 2.2 (2019-01-11)](release-notes/rl-2.2.md)
- [Release 2.1 (2018-09-02)](release-notes/rl-2.1.md)
- [Release 2.0 (2018-02-22)](release-notes/rl-2.0.md)
- [Release 1.11.10 (2017-06-12)](release-notes/rl-1.11.10.md)
- [Release 1.11 (2016-01-19)](release-notes/rl-1.11.md)
- [Release 1.10 (2015-09-03)](release-notes/rl-1.10.md)
- [Release 1.9 (2015-06-12)](release-notes/rl-1.9.md)
- [Release 1.8 (2014-12-14)](release-notes/rl-1.8.md)
- [Release 1.7 (2014-04-11)](release-notes/rl-1.7.md)
- [Release 1.6.1 (2013-10-28)](release-notes/rl-1.6.1.md)
- [Release 1.6 (2013-09-10)](release-notes/rl-1.6.md)
- [Release 1.5.2 (2013-05-13)](release-notes/rl-1.5.2.md)
- [Release 1.5 (2013-02-27)](release-notes/rl-1.5.md)
- [Release 1.4 (2013-02-26)](release-notes/rl-1.4.md)
- [Release 1.3 (2013-01-04)](release-notes/rl-1.3.md)
- [Release 1.2 (2012-12-06)](release-notes/rl-1.2.md)
- [Release 1.1 (2012-07-18)](release-notes/rl-1.1.md)
- [Release 1.0 (2012-05-11)](release-notes/rl-1.0.md)
- [Release 0.16 (2010-08-17)](release-notes/rl-0.16.md)
- [Release 0.15 (2010-03-17)](release-notes/rl-0.15.md)
- [Release 0.14 (2010-02-04)](release-notes/rl-0.14.md)
- [Release 0.13 (2009-11-05)](release-notes/rl-0.13.md)
- [Release 0.12 (2008-11-20)](release-notes/rl-0.12.md)
- [Release 0.11 (2007-12-31)](release-notes/rl-0.11.md)
- [Release 0.10.1 (2006-10-11)](release-notes/rl-0.10.1.md)
- [Release 0.10 (2006-10-06)](release-notes/rl-0.10.md)
- [Release 0.9.2 (2005-09-21)](release-notes/rl-0.9.2.md)
- [Release 0.9.1 (2005-09-20)](release-notes/rl-0.9.1.md)
- [Release 0.9 (2005-09-16)](release-notes/rl-0.9.md)
- [Release 0.8.1 (2005-04-13)](release-notes/rl-0.8.1.md)
- [Release 0.8 (2005-04-11)](release-notes/rl-0.8.md)
- [Release 0.7 (2005-01-12)](release-notes/rl-0.7.md)
- [Release 0.6 (2004-11-14)](release-notes/rl-0.6.md)
- [Release 0.5 and earlier](release-notes/rl-0.5.md)

Some files were not shown because too many files have changed in this diff Show more