Commit graph

15787 commits

Author SHA1 Message Date
julia 6c311a4afa Add a clearer error message for InvalidPathError during evaluation
Part of #270, #271

Change-Id: I864d7340f26d3c0f9c45db7b6b545face38d8294
2024-06-16 03:53:00 +00:00
julia 0fa289f559 Harmonise the Store::queryPathInfoUncached interface
This:
 - Consistently returns `nullptr` for a non-existent
   store path, instead of a mix of `nullptr` and
   throwing exceptions.

 - If a store returns "bad" store paths in response
   to a request (e.g. incorrect hash or name), don't
   cache this result. This removes some duplication
   of code at the cache-access layer of queryPathInfo()
   checking this, and ­allows us to provide more
   specific errors.

Part of #270.

Change-Id: I86612c6499b1a37ab872c712c2304d6a3ff19edb
2024-06-16 03:53:00 +00:00
jade 4734ce7831 version: update to 2.91-dev for main
This commit constitutes the branch-off for 2.91. The parent of this
commit will be the branch point for release-2.90.

Change-Id: I7f047545df29a9cff93346137c865dcbf1415488
2024-06-15 18:46:18 -07:00
jade 79404f7ffc releng: automatically add to the summary page
Also delete the obsolescent maintainers/release-notes script that is
unmaintained.

Change-Id: I3f4a75d790e8e00e970358ca8f32e8295c91aac3
2024-06-15 18:46:18 -07:00
jade f95a47e8c4 release: release notes for 2.90.0
For now we just need to put the release notes in the final spot. We will
have to fix the date on both 2.90 and 2.91 branches, but such as it is.

Release created with releng/create_release.xsh

Closes: lix-project/lix#318
Change-Id: I38e79b40e7f632c8a286f2f09865a84dc93eca90
2024-06-15 18:46:18 -07:00
jade 9923fb6dd9 version: update to 2.90.0-rc2
Change-Id: I7cfa8414fa937de940b1598cc300497ce2d47635
2024-06-14 20:29:59 -07:00
jade a9c610fe37 release: merge release 2.90.0-rc1 back to mainline
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh

Change-Id: I92296a1746b54a081004fe2bb23e9e37fd33b3e5
2024-06-14 19:40:14 -07:00
jade f82a2a9aaa release: 2.90.0-rc1 "Vanilla Ice Cream"
Release produced with releng/create_release.xsh

Change-Id: Ib8df5ea5096d47b25f74988447146c1b5072b869
2024-06-14 19:40:13 -07:00
jade e1059bfa34 Merge changes from topic "releng" into main
* changes:
  releng: add sha256 for the manual tarball
  releng: fix upload of multiarch images to forgejo
  releng: fix git checking
  releng: fix logging inside interactive xonsh
  releng: support multiple systems
  version: update to 2.90.0-rc1
2024-06-15 02:38:09 +00:00
jade d5c670ad01 releng: add sha256 for the manual tarball
Whoops.

Change-Id: Ic6f8cdcb074d679e9b1fc3323c106cc853328dcc
2024-06-13 17:14:06 -07:00
jade 16ea19ced8 releng: fix upload of multiarch images to forgejo
Forgejo appears to immediately delete registry content that is
overwritten. This means that we are forced to delete our previous
workaround of making a temporary tag and use a new, more absurd
workaround of making an entire temporary image that we basically only
need to create to get its hash.

However, on the plus side, the new workaround doesn't create garbage
tags to begin with, which means that we don't have to deal with GitHub
not implementing the standardized tag delete endpoint and instead
only implementing a proprietary one.

Upstream-Bug: https://github.com/containers/skopeo/issues/2354
Change-Id: I220e7ce9a17fd230c38882f12c009a166dcc9336
2024-06-13 17:12:45 -07:00
jade 7be0d237e0 releng: fix git checking
Change-Id: I82ddd918311b48e596adb807b81221973113fe7a
2024-06-13 15:24:21 -07:00
jade e715e5fd31 releng: fix logging inside interactive xonsh
I don't know when this broke, it seems like it happened since the 24.05
upgrade, so xonsh 0.15.

What happened is that xonsh was trying to intercept log output, which
explodes if you have the logger survive past one command input. This is,
however, impossible to avoid if you are trying to use logging when you
import releng from inside xonsh for interactive use!

The error below is because the memory handler backing the stdout/stderr
of the one command that's just been run was closed after the command
completed.

Change-Id: I2be642aebf93da9818d08ff8b97c2e72ba5ac581

--- Logging error ---
Traceback (most recent call last):
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/logging/__init__.py", line 1113, in emit
    stream.write(msg + self.terminator)
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/base_shell.py", line 183, in write
    self.mem.write(s)
ValueError: I/O operation on closed file.
Call stack:
  File "/nix/store/xgdp1p1gv8ni1awnkzyqasnn6gz5wlvx-xonsh-0.15.1/bin/xonsh", line 8, in <module>
    sys.exit(main())
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/main.py", line 470, in main
    sys.exit(main_xonsh(args))
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/main.py", line 514, in main_xonsh
    shell.shell.cmdloop()
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/ptk_shell/shell.py", line 406, in cmd
loop
    line = self.singleline(auto_suggest=auto_suggest)
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/ptk_shell/shell.py", line 374, in sin
gleline
    line = self.prompter.prompt(**prompt_args)
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/prompt_toolkit/shortcuts/prompt.py", line 1
026, in prompt
    return self.app.run(
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/prompt_toolkit/application/application.py",
 line 1002, in run
    return asyncio.run(coro)
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/runners.py", line 189, in run
    with Runner(debug=debug) as runner:
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/runners.py", line 59, in __enter__
    self._lazy_init()
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/runners.py", line 137, in _lazy_init
    self._loop = events.new_event_loop()
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/events.py", line 810, in new_event_loop
    return get_event_loop_policy().new_event_loop()
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/events.py", line 699, in new_event_loop
    return self._loop_factory()
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/unix_events.py", line 64, in __init__
    super().__init__(selector)
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/selector_events.py", line 54, in __init__
    logger.debug('Using selector: %s', selector.__class__.__name__)
Message: 'Using selector: %s'
Arguments: ('EpollSelector',)

Change-Id: I90959809129aaf96aad4577599031688599ed85e
2024-06-13 15:17:44 -07:00
jade 068576042b packaging: Move binaryTarball to a passthru attr in package.nix
This is motivated by flakes being bad and all the stuff that calls
things by "system" being utterly unable to cope with cross compilation.
So if we go shove it in package.nix it is suddenly usable from cross
contexts.

Usage:

```
nix build -L .#nix-riscv64-linux.binaryTarball
```

Change-Id: I702ebf2ac5bd9d1c57662f968b000073134df336
2024-06-13 15:14:22 -07:00
jade d194939ff5 flake.nix: add riscv64 cross target
It builds. I have not tested the binaries since I don't have hardware,
but I would be rather surprised if it were broken, given that nix *runs*
on this platform.

Change-Id: I0b474ffcd4a431bf117a303d0b65fa6532113f48
2024-06-13 15:04:06 -07:00
jade 74fb2e8c47 releng: support multiple systems
I guess this is kind of important to being able to "release it".

Change-Id: Id6f295d0b4944fa1203783a400a246727dbd94b6
2024-06-13 14:36:03 -07:00
jade ac28cff28f version: update to 2.90.0-rc1
Change-Id: I913852cfb88b3b300ffb1050a91784be659ae66a
2024-06-13 13:03:25 -07:00
alois31 1d6fd94cf9 Merge "tests/libcmd: set HOME to a temporary directory" into main 2024-06-13 05:27:55 +00:00
jade d0b28f0e74 releng: fix docs upload
There were two bugs I found:
1. If the build isn't already done in the store, nix-store --realise
   does not know how to build it. You have to just give it the
   derivation and I guess it will realise all outputs, which is fine.
2. cp without -T will not overwrite an existing manual directory,
   creating a path manual/manual.

Change-Id: Ibebfd136a266da5330944a985e636ebb776f1909
2024-06-12 15:34:23 -07:00
jade 38e4e69633 .github: clean up stale things and wrong references
We do not need a stale bot. We do not need dependabot.

Fixes: lix-project/lix#391
Change-Id: I983fae4dc4cd9022b12f70e330b5c984c5fc1b9d
2024-06-12 15:34:23 -07:00
Pierre Bourdon 248ecb11af dev shell: Add bashInteractive
This was bothersome to me as a zsh+direnv user.

Change-Id: Ia5b54cc63647a5c6ced2b5412e972dac1abf8184
2024-06-12 15:34:23 -07:00
jade 73898cad0e tests/flake-registry: Fix occasional deadlocks
This seems to have been caused by having the wrong PID. I don't know why
it worked before in the sandbox, but the code was definitely wrong
before, so let's just fix it.

Change-Id: I556580bdf614c716566310e975a36daa6d6c9a91
2024-06-12 15:34:23 -07:00
jade 59b5965bbf doc: Write an index of environment variables used in testsuite
This was originally going to be just the testsuite but I kinda just
documented all of them.

I am tired of us not documenting these. This is a starting point to
producing an actually good index. I would like to enforce it in a
pre-commit hook eventually that we document all environment variables
used in Lix itself, even if it is terse dev facing docs.

This is full of a bunch of TODOs caused by auditing code. They should
probably be done at some point.

Change-Id: I7c0d3b257e19bae23d47d1efbd7361d203bccb0e
2024-06-12 15:34:23 -07:00
jade 211f79d4a2 doc/testing.md: Rewrite some outdated sections for meson and current source layout
Change-Id: Ia23f82c9a564b55bd799afbda59c28c9b0a65c13
2024-06-12 15:34:23 -07:00
jade 5f6eb6eb44 doc: rewrite the multi-user documentation to actually talk about security
It's in the security section, and it was totally outdated anyway.

I took the opportunity to write down the stuff we already believed.

Change-Id: I73e62ae85a82dad13ef846e31f377c3efce13cb0
2024-06-12 15:34:23 -07:00
jade d9345d8836 tests: verify that NIX_DAEMON_PACKAGE's version looks like a version
Followup to https://gerrit.lix.systems/c/lix/+/1417 to ensure that this
parser will never take something that doesn't look like a version.

It turns out this problem is less alarming than initially thought
because it only applies to the testsuite in a non-default mode.

Change-Id: I26aba24aaf0215f2b782966314b94784db766266
2024-06-12 15:34:23 -07:00
jade 6939ffc9f9 Check devShells in CI
We should not let these regress in CI by having broken dependencies or
similar. Still need to fix the evaluation error checking in
buildbot-nix, but this is a useful step regardless.

Fixes: lix-project/lix#383

Change-Id: I3883184165440e66256c989117f2ab2e54c3aafd
2024-06-12 15:34:23 -07:00
jade 479055aee8 Misc workaround removals since 24.05 upgrade
Change-Id: I9491b103333cb0e25c245199e88365ded7800d2e
2024-06-12 15:34:23 -07:00
Pierre Bourdon f7b6552699 [resubmit] flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)
-- message from cl/1418 --

The boehmgc changes are bundled into this commit because doing otherwise
would require an annoying dance of "adding compatibility for < 8.2.6 and
>= 8.2.6" then updating the pin then removing the (now unneeded)
compatibility. It doesn't seem worth the trouble to me given the low
complexity of said changes.

Rebased coroutine-sp-fallback.diff patch taken from https://github.com/NixOS/nixpkgs/pull/317227

-- jade resubmit changes --

This is a resubmission of https://gerrit.lix.systems/c/lix/+/1418, which
was reverted in https://gerrit.lix.systems/c/lix/+/1432 for breaking CI
evaluation without being detected.

I have run `nix flake check -Lv` on this one before submission and it
passes on my machine and crucially without eval errors, so the CI result
should be accurate.

It seems like someone renamed forbiddenDependenciesRegex to
forbiddenDependenciesRegexes in nixpkgs and also changed the type
incompatibly. That's pretty silly, but at least it's just an eval error.

Also, `xonsh` regressed the availability of `xonsh-unwrapped`, but it
was fixed by us in https://github.com/NixOS/nixpkgs/pull/317636, which
is now in our channel, so we update nixpkgs compared to the original
iteration of this to simply get that.

We originally had a regression related to some reorganization of the
nixpkgs lib test suite in which there was broken parameter passing.
This, too, we got quickfixed in nixpkgs, so we don't need any changes
for it: https://github.com/NixOS/nixpkgs/pull/317772

Related: https://gerrit.lix.systems/c/lix/+/1428
Fixes: lix-project/lix#385

Change-Id: I26d41ea826fec900ebcad0f82a727feb6bcd28f3
2024-06-12 15:34:22 -07:00
alois31 3c0434999e tests/libcmd: set HOME to a temporary directory
The libcmd unit test creates files (more specifically, the fetcher cache) in
its home directory. In the single-user sandbox, this leads to the creation of
/homeless-shelter, since this is the default HOME and the root is writable.
Unfortunately, this conflicts with the assumption of the functional tests that
this directory does not exist. Use a different home directory to prevent these
test failures, and thus restore the ability to build inside the single-user
sandbox.

Fixes: lix-project/lix#365
Change-Id: I4df8c53d043234b95a7c0ac45fc5ee89e8d46aff
2024-06-12 22:13:55 +00:00
Qyriad f46194faa2 build: remove unused 'deps' variable
This never actually got used

Change-Id: I8f3f1d413124b27913d59a75cff42319cbaac385
2024-06-12 21:40:57 +00:00
jade 8a3d063a49 Merge changes from topic "releng" into main
* changes:
  releng: add prod environment, ready for release
  releng: automatically figure out if we should tag latest for docker
  releng: support multiarch docker images
  manual: rewrite the docker guide now that we have images
  Rewrite docker to be sensible and smaller
  Implement docker upload in the releng tools
2024-06-11 04:45:12 +00:00
jade f432e464dd Merge "tests: fix daemon version in isDaemonNewer function" into main 2024-06-10 23:22:05 +00:00
jade a986a8dfa1 Merge "Revert "flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)"" into main 2024-06-10 04:48:12 +00:00
jade 8a09465c3a Revert "flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)"
This reverts commit 28a079f841.

Reason for revert: This caused a pile of regressions in CI, and does not pass nix flake check. Some number of them are fixed in CL: https://gerrit.lix.systems/c/lix/+/1429 but there's more to be fixed.

We should defer this after 2.90.

Change-Id: Ib839d0fcb08eb52094af2b521145e3c1b4e0556f
2024-06-10 04:29:13 +00:00
jade 82dc712d93 releng: add prod environment, ready for release
I am *reasonably* confident that this releng infrastructure can actually
build a Lix 2.90 and release it successfully. Let's make it possible to
do, and add some cute colours to the confirmation message.

Change-Id: I85e498b6fb49ffc5e75c0a72c5e45fb1f69030d3
2024-06-09 20:33:24 -07:00
jade ce71d0e9ab releng: automatically figure out if we should tag latest for docker
For example, when releasing from release-2.90, if `main` has a 2.91 tag
ancestor, we know that 2.91 was released, so we should *not* tag latest.

Change-Id: Ia56b17a2ee03bbec74b7c271c742858c690d450d
2024-06-09 20:33:24 -07:00
jade 9aeb314e6a releng: support multiarch docker images
If we don't want to have separate registry tags by architecture (EWWWW),
we need to be able to build multiarch docker images. This is pretty
simple, and just requires making a manifest pointing to each of the
component images.

I was *going* to just do this API prodding with manifest-tool, but it
doesn't support putting metadata on the outer manifest, which is
actually kind of a problem because it then doesn't render the metadata
on github. So I guess we get a simple little containers API
implementation that is 90% auth code.

Change-Id: I8bdd118d4cbc13b23224f2fb174b232432686bea
2024-06-09 20:33:24 -07:00
jade 4392d89eea manual: rewrite the docker guide now that we have images
Change-Id: I5bdf47e67059ae4099552750a47ae070dbe094df
2024-06-09 20:33:24 -07:00
jade 9bb7fb8f69 Rewrite docker to be sensible and smaller
I have checked the image can build things and inspected `diff -ru`
compared to the old image. As far as I can tell it is more or less
the same besides the later git change.

Layers are now 65MB or less, and we aren't against the maxLayers limit
for the broken automatic layering to do anything but shove one store
path in a layer (which is good behaviour, actually).

This uses nix2container which streams images, so the build time is much
shorter.

I have also taken the opportunity to, in addition to fixing the 400MB
single layer (terrible, and what motivated this in the first place),
delete about 200MB of closure size inflicted by git vs gitMinimal
causing both perl and python to get into closure.

People mostly use this thing for CI, so I don't really think you need
advanced git operations, and large git can be added at the user side if
really motivated.

With love for whichever container developer somewhat ironically assumed
that one would not run skopeo in a minimal container that doesn't have a
/var/tmp.

Fixes: lix-project/lix#378

Change-Id: Icc3aa20e64446276716fbbb87535fd5b50628010
2024-06-09 20:33:24 -07:00
jade 7dfa2a761e Merge changes from topic "releng" into main
* changes:
  releng: support pushing the manual to docs also
  Expose officialRelease from the flake
  Put into place initial release engineering
2024-06-09 08:28:52 +00:00
jade ff95b980d4 Implement docker upload in the releng tools
This uses skopeo to not think about docker daemons. I, however, noticed
that the docker image we had would have totally terrible cache hits, so
I rewrote it.

Fixes: lix-project/lix#252

Change-Id: I3c5b6c1f3ba0b9dfcac212b2148f390e0cd542b7
2024-06-09 00:30:12 -07:00
Pierre Bourdon 28a079f841
flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)
The boehmgc changes are bundled into this commit because doing otherwise
would require an annoying dance of "adding compatibility for < 8.2.6 and
>= 8.2.6" then updating the pin then removing the (now unneeded)
compatibility. It doesn't seem worth the trouble to me given the low
complexity of said changes.

Rebased coroutine-sp-fallback.diff patch taken from https://github.com/NixOS/nixpkgs/pull/317227

Change-Id: I8c590e9fe25c0f566d0cfeacb96d8cf50abf12e8
2024-06-09 01:25:53 +02:00
Pierre Bourdon 9281a12532
tests/nixos/nix-copy: fix NixOS >= 24.05 compatibility
4b128008c5d9fde881ce1b0a25e60ae0415a14d5 in nixpkgs introduced a default
hashedPasswordFile for root in NixOS tests, which takes precedence over
the password option set in the nix-copy test.

Change-Id: Iffaebec5992e50614b854033f0d14312c8d275b5
2024-06-08 17:59:08 +02:00
Mario Rodas a05de58ebd tests: fix daemon version in isDaemonNewer function
Since ad8a4b380e, the version printer returns "nix (Lix, like Nix) 2.x",
hence the `daemonVersion` was being set to the string "like".

Using `compareVersions` with a letter compares them lexicographically:

   builtins.compareVersions "like" "2.12pre20230103"  // => -1
   builtins.compareVersions "like" "2.16.0"           // => -1

This caused that `isDaemonNewer` always returned 1, falsy in Bash terms.
Therefore, the test suite skipped those tests where they use it.

Fixes lix-project/lix#324

Change-Id: If6682515bf0bf8b8add641af9a4e98b50a9acb51
2024-06-08 04:20:00 +00:00
jade 4f94531209 Merge changes from topic "releng" into main
* changes:
  Add meson release note
  Move version to a JSON file so we can have release names
  Remove rl-next-dev
2024-06-07 03:53:31 +00:00
jade 98e8475147 releng: support pushing the manual to docs also
Change-Id: Ifd0b51425ee4955e0230fb2804a6f54ef0fe16e9
2024-06-06 20:53:08 -07:00
jade bdf1b264ad Expose officialRelease from the flake
Change-Id: If87beb3f31dfb5d59862294ac2e1c821ea864277
2024-06-06 20:53:08 -07:00
jade c32a01f9eb Put into place initial release engineering
This can release x86_64-linux binaries to staging, with ephemeral keys.
I think it's good enough to review at least at this point, so we don't
keep adding more stuff to it to make it harder to review.

Change-Id: Ie95e8f35d1252f5d014e819566f170b30eda152e
2024-06-06 20:53:08 -07:00
Qyriad ec768df004 Merge changes Ic4be41eb,I48db2385 into main
* changes:
  devshells: only enable pch for clang
  build: expose option to enable or disable precompiled std headers
2024-06-06 22:21:52 +00:00