update-input alternative for build subcommand #400

New issue

Closed

opened 2024-06-18 17:00:15 +00:00 by benaryorg · 2 comments

benaryorg commented

2024-06-18 17:00:15 +00:00

My systems are generally flake based (and the associated self-hosted hydra is testing the flake) for a variety of reasons. However the production machines themselves are supposed to run unattended updates.

Nix allows me to use this in the nixos-upgrade.service:

nixos-rebuild switch --update-input nixpkgs --refresh --flake /etc/nixos --upgrade

Where the flake in /etc/nixos is merely a shim¹ that allows me to replace the git+https:// URl of nixpkgs with the much more lightweight tarball+https:// one (considering there's no specialized fetcher for cgit):

Now that I'm in the process of switching to Lix, how would I solve this issue in the most Lix way possible?
Having multiple commands in the upgrade unit that run nix flake update before running the nixos-rebuild feels a bit clunky, however personally I'd be fine with this feature request (I was torn between RFE and doc issue) being closed with a reference to the multi-command way, however I feel like something's lost by not allowing there to be something like the Nix --no-update-lock-file --update-input INPUT way, since it does not allow on-the-fly testing with different versions (unless using override-input with a specific rev each) and requires on-disk state instead.

(I realize that was everything but concise, sorry 'bout that)

Describe the solution you'd like

A way to run a single command to build a flake with one or more inputs updated to the most current version available on the remote.

Describe alternatives you've considered

using override-input with the specific revisions for each command, which lacks much of the dynamic nature though (and requires several commands prior to determine which revision is the current one)
running a separate command which would work in my specific use-case but may not work generally

Additional context

¹: the shim mentioned above:

{
  inputs.benaryorg.url = "git+https://git.shell.bsocat.net/infra?ref=update";
  inputs.nixpkgs.url = "tarball+https://git.shell.bsocat.net/nixpkgs/snapshot/nixpkgs-nixos-24.05.tar.gz";
  inputs.benaryorg.inputs.nixpkgs.follows = "nixpkgs";

  outputs = { benaryorg, ... }:
  {
    inherit (benaryorg) nixosConfigurations;
  };
}

## Is your feature request related to a problem? Please describe. My systems are generally flake based (and the associated self-hosted hydra is testing the flake) for a variety of reasons. However the production machines themselves are supposed to run unattended updates. Nix allows me to use this in the *nixos-upgrade.service*: ```bash nixos-rebuild switch --update-input nixpkgs --refresh --flake /etc/nixos --upgrade ``` Where the flake in */etc/nixos* is merely a shim¹ that allows me to replace the `git+https://` URl of nixpkgs with the much more lightweight `tarball+https://` one (considering there's no specialized fetcher for cgit): Now that I'm in the process of switching to Lix, how would I solve this issue in the most Lix way possible? Having multiple commands in the upgrade unit that run `nix flake update` before running the *nixos-rebuild* feels a bit clunky, however personally I'd be fine with this feature request (I was torn between RFE and doc issue) being closed with a reference to the multi-command way, however I feel like something's lost by not allowing there to be something like the Nix `--no-update-lock-file --update-input INPUT` way, since it does not allow on-the-fly testing with different versions (unless using override-input with a specific rev each) and requires on-disk state instead. (I realize that was *everything* but concise, sorry 'bout that) ## Describe the solution you'd like A way to run a single command to build a flake with one or more inputs updated to the most current version available on the remote. ## Describe alternatives you've considered - using override-input with the specific revisions for each command, which lacks much of the dynamic nature though (and requires several commands prior to determine which revision is the current one) - running a separate command which would work in my specific use-case but may not work generally ## Additional context ¹: the shim mentioned above: ```nix { inputs.benaryorg.url = "git+https://git.shell.bsocat.net/infra?ref=update"; inputs.nixpkgs.url = "tarball+https://git.shell.bsocat.net/nixpkgs/snapshot/nixpkgs-nixos-24.05.tar.gz"; inputs.benaryorg.inputs.nixpkgs.follows = "nixpkgs"; outputs = { benaryorg, ... }: { inherit (benaryorg) nixosConfigurations; }; } ```

jade commented

2024-06-18 18:26:45 +00:00

Owner

FYI this complaint also applies to CppNix 2.22 to the best of our knowledge because we just cherry picked their redesign.

hort answer: the new semantics are that normal commands can only automatically apply necessary updates to match declarations in flake.nix but cannot otherwise update it. This aligns, at least, with how npm and cargo do their lock files; i don't think there's a command to change the version while invoking a build.

It is somewhat unfortunate that certain invocations now are two commands but it does also make the usage clearer. I think particularly with nixos-rebuild, those invocations may have been somewhat suspiciously implemented since it's plausible that those arguments got passed to multiple invocations of nix build, leading to unclear semantics (since nixos-rebuild reexecs itself after grabbing the nixos-rebuild of the new config). And furthermore, since nixos-rebuild is typically invoked as root, #47 can be avoided by updating the lock file first as a normal user.

FYI this complaint also applies to CppNix 2.22 to the best of our knowledge because we just cherry picked their redesign. hort answer: the new semantics are that normal commands can only automatically apply necessary updates to match declarations in flake.nix but cannot otherwise update it. This aligns, at least, with how npm and cargo do their lock files; i don't think there's a command to change the version while invoking a build. It is somewhat unfortunate that certain invocations now are two commands but it does also make the usage clearer. I think particularly with nixos-rebuild, those invocations may have been somewhat suspiciously implemented since it's plausible that those arguments got passed to multiple invocations of nix build, leading to unclear semantics (since nixos-rebuild reexecs itself after grabbing the nixos-rebuild of the new config). And furthermore, since nixos-rebuild is typically invoked as root, https://git.lix.systems/lix-project/lix/issues/47 can be avoided by updating the lock file first as a normal user.

benaryorg commented

2024-06-20 00:59:18 +00:00

Author

FYI this complaint also applies to CppNix 2.22 to the best of our knowledge because we just cherry picked their redesign.

Ah. I see.
I have had no run-in with any newer version of CppNix since none of them were ever marked as stable by nixpkgs IIRC (and I ran into other issues with CppNix when updating to 24.05 anyways).

This aligns, at least, with how npm and cargo do their lock files

That comparison hit me yesterday, because indeed it seems to me that the lock file is indeed the culprit here, so removing it (and adding --no-write-lock-file) should fix my issue in particular. The same way that Cargo.lock is serves reproducibility only, while the actual dependency management is handled in Cargo.toml, removing the lock file may lower reproducibility (which I want in this case) but makes stuff work with newer versions out of the box (for non-reproducible use-cases of course).
Doing that, the caching seems to still work, both for evaluations and all git sources, and with the flake I have as a shim to override the nixpkgs dependency via a follows I don't actually have to override or update the lock file of the repository anyway.
So as long as all my sources are git+https (or anything that can be properly cached, unlike the cgit tarball export, even though technically it has an etag in the HTTP response) those will remain cached, and the tarball will regrettably be refetched every single time (getting a 304 Not Modified would be nice, but I don't see where the etag could be properly cached on the client side of things to send along in the request header I guess).
Effectively, at the expense of refetching those 40MiB of data (however horrible that is on the cgit end, considering that nixpkgs is a pain for git to handle in general) I can bypass any caching for that file, so it does what I need, albeit not as generically as I had hoped.

So.… my very specific itch is scratched by deleting the lock file, which means I don't really have a use-case for this any longer where running a separate command wouldn't work better anyway.
Thanks for the explanation and context, I'll close this then.

> FYI this complaint also applies to CppNix 2.22 to the best of our knowledge because we just cherry picked their redesign. Ah. I see. I have had no run-in with any newer version of CppNix since none of them were ever marked as stable by nixpkgs IIRC (and I ran into other issues with CppNix when updating to 24.05 anyways). > This aligns, at least, with how npm and cargo do their lock files That comparison hit me yesterday, because indeed it *seems* to me that the lock file is indeed the culprit here, so removing it (and adding `--no-write-lock-file`) should fix my issue in particular. The same way that *Cargo.lock* is serves reproducibility only, while the actual dependency management is handled in *Cargo.toml*, removing the lock file may lower reproducibility (which I *want* in this case) but makes stuff work with newer versions out of the box (for non-reproducible use-cases of course). Doing that, the caching seems to still work, both for evaluations and all git sources, and with the flake I have as a shim to override the nixpkgs dependency via a follows I don't actually have to override or update the lock file of the repository anyway. So as long as all my sources are `git+https` (or anything that can be properly cached, unlike the cgit tarball export, even though technically it has an etag in the HTTP response) those will remain cached, and the tarball will regrettably be refetched every single time (getting a 304 Not Modified would be nice, but I don't see where the etag could be properly cached on the client side of things to send along in the request header I guess). Effectively, at the expense of refetching those 40MiB of data (however horrible that is on the cgit end, considering that nixpkgs is a pain for git to handle in general) I can bypass any caching for that file, so it does what I need, albeit not as generically as I had hoped. So.… my very specific itch is scratched by deleting the lock file, which means I don't really have a use-case for this any longer where running a separate command wouldn't work better anyway. Thanks for the explanation and context, I'll close this then.