lix-project/lix

Fork 31

update-input alternative for build subcommand #400

New issue

Closed

opened 2024-06-18 17:00:15 +00:00 by benaryorg · 6 comments

benaryorg commented

2024-06-18 17:00:15 +00:00

My systems are generally flake based (and the associated self-hosted hydra is testing the flake) for a variety of reasons. However the production machines themselves are supposed to run unattended updates.

Nix allows me to use this in the nixos-upgrade.service:

nixos-rebuild switch --update-input nixpkgs --refresh --flake /etc/nixos --upgrade

Where the flake in /etc/nixos is merely a shim¹ that allows me to replace the git+https:// URl of nixpkgs with the much more lightweight tarball+https:// one (considering there's no specialized fetcher for cgit):

Now that I'm in the process of switching to Lix, how would I solve this issue in the most Lix way possible?
Having multiple commands in the upgrade unit that run nix flake update before running the nixos-rebuild feels a bit clunky, however personally I'd be fine with this feature request (I was torn between RFE and doc issue) being closed with a reference to the multi-command way, however I feel like something's lost by not allowing there to be something like the Nix --no-update-lock-file --update-input INPUT way, since it does not allow on-the-fly testing with different versions (unless using override-input with a specific rev each) and requires on-disk state instead.

(I realize that was everything but concise, sorry 'bout that)

Describe the solution you'd like

A way to run a single command to build a flake with one or more inputs updated to the most current version available on the remote.

Describe alternatives you've considered

using override-input with the specific revisions for each command, which lacks much of the dynamic nature though (and requires several commands prior to determine which revision is the current one)
running a separate command which would work in my specific use-case but may not work generally

Additional context

¹: the shim mentioned above:

{
  inputs.benaryorg.url = "git+https://git.shell.bsocat.net/infra?ref=update";
  inputs.nixpkgs.url = "tarball+https://git.shell.bsocat.net/nixpkgs/snapshot/nixpkgs-nixos-24.05.tar.gz";
  inputs.benaryorg.inputs.nixpkgs.follows = "nixpkgs";

  outputs = { benaryorg, ... }:
  {
    inherit (benaryorg) nixosConfigurations;
  };
}

## Is your feature request related to a problem? Please describe. My systems are generally flake based (and the associated self-hosted hydra is testing the flake) for a variety of reasons. However the production machines themselves are supposed to run unattended updates. Nix allows me to use this in the *nixos-upgrade.service*: ```bash nixos-rebuild switch --update-input nixpkgs --refresh --flake /etc/nixos --upgrade ``` Where the flake in */etc/nixos* is merely a shim¹ that allows me to replace the `git+https://` URl of nixpkgs with the much more lightweight `tarball+https://` one (considering there's no specialized fetcher for cgit): Now that I'm in the process of switching to Lix, how would I solve this issue in the most Lix way possible? Having multiple commands in the upgrade unit that run `nix flake update` before running the *nixos-rebuild* feels a bit clunky, however personally I'd be fine with this feature request (I was torn between RFE and doc issue) being closed with a reference to the multi-command way, however I feel like something's lost by not allowing there to be something like the Nix `--no-update-lock-file --update-input INPUT` way, since it does not allow on-the-fly testing with different versions (unless using override-input with a specific rev each) and requires on-disk state instead. (I realize that was *everything* but concise, sorry 'bout that) ## Describe the solution you'd like A way to run a single command to build a flake with one or more inputs updated to the most current version available on the remote. ## Describe alternatives you've considered - using override-input with the specific revisions for each command, which lacks much of the dynamic nature though (and requires several commands prior to determine which revision is the current one) - running a separate command which would work in my specific use-case but may not work generally ## Additional context ¹: the shim mentioned above: ```nix { inputs.benaryorg.url = "git+https://git.shell.bsocat.net/infra?ref=update"; inputs.nixpkgs.url = "tarball+https://git.shell.bsocat.net/nixpkgs/snapshot/nixpkgs-nixos-24.05.tar.gz"; inputs.benaryorg.inputs.nixpkgs.follows = "nixpkgs"; outputs = { benaryorg, ... }: { inherit (benaryorg) nixosConfigurations; }; } ```

jade commented

2024-06-18 18:26:45 +00:00

Owner

FYI this complaint also applies to CppNix 2.22 to the best of our knowledge because we just cherry picked their redesign.

hort answer: the new semantics are that normal commands can only automatically apply necessary updates to match declarations in flake.nix but cannot otherwise update it. This aligns, at least, with how npm and cargo do their lock files; i don't think there's a command to change the version while invoking a build.

It is somewhat unfortunate that certain invocations now are two commands but it does also make the usage clearer. I think particularly with nixos-rebuild, those invocations may have been somewhat suspiciously implemented since it's plausible that those arguments got passed to multiple invocations of nix build, leading to unclear semantics (since nixos-rebuild reexecs itself after grabbing the nixos-rebuild of the new config). And furthermore, since nixos-rebuild is typically invoked as root, #47 can be avoided by updating the lock file first as a normal user.

FYI this complaint also applies to CppNix 2.22 to the best of our knowledge because we just cherry picked their redesign. hort answer: the new semantics are that normal commands can only automatically apply necessary updates to match declarations in flake.nix but cannot otherwise update it. This aligns, at least, with how npm and cargo do their lock files; i don't think there's a command to change the version while invoking a build. It is somewhat unfortunate that certain invocations now are two commands but it does also make the usage clearer. I think particularly with nixos-rebuild, those invocations may have been somewhat suspiciously implemented since it's plausible that those arguments got passed to multiple invocations of nix build, leading to unclear semantics (since nixos-rebuild reexecs itself after grabbing the nixos-rebuild of the new config). And furthermore, since nixos-rebuild is typically invoked as root, https://git.lix.systems/lix-project/lix/issues/47 can be avoided by updating the lock file first as a normal user.

benaryorg commented

2024-06-20 00:59:18 +00:00

Author

FYI this complaint also applies to CppNix 2.22 to the best of our knowledge because we just cherry picked their redesign.

Ah. I see.
I have had no run-in with any newer version of CppNix since none of them were ever marked as stable by nixpkgs IIRC (and I ran into other issues with CppNix when updating to 24.05 anyways).

This aligns, at least, with how npm and cargo do their lock files

That comparison hit me yesterday, because indeed it seems to me that the lock file is indeed the culprit here, so removing it (and adding --no-write-lock-file) should fix my issue in particular. The same way that Cargo.lock is serves reproducibility only, while the actual dependency management is handled in Cargo.toml, removing the lock file may lower reproducibility (which I want in this case) but makes stuff work with newer versions out of the box (for non-reproducible use-cases of course).
Doing that, the caching seems to still work, both for evaluations and all git sources, and with the flake I have as a shim to override the nixpkgs dependency via a follows I don't actually have to override or update the lock file of the repository anyway.
So as long as all my sources are git+https (or anything that can be properly cached, unlike the cgit tarball export, even though technically it has an etag in the HTTP response) those will remain cached, and the tarball will regrettably be refetched every single time (getting a 304 Not Modified would be nice, but I don't see where the etag could be properly cached on the client side of things to send along in the request header I guess).
Effectively, at the expense of refetching those 40MiB of data (however horrible that is on the cgit end, considering that nixpkgs is a pain for git to handle in general) I can bypass any caching for that file, so it does what I need, albeit not as generically as I had hoped.

So.… my very specific itch is scratched by deleting the lock file, which means I don't really have a use-case for this any longer where running a separate command wouldn't work better anyway.
Thanks for the explanation and context, I'll close this then.

> FYI this complaint also applies to CppNix 2.22 to the best of our knowledge because we just cherry picked their redesign. Ah. I see. I have had no run-in with any newer version of CppNix since none of them were ever marked as stable by nixpkgs IIRC (and I ran into other issues with CppNix when updating to 24.05 anyways). > This aligns, at least, with how npm and cargo do their lock files That comparison hit me yesterday, because indeed it *seems* to me that the lock file is indeed the culprit here, so removing it (and adding `--no-write-lock-file`) should fix my issue in particular. The same way that *Cargo.lock* is serves reproducibility only, while the actual dependency management is handled in *Cargo.toml*, removing the lock file may lower reproducibility (which I *want* in this case) but makes stuff work with newer versions out of the box (for non-reproducible use-cases of course). Doing that, the caching seems to still work, both for evaluations and all git sources, and with the flake I have as a shim to override the nixpkgs dependency via a follows I don't actually have to override or update the lock file of the repository anyway. So as long as all my sources are `git+https` (or anything that can be properly cached, unlike the cgit tarball export, even though technically it has an etag in the HTTP response) those will remain cached, and the tarball will regrettably be refetched every single time (getting a 304 Not Modified would be nice, but I don't see where the etag could be properly cached on the client side of things to send along in the request header I guess). Effectively, at the expense of refetching those 40MiB of data (however horrible that is on the cgit end, considering that nixpkgs is a pain for git to handle in general) I can bypass any caching for that file, so it does what I need, albeit not as generically as I had hoped. So.… my very specific itch is scratched by deleting the lock file, which means I don't really have a use-case for this any longer where running a separate command wouldn't work better anyway. Thanks for the explanation and context, I'll close this then.

benaryorg closed this issue

2024-06-20 00:59:19 +00:00

cleeyv commented

2024-07-17 18:48:28 +00:00

I had a similar problem with the --update-input flags on nixos-rebuild breaking for the nixos-upgrade service after I switched to Lix. The workaround for me was to add the required nix flake update as a preStart (aka ExecStartPre) script for the nixos-upgrade service. Here is the nixos config diff of that change for anyone else running into the same problem:

   system.autoUpgrade = {
     enable = true;
     operation = "boot";
     flake = "/etc/nixos";
     flags = [
-      "--update-input"
-      "nixpkgs"
-      "--update-input"
-      "nixpkgs-unstable"
-      "--update-input"
-      "nur"
       "-L"
     ];
     dates = "daily";
     persistent = true;
   };
 
+  systemd.services.nixos-upgrade.preStart = ''
+    nix flake update --flake /etc/nixos nixpkgs nixpkgs-unstable nur
+  '';

I had a similar problem with the `--update-input` flags on `nixos-rebuild` breaking for the nixos-upgrade service after I switched to Lix. The workaround for me was to add the required `nix flake update` as a `preStart` (aka `ExecStartPre`) script for the nixos-upgrade service. Here is the nixos config diff of that change for anyone else running into the same problem: ``` system.autoUpgrade = { enable = true; operation = "boot"; flake = "/etc/nixos"; flags = [ - "--update-input" - "nixpkgs" - "--update-input" - "nixpkgs-unstable" - "--update-input" - "nur" "-L" ]; dates = "daily"; persistent = true; }; + systemd.services.nixos-upgrade.preStart = '' + nix flake update --flake /etc/nixos nixpkgs nixpkgs-unstable nur + ''; ```

pbsds commented

2024-09-13 21:31:07 +00:00

cleeyv's solution assumes the flake checkout is writeable, which is not the case if system.autoUpgrade.flake is a url or a flake input/store path. This is the fix I ended up applying on my setup:

   system.autoUpgrade.flags = [
     "-L" # print build logs
     "--no-write-lock-file" # no write new flakelock, as the in-store flake is read-only
     # fetch new inputs
-    "--update-input" "nixpkgs-edge" "--update-input" "home-manager-edge"
-    "--update-input" "nixpkgs-2405" "--update-input" "home-manager-2405"
-    "--update-input" "nixpkgs-2311" "--update-input" "home-manager-2311"
-    "--update-input" "nix-index-database"
-    "--update-input" "nixos-hardware"
+    #"--update-input" "nixpkgs-edge" # deprecated in nix 2.22, removed in lix 2.90
+    "--refresh"
+    "--override-input" "nixpkgs-edge" "github:NixOS/nixpkgs/nixos-unstable"
+    "--override-input" "nixpkgs-2405" "github:NixOS/nixpkgs/nixos-24.05"
+    "--override-input" "nixpkgs-2311" "github:NixOS/nixpkgs/nixos-23.11"
+    "--override-input" "home-manager-edge" "github:nix-community/home-manager/master"
+    "--override-input" "home-manager-2405" "github:nix-community/home-manager/release-24.05"
+    "--override-input" "home-manager-2311" "github:nix-community/home-manager/release-23.11"
+    "--override-input" "nix-index-database" "github:Mic92/nix-index-database"
+    "--override-input" "nixos-hardware" "github:NixOS/nixos-hardware"
   ];

It's a bit annoying having to repeat the flake inputs, but hey it works! If the input urls were accessible from inside the the flake, then I could for example have written "--override-input" "nixpkgs-edge" inputs.nixpkgs-edge.sourceInfo.url

BTW, it seems that --update-input is still only deprecated on nix 2.24.6. Did lix drop it early by mistake or by intention?

cleeyv's solution assumes the flake checkout is writeable, which is not the case if `system.autoUpgrade.flake` is a url or a flake input/store path. This is the fix I ended up applying on my setup: ```patch system.autoUpgrade.flags = [ "-L" # print build logs "--no-write-lock-file" # no write new flakelock, as the in-store flake is read-only # fetch new inputs - "--update-input" "nixpkgs-edge" "--update-input" "home-manager-edge" - "--update-input" "nixpkgs-2405" "--update-input" "home-manager-2405" - "--update-input" "nixpkgs-2311" "--update-input" "home-manager-2311" - "--update-input" "nix-index-database" - "--update-input" "nixos-hardware" + #"--update-input" "nixpkgs-edge" # deprecated in nix 2.22, removed in lix 2.90 + "--refresh" + "--override-input" "nixpkgs-edge" "github:NixOS/nixpkgs/nixos-unstable" + "--override-input" "nixpkgs-2405" "github:NixOS/nixpkgs/nixos-24.05" + "--override-input" "nixpkgs-2311" "github:NixOS/nixpkgs/nixos-23.11" + "--override-input" "home-manager-edge" "github:nix-community/home-manager/master" + "--override-input" "home-manager-2405" "github:nix-community/home-manager/release-24.05" + "--override-input" "home-manager-2311" "github:nix-community/home-manager/release-23.11" + "--override-input" "nix-index-database" "github:Mic92/nix-index-database" + "--override-input" "nixos-hardware" "github:NixOS/nixos-hardware" ]; ``` It's a bit annoying having to repeat the flake inputs, but hey it works! If the input urls were accessible from inside the the flake, then I could for example have written `"--override-input" "nixpkgs-edge" inputs.nixpkgs-edge.sourceInfo.url` BTW, it seems that `--update-input` is still only deprecated on nix 2.24.6. Did lix drop it early by mistake or by intention?

jade commented

2024-09-13 23:34:48 +00:00

Owner

The way of doing auto update mentioned above is .... it's a choice. I would not implement it this way: this means that there is no way to have any idea what versions of anything that the machine is running, since override-input does not write the lock file. So it eliminates all the reproducibility enabled by flakes in NixOS configurations.

The real bug here is that nix-update is broken and, I think, at least, not very actively maintained?

Nix broke the old command with no diagnostic a while ago and my bad memory of it was that we backported their change with an added diagnostic to tell you to use the new thing, because writing lock files from arbitrary other Nix commands is kind of sketchy! It's possible that someone wrote a thing that corrects that, which might be reasonable to backport, but otoh, we kind of already accepted all the pain caused by simply breaking it, and the old way was definitely a bad idea. The way of doing auto update mentioned above is .... it's a choice. I would not implement it this way: this means that there is no way to have any idea what versions of anything that the machine is running, since override-input does not write the lock file. So it eliminates all the reproducibility enabled by flakes in NixOS configurations. The real bug here is that nix-update is broken and, I think, at least, not very actively maintained?

pbsds commented

2024-09-15 16:28:55 +00:00

EDIT: i nerd-sniped myself

  environment.etc."current-system-flake-inputs.json".source
    = pkgs.writers.writeJSON "flake-inputs.json" (
      lib.flip lib.mapAttrs inputs (name: input:
        # inputs.*.sourceInfo sans outPath, since writeJSON will otherwise serialize sourceInfo like a derivation
        lib.removeAttrs (input.sourceInfo or {}) [ "outPath" ] 
          // { store-path = input.outPath; } # comment this line if you don't want to retain a store reference to the flake inputs
      )
    );

autoUpgrade not retaining the flake lock file is a known problem, and various comments in https://github.com/NixOS/nix/issues/6895 discuss a workaround where they retain store references to the flake inputs. It should be easy to make a json dump of the flake `inputs.*.sourceInfo` and put it in `/etc` for example. EDIT: i nerd-sniped myself ```nix environment.etc."current-system-flake-inputs.json".source = pkgs.writers.writeJSON "flake-inputs.json" ( lib.flip lib.mapAttrs inputs (name: input: # inputs.*.sourceInfo sans outPath, since writeJSON will otherwise serialize sourceInfo like a derivation lib.removeAttrs (input.sourceInfo or {}) [ "outPath" ] // { store-path = input.outPath; } # comment this line if you don't want to retain a store reference to the flake inputs ) ); ```

No milestone

No project

No assignees

4 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#400

No description provided.