lix-project/lix

Fork 54

Show Download Progress (at least for large downloads) #737

New issue

Open

opened 2025-03-14 10:38:50 +00:00 by dwt · 9 comments

dwt commented

2025-03-14 10:38:50 +00:00

When lix downloads large files, for example for closed source packages like ut1999, then it looks like the download just hangs. TUI Shot:

❯ NIXPKGS_ALLOW_UNFREE=1 nix run --impure  nixpkgs-unstable#ut1999 --refresh --print-build-logs
UT_GOTY_CD1.iso> trying https://archive.org/download/ut-goty/UT_GOTY_CD1.iso
UT_GOTY_CD1.iso>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
UT_GOTY_CD1.iso>                                  Dload  Upload   Total   Spent    Left  Speed
UT_GOTY_CD1.iso>   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[1/1/4 built, 26 copied (67.6 MiB), 12.7 MiB DL] building UT_GOTY_CD1.iso

Describe the solution you'd like

I would like the download progress to show, so one can see that the download is progressing.

Some solutions I can see:

Update the download stats from curl shown above while downloading
Show the download progress (bar?) as part of the nix status line somehow
Enhance the (json) log messages to carry progress, so tools like nix-output-monitor can show the download progress.

There for sure are other / better methods to achieve this, but it would be really good to not seem like nix just hangs.

Describe alternatives you've considered

nix-output-monitor often provides great visualization, but not in this case.

## Is your feature request related to a problem? Please describe. When lix downloads large files, for example for closed source packages like [ut1999](https://search.nixos.org/packages?channel=24.11&show=ut1999&from=0&size=50&sort=relevance&type=packages&query=ut1999), then it looks like the download just hangs. TUI Shot: ```log ❯ NIXPKGS_ALLOW_UNFREE=1 nix run --impure nixpkgs-unstable#ut1999 --refresh --print-build-logs UT_GOTY_CD1.iso> trying https://archive.org/download/ut-goty/UT_GOTY_CD1.iso UT_GOTY_CD1.iso> % Total % Received % Xferd Average Speed Time Time Time Current UT_GOTY_CD1.iso> Dload Upload Total Spent Left Speed UT_GOTY_CD1.iso> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [1/1/4 built, 26 copied (67.6 MiB), 12.7 MiB DL] building UT_GOTY_CD1.iso ``` ## Describe the solution you'd like I would like the download progress to show, so one can see that the download is progressing. Some solutions I can see: - Update the download stats from curl shown above while downloading - Show the download progress (bar?) as part of the nix status line somehow - Enhance the (json) log messages to carry progress, so tools like nix-output-monitor can show the download progress. There for sure are other / better methods to achieve this, but it would be really good to not seem like nix just hangs. ## Describe alternatives you've considered nix-output-monitor often provides great visualization, but not in this case.

pennae commented

2025-03-14 11:03:50 +00:00

Owner

that's actually not even a lix problem, but a nixpkgs problem. the download you're seeing isn't done by lix itself but from inside a fixed-output derivation, and that derivation doesn't give progress reports on its stdout (which is all lix can observe about this download). we agree that there could be better integration of progress reporting in lix with this, but the easier fix would be to ask nixpkgs about adding a bit of progress reporting to their FOD fetchers

dwt commented

2025-03-14 11:11:55 +00:00

Author

As far as I can tell the download is done by builtin.fetchurl() I was under the impression that is something that lix provides. Is that not the case?

As far as I can tell the download is done by `builtin.fetchurl()` I was under the impression that is something that lix provides. Is that not the case?

pennae commented

2025-03-14 11:14:23 +00:00

Owner

builtins.fetchurl is done using curl, but the fetcher used here isn't that. what you're seeing is nixpkgs' version of fetchurl that runs curl inside derivation sandbox (hence the UT_GOTY_CD1.iso> prefix in log lines); our builtin fetchers cannot print statistics in the shape you're seeing here

`builtins.fetchurl` is done using curl, but the fetcher used here isn't that. what you're seeing is nixpkgs' version of fetchurl that runs curl inside derivation sandbox (hence the `UT_GOTY_CD1.iso>` prefix in log lines); our builtin fetchers cannot print statistics in the shape you're seeing here

👍 1

dwt commented

2025-03-14 12:15:34 +00:00

Author

Is there any infrastructure in lix that allows nixpkgs to actually show progress? I mean, can this actually be solved by nixpkgs and if yes, how?

dwt commented

2025-03-14 12:21:12 +00:00

Author

See https://github.com/NixOS/nixpkgs/issues/389794

❤️ 1

pennae commented

2025-03-14 12:33:15 +00:00

Owner

@dwt wrote in #737 (comment):

Is there any infrastructure in lix that allows nixpkgs to actually show progress? I mean, can this actually be solved by nixpkgs and if yes, how?

there is none at this point (though we conceivably could and should add some in the future). for now the nixpkgs FOD fetchers can output some status information regularly (e.g. every second) though, which will show up in the build logs and multiline log mode (albeit not in the lix download meter).

@dwt wrote in https://git.lix.systems/lix-project/lix/issues/737#issuecomment-9482: > Is there any infrastructure in lix that allows nixpkgs to actually show progress? I mean, can this actually be solved by nixpkgs and if yes, how? there is none at this point (though we conceivably could *and should* add some in the future). for now the nixpkgs FOD fetchers can output some status information regularly (e.g. every second) though, which will show up in the build logs and multiline log mode (albeit not in the lix download meter).

jade commented

2025-03-14 15:57:23 +00:00

Owner

I think in theory they could emit progress events to NIX_LOG_FD which would drive the progress bar? idk how they'd implement it in nixpkgs though.

pennae commented

2025-03-14 16:08:59 +00:00

Owner

that requires an activity ID though, and there's currently no way to generate one inside a build sandbox that is guaranteed to not collide with other ids (although the current mechanism does allow for "very likely no collision"s). even when you have such an id though you'd want to parent it to the build activity, which we cannot do :(