Store path substitution very badly handles unreachable/broken substituters #339

Open
opened 2024-05-22 23:58:34 +00:00 by delroth · 0 comments
Member

Describe the bug

Certain errors raised by substituters end up being "fatal" and causing Lix to not try further configured substituters or local builds. This needs investigation to figure out in which exact situation this happens, but here's an example log from my system with "substituters = s3://some-cache-which-was-down/ https://cache.nixos.org/":

$ nix-shell
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms
error: AWS error fetching 'gchwj6dfag4vkbbvm01338zs8pkai7n6.narinfo': curlCode: 7, Couldn't connect to server

Steps To Reproduce

To be determined.

Expected behavior

More concise error handling and better recovery: attempt fetching from other substituters as well as locally building.

nix --version output

nix (Lix, like Nix) 2.90.0-lixpre20240506-106b959

Additional context

This Nix PR seems related but was ignored by the Nix devteam and never merged: https://github.com/NixOS/nix/pull/8983

## Describe the bug Certain errors raised by substituters end up being "fatal" and causing Lix to not try further configured substituters or local builds. This needs investigation to figure out in which exact situation this happens, but here's an example log from my system with "substituters = s3://some-cache-which-was-down/ https://cache.nixos.org/": ``` $ nix-shell AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms error: AWS error fetching 'gchwj6dfag4vkbbvm01338zs8pkai7n6.narinfo': curlCode: 7, Couldn't connect to server ``` ## Steps To Reproduce To be determined. ## Expected behavior More concise error handling and better recovery: attempt fetching from other substituters as well as locally building. ## `nix --version` output nix (Lix, like Nix) 2.90.0-lixpre20240506-106b959 ## Additional context This Nix PR seems related but was ignored by the Nix devteam and never merged: https://github.com/NixOS/nix/pull/8983
delroth added the
bug
label 2024-05-22 23:58:34 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#339
No description provided.