lix-project/lix
17
37
Fork 14
Code Issues 286 Code Review (Gerrit) Projects 1 Releases Wiki Activity

Store path substitution very badly handles unreachable/broken substituters #339

New issue
Open
opened 2024-05-22 23:58:34 +00:00 by delroth · 0 comments
delroth commented 2024-05-22 23:58:34 +00:00
Member
Copy link

Describe the bug

Certain errors raised by substituters end up being "fatal" and causing Lix to not try further configured substituters or local builds. This needs investigation to figure out in which exact situation this happens, but here's an example log from my system with "substituters = s3://some-cache-which-was-down/ https://cache.nixos.org/":

$ nix-shell
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms
AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms
error: AWS error fetching 'gchwj6dfag4vkbbvm01338zs8pkai7n6.narinfo': curlCode: 7, Couldn't connect to server

Steps To Reproduce

To be determined.

Expected behavior

More concise error handling and better recovery: attempt fetching from other substituters as well as locally building.

nix --version output

nix (Lix, like Nix) 2.90.0-lixpre20240506-106b959

Additional context

This Nix PR seems related but was ignored by the Nix devteam and never merged: https://github.com/NixOS/nix/pull/8983

## Describe the bug Certain errors raised by substituters end up being "fatal" and causing Lix to not try further configured substituters or local builds. This needs investigation to figure out in which exact situation this happens, but here's an example log from my system with "substituters = s3://some-cache-which-was-down/ https://cache.nixos.org/": ``` $ nix-shell AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 0 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 50 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 100 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 1600 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 3200 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 6400 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms AWS error '' (curlCode: 7, Couldn't connect to server), will retry in 12800 ms error: AWS error fetching 'gchwj6dfag4vkbbvm01338zs8pkai7n6.narinfo': curlCode: 7, Couldn't connect to server ``` ## Steps To Reproduce To be determined. ## Expected behavior More concise error handling and better recovery: attempt fetching from other substituters as well as locally building. ## `nix --version` output nix (Lix, like Nix) 2.90.0-lixpre20240506-106b959 ## Additional context This Nix PR seems related but was ignored by the Nix devteam and never merged: https://github.com/NixOS/nix/pull/8983
delroth added the
bug
 label 2024-05-22 23:58:34 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
git-series/activity-cleanup
release-2.90
sb/tom-hubrecht/primops.cc-v2
sb/bacchanalia/cache-strlen%topic=substr-perf
sb/arcuru/commit-msg
sb/qyriad/beta-1
sb/qyriad/2.90-beta.0
rbt/pre-commit-update
sb/rbt/test-branch
sb/rbt/justfile
sb/rbt/repl-overlays
sb/fuck/what
sb/rbt/pre-commit
sb/rbt/makeflags
sb/pennae/io-rewrite
sb/qyriad/maint-meson
sb/qyriad/maint/meson
sb/puck/meson-c-api
sb/pennae/parser-rewrite
sb/jade/burn-it-all
dependabot/github_actions/zeebe-io/backport-action-2.4.1
dependabot/github_actions/cachix/install-nix-action-25
dependabot/github_actions/actions/labeler-5
dependabot/github_actions/cachix/cachix-action-14
2.90.0-rc1
2.90-beta.1
2.90-beta.0
Labels
Clear labels   
Area/build-packaging

regarding the building or packaging of Lix itself

  
Area/evaluator

libexpr and anything else related to the evaluation of Nix expressions

  
Area/flakes

   
Area/profiles

profiles: nix-env, nix profile, and installer flavors all alike

  
Area/remote-builds

remote builds

  
Area/repl

nix repl

  
Area/store

libstore and anything else relating to the Nix store

  
bug

it's a bug

  
Cross Compilation

   
devx

development experience issues

  
docs

   
Downstream Dependents

regarding an external tool's usage with Lix (e.g. nix-eval-jobs)

  
E/easy

bug is relatively simple and could simply be fixed with some minor effort

  
E/hard

this issue should probably be solved with close involvement of an expert

  
E/help wanted

maintainers may not get to this soon, but we want it fixed

  
E/reproducible

This bug is reproducible and obvious.

  
E/requires rearchitecture

this bug cannot be fixed properly without fixing a known architectural flaw

  
imported

Issue imported from upstream nix (please do not remove this label, it will cause dupes if import is rerun)

  
Needs Langver

We can't get to this until we implement language versioning

  
OS/Linux

Affects specifically Lix on Linux

  
OS/macOS

Affects specifically Lix on macOS

  
performance

make lix go faster

  
regression

This is a regression since some previous version of Lix

  
release-blocker

Blocks public release

  
RFD

Request for discussion

  
stability

Fixing this makes the software more stable

  
Status
blocked
Blocked on something

  
Status
invalid
not our bug, duplicate, similar

  
Status
postponed
closed but should maybe look at later (generally with S-wontfix)

  
Status
wontfix
We chose not to change this (see issue comments for more detail).

  
testing

Tests that should be added

  
ux

makes the user experience better :)

No labels
Area/build-packaging
Area/evaluator
Area/flakes
Area/profiles
Area/remote-builds
Area/repl
Area/store
bug
Cross Compilation
devx
docs
Downstream Dependents
E/easy
E/hard
E/help wanted
E/reproducible
E/requires rearchitecture
imported
Needs Langver
OS/Linux
OS/macOS
performance
regression
release-blocker
RFD
stability
Status
blocked
Status
invalid
Status
postponed
Status
wontfix
testing
ux
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#339
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?