Builds shouldn't fail if a subsitutor is returning 500 #1161

Open
opened 2026-03-19 13:12:32 +00:00 by jappeace · 1 comment

Describe the bug

I couldn't disable my substitor config even because I couldn't switch!
The problem was that my server ran an ancient nix daemon (2023) which is incompatible with recent lix.

I'd expect it to just continue after getting 500 errors from a substitor

🙂jappie@panorama-tower 🕙2026-03-19 13:37:07
✦ ~ ❯ nixos test
++ cd /linux-config/
+++ nix-instantiate --eval -E 'let sources = import ./npins; in sources.nixpkgs.outPath'
+++ jq -r .
++ nixpkgs=/nix/store/wdqrnd6fkxdsf4xn2kkfa8f42shfw0i3-source
++ sudo nixos-rebuild test -I nixpkgs=/nix/store/wdqrnd6fkxdsf4xn2kkfa8f42shfw0i3-source -I nixos-config=/etc/nixos/configuration.nix --fast
[sudo] wachtwoord voor jappie: 
Sorry, probeer het opnieuw.
[sudo] wachtwoord voor jappie: 
nixos-rebuild: warning: --fast is deprecated, use --no-reexec instead
building the system configuration...
fetching git input 'git+https://github.com/jappeace/agda-mode'
warning: error: unable to download 'https://nix-cache.jappie.me/q4f5h9fihip1za87cjnwxvmwlx4dc1wy.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 313 ms (attempt 1/5)
warning: error: unable to download 'https://nix-cache.jappie.me/q4f5h9fihip1za87cjnwxvmwlx4dc1wy.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 632 ms (attempt 2/5)
warning: error: unable to download 'https://nix-cache.jappie.me/npwkyxy3vs85ibqnw1xzslaxy9khvbdc.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 281 ms (attempt 1/5)
warning: error: unable to download 'https://nix-cache.jappie.me/vqzwiangd9q8zk2ydy30y5zizdjwi55s.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 335 ms (attempt 1/5)
warning: error: unable to download 'https://nix-cache.jappie.me/8jifb3sm1r84dwg9n1fjrhrrwwv1g719.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 346 ms (attempt 1/5)
warning: error: unable to download 'https://nix-cache.jappie.me/npwkyxy3vs85ibqnw1xzslaxy9khvbdc.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 519 ms (attempt 2/5)
warning: error: unable to download 'https://nix-cache.jappie.me/vqzwiangd9q8zk2ydy30y5zizdjwi55s.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 518 ms (attempt 2/5)
warning: error: unable to download 'https://nix-cache.jappie.me/8jifb3sm1r84dwg9n1fjrhrrwwv1g719.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 516 ms (attempt 2/5)
warning: error: unable to download 'https://nix-cache.jappie.me/q4f5h9fihip1za87cjnwxvmwlx4dc1wy.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 1108 ms (attempt 3/5)
warning: error: unable to download 'https://nix-cache.jappie.me/npwkyxy3vs85ibqnw1xzslaxy9khvbdc.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 1038 ms (attempt 3/5)
warning: error: unable to download 'https://nix-cache.jappie.me/vqzwiangd9q8zk2ydy30y5zizdjwi55s.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 1187 ms (attempt 3/5)
warning: error: unable to download 'https://nix-cache.jappie.me/8jifb3sm1r84dwg9n1fjrhrrwwv1g719.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 2340 ms (attempt 4/5)
warning: error: unable to download 'https://nix-cache.jappie.me/npwkyxy3vs85ibqnw1xzslaxy9khvbdc.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 2429 ms (attempt 4/5)
warning: error: unable to download 'https://nix-cache.jappie.me/8jifb3sm1r84dwg9n1fjrhrrwwv1g719.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 2404 ms (attempt 4/5)
warning: error: unable to download 'https://nix-cache.jappie.me/vqzwiangd9q8zk2ydy30y5zizdjwi55s.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error; retrying in 2121 ms (attempt 4/5)
error: unable to download 'https://nix-cache.jappie.me/q4f5h9fihip1za87cjnwxvmwlx4dc1wy.narinfo': HTTP error 500 ()

       response body:

       Internal Server Error
┏━ 1 Errors: 
┃ error: unable to download 'https://nix-cache.jappie.me/q4f5h9fihip1za87cjnwxvmwlx4dc1wy.narinfo': HTTP error 500 ()
┃ 
┃        response body:
┃ 
┃        Internal Server Error
┣━━━                                                             
┗━ ∑ ⚠ Exited with 1 errors reported by nix at 13:38:00 after 27s
Command 'nix-build '<nixpkgs/nixos>' --attr config.system.build.toplevel --no-out-link --include nixpkgs=/nix/store/wdqrnd6fkxdsf4xn2kkfa8f42shfw0i3-source --include nixos-config=/etc/nixos/configuration.nix' returned non-zero exit status 1.
[ble: exit 1]

Steps To Reproduce

you cna reproduce by making a faulty substitor with netcat:

while true; do echo -e "HTTP/1.1 500 Internal Server Error\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" | nc -l 80; done

Then you just set your normal nix config to use that substitor and it'll get 500 errors on any substitute attempt

Expected behavior

It should figure out the substitor is faulty and at least disable it for the current build.
Having a single substitor fail breaking my entire build system isn't acceptable.

nix --version output


🙂jappie@panorama-tower 🕙2026-03-19 14:11:45
✦ ~ ❯ nix --version
nix-output-monitor 2.1.8
nix (Lix, like Nix) 2.93.3
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/jappie/.config/nix/nix.conf:/home/jappie/.config/kdedefaults/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/jappie/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/jappie/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/jappie/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/kg6pv5f01vj5w6k7scy2zfvnkcgbifcw-lix-2.93.3/share

Additional context

I understand these situations are rare but they do happen!

## Describe the bug I couldn't disable my substitor config even because I couldn't switch! The problem was that my server ran an ancient nix daemon (2023) which is incompatible with recent lix. I'd expect it to just continue after getting 500 errors from a substitor ``` 🙂jappie@panorama-tower 🕙2026-03-19 13:37:07 ✦ ~ ❯ nixos test ++ cd /linux-config/ +++ nix-instantiate --eval -E 'let sources = import ./npins; in sources.nixpkgs.outPath' +++ jq -r . ++ nixpkgs=/nix/store/wdqrnd6fkxdsf4xn2kkfa8f42shfw0i3-source ++ sudo nixos-rebuild test -I nixpkgs=/nix/store/wdqrnd6fkxdsf4xn2kkfa8f42shfw0i3-source -I nixos-config=/etc/nixos/configuration.nix --fast [sudo] wachtwoord voor jappie: Sorry, probeer het opnieuw. [sudo] wachtwoord voor jappie: nixos-rebuild: warning: --fast is deprecated, use --no-reexec instead building the system configuration... fetching git input 'git+https://github.com/jappeace/agda-mode' warning: error: unable to download 'https://nix-cache.jappie.me/q4f5h9fihip1za87cjnwxvmwlx4dc1wy.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 313 ms (attempt 1/5) warning: error: unable to download 'https://nix-cache.jappie.me/q4f5h9fihip1za87cjnwxvmwlx4dc1wy.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 632 ms (attempt 2/5) warning: error: unable to download 'https://nix-cache.jappie.me/npwkyxy3vs85ibqnw1xzslaxy9khvbdc.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 281 ms (attempt 1/5) warning: error: unable to download 'https://nix-cache.jappie.me/vqzwiangd9q8zk2ydy30y5zizdjwi55s.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 335 ms (attempt 1/5) warning: error: unable to download 'https://nix-cache.jappie.me/8jifb3sm1r84dwg9n1fjrhrrwwv1g719.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 346 ms (attempt 1/5) warning: error: unable to download 'https://nix-cache.jappie.me/npwkyxy3vs85ibqnw1xzslaxy9khvbdc.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 519 ms (attempt 2/5) warning: error: unable to download 'https://nix-cache.jappie.me/vqzwiangd9q8zk2ydy30y5zizdjwi55s.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 518 ms (attempt 2/5) warning: error: unable to download 'https://nix-cache.jappie.me/8jifb3sm1r84dwg9n1fjrhrrwwv1g719.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 516 ms (attempt 2/5) warning: error: unable to download 'https://nix-cache.jappie.me/q4f5h9fihip1za87cjnwxvmwlx4dc1wy.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 1108 ms (attempt 3/5) warning: error: unable to download 'https://nix-cache.jappie.me/npwkyxy3vs85ibqnw1xzslaxy9khvbdc.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 1038 ms (attempt 3/5) warning: error: unable to download 'https://nix-cache.jappie.me/vqzwiangd9q8zk2ydy30y5zizdjwi55s.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 1187 ms (attempt 3/5) warning: error: unable to download 'https://nix-cache.jappie.me/8jifb3sm1r84dwg9n1fjrhrrwwv1g719.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 2340 ms (attempt 4/5) warning: error: unable to download 'https://nix-cache.jappie.me/npwkyxy3vs85ibqnw1xzslaxy9khvbdc.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 2429 ms (attempt 4/5) warning: error: unable to download 'https://nix-cache.jappie.me/8jifb3sm1r84dwg9n1fjrhrrwwv1g719.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 2404 ms (attempt 4/5) warning: error: unable to download 'https://nix-cache.jappie.me/vqzwiangd9q8zk2ydy30y5zizdjwi55s.narinfo': HTTP error 500 () response body: Internal Server Error; retrying in 2121 ms (attempt 4/5) error: unable to download 'https://nix-cache.jappie.me/q4f5h9fihip1za87cjnwxvmwlx4dc1wy.narinfo': HTTP error 500 () response body: Internal Server Error ┏━ 1 Errors: ┃ error: unable to download 'https://nix-cache.jappie.me/q4f5h9fihip1za87cjnwxvmwlx4dc1wy.narinfo': HTTP error 500 () ┃ ┃ response body: ┃ ┃ Internal Server Error ┣━━━ ┗━ ∑ ⚠ Exited with 1 errors reported by nix at 13:38:00 after 27s Command 'nix-build '<nixpkgs/nixos>' --attr config.system.build.toplevel --no-out-link --include nixpkgs=/nix/store/wdqrnd6fkxdsf4xn2kkfa8f42shfw0i3-source --include nixos-config=/etc/nixos/configuration.nix' returned non-zero exit status 1. [ble: exit 1] ``` ## Steps To Reproduce you cna reproduce by making a faulty substitor with netcat: ``` while true; do echo -e "HTTP/1.1 500 Internal Server Error\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" | nc -l 80; done ``` Then you just set your normal nix config to use that substitor and it'll get 500 errors on any substitute attempt ## Expected behavior It should figure out the substitor is faulty and at least disable it for the current build. Having a single substitor fail breaking my entire build system isn't acceptable. ## `nix --version` output ``` 🙂jappie@panorama-tower 🕙2026-03-19 14:11:45 ✦ ~ ❯ nix --version nix-output-monitor 2.1.8 nix (Lix, like Nix) 2.93.3 System type: x86_64-linux Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /home/jappie/.config/nix/nix.conf:/home/jappie/.config/kdedefaults/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/jappie/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/jappie/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/jappie/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/kg6pv5f01vj5w6k7scy2zfvnkcgbifcw-lix-2.93.3/share ``` ## Additional context I understand these situations are rare but they do happen!
Owner

running your reproducer on 2.95 we get this instead:

❯ nix run --store /tmp/store nixpkgs#hello --substituters 'http://localhost:8080 https://cache.nixos.org' -j0
warning: error: unable to download 'http://localhost:8080/nix-cache-info': HTTP error 500 (Internal Server Error)

       response body:

       ; retrying in 530 ms (attempt 1/5)
warning: error: unable to download 'http://localhost:8080/nix-cache-info': HTTP error 500 (Internal Server Error)

       response body:

       ; retrying in 1051 ms (attempt 2/5)
warning: error: unable to download 'http://localhost:8080/nix-cache-info': HTTP error 500 (Internal Server Error)

       response body:

       ; retrying in 1974 ms (attempt 3/5)
warning: error: unable to download 'http://localhost:8080/nix-cache-info': HTTP error 500 (Internal Server Error)

       response body:

       ; retrying in 4054 ms (attempt 4/5)
warning: Failed to setup the substituter at URI 'http://localhost:8080': error: unable to download 'http://localhost:8080/nix-cache-info': HTTP error 500 (Internal Server Error)

                response body:

                
Hello, world!

extending reproducer slightly to produce a cache with valid metadata by having

# srv.sh
read line                                                                                                                                                                                                           
echo "$line" >&2                                                                                                                                                                                                    
if [[ "$line" = *nix-cache-info* ]]; then                                                                                                                                                                           
    echo -en 'HTTP/1.1 200\r\nConnection: close\r\n\r\nStoreDir: /nix/store\n'                                                                                                                                      
else                                                                                                                                                                                                                
    echo -en 'HTTP/1.1 500 Internal Server Error\r\nConnection: close\r\n\r\n'                                                                                                                                      
fi | tee /dev/stderr

# server
socat tcp-listen:8080,fork system:'bash srv.sh'

# test command
nix run --store /tmp/store nixpkgs#hello --substituters 'http://localhost:8080 https://cache.nixos.org' -j0

does not cause a failure, but a long list of retries (too long, we do consider that a bug):

warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error)

       response body:

       ; retrying in 546 ms (attempt 1/5)
warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error)

       response body:

       ; retrying in 921 ms (attempt 2/5)
warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error)

       response body:

       ; retrying in 1893 ms (attempt 3/5)
warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error)

       response body:

       ; retrying in 3984 ms (attempt 4/5)
warning: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error)

         response body:

         
warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error)

       response body:

       ; retrying in 376 ms (attempt 1/5)
warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error)

       response body:

       ; retrying in 983 ms (attempt 2/5)
warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error)

       response body:

... repeated a lot for more retries and narinfos ...

       ; retrying in 3922 ms (attempt 4/5)
warning: unable to download 'http://localhost:8080/pkphs076yz5ajnqczzj0588n6miph269.narinfo': HTTP error 500 (Internal Server Error)

         response body:

         
Hello, world!

so we'd consider your problem fixed, but we probably should not be retrying the same substituter quite this often if it's failing

running your reproducer on 2.95 we get this instead: ``` ❯ nix run --store /tmp/store nixpkgs#hello --substituters 'http://localhost:8080 https://cache.nixos.org' -j0 warning: error: unable to download 'http://localhost:8080/nix-cache-info': HTTP error 500 (Internal Server Error) response body: ; retrying in 530 ms (attempt 1/5) warning: error: unable to download 'http://localhost:8080/nix-cache-info': HTTP error 500 (Internal Server Error) response body: ; retrying in 1051 ms (attempt 2/5) warning: error: unable to download 'http://localhost:8080/nix-cache-info': HTTP error 500 (Internal Server Error) response body: ; retrying in 1974 ms (attempt 3/5) warning: error: unable to download 'http://localhost:8080/nix-cache-info': HTTP error 500 (Internal Server Error) response body: ; retrying in 4054 ms (attempt 4/5) warning: Failed to setup the substituter at URI 'http://localhost:8080': error: unable to download 'http://localhost:8080/nix-cache-info': HTTP error 500 (Internal Server Error) response body: Hello, world! ``` extending reproducer slightly to produce a cache with valid metadata by having ```sh # srv.sh read line echo "$line" >&2 if [[ "$line" = *nix-cache-info* ]]; then echo -en 'HTTP/1.1 200\r\nConnection: close\r\n\r\nStoreDir: /nix/store\n' else echo -en 'HTTP/1.1 500 Internal Server Error\r\nConnection: close\r\n\r\n' fi | tee /dev/stderr # server socat tcp-listen:8080,fork system:'bash srv.sh' # test command nix run --store /tmp/store nixpkgs#hello --substituters 'http://localhost:8080 https://cache.nixos.org' -j0 ``` does not cause a failure, but a long list of retries (too long, we do consider that a bug): ``` warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error) response body: ; retrying in 546 ms (attempt 1/5) warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error) response body: ; retrying in 921 ms (attempt 2/5) warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error) response body: ; retrying in 1893 ms (attempt 3/5) warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error) response body: ; retrying in 3984 ms (attempt 4/5) warning: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error) response body: warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error) response body: ; retrying in 376 ms (attempt 1/5) warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error) response body: ; retrying in 983 ms (attempt 2/5) warning: error: unable to download 'http://localhost:8080/mi08jhbcjib1i1kgvbd0fxn2yrnzdv4a.narinfo': HTTP error 500 (Internal Server Error) response body: ... repeated a lot for more retries and narinfos ... ; retrying in 3922 ms (attempt 4/5) warning: unable to download 'http://localhost:8080/pkphs076yz5ajnqczzj0588n6miph269.narinfo': HTTP error 500 (Internal Server Error) response body: Hello, world! ``` so we'd consider your problem fixed, but we probably should not be retrying the same substituter quite this often if it's failing
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lix-project/lix#1161
No description provided.