lix-project/lix
19
97
Fork 70
Code Issues 543 Code Review (Gerrit) Projects 8 Releases Packages Wiki Activity

Failed writing received data to disk/application during substitution (HTTP2 issue) #1222

New issue
Open
opened 2026-06-01 10:52:57 +00:00 by mweinelt · 2 comments
mweinelt commented 2026-06-01 10:52:57 +00:00
Member
Copy link

Describe the bug

The build farm I maintain substitutes from cache.nixos-cuda.org for supporting machines with cudaSupport enabled.

Regularly when substituting onnxruntime from that cache I run into a curl error 23.

This reproduces not only on our build farm, but also on my desktop machine, but not on my Rock 5B. All use nixos-unstable with a Lix version from last week or newer.

Steps To Reproduce

  1. Configure `cache.nixos-cuda.org
  2. nix-store --realise /nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug

Expected behavior

A clear and concise description of what you expected to happen.

nix --version output

nix (Lix, like Nix) 2.96.0-dev-pre20260531-dev_003946f
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /root/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/root/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/root/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/root/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/ig7j7ky4df4ar83r0cv560r9rj40qaqw-lix-2.96.0-dev-pre20260531-dev_003946f/share

curl -V output

curl 8.20.0 (x86_64-pc-linux-gnu) libcurl/8.20.0 OpenSSL/3.6.2 zlib/1.3.2 brotli/1.2.0 zstd/1.5.7 libidn2/2.3.8 libpsl/0.21.5 libssh2/1.11.1 nghttp2/1.69.0 ngtcp2/1.22.1 nghttp3/1.15.0 mit-krb5/1.22.1
Release-Date: 2026-04-29
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt mqtts pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Additional context

# nix-store --realise /nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug -vvvvvvvv
performing daemon worker op: 40
querying info about missing paths...
checking substituter 'https://cache.nixos.org' for path '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug'
checking substituter 'https://nix-community.cachix.org' for path '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug'
checking substituter 'https://cache.nixos-cuda.org' for path '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug'
this path will be fetched (1091.10 MiB download, 1091.10 MiB unpacked):
  /nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug
performing daemon worker op: 9
running command: '/nix/store/d8i768l3mi5ynrzhk5j15hbpwg4y226n-lix-2.96.0-dev-pre20260523-dev_b113be5/libexec/lix/check-namespace-support' '16'
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': created
entered goal loop
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': init
acquired write lock on '/nix/var/nix/temproots/3566132'
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter
substitution of '/nix/store/7lmd80pj8j2yw1l54xsnhprd0a3sagv9-onnxruntime-1.26.0': created
substitution of '/nix/store/fb67sl8v628y15xqpbbarvzpz5kf3f8y-source': created
substitution of '/nix/store/7lmd80pj8j2yw1l54xsnhprd0a3sagv9-onnxruntime-1.26.0': init
substitution of '/nix/store/fb67sl8v628y15xqpbbarvzpz5kf3f8y-source': init
substitution of '/nix/store/7lmd80pj8j2yw1l54xsnhprd0a3sagv9-onnxruntime-1.26.0': done
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': waitee 'substitution of '/nix/store/7lmd80pj8j2yw1l54xsnhprd0a3sagv9-onnxruntime-1.26.0'' done; 1 left
substitution of '/nix/store/fb67sl8v628y15xqpbbarvzpz5kf3f8y-source': done
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': waitee 'substitution of '/nix/store/fb67sl8v628y15xqpbbarvzpz5kf3f8y-source'' done; 0 left
substitution of '/nix/store/7lmd80pj8j2yw1l54xsnhprd0a3sagv9-onnxruntime-1.26.0': goal destroyed
substitution of '/nix/store/fb67sl8v628y15xqpbbarvzpz5kf3f8y-source': goal destroyed
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': all references realised
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying to run
copying path '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug' from 'https://cache.nixos-cuda.org'...
downloading 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r'...
starting download of https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r
curl: Could not find host cache.nixos-cuda.org in the /etc/nix/netrc file; using defaults
curl:   Trying [2a01:4f9:3071:1108::1]:443...
curl: Host cache.nixos-cuda.org:443 was resolved.
curl: IPv6: 2a01:4f9:3071:1108::1
curl: IPv4: 37.27.129.22
curl: ALPN: curl offers h2,http/1.1
curl: TLSv1.3 (OUT), TLS handshake, Client hello (1):
curl: SSL Trust Anchors:
curl:   CAfile: /etc/ssl/certs/ca-certificates.crt
curl: TLSv1.3 (IN), TLS handshake, Server hello (2):
curl: TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
curl: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
curl: TLSv1.3 (IN), TLS handshake, Certificate (11):
curl: TLSv1.3 (IN), TLS handshake, CERT verify (15):
curl: TLSv1.3 (IN), TLS handshake, Finished (20):
curl: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
curl: TLSv1.3 (OUT), TLS handshake, Finished (20):
curl: SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519MLKEM768 / id-ecPublicKey
curl: ALPN: server accepted h2
curl: Server certificate:
curl:   subject: CN=cache.nixos-cuda.org
curl:   start date: Apr 24 14:18:06 2026 GMT
curl:   expire date: Jul 23 14:18:05 2026 GMT
curl:   issuer: C=US; O=Let's Encrypt; CN=E7
curl:   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
curl:   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
curl:   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
curl:   subjectAltName: "cache.nixos-cuda.org" matches cert's "cache.nixos-cuda.org"
curl: OpenSSL verify result: 0
curl: SSL certificate verified via OpenSSL.
curl: Established connection to cache.nixos-cuda.org (2a01:4f9:3071:1108::1 port 443) from 2001:41b8:83f:4242:202a:b300:1572:59b3 port 46100
curl: using HTTP/2
curl: [HTTP/2] [1] OPENED stream for https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r
curl: [HTTP/2] [1] [:method: GET]
curl: [HTTP/2] [1] [:scheme: https]
curl: [HTTP/2] [1] [:authority: cache.nixos-cuda.org]
curl: [HTTP/2] [1] [:path: /nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r]
curl: [HTTP/2] [1] [user-agent: curl/8.20.0 Lix/2.96.0-dev-pre20260523-dev_b113be5]
curl: [HTTP/2] [1] [accept: */*]
curl: [HTTP/2] [1] [accept-encoding: deflate, gzip, br, zstd]
curl: Request completely sent off
curl: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': HTTP/2 200
got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': accept-ranges: bytes
got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': alt-svc: h3=":443"; ma=2592000
got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': cache-control: max-age=31536000
got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': content-encoding: zstd
got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': content-type: application/x-nix-archive
got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': date: Mon, 01 Jun 2026 10:49:51 GMT
got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': vary: accept-encoding
got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': via: 1.1 Caddy
got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': 
locking path '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug'
lock acquired on '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug.lock'
curl: Connection #0 to host cache.nixos-cuda.org:443 left intact
finished download of 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r'; curl status = 23, HTTP status = 200, body = 22872064 bytes
lock released on '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug.lock'
error: unable to download 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': HTTP error 200 () (curl error code=23: Failed writing received data to disk/application)
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter
path '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug' is required, but there is no substituter that can build it
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': done
substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': goal destroyed
error: build of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug' failed
## Describe the bug The build farm I maintain substitutes from cache.nixos-cuda.org for supporting machines with `cudaSupport` enabled. Regularly when substituting `onnxruntime` from that cache I run into a curl error 23. This reproduces not only on our build farm, but also on my desktop machine, but not on my Rock 5B. All use nixos-unstable with a Lix version from last week or newer. ## Steps To Reproduce 1. Configure `cache.nixos-cuda.org 2. `nix-store --realise /nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug` ## Expected behavior A clear and concise description of what you expected to happen. ## `nix --version` output ``` nix (Lix, like Nix) 2.96.0-dev-pre20260531-dev_003946f System type: x86_64-linux Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /root/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/root/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/root/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/root/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/ig7j7ky4df4ar83r0cv560r9rj40qaqw-lix-2.96.0-dev-pre20260531-dev_003946f/share ``` ## `curl -V` output ``` curl 8.20.0 (x86_64-pc-linux-gnu) libcurl/8.20.0 OpenSSL/3.6.2 zlib/1.3.2 brotli/1.2.0 zstd/1.5.7 libidn2/2.3.8 libpsl/0.21.5 libssh2/1.11.1 nghttp2/1.69.0 ngtcp2/1.22.1 nghttp3/1.15.0 mit-krb5/1.22.1 Release-Date: 2026-04-29 Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt mqtts pop3 pop3s rtsp scp sftp smtp smtps telnet tftp Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd ``` ## Additional context ``` # nix-store --realise /nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug -vvvvvvvv performing daemon worker op: 40 querying info about missing paths... checking substituter 'https://cache.nixos.org' for path '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug' checking substituter 'https://nix-community.cachix.org' for path '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug' checking substituter 'https://cache.nixos-cuda.org' for path '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug' this path will be fetched (1091.10 MiB download, 1091.10 MiB unpacked): /nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug performing daemon worker op: 9 running command: '/nix/store/d8i768l3mi5ynrzhk5j15hbpwg4y226n-lix-2.96.0-dev-pre20260523-dev_b113be5/libexec/lix/check-namespace-support' '16' substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': created entered goal loop substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': init acquired write lock on '/nix/var/nix/temproots/3566132' substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter substitution of '/nix/store/7lmd80pj8j2yw1l54xsnhprd0a3sagv9-onnxruntime-1.26.0': created substitution of '/nix/store/fb67sl8v628y15xqpbbarvzpz5kf3f8y-source': created substitution of '/nix/store/7lmd80pj8j2yw1l54xsnhprd0a3sagv9-onnxruntime-1.26.0': init substitution of '/nix/store/fb67sl8v628y15xqpbbarvzpz5kf3f8y-source': init substitution of '/nix/store/7lmd80pj8j2yw1l54xsnhprd0a3sagv9-onnxruntime-1.26.0': done substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': waitee 'substitution of '/nix/store/7lmd80pj8j2yw1l54xsnhprd0a3sagv9-onnxruntime-1.26.0'' done; 1 left substitution of '/nix/store/fb67sl8v628y15xqpbbarvzpz5kf3f8y-source': done substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': waitee 'substitution of '/nix/store/fb67sl8v628y15xqpbbarvzpz5kf3f8y-source'' done; 0 left substitution of '/nix/store/7lmd80pj8j2yw1l54xsnhprd0a3sagv9-onnxruntime-1.26.0': goal destroyed substitution of '/nix/store/fb67sl8v628y15xqpbbarvzpz5kf3f8y-source': goal destroyed substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': all references realised substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying to run copying path '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug' from 'https://cache.nixos-cuda.org'... downloading 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r'... starting download of https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r curl: Could not find host cache.nixos-cuda.org in the /etc/nix/netrc file; using defaults curl: Trying [2a01:4f9:3071:1108::1]:443... curl: Host cache.nixos-cuda.org:443 was resolved. curl: IPv6: 2a01:4f9:3071:1108::1 curl: IPv4: 37.27.129.22 curl: ALPN: curl offers h2,http/1.1 curl: TLSv1.3 (OUT), TLS handshake, Client hello (1): curl: SSL Trust Anchors: curl: CAfile: /etc/ssl/certs/ca-certificates.crt curl: TLSv1.3 (IN), TLS handshake, Server hello (2): curl: TLSv1.3 (IN), TLS change cipher, Change cipher spec (1): curl: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): curl: TLSv1.3 (IN), TLS handshake, Certificate (11): curl: TLSv1.3 (IN), TLS handshake, CERT verify (15): curl: TLSv1.3 (IN), TLS handshake, Finished (20): curl: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): curl: TLSv1.3 (OUT), TLS handshake, Finished (20): curl: SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519MLKEM768 / id-ecPublicKey curl: ALPN: server accepted h2 curl: Server certificate: curl: subject: CN=cache.nixos-cuda.org curl: start date: Apr 24 14:18:06 2026 GMT curl: expire date: Jul 23 14:18:05 2026 GMT curl: issuer: C=US; O=Let's Encrypt; CN=E7 curl: Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384 curl: Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption curl: Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption curl: subjectAltName: "cache.nixos-cuda.org" matches cert's "cache.nixos-cuda.org" curl: OpenSSL verify result: 0 curl: SSL certificate verified via OpenSSL. curl: Established connection to cache.nixos-cuda.org (2a01:4f9:3071:1108::1 port 443) from 2001:41b8:83f:4242:202a:b300:1572:59b3 port 46100 curl: using HTTP/2 curl: [HTTP/2] [1] OPENED stream for https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r curl: [HTTP/2] [1] [:method: GET] curl: [HTTP/2] [1] [:scheme: https] curl: [HTTP/2] [1] [:authority: cache.nixos-cuda.org] curl: [HTTP/2] [1] [:path: /nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r] curl: [HTTP/2] [1] [user-agent: curl/8.20.0 Lix/2.96.0-dev-pre20260523-dev_b113be5] curl: [HTTP/2] [1] [accept: */*] curl: [HTTP/2] [1] [accept-encoding: deflate, gzip, br, zstd] curl: Request completely sent off curl: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': HTTP/2 200 got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': accept-ranges: bytes got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': alt-svc: h3=":443"; ma=2592000 got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': cache-control: max-age=31536000 got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': content-encoding: zstd got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': content-type: application/x-nix-archive got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': date: Mon, 01 Jun 2026 10:49:51 GMT got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': vary: accept-encoding got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': via: 1.1 Caddy got header for 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': locking path '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug' lock acquired on '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug.lock' curl: Connection #0 to host cache.nixos-cuda.org:443 left intact finished download of 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r'; curl status = 23, HTTP status = 200, body = 22872064 bytes lock released on '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug.lock' error: unable to download 'https://cache.nixos-cuda.org/nar/0848r90lzrc45725mil8vpzgn38hh4qhvdp540yyrdgy35d6y8wd.nar?hash=l673bdzbsfqy5arydhjbmx3lj29spj8r': HTTP error 200 () (curl error code=23: Failed writing received data to disk/application) substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': trying next substituter path '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug' is required, but there is no substituter that can build it substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': done substitution of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug': goal destroyed error: build of '/nix/store/l673bdzbsfqy5arydhjbmx3lj29spj8r-onnxruntime-1.26.0-debug' failed ```
mweinelt commented 2026-06-01 11:47:46 +00:00
Author
Member
Copy link

This started working again once I managed to disable http2.

This started working again once I managed to disable `http2`.
mweinelt changed title from Failed writing received data to disk/application during substitution to Failed writing received data to disk/application during substitution (HTTP2 issue) 2026-06-01 11:54:38 +00:00
pennae commented 2026-06-04 11:17:24 +00:00
Owner
Copy link

this is a curl bug. we managed to reproduce the failure in rr, and the things break the moment we return CURL_WRITEFUNC_PAUSE from our write callback the way the curl docs tell us to do to pause a transfer. so fucking sick of this shit. the only way forward seems to either disable h2/h3 completely or not pretend that curl has a useful async interface at all (ie roll back to pre-2.93 behavior of the curl wrapper)

this is a curl bug. we managed to reproduce the failure in rr, and the things break the moment we return `CURL_WRITEFUNC_PAUSE` from our write callback *the way the curl docs tell us to do to pause a transfer*. so fucking sick of this shit. the only way forward seems to either disable h2/h3 completely or not pretend that curl has a useful async interface at all (ie roll back to pre-2.93 behavior of the curl wrapper)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
release-2.95
release-2.93
release-2.94
sb/jade/coverage
ci-config
sb/raitobezarius/cgroups-observability
sb/jade/wip-tars
sb/raito/flakes-out-of-core
sb/raito/eval-explorations
sb/pennae/ci-config
release-2.91
release-2.92
sb/raito/darwin-kqueue-disablement
sb/raito/hydra-merge
sb/raito/phantom-referrers-gc
repl-overlay-example
reuse-compliance
release-2.90
sb/pennae/io-rewrite
git-series/activity-cleanup
sb/tom-hubrecht/primops.cc-v2
sb/bacchanalia/cache-strlen%topic=substr-perf
sb/qyriad/beta-1
sb/qyriad/2.90-beta.0
sb/rbt/test-branch
sb/rbt/justfile
sb/rbt/repl-overlays
sb/qyriad/maint-meson
sb/qyriad/maint/meson
sb/puck/meson-c-api
sb/jade/burn-it-all
dependabot/github_actions/zeebe-io/backport-action-2.4.1
dependabot/github_actions/cachix/install-nix-action-25
dependabot/github_actions/actions/labeler-5
dependabot/github_actions/cachix/cachix-action-14
2.95.3
2.93.4
2.94.2
2.95.2
2.95.1
2.94.1
2.95.0
2.94.0
2.93.3
2.92.3
2.93.2
2.91.3
2.92.2
2.91.2
2.93.1
2.93.0
2.92.0
2.91.1
2.91.0
2.90.0
2.90.0-rc1
2.90-beta.1
2.90-beta.0
1.11.1
1.11
1.10
1.9
1.8
1.7
1.6.1
1.6
1.5.3
1.5.2
1.5.1
1.5
1.4
1.3
1.2
1.1
1.0
Labels
Clear labels   
Affects/CppNix

Probably affects CppNix. This is a best-effort tag for troubles in old code; it exists to inform the CppNix team about problems they might care about also.

  
Affects/Nightly

Found in Lix nightly.

  
Affects/Only nightly

Tag for regressions in Lix nightly. We intend to fix all of these prior to releases.

  
Affects/Stable

Affects the stable release of Lix. Usually such issues affect nightly too but there is no guarantee.

  
Area/build-packaging

regarding the building or packaging of Lix itself

  
Area/cli

The command line interface of Lix

  
Area/evaluator

libexpr and anything else related to the evaluation of Nix expressions

  
Area/fetching

Fetching sources or other materials from the World Wide Web in Lix

  
Area/flakes

   
Area/language

language design, features, extensions, etc

  
Area/lix ci

Lix's own CI and deployment pipeline

  
Area/nix-eval-jobs

nix-eval-jobs for Lix

  
Area/profiles

profiles: nix-env, nix profile, and installer flavors all alike

  
Area/protocol

The protocol, used for communicating between the client and the daemon

  
Area/releng

Release engineering

  
Area/remote-builds

remote builds

  
Area/repl

nix repl

  
Area/repl/debugger

The --debugger that brings up a Lix repl

  
Area/store

libstore and anything else relating to the Nix store

  
awaiting
author
needs more information from the issue author

  
awaiting
contributors
needs more information from contributors

  
bug

it's a bug

  
Context
contributors
This issue might require some context that isn't written down. It might be possible to handle as a person who already worked on this area.

  
Context
drive-by
This issue contains all the necessary information to do a drive-by contribution: that is, you don't have to talk with anyone at all to successfully solve it!

  
Context
maintainers
This issue requires a lot of know-how. Attention from maintainers is needed to proceed.

  
Context
RFD
This issue needs discussion from multiple maintainers and interested parties: it's a Request For Discussion.

  
crash 💥

Lix crashes (possibly bad error handling)

  
Cross Compilation

   
devx

development experience issues

  
diagnostics

Misleading error messages and the like

  
docs

requires or is about documentation changes

  
Downstream Dependents

regarding an external tool's usage with Lix (e.g. nix-eval-jobs)

  
E/easy

bug is relatively simple and could simply be fixed with some minor effort

  
E/hard

this issue should probably be solved with close involvement of an expert

  
E/help wanted

maintainers may not get to this soon, but we want it fixed

  
E/reproducible

This bug is reproducible and obvious.

  
E/requires rearchitecture

this bug cannot be fixed properly without fixing a known architectural flaw

  
Feature/S3

S3 support in Lix

  
Importance
High
Very important to Lix's goals, not to be confused with urgent

  
Importance
Low
Less vital to Lix's goals. Not to be confused with non-urgent

  
imported

Issue imported from upstream nix (please do not remove this label, it will cause dupes if import is rerun)

  
Language/Bash

   
Language/C++

   
Language/NixLang

   
Language/Python

   
Language/Rust

   
Needs Langver

We can't get to this until we implement language versioning

  
OS/Linux

Affects specifically Lix on Linux

  
OS/macOS

Affects specifically Lix on macOS

  
performance

make lix go faster

  
regression

This is a regression since some previous version of Lix

  
Release Blocking
Non-urgent
We can't finish a release without this, but HEAD is okay

  
Release Blocking
Urgent
Disruptive AND blocking release. Something must be done!

  
stability

Fixing this makes the software more stable

  
Status
blocked
Blocked on something

  
Status
invalid
not our bug, duplicate, similar

  
Status
postponed
closed but should maybe look at later (generally with S-wontfix)

  
Status
wontfix
We chose not to change this (see issue comments for more detail).

  
testing

Tests that should be added

  
testing/flakey

Tests that are unreliable/flakey (not to be confused with flakes that test)

  
Topic/Large Scale Installations

Large scale installations of Lix, e.g. NixOS Hydra, Lix CI, big corporate installations etc.

  
Urgency
High
Disruptive and needs addressing quickly

  
Urgency
Low
Does not require *immediate* attention

  
ux

makes the user experience better :)

No labels
Affects/CppNix
Affects/Nightly
Affects/Only nightly
Affects/Stable
Area/build-packaging
Area/cli
Area/evaluator
Area/fetching
Area/flakes
Area/language
Area/lix ci
Area/nix-eval-jobs
Area/profiles
Area/protocol
Area/releng
Area/remote-builds
Area/repl
Area/repl/debugger
Area/store
awaiting
author
awaiting
contributors
bug
Context
contributors
Context
drive-by
Context
maintainers
Context
RFD
crash 💥
Cross Compilation
devx
diagnostics
docs
Downstream Dependents
E/easy
E/hard
E/help wanted
E/reproducible
E/requires rearchitecture
Feature/S3
Importance
High
Importance
Low
imported
Language/Bash
Language/C++
Language/NixLang
Language/Python
Language/Rust
Needs Langver
OS/Linux
OS/macOS
performance
regression
Release Blocking
Non-urgent
Release Blocking
Urgent
stability
Status
blocked
Status
invalid
Status
postponed
Status
wontfix
testing
testing/flakey
Topic/Large Scale Installations
Urgency
High
Urgency
Low
ux
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lix-project/lix#1222
No description provided.