`nix copy --to s3://...` very slow, hangs #945

New issue

Open

opened 2025-07-31 04:17:51 +00:00 by cobaltcause · 1 comment

cobaltcause commented

2025-07-31 04:17:51 +00:00

Member

Description

I have a script that contains this:

if command -v nom &> /dev/null; then
    nom build "$@"
else
    nix build "$@"
fi

# Find all output paths of the installables and their build dependencies.
readarray -t derivations < <(nix path-info --derivation "$@")
readarray -t upload_paths < <(
    xargs \
        nix-store --query --requisites --include-outputs \
        <<< "${derivations[*]}"
)

echo "Found ${#upload_paths[@]} paths to upload"

if [ -z ${NIX_SIGNING_PRIVATE_KEY+x} ]; then
    echo "Signing key not found, skipping uploading to the binary cache"
    return
fi

# Sign the paths. We have to put the secret in an actual file because Lix
# doesn't like process substitution.
key_file="$(mktemp)"
echo "$NIX_SIGNING_PRIVATE_KEY" > "$key_file"
nix store sign --key-file "$key_file" --stdin --recursive \
    <<< "${upload_paths[*]}"
rm "$key_file"

dst="s3://grapevine-nix"
dst+="?scheme=https"
dst+="&endpoint=garage.computer.surgery"
dst+="&region=garage"
dst+="&multipart-upload=true"
dst+="&buffer-size=67108864"

# Upload the paths. Environment variable disables some AWS-specific stuff
# that causes a 10 second delay before uploading starts.
env AWS_EC2_METADATA_DISABLED=true \
    nix copy --stdin --to "$dst" <<< "${upload_paths[*]}"

For credentials, the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set. No AWS configuration files exist. I'm copying to a self hosted Garage instance on the same network with a 2.5Gbps link.

I was able to successfully run this with CppNix 2.28.4 which resulted in ~7900 objects getting stored in the bucket at a total of ~3GB, with peak transfer speeds of ~100Mbps according to btm.

Trying this with Lix, however, doesn't seem to work very well. It can successfully upload some stuff, but it seems to get stuck on other stuff. If left running long enough, its own upload progress counter will stop. Transfer rate is low (Kbps range) to nonexistent according to btm. Additionally, when Lix hangs like that, hitting control+c doesn't successfully shut down and exit Lix, at least not within ~30 seconds; I always have to send a second control+c to kill it.

Uploading a single large file to a different bucket via awscli2 yields a throughput of ~850Mbps, so there's a lot of room for improvement it seems.

Expected behavior

Lix should be able to upload to an S3 binary cache. It would be cool if it were faster than CppNix too.

`nix --version` output

$ nix --version
nix (Lix, like Nix) 2.94.0-dev
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux, x86_64-v4-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/charles/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/charles/.local/share/flatpak/exports/etc/xdg/nix/nix.conf:/var/lib/flatpak/exports/etc/xdg/nix/nix.conf:/home/charles/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/charles/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/charles/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/kcn0iyrzzb8hri3bx3nwgkidr8ym4mri-lix-2.94.0-dev/share

Additional context

My version of Lix is built from commit 2d0109898a.

## Description I have a script that contains this: ```bash if command -v nom &> /dev/null; then nom build "$@" else nix build "$@" fi # Find all output paths of the installables and their build dependencies. readarray -t derivations < <(nix path-info --derivation "$@") readarray -t upload_paths < <( xargs \ nix-store --query --requisites --include-outputs \ <<< "${derivations[*]}" ) echo "Found ${#upload_paths[@]} paths to upload" if [ -z ${NIX_SIGNING_PRIVATE_KEY+x} ]; then echo "Signing key not found, skipping uploading to the binary cache" return fi # Sign the paths. We have to put the secret in an actual file because Lix # doesn't like process substitution. key_file="$(mktemp)" echo "$NIX_SIGNING_PRIVATE_KEY" > "$key_file" nix store sign --key-file "$key_file" --stdin --recursive \ <<< "${upload_paths[*]}" rm "$key_file" dst="s3://grapevine-nix" dst+="?scheme=https" dst+="&endpoint=garage.computer.surgery" dst+="&region=garage" dst+="&multipart-upload=true" dst+="&buffer-size=67108864" # Upload the paths. Environment variable disables some AWS-specific stuff # that causes a 10 second delay before uploading starts. env AWS_EC2_METADATA_DISABLED=true \ nix copy --stdin --to "$dst" <<< "${upload_paths[*]}" ``` For credentials, the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables are set. No AWS configuration files exist. I'm copying to a self hosted Garage instance on the same network with a 2.5Gbps link. I was able to successfully run this with CppNix 2.28.4 which resulted in ~7900 objects getting stored in the bucket at a total of ~3GB, with peak transfer speeds of ~100Mbps according to `btm`. Trying this with Lix, however, doesn't seem to work very well. It can successfully upload some stuff, but it seems to get stuck on other stuff. If left running long enough, its own upload progress counter will stop. Transfer rate is low (Kbps range) to nonexistent according to `btm`. Additionally, when Lix hangs like that, hitting control+c doesn't successfully shut down and exit Lix, at least not within ~30 seconds; I always have to send a second control+c to kill it. Uploading a single large file to a different bucket via awscli2 yields a throughput of ~850Mbps, so there's a lot of room for improvement it seems. ## Expected behavior Lix should be able to upload to an S3 binary cache. It would be cool if it were faster than CppNix too. ## `nix --version` output ``` $ nix --version nix (Lix, like Nix) 2.94.0-dev System type: x86_64-linux Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux, x86_64-v4-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /home/charles/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/charles/.local/share/flatpak/exports/etc/xdg/nix/nix.conf:/var/lib/flatpak/exports/etc/xdg/nix/nix.conf:/home/charles/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/charles/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/charles/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/kcn0iyrzzb8hri3bx3nwgkidr8ym4mri-lix-2.94.0-dev/share ``` ## Additional context My version of Lix is built from commit 2d0109898a65884e8953813c0391ad8b3be0d929.