AWS SDK tries to derive credentials from EC2 metadata endpoint before looking into the environment #1125

New issue

Open

opened 2026-02-08 11:49:00 +00:00 by ma27 · 0 comments

ma27 commented

2026-02-08 11:49:00 +00:00

Member

Describe the bug

When trying to connect to an S3 store (e.g. by having store_uri = s3://... in Hydra or just --store s3://... on the Nix CLI), several connection attempts to the metadata endpoint are being made before falling back to fetching credentials from the environment:

AWS: [WARN] 2026-02-08 10:48:38.651 EC2MetadataClient [140420637823552] Request failed, now waiting 0 ms before attempting again.
AWS: [DEBUG] 2026-02-08 10:48:38.651 CurlHandleContainer [140420637823552] Attempting to acquire curl connection.
AWS: [DEBUG] 2026-02-08 10:48:38.651 CurlHandleContainer [140420637823552] Connection has been released. Continuing.
AWS: [DEBUG] 2026-02-08 10:48:38.651 CurlHandleContainer [140420637823552] Returning connection handle 0x56055e8e74b0
AWS: [DEBUG] 2026-02-08 10:48:38.651 CurlHttpClient [140420637823552] Obtained connection handle 0x56055e8e74b0
AWS: [DEBUG] 2026-02-08 10:48:38.651 CURL [140420637823552] (Text)   Trying 169.254.169.254:80...
AWS: [DEBUG] 2026-02-08 10:48:39.652 CURL [140420637823552] (Text) Connection timed out after 1001 milliseconds
AWS: [DEBUG] 2026-02-08 10:48:39.652 CURL [140420637823552] (Text) closing connection #0
AWS: [ERROR] 2026-02-08 10:48:39.652 CurlHttpClient [140420637823552] Curl returned error: curlCode: 28, Timeout was reached; Details: Connection timed out after 1001 milliseconds
AWS: [DEBUG] 2026-02-08 10:48:39.653 CurlHandleContainer [140420637823552] Destroy curl handle: 0x56055e8e74b0
AWS: [DEBUG] 2026-02-08 10:48:39.653 CurlHandleContainer [140420637823552] Created replacement handle and released to pool: 0x56055e8e74b0
AWS: [ERROR] 2026-02-08 10:48:39.653 EC2MetadataClient [140420637823552] Http request to retrieve credentials failed
AWS: [ERROR] 2026-02-08 10:48:39.653 EC2MetadataClient [140420637823552] Can not retrieve resource from http://169.254.169.254/latest/meta-data/iam/security-credentials

Ways to work around this are

Passing region via env (export AWS_REGION=eu-west-1)
Setting AWS_EC2_METADATA_DISABLED=true in your env (see also https://cl.afnix.fr/c/infra/+/450)

Steps To Reproduce

Set AWS_AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY to your env
Run nix store ping --store "s3://bucket?endpoint=example.com&region=eu-west-1"

Expected behavior

I wouldn't expect the SDK trying to connect to the EC2 metadata endpoint in the first place.

Or at least coverage in the documentation.

I'm open for discussion on what the most sensible behavior would be, but I'd like to not have 10s of silence before any store operations happen on my machines without any interaction.

`nix --version` output

nix (Lix, like Nix) 2.95.0pre20260127_9861915
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux, x86_64-v4-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/ma27/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/ma27/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/ma27/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/ma27/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/9sygj9jajkg277a1z3qy57lcgampavsc-lix-2.95.0pre20260127_9861915/share

Additional context

Add any other context about the problem here.

## Describe the bug When trying to connect to an S3 store (e.g. by having `store_uri = s3://...` in Hydra or just `--store s3://...` on the Nix CLI), several connection attempts to the metadata endpoint are being made before falling back to fetching credentials from the environment: ``` AWS: [WARN] 2026-02-08 10:48:38.651 EC2MetadataClient [140420637823552] Request failed, now waiting 0 ms before attempting again. AWS: [DEBUG] 2026-02-08 10:48:38.651 CurlHandleContainer [140420637823552] Attempting to acquire curl connection. AWS: [DEBUG] 2026-02-08 10:48:38.651 CurlHandleContainer [140420637823552] Connection has been released. Continuing. AWS: [DEBUG] 2026-02-08 10:48:38.651 CurlHandleContainer [140420637823552] Returning connection handle 0x56055e8e74b0 AWS: [DEBUG] 2026-02-08 10:48:38.651 CurlHttpClient [140420637823552] Obtained connection handle 0x56055e8e74b0 AWS: [DEBUG] 2026-02-08 10:48:38.651 CURL [140420637823552] (Text) Trying 169.254.169.254:80... AWS: [DEBUG] 2026-02-08 10:48:39.652 CURL [140420637823552] (Text) Connection timed out after 1001 milliseconds AWS: [DEBUG] 2026-02-08 10:48:39.652 CURL [140420637823552] (Text) closing connection #0 AWS: [ERROR] 2026-02-08 10:48:39.652 CurlHttpClient [140420637823552] Curl returned error: curlCode: 28, Timeout was reached; Details: Connection timed out after 1001 milliseconds AWS: [DEBUG] 2026-02-08 10:48:39.653 CurlHandleContainer [140420637823552] Destroy curl handle: 0x56055e8e74b0 AWS: [DEBUG] 2026-02-08 10:48:39.653 CurlHandleContainer [140420637823552] Created replacement handle and released to pool: 0x56055e8e74b0 AWS: [ERROR] 2026-02-08 10:48:39.653 EC2MetadataClient [140420637823552] Http request to retrieve credentials failed AWS: [ERROR] 2026-02-08 10:48:39.653 EC2MetadataClient [140420637823552] Can not retrieve resource from http://169.254.169.254/latest/meta-data/iam/security-credentials ``` Ways to work around this are * Passing region via env (`export AWS_REGION=eu-west-1`) * Setting `AWS_EC2_METADATA_DISABLED=true` in your env (see also https://cl.afnix.fr/c/infra/+/450) ## Steps To Reproduce 1. Set `AWS_AWS_ACCESS_KEY_ID` & `AWS_SECRET_ACCESS_KEY` to your env 2. Run `nix store ping --store "s3://bucket?endpoint=example.com&region=eu-west-1"` ## Expected behavior I wouldn't expect the SDK trying to connect to the EC2 metadata endpoint in the first place. Or at least coverage in the documentation. I'm open for discussion on what the most sensible behavior would be, but I'd like to not have 10s of silence before any store operations happen on my machines without any interaction. ## `nix --version` output ``` nix (Lix, like Nix) 2.95.0pre20260127_9861915 System type: x86_64-linux Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux, x86_64-v4-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /home/ma27/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/ma27/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/ma27/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/ma27/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/9sygj9jajkg277a1z3qy57lcgampavsc-lix-2.95.0pre20260127_9861915/share ``` ## Additional context Add any other context about the problem here.