S3 + AWS KMS (specifically SSE-KMS) support is broken

rv32ima commented

2025-10-03 18:56:29 +00:00

Describe the bug

Pushing to a S3 bucket which has Server-Side Encryption with AWS KMS enabled with nix copy --to is broken, and results in this error:

; ./scripts/build_bidder_nix_store.sh 
warning: Git tree '/Users/eford/work/ds' is dirty
trace: evaluation warning: You have set specialArgs.pkgs, which means that options like nixpkgs.config
and nixpkgs.overlays will be ignored. If you wish to reuse an already created
pkgs, which you know is configured correctly for this NixOS configuration,
please import the `nixosModules.readOnlyPkgs` module from the nixpkgs flake or
`(modulesPath + "/misc/nixpkgs/read-only.nix"), and set `{ nixpkgs.pkgs = <your pkgs>; }`.
This properly disables the ignored options to prevent future surprises.

error: AWS error uploading 'nix-cache-info': Unable to parse ExceptionName: InvalidArgument Message: Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.

Interestingly, this only happens locally using AWS SSO + the AWS CLI, and doesn't happen in my $work's CI, where we use AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY derived from an OAuth identity principal (specifically GitHub Actions).

Steps To Reproduce

Create a new customer-managed AWS KMS symmetric key-pair, and configure the key policies to allow S3 to use this key. A good example key policy is as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:Encrypt",
        "kms:DescribeKey",
        "kms:Decrypt",
        "kms:CreateGrant"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "s3.(YOUR AWS REGION HERE).amazonaws.com",
          "kms:CallerAccount": "(YOUR AWS ACCOUNT ID HERE)"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::(YOUR AWS ACCOUNT ID HERE):root"
      },
      "Action": "kms:*",
      "Resource": "*"
    }
  ]
}

Create an S3 bucket, and configure it to use the AWS KMS symmetric key-pair created in step 1.
Configure AWS SSO, and use it to log in to your local CLI (aws sso login)
Attempt to push to the bucket using nix copy --to:

#!/usr/bin/env bash
set -eu

NIX_STORE_BUCKET=${NIX_STORE_BUCKET:-tvs-ds-bidder-nix-store}
REGION=${REGION:-us-west-2}

nix copy --to "s3://${NIX_STORE_BUCKET}?region=${REGION}" \
    .#packages.x86_64-linux.my-nixos-derivation.config.system.build.toplevel

Shed a tear or two in private

Expected behavior

Expected behavior is that this derivation (and all of it's dependencies) are properly copied to the S3 bucket using the credentials from the AWS SDK. Note that upstream CppNix also has this same error, albeit in a slightly different way:

; ./scripts/build_bidder_nix_store.sh
warning: Git tree '/Users/eford/work/ds' is dirty
evaluation warning: You have set specialArgs.pkgs, which means that options like nixpkgs.config
                    and nixpkgs.overlays will be ignored. If you wish to reuse an already created
                    pkgs, which you know is configured correctly for this NixOS configuration,
                    please import the `nixosModules.readOnlyPkgs` module from the nixpkgs flake or
                    `(modulesPath + "/misc/nixpkgs/read-only.nix"), and set `{ nixpkgs.pkgs = <your pkgs>; }`.
                    This properly disables the ignored options to prevent future surprises.
error: AWS error uploading 'nix-cache-info': Unable to parse ExceptionName: InvalidArgument Message: Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4. (request id: [censored])
; nix --version
nix (Nix) 2.28.5

`nix --version` output

; nix --version
nix (Lix, like Nix) 2.91.3

Additional context

Related: #272

## Describe the bug Pushing to a S3 bucket which has [Server-Side Encryption with AWS KMS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-kms-encryption.html) enabled with `nix copy --to` is broken, and results in this error: ``` ; ./scripts/build_bidder_nix_store.sh warning: Git tree '/Users/eford/work/ds' is dirty trace: evaluation warning: You have set specialArgs.pkgs, which means that options like nixpkgs.config and nixpkgs.overlays will be ignored. If you wish to reuse an already created pkgs, which you know is configured correctly for this NixOS configuration, please import the `nixosModules.readOnlyPkgs` module from the nixpkgs flake or `(modulesPath + "/misc/nixpkgs/read-only.nix"), and set `{ nixpkgs.pkgs = <your pkgs>; }`. This properly disables the ignored options to prevent future surprises. error: AWS error uploading 'nix-cache-info': Unable to parse ExceptionName: InvalidArgument Message: Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4. ``` Interestingly, this only happens locally using AWS SSO + the AWS CLI, and *doesn't* happen in my $work's CI, where we use `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` derived from an OAuth identity principal (specifically GitHub Actions). ## Steps To Reproduce 1. Create a new customer-managed AWS KMS symmetric key-pair, and configure the key policies to allow S3 to use this key. A good example key policy is as follows: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": [ "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:Encrypt", "kms:DescribeKey", "kms:Decrypt", "kms:CreateGrant" ], "Resource": "*", "Condition": { "StringEquals": { "kms:ViaService": "s3.(YOUR AWS REGION HERE).amazonaws.com", "kms:CallerAccount": "(YOUR AWS ACCOUNT ID HERE)" } } }, { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::(YOUR AWS ACCOUNT ID HERE):root" }, "Action": "kms:*", "Resource": "*" } ] } ``` 2. Create an S3 bucket, and configure it to use the AWS KMS symmetric key-pair created in step 1. 3. Configure AWS SSO, and use it to log in to your local CLI (`aws sso login`) 4. Attempt to push to the bucket using `nix copy --to`: ``` #!/usr/bin/env bash set -eu NIX_STORE_BUCKET=${NIX_STORE_BUCKET:-tvs-ds-bidder-nix-store} REGION=${REGION:-us-west-2} nix copy --to "s3://${NIX_STORE_BUCKET}?region=${REGION}" \ .#packages.x86_64-linux.my-nixos-derivation.config.system.build.toplevel ``` 5. Shed a tear or two in private ## Expected behavior Expected behavior is that this derivation (and all of it's dependencies) are properly copied to the S3 bucket using the credentials from the AWS SDK. Note that upstream CppNix also has this same error, albeit in a slightly different way: ``` ; ./scripts/build_bidder_nix_store.sh warning: Git tree '/Users/eford/work/ds' is dirty evaluation warning: You have set specialArgs.pkgs, which means that options like nixpkgs.config and nixpkgs.overlays will be ignored. If you wish to reuse an already created pkgs, which you know is configured correctly for this NixOS configuration, please import the `nixosModules.readOnlyPkgs` module from the nixpkgs flake or `(modulesPath + "/misc/nixpkgs/read-only.nix"), and set `{ nixpkgs.pkgs = <your pkgs>; }`. This properly disables the ignored options to prevent future surprises. error: AWS error uploading 'nix-cache-info': Unable to parse ExceptionName: InvalidArgument Message: Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4. (request id: [censored]) ; nix --version nix (Nix) 2.28.5 ``` ## `nix --version` output ``` ; nix --version nix (Lix, like Nix) 2.91.3 ``` ## Additional context Related: #272

rv32ima added the

bug

label

2025-10-03 18:56:29 +00:00

rv32ima commented

2025-10-03 19:01:09 +00:00

Author

fascinating. I have a workaround, which is to NOT use AWS SSO, and instead use a temporary key-pair for this which... fixes things.

; export AWS_ACCESS_KEY_ID="[CENSORED]"
  export AWS_SECRET_ACCESS_KEY="[CENSORED]"
  export AWS_SESSION_TOKEN="[CENSORED]"
; ./scripts/build_bidder_nix_store.sh
warning: Git tree '/Users/eford/work/ds' is dirty
trace: evaluation warning: You have set specialArgs.pkgs, which means that options like nixpkgs.config
and nixpkgs.overlays will be ignored. If you wish to reuse an already created
pkgs, which you know is configured correctly for this NixOS configuration,
please import the `nixosModules.readOnlyPkgs` module from the nixpkgs flake or
`(modulesPath + "/misc/nixpkgs/read-only.nix"), and set `{ nixpkgs.pkgs = <your pkgs>; }`.
This properly disables the ignored options to prevent future surprises.

warning: Git tree '/Users/eford/work/ds' is dirty
trace: evaluation warning: You have set specialArgs.pkgs, which means that options like nixpkgs.config
and nixpkgs.overlays will be ignored. If you wish to reuse an already created
pkgs, which you know is configured correctly for this NixOS configuration,
please import the `nixosModules.readOnlyPkgs` module from the nixpkgs flake or
`(modulesPath + "/misc/nixpkgs/read-only.nix"), and set `{ nixpkgs.pkgs = <your pkgs>; }`.
This properly disables the ignored options to prevent future surprises.

; nix --version
nix (Lix, like Nix) 2.91.3

fascinating. I have a workaround, which is to *NOT* use AWS SSO, and instead use a temporary key-pair for this which... fixes things. ```shell ; export AWS_ACCESS_KEY_ID="[CENSORED]" export AWS_SECRET_ACCESS_KEY="[CENSORED]" export AWS_SESSION_TOKEN="[CENSORED]" ; ./scripts/build_bidder_nix_store.sh warning: Git tree '/Users/eford/work/ds' is dirty trace: evaluation warning: You have set specialArgs.pkgs, which means that options like nixpkgs.config and nixpkgs.overlays will be ignored. If you wish to reuse an already created pkgs, which you know is configured correctly for this NixOS configuration, please import the `nixosModules.readOnlyPkgs` module from the nixpkgs flake or `(modulesPath + "/misc/nixpkgs/read-only.nix"), and set `{ nixpkgs.pkgs = <your pkgs>; }`. This properly disables the ignored options to prevent future surprises. warning: Git tree '/Users/eford/work/ds' is dirty trace: evaluation warning: You have set specialArgs.pkgs, which means that options like nixpkgs.config and nixpkgs.overlays will be ignored. If you wish to reuse an already created pkgs, which you know is configured correctly for this NixOS configuration, please import the `nixosModules.readOnlyPkgs` module from the nixpkgs flake or `(modulesPath + "/misc/nixpkgs/read-only.nix"), and set `{ nixpkgs.pkgs = <your pkgs>; }`. This properly disables the ignored options to prevent future surprises. ; nix --version nix (Lix, like Nix) 2.91.3

jade added the

Feature/S3

label

2025-10-04 07:27:32 +00:00

jade commented

2025-10-04 07:27:42 +00:00

Owner

@arianvp you may be interested in this report

arianvp commented

2025-10-04 09:39:41 +00:00

Member

The error you're getting is the fallback behaviour of trying to talk to S3 without any auth at all.

Credentials need to be available to the nix daemon. Lix doesn't do credential propagation from (trusted) users to the daemon.

Your CI most likely works because you're using nix in single user mode or setting the env vars in a way that the daemon can pick them up.

Your credentials from AWS CLI locally don't magically make it to the daemon unfortunately.

You'll have to export your local credentials and set them in the .aws/credentials file of the nix daemon user.

The error you're getting is the fallback behaviour of trying to talk to S3 without any auth at all. Credentials need to be available to the nix daemon. Lix doesn't do credential propagation from (trusted) users to the daemon. Your CI most likely works because you're using nix in single user mode or setting the env vars in a way that the daemon can pick them up. Your credentials from AWS CLI locally don't magically make it to the daemon unfortunately. You'll have to export your local credentials and set them in the `.aws/credentials` file of the nix daemon user.

arianvp commented

2025-10-04 09:47:12 +00:00

Member

Oh no wait. For nix copy it does use the local credentials. Maybe AWS SSO is not in the default credential chain or something.

arianvp commented

2025-10-05 12:44:00 +00:00

Member

Could you please run this wit maximum verbosity? It'll show the AWS SDK logs. Make sure to sanitize any credentials as they're logged at this level

rv32ima commented

2025-10-05 19:21:42 +00:00

Author

Could you please run this wit maximum verbosity? It'll show the AWS SDK logs. Make sure to sanitize any credentials as they're logged at this level

sure! will do on monday when i'm back on my work laptop.

> Could you please run this wit maximum verbosity? It'll show the AWS SDK logs. Make sure to sanitize any credentials as they're logged at this level sure! will do on monday when i'm back on my work laptop.

lunaphied commented

2025-10-22 18:23:30 +00:00

Owner

@rv32ima any updates on this?

rv32ima commented

2025-10-22 19:01:20 +00:00

Author

can't reproduce after upgrading to using a nightly version of 2.94.0

arianvp commented

2025-10-23 18:00:23 +00:00

Member

So, the AWS CRT doesn't pick up the AWS SSO credential chain at all it seems:

https://github.com/awslabs/aws-c-auth/issues/228
https://github.com/NixOS/nix/issues/5525

The workaround for copying to cache is:

source $(aws configure export-credentials)

For substitution you will need to get those credentials to be picked up by the nix-daemon instead.

You can do that with:

sudo -i aws configure set aws_access_key_id "${AWS_ACCESS_KEY_ID}"
sudo -i aws configure set aws_secret_access_key "${AWS_SECRET_ACCESS_KEY}"
sudo -i aws configure set aws_session_token "${AWS_SESSION_TOKEN}"
sudo -i aws configure set region "${AWS_REGION}"

But it's all kinda hacky. As the SSO credential chain can be renewed. But the raw AWS credentials expire after 15 mins (by default; can be made longer).

So, the AWS CRT doesn't pick up the AWS SSO credential chain at all it seems: https://github.com/awslabs/aws-c-auth/issues/228 https://github.com/NixOS/nix/issues/5525 The workaround for copying to cache is: ``` source $(aws configure export-credentials) ``` For substitution you will need to get those credentials to be picked up by the nix-daemon instead. You can do that with: ``` sudo -i aws configure set aws_access_key_id "${AWS_ACCESS_KEY_ID}" sudo -i aws configure set aws_secret_access_key "${AWS_SECRET_ACCESS_KEY}" sudo -i aws configure set aws_session_token "${AWS_SESSION_TOKEN}" sudo -i aws configure set region "${AWS_REGION}" ``` But it's all kinda hacky. As the SSO credential chain can be renewed. But the raw AWS credentials expire after 15 mins (by default; can be made longer).

Rows
Columns

S3 + AWS KMS (specifically SSE-KMS) support is broken #1001

Describe the bug

Steps To Reproduce

Expected behavior

nix --version output

Additional context

`nix --version` output