forked from lix-project/lix
Merge how-to section on S3 buckets into S3 store docs (#7972)
Rather than having a misc tutorial page in the grab-bag "package management" section, this information should just be part of the S3 store docs. --------- Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
This commit is contained in:
Valentin Gagarin
GitHub
parent
c9528d2081
commit
cd680bd53d
|
|
@ -24,7 +24,6 @@
|
||||||
- [Serving a Nix store via HTTP](package-management/binary-cache-substituter.md)
|
- [Serving a Nix store via HTTP](package-management/binary-cache-substituter.md)
|
||||||
- [Copying Closures via SSH](package-management/copy-closure.md)
|
- [Copying Closures via SSH](package-management/copy-closure.md)
|
||||||
- [Serving a Nix store via SSH](package-management/ssh-substituter.md)
|
- [Serving a Nix store via SSH](package-management/ssh-substituter.md)
|
||||||
- [Serving a Nix store via S3](package-management/s3-substituter.md)
|
|
||||||
- [Nix Language](language/index.md)
|
- [Nix Language](language/index.md)
|
||||||
- [Data Types](language/values.md)
|
- [Data Types](language/values.md)
|
||||||
- [Language Constructs](language/constructs.md)
|
- [Language Constructs](language/constructs.md)
|
||||||
|
|
|
||||||
|
|
@ -17,9 +17,8 @@ the build loop.
|
||||||
|
|
||||||
# Prerequisites
|
# Prerequisites
|
||||||
|
|
||||||
This tutorial assumes you have [configured an S3-compatible binary
|
This tutorial assumes you have configured an [S3-compatible binary cache](@docroot@/command-ref/new-cli/nix3-help-stores.md#s3-binary-cache-store) as a [substituter](../command-ref/conf-file.md#conf-substituters),
|
||||||
cache](../package-management/s3-substituter.md), and that the `root`
|
and that the `root` user's default AWS profile can upload to the bucket.
|
||||||
user's default AWS profile can upload to the bucket.
|
|
||||||
|
|
||||||
# Set up a Signing Key
|
# Set up a Signing Key
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,115 +0,0 @@
|
||||||
# Serving a Nix store via S3
|
|
||||||
|
|
||||||
Nix has [built-in support](@docroot@/command-ref/new-cli/nix3-help-stores.md#s3-binary-cache-store)
|
|
||||||
for storing and fetching store paths from
|
|
||||||
Amazon S3 and S3-compatible services. This uses the same *binary*
|
|
||||||
cache mechanism that Nix usually uses to fetch prebuilt binaries from
|
|
||||||
[cache.nixos.org](https://cache.nixos.org/).
|
|
||||||
|
|
||||||
In this example we will use the bucket named `example-nix-cache`.
|
|
||||||
|
|
||||||
## Anonymous Reads to your S3-compatible binary cache
|
|
||||||
|
|
||||||
If your binary cache is publicly accessible and does not require
|
|
||||||
authentication, the simplest and easiest way to use Nix with your S3
|
|
||||||
compatible binary cache is to use the HTTP URL for that cache.
|
|
||||||
|
|
||||||
For AWS S3 the binary cache URL for example bucket will be exactly
|
|
||||||
<https://example-nix-cache.s3.amazonaws.com> or
|
|
||||||
<s3://example-nix-cache>. For S3 compatible binary caches, consult that
|
|
||||||
cache's documentation.
|
|
||||||
|
|
||||||
Your bucket will need the following bucket policy:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"Id": "DirectReads",
|
|
||||||
"Version": "2012-10-17",
|
|
||||||
"Statement": [
|
|
||||||
{
|
|
||||||
"Sid": "AllowDirectReads",
|
|
||||||
"Action": [
|
|
||||||
"s3:GetObject",
|
|
||||||
"s3:GetBucketLocation"
|
|
||||||
],
|
|
||||||
"Effect": "Allow",
|
|
||||||
"Resource": [
|
|
||||||
"arn:aws:s3:::example-nix-cache",
|
|
||||||
"arn:aws:s3:::example-nix-cache/*"
|
|
||||||
],
|
|
||||||
"Principal": "*"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Authenticated Reads to your S3 binary cache
|
|
||||||
|
|
||||||
For AWS S3 the binary cache URL for example bucket will be exactly
|
|
||||||
<s3://example-nix-cache>.
|
|
||||||
|
|
||||||
Nix will use the [default credential provider
|
|
||||||
chain](https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/credentials.html)
|
|
||||||
for authenticating requests to Amazon S3.
|
|
||||||
|
|
||||||
Nix supports authenticated reads from Amazon S3 and S3 compatible binary
|
|
||||||
caches.
|
|
||||||
|
|
||||||
Your bucket will need a bucket policy allowing the desired users to
|
|
||||||
perform the `s3:GetObject` and `s3:GetBucketLocation` action on all
|
|
||||||
objects in the bucket. The [anonymous policy given
|
|
||||||
above](#anonymous-reads-to-your-s3-compatible-binary-cache) can be
|
|
||||||
updated to have a restricted `Principal` to support this.
|
|
||||||
|
|
||||||
## Authenticated Writes to your S3-compatible binary cache
|
|
||||||
|
|
||||||
Nix support fully supports writing to Amazon S3 and S3 compatible
|
|
||||||
buckets. The binary cache URL for our example bucket will be
|
|
||||||
<s3://example-nix-cache>.
|
|
||||||
|
|
||||||
Nix will use the [default credential provider
|
|
||||||
chain](https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/credentials.html)
|
|
||||||
for authenticating requests to Amazon S3.
|
|
||||||
|
|
||||||
Your account will need the following IAM policy to upload to the cache:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"Version": "2012-10-17",
|
|
||||||
"Statement": [
|
|
||||||
{
|
|
||||||
"Sid": "UploadToCache",
|
|
||||||
"Effect": "Allow",
|
|
||||||
"Action": [
|
|
||||||
"s3:AbortMultipartUpload",
|
|
||||||
"s3:GetBucketLocation",
|
|
||||||
"s3:GetObject",
|
|
||||||
"s3:ListBucket",
|
|
||||||
"s3:ListBucketMultipartUploads",
|
|
||||||
"s3:ListMultipartUploadParts",
|
|
||||||
"s3:PutObject"
|
|
||||||
],
|
|
||||||
"Resource": [
|
|
||||||
"arn:aws:s3:::example-nix-cache",
|
|
||||||
"arn:aws:s3:::example-nix-cache/*"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Examples
|
|
||||||
|
|
||||||
To upload with a specific credential profile for Amazon S3:
|
|
||||||
|
|
||||||
```console
|
|
||||||
$ nix copy nixpkgs.hello \
|
|
||||||
--to 's3://example-nix-cache?profile=cache-upload®ion=eu-west-2'
|
|
||||||
```
|
|
||||||
|
|
||||||
To upload to an S3-compatible binary cache:
|
|
||||||
|
|
||||||
```console
|
|
||||||
$ nix copy nixpkgs.hello --to \
|
|
||||||
's3://example-nix-cache?profile=cache-upload&scheme=https&endpoint=minio.example.com'
|
|
||||||
```
|
|
||||||
|
|
@ -2,7 +2,103 @@ R"(
|
||||||
|
|
||||||
**Store URL format**: `s3://`*bucket-name*
|
**Store URL format**: `s3://`*bucket-name*
|
||||||
|
|
||||||
This store allows reading and writing a binary cache stored in an AWS
|
This store allows reading and writing a binary cache stored in an AWS S3 (or S3-compatible service) bucket.
|
||||||
S3 bucket.
|
This store shares many idioms with the [HTTP Binary Cache Store](#http-binary-cache-store).
|
||||||
|
|
||||||
|
For AWS S3, the binary cache URL for a bucket named `example-nix-cache` will be exactly <s3://example-nix-cache>.
|
||||||
|
For S3 compatible binary caches, consult that cache's documentation.
|
||||||
|
|
||||||
|
### Anonymous reads to your S3-compatible binary cache
|
||||||
|
|
||||||
|
> If your binary cache is publicly accessible and does not require authentication,
|
||||||
|
> it is simplest to use the [HTTP Binary Cache Store] rather than S3 Binary Cache Store with
|
||||||
|
> <https://example-nix-cache.s3.amazonaws.com> instead of <s3://example-nix-cache>.
|
||||||
|
|
||||||
|
Your bucket will need a
|
||||||
|
[bucket policy](https://docs.aws.amazon.com/AmazonS3/v1/userguide/bucket-policies.html)
|
||||||
|
like the following to be accessible:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"Id": "DirectReads",
|
||||||
|
"Version": "2012-10-17",
|
||||||
|
"Statement": [
|
||||||
|
{
|
||||||
|
"Sid": "AllowDirectReads",
|
||||||
|
"Action": [
|
||||||
|
"s3:GetObject",
|
||||||
|
"s3:GetBucketLocation"
|
||||||
|
],
|
||||||
|
"Effect": "Allow",
|
||||||
|
"Resource": [
|
||||||
|
"arn:aws:s3:::example-nix-cache",
|
||||||
|
"arn:aws:s3:::example-nix-cache/*"
|
||||||
|
],
|
||||||
|
"Principal": "*"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Authentication
|
||||||
|
|
||||||
|
Nix will use the
|
||||||
|
[default credential provider chain](https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/credentials.html)
|
||||||
|
for authenticating requests to Amazon S3.
|
||||||
|
|
||||||
|
Note that this means Nix will read environment variables and files with different idioms than with Nix's own settings, as implemented by the AWS SDK.
|
||||||
|
Consult the documentation linked above for further details.
|
||||||
|
|
||||||
|
### Authenticated reads to your S3 binary cache
|
||||||
|
|
||||||
|
Your bucket will need a bucket policy allowing the desired users to perform the `s3:GetObject` and `s3:GetBucketLocation` action on all objects in the bucket.
|
||||||
|
The [anonymous policy given above](#anonymous-reads-to-your-s3-compatible-binary-cache) can be updated to have a restricted `Principal` to support this.
|
||||||
|
|
||||||
|
### Authenticated writes to your S3-compatible binary cache
|
||||||
|
|
||||||
|
Your account will need an IAM policy to support uploading to the bucket:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"Version": "2012-10-17",
|
||||||
|
"Statement": [
|
||||||
|
{
|
||||||
|
"Sid": "UploadToCache",
|
||||||
|
"Effect": "Allow",
|
||||||
|
"Action": [
|
||||||
|
"s3:AbortMultipartUpload",
|
||||||
|
"s3:GetBucketLocation",
|
||||||
|
"s3:GetObject",
|
||||||
|
"s3:ListBucket",
|
||||||
|
"s3:ListBucketMultipartUploads",
|
||||||
|
"s3:ListMultipartUploadParts",
|
||||||
|
"s3:PutObject"
|
||||||
|
],
|
||||||
|
"Resource": [
|
||||||
|
"arn:aws:s3:::example-nix-cache",
|
||||||
|
"arn:aws:s3:::example-nix-cache/*"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Examples
|
||||||
|
|
||||||
|
With bucket policies and authentication set up as described above, uploading works via [`nix copy`](@docroot@/command-ref/new-cli/nix3-copy.md) (experimental).
|
||||||
|
|
||||||
|
- To upload with a specific credential profile for Amazon S3:
|
||||||
|
|
||||||
|
```console
|
||||||
|
$ nix copy nixpkgs.hello \
|
||||||
|
--to 's3://example-nix-cache?profile=cache-upload®ion=eu-west-2'
|
||||||
|
```
|
||||||
|
|
||||||
|
- To upload to an S3-compatible binary cache:
|
||||||
|
|
||||||
|
```console
|
||||||
|
$ nix copy nixpkgs.hello --to \
|
||||||
|
's3://example-nix-cache?profile=cache-upload&scheme=https&endpoint=minio.example.com'
|
||||||
|
```
|
||||||
|
|
||||||
)"
|
)"
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue