forked from lix-project/lix
Merge pull request #2456 from grahamc/s3-substituters
Update docs to describe how s3:// URLS does in fact support endpoint, region, and profile for upload
This commit is contained in:
Eelco Dolstra
GitHub
commit
aa5e47b2f4
|
|
@ -12,95 +12,10 @@ from Amazon S3 and S3 compatible services. This uses the same
|
||||||
<emphasis>binary</emphasis> cache mechanism that Nix usually uses to
|
<emphasis>binary</emphasis> cache mechanism that Nix usually uses to
|
||||||
fetch prebuilt binaries from <uri>cache.nixos.org</uri>.</para>
|
fetch prebuilt binaries from <uri>cache.nixos.org</uri>.</para>
|
||||||
|
|
||||||
<para>In this example we will use the bucket named
|
<para>The following options can be specified as URL parameters to
|
||||||
<literal>example-bucket</literal>.</para>
|
the S3 URL:</para>
|
||||||
|
|
||||||
<section xml:id="ssec-s3-substituter-anonymous-reads">
|
<variablelist>
|
||||||
<title>Anonymous Reads to your S3-compatible binary cache</title>
|
|
||||||
|
|
||||||
<para>If your binary cache is publicly accessible and does not
|
|
||||||
require authentication, the simplest and easiest way to use Nix with
|
|
||||||
your S3 compatible binary cache is to use the HTTP URL for that
|
|
||||||
cache.</para>
|
|
||||||
|
|
||||||
<para>For AWS S3 the binary cache URL for example bucket will be
|
|
||||||
exactly <uri>https://example-bucket.s3.amazonaws.com</uri>. For S3
|
|
||||||
compatible binary caches ago have to consult your software's
|
|
||||||
documentation.</para>
|
|
||||||
|
|
||||||
<para>Your bucket will need the following bucket policy:</para>
|
|
||||||
|
|
||||||
<programlisting>
|
|
||||||
<![CDATA[
|
|
||||||
{
|
|
||||||
"Id": "DirectReads",
|
|
||||||
"Version": "2012-10-17",
|
|
||||||
"Statement": [
|
|
||||||
{
|
|
||||||
"Sid": "AlowDirectReads",
|
|
||||||
"Action": [
|
|
||||||
"s3:GetObject"
|
|
||||||
],
|
|
||||||
"Effect": "Allow",
|
|
||||||
"Resource": "arn:aws:s3:::example-bucket/*",
|
|
||||||
"Principal": "*"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]]>
|
|
||||||
</programlisting>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<section xml:id="ssec-s3-substituter-authenticated-reads">
|
|
||||||
<title>Authenticated Reads to your S3 binary cache</title>
|
|
||||||
|
|
||||||
<para>For AWS S3 the binary cache URL for example bucket will be
|
|
||||||
exactly <uri>s3://example-bucket</uri>.</para>
|
|
||||||
|
|
||||||
<para>Nix will use the <link
|
|
||||||
xlink:href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default.">default
|
|
||||||
credential provider chain</link> for authenticating requests to
|
|
||||||
Amazon S3.</para>
|
|
||||||
|
|
||||||
<para>Nix supports authenticated writes to S3 compatible binary
|
|
||||||
caches but only supports Authenticated reads from Amazon S3.
|
|
||||||
Additionally, the following limitations are in place for
|
|
||||||
authenticated reads:</para>
|
|
||||||
|
|
||||||
<itemizedlist>
|
|
||||||
<listitem><para>The bucket must actually be hosted by Amazon S3 and
|
|
||||||
<emphasis>not</emphasis> an S3 compatible
|
|
||||||
service.</para></listitem>
|
|
||||||
|
|
||||||
<listitem><para>The bucket must be within the
|
|
||||||
<literal>us-east-1</literal> region.</para></listitem>
|
|
||||||
|
|
||||||
<listitem><para>The Amazon credentials, if stored in a credential
|
|
||||||
profile, must be stored in the <literal>default</literal>
|
|
||||||
profile.</para></listitem>
|
|
||||||
</itemizedlist>
|
|
||||||
|
|
||||||
<para>Your bucket will need a bucket policy allowing the desired
|
|
||||||
users to perform the <literal>s3:GetObject</literal> action on all
|
|
||||||
objects in the bucket.</para>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
|
|
||||||
<section xml:id="ssec-s3-substituter-authenticated-writes">
|
|
||||||
<title>Authenticated Writes to your S3-compatible binary cache</title>
|
|
||||||
|
|
||||||
<para>Nix support fully supports writing to Amazon S3 and S3
|
|
||||||
compatible buckets. The binary cache URL for our example bucket will
|
|
||||||
be <uri>s3://example-bucket</uri>.</para>
|
|
||||||
|
|
||||||
<para>Nix will use the <link
|
|
||||||
xlink:href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default.">default
|
|
||||||
credential provider chain</link> for authenticating requests to
|
|
||||||
Amazon S3.</para>
|
|
||||||
|
|
||||||
<para>The following options can be specified as URL parameters to
|
|
||||||
the S3 URL:</para>
|
|
||||||
<variablelist>
|
|
||||||
<varlistentry><term><literal>profile</literal></term>
|
<varlistentry><term><literal>profile</literal></term>
|
||||||
<listitem>
|
<listitem>
|
||||||
<para>
|
<para>
|
||||||
|
|
@ -116,6 +31,11 @@ fetch prebuilt binaries from <uri>cache.nixos.org</uri>.</para>
|
||||||
The region of the S3 bucket. <literal>us–east-1</literal> by
|
The region of the S3 bucket. <literal>us–east-1</literal> by
|
||||||
default.
|
default.
|
||||||
</para>
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
If your bucket is not in <literal>us–east-1</literal>, you
|
||||||
|
should always explicitly specify the region parameter.
|
||||||
|
</para>
|
||||||
</listitem>
|
</listitem>
|
||||||
</varlistentry>
|
</varlistentry>
|
||||||
|
|
||||||
|
|
@ -131,29 +51,121 @@ fetch prebuilt binaries from <uri>cache.nixos.org</uri>.</para>
|
||||||
addressing.</para></note>
|
addressing.</para></note>
|
||||||
</listitem>
|
</listitem>
|
||||||
</varlistentry>
|
</varlistentry>
|
||||||
</variablelist>
|
</variablelist>
|
||||||
|
|
||||||
<example><title>Uploading with non-default credential profile for Amazon S3</title>
|
<para>In this example we will use the bucket named
|
||||||
<para><command>nix copy --to ssh://machine nixpkgs.hello s3://example-bucket?profile=cache-upload</command></para>
|
<literal>example-nix-cache</literal>.</para>
|
||||||
|
|
||||||
|
<section xml:id="ssec-s3-substituter-anonymous-reads">
|
||||||
|
<title>Anonymous Reads to your S3-compatible binary cache</title>
|
||||||
|
|
||||||
|
<para>If your binary cache is publicly accessible and does not
|
||||||
|
require authentication, the simplest and easiest way to use Nix with
|
||||||
|
your S3 compatible binary cache is to use the HTTP URL for that
|
||||||
|
cache.</para>
|
||||||
|
|
||||||
|
<para>For AWS S3 the binary cache URL for example bucket will be
|
||||||
|
exactly <uri>https://example-nix-cache.s3.amazonaws.com</uri> or
|
||||||
|
<uri>s3://example-nix-cache</uri>. For S3 compatible binary caches,
|
||||||
|
consult that cache's documentation.</para>
|
||||||
|
|
||||||
|
<para>Your bucket will need the following bucket policy:</para>
|
||||||
|
|
||||||
|
<programlisting><![CDATA[
|
||||||
|
{
|
||||||
|
"Id": "DirectReads",
|
||||||
|
"Version": "2012-10-17",
|
||||||
|
"Statement": [
|
||||||
|
{
|
||||||
|
"Sid": "AlowDirectReads",
|
||||||
|
"Action": [
|
||||||
|
"s3:GetObject",
|
||||||
|
"s3:GetBucketLocation"
|
||||||
|
],
|
||||||
|
"Effect": "Allow",
|
||||||
|
"Resource": [
|
||||||
|
"arn:aws:s3:::example-nix-cache",
|
||||||
|
"arn:aws:s3:::example-nix-cache/*"
|
||||||
|
],
|
||||||
|
"Principal": "*"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]]></programlisting>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section xml:id="ssec-s3-substituter-authenticated-reads">
|
||||||
|
<title>Authenticated Reads to your S3 binary cache</title>
|
||||||
|
|
||||||
|
<para>For AWS S3 the binary cache URL for example bucket will be
|
||||||
|
exactly <uri>s3://example-nix-cache</uri>.</para>
|
||||||
|
|
||||||
|
<para>Nix will use the <link
|
||||||
|
xlink:href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default.">default
|
||||||
|
credential provider chain</link> for authenticating requests to
|
||||||
|
Amazon S3.</para>
|
||||||
|
|
||||||
|
<para>Nix supports authenticated reads from Amazon S3 and S3
|
||||||
|
compatible binary caches.</para>
|
||||||
|
|
||||||
|
<para>Your bucket will need a bucket policy allowing the desired
|
||||||
|
users to perform the <literal>s3:GetObject</literal> and
|
||||||
|
<literal>s3:GetBucketLocation</literal> action on all objects in the
|
||||||
|
bucket. The anonymous policy in <xref
|
||||||
|
linkend="ssec-s3-substituter-anonymous-reads" /> can be updated to
|
||||||
|
have a restricted <literal>Principal</literal> to support
|
||||||
|
this.</para>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
|
||||||
|
<section xml:id="ssec-s3-substituter-authenticated-writes">
|
||||||
|
<title>Authenticated Writes to your S3-compatible binary cache</title>
|
||||||
|
|
||||||
|
<para>Nix support fully supports writing to Amazon S3 and S3
|
||||||
|
compatible buckets. The binary cache URL for our example bucket will
|
||||||
|
be <uri>s3://example-nix-cache</uri>.</para>
|
||||||
|
|
||||||
|
<para>Nix will use the <link
|
||||||
|
xlink:href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default.">default
|
||||||
|
credential provider chain</link> for authenticating requests to
|
||||||
|
Amazon S3.</para>
|
||||||
|
|
||||||
|
<para>Your account will need the following IAM policy to
|
||||||
|
upload to the cache:</para>
|
||||||
|
|
||||||
|
<programlisting><![CDATA[
|
||||||
|
{
|
||||||
|
"Version": "2012-10-17",
|
||||||
|
"Statement": [
|
||||||
|
{
|
||||||
|
"Sid": "UploadToCache",
|
||||||
|
"Effect": "Allow",
|
||||||
|
"Action": [
|
||||||
|
"s3:AbortMultipartUpload",
|
||||||
|
"s3:GetBucketLocation",
|
||||||
|
"s3:GetObject",
|
||||||
|
"s3:ListBucket",
|
||||||
|
"s3:ListBucketMultipartUploads",
|
||||||
|
"s3:ListMultipartUploadParts",
|
||||||
|
"s3:ListObjects",
|
||||||
|
"s3:PutObject"
|
||||||
|
],
|
||||||
|
"Resource": [
|
||||||
|
"arn:aws:s3:::example-nix-cache",
|
||||||
|
"arn:aws:s3:::example-nix-cache/*"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]]></programlisting>
|
||||||
|
|
||||||
|
|
||||||
|
<example><title>Uploading with a specific credential profile for Amazon S3</title>
|
||||||
|
<para><command>nix copy --to 's3://example-nix-cache?profile=cache-upload&region=eu-west-2' nixpkgs.hello</command></para>
|
||||||
</example>
|
</example>
|
||||||
|
|
||||||
<example><title>Uploading to an S3-Compatible Binary Cache</title>
|
<example><title>Uploading to an S3-Compatible Binary Cache</title>
|
||||||
<para><command>nix copy --to ssh://machine nixpkgs.hello s3://example-bucket?profile=cache-upload&endpoint=minio.example.com</command></para>
|
<para><command>nix copy --to 's3://example-nix-cache?profile=cache-upload&endpoint=minio.example.com' nixpkgs.hello</command></para>
|
||||||
</example>
|
</example>
|
||||||
|
|
||||||
<para>The user writing to the bucket will need to perform the
|
|
||||||
following actions against the bucket:</para>
|
|
||||||
|
|
||||||
<itemizedlist>
|
|
||||||
<listitem><para><literal>s3:ListBucket</literal></para></listitem>
|
|
||||||
<listitem><para><literal>s3:GetBucketLocation</literal></para></listitem>
|
|
||||||
<listitem><para><literal>s3:ListObjects</literal></para></listitem>
|
|
||||||
<listitem><para><literal>s3:GetObject</literal></para></listitem>
|
|
||||||
<listitem><para><literal>s3:PutObject</literal></para></listitem>
|
|
||||||
<listitem><para><literal>s3:ListBucketMultipartUploads</literal></para></listitem>
|
|
||||||
<listitem><para><literal>s3:CreateMultipartUpload</literal></para></listitem>
|
|
||||||
<listitem><para><literal>s3:ListMultipartUploadParts</literal></para></listitem>
|
|
||||||
<listitem><para><literal>s3:AbortMultipartUpload</literal></para></listitem>
|
|
||||||
</itemizedlist>
|
|
||||||
</section>
|
</section>
|
||||||
</section>
|
</section>
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue