lix-project/lix

Fork 31

kill aws-sdk-cpp with fire #272

New issue

Open

opened 2024-05-06 05:09:39 +00:00 by jade · 5 comments

jade commented

2024-05-06 05:09:39 +00:00

Owner

lix can't just remove it and use the libcurl builtin s3 support because handling aws credential files is hard

@artemist

challenge accepted. i would even go so far as saying we could just take the aws sdk code for credential files, vendor it, then toss the entire rest in the bin, if that's all we need.

see also #246

> lix can't just remove it and use the libcurl builtin s3 support because handling aws credential files is hard - @artemist challenge accepted. i would even go so far as saying we could just take the aws sdk code for credential files, vendor it, then toss the entire rest in the bin, if that's all we need. see also https://git.lix.systems/lix-project/lix/issues/246

👍 1

jade added the

devx

E/help wanted

labels 2024-05-06 05:09:39 +00:00

artemist commented

2024-05-06 05:10:20 +00:00

Member

please... cross-compiling aws-cpp-sdk from linux to freebsd is a huge pain

puck commented

2024-05-06 05:27:43 +00:00

Owner

Related: using the curl support to add s3 capability https://gerrit.lix.systems/c/lix/+/926

jade commented

2024-05-06 06:09:03 +00:00

Author

Owner

https://github.com/rusoto/rusoto/tree/master/rusoto/credential maybe of interest? could we replace this (actually very complex) creds code with statically linking more rust into our process?

👀 1

delroth commented

2024-05-20 11:05:15 +00:00

Member

(Rusoto implements that too, but only if you also depend on stuff like Hyper, and uh do we really want another HTTP client statically linked? Doubtful that it would do much good to the compile times / dependency chains at this point.)

IMO what we should however do is:

Support "simple" and explicit credentials loading natively (e.g. s3://mycache?creds-file=/run/secrets/creds.txt). This has other benefits, like not relying on ambient environment (IMO a pretty bad misfeature in most cases...).
Making the aws-sdk-cpp dependency lighter by making it an ambient creds loader only and using curl instead of S3Client (#246) for s3:// URLs.

The combination of (1) and (2) allows for Lix builds without aws-sdk-cpp that can still do basic S3 usage, especially in non-AWS environments.

I'm writing a short design to describe all that and make sure we agree on the details and unless that conflicts with anyone else's plans I'll probably work on it in the following days/weeks.

We discussed about this topic yesterday with @puck and one thing that came up is that entirely replacing aws-sdk-cpp with a custom implementation is likely out of the question. In particular, we know some users rely on features like "fetching instance credentials from IMDS for EC2 machines", and... yeah, you *can* reimplement that, but it starts getting hairy. Long term Lix could make the decision to drop those features, but this is something that should be carefully assessed. (Rusoto implements that too, but only if you also depend on stuff like Hyper, and uh do we really want another HTTP client statically linked? Doubtful that it would do much good to the compile times / dependency chains at this point.) IMO what we should however do is: 1. Support "simple" and explicit credentials loading natively (e.g. `s3://mycache?creds-file=/run/secrets/creds.txt`). This has other benefits, like not relying on ambient environment (IMO a pretty bad misfeature in most cases...). 2. Making the aws-sdk-cpp dependency lighter by making it an ambient creds loader only and using curl instead of S3Client (#246) for s3:// URLs. The combination of (1) and (2) allows for Lix builds without aws-sdk-cpp that can still do basic S3 usage, especially in non-AWS environments. I'm writing a short design to describe all that and make sure we agree on the details and unless that conflicts with anyone else's plans I'll probably work on it in the following days/weeks.

kfearsoff commented

2024-09-17 11:07:07 +00:00

Member

AWS authentication is really complex and feature-rich. There's SSO, MFA, IMDS, IAM... aws-vault does a good job describing those. Maintaining that ourselves is impossible, vendoring is problematic long-term, and including aws-sdk-cpp is overkill.

I'd like to see a detailed proposal. But roughly speaking, I think we should support three auth schemas:

No auth
Basic auth
External auth (read creds from file? launch external process for auth?)

The one that puzzles me is number 3. With AWS S3, it's quite easy: you can just export AWS env vars and run your S3 queries. Not sure if it works for other S3-compatible solutions.

AWS authentication is really complex and feature-rich. There's SSO, MFA, IMDS, IAM... [aws-vault](https://github.com/99designs/aws-vault/blob/master/USAGE.md) does a good job describing those. Maintaining that ourselves is impossible, vendoring is problematic long-term, and including aws-sdk-cpp is overkill. I'd like to see a detailed proposal. But roughly speaking, I think we should support three auth schemas: 1. No auth 2. Basic auth 3. External auth (read creds from file? launch external process for auth?) The one that puzzles me is number 3. With AWS S3, it's quite easy: you can just export AWS env vars and run your S3 queries. Not sure if it works for other S3-compatible solutions.

jade added the

Area/fetching

label 2024-10-20 01:04:57 +00:00