lix-project/lix

Fork 14

kill aws-sdk-cpp with fire #272

New issue

Open

opened 2024-05-06 05:09:39 +00:00 by jade · 4 comments

jade commented

2024-05-06 05:09:39 +00:00

Owner

lix can't just remove it and use the libcurl builtin s3 support because handling aws credential files is hard

@artemist

challenge accepted. i would even go so far as saying we could just take the aws sdk code for credential files, vendor it, then toss the entire rest in the bin, if that's all we need.

see also #246

> lix can't just remove it and use the libcurl builtin s3 support because handling aws credential files is hard - @artemist challenge accepted. i would even go so far as saying we could just take the aws sdk code for credential files, vendor it, then toss the entire rest in the bin, if that's all we need. see also https://git.lix.systems/lix-project/lix/issues/246

👍 1

jade added the

devx

E/help wanted

labels 2024-05-06 05:09:39 +00:00

artemist commented

2024-05-06 05:10:20 +00:00

Member

please... cross-compiling aws-cpp-sdk from linux to freebsd is a huge pain

puck commented

2024-05-06 05:27:43 +00:00

Owner

Related: using the curl support to add s3 capability https://gerrit.lix.systems/c/lix/+/926

jade commented

2024-05-06 06:09:03 +00:00

Author

Owner

https://github.com/rusoto/rusoto/tree/master/rusoto/credential maybe of interest? could we replace this (actually very complex) creds code with statically linking more rust into our process?

👀 1

delroth commented

2024-05-20 11:05:15 +00:00

Member

(Rusoto implements that too, but only if you also depend on stuff like Hyper, and uh do we really want another HTTP client statically linked? Doubtful that it would do much good to the compile times / dependency chains at this point.)

IMO what we should however do is:

Support "simple" and explicit credentials loading natively (e.g. s3://mycache?creds-file=/run/secrets/creds.txt). This has other benefits, like not relying on ambient environment (IMO a pretty bad misfeature in most cases...).
Making the aws-sdk-cpp dependency lighter by making it an ambient creds loader only and using curl instead of S3Client (#246) for s3:// URLs.

The combination of (1) and (2) allows for Lix builds without aws-sdk-cpp that can still do basic S3 usage, especially in non-AWS environments.

I'm writing a short design to describe all that and make sure we agree on the details and unless that conflicts with anyone else's plans I'll probably work on it in the following days/weeks.

We discussed about this topic yesterday with @puck and one thing that came up is that entirely replacing aws-sdk-cpp with a custom implementation is likely out of the question. In particular, we know some users rely on features like "fetching instance credentials from IMDS for EC2 machines", and... yeah, you *can* reimplement that, but it starts getting hairy. Long term Lix could make the decision to drop those features, but this is something that should be carefully assessed. (Rusoto implements that too, but only if you also depend on stuff like Hyper, and uh do we really want another HTTP client statically linked? Doubtful that it would do much good to the compile times / dependency chains at this point.) IMO what we should however do is: 1. Support "simple" and explicit credentials loading natively (e.g. `s3://mycache?creds-file=/run/secrets/creds.txt`). This has other benefits, like not relying on ambient environment (IMO a pretty bad misfeature in most cases...). 2. Making the aws-sdk-cpp dependency lighter by making it an ambient creds loader only and using curl instead of S3Client (#246) for s3:// URLs. The combination of (1) and (2) allows for Lix builds without aws-sdk-cpp that can still do basic S3 usage, especially in non-AWS environments. I'm writing a short design to describe all that and make sure we agree on the details and unless that conflicts with anyone else's plans I'll probably work on it in the following days/weeks.

No milestone

No project

No assignees

4 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#272

No description provided.