lix-project/lix

Fork 32

merge/implement CppNix 2.19+ feature: builtins.convertHash #602

New issue

Open

opened 2024-12-17 12:16:03 +00:00 by benaryorg · 5 comments

benaryorg commented

2024-12-17 12:16:03 +00:00

I wanted to take a shot at implementing a Rust build system on crate-level that doesn't require IFD when provided with a Cargo.lock (which should be possible since the entire spec can be generated from the lockfile, that's what its purpose is).
However, not being able to convert the native sha256 to SRI hashes is a showstopper.

The feature exists in CppNix 2.19+ however.

Describe the solution you'd like

Merging that specific feature, kinda like:

# the code is provided "as is" and the author disclaims all warranties (not sure if it works)
git fetch cppnix refs/pull/7708/head && git rebase --onto main 201c115c3e242e3db60a2cd1583223d4e196d6e5 FETCH_HEAD

Though I haven't checked if there are merge conflicts, and due to ongoing Lix development there may be other reasons to not just merge it of course.

Describe alternatives you've considered

I do not want to reimplement hash conversion in a language where parsing octal numbers is pretty much impossible without manual parsing
not using IFDs to shell out and get it converted with shellscripts was the entire point of the endeavor

## Is your feature request related to a problem? Please describe. I wanted to take a shot at implementing a Rust build system on crate-level that *doesn't* require IFD when provided with a *Cargo.lock* (which should be possible since the entire spec can be generated from the lockfile, that's what its purpose is). However, not being able to convert the native sha256 to SRI hashes is a showstopper. The [feature exists in CppNix 2.19+](https://github.com/NixOS/nix/pull/7708) however. ## Describe the solution you'd like Merging that specific feature, kinda like: ```bash # the code is provided "as is" and the author disclaims all warranties (not sure if it works) git fetch cppnix refs/pull/7708/head && git rebase --onto main 201c115c3e242e3db60a2cd1583223d4e196d6e5 FETCH_HEAD ``` Though I haven't checked if there are merge conflicts, and due to ongoing Lix development there may be other reasons to not just merge it of course. ## Describe alternatives you've considered - I do *not* want to reimplement hash conversion in a language where parsing octal numbers is pretty much impossible without manual parsing - not using IFDs to shell out and get it converted with shellscripts was the entire point of the endeavor

lix-bot commented

2024-12-17 14:23:40 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

commit message in cl/2313 ("libexpr: Add builtins.convertHash")

This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/2313](https://gerrit.lix.systems/c/lix/+/2313) ("libexpr: Add builtins.convertHash")

raito commented

2024-12-23 17:19:59 +00:00

Owner

Hi there, what about generalizing the built-in to allow arbitrary baseXYZ transformations? This would be useful for much more than builtins.convertHash which would be a specialization of that general builtin.

Hi there, what about generalizing the built-in to allow arbitrary baseXYZ transformations? This would be useful for much more than `builtins.convertHash` which would be a specialization of that general builtin.

raito commented

2024-12-23 17:20:05 +00:00

Owner

cc @tom-hubrecht

benaryorg commented

2024-12-25 00:19:21 +00:00

Author

Hi there, what about generalizing the built-in to allow arbitrary baseXYZ transformations?

As a potential user of said code; yes, but also "arbitrary baseXYZ transformations" may be a headache to provide since I know at least three different ways to do base32 alone.
Since CppNix already had the hash conversion I figured porting that made the most sense (since it provides compatibility too).

IMHO any additional APIs beyond that one would probably be better expressed in a design document featuring current annoyances in general so that some general patterns can be found.
Remember how I mentioned that octal numbers are basically just not a thing in Nix, so providing something as simple as a radix based conversion (with an optional mapping table) would make implementing a custom baseXYZ implementation a lot easier (exploding the string, using the x:xs pattern using the head and tail builtins and recursion, and shoving the x into the radix conversion function) and it would solve two issues at once.
Like, have you ever tried to properly parse/format an IPv6 address in Nix?
Half my code in that regard makes strict assumptions about format and prefix length just because anything else would be awful to implement (currently), and parsing and formatting hex numbers is the minimum required to even start doing that and that one is hidden within nixpkgs' lib if memory serves (then again I'd prefer being able to treat an IPv6 address as a single 128 bit number for "easy math" reasons).
Same with a lot of other formatting things like padding (space or zero) for whatever reason, etc.
I think there should be a larger collection of annoyances (similar to the Wiki's Nix lang v2 page) so that they can be

TL;DR: I think generic baseXYZ conversions might be hard to implement (for Layer 8 reasons) and would solve a specific subset of problems potentially leading to more feature creep than a more generic approach.

> Hi there, what about generalizing the built-in to allow arbitrary baseXYZ transformations? As a potential user of said code; yes, but also "arbitrary baseXYZ transformations" may be a headache to provide since I know at least three different ways to do base32 alone. Since CppNix already had the hash conversion I figured porting that made the most sense (since it provides compatibility too). IMHO any additional APIs beyond that one would probably be better expressed in a design document featuring current annoyances in general so that some general patterns can be found. Remember how I mentioned that octal numbers are basically just not a thing in Nix, so providing something as simple as a radix based conversion (with an optional mapping table) would make implementing a custom baseXYZ implementation a lot easier (exploding the string, using the `x:xs` pattern using the *head* and *tail* builtins and recursion, and shoving the *x* into the radix conversion function) and it would solve two issues at once. Like, have you ever tried to properly parse/format an IPv6 address in Nix? Half my code in that regard makes strict assumptions about format and prefix length just because anything else would be awful to implement (currently), and parsing and formatting hex numbers is the minimum required to even start doing that and that one is hidden within nixpkgs' *lib* if memory serves (then again I'd prefer being able to treat an IPv6 address as a single 128 bit number for "easy math" reasons). Same with a lot of [other formatting things](https://github.com/NixOS/nix/issues/5031) like padding (space or zero) for whatever reason, etc. I think there should be a larger collection of annoyances (similar to the [Wiki's Nix lang v2 page](https://wiki.lix.systems/books/lix-contributors/page/nix-lang-v2)) so that they can be TL;DR: I think generic baseXYZ conversions might be hard to implement (for Layer 8 reasons) and would solve a specific subset of problems *potentially* leading to more feature creep than a more generic approach.

raito commented

2024-12-25 14:07:35 +00:00

Owner

I agree with you regarding base32 alone (I worked on the base32 implementation in Tvix and I saw terrible things.).

Like, have you ever tried to properly parse/format an IPv6 address in Nix?

Hahahaha… :'(

Remember how I mentioned that octal numbers are basically just not a thing in Nix, so providing something as simple as a radix based conversion (with an optional mapping table) would make implementing a custom baseXYZ implementation a lot easier (exploding the string, using the x:xs pattern using the head and tail builtins and recursion, and shoving the x into the radix conversion function) and it would solve two issues at once.

TL;DR: I think generic baseXYZ conversions might be hard to implement (for Layer 8 reasons) and would solve a specific subset of problems potentially leading to more feature creep than a more generic approach.

What I gather from here is that you are for an even more generic way to generate baseXYZ conversions, e.g. radix based conversions, I think we are in agreement. Nonetheless, I believe this will be slower, right? Are you fine with that?

Since CppNix already had the hash conversion I figured porting that made the most sense (since it provides compatibility too).

IMHO any additional APIs beyond that one would probably be better expressed in a design document featuring current annoyances in general so that some general patterns can be found.

You are totally right, though we can have compatibility just by shipping a compat library in Nix code and this makes it easier for us to do things. We can also maybe invest in technology to hide certain builtins to prevent folks to depend on them and only have the libcompat Nix code access them. This is more work but this is well invested efforts rather than copying blindly CppNix who are effectively piling features without any reasoning on the global level (even though some of their developers try hard to keep things clean).

Would you be willing to drive this work? On our side, we can help on the design, review, etc. getting this to the finish line.

I agree with you regarding base32 alone (I worked on the base32 implementation in Tvix and I saw terrible things.). > Like, have you ever tried to properly parse/format an IPv6 address in Nix? Hahahaha… :'( > Remember how I mentioned that octal numbers are basically just not a thing in Nix, so providing something as simple as a radix based conversion (with an optional mapping table) would make implementing a custom baseXYZ implementation a lot easier (exploding the string, using the x:xs pattern using the head and tail builtins and recursion, and shoving the x into the radix conversion function) and it would solve two issues at once. > TL;DR: I think generic baseXYZ conversions might be hard to implement (for Layer 8 reasons) and would solve a specific subset of problems potentially leading to more feature creep than a more generic approach. What I gather from here is that you are for an even more generic way to generate baseXYZ conversions, e.g. radix based conversions, I think we are in agreement. Nonetheless, I believe this will be slower, right? Are you fine with that? > Since CppNix already had the hash conversion I figured porting that made the most sense (since it provides compatibility too). > IMHO any additional APIs beyond that one would probably be better expressed in a design document featuring current annoyances in general so that some general patterns can be found. You are totally right, though we can have compatibility just by shipping a compat library in Nix code and this makes it easier for us to do things. We can also maybe invest in technology to hide certain builtins to prevent folks to depend on them and only have the libcompat Nix code access them. This is more work but this is well invested efforts rather than copying blindly CppNix who are effectively piling features without any reasoning on the global level (even though some of their developers try hard to keep things clean). Would you be willing to drive this work? On our side, we can help on the design, review, etc. getting this to the finish line.