lix-project/lix

Fork 54

merge/implement CppNix 2.19+ feature: builtins.convertHash #602

New issue

Open

opened 2024-12-17 12:16:03 +00:00 by benaryorg · 9 comments

benaryorg commented

2024-12-17 12:16:03 +00:00

I wanted to take a shot at implementing a Rust build system on crate-level that doesn't require IFD when provided with a Cargo.lock (which should be possible since the entire spec can be generated from the lockfile, that's what its purpose is).
However, not being able to convert the native sha256 to SRI hashes is a showstopper.

The feature exists in CppNix 2.19+ however.

Describe the solution you'd like

Merging that specific feature, kinda like:

# the code is provided "as is" and the author disclaims all warranties (not sure if it works)
git fetch cppnix refs/pull/7708/head && git rebase --onto main 201c115c3e242e3db60a2cd1583223d4e196d6e5 FETCH_HEAD

Though I haven't checked if there are merge conflicts, and due to ongoing Lix development there may be other reasons to not just merge it of course.

Describe alternatives you've considered

I do not want to reimplement hash conversion in a language where parsing octal numbers is pretty much impossible without manual parsing
not using IFDs to shell out and get it converted with shellscripts was the entire point of the endeavor

## Is your feature request related to a problem? Please describe. I wanted to take a shot at implementing a Rust build system on crate-level that *doesn't* require IFD when provided with a *Cargo.lock* (which should be possible since the entire spec can be generated from the lockfile, that's what its purpose is). However, not being able to convert the native sha256 to SRI hashes is a showstopper. The [feature exists in CppNix 2.19+](https://github.com/NixOS/nix/pull/7708) however. ## Describe the solution you'd like Merging that specific feature, kinda like: ```bash # the code is provided "as is" and the author disclaims all warranties (not sure if it works) git fetch cppnix refs/pull/7708/head && git rebase --onto main 201c115c3e242e3db60a2cd1583223d4e196d6e5 FETCH_HEAD ``` Though I haven't checked if there are merge conflicts, and due to ongoing Lix development there may be other reasons to not just merge it of course. ## Describe alternatives you've considered - I do *not* want to reimplement hash conversion in a language where parsing octal numbers is pretty much impossible without manual parsing - not using IFDs to shell out and get it converted with shellscripts was the entire point of the endeavor

lix-bot commented

2024-12-17 14:23:40 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

commit message in cl/2313 ("libexpr: Add builtins.convertHash")

This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/2313](https://gerrit.lix.systems/c/lix/+/2313) ("libexpr: Add builtins.convertHash")

raito commented

2024-12-23 17:19:59 +00:00

Owner

Hi there, what about generalizing the built-in to allow arbitrary baseXYZ transformations? This would be useful for much more than builtins.convertHash which would be a specialization of that general builtin.

Hi there, what about generalizing the built-in to allow arbitrary baseXYZ transformations? This would be useful for much more than `builtins.convertHash` which would be a specialization of that general builtin.

raito commented

2024-12-23 17:20:05 +00:00

Owner

cc @tom-hubrecht

benaryorg commented

2024-12-25 00:19:21 +00:00

Author

Hi there, what about generalizing the built-in to allow arbitrary baseXYZ transformations?

As a potential user of said code; yes, but also "arbitrary baseXYZ transformations" may be a headache to provide since I know at least three different ways to do base32 alone.
Since CppNix already had the hash conversion I figured porting that made the most sense (since it provides compatibility too).

IMHO any additional APIs beyond that one would probably be better expressed in a design document featuring current annoyances in general so that some general patterns can be found.
Remember how I mentioned that octal numbers are basically just not a thing in Nix, so providing something as simple as a radix based conversion (with an optional mapping table) would make implementing a custom baseXYZ implementation a lot easier (exploding the string, using the x:xs pattern using the head and tail builtins and recursion, and shoving the x into the radix conversion function) and it would solve two issues at once.
Like, have you ever tried to properly parse/format an IPv6 address in Nix?
Half my code in that regard makes strict assumptions about format and prefix length just because anything else would be awful to implement (currently), and parsing and formatting hex numbers is the minimum required to even start doing that and that one is hidden within nixpkgs' lib if memory serves (then again I'd prefer being able to treat an IPv6 address as a single 128 bit number for "easy math" reasons).
Same with a lot of other formatting things like padding (space or zero) for whatever reason, etc.
I think there should be a larger collection of annoyances (similar to the Wiki's Nix lang v2 page) so that they can be

TL;DR: I think generic baseXYZ conversions might be hard to implement (for Layer 8 reasons) and would solve a specific subset of problems potentially leading to more feature creep than a more generic approach.

> Hi there, what about generalizing the built-in to allow arbitrary baseXYZ transformations? As a potential user of said code; yes, but also "arbitrary baseXYZ transformations" may be a headache to provide since I know at least three different ways to do base32 alone. Since CppNix already had the hash conversion I figured porting that made the most sense (since it provides compatibility too). IMHO any additional APIs beyond that one would probably be better expressed in a design document featuring current annoyances in general so that some general patterns can be found. Remember how I mentioned that octal numbers are basically just not a thing in Nix, so providing something as simple as a radix based conversion (with an optional mapping table) would make implementing a custom baseXYZ implementation a lot easier (exploding the string, using the `x:xs` pattern using the *head* and *tail* builtins and recursion, and shoving the *x* into the radix conversion function) and it would solve two issues at once. Like, have you ever tried to properly parse/format an IPv6 address in Nix? Half my code in that regard makes strict assumptions about format and prefix length just because anything else would be awful to implement (currently), and parsing and formatting hex numbers is the minimum required to even start doing that and that one is hidden within nixpkgs' *lib* if memory serves (then again I'd prefer being able to treat an IPv6 address as a single 128 bit number for "easy math" reasons). Same with a lot of [other formatting things](https://github.com/NixOS/nix/issues/5031) like padding (space or zero) for whatever reason, etc. I think there should be a larger collection of annoyances (similar to the [Wiki's Nix lang v2 page](https://wiki.lix.systems/books/lix-contributors/page/nix-lang-v2)) so that they can be TL;DR: I think generic baseXYZ conversions might be hard to implement (for Layer 8 reasons) and would solve a specific subset of problems *potentially* leading to more feature creep than a more generic approach.

raito commented

2024-12-25 14:07:35 +00:00

Owner

I agree with you regarding base32 alone (I worked on the base32 implementation in Tvix and I saw terrible things.).

Like, have you ever tried to properly parse/format an IPv6 address in Nix?

Hahahaha… :'(

Remember how I mentioned that octal numbers are basically just not a thing in Nix, so providing something as simple as a radix based conversion (with an optional mapping table) would make implementing a custom baseXYZ implementation a lot easier (exploding the string, using the x:xs pattern using the head and tail builtins and recursion, and shoving the x into the radix conversion function) and it would solve two issues at once.

TL;DR: I think generic baseXYZ conversions might be hard to implement (for Layer 8 reasons) and would solve a specific subset of problems potentially leading to more feature creep than a more generic approach.

What I gather from here is that you are for an even more generic way to generate baseXYZ conversions, e.g. radix based conversions, I think we are in agreement. Nonetheless, I believe this will be slower, right? Are you fine with that?

Since CppNix already had the hash conversion I figured porting that made the most sense (since it provides compatibility too).

IMHO any additional APIs beyond that one would probably be better expressed in a design document featuring current annoyances in general so that some general patterns can be found.

You are totally right, though we can have compatibility just by shipping a compat library in Nix code and this makes it easier for us to do things. We can also maybe invest in technology to hide certain builtins to prevent folks to depend on them and only have the libcompat Nix code access them. This is more work but this is well invested efforts rather than copying blindly CppNix who are effectively piling features without any reasoning on the global level (even though some of their developers try hard to keep things clean).

Would you be willing to drive this work? On our side, we can help on the design, review, etc. getting this to the finish line.

I agree with you regarding base32 alone (I worked on the base32 implementation in Tvix and I saw terrible things.). > Like, have you ever tried to properly parse/format an IPv6 address in Nix? Hahahaha… :'( > Remember how I mentioned that octal numbers are basically just not a thing in Nix, so providing something as simple as a radix based conversion (with an optional mapping table) would make implementing a custom baseXYZ implementation a lot easier (exploding the string, using the x:xs pattern using the head and tail builtins and recursion, and shoving the x into the radix conversion function) and it would solve two issues at once. > TL;DR: I think generic baseXYZ conversions might be hard to implement (for Layer 8 reasons) and would solve a specific subset of problems potentially leading to more feature creep than a more generic approach. What I gather from here is that you are for an even more generic way to generate baseXYZ conversions, e.g. radix based conversions, I think we are in agreement. Nonetheless, I believe this will be slower, right? Are you fine with that? > Since CppNix already had the hash conversion I figured porting that made the most sense (since it provides compatibility too). > IMHO any additional APIs beyond that one would probably be better expressed in a design document featuring current annoyances in general so that some general patterns can be found. You are totally right, though we can have compatibility just by shipping a compat library in Nix code and this makes it easier for us to do things. We can also maybe invest in technology to hide certain builtins to prevent folks to depend on them and only have the libcompat Nix code access them. This is more work but this is well invested efforts rather than copying blindly CppNix who are effectively piling features without any reasoning on the global level (even though some of their developers try hard to keep things clean). Would you be willing to drive this work? On our side, we can help on the design, review, etc. getting this to the finish line.

jade added the

Area/language

label

2025-01-07 17:24:56 +00:00

jade added the

Status

wontfix

label

2025-01-19 01:45:05 +00:00

jade commented

2025-01-19 01:45:30 +00:00

Owner

I think the status of the discussion above is that we want to implement something different. Is that a correct read?

I *think* the status of the discussion above is that we want to implement something different. Is that a correct read?

jade closed this issue

2025-01-19 01:45:30 +00:00

raito commented

2025-01-19 12:41:08 +00:00

Owner

I think the status of the discussion above is that we want to implement something different. Is that a correct read?

Yeah, we should have the primitive that enable us to implement builtins.convertHash in pure Nix.

> I *think* the status of the discussion above is that we want to implement something different. Is that a correct read? Yeah, we should have the primitive that enable us to implement `builtins.convertHash` in pure Nix.

benaryorg commented

2025-01-19 17:43:58 +00:00

Author

Ah, sorry for being unresponsive.

Being able to implement this in pure Nix would be great, however I do not have the capacity to do any of that right now (and I am forever grateful for every single person contributing to the software I use when I can't).

Personally worry about compat to CppNix if (big If, and also when (big When, given e.g. nixpkgs compat)) the convertHash builtin sees practical use elsewhere, but given that there is a plan to implement it (in pure Nix), I don't see this being a problem.

As for the underlying design of a conversion function (and potential radix builtins/functions) I tried to think about it but couldn't quite figure out a way that isn't a big footgun in some shape or form given that a hash isn't much more than a multi-byte number in some radix representation, and integer numbers not being arbitrarily big in the Nix language, so a conversion function of radix' would either have to actually convert from one to the other directly, or expose an opaque intermediate type, or have some limits in respects to length or size (requiring user-side chunking or such).

Ah, sorry for being unresponsive. Being *able* to implement this in pure Nix would be great, however I do not have the capacity to do any of that right now (and I am forever grateful for every single person contributing to the software I use when I can't). Personally worry about compat to CppNix if (big If, and also when (big When, given e.g. nixpkgs compat)) the *convertHash* builtin sees practical use elsewhere, but given that there is a plan to implement it (in pure Nix), I don't see this being a problem. As for the underlying design of a conversion function (and potential radix builtins/functions) I tried to think about it but couldn't quite figure out a way that isn't a big footgun in some shape or form given that a hash isn't much more than a multi-byte number in some radix representation, and integer numbers not being arbitrarily big in the Nix language, so a conversion function of radix' would either have to actually convert from one to the other directly, or expose an opaque intermediate type, or have some limits in respects to length or size (requiring user-side chunking or such).

jade referenced this issue

2025-07-31 06:50:34 +00:00

Implement builtins.convertHash #946

raito commented

2025-08-03 14:15:22 +00:00

Owner

Reopening as I do not consider this a true "wontfix". This can be re-implemented in pure Nix using another builtin radixConvert, with minimal built-in support. radixConvert has massive usefulness beyond just converting hashes.

Current status

convertHash has been ported in CL 2313, but rejected because we do not deem this should be a built-in.
A general-purpose radixConvert primitive is introduced in CL 2585, which should enable base conversions between base16, base32, base64, and raw binary.
radixConvert is not yet merged, it needs rework according to the -2 message.

Next steps

Finalize and merge radixConvert ideally with a convertHash as a validation that the built-in works, perhaps some IPv4/IPv6 exercises for it as well
Rebase and clean up CL 2313 to remove native code and use pure Nix
Add a thin wrapper as a built-in or nixpkgs lib helper for convertHash

I'm happy to review anything on that area, but as I do not use convertHash at all, I fail to find the time to take care of this.

Reopening as I do not consider this a true "wontfix". This can be re-implemented in pure Nix using another builtin `radixConvert`, with minimal built-in support. `radixConvert` has massive usefulness beyond just converting hashes. ## Current status * `convertHash` has been ported in [CL 2313](https://gerrit.lix.systems/c/lix/+/2313), but rejected because we do not deem this should be a built-in. * A general-purpose `radixConvert` primitive is introduced in [CL 2585](https://gerrit.lix.systems/c/lix/+/2585), which **should** enable base conversions between `base16`, `base32`, `base64`, and raw binary. * `radixConvert` is not yet merged, it needs rework according to the -2 message. ## Next steps * Finalize and merge `radixConvert` ideally with a `convertHash` as a validation that the built-in works, perhaps some IPv4/IPv6 exercises for it as well * Rebase and clean up CL 2313 to remove native code and use pure Nix * Add a thin wrapper as a built-in or nixpkgs lib helper for `convertHash` I'm happy to review anything on that area, but as I do not use `convertHash` at all, I fail to find the time to take care of this.