shell.nix for new cli

pennae commented

2024-08-06 21:24:17 +00:00

Owner

I asked about an issue because there’re some design considerations I’d like to discuss (I’ll share them here, but feel free to point me to a more suitable venue).

old-style shell.nix doesn’t currently accept any arguments, it just returns a derivation (you kinda can pass arguments via --arg, but they require default values). that means that adding auto-filling for arguments with no default values is not a breaking change, which is nice. there’re some options as to which arguments get passed though:

flakes from the registry (so basically only nixpkgs for most shells). requires the shell to import nixpkgs manually every time, but allows e.g. overlays, so is the most flexible.
packages via callPackage, as nix-build does. very concise and pleasant for simple shells, but less flexible and doesn’t provide a clean way to access other flakes from the registry.
somehow both? placing them into the same namespace feels bad though, name conflicts are bound to happen. something like
```
{ flakes = { someFlake }, hello, mkDerivation }: ...
```
could potentially solve this problem, but that’s very much not a syntax today.

I’m leaning towards the first option, as it’s flexible, has simple semantics and doesn’t require new syntax, and I’ll try to write a basic implementation to see how that works. I’d be interested to hear another opinions on the issue though.

Originally posted by @goldstein in #170 (comment)

I asked about an issue because there’re some design considerations I’d like to discuss (I’ll share them here, but feel free to point me to a more suitable venue). old-style `shell.nix` doesn’t currently accept any arguments, it just returns a derivation (you kinda can pass arguments via `--arg`, but they require default values). that means that adding auto-filling for arguments with no default values is not a breaking change, which is nice. there’re some options as to which arguments get passed though: - flakes from the registry (so basically only `nixpkgs` for most shells). requires the shell to `import nixpkgs` manually every time, but allows e.g. overlays, so is the most flexible. - packages via `callPackage`, as `nix-build` does. very concise and pleasant for simple shells, but less flexible and doesn’t provide a clean way to access other flakes from the registry. - somehow both? placing them into the same namespace feels bad though, name conflicts are bound to happen. something like ```nix { flakes = { someFlake }, hello, mkDerivation }: ... ``` could potentially solve this problem, but that’s very much not a syntax today. I’m leaning towards the first option, as it’s flexible, has simple semantics and doesn’t require new syntax, and I’ll try to write a basic implementation to see how that works. I’d be interested to hear another opinions on the issue though. _Originally posted by @goldstein in https://git.lix.systems/lix-project/lix/issues/170#issuecomment-5464_

❤️ 1

goldstein commented

2024-08-06 21:25:58 +00:00

Member

I’ll try to tinker with possible implementations.

pennae commented

2024-08-06 21:28:41 +00:00

Author

Owner

the first option seems like a good thing to try first, yes. it's also closest to the current old-cli shell.nix pattern of starting everything with { pkgs ? import <nixpkgs> {} }:, and the additional let needed to actually import things seems fine for letting us avoid the NIX_PATH shenanigans the old style relied upon. (and we can even use flakes as flakes by getFlakeing the args if needed, so we don't lose anything)

as long as it's behind an experimental feature gate we're free to experiment and break things if a better way becomes evident. :)

the first option seems like a good thing to try first, yes. it's also closest to the current old-cli shell.nix pattern of starting everything with `{ pkgs ? import <nixpkgs> {} }:`, and the additional let needed to actually import things seems fine for letting us avoid the `NIX_PATH` shenanigans the old style relied upon. (and we can even use flakes as flakes by `getFlake`ing the args if needed, so we don't lose anything) as long as it's behind an experimental feature gate we're free to experiment and break things if a better way becomes evident. :)

jade commented

2024-08-06 23:06:19 +00:00

Owner

fyi you may be interested in qyriad's project xil https://github.com/qyriad/xil

❤️ 1

jade added the

ux

label 2024-08-07 06:02:03 +00:00

goldstein commented

2024-08-07 13:04:35 +00:00

Member

another consideration is whether this should work as nix shell or as nix-shell. I’d personally like nix-shell/nix develop-like behaviour, so e.g. C headers get propagated. I could just add new semantics to nix develop instead of nix shell, but it isn’t quite right either, because nix develop ignores $SHELL by default so it can inject bash functions into the environment. I started writing it as a separate command for a proof of concept, but it should probably be integrated into either nix shell or nix develop later.

another consideration is whether this should work as `nix shell` or as `nix-shell`. I’d personally like `nix-shell`/`nix develop`-like behaviour, so e.g. C headers get propagated. I could just add new semantics to `nix develop` instead of `nix shell`, but it isn’t quite right either, because `nix develop` ignores `$SHELL` by default so it can inject bash functions into the environment. I started writing it as a separate command for a proof of concept, but it should probably be integrated into either `nix shell` or `nix develop` later.

goldstein commented

2024-08-30 08:25:16 +00:00

Member

I played around with this for a bit, here’re some thoughts:

it’s kind of easier to implement this out-of-tree than in-tree. I thought to reuse nix develop logic, but nix develop basically generates some bespoke Bash scripts and then evaluates them. we can generate similar scripts with nix print-dev-env. it’s still very doable in-tree, but I experimented mostly out-of-tree because I’m not that comfortable with Lix codebase yet.
“half-locking” (locking inputs that do not come from registry) can be achieved with a syntax like
```
{
  nixpkgs,
  someInput ? locked "github:some/input",
}: ...
```
where locked is a function that resolves input via a local lockfile, or falls back to getFlake if needed.
I really want this to be an impure evaluation so we can avoid preemptively copying all the files to store (this is a big slowdown with my current flake-based workflow), but impure evaluations are not cached. out-of-tree tool can just cache evaluations (+ instantiations) in some local database, but it would be kinda weird to do this for a proper Lix subcommand.
the big problem is LSP integration. nixd supports flakes, but any new bespoke format would have to be explicitly supported so it knows what gets passed to the closure arguments. this is a point in favour of in-tree implementation, because supporting an “official” feature makes more sense for LSP than supporting a random external tool. I’ll have to look into nixd code though, maybe there is a way to teach it a custom format without literally hardcoding it.

I played around with this for a bit, here’re some thoughts: - it’s kind of easier to implement this out-of-tree than in-tree. I thought to reuse `nix develop` logic, but `nix develop` basically generates some bespoke Bash scripts and then evaluates them. we can generate similar scripts with `nix print-dev-env`. it’s still very doable in-tree, but I experimented mostly out-of-tree because I’m not that comfortable with Lix codebase yet. - “half-locking” (locking inputs that do not come from registry) can be achieved with a syntax like ```nix { nixpkgs, someInput ? locked "github:some/input", }: ... ``` where `locked` is a function that resolves input via a local lockfile, or falls back to `getFlake` if needed. - I really want this to be an impure evaluation so we can avoid preemptively copying all the files to store (this is a big slowdown with my current flake-based workflow), but impure evaluations are not cached. out-of-tree tool can just cache evaluations (+ instantiations) in some local database, but it would be kinda weird to do this for a proper Lix subcommand. - the big problem is LSP integration. nixd supports flakes, but any new bespoke format would have to be explicitly supported so it knows what gets passed to the closure arguments. this is a point in favour of in-tree implementation, because supporting an “official” feature makes more sense for LSP than supporting a random external tool. I’ll have to look into nixd code though, maybe there is a way to teach it a custom format without literally hardcoding it.

jade commented

2024-08-30 10:30:45 +00:00

Owner

there is an external tool here that implements a plainly better eval cache by just tracing nix io: https://github.com/xzfc/cached-nix-shell

this, ironically, actually is better for cache behaviour than flakes in a certain sense because flakes get moved to the store under different paths all the time and so this strategy simply wouldn't work because the absolute paths keep changing (and this is visible to nix code too; ouch).

there is an external tool here that implements a plainly better eval cache by just tracing nix io: https://github.com/xzfc/cached-nix-shell this, ironically, actually is better for cache behaviour than flakes in a certain sense because flakes get moved to the store under different paths all the time and so this strategy simply wouldn't work because the absolute paths keep changing (and this is visible to nix code too; ouch).

goldstein commented

2024-08-30 10:41:22 +00:00

Member

“just tracing all the files Nix accesses” is a cool strategy, thanks for the link! it’s somewhat more complex than what I had in mind, but it’s even better. I’ll try to reuse it if I make this as an external tool.

it still would feel somewhat weird for a built-in command (and it’s hard to imagine how that would look in code). AFAIK Lix doesn’t currently store stuff (aside from repl history) outside of the, well, store, so it would be a big exception to the general rules.

I wonder how hard it is to “internally intercept” file accesses of the evaluator to make a list without syscall hooking. even if implementing “new shell.nix” as an external tool, this could be provided as an API like nix eval --dump-file-deps.

“just tracing all the files Nix accesses” is a cool strategy, thanks for the link! it’s somewhat more complex than what I had in mind, but it’s even better. I’ll try to reuse it if I make this as an external tool. it still would feel somewhat weird for a built-in command (and it’s hard to imagine how that would look in code). AFAIK Lix doesn’t currently store stuff (aside from repl history) outside of the, well, store, so it would be a big exception to the general rules. I wonder how hard it is to “internally intercept” file accesses of the evaluator to make a list without syscall hooking. even if implementing “new shell.nix” as an external tool, this could be provided as an API like `nix eval --dump-file-deps`.

jade commented

2024-08-30 18:24:14 +00:00

Owner

.local/share/nix, .cache/nix beg to differ. fetchGit, fetchTarball (without hashes; there's a cache of what hash was last obtained when no hash is given), flake eval cache, etc do all store things.

we would never ship a dylib crimes tracer like that though; much easier to make the evaluator io itself possible to intercept, also enabling a lazy trees like optimization one day.

.local/share/nix, .cache/nix beg to differ. fetchGit, fetchTarball (without hashes; there's a cache of what hash was last obtained when no hash is given), flake eval cache, etc do all store things. we would never ship a dylib crimes tracer like that though; much easier to make the evaluator io itself possible to intercept, also enabling a lazy trees like optimization one day.

goldstein commented

2024-08-30 19:38:05 +00:00

Member

TIL about fetcher cache, thanks. intercepting a bunch of primops is easy (I think it’s just __read{FileType,File,Dir}?), but there’re also fetchers which can read from disk, including indrectly, like via git. fetchers are already cached though, so maybe it’s fine to ignore them.

TIL about fetcher cache, thanks. intercepting a bunch of primops is easy (I think it’s just `__read{FileType,File,Dir}`?), but there’re also fetchers which can read from disk, including indrectly, like via `git`. fetchers are already cached though, so maybe it’s fine to ignore them.

shell.nix for new cli #463