lix-project/lix

Fork 31

An easy way to get a temporary shell with `withPackages` #462

New issue

Open

opened 2024-08-05 11:25:54 +00:00 by goldstein · 1 comment

goldstein commented

2024-08-05 11:25:54 +00:00

Member

I often want to get a nix shell with a bunch of Python packages. The current way to do it is something sad like

nix shell --impure --expr '{x = (import <nixpkgs> {}).python3.withPackages (y: with y; [ brotli fonttools ]);}' x

which is really annoying to type (I usually just search it in shell history, but still).

Describe the solution you'd like

nix-super has a bunch of goodies for Nix CLI interface, including

nix shell n#python3 --with '[ brotli fonttools ]'

which I think is really concise and nice.

Describe alternatives you've considered

This is kind of a band-aid though. A more general solution would be something like

nix shell 'n#python3.withPackages (p: with p; [brotli fonttools ])'

which is somewhat longer, but still gets rid of the most unpleasant parts like --impure and (import <nixpkgs> {}),
and also allows other fun stuff like overrides.

The problem here is that the word python3 in this invocation is overloaded: it’s simultaneously the name of a flake output and the start of the expression, which would require CLI to, like, half-parse the expression to split it. I don’t think it’s a big deal though, I think just splitting on the first dot (or |> to allow arbitrary functions) would be enough.

Another alternative would be to provide a separate syntax to map over chosen derivation, like

nix shell n#python3 --map '(py: py.withPackages (p: with p; [ brotli fonttools ]))'

This is even longer but just as expressive and sidesteps the parsing issue.

Additional context

If any of these gets accepted, I’d try to implement it.

## Is your feature request related to a problem? Please describe. I often want to get a nix shell with a bunch of Python packages. The current way to do it is something sad like ```shell nix shell --impure --expr '{x = (import <nixpkgs> {}).python3.withPackages (y: with y; [ brotli fonttools ]);}' x ``` which is really annoying to type (I usually just search it in shell history, but still). ## Describe the solution you'd like [nix-super](https://github.com/privatevoid-net/nix-super) has a bunch of goodies for Nix CLI interface, including ```shell nix shell n#python3 --with '[ brotli fonttools ]' ``` which I think is really concise and nice. ## Describe alternatives you've considered This is kind of a band-aid though. A more general solution would be something like ```shell nix shell 'n#python3.withPackages (p: with p; [brotli fonttools ])' ``` which is somewhat longer, but still gets rid of the most unpleasant parts like `--impure` and `(import <nixpkgs> {})`, and also allows other fun stuff like overrides. The problem here is that the word `python3` in this invocation is overloaded: it’s simultaneously the name of a flake output and the start of the expression, which would require CLI to, like, half-parse the expression to split it. I don’t think it’s a big deal though, I think just splitting on the first dot (or `|>` to allow arbitrary functions) would be enough. Another alternative would be to provide a separate syntax to map over chosen derivation, like ```shell nix shell n#python3 --map '(py: py.withPackages (p: with p; [ brotli fonttools ]))' ``` This is even longer but just as expressive and sidesteps the parsing issue. ## Additional context If any of these gets accepted, I’d try to implement it.

jade added the

label 2024-08-07 06:02:10 +00:00

ifreilicht commented

2024-08-11 08:17:25 +00:00

Member

There is an open issue on the NixCpp issue tracker for this since 2021. You can have a look at my write-up of the discussion there, but I'll also paste the most important part of it here for your convenience, adapted to include an additional suggestion made later in the thread:

This looks like a high-demand feature that could bridge the gap between the simple nix shell nixpkgs#asdf commands and writing a full-on flake. Just to make sure I'm grasping the problem and solution space correctly, here are my thoughts and the two potential solutions:

`--apply`, solution with precedent

The least controversial (but also least "nice") solution would be to port the --apply flag from nix eval to nix shell and nix build (or to any command that takes installables in general):

$ nix shell nixpkgs#python3 --apply 'py: (py.withPackages (p: [ p.flask ]))'

Differences to `nix eval`

This solution should not require the .outPath workaround you need for nix eval, it should not require --impure because the function itself is pure, and it should work for multiple installables, but nix eval applies the function passed to --apply to all of its inputs, which will very rarely be useful here. Rather, we want to be able to apply different functions to different installables.

There are two potential solutions for this:

Flake-like outputs functions

This is the option proposed by @YellowOnion:

$ nix shell nixpkgs github:somerepo#pkg --apply '{nixpkgs, pkg}: nixpkgs.pkg2.override {inherit pkg;}'

Unused inputs would just be put into the shell without applying the function:

$ nix shell nixpkgs#python3 nixpkgs#gnugrep --apply '{python3}: (python3.withPackages (p: [ p.flask ]))'

This is cool because it is very similar to how flakes work, making the transition easy and giving users a familiar interface. However, you have to type the name of the input three or more times, which is very annoying.

Split options

Basically the way --apply currently works, but extended to an arbitrary number of functions and installables.

$ nix shell nixpkgs github:somerepo#pkg --apply 'pkgs: p: pkgs.pkg2.override {pkg: p;}'

This is nice because it can be much terser. Again, if some inputs are unused, they will just be used verbatim.

$ nix shell nixpkgs#python3 --apply 'py: (py.withPackages (p: [ p.flask ]))' nixpkgs#gnugrep

Note that this order is not mandatory, but good style as it puts the function close to its input. It's also important to note that now the evaluation of the functions depends on the order of the installables. This is not the case with the flake-like solution.

Closer to `nix repl`: `--expr`

I do like the idea from @bobvanderlinden as well to align more with the interface of nix repl:

$ nix repl .
nix-repl> python3.withPackages (pkgs: [ pkgs.pjsua2 ])
«derivation /nix/store/kchdd811lan46mlk8bdhqq70i87lv2r6-python3-3.10.9-env.drv»
$ nix shell . --expr 'python3.withPackages (pkgs: [ pkgs.pjsua2 ])'
# or
$ nix shell nixpkgs#python3 --expr 'withPackages (pkgs: [ pkgs.pjsua2 ])'

Very clean, perfect for this simple usecase. I can't think of a way to really make this work with multiple installables, though.

Additionally, this doesn't cover all use-cases, it just makes the simple ones even simpler and allows jumping directly from a working repl statement to a nix shell invocation. So if this was added, I think it would have to be done in addition to --apply, not as a replacement.

Collision with current `--expr` installables

This solution completely reverses the current meaning of --expr, which replaces the initial part of the installable:

#           <installable> <                    output                         >
$ nix shell       nixpkgs#python3 --expr 'withPackages (pkgs: [ pkgs.pjsua2 ])'
#           <        installable       >  <output>
$ nix shell --expr 'import <nixpkgs> {}'  hello

Alternative 1: just call it `--eval`

Somewhat arbitrary, just call it --eval instead of --expr. Not a great solution, but not terrible either.

Alternative 2: two argument flag `--with`

A better option might be to implement this as 2-argument flag --with that can mirror the behavior of with inside the nix language:

$ nix shell nixpkgs#gnugrep --with nixpkgs#python3 'withPackages (p: [ p.flask ])'

This makes it very clear which function is applied to which flake, and makes confusing ordering an explicit error.

Complete overhaul: Arbitrary expressions in flake URLs

There was already an addition to flake URL syntax in stating outputs explicitly: nix shell nixpkgs#openssl^bin which is basically openssl.bin in nix repl. This already is kind of confusing, as nix shell nixpkgs#openssl.bin is equal.

If the suffix of # could instead be any Nix expression inside the context of the flake, then you would be able to even use trivial builders to write derivations directly:

nix shell "nixpkgs#openssl.override { withZlib = false; }"
nix shell "nixpkgs#python3.withPackages (ps: [ ps.pandas ])"
nix build 'nixpkgs#writeShellScript "fancyapp" "echo hello"'

Because it is part of the URL, it'll work for build, shell, profile, though there will have to be some adaptations for parsing names from these URLs correctly. For nix shell it'll support multiple expressions basically by default.

This might even make 'multiple output'-syntax for URLs (nixpkgs#openssl^bin,lib) obsolete and replace it with an array (nixpkgs#with openssl; [ bin lib ]), though that it is open for discussion.

Interop

With other installables

Apart from flakes outputs, we also have store paths, files, and (as already mentioned) expressions.

For store paths, this feature doesn't seem to be applicable.
For files, the --apply and --eval functions could operate on the output of those files. Only the proposed --with would not be compatible with file installables due to argument ordering.
For expressions, the feature doesn't make a lot of sense, because you can already put everything you need into the expression.

With other commands operating on installables

nix shell and nix build are the obvious cases we care about. nix profile install is a non-obvious case, but should work as well. nix copy should also work. nix edit might work, but I doubt it will ever be used. nix eval makes sense in some cases for debugging and should. nix run might also be useful for running shell scripts in a known environment?

There is an open issue on the NixCpp issue tracker for this since 2021. You can have a look at [my write-up of the discussion](https://github.com/NixOS/nix/issues/5567#issuecomment-1781442321) there, but I'll also paste the most important part of it here for your convenience, adapted to include [an additional suggestion made later in the thread](https://github.com/NixOS/nix/issues/5567#issuecomment-1782723120): --- This looks like a high-demand feature that could bridge the gap between the simple `nix shell nixpkgs#asdf` commands and writing a full-on flake. Just to make sure I'm grasping the problem and solution space correctly, here are my thoughts and the two potential solutions: # `--apply`, solution with precedent The least controversial (but also least "nice") solution would be to port the `--apply` flag from [`nix eval`](https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix3-eval) to `nix shell` and `nix build` (or to any command that takes installables in general): ```sh $ nix shell nixpkgs#python3 --apply 'py: (py.withPackages (p: [ p.flask ]))' ``` ## Differences to `nix eval` This solution should *not* require the `.outPath` workaround you need for `nix eval`, it should *not* require `--impure` because the function itself is pure, and it *should* work for multiple installables, but `nix eval` applies the function passed to `--apply` to *all* of its inputs, which will very rarely be useful here. Rather, we want to be able to apply different functions to different installables. There are two potential solutions for this: ### Flake-like outputs functions This is the option proposed by @YellowOnion: ```sh $ nix shell nixpkgs github:somerepo#pkg --apply '{nixpkgs, pkg}: nixpkgs.pkg2.override {inherit pkg;}' ``` Unused inputs would just be put into the shell without applying the function: ```sh $ nix shell nixpkgs#python3 nixpkgs#gnugrep --apply '{python3}: (python3.withPackages (p: [ p.flask ]))' ``` This is cool because it is very similar to how flakes work, making the transition easy and giving users a familiar interface. However, you have to type the name of the input three or more times, which is very annoying. ### Split options Basically the way `--apply` currently works, but extended to an arbitrary number of functions and installables. ```sh $ nix shell nixpkgs github:somerepo#pkg --apply 'pkgs: p: pkgs.pkg2.override {pkg: p;}' ``` This is nice because it can be much terser. Again, if some inputs are unused, they will just be used verbatim. ```sh $ nix shell nixpkgs#python3 --apply 'py: (py.withPackages (p: [ p.flask ]))' nixpkgs#gnugrep ``` Note that this order is not mandatory, but good style as it puts the function close to its input. It's also important to note that now the evaluation of the functions depends on the order of the installables. This is not the case with the flake-like solution. # Closer to `nix repl`: `--expr` I do like the idea from @bobvanderlinden as well to align more with the interface of `nix repl`: ```sh $ nix repl . nix-repl> python3.withPackages (pkgs: [ pkgs.pjsua2 ]) «derivation /nix/store/kchdd811lan46mlk8bdhqq70i87lv2r6-python3-3.10.9-env.drv» $ nix shell . --expr 'python3.withPackages (pkgs: [ pkgs.pjsua2 ])' # or $ nix shell nixpkgs#python3 --expr 'withPackages (pkgs: [ pkgs.pjsua2 ])' ``` Very clean, perfect for this simple usecase. I can't think of a way to really make this work with multiple installables, though. Additionally, this doesn't cover all use-cases, it just makes the simple ones even simpler and allows jumping directly from a working repl statement to a `nix shell` invocation. So if this was added, I think it would have to be done in addition to `--apply`, not as a replacement. ## Collision with current `--expr` installables This solution completely reverses the current meaning of `--expr`, which replaces the initial part of the installable: ```sh # <installable> < output > $ nix shell nixpkgs#python3 --expr 'withPackages (pkgs: [ pkgs.pjsua2 ])' # < installable > <output> $ nix shell --expr 'import <nixpkgs> {}' hello ``` ## Alternative 1: just call it `--eval` Somewhat arbitrary, just call it `--eval` instead of `--expr`. Not a great solution, but not terrible either. ## Alternative 2: two argument flag `--with` A better option might be to implement this as 2-argument flag `--with` that can mirror the behavior of `with` inside the nix language: ```sh $ nix shell nixpkgs#gnugrep --with nixpkgs#python3 'withPackages (p: [ p.flask ])' ``` This makes it very clear which function is applied to which flake, and makes confusing ordering an explicit error. # Complete overhaul: Arbitrary expressions in flake URLs There was already an addition to flake URL syntax in stating outputs explicitly: `nix shell nixpkgs#openssl^bin` which is basically `openssl.bin` in nix repl. This already is kind of confusing, as `nix shell nixpkgs#openssl.bin` is equal. If the suffix of # could instead be any Nix expression inside the context of the flake, then you would be able to even use trivial builders to write derivations directly: ``` nix shell "nixpkgs#openssl.override { withZlib = false; }" nix shell "nixpkgs#python3.withPackages (ps: [ ps.pandas ])" nix build 'nixpkgs#writeShellScript "fancyapp" "echo hello"' ``` Because it is part of the URL, it'll work for build, shell, profile, though there will have to be some adaptations for parsing names from these URLs correctly. For nix shell it'll support multiple expressions basically by default. This might even make 'multiple output'-syntax for URLs (`nixpkgs#openssl^bin,lib`) obsolete and replace it with an array (`nixpkgs#with openssl; [ bin lib ]`), though that it is open for discussion. # Interop ## With other installables Apart from [flakes outputs](https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix#flake-output-attribute), we also have [store paths](https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix#store-path), [files](https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix#nix-file), and (as already mentioned) [expressions](https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix#nix-expression). For store paths, this feature doesn't seem to be applicable. For files, the `--apply` and `--eval` functions could operate on the output of those files. Only the proposed `--with` would not be compatible with file installables due to argument ordering. For expressions, the feature doesn't make a lot of sense, because you can already put everything you need into the expression. ## With other commands operating on installables `nix shell` and `nix build` are the obvious cases we care about. `nix profile install` is a non-obvious case, but should work as well. `nix copy` should also work. `nix edit` might work, but I doubt it will ever be used. `nix eval` makes sense in some cases for debugging and should. `nix run` might also be useful for running shell scripts in a known environment?

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#462

No description provided.

An easy way to get a temporary shell with withPackages #462

Is your feature request related to a problem? Please describe.