Commit graph

8997 commits

Author SHA1 Message Date
John Ericson 27597f8131 Rename files to reflect new nix derivation show name
This will match the files we added for `nix add derivation` in the rest
of this PR.
2023-04-07 08:34:58 -04:00
John Ericson 2b98af2e62 nix show-derivation -> nix derivation show 2023-04-07 08:34:58 -04:00
John Ericson 4e9f32f993 Liberate checkDerivationOutputs from LocalStore
Make it instead a method on `Derivation` that can work with any store.
We will need this for a CLI command to create a derivation.
2023-04-07 08:34:58 -04:00
John Ericson b200784cec Include the name in the JSON for derivations
This is non-breaking change in the to-JSON direction. This *is* a
breaking change in the from-JSON direction, but we don't care, as that
is brand new in this PR.

`nix show-derivation --help` currently has the sole public documentation
of this format, it is updated accordingly.
2023-04-07 08:34:58 -04:00
John Ericson fe9cbe838c Create Derivation::fromJSON
And test, of course
2023-04-07 08:34:58 -04:00
Eelco Dolstra 6960de73af
Typo 2023-04-07 13:08:32 +02:00
Théophane Hufschmitt 214f1d6791 Rename and protect BufferedSink::write
The `write` name is ambiguous and could lead to some funny bugs like
https://github.com/NixOS/nix/pull/8173#issuecomment-1500009480. So
rename it to the more explicit `writeUnbuffered`.
Besides, this method shouldn't be (and isn't) used outside of the class
implementation, so mark it `protected`.

This makes it more symetrical to `BufferedSource` which uses a
`protected readUnbuffered` method.
2023-04-07 09:21:50 +02:00
matthewcroughan 9207f94582 Add Store::isTrustedClient()
This function returns true or false depending on whether the Nix client
is trusted or not. Mostly relevant when speaking to a remote store with
a daemon.

We include this information in `nix ping store` and `nix doctor`

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2023-04-06 19:59:57 -04:00
John Ericson bc192a95ef Describe active experimental features in the contributing guide
They are put in the manual separate pages under the new overarching
description of experimental features.

The settings page just lists the valid experimental feature names (so
people know what a valid setting entry looks like), with links to those
pages. It doesn't attempt to describe each experimental feature as that
is too much information for the configuration settings section.
2023-04-06 18:07:59 -04:00
John Ericson 6c4049b38a Link the new general documentation on xp features on the setting 2023-04-06 18:07:58 -04:00
Noah Snelson e399cb49c2
Fix typo in no-url-literals experimental feature docs
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-04-06 15:02:19 -07:00
Noah Snelson 8a7790f46a Expand documentation for experimental-features
Adds examples and additional information to the `impure-derivations`,
`recursive-nix`, and `no-url-literals` experimental feature
documentation.
2023-04-06 14:52:34 -07:00
Eelco Dolstra a9759407e5 Origin: Use SourcePath 2023-04-06 15:25:06 +02:00
Eelco Dolstra 94812cca98 Backport SourcePath from the lazy-trees branch
This introduces the SourcePath type from lazy-trees as an abstraction
for accessing files from inputs that may not be materialized in the
real filesystem (e.g. Git repositories). Currently, however, it's just
a wrapper around CanonPath, so it shouldn't change any behaviour. (On
lazy-trees, SourcePath is a <InputAccessor, CanonPath> tuple.)
2023-04-06 13:15:50 +02:00
Théophane Hufschmitt faefaac875 Explicitely define LockFile::operator!=
It should be syntethised in terms of `operator==`, but the GCC version
used on aarch64-linux doesn't implement that (see
https://hydra.nixos.org/build/214848896=. So explicitely define it.

Fix https://github.com/NixOS/nix/issues/8159
2023-04-05 17:20:04 +02:00
John Ericson 53d0836347 Assemble experimental feature docs outside of Nix itself
Instead of constructing a markdown list in C++ (which involved all sorts
of nasty string literals), export some JSON and assemble it with the
manual build system.

Besides following the precedent set with other dumped data, this is a
better separate of content and presentation; if we decide for example we
want to display this information in a different way, or in a different
section of the manual, it will become much easier to do so.
2023-04-04 22:57:11 -04:00
John Ericson 3f98353f19 Merge remote-tracking branch 'upstream/master' into list-experimental-features 2023-04-04 21:34:20 -04:00
John Ericson bdeeffff96
Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-04-04 19:16:10 -04:00
Théophane Hufschmitt d93c1514eb
Merge pull request #7944 from yorickvP/fix-curRepl
NixRepl::mainLoop: restore old curRepl on function exit
2023-04-04 09:44:47 +02:00
Théophane Hufschmitt 523c3f7225
Merge pull request #8157 from hercules-ci/switch-bugs
Fix current and future `switch` bugs
2023-04-03 20:43:02 +02:00
Eelco Dolstra 81491e1379
Merge pull request #8042 from lheckemann/alt-lockfiles
Allow specifying alternative paths for reading/writing flake locks
2023-04-03 19:28:09 +02:00
Robert Hensing bf2c5c3958 nix-store.cc: Refactor, remove qDefault 2023-04-03 18:45:20 +02:00
Robert Hensing fba7be80eb Enable -Werror=switch-enum
switch statements must now match all enum values or disable the
warning.

Explicit is good. This has helped us find two bugs, after solving
another one by debugging.

From now on, adding to an enum will raise errors where they are
not explicitly handled, which is good for productivity, and helps
us decide the correct behavior in all usages.

Notably still excluded from this though are the cases where the
warning is disabled by local pragmas.

fromTOML.cc did not build despite a top-level pragma, so I've had
to resort to a makefile solution for that.
2023-04-03 18:45:20 +02:00
Robert Hensing 9470ee877d Allow open switch-enum in 5 places 2023-04-03 18:45:20 +02:00
Robert Hensing 3dac4c7874 Add explicit case statements where -Wswitch-enum would report them 2023-04-03 18:17:32 +02:00
Robert Hensing ed7885017c Fix systemd logging for lvlNotice: eqv to lvlInfo, not lvlVomit 2023-04-03 18:17:32 +02:00
Robert Hensing 62cacc371f Fix BuildResult.toString() for NoSubstituters 2023-04-03 18:17:30 +02:00
John Ericson 4a0b893d5e Stuctured command stability
Prior to this, there was an ad-hoc whitelist in `main.cc`. Now, every
command states its stability.

In a future PR, we will adjust the manual to take advantage of this new
information in the JSON.
(It will be easier to do that once we have some experimental feature
docs to link too; see #5930 and #7798.)
2023-04-03 11:48:21 -04:00
Robert Hensing f3a6de6ba9
Merge pull request #8148 from hercules-ci/fix-issue-8119-printValue-tBlackhole-abort
eval: Fix crash on missing printValue tBlackhole case
2023-04-03 16:05:55 +02:00
Eelco Dolstra ac4318a1b9
Merge pull request #8152 from obsidiansystems/move-querySubstitutablePathInfos
Move `querySubstitutablePathInfos` from `LocalStore` to `Store`
2023-04-03 15:36:33 +02:00
Robert Hensing 1c55544a42 eval: Fix crash on missing printValue tBlackhole case
Fixes #8119
2023-04-03 15:32:42 +02:00
Théophane Hufschmitt 70bb7b7289
Merge pull request #7610 from obsidiansystems/gate-default-settings
Punt on improper global flags for now
2023-04-03 14:02:45 +02:00
John Ericson 9383520b75 Move querySubstitutablePathInfos from LocalStore to Store
The code is not local-store-specific, so we should share it with all
stores. More uniform behavior is better, and a less store-specific
functionality is more maintainable.

This fixes a FIXME added in f73d911628 by @edolstra himself.
2023-04-02 20:32:01 -04:00
John Ericson 32d72b1696 Add more API docs to experimental-features.hh 2023-04-02 18:57:46 -04:00
John Ericson 2585bcaa50 Rework a few things with the experimental features list
- Use struct not `std::pair`, designated initializers

- Use `constexpr` array that we can index by enum tag

- It no longer segfaults; not sure why.
2023-04-02 18:17:45 -04:00
John Ericson b2c9315bf2 Merge remote-tracking branch 'upstream/master' into list-experimental-features 2023-04-02 16:21:38 -04:00
John Ericson 5d56e2daf7 Add comparison methods for content addresses 2023-04-01 16:52:23 -04:00
John Ericson eeecfacb43 Merge branch 'path-info' into ca-drv-exotic 2023-04-01 16:40:32 -04:00
John Ericson 1f8e1edba9 Merge commit 'a6d00a7bfb18e7ec461ac1d54203cc628aca5c66' into ca-drv-exotic 2023-04-01 15:15:38 -04:00
John Ericson f7f44f7c96 Merge commit 'aa99005004bccc9be506a2a2f162f78bad4bcb41' into ca-drv-exotic 2023-04-01 15:15:32 -04:00
John Ericson 2ef99cd104
Merge pull request #8122 from bjornfor/use-nix-store-l-unless-experimental-enabled
Don't recommend 'nix log' unless experimental feature is enabled
2023-04-01 14:50:16 -04:00
John Ericson f4ab297b31 Ensure all headers have #pragma once and are in API docs
`///@file` makes them show up in the internal API dos. A tiny few were
missing `#pragma once`.
2023-03-31 23:19:44 -04:00
John Ericson abd5e7dec0 Extend internal API docs, part 2
Picking up from #8111.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-03-31 23:01:40 -04:00
Patrick Jackson 2b905d1d35 Replace unnecessary Sync<uint64_t> with std::atomic<uint64_t> 2023-03-31 17:56:25 -07:00
Eelco Dolstra 936e7c66eb Indentation 2023-03-31 17:15:03 +02:00
Eelco Dolstra 7ebaf0252a Add CanonPath::makeRelative() 2023-03-31 17:15:03 +02:00
Théophane Hufschmitt bb6e1244a3
Merge pull request #7406 from amesgen/gitlab-fetcher
Document `gitlab` input scheme
2023-03-31 13:21:55 +02:00
Théophane Hufschmitt e32ca3cf16
Merge pull request #8018 from tweag/ssh-password-prompt
SSH: don't erase password prompt if it is displayed
2023-03-31 12:06:10 +02:00
Patrick Jackson 804180ad52 Only lock once 2023-03-30 18:06:27 -07:00
Patrick Jackson 309753ebb5 Fix data race in copyPaths 2023-03-30 16:59:34 -07:00
John Ericson c51d554c93 Use "raw pattern" for content address types
We weren't because this ancient PR predated it!

This is actually a new version of the pattern which addresses some
issues identified in #7479.
2023-03-30 17:12:49 -04:00
John Ericson a6d00a7bfb Fix warning 2023-03-30 16:29:13 -04:00
John Ericson aa99005004 Merge remote-tracking branch 'upstream/master' into path-info
Also improve content-address.hh API docs.
2023-03-30 16:28:53 -04:00
John Ericson 06d87b95bc
Merge pull request #8111 from obsidiansystems/improve-internal-api-docs
Convert a bunch of comments in headers to Doxygen documentation
2023-03-30 09:40:22 -04:00
Valentin Gagarin 368c5b8b9a
Merge pull request #7518 from fricklerhandwerk/doc-commands
separate man pages for `nix-store` and `nix-env` subcommands
2023-03-30 15:33:08 +02:00
Eelco Dolstra 1cc5e1d5b6
Merge pull request #8123 from nomeata/import-flake-docs
Docs: Explain why `import nixpkgs` works in flakes
2023-03-30 11:20:55 +02:00
Eelco Dolstra 02050bd055
Typo 2023-03-30 10:49:21 +02:00
Joachim Breitner 1b6cfe6fa1 More precise wording 2023-03-30 10:40:04 +02:00
Alexander Bantyev 36b059748d Split nix-env and nix-store documentation per-subcommand
Documentation on "classic" commands with many sub-commands are
notoriously hard to discover due to lack of overview and anchor links.
Additionally the information on common options and environment variables
is not accessible offline in man pages, and therefore often overlooked
by readers.

With this change, each sub-command of nix-store and nix-env gets its
own page in the manual (listed in the table of contents), and each own
man page.

Also, man pages for each subcommand now (again) list common options
and environment variables. While this makes each page quite long and
some common parameters don't apply, this should still make it easier
to navigate as that additional information was not accessible on the
command line at all.

It is now possible to run 'nix-store --<subcommand> --help` to display
help pages for the given subcommand.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-03-30 09:46:28 +02:00
John Ericson 8cab89a94b Convert a bunch of comments in headers to Doxygen documentation
The internal API docs now contain more useful information.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-03-29 23:22:07 -04:00
figsoda 081aadb76c Allow @ in git refs 2023-03-29 15:27:19 -04:00
Joachim Breitner 2e6b95dac7
Update src/libexpr/primops.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-03-29 13:19:51 +02:00
Vaci Koblizek a4dd87a2b3 avoid a string copy in printHash16 2023-03-29 10:54:27 +01:00
Vaci Koblizek ccf7ce26fe return string_view from printHashType rather than string 2023-03-29 10:44:22 +01:00
Bjørn Forsman 74d94b2a13 Don't recommend 'nix log' unless experimental feature is enabled
This fixes the issue that `nix-build`, without experimental feature
'nix-command' enabled, recommends the experimental CLI `nix log` to view
build logs. Now it'll recommend the stable `nix-store -l` CLI instead.

Fixes https://github.com/NixOS/nix/issues/8118
2023-03-28 21:02:08 +02:00
Vaci Koblizek 957f832074 Avoid a string copy during Hash::to_string 2023-03-28 17:20:33 +01:00
Joachim Breitner 3ce6d72b2b
Update src/libexpr/primops.cc
Co-authored-by: asymmetric <lorenzo@mailbox.org>
2023-03-28 14:38:39 +02:00
Joachim Breitner 605c8f7789 Docs: Explain why import nixpkgs works in flakes 2023-03-28 09:35:49 +02:00
Noah Snelson d1d1ae7a3b Documentation: list experimental features in manual
Lists all current experimental features in the `nix.conf` manual.
2023-03-27 17:12:49 -07:00
John Ericson 8df6f7bb0f Fix more profile dirs in docs
Picking up where #8078 left off.
2023-03-27 12:10:32 -04:00
Eelco Dolstra 5e3f855526
Merge pull request #7763 from obsidiansystems/installable-wide-info
Stratify `ExtraPathInfo` along `Installable` hierarchy
2023-03-27 17:04:08 +02:00
Eelco Dolstra 237587bc0a
Merge pull request #8084 from edolstra/store-docs
Auto-generate store documentation
2023-03-27 15:46:18 +02:00
John Ericson 1d539aa287
Merge pull request #8073 from tweag/fix-root-channels-location
Fix root channels location
2023-03-27 09:37:53 -04:00
Eelco Dolstra 5a0f5b5c34
Merge pull request #8062 from edolstra/ssl-cert-file
Add a setting for configuring the SSL certificates file
2023-03-27 15:37:15 +02:00
John Ericson 7c4dea3cf3 Punt on improper global flags for now
See the note in the test.

We don't want these flags showing up for commands where they are
irrelevant.

Eventually, this needs a proper fix, but it need not be a blocker for
stabilize: for a quick-n-dirty punt, just put these flags behind the
`nix-command` unstable feature.

This is fine because they are only relevant for commands which we don't
need to stabilize for a while.
2023-03-27 09:21:50 -04:00
John Ericson 570829d67e
Merge pull request #7609 from obsidiansystems/hide-experimental-settings
Hide experimental settings
2023-03-27 09:19:29 -04:00
Eelco Dolstra a519436e9d ssl-cert-file: Use lists 2023-03-27 14:08:39 +02:00
Eelco Dolstra 2523448bad
Merge pull request #8104 from avdv/graphml-use-long-for-narsize
nix-store: Use `long` for `narSize` in graphml output
2023-03-27 13:10:28 +02:00
Théophane Hufschmitt 128994509f Make some comments in profiles.hh doxygen-enabled
These are proper documentation of the API, so they deserve to be here
2023-03-27 10:02:10 +02:00
oxalica 2941a599fa Catch eval errors in hasContent
`legacyPackages` of nixpkgs trigger eval errors in `hasContent`, causing
the whole `legacyPackages` being skipped. We should treat it as
has-content in that case.
2023-03-26 23:05:29 +08:00
Claudio Bley e7b9dc7c89 nix-store: Use long for narSize in graphml output 2023-03-24 17:25:17 +01:00
John Ericson 256f3e3063 Stratify ExtraPathInfo along Installable hierarchy
Instead of having a bunch of optional fields, have a few subclasses
which can have mandatory fields.

Additionally, the new `getExtraPathInfo`, and `nixpkgsFlakeRef`, are
moved to `InstallableValue`.

I did these things because https://github.com/NixOS/rfcs/pull/134 ; with
these things moved to `InstallableValue`, the base `Installable` no
longer depends on libexpr! This is a major step towards that.

Also, add a bunch of doc comments for sake of the internal API docs.
2023-03-24 12:22:40 -04:00
Eelco Dolstra e00abd3f56
Merge pull request #7757 from obsidiansystems/more-installable-value
Move value-only methods to `InstallableValue`
2023-03-24 14:07:18 +01:00
Théophane Hufschmitt a4c2cd3c06
Merge pull request #8098 from hercules-ci/error-cc-Only-suggest-show-trace-when-truncated-trace-items-would-be-pri
error.cc: Only suggest `--show-trace` when relevant
2023-03-24 12:07:44 +01:00
Robert Hensing 233c4cf30f error.cc: Only suggest show-trace when truncated trace items would be printed
Otherwise, a trace consisting of

frame
frame
frame
non-frame

... would reach the non-frame and print the suggestion, even though
it would have ignored the non-frame anyway.

This resulted in a peculariar situation where --show-trace would have
no apparent effect, as the trace was actually already complete.
2023-03-23 17:29:06 +01:00
Eelco Dolstra 168b6021c5 Tweaks 2023-03-23 15:32:59 +01:00
Eelco Dolstra b79df9dedc Register LocalStore to ensure it's included in the manual 2023-03-23 15:23:13 +01:00
Eelco Dolstra 161f4b0dea Document store URLs 2023-03-23 10:38:48 +01:00
Théophane Hufschmitt fb67c1a1fb Factor out the generation of the profile/channel directory
Make sure that all the code paths use the same one, and that the
backwards-compatibility measures are probably in place when needed
2023-03-23 10:22:42 +01:00
Théophane Hufschmitt 82bd9535dd nix-channel: Restore the old root channels directory 2023-03-23 10:22:42 +01:00
Eelco Dolstra abc449bc30
Update src/libstore/local-store.md
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-03-23 10:12:57 +01:00
Eelco Dolstra 80f0b8d307 Fix SSHStore 2023-03-23 09:35:35 +01:00
Eelco Dolstra b134546f08 Fix clang build 2023-03-23 09:11:15 +01:00
John Ericson e5146a6a50
Merge pull request #8078 from obsidiansystems/profile-md
Fix `nix profile --help`
2023-03-22 19:36:12 -04:00
Eelco Dolstra 5691bac202 Improve store setting descriptions / Markdown formatting 2023-03-22 14:23:36 +01:00
Alexander Bantyev 85a2d1d94f
Add a test for nix copy over ssh
Check that nix copy can copy stuff, refuses to copy unsigned paths by
default, and doesn't hide the ssh password prompt.
2023-03-22 09:45:08 +04:00
Alexander Bantyev 5291a82cd9
SSHMaster: pause logger to show password prompt
Pause logger before starting SSH connections, and resume it after the
connection is established, so that SSH password prompts are not erased
by the logger's updates.
2023-03-22 09:45:08 +04:00
Alexander Bantyev 85df7e7ea2
Logger, ProgressBar: add a way to pause/resume
Add new virtual methods pause and resume to the Logger class, and
implement them in ProgressBar to allow to pause the bar refreshing.
2023-03-22 09:45:08 +04:00
Eelco Dolstra 16cb9b9f04 Fix rendering of fetchGit documentation
stripIndentation() doesn't support tabs, so the entire markdown ended
up indented and thus rendered as a code block.
2023-03-21 14:48:15 +01:00
Eelco Dolstra c967c29290 Add a "help" category
This makes the help commands show up prominently at the top of the
'nix' manpage.
2023-03-21 14:43:58 +01:00
Eelco Dolstra 233b063b08 Move store docs to 'nix help-stores'
Why not 'nix help stores'? Well, 'nix help <arg>' already means 'show
help on the "arg" subcommand'.
2023-03-21 14:37:09 +01:00
Eelco Dolstra 9eb53bbf17 Support per-store Markdown documentation 2023-03-21 14:03:40 +01:00
Eelco Dolstra 7704118d28 nix describe-stores: Remove
This command was intended for docs generation, but it was never used
for that and we don't need it.
2023-03-21 14:03:40 +01:00
Eelco Dolstra 8d6d59cb1b nix store --help: Include store type documentation 2023-03-21 14:03:40 +01:00
Eelco Dolstra 3ac3e31c4d Fix misrendering of 'nix store --help'
There are no categories underneath 'nix store', so having 'nix store
copy-log' in a category rendered as ':'.
2023-03-21 12:02:19 +01:00
John Ericson f87f1bbfe6 Fix nix profile --help
The documentation hadn't be updated to reflect the new default profile
location.
2023-03-20 14:22:23 -04:00
Eelco Dolstra 515662ad70 Cleanup 2023-03-20 18:06:08 +01:00
Eelco Dolstra 16db8dc96f Open slave pseudoterminal before CLONE_NEWUSER
Otherwise, when running as root and user namespaces are enabled,
opening the slave fails with EPERM.

Fixes "opening pseudoterminal slave: Permission denied" followed by a
hang (https://hydra.nixos.org/build/213104244), and "error: getting
sandbox mount namespace: No such file or directory" (#8072), which
happens when the child fails very quickly and consequently reading
/proc/<child>/ns fails.
2023-03-20 17:58:36 +01:00
John Ericson 4607ac7aed Fix handling of experimental features mid-parse
If we conditionally "declare" the argument, as we did before, based upon
weather the feature is enabled, commands like

    nix --experimental-features=foo ... --thing-gated-on-foo

won't work, because the experimental feature isn't enabled until *after*
we start parsing.

Instead, allow arguments to also be associated with experimental
features (just as we did for builtins and settings), and then the
command line parser will filter out the experimental ones.

Since the effects of arguments (handler functions) are performed right
away, we get the required behavior: earlier arguments can enable later
arguments enabled!

There is just one catch: we want to keep non-positional
flags...non-positional. So if

    nix --experimental-features=foo ... --thing-gated-on-foo

works, then

    nix --thing-gated-on-foo --experimental-features=foo ...

should also work.

This is not my favorite long-term solution, but for now this is
implemented by delaying the requirement of needed experimental features
until *after* all the arguments have been parsed.
2023-03-20 11:35:34 -04:00
John Ericson aa663b7e89 Mark experimental features on settings
We hide them in various ways if the experimental feature isn't enabled.

To do this, we had to move the experimental features list out of
libnixstore, because the setting machinary itself depends on it. To do
that, we made a new `ExperimentalFeatureSettings`.
2023-03-20 11:06:40 -04:00
John Ericson 296831f641 Move enabled experimental feature to libutil struct
This is needed in subsequent commits to allow the settings and CLI args
infrastructure itself to read this setting.
2023-03-20 11:05:22 -04:00
figsoda 54b8da0b10 docs: quote URL literals 2023-03-19 11:28:10 -04:00
Linus Heckemann f1c9d83697 Only allow reference lock files when allow-dirty is set 2023-03-19 14:12:49 +01:00
Eelco Dolstra e53e5c38d4 Add a setting for configuring the SSL certificates file
This provides a platform-independent way to configure the SSL
certificates file in the Nix daemon. Previously we provided
instructions for overriding the environment variable in launchd, but
that obviously doesn't work with systemd. Now we can just tell users
to add

  ssl-cert-file = /etc/ssl/my-certificate-bundle.crt

to their nix.conf.
2023-03-17 18:32:18 +01:00
John Ericson 790dd2555b
Merge pull request #7891 from mupdt/shared-fs-out-links
no-op refactor: extract outLink generation into a function
2023-03-17 10:57:07 -04:00
John Ericson c998e0172f Move value-only methods to InstallableValue
These methods would previously fail on the other `Installable`s, so
moving them to this class is more correct as to where they actually
work.

Additionally, a `InstallableValueCommand` is created to make it easier
(or rather no worse than before) to write commands that just work on
`InstallableValue`s.

Besides being a cleanup to avoid failing default methods, this gets us
closer to https://github.com/NixOS/rfcs/pull/134.
2023-03-16 20:06:46 -04:00
Eelco Dolstra 7f46ebcf90
Merge pull request #8049 from edolstra/unexpected-eof
Fix "unexpected EOF" errors on macOS
2023-03-16 16:13:42 +01:00
Linus Heckemann af4cbdafe7 LocalDerivationGoal: set NIX_ATTRS_*_FILE correctly for sandboxed builds 2023-03-16 15:26:52 +01:00
John Ericson eb56cb7cc7
Merge pull request #7750 from obsidiansystems/no-args-prepare
Make command infra less stateful and more regular
2023-03-15 17:23:40 -04:00
John Ericson bc23a44c54 Make command infra less stateful and more regular
Already, we had classes like `BuiltPathsCommand` and `StorePathsCommand`
which provided alternative `run` virtual functions providing the
implementation with more arguments. This was a very nice and easy way to
make writing command; just fill in the virtual functions and it is
fairly clear what to do.

However, exception to this pattern were `Installable{,s}Command`. These
two classes instead just had a field where the installables would be
stored, and various side-effecting `prepare` and `load` machinery too
fill them in. Command would wish out those fields.

This isn't so clear to use.

What this commit does is make those command classes like the others,
with richer `run` functions.

Not only does this restore the pattern making commands easier to write,
it has a number of other benefits:

- `prepare` and `load` are gone entirely! One command just hands just
  hands off to the next.

- `useDefaultInstallables` because `defaultInstallables`. This takes
  over `prepare` for the one case that needs it, and provides enough
  flexiblity to handle `nix repl`'s idiosyncratic migration.

- We can use `ref` instead of `std::shared_ptr`. The former must be
  initialized (so it is like Rust's `Box` rather than `Option<Box>`,
  This expresses the invariant that the installable are in fact
  initialized much better.

  This is possible because since we just have local variables not
  fields, we can stop worrying about the not-yet-initialized case.

- Fewer lines of code! (Finally I have a large refactor that makes the
  number go down not up...)

- `nix repl` is now implemented in a clearer way.

The last item deserves further mention. `nix repl` is not like the other
installable commands because instead working from once-loaded
installables, it needs to be able to load them again and again.

To properly support this, we make a new superclass
`RawInstallablesCommand`. This class has the argument parsing and
completion logic, but does *not* hand off parsed installables but
instead just the raw string arguments.

This is exactly what `nix repl` needs, and allows us to instead of
having the logic awkwardly split between `prepare`,
`useDefaultInstallables,` and `load`, have everything right next to each
other. I think this will enable future simplifications of that argument
defaulting logic, but I am saving those for a future PR --- best to keep
code motion and more complicated boolean expression rewriting separate
steps.

The "diagnostic ignored `-Woverloaded-virtual`" pragma helps because C++
doesn't like our many `run` methods. In our case, we don't mind the
shadowing it all --- it is *intentional* that the derived class only
provides a `run` method, and doesn't call any of the overridden `run`
methods.

Helps with https://github.com/NixOS/rfcs/pull/134
2023-03-15 16:29:07 -04:00
John Ericson 0a140a92bb
Merge pull request #7690 from ShamrockLee/nix-hash-sri
nix-hash: support base-64 and SRI format
2023-03-15 16:02:46 -04:00
Yueh-Shun Li c27d358abb nix-hash: support base-64 and SRI format
Add the --base64 and --sri flags for the Base64 and SRI format output.

Add the --base16 flag to explicitly specify the hexadecimal format.

Add the --to-base64 and --to-sri flag to convert a hash to the above
mentioned format.
2023-03-16 03:08:42 +08:00
figsoda 31d588db8b respect allRefs=1 when using nix flake prefetch 2023-03-15 11:29:01 -04:00
Eelco Dolstra 5eb8bfd0f1 Remove a variable in LocalDerivationGoal 2023-03-15 10:59:10 +01:00
Eelco Dolstra 6029c763c2 Change builderOut from Pipe to AutoCloseFD 2023-03-15 10:59:10 +01:00
Eelco Dolstra c536e00c9d Move pseudoterminal slave open to child
Hopefully this fixes "unexpected EOF" failures on macOS
(#3137, #3605, #7242, #7702).

The problem appears to be that under some circumstances, macOS
discards the output written to the slave side of the
pseudoterminal. Hence the parent never sees the "sandbox initialized"
message from the child, even though it succeeded. The conditions are:

* The child finishes very quickly. That's why this bug is likely to
  trigger in nix-env tests, since that uses a builtin builder. Adding
  a short sleep before the child exits makes the problem go away.

* The parent has closed its duplicate of the slave file
  descriptor. This shouldn't matter, since the child has a duplicate
  as well, but it does. E.g. moving the close to the bottom of
  startBuilder() makes the problem go away. However, that's not a
  solution because it would make Nix hang if the child dies before
  sending the "sandbox initialized" message.

* The system is under high load. E.g. "make installcheck -j16" makes
  the issue pretty reproducible, while it's very rare under "make
  installcheck -j1".

As a fix/workaround, we now open the pseudoterminal slave in the
child, rather than the parent. This removes the second condition
(i.e. the parent no longer needs to close the slave fd) and I haven't
been able to reproduce the "unexpected EOF" with this.
2023-03-15 10:58:37 +01:00
Eelco Dolstra 19326ac297 Simplify commonChildInit() 2023-03-14 18:58:12 +01:00
Linus Heckemann 3a1de4c3fe Apply review suggestions
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-03-14 12:02:03 +01:00
Linus Heckemann e91596eb69 Allow specifying alternative paths for reading/writing flake locks
This allows having multiple separate lockfiles for a single
project, which can be useful for testing against different versions of
nixpkgs; it also allows tracking custom input overrides for remote
flakes without requiring local clones of these flakes.

For example, if I want to build Nix against my locally pinned nixpkgs,
and have a lock file tracking this override independently of future
updates to said nixpkgs:

nix flake lock --output-lock-file /tmp/nix-flake.lock --override-input nixpkgs flake:nixpkgs
nix build --reference-lock-file /tmp/nix-flake.lock

Co-Authored-By: Will Fancher <elvishjerricco@gmail.com>
2023-03-13 21:08:52 +01:00
Robert Hensing a387f46967
Merge pull request #8033 from lbodor/stop-adding-dot-to-nix-dev-env-path
`print-dev-env`: stop inadvertently adding `.` to `PATH`
2023-03-13 19:52:41 +01:00
Eelco Dolstra fcd0b0fbd5 Fix macOS warning 2023-03-13 05:31:03 -07:00
Lazar Bodor 526bdbda3c print-dev-env: stop inadvertently adding . to PATH 2023-03-12 22:40:47 +11:00
Théophane Hufschmitt 99af56cd0d
Merge pull request #8015 from tweag/progress-during-nix-copy
Display progress when running copyPaths (nix copy)
2023-03-10 16:23:22 +01:00
Eelco Dolstra 4f3a4b732c Make findRuntimeRoots() more resilient to disappearing processes
I saw this random failure in https://hydra.nixos.org/build/211811692:

  error: opening /proc/15307/fd: No such process

while running nix-collect-garbage in a readfile-context.sh. This is
because we're not handling ESRCH errors reading /proc/<pid>/fd. So
just move the read inside the try/catch where we do handle it.
2023-03-09 16:44:51 +01:00
Eelco Dolstra 8f96720b9f Fix another uninitialized variable
https://hydra.nixos.org/build/211811494
2023-03-09 15:30:53 +01:00
Alexander Bantyev 4bef2016a1
Display progress when running copyPaths (nix copy)
`nix copy` operations did not show progress. This is quite confusing.

Add a `progressSink` which displays the progress during `copyPaths`,
pretty much copied from `copyStorePath`.

Fixes https://github.com/NixOS/nix/issues/8000
2023-03-09 16:03:48 +04:00
Théophane Hufschmitt 39700c5cbe
Revert "Disable GC during coroutine execution + test" 2023-03-08 20:47:52 +01:00
Eelco Dolstra 1607ad5076 Fix uninitialized readFromStdIn variable
This was causing random failures in tests/ca/substitute.ca: 'nix copy
--file ./content-addressed.nix' wouldn't get the default installable
'.' applied in InstallablesCommand::load(), so it would do nothing.
2023-03-08 13:53:20 +01:00
Eelco Dolstra df48040786
Merge pull request #7994 from edolstra/fix-ca-crash
Fix crash/hang with CA derivations
2023-03-08 12:15:55 +01:00
Eelco Dolstra 6b243ef2d5
Merge pull request #7993 from tweag/fix-profile-gc
Fix `nix-collect-garbage -d` with the new profile location
2023-03-08 11:10:45 +01:00
Eelco Dolstra 7bfed34367 Fix crash/hang with CA derivations
The curl download can outlive DrvOutputSubstitutionGoal (if some other
error occurs), so at shutdown setting the promise to an exception will
fail because 'this' is no longer valid in the callback. This can
manifest itself as a segfault, "corrupted double-linked list" or hang.
2023-03-08 11:09:15 +01:00
Théophane Hufschmitt 4a6244dcf7
Merge pull request #7725 from yorickvP/check-coro-gc
Disable GC during coroutine execution + test
2023-03-08 09:51:46 +01:00
Théophane Hufschmitt 2272bc6049 Fix nix-collect-garbage -d with the new profile location
Low-hanging fix for https://github.com/NixOS/nix/pull/5226#issuecomment-1454669399
2023-03-07 13:08:04 +01:00
Théophane Hufschmitt ba0486f045
Merge pull request #7889 from sidkshatriya/sorted-fetch-paths
Print the store paths to be fetched sorted by StorePath name()
2023-03-07 11:58:10 +01:00
mupdt 762cf7f214 no-op refactor: extract outLink generation into a function 2023-03-06 10:20:21 -05:00
Eelco Dolstra 3f4dc61f7f
Merge pull request #7958 from amesgen/issue-7955
InstallableFlake: Apply nix config in `getCursors`
2023-03-06 11:08:51 +01:00
Eelco Dolstra 92771ca1be
Merge pull request #7975 from fricklerhandwerk/fix-fetchgit
remove indentation in `fetchGit` attribute listing
2023-03-06 11:00:40 +01:00
Valentin Gagarin dfeb83cac1 output paths of store derivations are now addressed explicitly 2023-03-06 03:32:14 +01:00
Valentin Gagarin f69d453c3e remove indentation in fetchGit attribute listing
also reword a confusing sentence and add links to Git terminology
2023-03-05 04:44:11 +01:00
Valentin Gagarin 2af9fd20c6 clarify definition of "installable"
the term was hard to discover, as its definition and explanation were in
a very long document lacking an overview section.
search did not help because it occurs so often.

- clarify wording in the definition
- add an overview of installable types
- add "installable" to glossary
- link to definition from occurrences of the term
- be more precise about where store derivation outputs are processed
- installable Nix expressions must evaluate to a derivation

Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
2023-03-05 01:46:17 +01:00
Valentin Gagarin 1e87d5f1ea make descriptions of each installable type an own subsection
this is easier to edit, provides anchors for free, and renders correctly
on the terminal without additional effort.
2023-03-05 01:46:17 +01:00
amesgen 77bb950566
InstallableFlake: Apply nix config in getCursors 2023-03-04 12:52:20 +01:00
Sidharth Kshatriya 427555861b Print the store paths to be fetched sorted by StorePath name() and not baseName
Presently when nix says something like:

```
these 486 paths will be fetched (511.54 MiB download, 6458.64 MiB unpacked):
 ...path1
 ...path2
 ...path3
    ...
    ...
 ...path486
```

It sorts path1, path2, path3, ..., path486 in lexicographic order of the
store path.

After this commit, nix will show path1, path2, path3, ..., path486 sorted by
StorePath name() (basically everything after the hash) rather than the store path.

This makes it easier to review what exactly is being downloaded at a glance,
especially when many paths need to be fetched.
2023-03-03 21:46:35 +05:30
Eelco Dolstra 0507462c06
Merge pull request #7918 from zimbatm/fix-empty-nix-store-env
treat empty NIX_STORE_DIR env vars as unset
2023-03-03 13:49:56 +01:00
Yorick 2683734936
Add talkative msg for coro gc debug 2023-03-03 11:43:47 +01:00
Jonas Chevalier dc8820c71f
fixup: use same style as getEnv 2023-03-03 11:34:36 +01:00
Jonas Chevalier b96d9c1687
fixup: remove warning entirely
fixes 72e1e23051 (r1124211067)
2023-03-03 11:32:06 +01:00
Eelco Dolstra f0908f592c
Merge pull request #7942 from edolstra/remove-format
Remove FormatOrString and remaining uses of format()
2023-03-03 10:22:11 +01:00
Valentin Gagarin 37c61907d7
Merge pull request #7605 from tweag/comments
add comments
2023-03-03 06:38:29 +01:00
Théophane Hufschmitt 1f394d2107
Merge branch 'master' into paths-from-stdin 2023-03-02 19:20:51 +01:00
Yorick 5bb6e3bfaf
NixRepl::mainLoop: restore old curRepl on function exit
This fixes completion callbacks after entering and leaving a nested
debugger.
2023-03-02 17:24:58 +01:00
Eelco Dolstra 687d86f8ab
Merge pull request #7940 from tweag/fix-http-errors-encoding
Log the decompressed body in case of http errors
2023-03-02 16:25:09 +01:00
Jonas Chevalier 72e1e23051
Update src/libutil/util.cc
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-03-02 16:17:20 +01:00
Eelco Dolstra 29abc8e764 Remove FormatOrString and remaining uses of format() 2023-03-02 15:57:54 +01:00
Eelco Dolstra b69a73a230 Get rid of some unchecked calls to std::cout 2023-03-02 15:02:24 +01:00
Théophane Hufschmitt 1a825b6919 Log the decompressed body in case of http errors
Don't show the users the raw (possibly compressed) error message as
everyone isn't able to decompress brotli in their brain.
2023-03-02 14:59:15 +01:00
Eelco Dolstra 09f5975c6a Logger::cout: Use fmt()
This ensures that in cout(s), 's' does not get interpreted as a format
string.
2023-03-02 14:52:37 +01:00
Eelco Dolstra 989b823ac5 nix store cat: Use writeFull()
Fixes #7939.
2023-03-02 14:51:14 +01:00
Eelco Dolstra c13cbd20ab Logger::writeToStdout(): Use writeFull()
This ensures that write errors do not get ignored.
2023-03-02 14:46:28 +01:00
Eelco Dolstra 767974f411
Merge pull request #7924 from mkenigs/valid
Always set valid in path-info --json output
2023-03-02 09:58:20 +01:00
Félix Baylac Jacqué 25300c0ecd
Treat empty env var paths as unset
We make sure the env var paths are actually set (ie. not "") before
sending them to the canonicalization function. If we forget to do so,
the user will end up facing a puzzled failed assertion internal error.

We issue a non-failing warning as a stop-gap measure. We could want to
revisit this to issue a detailed failing error message in the future.
2023-03-01 20:50:07 +01:00
Théophane Hufschmitt 182129d985
Merge pull request #7932 from obsidiansystems/remove-sameMachine
Remove dead code `RemoteStore::sameMachine`
2023-03-01 20:05:39 +01:00
Valentin Gagarin 651dab55da
Merge pull request #7854 from aameen-tulip/patch-1 2023-03-01 18:24:53 +01:00
John Ericson b7f01a82a9 Remove dead code RemoteStore::sameMachine
It has been dead code since 9747ea84b4.
2023-03-01 11:10:30 -05:00
Yorick 176005749c
Always disable GC in a coroutine unless the patch is applied 2023-03-01 15:07:00 +01:00
Yorick 4c73eab923
DisableGC: replace by CoroutineContext, std::shared_ptr<void> 2023-03-01 13:55:41 +01:00
Yorick 53bb4a5327
tests/coro-gc: refactor and split into 3 tests 2023-03-01 13:55:41 +01:00
Yorick eaeb994d8b
Disable GC inside coroutines on mac OS 2023-03-01 13:55:41 +01:00
Yorick 0fd8f542a8
tests/coro-gc: create test for boehm stack patch
Regression test for #7679
2023-03-01 13:55:37 +01:00
Valentin Gagarin 306e5c5ce5
Merge pull request #7788 from bobvanderlinden/pr-improve-nix-profile-install-error
Improve error on conflict for nix profile install
2023-03-01 11:48:43 +01:00
Bob van der Linden 12538605fd
nix-profile: add FIXME about using C++20 std::ranges 2023-03-01 07:40:53 +01:00
Matthew Kenigsberg f86f2b973f Always set valid in path-info --json output
Currently the valid key is only present when the path is invalid, which
makes checking path validity more complex than it should be. With this
change, the valid key can always be used to check if a path is valid
2023-02-28 16:04:41 -07:00
John Ericson ea0adfc582 Get rid of .drv special-casing for store path installables
The release notes document the change in behavior, I don't include it
here so there is no risk to it getting out of sync.

> Motivation

>> Plumbing CLI should be simple

Store derivation installations are intended as "plumbing": very simple
utilities for advanced users and scripts, and not what regular users
interact with. (Similarly, regular Git users will use branch and tag
names not explicit hashes for most things.)

The plumbing CLI should prize simplicity over convenience; that is its
raison d'etre. If the user provides a path, we should treat it the same
way not caring what sort of path it is.

>> Scripting

This is especially important for the scripting use-case. when arbitrary
paths are sent to e.g. `nix copy` and the script author wants consistent
behavior regardless of what those store paths are. Otherwise the script
author needs to be careful to filter out `.drv` ones, and then run `nix
copy` again with those paths and `--derivation`. That is not good!

>> Surprisingly low impact

Only two lines in the tests need changing, showing that the impact of
this is pretty light.

Many command, like `nix log` will continue to work with just the
derivation passed as before. This because we used to:

- Special case the drv path and replace it with it's outputs (what this
  gets rid of).

- Turn those output path *back* into the original drv path.

Now we just skip that entire round trip!

> Context

Issue #7261 lays out a broader vision for getting rid of `--derivation`,
and has this as one of its dependencies. But we can do this with or
without that.

`Installable::toDerivations` is changed to handle the case of a
`DerivedPath::Opaque` ending in `.drv`, which is new: it simply doesn't
need to do any extra work in that case. On this basis, commands like
`nix {show-derivation,log} /nix/store/...-foo.drv` still work as before,
as described above.

When testing older daemons, the post-build-hook will be run against the
old CLI, so we need the old version of the post-build-hook to support
that use-case.

Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-28 17:07:05 -05:00
Timothy DeHerrera df643051e2
nix-store: read paths from standard input
Resolves #7437 for new `nix-store` by adding a `--stdin` flag.
2023-02-28 12:29:16 -07:00
Timothy DeHerrera 269caa5317
feat: read installable paths from stdin
Resolves #7437 for new `nix` commands only by adding a `--stdin` flag.

If paths are also passed on the cli they will be combined with the ones
from standard input.
2023-02-28 12:29:15 -07:00
John Ericson 5abd643c6d Merge branch 'path-info' into ca-drv-exotic 2023-02-28 12:46:00 -05:00
John Ericson d381248ec0 No inheritance for TextInfo and FixedOutputInfo 2023-02-28 12:14:11 -05:00
John Ericson 85bb865d20 Revert "Remove some designated initializers"
This reverts commit ee9eb83a84.
2023-02-28 11:57:20 -05:00
John Ericson 123b11ff83 Clarify store path grammar and improve comment on makeType 2023-02-28 11:49:13 -05:00
John Ericson d12f57c2c0 Merge remote-tracking branch 'upstream/master' into path-info 2023-02-28 11:34:34 -05:00
John Ericson c36b584f8e Fix typo in the method name 2023-02-28 11:34:18 -05:00
Eelco Dolstra a4a5d828e2
Merge pull request #7793 from layus/interrupt_downloads
Check interrupts even when download stalled
2023-02-28 13:29:29 +01:00
Eelco Dolstra da1f49c4e3
Merge pull request #7904 from anatol/patch-1
Example uses gitlab.com thus clarify the comment
2023-02-28 13:29:10 +01:00
Bob van der Linden 872cdb4346
nix-profile-install: show helpful error upon package conflict
Whenever a file conflict happens during "nix profile install" an error
is shown that was previously thrown inside builtins.buildEnv.

We catch BuildProfileConflictError here so that we can provide the user
with more useful instructions on what to do next.

Most notably, we give the user concrete commands to use with all
parameters  already filled in. This avoids the need for the user to look
up these commands in manual pages.
2023-02-28 09:28:05 +01:00
Eelco Dolstra 3d15dbadc2
Merge pull request #7911 from edolstra/revert-7689
Revert #7689
2023-02-28 08:46:55 +01:00
Bob van der Linden 3113b13df9
buildenv: throw BuildEnvFileConflictError with more context
At the moment an Error is thrown that only holds an error message
regarding `nix-env` and `nix profile`. These tools make use of
builtins.buildEnv, but buildEnv is also used in other places. These
places are unrelated to Nix profiles, so the error shouldn't mention
these tools.

This generic error is now BuildEnvFileConflictError, which holds more
contextual information about the files that were conflicting while
building the environment.
2023-02-27 21:39:34 +01:00
Théophane Hufschmitt eae89aca1b
Merge pull request #7776 from yorickvP/fix-path-escapes-7707
Properly escape local paths into URLs in fetchTree
2023-02-27 21:10:25 +01:00
Eelco Dolstra f08ad5bdba
Merge pull request #7913 from fricklerhandwerk/master
add information on the `build-hook` setting
2023-02-27 17:33:25 +01:00
Valentin Gagarin fd0e21475c add information on the build-hook setting
add a warning that you probably never want to change this.
2023-02-27 16:27:56 +01:00
Yorick 2c0866fc3f
fetchTree: convert fs path to url via ParsedURL::to_string 2023-02-27 15:30:04 +01:00
Yorick 0844856c84
url: make percentEncode stricter, expose and unit test it 2023-02-27 15:30:00 +01:00
Eelco Dolstra dd93c12c6a Revert "getDefaultNixPath: actually respect {restrict,pure}-eval"
This reverts commit 1cba5984a6.
2023-02-27 15:11:36 +01:00
Eelco Dolstra e928c72cf9 Revert "Document default nix-path value"
This reverts commit dba9173a1d.
2023-02-27 14:16:49 +01:00
Théophane Hufschmitt 995bfeef3b
Merge pull request #7796 from hercules-ci/fix-7263
Ensure that `self.outPath == ./.`
2023-02-27 10:26:02 +01:00
Anatol Pomozov d731235f6b
Example uses gitlab.com thus clarify the comment 2023-02-24 07:17:47 -08:00
Théophane Hufschmitt 4a921ba43b
Merge pull request #7764 from yorickvP/build-remote-warning
build-remote: don't warn when all local build slots are taken
2023-02-22 17:43:20 +01:00
Robert Hensing 5d834c40d0 flakes: Differentiate self.outPath and self.sourceInfo.outPath
It would be incorrect to say that the `sourceInfo` has an `outPath`
that isn't the root. `sourceInfo` is about the root, whereas only
the flake may not be about the root. Thanks Eelco for pointing that
out.
2023-02-22 03:31:24 +01:00
Robert Hensing 904a107d16 flakes: Ensure that self.outPath == ./.
Users expect `self` to refer to the directory where the `flake.nix`
file resides.
2023-02-22 03:30:47 +01:00
Théophane Hufschmitt 8418d22ac1
Merge pull request #7874 from obsidiansystems/fix-no-gc-build
Fix the build without GC
2023-02-21 16:14:51 +01:00
Théophane Hufschmitt c7885ab6f2
Merge pull request #7755 from obsidiansystems/mix-read-only-mode
Make `--read-only` a separate mixin
2023-02-21 16:13:48 +01:00
John Ericson 5b0175e81d Fix the build without GC
I had given it an improper trailing comma in
1bd03ad100.
2023-02-21 09:38:46 -05:00
Théophane Hufschmitt 532c70f531
Merge pull request #7856 from yorickvP/fix-nsswitch
Wait with making /etc unwritable until after build env setup
2023-02-21 09:39:10 +01:00
John Ericson 208c8d326d Derivation::toJSON: fix bug!
When I moved this code from the binary to libnixstore #7863, I forgot to
display the environment variables!
2023-02-20 17:38:57 -05:00
Robert Hensing c7bd3a874f
Merge pull request #7863 from obsidiansystems/test-derivation-to-json
Move Derivation JSON printing logic to lib and test it
2023-02-20 23:21:50 +01:00
John Ericson 0258ac9c2a Make --read-only a separate mixin
It is independent of SourceExprCommand, which is about parsing
installables, except for the fact that parsing installables is one of
the many things influenced by read-only mode.
2023-02-20 10:39:18 -05:00
John Ericson 1bd03ad100 Split out CmdRepl and editorFor
The REPL itself and the `nix repl` CLI are conceptually different
things, and thus deserve to be in different files.
2023-02-20 09:45:29 -05:00
John Ericson 57a2e46ee0 Slight cleanup of InstallablesCommand::load 2023-02-20 09:09:11 -05:00
John Ericson fa4733fce5 Split out InstallableFlake and InstallableAttrPath 2023-02-20 09:09:11 -05:00
Valentin Gagarin dda83a59c1
Merge pull request #7158 from sternenseemann/foldl-strict-accumulation-value 2023-02-19 23:54:14 +01:00
John Ericson 7998686c00 Test toJSON of DerivationOutput and Derivation 2023-02-19 11:12:12 -05:00
John Ericson cd583362ec Move Derivation toJSON logic to libnixstore 2023-02-19 10:06:40 -05:00
Yorick bbba49b3e4
Wait with making /etc unwritable until after build env setup
This fixes /etc/nsswitch.conf
2023-02-17 16:34:45 +01:00
aameen-tulip b31d4b689c
Document hasAllInfo
If this documentation is inaccurate in any way please do not hesitate to suggest corrections.

My understanding of this function is strictly from reading the source code and some limited experience implementing fetchers.
2023-02-16 18:47:45 -06:00
Robert Hensing a88ae62bc0
Merge pull request #7811 from Et7f3/fix_memory_leaks
Reduce memory leaks
2023-02-16 21:21:54 +01:00
Et7f3 cec23f5dda ExprOpHasAttr,ExprSelect,stripIndentation,binds,formals: delete losts objects
We are looking for *$ because it indicate that it was constructed with a new but
not release. De-referencing shallow copy so deleting as whole might create
dangling pointer that's why we move it so we delete a empty containers + the
nice perf boost.
2023-02-16 19:53:55 +01:00
Yorick 49fd72a903
Make /etc writability conditional on uid-range feature 2023-02-14 13:55:41 +01:00
Yorick db41f74af3
Don't allow writing to /etc 2023-02-14 12:03:34 +01:00
tomberek 601faa00d7
Merge pull request #7744 from obsidiansystems/split-installable-store-path
Factor out `InstallableStorePath` to its own file, dedup
2023-02-13 08:57:19 -05:00
Eelco Dolstra c205d10c66
Merge pull request #7616 from hercules-ci/fix-3898
Fix foreign key error inserting into NARs #3898
2023-02-13 13:02:19 +01:00
Eelco Dolstra 2037f8a3ee
Merge pull request #7804 from PJungkamp/fix-completions
Infer short completion descriptions for commandline flags
2023-02-13 11:26:38 +01:00
Et7f3 fa89d317b7 ExprString: Avoid copy of string 2023-02-12 05:49:45 +01:00
Et7f3 3d16f2a281 parser: use implicit rule 2023-02-12 05:49:45 +01:00
Philipp Jungkamp 30edd7af53 Completions::add use libutil trim() 2023-02-10 22:17:09 +01:00
Eelco Dolstra b3d29e80e0
Merge pull request #7805 from edolstra/c++2a
Fix building with GCC 9
2023-02-10 20:41:29 +01:00
Eelco Dolstra 67451d8ed7
Merge pull request #7802 from edolstra/fix-7783
Fix PID namespace support check
2023-02-10 20:41:13 +01:00
Eelco Dolstra 5978ceb271 Fix building with GCC 9
Nixpkgs on aarch64-linux is currently stuck on GCC 9
(https://github.com/NixOS/nixpkgs/issues/208412) and using gcc11Stdenv
doesn't work either.

So use c++2a instead of c++20 for now. Unfortunately this means we
can't use some C++20 features for now (like std::span).
2023-02-10 18:38:57 +01:00
Théophane Hufschmitt 9ebbe35817
Merge pull request #5588 from tweag/balsoft/xdg
Follow XDG Base Directory standard
2023-02-10 18:05:50 +01:00
Philipp Jungkamp a537095e1f Infer short completion descriptions for commandline flags
Descriptions for commandline flags may not include newlines and should
be rather short for display in a shell. Truncate the description string
of a flag on '\n' or '.' to and add an ellipsis if needed.
2023-02-10 18:03:19 +01:00
Alexander Bantyev 2384d36083
A setting to follow XDG Base Directory standard
XDG Base Directory is a standard for locations for storing various
files. Nix has a few files which seem to fit in the standard, but
currently use a custom location directly in the user's ~, polluting
it:

- ~/.nix-profile
- ~/.nix-defexpr
- ~/.nix-channels

This commit adds a config option (use-xdg-base-directories) to follow
the XDG spec and instead use the following locations:

- $XDG_STATE_HOME/nix/profile
- $XDG_STATE_HOME/nix/defexpr
- $XDG_STATE_HOME/nix/channels

If $XDG_STATE_HOME is not set, it is assumed to be ~/.local/state.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
Co-authored-by: Tim Fenney <kodekata@gmail.com>
Co-authored-by: pasqui23 <pasqui23@users.noreply.github.com>
Co-authored-by: Artturin <Artturin@artturin.com>
Co-authored-by: John Ericson <Ericson2314@Yahoo.com>
2023-02-10 20:14:06 +04:00
Eelco Dolstra c49b7472ea Fix macOS build 2023-02-10 17:08:33 +01:00
Eelco Dolstra 3e6e34cdf5 LocalDerivationGoal::startBuilder(): Use startProcess() to clone 2023-02-10 14:44:25 +01:00
Robert Hensing 37b1e93f4b daemon.cc: Rename UserSettings -> AuthorizationSettings
This is a bit more accurate.

It's a private name, but before you know it, someone might make it public!
2023-02-10 14:41:39 +01:00
Eelco Dolstra f094ba7386 Simplify the PID namespace check: just try to mount /proc
Fixes #7783.
2023-02-10 14:38:14 +01:00
Guillaume Maudoux e6ad8e8440 nit: cleaner diff 2023-02-10 00:57:56 +01:00
Guillaume Maudoux aa18404ecb Flush data when download ends 2023-02-10 00:54:29 +01:00
Théophane Hufschmitt 5597d68e2d
Merge pull request #7754 from obsidiansystems/narrower-scope-derivation-flag
Scope down `--derivation` to just the commands that use it
2023-02-09 19:51:43 +01:00
Eelco Dolstra 0a7071ed33
Merge pull request #7774 from edolstra/submodule-fixes
Git submodule fixes
2023-02-09 17:19:48 +01:00
Eelco Dolstra e46429f674
Merge pull request #7712 from Mic92/advertise-compressions
advertise transport encoding in http transfers to
2023-02-09 17:15:25 +01:00
Eelco Dolstra 862e56c23d
Improve comment
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-02-09 16:42:45 +01:00
Eelco Dolstra 15313bfdb7
Fix activity message
Co-authored-by: Josef Kemetmüller <josef.kemetmueller@gmail.com>
2023-02-09 16:42:14 +01:00
Guillaume Maudoux 78fea899e0 Check interupts even when download stalled 2023-02-09 13:56:50 +01:00
Joachim Breitner e4a2a5c074
Documentation: builtins.fetchGit when used on a local path (#7706)
* Documentation:  builtins.fetchGit when used on a local path

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-08 11:53:28 +00:00
Robert Hensing 19b495a48a NarInfoDiskCache: Also test id consistency with updated fields
And clarify test
2023-02-07 23:34:36 +01:00
Robert Hensing fb94d5cabd NarInfoDiskCache: Keep BinaryCache.id stable and improve test
Fixes #3898

The entire `BinaryCaches` row used to get replaced after it became
stale according to the `timestamp` column. In a concurrent scenario,
this leads to foreign key conflicts as different instances of the
in-process `state.caches` cache now differ, with the consequence that
the older process still tries to use the `id` number of the old record.

Furthermore, this phenomenon appears to have caused the cache for
actual narinfos to be erased about every week, while the default
ttl for narinfos was supposed to be 30 days.
2023-02-07 23:34:36 +01:00
Robert Hensing 2ceece3ef3 NarInfoDiskCache: Prepare reproducer for #3898 2023-02-07 23:34:36 +01:00
Robert Hensing 79f62d2dda NarInfoDiskCacheImpl: Make dbPath a parameter
This allows testing with a clean database.
2023-02-07 23:34:36 +01:00
Robert Hensing 29f0b196f4 NarInfoDiskCache: Rename cacheExists -> upToDateCacheExists
This is slightly more accurate considering that an outdated record
may exist in the persistent cache. Possibly-outdated records are
quite relevant as they may be foreign keys to more recent information
that we want to keep, but we will not return them here.
2023-02-07 23:34:36 +01:00
Robert Hensing 8a0ef5d58e sqlite.cc: Add SQL tracing
Set environment variable NIX_DEBUG_SQLITE_TRACES=1 to log all sql statements.
2023-02-07 23:34:36 +01:00
Eelco Dolstra 0a70b411e1 Print debug message if a namespace test fails 2023-02-07 23:01:39 +01:00
Eelco Dolstra c5c0617d6f Mention --no-sandbox if sandboxing is unsupported 2023-02-07 22:59:46 +01:00
Eelco Dolstra 4e61877b5c More #ifdef 2023-02-07 22:51:53 +01:00
Eelco Dolstra d834de2894 Fix macOS build 2023-02-07 22:51:53 +01:00
Eelco Dolstra bc1d9fd8b5 Check whether we can use PID namespaces
In unprivileged podman containers, /proc is not fully visible (there
are other filesystems mounted on subdirectories of /proc). Therefore
we can't mount a new /proc in the sandbox that matches the PID
namespace of the sandbox. So this commit automatically disables
sandboxing if /proc is not fully visible.
2023-02-07 22:51:53 +01:00
Eelco Dolstra fb2f7f5dcc Fix auto-uid-allocation in Docker containers
This didn't work because sandboxing doesn't work in Docker. However,
the sandboxing check is done lazily - after clone(CLONE_NEWNS) fails,
we retry with sandboxing disabled. But at that point, we've already
done UID allocation under the assumption that user namespaces are
enabled.

So let's get rid of the "goto fallback" logic and just detect early
whether user / mount namespaces are enabled.

This commit also gets rid of a compatibility hack for some ancient
Linux kernels (<2.13).
2023-02-07 22:51:53 +01:00
Eelco Dolstra 7a6daf61e8 Fix activity message 2023-02-07 22:22:50 +01:00
Eelco Dolstra a8fe0dc16c Speed up fetching submodules
Previously we would completely refetch the submodules from the
network, even though the repo might already have them. Now we copy the
.git/modules directory from the repo as an optimisation. This speeds
up evaluating

  builtins.fetchTree { type = "git"; url = "/path/to/blender"; submodules = true; }

(where /path/to/blender already has the needed submodules) from 121s
to 57s.

This is still pretty inefficient and a hack, but a better solution is
best done on the lazy-trees branch.

This change also help in the case where the repo already has the
submodules but the origin is unfetchable for whatever reason
(e.g. there have been cases where Nix in a GitHub action doesn't have
the right authentication set up).
2023-02-07 16:01:36 +01:00
Eelco Dolstra 2edd5cf618 Fix the origin URL used for fetching submodules
We cannot use 'actualUrl', because for file:// repos that's not the
original URL that the repo was fetched from. This is a problem since
submodules may be relative to the original URL.

Fixes e.g.

  nix eval --impure --json --expr 'builtins.fetchTree { type = "git"; url = "/path/to/blender"; submodules = true; }'

where /path/to/blender is a clone of
https://github.com/blender/blender.git (which has several relative
submodules like '../blender-addons.git').
2023-02-07 16:01:36 +01:00
Eelco Dolstra 81e75e4bf6 Add some progress indication when fetching submodules 2023-02-07 16:01:36 +01:00
Yorick 631ba6442a
build-remote: store maxBuildJobs before forcing it to 1 2023-02-07 12:08:00 +01:00
Yorick 3050005211
build-remote: don't warn when all local build slots are taken
Previously, build-remote would show a warning if all build slots were
taken, even if they would open up later. This caused a lot of spam in
the logs. Disable this warning when maxJobs > 0.

See #6263
2023-02-06 17:53:03 +01:00
John Ericson 44bea52ae3 Scope down --derivation to just the commands that use it
Per the old FIXME, this flag was on too many commands, and mostly
ignored. Now it is just on the commands where it actually has an effect.

Per https://github.com/NixOS/nix/issues/7261, I would still like to get
rid of it entirely, but that is a separate project. This change should
be good with or without doing that.
2023-02-04 18:30:02 -05:00
John Ericson 45fa297e40 Factor out InstallableStorePath to its own file, dedup
`nix app` had something called `InstallableDerivedPath` which is
actually the same thing. We go with the later's name because it has
become more correct.

I originally did this change (more hurriedly) as part of #6225 --- a
mini store-only Nix and a full Nix need to share this code. In the first
RFC meeting for https://github.com/NixOS/rfcs/pull/134 we discussed how
some splitting of the massive `installables.cc` could begin prior, as
that is a good thing anyways. (@edolstra's words, not mine!) This would
be one such PR.
2023-02-03 11:26:39 -05:00
Jörg Thalheim f20d3726dd advertise transport encoding in http transfers to
tl;dr: With this 1 line change I was able to get a speedup of 1.5x on 1Gbit/s
wan connections by enabling zstd compression in nginx.

Also nix already supported all common compression format for http
transfer, webservers usually only enable them if they are advertised
through the Accept-Encoding header.

This pull requests makes nix advertises content compression support for
zstd, br, gzip and deflate.

It's particular useful to add transparent compression for binary caches
that serve packages from the host nix store in particular nix-serve,
nix-serve-ng and harmonia.

I tried so far gzip, brotli and zstd, whereas only zstd was able to bring
me performance improvements for 1Gbit/s WAN connections.

The following nginx configuration was used in combination with the
[zstd module](https://github.com/tokers/zstd-nginx-module) and
[harmonia](https://github.com/nix-community/harmonia/)

```nix
{
  services.nginx.virtualHosts."cache.yourhost.com" = {
    locations."/".extraConfig = ''
      proxy_pass http://127.0.0.1:5000;
      proxy_set_header Host $host;
      proxy_redirect http:// https://;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;

      zstd on;
      zstd_types application/x-nix-archive;
    '';
  };
}
```

For testing I unpacked a linux kernel tarball to the nix store using
this command `nix-prefetch-url --unpack https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.8.tar.gz`.

Before:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwbb88sw36fvah-linux-6.1.8.tar.gz'
warning: Git tree '/scratch/joerg/nix' is dirty

real    0m18,375s
user    0m2,889s
sys     0m1,558s
```

After:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwb
b88sw36fvah-linux-6.1.8.tar.gz'

real    0m11,884s
user    0m4,130s
sys     0m1,439s
```

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>

Update src/libstore/filetransfer.cc

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-02-03 12:33:38 +00:00
Eelco Dolstra dbe0748f97
Merge pull request #7739 from obsidiansystems/user-settings
Move `trustedUsers` and `allowedUsers` to separate config struct
2023-02-03 11:55:37 +01:00
John Ericson a47e055e09 Move trustedUsers and allowedUsers to separate config struct
These settings are not needed for libstore at all, they are just used by
the nix daemon *command* for authorization on unix domain sockets. My
moving them to a new configuration struct just in that file, we avoid
them leaking anywhere else.

Also, it is good to break up the mammoth `Settings` struct in general.
Issue #5638 tracks this.

The message is not changed because I do not want to regress in
convenience to the user. Just saying "this connection is not trusted"
doesn't tell them out to fix the issue. The ideal thing to do would be
to somehow parameterize `processCommand` on how the error should be
displayed, so different sorts of connections can display different
information to the user based on how authentication is performed for the
connection in question. This, however, is a good bit more work, so it is
left for the future.

This came up with me thinking about the tcp:// store (#5265). The larger
project is not TCP *per se*, but the idea that it should be possible for
something else to manage access control to services like the Nix Daemon,
and those services simply trust or trust the incoming connection as they
are told. This is a more capability-oriented way of thinking about trust
than "every server implements its own auth separately" as we are used to today.

Its very great that libstore itself already implements just this model,
and so via this refactor I basically want to "enshrine" that so it
continues to be the case.
2023-02-02 14:17:24 -05:00
John Ericson 479c011784 Get rid of the authHook parameter on processConnection
This is (morally) dead code.

As @edolstra pointed out in
https://github.com/NixOS/nix/pull/5226#discussion_r1073470813, this is
no longer needed.

I created this in 8d4162ff9e, so it is
fitting that I now destroy it :).
2023-02-02 12:02:03 -05:00
John Ericson 87ca46263d Merge branch 'master' into path-info 2023-02-02 09:55:07 -05:00
Eelco Dolstra b574c70ccb
Merge pull request #7736 from shlevy/plugin-files-daemon
Don't send plugin-files to the daemon.
2023-02-02 12:35:51 +01:00
Shea Levy 895c525d04
daemon: Warn on old clients passing unexpected plugin-files.
The setting itself was already ignored due to exception trying to set pluginFiles.
2023-02-02 06:03:45 -05:00
Shea Levy 92edc38369
Don't send plugin-files to the daemon.
This is radically unsafe and the daemon has already loaded its plugins
anyway.

Fixes cachix/devenv#276
2023-02-01 20:05:56 -05:00
Jamie Quigley 32ca59649b
nix-shell: Colour the prompt red if the user is root
This matches the nixos prompt colours - green for standard user, red for
root
2023-02-01 20:50:44 +00:00
Eelco Dolstra e8ca49f6ef Fix clang compilation 2023-02-01 20:34:44 +01:00
Eelco Dolstra 57a4258426 Remove an unused capture 2023-02-01 20:27:35 +01:00
John Ericson ee9eb83a84 Remove some designated initializers
With the switch to C++20, the rules became more strict, and we can no
longer initialize base classes. Make them comments instead.

(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)
2023-02-01 11:25:56 -05:00
Eelco Dolstra 7a09bfbcb6
Merge pull request #7723 from yorickvP/nix-store-ping-json
nix store ping: add --json flag
2023-02-01 17:11:34 +01:00
John Ericson 59d3175649 Put back TODO
I don't think the `narHash` is in need of documentation more than the
other undocumented fields, but regardless this change has nothing to do
with that field and so we should leave the comment as is.
2023-02-01 10:09:25 -05:00
John Ericson db759b1bc2 Undo style change
`&` without space before is far more common on this codebase than I
thought, so it is not worth changing just this one file. Maybe we will
adopt a formatter someday but until then this is fine.
2023-02-01 10:07:54 -05:00
John Ericson 0983a0bd30 Shrink diff in one place 2023-02-01 10:04:28 -05:00
John Ericson 43414738a0 Merge remote-tracking branch 'upstream/master' into path-info 2023-02-01 10:02:30 -05:00
Théophane Hufschmitt 518da6c6a3
Merge pull request #7716 from obsidiansystems/small-storePath-cleanups
Separate `path.hh` from `content-address.hh`
2023-02-01 16:00:28 +01:00
Eelco Dolstra b55a946d8d
Merge pull request #7717 from obsidiansystems/delete-dead-code
Delete dead code
2023-02-01 15:57:04 +01:00
Théophane Hufschmitt e32c5c2c77
Merge pull request #7667 from dramforever/flake-search-attr
Better error message for nix search when attr is not found
2023-02-01 15:56:22 +01:00
Eelco Dolstra 845b7f067d
Merge pull request #7726 from hercules-ci/flake-show-when-empty-hide
nix flake show: Ignore empty attrsets
2023-02-01 15:49:13 +01:00
Eelco Dolstra 14b0b9ea5a
Merge pull request #7203 from graham33/feature/cpp20
Proposal: Use C++20
2023-02-01 15:41:04 +01:00
Robert Hensing 60d48eda23 nix flake show: Ignore empty attrsets
For frameworks it's important that structures are as lazy as possible
to prevent infinite recursions, performance issues and errors that
aren't related to the thing to evaluate. As a consequence, they have
to emit more attributes than strictly (sic) necessary.
However, these attributes with empty values are not useful to the user
so we omit them.
2023-01-31 18:20:26 +01:00
Yorick 4f4a6074e4
nix store ping: try to print json if connect() fails aswell 2023-01-31 15:10:39 +01:00
Yorick 28648ed784
nix store ping: add --json flag 2023-01-31 13:24:23 +01:00
John Ericson b6c98752f6 Merge remote-tracking branch 'upstream/master' into path-info 2023-01-30 18:04:54 -05:00
Robert Hensing c9b9260f34
Merge pull request #7713 from obsidiansystems/more-rapid-check
Add more property tests
2023-01-30 18:54:53 +01:00
John Ericson e21aa43212 Delete dead code
The references set seems to have been unused since `LegacySSHStore`
references were first created in
caa5793b4a.

The method decls never were upstream, and accidentally added by me in
062533f7cd (probably due to `git rerere`).
Sorry!

This reduces the diff from #3746.
2023-01-30 11:29:01 -05:00