Commit graph

1680 commits

Author SHA1 Message Date
Eelco Dolstra 1e6a5d1ff6 Clean up cgroup handling in getMaxCPU()
Also, don't assume in LocalDerivationGoal that cgroups are mounted on
/sys/fs/cgroup.
2022-12-02 12:59:13 +01:00
Eelco Dolstra 1211e59a03 Move cgroup.{cc,hh} to libutil 2022-12-02 12:38:03 +01:00
Eelco Dolstra fbc53e97ed
Merge pull request #3600 from NixOS/auto-uid-allocation
Automatic UID allocation
2022-11-29 14:01:42 +01:00
Eelco Dolstra 653b32a78f Merge remote-tracking branch 'origin/master' into auto-uid-allocation 2022-11-21 11:33:23 +01:00
Eelco Dolstra 128910ba23 Separate cgroup support from auto-uid-allocation
The new experimental feature 'cgroups' enables the use of cgroups for
all builds. This allows better containment and enables setting
resource limits and getting some build stats.
2022-11-18 10:39:28 +01:00
Eelco Dolstra f1ab082ac4 createTempDir(): Use std::atomic 2022-11-18 09:37:11 +01:00
Yorick 09f00dd4d0
Replace src/libutil/json.cc with nlohmann json generation 2022-11-16 16:50:50 +01:00
Eelco Dolstra 6c6eff8ac4 Remove the SystemdCgroup feature 2022-11-10 17:24:12 +01:00
Eelco Dolstra 40911d7dec Remove stray tab 2022-11-04 13:30:35 +01:00
Eelco Dolstra b95faccf03 Merge remote-tracking branch 'origin/master' into auto-uid-allocation 2022-11-03 17:43:40 +01:00
Yorick 34ea0e2e7b
tarfile: set directory mode to at least 0500, don't extract fflags
We don't need SGID, or any ACL's. We also want to keep every dir +rx.
2022-11-01 16:01:38 +01:00
Guillaume Maudoux e93bf69b44 Rework error throwing, and test it 2022-10-25 01:46:10 +02:00
Graham Bennett 4563e80363 Fix C++20 warnings 2022-10-22 15:16:46 +01:00
Guillaume Maudoux b945b844a9 Initial frames support 2022-10-17 03:05:02 +02:00
Guillaume Maudoux 3f9f6ae127 Merge remote-tracking branch 'origin/master' into coerce-string 2022-10-16 20:39:19 +02:00
squalus 223f8dace0 archive: check close errors when extracting nars 2022-09-22 12:50:32 -07:00
squalus 1b595026e1 Improve durability of schema version file writes
- call close explicitly in writeFile to prevent the close exception
  from being ignored
- fsync after writing schema file to flush data to disk
- fsync schema file parent to flush metadata to disk

https://github.com/NixOS/nix/issues/7064
2022-09-19 20:13:30 -07:00
Eelco Dolstra d365cced4f
Trim option descriptions
This removes unintended blank lines in Markdown when the description
is a multiline string literal.
2022-09-13 16:58:32 +02:00
Guillaume Maudoux eb460a9529 WIP: broken merge but need a git checkpoint 2022-09-07 00:34:03 +02:00
Eelco Dolstra bb411e4ae1 Fix progress bar flicker with -L
This was caused by -L calling setLogFormat() again, which caused the
creation of a new progress bar without destroying the old one. So we
had two progress bars clobbering each other.

We should change 'logger' to be a smart pointer, but I'll do that in a
future PR.

Fixes #6931.
2022-08-24 22:36:40 +02:00
Naïm Favier 8188b1d0ab
json: write null on abnormal placeholder destruction
Avoids leaving dangling attributes like

    {
        "foo":
    }

in case of exceptions.
2022-08-19 01:00:10 +02:00
Eelco Dolstra 53e7b7e8ac Remove warnLargeDump()
This message was unhelpful (#1184) and probably misleading since
memory is O(1) in most cases now.
2022-08-17 11:32:01 +02:00
Théophane Hufschmitt 5192bb093a Merge remote-tracking branch 'upstream/master' into fix-mv-in-different-filesystems 2022-08-08 15:42:56 +02:00
Théophane Hufschmitt f4a8426098 Remove the explicit c++fs linkage on darwin
Doesn't seem needed on a recent-enough clang anymore (and even seems to
break stuff)
2022-08-08 14:34:22 +02:00
Dave Nicponski cb6794a0d9 Do not spam logs if the owned-homedir check results in a noop 2022-08-07 10:13:11 -04:00
Eelco Dolstra ccbd906c86 Fix NIX_COUNT_CALLS=1
Also, make the JSON writer support std::string_view.

Fixes #6857.
2022-08-03 17:46:51 +02:00
Théophane Hufschmitt 1ba5b3e001 Make moveFile more atomic
Rather than directly copying the source to its dest, copy it first to a
temporary location, and eventually move that temporary.
That way, the move is at least atomic from the point-of-view of the destination
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt 90f9680733 Only use renameFile where needed
In most places the fallback to copying isn’t needed and can actually be
bad, so we’d rather not transparently fallback
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt d71d9e9fbf moveFile -> renameFile
`move` tends to have this `mv` connotation of “I will copy it for you if
needs be”
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt a4f0fd633c Link against c++fs on darwin
Required by the old clang version
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt c5db1821a9 Re-implement the recursive directory copy
The recursive copy from the stl doesn’t exactly do what we need because
1. It doesn’t delete things as we go
2. It doesn’t keep the mtime, which change the nars

So re-implement it ourselves. A bit dull, but that way we have what we want
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt 6f89fb6008 rename: Fallback to a copy if the filesystems mismatch
In `nix::rename`, if the call to `rename` fails with `EXDEV` (failure
because the source and the destination are in a different filesystems)
switch to copying and removing the source.

To avoid having to re-implement the copy manually, I switched the
function to use the c++17 `filesystem` library (which has a `copy`
function that should do what we want).

Fix #6262
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt c2de0a232c Create a wrapper around stdlib’s rename
Directly takes some c++ strings, and gently throws an exception on error
(rather than having to inline this logic everywhere)
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt 8119390abc Move some fs-related functions to their own file
Unclutter `util.cc` a bit
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt 2805439335
Merge pull request #6814 from amjoseph-nixpkgs/pr/sandbox-error-messages
local-derivation-goal.cc: improve error messages when sandboxing fails
2022-07-22 13:27:52 +02:00
Alex Wied 722de8ddcc libstore/globals.cc: Move cgroup detection to libutil 2022-07-19 16:25:53 -04:00
Adam Joseph a9e75eca00 error.hh: add additional constructor with explicit errno argument 2022-07-19 03:49:33 -07:00
Théophane Hufschmitt f071eb3702
Merge pull request #6693 from ncfavier/complete-flake-inputs
Improve shell completion of flake inputs
2022-07-12 09:12:00 +02:00
Naïm Favier 711b2e1f48
Fix flake input completion for InstallablesCommands
Defers completion of flake inputs until the whole command line is parsed
so that we know what flakes we need to complete the inputs of.

Previously, `nix build flake --update-input <Tab>` always behaved like
`nix build . --update-input <Tab>`.
2022-07-11 15:13:51 +02:00
Théophane Hufschmitt 4c8210095e
Merge pull request #6233 from flox/nix-repl-flakes
Nix repl flakes
2022-06-29 17:59:22 +02:00
Eelco Dolstra 1e55ee2961 getSelfExe(): Support macOS 2022-06-23 01:32:46 +02:00
Eelco Dolstra d3176ce076 Fix build-remote in nix-static
'build-remote' is now executed via /proc/self/exe so it always works.
2022-06-23 01:32:46 +02:00
Eelco Dolstra f6cf644e5f Style 2022-06-22 15:35:52 +02:00
Dave Nicponski ca2be509b9 Verify $HOME is owned by current user in getHome(), if it exists.
Useful because a default `sudo` on darwin doesn't clear `$HOME`, so things like `sudo nix-channel --list`
will surprisingly return the USER'S channels, rather than `root`'s.

Other counterintuitive outcomes can be seen in this PR description:
  https://github.com/NixOS/nix/pull/6622
2022-06-17 17:42:08 -04:00
Tom Bereknyei 51268ceb79 Merge branch 'master' into nix-repl-flakes 2022-06-15 09:17:03 -04:00
Eelco Dolstra 798efdf10b
Merge pull request #6027 from Ma27/pure-repl
repl: `--option pure-eval true` actually enables pure eval mode
2022-06-13 16:11:46 +02:00
Fishhh d137ceccef
Fix incorrect comment in hiliteMatches 2022-06-05 20:30:18 +02:00
Tom Bereknyei ffd41d1757 Merge branch 'master' into nix-repl-flakes 2022-06-02 16:58:35 -04:00
Eelco Dolstra 28e08822a3
Avoid unnecessary string copy 2022-06-02 21:26:28 +02:00
Eelco Dolstra abb80cfa4c
Add operator for concatenating strings and string_views 2022-06-02 21:26:21 +02:00
Maximilian Bosch 159b5815b5
repl: --option pure-eval true actually enables pure eval mode
To quote Eelco in #5867:

> Unfortunately we can't do
>
>     evalSettings.pureEval.setDefault(false);
>
> because then we have to do the same in main.cc (where
> pureEval is set to true), and that would allow pure-eval
> to be disabled globally from nix.conf.

Instead, a command should specify that it should be impure by
default. Then, `evalSettings.pureEval` will be set to `false;` unless
it's overridden by e.g. a CLI flag.

In that case it's IMHO OK to be (theoretically) able to override
`pure-eval` via `nix.conf` because it doesn't have an effect on commands
where `forceImpureByDefault` returns `false` (i.e. everything where pure
eval actually matters).

Closes #5867
2022-05-31 22:20:11 +02:00
Sergei Trofimovich b36d5172cb src/libutil/json.cc: add missing <cstdint> include for gcc-13
Without the change llvm build fails on this week's gcc-13 snapshot as:

    src/libutil/json.cc: In function 'void nix::toJSON(std::ostream&, const char*, const char*)':
    src/libutil/json.cc:33:22: error: 'uint16_t' was not declared in this scope
       33 |             put(hex[(uint16_t(*i) >> 12) & 0xf]);
          |                      ^~~~~~~~
    src/libutil/json.cc:5:1: note: 'uint16_t' is defined in header '<cstdint>'; did you forget to '#include <cstdint>'?
        4 | #include <cstring>
      +++ |+#include <cstdint>
        5 |
2022-05-26 18:37:15 +01:00
Robert Hensing c156155239 createUnixDomainSocket: listen(unix, 5 -> 100)
This solves the error

    error: cannot connect to socket at '/nix/var/nix/daemon-socket/socket': Connection refused

on build farm systems that are loaded but operating normally.

I've seen this happen on an M1 mac running a loaded hercules-ci-agent.
Hercules CI uses multiple worker processes, which may connect to
the Nix daemon around the same time. It's not unthinkable that
the Nix daemon listening process isn't scheduled until after 6
workers try to connect, especially on a system under load with
many workers.

Is the increase safe?

The number is the number of connections that the kernel will buffer
while the listening process hasn't `accept`-ed them yet.
It did not - and will not - restrict the total number of daemon
forks that a client can create.

History

The number 5 has remained unchanged since the introduction in
nix-worker with 0130ef88ea in 2006.
2022-05-26 11:24:04 +02:00
Ben Burdette 9a5ea6c359
Merge branch 'master' into debug-exploratory-PR 2022-05-25 10:41:10 -06:00
Tom Bereknyei 7a04fb1c56 repl: add repl-flake experimental feature for gating 2022-05-20 08:20:00 -04:00
Tony Olagbaiye 5b8c1deb18 fetchTree: Allow fetching plain files
Add a new `file` fetcher type, which will fetch a plain file over
http(s), or from the local file.

Because plain `http(s)://` or `file://` urls can already correspond to
`tarball` inputs (if the path ends-up with a know archive extension),
the URL parsing logic is a bit convuluted in that:

- {http,https,file}:// urls will be interpreted as either a tarball or a
  file input, depending on the extensions of the path part (so
  `https://foo.com/bar` will be a `file` input and
  `https://foo.com/bar.tar.gz` as a `tarball` input)
- `file+{something}://` urls will be interpreted as `file` urls (with
  the `file+` part removed)
- `tarball+{something}://` urls will be interpreted as `tarball` urls (with
  the `tarball+` part removed)

Fix #3785

Co-Authored-By: Tony Olagbaiye <me@fron.io>
2022-05-19 18:24:49 +02:00
Ben Burdette 667074b586 first whack at passing evalState as an arg to debuggerHook. 2022-05-16 09:20:51 -06:00
Ben Burdette 7cd7c7c91a
Merge branch 'master' into debug-exploratory-PR 2022-05-09 09:30:44 -06:00
Théophane Hufschmitt e68676e6c8 Fix the parsing of the sourcehut refs file
Since a26be9f3b8, the same parser is used
to parse the result of sourcehut’s `HEAD` endpoint (coming from [git
dumb protocol]) and the output of `git ls-remote`. However, they are very
slightly different (the former doesn’t specify the current reference
since it’s implied to be `HEAD`).

Unify both, and make the parser a bit more robust and understandable (by
making it more typed and adding tests for it)

[git dumb protocol]: https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols#_the_dumb_protocol
2022-05-04 14:38:59 +02:00
Eelco Dolstra c98648bef0 Merge remote-tracking branch 'origin/master' into debug-exploratory-PR 2022-05-04 14:10:21 +02:00
Eelco Dolstra 3e87c8e62b Move json stuff out of util.cc 2022-05-04 11:22:06 +02:00
Alain Zscheile 1385b20078
Get rid of most .at calls (#6393)
Use one of `get` or `getOr` instead which will either return a null-pointer (with a nicer error message) or a default value when the key is missing.
2022-05-04 07:44:32 +02:00
Eelco Dolstra a3c6c5b1c7 nix profile: Support overriding outputs 2022-05-03 15:00:34 +02:00
Ben Burdette c941803861 spacing 2022-04-29 11:27:38 -06:00
Guillaume Maudoux e93b59fbc5 Merge remote-tracking branch 'origin/master' into coerce-string 2022-04-29 00:12:25 +02:00
Ben Burdette 6e19947993 Merge branch 'master' into debug-merge-master 2022-04-28 12:32:57 -06:00
Guillaume Maudoux 402ee8ab64 No point in passing string_views by reference 2022-04-28 13:02:39 +02:00
Guillaume Maudoux 5ef88457b8 Better document error location indent 2022-04-28 13:00:24 +02:00
Théophane Hufschmitt 7b889f31ea Fix the darwin build
Looks like the auto-merge is indeed quite broken and merges even when the CI fails
2022-04-22 10:56:56 +02:00
Théophane Hufschmitt 484badfa09 Add some tests for ChunkedVector 2022-04-22 10:03:44 +02:00
Théophane Hufschmitt 7ca6fbc8ca Move ChunkedVector to its own header 2022-04-22 10:01:02 +02:00
pennae 8adaa6acb5 remove pos<T>
it's no longer needed now that positions aren't really pointers any
more.
2022-04-21 21:56:34 +02:00
pennae 8775be3393 store Symbols in a table as well, like positions
this slightly increases the amount of memory used for any given symbol, but this
increase is more than made up for if the symbol is referenced more than once in
the EvalState that holds it. on average every symbol should be referenced at
least twice (once to introduce a binding, once to use it), so we expect no
increase in memory on average.

symbol tables are limited to 2³² entries like position tables, and similar
arguments apply to why overflow is not likely: 2³² symbols would require as many
string instances (at 24 bytes each) and map entries (at 24 bytes or more each,
assuming that the map holds on average at most one item per bucket as the docs
say). a full symbol table would require at least 192GB of memory just for
symbols, which is well out of reach. (an ofborg eval of nixpks today creates
less than a million symbols!)
2022-04-21 21:56:31 +02:00
pennae 00a3280232 don't use Symbol in Pos to represent a path
PosTable deduplicates origin information, so using symbols for paths is no
longer necessary. moving away from path Symbols also reduces the usage of
symbols for things that are not keys in attribute sets, which will become
important in the future when we turn symbols into indices as well.
2022-04-21 21:46:10 +02:00
pennae 6526d1676b replace most Pos objects/ptrs with indexes into a position table
Pos objects are somewhat wasteful as they duplicate the origin file name and
input type for each object. on files that produce more than one Pos when parsed
this a sizeable waste of memory (one pointer per Pos). the same goes for
ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate
a position would need at least 8GB of input and 64GB of expression memory. it's
not likely that we'll hit that any time soon, so we can use a uint32_t index to
locate positions instead.
2022-04-21 21:46:06 +02:00
Eelco Dolstra 3b9d31b88c Rename fmt test -> hilte 2022-04-21 13:00:50 +02:00
Eelco Dolstra f1eee873ea Fix fmt test 2022-04-21 13:00:24 +02:00
Eelco Dolstra f05e1f6fbb Move hiliteMatches into a separate header
This is mostly so that we don't #include <regex> everywhere (which
adds quite a bit of compilation time).
2022-04-21 12:06:29 +02:00
Théophane Hufschmitt 9345b4e9ca
Merge pull request #3720 from obsidiansystems/fix-url-format
Avoid `fmt` when constructor already does it
2022-04-20 20:01:37 +02:00
John Ericson 3c220442ff Merge remote-tracking branch 'upstream/master' into fix-url-format 2022-04-20 16:53:16 +00:00
Yorick ebf2fd76b1
Add custom to_json and from_json functions for ExperimentalFeature
nix show-config --json was serializing experimental features as ints.
nlohmann::json will automatically use these definitions to serialize
and deserialize ExperimentalFeatures.

Strictly, we don't use the from_json instance yet, it's provided for
completeness and hopefully future use.
2022-04-20 15:41:01 +02:00
Eelco Dolstra 51712bf012
Merge pull request #6128 from ncfavier/fix-completion
Shell completion improvements
2022-04-19 13:45:33 +02:00
John Ericson 75b62e5260 Avoid fmt when constructor already does it
There is a correctnes issue here, but #3724 will fix that. This is just
a cleanup for brevity's sake.
2022-04-19 01:44:11 +00:00
Ben Burdette 93b8d31508 Merge branch 'master' into debug-exploratory-PR 2022-04-15 14:49:08 -06:00
Eelco Dolstra 092f6d2e7a
Merge pull request #6380 from thufschmitt/fix-double-slahsh-in-uri
Allow empty path segments in urls
2022-04-11 11:15:14 +02:00
Théophane Hufschmitt 646af7325d
Merge pull request #6376 from Uthar/master
don't assume that rev is a SHA1 hash
2022-04-08 17:56:27 +02:00
Théophane Hufschmitt f3d3587ab3 Allow empty path segments in urls
Valid per https://datatracker.ietf.org/doc/html/rfc3986#section-3.3 (and
also somewhat frequently happening for local paths)
2022-04-08 16:09:49 +02:00
Eelco Dolstra 8bd9ebf52c Error: Remove unused sname() method 2022-04-08 11:31:51 +02:00
Eelco Dolstra 168ef9f3ab Remove unused Error.name field 2022-04-08 11:31:51 +02:00
Ben Burdette 1a93ac8133 Merge remote-tracking branch 'upstream/master' into upstream-merge 2022-04-07 13:42:01 -06:00
Kasper Gałkowski 2c2fd4946f don't assume that rev is a SHA1 hash
This was a problem when writing a fetcher that uses e.g. sha256 hashes
for revisions. This doesn't actually do anything new, but allows for
creating such fetchers in the future (perhaps when support for Git's
SHA256 object format gains more popularity).
2022-04-07 19:49:47 +02:00
Eelco Dolstra 8b1e328d5d
Merge pull request #6348 from cole-h/fix-restoring-mount-namespace
libutil: Fix restoring mount namespace
2022-04-07 18:15:33 +02:00
Eelco Dolstra c0ad86f681
Merge pull request #6366 from danpls/base64-reserve
libutil: Reserve memory when en/decoding base64
2022-04-05 23:20:33 +02:00
Daniel Pauls 513652d594 tokenizeString: Fix semantic mistake
`string_view::find_first_not_of(...)` and
`string_view::find_first_of(...)` return `string_view::npos` on error
not `string::npos`.
2022-04-05 22:33:03 +02:00
Daniel Pauls 1fa0393479 libutil: Reserve memory when en/decoding base64
The size of the output when encoding to and decoding from base64 is
(roughly) known so we can allocate it in advance to prevent
reallocation.
2022-04-05 21:30:50 +02:00
Cole Helbling 56009b2639 libutil: don't save cwd fd, use path instead
Saving the cwd fd didn't actually work well -- prior to this commit, the
following would happen:

    : ~/w/vc/nix ; doas outputs/out/bin/nix --experimental-features 'nix-command flakes' run nixpkgs#coreutils -- --coreutils-prog=pwd
    pwd: couldn't find directory entry in ‘../../../..’ with matching i-node
    : ~/w/vc/nix ; doas outputs/out/bin/nix --experimental-features 'nix-command flakes' develop -c pwd
    pwd: couldn't find directory entry in ‘../../../..’ with matching i-node
2022-04-04 10:28:08 -07:00
Cole Helbling 10b9c1b2b2 libutil: save cwd fd in restoreMountNamespace
This doesn't work very well (maybe I'm misunderstanding the desired
implementation):

    : ~/w/vc/nix ; doas outputs/out/bin/nix --experimental-features 'nix-command flakes' develop -c pwd
    pwd: couldn't find directory entry in ‘../../../..’ with matching i-node
2022-04-04 10:28:00 -07:00
Cole Helbling f89b0f7846 libutil: try restoring the cwd from fdSavedCwd 2022-04-04 08:33:59 -07:00
Cole Helbling e135d223f6 libutil: save fd to cwd instead of cwd itself 2022-04-04 08:32:45 -07:00
Cole Helbling e5b70d47aa libutil: cleanup savedCwd logic
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-04-04 08:20:11 -07:00
Cole Helbling 2a45cf54e4 libutil: Properly guard self-allocating getcwd on GNU
It's a GNU extension, as pointed out by pennae.
2022-04-01 12:20:34 -07:00
Cole Helbling 7f5caaa7c0 libutil: Don't use std::filesystem
Just in case making libutil depend on std::filesystem is unacceptable,
here is the non-filesystem approach.
2022-04-01 10:24:31 -07:00
aszlig 435848cef1 libutil: Fix restoring mount namespace
I regularly pass around simple scripts by using nix-shell as the script
interpreter, eg. like this:

    #!/usr/bin/env nix-shell
    #!nix-shell -p dd_rescue coreutils bash -i bash

While this works most of the time, I recently had one occasion where it
would not and the above would result in the following:

    $ sudo ./myscript.sh
    bash: ./myscript.sh: No such file or directory

Note the "sudo" here, because this error only occurs if we're root.

The reason for the latter is because running Nix as root means that we
can directly access the store, which makes sure we use a filesystem
namespace to make the store writable. XXX - REWORD!

So when stracing the process, I stumbled on the following sequence:

    openat(AT_FDCWD, "/proc/self/ns/mnt", O_RDONLY) = 3
    unshare(CLONE_NEWNS)                            = 0
    ... later ...
    getcwd("/the/real/cwd", 4096)                   = 14
    setns(3, CLONE_NEWNS)                           = 0
    getcwd("/", 4096)                               = 2

In the whole strace output there are no calls to chdir() whatsoever, so
I decided to look into the kernel source to see what else could change
directories and found this[1]:

    /* Update the pwd and root */
    set_fs_pwd(fs, &root);
    set_fs_root(fs, &root);

The set_fs_pwd() call is roughly equivalent to a chdir() syscall and
this is called when the setns() syscall is invoked[2].

[1]: b14ffae378/fs/namespace.c (L4659)
[2]: b14ffae378/kernel/nsproxy.c (L346)
2022-04-01 09:30:52 -07:00
Eelco Dolstra c9a29d0d92
Merge pull request #6227 from NixOS/impure-derivations-ng
Impure derivations
2022-03-31 19:58:35 +02:00
Eelco Dolstra 5cd72598fe Add support for impure derivations
Impure derivations are derivations that can produce a different result
every time they're built. Example:

  stdenv.mkDerivation {
    name = "impure";
    __impure = true; # marks this derivation as impure
    outputHashAlgo = "sha256";
    outputHashMode = "recursive";
    buildCommand = "date > $out";
  };

Some important characteristics:

* This requires the 'impure-derivations' experimental feature.

* Impure derivations are not "cached". Thus, running "nix-build" on
  the example above multiple times will cause a rebuild every time.

* They are implemented similar to CA derivations, i.e. the output is
  moved to a content-addressed path in the store. The difference is
  that we don't register a realisation in the Nix database.

* Pure derivations are not allowed to depend on impure derivations. In
  the future fixed-output derivations will be allowed to depend on
  impure derivations, thus forming an "impurity barrier" in the
  dependency graph.

* When sandboxing is enabled, impure derivations can access the
  network in the same way as fixed-output derivations. In relaxed
  sandboxing mode, they can access the local filesystem.
2022-03-31 13:43:20 +02:00
Eelco Dolstra 0fe8849914
Merge pull request #6337 from danpls/fix-to-json-repl
libexpr: Throw the correct error in toJSON
2022-03-31 11:33:16 +02:00
Eelco Dolstra 28309352d9 replaceEnv(): Pass newEnv by reference 2022-03-31 10:39:53 +02:00
Daniel Pauls 629edd43ba libutil: Change return value of addTrace to void
The return value of BaseError::addTrace(...) is never used and
error-prone as subclasses calling it will return a BaseError instead of
the subclass.
This commit changes its return value to be void.
2022-03-30 18:37:32 +02:00
Maximilian Bosch c85467a1b6
Revert "TarArchive: Small refactoring"
This reverts commit 50a35860ee.

With this change Nix fails to open bzip2 logfiles that were created from
builds with no stdout/stderr.
2022-03-24 22:30:46 +01:00
Eelco Dolstra f902f3c2cb Add experimental feature 'fetch-closure' 2022-03-24 21:33:33 +01:00
Guillaume Maudoux 963b8aa39b Explain current error trace impl 2022-03-18 23:17:50 +01:00
Guillaume Maudoux ca5c3e86ab Merge remote-tracking branch 'origin/master' into coerce-string 2022-03-18 01:25:55 +01:00
Guillaume Maudoux e6d07e0d89 Refactor to use more traces and less string manipulations 2022-03-18 00:58:09 +01:00
Eelco Dolstra a53c1dc96d
Merge pull request #6245 from trofi/fix-nix-store-gc-limit-auto-optimise-store
nix store gc: account for auto-optimised store
2022-03-16 09:19:57 +01:00
Maximilian Bosch 34e20c164c
libfetchers/path: set lastModified to path's mtime
When importing e.g. a local `nixpkgs` in a flake to test a change like

    {
      inputs.nixpkgs.url = path:/home/ma27/Projects/nixpkgs;
      outputs = /* ... */
    }

then the input is missing a `lastModified`-field that's e.g. used in
`nixpkgs.lib.nixosSystem`. Due to the missing `lastMoified`-field, the
mtime is set to 19700101:

    result -> /nix/store/b7dg1lmmsill2rsgyv2w7b6cnmixkvc1-nixos-system-nixos-22.05.19700101.dirty

With this change, the `path`-fetcher now sets a `lastModified` attribute
to the `mtime` just like it's the case in the `tarball`-fetcher already.
When building NixOS systems with `nixpkgs` being a `path`-input and this
patch, the output-path now looks like this:

    result -> /nix/store/ld2qf9c1s98dxmiwcaq5vn9k5ylzrm1s-nixos-system-nixos-22.05.20220217.dirty
2022-03-15 12:32:11 +01:00
Sergei Trofimovich 6b1872312f nix store gc: account for auto-optimised store
Before the change on a system with `auto-optimise-store = true`:

    $ nix store gc --verbose --max 1

deleted all the paths instead of one path (we requested 1 byte limit).

It happens because every file in `auto-optimise-store = true` has at
least 2 links: file itself and a link in /nix/store/.links/ directory.

The change conservatively assumes that any file that has one (as before)
or two links (assume auto-potimise mode) will free space.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2022-03-13 07:24:48 +00:00
Eelco Dolstra 31a392dfe2 Merge pull request #5865 from pennae/memory-friendliness
be more memory friendly
2022-03-11 11:52:39 +01:00
pennae 8e2eaaaf69 make Finally more local
no need for function<> with c++17 deduction. this saves allocations and virtual
calls, but has the same semantics otherwise. not going through function has the
side effect of giving compilers more insight into the cleanup code, so we need a
few local warning disables.
2022-03-09 00:16:50 +01:00
pennae 47baa9d43c make Pos smaller
reduces peak hep memory use on eval of our test system from 264.4MB to 242.3MB,
possibly also a slight performance boost.

theoretically memory use could be cut down by another eight bytes per Pos on
average by turning it into a tuple containing an index into a global base
position table with row and column offsets, but that doesn't seem worth the
effort at this point.
2022-03-08 23:30:18 +01:00
Eelco Dolstra 30ddd37873 Merge branch 'cli-suggestions' of https://github.com/thufschmitt/nix 2022-03-07 19:47:45 +01:00
regnat dcf4f77fac Merge or-suggestions.hh into suggestions.hh
No real need for keeping a separate header for such a simple class.

This requires changing a bit `OrSuggestions<T>::operator*` to not throw
an `Error` to prevent a cyclic dependency. But since this error is only
thrown on programmer error, we can replace the whole method by a direct
call to `std::get` which will raise its own assertion if needs be.
2022-03-07 17:49:02 +01:00
Naïm Favier 55c6906701
Perform tilde expansion when completing flake fragments
Allows completing `nix build ~/flake#<Tab>`.
We can implement expansion for `~user` later if needed.
Not using wordexp(3) since that expands way too much.
2022-03-07 12:01:54 +01:00
Naïm Favier 5461ff532d
Make completeDir follow symlinks
Allows completing `nix why-depends /run/cur<Tab>` to /run/current-system
2022-03-07 12:01:54 +01:00
Naïm Favier a6d7cd4183
Ensure the completion marker is not processed beyond completion
I was surprised to see an error mentioning ___COMPLETE___ when trying to
complete a flag argument that had no completer implemented
2022-03-07 12:01:54 +01:00
regnat 313bbc07a8 Implement operator<< for Suggestions
That way there’s no need to explicitely convert it to a string when
printing it
2022-03-07 10:09:10 +01:00
regnat fd45d85b41 Move OrSuggestions to its own header
Prevents a recursive inclusion
2022-03-07 10:09:10 +01:00
regnat 98e361ad4c Also display suggestions for the commands using the eval cache
Make `nix build .#nix-armv8l-linux` work for example
2022-03-07 10:09:10 +01:00
regnat 2405bbbb5e Add some tests for the suggestions 2022-03-07 10:09:10 +01:00
regnat c0792b1546 Implement a suggestions mechanism
Each `Error` class now includes a set of suggestions, and these are printed by
the top-level handler.
2022-03-07 10:09:09 +01:00
Théophane Hufschmitt a2ace54fe4
Merge pull request #6029 from Ma27/nix-log-ssh-ng
ssh-ng: also store build logs to make them accessible by `nix log`
2022-03-07 09:51:40 +01:00
John Ericson 6636202356 Factor out a GcStore interface
Starts progress on #5729.

The idea is that we should not have these default methods throwing
"unimplemented". This is a small step in that direction.

I kept `addTempRoot` because it is a no-op, rather than failure. Also,
as a practical matter, it is called all over the place, while doing
other tasks, so the downcasting would be annoying.

Maybe in the future I could move the "real" `addTempRoot` to `GcStore`,
and the existing usecases use a `tryAddTempRoot` wrapper to downcast or
do nothing, but I wasn't sure whether that was a good idea so with a
bias to less churn I didn't do it yet.
2022-03-03 19:01:25 +00:00
Eelco Dolstra 6097790863 Fix segfault in headerCallback()
https://hydra.nixos.org/build/168594664
2022-03-03 11:11:16 +01:00
Anders Kaseorg b5cd3e2d5c filterANSIEscapes: Ignore BEL character
GCC is not as good at music as it seems to think it is.  Fixes #4546.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2022-03-01 15:35:42 -08:00
John Ericson dc92b01885 Merge remote-tracking branch 'upstream/master' into auto-uid-allocation 2022-02-28 23:54:20 +00:00
Maximilian Bosch 6a8f1b548f
logging.hh: json.hpp -> json_fwd.hpp 2022-02-28 17:27:52 +01:00
Maximilian Bosch cd92ea5885
libstore/derivation-goal: avoid double-parsing of JSON messages
To avoid that JSON messages are parsed twice in case of
remote builds with `ssh-ng://`, I split up the original
`handleJSONLogMessage` into three parts:

* `parseJSONMessage(const std::string&)` checks if it's a message in the
  form of `@nix {...}` and tries to parse it (and prints an error if the
  parsing fails).
* `handleJSONLogMessage(nlohmann::json&, ...)` reads the fields from the
  message and passes them to the logger.
* `handleJSONLogMessage(const std::string&, ...)` behaves as before, but
  uses the two functions mentioned above as implementation.

In case of `ssh-ng://`-logs the first two methods are invoked manually.
2022-02-28 17:27:52 +01:00
Eelco Dolstra df552ff53e Remove std::string alias (for real this time)
Also use std::string_view in a few more places.
2022-02-25 16:13:02 +01:00
Eelco Dolstra 14b38d0887 xml-writer: Remove std aliases 2022-02-25 16:13:02 +01:00
Eelco Dolstra 36c7b12f33 Remove std::string alias 2022-02-21 16:37:25 +01:00
Eelco Dolstra 1ac2664472 Remove std::vector alias 2022-02-21 16:32:34 +01:00
Eelco Dolstra fe9afb65bb Remove std::set alias 2022-02-21 16:28:23 +01:00
Eelco Dolstra afcdc7606c Remove std::list alias 2022-02-21 16:25:12 +01:00
Eelco Dolstra e2422c4582
Merge pull request #6052 from hercules-ci/issue-3294-fix-interruptCallback-deadlock
Fix deadlocked nix-daemon zombies on darwin #3294
2022-02-21 16:21:45 +01:00
Robert Hensing ddb6740e7d triggerInterrupt: Refactor to use break 2022-02-21 15:43:43 +01:00
Sergei Trofimovich 3ec02deb20 Make sure no exceptions leave ignoreException()
I noticed that occasional Ctrl-C leaves *.lock files around.
`nix-daemon`'s journal logs contained crashes like:

    nix-daemon[30416]: terminate called after throwing an instance of 'nix::SysError'
    nix-daemon[30416]:   what():  error: writing to file: Broken pipe

And core dump backtraces pointed at `teriminate()` call from
destructors:

    ...
    _Unwind_Resume ()
    nix::ignoreException() ()
    nix::LocalDerivationGoal::~LocalDerivationGoal()
    ...

    void ignoreException()
    {
        try {
            throw;
        } catch (std::exception & e) {
            printError("error (ignored): %1%", e.what());
        }
    }

The crashes happen when client side closes early and printError() throws
an IO error.

The change wraps `ignoreException()` into blanket `try { ... } catch (...) {}`.

Closes: https://github.com/NixOS/nix/issues/6046
2022-02-07 16:20:56 +00:00
Robert Hensing c3b942e0fc Don't hold interruptCallbacks lock during interrupt handling
This changes the representation of the interrupt callback list to
be safe to use during interrupt handling.

Holding a lock while executing arbitrary functions is something to
avoid in general, because of the risk of deadlock.

Such a deadlock occurs in https://github.com/NixOS/nix/issues/3294
where ~CurlDownloader tries to deregister its interrupt callback.

This happens during what seems to be a triggerInterrupt() by the
daemon connection's MonitorFdHup thread. This bit I can not confirm
based on the stack trace though; it's based on reading the code,
so no absolute certainty, but a smoking gun nonetheless.
2022-02-06 13:53:28 +01:00
Ben Burdette dbe3fd3735 Merge branch 'master' into debug-step 2022-02-04 15:09:40 -07:00
Eelco Dolstra cd35bbbeef Merge branch 'more-stringviews' of https://github.com/pennae/nix 2022-02-02 12:38:37 +01:00
Will Dietz a0357abda7 canonPath: fix missing slash when resolving links
Fixes #6017
2022-01-29 16:32:27 -06:00
pennae d439dceb3b optionally return string_view from coerceToString
we'll retain the old coerceToString interface that returns a string, but callers
that don't need the returned value to outlive the Value it came from can save
copies by using the new interface instead. for values that weren't stringy we'll
pass a new buffer argument that'll be used for storage and shouldn't be
inspected.
2022-01-27 22:15:30 +01:00
pennae 41d70a2fc8 return string_views from forceString*
once a string has been forced we already have dynamic storage allocated for it,
so we can easily reuse that storage instead of copying.
2022-01-27 17:15:43 +01:00
pennae 0d7fae6a57 convert a for more utilities to string_view 2022-01-27 17:15:43 +01:00
Eelco Dolstra fcf3528ad1 Remove unused Rust stuff
In particular we were still compiling rust-ffi.cc even though we're
not using it.
2022-01-25 11:58:00 +01:00
Eelco Dolstra 7afbdf2545 hiliteMatches(): Style fixes, pass more stuff by reference 2022-01-24 14:47:34 +01:00
regnat ffb28eaa1e Add a small documentation for hiliteMatches 2022-01-24 10:07:02 +01:00
Fishhh f82a426502
Add some tests for hiliteMatches in libutil 2022-01-21 20:10:54 +01:00
Fishhh 1e0b7cdc3f
Move hilite_all into libutil and rename it to hiliteMatches
The signature was also changed so the function now accepts a vector
instead of an iterator
2022-01-21 20:10:46 +01:00
Eelco Dolstra 4af88a4c91
Merge pull request #5906 from pennae/primops-optimization
optimize primops and utils by caching more and copying less
2022-01-18 19:43:28 +01:00
Eelco Dolstra 3157028fc1
Merge pull request #5932 from edolstra/remove-shared-strings
Remove shared strings
2022-01-18 11:14:18 +01:00
Eelco Dolstra d62a9390fc Get rid of std::shared_ptr<std::string> and ref<std::string>
These were needed back in the pre-C++11 era because we didn't have
move semantics. But now we do.
2022-01-18 11:12:30 +01:00
Eelco Dolstra 52ee7ec002 StringSource: Use std::string_view 2022-01-17 22:20:23 +01:00
Eelco Dolstra 776eb97a43 serialise.hh: Use std::string_view 2022-01-17 22:20:23 +01:00
Eelco Dolstra fc2443a67c
Merge pull request #5812 from pennae/small-perf-improvements
improve parser performance a bit
2022-01-17 19:49:52 +01:00
pennae 34e3bd10e3 avoid copies of parser input data
when given a string yacc will copy the entire input to a newly allocated
location so that it can add a second terminating NUL byte. since the
parser is a very internal thing to EvalState we can ensure that having
two terminating NUL bytes is always possible without copying, and have
the parser itself merely check that the expected NULs are present.

 # before

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     572.4 ms ±   2.3 ms    [User: 563.4 ms, System: 8.6 ms]
  Range (min … max):   566.9 ms … 579.1 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     381.7 ms ±   1.0 ms    [User: 348.3 ms, System: 33.1 ms]
  Range (min … max):   380.2 ms … 387.7 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.936 s ±  0.005 s    [User: 2.715 s, System: 0.221 s]
  Range (min … max):    2.923 s …  2.946 s    50 runs

 # after

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     571.7 ms ±   2.4 ms    [User: 563.3 ms, System: 8.0 ms]
  Range (min … max):   566.7 ms … 579.7 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     376.6 ms ±   1.0 ms    [User: 345.8 ms, System: 30.5 ms]
  Range (min … max):   374.5 ms … 379.1 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.922 s ±  0.006 s    [User: 2.707 s, System: 0.215 s]
  Range (min … max):    2.906 s …  2.934 s    50 runs
2022-01-13 18:06:15 +01:00
pennae 44c92a1667 use more string_view in utils
there's a couple places that can be easily converted from using strings to using
string_views instead. gives a slight (~1%) boost to system eval.

 # before

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.946 s ±  0.026 s    [User: 2.655 s, System: 0.209 s]
    Range (min … max):    2.905 s …  2.995 s    20 runs

 # after

    Time (mean ± σ):      2.928 s ±  0.024 s    [User: 2.638 s, System: 0.211 s]
    Range (min … max):    2.893 s …  2.970 s    20 runs
2022-01-13 13:51:29 +01:00
Théophane Hufschmitt e61c4bc25a
Merge pull request #5887 from pennae/avoid-streams
avoid std::?stream overhead when it's not helpful
2022-01-12 10:52:40 +01:00
pennae 73fcc40fa4 use boost::lexical_cast for string2*
this avoids one copy from `s` into `str`, and possibly another copy needed to
construct `s` at the call site. lexical_cast is also more efficient in general.
2022-01-12 10:07:21 +01:00
Eelco Dolstra 8a446aff75
Merge pull request #5898 from layus/repair-path-links
Make --repair-path also repair corrupt optimised links
2022-01-11 14:14:44 +01:00
Guillaume Maudoux e9a4abdb5d Make --repair-path also repair corrupt optimised links
There already existed a smoke test for the link content length,
but it appears that there exists some corruptions pernicious enough
to replace the file content with zeros, and keeping the same length.

--repair-path now goes as far as checking the content of the link,
making it true to its name and actually repairing the path for such
coruption cases.
2022-01-11 11:57:45 +01:00
Eelco Dolstra fcddaa4b9b dump(): Use emplace() 2022-01-07 13:23:17 +01:00
Eelco Dolstra e1a0359b59 isInDir() / isDirOrInDir(): Use std::string_view 2022-01-07 13:23:00 +01:00
pennae 26a8b220eb avoid ostream sentries per json string character
we don't have to create an ostream sentry object for every character of a JSON
string we write. format a bunch of characters and flush them to the stream all
at once instead.

this doesn't affect small numbers of string characters, but larger numbers of
total JSON string characters written gain a lot. at 1MB of total string written
we gain almost 30%, at 16MB it's almost a factor of 3x. large numbers of JSON
string characters do occur naturally in a nixos system evaluation to generate
documentation (though this is now somewhat mitigated by caching the largest part
of nixos option docs).

benchmarked with

  hyperfine 'nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) {e})"' --warmup 1 -L e 1,4,256,4096,65536

before:

  Benchmark 1: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 1)"
    Time (mean ± σ):      12.5 ms ±   0.2 ms    [User: 9.2 ms, System: 4.0 ms]
    Range (min … max):    11.9 ms …  13.1 ms    223 runs

  Benchmark 2: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 4)"
    Time (mean ± σ):      12.5 ms ±   0.2 ms    [User: 9.3 ms, System: 3.8 ms]
    Range (min … max):    11.9 ms …  13.2 ms    220 runs

  Benchmark 3: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 256)"
    Time (mean ± σ):      13.2 ms ±   0.3 ms    [User: 9.8 ms, System: 4.0 ms]
    Range (min … max):    12.6 ms …  14.3 ms    205 runs

  Benchmark 4: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 4096)"
    Time (mean ± σ):      24.0 ms ±   0.4 ms    [User: 19.4 ms, System: 5.2 ms]
    Range (min … max):    22.7 ms …  25.8 ms    119 runs

  Benchmark 5: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 65536)"
    Time (mean ± σ):     196.0 ms ±   3.7 ms    [User: 171.2 ms, System: 25.8 ms]
    Range (min … max):   190.6 ms … 201.5 ms    14 runs

after:

  Benchmark 1: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 1)"
    Time (mean ± σ):      12.4 ms ±   0.3 ms    [User: 9.1 ms, System: 4.0 ms]
    Range (min … max):    11.7 ms …  13.3 ms    204 runs

  Benchmark 2: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 4)"
    Time (mean ± σ):      12.4 ms ±   0.2 ms    [User: 9.2 ms, System: 3.9 ms]
    Range (min … max):    11.8 ms …  13.0 ms    214 runs

  Benchmark 3: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 256)"
    Time (mean ± σ):      12.6 ms ±   0.2 ms    [User: 9.5 ms, System: 3.8 ms]
    Range (min … max):    12.1 ms …  13.3 ms    209 runs

  Benchmark 4: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 4096)"
    Time (mean ± σ):      15.9 ms ±   0.2 ms    [User: 11.4 ms, System: 5.1 ms]
    Range (min … max):    15.2 ms …  16.4 ms    171 runs

  Benchmark 5: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 65536)"
    Time (mean ± σ):      69.0 ms ±   0.9 ms    [User: 44.3 ms, System: 25.3 ms]
    Range (min … max):    67.2 ms …  70.9 ms    42 runs
2022-01-07 06:53:47 +01:00
Ben Burdette a47de1ac37 Merge branch 'master' into debug-exploratory-PR 2022-01-03 16:08:28 -07:00
Alexander Bantyev 581f774284
BaseError::calcWhat: take loggerSettings.showTrace into account
Text representation for errors should include the trace if
--show-trace is passed.
2021-12-28 15:53:21 +03:00
Ben Burdette 5954cbf3e9 more cleanup 2021-12-27 18:29:55 -07:00
Ben Burdette 4610e02d04 remove debug code 2021-12-27 18:12:46 -07:00
Ben Burdette bc20e54e00 stack traces basically working 2021-12-22 19:40:08 -07:00
Ben Burdette b4a59a5eec DebugStackTracker class in one place 2021-12-22 15:38:49 -07:00
Eelco Dolstra 7feb741e00
Merge pull request #5821 from edolstra/remove-affinity
Remove CPU locking
2021-12-22 20:31:19 +01:00
Naïm Favier 1da1b2b345
Don't insert spaces when completing attribute paths 2021-12-22 16:17:01 +01:00
Eelco Dolstra 9747ea84b4 Remove CPU locking
This was already accidentally disabled in ba87b08. It also no longer
appears to be beneficial, and in fact slow things down, e.g. when
evaluating a NixOS system configuration:

  elapsed time:       median =      3.8170  mean =      3.8202  stddev =      0.0195  min =      3.7894  max =      3.8600  [rejected, p=0.00000, Δ=0.36929±0.02513]
2021-12-22 15:56:25 +01:00
Eelco Dolstra ec8f24ed3a Ignore EPERM when unsharing FS state
On Docker (but not podman), unshare(CLONE_FS) fails with EPERM. So
let's ignore it and hope nothing bad happens.

Attempted fix for #5777.
2021-12-16 22:02:50 +01:00
Théophane Hufschmitt 6234e1c811
Merge pull request #5702 from baloo/baloo/hide-non-reproducible-settings
reproducibility: hide non-reproducible settings from manual
2021-12-14 09:14:25 +01:00
Yorick 0e0de90b35
extract_archive: fix "Hard-link target '...'" error
Fixes #5741
2021-12-10 17:03:51 +01:00
Yorick be1055f2cc
extract_archive: use copy_pathname instead of set_pathname.
Libarchive documentation mentions that archive_entry_set_pathname
expects us to keep the passed string alive, which we don't.
2021-12-10 17:00:32 +01:00
Eelco Dolstra bc57b3854e
Merge pull request #5695 from obsidiansystems/tidy-logging
Tidy up the logging
2021-12-09 15:31:08 +01:00
Sergei Trofimovich c32a5f4d38 src/libutil/util.hh: fix build on gcc-11
Due to missing <atomic> declaration the build fails as:

    src/libutil/util.hh:350:24: error: no match for 'operator||' (operand types are 'std::atomic<bool>' and 'bool')
      350 |     if (_isInterrupted || (interruptCheck && interruptCheck()))
          |         ~~~~~~~~~~~~~~ ^~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          |         |                                 |
          |         std::atomic<bool>                 bool
2021-12-01 22:08:05 +00:00
Arthur Gautier 21520297da reproducibility: hide non-reproducible settings from manual
Because the manual is generated from default values which are themselves
generated from various sources (cpuid, bios settings (kvm), number of
cores). This commit hides non-reproducible settings from the manual
output.
2021-12-01 17:25:58 +01:00
John Ericson 1f15441103 Tidy up the logging
Use the macros more, so we properly skip work when the log level
excludes. Also log the daemon operation number on the daemon side.
2021-11-30 20:23:13 +00:00
Eelco Dolstra 6e0cbc666b createTempFile(): Mark file as CLOEEXEC
Fixes #5674.
2021-11-29 11:20:50 +01:00
Ben Burdette 64c4ba8f66 Merge branch 'master' into debug-merge 2021-11-25 08:53:59 -07:00
Alex Shabalin 2970ca18bf Fix a minor data race with _isInterrupted 2021-11-24 14:50:08 +01:00
Jan Tojnar e96faadcd6 Fix XDG_CONFIG_DIRS fallback
According to XDG Base Directory Specification,
it should fall back to /etc/xdg when the env var is not present.
2021-11-17 14:31:15 +01:00
Eelco Dolstra 8c93a481af Ignore errors unsharing/restoring the mount namespace
This prevents Nix from barfing when run in a container where it
doesn't have the appropriate privileges.
2021-11-16 14:48:44 +01:00
Eelco Dolstra 7a71621b7c Merge branch 'fix-writable-shell' of https://github.com/yorickvP/nix 2021-11-08 21:12:51 +01:00
Eelco Dolstra 6e30d9b69f Merge branch 'master' of https://github.com/alekswn/nix 2021-10-29 14:42:26 +02:00
Eelco Dolstra a7d4f3411e Merge remote-tracking branch 'origin/master' into non-blocking-gc 2021-10-28 14:56:55 +02:00
Eelco Dolstra 13a7a24ba5 Style 2021-10-27 13:02:37 +02:00
Eelco Dolstra 0d9e050ba7 parseExperimentalFeature(): Initialize atomically 2021-10-26 14:29:48 +02:00
regnat af99941279 Make experimental-features a proper type
Rather than having them plain strings scattered through the whole
codebase, create an enum containing all the known experimental features.

This means that
- Nix can now `warn` when an unkwown experimental feature is passed
  (making it much nicer to spot typos and spot deprecated features)
- It’s now easy to remove a feature altogether (once the feature isn’t
  experimental anymore or is dropped) by just removing the field for the
  enum and letting the compiler point us to all the now invalid usages
  of it.
2021-10-26 07:02:31 +02:00
Alexey Novikov 64a3b045c1 Fix error detection in 'base64Decode()'
Fixed a bug in initialization of 'base64DecodeChars' variable.
Currently decoder do not fail on invalid Base64 strings.
Added test-case to verify the fix.

Also have made 'base64DecodeChars' to be computed at compile time.
And added a test case to encode/decode string with non-printable charactes.
2021-10-17 12:45:26 +04:00
Yorick fcb8af550f
Restore parent mount namespace in restoreProcessContext
This ensures any started processes can't write to /nix/store (except
during builds). This partially reverts 01d07b1e, which happened because
of #2646.

The problem was only happening after nix downloads anything, causing
me to suspect the download thread. The problem turns out to be:
"A  process  can't  join a new mount namespace if it is sharing
filesystem-related attributes with another process", in this case this
process is the curl thread.

Ideally, we might kill it before spawning the shell process, but it's
inside a static variable in the getFileTransfer() function. So
instead, stop it from sharing FS state using unshare(). A strategy
such as the one from #5057 (single-threaded chroot helper binary) is
also very much on the table.

Fixes #4337.
2021-10-15 16:25:49 +02:00
Eelco Dolstra eab934cb2a Make the canReachRoots() traversal non-recursive 2021-10-14 12:34:32 +02:00
kvtb eae29b0385
fix build with gcc11 2021-10-13 18:03:33 +00:00
Eelco Dolstra dced45f146 strcpy -> memcpy
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
2021-10-13 12:12:44 +02:00
Eelco Dolstra 262520fcfe Use a thread per connection 2021-10-13 12:12:44 +02:00
Eelco Dolstra 8614cf1334 Non-blocking garbage collector
The garbage collector no longer blocks other processes from
adding/building store paths or adding GC roots. To prevent the
collector from deleting store paths just added by another process,
processes need to connect to the garbage collector via a Unix domain
socket to register new temporary roots.
2021-10-13 12:12:44 +02:00
Eelco Dolstra 0fac86fd6f Style tweaks 2021-10-13 11:00:10 +02:00
Tom Bereknyei 03bb8f84e0 Add compression level for NARs
Based off on @dtzWill's #2276
2021-10-12 02:14:36 -04:00
Eelco Dolstra c6718a9d95 Don't reset the logger in a vfork
9c766a40cb broke logging from the
daemon, because commonChildInit is called when starting the build hook
in a vfork, so it ends up resetting the parent's logger. So don't
vfork.

It might be best to get rid of vfork altogether, but that may cause
problems, e.g. when we call an external program like git from the
evaluator.
2021-10-06 13:54:59 +02:00
Eelco Dolstra e4a5d64a81 Show failing PID 2021-10-05 13:19:55 +02:00
Eelco Dolstra b299560872 Typo 2021-10-05 11:24:09 +02:00
Eelco Dolstra 43d4d75e22 Connect/bind Unix domain sockets in a child process
In the child process, we can do a chdir() and avoid the problem of the
path not fitting into sockaddr_un.
2021-10-05 10:44:59 +02:00
kvtb 638c73776a
fix creation of NAR files >4GB on 32-bit platforms
`size_t` is 32-bit on 32-bit platforms, while the file size can be bigger
2021-10-02 21:04:01 +00:00
Eelco Dolstra 9c766a40cb Fix 'error: reading a line: Input/output error' in startBuilder()
With -vvvv, the ProgressBar was polluting the stderr of the child,
messing up its \2 message to the parent.
2021-09-27 14:44:21 +02:00
Eelco Dolstra d8c10028d9 Make setDefault() typed 2021-09-22 14:15:35 +02:00
Eelco Dolstra 817562e694 Add "nix profile rollback" command 2021-09-14 19:32:33 +02:00
Eelco Dolstra a4ae601b44
Merge pull request #5245 from edolstra/warnings
Change warning messages from yellow to magenta
2021-09-14 11:46:34 +02:00
Eelco Dolstra 16d4922dd2
Merge pull request #5240 from edolstra/builtin-help
nix --help: Display help using lowdown instead of man
2021-09-14 11:31:23 +02:00
Eelco Dolstra 58b5036c54 Change warnings from yellow to magenta
This matches gcc and clang.
2021-09-14 10:42:29 +02:00
Eelco Dolstra 4ffda0af7c ANSI_YELLOW -> ANSI_WARNING 2021-09-14 10:42:29 +02:00
Eelco Dolstra 76e368a3b4 Fix macOS build 2021-09-14 10:07:31 +02:00
Eelco Dolstra 1ba993d07c Fix clang warning 2021-09-14 08:15:33 +02:00
Eelco Dolstra 4ed66735b6 RunOptions: Use designated initializers
Also get rid of _killStderr because it wasn't actually checked
anywhere.
2021-09-13 23:31:04 +02:00
Ben Burdette 176911102c printEnvPosChain 2021-09-13 11:57:25 -06:00
Eelco Dolstra 49a932fb18 nix --help: Display help using lowdown instead of man
Fixes #4476.
Fixes #5231.
2021-09-13 14:45:21 +02:00
Naïm Favier 7f0d177ce7
Add missing include in util.cc 2021-09-08 12:20:08 +02:00
Eelco Dolstra 50a35860ee TarArchive: Small refactoring 2021-08-30 17:02:51 +02:00
Eelco Dolstra c7a7652725 Don't segfault if archive_entry_pathname() returns null
Issues #4499.
2021-08-30 17:02:39 +02:00
Eelco Dolstra eda0fee160
Merge pull request #5175 from Pamplemousse/make
Don't overwrite user provided `lib*_LDFLAGS`
2021-08-30 12:44:29 +02:00
Robert Hensing f10465774f Force all Pos* to be non-null
This fixes a class of crashes and introduces ptr<T> to make the
code robust against this failure mode going forward.

Thanks regnat for the idea of a ref<T> without overhead!

Closes #4895
Closes #4893
Closes #5127
Closes #5113
2021-08-29 18:11:58 +02:00
Pamplemousse a4c6d319a8 Don't overwrite user provided lib*_LDFLAGS
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-25 08:59:19 -07:00
Yaroslav Bolyukin bbb3bcb165
Replace invalid characters in json logger
Fixes #5159

Signed-off-by: Yaroslav Bolyukin <iam@lach.pw>
2021-08-22 00:26:22 +03:00
Pamplemousse 0da416f820 15f4d4f follow up
* libstore: `bz2` should not be linked
  * libutil: `zlib.h` should not be included

Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-10 16:04:53 -07:00
Andreas Rammhold b9c9c25766
libutil: initialize the base64 decode array only once
Previously, despite having a boolean that tracked initialization, the
decode characters have been "calculated" every single time a base64
string was being decoded.

With this change we only initialize the decode array once in a
thread-safe manner.
2021-07-30 21:07:32 +02:00
Alyssa Ross 369ed71858
libutil: use uniform initialization in _deletePath
Otherwise I get a compiler error when building for NetBSD:

src/libutil/util.cc: In function 'void nix::_deletePath(const Path&, uint64_t&)':
src/libutil/util.cc:438:17: error: base operand of '->' is not a pointer
  438 |     AutoCloseFD dirfd(open(dir.c_str(), O_RDONLY));
      |                 ^~~~~
src/libutil/util.cc:439:10: error: 'dirfd' was not declared in this scope
  439 |     if (!dirfd) {
      |          ^~~~~
src/libutil/util.cc:444:17: error: 'dirfd' was not declared in this scope
  444 |     _deletePath(dirfd.get(), path, bytesFreed);
      |                 ^~~~~
2021-07-24 09:19:48 +00:00
Eelco Dolstra 140ccf1368 deletePath(): Return ENFILE instead of EBADF when out of file descriptors
Also remove an erroneous comment.
2021-07-20 20:59:45 +02:00
regnat 9b1f3cbc13 Forward the whole Nix config to the post-build-hook
Fill `NIX_CONFIG` with the value of the current Nix configuration before
calling the post-build-hook.
That way the whole configuration (including the possible
`experimental-features`, a possibly `--store` option or whatever) will
be made available to the hook
2021-07-15 18:41:56 +02:00
Eelco Dolstra 07790fdddf ref: Add equality operators 2021-07-09 14:03:48 +02:00
Domen Kožar f9d72855ae
Merge pull request #4967 from Pamplemousse/specific_errors
Prefer to throw specific errors
2021-07-03 10:34:56 +02:00
Yestin L. Harrison 0fe84bef72 Add $NO_COLOR check to ANSI escape conditions 2021-07-02 09:33:54 -06:00
Yestin L. Harrison 20cce079f2 Respect TERM=dumb more consistently 2021-07-01 18:19:01 -06:00
Pamplemousse 4a7a8b87cd Prefer to throw specific errors
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-01 11:09:31 -07:00
regnat a22755721b Recursively substitute the realisations
Make sure that whenever we substitute a realisation, we also substitute
its entire closure
2021-05-26 18:44:17 +02:00
regnat 184558834a Extract a generic computeClosure function
Move the `closure` logic of `computeFSClosure` to its own (templated) function.

This doesn’t bring much by itself (except for the ability to properly
test the “closure” functionality independently from the rest), but it
allows reusing it (in particular for the realisations which will require
a very similar closure computation)
2021-05-19 11:44:58 +02:00
Matthew Bauer 3d90ab9345 Fix extra slash in canonPath output
When you have a symlink like:

  /tmp -> ./private/tmp

you need to resolve ./private/tmp relative to /tmp’s dir: ‘/’. Unlike
any other path output by dirOf, / ends with a slash. We don’t want
trailing slashes here since we will append another slash in the next
comoponent, so clear s like we would if it was a symlink to an absoute
path.

This should fix at least part of the issue in
https://github.com/NixOS/nix/issues/4822, will need confirmation that
it actually fixes the problem to close though.

Introduced in f3f228700a.
2021-05-18 16:38:55 -05:00
Eelco Dolstra 5385755227 Remove makeDecompressionSource()
This function doesn't support all compression methods (i.e. 'none' and
'br') so it shouldn't be exposed.

Also restore the original decompress() as a wrapper around
makeDecompressionSink().
2021-04-22 10:23:20 +02:00
Luke Granger-Brown 97dde3cdd9 libutil: allow decompression with none/empty method
The S3 store relies on the ability to be able to decompress things with
an empty method, because it just passes the value of the Content-Encoding
directly to decompress.

If the file is not compressed, then this will cause the compression
routine to get confused.

This caused NixOS/nixpkgs#120120.
2021-04-22 02:33:59 +00:00
Eelco Dolstra 8d651a1f68 Fix brotli compression of files > 128 KiB
This has been broken since faa31f4084.
2021-04-20 22:54:49 +02:00
Eelco Dolstra 3ee0ecdda0
Merge pull request #4679 from ony/feature/one-pass-canon-path
Optimize canonPath to resolve relative symlinks in one pass
2021-04-15 14:11:51 +02:00
Eelco Dolstra 15f4d4fd43 Drop libbz2 / zlib / lzma dependency + style fixes 2021-04-15 13:55:22 +02:00
Eelco Dolstra 6fb7582413 Merge branch 'libarchive-decompress' of https://github.com/serokell/nix 2021-04-15 13:39:04 +02:00
Eelco Dolstra 906adadacd Restore stack size in child processes
Fixes #4673.
2021-04-07 13:40:13 +02:00
Eelco Dolstra 9b9e703df4 restoreSignals() + restoreAffinity() -> restoreProcessContext() 2021-04-07 13:10:02 +02:00
Eelco Dolstra 8a29052cb2 PathSubstitutionGoal: Clean up pipe
If there were many top-level goals (which are not destroyed until the
very end), commands like

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' \
    /run/current-system --no-check-sigs --substitute-on-destination

could fail with "Too many open files". So now we do some explicit
cleanup from amDone(). It would be cleaner to separate goals from
their temporary internal state, but that would be a bigger refactor.
2021-04-07 12:21:31 +02:00
Mykola Orliuk f3f228700a canonPath in one pass 2021-03-31 04:58:49 +02:00
Eelco Dolstra 4638bcfb2c Fix some typos
Fixes #4671.
2021-03-26 16:14:38 +01:00
Maximilian Bosch ddcef3bcbb
Fix Nix to properly work with stores using a scoped IPv6 address
According to RFC4007[1], IPv6 addresses can have a so-called zone_id
separated from the actual address with `%` as delimiter. In contrast to
Nix 2.3, the version on `master` doesn't recognize it as such:

    $ nix ping-store --store ssh://root@fe80::1%18 --experimental-features nix-command
    warning: 'ping-store' is a deprecated alias for 'store ping'
    error: --- Error ----------------------------------------------------------------- nix
    don't know how to open Nix store 'ssh://root@fe80::1%18'

I modified the IPv6 match-regex accordingly to optionally detect this
part of the address. As we don't seem to do anything special with it, I
decided to leave it as part of the URL for now.

Fixes #4490

[1] https://tools.ietf.org/html/rfc4007
2021-03-16 19:14:42 +01:00
Yorick 8a0c00b856
Use libarchive for all compression 2021-03-10 22:34:29 +01:00
Eelco Dolstra 453c3a603f nix flake update: Recreate the lock file
This is probably what most people expect it to do. Fixes #3781.

There is a new command 'nix flake lock' that has the old behaviour of
'nix flake update', i.e. it just adds missing lock file entries unless
overriden using --update-input.
2021-02-26 14:55:54 +01:00
Eelco Dolstra 5edab777d1
Merge pull request #4530 from alyssais/kill
libutil: EPERM from kill(-1, ...) is fine
2021-02-26 12:19:16 +01:00
Shea Levy f6c5b05488
Respect command registrations in plugins. 2021-02-24 08:25:45 -05:00
Shea Levy 98d1b64400
Initialize plugins after handling initial command line flags
This is technically a breaking change, since attempting to set plugin
files after the first non-flag argument will now throw an error. This
is acceptable given the relative lack of stability in a plugin
interface and the need to tie the knot somewhere once plugins can
actually define new subcommands.
2021-02-24 08:22:17 -05:00
Daniël de Kok 2de232d2b3 Add x86_64 compute levels as additional system types
When performing distributed builds of machine learning packages, it
would be nice if builders without the required SIMD instructions can
be excluded as build nodes.

Since x86_64 has accumulated a large number of different instruction
set extensions, listing all possible extensions would be unwieldy.
AMD, Intel, Red Hat, and SUSE have recently defined four different
microarchitecture levels that are now part of the x86-64 psABI
supplement and will be used in glibc 2.33:

https://gitlab.com/x86-psABIs/x86-64-ABI
https://lwn.net/Articles/844831/

This change uses libcpuid to detect CPU features and then uses them to
add the supported x86_64 levels to the additional system types. For
example on a Ryzen 3700X:

$ ~/aps/bin/nix -vv --version | grep "Additional system"
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
2021-02-22 09:11:15 +01:00
Eelco Dolstra 37352aa7e1 Support --no-net for backwards compatibility 2021-02-07 20:44:56 +01:00
Alyssa Ross 7c112351d9
libutil: EPERM from kill(-1, ...) is fine
I tested a trivial program that called kill(-1, SIGKILL), which was
run as the only process for an unpriveleged user, on Linux and
FreeBSD.  On Linux, kill reported success, while on FreeBSD it failed
with EPERM.

POSIX says:

> If pid is -1, sig shall be sent to all processes (excluding an
> unspecified set of system processes) for which the process has
> permission to send that signal.

and

> The kill() function is successful if the process has permission to
> send sig to any of the processes specified by pid.  If kill() fails,
> no signal shall be sent.

and

> [EPERM]
>     The process does not have permission to send the signal to any
>     receiving process.

My reading of this is that kill(-1, ...) may fail with EPERM when
there are no other processes to kill (since the current process is
ignored).  Since kill(-1, ...) only attempts to kill processes the
user has permission to kill, it can't mean that we tried to do
something we didn't have permission to kill, so it should be fine to
interpret EPERM the same as success here for any POSIX-compliant
system.

This fixes an issue that Mic92 encountered[1] when he tried to review a
Nixpkgs PR on FreeBSD.

[1]: https://github.com/NixOS/nixpkgs/pull/81459#issuecomment-606073668
2021-02-07 13:56:50 +00:00
regnat d2091af231 Move the GENERATE_CMP macro to its own file
Despite being an ugly hack, it can probably be useful in a couple extra
places
2021-02-05 11:42:33 +01:00
Eelco Dolstra 8e758d402b Remove mkFlag() 2021-01-27 12:06:03 +01:00
Eelco Dolstra 36c4d6f592 Group common options 2021-01-25 19:03:13 +01:00
Eelco Dolstra 807d963ee8 Group subcommands by category 2021-01-25 18:19:32 +01:00
Eelco Dolstra b159d23800 Make '--help' do the same as 'help' (i.e. show a manpage) 2021-01-25 14:38:15 +01:00
Eelco Dolstra 488a826842
Merge pull request #4467 from edolstra/error-formatting
Improve error formatting
2021-01-25 12:50:57 +01:00
Eelco Dolstra b7bfc7ee52
Add FIXME 2021-01-22 12:36:50 +01:00
Eelco Dolstra 55849e153e Change error position formatting
It's now

  at /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix:7:7:

instead of

  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

The new format is more standard and clickable.
2021-01-21 11:02:09 +01:00
Eelco Dolstra 40608342cb Remove trailing whitespace 2021-01-21 11:02:09 +01:00
Eelco Dolstra 8d4268d190 Improve error formatting
Changes:

* The divider lines are gone. These were in practice a bit confusing,
  in particular with --show-trace or --keep-going, since then there
  were multiple lines, suggesting a start/end which wasn't the case.

* Instead, multi-line error messages are now indented to align with
  the prefix (e.g. "error: ").

* The 'description' field is gone since we weren't really using it.

* 'hint' is renamed to 'msg' since it really wasn't a hint.

* The error is now printed *before* the location info.

* The 'name' field is no longer printed since most of the time it
  wasn't very useful since it was just the name of the exception (like
  EvalError). Ideally in the future this would be a unique, easily
  googleable error ID (like rustc).

* "trace:" is now just "…". This assumes error contexts start with
  something like "while doing X".

Example before:

  error: --- AssertionError ---------------------------------------------------------------------------------------- nix
  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

       6|
       7|   x = assert false; 1;
        |       ^
       8|

  assertion 'false' failed
  ----------------------------------------------------- show-trace -----------------------------------------------------
  trace: while evaluating the attribute 'x' of the derivation 'hello-2.10'
  at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

     191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
     192|           name = "${attrs.pname}-${attrs.version}";
        |           ^
     193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {

Example after:

  error: assertion 'false' failed

         at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

              6|
              7|   x = assert false; 1;
               |       ^
              8|

         … while evaluating the attribute 'x' of the derivation 'hello-2.10'

         at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix

            191|         // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) {
            192|           name = "${attrs.pname}-${attrs.version}";
               |           ^
            193|         } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2021-01-21 11:02:09 +01:00
Eelco Dolstra 1bbc66f865 Merge branch 'slashes-in-github-branches' of https://github.com/Ma27/nix 2021-01-18 12:55:06 +01:00
Domen Kožar 00f99fdfe6
Merge pull request #4240 from bburdette/2259-error-message
2259 error message - "auto-call" error
2021-01-15 18:26:21 +00:00
Eelco Dolstra e21aee58f6 Fix tests 2021-01-08 14:17:06 +01:00
Eelco Dolstra 17beae299d Support binary unit prefixes in command line arguments 2021-01-08 12:51:19 +01:00
Eelco Dolstra 6548b89cc4 string2Int(): Return std::optional 2021-01-08 12:22:21 +01:00
Eelco Dolstra 29a445840a Remove unused mkFlag1 2021-01-08 11:42:44 +01:00
Eelco Dolstra 1d4954e73e Remove mkFlag integer specialisation 2021-01-08 11:40:36 +01:00
Eelco Dolstra 48a9be2aab Remove mkIntFlag 2021-01-08 10:44:55 +01:00
Eelco Dolstra a93916b190
Merge pull request #4336 from NixOS/manpages
Documentation for nix subcommands
2020-12-23 21:10:32 +01:00
Eelco Dolstra 1047cb1e53
Command: Remove examples() 2020-12-23 18:26:40 +01:00
Eelco Dolstra e27044216b Fix tests 2020-12-22 16:23:57 +01:00
Eelco Dolstra 724b7f4fb6 Don't log from inside the logger
This deadlocks ProgressBar, e.g.

  # nix run --impure --no-substitute --store '/tmp/nix2?store=/foo' --expr 'derivation { builder = /nix/store/zi90rxslsm4mlr46l2xws1rm94g7pk8p-busybox-1.31.1-x86_64-unknown-linux-musl/bin/busybox; }'

leads to

  Thread 1 (Thread 0x7ffff6126e80 (LWP 12250)):
  #0  0x00007ffff7215d62 in __lll_lock_wait () from /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libpthread.so.0
  #1  0x00007ffff720e721 in pthread_mutex_lock () from /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libpthread.so.0
  #2  0x00007ffff7ad17fa in __gthread_mutex_lock (__mutex=0x6c5448) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:749
  #3  std::mutex::lock (this=0x6c5448) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/std_mutex.h:100
  #4  std::unique_lock<std::mutex>::lock (this=0x7fffffff09a8, this=0x7fffffff09a8) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/unique_lock.h:141
  #5  std::unique_lock<std::mutex>::unique_lock (__m=..., this=0x7fffffff09a8) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/unique_lock.h:71
  #6  nix::Sync<nix::ProgressBar::State, std::mutex>::Lock::Lock (s=0x6c5448, this=0x7fffffff09a0) at src/libutil/sync.hh:45
  #7  nix::Sync<nix::ProgressBar::State, std::mutex>::lock (this=0x6c5448) at src/libutil/sync.hh:85
  #8  nix::ProgressBar::logEI (this=0x6c5440, ei=...) at src/libmain/progress-bar.cc:131
  #9  0x00007ffff7608cfd in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x6c5440) at src/libutil/logging.hh:88
  #10 nix::getCodeLines (errPos=...) at src/libutil/error.cc:66
  #11 0x00007ffff76073f2 in nix::showErrorInfo (out=..., einfo=..., showTrace=<optimized out>) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/optional:897
  #12 0x00007ffff7ad19e7 in nix::ProgressBar::logEI (this=0x6c5440, ei=...) at src/libmain/progress-bar.cc:134
  #13 0x00007ffff7ab9d10 in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x6c5440) at src/libutil/logging.hh:88
  #14 nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) (programName="/home/eelco/Dev/nix/outputs/out/bin/nix", fun=...) at src/libmain/shared.cc:328
  #15 0x000000000046226b in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/ext/new_allocator.h:80
2020-12-22 11:15:29 +01:00
Eelco Dolstra 346baec783
Move doc() to Args 2020-12-21 13:32:23 +01:00
Eelco Dolstra a8f533b664 Add lvlNotice log level
This is like syslog's LOG_NOTICE: "normal, but significant,
condition".
2020-12-10 16:41:24 +01:00
Maximilian Bosch 93a8a005de
libstore/openStore: fix stores with IPv6 addresses
In `nixStable` (2.3.7 to be precise) it's possible to connect to stores
using an IPv6 address:

  nix ping-store --store ssh://root@2001:db8::1

This is also useful for `nixops(1)` where you could specify an IPv6
address in `deployment.targetHost`.

However, this behavior is broken on `nixUnstable` and fails with the
following error:

  $ nix store ping --store ssh://root@2001:db8::1
  don't know how to open Nix store 'ssh://root@2001:db8::1'

This happened because `openStore` from `libstore` uses the `parseURL`
function from `libfetchers` which expects a valid URL as defined in
RFC2732. However, this is unsupported by `ssh(1)`:

  $ nix store ping --store 'ssh://root@[2001:db8::1]'
  cannot connect to 'root@[2001:db8::1]'

This patch now allows both ways of specifying a store (`root@2001:db8::1`) and
also `root@[2001:db8::1]` since the latter one is useful to pass query
parameters to the remote store.

In order to achieve this, the following changes were made:

* The URL regex from `url-parts.hh` now allows an IPv6 address in the
  form `2001:db8::1` and also `[2001:db8::1]`.

* In `libstore`, a new function named `extractConnStr` ensures that a
  proper URL is passed to e.g. `ssh(1)`:

  * If a URL looks like either `[2001:db8::1]` or `root@[2001:db8::1]`,
    the brackets will be removed using a regex. No additional validation
    is done here as only strings parsed by `parseURL` are expected.

  * In any other case, the string will be left untouched.

* The rules above only apply for `LegacySSHStore` and `SSHStore` (a.k.a
  `ssh://` and `ssh-ng://`).

Unresolved questions:

* I'm not really sure whether we want to allow both variants of IPv6
  addresses in the URL parser. However it should be noted that both seem
  to be possible according to RFC2732:

  > This document incudes an update to the generic syntax for Uniform
  > Resource Identifiers defined in RFC 2396 [URL].  It defines a syntax
  > for IPv6 addresses and allows the use of "[" and "]" within a URI
  > explicitly for this reserved purpose.

* Currently, it's not supported to specify a port number behind the
  hostname, however it seems as this is not really supported by the URL
  parser. Hence, this is probably out of scope here.
2020-12-09 12:23:29 +01:00
Eelco Dolstra af373c2ece Add deprecated aliases for renamed commands 2020-12-03 22:45:44 +01:00
regnat a8a96dbaf8 Add forgotten override annotation 2020-12-02 14:23:38 +01:00
Eelco Dolstra 1b79b5b983 read(): Use char * instead of unsigned char *
This gets rid of some pointless casts.
2020-12-02 14:17:27 +01:00
Eelco Dolstra faa31f4084 Sink: Use std::string_view 2020-12-02 14:17:27 +01:00
Eelco Dolstra aa68486112 writeFull/writeFile: Use std::string_view 2020-12-02 14:17:27 +01:00
Eelco Dolstra 88798613ee replaceStrings(): Use std::string_view 2020-12-01 13:45:43 +01:00
Eelco Dolstra c0d1354b7d Macro hygiene 2020-12-01 13:45:06 +01:00
Eelco Dolstra e224c16d28 Macro hygiene 2020-12-01 13:43:33 +01:00
Dominique Martinet 1fd13d67e8 archive: disable preallocate-contents by default
using fallocate() to preallocate files space does more harm than good:
 - breaks compression on btrfs
 - has been called "not the right thing to do" by xfs developers
(because delayed allocation that most filesystems implement leads to smarter
allocation than what the filesystem needs to do if we upfront fallocate files)
2020-11-26 14:26:57 +01:00