Commit graph

9740 commits

Author SHA1 Message Date
Nikodem Rabuliński 5d3910330d Show message about --update-input being replaced by nix flake update
Fixes: lix-project/lix#283

Change-Id: I6ee23874cb09f51d788521273076a25ba8764859
2024-06-03 21:50:33 +00:00
Nikodem Rabuliński cc3674ea93 Accept multiple arguments to nix flake update
Fixes: lix-project/lix#194

Change-Id: Ia7bd4f7640384be9827dbb7e2c594f0aa5f1aff8
2024-06-03 21:50:33 +00:00
Qyriad c55e93ca23 Revert "nix3: always use the same verbosity default (info)"
This reverts commit d0390b5cf2.

Other parts of the codebase will need to be adjusted in response to a
default verbosity change. Let's just push this to after 2.90.

Fixes #362.
Fixes #367.

Change-Id: I04648473579146851bda41d764adc1ef954c355d
2024-06-01 18:29:19 -06:00
Qyriad d374a9908f Merge "build: fix static linking with a hack" into main 2024-06-01 19:17:13 +00:00
raito b8cb7abcf0 chore: rebrand Nix to Lix when it makes sense
Here's my guide so far:

$ rg '((?!(recursive).*) Nix
(?!(daemon|store|expression|Rocks!|Packages|language|derivation|archive|account|user|sandbox|flake).*))'
-g '!doc/' --pcre2

All items from this query have been tackled. For the documentation side:
that's for lix-project/lix#162.

Additionally, all remaining references to github.com/NixOS/nix which
were not relevant were also replaced.

Fixes: lix-project/lix#148.
Fixes: lix-project/lix#162.
Change-Id: Ib3451fae5cb8ab8cd9ac9e4e4551284ee6794545
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-06-01 20:31:24 +02:00
Linus Heckemann 5312e60be6 Merge "libfetchers: allow fetching gitlab refs with >1 commit" into main 2024-06-01 09:54:11 +00:00
Qyriad e54d4c9381 build: fix static linking with a hack
This causes libstore, libexpr, libfetchers, and libutil to be linked
with -Wl,--whole-archive to executables, when building statically.

libstore for the store backends, libexpr for the primops, libfetchers
for the fetcher backends I assume(?), and libutil for the nix::logger
initializer (which notably shows in pre-main constructors when HOME is
not owned by the user. cursed.).

This workaround should be removed when #359 is fixed.

Fixes #306.

Change-Id: Ie9ef0154e09a6ed97920ee8ab23810ca5e2de84c
2024-05-31 21:47:16 -06:00
jade c7ca87461d Merge "build-remote: truncate+hash store URI used in lockfile paths" into main 2024-05-31 19:22:32 +00:00
jade 7081889faa Merge "truncate WAL files on exit" into main 2024-05-31 19:21:30 +00:00
jade adedac70fa Merge changes Ifcb0d310,I664366b8,Ibe7cf546 into main
* changes:
  gitignore: delete 90% of it
  build-time: remove 20% more by PCH'ing C++ stdlib
  shellHook: make it actually run
2024-05-31 19:19:29 +00:00
Linus Heckemann 82de36f77a libfetchers: allow fetching gitlab refs with >1 commit
Change-Id: I945c4c5512def9eff728bb67fe3c03ae17f99d6d
2024-05-31 21:12:04 +02:00
jade a75d7a5777 Merge "libutil: fix args assert being thrown on Darwin in nix-eval-jobs" into main 2024-05-31 18:57:13 +00:00
annalee 713cd7e9e7 truncate WAL files on exit
Fix for https://github.com/NixOS/nix/issues/10300

18a2620273  enabled persistent WAL files that will never get truncated. to fix this, journal_size_limit is set to 2^40, which results in the WAL files being truncated to 0 on exit, as well as limiting the WAL files to 2^40 bytes following a checkpoint.

this aligns lix with the nix change: https://github.com/NixOS/nix/pull/10301

https://www.sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpersistwal
https://www.sqlite.org/pragma.html#pragma_journal_size_limit
ed517a7082/src/wal.c (L2518)

PR-Link: https://github.com/lix-project/lix/pull/9

Co-Authored-By: paparodeo <170618376+paparodeo@users.noreply.github.com>
Change-Id: I90ec1a467c92c582ff8c07dd363a4cf789782214
2024-05-31 12:22:15 +00:00
Lunaphied d4b7e6baca build-remote: truncate+hash store URI used in lockfile paths
Fixes: lix-project/lix#157
Fixes: lix-project/lix#221

Previously the entire escaped store URI was included. This would cause
build failures if a very long or deeply nested path was being used in
the store.

Now, we use the first 48 characters of the URL (escaped), then 16 bytes
of hash of the entire URL. This should never collide and limits the
length of the file name to a bit over 64, which is fine.

Change-Id: Ic1ba690a94e83749567c2c29460b8d1bcf2ac413
2024-05-31 12:18:24 +00:00
jade ac78c1dcd5 libutil: fix args assert being thrown on Darwin in nix-eval-jobs
This is because a dynamic_cast<nix::RootArgs *> of a (n-e-j) MyArgs
returns nullptr even though MyArgs has virtual nix::RootArgs as a
parent.

class MyArgs : virtual public nix::MixEvalArgs,
               virtual public nix::MixCommonArgs,
               virtual nix::RootArgs { ... };

So this should work right?? But it does not. We found out that it's
caused by -fvisibility=hidden in n-e-j, but honestly this code was bad
anyway.

The trivial solution is to simply stop relying on RTTI working properly
here, which is probably better OO architecture anyway. However, I am not
100% confident *this* is sound, since we have this horrible hierarchy:

      Args (defines getRoot)
     /        |           \
RootArgs  MixCommonArgs  MixEvalArgs
(overrides)

I am not confident that this is guaranteed to resolve from Args always
in the case of this override.

Assertion failed: (res), function getRoot, file src/libutil/args.cc, line 67.
6MyArgsProcess 60503 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #4: 0x0000000100b1a41c liblixutil.dylib`nix::Args::processArgs(std::__1::list<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, bool) [inlined] nix::Args::getRoot(this=0x00000001000d0688) at args.cc:67:5 [opt]
   64       std::cout << typeid(*p).name();
   65
   66       auto * res = dynamic_cast<RootArgs *>(p);
-> 67       assert(res);
   68       return *res;
   69   }
   70
Target 0: (nix-eval-jobs) stopped.

(lldb) p this
(MyArgs *) 0x00000001000d0688
(lldb) p *this
(nix::Args) {
  longFlags = size=180  { ... }
  shortFlags = size=4  { ... }
  expectedArgs = size=1  { ... }
  processedArgs = size=0 {}
  hiddenCategories = size=1 {
    [0] = "Options to override configuration settings"
  }
  parent = nullptr
}

We also found that if we did this:
class [[gnu::visibility("default")]] RootArgs : virtual public Args

it would work properly (???!). This is of course, very strange, because
objdump -Ct output on liblixexpr.dylib is identical both with and
without it.

Possibly related: https://www.qt.io/blog/quality-assurance/one-way-dynamic_cast-across-library-boundaries-can-fail-and-how-to-fix-it

Fixes: lix-project/nix-eval-jobs#2
Change-Id: I6b9ed968ed56420a9c4d2dffd18999d78c2761bd
2024-05-31 12:17:06 +00:00
alois31 cf756fdf3c libstore/build: copy ca-certificates too
In b469c6509b, the ca-certificates file was
missed. It should be copied too so that we don't end up bind-mounting a broken
symlink.

Change-Id: Ic9b292d602eb94b0e78f77f2a27a19d24665783c
2024-05-31 07:54:18 +00:00
jade 0f99ed43f1 build-time: remove 20% more by PCH'ing C++ stdlib
It seems like someone implemented precompiled headers a long time ago
and then it never got ported to meson or maybe didn't work at all.

This is, however, blessedly easy to simply implement. I went looking for
`#define` that could affect the result of precompiling the headers, and
as far as I can tell we aren't doing any of that, so this should truly
just be free build time savings.

Previous state:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

New state:
**** Time summary:
Compilation (567 times):
  Parsing (frontend):         1123.0 s
  Codegen & opts (backend):   1078.1 s

I wonder if the "regression" in codegen time is just doing the PCH
operation a few times, because meson does it per-target.

Change-Id: I664366b8069bab4851308b3a7571bea97ac64022
2024-05-30 21:54:21 +00:00
Linus Heckemann 3df013597d libfetchers: handle nonexistent refs in GitLab repos more gracefully
Before:

$ nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: [json.exception.type_error.302] type must be string, but is null

After:

$ outputs/out/bin/nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error:
       … while updating the lock file of flake 'git+file:///home/linus/projects/lix?ref=refs/heads/fix-gitlab-nonexistent&rev=915f16a619a36237a099b9aa9afed6d14ff613b4'

       … while updating the flake input 'nixpkgs'

       … while fetching the input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'

       error: No commits returned by GitLab API -- does the ref really exist?

Change-Id: Id9bc79d98348500e152ed519bb3ac79a3d15c38d
2024-05-30 21:53:51 +00:00
jade 18aa3e1d57 Merge "Remove 100s of CPU time (10%) from build times (1465s -> 1302s)" into main 2024-05-30 14:57:37 +00:00
jade 53d40888ff Merge "unix-domain-socket.cc: add comment explaining why bindConnectProcHelper" into main 2024-05-30 14:56:57 +00:00
Ilya K da95bf8c82 libstore/filetransfer: remove debug print
foo.

Change-Id: I7d7db22f68046d2ecf3b594b4ee6fd9c9dac4be1
2024-05-30 16:42:45 +03:00
Jörg Thalheim 194b6cc611 docs: expand importNative/exec example (#10803)
Co-authored-by: Qyriad <qyriad@qyriad.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
(cherry picked from commit 5786e1ae7c300b3c7434e7df99b41f180dc42e37)
Change-Id: I16b408ba7c70dca985c05c71bf6195fe9f0b5841
2024-05-30 08:12:03 +02:00
jade 7575db522e Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.

Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1465.3 s
  Codegen & opts (backend):   1110.9 s

<snip>

**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: serialise.hh strings.hh
  6x: <direct include>
  6x: archive.hh serialise.hh strings.hh
  ...

173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
  14x: store-api.hh nar-info.hh hash.hh
  11x: <direct include>
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh
  6x: fetchers.hh hash.hh
  ...

169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-derived-path.hh
  1x: installable-value.hh
  ...

159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
  19x: <direct include>
  14x: store-api.hh nar-info.hh hash.hh serialise.hh
  11x: serialise.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
  ...

156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
  19x: file-system.hh
  11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  ...

132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

  done in 0.6s.
```

After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
  done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

<snip>

**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: <direct include>
  6x: serialise.hh strings.hh
  6x: fetchers.hh hash.hh serialise.hh strings.hh
  ...

154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-value.hh
  1x: installable-derived-path.hh
  ...

136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
  37x: command.hh installable-value.hh installables.hh
  25x: store-api.hh realisation.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: libexpr.hh value.hh context.hh
  6x: shared.hh
  ...

87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
  35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
  12x: regex sstream istream
  10x: file-system.hh types.hh ref.hh memory unique_ptr.h
  10x: gtest.h memory unique_ptr.h
  10x: globals.hh types.hh ref.hh memory unique_ptr.h
  6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
  ...

85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
  37x: command.hh installable-value.hh installables.hh derived-path.hh
  20x: globals.hh
  20x: logging.hh
  16x: store-api.hh logging.hh
  6x: <direct include>
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
  ...

  done in 0.5s.
```

Change-Id: I27f0a2d566db17832cd9be935f12efe7f95b92d0
2024-05-29 22:16:15 -07:00
Cole Helbling 031d924116 libutil/args: warn on unknown settings after parsing all flags
Upstream change: https://github.com/NixOS/nix/pull/10701

Change-Id: Icf271df57ec529dd8c64667d1ef9f6dbf02d33d3
2024-05-30 03:07:21 +00:00
jade 26b3a1b9ce unix-domain-socket.cc: add comment explaining why bindConnectProcHelper
We reviewed this code a while ago, and we neglected to get a comment in
saying why it's Like This at the time. Let's fix that, since it is code
that looks very absurd at first glance.

Change-Id: Ib67b49605ef9ef1c84ecda1db16be74fc9105398
2024-05-29 19:50:04 -07:00
jade 562ff516ab Merge changes from topic "libutil-split" into main
* changes:
  util.hh: Delete remaining file and clean up headers
  util.hh: Move nativeSystem to local-derivation-goal.cc
  util.hh: Move stuff to types.hh
  util.cc: Delete remaining file
  util.{hh,cc}: Move ignoreException to error.{hh,cc}
  util.{hh,cc}: Split out namespaces.{hh,cc}
  util.{hh,cc}: Split out users.{hh,cc}
  util.{hh,cc}: Split out strings.{hh,cc}
  util.{hh,cc}: Split out unix-domain-socket.{hh,cc}
  util.{hh,cc}: Split out child.{hh,cc}
  util.{hh,cc}: Split out current-process.{hh,cc}
  util.{hh,cc}: Split out processes.{hh,cc}
  util.{hh,cc}: Split out file-descriptor.{hh,cc}
  util.{hh,cc}: Split out file-system.{hh,cc}
  util.{hh,cc}: Split out terminal.{hh,cc}
  util.{hh,cc}: Split out environment-variables.{hh,cc}
2024-05-30 02:33:05 +00:00
eldritch horrors dd4a2c1759 libstore: fix http abuses no longer working
while refactoring the curl wrapper we inadvertently broken the immutable
flake protocol, because the immutable flake protocol accumulates headers
across the entire redirect chain instead of using only the headers given
in the final response of the chain. this is a problem because Some Known
Providers Of Flake Infrastructure set rel=immutable link headers only in
the penultimate entry of the redirect chain, and curl does not regard it
as worth returning to us via its response header enumeration mechanisms.

fixes lix-project/lix#358

Change-Id: I645c3932b465cde848bd6a3565925a1e3cbcdda0
2024-05-29 22:35:29 +00:00
Tom Hubrecht a39ba22ff7 util.hh: Delete remaining file and clean up headers
Change-Id: Ic1f68e6af658e94ef7922841dd3ad4c69551ef56
2024-05-29 12:38:51 +02:00
Tom Hubrecht f0eb650ee8 util.hh: Move nativeSystem to local-derivation-goal.cc
Change-Id: I74565fbfd3aeedef8f50465808fac712b84e47ad
2024-05-29 11:42:42 +02:00
Tom Hubrecht d73c40ff3d util.hh: Move stuff to types.hh
Change-Id: Ia852306a4b8aac6856dc42bc69e4b58b53a0d67c
2024-05-29 11:42:42 +02:00
Tom Hubrecht 74513483bc util.cc: Delete remaining file
Change-Id: I2b47848904f2ce7bd78b83738e99a4c9da627751
2024-05-29 11:42:42 +02:00
Tom Hubrecht 93ebb3e7df util.{hh,cc}: Move ignoreException to error.{hh,cc}
Change-Id: Iae6464217a55c313a983e5c651b26a4a1e446706
2024-05-29 11:42:42 +02:00
Tom Hubrecht 8b6d2d3915 util.{hh,cc}: Split out namespaces.{hh,cc}
Change-Id: I8fd3f3b50c15ede29d489066b4e8d99c2c4636a6
2024-05-29 11:41:16 +02:00
Tom Hubrecht f79ee66646 util.{hh,cc}: Split out users.{hh,cc}
Change-Id: I1bd92479a2cb7e5c2c2e1541b80474adb05ea0df
2024-05-29 11:01:34 +02:00
Tom Hubrecht b910551120 util.{hh,cc}: Split out strings.{hh,cc}
Change-Id: I4f642d1046d56b5db26f1b0296ee16a0e02d444a
2024-05-29 11:01:34 +02:00
Tom Hubrecht 5b5a75979a util.{hh,cc}: Split out unix-domain-socket.{hh,cc}
Change-Id: I3f9a628e0f8998b6146f5caa8ae9842361a66b8b
2024-05-29 11:01:34 +02:00
Tom Hubrecht e81ed5f12d util.{hh,cc}: Split out child.{hh,cc}
Change-Id: Iec4824e071f537b17dd62dbb8c01b8eec14e9783
2024-05-29 11:01:34 +02:00
Tom Hubrecht 2473e1253d util.{hh,cc}: Split out current-process.{hh,cc}
Change-Id: I77095b9d37e85310075bada7a076ccd482c28e47
2024-05-29 11:01:34 +02:00
Tom Hubrecht 9a52e4688c util.{hh,cc}: Split out processes.{hh,cc}
Change-Id: I39280dc40ca3f7f9007bc6c898ffcf760e2238b7
2024-05-29 11:01:34 +02:00
Tom Hubrecht 8cd9aa24a8 util.{hh,cc}: Split out file-descriptor.{hh,cc}
Change-Id: I0dd0f9a9c2003fb887e076127e7f825fd3289c76
2024-05-29 09:54:47 +02:00
Tom Hubrecht 6b5078c815 util.{hh,cc}: Split out file-system.{hh,cc}
Change-Id: Ifa89a529e7e34e7291eca87d802d2f569cf2493e
2024-05-29 09:54:47 +02:00
Tom Hubrecht 81bdf8d2d6 util.{hh,cc}: Split out terminal.{hh,cc}
Change-Id: I9de2296b4012d50f540124001d54d6ca3be4c6da
2024-05-29 09:54:47 +02:00
Tom Hubrecht 6fd6795bc4 util.{hh,cc}: Split out environment-variables.{hh,cc}
Change-Id: Icff0aa33fda5147bd5dbe256a0b9d6a6c8a2c3f6
2024-05-28 11:29:29 +02:00
Qyriad 2cd1ef2201 Merge "libfetchers: fix URL logging" into main 2024-05-27 14:20:15 +00:00
Qyriad ec5039653d Merge "Revert "bump the extra --version info from info to notice, -vv -> -v"" into main 2024-05-26 21:51:05 +00:00
eldritch horrors ed6b3165ea worker: process timeouts first, and exclusively
check goals for timeouts first, and their activity fds only if no
timeout has occurred. checking for timeouts *after* activity sets
us up for assertion failures by running multiple build completion
notifiers, the first of which will kill/reap the the goal process
and consuming the Pid instance. when the second notifier attempts
to do the same it will core dump with an assertion failure in Pid
and take down not only the single goal, but the entire daemon and
all goals it was building. luckily this is rare in practice since
it requires a build to both finish and time out at the same time.

writing a test for this is not feasible due to how much it relies
on scheduling to actually trigger the underlying bug, but on idle
machines it can usually be triggered by running multiple sleeping
builds with timeout set to the sleep duration and `--keep-going`:

    nix-build --timeout 10 --builders '' --keep-going -E '
      with import <nixpkgs> {};
      builtins.genList
        (i: runCommand "foo-${toString i}" {} "sleep 10")
        100
    '

Change-Id: I394d36b2e5ffb909cf8a19977d569bbdb71cb67b
2024-05-26 20:57:09 +00:00
Qyriad 742c62a6eb docstrings: NixRepl::getDerivationPath: exceptions directly thrown
getDerivationPath() directly throws nix::Error for invalid derivations

Change-Id: I81ead950060b789794fa683b61c6349fece1690d
2024-05-26 17:19:26 +00:00
Qyriad 9f16a20f3d Revert "bump the extra --version info from info to notice, -vv -> -v"
This reverts commit d003dcd7f4.

d0390b5cf¹ bumped the default verbosity of nix3 commands, making this
change obsolete.

[1]: d0390b5cf2

Change-Id: I1ddc36ea4514cd8d2855ff334accf6e98c210879
2024-05-26 11:15:10 -06:00
Maximilian Bosch 5986a720d4 Merge "git putFile: support flake maximalists" into main 2024-05-26 10:16:40 +00:00
Qyriad ebd00b2d0b libfetchers: fix URL logging
8c06b7b43¹ made libfetchers log the URL being fetched just before the
actual fetch, particularly in case something freezes. This used the base
URL, to not include query parameters, as the Nixpkgs lib tests assume
that stderr logs will be equal across shallow and non-shallow git
fetches (and shallow fetches have the ?shallow=1 query parameter).

8c06b7b43 assumed that the `base` field of ParsedURL would be populated,
as the comment simply says "URL without query/fragment"... but
apparently it is not populated when the URL being fetched is *already*
fetched, which caused libfetchers to log things like

fetching gitlab input ''

which is. silly. but you know, busted lix be busted.

Anyway, with this commit we just remove the query params before printing
instead, which seems to do the right thing

[1]: 8c06b7b431

Change-Id: I9b9988992029aa6abef786f20b66e68c2ebb97d4
2024-05-25 19:37:38 -06:00
Artemis Tosini 53e2b0740c
libstore: remove duplicate builder variable
The `builder` local variable and duplicate `args.push_back` are no
longer required since the Darwin sandbox stopped using `sandbox-exec`.
The `drv->isBuiltin` check is not required either, as args are not
accessed when the builder is builtin.

Change-Id: I80b939bbd6f727b01793809921810ff09b579d54
2024-05-25 22:14:55 +00:00
jade dd53bce476 Merge "libstore: parse the buildMode instead of unchecked cast" into main 2024-05-25 17:42:09 +00:00
alois31 ddfe379a6b Merge "libstore/build: always enable seccomp filtering and no-new-privileges" into main 2024-05-25 04:21:53 +00:00
jade 2a7a824d83 libstore: parse the buildMode instead of unchecked cast
Change-Id: Icf6af7935e8f139bef36b40ad475e973aa48855c
2024-05-24 20:45:05 -06:00
Qyriad 5b4b216fac Merge "nix3-upgrade-nix: fix when there are differing pnames" into main 2024-05-25 02:20:24 +00:00
Qyriad 1c0f3c540e Merge changes I3ab84cc5,Iba34ad42 into main
* changes:
  nix3: always use the same verbosity default (info)
  libfetchers: log fetches by URL just before they happen
2024-05-25 01:52:44 +00:00
Qyriad f3f68fcfac nix3-upgrade-nix: fix when there are differing pnames
Change-Id: I19c7e24a4d46137127e76b7bb133e0184d73d1b6
2024-05-25 01:50:32 +00:00
Qyriad 076c19e0d1 change "evaluating file" logs to debug
I can't imagine wanting this unless you are debugging something (in
which case it's very useful)

Change-Id: I90c6f182c18486e9f6b15a59379bbb8e88fb8e7f
2024-05-24 23:03:08 +00:00
alois31 f047e4357b libstore/build: always enable seccomp filtering and no-new-privileges
Seccomp filtering and the no-new-privileges functionality improve the security
of the sandbox, and have been enabled by default for a long time. In
lix-project/lix#265 it was decided that they
should be enabled unconditionally. Accordingly, remove the allow-new-privileges
(which had weird behavior anyway) and filter-syscall settings, and force the
security features on. Syscall filtering can still be enabled at build time to
support building on architectures libseccomp doesn't support.

Change-Id: Iedbfa18d720ae557dee07a24f69b2520f30119cb
2024-05-24 21:19:29 +00:00
Qyriad d0390b5cf2 nix3: always use the same verbosity default (info)
Change-Id: I3ab84cc583e3e8b1c05a8ae1a7a087204f513d03
2024-05-24 15:15:42 -06:00
Qyriad 8c06b7b431 libfetchers: log fetches by URL just before they happen
Addresses but does not close #305, as we still need an indicator for
frozen fetches.

Change-Id: Iba34ad42dc1c8772f7da249b90fe794b041bbf73
2024-05-24 15:15:42 -06:00
Qyriad 00bf2b105d Merge changes I462a8cf0,I3b0bcea3,I2acd56e7,Ifc149764,I9e2ef170 into main
* changes:
  docs: linkify nix3-build mention in nix-build.md
  build: make internal-api-docs PHONY
  cleanup lookupFileArg
  add docstring to lookupFileArg
  add libcmd test for lookupFileArg
2024-05-23 23:28:42 +00:00
jade 9530b7f2b2 Merge "packaging: rename nixexpr -> lixexpr and so on" into main 2024-05-23 23:19:28 +00:00
jade c97e17144e packaging: rename nixexpr -> lixexpr and so on
This breaks downstreams linking to us on purpose to make sure that if
someone is linking to Lix they're doing it on purpose and crucially not
mixing up Nix and Lix versions in compatibility code.

We still need to fix the internal includes to follow the same schema so
we can drop the single-level include system entirely. However, this
requires a little more effort.

This adds pkg-config for libfetchers and config.h.

Migration path:
expr.hh      -> lix/libexpr/expr.hh
nix/config.h -> lix/config.h

To apply this migration automatically, remove all `<nix/>` from
includes, so: `#include <nix/expr.hh>` -> `#include <expr.hh>`. Then,
the correct paths will be resolved from the tangled mess, and the
clang-tidy automated fix will work.

Then run the following for out of tree projects:

```
lix_root=$HOME/lix
(cd $lix_root/clang-tidy && nix develop -c 'meson setup build && ninja -C build')
run-clang-tidy -checks='-*,lix-fixincludes' -load=$lix_root/clang-tidy/build/liblix-clang-tidy.so -p build/ -fix src
```

Related: lix-project/nix-eval-jobs#5
Fixes: lix-project/lix#279
Change-Id: I7498e903afa6850a731ef8ce77a70da6b2b46966
2024-05-23 16:45:23 -06:00
Qyriad a0dcfbb084 cleanup lookupFileArg
Change-Id: I2acd56e7a542b12138f43c95af78fdd50e944619
2024-05-23 14:11:10 -06:00
Qyriad 3a597f1d0b add docstring to lookupFileArg
Change-Id: Ifc149764f5a15725d3d630677c6da29def4b0f3e
2024-05-23 14:11:10 -06:00
Qyriad 677cf75473 repl: log errors writing to history file
These errors are now logged and explicitly ignored, rather than
implicitly ignored.

Change-Id: Ia26015466a17f2b11952df5317a4d150d79dc184
2024-05-23 18:10:23 +00:00
Pierre Bourdon 79121e0c44 Merge "repl: do not crash when tab-completing import errors" into main 2024-05-23 10:48:38 +00:00
eldritch horrors f281727e70 Merge "libstore: remove unused copyPath function" into main 2024-05-23 02:31:50 +00:00
Pierre Bourdon d8bc3bfb6d
repl: do not crash when tab-completing import errors
File not found while importing causes a SysError, not an EvalError,
which is not currently caught by the tab-completion handler. Ignoring
all SysErrors might seem "dangerous" but this is the tab-completion
handler, any exception being bubbled up from there causes unexpected
behavior (causes the whole repl to exit).

Fixes #340.

Change-Id: I643048a47935e77f582decc539d9e51bdb96c890
2024-05-23 02:52:54 +02:00
jade d05e0b9f1f fix: enlarge envSize by several times to not be close to nixpkgs size
nixpkgs has 23000 attributes, and our previous limit would be hit if you
have more than one nixpkgs in the environment, for example, because
`repl-overlays` will load the new stuff from the environment on top of
the existing environment.

This is not really testable since if we did write such a test, it would
just be testing this constant tbh...

Fixes: lix-project/lix#337
Change-Id: I49197bfb4db55b082f914f0d70e84f5f5f110954
2024-05-22 17:08:37 -07:00
Qyriad 06c1375e52 Merge "libfetchers: fallback to memory SQLite if fs IO fails" into main 2024-05-22 21:17:46 +00:00
jade dcc7ea5498 release notes: add a bunch of them
Also fix typos introduced by the commits I read.

I have run the addDrvOutputDependencies release note past Ericson since
I was confused by what the heck it was doing, and he was saying it was
reasonable.

Change-Id: Id015353b00938682f7faae7de43df7f991a5237e
2024-05-22 21:13:56 +02:00
eldritch horrors 47523944c5 libstore: remove unused copyPath function
Change-Id: Ibfb0848d8b337bd38947a745c240838865cea401
2024-05-22 18:26:01 +02:00
Qyriad 6881476232 libfetchers: fallback to memory SQLite if fs IO fails
nix::fetchers::CacheImpl uses $XDG_CACHE_HOME, or its default based on
$HOME, to store its SQLite database. If the current process can't write
to that directory for whatever reason, though, any eval-time fetching
would fail just initializing the cache.

With this change, IO errors initializing the fetcher cache are logged
but ignored, and nix::fetchers::CacheImpl falls back to an in-memory¹
database instead.

Notably, this will fix any uses eval fetching while Lix itself is being
run in a derivation builder (such as during tests), as the derivation
builder does not set $XDG_CACHE_HOME, and sets $HOME to the non-existent
directory /homeless-shelter.

Before:

$ env -u XDG_CACHE_HOME HOME=/homeless-shelter nix -Lv eval --impure -E 'fetchTarball "https://git.lix.systems/lix-project/lix/archive/main.tar.gz"'
error:
       … while calling the 'fetchTarball' builtin
         at «string»:1:1:
            1| fetchTarball "https://git.lix.systems/lix-project/lix/archive/main.tar.gz"
             | ^

       error: creating directory '/homeless-shelter': Permission denied

After:

$ env -u XDG_CACHE_HOME HOME=/homeless-shelter nix -Lv eval --impure -E 'fetchTarball "https://git.lix.systems/lix-project/lix/archive/main.tar.gz"'
warning: ignoring error initializing Lix fetcher cache: error: creating directory '/homeless-shelter': Permission denied
"/nix/store/s9lxdnn0awp37n560bg4fgr497ah4hvw-source"

¹: https://www.sqlite.org/inmemorydb.html

Change-Id: I15c38c9baaf215fc6e192b8a4c70b9692a69bc22
2024-05-21 09:30:25 -06:00
Qyriad 20981461d4 print type and value in "flake attr is not a derivation" errors
This turns errors like:

error: flake output attribute 'hydraJobs' is not a derivation or path

into errors like:

error: expected flake output attribute 'hydraJobs' to be a derivation or
path but found a set: { binaryTarball = «thunk»; build = «thunk»; etc> }

This change affects all InstallableFlake commands.

Change-Id: I899757af418b6f98201006ec6ee13a448c07077c
2024-05-21 05:55:13 -06:00
Artemis Tosini 3de77e6dbd Merge "libutil: Create chmodPath function" into main 2024-05-20 15:13:53 +00:00
Graham Christensen 8220da8a53
git putFile: support flake maximalists
Passing the commit message as an argument causes update failures on repositories with
lots of flake inputs. In some cases, the commit message is over 250,000 bytes.

Upstream PR: https://github.com/NixOS/nix/pull/10686

(cherry picked from commit 8b5e8f4fba5728f2b3e90fcd1ab15df77e3ea0e8)
Change-Id: I2c196a21cc9bedc24d57a828a0c5b9467e072f76
2024-05-20 13:51:29 +02:00
Qyriad 589953e832 Merge "fix -Wdeprecated-copy on clang (BaseError copy assignment)" into main 2024-05-20 00:11:12 +00:00
puck bfb91db4f6 repl-interacter: save history after entering every line
Fixes: lix-project/lix#328
Change-Id: Iedd79ff5f72e84766ebd234c63856170afc624f0
2024-05-19 22:47:45 +00:00
Artemis Tosini 5411fbf204
libutil: Create chmodPath function
Move the identical static `chmod_` functions in libstore to
libutil. the function is called `chmodPath` instead of `chmod`
as otherwise it will shadow the standard library chmod in the nix
namespace, which is somewhat confusing.

Change-Id: I7b5ce379c6c602e3d3a1bbc49dbb70b1ae8f7bad
2024-05-19 22:07:58 +00:00
Qyriad 4eb6779ea8 fix -Wdeprecated-copy on clang (BaseError copy assignment)
2bbe3efd1¹ added the -Wdeprecated-copy warning, and fixed the instances
of it which GCC warned about, in HintFmt and ref<T>. However, when
building with Clang, there is an additional deprecated-copy warning in
BaseError. This commit explicitly defaults the copy assignment operator
for BaseError and silences this warning.

1: 2bbe3efd16
Change-Id: I50aa4a7ab1a7aae5d7b31f765994abd3db06379d
2024-05-19 12:32:13 -06:00
eldritch horrors 774c56094f libstore: fix old RemoteStore::addToStore serializer
having the serializer write into `*conn` is not legal because we are
in a sinkToSource that will be drained by the remote we're connected
to. writing into `*conn` directly can break the framing protocol. it
is unlikely this code was ever run: to protocol it caters to is from
2016(!) and thoroughly untested in-tree, and since it's been present
since nix 2.17 and the 1.18 protocol broken here is nix 2.0 we might
safely assume that daemons older than nix 2.1 are no longer used now

see also #325 (though that wants <2.3 gone, this is sadly only <2.1)

Change-Id: I9d674c18f6d802f61c5d85dfd9608587b73e70a5
2024-05-19 11:57:55 +00:00
Alyssa Ross 139d31f876 Improve nix-store --delete failure message
On several occasions I've found myself confused when trying to delete
a store path, because I am told it's still alive, but
nix-store --query --roots doesn't show anything.  Let's save future
users this confusion by mentioning that a path might be alive due to
having referrers, not just roots.

(cherry picked from commit 979a019014569eee7d0071605f6ff500b544f6ac)

Upstream-PR: https://github.com/NixOS/nix/pull/10733
Change-Id: I54ae839a85f3de3393493fba27fd40d7d3af0516
2024-05-18 14:49:40 -06:00
puck 1fe58bd8a7 nix cat/dump-path/key: stop progress bar before writeFull
These commands outputs data that may not end with a newline. This
causes problems when the progress bar redraws, as that completely
wipes the last line of output. As nix key generate-secret outputs
a single line of text with no output, it shows up entirely blank,
making it look like nothing happened.

Fixes: lix-project/lix#320
Change-Id: I5ac706d71d839b6dfa760b60a351414cd96297cf
2024-05-18 17:51:16 +00:00
Pierre Bourdon d1c8fd3b09 Merge "derived-path: refuse built derived path with a non-derivation base" into main 2024-05-18 07:26:26 +00:00
julia 7a3745b076
Deprecate the online flake registries and vendor the default registry
Fixes #183, #110, #116.

The default flake-registry option becomes 'vendored', and refers
to a vendored flake-registry.json file in the install path.

Vendored copy of the flake-registry is from github:NixOS/flake-registry
at commit 9c69f7bd2363e71fe5cd7f608113290c7614dcdd.

Change-Id: I752b81c85ebeaab4e582ac01c239d69d65580f37
2024-05-18 12:27:23 +10:00
puck 23c92f0815 Merge "primops: change to std::function, allowing the passing of user data" into main 2024-05-17 21:37:41 +00:00
puck 92e1df23b3 Merge "Loosen constness on listElems() result" into main 2024-05-17 21:37:35 +00:00
Qyriad 93b7edfd07 Merge "docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation" into main 2024-05-17 18:17:05 +00:00
julia 005ee33a9a Merge "Allow enabling core dumps from builds for nix & child processes" into main 2024-05-17 01:10:44 +00:00
Qyriad 5ff076d8ad docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation
Both of these still needs their own actual documentation, but they are
at least now mentioned that they exist and what they're enabled by.

Change-Id: I235b9e8e627e04ed06611423c8e67a8eca233120
2024-05-17 00:41:35 +00:00
Pierre Bourdon 5a1824ebe1
derived-path: refuse built derived path with a non-derivation base
Example: /nix/store/dr53sp25hyfsnzjpm8mh3r3y36vrw3ng-neovim-0.9.5^out

This is nonsensical since selecting outputs can only be done for a
buildable derivation, not for a realised store path. The build worker
side of things ends up crashing with an assertion when trying to handle
such malformed paths.

Change-Id: Ia3587c71fe3da5bea45d4e506e1be4dd62291ddf
2024-05-17 02:16:15 +02:00
julia 5b7dcb3005 Allow enabling core dumps from builds for nix & child processes
Fixes lix-project/lix#268

Change-Id: I3f1b0ddf064f891cca8b53229c5c31c74cea3d9f
2024-05-16 17:11:21 -07:00
Qyriad 5756be19f6 Merge "add clarifying doc-comments on {get,create}NixStateDir()" into main 2024-05-17 00:10:38 +00:00
jade 9322a1cbe7 Merge "builtins: fix builtins.langVersion docs to state it's deprecated" into main 2024-05-16 21:19:15 +00:00
Yorick 194654c96f primops: change to std::function, allowing the passing of user data
(cherry picked from commit 48aa57549d514432d6621c1e29f051951eca2d7f)
Change-Id: Ib7d5c6514031ceb6c42ac44588be6b0c1c3c225b
2024-05-16 13:01:40 +00:00
puck c6bb377c91 Loosen constness on listElems() result
Change-Id: I1caff000362c83e5172413a036c22a2e9ed3ede8
2024-05-16 13:01:40 +00:00
jade 6d84d6bdc1 builtins: fix builtins.langVersion docs to state it's deprecated
This builtin was always a problem and nixpkgs uses it in exactly one
place, to give up if the Nix version is absurdly old. It has no other
use cases, and doesn't work in a multi-implementation world anyway.

Change-Id: I03c36e118591029e2ef14b091fe14a311c66a08a
2024-05-15 21:54:12 -07:00
Mel Zuser 9249c89dc6 build: fix build when gc is disabled
Change-Id: I8d3eb8874a4138668011b525c3b400a55a1f4866
2024-05-15 18:48:00 -07:00
Maximilian Bosch aabc902040 Merge "nix3-build: show all FOD errors with --keep-going" into main 2024-05-15 18:46:06 +00:00
alois31 0903a99bad Merge changes I8456c47b,I48253f5f into main
* changes:
  repl: clear the interrupt before reading the next line
  libutil: remove the interrupt-blocking code
2024-05-15 15:46:11 +00:00
Maximilian Bosch e146393183
nix3-build: show all FOD errors with --keep-going
Basically I'd expect the same behavior as with `nix-build`, i.e.
with `--keep-going` the hash-mismatch error of each failing
fixed-output derivation is shown.

The approach is derived from `Store::buildPaths` (`entry-point.cc`):
instead of throwing the first build-result, check if there are any build
errors and if so, display all of them and throw after that.

Unfortunately, the BuildResult struct doesn't have an `ErrorInfo`
(there's a FIXME for that at least), so I have to construct my own here.
This is a rather cheap bugfix and I decided against touching too many
parts of libstore for that (also I don't know if that's in line with the
ongoing refactoring work).

Closes lix-project/lix#302

Change-Id: I378ab984fa271e6808c6897c45e0f070eb4c6fac
2024-05-15 15:35:18 +02:00
alois31 eeb7e71810
repl: clear the interrupt before reading the next line
Otherwise, it will be thrown again during exit when the repl is terminated by
end-of-input after the last command was interrupted.

Change-Id: I8456c47bc36cfb0892efdad5420f318f7e6526d5
2024-05-13 09:04:05 +02:00
alois31 914b0febf7
libutil: remove the interrupt-blocking code
The interrupt-blocking code was originally introduced 20 years ago so that
trying to log an error message does not result in an interrupt exception being
thrown and then going unhandled (c8d3882cdc).
However, the logging code does not check for interrupts any more
(054be50257), so this reasoning is no longer
applicable. Delete this code so that later interrupts are unblocked again, for
example in the next line entered into the repl.

Closes: lix-project/lix#296
Change-Id: I48253f5f4272e75001148c13046e709ef5427fbd
2024-05-13 09:03:53 +02:00
Artemis Tosini 4b35e6a75e Merge "libstore: Fix sandbox=relaxed" into main 2024-05-12 03:51:19 +00:00
Qyriad 88d9b70f67 add clarifying doc-comments on {get,create}NixStateDir()
NixStateDir() != NIX_STATE_DIR. These functions should honestly probably
be renamed.

Change-Id: I00f54b742bba6188bbc7f2410956d956780b99d3
2024-05-11 15:40:00 -06:00
Pierre Bourdon 38d825b21e
filetransfer: fix decompression regression from 121edecf
121edecf65 added a new state field to
carry over content encoding settings from transfer to sink creation, but
never actually set that field.

Change-Id: I714b2efe745561e851b78a4791479b3501db8c72
2024-05-11 14:49:23 +02:00
Qyriad f6dc40cd1c Merge "bump the extra --version info from info to notice, -vv -> -v" into main 2024-05-10 21:56:09 +00:00
raito 4ebbd4362f Merge "feat(libstore): print the first line of stdout of SSH in case of failure" into main 2024-05-10 19:33:36 +00:00
raito 8404a1f66d feat(libstore): print the first line of stdout of SSH in case of failure
In case of failure to connect as can be seen in
https://buildbot.lix.systems/#/builders/39/builds/1386/steps/1/logs/stdio

It is difficult to understand what happened, if we enabled the talkative
verbose level, we could learn about the first line SSH sent us.

In practice, this is not workable, we can just make it warn all the
time.

Change-Id: Iaaf56894060a58f2dfc78254bb60b1c43482f9bb
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-05-10 20:22:47 +02:00
Qyriad d003dcd7f4 bump the extra --version info from info to notice, -vv -> -v
requiring *two* --verbose to print extra information on --version is a
weird flex

Change-Id: I05d043da1bf583f34e9d1fc206144ea4ca9a859d
2024-05-10 11:55:17 -06:00
eldritch horrors ceccac835c libutil: remove callback.hh
it's no longer used. it really shouldn't have existed this long since it
was just a mashup of both std::promise and std::packaged_task in a shape
that makes composition unnecessarily difficult. all but a single case of
Callback pattern calls were fully synchronous anyway, and even this sole
outlier was by far not important enough to justify the extra complexity.

Change-Id: I208aec4572bf2501cdbd0f331f27d505fca3a62f
2024-05-10 02:21:11 +02:00
eldritch horrors b66451ae7f libstore: de-callback-ify FileTransfer
also add a few more tests for exception propagation behavior. using
packaged_tasks and futures (which only allow a single call to a few
of their methods) introduces error paths that weren't there before.

Change-Id: I42ca5236f156fefec17df972f6e9be45989cf805
2024-05-10 02:21:11 +02:00
eldritch horrors 28a98d152c libstore: de-callback-ify Store::queryRealisation
Change-Id: I8d74745c519518f163f51dfaa39063836f17599e
2024-05-09 23:18:05 +02:00
eldritch horrors 17965bf11c libstore: un-callback-ify Store::queryRealisationUncached
Change-Id: I4a328f46eaac3bb8b19ddc091306de83348be9cf
2024-05-09 23:18:05 +02:00
eldritch horrors 2f4a1dd6e0 libstore: de-callback-ify Store::queryPathInfoUncached
Change-Id: I23a156aaff5328f67ca16ccd85c0ea1711b21e35
2024-05-09 23:18:05 +02:00
eldritch horrors c77bd88259 libstore: de-callback-ify BinaryCacheStore::getFile
Change-Id: I36b3eb9f645aa04058151e7b2353e15e6f29057b
2024-05-09 23:18:05 +02:00
eldritch horrors 1a002d1a11 libstore: de-callback-ify CA realisation substitution
this is the *only* real user of file transfer download completion
callbacks, and a pretty spurious user at that (seeing how nothing
here is even turned on by default and indeed a dependency of path
substitution which *isn't* async, and concurrency-limited). it'll
be a real pain to keep this around, and realistically it would be
a lot better to overhaul substitution in general to be *actually*
async. that requires a proper async framework footing though, and
we don't have anything of the sort, but it's also blocking *that*

Change-Id: I1bf671f217c654a67377087607bf608728cbfc83
2024-05-09 23:18:05 +02:00
Qyriad 1f9b0fba23 add and fix -Wignored-qualifiers
Change-Id: I4bffa766ae04dd80355f9b8c10e59700e4b406da
2024-05-09 07:08:44 -06:00
Qyriad 2bbe3efd16 add and fix -Wdeprecated-copy
*so* many warnings, from only two definitions

Change-Id: If2561cd500c05a1e33cce984faf9f3e42a8a95ac
2024-05-09 13:02:45 +00:00
Qyriad 010bbd1b0e Merge "nix3-eval: don't elide top-level errors" into main 2024-05-09 11:46:13 +00:00
Qyriad 54322f09d3 nix3-eval: don't elide top-level errors
Fixes #276.

Change-Id: I83e71beb5c35d6f3b10a4186caa5e52a2f95b510
2024-05-08 13:37:20 -06:00
Théophane Hufschmitt adea821d87
libstore: Fix sandbox=relaxed
The fix for the Darwin vulnerability in ecdbc3b207
also broke setting `__sandboxProfile` when `sandbox=relaxed` or
`sandbox=false`. This cppnix change fixes `sandbox=relaxed` and
adds a suitable test.

Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
Change-Id: I40190f44f3e1d61846df1c7b89677c20a1488522
2024-05-08 19:31:43 +00:00
alois31 243c0f18da
repl: show a progress bar while performing builds
In commit 946fc12e4e, the progress bar in the
repl was disabled again because it was observed to erase incremental output
from attrset evaluations from the terminal. Let's try adding the progress bar
again, this time showing up only when a build is initiated, which does not have
incremental output that could be destroyed to begin with. While this does mean
that we won't have a progress bar for eval-time fetching or IFD, it's still
better than nothing.

Change-Id: If4eb1035cd0c876f5b4ff1e2434b9baf99f150ac
2024-05-08 20:54:41 +02:00
Qyriad b9be46fb31 remove the autoconf+Make buildsystem
We're not using it anymore. Any leftover bugs in the Meson buildsystem
are now just bugs.

Closes #249.

Change-Id: I0465a0c37ae819f94d40e7829f5bff046aa63d73
2024-05-07 17:04:30 -06:00
Nikodem Rabuliński e8a603fb2f Merge changes Icf26010a,Ib6161567 into main
* changes:
  Always initialize curl in parent process on darwin
  Fix failing darwin tests
2024-05-07 21:26:24 +00:00
Nikodem Rabuliński f894cce79b
Always initialize curl in parent process on darwin
Because of an objc quirk[1], calling curl_global_init for the first time
after fork() will always result in a crash.
Up until now the solution has been to set
OBJC_DISABLE_INITIALIZE_FORK_SAFETY for every nix process to ignore
that error.
This is less than ideal because we were setting it in package.nix,
which meant that running nix tests locally would fail because
that variable was not set.
Instead of working around that error we address it at the core -
by calling curl_global_init inside initLibStore, which should mean
curl will already have been initialized by the time we try to do so in
a forked process.

[1] 01edf1705f/runtime/objc-initialize.mm (L614-L636)

Change-Id: Icf26010a8be655127cc130efb9c77b603a6660d0
2024-05-07 20:43:17 +02:00
eldritch horrors 964ac8b0e8 libutil: de-callback-ify computeClosure
only two users of this function exist. only one used it in a way that
even bears resemblance to asynchronicity, and even that one didn't do
it right. fully async and parallel computation would have only worked
if any getEdgesAsync never calls the continuation it receives itself,
only from more derived callbacks running on other threads. calling it
directly would cause the decoupling promise to be awaited immediately
*on the original thread*, completely negating all nice async effects.

Change-Id: I0aa640950cf327533a32dee410105efdabb448df
2024-05-07 14:35:20 +00:00
eldritch horrors 230860dbb8 libstore: limit CA realisation info substitution concurrency
this seems to be an oversight, considering that regular substitutions
are concurrency-limited. while not particularly necessary at present,
once we've removed the `Callback` based interfaces it will be needed.

Change-Id: Ide2d08169fcc24752cbd07a1d33fb8482f7034f5
2024-05-07 14:35:20 +00:00
eldritch horrors 29f93e1e0d libutil: throw EndOfFile at sourceToSink end
... how has this never broken anything before

Change-Id: If3789c02028e8f929481514f63d76b0b46bfc182
2024-05-07 14:35:20 +00:00
Qyriad 005b2b61e6 fix fallback chroot store creation after b247ef72d
When /nix/var (or, more precisely, NIX_STATE_DIR) does not exist at all,
Lix falls back to creating an adhoc chroot store in XDG_DATA_HOME.

b247ef72d[1] changed the way Store classes are initialized, and in the
migration, a `params2` was accidentally changed to `params`. This commit
restores the correct behavior, and in lieu of a single *character* fix,
this commit also changes the variable name to something more reasonable.

Fixes #274.

[1]: b247ef72dc

n.b., this code might deserve some more looking at anyway. this fallback
store creation throws away *all* Store params passed to
openFromNonUri() in favor of an entirely new set which only contains
the `root` param, which may or may not be the correct behavior

Change-Id: Ibea559b88a50e6d6e75a1f87d9d7816cabb2a8f3
2024-05-06 20:54:21 -06:00
eldritch horrors f75d0752ce filetransfer: correctly abort empty transfers
returning 0 from the callback for errors signals successful transfer if
the source returned no data even though the exception we've just caught
clearly disagrees. while this is not all that important (since the only
viable cause of such errors will be dataCallback, and the sole instance
of it being used already takes care of exceptions) we can just do this.

Change-Id: I2bb150eff447121d82e8e3aa4e00057c40523ac6
2024-05-06 21:18:23 +02:00
eldritch horrors 121edecf65 filetransfer: extract decompressor creation
this will be necessary if we want download() to return a source instead
of consuming a sink, which will in turn be needed to remove coroutines.

Change-Id: I34ec241e9bbc5d32fbcd243b244e29c3757533aa
2024-05-06 20:19:03 +02:00
jade 748d8310fa Fix the pages in the manual for Lix
This doesn't comprehensively fix everything outdated in the manual, or
make the manual greatly better, but it does note down where at least
jade noticed it was wrong, and it does fix all the instances of
referencing Nix to conform to the style guide to the best of our
ability.

A lot of things have been commented out for being wrong, and there are
three types of FIXME introduced:

- FIXME(Lix): generically Lix needs to fix it
- FIXME(Qyriad): re lix-project/lix#215
- FIXME(meson): docs got outdated by meson changes and need rewriting

I did fix a bunch of it that I could, but there could certainly be
mistakes and this is definitely just an incremental improvement.

Fixes: lix-project/lix#266
Change-Id: I5993c4603d7f026a887089fce77db08394362135
2024-05-05 16:11:01 -07:00
Qyriad 99903f24eb fix integer overflow on i686 with high phys memory
sizeof(long) is 4 bytes on i686 GCC.
With ~32 GiB of memory and a page size of 4096, there are 7988420 pages.
(7988420 * 4096) is bigger than INT32_MAX folks.

This has gone unnoticed for 9 years, and only came up thanks to
94ea517db[1] adding integer overflow sensitization checks, which caused
this broken code to emit an illegal instruction, crashing Lix the
instant the buildsystem ran Lix to generate the docs files.

[1]: 94ea517dbe

Change-Id: I50bb9ea072aac11b449d79e5d55525887a6e5a99
2024-05-05 19:47:46 +00:00
Qyriad 4998699e1a Merge "point nix3-upgrade-nix to releases.lix.systems/manifest.nix" into main 2024-05-05 19:47:36 +00:00
jade 4fd32351f3 Merge "Warn on untrusted client settings being ignored" into main 2024-05-05 19:35:35 +00:00
eldritch horrors 6b08138929 filetransfer: abort transfer on receiver exception
not doing this will cause transfers that had their readers disappear to
linger. with lingering transfers the curl thread can't shut down, which
will cause nix itself to not shut down until the transfer finishes some
other way (most likely network timeouts). also add a new test for this.

Change-Id: Id2401b3ac85731c824db05918d4079125be25b57
2024-05-05 18:09:31 +00:00
Qyriad 7ab076f21c point nix3-upgrade-nix to releases.lix.systems/manifest.nix
This file is currently manually managed, but will be automated along
with the rest of the release process.

Change-Id: I77839919549aaac73de582b2e563ce3ef914a8cb
2024-05-05 17:45:50 +00:00
jade 2e8f9ac944 Warn on untrusted client settings being ignored
These are such a footgun and trip people up a lot. Let's make Lix louder
about this.

Related: lix-project/lix#261
Change-Id: I6a8d57c9817caaa6b0cbf886c615dda51038f628
2024-05-04 21:30:26 -07:00
jade 47fb494676 Merge "Actually try making a userns before assuming they don't work" into main 2024-05-05 03:58:44 +00:00
jade fb5d6f325b Merge "Fix /etc/group having desynced IDs from the actual UID in the sandbox" into main 2024-05-05 03:57:29 +00:00
jade e3b702fa22 Actually try making a userns before assuming they don't work
If unprivileged userns are *believed* to be disabled (such as with
"kernel.unprivileged_userns_clone = 0"), Lix would previously *give up*
on trying to use a user namespace before actually trying it, even if, in
cases such as unprivileged_userns_clone, it would actually be allowed
since Nix has CAP_SYS_ADMIN when running as daemon.

(see, e.g. 25d4709a4f)

We changed it to actually try it first, and then diagnose possible
causes, and also to be more loud about the whole thing, using warnings
instead of debugs. These warnings will only print on the first build run
by the daemon, which is, tbh, eh, shrug.

This is what led to us realizing that no-userns was a poorly exercised
condition.

Change-Id: I8e4f21afc89c574020dc7e89a560cc740ce6573a
2024-05-05 00:37:24 +00:00
jade 9909a175bf Fix /etc/group having desynced IDs from the actual UID in the sandbox
This was found when `logrotate.conf` failed to build in a NixOS system
with:

    /nix/store/26zdl4pyw5qazppj8if5lm8bjzxlc07l-coreutils-9.3/bin/id: cannot find name for group ID 30000

This was surprising because it seemed to mean that /etc/group was busted
in the sandbox. Indeed it was:

    root0:
    nixbld:!💯
    nogroup65534:

We diagnosed this to sandboxUid() being called before
usingUserNamespace() was called, in setting up /etc/group inside the
sandbox. This code desperately needs refactoring.

We also moved the /etc/group code to be with the /etc/passwd code, but
honestly this code is all spaghetti'd all over the place and needs some
more serious tidying than we did here.

We also moved some checks to be earlier to improve locality with where
the things they are checking come from.

Change-Id: Ie29798771f3593c46ec313a32960fa955054aceb
2024-05-04 17:36:50 -07:00
jade 4886d4592b Remove a URL literal from fetchTarball docs
Change-Id: I254b793b42f77ffe9f357f3b376683e5758f23b5
2024-05-04 16:55:27 -07:00
Maximilian Bosch 799d0132f0 Merge "Revert "Revert "Merge pull request #6621 from Kha/nested-follows""" into main 2024-05-04 08:52:29 +00:00
Maximilian Bosch 79d0ae6670 Merge "libstore/local-derivation-goal: prohibit creating setuid/setgid binaries" into main 2024-05-04 07:26:15 +00:00
Maximilian Bosch 0e38720502 Revert "Revert "Merge pull request #6621 from Kha/nested-follows""
This reverts commit a8b3d777fb.

This undoes the revert of PR#6621, which allows nested `follows`, i.e.

    {
      inputs = {
        foo.url = "github:bar/foo";
        foo.inputs.bar.inputs.nixpkgs = "nixpkgs";
      };
    }

does the expected thing now. This is useful to avoid the 1000 instances
of nixpkgs problem without having each flake in the dependency tree to
expose all of its transitive dependencies for modification.

This was in fact part of Nix before and the C++ changes applied w/o
conflicts. However, it got reverted then because people didn't want to
merge lazy-trees against it which was supposed to be merged soon back in
October 2022.

Fixes: lix-project/lix#201

Change-Id: I5ddef914135b695717b2ef88862d57ced5e7aa3c
2024-05-03 22:54:38 +00:00
Maximilian Bosch f8617f9dc6 Merge "Rename nix show-config to nix config show" into main 2024-05-03 22:07:33 +00:00