Commit graph

15973 commits

Author SHA1 Message Date
eldritch horrors d477b34d1d libutil: remove runProgram2 stdin functionality
this was only used in one place, and that place has been rewritten to
use a temporary file instead. keeping this around is not very helpful
at this time, and in any case we'd be better off rewriting subprocess
handling in rust where we not only have a much safer library for such
things but also async frameworks necessary for this easily available.

Change-Id: I6f8641b756857c84ae2602cdf41f74ee7a1fda02
2024-06-23 17:29:40 +00:00
eldritch horrors 2bbdaf0b19 libfetchers: write git commit message to tempfile
we want to remove runProgram's ability to provide stdin to a process
because the concurrency issues of handling both stdin and stdout are
much more pronounced once runProgram returns not is collected output
but a source. this is possible in the current c++ framework, however
it isn't necessary in almost all cases (as demonstrated by only this
single user existing) and in much better handled with a proper async
concurrency model that lets the caller handle both at the same time.

Change-Id: I29da1e1ad898d45e2e33a7320b246d5003e7712b
2024-06-23 17:29:40 +00:00
eldritch horrors 2fe9157808 flakes: add --commit-lock-file message test
we had no test to ensure that we generated a commit message at all?

Change-Id: Ic9aa8fde92b83e1ea6f61cd2a21867aa73d4e885
2024-06-23 17:29:40 +00:00
Qyriad 521e08cbde Merge changes I697f4f39,I9f25235d into main
* changes:
  doc-comment Fields for Activity and Result types
  mildly refactor the renderActivity if hell-chain
2024-06-23 16:55:36 +00:00
Maximilian Bosch 7c3b8229cc Merge "libstore: reduce loglevel of waiting for a machine to build" into main 2024-06-23 16:33:44 +00:00
Qyriad 701eb332bd doc-comment Fields for Activity and Result types
The ones we were able to figure out, at least.

Change-Id: I697f4f3942e35a7adf1b2a6cc28b3168d1de111c
2024-06-23 16:30:48 +00:00
Qyriad ce4dee0fa5 mildly refactor the renderActivity if hell-chain
This is primarily for readability, but iwrc chaining std::string's
operator+ is also pretty scuffed performance-wise, and this was doing a
lot of operator+ chainging.

Change-Id: I9f25235df153eb2bbb491f1ff7ebbeed9a8ec985
2024-06-23 16:30:38 +00:00
Maximilian Bosch 5f0062285c Merge "libfetchers: make attribute / URL query handling consistent" into main 2024-06-23 15:51:34 +00:00
eldritch horrors ce6cb14995 libutil: return Pid from startProcess, not pid_t
Change-Id: Icc8a15090c77f54ea7d9220aadedcd4a19922814
2024-06-23 11:52:49 +00:00
eldritch horrors 3d155fc509 libutil: give Pid proper resource semantics
copy-constructing or assigning from pid_t can easily lead to duplicate
Pid instances for the same process if a pid_t was used carelessly, and
Pid itself was copy-constructible. both could cause surprising results
such as killing processes twice (which could become very problemantic,
but luckily modern systems don't reuse PIDs all that quickly), or more
than one piece of the code believing it owns a process when neither do

Change-Id: Ifea7445f84200b34c1a1d0acc2cdffe0f01e20c6
2024-06-23 11:52:49 +00:00
eldritch horrors b43a2e84c4 libutil: make Pid -> pid_t operations explicit
Change-Id: I3137cc140590001fe7ba542844e735944a0a9255
2024-06-23 11:52:49 +00:00
eldritch horrors a9949f4760 libutil: add some serialize.hh serializer tests
Change-Id: I0116265a18bc44bba16c07bf419af70d5195f07d
2024-06-23 11:52:49 +00:00
eldritch horrors 39a1e248c9 libutil: remove sinkToSource eof callback
this is only used in one place, and only to set a nicer error message on
EndOfFile. the only caller that actually *catches* this exception should
provide an error message in that catch block rather than forcing support
for setting error message so deep into the stack. copyStorePath is never
called outside of PathSubstitutionGoal anyway, which catches everything.

Change-Id: Ifbae8706d781c388737706faf4c8a8b7917ca278
2024-06-23 11:52:49 +00:00
Artemis Tosini f80d95e36d Merge "libstore: Start creating LocalDerivationGoal subclasses" into main 2024-06-23 08:11:51 +00:00
Artemis Tosini 12f5d27363 libstore: Start creating LocalDerivationGoal subclasses
LocalDerivationGoal includes a large number of low-level sandboxing
primitives for Darwin and Linux, intermingled with ifdefs.
Start creating platform-specific classes to make it easier to add new
platforms and review platform-specific code.

This change only creates support infrastructure and moves two function,
more functions will be moved in future changes.

Change-Id: I9fc29fa2a7345107d4fc96c46fa90b4eabf6bb89
2024-06-23 03:33:07 +00:00
Qyriad fb8553f63c mildly cleanup ExprSelect::eval
Better variable names, some comments, and a slight logic rearrange.

Change-Id: I9685ae252f83217aa85f06432234159c9ad19d1c
2024-06-22 18:52:57 -06:00
Qyriad e09cc60df9 doc-comment ExprSelect's fields
Change-Id: I63e79991a4bab93421266785e9258e0f5bb89b8f
2024-06-22 18:52:57 -06:00
Maximilian Bosch fc6a1451af
libstore: reduce loglevel of waiting for a machine to build
This comes quite often when the available job slots on all remote
builders are exhausted and this is pretty spammy.

This isn't really an issue, but expected behavior.

A better way to display this is a nom-like approach where all scheduled
builds are shown in a tree and pending builds are being marked as such
IMHO.

Change-Id: I6bc14e6054f84e3eb0768127b490e263d8cdcf89
2024-06-22 17:38:25 +02:00
Maximilian Bosch 35eec921af
libfetchers: make attribute / URL query handling consistent
The original idea was to fix lix#174, but for a user friendly solution,
I figured that we'd need more consistency:

* Invalid query params will cause an error, just like invalid
  attributes. This has the following two consequences:

  * The `?dir=`-param from flakes will be removed before the URL to be
    fetched is passed to libfetchers.

  * The tarball fetcher doesn't allow URLs with custom query params
    anymore. I think this was questionable anyways given that an
    arbitrary set of query params was silently removed from the URL you
    wanted to fetch. The correct way is to use an attribute-set
    with a key `url` that contains the tarball URL to fetch.

  * Same for the git & mercurial fetchers: in that case it doesn't even
    matter though: both fetchers added unused query params to the URL
    that's passed from the input scheme to the fetcher (`url2` in the code).
    It turns out that this was never used since the query parameters were
    erased again in `getActualUrl`.

* Validation happens for both attributes and URLs. Previously, a lot of
  fetchers validated e.g. refs/revs only when specified in a URL and
  the validity of attribute names only in `inputFromAttrs`.

  Now, all the validation is done in `inputFromAttrs` and `inputFromURL`
  constructs attributes that will be passed to `inputFromAttrs`.

* Accept all attributes as URL query parameters. That also includes
  lesser used ones such as `narHash`.

  And "output" attributes like `lastModified`: these could be declared
  already when declaring inputs as attribute rather than URL. Now the
  behavior is at least consistent.

  Personally, I think we should differentiate in the future between
  "fetched input" (basically the attr-set that ends up in the lock-file)
  and "unfetched input" earlier: both inputFrom{Attrs,URL} entrypoints
  are probably OK for unfetched inputs, but for locked/fetched inputs
  a custom entrypoint should be used. Then, the current entrypoints
  wouldn't have to allow these attributes anymore.

Change-Id: I1be1992249f7af8287cfc37891ab505ddaa2e8cd
2024-06-22 14:42:43 +02:00
kloenk da4e46dd1f libmain: add progress bar with multiple status lines
Add the log-formats `multiline` and `multiline-with-logs` which offer
multiple current active building status lines.

Change-Id: Idd8afe62f8591b5d8b70e258c5cefa09be4cab03
2024-06-22 14:20:27 +02:00
Qyriad 21865ccce0 Merge "add a basic libmain test for the progress bar rendering" into main 2024-06-22 02:04:14 +00:00
jade 375f4c0337 Merge "libstore: remove operations that are never called by supported clients" into main 2024-06-21 20:46:18 +00:00
Qyriad fd250c51ed add a basic libmain test for the progress bar rendering
Hooray for leaky abstraction allowing us to test this particular part of
the render pipeline.

Change-Id: Ie0f251ff874f63324e6a9c6388b84ec6507eeae2
2024-06-20 13:56:53 -06:00
Qyriad e44dcd63c4 remove InstallableValueCommand class
Change-Id: Id12383f4741cba07159712700ebcfbe37e61560c
2024-06-20 17:00:06 +00:00
Qyriad 6515b1a495 de-inheritance CmdSearch for InstallableValueCommand
Change-Id: I125c8cac05c8e924b55e4eb1060496e35ea4e941
2024-06-20 17:00:06 +00:00
Qyriad 50be55ffca de-inheritance CmdEdit for InstallableValueCommand
Change-Id: If85ea78954a45470b0b25c08dc7d40bfebd53610
2024-06-20 17:00:06 +00:00
Qyriad 079eeb1de7 de-inheritance CmdRun for InstallableValueCommand
Change-Id: Ief858c1488197211e2ee8b70aa40ed6c65743558
2024-06-20 17:00:06 +00:00
Qyriad b9e9235ac0 de-inheritance CmdBundle for InstallableValueCommand
Change-Id: Icbac4ef927ddcaf0d2a74b376e5a77299529cd34
2024-06-20 17:00:06 +00:00
Qyriad 1e5f134560 de-inheritance CmdEval for InstallableValueCommand
Change-Id: I08b1702310e863d15de26dc838eb0bcb62417c10
2024-06-20 17:00:06 +00:00
Qyriad 8ba1939540 use a type alias for ProgressBar's chosen time point type
Change-Id: I621a455b1daba806fc498958aee7931fbfc55445
2024-06-20 15:24:27 +00:00
Qyriad f9594b592b extract ProgressBar declaration into its header file
Change-Id: Ica9e2ec41d99eaa196a0d535501edf45c589b2b6
2024-06-20 15:24:27 +00:00
Qyriad 3a4c21fc9e slight cleanup to ProgressBar::getStatus()
Binaries were identical before and after this commit on our machine

Change-Id: I6f8bfbe3298d6c5f43d5702c7a1e05cb180226cc
2024-06-20 15:24:27 +00:00
Ilya K 697ef65c14 Merge "BrotliDecompressionSource: don't bail out too early" into main 2024-06-20 07:06:52 +00:00
Ilya K 7d52d74bbe BrotliDecompressionSource: don't bail out too early
If we've consumed the entire input, that doesn't actually mean we're
done decompressing - there might be more output left. This worked (?)
in most cases because the input and output sizes are pretty comparable,
but sometimes they're not and then things get very funny.

Change-Id: I73435a654a911b8ce25119f713b80706c5783c1b
2024-06-20 09:21:13 +03:00
jade 6c29a2a6fc Merge "libstore: fix queryValidPaths concurrency" into main 2024-06-20 05:55:08 +00:00
jade 50472aa5be libstore: remove operations that are never called by supported clients
I did a whole bunch of `git log -S` to find out exactly when all these
things were obsoleted and found the commit in which their usage was
removed, which I have added in either the error message or a comment.

I've also made *some* of the version checks into static asserts for when
we update the minimum supported protocol version.

In the end this is not a lot of code we are deleting, but it's code that
we will never have to support into the future when we build a protocol
bridge, which is why I did it. It is not in the support baseline.

Change-Id: Iea3c80795c75ea74f328cf7ede7cbedf8c41926b
2024-06-19 19:41:04 -07:00
Eelco Dolstra fb7d315411 Merge pull request #10570 from layus/shared_caches
Share evaluation caches across installables

Before:

$ rm -rf ~/.cache/nix && time -f '%E' nix build --dry-run \
  'nixpkgs#hello' \
  'nixpkgs#clang' \
  'nixpkgs#cargo' \
  'nixpkgs#rustup' \
  'nixpkgs#bear' \
  'nixpkgs#firefox' \
  'nixpkgs#git-revise' \
  'nixpkgs#hyperfine' \
  'nixpkgs#curlie' \
  'nixpkgs#xz' \
  'nixpkgs#ripgrep'
0:03.61

After:

$ rm -rf ~/.cache/nix && time -f '%E' nix build --dry-run \
  'nixpkgs#hello' \
  'nixpkgs#clang' \
  'nixpkgs#cargo' \
  'nixpkgs#rustup' \
  'nixpkgs#bear' \
  'nixpkgs#firefox' \
  'nixpkgs#git-revise' \
  'nixpkgs#hyperfine' \
  'nixpkgs#curlie' \
  'nixpkgs#xz' \
  'nixpkgs#ripgrep'
0:01.46

This could probably use a more proper benchmark...

Fixes #313

(cherry picked from commit de51e5c335865e3e0a8cccd283fec1a52cce243f)
Change-Id: I9350bebd462b6af12c51db5bf432321abfe84a16
2024-06-19 18:39:11 +00:00
eldritch horrors c55dcc6c13 filetransfer: return a Source from download()
without this we will not be able to get rid of makeDecompressionSink,
which in turn will be necessary to get rid of sourceToSink (since the
libarchive archive wrapper *must* be a Source due to api limitations)

Change-Id: Iccd3d333ba2cbcab49cb5a1d3125624de16bce27
2024-06-19 10:50:12 +00:00
eldritch horrors 11f4a5bc7e libutil: return a source from readFile
don't consume a sink, return a source instead. the only reason to not do
this is a very slight reduction in dynamic allocations, but since we are
going to *at least* do disk io that will not be a lot of overhead anyway

Change-Id: Iae2f879ec64c3c3ac1d5310eeb6a85e696d4614a
2024-06-19 10:50:12 +00:00
eldritch horrors 67f778670c libutil: add makeDecompressionSource
Change-Id: Iac7f24d79e24417436b9b5cbefd6af051aeea0a6
2024-06-19 10:50:12 +00:00
eldritch horrors 3425e90d76 libstore: BinaryCacheStore::getFile{ -> Contents}
if we want have getFile return a source instead of consuming a sink
we'll have to disambiguate this overload another way, eg like this.

Change-Id: Ia26de2020c309a37e7ccc3775c1ad1f32e0a778b
2024-06-19 10:50:12 +00:00
alois31 fed34594d8 Merge "libfetchers: represent unfetched submodules consistently" into main 2024-06-19 07:08:19 +00:00
jade 85f282ef57 Merge changes Id0e651e4,I0ed20da8,I76bd6d22,I5d8ffb7b into main
* changes:
  store: fix null reference from DerivationGoal::waiteeDone
  libmain: fix UB in verbosity assignment
  build: make UBSan work :)
  libexpr: fix accessing uninitialized values and fix pure-eval docs
2024-06-19 03:31:47 +00:00
Qyriad b338435b75 Merge "refactor lambda formals handling" into main 2024-06-19 03:13:46 +00:00
jade c897fba787 store: fix null reference from DerivationGoal::waiteeDone
This happened during a PathSubstitutionGoal of a .drv file:

substitution of '/tmp/jade/nix-test/ca/eval-store/store/1lj7lsq5y0f25mfbnq6d3zd0bw5ay33n-dependencies-input-2.drv'

What happened here is that since PathSubstitutionGoal is not a
DerivationGoal, in production builds, the UB was not caught, since it
would early-exit from failing a dynamic_cast to DerivationGoal * on the
very next line, but before the null reference was ever used.

This was nonetheless UB. The fix should be to just rearrange the two
lines; I don't think there is a further bug there, since *substituting a
.drv* **necessarily** means you cannot have the representation of
the derivation as would be necessary for drv to not be null there.

Test failure:

++(eval-store.sh:12) _RR_TRACE_DIR=/home/jade/.local/share/rr rr record -- nix build -f dependencies.nix --eval-store /tmp/jade/nix-test/ca/eval-store/eval-store -o /tmp/jade/nix-test/ca/eval-store/result
don't know how to build these paths:
  /tmp/jade/nix-test/ca/eval-store/store/6y51mf0p57ggipgab6hdjabbvplzsicq-dependencies-top.drv
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/8027afyvqb87y1sf5xhdkqsflqn1ziy8-dependencies.builder0.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/7r5pqyncvfgrryf9gzy1z56z3xigi61x-builder-dependencies-input-0.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/nhmgm87zlqy3ks96dxrn7l37b72azi99-builder-dependencies-input-1.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/nq4qa2j6y8ajqazlfq6h46ck637my1n6-builder-dependencies-input-2.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/6vh0vna9l5afck01y7iaks3hm9ikwqyj-builder-fod-input.sh' to 'local'...
building '/tmp/jade/nix-test/ca/eval-store/store/gy91pqymf2nc5v7ld1bad94xpwxdi25s-dependencies-input-0.drv'...
building '/tmp/jade/nix-test/ca/eval-store/store/w7wlkjx97ivmnrymkac5av3nyp94hzvq-dependencies-input-1.drv'...
../src/libstore/build/derivation-goal.cc:1556:22: runtime error: reference binding to null pointer of type 'Derivation'
    0 0x734ba59a6886 in nix::DerivationGoal::waiteeDone(std::shared_ptr<nix::Goal>, nix::Goal::ExitCode) /home/jade/lix/lix2/build/src/libstore/build/derivation-goal.cc:1556:12
    1 0x734ba59c0962 in nix::Goal::amDone(nix::Goal::ExitCode, std::optional<nix::Error>) /home/jade/lix/lix2/build/src/libstore/build/goal.cc:95:25
    2 0x734ba5a1c44a in nix::PathSubstitutionGoal::done(nix::Goal::ExitCode, nix::BuildResult::Status, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>) /home/jade/lix/lix2/build/src/libstore/build/substitution-goal.cc:38:5
    3 0x734ba5a1b454 in nix::PathSubstitutionGoal::init() /home/jade/lix/lix2/build/src/libstore/build/substitution-goal.cc:56:9
    4 0x734ba5a2a6c6 in nix::Worker::run(std::set<std::shared_ptr<nix::Goal>, nix::CompareGoalPtrs, std::allocator<std::shared_ptr<nix::Goal>>> const&) /home/jade/lix/lix2/build/src/libstore/build/worker.cc:320:23
    5 0x734ba59b93d8 in nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath>> const&, nix::BuildMode, std::shared_ptr<nix::Store>) /home/jade/lix/lix2/build/src/libstore/build/entry-points.cc:60:12
    6 0x734ba663c107 in nix::Installable::build2(nix::ref<nix::Store>, nix::ref<nix::Store>, nix::Realise, std::vector<nix::ref<nix::Installable>, std::allocator<nix::ref<nix::Installable>>> const&, nix::BuildMode) /home/jade/lix/lix2/build/src/libcmd/installables.cc:637:36

Change-Id: Id0e651e480bebf6356733b01bc639e9bb59c7bd0
2024-06-18 19:26:54 -07:00
jade f2fff1faa4 libmain: fix UB in verbosity assignment
This was generating an out-of-range verbosity value. We should just
process it as an int and then convert to verbosity with a clamping
function, which trivially avoids any domain type violations.

Change-Id: I0ed20da8e1496a1225ff3008b76827d99265d404
2024-06-18 19:26:54 -07:00
jade e0a3a5f226 build: make UBSan work :)
This is really just a question of turning off the production sanitizer
configuration so we get nice diagnostics. Not much else to say.

Change-Id: I76bd6d225320056ed95bd89955f00beff2db0d2f
2024-06-18 19:25:58 -07:00
jade 1eef1927b6 libexpr: fix accessing uninitialized values and fix pure-eval docs
We got UBSan working on Lix, so we of course immediately found a bug and
some definitely nonsense behaviour.

Accessing `pureEval` or `restrictEval` from a default setting value is
nonsense, since they would never be actually set by the time that value
is set so they are not going to do anything. The configuration is not
applied in an initializer (and even if it were, it's not going to be in
the right order).

After looking into *that*, we hunted down what actually was applying
these, since clearly this code did not do anything. The EvalState
constructor should have a "search path added and removed here :)" sign
on it, because that's where it is done. We added an explicit
initialization of the optional in there because it was otherwise unclear
why pureEval also has the search path to allowed paths setup code run.

We then realized that the `pureEval` documentation was *also* bogus, and
we rewrote it. In so doing, we realized that we forgot to file a bug to
make `builtins.storePath` work in pure eval mode, so we filed one of
those: lix-project/lix#402

Yaks have been thoroughly shorn.

UBSan report:

    ../src/libexpr/eval-settings.cc:66:10: runtime error: member call on address 0x752fa9a13060 which does not point to an object of type 'nix::BaseSetting<b
    ool>'
    0x752fa9a13060: note: object has invalid vptr
     00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00
                  ^~~~~~~~~~~~~~~~~~~~~~~
                  invalid vptr
        0 0x752fa95106a6 in nix::EvalSettings::getDefaultNixPath[abi:cxx11]() /home/jade/lix/lix2/build/src/libexpr/eval-settings.cc:66:10
        1 0x752fa950e420 in nix::EvalSettings::EvalSettings() /home/jade/lix/lix2/build/src/libexpr/eval-settings.hh:36:15
        2 0x752fa9469f1f in __cxx_global_var_init.50 /home/jade/lix/lix2/build/src/libexpr/eval-settings.cc:98:14
        3 0x752fa9469f1f in _GLOBAL__sub_I_eval_settings.cc /home/jade/lix/lix2/build/src/libexpr/eval-settings.cc
        4 0x752fabbd308d in call_init (/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2+0x508d) (BuildId: a5b8228edc9f16078ac3c894af964eeb990ecb4c)
        5 0x752fabbd317b in _dl_init (/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2+0x517b) (BuildId: a5b8228edc9f16078ac3c894af964eeb990ecb4c)
        6 0x752fabbe9c2f in _dl_start_user (/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2+0x1bc2f) (BuildId: a5b8228edc9f16078ac3c894af964eeb990ecb4c)

Change-Id: I5d8ffb7bfbe24b6584020ac74eed93d9f2e6d111
2024-06-18 19:25:35 -07:00
Qyriad 54d2c189ae refactor lambda formals handling
Change-Id: Iebffd5436109da270ee870670a20f5ee7db9a204
2024-06-19 00:57:56 +00:00
eldritch horrors a960576f58 libutil: BrotliDecompression{Sink -> Source}
Change-Id: I9579dd08f7bd0f927bde9d3128515b0cee15f320
2024-06-19 00:54:06 +00:00