`--keep-failed` across remote builders does not work #933

New issue

Open

opened 2025-07-26 19:56:47 +00:00 by raito · 1 comment

raito commented

2025-07-26 19:56:47 +00:00

Owner

Describe the bug

--keep-failed across a remote builder historically kept the failed directory on the remote builder.

This is not the case anymore.

Steps To Reproduce

Build a failing derivation with keep failed with -j0.
SSH on the remote server and search for it in /tmp or /nix/var/nix/builds.

Expected behavior

keep-failed should be transfered along.
In case of remote building, it's better to add a nicer message for this to reduce confusion from users.

`nix --version` output

Lix 2.94.0 57b1c289b5.

Additional context

I may fix it once I have some time.

## Describe the bug `--keep-failed` across a remote builder historically kept the failed directory on the remote builder. This is not the case anymore. ## Steps To Reproduce 1. Build a failing derivation with keep failed with -j0. 2. SSH on the remote server and search for it in /tmp or /nix/var/nix/builds. ## Expected behavior - `keep-failed` should be transfered along. - In case of remote building, it's better to add a nicer message for this to reduce confusion from users. ## `nix --version` output Lix 2.94.0 57b1c289b59240d8cc34adc4c4c59bb0042c612d. ## Additional context I may fix it once I have some time.

raito added the

bug

label

2025-07-26 19:56:47 +00:00

pennae commented

2025-07-27 12:15:57 +00:00

Owner

ssh-ng doesn't do this fully on purpose, though with a todo to fix it at some point:

     kj::Promise<Result<void>> setOptions(RemoteStore::Connection & conn) override
     {
         /* TODO Add a way to explicitly ask for some options to be
            forwarded. One option: A way to query the daemon for its
            settings, and then a series of params to SSHStore like
            forward-cores or forward-overridden-cores that only
            override the requested settings.
         */
         return {result::success()};
     };

on ssh cannot do this when building through a daemon, only through a local store:

     auto getBuildSettings = [&]() {
         // FIXME: changing options here doesn't work if we're
         // building through the daemon.
         verbosity = lvlError;
         settings.keepLog.override(false);
         settings.useSubstitutes.override(false);
         settings.maxSilentTime.override(readInt(in));
         settings.buildTimeout.override(readInt(in));
         if (GET_PROTOCOL_MINOR(clientVersion) >= 2)
             settings.maxLogSize.override(readNum<unsigned long>(in));
         if (GET_PROTOCOL_MINOR(clientVersion) >= 3) {
             auto nrRepeats = readInt(in);
             if (nrRepeats != 0) {
                 throw Error("client requested repeating builds, but this is not currently implemented");
             }
             // Ignore 'enforceDeterminism'. It used to be true by
             // default, but also only never had any effect when
             // `nrRepeats == 0`.  We have already asserted that
             // `nrRepeats` in fact is 0, so we can safely ignore this
             // without doing something other than what the client
             // asked for.
             readInt(in);
             settings.runDiffHook.override(true);
         }
         if (GET_PROTOCOL_MINOR(clientVersion) >= 7) {
             settings.keepFailed.override((bool) readInt(in));
         }
     };

this is in theory fixable by sending an options packet before every build, but then the daemon would just reject the build anyway because some of the setting nix-store overrides unconditionally are trusted settings.

we're going to file this under WONTFIX before rpc lands

`ssh-ng` doesn't do this fully on purpose, though with a todo to fix it at some point: https://git.lix.systems/lix-project/lix/src/commit/ef94901156c0c688ffffaa2c5caf1498119f01d4/lix/libstore/ssh-store.cc#L110-L119 on `ssh` cannot do this when building through a daemon, only through a local store: https://git.lix.systems/lix-project/lix/src/commit/ef94901156c0c688ffffaa2c5caf1498119f01d4/lix/legacy/nix-store.cc#L915-L943 this is in theory fixable by sending an options packet before every build, but then the daemon would just reject the build anyway because some of the setting nix-store overrides unconditionally are trusted settings. we're going to file this under WONTFIX before rpc lands