(impossible?) spurious test failure of hydraJobs.tests.nixpkgsLibTests.aarch64-darwin #331

Open
opened 2024-05-19 15:07:42 +00:00 by puck · 3 comments
Owner
building '/nix/store/hcn5zi261krdp3dzbx9z8hmiplb21y68-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv' on 'ssh-ng://m1@build01.aarch64-darwin.lix.systems'...
copying 0 paths...
building '/nix/store/hcn5zi261krdp3dzbx9z8hmiplb21y68-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv'...
nixpkgs-lib-tests-nix> Running lib/tests/modules.sh
nixpkgs-lib-tests-nix> ====== module tests ======
nixpkgs-lib-tests-nix> 230 Pass
nixpkgs-lib-tests-nix> 0 Fail
nixpkgs-lib-tests-nix> Running lib/tests/filesystem.sh
nixpkgs-lib-tests-nix> tests ok
nixpkgs-lib-tests-nix> Running lib/tests/sources.sh
nixpkgs-lib-tests-nix> tests ok
nixpkgs-lib-tests-nix> Running lib/fileset/tests.sh
nixpkgs-lib-tests-nix> Warning: Cannot check for paths not getting read since the inotifywait command (from the inotify-tools package) is not available
nixpkgs-lib-tests-nix> error:
nixpkgs-lib-tests-nix>        … from call site
nixpkgs-lib-tests-nix>          at «string»:11:3:
nixpkgs-lib-tests-nix>            10|   with lib.fileset;
nixpkgs-lib-tests-nix>            11|  (toSource { root = ./.; fileset = gitTrackedWith { } ./.; })
nixpkgs-lib-tests-nix>              |   ^
nixpkgs-lib-tests-nix>        … while calling 'toSource'
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/default.nix:343:14:
nixpkgs-lib-tests-nix>           342|   */
nixpkgs-lib-tests-nix>           343|   toSource = {
nixpkgs-lib-tests-nix>              |              ^
nixpkgs-lib-tests-nix>           344|     /*
nixpkgs-lib-tests-nix>        … while calling the 'seq' builtin
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/default.nix:413:7:
nixpkgs-lib-tests-nix>           412|     else
nixpkgs-lib-tests-nix>           413|       seq sourceFilter
nixpkgs-lib-tests-nix>              |       ^
nixpkgs-lib-tests-nix>           414|       cleanSourceWith {
nixpkgs-lib-tests-nix>        … from call site
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/default.nix:376:22:
nixpkgs-lib-tests-nix>           375|       filesetFilesystemRoot = (splitRoot fileset._internalBase).root;
nixpkgs-lib-tests-nix>           376|       sourceFilter = _toSourceFilter fileset;
nixpkgs-lib-tests-nix>              |                      ^
nixpkgs-lib-tests-nix>           377|     in
nixpkgs-lib-tests-nix>        … while calling '_toSourceFilter'
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:398:21:
nixpkgs-lib-tests-nix>           397|   # Type: fileset -> (String -> String -> Bool)
nixpkgs-lib-tests-nix>           398|   _toSourceFilter = fileset:
nixpkgs-lib-tests-nix>              |                     ^
nixpkgs-lib-tests-nix>           399|     let
nixpkgs-lib-tests-nix>        … while evaluating a branch condition
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:486:5:
nixpkgs-lib-tests-nix>           485|     # This also forces the tree before returning the filter, leads to earlier error messages
nixpkgs-lib-tests-nix>           486|     if fileset._internalIsEmptyWithoutBase || tree == null then
nixpkgs-lib-tests-nix>              |     ^
nixpkgs-lib-tests-nix>           487|       empty
nixpkgs-lib-tests-nix>        … in the right operand of the OR (||) operator
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:486:44:
nixpkgs-lib-tests-nix>           485|     # This also forces the tree before returning the filter, leads to earlier error messages
nixpkgs-lib-tests-nix>           486|     if fileset._internalIsEmptyWithoutBase || tree == null then
nixpkgs-lib-tests-nix>              |                                            ^
nixpkgs-lib-tests-nix>           487|       empty
nixpkgs-lib-tests-nix>        … from call site
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:402:14:
nixpkgs-lib-tests-nix>           401|       # which has the effect that they aren't included in the result
nixpkgs-lib-tests-nix>           402|       tree = _normaliseTreeFilter fileset._internalBase fileset._internalTree;
nixpkgs-lib-tests-nix>              |              ^
nixpkgs-lib-tests-nix>           403|
nixpkgs-lib-tests-nix>        … while calling '_normaliseTreeFilter'
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:274:32:
nixpkgs-lib-tests-nix>           273|   */
nixpkgs-lib-tests-nix>           274|   _normaliseTreeFilter = path: tree:
nixpkgs-lib-tests-nix>              |                                ^
nixpkgs-lib-tests-nix>           275|     if tree == "directory" || isAttrs tree then
nixpkgs-lib-tests-nix>        … while evaluating a branch condition
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:275:5:
nixpkgs-lib-tests-nix>           274|   _normaliseTreeFilter = path: tree:
nixpkgs-lib-tests-nix>           275|     if tree == "directory" || isAttrs tree then
nixpkgs-lib-tests-nix>              |     ^
nixpkgs-lib-tests-nix>           276|       let
nixpkgs-lib-tests-nix>        … in the left operand of the OR (||) operator
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:275:28:
nixpkgs-lib-tests-nix>           274|   _normaliseTreeFilter = path: tree:
nixpkgs-lib-tests-nix>           275|     if tree == "directory" || isAttrs tree then
nixpkgs-lib-tests-nix>              |                            ^
nixpkgs-lib-tests-nix>           276|       let
nixpkgs-lib-tests-nix>        … while evaluating the attribute '_internalTree'
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:154:7:
nixpkgs-lib-tests-nix>           153|       _internalBaseComponents = components parts.subpath;
nixpkgs-lib-tests-nix>           154|       _internalTree = tree;
nixpkgs-lib-tests-nix>              |       ^
nixpkgs-lib-tests-nix>           155|
nixpkgs-lib-tests-nix>        … from call site
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:890:8:
nixpkgs-lib-tests-nix>           889|     _create localPath
nixpkgs-lib-tests-nix>           890|       (recurse storePath);
nixpkgs-lib-tests-nix>              |        ^
nixpkgs-lib-tests-nix>           891|
nixpkgs-lib-tests-nix>        … while calling 'recurse'
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:881:17:
nixpkgs-lib-tests-nix>           880|     let
nixpkgs-lib-tests-nix>           881|       recurse = focusedStorePath:
nixpkgs-lib-tests-nix>              |                 ^
nixpkgs-lib-tests-nix>           882|         mapAttrs (name: type:
nixpkgs-lib-tests-nix>        … while calling the 'mapAttrs' builtin
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:882:9:
nixpkgs-lib-tests-nix>           881|       recurse = focusedStorePath:
nixpkgs-lib-tests-nix>           882|         mapAttrs (name: type:
nixpkgs-lib-tests-nix>              |         ^
nixpkgs-lib-tests-nix>           883|           if type == "directory" then
nixpkgs-lib-tests-nix>        … while calling the 'readDir' builtin
nixpkgs-lib-tests-nix>          at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:887:12:
nixpkgs-lib-tests-nix>           886|             type
nixpkgs-lib-tests-nix>           887|         ) (builtins.readDir focusedStorePath);
nixpkgs-lib-tests-nix>              |            ^
nixpkgs-lib-tests-nix>           888|     in
nixpkgs-lib-tests-nix>        error: opening directory '/private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/test-tmp/store/k0mpzqpyqxb2jy1xqa8hga9inzvbbfs0-source': No such file or directory
nixpkgs-lib-tests-nix> test case at lib/fileset/tests.sh:1474 failed: toSource { root = ./.; fileset = gitTrackedWith { } ./.; } failed to evaluate, but it was expected to succeed

My guess is this got caught in a GC run, but this feels like part of a larger issue that we should look at, to make sure this can't happen again.

``` building '/nix/store/hcn5zi261krdp3dzbx9z8hmiplb21y68-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv' on 'ssh-ng://m1@build01.aarch64-darwin.lix.systems'... copying 0 paths... building '/nix/store/hcn5zi261krdp3dzbx9z8hmiplb21y68-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv'... nixpkgs-lib-tests-nix> Running lib/tests/modules.sh nixpkgs-lib-tests-nix> ====== module tests ====== nixpkgs-lib-tests-nix> 230 Pass nixpkgs-lib-tests-nix> 0 Fail nixpkgs-lib-tests-nix> Running lib/tests/filesystem.sh nixpkgs-lib-tests-nix> tests ok nixpkgs-lib-tests-nix> Running lib/tests/sources.sh nixpkgs-lib-tests-nix> tests ok nixpkgs-lib-tests-nix> Running lib/fileset/tests.sh nixpkgs-lib-tests-nix> Warning: Cannot check for paths not getting read since the inotifywait command (from the inotify-tools package) is not available nixpkgs-lib-tests-nix> error: nixpkgs-lib-tests-nix> … from call site nixpkgs-lib-tests-nix> at «string»:11:3: nixpkgs-lib-tests-nix> 10| with lib.fileset; nixpkgs-lib-tests-nix> 11| (toSource { root = ./.; fileset = gitTrackedWith { } ./.; }) nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> … while calling 'toSource' nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/default.nix:343:14: nixpkgs-lib-tests-nix> 342| */ nixpkgs-lib-tests-nix> 343| toSource = { nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 344| /* nixpkgs-lib-tests-nix> … while calling the 'seq' builtin nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/default.nix:413:7: nixpkgs-lib-tests-nix> 412| else nixpkgs-lib-tests-nix> 413| seq sourceFilter nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 414| cleanSourceWith { nixpkgs-lib-tests-nix> … from call site nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/default.nix:376:22: nixpkgs-lib-tests-nix> 375| filesetFilesystemRoot = (splitRoot fileset._internalBase).root; nixpkgs-lib-tests-nix> 376| sourceFilter = _toSourceFilter fileset; nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 377| in nixpkgs-lib-tests-nix> … while calling '_toSourceFilter' nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:398:21: nixpkgs-lib-tests-nix> 397| # Type: fileset -> (String -> String -> Bool) nixpkgs-lib-tests-nix> 398| _toSourceFilter = fileset: nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 399| let nixpkgs-lib-tests-nix> … while evaluating a branch condition nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:486:5: nixpkgs-lib-tests-nix> 485| # This also forces the tree before returning the filter, leads to earlier error messages nixpkgs-lib-tests-nix> 486| if fileset._internalIsEmptyWithoutBase || tree == null then nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 487| empty nixpkgs-lib-tests-nix> … in the right operand of the OR (||) operator nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:486:44: nixpkgs-lib-tests-nix> 485| # This also forces the tree before returning the filter, leads to earlier error messages nixpkgs-lib-tests-nix> 486| if fileset._internalIsEmptyWithoutBase || tree == null then nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 487| empty nixpkgs-lib-tests-nix> … from call site nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:402:14: nixpkgs-lib-tests-nix> 401| # which has the effect that they aren't included in the result nixpkgs-lib-tests-nix> 402| tree = _normaliseTreeFilter fileset._internalBase fileset._internalTree; nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 403| nixpkgs-lib-tests-nix> … while calling '_normaliseTreeFilter' nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:274:32: nixpkgs-lib-tests-nix> 273| */ nixpkgs-lib-tests-nix> 274| _normaliseTreeFilter = path: tree: nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 275| if tree == "directory" || isAttrs tree then nixpkgs-lib-tests-nix> … while evaluating a branch condition nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:275:5: nixpkgs-lib-tests-nix> 274| _normaliseTreeFilter = path: tree: nixpkgs-lib-tests-nix> 275| if tree == "directory" || isAttrs tree then nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 276| let nixpkgs-lib-tests-nix> … in the left operand of the OR (||) operator nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:275:28: nixpkgs-lib-tests-nix> 274| _normaliseTreeFilter = path: tree: nixpkgs-lib-tests-nix> 275| if tree == "directory" || isAttrs tree then nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 276| let nixpkgs-lib-tests-nix> … while evaluating the attribute '_internalTree' nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:154:7: nixpkgs-lib-tests-nix> 153| _internalBaseComponents = components parts.subpath; nixpkgs-lib-tests-nix> 154| _internalTree = tree; nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 155| nixpkgs-lib-tests-nix> … from call site nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:890:8: nixpkgs-lib-tests-nix> 889| _create localPath nixpkgs-lib-tests-nix> 890| (recurse storePath); nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 891| nixpkgs-lib-tests-nix> … while calling 'recurse' nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:881:17: nixpkgs-lib-tests-nix> 880| let nixpkgs-lib-tests-nix> 881| recurse = focusedStorePath: nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 882| mapAttrs (name: type: nixpkgs-lib-tests-nix> … while calling the 'mapAttrs' builtin nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:882:9: nixpkgs-lib-tests-nix> 881| recurse = focusedStorePath: nixpkgs-lib-tests-nix> 882| mapAttrs (name: type: nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 883| if type == "directory" then nixpkgs-lib-tests-nix> … while calling the 'readDir' builtin nixpkgs-lib-tests-nix> at /private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/lib/fileset/internal.nix:887:12: nixpkgs-lib-tests-nix> 886| type nixpkgs-lib-tests-nix> 887| ) (builtins.readDir focusedStorePath); nixpkgs-lib-tests-nix> | ^ nixpkgs-lib-tests-nix> 888| in nixpkgs-lib-tests-nix> error: opening directory '/private/tmp/nix-build-nixpkgs-lib-tests-nix-2.90.0pre20240519_73ef8cd.drv-0/test-tmp/store/k0mpzqpyqxb2jy1xqa8hga9inzvbbfs0-source': No such file or directory nixpkgs-lib-tests-nix> test case at lib/fileset/tests.sh:1474 failed: toSource { root = ./.; fileset = gitTrackedWith { } ./.; } failed to evaluate, but it was expected to succeed ``` My guess is this got caught in a GC run, but this feels like part of a larger issue that we _should_ look at, to make sure this can't happen again.
puck added the
stability
bug
Area/build-packaging
Area/store
labels 2024-05-19 15:07:42 +00:00
Owner

this has happened on multiple occasions already fwiw. it seems eminently possible that there's a race condition between GC and drv uploads?

this has happened on multiple occasions already fwiw. it seems eminently possible that there's a race condition between GC and drv uploads?
Author
Owner

I think this might be a bit more complex, because this involves the drv's build dir and seeing as ./. failed I'm a bit more worried something weirder happened..

I think this might be a bit more complex, because this involves the drv's build dir and seeing as `./.` failed I'm a bit more worried something weirder happened..
Owner

oh wait right this was darwin
didn't somefew say that darwin has really strange /tmp cleanup behavior? can't remember >,<

oh wait right this was *darwin* didn't somefew say that darwin has *really strange* /tmp cleanup behavior? can't remember >,<
jade added the
testing/flakey
label 2024-09-14 00:05:55 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#331
No description provided.