functional-ca-repl flakey test? #690

Open
opened 2025-02-22 01:44:46 +00:00 by jade · 2 comments
Owner

Have not closely inspected this, but it did fail in CI for seemingly no reason on https://gerrit.lix.systems/c/lix/+/2613: https://buildbot.lix.systems/#/builders/457/builds/2422/steps/1/logs/stdio

Leads me to believe it is probably flakey given the change had nothing to do with it:

lix> Listing only the last 100 lines from a long log.
lix> ++(repl.sh:63) shift
lix> ++(repl.sh:64) local 'commands=
lix> drvPath
lix> '
lix> ++(repl.sh:64) shift
lix> +++(repl.sh:69) printf %s '".*-simple.drv"'
lix> +++(repl.sh:69) awk 1 ORS=TEST_NEWLINE
lix> +++(repl.sh:69) sed 's/TEST_NEWLINE$//'
lix> ++(repl.sh:69) local 'expectedResponse=".*-simple.drv"'
lix> ++(repl.sh:69) shift
lix> +++(repl.sh:71) nix repl --file /build/source/build/tests/functional/simple.nix --experimental-features ca-derivations
lix> +++(repl.sh:71) stripColors
lix> +++(repl.sh:59) sed -E 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g'
lix> +++(repl.sh:71) awk 1 ORS=TEST_NEWLINE
lix> ++(repl.sh:71) local 'response=warning: you don'\''t have Internet access; disabling some network-dependent featuresTEST_NEWLINELix 2.93.0-devTEST_NEWLINEType :? for help.TEST_NEWLINELoading installable '\'''\''...TEST_NEWLINEAdded 17 variables.TEST_NEWLINETEST_NEWLINE"/build/nix-test/ca/repl/store/n2fbzncxhgp5w14jamvdhdh173x5pbfc-simple.drv"TEST_NEWLINETEST_NEWLINETEST_NEWLINETEST_NEWLINE'
lix> ++(repl.sh:72) grepQuiet --basic-regexp -s '".*-simple.drv"'
lix> ++(repl.sh:72) printf %s 'warning: you don'\''t have Internet access; disabling some network-dependent featuresTEST_NEWLINELix 2.93.0-devTEST_NEWLINEType :? for help.TEST_NEWLINELoading installable '\'''\''...TEST_NEWLINEAdded 17 variables.TEST_NEWLINETEST_NEWLINE"/build/nix-test/ca/repl/store/n2fbzncxhgp5w14jamvdhdh173x5pbfc-simple.drv"TEST_NEWLINETEST_NEWLINETEST_NEWLINETEST_NEWLINE'
lix> ++(/build/source/build/tests/functional/common/vars-and-functions.sh:273) grep --basic-regexp -s '".*-simple.drv"'
lix> ++(repl.sh:127) testReplResponse '
lix> drvPath
lix> ' '".*-simple.drv"' --file /build/source/build/tests/functional/simple.nix --extra-experimental-features 'repl-flake ca-derivations'
lix> ++(repl.sh:93) testReplResponseGeneral --basic-regexp '
lix> drvPath
lix> ' '".*-simple.drv"' --file /build/source/build/tests/functional/simple.nix --extra-experimental-features 'repl-flake ca-derivations'
lix> ++(repl.sh:63) local grepMode=--basic-regexp
lix> ++(repl.sh:63) shift
lix> ++(repl.sh:64) local 'commands=
lix> drvPath
lix> '
lix> ++(repl.sh:64) shift
lix> +++(repl.sh:69) sed 's/TEST_NEWLINE$//'
lix> +++(repl.sh:69) printf %s '".*-simple.drv"'
lix> +++(repl.sh:69) awk 1 ORS=TEST_NEWLINE
lix> ++(repl.sh:69) local 'expectedResponse=".*-simple.drv"'
lix> ++(repl.sh:69) shift
lix> +++(repl.sh:71) nix repl --file /build/source/build/tests/functional/simple.nix --extra-experimental-features 'repl-flake ca-derivations'
lix> +++(repl.sh:71) stripColors
lix> +++(repl.sh:59) sed -E 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g'
lix> +++(repl.sh:71) awk 1 ORS=TEST_NEWLINE
lix> ++(repl.sh:71) local 'response=warning: you don'\''t have Internet access; disabling some network-dependent featuresTEST_NEWLINELix 2.93.0-devTEST_NEWLINEType :? for help.TEST_NEWLINELoading installable '\'''\''...TEST_NEWLINEAdded 17 variables.TEST_NEWLINETEST_NEWLINE"/build/nix-test/ca/repl/store/n2fbzncxhgp5w14jamvdhdh173x5pbfc-simple.drv"TEST_NEWLINETEST_NEWLINETEST_NEWLINETEST_NEWLINE'
lix> ++(repl.sh:72) grepQuiet --basic-regexp -s '".*-simple.drv"'
lix> ++(repl.sh:72) printf %s 'warning: you don'\''t have Internet access; disabling some network-dependent featuresTEST_NEWLINELix 2.93.0-devTEST_NEWLINEType :? for help.TEST_NEWLINELoading installable '\'''\''...TEST_NEWLINEAdded 17 variables.TEST_NEWLINETEST_NEWLINE"/build/nix-test/ca/repl/store/n2fbzncxhgp5w14jamvdhdh173x5pbfc-simple.drv"TEST_NEWLINETEST_NEWLINETEST_NEWLINETEST_NEWLINE'
lix> ++(/build/source/build/tests/functional/common/vars-and-functions.sh:273) grep --basic-regexp -s '".*-simple.drv"'
lix> ++(repl.sh:130) mkdir -p flake
lix> ++(repl.sh:130) cat
lix> ++(repl.sh:142) testReplResponse '
lix> foo + baz
lix> ' 3 ./flake ./flake#bar --experimental-features 'flakes repl-flake'
lix> ++(repl.sh:93) testReplResponseGeneral --basic-regexp '
lix> foo + baz
lix> ' 3 ./flake ./flake#bar --experimental-features 'flakes repl-flake'
lix> ++(repl.sh:63) local grepMode=--basic-regexp
lix> ++(repl.sh:63) shift
lix> ++(repl.sh:64) local 'commands=
lix> foo + baz
lix> '
lix> ++(repl.sh:64) shift
lix> +++(repl.sh:69) awk 1 ORS=TEST_NEWLINE
lix> +++(repl.sh:69) printf %s 3
lix> +++(repl.sh:69) sed 's/TEST_NEWLINE$//'
lix> ++(repl.sh:69) local expectedResponse=3
lix> ++(repl.sh:69) shift
lix> +++(repl.sh:71) nix repl ./flake ./flake#bar --experimental-features 'flakes repl-flake'
lix> +++(repl.sh:71) stripColors
lix> +++(repl.sh:59) sed -E 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g'
lix> +++(repl.sh:71) awk 1 ORS=TEST_NEWLINE
lix> ++(repl.sh:71) local 'response=warning: you don'\''t have Internet access; disabling some network-dependent featuresTEST_NEWLINELix 2.93.0-devTEST_NEWLINEType :? for help.TEST_NEWLINEfetching path input '\''path:/build/nix-test/ca/repl/flake'\''TEST_NEWLINEfetching path input '\''path:/build/nix-test/ca/repl/flake'\''TEST_NEWLINELoading installable '\''path:/build/nix-test/ca/repl/flake#'\''...TEST_NEWLINEAdded 3 variables.TEST_NEWLINELoading installable '\''path:/build/nix-test/ca/repl/flake#bar'\''...TEST_NEWLINEAdded 1 variables.TEST_NEWLINETEST_NEWLINE3TEST_NEWLINETEST_NEWLINETEST_NEWLINETEST_NEWLINE'
lix> ++(repl.sh:72) grepQuiet --basic-regexp -s 3
lix> ++(repl.sh:72) printf %s 'warning: you don'\''t have Internet access; disabling some network-dependent featuresTEST_NEWLINELix 2.93.0-devTEST_NEWLINEType :? for help.TEST_NEWLINEfetching path input '\''path:/build/nix-test/ca/repl/flake'\''TEST_NEWLINEfetching path input '\''path:/build/nix-test/ca/repl/flake'\''TEST_NEWLINELoading installable '\''path:/build/nix-test/ca/repl/flake#'\''...TEST_NEWLINEAdded 3 variables.TEST_NEWLINELoading installable '\''path:/build/nix-test/ca/repl/flake#bar'\''...TEST_NEWLINEAdded 1 variables.TEST_NEWLINETEST_NEWLINE3TEST_NEWLINETEST_NEWLINETEST_NEWLINETEST_NEWLINE'
lix> ++(/build/source/build/tests/functional/common/vars-and-functions.sh:273) grep --basic-regexp -s 3
lix> +++(repl.sh:160) nix repl ./flake --experimental-features 'flakes repl-flake'
lix> +++(repl.sh:156) echo changingThing
lix> +++(repl.sh:157) sleep 1
lix> +++(repl.sh:158) sed -i s/beforeChange/afterChange/ flake/flake.nix
lix> +++(repl.sh:159) echo :reload
lix> +++(repl.sh:160) echo changingThing
lix> warning: you don't have Internet access; disabling some network-dependent features
lix> Lix 2.93.0-dev
lix> Type :? for help.
lix> fetching path input 'path:/build/nix-test/ca/repl/flake'
lix> Loading installable 'path:/build/nix-test/ca/repl/flake#'...
lix> Added 3 variables.
lix> fetching path input 'path:/build/nix-test/ca/repl/flake'
lix> Loading installable 'path:/build/nix-test/ca/repl/flake#'...
lix> Added 3 variables.
lix> ++(repl.sh:156) replResult='"afterChange"
lix> "afterChange"'
lix> ++(repl.sh:157) echo '"afterChange"
lix> "afterChange"'
lix> ++(repl.sh:157) grepQuiet -s beforeChange
lix> ++(/build/source/build/tests/functional/common/vars-and-functions.sh:273) grep -s beforeChange
lix> +++(repl.sh:157) onError
lix> +++(/build/source/build/tests/functional/common/vars-and-functions.sh:244) set +x
lix> repl.sh: test failed at:
lix>   source in repl.sh:157
lix>   main in repl.sh:5
Have not closely inspected this, but it did fail in CI for seemingly no reason on https://gerrit.lix.systems/c/lix/+/2613: https://buildbot.lix.systems/#/builders/457/builds/2422/steps/1/logs/stdio Leads me to believe it is probably flakey given the change had nothing to do with it: ``` lix> Listing only the last 100 lines from a long log. lix> ++(repl.sh:63) shift lix> ++(repl.sh:64) local 'commands= lix> drvPath lix> ' lix> ++(repl.sh:64) shift lix> +++(repl.sh:69) printf %s '".*-simple.drv"' lix> +++(repl.sh:69) awk 1 ORS=TEST_NEWLINE lix> +++(repl.sh:69) sed 's/TEST_NEWLINE$//' lix> ++(repl.sh:69) local 'expectedResponse=".*-simple.drv"' lix> ++(repl.sh:69) shift lix> +++(repl.sh:71) nix repl --file /build/source/build/tests/functional/simple.nix --experimental-features ca-derivations lix> +++(repl.sh:71) stripColors lix> +++(repl.sh:59) sed -E 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' lix> +++(repl.sh:71) awk 1 ORS=TEST_NEWLINE lix> ++(repl.sh:71) local 'response=warning: you don'\''t have Internet access; disabling some network-dependent featuresTEST_NEWLINELix 2.93.0-devTEST_NEWLINEType :? for help.TEST_NEWLINELoading installable '\'''\''...TEST_NEWLINEAdded 17 variables.TEST_NEWLINETEST_NEWLINE"/build/nix-test/ca/repl/store/n2fbzncxhgp5w14jamvdhdh173x5pbfc-simple.drv"TEST_NEWLINETEST_NEWLINETEST_NEWLINETEST_NEWLINE' lix> ++(repl.sh:72) grepQuiet --basic-regexp -s '".*-simple.drv"' lix> ++(repl.sh:72) printf %s 'warning: you don'\''t have Internet access; disabling some network-dependent featuresTEST_NEWLINELix 2.93.0-devTEST_NEWLINEType :? for help.TEST_NEWLINELoading installable '\'''\''...TEST_NEWLINEAdded 17 variables.TEST_NEWLINETEST_NEWLINE"/build/nix-test/ca/repl/store/n2fbzncxhgp5w14jamvdhdh173x5pbfc-simple.drv"TEST_NEWLINETEST_NEWLINETEST_NEWLINETEST_NEWLINE' lix> ++(/build/source/build/tests/functional/common/vars-and-functions.sh:273) grep --basic-regexp -s '".*-simple.drv"' lix> ++(repl.sh:127) testReplResponse ' lix> drvPath lix> ' '".*-simple.drv"' --file /build/source/build/tests/functional/simple.nix --extra-experimental-features 'repl-flake ca-derivations' lix> ++(repl.sh:93) testReplResponseGeneral --basic-regexp ' lix> drvPath lix> ' '".*-simple.drv"' --file /build/source/build/tests/functional/simple.nix --extra-experimental-features 'repl-flake ca-derivations' lix> ++(repl.sh:63) local grepMode=--basic-regexp lix> ++(repl.sh:63) shift lix> ++(repl.sh:64) local 'commands= lix> drvPath lix> ' lix> ++(repl.sh:64) shift lix> +++(repl.sh:69) sed 's/TEST_NEWLINE$//' lix> +++(repl.sh:69) printf %s '".*-simple.drv"' lix> +++(repl.sh:69) awk 1 ORS=TEST_NEWLINE lix> ++(repl.sh:69) local 'expectedResponse=".*-simple.drv"' lix> ++(repl.sh:69) shift lix> +++(repl.sh:71) nix repl --file /build/source/build/tests/functional/simple.nix --extra-experimental-features 'repl-flake ca-derivations' lix> +++(repl.sh:71) stripColors lix> +++(repl.sh:59) sed -E 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' lix> +++(repl.sh:71) awk 1 ORS=TEST_NEWLINE lix> ++(repl.sh:71) local 'response=warning: you don'\''t have Internet access; disabling some network-dependent featuresTEST_NEWLINELix 2.93.0-devTEST_NEWLINEType :? for help.TEST_NEWLINELoading installable '\'''\''...TEST_NEWLINEAdded 17 variables.TEST_NEWLINETEST_NEWLINE"/build/nix-test/ca/repl/store/n2fbzncxhgp5w14jamvdhdh173x5pbfc-simple.drv"TEST_NEWLINETEST_NEWLINETEST_NEWLINETEST_NEWLINE' lix> ++(repl.sh:72) grepQuiet --basic-regexp -s '".*-simple.drv"' lix> ++(repl.sh:72) printf %s 'warning: you don'\''t have Internet access; disabling some network-dependent featuresTEST_NEWLINELix 2.93.0-devTEST_NEWLINEType :? for help.TEST_NEWLINELoading installable '\'''\''...TEST_NEWLINEAdded 17 variables.TEST_NEWLINETEST_NEWLINE"/build/nix-test/ca/repl/store/n2fbzncxhgp5w14jamvdhdh173x5pbfc-simple.drv"TEST_NEWLINETEST_NEWLINETEST_NEWLINETEST_NEWLINE' lix> ++(/build/source/build/tests/functional/common/vars-and-functions.sh:273) grep --basic-regexp -s '".*-simple.drv"' lix> ++(repl.sh:130) mkdir -p flake lix> ++(repl.sh:130) cat lix> ++(repl.sh:142) testReplResponse ' lix> foo + baz lix> ' 3 ./flake ./flake#bar --experimental-features 'flakes repl-flake' lix> ++(repl.sh:93) testReplResponseGeneral --basic-regexp ' lix> foo + baz lix> ' 3 ./flake ./flake#bar --experimental-features 'flakes repl-flake' lix> ++(repl.sh:63) local grepMode=--basic-regexp lix> ++(repl.sh:63) shift lix> ++(repl.sh:64) local 'commands= lix> foo + baz lix> ' lix> ++(repl.sh:64) shift lix> +++(repl.sh:69) awk 1 ORS=TEST_NEWLINE lix> +++(repl.sh:69) printf %s 3 lix> +++(repl.sh:69) sed 's/TEST_NEWLINE$//' lix> ++(repl.sh:69) local expectedResponse=3 lix> ++(repl.sh:69) shift lix> +++(repl.sh:71) nix repl ./flake ./flake#bar --experimental-features 'flakes repl-flake' lix> +++(repl.sh:71) stripColors lix> +++(repl.sh:59) sed -E 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' lix> +++(repl.sh:71) awk 1 ORS=TEST_NEWLINE lix> ++(repl.sh:71) local 'response=warning: you don'\''t have Internet access; disabling some network-dependent featuresTEST_NEWLINELix 2.93.0-devTEST_NEWLINEType :? for help.TEST_NEWLINEfetching path input '\''path:/build/nix-test/ca/repl/flake'\''TEST_NEWLINEfetching path input '\''path:/build/nix-test/ca/repl/flake'\''TEST_NEWLINELoading installable '\''path:/build/nix-test/ca/repl/flake#'\''...TEST_NEWLINEAdded 3 variables.TEST_NEWLINELoading installable '\''path:/build/nix-test/ca/repl/flake#bar'\''...TEST_NEWLINEAdded 1 variables.TEST_NEWLINETEST_NEWLINE3TEST_NEWLINETEST_NEWLINETEST_NEWLINETEST_NEWLINE' lix> ++(repl.sh:72) grepQuiet --basic-regexp -s 3 lix> ++(repl.sh:72) printf %s 'warning: you don'\''t have Internet access; disabling some network-dependent featuresTEST_NEWLINELix 2.93.0-devTEST_NEWLINEType :? for help.TEST_NEWLINEfetching path input '\''path:/build/nix-test/ca/repl/flake'\''TEST_NEWLINEfetching path input '\''path:/build/nix-test/ca/repl/flake'\''TEST_NEWLINELoading installable '\''path:/build/nix-test/ca/repl/flake#'\''...TEST_NEWLINEAdded 3 variables.TEST_NEWLINELoading installable '\''path:/build/nix-test/ca/repl/flake#bar'\''...TEST_NEWLINEAdded 1 variables.TEST_NEWLINETEST_NEWLINE3TEST_NEWLINETEST_NEWLINETEST_NEWLINETEST_NEWLINE' lix> ++(/build/source/build/tests/functional/common/vars-and-functions.sh:273) grep --basic-regexp -s 3 lix> +++(repl.sh:160) nix repl ./flake --experimental-features 'flakes repl-flake' lix> +++(repl.sh:156) echo changingThing lix> +++(repl.sh:157) sleep 1 lix> +++(repl.sh:158) sed -i s/beforeChange/afterChange/ flake/flake.nix lix> +++(repl.sh:159) echo :reload lix> +++(repl.sh:160) echo changingThing lix> warning: you don't have Internet access; disabling some network-dependent features lix> Lix 2.93.0-dev lix> Type :? for help. lix> fetching path input 'path:/build/nix-test/ca/repl/flake' lix> Loading installable 'path:/build/nix-test/ca/repl/flake#'... lix> Added 3 variables. lix> fetching path input 'path:/build/nix-test/ca/repl/flake' lix> Loading installable 'path:/build/nix-test/ca/repl/flake#'... lix> Added 3 variables. lix> ++(repl.sh:156) replResult='"afterChange" lix> "afterChange"' lix> ++(repl.sh:157) echo '"afterChange" lix> "afterChange"' lix> ++(repl.sh:157) grepQuiet -s beforeChange lix> ++(/build/source/build/tests/functional/common/vars-and-functions.sh:273) grep -s beforeChange lix> +++(repl.sh:157) onError lix> +++(/build/source/build/tests/functional/common/vars-and-functions.sh:244) set +x lix> repl.sh: test failed at: lix> source in repl.sh:157 lix> main in repl.sh:5 ```
Member

This issue was mentioned on Gerrit on the following CLs:

  • comment in cl/2613 ("deps: backport capnproto promise nodiscard PR")
<!-- GERRIT_LINKBOT: {"cls": [{"backlink": "https://gerrit.lix.systems/c/lix/+/2613", "number": 2613, "kind": "comment"}], "cl_meta": {"2613": {"change_title": "deps: backport capnproto promise nodiscard PR"}}} --> This issue was mentioned on Gerrit on the following CLs: * comment in [cl/2613](https://gerrit.lix.systems/c/lix/+/2613) ("deps: backport capnproto promise nodiscard PR")
Owner

we looked at this and it's almost certainly a load-induced race. the test checks whether the repl actually reloads files when commanded by writing one thing to a file, loading it, printing it, sleeping a while, writing something else, reloading, and printing again. that's clearly jank and should be done with the repl testing framework instead of relying on sleep-based synchronization

we looked at this and it's almost certainly a load-induced race. the test checks whether the repl actually reloads files when commanded by writing one thing to a file, loading it, printing it, sleeping a while, writing something else, reloading, and printing again. that's clearly jank and should be done with the repl testing framework instead of relying on sleep-based synchronization
Sign in to join this conversation.
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#690
No description provided.