lix-project/buildbot-nix
15
0
Fork 0
Code Issues 13 Pull requests 7 Packages Projects Releases Wiki Activity

Signing paths on builder nodes is unsound with respect to gcroots #28

New issue
Open
opened 2024-10-18 22:25:28 +00:00 by jade · 1 comment
jade commented 2024-10-18 22:25:28 +00:00
Owner
Copy link

It appears that there is no gcroot on paths that are being signed on workers, which causes random CI failures: https://buildbot.lix.systems/#/buildrequests/196354

Failure:

 argv: [b'nix', b'store', b'sign', b'--store', b'ssh-ng://nix@build01.aarch64.lix.systems?ssh-key=/run/agenix/buildbot-remote-builder-key&base64-ssh-public-host-key=c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUNDNjlOWkQvemhJQi93VWI1b2RnNDZic3M1ZzhoSDJmRGwyMmJrNHFlU1c=', b'--key-file', b'/run/agenix/buildbot-signing-key', b'/nix/store/3nkf8bihgf76w0qxgbzqprn9pbl5sfkc-lix-2.91.1pre20241017_dcdeefd.drv^*']
 environment:
  BUILDBOT_DIR=/var/lib/buildbot-worker/worker
  CREDENTIALS_DIRECTORY=/run/credentials/buildbot-worker.service
  HOME=/var/lib/buildbot-worker
  INVOCATION_ID=c6072ec4d4b84054b8d7966ed7fc5a16
  JOURNAL_STREAM=8:7746687
  LANG=en_US.UTF-8
  LOCALE_ARCHIVE=/nix/store/wdrnjaw2sgqpp1w7kbd1jv5whcah0v5q-glibc-locales-2.39-52/lib/locale/locale-archive
  LOGNAME=buildbot-worker
  MASTER_URL=tcp:host=localhost:port=9989
  MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/buildbot-worker.service/memory.pressure
  MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
  PATH=/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin:/nix/store/dj1489kq00csdvxh1lzyr3kgj6f1ahc8-python3.12-twisted-24.7.0/bin:/nix/store/g0rr15zdsdbnbbcli08x3zx170j1kc68-python3.12-automat-22.10.0/bin:/nix/store/0l63mqr7qx1asjxfz86j4zbs4758gn98-git-2.46.0/bin:/nix/store/2wwfdill3872xircv1vwmlj8gxghg5f9-openssh-9.8p1/bin:/nix/store/cpad9d9fdvn1fazzmk9q3a06hlmwamx6-lix-2.92.0-devpre20241005_ed9b7f4/bin:/nix/store/w5fz6climh0a26k4vjkdm8l5ggzjbqm3-nix-eval-jobs-2.24.0/bin:/nix/store/izpf49b74i15pcr9708s3xdwyqs4jxwl-bash-5.2p32/bin:/nix/store/0kg70swgpg45ipcz3pr2siidq9fn6d77-coreutils-9.5/bin:/nix/store/fnq94lw19pnwdl9p8zhfrad4jmlgxlbr-findutils-4.10.0/bin:/nix/store/vsyc8jhsr4d9lm2r8yqq9n3j4i66inlj-gnugrep-3.11/bin:/nix/store/gjlh1zvckhz0qv795lnzgw2zciklbzj2-gnused-4.9/bin:/nix/store/1lbc6v5p1a3rn4rjaqnz0694xfbq8dxq-systemd-256.4/bin:/nix/store/0l63mqr7qx1asjxfz86j4zbs4758gn98-git-2.46.0/sbin:/nix/store/2wwfdill3872xircv1vwmlj8gxghg5f9-openssh-9.8p1/sbin:/nix/store/cpad9d9fdvn1fazzmk9q3a06hlmwamx6-lix-2.92.0-devpre20241005_ed9b7f4/sbin:/nix/store/w5fz6climh0a26k4vjkdm8l5ggzjbqm3-nix-eval-jobs-2.24.0/sbin:/nix/store/izpf49b74i15pcr9708s3xdwyqs4jxwl-bash-5.2p32/sbin:/nix/store/0kg70swgpg45ipcz3pr2siidq9fn6d77-coreutils-9.5/sbin:/nix/store/fnq94lw19pnwdl9p8zhfrad4jmlgxlbr-findutils-4.10.0/sbin:/nix/store/vsyc8jhsr4d9lm2r8yqq9n3j4i66inlj-gnugrep-3.11/sbin:/nix/store/gjlh1zvckhz0qv795lnzgw2zciklbzj2-gnused-4.9/sbin:/nix/store/1lbc6v5p1a3rn4rjaqnz0694xfbq8dxq-systemd-256.4/sbin
  PWD=/var/lib/buildbot-worker/worker-004/aarch64-linux/lix_nix-build_aarch64-linux/build
  PYTHONNOUSERSITE=true
  PYTHONPATH=/nix/store/p93sqnkspfkg3vn24xbm62ivmnr6qhy6-python3-3.12.5-env/lib/python3.12/site-packages
  SHELL=/run/current-system/sw/bin/zsh
  SHLVL=0
  SYSTEMD_EXEC_PID=4047704
  TZDIR=/nix/store/897xqnq52vw76991r5m80h9j91370vj9-tzdata-2024a/share/zoneinfo
  USER=buildbot-worker
  WORKER_ARCH_LIST=aarch64-darwin=2,aarch64-linux=6,other=8,x86_64-linux=16
  WORKER_PASSWORD_FILE=/run/credentials/buildbot-worker.service/worker-password-file
 using PTY: False
don't know how to build these paths:
  /nix/store/3nkf8bihgf76w0qxgbzqprn9pbl5sfkc-lix-2.91.1pre20241017_dcdeefd.drv
error: unexpected EOF from daemon socket
error: Nix daemon disconnected unexpectedly (maybe it crashed?)

cc @raito

It appears that there is no gcroot on paths that are being signed on workers, which causes random CI failures: https://buildbot.lix.systems/#/buildrequests/196354 Failure: ``` argv: [b'nix', b'store', b'sign', b'--store', b'ssh-ng://nix@build01.aarch64.lix.systems?ssh-key=/run/agenix/buildbot-remote-builder-key&base64-ssh-public-host-key=c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUNDNjlOWkQvemhJQi93VWI1b2RnNDZic3M1ZzhoSDJmRGwyMmJrNHFlU1c=', b'--key-file', b'/run/agenix/buildbot-signing-key', b'/nix/store/3nkf8bihgf76w0qxgbzqprn9pbl5sfkc-lix-2.91.1pre20241017_dcdeefd.drv^*'] environment: BUILDBOT_DIR=/var/lib/buildbot-worker/worker CREDENTIALS_DIRECTORY=/run/credentials/buildbot-worker.service HOME=/var/lib/buildbot-worker INVOCATION_ID=c6072ec4d4b84054b8d7966ed7fc5a16 JOURNAL_STREAM=8:7746687 LANG=en_US.UTF-8 LOCALE_ARCHIVE=/nix/store/wdrnjaw2sgqpp1w7kbd1jv5whcah0v5q-glibc-locales-2.39-52/lib/locale/locale-archive LOGNAME=buildbot-worker MASTER_URL=tcp:host=localhost:port=9989 MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/buildbot-worker.service/memory.pressure MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA= PATH=/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin:/nix/store/dj1489kq00csdvxh1lzyr3kgj6f1ahc8-python3.12-twisted-24.7.0/bin:/nix/store/g0rr15zdsdbnbbcli08x3zx170j1kc68-python3.12-automat-22.10.0/bin:/nix/store/0l63mqr7qx1asjxfz86j4zbs4758gn98-git-2.46.0/bin:/nix/store/2wwfdill3872xircv1vwmlj8gxghg5f9-openssh-9.8p1/bin:/nix/store/cpad9d9fdvn1fazzmk9q3a06hlmwamx6-lix-2.92.0-devpre20241005_ed9b7f4/bin:/nix/store/w5fz6climh0a26k4vjkdm8l5ggzjbqm3-nix-eval-jobs-2.24.0/bin:/nix/store/izpf49b74i15pcr9708s3xdwyqs4jxwl-bash-5.2p32/bin:/nix/store/0kg70swgpg45ipcz3pr2siidq9fn6d77-coreutils-9.5/bin:/nix/store/fnq94lw19pnwdl9p8zhfrad4jmlgxlbr-findutils-4.10.0/bin:/nix/store/vsyc8jhsr4d9lm2r8yqq9n3j4i66inlj-gnugrep-3.11/bin:/nix/store/gjlh1zvckhz0qv795lnzgw2zciklbzj2-gnused-4.9/bin:/nix/store/1lbc6v5p1a3rn4rjaqnz0694xfbq8dxq-systemd-256.4/bin:/nix/store/0l63mqr7qx1asjxfz86j4zbs4758gn98-git-2.46.0/sbin:/nix/store/2wwfdill3872xircv1vwmlj8gxghg5f9-openssh-9.8p1/sbin:/nix/store/cpad9d9fdvn1fazzmk9q3a06hlmwamx6-lix-2.92.0-devpre20241005_ed9b7f4/sbin:/nix/store/w5fz6climh0a26k4vjkdm8l5ggzjbqm3-nix-eval-jobs-2.24.0/sbin:/nix/store/izpf49b74i15pcr9708s3xdwyqs4jxwl-bash-5.2p32/sbin:/nix/store/0kg70swgpg45ipcz3pr2siidq9fn6d77-coreutils-9.5/sbin:/nix/store/fnq94lw19pnwdl9p8zhfrad4jmlgxlbr-findutils-4.10.0/sbin:/nix/store/vsyc8jhsr4d9lm2r8yqq9n3j4i66inlj-gnugrep-3.11/sbin:/nix/store/gjlh1zvckhz0qv795lnzgw2zciklbzj2-gnused-4.9/sbin:/nix/store/1lbc6v5p1a3rn4rjaqnz0694xfbq8dxq-systemd-256.4/sbin PWD=/var/lib/buildbot-worker/worker-004/aarch64-linux/lix_nix-build_aarch64-linux/build PYTHONNOUSERSITE=true PYTHONPATH=/nix/store/p93sqnkspfkg3vn24xbm62ivmnr6qhy6-python3-3.12.5-env/lib/python3.12/site-packages SHELL=/run/current-system/sw/bin/zsh SHLVL=0 SYSTEMD_EXEC_PID=4047704 TZDIR=/nix/store/897xqnq52vw76991r5m80h9j91370vj9-tzdata-2024a/share/zoneinfo USER=buildbot-worker WORKER_ARCH_LIST=aarch64-darwin=2,aarch64-linux=6,other=8,x86_64-linux=16 WORKER_PASSWORD_FILE=/run/credentials/buildbot-worker.service/worker-password-file using PTY: False don't know how to build these paths: /nix/store/3nkf8bihgf76w0qxgbzqprn9pbl5sfkc-lix-2.91.1pre20241017_dcdeefd.drv error: unexpected EOF from daemon socket error: Nix daemon disconnected unexpectedly (maybe it crashed?) ``` cc @raito
jade commented 2024-11-13 01:37:22 +00:00
Author
Owner
Copy link

Still reproduces https://buildbot.lix.systems/#/builders/39/builds/4223/steps/1/logs/stdio

Still reproduces https://buildbot.lix.systems/#/builders/39/builds/4223/steps/1/logs/stdio
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
forkos
faster-depinfo
profiler
jade/report-eval-errors
bring-back-old-gerrit-reporting
fix-reporting
incoming-ref-for-non-flakes
local-stores
non-flakes-rerun
prepare-for-non-flakes
non-flakes
jade/deny-flake-config
fallable-binary-cache
gerrit
better-gerrit
puck/gerrit-changes
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/buildbot-nix#28
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?