lix-project/lix
22
79
Fork 54
Code Issues 518 Code Review (Gerrit) Projects 8 Releases Packages Wiki Activity

nix build --system with remote builder tries to execute wrong-architecture binaries locally #692

New issue
Closed
opened 2025-02-22 11:51:08 +00:00 by dwt · 8 comments
dwt commented 2025-02-22 11:51:08 +00:00
Copy link

Describe the bug

I have a linux builder set up on my darwin machine to be able to compile linux packages. For details of my setup, see my nix darwin config.

When I try to build nixpkgs#ioquake3 it fails with a strange error:

❯ nix build --system aarch64-linux nixpkgs#ioquake3 --print-build-logs --rebuild
IOQuake3.desktop> error: executing '/nix/store/wnmp2svv1amz605gpapxd6400f7nclfg-bash-5.2p37/bin/bash': Undefined error: 0
error: builder for '/nix/store/g354lsk48pbsi9f8mg9mkyrrg40ky6lq-IOQuake3.desktop.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/nix/store/wnmp2svv1amz605gpapxd6400f7nclfg-bash-5.2p37/bin/bash': Undefined error: 0
       For full logs, run 'nix log /nix/store/g354lsk48pbsi9f8mg9mkyrrg40ky6lq-IOQuake3.desktop.drv'.
error: 1 dependencies of derivation '/nix/store/hsqq7lhx4965nqjj9y2xpfxpqc6bygqa-ioquake3-unstable-2023-08-13.drv' failed to build

Looking at that bash file, that is indeed a linux binary:

❯ file /nix/store/wnmp2svv1amz605gpapxd6400f7nclfg-bash-5.2p37/bin/bash
/nix/store/wnmp2svv1amz605gpapxd6400f7nclfg-bash-5.2p37/bin/bash: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/8d9rkvllf04pyz790vk6wd4k8mnc5c64-glibc-2.40-36/lib/ld-linux-aarch64.so.1, BuildID[sha1]=baace6f679f9c2b65933a6ccb0b85867e7c0816d, for GNU/Linux 3.10.0, not stripped

It seems as if nix tries to execute this on my local machine, instead of using my linux-builder?

Steps To Reproduce

  1. Go to somewhere. (mktemp -d) on a mac
  2. nix build --system aarch64-linux nixpkgs#ioquake3 --print-build-logs --rebuild
  3. See error

Expected behavior

It should build on my linux builder. I use https://github.com/cpick/nix-rosetta-builder for that.

nix --version output

❯ nix --version
nix (Lix, like Nix) 2.92.0
System type: aarch64-darwin
Additional system types: aarch64-darwin, x86_64-darwin
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /Users/dwt/.config/nix/nix.conf:/Users/dwt/.nix-profile/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/vs6s56c7crvj75fmmyrc4f1wh4q7pbns-lix-2.92.0/share

Additional context

Please ask for anything that can be helpful.

## Describe the bug I have a linux builder set up on my darwin machine to be able to compile linux packages. For details of my setup, see [my nix darwin config](https://github.com/dwt/nix-darwin-config). When I try to build nixpkgs#ioquake3 it fails with a strange error: ``` ❯ nix build --system aarch64-linux nixpkgs#ioquake3 --print-build-logs --rebuild IOQuake3.desktop> error: executing '/nix/store/wnmp2svv1amz605gpapxd6400f7nclfg-bash-5.2p37/bin/bash': Undefined error: 0 error: builder for '/nix/store/g354lsk48pbsi9f8mg9mkyrrg40ky6lq-IOQuake3.desktop.drv' failed with exit code 1; last 1 log lines: > error: executing '/nix/store/wnmp2svv1amz605gpapxd6400f7nclfg-bash-5.2p37/bin/bash': Undefined error: 0 For full logs, run 'nix log /nix/store/g354lsk48pbsi9f8mg9mkyrrg40ky6lq-IOQuake3.desktop.drv'. error: 1 dependencies of derivation '/nix/store/hsqq7lhx4965nqjj9y2xpfxpqc6bygqa-ioquake3-unstable-2023-08-13.drv' failed to build ``` Looking at that bash file, that is indeed a linux binary: ``` ❯ file /nix/store/wnmp2svv1amz605gpapxd6400f7nclfg-bash-5.2p37/bin/bash /nix/store/wnmp2svv1amz605gpapxd6400f7nclfg-bash-5.2p37/bin/bash: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/8d9rkvllf04pyz790vk6wd4k8mnc5c64-glibc-2.40-36/lib/ld-linux-aarch64.so.1, BuildID[sha1]=baace6f679f9c2b65933a6ccb0b85867e7c0816d, for GNU/Linux 3.10.0, not stripped ``` It seems as if nix tries to execute this on my local machine, instead of using my linux-builder? ## Steps To Reproduce 1. Go to somewhere. (mktemp -d) on a mac 2. `nix build --system aarch64-linux nixpkgs#ioquake3 --print-build-logs --rebuild` 4. See error ## Expected behavior It should build on my linux builder. I use https://github.com/cpick/nix-rosetta-builder for that. ## `nix --version` output ```shell ❯ nix --version nix (Lix, like Nix) 2.92.0 System type: aarch64-darwin Additional system types: aarch64-darwin, x86_64-darwin Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /Users/dwt/.config/nix/nix.conf:/Users/dwt/.nix-profile/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/vs6s56c7crvj75fmmyrc4f1wh4q7pbns-lix-2.92.0/share ``` ## Additional context Please ask for anything that can be helpful.
jade commented 2025-02-22 20:40:21 +00:00
Owner
Copy link

--system is a big gun pointed directly at your foot. you want --eval-system here which will fix your issue.

--system should probably be deprecated. I think there might be other issues; I filed a related one about flakes recently; it's tagged flakes.

--system is a big gun pointed directly at your foot. you want --eval-system here which will fix your issue. --system should probably be deprecated. I think there might be other issues; I filed a related one about flakes recently; it's tagged flakes.
jade changed title from nix build --rebuild with remote builder breaks on some packages to nix build --system with remote builder tries to execute wrong-architecture binaries locally 2025-02-22 20:41:15 +00:00
jade commented 2025-02-22 20:43:48 +00:00
Owner
Copy link

(the reason this breaks is that it tells the daemon that it is a different operating system itself and then it tries to actually be that different operating system. --eval-system just affects the evaluator and then sending it to the daemon dispatches to remote builders. i believe there is an extra-systems setting or something?)

(the reason this breaks is that it tells the daemon that it is a different operating system itself and then it tries to actually be that different operating system. --eval-system just affects the evaluator and then sending it to the daemon dispatches to remote builders. i believe there is an extra-systems setting or something?)
jade closed this issue 2025-02-22 20:43:48 +00:00
jade commented 2025-02-22 20:44:35 +00:00
Owner
Copy link

flakes issue i found #673

flakes issue i found https://git.lix.systems/lix-project/lix/issues/673
dwt commented 2025-02-24 07:11:49 +00:00
Author
Copy link

Oh that is useful, I'll try that out tonight. Strange that all of the internet suggests using --system to do builds using a nix-builder that is configured on the local system.

Thanks!

Oh that is useful, I'll try that out tonight. Strange that all of the internet suggests using `--system` to do builds using a nix-builder that is configured on the local system. Thanks!
dwt commented 2025-02-24 07:19:18 +00:00
Author
Copy link

Hm, that might not be enough, when I try this:

❯ nix build --eval-system aarch64-linux .#ioquake3
error:
       … in the condition of the assert statement
         at /nix/store/cqkfmgcc8ivdg74718zm253fqmxh7alj-source/lib/customisation.nix:419:9:
          418|       drvPath =
          419|         assert condition;
             |         ^
          420|         drv.drvPath;

       … while evaluating the attribute 'handled'
         at /nix/store/cqkfmgcc8ivdg74718zm253fqmxh7alj-source/pkgs/stdenv/generic/check-meta.nix:507:7:
          506|       # or, alternatively, just output a warning message.
          507|       handled =
             |       ^
          508|         (

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: Package ‘ioquake3-unstable-2023-08-13’ in /nix/store/cqkfmgcc8ivdg74718zm253fqmxh7alj-source/pkgs/by-name/io/ioquake3/package.nix:88 is not available on the requested hostPlatform:
         hostPlatform.config = "aarch64-apple-darwin"
         package.meta.platforms = [
           "aarch64-linux"
           "armv5tel-linux"
           "armv6l-linux"
           "armv7a-linux"
           "armv7l-linux"
           "i686-linux"
           "loongarch64-linux"
           "m68k-linux"
           "microblaze-linux"
           "microblazeel-linux"
           "mips-linux"
           "mips64-linux"
           "mips64el-linux"
           "mipsel-linux"
           "powerpc64-linux"
           "powerpc64le-linux"
           "riscv32-linux"
           "riscv64-linux"
           "s390-linux"
           "s390x-linux"
           "x86_64-linux"
         ]
         package.meta.badPlatforms = [ ]
       , refusing to evaluate.

       a) To temporarily allow packages that are unsupported for this system, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1

          Note: When using `nix shell`, `nix build`, `nix develop`, etc with a flake,
                then pass `--impure` in order to allow use of environment variables.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowUnsupportedSystem = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowUnsupportedSystem = true; }
       to ~/.config/nixpkgs/config.nix.

That does not seem to work yet.

Some excerpts from my nix config:

❯ nix config show|rg sub
always-allow-substitutes = false
builders-use-substitutes = true
max-substitution-jobs = 16
substitute = true
substituters = https://cache.nixos.org https://cache.lix.systems https://cache.nixos.org/
trusted-substituters = 

❯ nix config show|rg builders
builders = @/etc/nix/machines
builders-use-substitutes = true

❯ cat -p  /etc/nix/machines
ssh-ng://rosetta-builder aarch64-linux,x86_64-linux - 8 1 benchmark,big-parallel,kvm - -

❯ nix config show|rg extra-platforms
extra-platforms = aarch64-darwin x86_64-darwin

❯ nix config show|rg system

substituters = https://cache.nixos.org https://cache.lix.systems https://cache.nixos.org/
system = aarch64-darwin
system-features = apple-virt benchmark big-parallel nixos-test
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.lix.systems:aBnZUw8zA7H35Cz2RyKFVs3H4PlGTLawyY5KRbvJR8o=
Hm, that might not be enough, when I try this: ```shell ❯ nix build --eval-system aarch64-linux .#ioquake3 error: … in the condition of the assert statement at /nix/store/cqkfmgcc8ivdg74718zm253fqmxh7alj-source/lib/customisation.nix:419:9: 418| drvPath = 419| assert condition; | ^ 420| drv.drvPath; … while evaluating the attribute 'handled' at /nix/store/cqkfmgcc8ivdg74718zm253fqmxh7alj-source/pkgs/stdenv/generic/check-meta.nix:507:7: 506| # or, alternatively, just output a warning message. 507| handled = | ^ 508| ( (stack trace truncated; use '--show-trace' to show the full trace) error: Package ‘ioquake3-unstable-2023-08-13’ in /nix/store/cqkfmgcc8ivdg74718zm253fqmxh7alj-source/pkgs/by-name/io/ioquake3/package.nix:88 is not available on the requested hostPlatform: hostPlatform.config = "aarch64-apple-darwin" package.meta.platforms = [ "aarch64-linux" "armv5tel-linux" "armv6l-linux" "armv7a-linux" "armv7l-linux" "i686-linux" "loongarch64-linux" "m68k-linux" "microblaze-linux" "microblazeel-linux" "mips-linux" "mips64-linux" "mips64el-linux" "mipsel-linux" "powerpc64-linux" "powerpc64le-linux" "riscv32-linux" "riscv64-linux" "s390-linux" "s390x-linux" "x86_64-linux" ] package.meta.badPlatforms = [ ] , refusing to evaluate. a) To temporarily allow packages that are unsupported for this system, you can use an environment variable for a single invocation of the nix tools. $ export NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1 Note: When using `nix shell`, `nix build`, `nix develop`, etc with a flake, then pass `--impure` in order to allow use of environment variables. b) For `nixos-rebuild` you can set { nixpkgs.config.allowUnsupportedSystem = true; } in configuration.nix to override this. c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add { allowUnsupportedSystem = true; } to ~/.config/nixpkgs/config.nix. ``` That does not seem to work yet. Some excerpts from my nix config: ```shell ❯ nix config show|rg sub always-allow-substitutes = false builders-use-substitutes = true max-substitution-jobs = 16 substitute = true substituters = https://cache.nixos.org https://cache.lix.systems https://cache.nixos.org/ trusted-substituters = ❯ nix config show|rg builders builders = @/etc/nix/machines builders-use-substitutes = true ❯ cat -p /etc/nix/machines ssh-ng://rosetta-builder aarch64-linux,x86_64-linux - 8 1 benchmark,big-parallel,kvm - - ❯ nix config show|rg extra-platforms extra-platforms = aarch64-darwin x86_64-darwin ❯ nix config show|rg system substituters = https://cache.nixos.org https://cache.lix.systems https://cache.nixos.org/ system = aarch64-darwin system-features = apple-virt benchmark big-parallel nixos-test trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.lix.systems:aBnZUw8zA7H35Cz2RyKFVs3H4PlGTLawyY5KRbvJR8o= ```
dwt commented 2025-02-24 07:20:00 +00:00
Author
Copy link

Also I just noticed, that because this is not GitHub, linking to mentions doesn't work automatically. this is where I came from: https://github.com/cpick/nix-rosetta-builder/issues/19

Also I just noticed, that because this is not GitHub, linking to mentions doesn't work automatically. this is where I came from: https://github.com/cpick/nix-rosetta-builder/issues/19
jade commented 2025-02-24 17:11:43 +00:00
Owner
Copy link

oh this is a flakes being bad moment! i think #673 is highly related if not the same bug.

oh this is a flakes being bad moment! i think #673 is highly related if not the same bug.
👍 1
lix-bot commented 2025-02-25 02:09:30 +00:00
Member
Copy link

This issue was mentioned on Gerrit on the following CLs:

  • commit message in cl/2657 ("fix: flakes now obey --eval-system")
<!-- GERRIT_LINKBOT: {"cls": [{"backlink": "https://gerrit.lix.systems/c/lix/+/2657", "number": 2657, "kind": "commit message"}], "cl_meta": {"2657": {"change_title": "fix: flakes now obey --eval-system"}}} --> This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/2657](https://gerrit.lix.systems/c/lix/+/2657) ("fix: flakes now obey --eval-system")
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
ci-config
release-2.91
release-2.92
release-2.93
sb/raito/darwin-kqueue-disablement
sb/raito/hydra-merge
sb/pennae/ci-config
sb/raito/phantom-referrers-gc
repl-overlay-example
reuse-compliance
release-2.90
sb/pennae/io-rewrite
git-series/activity-cleanup
sb/tom-hubrecht/primops.cc-v2
sb/bacchanalia/cache-strlen%topic=substr-perf
sb/qyriad/beta-1
sb/qyriad/2.90-beta.0
sb/rbt/test-branch
sb/rbt/justfile
sb/rbt/repl-overlays
sb/qyriad/maint-meson
sb/qyriad/maint/meson
sb/puck/meson-c-api
sb/jade/burn-it-all
dependabot/github_actions/zeebe-io/backport-action-2.4.1
dependabot/github_actions/cachix/install-nix-action-25
dependabot/github_actions/actions/labeler-5
dependabot/github_actions/cachix/cachix-action-14
2.93.3
2.92.3
2.93.2
2.91.3
2.92.2
2.91.2
2.93.1
2.93.0
2.92.0
2.91.1
2.91.0
2.90.0
2.90.0-rc1
2.90-beta.1
2.90-beta.0
Labels
Clear labels   
Affects/CppNix

Probably affects CppNix. This is a best-effort tag for troubles in old code; it exists to inform the CppNix team about problems they might care about also.

  
Affects/Nightly

Found in Lix nightly.

  
Affects/Only nightly

Tag for regressions in Lix nightly. We intend to fix all of these prior to releases.

  
Affects/Stable

Affects the stable release of Lix. Usually such issues affect nightly too but there is no guarantee.

  
Area/build-packaging

regarding the building or packaging of Lix itself

  
Area/cli

The command line interface of Lix

  
Area/evaluator

libexpr and anything else related to the evaluation of Nix expressions

  
Area/fetching

Fetching sources or other materials from the World Wide Web in Lix

  
Area/flakes

   
Area/language

language design, features, extensions, etc

  
Area/lix ci

Lix's own CI and deployment pipeline

  
Area/nix-eval-jobs

nix-eval-jobs for Lix

  
Area/profiles

profiles: nix-env, nix profile, and installer flavors all alike

  
Area/protocol

The protocol, used for communicating between the client and the daemon

  
Area/releng

Release engineering

  
Area/remote-builds

remote builds

  
Area/repl

nix repl

  
Area/repl/debugger

The --debugger that brings up a Lix repl

  
Area/store

libstore and anything else relating to the Nix store

  
bug

it's a bug

  
Context
contributors
This issue might require some context that isn't written down. It might be possible to handle as a person who already worked on this area.

  
Context
drive-by
This issue contains all the necessary information to do a drive-by contribution: that is, you don't have to talk with anyone at all to successfully solve it!

  
Context
maintainers
This issue requires a lot of know-how. Attention from maintainers is needed to proceed.

  
Context
RFD
This issue needs discussion from multiple maintainers and interested parties: it's a Request For Discussion.

  
crash 💥

Lix crashes (possibly bad error handling)

  
Cross Compilation

   
devx

development experience issues

  
docs

requires or is about documentation changes

  
Downstream Dependents

regarding an external tool's usage with Lix (e.g. nix-eval-jobs)

  
E/easy

bug is relatively simple and could simply be fixed with some minor effort

  
E/hard

this issue should probably be solved with close involvement of an expert

  
E/help wanted

maintainers may not get to this soon, but we want it fixed

  
E/reproducible

This bug is reproducible and obvious.

  
E/requires rearchitecture

this bug cannot be fixed properly without fixing a known architectural flaw

  
imported

Issue imported from upstream nix (please do not remove this label, it will cause dupes if import is rerun)

  
Language/Bash

   
Language/C++

   
Language/NixLang

   
Language/Python

   
Language/Rust

   
Needs Langver

We can't get to this until we implement language versioning

  
OS/Linux

Affects specifically Lix on Linux

  
OS/macOS

Affects specifically Lix on macOS

  
performance

make lix go faster

  
regression

This is a regression since some previous version of Lix

  
release-blocker

Blocks public release

  
stability

Fixing this makes the software more stable

  
Status
blocked
Blocked on something

  
Status
invalid
not our bug, duplicate, similar

  
Status
postponed
closed but should maybe look at later (generally with S-wontfix)

  
Status
wontfix
We chose not to change this (see issue comments for more detail).

  
testing

Tests that should be added

  
testing/flakey

Tests that are unreliable/flakey (not to be confused with flakes that test)

  
Topic/Large Scale Installations

Large scale installations of Lix, e.g. NixOS Hydra, Lix CI, big corporate installations etc.

  
ux

makes the user experience better :)

No labels
Affects/CppNix
Affects/Nightly
Affects/Only nightly
Affects/Stable
Area/build-packaging
Area/cli
Area/evaluator
Area/fetching
Area/flakes
Area/language
Area/lix ci
Area/nix-eval-jobs
Area/profiles
Area/protocol
Area/releng
Area/remote-builds
Area/repl
Area/repl/debugger
Area/store
bug
Context
contributors
Context
drive-by
Context
maintainers
Context
RFD
crash 💥
Cross Compilation
devx
docs
Downstream Dependents
E/easy
E/hard
E/help wanted
E/reproducible
E/requires rearchitecture
imported
Language/Bash
Language/C++
Language/NixLang
Language/Python
Language/Rust
Needs Langver
OS/Linux
OS/macOS
performance
regression
release-blocker
stability
Status
blocked
Status
invalid
Status
postponed
Status
wontfix
testing
testing/flakey
Topic/Large Scale Installations
ux
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#692
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?