Cannot compile Lix on my rackmount server, but I can on my laptop. #569

Open
opened 2024-11-04 18:10:16 +00:00 by johnhamelink · 3 comments

Describe the bug

A clear and concise description of what the bug is.

When building 5il3pn3sr03yvjx95ykhg4m3ca7c85lk-lix-2.91.1.drv, the tests fail to conclude and the build fails, but only on my rackmount server (called Sun): https://0x0.st/XG0P.txt

On my laptop (called Saturn), I have /nix/store/5il3pn3sr03yvjx95ykhg4m3ca7c85lk-lix-2.91.1.drv as well, which builds OK: https://0x0.st/XG02.txt

If you have a problem with a specific package or NixOS,
you probably want to file an issue at https://github.com/NixOS/nixpkgs/issues.

Steps To Reproduce

(I have found that running nixos-rebuild --flake .\#sun build on saturn does not yield an error) because I have a successful build in my store already, possibly?)

  1. On Saturn (laptop):
    1.1. git clone https://git.sr.ht/~johnhamelink/nix && cd nix && git checkout f28e9788d3e8c668ceacdaeceeb145768a2cf691
    1.2. nixos-rebuild --flake .\#sun build - Succeeds
    1.4. nixos-rebuild --flake .\#saturn build - Succeeds
  2. On Sun (server):
    1.1. git clone https://git.sr.ht/~johnhamelink/nix && cd nix && git checkout f28e9788d3e8c668ceacdaeceeb145768a2cf691
    1.2. nixos-rebuild --flake .\#sun build - Fails
    1.3. See error: https://0x0.st/XG0P.txt
    1.4. nixos-rebuild --flake .\#saturn build - Fails in exactly the same way

Expected behavior

A clear and concise description of what you expected to happen.

I would expect the build to complete successfully just as it does on my laptop.

nix --version output

  • Sun: nix (Nix) 2.24.10
  • Saturn: nix (Lix, like Nix) 2.91.1

Additional context

Add any other context about the problem here.

## Describe the bug > A clear and concise description of what the bug is. When building `5il3pn3sr03yvjx95ykhg4m3ca7c85lk-lix-2.91.1.drv`, the tests fail to conclude and the build fails, but only on my rackmount server (called Sun): https://0x0.st/XG0P.txt On my laptop (called Saturn), I have `/nix/store/5il3pn3sr03yvjx95ykhg4m3ca7c85lk-lix-2.91.1.drv` as well, which builds OK: https://0x0.st/XG02.txt > If you have a problem with a specific package or NixOS, > you probably want to file an issue at https://github.com/NixOS/nixpkgs/issues. ## Steps To Reproduce (I have found that running `nixos-rebuild --flake .\#sun build` on saturn does not yield an error) because I have a successful build in my store already, possibly?) 1. On Saturn (laptop): 1.1. `git clone https://git.sr.ht/~johnhamelink/nix && cd nix && git checkout f28e9788d3e8c668ceacdaeceeb145768a2cf691` 1.2. `nixos-rebuild --flake .\#sun build` - Succeeds 1.4. `nixos-rebuild --flake .\#saturn build` - Succeeds 2. On Sun (server): 1.1. `git clone https://git.sr.ht/~johnhamelink/nix && cd nix && git checkout f28e9788d3e8c668ceacdaeceeb145768a2cf691` 1.2. `nixos-rebuild --flake .\#sun build` - Fails 1.3. See error: https://0x0.st/XG0P.txt 1.4. `nixos-rebuild --flake .\#saturn build` - Fails in exactly the same way ## Expected behavior > A clear and concise description of what you expected to happen. I would expect the build to complete successfully just as it does on my laptop. ## `nix --version` output - Sun: `nix (Nix) 2.24.10` - Saturn: `nix (Lix, like Nix) 2.91.1` ## Additional context > Add any other context about the problem here. - `nix log /nix/store/5il3pn3sr03yvjx95ykhg4m3ca7c85lk-lix-2.91.1.drv`: https://0x0.st/XG0P.txt
johnhamelink added the
bug
label 2024-11-04 18:10:16 +00:00
Owner

Hello, thanks for the information, can you provide more information about the Sun system, e.g. architecture, what is your /tmp filesystem, what is your /nix/store filesystem, can you share the /etc/nix/nix.conf file from that system, enabled experimental features.

If that's not sufficient, we may ask you for more information to try to understand, as we cannot reproduce this error (while building via Lix, I need to see if Nix 2.24.10 is the simple reproducer).

Hello, thanks for the information, can you provide more information about the Sun system, e.g. architecture, what is your /tmp filesystem, what is your /nix/store filesystem, can you share the /etc/nix/nix.conf file from that system, enabled experimental features. If that's not sufficient, we may ask you for more information to try to understand, as we cannot reproduce this error (while building via Lix, I need to see if Nix 2.24.10 is the simple reproducer).
Author

Ah yes, certainly - my server is running a bit different in this regard. I should've thought to include this initially!

Firstly here's the output of inxi -F:

System:
  Host: sun Kernel: 6.6.59 arch: x86_64 bits: 64
  Console: pty pts/0 Distro: NixOS 24.11 (Vicuna)
Machine:
  Type: Server System: Supermicro product: Super Server v: 0123456789 serial: <superuser required>
  Mobo: Supermicro model: X10DRU-i+ serial: <superuser required> UEFI: American Megatrends
    v: 3.5 date: 04/27/2022
CPU:
  Info: 2x 16-core model: Intel Xeon E5-2683 v4 bits: 64 type: MT MCP SMP cache:
    L2: 2x 4 MiB (8 MiB)
  Speed (MHz): avg: 1215 min/max: 1200/3000 cores: 1: 1201 2: 1200 3: 1200 4: 1200 5: 1200
    6: 1200 7: 1200 8: 1200 9: 1200 10: 1200 11: 1200 12: 1200 13: 1200 14: 1200 15: 1200 16: 1200
    17: 1200 18: 1200 19: 1201 20: 1700 21: 1200 22: 1200 23: 1200 24: 1200 25: 1700 26: 1200
    27: 1200 28: 1200 29: 1200 30: 1200 31: 1200 32: 1200 33: 1200 34: 1200 35: 1200 36: 1200
    37: 1200 38: 1200 39: 1200 40: 1200 41: 1200 42: 1200 43: 1200 44: 1200 45: 1200 46: 1200
    47: 1200 48: 1201 49: 1200 50: 1200 51: 1200 52: 1200 53: 1200 54: 1200 55: 1200 56: 1200
    57: 1200 58: 1200 59: 1200 60: 1200 61: 1200 62: 1200 63: 1200 64: 1200
Graphics:
  Device-1: ASPEED Graphics Family driver: ast v: kernel
  Display: unspecified server: N/A driver: gpu: ast tty: 137x67
  API: N/A Message: No API data available in console. Headless machine?
Audio:
  Message: No device data found.
Network:
  Device-1: Intel Ethernet 10-Gigabit X540-AT2 driver: ixgbe
  IF: enp1s0f0 state: down mac: 0c:c4:7a:a4:33:e8
  Device-2: Intel Ethernet 10-Gigabit X540-AT2 driver: ixgbe
  IF: enp1s0f1 state: down mac: 0c:c4:7a:a4:33:e9
  Device-3: Intel Ethernet 10-Gigabit X540-AT2 driver: ixgbe
  IF: enp2s0f0 state: up speed: 1000 Mbps duplex: full mac: 0c:c4:7a:a4:33:ea
  Device-4: Intel Ethernet 10-Gigabit X540-AT2 driver: ixgbe
  IF: enp2s0f1 state: down mac: 0c:c4:7a:a4:33:eb
RAID:
  Device-1: cache type: zfs status: - level: linear zfs-fs: size: 32.25 TiB free: raw:
    size: 466 GiB free: 47.3 GiB
  Components: Online: 1: sdl1
  Device-2: pool type: zfs status: ONLINE raw: size: 47.3 TiB free: 3.16 TiB zfs-fs:
    size: 32.25 TiB free: 1.95 TiB
  Array-1: raidz1-0 status: N/A raw: size: 14.5 TiB free: 242 GiB
  Components: Online: 1: sdb 2: sdj 3: sdm 4: sdp
  Array-2: raidz1-1 status: N/A raw: size: 32.7 TiB free: 2.93 TiB
  Components: Online: 1: sdd 2: sdf 3: sdi2
Drives:
  Local Storage: total: raw: 102.89 TiB usable: 84.21 TiB used: 30.05 TiB (35.7%)
  ID-1: /dev/sda vendor: Western Digital model: WD40EFRX-68N size: 3.64 TiB
  ID-2: /dev/sdb vendor: Western Digital model: WD60EFRX-68L0BN1 size: 5.46 TiB
  ID-3: /dev/sdc vendor: Western Digital model: WD60EFRX-68L size: 5.46 TiB
  ID-4: /dev/sdd vendor: Seagate model: ST12000VN0008-2PH103 size: 10.91 TiB
  ID-5: /dev/sde vendor: Seagate model: ST12000VN0008-2PH103 size: 10.91 TiB
  ID-6: /dev/sdf vendor: Seagate model: ST12000VN0008-2PH103 size: 10.91 TiB
  ID-7: /dev/sdg vendor: Seagate model: ST12000VN0008-2P size: 10.91 TiB
  ID-8: /dev/sdh vendor: Western Digital model: WD60EFRX-68L size: 5.46 TiB
  ID-9: /dev/sdi vendor: Seagate model: ST12000VN0008-2P size: 10.91 TiB
  ID-10: /dev/sdj vendor: Western Digital model: WD40EFRX-68N32N0 size: 3.64 TiB
  ID-11: /dev/sdk vendor: Crucial model: CT500BX500SSD1 size: 465.76 GiB
  ID-12: /dev/sdl vendor: Crucial model: CT500BX500SSD1 size: 465.76 GiB
  ID-13: /dev/sdm vendor: Western Digital model: WD60EFRX-68L0BN1 size: 5.46 TiB
  ID-14: /dev/sdn vendor: Seagate model: ST12000VN0008-2P size: 10.91 TiB
  ID-15: /dev/sdo vendor: Western Digital model: WD40EFRX-68N size: 3.64 TiB
  ID-16: /dev/sdp vendor: Western Digital model: WD40EFRX-68N32N0 size: 3.64 TiB
  ID-17: /dev/sdq model: USB DISK 3.0 size: 115.63 GiB type: USB
Partition:
  ID-1: / size: 107.25 GiB used: 18.07 GiB (16.9%) fs: btrfs dev: /dev/sdq1
  ID-2: /boot size: 486 MiB used: 88.1 MiB (18.1%) fs: vfat dev: /dev/sdq2
  ID-3: /var size: 1.98 TiB used: 30.52 GiB (1.5%) fs: zfs logical: pool/nix-var
Swap:
  Alert: No swap data was found.
Sensors:
  Src: ipmi Permissions: Unable to run ipmi sensors. Root privileges required.
  Src: lm-sensors System Temperatures: cpu: 30.0 C mobo: N/A
  Fan Speeds (rpm): N/A
Info:
  Memory: total: 256 GiB available: 251.76 GiB used: 151.65 GiB (60.2%)
  Processes: 982 Uptime: 1d 19h 9m Init: systemd Shell: Zsh inxi: 3.3.35

I am using ZFS to mount my nix store:

/dev/disk/by-uuid/1e2755f9-3897-40a6-8687-711934434b98 / btrfs x-initrd.mount 0 0
/dev/disk/by-uuid/6344-37FA /boot vfat fmask=0077,dmask=0077 0 2
pool/nix /nix zfs x-initrd.mount 0 0
pool/nix-var /var zfs x-initrd.mount 0 0

The configuration of which can be found here: https://git.sr.ht/~johnhamelink/nix/tree/master/item/os/hosts/sun/zfs.nix and here: https://git.sr.ht/~johnhamelink/nix/tree/master/item/os/hosts/sun/hardware-configuration.nix#L37

My nix.conf is (I think) fairly vanilla:

allowed-users = *
auto-optimise-store = false
builders = 
cores = 0
experimental-features = nix-command flakes
max-jobs = auto
require-sigs = true
sandbox = true
sandbox-fallback = false
secret-key-files = /run/secrets/distributed_builds/sun/nix-bin-cache-priv-key.pem
substituters = https://cache.nixos.org/
system-features = nixos-test benchmark big-parallel kvm
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= sun:<REDACTED> cache.lix.systems:aBnZUw8zA7H35Cz2RyKFVs3H4PlGTLawyY5KRbvJR8o= nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
trusted-substituters = 
trusted-users = root root johnhamelink john @wheel
extra-sandbox-paths = 
extra-substituters = https://cache.lix.systems https://nix-community.cachix.org

If there's anything else I could provide which would be helpful please do let me know 👍

Ah yes, certainly - my server is running a bit different in this regard. I should've thought to include this initially! Firstly here's the output of `inxi -F`: ``` System: Host: sun Kernel: 6.6.59 arch: x86_64 bits: 64 Console: pty pts/0 Distro: NixOS 24.11 (Vicuna) Machine: Type: Server System: Supermicro product: Super Server v: 0123456789 serial: <superuser required> Mobo: Supermicro model: X10DRU-i+ serial: <superuser required> UEFI: American Megatrends v: 3.5 date: 04/27/2022 CPU: Info: 2x 16-core model: Intel Xeon E5-2683 v4 bits: 64 type: MT MCP SMP cache: L2: 2x 4 MiB (8 MiB) Speed (MHz): avg: 1215 min/max: 1200/3000 cores: 1: 1201 2: 1200 3: 1200 4: 1200 5: 1200 6: 1200 7: 1200 8: 1200 9: 1200 10: 1200 11: 1200 12: 1200 13: 1200 14: 1200 15: 1200 16: 1200 17: 1200 18: 1200 19: 1201 20: 1700 21: 1200 22: 1200 23: 1200 24: 1200 25: 1700 26: 1200 27: 1200 28: 1200 29: 1200 30: 1200 31: 1200 32: 1200 33: 1200 34: 1200 35: 1200 36: 1200 37: 1200 38: 1200 39: 1200 40: 1200 41: 1200 42: 1200 43: 1200 44: 1200 45: 1200 46: 1200 47: 1200 48: 1201 49: 1200 50: 1200 51: 1200 52: 1200 53: 1200 54: 1200 55: 1200 56: 1200 57: 1200 58: 1200 59: 1200 60: 1200 61: 1200 62: 1200 63: 1200 64: 1200 Graphics: Device-1: ASPEED Graphics Family driver: ast v: kernel Display: unspecified server: N/A driver: gpu: ast tty: 137x67 API: N/A Message: No API data available in console. Headless machine? Audio: Message: No device data found. Network: Device-1: Intel Ethernet 10-Gigabit X540-AT2 driver: ixgbe IF: enp1s0f0 state: down mac: 0c:c4:7a:a4:33:e8 Device-2: Intel Ethernet 10-Gigabit X540-AT2 driver: ixgbe IF: enp1s0f1 state: down mac: 0c:c4:7a:a4:33:e9 Device-3: Intel Ethernet 10-Gigabit X540-AT2 driver: ixgbe IF: enp2s0f0 state: up speed: 1000 Mbps duplex: full mac: 0c:c4:7a:a4:33:ea Device-4: Intel Ethernet 10-Gigabit X540-AT2 driver: ixgbe IF: enp2s0f1 state: down mac: 0c:c4:7a:a4:33:eb RAID: Device-1: cache type: zfs status: - level: linear zfs-fs: size: 32.25 TiB free: raw: size: 466 GiB free: 47.3 GiB Components: Online: 1: sdl1 Device-2: pool type: zfs status: ONLINE raw: size: 47.3 TiB free: 3.16 TiB zfs-fs: size: 32.25 TiB free: 1.95 TiB Array-1: raidz1-0 status: N/A raw: size: 14.5 TiB free: 242 GiB Components: Online: 1: sdb 2: sdj 3: sdm 4: sdp Array-2: raidz1-1 status: N/A raw: size: 32.7 TiB free: 2.93 TiB Components: Online: 1: sdd 2: sdf 3: sdi2 Drives: Local Storage: total: raw: 102.89 TiB usable: 84.21 TiB used: 30.05 TiB (35.7%) ID-1: /dev/sda vendor: Western Digital model: WD40EFRX-68N size: 3.64 TiB ID-2: /dev/sdb vendor: Western Digital model: WD60EFRX-68L0BN1 size: 5.46 TiB ID-3: /dev/sdc vendor: Western Digital model: WD60EFRX-68L size: 5.46 TiB ID-4: /dev/sdd vendor: Seagate model: ST12000VN0008-2PH103 size: 10.91 TiB ID-5: /dev/sde vendor: Seagate model: ST12000VN0008-2PH103 size: 10.91 TiB ID-6: /dev/sdf vendor: Seagate model: ST12000VN0008-2PH103 size: 10.91 TiB ID-7: /dev/sdg vendor: Seagate model: ST12000VN0008-2P size: 10.91 TiB ID-8: /dev/sdh vendor: Western Digital model: WD60EFRX-68L size: 5.46 TiB ID-9: /dev/sdi vendor: Seagate model: ST12000VN0008-2P size: 10.91 TiB ID-10: /dev/sdj vendor: Western Digital model: WD40EFRX-68N32N0 size: 3.64 TiB ID-11: /dev/sdk vendor: Crucial model: CT500BX500SSD1 size: 465.76 GiB ID-12: /dev/sdl vendor: Crucial model: CT500BX500SSD1 size: 465.76 GiB ID-13: /dev/sdm vendor: Western Digital model: WD60EFRX-68L0BN1 size: 5.46 TiB ID-14: /dev/sdn vendor: Seagate model: ST12000VN0008-2P size: 10.91 TiB ID-15: /dev/sdo vendor: Western Digital model: WD40EFRX-68N size: 3.64 TiB ID-16: /dev/sdp vendor: Western Digital model: WD40EFRX-68N32N0 size: 3.64 TiB ID-17: /dev/sdq model: USB DISK 3.0 size: 115.63 GiB type: USB Partition: ID-1: / size: 107.25 GiB used: 18.07 GiB (16.9%) fs: btrfs dev: /dev/sdq1 ID-2: /boot size: 486 MiB used: 88.1 MiB (18.1%) fs: vfat dev: /dev/sdq2 ID-3: /var size: 1.98 TiB used: 30.52 GiB (1.5%) fs: zfs logical: pool/nix-var Swap: Alert: No swap data was found. Sensors: Src: ipmi Permissions: Unable to run ipmi sensors. Root privileges required. Src: lm-sensors System Temperatures: cpu: 30.0 C mobo: N/A Fan Speeds (rpm): N/A Info: Memory: total: 256 GiB available: 251.76 GiB used: 151.65 GiB (60.2%) Processes: 982 Uptime: 1d 19h 9m Init: systemd Shell: Zsh inxi: 3.3.35 ``` I am using ZFS to mount my nix store: ``` /dev/disk/by-uuid/1e2755f9-3897-40a6-8687-711934434b98 / btrfs x-initrd.mount 0 0 /dev/disk/by-uuid/6344-37FA /boot vfat fmask=0077,dmask=0077 0 2 pool/nix /nix zfs x-initrd.mount 0 0 pool/nix-var /var zfs x-initrd.mount 0 0 ``` The configuration of which can be found here: https://git.sr.ht/~johnhamelink/nix/tree/master/item/os/hosts/sun/zfs.nix and here: https://git.sr.ht/~johnhamelink/nix/tree/master/item/os/hosts/sun/hardware-configuration.nix#L37 My `nix.conf` is (I think) fairly vanilla: ``` allowed-users = * auto-optimise-store = false builders = cores = 0 experimental-features = nix-command flakes max-jobs = auto require-sigs = true sandbox = true sandbox-fallback = false secret-key-files = /run/secrets/distributed_builds/sun/nix-bin-cache-priv-key.pem substituters = https://cache.nixos.org/ system-features = nixos-test benchmark big-parallel kvm trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= sun:<REDACTED> cache.lix.systems:aBnZUw8zA7H35Cz2RyKFVs3H4PlGTLawyY5KRbvJR8o= nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs= trusted-substituters = trusted-users = root root johnhamelink john @wheel extra-sandbox-paths = extra-substituters = https://cache.lix.systems https://nix-community.cachix.org ``` If there's anything else I could provide which would be helpful please do let me know 👍
Owner

Can we bother you to try to reproduce the build issue with Lix itself, @johnhamelink ?

A "way to achieve" this is to determine what should be the .drv to build, build it on your local system, then nix copy --to $your_server $lix_store_path and then you can use it to rebuild your server, and it will automatically pick up this store path without building it on the server and thus failing.

Otherwise, we will read your report in parallel and try to figure out things up.

Can we bother you to try to reproduce the build issue with Lix itself, @johnhamelink ? A "way to achieve" this is to determine what should be the .drv to build, build it on your local system, then `nix copy --to $your_server $lix_store_path` and then you can use it to rebuild your server, and it will automatically pick up this store path without building it on the server and thus failing. Otherwise, we will read your report in parallel and try to figure out things up.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#569
No description provided.