[Nix#9705] Nix fails to remount the store with all the required flags #72

New issue

Open

opened 2024-03-16 06:44:46 +00:00 by lix-bot · 0 comments

lix-bot commented

2024-03-16 06:44:46 +00:00

Member

Upstream-Issue: NixOS/nix#9705

Normally /nix/store is bind-mounted as read-only. That's to avoid accidental changes to the Nix store even by root. nix/nix-daemon though try to remount the nix-store to read-write in its own namespace, but the implementation is simplistic[1] and will fail in different scenarios.
[1]
https://github.com/NixOS/nix/blob/master/src/libstore/local-store.cc#L581-L582

I'm running nixos as a systemd-nspawn container, where the whole nixos filesystem lives on the host system (in /var/lib/machines/nixos), and the container is then run with it as a root filesystem ("booting" /sbin/init which execs /nix/store/d842nhr0ckaw7ijlf54q9dnm37dnn00p-nixos-system-nix-container-24.05.20240102.bd645e8/init).

In that case, the root and the nix-store are mounted with some additional flags compared to "normal" nixos (nodev and idmapped):

$ findmnt /
TARGET
  SOURCE                                    FSTYPE OPTIONS
/ /dev/mapper/root[/var/lib/machines/nixos] ext4   rw,nodev,relatime,idmapped

$ findmnt /nix/store/
TARGET     SOURCE                                              FSTYPE OPTIONS
/nix/store /dev/mapper/root[/var/lib/machines/nixos/nix/store] ext4   ro,nodev,relatime,idmapped

The problem is nix(nix-daemon) fails to remount the store to read-write:

$ strace -f -e mount nix-daemon 
strace: Process 206 attached
accepted connection from pid 230, user root (trusted)
strace: Process 239 attached
[pid   239] mount(NULL, "/nix/store", 0x7fa5ce409bd5, MS_REMOUNT|MS_BIND, NULL) = -1 EPERM (Operation not permitted)
unexpected Nix daemon error: error: remounting /nix/store writable: Operation not permitted

The most simple hack to solve the issue was to add the MS_NODEV flag in the remount call:

diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc
index 0f3c37c8a..ce6dd13ae 100644
--- a/src/libstore/local-store.cc
+++ b/src/libstore/local-store.cc
@@ -579,7 +579,7 @@ void LocalStore::makeStoreWritable()
         throw SysError("getting info about the Nix store mount point");
 
     if (stat.f_flag & ST_RDONLY) {
-        if (mount(0, realStoreDir.get().c_str(), "none", MS_REMOUNT | MS_BIND, 0) == -1)
+        if (mount(0, realStoreDir.get().c_str(), "none", MS_REMOUNT | MS_BIND | MS_NODEV, 0) == -1)
             throw SysError("remounting %1% writable", realStoreDir);
     }
 #endif

after this change, nix/nix-daemon successfully can remount the /nix/store.

Now a more robust solution would probably be to first introspect all the mount flags and then only flip the read-only flag - other mount cli tools seem to do that (busybox or util-linux mount for ex).

I can try to pursue that fix, if it's acceptable.

Upstream-Issue: https://git.lix.systems/NixOS/nix/issues/9705 Normally `/nix/store` is bind-mounted as read-only. That's to avoid accidental changes to the Nix store even by root. `nix`/`nix-daemon` though try to remount the nix-store to read-write in its own namespace, but the implementation is simplistic[1] and will fail in different scenarios. [1] https://github.com/NixOS/nix/blob/master/src/libstore/local-store.cc#L581-L582 I'm running nixos as a `systemd-nspawn` container, where the whole nixos filesystem lives on the host system (in `/var/lib/machines/nixos`), and the container is then run with it as a root filesystem ("booting" `/sbin/init` which execs `/nix/store/d842nhr0ckaw7ijlf54q9dnm37dnn00p-nixos-system-nix-container-24.05.20240102.bd645e8/init`). In that case, the root and the nix-store are mounted with some additional flags compared to "normal" nixos (nodev and idmapped): ``` $ findmnt / TARGET SOURCE FSTYPE OPTIONS / /dev/mapper/root[/var/lib/machines/nixos] ext4 rw,nodev,relatime,idmapped $ findmnt /nix/store/ TARGET SOURCE FSTYPE OPTIONS /nix/store /dev/mapper/root[/var/lib/machines/nixos/nix/store] ext4 ro,nodev,relatime,idmapped ``` The problem is nix(nix-daemon) fails to remount the store to read-write: ``` $ strace -f -e mount nix-daemon strace: Process 206 attached accepted connection from pid 230, user root (trusted) strace: Process 239 attached [pid 239] mount(NULL, "/nix/store", 0x7fa5ce409bd5, MS_REMOUNT|MS_BIND, NULL) = -1 EPERM (Operation not permitted) unexpected Nix daemon error: error: remounting /nix/store writable: Operation not permitted ``` The most simple hack to solve the issue was to add the MS_NODEV flag in the remount call: ```diff diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index 0f3c37c8a..ce6dd13ae 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -579,7 +579,7 @@ void LocalStore::makeStoreWritable() throw SysError("getting info about the Nix store mount point"); if (stat.f_flag & ST_RDONLY) { - if (mount(0, realStoreDir.get().c_str(), "none", MS_REMOUNT | MS_BIND, 0) == -1) + if (mount(0, realStoreDir.get().c_str(), "none", MS_REMOUNT | MS_BIND | MS_NODEV, 0) == -1) throw SysError("remounting %1% writable", realStoreDir); } #endif ``` after this change, nix/nix-daemon successfully can remount the /nix/store. Now a more robust solution would probably be to first introspect all the mount flags and then only flip the read-only flag - other `mount` cli tools seem to do that (busybox or util-linux mount for ex). I can try to pursue that fix, if it's acceptable.