bug#70165: D-Bus system service breaks reconfiguration when /var/run/dbus is present + /run and /var/run are on separate file systems.

bug#70165: D-Bus system service breaks reconfiguration when /var/run/dbus is present + /run and /var/run are on separate file systems.

[Hilton Chain via Bug reports for GNU Guix]

Hi,

I have /var/run and /run on separate file systems, recently I noticed system
reconfiguration stopped with "guix system: error: rename-file: Invalid
cross-device link":

--8<---------------cut here---------------start------------->8---
newfstatat(AT_FDCWD, "/run", {st_mode=S_IFDIR|0755, st_size=440, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/run/dbus", {st_mode=S_IFDIR|0700, st_size=40, ...}, AT_SYMLINK_NOFOLLOW) = 0
mkdir("/run", 0777)                     = -1 EEXIST (File exists)
mkdir("/run/dbus", 0777)                = -1 EEXIST (File exists)
chown("/run/dbus", 988, 983)            = 0
chmod("/run/dbus", 0755)                = 0
symlink("/run/dbus", "/var/run/dbus")   = -1 EEXIST (File exists)
readlink("/var/run/dbus", 0x1634730, 100) = -1 EINVAL (Invalid argument)
openat(AT_FDCWD, "/var/run/dbus", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 17
newfstatat(17, "", {st_mode=S_IFDIR|0755, st_size=80, ...}, AT_EMPTY_PATH) = 0
getdents64(17, 0x16dfe10 /* 4 entries */, 32768) = 112
rename("/var/run/dbus/system_bus_socket", "/run/dbus/system_bus_socket") = -1 EXDEV (Invalid cross-device link)
close(13)                               = 0
write(2, "\33[1m\33[0mguix system: error: rena"..., 67^[[1m^[[0mguix system: error: rename-file: Invalid cross-device link
) = 67
exit_group(1)                           = ?
+++ exited with 1 +++
--8<---------------cut here---------------end--------------->8---

It's because /var/run/dbus was used for dbus service before, and now migration
to /run/dbus is done with ‘rename-file’:

--8<---------------cut here---------------start------------->8---
(rename-file (string-append "/var/run/dbus/" next)
             (string-append "/run/dbus/" next))
--8<---------------cut here---------------end--------------->8---

I think the logic can be improved for this case, but not sure how at the moment.
What do you think?

Thanks

bug#70165: D-Bus system service breaks reconfiguration when /var/run/dbus is present + /run and /var/run are on separate file systems.

[Liliana Marie Prikler]

Am Donnerstag, dem 04.04.2024 um 00:36 +0800 schrieb Hilton Chain:
> Hi,
> 
> I have /var/run and /run on separate file systems, recently I noticed
> system reconfiguration stopped with "guix system: error: rename-file:
> Invalid cross-device link":
> 
> --8<---------------cut here---------------start------------->8---
> newfstatat(AT_FDCWD, "/run", {st_mode=S_IFDIR|0755, st_size=440,
> ...}, AT_SYMLINK_NOFOLLOW) = 0
> newfstatat(AT_FDCWD, "/run/dbus", {st_mode=S_IFDIR|0700, st_size=40,
> ...}, AT_SYMLINK_NOFOLLOW) = 0
> mkdir("/run", 0777)                     = -1 EEXIST (File exists)
> mkdir("/run/dbus", 0777)                = -1 EEXIST (File exists)
> chown("/run/dbus", 988, 983)            = 0
> chmod("/run/dbus", 0755)                = 0
> symlink("/run/dbus", "/var/run/dbus")   = -1 EEXIST (File exists)
> readlink("/var/run/dbus", 0x1634730, 100) = -1 EINVAL (Invalid
> argument)
> openat(AT_FDCWD, "/var/run/dbus",
> O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 17
> newfstatat(17, "", {st_mode=S_IFDIR|0755, st_size=80, ...},
> AT_EMPTY_PATH) = 0
> getdents64(17, 0x16dfe10 /* 4 entries */, 32768) = 112
> rename("/var/run/dbus/system_bus_socket",
> "/run/dbus/system_bus_socket") = -1 EXDEV (Invalid cross-device link)
> close(13)                               = 0
> write(2, "\33[1m\33[0mguix system: error: rena"..., 67^[[1m^[[0mguix
> system: error: rename-file: Invalid cross-device link
> ) = 67
> exit_group(1)                           = ?
> +++ exited with 1 +++
> --8<---------------cut here---------------end--------------->8---
> 
> It's because /var/run/dbus was used for dbus service before, and now
> migration to /run/dbus is done with ‘rename-file’:
> 
> --8<---------------cut here---------------start------------->8---
> (rename-file (string-append "/var/run/dbus/" next)
>              (string-append "/run/dbus/" next))
> --8<---------------cut here---------------end--------------->8---
> 
> I think the logic can be improved for this case, but not sure how at
> the moment.
> What do you think?
Is there a *good* reason to have those be separate file systems? :D
What does the FHS say?

Anyhow, cross-fs symlinks and rename-file ought to work assuming that
both file-systems are already mounted.  Is one of them not needed for
boot?

Cheers

bug#70165: D-Bus system service breaks reconfiguration when /var/run/dbus is present + /run and /var/run are on separate file systems.

[Hilton Chain via Bug reports for GNU Guix]

On Fri, 05 Apr 2024 02:52:09 +0800,
Liliana Marie Prikler wrote:
>
> Am Donnerstag, dem 04.04.2024 um 00:36 +0800 schrieb Hilton Chain:
> > Hi,
> >
> > I have /var/run and /run on separate file systems, recently I noticed
> > system reconfiguration stopped with "guix system: error: rename-file:
> > Invalid cross-device link":
> >
> > --8<---------------cut here---------------start------------->8---
> > newfstatat(AT_FDCWD, "/run", {st_mode=S_IFDIR|0755, st_size=440,
> > ...}, AT_SYMLINK_NOFOLLOW) = 0
> > newfstatat(AT_FDCWD, "/run/dbus", {st_mode=S_IFDIR|0700, st_size=40,
> > ...}, AT_SYMLINK_NOFOLLOW) = 0
> > mkdir("/run", 0777)                     = -1 EEXIST (File exists)
> > mkdir("/run/dbus", 0777)                = -1 EEXIST (File exists)
> > chown("/run/dbus", 988, 983)            = 0
> > chmod("/run/dbus", 0755)                = 0
> > symlink("/run/dbus", "/var/run/dbus")   = -1 EEXIST (File exists)
> > readlink("/var/run/dbus", 0x1634730, 100) = -1 EINVAL (Invalid
> > argument)
> > openat(AT_FDCWD, "/var/run/dbus",
> > O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 17
> > newfstatat(17, "", {st_mode=S_IFDIR|0755, st_size=80, ...},
> > AT_EMPTY_PATH) = 0
> > getdents64(17, 0x16dfe10 /* 4 entries */, 32768) = 112
> > rename("/var/run/dbus/system_bus_socket",
> > "/run/dbus/system_bus_socket") = -1 EXDEV (Invalid cross-device link)
> > close(13)                               = 0
> > write(2, "\33[1m\33[0mguix system: error: rena"..., 67^[[1m^[[0mguix
> > system: error: rename-file: Invalid cross-device link
> > ) = 67
> > exit_group(1)                           = ?
> > +++ exited with 1 +++
> > --8<---------------cut here---------------end--------------->8---
> >
> > It's because /var/run/dbus was used for dbus service before, and now
> > migration to /run/dbus is done with ‘rename-file’:
> >
> > --8<---------------cut here---------------start------------->8---
> > (rename-file (string-append "/var/run/dbus/" next)
> >              (string-append "/run/dbus/" next))
> > --8<---------------cut here---------------end--------------->8---
> >
> > I think the logic can be improved for this case, but not sure how at
> > the moment.
> > What do you think?
> Is there a *good* reason to have those be separate file systems? :D
> What does the FHS say?
>
> Anyhow, cross-fs symlinks and rename-file ought to work assuming that
> both file-systems are already mounted.  Is one of them not needed for
> boot?

Specifically, I'm using tmpfs for them:

--8<---------------cut here---------------start------------->8---
(file-system
  (type "tmpfs")
  (mount-point "/run")
  (device "tmpfs")
  (flags '(no-dev strict-atime))
  (options "mode=0755,nr_inodes=800k,size=20%")
  (needed-for-boot? #t)
  (check? #f))

(file-system
  (type "tmpfs")
  (mount-point "/var/run")
  (device "tmpfs")
  (flags '(no-suid no-dev strict-atime))
  (options "mode=0755,nr_inodes=800k,size=20%")
  (needed-for-boot? #t)
  (check? #f))
--8<---------------cut here---------------end--------------->8---


And here's a reproducer:

--8<---------------cut here---------------start------------->8---
test_dir="$(mktemp --directory)"

sudo mount --type tmpfs none "$test_dir"
touch "$test_dir/test"

guile -c "(rename-file \"$test_dir/test\" \"/tmp/test\")"

sudo umount "$test_dir"
rmdir "$test_dir"
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
Backtrace:
In ice-9/boot-9.scm:
  1752:10  6 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
           5 (apply-smob/0 #<thunk 7061ede8c300>)
In ice-9/boot-9.scm:
    724:2  4 (call-with-prompt ("prompt") #<procedure 7061ede99320 …> …)
In ice-9/eval.scm:
    619:8  3 (_ #(#(#<directory (guile-user) 7061ede8fc80>)))
In ice-9/command-line.scm:
   185:19  2 (_ #<input: string 7061ede89850>)
In unknown file:
           1 (eval (rename-file "/tmp/tmp.9wyzRfQ28l/test" "/tmp/…") #)
           0 (rename-file "/tmp/tmp.9wyzRfQ28l/test" "/tmp/test")

ERROR: In procedure rename-file:
In procedure rename-file: Invalid cross-device link
--8<---------------cut here---------------end--------------->8---

bug#70165: D-Bus system service breaks reconfiguration when /var/run/dbus is present + /run and /var/run are on separate file systems.

[Liliana Marie Prikler]

Am Freitag, dem 05.04.2024 um 09:53 +0800 schrieb Hilton Chain:
> [H]ere's a reproducer:
> 
> --8<---------------cut here---------------start------------->8---
> test_dir="$(mktemp --directory)"
> 
> sudo mount --type tmpfs none "$test_dir"
> touch "$test_dir/test"
> 
> guile -c "(rename-file \"$test_dir/test\" \"/tmp/test\")"
> 
> sudo umount "$test_dir"
> rmdir "$test_dir"
> --8<---------------cut here---------------end--------------->8---
> 
> --8<---------------cut here---------------start------------->8---
> Backtrace:
> In ice-9/boot-9.scm:
>   1752:10  6 (with-exception-handler _ _ #:unwind? _ # _)
> In unknown file:
>            5 (apply-smob/0 #<thunk 7061ede8c300>)
> In ice-9/boot-9.scm:
>     724:2  4 (call-with-prompt ("prompt") #<procedure 7061ede99320 …>
> …)
> In ice-9/eval.scm:
>     619:8  3 (_ #(#(#<directory (guile-user) 7061ede8fc80>)))
> In ice-9/command-line.scm:
>    185:19  2 (_ #<input: string 7061ede89850>)
> In unknown file:
>            1 (eval (rename-file "/tmp/tmp.9wyzRfQ28l/test" "/tmp/…")
> #)
>            0 (rename-file "/tmp/tmp.9wyzRfQ28l/test" "/tmp/test")
> 
> ERROR: In procedure rename-file:
> In procedure rename-file: Invalid cross-device link
> --8<---------------cut here---------------end--------------->8---
If I understand this reproducer correctly, there aren't even symbolic
links involved, are there?

Adding Ludo to CC, because this looks like a Guile bug to me.

Cheers

bug#70165: D-Bus system service breaks reconfiguration when /var/run/dbus is present + /run and /var/run are on separate file systems.

[Ludovic Courtès]

Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:

> Am Freitag, dem 05.04.2024 um 09:53 +0800 schrieb Hilton Chain:

[...]

>>            0 (rename-file "/tmp/tmp.9wyzRfQ28l/test" "/tmp/test")
>> 
>> ERROR: In procedure rename-file:
>> In procedure rename-file: Invalid cross-device link
>> --8<---------------cut here---------------end--------------->8---
> If I understand this reproducer correctly, there aren't even symbolic
> links involved, are there?
>
> Adding Ludo to CC, because this looks like a Guile bug to me.

‘rename-file’ merely wraps rename(2), which errors out with EXDEV
(“Invalid cross-link device”) when the source and targets are on
different file systems.

So the case above is behaving as expected.

What that means is that probably we shouldn’t be using ‘rename-file’
directly in cases where the source and target might be on different file
systems, and instead do something like:

  (define (rename-file* old new)
    ;; Like rename-file, but handle the case when OLD and NEW are on
    ;; different file systems.
    (catch 'system-error
      (lambda ()
        (rename-file old new))
      (lambda args
        (if (= EXDEV (system-error-errno args))
            (begin
              (copy-file old new)
              (delete-file old))
            (apply throw args)))))

(Untested.)

HTH!

Ludo’.

bug#70165: D-Bus system service breaks reconfiguration when /var/run/dbus is present + /run and /var/run are on separate file systems.

[Collin J. Doering via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 410 bytes --]

Hi there,

I recently hit this issue, and was able work around it by:

1. Get root shell (via sudo or otherwise)
2. Disable dbus-system: 'herd stop guix-system'
3. Update guix: 'guix system reconfigure ...'
4. Reboot

Thought I would note it just in case its useful to someone else.

Kind regards,

-- 
Collin J. Doering

http://rekahsoft.ca
http://blog.rekahsoft.ca
http://git.rekahsoft.ca

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]