bug#59823: an installer dump was sent

bug#59823: an installer dump was sent

[So'n Typ im Internet]

[-- Attachment #1: Type: text/plain, Size: 97 bytes --]

Hello,
I just sent an installer dump to you. The name is "installer-dump-2a14e489"

best regards

[-- Attachment #2: Type: text/html, Size: 177 bytes --]

bug#59823: an installer dump was sent

[Mathieu Othacehe]

[-- Attachment #1: Type: text/plain, Size: 3672 bytes --]


Hello,

Thanks for reporting! So the error is:

--8<---------------cut here---------------start------------->8---
mathieu@meije ~$ cat dump.2022-12-04.10.54.06/installer-backtrace
In ./gnu/installer/steps.scm:
   150:13 19 (run ((locale . "de_DE.utf8")) #:todo-steps _ #:done-steps _)
   150:13 18 (run ((welcome . #t) (locale . "de_DE.utf8")) #:todo-steps _ #:done-steps _)
   150:13 17 (run ((timezone . "Europe/Berlin") (welcome . #t) (locale . "de_DE.utf8")) #:todo-steps _ #:done-steps _)
   150:13 16 (run ((keymap "de" #f #f) (timezone . "Europe/Berlin") (welcome . #t) (locale . "de_DE.utf8")) #:todo-steps _ #:done-steps _)
   150:13 15 (run ((hostname . "guix-hp") (keymap "de" #f #f) (timezone . "Europe/Berlin") (welcome . #t) (locale . "de_DE.utf8")) #:todo-steps _ #:done-steps _)
   150:13 14 (run ((network (select-technology . #<<technology> name: "Wired" type: "ethernet" powered?: #t connected?: #t>) (power-technology . #<unspecified>) (connect-service . #<<servi…>) …) …) …)
   150:13 13 (run ((substitutes #t) (network (select-technology . #<<technology> name: "Wired" type: "ethernet" powered?: #t connected?: #t>) (power-technology . #<unspecified>) (# . #<<se…>) …) …) …)
   150:13 12 (run ((user #<<user> name: "root" real-name: "" group: "users" password: <secret> home-directory: "/root"> #<<user> name: "typ" real-name: "Typ" group: "users" password: <secret…>) …) …)
   148:23 11 (run ((services #<<system-service> name: "GNOME" type: desktop recommended?: #f snippet: ((service gnome-desktop-service-type)) packages: ()> #<<system-service> name: "OpenSSH …> …) …) …)
In ./gnu/installer/newt/partition.scm:
    814:4 10 (run-partitioning-page)
In srfi/srfi-1.scm:
    634:9  9 (for-each #<procedure 7fc5c4397698 at ./gnu/installer/parted.scm:1524:14 (file-name)> _)
In ./gnu/installer/parted.scm:
  1528:22  8 (_ #<time type: time-duration nanosecond: 261179000 second: 4> #t)
In ice-9/boot-9.scm:
  1685:16  7 (raise-exception _ #:continuable? _)
  1780:13  6 (_ #<&compound-exception components: (#<&error> #<&origin origin: #f> #<&message message: "~A"> #<&irritants irritants: ("Gerät /dev/sdd wird noch verwendet.")> #<&exception-with-kind…>)
In ice-9/eval.scm:
    619:8  5 (_ #(#(#(#<directory (guile-user) 7fc5d77d0c80>) misc-error (#f "~A" ("Gerät /dev/sdd wird noch verwendet.") #f)) #<variable 7fc5c0c7d150 value: #<unspecified>> #<variable 7fc5c0c7…> …))
   626:19  4 (_ #(#(#(#<directory (guile-user) 7fc5d77d0c80>) misc-error (#f "~A" ("Gerät /dev/sdd wird noch verwendet.") #f)) #<variable 7fc5c0c7d150 value: #<unspecified>> #<variable 7fc5c0c7…> …))
In ./gnu/installer/dump.scm:
     58:4  3 (prepare-dump misc-error (#f "~A" ("Gerät /dev/sdd wird noch verwendet.") #f) #:result _)
In ice-9/ports.scm:
   433:17  2 (call-with-output-file _ _ #:binary _ #:encoding _)
In ./gnu/installer/dump.scm:
    60:27  1 (_ #<output: installer-backtrace 20>)
In unknown file:
           0 (make-stack #t)
./gnu/installer/dump.scm:62:36: Gerät /dev/sdd wird noch verwendet.
--8<---------------cut here---------------end--------------->8---

Which means that the delay in the "with-delay-device-in-use?" procedure
is probably not high enough.

Here is an attached patch to bump it from 16 to 96 seconds. I also
uploaded an image built on top of of the version-1.4.0 with this patch
if you are up for a retry :), you can download it this way:

--8<---------------cut here---------------start------------->8---
wget https://othacehe.org/files/installer.iso
--8<---------------cut here---------------end--------------->8---

Thanks,

Mathieu

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-installer-Dump-the-device-in-use-retry-count-to-96.patch --]
[-- Type: text/x-patch, Size: 1029 bytes --]

From b53d7f0c930f029d6b17be92dfa408b74615c1a5 Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <othacehe@gnu.org>
Date: Mon, 5 Dec 2022 08:56:43 +0100
Subject: [PATCH 1/1] installer: Dump the device in use retry count to 96.

Fixes: <https://issues.guix.gnu.org/59823>

* gnu/installer/parted.scm (with-delay-device-in-use): Bump it.
---
 gnu/installer/parted.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/installer/parted.scm b/gnu/installer/parted.scm
index 82375d29e3..518b40a7ea 100644
--- a/gnu/installer/parted.scm
+++ b/gnu/installer/parted.scm
@@ -348,7 +348,7 @@ (define (with-delay-device-in-use? file-name)
 fail. See rereadpt function in wipefs.c of util-linux for an explanation."
   ;; Kernel always return EINVAL for BLKRRPART on loopdevices.
   (and (not (string-match "/dev/loop*" file-name))
-       (let loop ((try 16))
+       (let loop ((try 96))
          (usleep 250000)
          (let ((in-use? (device-in-use? file-name)))
            (if (and in-use? (> try 0))
-- 
2.38.1

bug#59823: an installer dump was sent

[So'n Typ im Internet]

[-- Attachment #1: Type: text/plain, Size: 3989 bytes --]

Hi Mathieu,

Thanks so much dir the help.

Best regards,

Typ

Mathieu Othacehe <othacehe@gnu.org> schrieb am Mo. 5. Dez. 2022 um 09:18:

>
> Hello,
>
> Thanks for reporting! So the error is:
>
> --8<---------------cut here---------------start------------->8---
> mathieu@meije ~$ cat dump.2022-12-04.10.54.06/installer-backtrace
> In ./gnu/installer/steps.scm:
>    150:13 19 (run ((locale . "de_DE.utf8")) #:todo-steps _ #:done-steps _)
>    150:13 18 (run ((welcome . #t) (locale . "de_DE.utf8")) #:todo-steps _
> #:done-steps _)
>    150:13 17 (run ((timezone . "Europe/Berlin") (welcome . #t) (locale .
> "de_DE.utf8")) #:todo-steps _ #:done-steps _)
>    150:13 16 (run ((keymap "de" #f #f) (timezone . "Europe/Berlin")
> (welcome . #t) (locale . "de_DE.utf8")) #:todo-steps _ #:done-steps _)
>    150:13 15 (run ((hostname . "guix-hp") (keymap "de" #f #f) (timezone .
> "Europe/Berlin") (welcome . #t) (locale . "de_DE.utf8")) #:todo-steps _
> #:done-steps _)
>    150:13 14 (run ((network (select-technology . #<<technology> name:
> "Wired" type: "ethernet" powered?: #t connected?: #t>) (power-technology .
> #<unspecified>) (connect-service . #<<servi…>) …) …) …)
>    150:13 13 (run ((substitutes #t) (network (select-technology .
> #<<technology> name: "Wired" type: "ethernet" powered?: #t connected?: #t>)
> (power-technology . #<unspecified>) (# . #<<se…>) …) …) …)
>    150:13 12 (run ((user #<<user> name: "root" real-name: "" group:
> "users" password: <secret> home-directory: "/root"> #<<user> name: "typ"
> real-name: "Typ" group: "users" password: <secret…>) …) …)
>    148:23 11 (run ((services #<<system-service> name: "GNOME" type:
> desktop recommended?: #f snippet: ((service gnome-desktop-service-type))
> packages: ()> #<<system-service> name: "OpenSSH …> …) …) …)
> In ./gnu/installer/newt/partition.scm:
>     814:4 10 (run-partitioning-page)
> In srfi/srfi-1.scm:
>     634:9  9 (for-each #<procedure 7fc5c4397698 at
> ./gnu/installer/parted.scm:1524:14 (file-name)> _)
> In ./gnu/installer/parted.scm:
>   1528:22  8 (_ #<time type: time-duration nanosecond: 261179000 second:
> 4> #t)
> In ice-9/boot-9.scm:
>   1685:16  7 (raise-exception _ #:continuable? _)
>   1780:13  6 (_ #<&compound-exception components: (#<&error> #<&origin
> origin: #f> #<&message message: "~A"> #<&irritants irritants: ("Gerät
> /dev/sdd wird noch verwendet.")> #<&exception-with-kind…>)
> In ice-9/eval.scm:
>     619:8  5 (_ #(#(#(#<directory (guile-user) 7fc5d77d0c80>) misc-error
> (#f "~A" ("Gerät /dev/sdd wird noch verwendet.") #f)) #<variable
> 7fc5c0c7d150 value: #<unspecified>> #<variable 7fc5c0c7…> …))
>    626:19  4 (_ #(#(#(#<directory (guile-user) 7fc5d77d0c80>) misc-error
> (#f "~A" ("Gerät /dev/sdd wird noch verwendet.") #f)) #<variable
> 7fc5c0c7d150 value: #<unspecified>> #<variable 7fc5c0c7…> …))
> In ./gnu/installer/dump.scm:
>      58:4  3 (prepare-dump misc-error (#f "~A" ("Gerät /dev/sdd wird noch
> verwendet.") #f) #:result _)
> In ice-9/ports.scm:
>    433:17  2 (call-with-output-file _ _ #:binary _ #:encoding _)
> In ./gnu/installer/dump.scm:
>     60:27  1 (_ #<output: installer-backtrace 20>)
> In unknown file:
>            0 (make-stack #t)
> ./gnu/installer/dump.scm:62:36: Gerät /dev/sdd wird noch verwendet.
> --8<---------------cut here---------------end--------------->8---
>
> Which means that the delay in the "with-delay-device-in-use?" procedure
> is probably not high enough.
>
> Here is an attached patch to bump it from 16 to 96 seconds. I also
> uploaded an image built on top of of the version-1.4.0 with this patch
> if you are up for a retry :), you can download it this way:
>
> --8<---------------cut here---------------start------------->8---
> wget https://othacehe.org/files/installer.iso
> --8<---------------cut here---------------end--------------->8---
>
> Thanks,
>
> Mathieu
>

[-- Attachment #2: Type: text/html, Size: 5218 bytes --]

bug#59823: an installer dump was sent

[Mathieu Othacehe]

Hello,

So I had a closer look and we do have this strange kernel warning: 

--8<---------------cut here---------------start------------->8---
Dec  4 10:49:05 localhost vmunix: [ 1351.610773] device-mapper: ioctl: remove_all left 1 open device(s)
--8<---------------cut here---------------end--------------->8---

I also noticed that we do not have the following trace:

--8<---------------cut here---------------start------------->8---
/dev/sdX is not eligible because it is the installation device. 
--8<---------------cut here---------------end--------------->8---

which indicates that the "eligible-devices" procedure failed to identify
the installation device. So /dev/sdd could be the installation device
and the "with-delay-device-in-use?" always reports true because it is in
fact always in use.

My proposed patch probably won't help and the question is, why is the
installation device not detected?

Would it be possible for you to try the installation with an
instrumented installer that I would provide you with?

Thanks,

Mathieu

bug#59823: an installer dump was sent

[So'n Typ im Internet]

[-- Attachment #1: Type: text/plain, Size: 1505 bytes --]

Hallo Mathieu,

Thank you so much for your effort.
I already managed the installation by burning the iso directly to my usb
drive and not via Ventoy.
Everything works fine.
I should have try this in the first place. Sometimes the ways of the lord
gnu are inscrutable.

Big thanks again and a warm regards to you and your team.

Typ.


Mathieu Othacehe <othacehe@gnu.org> schrieb am Mi. 7. Dez. 2022 um 10:31:

>
> Hello,
>
> So I had a closer look and we do have this strange kernel warning:
>
> --8<---------------cut here---------------start------------->8---
> Dec  4 10:49:05 localhost vmunix: [ 1351.610773] device-mapper: ioctl:
> remove_all left 1 open device(s)
> --8<---------------cut here---------------end--------------->8---
>
> I also noticed that we do not have the following trace:
>
> --8<---------------cut here---------------start------------->8---
> /dev/sdX is not eligible because it is the installation device.
> --8<---------------cut here---------------end--------------->8---
>
> which indicates that the "eligible-devices" procedure failed to identify
> the installation device. So /dev/sdd could be the installation device
> and the "with-delay-device-in-use?" always reports true because it is in
> fact always in use.
>
> My proposed patch probably won't help and the question is, why is the
> installation device not detected?
>
> Would it be possible for you to try the installation with an
> instrumented installer that I would provide you with?
>
> Thanks,
>
> Mathieu
>

[-- Attachment #2: Type: text/html, Size: 2217 bytes --]

bug#59823: an installer dump was sent

[Mathieu Othacehe]

Hello,

It's really good that you managed to install it anyway. Thanks for
persevering :) Nevertheless we need to fix the problem.

> I already managed the installation by burning the iso directly to my
> usb drive and not via Ventoy.

Turns out Ventoy was the crux of the issue here. Ventox allows to
install multiple iso images just by copying them in a directory of the
drive. It then creates a device mapping which look like that:

NAME       MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda          8:0    0 238.5G  0 disk 
├─sda1       8:1    0   512M  0 part 
└─sda2       8:2    0   238G  0 part 
sdb          8:16   0  14.6G  0 disk 
├─sdb1       8:17   0  14.6G  0 part 
│ └─ventoy 253:0    0 842.9M  1 dm   
└─sdb2       8:18   0    32M  0 part 

This device mapping defeats our "eligible-devices" procedure. It's
because the UUID that is passed as the root=xxx argument of the Linux
command line is in fact related to /dev/dm-0 which is different from
/dev/sdb.

I had a look to the parted sources and it should detect mapped devices
but for some reason it doesn't. Figuring it out and fixing it is maybe a
bit risky before the release.

So I'm trying to figure out a (cheap) way to make the correlation
between /dev/dm-0 and /dev/sdb to exclude the latter in the eligible
devices procedure.

Any idea is welcomed!

Thanks,

Mathieu

bug#59823: [1.4.0rc1] Installer fails to identify installation device on Ventoy-made images

[Mathieu Othacehe]

[-- Attachment #1: Type: text/plain, Size: 170 bytes --]


Hello,

The attached patch fixes it for me. We could maybe use libdevmapper
instead of the plain "dmsetup" call but that's not critical in my
opinion.

Thanks,

Mathieu

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-installer-Detect-mapped-installation-devices.patch --]
[-- Type: text/x-patch, Size: 2511 bytes --]

From 0afda5b3ed32e73bece9db96ab970d83f9f2e74b Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <othacehe@gnu.org>
Date: Thu, 8 Dec 2022 13:24:02 +0100
Subject: [PATCH 1/1] installer: Detect mapped installation devices.

Fixes: <https://issues.guix.gnu.org/59823>

* gnu/installer/parted.scm (mapped-device?,
mapped-device->parent-partition-path): New procedures.
(eligible-devices): Detect mapped installation devices using the new
procedures.
---
 gnu/installer/parted.scm | 35 ++++++++++++++++++++++++++++++++++-
 1 file changed, 34 insertions(+), 1 deletion(-)

diff --git a/gnu/installer/parted.scm b/gnu/installer/parted.scm
index 82375d29e3..058f2a8dab 100644
--- a/gnu/installer/parted.scm
+++ b/gnu/installer/parted.scm
@@ -379,12 +379,45 @@ (define (installer-root-partition-path)
 (define %min-device-size
   (* 2 GIBIBYTE-SIZE)) ;2GiB
 
+(define (mapped-device? device)
+  "Return #true if DEVICE is a mapped device, false otherwise."
+  (string-prefix? "/dev/dm-" device))
+
+(define (mapped-device->parent-partition-path device)
+  "Return the parent partition path of the mapped DEVICE."
+  (let* ((command `("dmsetup" "deps" ,device "-o" "devname"))
+         (parent #f)
+         (handler
+          (lambda (input)
+            (let ((result
+                   (string-match "\\(([^\\)]+)\\)"
+                                 (get-string-all input))))
+              (and result
+                   (set! parent
+                         (format #f "/dev/~a"
+                                 (match:substring result 1))))))))
+    (run-external-command-with-handler handler command)
+    parent))
+
 (define (eligible-devices)
   "Return all the available devices except the install device and the devices
 which are smaller than %MIN-DEVICE-SIZE."
 
   (define the-installer-root-partition-path
-    (installer-root-partition-path))
+    (let ((root-path
+           (installer-root-partition-path)))
+      (cond
+       ((mapped-device? root-path)
+        ;; If the partition is a mapped device (/dev/dm-X), locate the parent
+        ;; partition.  It is the case when Ventoy is used to host the
+        ;; installation image.
+        (let ((parent-path
+               (mapped-device->parent-partition-path root-path)))
+          (installer-log-line "mapped device ~a -> ~a"
+                              parent-path root-path)
+          parent-path))
+       (else
+        root-path))))
 
   (define (small-device? device)
     (let ((length (device-length device))
-- 
2.38.1

bug#59823: [1.4.0rc1] Installer fails to identify installation device on Ventoy-made images

[Ludovic Courtès]

Hi!

Mathieu Othacehe <othacehe@gnu.org> skribis:

>>From 0afda5b3ed32e73bece9db96ab970d83f9f2e74b Mon Sep 17 00:00:00 2001
> From: Mathieu Othacehe <othacehe@gnu.org>
> Date: Thu, 8 Dec 2022 13:24:02 +0100
> Subject: [PATCH 1/1] installer: Detect mapped installation devices.
>
> Fixes: <https://issues.guix.gnu.org/59823>
>
> * gnu/installer/parted.scm (mapped-device?,
> mapped-device->parent-partition-path): New procedures.
> (eligible-devices): Detect mapped installation devices using the new
> procedures.

Well done!

> +(define (mapped-device->parent-partition-path device)
> +  "Return the parent partition path of the mapped DEVICE."

Maybe ‘mapped-device-parent-partition’?  (“Path” being used for search
paths, not for file names.)

> +  (let* ((command `("dmsetup" "deps" ,device "-o" "devname"))
> +         (parent #f)
> +         (handler
> +          (lambda (input)
> +            (let ((result
> +                   (string-match "\\(([^\\)]+)\\)"
> +                                 (get-string-all input))))
> +              (and result
> +                   (set! parent
> +                         (format #f "/dev/~a"
> +                                 (match:substring result 1))))))))
> +    (run-external-command-with-handler handler command)
> +    parent))

The advantage of ‘run-external-command-with-handler’ is logging I guess,
but that leads to an awkward style compared to using ‘open-input-pipe’,
which I would favor.

Then we want to parse a line that looks like this:

--8<---------------cut here---------------start------------->8---
1 dependencies  : (sda2)
--8<---------------cut here---------------end--------------->8---

I’m not super confident relying on this format.

‘dmsetup deps …’ does this (here /dev/dm-0 is the same as
/dev/mapper/root):

--8<---------------cut here---------------start------------->8---
ioctl(3, DM_VERSION, {version=[4, 0, 0], data_size=16384, flags=DM_EXISTS_FLAG} => {version=[4, 47, 0], data_size=16384, flags=DM_EXISTS_FLAG}) = 0
newfstatat(AT_FDCWD, "/dev/dm-0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0xfd, 0), ...}, 0) = 0
newfstatat(AT_FDCWD, "/dev/mapper/dm-0", 0x7ffe78f99020, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/mapper", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
newfstatat(4, "", {st_mode=S_IFDIR|0755, st_size=80, ...}, AT_EMPTY_PATH) = 0
getdents64(4, 0x563c1d72f8b0 /* 4 entries */, 32768) = 104
newfstatat(AT_FDCWD, "/dev/mapper/root", {st_mode=S_IFBLK|0600, st_rdev=makedev(0xfd, 0), ...}, 0) = 0
close(4)                                = 0
ioctl(3, DM_TABLE_DEPS, {version=[4, 0, 0], data_size=16384, data_start=312, name="root", flags=DM_EXISTS_FLAG} => {version=[4, 47, 0], data_size=328, data_start=312, dev=makedev(0xfd, 0), name="root", uuid="CRYPT-LUKS1-xxxxxx-root", target_count=1, open_count=1, event_nr=0, flags=DM_EXISTS_FLAG|DM_ACTIVE_PRESENT_FLAG, ...}) = 0
newfstatat(1, "", {st_mode=S_IFCHR|0622, st_rdev=makedev(0x88, 0), ...}, AT_EMPTY_PATH) = 0
readlink("/sys/dev/block/8:2", "../../devices/pci0000:00/0000:00:17.0/ata3/host2/target2:0:0/2:0:0:0/block/sda/sda2", 4095) = 83
write(1, "1 dependencies\t: (sda2)\n", 24) = 24
--8<---------------cut here---------------end--------------->8---

We could implement those ioctls, but not in the time frame we’re talking
about…

The libdevmapper interface doesn’t look more pleasant than the raw
ioctl.

Anyway, all in all, calling out to dmsetup looks reasonable for now; I
have a slight preference for using ‘open-pipe* OPEN_READ’, but no big
deal.  Perhaps add a comment showing what the line we’re parsing should
look like.

>  (define (eligible-devices)
>    "Return all the available devices except the install device and the devices
>  which are smaller than %MIN-DEVICE-SIZE."
>  
>    (define the-installer-root-partition-path
> -    (installer-root-partition-path))
> +    (let ((root-path
> +           (installer-root-partition-path)))

Just ‘root’.  :-)

Thanks a lot for fixing this!  You can push to ‘version-1.4.0’.
Hopefully I’ll prepare RC2 tomorrow evening.

Ludo’.

bug#59823: [1.4.0rc1] Installer fails to identify installation device on Ventoy-made images

[Mathieu Othacehe]

Hola,

> Anyway, all in all, calling out to dmsetup looks reasonable for now; I
> have a slight preference for using ‘open-pipe* OPEN_READ’, but no big
> deal.  Perhaps add a comment showing what the line we’re parsing should
> look like.

Yeah, I agree that open-pipe would be a bit clearer, but as this is
already tested as-is plus we'll switch to the ioctl after the release, I
think we can proceed.

>>  (define (eligible-devices)
>>    "Return all the available devices except the install device and the devices
>>  which are smaller than %MIN-DEVICE-SIZE."
>>  
>>    (define the-installer-root-partition-path
>> -    (installer-root-partition-path))
>> +    (let ((root-path
>> +           (installer-root-partition-path)))
>
> Just ‘root’.  :-)

Fixed, added a few comments and pushed.

Thanks for having a look!

Mathieu

bug#59823: [1.4.0rc1] Installer fails to identify installation device on Ventoy-made images

[Ludovic Courtès]

¡Hola!

Mathieu Othacehe <othacehe@gnu.org> skribis:

> Yeah, I agree that open-pipe would be a bit clearer, but as this is
> already tested as-is plus we'll switch to the ioctl after the release, I
> think we can proceed.

Yup, that makes sense to me.

> Fixed, added a few comments and pushed.

Thank you!

Ludo’.