bug#42053: hidden derivation build with inferiors

bug#42053: hidden derivation build with inferiors

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 1440 bytes --]

Hi,

someone (kernel-help) on IRC today was having issues using an inferior
kernel with their os configuration and reported that guix was
"hanging". It turns out that guix was actually building the kernel in
the background, but did not report anything to the user. kernel-help
was kind enough to share their config, and I can confirm that guix is
building the kernel, but doesn't tell anything.

I was able to track this done to a call to operating-system-derivation,
which is supposed to return the path to a .drv file. I did not expect
it to have to build anything, since it's only computing a derivation
name.

Using the attached operating system declaration, defining it as "os",
you can see that calling ((operating-system-derivation os)
(open-connection)) "hangs" with no message, and a build user is
building the kernel (unless you already have the kernel built).

Note that the build process continues even if you interrupt that
procedure, and only stops after you exit guix repl entirely.

Going one step further, operating-system-derivation calls fold-services
(which is fine), then service-value (which is also fine) and calls that
value with the store, which builds the kernel without saying anything.
That value comes from the root service type, system-service-type and its
system-derivation procedure.

With some pk, I was able to find that mapm/accumulate-builds was the
culprit, but I don't know how to investigate further.

[-- Attachment #2: test-inferior.scm --]
[-- Type: text/x-scheme, Size: 2612 bytes --]

(use-modules (gnu)
	     (guix channels)
	     (srfi srfi-1)
	     (guix inferior)
	     (gnu packages linux)
	     (srfi srfi-1)
	     (ice-9 match)
	     (guix packages)
	     (gnu packages gnome))
(use-service-modules desktop networking ssh xorg)

(define kernel-inferior-channel
  ;; This is the old revision from which
  ;; we will obtain our desired kernel version.
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git")
	 (commit
	  "4f33bf5709a77a98f3d0330e45753d54675239ff"))))

(define linux-libre-4.19-inferior
  (first
   (lookup-inferior-packages
    (inferior-for-channels kernel-inferior-channel)
    "linux-libre" "4.19.129")))

(define-public gnome-minimal
  (package (inherit gnome)
	   (name "gnome-minimal")
	   (propagated-inputs
	    (remove (match-lambda
		      ((name _)
		       (member name '("baobab"
				      "cheese"
				      "eog"
				      "epiphany"
				      "evince"
				      "file-roller"
				      "gedit"
				      "gnome-boxes"
				      "gnome-calculator"
				      "gnome-calendar"
				      "gnome-characters"
				      "gnome-clocks"
				      "gnome-contacts"
				      "gnome-disk-utility"
				      "gnome-font-viewer"
				      "gnome-maps"
				      "gnome-weather"
				      "simple-scan"
				      "totem"
				      "gnome-default-applications"))))
		    (package-propagated-inputs gnome)))))
	   
(operating-system
 (locale "en_US.utf8")
 (kernel linux-libre-4.19-inferior)
 (timezone "America/Chicago")
 (keyboard-layout (keyboard-layout "us"))
 (host-name "t430")
 (users (cons* (user-account
		(name "brettg")
		(comment "Redacted")
		(group "users")
		(home-directory "/home/redacted")
		(supplementary-groups
		 '("wheel" "netdev" "audio" "video")))
	       %base-user-accounts))
 (packages
  (append
   (list (specification->package "nss-certs"))
   %base-packages))
 (services
  (append
   (list (service gnome-desktop-service-type
		  (gnome-desktop-configuration
		   (inherit config)
		   (gnome gnome-minimal)))
	 (service openssh-service-type)
	 (set-xorg-configuration
	  (xorg-configuration
	   (keyboard-layout keyboard-layout))))
   %desktop-services))
 (bootloader
  (bootloader-configuration
   (bootloader grub-efi-bootloader)
   (target "/boot/efi")
   (keyboard-layout keyboard-layout)))
 (swap-devices (list "/dev/sda2"))
 (file-systems
  (cons* (file-system
	  (mount-point "/boot/efi")
	  (device (uuid "0721-45AE" 'fat32))
	  (type "vfat"))
	 (file-system
	  (mount-point "/")
	  (device
	   (uuid "ce05e959-56a7-4520-ad73-556c82861537"
		 'ext4))
	  (type "ext4"))
	 %base-file-systems)))

bug#42053: hidden derivation build with inferiors

[Ludovic Courtès]

Hi,

Julien Lepiller <julien@lepiller.eu> skribis:

> someone (kernel-help) on IRC today was having issues using an inferior
> kernel with their os configuration and reported that guix was
> "hanging". It turns out that guix was actually building the kernel in
> the background, but did not report anything to the user. kernel-help
> was kind enough to share their config, and I can confirm that guix is
> building the kernel, but doesn't tell anything.

Aaah right.  What happens here is that grafts lead the inferior to start
building the kernel (this is expected).  However, ‘proxy’ in (guix
inferior) doesn’t forward build output to ‘current-build-output-port’.
That’s why there’s no message saying what’s being built, no spinner, etc.

I’ll see what can be done.

The problem has always been there but it was perhaps less visible before
710854304b1ab29332edcb76f3de532e0724c197, at least when substitutes were
available, because then the inferior wouldn’t start building things.

Ludo’.

bug#42053: hidden derivation build with inferiors

[Julien Lepiller]

Le 26 juin 2020 06:17:28 GMT-04:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :
>Hi,
>
>Julien Lepiller <julien@lepiller.eu> skribis:
>
>> someone (kernel-help) on IRC today was having issues using an
>inferior
>> kernel with their os configuration and reported that guix was
>> "hanging". It turns out that guix was actually building the kernel in
>> the background, but did not report anything to the user. kernel-help
>> was kind enough to share their config, and I can confirm that guix is
>> building the kernel, but doesn't tell anything.
>
>Aaah right.  What happens here is that grafts lead the inferior to
>start
>building the kernel (this is expected).  However, ‘proxy’ in (guix
>inferior) doesn’t forward build output to ‘current-build-output-port’.
>That’s why there’s no message saying what’s being built, no spinner,
>etc.
>
>I’ll see what can be done.
>
>The problem has always been there but it was perhaps less visible
>before
>710854304b1ab29332edcb76f3de532e0724c197, at least when substitutes
>were
>available, because then the inferior wouldn’t start building things.
>
>Ludo’.

Thanks for the answer. The build also starts with -n, which I didn't expect either. I suppose this is also because the inferior doesn't report anything?

bug#42053: hidden derivation build with inferiors

[Ludovic Courtès]

Hi,

Julien Lepiller <julien@lepiller.eu> skribis:

> Le 26 juin 2020 06:17:28 GMT-04:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :

[...]

>>The problem has always been there but it was perhaps less visible
>>before
>>710854304b1ab29332edcb76f3de532e0724c197, at least when substitutes
>>were
>>available, because then the inferior wouldn’t start building things.
>>
>>Ludo’.
>
> Thanks for the answer. The build also starts with -n, which I didn't expect either. I suppose this is also because the inferior doesn't report anything?

Yes, that and the fact that ‘-n’ no longer implies ‘--no-grafts’.

Ludo’.

bug#42053: hidden derivation build with inferiors

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> Julien Lepiller <julien@lepiller.eu> skribis:
>
>> someone (kernel-help) on IRC today was having issues using an inferior
>> kernel with their os configuration and reported that guix was
>> "hanging". It turns out that guix was actually building the kernel in
>> the background, but did not report anything to the user. kernel-help
>> was kind enough to share their config, and I can confirm that guix is
>> building the kernel, but doesn't tell anything.
>
> Aaah right.  What happens here is that grafts lead the inferior to start
> building the kernel (this is expected).  However, ‘proxy’ in (guix
> inferior) doesn’t forward build output to ‘current-build-output-port’.
> That’s why there’s no message saying what’s being built, no spinner, etc.
>
> I’ll see what can be done.

Not surprisingly, to fix it, we need to implement a proxy that
interprets all the messages exchanged so that it can write to
‘current-build-output-port’ when appropriate (that’s the usual
all-or-nothing problem with proxies.)

For the daemon rewrite in Scheme, we’ll need to have server stubs
anyway, so perhaps that’s an opportunity to fiddle with that.  Caleb?

Ludo’.