bug#42164: Combining file-append with gexps results in incomprehensible errors

bug#42164: Combining file-append with gexps results in incomprehensible errors

[Jakub Kądziołka]

[-- Attachment #1: Type: text/plain, Size: 2784 bytes --]

Consider this minimal operating-system definition:

(use-modules (gnu))
(use-package-modules gcc)

(operating-system
  (host-name "test")
  (timezone "UTC")
  (bootloader (bootloader-configuration
                (bootloader grub-efi-bootloader)
                (target "/boot/efi")))
  (file-systems (cons*
                  (file-system
                    (device (file-system-label "root"))
                    (mount-point "/")
                    (type "ext4"))
                  %base-file-systems))
  (packages '())
  (services (cons*
              (service special-files-service-type
                       `(("/lib64" ,(directory-union "rustup-libs"
                                      (list
                                        (file-append glibc "/lib")
                                        (file-append #~#$gcc:lib "/lib"))))))
              %base-services)))

I would expect this way of specifying a specific output of a package to
work, but it results in the following error:

ice-9/boot-9.scm:1515:18: object is not an exception of the right type #<&gexp-input-error input: #<gexp #<gexp-input #<package gcc@7.5.0 gnu/packages/gcc.scm:520 7f06c996c960>:lib> 7f06c6b06990>> #<record-type &package-input-error>

This also happens when I omit the directory-union:

              (service special-files-service-type
                       `(("/lib64" ,(file-append #~#$gcc:lib "/lib"))))

What *does* work, is this variant with string-append:

              (service special-files-service-type
                       `(("/lib64" ,#~(string-append #$gcc:lib "/lib"))))

However, using it in the directory-union breaks again:

              (service special-files-service-type
                       `(("/lib64" ,(directory-union "rustup-libs"
                                      (list
                                        (file-append glibc "/lib")
                                        #~(string-append #$gcc:lib "/lib"))))))

ERROR: In procedure opendir:
Wrong type (expecting string): (string-append "/gnu/store/mdxmdhrlkgdik6ay9pzmmy8mjcbibpwb-gcc-7.5.0-lib" "/lib")
builder for `/gnu/store/p5hf7hqxn35fgsb75s5i7326vyzb8jkr-rustup-libs.drv' failed with exit code 1

I have figured out that the following does work:

              (service special-files-service-type
                       `(("/lib64" ,#~(directory-union "rustup-libs"
                                        (list
                                          (string-append #$glibc "/lib")
                                          (string-append #$gcc:lib "/lib"))))))

However, I would expect the other variants to work as well. The
documentation and error messages are lacking in this regard.

Regards,
Jakub Kądziołka

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#42164: Combining file-append with gexps results in incomprehensible errors

[Maxim Cournoyer]

Hi Jakub!

Jakub Kądziołka <kuba@kadziolka.net> writes:

> Consider this minimal operating-system definition:
>
> (use-modules (gnu))
> (use-package-modules gcc)
>
> (operating-system
>   (host-name "test")
>   (timezone "UTC")
>   (bootloader (bootloader-configuration
>                 (bootloader grub-efi-bootloader)
>                 (target "/boot/efi")))
>   (file-systems (cons*
>                   (file-system
>                     (device (file-system-label "root"))
>                     (mount-point "/")
>                     (type "ext4"))
>                   %base-file-systems))
>   (packages '())
>   (services (cons*
>               (service special-files-service-type
>                        `(("/lib64" ,(directory-union "rustup-libs"
>                                       (list
>                                         (file-append glibc "/lib")
>                                         (file-append #~#$gcc:lib "/lib"))))))
>               %base-services)))
>
> I would expect this way of specifying a specific output of a package to
> work, but it results in the following error:
>
> ice-9/boot-9.scm:1515:18: object is not an exception of the right type #<&gexp-input-error input: #<gexp #<gexp-input #<package gcc@7.5.0 gnu/packages/gcc.scm:520 7f06c996c960>:lib> 7f06c6b06990>> #<record-type &package-input-error>

[...]

> I have figured out that the following does work:
>
>               (service special-files-service-type
>                        `(("/lib64" ,#~(directory-union "rustup-libs"
>                                         (list
>                                           (string-append #$glibc "/lib")
>                                           (string-append #$gcc:lib "/lib"))))))
>
> However, I would expect the other variants to work as well. The
> documentation and error messages are lacking in this regard.

This seems to come up every time I have to specify a non-default package
output; I've investigated the issue a bit and it seems to be caused by
the lack of a gexp-compiler for the gexp object resulting from
#$package:output, which is of type gexp-input as represented by, for
example:

#<gexp-input #<package opendht@2.2.0rc4 /home/maxim/src/guix/gnu/packages/networking.scm:3426 7f2dbcae1aa0>:tools>

From the REPL

(lower-object (gexp (ungexp opendht "tools")))
$17 = #<procedure 7f2db56ed100 at guix/gexp.scm:246:2 (state)>
scheme@(gnu services jami)> ,run-in-store $17
While executing meta-command:
ERROR:
  1. &gexp-input-error: #<gexp #<gexp-input #<package opendht@2.2.0rc4
  /home/maxim/src/guix/gnu/packages/networking.scm:3426
  7f2dbcae1aa0>:tools> 7f2db7427690>
  
So if we were to define/register a gexp-compiler for this kind of
gexp-input, we could make it work, but I haven't yet investigated how
easy/difficult that would be.

Seems worthwhile though, this issue is quite annoying when attempting to
work with specific outputs!

Thanks for the detailed report.

Maxim

bug#42164: Combining file-append with gexps results in incomprehensible errors

[Brian Cully]

	I was having a similar issue, so I dug into this issue a bit,
and it turns out that you *can* select an output from file-append, but
the syntax is a bit wonky:

---[snip]---
(let ((fa (file-append pkg "/path")))
  $~$#fa:output)
---[snip]---

	I’ve tried to remove the let:

---[snip]---
$~$#(file-append pkg "/path"):output
---[snip]---

	But this confuses the reader, an it things ‘:output’ is a
variable. Not too surprising, but it does make the Guix configuration
files more awkward, since now you have to ‘define’ the file-append
operations at the top level so you can refer to them with the syntax
above.

	I would prefer to have something like any of the following (in
order of preference):

---[snip]---
(file-append pkg:output "/path")
(file-append (list pkg "output") "/path")
(file-append $~$#pkg:output "/path")
---[snip]---

	The reason being that it’s awkward to select the output after
the file-append due to syntax, and it’s less intuitive than trying to
select the output directly from the package inside the file-append (and
it seems I’m not alone in this). I’ve been through the code, so to some
degree I realize why things are being done the way they are, but that
doesn’t change my current preferences.

	I prefer the list over the gexp/ungexp because we already use
lists to select outputs in other parts of the config declarations, and
the gexp/ungexp stuff is harder to understand from the point-of-view of
a user, rather than package author. It’s also not present in the
majority of configs, I’d wager.

	Before I found the “correct” way to do things, I did cobble
together some code that allowed the (list pkg "output") form to work,
and would be happy to submit it upstream. It’s not perfect, since it
ignores the ‘output’ argument in the gexp expander in favor of the
stored output in the file-append, but I don’t think that’s an actual
issue in practice.

-bjc

bug#42164: Combining file-append with gexps results in incomprehensible errors

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1740 bytes --]

Brian Cully schreef op za 23-04-2022 om 12:03 [-0400]:
> 	I was having a similar issue, so I dug into this issue a bit,
> and it turns out that you *can* select an output from file-append, but
> the syntax is a bit wonky:
> 
> ---[snip]---
> (let ((fa (file-append pkg "/path")))
>   $~$#fa:output)
> ---[snip]---
> 
> 	I’ve tried to remove the let:
> 
> ---[snip]---
> $~$#(file-append pkg "/path"):output
> ---[snip]---

Do you mean #~#$fa:output here?

> 
> 	But this confuses the reader, an it things ‘:output’ is a
> variable. Not too surprising, but it does make the Guix configuration
> files more awkward, since now you have to ‘define’ the file-append
> operations at the top level so you can refer to them with the syntax
> above.
> 
> 	I would prefer to have something like any of the following (in
> order of preference):
> 
> ---[snip]---
> (file-append pkg:output "/path")

This one is only possible if file-append becomes syntax instead of a
procedure.

> (file-append (list pkg "output") "/path")

Possible, but seems semantically wrong to me -- what's a list doing in
a file-manipulating procedure?

> (file-append $~$#pkg:output "/path")

I assume this would be #~#$pkg:output?

My bikeshed proposal:

  (file-append (output glibc "static") "/lib/...")

Here, 'output' is a procedure constructing some kind of object.
This would also be useful outside 'file-append', as it allows
programmatically constructing equivalents #~#$pkg:output for multiple
'output' (there was some package definition some while back where this
would have been useful).

This 'output' could perhaps also be used in 'inputs' / 'native-inputs'
fields.  YMMV.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#42164: Combining file-append with gexps results in incomprehensible errors

[Brian Cully]

Maxime Devos <maximedevos@telenet.be> writes:

> Do you mean #~#$fa:output here?

	Yes, I did. Sorry, my fingers have minds of their own, and the
gexp syntax, in particular, really runs counter to their memory.

>> (file-append pkg:output "/path")
>
> This one is only possible if file-append becomes syntax instead of a
> procedure.

	I did not expect this to be particularly possible, but I find it
aesthetically and intuitively pleasing enough to propose it anyway.

>> (file-append (list pkg "output") "/path")
>
> Possible, but seems semantically wrong to me -- what's a list doing in
> a file-manipulating procedure?

	It need not be a list, but I do think it makes sense inside the
file-append, rather than a modification to file-append’s results, as is
currently the case. I picked list because it was being used elsewhere to
explicitly select a derivation’s output.

> My bikeshed proposal:
>
>   (file-append (output glibc "static") "/lib/...")
>
> Here, 'output' is a procedure constructing some kind of object.
> This would also be useful outside 'file-append', as it allows
> programmatically constructing equivalents #~#$pkg:output for multiple
> 'output' (there was some package definition some while back where this
> would have been useful).
>
> This 'output' could perhaps also be used in 'inputs' / 'native-inputs'
> fields.  YMMV.

	I like this, and would prefer it over the ‘list’ proposal. I’m
not sure how to implement it given my limited knowledge of Guix
internals, though. Special-casing a list inside of file-append was
fairly straightforward and solved the immediate issue.

	I’d be happy to take a look at implementing this, but I’m a bit
clueless where to begin, as it’s much more general than my previous
attempt. If you (or others) have the time, I’d appreciate any pointers
that’d make it easier to begin.

	For instance: I would expect the ‘output’ form to be lowerable,
which would be done in the ‘compile’ branch of a gexp-compiler, but then
it would have to be expanded later in the ‘expand’ branch, which
presents an issue: the third argument to the expander procedure is
already an output. Would we do away with that argument? Somehow fill it
in based on the ‘output’ form (either expressed or implicit)? Let the
‘output’ form, if present, override it?

-bjc

bug#42164: Combining file-append with gexps results in incomprehensible errors

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 540 bytes --]

Brian Cully schreef op za 23-04-2022 om 13:29 [-0400]:
> 	I like this, and would prefer it over the ‘list’ proposal. I’m
> not sure how to implement it given my limited knowledge of Guix
> internals, though. Special-casing a list inside of file-append was
> fairly straightforward and solved the immediate issue.

Proposed implementation:

(define (output thing output)
  (gexp-input package output))

Seems like this thing already exists, it's just not well-known and not
documented in the manual.

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#42164: Combining file-append with gexps results in incomprehensible errors

[Brian Cully]

Maxime Devos <maximedevos@telenet.be> writes:

> Proposed implementation:
>
> (define (output thing output)
>   (gexp-input package output))
>
> Seems like this thing already exists, it's just not well-known and not
> documented in the manual.

	Where is this in the code? I couldn’t find it myself (grepping
for these keywords turns up a lot of false positives). On the off chance
it might work, I did try ‘(file-append (output pkg "lib") "/path")’ and
was told that ‘output’ is an unbound variable.

-bjc

bug#42164: Combining file-append with gexps results in incomprehensible errors

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 329 bytes --]

Brian Cully schreef op za 23-04-2022 om 16:40 [-0400]:
> 	Where is this in the code?

Nowhere.  I tried implementing 'output', and noticed I just ended up
with something equivalent to a simple call to 'gexp-input' (hence,
‘this thing already exists’).  'gexp-input' can be found in (guix
gexp).

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#42164: Combining file-append with gexps results in incomprehensible errors

[Brian Cully]

Maxime Devos <maximedevos@telenet.be> writes:

> Nowhere.  I tried implementing 'output', and noticed I just 
> ended up
> with something equivalent to a simple call to 'gexp-input' 
> (hence,
> ‘this thing already exists’).  'gexp-input' can be found in 
> (guix
> gexp).

	I finally had a chance to mess around with this, and it 
	does, indeed, do the job. Since it wasn’t clear to me how 
	it worked, I’d like to document it here.

	To accomplish the goal at the begging of this thread, 
	rather than using:
--8<---------------cut here---------------start------------->8---
(file-append #~#$gcc:lib “/lib”)
--8<---------------cut here---------------end--------------->8---

	The “correct” way is this:
--8<---------------cut here---------------start------------->8---
(gexp-input (file-append gcc "/lib") "lib")
--8<---------------cut here---------------end--------------->8---

	The name seems a little weird, since we’re selecting an 
	output, but it makes sense from the other end: we’re 
	creating an input for the higher-level expression (in this 
	case, the special-file-services-type).

	Anyway, this works for me, so this ticket can probably be 
	closed.

-bjc

bug#42164: Combining file-append with gexps results in incomprehensible errors

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 636 bytes --]

Brian Cully schreef op zo 24-04-2022 om 19:27 [-0400]:
> --8<---------------cut here---------------start------------->8---
> (file-append #~#$gcc:lib “/lib”)
> --8<---------------cut here---------------end--------------->8---
> 
> 	The “correct” way is this:
> --8<---------------cut here---------------start------------->8---
> (gexp-input (file-append gcc "/lib") "lib")
> --8<---------------cut here---------------end--------------->8---

FWIW, I was actually thinking of

  (file-append (gexp-input gcc "lib") "/lib")

(First select the output with 'gexp-input', then append a suffix.)

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]