How to write a compute-file producing utf8 output?

How to write a compute-file producing utf8 output?

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 968 bytes --]

Hello Guix,

I am currently struggling when trying to create a computed-file producing
internationalized content.  Simple reproducer is:

      (use-modules (gnu packages base)
                   (guix gexp))

      (computed-file
       "utf8-display"
       #~(with-output-to-file #$output
           (λ ()
             (display "猫\n"))))

This, when executed and printed, gives following result:

      /tmp $ guix build -e '(load "/tmp/yy.scm")'
      /gnu/store/fyr9h82v9hhbjm4f7lqsdnhs4c45sn3y-utf8-display
      /tmp $ cat /gnu/store/fyr9h82v9hhbjm4f7lqsdnhs4c45sn3y-utf8-display
      ?

As you can see, ? does not equal 猫.  I would expect the root cause to be the
same as #66777.

Any ideas what can I do about this?  Or, even better, why this does not just
work out of the box?

Thanks for your help,
Tomas

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: How to write a compute-file producing utf8 output?

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Tomas,

On Thu, May 23 2024, Tomas Volf wrote:

> As you can see, ? does not equal 猫.

Maybe it helps you that this file works:

    (use-modules (gnu packages base)
                 (guix gexp))
    
    (computed-file
     "utf8-display"
     #~(with-output-to-file #$output
         (λ ()
           (use-modules (ice-9 binary-ports)
                        (rnrs bytevectors))
           (put-bytevector (current-output-port)
                           (string->utf8 "猫\n")))))

Diagnosing encoding issues can be tricky.  Therefore:

    $ guix shell xxd -- xxd /gnu/store/qqljfpzks62kph7qqdv26aj76r2q14nz-utf8-display
    00000000: e78c ab0a

The present issue reminds me---and perhaps you---of
https://issues.guix.gnu.org/69381.

Kind regards
Felix

Re: How to write a compute-file producing utf8 output?

[Ludovic Courtès]

Tomas Volf <~@wolfsden.cz> skribis:

> I am currently struggling when trying to create a computed-file producing
> internationalized content.  Simple reproducer is:

I think you found out in the meantime :-), but for posterity, one
possibility is to set ‘%default-port-encoding’:

  (fluid-set! %default-port-encoding "UTF-8")

Ludo’.

Re: How to write a compute-file producing utf8 output?

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 1088 bytes --]

On 2024-06-01 15:39:56 +0200, Ludovic Courtès wrote:
> Tomas Volf <~@wolfsden.cz> skribis:
>
> > I am currently struggling when trying to create a computed-file producing
> > internationalized content.  Simple reproducer is:
>
> I think you found out in the meantime :-), but for posterity, one
> possibility is to set ‘%default-port-encoding’:
>
>   (fluid-set! %default-port-encoding "UTF-8")

Thanks for the information :)

While this approach works (even the set-port-encoding! does), the problem with
this approach is that this needs to be put into every computed-file, making it
easy to forget (especially with #$random-variable, where you might not even know
if Unicode is present or not).  And is not possible at all with likes of
program-file.

So I still hope someone manages to take a look at
https://issues.guix.gnu.org/71257 which should just fix it for everyone across
majority of the G-exp forms.

Have a nice day,
Tomas Volf

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]