bug#21536: Texi, Unicode and Emacs interface

bug#21536: Texi, Unicode and Emacs interface

[Alex Kost]

[-- Attachment #1: Type: text/plain, Size: 1404 bytes --]

There is an interesting "bug" that leads to a problem with Emacs
interface for Guix packages.

Try "M-x guix-search-by-name RET mpfrcx".  You will get an
'encoding-error'.  This happens because:

1. Description of the 'mpfrcx' package contains a unicode symbol "–".

2. For some reason unicode symbols are not handled properly in REPLs
   connected to a guile server.

To illustrate this problem, here is the recipe:

1. Run guile server:

  guile --listen

Evaluate there:

  ((@@ (guix ui) texi->plain-text) "foo \u2015 bar.")

So far so good.

2. Now connect to it, either with:

   - netcat: "netcat localhost 37146"

   - or Geiser: "M-x connect-to-guile"

and evaluate the same expression.  This time you will get the error.

(This should probably be counted as a Guile bug, I'm not sure)

To make the connected REPL handle such unicode strings you have to
manually evaluate (setlocale LC_ALL "") there (even (setlocale LC_ALL)
works!).  So I used this as a workaround for the emacs interface in the
attached patch (I don't know if there is a better workaround).


BTW, since we now use texi for the package descriptions, does it mean
that our intention is to get rid of using unicode symbols directly?

…Hm, I've just tried ((@@ (guix ui) texi->plain-text) "@U{2012}") and
got:

  Throw to key `parser-error' with args `(#f "Unknown command" U)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-emacs-Handle-unicode-symbols-in-texi-descriptions.patch --]
[-- Type: text/x-patch, Size: 1164 bytes --]

From 5226abeac27e5b6f79cc6e2b933f146040e2c2bd Mon Sep 17 00:00:00 2001
From: Alex Kost <alezost@gmail.com>
Date: Wed, 23 Sep 2015 13:42:44 +0300
Subject: [PATCH] emacs: Handle unicode symbols in texi descriptions.

Fixes <http://bugs.gnu.org/XXXXX> (FIXME).

* emacs/guix-backend.el (guix-start-repl-maybe): Call 'setlocale' for
  the internal REPL.
---
 emacs/guix-backend.el | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/emacs/guix-backend.el b/emacs/guix-backend.el
index 412d648..32d449a 100644
--- a/emacs/guix-backend.el
+++ b/emacs/guix-backend.el
@@ -190,7 +190,11 @@ display messages."
                   guix-default-port (buffer-name repl))))
         (set repl-var repl)
         (and end-msg (message end-msg))
-        (unless internal
+        (if internal
+            ;; Without this, the REPL will error when package
+            ;; descriptions contain unicode symbols.  See
+            ;; <http://bugs.gnu.org/XXXXX> (FIXME) for details.
+            (guix-geiser-eval "(setlocale LC_ALL)" repl)
           (run-hooks 'guix-after-start-repl-hook))))))
 
 (defun guix-start-repl (buffer &optional address)
-- 
2.5.0

bug#21536: Texi, Unicode and Emacs interface

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 982 bytes --]

Alex Kost <alezost@gmail.com> skribis:

> Evaluate there:
>
>   ((@@ (guix ui) texi->plain-text) "foo \u2015 bar.")
>
> So far so good.
>
> 2. Now connect to it, either with:
>
>    - netcat: "netcat localhost 37146"
>
>    - or Geiser: "M-x connect-to-guile"
>
> and evaluate the same expression.  This time you will get the error.

The encoding error comes from the fact that ‘texi->plain-text’ uses a
string port, and string ports internally use the current locale encoding
or ‘%default-port-encoding’.

Consequently, when running in the “C” locale, string ports cannot
represent non-ASCII code points (something widely regarded as a bug in
Guile, and at the very least an annoyance.)

To work around that, you can type this in *Guix Internal REPL*:

  (fluid-set! %default-port-encoding "UTF-8")

I fixed in commit 2cad18a8 of guix-artwork.git, but perhaps a similar
hack is apparently needed elsewhere.

Could you test this patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 704 bytes --]

diff --git a/guix/ui.scm b/guix/ui.scm
index 4a3630f..67dd062 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -803,7 +803,10 @@ converted to a space; sequences of more than one line break are preserved."
 
 (define (texi->plain-text str)
   "Return a plain-text representation of texinfo fragment STR."
-  (stexi->plain-text (texi-fragment->stexi str)))
+  ;; 'texi-fragment->stexi' uses a string port so make sure it's a
+  ;; Unicode-capable one (see <http://bugs.gnu.org/11197>.)
+  (with-fluids ((%default-port-encoding "UTF-8"))
+    (stexi->plain-text (texi-fragment->stexi str))))
 
 (define (package-description-string package)
   "Return a plain-text representation of PACKAGE description field."

[-- Attachment #3: Type: text/plain, Size: 188 bytes --]


> BTW, since we now use texi for the package descriptions, does it mean
> that our intention is to get rid of using unicode symbols directly?

Not particularly.

Thanks,
Ludo’.

bug#21536: Texi, Unicode and Emacs interface

[Alex Kost]

Ludovic Courtès (2015-09-24 01:02 +0300) wrote:

> Alex Kost <alezost@gmail.com> skribis:
>
>> Evaluate there:
>>
>>   ((@@ (guix ui) texi->plain-text) "foo \u2015 bar.")
>>
>> So far so good.
>>
>> 2. Now connect to it, either with:
>>
>>    - netcat: "netcat localhost 37146"
>>
>>    - or Geiser: "M-x connect-to-guile"
>>
>> and evaluate the same expression.  This time you will get the error.
>
> The encoding error comes from the fact that ‘texi->plain-text’ uses a
> string port, and string ports internally use the current locale encoding
> or ‘%default-port-encoding’.
>
> Consequently, when running in the “C” locale, string ports cannot
> represent non-ASCII code points (something widely regarded as a bug in
> Guile, and at the very least an annoyance.)
>
> To work around that, you can type this in *Guix Internal REPL*:
>
>   (fluid-set! %default-port-encoding "UTF-8")
>
> I fixed in commit 2cad18a8 of guix-artwork.git, but perhaps a similar
> hack is apparently needed elsewhere.

Ah, I didn't follow 'guix-artwork' repo, now I see.

> Could you test this patch:
>
>
> diff --git a/guix/ui.scm b/guix/ui.scm
> index 4a3630f..67dd062 100644
> --- a/guix/ui.scm
> +++ b/guix/ui.scm
> @@ -803,7 +803,10 @@ converted to a space; sequences of more than one line break are preserved."
>  
>  (define (texi->plain-text str)
>    "Return a plain-text representation of texinfo fragment STR."
> -  (stexi->plain-text (texi-fragment->stexi str)))
> +  ;; 'texi-fragment->stexi' uses a string port so make sure it's a
> +  ;; Unicode-capable one (see <http://bugs.gnu.org/11197>.)
> +  (with-fluids ((%default-port-encoding "UTF-8"))
> +    (stexi->plain-text (texi-fragment->stexi str))))
>  
>  (define (package-description-string package)
>    "Return a plain-text representation of PACKAGE description field."

Yes, I confirm that with this change the problem with Emacs interface is
solved, thank you!

-- 
Alex

bug#21536: Texi, Unicode and Emacs interface

[Ludovic Courtès]

Alex Kost <alezost@gmail.com> skribis:

> Yes, I confirm that with this change the problem with Emacs interface is
> solved, thank you!

Pushed as 08d7e35, thanks!

Ludo’.