Problem with wide characters on upgrading to guile 2.x

Problem with wide characters on upgrading to guile 2.x

[Richard Shann]

GNU/Denemo has a number of guile scripts in which wide characters are
embedded in strings. These used to work in guile 1.8 but with guile 2.0
I am seeing the following error message trying to use a string with "
"
in it (if that string will get through the email software)

(format #t "~%=> ~A~%""  ")
; the throw arguments are

(scm_to_stringn cannot convert wide string to output locale 84 #f #f)

what is "locale 84" and how can I return to the desired behavior?

Richard Shann

Re: Problem with wide characters on upgrading to guile 2.x

[Panicz Maciej Godek]

[-- Attachment #1: Type: text/plain, Size: 844 bytes --]

2013/8/22 Richard Shann <richard.shann@virgin.net>

> GNU/Denemo has a number of guile scripts in which wide characters are
> embedded in strings. These used to work in guile 1.8 but with guile 2.0
> I am seeing the following error message trying to use a string with "
> "
> in it (if that string will get through the email software)
>
> (format #t "~%=> ~A~%""  ")
> ; the throw arguments are
>
> (scm_to_stringn cannot convert wide string to output locale 84 #f #f)
>
> what is "locale 84" and how can I return to the desired behavior?
>
>
I don't know much about the internal details, but did you try
to set the locale of the environment to e.g. C.UTF-8 like that

 $ LC_ALL="C.UTF-8" guile # or your particular script

(the list of possible values can be obtained by typing "locale -a"
on modern unix systems)

[-- Attachment #2: Type: text/html, Size: 1391 bytes --]

Re: Problem with wide characters on upgrading to guile 2.x

[Richard Shann]

Thank you for your reply. I didn't try altering anything in the program,
because this problem has arisen when changing from guile-1.8 to
guile-2.0 and so I am hoping it is an understood problem.
FWIW I see that the following code is executed at Denemo program startup

  setlocale (LC_ALL, "");
  bindtextdomain(GETTEXT_PACKAGE, get_system_locale_dir ());
  bind_textdomain_codeset(GETTEXT_PACKAGE, "UTF-8");
  textdomain(GETTEXT_PACKAGE);

so it would seem that LC_ALL is being specifically unset. The code works
both under GNU/Linux and Windows using guile-1.8.
Putting 

setlocale (LC_ALL, "C.UTF-8") 

in the above initialization sequence does not affect the behavior.

Richard

On Thu, 2013-08-22 at 14:50 +0200, Panicz Maciej Godek wrote:
> 2013/8/22 Richard Shann <richard.shann@virgin.net>
>         GNU/Denemo has a number of guile scripts in which wide
>         characters are
>         embedded in strings. These used to work in guile 1.8 but with
>         guile 2.0
>         I am seeing the following error message trying to use a string
>         with "
>         "
>         in it (if that string will get through the email software)
>         
>         (format #t "~%=> ~A~%""  ")
>         ; the throw arguments are
>         
>         (scm_to_stringn cannot convert wide string to output locale 84
>         #f #f)
>         
>         what is "locale 84" and how can I return to the desired
>         behavior?
>         
> 
> 
> I don't know much about the internal details, but did you try
> to set the locale of the environment to e.g. C.UTF-8 like that
> 
> 
>  $ LC_ALL="C.UTF-8" guile # or your particular script
> 
> 
> (the list of possible values can be obtained by typing "locale -a"
> on modern unix systems)
> 
> 



_______________________________________________
Denemo-devel mailing list
Denemo-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/denemo-devel

Re: Problem with wide characters on upgrading to guile 2.x

[Richard Shann]

I have been doing some experiments with the guile interpreter

 guile --version
guile (GNU Guile) 2.0.5-deb+1-3

and it seems to understand the wide characters, using the unicode e176
character with the string "hello" attached I get this:

scheme@(guile-user)> (string-append "  " "hello")
$1 = "  \ue176hello"
scheme@(guile-user)> (display  "  ")
  scheme@(guile-user)> (display  (string-append "  " "hello"))
  helloscheme@(guile-user)> 
I notice that I do not have LC_ALL set in my environment.

From inside Denemo scm_c_eval_string() is being used to evaluate a
string that includes this wide character and that is throwing the
exception.

Richard


On Thu, 2013-08-22 at 14:50 +0200, Panicz Maciej Godek wrote:
> 2013/8/22 Richard Shann <richard.shann@virgin.net>
>         GNU/Denemo has a number of guile scripts in which wide
>         characters are
>         embedded in strings. These used to work in guile 1.8 but with
>         guile 2.0
>         I am seeing the following error message trying to use a string
>         with "
>         "
>         in it (if that string will get through the email software)
>         
>         (format #t "~%=> ~A~%""  ")
>         ; the throw arguments are
>         
>         (scm_to_stringn cannot convert wide string to output locale 84
>         #f #f)
>         
>         what is "locale 84" and how can I return to the desired
>         behavior?
>         
> 
> 
> I don't know much about the internal details, but did you try
> to set the locale of the environment to e.g. C.UTF-8 like that
> 
> 
>  $ LC_ALL="C.UTF-8" guile # or your particular script
> 
> 
> (the list of possible values can be obtained by typing "locale -a"
> on modern unix systems)
> 
> 



_______________________________________________
Denemo-devel mailing list
Denemo-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/denemo-devel

Re: Problem with wide characters on upgrading to guile 2.x

[Mike Gran]

Hi Richard-


> scheme@(guile-user)> (string-append "  " "hello")

> $1 = "  \ue176hello"
> scheme@(guile-user)> (display  "  ")
>   scheme@(guile-user)> (display  (string-append "  " 
> "hello"))
>   helloscheme@(guile-user)> 
> I notice that I do not have LC_ALL set in my environment.
> 
> From inside Denemo scm_c_eval_string() is being used to evaluate a
> string that includes this wide character and that is throwing the
> exception.

I've been gone a while, so I don't know what the latest is.
But based on old knowledge...

1. Make sure that you Guile script files are UTF-8.  If they
are not, you'll need to explicitly put a "coding:"
declaration at the top of each file.

2. In the inner_main of your scm_with_guile call, 
try calling scm_setlocale.  Maybe something like this?
(This shouldn't make a difference, I think. 
But, if it does, it says something interesting.)

scm_setlocale( scm_variable_ref(scm_c_lookup("LC_ALL")),
scm_from_locale_string("") ); If that actually works, lemme know.

-Mike

Re: Problem with wide characters on upgrading to guile 2.x

[Richard Shann]

On Thu, 2013-08-22 at 11:45 -0700, Mike Gran wrote:
> Hi Richard-
> 
> 
> > scheme@(guile-user)> (string-append "  " "hello")
> 
> > $1 = "  \ue176hello"
> > scheme@(guile-user)> (display  "  ")
> >   scheme@(guile-user)> (display  (string-append "  " 
> > "hello"))
> >   helloscheme@(guile-user)> 
> > I notice that I do not have LC_ALL set in my environment.
> > 
> > From inside Denemo scm_c_eval_string() is being used to evaluate a
> > string that includes this wide character and that is throwing the
> > exception.
> 
> I've been gone a while, so I don't know what the latest is.
> But based on old knowledge...
> 
> 1. Make sure that you Guile script files are UTF-8.  If they
> are not, you'll need to explicitly put a "coding:"
> declaration at the top of each file.

I guess they are (else it would never have worked under 1.8) - typically
they are input to a Gtk widget which I think default to taking utf-8.

> 
> 2. In the inner_main of your scm_with_guile call, 
> try calling scm_setlocale.  Maybe something like this?
> (This shouldn't make a difference, I think. 
> But, if it does, it says something interesting.)
> 
> scm_setlocale( scm_variable_ref(scm_c_lookup("LC_ALL")), scm_from_locale_string("") );

>  If that actually works, lemme know.

I just pasted the above into the start of inner_main() and like magic,
it has fixed the problem. Is this call ok for guile-1.8 linking too?

Thanks very much for the magic

Richard


> 
> -Mike
> 



_______________________________________________
Denemo-devel mailing list
Denemo-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/denemo-devel

Re: Problem with wide characters on upgrading to guile 2.x

[Mike Gran]

>>  2. In the inner_main of your scm_with_guile call, 

>>  try calling scm_setlocale.  Maybe something like this?
>>  (This shouldn't make a difference, I think. 
>>  But, if it does, it says something interesting.)
>> 
>>  scm_setlocale( scm_variable_ref(scm_c_lookup("LC_ALL")), 
> scm_from_locale_string("") );
> 
>>   If that actually works, lemme know.
> 
> I just pasted the above into the start of inner_main() and like magic,
> it has fixed the problem. Is this call ok for guile-1.8 linking too?

I'm not 100% sure, but, I think so.

-Mike

Re: Problem with wide characters on upgrading to guile 2.x

[Ludovic Courtès]

Hello,

Richard Shann <richard.shann@virgin.net> skribis:

> GNU/Denemo has a number of guile scripts in which wide characters are
> embedded in strings. These used to work in guile 1.8 but with guile 2.0
> I am seeing the following error message trying to use a string with "
> "
> in it (if that string will get through the email software)
>
> (format #t "~%=> ~A~%""  ")
> ; the throw arguments are
>
> (scm_to_stringn cannot convert wide string to output locale 84 #f #f)
>
> what is "locale 84" and how can I return to the desired behavior?

You already got advice, but anyway: what this means is that the string
cannot be converted to the current locale.

In Guile 1.8, strings were just byte sequences, like in C.  In 2.0,
strings are sequences of Unicode codepoints, which means that they must
be encoded when written to an output port.

HTH,
Ludo’.

PS: Nitpicking: I guess it should be “GNU Denemo” (as in “the Denemo
    package of GNU”), rather than “GNU/Denemo” (“GNU over the Denemo
    kernel”?)  :-)

Re: Problem with wide characters on upgrading to guile 2.x

[Richard Shann]

On Thu, 2013-08-22 at 12:14 -0700, Mike Gran wrote:
> >>  2. In the inner_main of your scm_with_guile call, 
> 
> >>  try calling scm_setlocale.  Maybe something like this?
> >>  (This shouldn't make a difference, I think. 
> >>  But, if it does, it says something interesting.)
> >> 
> >>  scm_setlocale( scm_variable_ref(scm_c_lookup("LC_ALL")), 
> > scm_from_locale_string("") );
> > 
> >>   If that actually works, lemme know.
> > 
> > I just pasted the above into the start of inner_main() and like magic,
> > it has fixed the problem. Is this call ok for guile-1.8 linking too?
> 
> I'm not 100% sure, but, I think so.

I have located a problem which arises I think when LC_ALL is not set
(the actual circumstance I triggered by setting LANG to a language not
installed):

8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><
Backtrace:
In ice-9/boot-9.scm:
 149: 2 [catch #t #<catch-closure 19194e0> ...]
 157: 1 [#<procedure 18b80f0 ()>]
In unknown file:
   ?: 0 [catch-closure]

ERROR: In procedure catch-closure:
ERROR: In procedure setlocale: Invalid argument
8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><

This is a pretty obscure case, but I guess some small tweak could fix
it? I have installed en_US now, but I can test as follows.

rshann@DebianBox:~/local/bin$ export LANG=pt_BR.utf8 
rshann@DebianBox:~/local/bin$ ./denemo

(process:7073): Gtk-WARNING **: Locale not supported by C library.
	Using the fallback 'C' locale.
Backtrace:
In ice-9/boot-9.scm:
 149: 2 [catch #t #<catch-closure 216f4e0> ...]
 157: 1 [#<procedure 210e0f0 ()>]
In unknown file:
   ?: 0 [catch-closure]

ERROR: In procedure catch-closure:
ERROR: In procedure setlocale: Invalid argument

Richard

Re: Problem with wide characters on upgrading to guile 2.x

[Ludovic Courtès]

Richard Shann <richard.shann@virgin.net>
skribis:

> rshann@DebianBox:~/local/bin$ export LANG=pt_BR.utf8 
> rshann@DebianBox:~/local/bin$ ./denemo
>
> (process:7073): Gtk-WARNING **: Locale not supported by C library.
> 	Using the fallback 'C' locale.
> Backtrace:
> In ice-9/boot-9.scm:
>  149: 2 [catch #t #<catch-closure 216f4e0> ...]
>  157: 1 [#<procedure 210e0f0 ()>]
> In unknown file:
>    ?: 0 [catch-closure]
>
> ERROR: In procedure catch-closure:
> ERROR: In procedure setlocale: Invalid argument

Apparently pt_BR.utf8 is not a supported locale on your system.

To gracefully handle this case (like GTK+ does above), just wrap the
‘setlocale’ call:

  (catch 'system-error
    (lambda ()
      (setlocale LC_ALL ""))
    (lambda args
      (format (current-error-port)
              "warning: failed to install locale: ~a~%"
              (strerror (system-error-errno args))))

HTH,
Ludo’.

Re: Problem with wide characters on upgrading to guile 2.x

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

() ludo@gnu.org (Ludovic Courtès)
() Mon, 02 Sep 2013 22:01:03 +0200

   pt_BR.utf8 is not a supported locale on your system.

Could the spelling be part of the problem?  On my computer,
i see:

 $ locale -a
 C
 C.UTF-8
 POSIX
 it_IT.utf8

but also note that:

 $ printenv | grep LANG
 LANG=it_IT.UTF-8

and /usr/share/i18n/SUPPORTED is full of ".UTF-8", only.

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Problem with wide characters on upgrading to guile 2.x

[Richard Shann]

On Tue, 2013-09-03 at 19:53 +0200, Thien-Thi Nguyen wrote:
[...]
>    pt_BR.utf8 is not a supported locale on your system.

yes, that is why I chose it, so as to trigger the error. It is graceful
handling of this error that I was after - it should default to
untranslated, not abort the program.

> 
> Could the spelling be part of the problem? 

I strongly suspect this stuff is case-insensitive.

The problem surely lies in one of the calls in the line

 scm_setlocale( scm_variable_ref(scm_c_lookup("LC_ALL")), scm_from_locale_string("") );

I'll dig down into that when I understand it better.

Thank you for the response,

Richard Shann

Re: [Denemo-devel] Problem with wide characters on upgrading to guile 2.x

[Mark H Weaver]

Richard Shann <richard.shann@virgin.net> writes:

> On Tue, 2013-09-03 at 19:53 +0200, Thien-Thi Nguyen wrote:
> [...]
>>    pt_BR.utf8 is not a supported locale on your system.
>
> yes, that is why I chose it, so as to trigger the error. It is graceful
> handling of this error that I was after - it should default to
> untranslated, not abort the program.

I disagree.  If Guile does not know what encoding to use, that's a
serious error that should not be ignored, at least not by default.
Please keep in mind that Guile is used for many diverse tasks, including
scripts that are run when the user's not looking, e.g. cron jobs.  Guile
must not silently corrupt data, which could easily happen if it silently
ignores the error without knowing what character encoding to use.

For fully interactive programs, ignoring the error and defaulting to the
C locale (e.g. ASCII) is more reasonable, because any corruptions will
hopefully be noticed by the user.  To do that, please consider the code
that Ludovic suggested:

  (catch 'system-error
      (lambda ()
        (setlocale LC_ALL ""))
      (lambda args
        (format (current-error-port)
                "warning: failed to install locale: ~a~%"
                (strerror (system-error-errno args))))

However, it should be noted that if you do this, Guile 2 (unlike 1.8)
will still raise an error later if you try to read a byte outside of the
ASCII range, or if you try to write a non-ASCII character.

Note that Guile 1.8 conflated characters and bytes, like many older
non-i18n programs, and thus effectively chose Latin-1 by default.
However, since most modern GNU systems use UTF-8 by default (at least
outside of the CJK world) this guess is most likely wrong, and thus
likely to silently corrupt non-ASCII characters.

     Regards,
       Mark

Re: Problem with wide characters on upgrading to guile 2.x

[Richard Shann]

On Wed, 2013-09-04 at 12:17 -0400, Mark H Weaver wrote:
> Richard Shann <richard.shann-J1btbF6uMEmsTnJN9+BGXg@public.gmane.org> writes:
> 
> > On Tue, 2013-09-03 at 19:53 +0200, Thien-Thi Nguyen wrote:
> > [...]
> >>    pt_BR.utf8 is not a supported locale on your system.
> >
> > yes, that is why I chose it, so as to trigger the error. It is graceful
> > handling of this error that I was after - it should default to
> > untranslated, not abort the program.
> 
> I disagree.  If Guile does not know what encoding to use, that's a
> serious error that should not be ignored, at least not by default.
> Please keep in mind that Guile is used for many diverse tasks, including
> scripts that are run when the user's not looking, e.g. cron jobs.  Guile
> must not silently corrupt data, which could easily happen if it silently
> ignores the error without knowing what character encoding to use.

Ah, I see.

> 
> For fully interactive programs, ignoring the error and defaulting to the
> C locale (e.g. ASCII) is more reasonable, because any corruptions will
> hopefully be noticed by the user.  To do that, please consider the code
> that Ludovic suggested:
> 
>   (catch 'system-error
>       (lambda ()
>         (setlocale LC_ALL ""))
>       (lambda args
>         (format (current-error-port)
>                 "warning: failed to install locale: ~a~%"
>                 (strerror (system-error-errno args))))
> 
> However, it should be noted that if you do this, Guile 2 (unlike 1.8)
> will still raise an error later if you try to read a byte outside of the
> ASCII range, or if you try to write a non-ASCII character.

Hmm, this all started with guile barfing on utf8 characters in the
scripts it gets from the Denemo program. I'm afraid I've messed up the
title of the thread by a misused "Group Reply", but the setlocale is
being called from C to as a fix for that problem (barfing on utf8):

scm_setlocale( scm_variable_ref(scm_c_lookup("LC_ALL")), scm_from_locale_string("") );
 
as suggested by Mike Gran. This works when I use the program with a
(utf8) locale that I have installed, that is scripts with embedded utf8
characters work again. But guile aborts with the erroneous locale set.
Perhaps this is not so important after all.

Thanks

Richard

Re: Problem with wide characters on upgrading to guile 2.x

[Andy Wingo]

On Mon 02 Sep 2013 10:48, Richard Shann <richard.shann@virgin.net> writes:

> On Thu, 2013-08-22 at 12:14 -0700, Mike Gran wrote:
>> >>  2. In the inner_main of your scm_with_guile call, 
>> 
>> >>  try calling scm_setlocale.  Maybe something like this?
>> >>  (This shouldn't make a difference, I think. 
>> >>  But, if it does, it says something interesting.)
>> >> 
>> >>  scm_setlocale( scm_variable_ref(scm_c_lookup("LC_ALL")), 
>> > scm_from_locale_string("") );
>> > 
>> >>   If that actually works, lemme know.
>> > 
>> > I just pasted the above into the start of inner_main() and like magic,
>> > it has fixed the problem. Is this call ok for guile-1.8 linking too?
>> 
>> I'm not 100% sure, but, I think so.
>
> I have located a problem which arises I think when LC_ALL is not set
> (the actual circumstance I triggered by setting LANG to a language not
> installed):
>
> 8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><
> Backtrace:
> In ice-9/boot-9.scm:
>  149: 2 [catch #t #<catch-closure 19194e0> ...]
>  157: 1 [#<procedure 18b80f0 ()>]
> In unknown file:
>    ?: 0 [catch-closure]
>
> ERROR: In procedure catch-closure:
> ERROR: In procedure setlocale: Invalid argument
> 8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><

const char prog[] =
  "(catch #t
     (lambda () (setlocale LC_ALL \"\"))
     (lambda _
       (display \"Locale not supported by the C library.
Falling back to default \\\"C\\\" locale.\\n\"
                (current-error-port))))";
scm_c_eval_string (prog);

Andy
-- 
http://wingolog.org/

Re: Problem with wide characters on upgrading to guile 2.x

[Richard Shann]

On Sat, 2013-09-07 at 11:07 +0200, Andy Wingo wrote:
> On Mon 02 Sep 2013 10:48, Richard Shann <richard.shann-J1btbF6uMEmsTnJN9+BGXg@public.gmane.org> writes:
> 
> > On Thu, 2013-08-22 at 12:14 -0700, Mike Gran wrote:
> >> >>  2. In the inner_main of your scm_with_guile call, 
> >> 
> >> >>  try calling scm_setlocale.  Maybe something like this?
> >> >>  (This shouldn't make a difference, I think. 
> >> >>  But, if it does, it says something interesting.)
> >> >> 
> >> >>  scm_setlocale( scm_variable_ref(scm_c_lookup("LC_ALL")), 
> >> > scm_from_locale_string("") );
> >> > 
> >> >>   If that actually works, lemme know.
[...]
> >
> > ERROR: In procedure catch-closure:
> > ERROR: In procedure setlocale: Invalid argument
> > 8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><8><
> 
> const char prog[] =
>   "(catch #t
>      (lambda () (setlocale LC_ALL \"\"))
>      (lambda _
>        (display \"Locale not supported by the C library.
> Falling back to default \\\"C\\\" locale.\\n\"
>                 (current-error-port))))";
> scm_c_eval_string (prog);
> 
> Andy

Thank you! That does nicely.

Richard