* UTF8 string storage and retrieval in XML
@ 2016-02-01 15:05 Richard Shann
2016-02-01 15:36 ` Richard Shann
2016-02-01 18:16 ` Mark H Weaver
0 siblings, 2 replies; 6+ messages in thread
From: Richard Shann @ 2016-02-01 15:05 UTC (permalink / raw)
To: guile-user
Can anyone explain what is going on when you try to store strings with
non-ASCII characters? Here is an example:
guile> (define another-data "Čć")
guile> another-data
"�\x8c�\x87"
guile> (display another-data)
Čćguile>
I would expect evaluating another-data to give "Čć" not
"�\x8c�\x87" (what is that?)...
My problem comes with storing an association list with the second
element a string with a non-ASCII character in it. I'm storing the alist
as a string with
(format #f "'~s" myalist)
but when I try to store this string on disk using XML it barfs on
reloading.
Richard Shann
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: UTF8 string storage and retrieval in XML
2016-02-01 15:05 UTF8 string storage and retrieval in XML Richard Shann
@ 2016-02-01 15:36 ` Richard Shann
2016-02-01 15:43 ` Thompson, David
2016-02-01 18:16 ` Mark H Weaver
1 sibling, 1 reply; 6+ messages in thread
From: Richard Shann @ 2016-02-01 15:36 UTC (permalink / raw)
To: guile-user
I should have said:
guile --version
Guile 1.8.8
Richard
On Mon, 2016-02-01 at 15:05 +0000, Richard Shann wrote:
> Can anyone explain what is going on when you try to store strings with
> non-ASCII characters? Here is an example:
>
> guile> (define another-data "Čć")
> guile> another-data
> "�\x8c�\x87"
> guile> (display another-data)
> Čćguile>
>
> I would expect evaluating another-data to give "Čć" not
> "�\x8c�\x87" (what is that?)...
>
> My problem comes with storing an association list with the second
> element a string with a non-ASCII character in it. I'm storing the alist
> as a string with
>
> (format #f "'~s" myalist)
>
> but when I try to store this string on disk using XML it barfs on
> reloading.
>
> Richard Shann
>
>
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: UTF8 string storage and retrieval in XML
2016-02-01 15:36 ` Richard Shann
@ 2016-02-01 15:43 ` Thompson, David
2016-02-01 16:58 ` Richard Shann
0 siblings, 1 reply; 6+ messages in thread
From: Thompson, David @ 2016-02-01 15:43 UTC (permalink / raw)
To: richard; +Cc: Guile User
On Mon, Feb 1, 2016 at 10:36 AM, Richard Shann <richard@rshann.plus.com> wrote:
> I should have said:
> guile --version
> Guile 1.8.8
I don't think that you will find much help for this version of Guile
here. Guile 1.8.8 is over 5 years old and the 1.8.x series is no
longer maintained. Please upgrade to the Guile 2.0.x series and see
if your problems persist. Guile 2.0.11 is recommended.
Thanks!
- Dave
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: UTF8 string storage and retrieval in XML
2016-02-01 15:43 ` Thompson, David
@ 2016-02-01 16:58 ` Richard Shann
0 siblings, 0 replies; 6+ messages in thread
From: Richard Shann @ 2016-02-01 16:58 UTC (permalink / raw)
To: Thompson, David; +Cc: Guile User
On Mon, 2016-02-01 at 10:43 -0500, Thompson, David wrote:
> On Mon, Feb 1, 2016 at 10:36 AM, Richard Shann <richard@rshann.plus.com> wrote:
> > I should have said:
> > guile --version
> > Guile 1.8.8
>
> I don't think that you will find much help for this version of Guile
> here. Guile 1.8.8 is over 5 years old and the 1.8.x series is no
> longer maintained. Please upgrade to the Guile 2.0.x series and see
> if your problems persist. Guile 2.0.11 is recommended.
Thanks for the hint.
The version 1.8.8 is the default in Debian stable, I try to stick to the
stable release in order not to introduce too many uncertainties. There
are enough when developing an application without having an unstable
environment.
On building with 2.0.11 (which Debian stable offers as well), I see that
the bug (I presume it was that then) is fixed.
Is there a 1.8 series Guile in which the fix was introduced? We build
Denemo for several machines and I wouldn't like to say that we could use
2.x on all of them.
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: UTF8 string storage and retrieval in XML
2016-02-01 15:05 UTF8 string storage and retrieval in XML Richard Shann
2016-02-01 15:36 ` Richard Shann
@ 2016-02-01 18:16 ` Mark H Weaver
2016-02-01 18:22 ` Richard Shann
1 sibling, 1 reply; 6+ messages in thread
From: Mark H Weaver @ 2016-02-01 18:16 UTC (permalink / raw)
To: Richard Shann; +Cc: guile-user
Richard Shann <richard@rshann.plus.com> writes:
> Can anyone explain what is going on when you try to store strings with
> non-ASCII characters? Here is an example:
>
> guile> (define another-data "Čć")
> guile> another-data
> "�\x8c�\x87"
> guile> (display another-data)
> Čćguile>
I guess this is Guile 1.x, where strings are merely byte sequences.
Your terminal is using the UTF-8 encoding, where "Čć" is represented as
the byte sequence:
0xC4 0x8C 0xC4 0x87
When printing this using 'write' (which is how values are printed at the
REPL), Guile 1.x is treating this byte sequence as Latin-1. The 0xC4 is
the Latin-1 representation for the character "Ä", but 0x8C and 0x87 are
not characters in Latin-1 and so are escaped as "\x8c" and "\x87".
When printing using display, Guile is simply writing the bytes out
unescaped, which your terminal interprets as UTF-8.
Obviously this is terrible, which is why Guile 2.0+ strings are
sequences of unicode code points. Can you switch to Guile 2.0?
Mark
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: UTF8 string storage and retrieval in XML
2016-02-01 18:16 ` Mark H Weaver
@ 2016-02-01 18:22 ` Richard Shann
0 siblings, 0 replies; 6+ messages in thread
From: Richard Shann @ 2016-02-01 18:22 UTC (permalink / raw)
To: Mark H Weaver; +Cc: guile-user
On Mon, 2016-02-01 at 13:16 -0500, Mark H Weaver wrote:
> Richard Shann <richard@rshann.plus.com> writes:
>
> > Can anyone explain what is going on when you try to store strings with
> > non-ASCII characters? Here is an example:
> >
> > guile> (define another-data "Čć")
> > guile> another-data
> > "�\x8c�\x87"
> > guile> (display another-data)
> > Čćguile>
>
> I guess this is Guile 1.x, where strings are merely byte sequences.
> Your terminal is using the UTF-8 encoding, where "Čć" is represented as
> the byte sequence:
>
> 0xC4 0x8C 0xC4 0x87
>
> When printing this using 'write' (which is how values are printed at the
> REPL), Guile 1.x is treating this byte sequence as Latin-1. The 0xC4 is
> the Latin-1 representation for the character "Ä", but 0x8C and 0x87 are
> not characters in Latin-1 and so are escaped as "\x8c" and "\x87".
>
> When printing using display, Guile is simply writing the bytes out
> unescaped, which your terminal interprets as UTF-8.
>
> Obviously this is terrible, which is why Guile 2.0+ strings are
> sequences of unicode code points. Can you switch to Guile 2.0?
Thank you for the detailed explanation. I'm currently asking whether we
can switch, the problem will be that LilyPond is stuck at 1.8 because of
very complex issues I gather, and Denemo runs LilyPond to typeset the
music entered.
Is there any reasonable workaround - can I insert something to transform
the string?
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-02-01 18:22 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-02-01 15:05 UTF8 string storage and retrieval in XML Richard Shann
2016-02-01 15:36 ` Richard Shann
2016-02-01 15:43 ` Thompson, David
2016-02-01 16:58 ` Richard Shann
2016-02-01 18:16 ` Mark H Weaver
2016-02-01 18:22 ` Richard Shann
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).