Re: UTF8 string storage and retrieval in XML

[Richard Shann <richard@rshann.plus.com>]

On Mon, 2016-02-01 at 13:16 -0500, Mark H Weaver wrote:
> Richard Shann <richard@rshann.plus.com> writes:
> 
> > Can anyone explain what is going on when you try to store strings with
> > non-ASCII characters? Here is an example:
> >
> > guile> (define another-data "Čć")           
> > guile> another-data
> > "�\x8c�\x87"
> > guile> (display another-data)
> > Čćguile> 
> 
> I guess this is Guile 1.x, where strings are merely byte sequences.
> Your terminal is using the UTF-8 encoding, where "Čć" is represented as
> the byte sequence:
> 
>   0xC4 0x8C 0xC4 0x87
> 
> When printing this using 'write' (which is how values are printed at the
> REPL), Guile 1.x is treating this byte sequence as Latin-1.  The 0xC4 is
> the Latin-1 representation for the character "Ä", but 0x8C and 0x87 are
> not characters in Latin-1 and so are escaped as "\x8c" and "\x87".
> 
> When printing using display, Guile is simply writing the bytes out
> unescaped, which your terminal interprets as UTF-8.
> 
> Obviously this is terrible, which is why Guile 2.0+ strings are
> sequences of unicode code points.  Can you switch to Guile 2.0?

Thank you for the detailed explanation. I'm currently asking whether we
can switch, the problem will be that LilyPond is stuck at 1.8 because of
very complex issues I gather, and Denemo runs LilyPond to typeset the
music entered.
Is there any reasonable workaround - can I insert something to transform
the string?

Richard