From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Richard Shann Newsgroups: gmane.lisp.guile.user Subject: Re: UTF8 string storage and retrieval in XML Date: Mon, 01 Feb 2016 18:22:36 +0000 Message-ID: <1454350956.1344.80.camel@rshann.plus.com> References: <1454339138.1344.51.camel@rshann.plus.com> <87fuxci9dl.fsf@netris.org> Reply-To: richard@rshann.plus.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Trace: ger.gmane.org 1454351100 32731 80.91.229.3 (1 Feb 2016 18:25:00 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 1 Feb 2016 18:25:00 +0000 (UTC) Cc: guile-user@gnu.org To: Mark H Weaver Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Mon Feb 01 19:24:51 2016 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1aQJ9b-0001u1-KC for guile-user@m.gmane.org; Mon, 01 Feb 2016 19:24:47 +0100 Original-Received: from localhost ([::1]:53681 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aQJ9b-0006Ie-3a for guile-user@m.gmane.org; Mon, 01 Feb 2016 13:24:47 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:42103) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aQJ9S-0006He-9g for guile-user@gnu.org; Mon, 01 Feb 2016 13:24:39 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aQJ9O-0008WK-Sp for guile-user@gnu.org; Mon, 01 Feb 2016 13:24:38 -0500 Original-Received: from avasout08.plus.net ([212.159.14.20]:40681) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aQJ9O-0008WB-Ma for guile-user@gnu.org; Mon, 01 Feb 2016 13:24:34 -0500 Original-Received: from debian ([146.200.69.115]) by avasout08 with smtp id D6QY1s0042VDVCq016QZtJ; Mon, 01 Feb 2016 18:24:34 +0000 X-CM-Score: 0.00 X-CNFS-Analysis: v=2.1 cv=O6PEx0JW c=1 sm=1 tr=0 a=rwaWWJuvmNLHy6Ld/J3jDA==:117 a=rwaWWJuvmNLHy6Ld/J3jDA==:17 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=IkcTkHD0fZMA:10 a=EBOSESyhAAAA:8 a=CCJiPM-nF1rd7SasPsQA:9 a=QEXdDO2ut3YA:10 X-AUTH: rshann@:2500 In-Reply-To: <87fuxci9dl.fsf@netris.org> X-Mailer: Evolution 3.12.9-1+b1 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x (no timestamps) [generic] X-Received-From: 212.159.14.20 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Original-Sender: guile-user-bounces+guile-user=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.user:12339 Archived-At: On Mon, 2016-02-01 at 13:16 -0500, Mark H Weaver wrote: > Richard Shann writes: > > > Can anyone explain what is going on when you try to store strings with > > non-ASCII characters? Here is an example: > > > > guile> (define another-data "Čć") > > guile> another-data > > "�\x8c�\x87" > > guile> (display another-data) > > Čćguile> > > I guess this is Guile 1.x, where strings are merely byte sequences. > Your terminal is using the UTF-8 encoding, where "Čć" is represented as > the byte sequence: > > 0xC4 0x8C 0xC4 0x87 > > When printing this using 'write' (which is how values are printed at the > REPL), Guile 1.x is treating this byte sequence as Latin-1. The 0xC4 is > the Latin-1 representation for the character "Ä", but 0x8C and 0x87 are > not characters in Latin-1 and so are escaped as "\x8c" and "\x87". > > When printing using display, Guile is simply writing the bytes out > unescaped, which your terminal interprets as UTF-8. > > Obviously this is terrible, which is why Guile 2.0+ strings are > sequences of unicode code points. Can you switch to Guile 2.0? Thank you for the detailed explanation. I'm currently asking whether we can switch, the problem will be that LilyPond is stuck at 1.8 because of very complex issues I gather, and Denemo runs LilyPond to typeset the music entered. Is there any reasonable workaround - can I insert something to transform the string? Richard