Re: Wide strings

[Mike Gran <spk121@yahoo.com>]

> From: Ludovic Courtès ludo@gnu.org

I believe that we should aim for R6RS strings.

I think the most important thing is to have humility in the face of an
impossible problem: how to encode all textual information.  It is
important to "stand on the shoulders of giants" here.  It becomes a
matter of deciding which actively developed library of wide character
functions is to be used and how to integrate it.

There are 3 good, actively developed solutions of which I am aware.

1.  Use GNU libc functionality.  Encode wide strings as wchar_t.

2.  Use GLib functionality.  Encode wide strings as UTF-8.  Possibly
give up on O(1).  Possibly add indexing information to string to allow
O(1), which might negate the space advantage of UTF-8.
 
3.  Use IBM's ICU4c.  Encode wide strings as UTF-16.  Thus, add an
obscure dependency.

Option 3 is likely a non-starter, because it seems that Guile has
tried to avoid adding new non-GNU dependencies.  It is technologically
a great solution, IMHO.

Option 1 is probably the way to go, because it keeps Guile close to
the metal and keeps dependencies out of it.  Unfortunately, UTF-8
strings would require conversion.

>  1. IMO it'd be nice to have ASCII strings special-cased so that they
>    are always encoded in ASCII.  This would allow for memory savings
>    since, e.g., most symbols are expected to contain only ASCII
>    characters.  It might also simplify interaction with C in certain
>    cases; for instance, it would make it easy to have statically
>    initialized ASCII Scheme strings.

Why not?  It does solve the initialization problem of dealing with strings
before setlocale has been called.

Let's say that a string is a union of either an ASCII char vector or a
wchar_t vector.  A "character" then is just a Unicode codepoint.
String-ref returns a wchar_t.  This is all in line with R6RS as I
understand it.

There could then be a separate iterator and function set that does
(likely O(n)) operations on the grapheme clusters of strings.  A
grapheme cluster is a single written symbol which may be made up of
several codepoints.  Unicode Standard Annex #29 describes how to
partition a string into a set of graphemes.[1]

There is the problem of systems where wchar_t is 2 bytes instead of 4
bytes, like Cygwin.  For those systems, I'd recommend
restricting functionality to 16-bit characters instead of trying to
add an extra UTF-16 encoding/decoding step.  I think there should
always be a complete codepoint in each wchar_t.

-- 
Mike Gran

[1] http://www.unicode.org/reports/tr29/