Re: Wide strings status

[ludo@gnu.org (Ludovic Courtès)]

Hello!

Mike Gran <spk121@yahoo.com> writes:

> Strings are internally encoded either as "narrow" 8-bit ISO-8859-1
> strings or as "wide" UTF-32 strings.  Strings are usually created as
> narrow strings.  Narrow strings get automatically widened to wide
> strings if non-8-bit characters are set! or appended to them.

Great!

> The machine-readable "write" form of strings has been changed.  Before,
> non-printable characters were given as hex escapes, for example \xFF.
> Now there are three levels of hex escape for 8, 16, and 24 bit
> characters: \xFF, \uFFFF, \UFFFFFF.  This is a pretty common convention.
> But after I coded this, I noticed that R6RS has a different convention
> and I'll probably go with that.

OK.  I think it's probably good to follow R6RS when it has something to
say.

> The internal representation of strings seems to work already, but, the
> reader doesn't work yet.  For now, one can make wide strings like this:
>
>> (setlocale LC_ALL "")
> ==> "en_US.UTF-8"
>
>> (define str (apply string (map integer->char '(100 200 300 400 500))))
>
>> (write str)
> ==>"d\xc8\u012c\u0190\u01f4"
>
> (display str)
> ==>dÈĬƐǴ

Eh eh, looks nice.  Looking forward to typing `(λ (x y) (+ x y))'.  ;-)

> This is all going to be slower than before because of the string
> conversion operations, but, I didn't want to do any premature
> optimization.  First, I wanted to get it working, but, there is plenty
> of room for optimization later.

Good.  Maybe it'd be nice to add simple micro-benchmarks for
`string-ref', `string-set!' et al. under `benchmarks'.

> Character encoding needs to be a property of ports, so that not all
> string operations are done in the current locale.  This is necessary so
> that UTF-8-encoded source files are not interpreted differently based on
> the current locale.

You seem to imply that `scm_getc ()' will now return a Unicode
codepoint, is that right?  What about `scm_c_{read,write} ()', and
`scm_{get,put}s ()'?

> The VM and interpreter need to be updated to deal with wide chars and
> probably in other ways that are unclear to me now.  Wide strings are
> currently getting truncated to 8-bit somewhere in there.

The compiler could use bytevectors when dealing with bytecode.  Maybe
that would clarify things.

Thanks,
Ludo'.