Re: unicode status

[Mike Gran <spk121@yahoo.com>]

On Mon, 2009-09-14 at 00:08 +0200, Ludovic Courtès wrote:
> Hello!
> 
> Mike Gran <spk121@yahoo.com> writes:
> 
> > ** Ports do transcoding
> 
> Speaking of this, would you be willing to implement R6RS’ transcoder
> API in ‘r6rs-ports.c’?  :-)

Hard to say.  After September, my free time evaporates.  However, it
shouldn't be a very difficult task to do.  The difference between R6RS
ports and what we've done so far is the end-of-line conversions that
R6RS requires: CR, CR/LF, NEL, NEL/LF, LS, etc.

> 
> > * The i18n library hasn't been touched.  It should probably move to use
> > functions like u32_casecmp from libunistring for unicode-capable
> > locale-specific sorting.
> 
> Is u32_casecmp locale-dependent?

>From the docs

 -- Function: int u32_casecoll (const uint32_t *S1, size_t N1, const
          uint32_t *S2, size_t N2, const char *ISO639_LANGUAGE,
          uninorm_t NF, int *RESULTP)
     Compares S1 and S2, ignoring differences in case and normalization,
     using the collation rules of the current locale.

> 
> > But the #ifdef and locale madness in i18n is
> > deep.
> 
> Heh heh, it’s deep but needed.  It allows us to provide an API with
> first-class locale objects, akin to POSIX 2008’s ‘locale_t’, which is
> neat IMO.
> 
> At any rate, the parts you’re interested in can probably be modified
> without touching the #ifdef madness.

The libunistring way for sorting would be something like
1. set the locale
2. convert the strings to unistring u32 strings
3. get the locale's 'language' with uc_locale_language ()
4. use the language and strings as input to u32_strcoll or u32_casecoll
5. profit!

So once that problem of setting the locale and getting the
uc_locale_language is solved generically for one of the i18n funcs, the
rest should fall into place.  If, in your copious free time (LOL), you
want to figure out that for one func, I can do the rest by extension.
Otherwise, I'll get to it eventually.

You can't really do unicode sorting without also including the
normalization functions string-normalize-nfc, string-normalize-nfkc etc
from (rnrs unicode (6)) so those'll need to be added.  That also isn't
hard: libunistring does the low-level op.

> Overall, it seems to me that Unicode support is in a very good shape and
> the points above aren’t too worrying.

Thanks,

Mike