unicode status

unicode status

[Andy Wingo]

Hey Mike,

Would you mind posting to the list a "state of unicode & guile" summary?
I'm very excited about finally being able to say "Guile does unicode",
and was wondering what was left to do :)

Andy
-- 
http://wingolog.org/

Re: unicode status

[Mike Gran]

On Sun, 2009-09-06 at 12:45 +0200, Andy Wingo wrote:
> Hey Mike,
> 
> Would you mind posting to the list a "state of unicode & guile" summary?
> I'm very excited about finally being able to say "Guile does unicode",
> and was wondering what was left to do :)
> 
> Andy

OK.  

First, here's the stuff I've already put in NEWS

** Characters

Characters can take the whole Unicode range.  char-upcase and
char-downcase use default Unicode casing rules. Character comparisons
such as char<? and char-ci<? are now sorting based on Unicode code
points.  Combining characters are printed with dotted circles #\◌́

** Strings

String and SRFI-13 functions can operate on Unicode strings.  Strings
can contain the new string escapes \uHHHH and \UHHHHHH for 4 and 6 hex
digit characters.

** SRFI-14 char-sets are modified for Unicode

The default char-sets are not longer locale dependent and contain
characters from the whole Unicode range.  There is a new char-set,
char-set:designated, which contains all assigned Unicode characters.
There is a new debugging function: %char-set-dump.

** Ports do transcoding

Ports now have an associated character encoding, and port read/write
operations do conversion to/from locales automatically.  Ports also
have an associated strategy for how to deal with locale conversion
failures.  Four functions to support this: set-port-encoding!,
port-encoding, set-port-conversion-strategy!,
port-conversion-strategy.

** Non-ASCII source code files can be read, but require coding
   declarations

The default reader now handles source code files for some of the
non-ASCII character encodings, such as UTF-8.  A non-ASCII source file
should have an encoding declaration near the top of the file.  Also,
there is a new function 'file-encoding' that scans a port for a coding
declaration.

The pre-1.9.3 reader handled 8-bit clean but otherwise unspecified
source code.  This use is now discouraged.

-------------------------------------------------------------

Here's some stuff that is complete, but, not working quite right.

** There are undocumented things: %string-dump, %symbol-dump, setbinary,
and a discussion about why ISO-8859-1 is the fastest encoding to process
and why it should be used by default.

** Non-ASCII symbols and keywords are supported and variables and
procedures can have non-ASCII names.

These probably need wide-symbol and wide-keyword support in the VM,
instead of the locale-specific implementation that they have now to
avoid some corner cases where locales switch.

-------------------------------------------------------------

Here's the stuff left to be done, in no particular order

* The disassembler doesn't handle wide strings gracefully

* Some parts of Goops expect 8-bit strings.  This is probably fine for
now, but, needs to be documented.  I've avoided touching this because
I've never used goops for anything, so I'm not sure what does what.

* The i18n library hasn't been touched.  It should probably move to use
functions like u32_casecmp from libunistring for unicode-capable
locale-specific sorting.  But the #ifdef and locale madness in i18n is
deep.  I've avoided hacking it.  Also we'll have to write our own
functions for locale-string->double and locale-string->int.  Bruno has
some suggestions on how to do that at
http://savannah.gnu.org/support/?106998

* I haven't done any testing on readline or gettext

* Unicode-capable regex has not been implemented.  Libunistring might do
this someday.  Until then, there will probably have to be the hack where
strings are converted to UTF-8 encoding to pass through regex.  This
doesn't get you Unicode regex, but, it keeps non-ASCII from being
mangled by regex.

* EMACS has a lot of aliases that can be use in the "-*- coding: XXXXX
-*-" line, like latin-1, that aren't valid encoding names.  The reader
should be modified to understand the common ones.

* The whole issue of R6RS compliance will have to be dealt with some
day.  For example, I went with \xHH \uHHHH and \UHHHHHH escapes because
they were backwards compatible with the \xHH we already had.  R6RS uses
a variable length hex escape terminated by a semicolon: \xHH; \xHHH;.
These are not backward compatible.  There are some R6RS functions that
are missing: string-foldcase, string normalization routines.

Also, R6RS and R5RS seem to disagree on the definition of string-upcase
et al.  R6RS is clear that the result of string-upcase can have more
letters that its input, and it gets rid of string-upcase! for the same
reason.

That's all I remember off the top of my head.

Thanks,

Mike

Re: unicode status

[Ludovic Courtès]

Hello!

Mike Gran <spk121@yahoo.com> writes:

> ** Ports do transcoding

Speaking of this, would you be willing to implement R6RS’ transcoder
API in ‘r6rs-ports.c’?  :-)

> * The i18n library hasn't been touched.  It should probably move to use
> functions like u32_casecmp from libunistring for unicode-capable
> locale-specific sorting.

Is u32_casecmp locale-dependent?

> But the #ifdef and locale madness in i18n is
> deep.

Heh heh, it’s deep but needed.  It allows us to provide an API with
first-class locale objects, akin to POSIX 2008’s ‘locale_t’, which is
neat IMO.

At any rate, the parts you’re interested in can probably be modified
without touching the #ifdef madness.

> I've avoided hacking it.  Also we'll have to write our own functions
> for locale-string->double and locale-string->int.

We have ‘locale-string->integer’ and ‘locale-string->inexact’, which
currently use strtol(3) and strtod(3) respectively (info "(guile) Number
Input and Output").

> Bruno has some suggestions on how to do that at
> http://savannah.gnu.org/support/?106998

This suggestion could probably be implemented in Scheme, similarly to
‘number->locale-string’.

> * EMACS has a lot of aliases that can be use in the "-*- coding: XXXXX
> -*-" line, like latin-1, that aren't valid encoding names.  The reader
> should be modified to understand the common ones.

Also, currently ‘scm_scan_for_encoding ()’ searches for a “coding:”
string in the ‘SCM_ENCODING_SEARCH_SIZE’ first bytes of the file,
whereas Emacs searches in the first line (or second line if the first is
“#!...”) or in a local variable list less than 3000 characters from the
end of the file (info "(emacs) Specifying File Variables").

Overall, it seems to me that Unicode support is in a very good shape and
the points above aren’t too worrying.

Thanks,
Ludo’.

Re: unicode status

[Mike Gran]

On Mon, 2009-09-14 at 00:08 +0200, Ludovic Courtès wrote:
> Hello!
> 
> Mike Gran <spk121@yahoo.com> writes:
> 
> > ** Ports do transcoding
> 
> Speaking of this, would you be willing to implement R6RS’ transcoder
> API in ‘r6rs-ports.c’?  :-)

Hard to say.  After September, my free time evaporates.  However, it
shouldn't be a very difficult task to do.  The difference between R6RS
ports and what we've done so far is the end-of-line conversions that
R6RS requires: CR, CR/LF, NEL, NEL/LF, LS, etc.

> 
> > * The i18n library hasn't been touched.  It should probably move to use
> > functions like u32_casecmp from libunistring for unicode-capable
> > locale-specific sorting.
> 
> Is u32_casecmp locale-dependent?

>From the docs

 -- Function: int u32_casecoll (const uint32_t *S1, size_t N1, const
          uint32_t *S2, size_t N2, const char *ISO639_LANGUAGE,
          uninorm_t NF, int *RESULTP)
     Compares S1 and S2, ignoring differences in case and normalization,
     using the collation rules of the current locale.

> 
> > But the #ifdef and locale madness in i18n is
> > deep.
> 
> Heh heh, it’s deep but needed.  It allows us to provide an API with
> first-class locale objects, akin to POSIX 2008’s ‘locale_t’, which is
> neat IMO.
> 
> At any rate, the parts you’re interested in can probably be modified
> without touching the #ifdef madness.

The libunistring way for sorting would be something like
1. set the locale
2. convert the strings to unistring u32 strings
3. get the locale's 'language' with uc_locale_language ()
4. use the language and strings as input to u32_strcoll or u32_casecoll
5. profit!

So once that problem of setting the locale and getting the
uc_locale_language is solved generically for one of the i18n funcs, the
rest should fall into place.  If, in your copious free time (LOL), you
want to figure out that for one func, I can do the rest by extension.
Otherwise, I'll get to it eventually.

You can't really do unicode sorting without also including the
normalization functions string-normalize-nfc, string-normalize-nfkc etc
from (rnrs unicode (6)) so those'll need to be added.  That also isn't
hard: libunistring does the low-level op.

> Overall, it seems to me that Unicode support is in a very good shape and
> the points above aren’t too worrying.

Thanks,

Mike