Transcoder library

Transcoder library

[Mike Gran]

Hi-

R6RS describes a transcoder library that converts locale strings to
codepoints and back.  I put together something that is like the
library described in R6RS.

Basically, one would create a transcoder like this...

(define TC (make-transcoder "UTF-8"))

... and then use it either to read codepoints from encoded files ...

(read-codepoint PORT TC)
(write-codepoint CODEPOINT PORT TC)

... or convert to/from vectors of codepoints ...

(locale-string->u32vector STRING TC)
(u32vector->locale-string U32VECTOR TC)

Not very useful on its own, but, perhaps will have more utility in the
future if someone implements R6RS ports and Unicode-enable
Guile.  ;-)

There's more info about it at http://lonelycactus.com/transcoder.html

-Mike Gran

Re: Transcoder library

[Ludovic Courtès]

Hi Mike,

Mike Gran <spk121@yahoo.com> writes:

> R6RS describes a transcoder library that converts locale strings to
> codepoints and back.  I put together something that is like the
> library described in R6RS.

That's good news!

> Basically, one would create a transcoder like this...
>
> (define TC (make-transcoder "UTF-8"))

It's nice you already have a first stab at this API.

> ... and then use it either to read codepoints from encoded files ...
>
> (read-codepoint PORT TC)
> (write-codepoint CODEPOINT PORT TC)

That would be roughly equivalent to the procedures in R6RS Section
8.2.9, right?

> ... or convert to/from vectors of codepoints ...
>
> (locale-string->u32vector STRING TC)
> (u32vector->locale-string U32VECTOR TC)

FWIW, I implemented the `string->utf*' and `utf*->string' procedures in
Guile-R6RS-Libs [0,1] using the relevant Gnulib modules.  I'm planning
to integrate it in Guile `master' at some point; would that be helpful
to you to put it all together?

Thanks,
Ludo'.

[0] http://www.fdn.fr/~lcourtes/software/guile/guile-r6rs-libs-0.1.tar.gz
[1] http://repo.or.cz/w/guile-r6rs-libs.git

Re: Transcoder library

[Mike Gran]

On Wed, 2009-04-01 at 23:24 +0200, Ludovic Courtès wrote:
> Hi Mike,
> 
> Mike Gran <spk121@yahoo.com> writes:

> > Basically, one would create a transcoder like this...
> >
> > (define TC (make-transcoder "UTF-8"))
> > ... and then use it either to read codepoints from encoded files ...
> >
> > (read-codepoint PORT TC)
> > (write-codepoint CODEPOINT PORT TC)
> 
> That would be roughly equivalent to the procedures in R6RS Section
> 8.2.9, right?
> 

That was the intention.  R6RS has the transcoder as a property of the
port, so I wrote port and transcoder side-by-side to imply that
association.  

> > (locale-string->u32vector STRING TC)
> > (u32vector->locale-string U32VECTOR TC)
> 
> FWIW, I implemented the `string->utf*' and `utf*->string' procedures
in
> Guile-R6RS-Libs [0,1] using the relevant Gnulib modules.  I'm planning
> to integrate it in Guile `master' at some point; would that be helpful
> to you to put it all together?
> 

Funny.  I did actually look through Guile-R6RS before putting this
together, but, somehow I overlooked that.  Yours has the feature of
dealing with various sizes of int.  Nice.  Mine has some of the ugly
code required to deal with the conversion of line endings that R6RS
requests.

With respect to the greater goal of wide strings in Guile, I have
played with it quite a bit, though I haven't posted much code. Now, I
am convinced that the character encoding should be a property of the
port.  It would be helpful, then, to have R6RS ports integrated at some
point.

Thanks,

Mike Gran

Re: Transcoder library

[Ludovic Courtès]

Hi,

Mike Gran <spk121@yahoo.com> writes:

> Funny.  I did actually look through Guile-R6RS before putting this
> together, but, somehow I overlooked that.  Yours has the feature of
> dealing with various sizes of int.  Nice.  Mine has some of the ugly
> code required to deal with the conversion of line endings that R6RS
> requests.

That's nice!  I'm working towards merging Guile-R6RS-Libs in `master'.

> With respect to the greater goal of wide strings in Guile, I have
> played with it quite a bit, though I haven't posted much code. Now, I
> am convinced that the character encoding should be a property of the
> port.  It would be helpful, then, to have R6RS ports integrated at some
> point.

OK.

Thanks,
Ludo'.

Re: Transcoder library

[Ludovic Courtès]

Hi Mike!

I was about to add you as a member of the Guile project at Savannah but
it seems that you don't have an account there.  Could you create one?

We'll then create a `wide-strings' (or similar) branch, where you'll be
able to experiment.

Thanks,
Ludo'.