bug#26058: utf16->string and utf32->string don't conform to R6RS

[taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)]

Andy Wingo <wingo@pobox.com> writes:

> Hi,
>
> this is a tricky area that is not so amenable to quick patches :) Have
> you looked into what Guile already does for byte-order marks?  Can you
> explain how the R6RS specification relates to this?
>
>   https://www.gnu.org/software/guile/manual/html_node/BOM-Handling.html
>
> Andy

Hmm, interesting.  I noticed the utf{16,32}->string procedures ignoring
a BOM at the start of the given bytevector, but didn't look at it from a
ports perspective.

TL;DR of the below though: the R6RS semantics offer a strict enrichment
of the feature-set of the utfX->string procedures relative to the Guile
semantics, so at most we would end up with spurious features.  (The
optional ability to handle any possible BOM at the start of the
bytevector, with a fall-back endianness in case none is found.)


That said, let's see...

If I do a textual read from a port, I already get a string and not a
bytevector, so the behavior of utfX->string operations is irrelevant.

If I do binary I/O, the following situations are possible:

1. I'm guaranteed to get any possible bytes that happen to form a valid
   BOM at the start of the stream as-is in the returned bytevector; the
   binary I/O interface doesn't see such bytes as anything special, as
   it could simply be coincidence that the stream starts with such
   bytes.

2. I'm guaranteed *not* to get bytes that form a BOM at the start of the
   stream; instead they're consumed to set the port encoding for any
   future text I/O.

3. The behavior is unspecified and either of the above may happen.

In the case of #1, it's probably good for utfX->string procedures to be
able to handle BOMs, but also allow explicitly ignoring any possible
BOM.  The R6RS semantics cover this.

In the case of #2, the utfX->string procedures don't need to be able to
handle BOMs as far as we're talking about passing them bytevectors
returned by port I/O, but it also doesn't hurt if they optionally
support it.  The R6RS semantics are fine here as well I think.

As for #3... first of all it's bad IMO; the behavior ought to be
specified. :-) But in any case, the additional features of the R6RS
semantics won't hurt.

WDYT?  As far as I understand the page you linked, Guile currently
implements #3, which I think is unfortunate but can kinda understand
too.  In any case, the additional R6RS features won't hurt, right?

Taylan