%default-port-conversion-strategy and string ports

%default-port-conversion-strategy and string ports

[Ludovic Courtès]

Hello!

Commit b22e94db7c91d7661204e33f3bc2bfead002c9b7 adds
‘%default-port-conversion-strategy’, a natural friend of
‘%default-port-encoding’.

First, I’m wondering whether ‘port’ should be part of the name, given
that it’s also referred to by ‘scm_stringn’ & co.  It’s good to have it
in the name, for the symmetry with ‘%default-port-encoding’, but it’s
not accurate.

Second, in commit 9f6e3f5a997f484548bd03e7e7573c38a95c8d09, I changed
string ports to honor it, like other port types, instead of forcing
'error.  This seems like the right thing to me, for the sake of
consistency (in fact, I’d consider the previous behavior as a bug), but
it’s an observable change.

WDYT?

Thanks,
Ludo’.

Re: %default-port-conversion-strategy and string ports

[Mike Gran]

>Second, in commit 9f6e3f5a997f484548bd03e7e7573c38a95c8d09, I changed
>string ports to honor it, like other port types, instead of forcing
>'error.  This seems like the right thing to me, for the sake of
>consistency (in fact, I’d consider the previous behavior as a bug), but
>it’s an observable change.

Sounds fair.  The 'error behavior for was once coupled with UTF-8, making
the error mostly moot at the time, but, if string ports are honoring encoding
they should probably honor the conversion strategy as well.
 
-Mike

Re: %default-port-conversion-strategy and string ports

[David Kastrup]

ludo@gnu.org (Ludovic Courtès) writes:

> Hello!
>
> Commit b22e94db7c91d7661204e33f3bc2bfead002c9b7 adds
> ‘%default-port-conversion-strategy’, a natural friend of
> ‘%default-port-encoding’.
>
> First, I’m wondering whether ‘port’ should be part of the name, given
> that it’s also referred to by ‘scm_stringn’ & co.  It’s good to have it
> in the name, for the symmetry with ‘%default-port-encoding’, but it’s
> not accurate.
>
> Second, in commit 9f6e3f5a997f484548bd03e7e7573c38a95c8d09, I changed
> string ports to honor it, like other port types, instead of forcing
> 'error.  This seems like the right thing to me, for the sake of
> consistency (in fact, I’d consider the previous behavior as a bug), but
> it’s an observable change.
>
> WDYT?

Shouldn't strings be in "internal encoding" anyway?  The whole point of
a string is to be an array of characters.  Not an array of arbitrarily
encoded bytes.

-- 
David Kastrup

Re: %default-port-conversion-strategy and string ports

[Ludovic Courtès]

Hi,

David Kastrup <dak@gnu.org> skribis:

> Shouldn't strings be in "internal encoding" anyway?  The whole point of
> a string is to be an array of characters.  Not an array of arbitrarily
> encoded bytes.

Yes, but I was referring to “string ports”, which may actually be fed
arbitrary binary data, not just characters.

Thanks,
Ludo’.

Re: %default-port-conversion-strategy and string ports

[David Kastrup]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi,
>
> David Kastrup <dak@gnu.org> skribis:
>
>> Shouldn't strings be in "internal encoding" anyway?  The whole point of
>> a string is to be an array of characters.  Not an array of arbitrarily
>> encoded bytes.
>
> Yes, but I was referring to “string ports”, which may actually be fed
> arbitrary binary data, not just characters.

How so?  A string is an array of characters.  Arbitrary binary data is
an array of bytes.  Merrily mixing the two is not going to lead to
consistent results: you are going to have things accidentally decoded
more than once or not at all, and accidentally encoded more than once or
not at all.

Emacs _does_ have unibyte-string as a data structure of raw bytes for
efficiency reasons, but it is not clear that the hassle is worth it.
You _can_ read binary data into multibyte strings: non-utf-8 sequences
are then put into special code places so that they can be recovered
unchanged when encoding again.  So for strings, Emacs has the two kinds:
unibyte (raw data) and multibyte (conceptually an array of Unicode
characters in some hidden multibyte encoding incidentally quite close to
utf-8).

And that is all.  The rest is decoded (typically from unibyte) into
multibyte, and encoded back when writing it somewhere.  How are you even
supposed to deal with combining strings when they can be encoded
differently?

-- 
David Kastrup

Re: %default-port-conversion-strategy and string ports

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:
> David Kastrup <dak@gnu.org> skribis:
>
>> Shouldn't strings be in "internal encoding" anyway?  The whole point of
>> a string is to be an array of characters.  Not an array of arbitrarily
>> encoded bytes.
>
> Yes, but I was referring to “string ports”, which may actually be fed
> arbitrary binary data, not just characters.

For the record, I agree with David.  String ports should be textual
ports, not binary ports.  In particular, you should be able to
write/read _any_ character to/from a string port, regardless of what the
current default port encoding happens to be.

SRFI-6 (string ports) says nothing about port encodings, and yet
portable code written for SRFI-6 will fail on Guile 2.0 unless the
string is constrained to whatever the default port encoding happens to
be.  This is not just a theoretical issue; it has caused trouble in
practice, e.g.:

  http://bugs.gnu.org/11197

Unfortunately, we are now in an awkward situation.  The current behavior
of Guile 2.0 is conceptually broken and breaks portable SRFI-6 code, and
yet it is possible that some users have grown to depend on our current
behavior.

     Mark

Re: %default-port-conversion-strategy and string ports

[Ludovic Courtès]

Hi!

Mark H Weaver <mhw@netris.org> skribis:

> SRFI-6 (string ports) says nothing about port encodings, and yet
> portable code written for SRFI-6 will fail on Guile 2.0 unless the
> string is constrained to whatever the default port encoding happens to
> be.  This is not just a theoretical issue; it has caused trouble in
> practice, e.g.:
>
>   http://bugs.gnu.org/11197

Hey, there’s a patch for SRFI-6 there.  Could we resume the discussion
in that bug?

Guile ports are mixed textual/binary ports.  Whether this or separate
binary/textual ports as in R6 is best is an interesting question, but as
you note, we cannot really change that currently.

Thanks,
Ludo’.

Re: %default-port-conversion-strategy and string ports

[David Kastrup]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi!
>
> Mark H Weaver <mhw@netris.org> skribis:
>
>> SRFI-6 (string ports) says nothing about port encodings, and yet
>> portable code written for SRFI-6 will fail on Guile 2.0 unless the
>> string is constrained to whatever the default port encoding happens to
>> be.  This is not just a theoretical issue; it has caused trouble in
>> practice, e.g.:
>>
>>   http://bugs.gnu.org/11197
>
> Hey, there’s a patch for SRFI-6 there.  Could we resume the discussion
> in that bug?
>
> Guile ports are mixed textual/binary ports.  Whether this or separate
> binary/textual ports as in R6 is best is an interesting question, but as
> you note, we cannot really change that currently.

I don't understand this distinction.  A port transfers characters, like
strings contain characters.  The relation is 1:1.  The question of
encoding only concerns ports connected to a file, or a terminal, and
then textual/binary is a question of encoding/decoding.  A port that
stays within Guile has no business being concerned with encoding.  It
has to reproduce the characters from its input to its output without
change.

Things are complicated enough talking to the outside.  There is no point
in Guile being confused even when talking to itself.

-- 
David Kastrup

Re: %default-port-conversion-strategy and string ports

[Ludovic Courtès]

Hi David,

Ports in Guile can be used to write characters, or bytes, or both.  In
particular, every port (including string ports, void ports, etc.) has an
“encoding”, which is actually only used for textual I/O.

Conversely, an R6RS port is either textual or binary, but not both.

IMO, one advantage of mixed text/binary ports is to allow things like this:

  scheme@(guile-user)> (define (string->utf16 s)
                         (let ((p (with-fluids ((%default-port-encoding "UTF-16BE"))
                                    (open-input-string s))))
                           (get-bytevector-all p)))
  scheme@(guile-user)> (string->utf16 "hello")
  $4 = #vu8(0 104 0 101 0 108 0 108 0 111)
  scheme@(guile-user)> (use-modules(rnrs bytevectors))
  scheme@(guile-user)> (utf16->string $4)
  $5 = "hello"

Thanks,
Ludo’.

Re: %default-port-conversion-strategy and string ports

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:
> Ports in Guile can be used to write characters, or bytes, or both.  In
> particular, every port (including string ports, void ports, etc.) has an
> “encoding”, which is actually only used for textual I/O.
>
> Conversely, an R6RS port is either textual or binary, but not both.
>
> IMO, one advantage of mixed text/binary ports is to allow things like this:
>
>   scheme@(guile-user)> (define (string->utf16 s)
>                          (let ((p (with-fluids ((%default-port-encoding "UTF-16BE"))
>                                     (open-input-string s))))
>                            (get-bytevector-all p)))
>   scheme@(guile-user)> (string->utf16 "hello")
>   $4 = #vu8(0 104 0 101 0 108 0 108 0 111)
>   scheme@(guile-user)> (use-modules(rnrs bytevectors))
>   scheme@(guile-user)> (utf16->string $4)
>   $5 = "hello"

IMHO, this is a bad hack that exposes internal details of our
implementation of string ports, and whose only utility AFAIK is to
(partially) make up for our lack of a proper 'iconv' interface from
Scheme.  If we could enable this hack without compromising the
robustness of the _primary_ use case for string ports, then I would
merely frown at this leak of internal implementation details.
Unfortunately, it's worse than that.

Anyway, I suppose that we are now locked into this broken behavior, at
least for Guile 2.0.  I guess that your proposed solution (to make
SRFI-6 export alternative versions of 'open-input-string' and
'open-output-string' that always use UTF-8 for the port encoding) is the
best we can do now.

I have concerns about this solution, but I can't think of a better one,
given the unfortunate fact that our current semantics have been widely
deployed (and worse, that the above hack has been advertised on
guile-user as a way to work around our lack of 'iconv' from Scheme).

     Mark

Re: %default-port-conversion-strategy and string ports

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>> Ports in Guile can be used to write characters, or bytes, or both.  In
>> particular, every port (including string ports, void ports, etc.) has an
>> “encoding”, which is actually only used for textual I/O.
>>
>> Conversely, an R6RS port is either textual or binary, but not both.
>>
>> IMO, one advantage of mixed text/binary ports is to allow things like this:
>>
>>   scheme@(guile-user)> (define (string->utf16 s)
>>                          (let ((p (with-fluids ((%default-port-encoding "UTF-16BE"))
>>                                     (open-input-string s))))
>>                            (get-bytevector-all p)))
>>   scheme@(guile-user)> (string->utf16 "hello")
>>   $4 = #vu8(0 104 0 101 0 108 0 108 0 111)
>>   scheme@(guile-user)> (use-modules(rnrs bytevectors))
>>   scheme@(guile-user)> (utf16->string $4)
>>   $5 = "hello"
>
> IMHO, this is a bad hack that exposes internal details of our
> implementation of string ports

Which details?

It exposes the fact that ports in general are mixed textual/binary, but
nothing specific to string ports AFAICS.

> and whose only utility AFAIK is to (partially) make up for our lack
> of a proper 'iconv' interface from Scheme.

Well, ‘string->pointer’ and ‘scm_to_stringn’ can be thought of as an
iconv interface, no?

I understand you’re in favor of separate textual/binary ports.  It
may have its advantages; yet, it’s not clear to me that mixed
binary/textual ports are a kludge either.

We’d have to dig r6rs-discuss, but my recollection is that there were
arguments both in favor and against separate binary/textual ports.

Last but not least, mixed ports were decided a couple of years ago as a
reasonable option to allow a smooth transition from 1.8 where the API
was already mixed (think ‘write’, vs. ‘uniform-vector-write’,
vs. ‘scm_c_write’, etc.)  It may not be ideal, but I’m convinced it’s a
good compromise.  It’s not all black and white.

[...]

> I guess that your proposed solution (to make
> SRFI-6 export alternative versions of 'open-input-string' and
> 'open-output-string' that always use UTF-8 for the port encoding) is the
> best we can do now.

OK, I’ll do that.

> I have concerns about this solution, but I can't think of a better one,
> given the unfortunate fact that our current semantics have been widely
> deployed (and worse, that the above hack has been advertised on
> guile-user as a way to work around our lack of 'iconv' from Scheme).

Again, I’ve advertised it because I find it useful, and because I don’t
consider it such an ugly kludge.

Thanks,
Ludo’.

Re: %default-port-conversion-strategy and string ports

[David Kastrup]

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>> Ports in Guile can be used to write characters, or bytes, or both.  In
>>> particular, every port (including string ports, void ports, etc.) has an
>>> “encoding”, which is actually only used for textual I/O.
>>>
>>> Conversely, an R6RS port is either textual or binary, but not both.
>>>
>>> IMO, one advantage of mixed text/binary ports is to allow things like this:
>>>
>>>   scheme@(guile-user)> (define (string->utf16 s)
>>>                          (let ((p (with-fluids ((%default-port-encoding "UTF-16BE"))
>>>                                     (open-input-string s))))
>>>                            (get-bytevector-all p)))
>>>   scheme@(guile-user)> (string->utf16 "hello")
>>>   $4 = #vu8(0 104 0 101 0 108 0 108 0 111)
>>>   scheme@(guile-user)> (use-modules(rnrs bytevectors))
>>>   scheme@(guile-user)> (utf16->string $4)
>>>   $5 = "hello"
>>
>> IMHO, this is a bad hack that exposes internal details of our
>> implementation of string ports
>
> Which details?
>
> It exposes the fact that ports in general are mixed textual/binary, but
> nothing specific to string ports AFAICS.

If I can't rely on

(string= (with-output-to-string (display x)) x)

then the interface is seriously rotten.  There is lots of code around
that depends on the ability to bounce material between strings and the
default output port.

> We’d have to dig r6rs-discuss, but my recollection is that there were
> arguments both in favor and against separate binary/textual ports.

The question binary/textual concerns ports connected to a file.  String
ports and Scheme ports should be _transparent_: input and output
identical.  They are used for connecting character streams within Scheme
and should not tamper with them.

-- 
David Kastrup

Re: %default-port-conversion-strategy and string ports

[Daniel Krueger]

Hey,

I don't want to comment on what guile should choose to do in the
future but just wanted to say which interface would be clear to me.

In the first place I agree that ports should be seperated and not
mixed textual/binary (mind, I know it may be that this can't be just
changed that easily in guile). There you either work with characters
and don't mind how they are encoded, or you work on binary data, which
has some encoding, if it is for letters or something else like rgb
data etc.
Then there is are procedures which explicitly convert numbers<->chars
and bytevectors<->strings, which are partly there (number->char and
char->number afaik), but which lack the ability to specify which
encoding they should use (I don't know if they have to use a specific
encoding in rnrs).
And the last is, you have to have open-string-input/output-port, which
opens a textual port and open-bytevector-input/output-port, which
opens a binary port. So if you want to work with the encoded
characters of a string you say `(open-bytevector-input-port
(string->bytevector "my string" (character-encoding UTF-8)))`.

WDYT?

- Daniel

On Sat, Jun 2, 2012 at 4:55 PM, David Kastrup <dak@gnu.org> wrote:
> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> ludo@gnu.org (Ludovic Courtès) writes:
>>>> Ports in Guile can be used to write characters, or bytes, or both.  In
>>>> particular, every port (including string ports, void ports, etc.) has an
>>>> “encoding”, which is actually only used for textual I/O.
>>>>
>>>> Conversely, an R6RS port is either textual or binary, but not both.
>>>>
>>>> IMO, one advantage of mixed text/binary ports is to allow things like this:
>>>>
>>>>   scheme@(guile-user)> (define (string->utf16 s)
>>>>                          (let ((p (with-fluids ((%default-port-encoding "UTF-16BE"))
>>>>                                     (open-input-string s))))
>>>>                            (get-bytevector-all p)))
>>>>   scheme@(guile-user)> (string->utf16 "hello")
>>>>   $4 = #vu8(0 104 0 101 0 108 0 108 0 111)
>>>>   scheme@(guile-user)> (use-modules(rnrs bytevectors))
>>>>   scheme@(guile-user)> (utf16->string $4)
>>>>   $5 = "hello"
>>>
>>> IMHO, this is a bad hack that exposes internal details of our
>>> implementation of string ports
>>
>> Which details?
>>
>> It exposes the fact that ports in general are mixed textual/binary, but
>> nothing specific to string ports AFAICS.
>
> If I can't rely on
>
> (string= (with-output-to-string (display x)) x)
>
> then the interface is seriously rotten.  There is lots of code around
> that depends on the ability to bounce material between strings and the
> default output port.
>
>> We’d have to dig r6rs-discuss, but my recollection is that there were
>> arguments both in favor and against separate binary/textual ports.
>
> The question binary/textual concerns ports connected to a file.  String
> ports and Scheme ports should be _transparent_: input and output
> identical.  They are used for connecting character streams within Scheme
> and should not tamper with them.
>
> --
> David Kastrup
>
>

Separate textual/binary ports vs. mixed ports

[Ludovic Courtès]

Hi,

Daniel Krueger <keenbug@googlemail.com> skribis:

> In the first place I agree that ports should be seperated and not
> mixed textual/binary

Why?

(I understand how this would work in terms of APIs & co., I’m just
unsure what the rationale is.)

Thanks,
Ludo’.

Re: Separate textual/binary ports vs. mixed ports

[Daniel Krueger]

Hey,

I think if you only use them seperate there's a clearer distinction.
If you have it mixed you can do some, say hacking, where you see it
works but you can't see anywhere what you're exactly doing, most of it
is hidden in the guile implementation, which interprets
%default-port-conversion-strategy and gives you the coding. In one
case you maybe rely on %default-port-conversion-strategy normally
being UTF-8 and then someone sets it to something else, which could
give some hard to track errors. I think explicity just makes code much
clearer and I think seperating textual and binary ports leads to more
explicity, that's my point of view..

WDYT?

- Daniel

Re: Separate textual/binary ports vs. mixed ports

[Noah Lavine]

I agree that separate binary and textual ports are cleaner, but what
about using a port to deal with a mixed binary/textual protocol, like
HTTP? I think the cleanest way to deal with that would be to have a
port where you first read characters and then read binary data.

That doesn't directly address the string-port issue, though.

Noah

On Tue, Jun 5, 2012 at 5:31 AM, Daniel Krueger <keenbug@googlemail.com> wrote:
> Hey,
>
> I think if you only use them seperate there's a clearer distinction.
> If you have it mixed you can do some, say hacking, where you see it
> works but you can't see anywhere what you're exactly doing, most of it
> is hidden in the guile implementation, which interprets
> %default-port-conversion-strategy and gives you the coding. In one
> case you maybe rely on %default-port-conversion-strategy normally
> being UTF-8 and then someone sets it to something else, which could
> give some hard to track errors. I think explicity just makes code much
> clearer and I think seperating textual and binary ports leads to more
> explicity, that's my point of view..
>
> WDYT?
>
> - Daniel
>

Re: Separate textual/binary ports vs. mixed ports

[Ludovic Courtès]

Hi!

Noah Lavine <noah.b.lavine@gmail.com> skribis:

> I agree that separate binary and textual ports are cleaner, but what
> about using a port to deal with a mixed binary/textual protocol, like
> HTTP? I think the cleanest way to deal with that would be to have a
> port where you first read characters and then read binary data.

Yes, but I think ‘transcoded-port’ in R6 helps solve that (you open a
binary port, then use ‘transcoded-port’ to open a textual port above it.)

Thanks,
Ludo’.

Re: Separate textual/binary ports vs. mixed ports

[Ludovic Courtès]

Hi,

Daniel Krueger <keenbug@googlemail.com> skribis:

> If you have it mixed you can do some, say hacking, where you see it
> works but you can't see anywhere what you're exactly doing, most of it
> is hidden in the guile implementation, which interprets
> %default-port-conversion-strategy and gives you the coding. In one
> case you maybe rely on %default-port-conversion-strategy normally
> being UTF-8 and then someone sets it to something else, which could
> give some hard to track errors.

(You mean %default-port-encoding, right?)

I think this is an argument against %default-port-encoding, because the
“clarity” problem described here applies just the same for purely
textual ports.

> I think explicity just makes code much clearer and I think seperating
> textual and binary ports leads to more explicity, that's my point of
> view..

Yeah, I understand the idea, but I’m not sure how it translates to
concrete use cases.  ;-)

Thanks for your feedback!

Ludo’.