bug#10627: char-ready? is broken for multibyte encodings

bug#10627: char-ready? is broken for multibyte encodings

[Mark H Weaver]

The R5RS specifies that if 'char-ready?' returns #t, then the next
'read-char' operation is guaranteed not to hang.  This is not currently
the case for ports using a multibyte encoding.

'char-ready?' currently returns #t whenever at least one _byte_ is
available.  This is not correct in general.  It should return #t only if
there is a complete _character_ available.

     Mark

bug#10627: char-ready? is broken for multibyte encodings

[Andy Wingo]

On Sat 28 Jan 2012 11:21, Mark H Weaver <mhw@netris.org> writes:

> The R5RS specifies that if 'char-ready?' returns #t, then the next
> 'read-char' operation is guaranteed not to hang.  This is not currently
> the case for ports using a multibyte encoding.
>
> 'char-ready?' currently returns #t whenever at least one _byte_ is
> available.  This is not correct in general.  It should return #t only if
> there is a complete _character_ available.

This procedure is omitted in the R6RS because it is not a good
interface.  Besides its semantic difficulties, can you think of a sane
implementation for multibyte characters?

I suggest we document that this procedure only works correctly in
encodings with 1-byte characters and recommend that people use u8-ready?
instead.

Andy
-- 
http://wingolog.org/

bug#10627: char-ready? is broken for multibyte encodings

[Mark H Weaver]

Hi Andy,

Andy Wingo <wingo@pobox.com> writes:

> On Sat 28 Jan 2012 11:21, Mark H Weaver <mhw@netris.org> writes:
>
>> The R5RS specifies that if 'char-ready?' returns #t, then the next
>> 'read-char' operation is guaranteed not to hang.  This is not currently
>> the case for ports using a multibyte encoding.
>>
>> 'char-ready?' currently returns #t whenever at least one _byte_ is
>> available.  This is not correct in general.  It should return #t only if
>> there is a complete _character_ available.
>
> This procedure is omitted in the R6RS because it is not a good
> interface.  Besides its semantic difficulties, can you think of a sane
> implementation for multibyte characters?

Maybe I'm missing something, but I don't see any semantic problem here,
and it seems straightforward to implement.  'char-ready?' should simply
read bytes until either a complete character is available, or no more
bytes are ready.  In either case, all the bytes should then be 'unget'
before returning.  What's the problem?

The only reason I haven't yet fixed this is because it will require some
refactoring in ports.c.  I guess the most straightforward approach is to
generalize 'get_codepoint', 'get_utf8_codepoint', and
'get_iconv_codepoint' to support a non-blocking mode of operation.

What do you think?

  Regards,
    Mark

bug#10627: char-ready? is broken for multibyte encodings

[Andy Wingo]

Hi :)

On Sun 24 Feb 2013 21:14, Mark H Weaver <mhw@netris.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> On Sat 28 Jan 2012 11:21, Mark H Weaver <mhw@netris.org> writes:
>>
>>> The R5RS specifies that if 'char-ready?' returns #t, then the next
>>> 'read-char' operation is guaranteed not to hang.  This is not currently
>>> the case for ports using a multibyte encoding.
>>>
>>> 'char-ready?' currently returns #t whenever at least one _byte_ is
>>> available.  This is not correct in general.  It should return #t only if
>>> there is a complete _character_ available.
>>
>> This procedure is omitted in the R6RS because it is not a good
>> interface.  Besides its semantic difficulties, can you think of a sane
>> implementation for multibyte characters?
>
> Maybe I'm missing something, but I don't see any semantic problem here,
> and it seems straightforward to implement.  'char-ready?' should simply
> read bytes until either a complete character is available, or no more
> bytes are ready.  In either case, all the bytes should then be 'unget'
> before returning.  What's the problem?

The problem is that char-ready? should not read anything.  If you want
to peek, use peek-char.  Note that if the stream is at EOF, char-ready?
should return #t.

Andy
-- 
http://wingolog.org/

bug#10627: char-ready? is broken for multibyte encodings

[Mark H Weaver]

Andy Wingo <wingo@pobox.com> writes:

> On Sun 24 Feb 2013 21:14, Mark H Weaver <mhw@netris.org> writes:
>
>> Maybe I'm missing something, but I don't see any semantic problem here,
>> and it seems straightforward to implement.  'char-ready?' should simply
>> read bytes until either a complete character is available, or no more
>> bytes are ready.  In either case, all the bytes should then be 'unget'
>> before returning.  What's the problem?
>
> The problem is that char-ready? should not read anything.

Okay, but if all bytes read are later *unread*, and the reads never
block, then why does it matter?  The reads in my proposed implementation
are just an internal implementation detail, and it seems to me that the
user cannot tell the difference, as long as he does not peek underneath
the Scheme port abstraction.

If you prefer, perhaps a nicer way to think about it is that
'char-ready?' looks ahead in the putback buffer and/or the read buffer
(refilling it in a non-blocking mode if needed), and returns #t iff a
complete character is present in the buffer(s), or EOF is reached.
However, is seems to me that implementing this in terms of read-byte and
unget-byte is simpler, because it avoids duplication of the logic
regarding putback buffers and refilling of buffers.  Maybe there's some
reason why this is a bad idea, but I haven't heard one.

I agree that 'char-ready?' is an antiquated interface, but it is
nonetheless part of the R5RS (and Guile since approximately forever),
and it is the only way to do a non-blocking read in portable R5RS.  It
seems to me that we ought to try to implement it as well as we can, no?

> If you want to peek, use peek-char.

Okay, but that's a totally different tool with a different use case.
It cannot be used to do non-blocking reads.

> Note that if the stream is at EOF, char-ready? should return #t.

Agreed.

More thoughts?

   Thanks,
     Mark

bug#10627: char-ready? is broken for multibyte encodings

[Daniel Hartwig]

On 25 February 2013 08:06, Mark H Weaver <mhw@netris.org> wrote:
> Andy Wingo <wingo@pobox.com> writes:
>
>> On Sun 24 Feb 2013 21:14, Mark H Weaver <mhw@netris.org> writes:
>>
>>> Maybe I'm missing something, but I don't see any semantic problem here,
>>> and it seems straightforward to implement.  'char-ready?' should simply
>>> read bytes until either a complete character is available, or no more
>>> bytes are ready.  In either case, all the bytes should then be 'unget'
>>> before returning.  What's the problem?
>>
>> The problem is that char-ready? should not read anything.
>
> Okay, but if all bytes read are later *unread*, and the reads never
> block, then why does it matter?

Taking care to still use sf_input_waiting for soft ports?  Reading
bytes from a soft port could have side effects (i.e. logging action or
similar).

bug#10627: char-ready? is broken for multibyte encodings

[Andy Wingo]

Hi Mark,

Are you proposing that `char-ready?' do a nonblocking read if
the buffer is empty?  That could work.

On Mon 25 Feb 2013 01:06, Mark H Weaver <mhw@netris.org> writes:

> However, is seems to me that implementing this in terms of read-byte and
> unget-byte is simpler, because it avoids duplication of the logic
> regarding putback buffers and refilling of buffers.

Could work, if the port is nonblocking to begin with.

> I agree that 'char-ready?' is an antiquated interface, but it is
> nonetheless part of the R5RS (and Guile since approximately forever),
> and it is the only way to do a non-blocking read in portable R5RS.  It
> seems to me that we ought to try to implement it as well as we can, no?

Do what you like to do :)  But if it were my time, I would simply
document that it checks for a byte and not a character and move on.

Andy
-- 
http://wingolog.org/

bug#10627: char-ready? is broken for multibyte encodings

[Mark H Weaver]

Andy Wingo <wingo@pobox.com> writes:
> Are you proposing that `char-ready?' do a nonblocking read if
> the buffer is empty?  That could work.

Yes.  I suspect that something along these lines is already implemented,
because I don't see how 'u8-ready?' could work properly without it.

> Do what you like to do :)  But if it were my time, I would simply
> document that it checks for a byte and not a character and move on.

I'd like to fix it properly.  Let's keep this bug open until it's done.

     Thanks,
       Mark

bug#10627: char-ready? is broken for multibyte encodings

[Andy Wingo]

On Tue 26 Feb 2013 20:50, Mark H Weaver <mhw@netris.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>> Are you proposing that `char-ready?' do a nonblocking read if
>> the buffer is empty?  That could work.
>
> Yes.  I suspect that something along these lines is already implemented,
> because I don't see how 'u8-ready?' could work properly without it.

It does a poll with a timeout of 0.

Andy
-- 
http://wingolog.org/

bug#10627: char-ready? is broken for multibyte encodings

[Andy Wingo]

On Tue 26 Feb 2013 20:59, Andy Wingo <wingo@pobox.com> writes:

> On Tue 26 Feb 2013 20:50, Mark H Weaver <mhw@netris.org> writes:
>
>> Andy Wingo <wingo@pobox.com> writes:
>>> Are you proposing that `char-ready?' do a nonblocking read if
>>> the buffer is empty?  That could work.
>>
>> Yes.  I suspect that something along these lines is already implemented,
>> because I don't see how 'u8-ready?' could work properly without it.
>
> It does a poll with a timeout of 0.

In the end I added this to the manual:

    Note that @code{char-ready?} only works reliably for terminals and
    sockets with one-byte encodings.  Under the hood it will return
    @code{#t} if the port has any input buffered, or if the file descriptor
    that backs the port polls as readable, indicating that Guile can fetch
    more bytes from the kernel.  However being able to fetch one byte
    doesn't mean that a full character is available; @xref{Encoding}.  Also,
    on many systems it's possible for a file descriptor to poll as readable,
    but then block when it comes time to read bytes.  Note also that on
    Linux kernels, all file ports backed by files always poll as readable.
    For non-file ports, this procedure always returns @code{#t}, except for
    soft ports, which have a @code{char-ready?} handler.  @xref{Soft Ports}.

    In short, this is a legacy procedure whose semantics are hard to
    provide.  However it is a useful check to see if any input is buffered.
    @xref{Non-Blocking I/O}.

We could try a non-blocking read but at that point we should just
provide a non-blocking read-char, and allow users to unread-char.  That
would be a different bug :)

Andy