bug#35350: Some compile output still leaks through with --verbosity=1

["Ludovic Courtès" <ludo@gnu.org>]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> The third read(2) call here ends on a partial UTF-8 sequence for LEFT
>> SINGLE QUOTATION MARK (we get the first two bytes of a three byte
>> sequence.)
>>
>> What happens is that ‘process-stderr’ in (guix store) gets that byte
>> string from the daemon, passes it through ‘read-maybe-utf8-string’,
>> which replaces the last two bytes with REPLACEMENT CHARACTER, which is
>> itself a 3-byte sequence.
>
> It seems to me that what's needed here is to save the UTF-8 decoder
> state between calls to 'process-stderr'.

So there are two things.  To fix the issue you reported (build output
that goes through), I think we must simply turn off UTF-8 decoding from
‘process-stderr’ and leave that entirely to ‘build-event-output-port’.

However, ‘build-event-output-port’ would still fail to properly decode
split UTF-8 sequences, and for that we’d need to preserve decoder state
as you describe.

> Coincidentally, I also needed something like this a week ago, when I
> tried implementing R6RS custom textual input/output ports on top of
> R6RS custom binary input/output ports.
>
> To meet these needs, I've implemented a fairly efficient, purely
> functional UTF-8 decoder in Scheme that accepts a decoder state and an
> arbitrary range from a bytevector, and returns a new decoder state.
> There's a macro that allows arbitrary actions to be performed when a
> code point (or maximal subpart in the case of errors) is found.
>
> This macro is then used to implement a decoder (utf8->string!) that
> writes into an arbitrary range of an existing string.  Of course, it's
> not purely functional, but it avoids heap allocation when compiled with
> Guile.  On my Thinkpad X200, it can process around 10 megabytes per
> second.
>
> The state is represented as an exact integer between 0 and #xF48FBF
> inclusive, which are simply the bytes that have been seen so far in the
> current code sequence, in big-endian order, or 0 for the start state.
> For example, #xF48FBF represents the state where the bytes (F4 8F BF)
> have been read.  The state is always either 0 or a proper prefix of a
> valid UTF-8 byte sequence.

Awesome!  I think that’s something we should definitely add to Guile
proper.  We can use it in Guix before or after it’s included in Guile.

Thank you!

Ludo’.