From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Subject: bug#35350: Some compile output still leaks through with --verbosity=1 Date: Sat, 27 Apr 2019 18:36:34 +0200 Message-ID: <874l6jh0bx.fsf@gnu.org> References: <87mukkfd2j.fsf@netris.org> <87r29v2jz2.fsf@gnu.org> <87ftq9silk.fsf@netris.org> <87imv5jai5.fsf@gnu.org> <87k1fgh9c0.fsf@netris.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([209.51.188.92]:33736) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hKQKT-00017g-2P for bug-guix@gnu.org; Sat, 27 Apr 2019 12:37:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hKQK0-0005Ku-7C for bug-guix@gnu.org; Sat, 27 Apr 2019 12:37:06 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:50552) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hKQJy-0005HS-Bz for bug-guix@gnu.org; Sat, 27 Apr 2019 12:37:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hKQJy-0007je-6k for bug-guix@gnu.org; Sat, 27 Apr 2019 12:37:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87k1fgh9c0.fsf@netris.org> (Mark H. Weaver's message of "Fri, 26 Apr 2019 15:09:24 -0400") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Mark H Weaver Cc: 35350@debbugs.gnu.org Hi Mark, Mark H Weaver skribis: > Ludovic Court=C3=A8s writes: > >> The third read(2) call here ends on a partial UTF-8 sequence for LEFT >> SINGLE QUOTATION MARK (we get the first two bytes of a three byte >> sequence.) >> >> What happens is that =E2=80=98process-stderr=E2=80=99 in (guix store) ge= ts that byte >> string from the daemon, passes it through =E2=80=98read-maybe-utf8-strin= g=E2=80=99, >> which replaces the last two bytes with REPLACEMENT CHARACTER, which is >> itself a 3-byte sequence. > > It seems to me that what's needed here is to save the UTF-8 decoder > state between calls to 'process-stderr'. So there are two things. To fix the issue you reported (build output that goes through), I think we must simply turn off UTF-8 decoding from =E2=80=98process-stderr=E2=80=99 and leave that entirely to =E2=80=98build-= event-output-port=E2=80=99. However, =E2=80=98build-event-output-port=E2=80=99 would still fail to prop= erly decode split UTF-8 sequences, and for that we=E2=80=99d need to preserve decoder s= tate as you describe. > Coincidentally, I also needed something like this a week ago, when I > tried implementing R6RS custom textual input/output ports on top of > R6RS custom binary input/output ports. > > To meet these needs, I've implemented a fairly efficient, purely > functional UTF-8 decoder in Scheme that accepts a decoder state and an > arbitrary range from a bytevector, and returns a new decoder state. > There's a macro that allows arbitrary actions to be performed when a > code point (or maximal subpart in the case of errors) is found. > > This macro is then used to implement a decoder (utf8->string!) that > writes into an arbitrary range of an existing string. Of course, it's > not purely functional, but it avoids heap allocation when compiled with > Guile. On my Thinkpad X200, it can process around 10 megabytes per > second. > > The state is represented as an exact integer between 0 and #xF48FBF > inclusive, which are simply the bytes that have been seen so far in the > current code sequence, in big-endian order, or 0 for the start state. > For example, #xF48FBF represents the state where the bytes (F4 8F BF) > have been read. The state is always either 0 or a proper prefix of a > valid UTF-8 byte sequence. Awesome! I think that=E2=80=99s something we should definitely add to Guile proper. We can use it in Guix before or after it=E2=80=99s included in Gui= le. Thank you! Ludo=E2=80=99.