From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Newsgroups: gmane.emacs.help Subject: Re: Text copied from *grep* buffer has NUL (0x00) characters Date: Mon, 10 May 2021 09:10:30 +0200 Message-ID: <20210510071030.GB31080@tuxteam.de> References: <83bl9k8buk.fsf@gnu.org> <3e892a2e-1d04-7712-d129-e4f59382457b@yahoo.de> <83a6p37n15.fsf@gnu.org> <634e880a-b43b-8c59-eb4c-b0c07813bb12@yahoo.de> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="CUfgB8w4ZwR/yMy5" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="29683"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mutt/1.5.21 (2010-09-15) Cc: help-gnu-emacs@gnu.org, Stefan Monnier To: "R. Diez" Original-X-From: help-gnu-emacs-bounces+geh-help-gnu-emacs=m.gmane-mx.org@gnu.org Mon May 10 09:12:16 2021 Return-path: Envelope-to: geh-help-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lg05P-0007ab-Ks for geh-help-gnu-emacs@m.gmane-mx.org; Mon, 10 May 2021 09:12:15 +0200 Original-Received: from localhost ([::1]:48302 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lg05N-0007ox-Vy for geh-help-gnu-emacs@m.gmane-mx.org; Mon, 10 May 2021 03:12:14 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:35634) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lg03z-0007om-8y for help-gnu-emacs@gnu.org; Mon, 10 May 2021 03:10:47 -0400 Original-Received: from mail.tuxteam.de ([5.199.139.25]:36071) by eggs.gnu.org with esmtps (TLS1.2:DHE_RSA_AES_128_CBC_SHA1:128) (Exim 4.90_1) (envelope-from ) id 1lg03v-0005fl-Fx; Mon, 10 May 2021 03:10:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=tuxteam.de; s=mail; h=From:In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:Date; bh=hw8WI7AAOUwxN4NyGmfURj7URsUY1bT/N1RGd6MHF08=; b=PFXbI97p21xpA82cSlgkfU9+EfEjkyjB0ZgeChlo6vDl4EwZy/dBO7cSMI7ge2l5DkR4xBxyXcCBDJX+dPRVUQOql1jDA/F/T/MrN6HaTsgC8Mq6NAV8d2crNcyun6NUuZ5rU0ICQCP1WjicFL+ZowzGL4wEMTAvagTvyfM1lm/JzCBB6ojzvaj8dlKu/l5K2lpUMzYxCnc7H5bk7Dc9ESQg8/ZBc8ggravYGqaf+CgsULB3fcxFidIiM92/516SRfPVu7wbhEtCHJ8Ry9jxyMKbTV7J6OfvQohXQok+rfB7ZlzwD9KWsoIBjqzurjFR1+hKOeVv1aKUEntN8hUBPg==; Original-Received: from tomas by mail.tuxteam.de with local (Exim 4.80) (envelope-from ) id 1lg03i-0008Hr-Gh; Mon, 10 May 2021 09:10:30 +0200 Content-Disposition: inline In-Reply-To: <634e880a-b43b-8c59-eb4c-b0c07813bb12@yahoo.de> Received-SPF: pass client-ip=5.199.139.25; envelope-from=tomas@tuxteam.de; helo=mail.tuxteam.de X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: help-gnu-emacs@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Users list for the GNU Emacs text editor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-gnu-emacs-bounces+geh-help-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "help-gnu-emacs" Xref: news.gmane.io gmane.emacs.help:129650 Archived-At: --CUfgB8w4ZwR/yMy5 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, May 09, 2021 at 11:13:36PM +0200, R. Diez wrote: >=20 > EZ> That's not the same. the warning you saw is triggered by a failure to > EZ> convert to the external encoding, so it consumes no extra CPU cycles. >=20 > But it could be, from my (admittedly naive) point of view: >=20 > (convert-to-external-encoding but-with-some-extra-flag-to-warn-about-NUL= -chars) >=20 >=20 > EZ> Null bytes will not fail anything, so you should test for them > EZ> explicitly (and in some encodings, like UTF-16, they are necessary and > EZ> cannot be avoided). >=20 > I didn't know that about UTF-16, but I could not find any information abo= ut it either. Why is a NUL char necessary in UTF-16 and not UTF-8? UTF-16 [1] encodes characters using 16 bit "packets" called "code units". Like UTF-8, whenever one unit isn't sufficient, you use more. The bit pattern tells you whether there are more to come. In the case of UTF-16 "more" is at most two. For the "small" code points, 8 of those 16 bit are zero. Which one depends on endiannes, but this or that way, you end up with a lot of zero bytes in your text. That's how UTF-16BE (big endian) looks like: tomas@trotzki:~$ echo "hello, world" | iconv -f utf-8 -t UTF-16BE | hexdu= mp -C 00000000 00 68 00 65 00 6c 00 6c 00 6f 00 2c 00 20 00 77 |.h.e.l.l.o.,= =2E .w| 00000010 00 6f 00 72 00 6c 00 64 00 0a |.o.r.l.d..| 0000001a =2E.. so a bit like Swiss cheese. UTF-16 needs a BOM (byte order mark) to disambiguate on endianness. UTF-8 doesn't (is a byte stream), although Microsoftey-applications tend to sneak one in, just to annoy the rest of us. Or something. Cheers - t --CUfgB8w4ZwR/yMy5 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iEYEARECAAYFAmCY3GYACgkQBcgs9XrR2kYPkACeIx1tSc5xagZXLOU+3dhuiCE8 Fe8AniJlUl3YDq297M+YL/NdvxECV8FW =ysWq -----END PGP SIGNATURE----- --CUfgB8w4ZwR/yMy5--