unofficial mirror of help-gnu-emacs@gnu.org
 help / color / mirror / Atom feed
From: <tomas@tuxteam.de>
To: "R. Diez" <rdiezmail-emacs@yahoo.de>
Cc: help-gnu-emacs@gnu.org, Stefan Monnier <monnier@iro.umontreal.ca>
Subject: Re: Text copied from *grep* buffer has NUL (0x00) characters
Date: Mon, 10 May 2021 09:10:30 +0200	[thread overview]
Message-ID: <20210510071030.GB31080@tuxteam.de> (raw)
In-Reply-To: <634e880a-b43b-8c59-eb4c-b0c07813bb12@yahoo.de>

[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]

On Sun, May 09, 2021 at 11:13:36PM +0200, R. Diez wrote:
> 
> EZ> That's not the same.  the warning you saw is triggered by a failure to
> EZ> convert to the external encoding, so it consumes no extra CPU cycles.
> 
> But it could be, from my (admittedly naive) point of view:
> 
> (convert-to-external-encoding  but-with-some-extra-flag-to-warn-about-NUL-chars)
> 
> 
> EZ> Null bytes will not fail anything, so you should test for them
> EZ> explicitly (and in some encodings, like UTF-16, they are necessary and
> EZ> cannot be avoided).
> 
> I didn't know that about UTF-16, but I could not find any information about it either. Why is a NUL char necessary in UTF-16 and not UTF-8?

UTF-16 [1] encodes characters using 16 bit "packets" called "code
units". Like UTF-8, whenever one unit isn't sufficient, you use
more. The bit pattern tells you whether there are more to come.

In the case of UTF-16 "more" is at most two.

For the "small" code points, 8 of those 16 bit are zero. Which one
depends on endiannes, but this or that way, you end up with a lot
of zero bytes in your text. That's how UTF-16BE (big endian) looks
like:

  tomas@trotzki:~$ echo "hello, world" | iconv -f utf-8 -t UTF-16BE | hexdump -C
  00000000  00 68 00 65 00 6c 00 6c  00 6f 00 2c 00 20 00 77  |.h.e.l.l.o.,. .w|
  00000010  00 6f 00 72 00 6c 00 64  00 0a                    |.o.r.l.d..|
  0000001a

... so a bit like Swiss cheese.

UTF-16 needs a BOM (byte order mark) to disambiguate on endianness.
UTF-8 doesn't (is a byte stream), although Microsoftey-applications
tend to sneak one in, just to annoy the rest of us.

Or something.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

  reply	other threads:[~2021-05-10  7:10 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <ddcebd62-1eb7-d24c-9a85-dadeb62c6ea2.ref@yahoo.de>
2021-05-09  9:19 ` Text copied from *grep* buffer has NUL (0x00) characters R. Diez
2021-05-09 10:01   ` Eli Zaretskii
2021-05-09 18:47     ` R. Diez
2021-05-09 18:57       ` Eli Zaretskii
2021-05-09 21:13         ` R. Diez
2021-05-10  7:10           ` tomas [this message]
2021-05-09 19:47       ` Stefan Monnier
2021-05-09 13:57   ` Stefan Monnier via Users list for the GNU Emacs text editor

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.gnu.org/software/emacs/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210510071030.GB31080@tuxteam.de \
    --to=tomas@tuxteam.de \
    --cc=help-gnu-emacs@gnu.org \
    --cc=monnier@iro.umontreal.ca \
    --cc=rdiezmail-emacs@yahoo.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).