From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#3687: 23.1.50; inconsistency in multibyte eight-bit regexps [PATCH] Date: Fri, 28 Jun 2019 17:40:47 +0300 Message-ID: <83v9wphixc.fsf@gnu.org> References: <831rzdj1z9.fsf@gnu.org> <6138515E-3202-437D-8341-7A8856AD0AE9@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="102159"; mail-complaints-to="usenet@blaine.gmane.org" Cc: monnier@iro.umontreal.ca, 3687@debbugs.gnu.org To: Mattias =?UTF-8?Q?Engdeg=C3=A5rd?= Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Fri Jun 28 17:55:47 2019 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hgtE2-000QTQ-Et for geb-bug-gnu-emacs@m.gmane.org; Fri, 28 Jun 2019 17:55:46 +0200 Original-Received: from localhost ([::1]:33602 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hgtE1-0004Z0-DP for geb-bug-gnu-emacs@m.gmane.org; Fri, 28 Jun 2019 11:55:45 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:44606) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hgs4k-0006EB-HL for bug-gnu-emacs@gnu.org; Fri, 28 Jun 2019 10:42:08 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hgs4j-00007z-B1 for bug-gnu-emacs@gnu.org; Fri, 28 Jun 2019 10:42:06 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:57444) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hgs4f-0008Us-VB for bug-gnu-emacs@gnu.org; Fri, 28 Jun 2019 10:42:03 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hgs4f-0005Jp-QW for bug-gnu-emacs@gnu.org; Fri, 28 Jun 2019 10:42:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Fri, 28 Jun 2019 14:42:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 3687 X-GNU-PR-Package: emacs Original-Received: via spool by 3687-submit@debbugs.gnu.org id=B3687.156173289820401 (code B ref 3687); Fri, 28 Jun 2019 14:42:01 +0000 Original-Received: (at 3687) by debbugs.gnu.org; 28 Jun 2019 14:41:38 +0000 Original-Received: from localhost ([127.0.0.1]:42755 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hgs4F-0005Ix-PQ for submit@debbugs.gnu.org; Fri, 28 Jun 2019 10:41:37 -0400 Original-Received: from eggs.gnu.org ([209.51.188.92]:38607) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hgs4D-0005If-MW for 3687@debbugs.gnu.org; Fri, 28 Jun 2019 10:41:34 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]:56531) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hgs47-00081L-5p; Fri, 28 Jun 2019 10:41:27 -0400 Original-Received: from [176.228.60.248] (port=1645 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hgs44-0000s7-FO; Fri, 28 Jun 2019 10:41:26 -0400 In-reply-to: <6138515E-3202-437D-8341-7A8856AD0AE9@acm.org> (message from Mattias =?UTF-8?Q?Engdeg=C3=A5rd?= on Fri, 28 Jun 2019 16:05:07 +0200) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:161739 Archived-At: > From: Mattias Engdegård > Date: Fri, 28 Jun 2019 16:05:07 +0200 > Cc: mituharu@math.s.chiba-u.ac.jp, monnier@iro.umontreal.ca, > 3687@debbugs.gnu.org > > > 1. What do you mean by "raw bytes"? Is #xab a raw byte or a Unicode > > point U+00AB? IOW, how do we distinguish, in a regexp, between a > > raw byte and a character whose Unicode codepoint is that byte's > > value? And how does one go about concocting a regexp that matches > > raw bytes in a unibyte or multibyte buffer or string? > > Sorry, I should have been more clear. The terminology in the manual is a bit muddled; in this case I mean the characters (or whatever you prefer calling them) obtained with hex or octal escapes in the range 128-255, such as "\xff" or "\377", regardless of the string's type (unibyte or multibyte). > > Unicode characters in the range 128-255 can be generated using the \u00HH or \U000000HH notations, or by just including them literally. They are distinct from raw bytes. > > To match raw bytes, just write them. They are not special in regexp syntax and need no escaping. So this means \240 is no longer the same as NBSP and \300 is no longer the same as À? But \176 is still the same as ~? Doesn't this open a clear path for another bug report about inconsistencies in regexps? Also, which ways do you propose for specifying raw bytes? Only hex escapes? octal escapes as well? something else? > > 2. What is meant by "ranges from ASCII to raw bytes"? Which > > characters are included in such ranges? > > Ranges such as [A-\xb1] or [\000-\377], where the first endpoint is an ASCII character and the last endpoint is a raw byte as defined above. These should include all characters from the first endpoint up to and including ASCII 127, and all raw bytes from 128 to the last endpoint. This makes intuitive sense for unibyte strings where such an interval is contiguous in the underlying representation; extending them to multibyte is obvious. So you are saying that we will consider the raw bytes as if they followed ASCII characters in the lexicographical order? But non-ASCII characters whose codepoints start at 0x80? where are they in this order? > > 3. If ranges from non-ASCII characters to raw bytes make no sense, > > how would one go about specifying a range that includes all the > > characters and raw bytes supported by Emacs? > > "[\x00-\U0010ffff\x80-\xff]" This looks confusing, because to a naïve reader the first part already includes the second one. My point is that I'm afraid this proposal will replace one set of inconsistencies by another. I think the only way to avoid inconsistencies is to consider the likes of \177 mean different things depending on whether the text being matched is unibyte or multibyte. In particular, raw bytes in multibyte regexps should (if they are needed) be spelled out as #x3fff00, \17777400, etc. This, of course, has the disadvantage that one needs to know which text is being matched before one concocts the regexp.