bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Andrea Greselin]

[-- Attachment #1: Type: text/plain, Size: 5052 bytes --]

Steps to reproduce:

Launch `emacs -Q`, paste these four lines on a buffer

aaaaaaaaa'aaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa'aaaaaaaaaaaaaaaaaaa'aaaaaaaaaa
aaaaaaaaa’aaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa’aaaaaaaaaaaaaaaaaaa’aaaaaaaaaa

nnnnnnnnn'nnnnnnnnnnnnnnnnnnnnnnnnnnnn nnnn'nnnnnnnnnnnnnnnnnnn'nnnnnnnnnn
nnnnnnnnn’nnnnnnnnnnnnnnnnnnnnnnnnnnnn nnnn’nnnnnnnnnnnnnnnnnnn’nnnnnnnnnn

Search for the first one with char folding enabled. In my system Emacs
doesn’t match the second line. It matches after I hit ‘C-M-d’ to
delete the last character from the search query.

If you search with char folding enabled for the third line then it
does match the fourth, even though the third and the fourth lines are
just copies of the first two with the a’s replaced by n’s.

All the best,
Andrea


In GNU Emacs 27.2 (build 1, x86_64-redhat-linux-gnu, GTK+ Version 3.24.30,
cairo version 1.17.4)
 of 2021-08-07 built on buildvm-x86-29.iad2.fedoraproject.org
Windowing system distributor 'The X.Org Foundation', version 11.0.12101003
System Description: Fedora 34 (Workstation Edition)

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list... [2 times]

Configured using:
 'configure --build=x86_64-redhat-linux-gnu
 --host=x86_64-redhat-linux-gnu --program-prefix=
 --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr
 --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
 --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
 --libexecdir=/usr/libexec --localstatedir=/var
 --sharedstatedir=/var/lib --mandir=/usr/share/man
 --infodir=/usr/share/info --with-dbus --with-gif --with-jpeg --with-png
 --with-rsvg --with-tiff --with-xft --with-xpm --with-x-toolkit=gtk3
 --with-gpm=no --with-xwidgets --with-modules --with-harfbuzz
 --with-cairo --with-json build_alias=x86_64-redhat-linux-gnu
 host_alias=x86_64-redhat-linux-gnu CC=gcc 'CFLAGS=-DMAIL_USE_LOCKF -O2
 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches
 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
 -Wp,-D_GLIBCXX_ASSERTIONS
 -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong
 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
 -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-z,relro
 PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS XWIDGETS
LIBSYSTEMD JSON PDUMPER GMP

Important settings:
  value of $LANG: en_GB.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting
xwidget-internal cairo move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 45893 6606)
 (symbols 48 6007 1)
 (strings 32 15796 1378)
 (string-bytes 1 518486)
 (vectors 16 10238)
 (vector-slots 8 133527 10860)
 (floats 8 20 35)
 (intervals 56 271 0)
 (buffers 1000 12))

[-- Attachment #2: Type: text/html, Size: 5507 bytes --]

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Michael Heerdegen]

Andrea Greselin <greselin.andrea@gmail.com> writes:

> nnnnnnnnn'nnnnnnnnnnnnnnnnnnnnnnnnnnnn
> nnnn'nnnnnnnnnnnnnnnnnnn'nnnnnnnnnn
> nnnnnnnnn’nnnnnnnnnnnnnnnnnnnnnnnnnnnn
> nnnn’nnnnnnnnnnnnnnnnnnn’nnnnnnnnnn

A note for the next reader: the lines provided in the recipe are very
long, if you see more than four example lines, something like above, your
mail reader added newlines, and the example is probably broken.  There
are only two n...n lines.

Michael.

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Juri Linkov]

> Launch `emacs -Q`, paste these four lines on a buffer
>
> aaaaaaaaa'aaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa'aaaaaaaaaaaaaaaaaaa'aaaaaaaaaa
> aaaaaaaaa’aaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa’aaaaaaaaaaaaaaaaaaa’aaaaaaaaaa
>
> nnnnnnnnn'nnnnnnnnnnnnnnnnnnnnnnnnnnnn nnnn'nnnnnnnnnnnnnnnnnnn'nnnnnnnnnn
> nnnnnnnnn’nnnnnnnnnnnnnnnnnnnnnnnnnnnn nnnn’nnnnnnnnnnnnnnnnnnn’nnnnnnnnnn
>
> Search for the first one with char folding enabled. In my system Emacs
> doesn’t match the second line. It matches after I hit ‘C-M-d’ to
> delete the last character from the search query.
>
> If you search with char folding enabled for the third line then it
> does match the fourth, even though the third and the fourth lines are
> just copies of the first two with the a’s replaced by n’s.

Please note that when the search string is too long for char folding,
then it displays the message at the end of the Isearch prompt
before disabling char folding:

  "Too many words; switched to literal mode"

Char folding is regexp-based and converts the search string
into a very long regexp that matches all character variants.
But the regexp have limitations when matching runs out of
internal stack space.  This is described in the Info node:

  (info "(elisp) Regexp Problems")

For example, my stack is longer than yours, so the second line
of the described case matches with line length of 74 characters.
But the search still switches to literal mode on 87 characters.

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Andrea Greselin]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

> Please note that when the search string is too long for char folding,
> then it displays the message at the end of the Isearch prompt
> before disabling char folding:

Emacs doesn’t display any message, I’ve tried catching it with
‘(setq debug-on-message "Too")’ but got nothing.

> But the regexp have limitations when matching runs out of
> internal stack space.  This is described in the Info node:
>   (info "(elisp) Regexp Problems")

I’ve found the Info node on [1] but not in my Emacs nor on [2].
Could it be that both the node and the message are new features of
versions > 27.2?

1
https://github.com/emacs-mirror/emacs/blob/master/doc/lispref/elisp.texi#L1322
2
https://www.gnu.org/software/emacs/manual/html_node/elisp/Regular-Expressions.html

Thanks,
Andrea

[-- Attachment #2: Type: text/html, Size: 1095 bytes --]

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Juri Linkov]

>> Please note that when the search string is too long for char folding,
>> then it displays the message at the end of the Isearch prompt
>> before disabling char folding:
>
> Emacs doesn’t display any message, I’ve tried catching it with
> ‘(setq debug-on-message "Too")’ but got nothing.
>
>> But the regexp have limitations when matching runs out of
>> internal stack space.  This is described in the Info node:
>>   (info "(elisp) Regexp Problems")
>
> I’ve found the Info node on [1] but not in my Emacs nor on [2].
> Could it be that both the node and the message are new features of
> versions > 27.2?

Indeed, thanks for pointing to the version number 27.2.
That message was added in 28.1.  And also
the regexp stack size was increased too in 28.1.
And the new node exists also only in 28.1.

So unless someone has an idea how to implement the search
for regexps of arbitrary lengths, this report could be closed.

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Andrea Greselin]

[-- Attachment #1: Type: text/plain, Size: 572 bytes --]

I guess that removing equivalent characters would help make the string
that matches longer, right? In this case having the possibility of
building the sets of equivalent characters from zero would help. (For
me, just the equivalence of straight and curly quotes would cover 90%
of the use cases.) The options for customising these sets AFAIK are
‘char-fold-exclude’, but it’s not for building them from null, and
maybe the ‘char-fold-table’, but I’m not sure as to how to edit it, it
looks like it’s more for internal use than for the user to tweak.

[-- Attachment #2: Type: text/html, Size: 607 bytes --]

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Juri Linkov]

> I guess that removing equivalent characters would help make the string
> that matches longer, right? In this case having the possibility of
> building the sets of equivalent characters from zero would help. (For
> me, just the equivalence of straight and curly quotes would cover 90%
> of the use cases.) The options for customising these sets AFAIK are
> ‘char-fold-exclude’, but it’s not for building them from null, and
> maybe the ‘char-fold-table’, but I’m not sure as to how to edit it, it
> looks like it’s more for internal use than for the user to tweak.

Right, it's possible to customize char-fold-exclude to make the string longer.
As for how to do this without customization, internally maybe it would be
possible to add regexp back references to reuse parts of the previous regexps
constructed for the same letter.

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Andrea Greselin]

[-- Attachment #1: Type: text/plain, Size: 430 bytes --]

What about the option for resetting the default character
equivalence sets and rebuilding them from zero? A variable
that would allow me to write something like

  (setq char-fold-user-list '((?'  "‘" "’")
                              (?\" "“" "”")))

to have only quotes be considered equivalent. (When
‘char-fold-user-table’ is non-nil it overrides the
default table.)

Do you think it’s a good idea?

[-- Attachment #2: Type: text/html, Size: 543 bytes --]

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Juri Linkov]

> What about the option for resetting the default character
> equivalence sets and rebuilding them from zero? A variable
> that would allow me to write something like
>
>   (setq char-fold-user-list '((?'  "‘" "’")
>                               (?\" "“" "”")))
>
> to have only quotes be considered equivalent. (When
> ‘char-fold-user-table’ is non-nil it overrides the
> default table.)
>
> Do you think it’s a good idea?

We already have a variable ‘char-fold-include’ that adds equivalence sets,
so maybe it would be sufficient to add just a new boolean variable,
maybe with a name e.g. ‘char-fold-override’.  When customized to non-nil,
it will prevent from filling ‘char-fold-table’ with default equivalence sets.
Thus it will build equivalence sets from zero using ‘char-fold-include’ only,
like you need to do.

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Andrea Greselin]

[-- Attachment #1: Type: text/plain, Size: 243 bytes --]

Even better, yes. In addition to simplifying changing the default,
something like that would make it easy to create different levels of
folding and switch between them (by writing commands that add or
remove certain characters from the list).

[-- Attachment #2: Type: text/html, Size: 295 bytes --]

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]

> Even better, yes. In addition to simplifying changing the default,
> something like that would make it easy to create different levels of
> folding and switch between them (by writing commands that add or
> remove certain characters from the list).

Thanks for the suggestion,  This is implemented by the following patch:


[-- Attachment #2: char-fold-override.patch --]
[-- Type: text/x-diff, Size: 3108 bytes --]

diff --git a/lisp/char-fold.el b/lisp/char-fold.el
index e3ab7d5b64..12c9d2d4b1 100644
--- a/lisp/char-fold.el
+++ b/lisp/char-fold.el
@@ -26,6 +26,7 @@
 
 (eval-and-compile
   (put 'char-fold-table 'char-table-extra-slots 1)
+  (defconst char-fold--default-override nil)
   (defconst char-fold--default-include
     '((?\" "＂" "“" "”" "”" "„" "⹂" "〞" "‟" "‟" "❞" "❝" "❠" "“" "„" "〝" "〟" "🙷" "🙶" "🙸" "«" "»")
       (?' "❟" "❛" "❜" "‘" "’" "‚" "‛" "‚" "󠀢" "❮" "❯" "‹" "›")
@@ -40,7 +41,8 @@
       ))
   (defconst char-fold--default-symmetric nil)
   (defvar char-fold--previous
-    (list char-fold--default-include
+    (list char-fold--default-override
+          char-fold--default-include
           char-fold--default-exclude
           char-fold--default-symmetric)))
 
@@ -67,6 +69,8 @@
       ;; - A single char of the decomp might be allowed to match the
       ;;   character.
       ;; Some examples in the comments below.
+      (unless (or (bound-and-true-p char-fold-override)
+                  char-fold--default-override)
         (map-char-table
          (lambda (char decomp)
            (when (consp decomp)
@@ -135,7 +139,7 @@
                        (aset equiv (car simpler-decomp)
                              (cons (apply #'string decomp)
                                    (aref equiv (car simpler-decomp)))))))))))
-       table)
+         table))
 
       ;; Add some entries to default decomposition
       (dolist (it (or (bound-and-true-p char-fold-include)
@@ -232,7 +236,9 @@ char-fold-table
 \f
 (defun char-fold-update-table ()
   "Update char-fold-table only when one of the options changes its value."
-  (let ((new (list (or (bound-and-true-p char-fold-include)
+  (let ((new (list (or (bound-and-true-p char-fold-override)
+                       char-fold--default-override)
+                   (or (bound-and-true-p char-fold-include)
                        char-fold--default-include)
                    (or (bound-and-true-p char-fold-exclude)
                        char-fold--default-exclude)
@@ -242,6 +248,22 @@ char-fold-update-table
       (setq char-fold-table (char-fold--make-table)
             char-fold--previous new))))
 
+(defcustom char-fold-override char-fold--default-override
+  "Non-nil means to override all default folding characters.
+When nil, the equivalence table is populated with the default set
+of equivalent chars, and you can remove unneeded characters using
+`char-fold-exclude', and add own characters using `char-fold-include'.
+But when this variable is customized to non-nil, you start with
+an empty table where you can add only own characters
+using `char-fold-include'."
+  :type 'boolean
+  :initialize #'custom-initialize-default
+  :set (lambda (sym val)
+         (custom-set-default sym val)
+         (char-fold-update-table))
+  :group 'isearch
+  :version "29.1")
+
 (defcustom char-fold-include char-fold--default-include
   "Additional character foldings to include.
 Each entry is a list of a character and the strings that fold into it."

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Sun, 12 Dec 2021 19:19:14 +0200
> Cc: 52394@debbugs.gnu.org
> 
> > Even better, yes. In addition to simplifying changing the default,
> > something like that would make it easy to create different levels of
> > folding and switch between them (by writing commands that add or
> > remove certain characters from the list).
> 
> Thanks for the suggestion,  This is implemented by the following patch:

Doesn't this change the default behavior?

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Juri Linkov]

>> > Even better, yes. In addition to simplifying changing the default,
>> > something like that would make it easy to create different levels of
>> > folding and switch between them (by writing commands that add or
>> > remove certain characters from the list).
>> 
>> Thanks for the suggestion,  This is implemented by the following patch:
>
> Doesn't this change the default behavior?

No, no change in the default behavior.

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: greselin.andrea@gmail.com,  52394@debbugs.gnu.org
> Date: Sun, 12 Dec 2021 20:06:15 +0200
> 
> >> > Even better, yes. In addition to simplifying changing the default,
> >> > something like that would make it easy to create different levels of
> >> > folding and switch between them (by writing commands that add or
> >> > remove certain characters from the list).
> >> 
> >> Thanks for the suggestion,  This is implemented by the following patch:
> >
> > Doesn't this change the default behavior?
> 
> No, no change in the default behavior.

Then I guess the doc string of the defcustom is confusing?  Can you
make it more clear wrt what each value means in practice?

Thanks.

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Juri Linkov]

>> >> > Even better, yes. In addition to simplifying changing the default,
>> >> > something like that would make it easy to create different levels of
>> >> > folding and switch between them (by writing commands that add or
>> >> > remove certain characters from the list).
>> >> 
>> >> Thanks for the suggestion,  This is implemented by the following patch:
>> >
>> > Doesn't this change the default behavior?
>> 
>> No, no change in the default behavior.
>
> Then I guess the doc string of the defcustom is confusing?  Can you
> make it more clear wrt what each value means in practice?

Here is the doc string.  How do you propose to make it more clear?

    "Non-nil means to override all default folding characters.
  When nil, the equivalence table is populated with the default set
  of equivalent chars, and you can remove unneeded characters using
  `char-fold-exclude', and add own characters using `char-fold-include'.
  But when this variable is customized to non-nil, you start with
  an empty table where you can add only own characters
  using `char-fold-include'."

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: greselin.andrea@gmail.com,  52394@debbugs.gnu.org
> Date: Sun, 12 Dec 2021 20:30:10 +0200
> 
> >> > Doesn't this change the default behavior?
> >> 
> >> No, no change in the default behavior.
> >
> > Then I guess the doc string of the defcustom is confusing?  Can you
> > make it more clear wrt what each value means in practice?
> 
> Here is the doc string.  How do you propose to make it more clear?
> 
>     "Non-nil means to override all default folding characters.
>   When nil, the equivalence table is populated with the default set
>   of equivalent chars, and you can remove unneeded characters using
>   `char-fold-exclude', and add own characters using `char-fold-include'.
>   But when this variable is customized to non-nil, you start with
>   an empty table where you can add only own characters
>   using `char-fold-include'."

My reading of this is that the feature by default will not work at all:
since the default value is not nil, the doc string says that I "start
with an empty table", so no folding will happen.  Is that what you
meant?  And if so, how isn't this a change in behavior?  In Emacs 27
the folding works by default and finds equivalent characters without
any user intervention.

What am I missing?

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Juri Linkov]

>> Here is the doc string.  How do you propose to make it more clear?
>>
>>     "Non-nil means to override all default folding characters.
>>   When nil, the equivalence table is populated with the default set
>>   of equivalent chars, and you can remove unneeded characters using
>>   `char-fold-exclude', and add own characters using `char-fold-include'.
>>   But when this variable is customized to non-nil, you start with
>>   an empty table where you can add only own characters
>>   using `char-fold-include'."
>
> My reading of this is that the feature by default will not work at all:
> since the default value is not nil, the doc string says that I "start
> with an empty table", so no folding will happen.  Is that what you
> meant?  And if so, how isn't this a change in behavior?  In Emacs 27
> the folding works by default and finds equivalent characters without
> any user intervention.
>
> What am I missing?

But doesn't this line

  "when this variable is customized to non-nil"

imply that the default is nil?  Or maybe the nil default value
should be mentioned in the doc string?

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: greselin.andrea@gmail.com,  52394@debbugs.gnu.org
> Date: Sun, 12 Dec 2021 20:54:51 +0200
> 
> But doesn't this line
> 
>   "when this variable is customized to non-nil"
> 
> imply that the default is nil?

No, it doesn't.

> Or maybe the nil default value should be mentioned in the doc
> string?

That would help.  It would also help to make the default value nil,
literally, not via some defconst.  Why do we need to obfuscate the
source code?

bug#52394: 27.2; Isearch with char folding enabled doesn’t match equivalent strings

[Juri Linkov]

close 52394 29.0.50
thanks

>> Or maybe the nil default value should be mentioned in the doc string?
>
> That would help.

So now pushed to master with the doc string mentioning the default value.

> It would also help to make the default value nil, literally, not via
> some defconst.  Why do we need to obfuscate the source code?

This is necessary to not duplicate the default value in many places.
This was discussed in bug#35689.