bug#21278: 25.0.50; char-fold search is a screw, and I can't see how to disable

bug#21278: 25.0.50; char-fold search is a screw, and I can't see how to disable

[Richard Stallman]

It is a screw that char-fold isearch treats space and NL as equivalent.
They do not look alike.  Please disconnect them.




In GNU Emacs 25.0.50.1 (i686-pc-linux-gnu, GTK+ Version 2.24.23)
 of 2015-08-12 on freetop
Repository revision: 79a169684dfad2c0bbb9fdbae539c1f30d9f0ac3
System Description:	Trisquel GNU/Linux 7.0, Belenos

Configured using:
 `configure 'CFLAGS=-g -O0''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GCONF GSETTINGS NOTIFY
LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK2 X11

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Help

Minor modes in effect:
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  gpm-mouse-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Recent messages:
Mark set
Mark saved where search started [7 times]
Quit
Type C-x 1 to delete the help window, C-M-v to scroll help.
Mark saved where search started [2 times]
Auto-saving...
Mark saved where search started
Making completion list...
Quit
C-x 4 C-g is undefined

Load-path shadows:
None found.

Features:
(eieio-opt speedbar sb-image ezimage dframe find-func novice vc
vc-dispatcher vc-git bug-reference add-log pp shadow emacsbug
parse-time vc-cvs etags thingatpt xref project cal-move cal-menu
calendar cal-loaddefs wid-edit battery sgml-mode epa-file rmailkwd
ispell epa-mail epa derived epg diff-mode easy-mmode diff dired-aux
sh-script smie executable quail rmailsum conf-mode rmailout shr-color
color url-util url-parse auth-source cl-seq eieio byte-opt bytecomp
byte-compile cl-extra seq cconv eieio-core cl-macs gv gnus-util
password-cache url-vars shr dom subr-x browse-url misearch
multi-isearch dabbrev shell pcomplete grep compile comint ansi-color
ring mailalias qp rmailmm message sendmail format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mailabbrev gmm-utils mailheader
mail-parse rfc2231 rmail rfc2047 rfc2045 ietf-drums mm-util help-fns
help-mode cl-loaddefs pcase cl-lib mail-prsvr mail-utils dired t-mouse
view time-date paren cus-start cus-load advice finder-inf package
easymenu epg-config mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese charscript case-table epa-hook jka-cmpr-hook help simple
abbrev minibuffer cl-preloaded nadvice loaddefs button faces cus-face
macroexp files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote
dbusbind gfilenotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 8 387608 101135)
 (symbols 24 28231 3)
 (miscs 20 5511 4232)
 (strings 16 51815 17926)
 (string-bytes 1 1899171)
 (vectors 8 31455)
 (vector-slots 4 1538023 39042)
 (floats 8 453 880)
 (intervals 28 70732 2674)
 (buffers 520 156)
 (heap 1024 23258 2927))
[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

bug#21278: 25.0.50; char-fold search is a screw, and I can't see how to disable

[Artur Malabarba]

2015-08-17 6:04 GMT+01:00 Richard Stallman <rms@gnu.org>:
>
> It is a screw that char-fold isearch treats space and NL as equivalent.
> They do not look alike.  Please disconnect them.

As of 2 weeks ago, it doesn't. You can verify that by running
(character-fold-to-regexp " ")
The regexp it returns doesn't contain newline chars.

isearch-lax-whitespace does treat space as NL, so maybe that's what
you're seeing.
I believe there's a bug in `isearch-search-fun-default' where even if
you disable `isearch-lax-whitespace', isearch sometimes uses
lax-whitespace anyway.

I would have fixed this myself, but I don't quite understand what that
code is trying to do and last time I asked for help nobody answered.

bug#21278: 25.0.50; char-fold search is a screw, and I can't see how to disable

[Juri Linkov]

>> It is a screw that char-fold isearch treats space and NL as equivalent.
>> They do not look alike.  Please disconnect them.
>
> As of 2 weeks ago, it doesn't. You can verify that by running
> (character-fold-to-regexp " ")
> The regexp it returns doesn't contain newline chars.
>
> isearch-lax-whitespace does treat space as NL, so maybe that's what
> you're seeing.
> I believe there's a bug in `isearch-search-fun-default' where even if
> you disable `isearch-lax-whitespace', isearch sometimes uses
> lax-whitespace anyway.
>
> I would have fixed this myself, but I don't quite understand what that
> code is trying to do and last time I asked for help nobody answered.

Sorry for missing your questions.  ‘isearch-search-fun-default’ detects
the situation when the user is in the process of typing a word,
so it should not match incomplete words, otherwise the word search
will immediately fail and not allow the user to complete the word.
I guess this code is not necessary for char-fold search, but to
double-check I'm looking at this problem now.

bug#21278: 25.0.50; char-fold search is a screw, and I can't see how to disable

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 314 bytes --]

> I guess this code is not necessary for char-fold search, but to
> double-check I'm looking at this problem now.

Indeed, it's not necessary. The problem is that I was under the impression
that that lax variable referred to lax-whitespace, so I coded some things
on that assumption.

I've fixed that mistake now.

[-- Attachment #2: Type: text/html, Size: 391 bytes --]

bug#21278: char-fold isearch: space and newline

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> It is a screw that char-fold isearch treats space and NL as equivalent.
> They do not look alike.  Please disconnect them.

Artur Malabarba <bruce.connor.am@gmail.com> writes:

>> I guess this code is not necessary for char-fold search, but to
>> double-check I'm looking at this problem now.
>
> Indeed, it's not necessary. The problem is that I was under the impression
> that that lax variable referred to lax-whitespace, so I coded some things on
> that assumption. 
>
> I've fixed that mistake now.

As far as I can tell, char-fold isearch doesn't conflate space and
newline, so I guess that Artur's changes fixed this issue?  But the bug
report was left open, so I'm closing it now.  If I misunderstood, please
respond to the debbugs address and we'll reopen.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21278: char-fold isearch: space and newline still broken

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I am running master from May 11 and the failure still happens.

I put this text from your message in my buffer:

    As far as I can tell, char-fold isearch doesn't conflate space and
    newline, so I guess that Artur's changes fixed this issue?  But the bug
    report was left open, so I'm closing it now.  If I misunderstood, please
    respond to the debbugs address and we'll reopen.

I put point on "As far", a few lines above here, and type C-s and SPC.
It finds the "and" at the end of the line together with the newline
and spaces.  This buffer is in Mail mode.

However, when I insert the same text in *scratch*, the bug does not
happen there.

I don't know when I reported that bug, and I can't find the message
I originally sent without knowing the approximate date.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#21278: char-fold isearch: space and newline still broken

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Sun, 05 Sep 2021 23:09:36 -0400
> Cc: 21278@debbugs.gnu.org, bruce.connor.am@gmail.com
> 
> I am running master from May 11 and the failure still happens.
> 
> I put this text from your message in my buffer:
> 
>     As far as I can tell, char-fold isearch doesn't conflate space and
>     newline, so I guess that Artur's changes fixed this issue?  But the bug
>     report was left open, so I'm closing it now.  If I misunderstood, please
>     respond to the debbugs address and we'll reopen.
> 
> I put point on "As far", a few lines above here, and type C-s and SPC.
> It finds the "and" at the end of the line together with the newline
> and spaces.  This buffer is in Mail mode.
> 
> However, when I insert the same text in *scratch*, the bug does not
> happen there.

I think that's because in Mail mode we have:

  (char-syntax ?\n) => 32 ; SPC

whereas in *scratch*:

  (char-syntax ?\n) => 62 ; >

And the regexp in search-whitespace-regexp is "\\s-+".

bug#21278: char-fold isearch: space and newline still broken

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> I am running master from May 11 and the failure still happens.

OK; reopening.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21278: char-fold isearch: space and newline still broken

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think that's because in Mail mode we have:

  >   (char-syntax ?\n) => 32 ; SPC

  > whereas in *scratch*:

  >   (char-syntax ?\n) => 62 ; >

  > And the regexp in search-whitespace-regexp is "\\s-+".

That seems plausible.

I still say it is a bug.  Incremental search should not equate
newline with spaces, at least not by default.  It is right to
treat newline as whitespace in other commands, but not in search.

What would be a good fix?  Change the syntax of newline?
What would be a good choice?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#21278: char-fold isearch: space and newline still broken

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: larsi@gnus.org, 21278@debbugs.gnu.org, bruce.connor.am@gmail.com
> Date: Mon, 06 Sep 2021 23:17:45 -0400
> 
>   > I think that's because in Mail mode we have:
> 
>   >   (char-syntax ?\n) => 32 ; SPC
> 
>   > whereas in *scratch*:
> 
>   >   (char-syntax ?\n) => 62 ; >
> 
>   > And the regexp in search-whitespace-regexp is "\\s-+".
> 
> That seems plausible.
> 
> I still say it is a bug.  Incremental search should not equate
> newline with spaces, at least not by default.  It is right to
> treat newline as whitespace in other commands, but not in search.
> 
> What would be a good fix?  Change the syntax of newline?
> What would be a good choice?

No, I think we should leave the syntax of the newline alone, I bet too
much depends on it.

I think we should change the default value of search-whitespace-regexp
instead.  Building such mode-independent features on syntax properties
is IME asking for trouble.

Stefan, WDYT?

bug#21278: char-fold isearch: space and newline still broken

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Building such mode-independent features on syntax properties
> is IME asking for trouble.
> Stefan, WDYT?

Agreed.  I'd use something like [ \t] or [ \t\n] or even [\0-\s].


        Stefan

bug#21278: char-fold isearch: space and newline still broken

[Juri Linkov]

>> Building such mode-independent features on syntax properties
>> is IME asking for trouble.
>> Stefan, WDYT?
>
> Agreed.  I'd use something like [ \t] or [ \t\n] or even [\0-\s].

Seconded.  Everyone who wants to match a newline,
still needs to customize search-whitespace-regexp
to include \n, e.g. "\\(?:\\s-\\|\n\\)+", so an explicit
list of characters would be more deterministic.

bug#21278: char-fold isearch: space and newline still broken

[Lars Ingebrigtsen]

Juri Linkov <juri@linkov.net> writes:

>>> Building such mode-independent features on syntax properties
>>> is IME asking for trouble.
>>> Stefan, WDYT?
>>
>> Agreed.  I'd use something like [ \t] or [ \t\n] or even [\0-\s].
>
> Seconded.  Everyone who wants to match a newline,
> still needs to customize search-whitespace-regexp
> to include \n, e.g. "\\(?:\\s-\\|\n\\)+", so an explicit
> list of characters would be more deterministic.

It seems like everybody agrees that the old value was confusing, so I've
now changed it to [ \t] in Emacs 28.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21278: char-fold isearch: space and newline still broken

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Stefan Monnier
>  <monnier@iro.umontreal.ca>,  21278@debbugs.gnu.org,  rms@gnu.org,
>   bruce.connor.am@gmail.com
> Date: Fri, 10 Sep 2021 12:28:20 +0200
> 
> > Seconded.  Everyone who wants to match a newline,
> > still needs to customize search-whitespace-regexp
> > to include \n, e.g. "\\(?:\\s-\\|\n\\)+", so an explicit
> > list of characters would be more deterministic.
> 
> It seems like everybody agrees that the old value was confusing, so I've
> now changed it to [ \t] in Emacs 28.

Shouldn't it be [ \t]+ instead?

bug#21278: char-fold isearch: space and newline still broken

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> It seems like everybody agrees that the old value was confusing, so I've
>> now changed it to [ \t] in Emacs 28.
>
> Shouldn't it be [ \t]+ instead?

Oops.  Looks like I'm going for the "most errors in a trivial one-line
patch" record here...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21278: char-fold isearch: space and newline still broken

[Augusto Stoffel]

On Fri, 10 Sep 2021 at 12:57, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> It seems like everybody agrees that the old value was confusing, so I've
>>> now changed it to [ \t] in Emacs 28.
>>
>> Shouldn't it be [ \t]+ instead?
>
> Oops.  Looks like I'm going for the "most errors in a trivial one-line
> patch" record here...

For the record, search-whitespace-regexp could have more useful values
suggested in customize.  I can think of:

- not too lax: "[ \\t]+"
- match across lines: "[ \\t\\n]+"
- eat punctuation: "\\W+"
- fuzzy: ".*?"

bug#21278: char-fold isearch: space and newline still broken

[Kévin Le Gouguec]

Augusto Stoffel <arstoffel@gmail.com> writes:

> On Fri, 10 Sep 2021 at 12:57, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> It seems like everybody agrees that the old value was confusing, so I've
>>>> now changed it to [ \t] in Emacs 28.
>>>
>>> Shouldn't it be [ \t]+ instead?
>>
>> Oops.  Looks like I'm going for the "most errors in a trivial one-line
>> patch" record here...
>
> For the record, search-whitespace-regexp could have more useful values
> suggested in customize.  I can think of:
>
> - not too lax: "[ \\t]+"
> - match across lines: "[ \\t\\n]+"
> - eat punctuation: "\\W+"
> - fuzzy: ".*?"

Seconded.  I expect this to be a common customization: when searching
for compound expressions ("regular expression", "environment variable"…)
through prose filled using hard newlines (manpages, info nodes,
docstrings…), I have always found it useful for isearch to treat
newlines as regular whitespace.

If isearch no longer behaves like that by default, I'm sure users would
at least appreciate Customize suggesting some ready-made values.

bug#21278: char-fold isearch: space and newline still broken

[Lars Ingebrigtsen]

Augusto Stoffel <arstoffel@gmail.com> writes:

> For the record, search-whitespace-regexp could have more useful values
> suggested in customize.  I can think of:
>
> - not too lax: "[ \\t]+"
> - match across lines: "[ \\t\\n]+"
> - eat punctuation: "\\W+"
> - fuzzy: ".*?"

I'm thing that trying to assign names to "[ \t]+" as "not too lax" will
just be more confusing than saying "tabs and spaces", and the same goes
for the other alternatives, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21278: char-fold isearch: space and newline still broken

[Augusto Stoffel]

On Sat, 11 Sep 2021 at 14:12, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Augusto Stoffel <arstoffel@gmail.com> writes:
>
>> For the record, search-whitespace-regexp could have more useful values
>> suggested in customize.  I can think of:
>>
>> - not too lax: "[ \\t]+"
>> - match across lines: "[ \\t\\n]+"
>> - eat punctuation: "\\W+"
>> - fuzzy: ".*?"
>
> I'm thing that trying to assign names to "[ \t]+" as "not too lax" will
> just be more confusing than saying "tabs and spaces", and the same goes
> for the other alternatives, too.

Right, more serious names could be:

1. Tabs and spaces
2. Tabs, spaces and line breaks
3. Whitespace and punctuation
4. Any sequence of characters (fuzzy)

Description 3. might not be 100% accurate, but is probably more
meaningful than "Non-word characters".

bug#21278: char-fold isearch: space and newline still broken

[Lars Ingebrigtsen]

Augusto Stoffel <arstoffel@gmail.com> writes:

>>> - not too lax: "[ \\t]+"
>>> - match across lines: "[ \\t\\n]+"
>>> - eat punctuation: "\\W+"
>>> - fuzzy: ".*?"
>>
>> I'm thing that trying to assign names to "[ \t]+" as "not too lax" will
>> just be more confusing than saying "tabs and spaces", and the same goes
>> for the other alternatives, too.
>
> Right, more serious names could be:
>
> 1. Tabs and spaces
> 2. Tabs, spaces and line breaks

OK, added.

> 3. Whitespace and punctuation

That has the same problem as this bug report -- it uses a syntax class.
(And it doesn't seem to have much to do with whitespace, which is what
this variable is about.

> 4. Any sequence of characters (fuzzy)

That doesn't seem to have anything to do with whitespace.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no