bug#53758: 28.0.91; Recursive edit during dired-do-find-regexp-and-replace breaks isearch

bug#53758: 28.0.91; Recursive edit during dired-do-find-regexp-and-replace breaks isearch

[sbaugh]

isearch-forward (C-s) while in a recursive edit (C-r) triggered from
dired-do-find-regexp-and-replace (Q in a dired buffer) always fails to
find any matches for any string, even if there are matches in the
buffer.

Steps to reproduce:
1. Open a dired buffer in a directory containing at least one file which
contains some text (e.g. "Hello world")
2. Run dired-do-find-regexp-and-replace to replace "Hello" with
"Goodbye" ("Q Hello RET Goodbye RET"); this will switch buffers
to the first matching file.   
3. Type C-r to enter a recursive edit (I'm guessing this runs
(recursive-edit)?)
4. At the start of the buffer, run isearch-forward searching for "world"
("C-s world RET")
5. Note that the isearch fails despite "world" being in the buffer.

For what it's worth, this interestingly doesn't happen with
project-query-regexp-replace (which also does multi-file query-replace).


In GNU Emacs 28.0.91 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.30, cairo version 1.16.0)
Repository revision: 525dc6e5c428185b62c72d7958cd4fe17937f126
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: NixOS 21.05 (Okapi)

Configured using:
 'configure
 --prefix=/nix/store/rkpz8vsk15zlphra3hhngf96irwci7r3-emacs-git-20220115.0
 --disable-build-details --with-modules --with-x-toolkit=gtk3 --with-xft
 --with-cairo'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY
PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE
XIM XPM GTK3 ZLIB

Important settings:
  value of $EMACSLOADPATH: 
  value of $EMACSNATIVELOADPATH: /nix/store/fs7slsl0rz28h6dq8rnhgk4ddkk8dh0w-emacs-packages-deps/share/emacs/native-lisp:/nix/store/b854vn0hskfs7d2j4pr1vp71fhf4ynvc-emacs-packages-deps/share/emacs/native-lisp:::
  value of $LANG: C.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Dired by name

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/sbaugh/.nix-profile/share/emacs/site-lisp/site-start hides /nix/store/b854vn0hskfs7d2j4pr1vp71fhf4ynvc-emacs-packages-deps/share/emacs/site-lisp/site-start
/home/sbaugh/.nix-profile/share/emacs/site-lisp/site-start hides /nix/store/rkpz8vsk15zlphra3hhngf96irwci7r3-emacs-git-20220115.0/share/emacs/site-lisp/site-start
/nix/store/b854vn0hskfs7d2j4pr1vp71fhf4ynvc-emacs-packages-deps/share/emacs/site-lisp/elpa/transient-20220112.1305/transient hides /nix/store/rkpz8vsk15zlphra3hhngf96irwci7r3-emacs-git-20220115.0/share/emacs/28.0.91/lisp/transient
/nix/store/b854vn0hskfs7d2j4pr1vp71fhf4ynvc-emacs-packages-deps/share/emacs/site-lisp/elpa/let-alist-1.0.6/let-alist hides /nix/store/rkpz8vsk15zlphra3hhngf96irwci7r3-emacs-git-20220115.0/share/emacs/28.0.91/lisp/emacs-lisp/let-alist

Features:
(shadow sort mail-extr eieio-opt cl-extra speedbar ezimage dframe
find-func shortdoc help-fns radix-tree help-mode emacsbug message rmc
puny rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util
rmail rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
misearch multi-isearch bug-reference pulse color vc-mtn vc-hg vc-git
diff-mode easy-mmode vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs vc
vc-dispatcher find-dired grep compile text-property-search comint
ansi-color xref project ring thingatpt dired-aux cl-loaddefs cl-lib
dired dired-loaddefs seq byte-opt gv bytecomp byte-compile cconv
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 588633 721804)
 (symbols 48 9814 41)
 (strings 32 58972 20078)
 (string-bytes 1 2344778)
 (vectors 16 49128)
 (vector-slots 8 391332 666289)
 (floats 8 169 491)
 (intervals 56 104017 11468)
 (buffers 992 22))

bug#53758: 28.0.91; Recursive edit during dired-do-find-regexp-and-replace breaks isearch

[Juri Linkov]

> isearch-forward (C-s) while in a recursive edit (C-r) triggered from
> dired-do-find-regexp-and-replace (Q in a dired buffer) always fails to
> find any matches for any string, even if there are matches in the
> buffer.
>
> Steps to reproduce:
> 1. Open a dired buffer in a directory containing at least one file which
> contains some text (e.g. "Hello world")
> 2. Run dired-do-find-regexp-and-replace to replace "Hello" with
> "Goodbye" ("Q Hello RET Goodbye RET"); this will switch buffers
> to the first matching file.
> 3. Type C-r to enter a recursive edit (I'm guessing this runs
> (recursive-edit)?)
> 4. At the start of the buffer, run isearch-forward searching for "world"
> ("C-s world RET")
> 5. Note that the isearch fails despite "world" being in the buffer.
>
> For what it's worth, this interestingly doesn't happen with
> project-query-regexp-replace (which also does multi-file query-replace).

This is because of these lines in xref--query-replace-1:

         ;; Counteract the "do the next match now" hack in
         ;; `perform-replace'.  And still, it'll report that those
         ;; matches were "filtered out" at the end.
         (isearch-filter-predicate
          (lambda (beg end)
            (and current-beg
                 (>= beg current-beg)
                 (<= end current-end))))

Dmitry, could you please explain the comment above.
What I don't understand is where this
"the next match now" hack is in `perform-replace'?

bug#53758: 28.0.91; Recursive edit during dired-do-find-regexp-and-replace breaks isearch

[Dmitry Gutov]

Hi Juri,

On 06.02.2022 19:44, Juri Linkov wrote:
>> isearch-forward (C-s) while in a recursive edit (C-r) triggered from
>> dired-do-find-regexp-and-replace (Q in a dired buffer) always fails to
>> find any matches for any string, even if there are matches in the
>> buffer.
>>
>> Steps to reproduce:
>> 1. Open a dired buffer in a directory containing at least one file which
>> contains some text (e.g. "Hello world")
>> 2. Run dired-do-find-regexp-and-replace to replace "Hello" with
>> "Goodbye" ("Q Hello RET Goodbye RET"); this will switch buffers
>> to the first matching file.
>> 3. Type C-r to enter a recursive edit (I'm guessing this runs
>> (recursive-edit)?)
>> 4. At the start of the buffer, run isearch-forward searching for "world"
>> ("C-s world RET")
>> 5. Note that the isearch fails despite "world" being in the buffer.
>>
>> For what it's worth, this interestingly doesn't happen with
>> project-query-regexp-replace (which also does multi-file query-replace).
> This is because of these lines in xref--query-replace-1:
> 
>           ;; Counteract the "do the next match now" hack in
>           ;; `perform-replace'.  And still, it'll report that those
>           ;; matches were "filtered out" at the end.
>           (isearch-filter-predicate
>            (lambda (beg end)
>              (and current-beg
>                   (>= beg current-beg)
>                   (<= end current-end))))
> 
> Dmitry, could you please explain the comment above.
> What I don't understand is where this
> "the next match now" hack is in `perform-replace'?

It's referring to the comment on lines starting with 2938 and the 
subsequent code which uses 'looking-at' instead of 
replace-re-search-function.

Here's that comment in full:

	  ;; Otherwise, if matching a regular expression, do the next
	  ;; match now, since the replacement for this match may
	  ;; affect whether the next match is adjacent to this one.
	  ;; If that match is empty, don't use it.

I'm not sure why the result would be different between 
dired-do-find-regexp-and-replace and project-query-regexp-replace, though.

bug#53758: 28.0.91; Recursive edit during dired-do-find-regexp-and-replace breaks isearch

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

> It's referring to the comment on lines starting with 2938 and the
> subsequent code which uses 'looking-at' instead of
> replace-re-search-function.
>
> Here's that comment in full:
>
> 	  ;; Otherwise, if matching a regular expression, do the next
> 	  ;; match now, since the replacement for this match may
> 	  ;; affect whether the next match is adjacent to this one.
> 	  ;; If that match is empty, don't use it.

Thanks, now everything is clear.  Then it can be simplified by this patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: xref--query-replace-1.patch --]
[-- Type: text/x-diff, Size: 1848 bytes --]

diff --git a/lisp/progmodes/xref.el b/lisp/progmodes/xref.el
index 4efa652084..f334710a28 100644
--- a/lisp/progmodes/xref.el
+++ b/lisp/progmodes/xref.el
@@ -843,25 +843,7 @@ xref--query-replace-1
          (continue t)
          did-it-once buf-pairs pairs
          current-beg current-end
-         ;; Counteract the "do the next match now" hack in
-         ;; `perform-replace'.  And still, it'll report that those
-         ;; matches were "filtered out" at the end.
-         (isearch-filter-predicate
-          (lambda (beg end)
-            (and current-beg
-                 (>= beg current-beg)
-                 (<= end current-end))))
-         (replace-re-search-function
-          (lambda (from &optional _bound noerror)
-            (let (found pair)
-              (while (and (not found) pairs)
-                (setq pair (pop pairs)
-                      current-beg (car pair)
-                      current-end (cdr pair))
-                (goto-char current-beg)
-                (when (re-search-forward from current-end noerror)
-                  (setq found t)))
-              found))))
+         (region-extract-function (lambda (_) pairs)))
     (while (and continue (setq buf-pairs (funcall iter :next)))
       (if did-it-once
           ;; Reuse the same window for subsequent buffers.
@@ -870,8 +852,9 @@ xref--query-replace-1
          (pop-to-buffer (car buf-pairs)))
         (setq did-it-once t))
       (setq pairs (cdr buf-pairs))
+      (goto-char (point-min))
       (setq continue
-            (perform-replace from to t t nil nil multi-query-replace-map)))
+            (perform-replace from to t t nil nil multi-query-replace-map nil nil nil t)))
     (unless did-it-once (user-error "No suitable matches here"))
     (when (and continue (not buf-pairs))
       (message "All results processed"))))

[-- Attachment #3: Type: text/plain, Size: 420 bytes --]


> I'm not sure why the result would be different between
> dired-do-find-regexp-and-replace and project-query-regexp-replace, though.

project-query-regexp-replace doesn't use xref--query-replace-1.

Actually, the reported problem is not specific to xref.
Performing replacements on a rectangular region doesn't allow
searching outside the region on recursive edit too.
So here is a general fix that covers all cases:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: isearch-filter-predicate.patch --]
[-- Type: text/x-diff, Size: 595 bytes --]

diff --git a/lisp/replace.el b/lisp/replace.el
index 23e6483809..5add576d6b 100644
--- a/lisp/replace.el
+++ b/lisp/replace.el
@@ -3233,7 +3233,10 @@ perform-replace
 			       (last-command 'recenter-top-bottom))
 			   (recenter-top-bottom)))
 			((eq def 'edit)
-			 (let ((opos (point-marker)))
+			 (let ((opos (point-marker))
+			       (isearch-filter-predicate isearch-filter-predicate))
+			   (when region-filter
+			     (remove-function isearch-filter-predicate region-filter))
 			   (setq real-match-data (replace-match-data
 						  nil real-match-data
 						  real-match-data))

bug#53758: 28.0.91; Recursive edit during dired-do-find-regexp-and-replace breaks isearch

[Dmitry Gutov]

On 07.02.2022 21:27, Juri Linkov wrote:
> Thanks, now everything is clear.  Then it can be simplified by this patch:

Thanks!

Are these changes independent? That is, will the new version of xref 
continue to work with Emacs 26.1?

bug#53758: 28.0.91; Recursive edit during dired-do-find-regexp-and-replace breaks isearch

[Juri Linkov]

>> Thanks, now everything is clear.  Then it can be simplified by this patch:
>
> Thanks!
>
> Are these changes independent? That is, will the new version of xref
> continue to work with Emacs 26.1?

It should continue to work.  Could you suggest how this
independent change in xref could be tested?  I see that
dired-do-find-regexp-and-replace replaces all matches,
but xref--query-replace-1 tries to skip some matches.
What test could emulate the case where some matches
should be skipped?

bug#53758: 28.0.91; Recursive edit during dired-do-find-regexp-and-replace breaks isearch

[Dmitry Gutov]

On 08.02.2022 09:43, Juri Linkov wrote:
> I see that
> dired-do-find-regexp-and-replace replaces all matches,
> but xref--query-replace-1 tries to skip some matches.
> What test could emulate the case where some matches
> should be skipped?

Perhaps try xref-find-references-and-replace?

Search for some symbol which can also be included as part of the other 
symbols' name.

E.g. search for 'dired-do-find-regexp' and verify that 
'dired-do-find-regexp-and-replace' is not affected.

bug#53758: 28.0.91; Recursive edit during dired-do-find-regexp-and-replace breaks isearch

[Juri Linkov]

close 53758 29.0.50
thanks

>> I see that
>> dired-do-find-regexp-and-replace replaces all matches,
>> but xref--query-replace-1 tries to skip some matches.
>> What test could emulate the case where some matches
>> should be skipped?
>
> Perhaps try xref-find-references-and-replace?
>
> Search for some symbol which can also be included as part of the other
> symbols' name.
>
> E.g. search for 'dired-do-find-regexp' and verify that
> 'dired-do-find-regexp-and-replace' is not affected.

I see that these custom functions are needed in xref
to search/replace the specific regexp ".*".
Then this is the same problem like in bug#14013.

So I fixed perform-replace to allow using isearch in a recursive edit,
and closed this report.  Other problems could be discussed in bug#14013.