bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Drew Adams]

[-- Attachment #1: Type: text/plain, Size: 295 bytes --]

What's the status of this bug?  Apart from the request
by Juri for changes to be made to Org mode, is the code
Juri prescribed for this bug fix going to be applied,
or has it been applied?

Is fixing this bug (e.g. with Juri's code) somehow
dependent on also fixing some Org-mode code?


[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 13329 bytes --]

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

> What's the status of this bug?  Apart from the request
> by Juri for changes to be made to Org mode, is the code
> Juri prescribed for this bug fix going to be applied,
> or has it been applied?
>
> Is fixing this bug (e.g. with Juri's code) somehow
> dependent on also fixing some Org-mode code?

Fortunately, the isearch fix doesn't depend on org-mode.
So here is the complete implementation that was made possible
thanks to the suggestions in
https://lists.gnu.org/archive/html/emacs-devel/2022-06/msg01208.html

For example, searching for the word "nongnu" in etc/ORG-NEWS
when all outlines are collapsed with isearch-lazy-count=t
will show such prompts:

search-invisible = t      1/4 I-search:
search-invisible = open   1/2 (invisible 2) I-search:
search-invisible = nil    0/0 (invisible 4) I-search:

where such numbers as "1/2" show the matches that can be visited by isearch,
and "(invisible 2)" are the rest of matches unreachable by isearch.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: lazy-count-invisible.patch --]
[-- Type: text/x-diff, Size: 7164 bytes --]

diff --git a/lisp/isearch.el b/lisp/isearch.el
index db7b53c014..945dce029a 100644
--- a/lisp/isearch.el
+++ b/lisp/isearch.el
@@ -466,6 +466,12 @@ lazy-count-suffix-format
   :group 'lazy-count
   :version "27.1")
 
+(defvar lazy-count-invisible-format " (invisible %s)"
+  "Format of the number of invisible matches for the prompt.
+When invisible matches exist, their number is appended
+after the total number of matches.  Display nothing when
+this variable is nil.")
+
 \f
 ;; Define isearch help map.
 
@@ -1277,6 +1283,7 @@ isearch-mode
 
 	isearch-lazy-count-current nil
 	isearch-lazy-count-total nil
+	isearch-lazy-count-invisible nil
 
 	;; Save the original value of `minibuffer-message-timeout', and
 	;; set it to nil so that isearch's messages don't get timed out.
@@ -3529,7 +3536,12 @@ isearch-lazy-count-format
                     (- isearch-lazy-count-total
                        isearch-lazy-count-current
                        -1)))
-                (or isearch-lazy-count-total "?"))
+                (if (and isearch-lazy-count-invisible
+                         lazy-count-invisible-format)
+                    (concat (format "%s" (or isearch-lazy-count-total "?"))
+                            (format lazy-count-invisible-format
+                                    isearch-lazy-count-invisible))
+                  (or isearch-lazy-count-total "?")))
       "")))
 
 \f
@@ -3780,10 +3792,11 @@ isearch-range-invisible
     (save-excursion
       (goto-char beg)
       (let (;; can-be-opened keeps track if we can open some overlays.
-	    (can-be-opened (eq search-invisible 'open))
+	    (can-be-opened (memq search-invisible '(open can-be-opened)))
 	    ;; the list of overlays that could be opened
 	    (crt-overlays nil))
-	(when (and can-be-opened isearch-hide-immediately)
+	(when (and can-be-opened isearch-hide-immediately
+                   (not (eq search-invisible 'can-be-opened)))
 	  (isearch-close-unnecessary-overlays beg end))
 	;; If the following character is currently invisible,
 	;; skip all characters with that same `invisible' property value.
@@ -3822,9 +3835,10 @@ isearch-range-invisible
 	(if (>= (point) end)
 	    (if (and can-be-opened (consp crt-overlays))
 		(progn
-		  (setq isearch-opened-overlays
-			(append isearch-opened-overlays crt-overlays))
-		  (mapc 'isearch-open-overlay-temporary crt-overlays)
+		  (unless (eq search-invisible 'can-be-opened)
+                    (setq isearch-opened-overlays
+			  (append isearch-opened-overlays crt-overlays))
+		    (mapc 'isearch-open-overlay-temporary crt-overlays))
 		  nil)
 	      (setq isearch-hidden t)))))))
 
@@ -4008,6 +4022,7 @@ isearch-lazy-highlight-forward
 (defvar isearch-lazy-highlight-error nil)
 (defvar isearch-lazy-count-current nil)
 (defvar isearch-lazy-count-total nil)
+(defvar isearch-lazy-count-invisible nil)
 (defvar isearch-lazy-count-hash (make-hash-table))
 (defvar lazy-count-update-hook nil
   "Hook run after new lazy count results are computed.")
@@ -4086,7 +4101,8 @@ isearch-lazy-highlight-new-loop
         ;; Reset old counter before going to count new numbers
         (clrhash isearch-lazy-count-hash)
         (setq isearch-lazy-count-current nil
-              isearch-lazy-count-total nil)
+              isearch-lazy-count-total nil
+              isearch-lazy-count-invisible nil)
         ;; Delay updating the message if possible, to avoid flicker
         (when (string-equal isearch-string "")
           (when (and isearch-mode (null isearch-message-function))
@@ -4166,11 +4182,13 @@ isearch-lazy-highlight-search
 	    (isearch-regexp-lax-whitespace
 	     isearch-lazy-highlight-regexp-lax-whitespace)
 	    (isearch-forward isearch-lazy-highlight-forward)
-	    ;; Don't match invisible text unless it can be opened
-	    ;; or when counting matches and user can visit hidden matches
-	    (search-invisible (or (eq search-invisible 'open)
-				  (and isearch-lazy-count search-invisible)))
-	    (retry t)
+            ;; Count all invisible matches, but highlight only
+            ;; according to search-invisible without opening overlays.
+	    (search-invisible (or isearch-lazy-count
+                                  (if (eq search-invisible 'open)
+                                      'can-be-opened
+                                    search-invisible)))
+            (retry t)
 	    (success nil))
 	;; Use a loop like in `isearch-search'.
 	(while retry
@@ -4186,15 +4204,23 @@ isearch-lazy-highlight-search
     (error nil)))
 
 (defun isearch-lazy-highlight-match (mb me)
-  (let ((ov (make-overlay mb me)))
-    (push ov isearch-lazy-highlight-overlays)
-    ;; 1000 is higher than ediff's 100+,
-    ;; but lower than isearch main overlay's 1001
-    (overlay-put ov 'priority 1000)
-    (overlay-put ov 'face 'lazy-highlight)
-    (unless (or (eq isearch-lazy-highlight 'all-windows)
-                isearch-lazy-highlight-buffer)
-      (overlay-put ov 'window (selected-window)))))
+  (unless (and isearch-lazy-count
+               ;; Recheck the match that possibly was intended
+               ;; for counting only, but not for highlighting
+               (not (let ((search-invisible
+                           (if (eq search-invisible 'open)
+                               'can-be-opened
+                             search-invisible)))
+                      (funcall isearch-filter-predicate mb me))))
+    (let ((ov (make-overlay mb me)))
+      (push ov isearch-lazy-highlight-overlays)
+      ;; 1000 is higher than ediff's 100+,
+      ;; but lower than isearch main overlay's 1001
+      (overlay-put ov 'priority 1000)
+      (overlay-put ov 'face 'lazy-highlight)
+      (unless (or (eq isearch-lazy-highlight 'all-windows)
+                  isearch-lazy-highlight-buffer)
+        (overlay-put ov 'window (selected-window))))))
 
 (defun isearch-lazy-highlight-start ()
   "Start a new lazy-highlight updating loop."
@@ -4328,11 +4354,19 @@ isearch-lazy-highlight-buffer-update
 				(setq found nil)
 			      (forward-char -1)))
 			(when isearch-lazy-count
-			  (setq isearch-lazy-count-total
-				(1+ (or isearch-lazy-count-total 0)))
-			  (puthash (if isearch-lazy-highlight-forward me mb)
-				   isearch-lazy-count-total
-				   isearch-lazy-count-hash))
+                          ;; Count as invisible when can't open overlay
+			  (if (not (let ((search-invisible
+                                          (if (eq search-invisible 'open)
+                                              'can-be-opened
+                                            search-invisible)))
+                                     (funcall isearch-filter-predicate mb me)))
+			      (setq isearch-lazy-count-invisible
+				    (1+ (or isearch-lazy-count-invisible 0)))
+			    (setq isearch-lazy-count-total
+				  (1+ (or isearch-lazy-count-total 0)))
+			    (puthash (if isearch-lazy-highlight-forward me mb)
+				     isearch-lazy-count-total
+				     isearch-lazy-count-hash)))
 			;; Don't highlight the match when this loop is used
 			;; only to count matches or when matches were already
 			;; highlighted within the current window boundaries

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Juri Linkov]

close 40808 29.0.50
thanks

>> What's the status of this bug?  Apart from the request
>> by Juri for changes to be made to Org mode, is the code
>> Juri prescribed for this bug fix going to be applied,
>> or has it been applied?
>>
>> Is fixing this bug (e.g. with Juri's code) somehow
>> dependent on also fixing some Org-mode code?
>
> Fortunately, the isearch fix doesn't depend on org-mode.
> So here is the complete implementation that was made possible

Now pushed and closed since org-mode problems should be
solved independently by org-mode developers.

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 658 bytes --]

> So here is the complete implementation that was made possible
> thanks to the suggestions in
> https://lists.gnu.org/archive/html/emacs-devel/2022-06/msg01208.html
>
> For example, searching for the word "nongnu" in etc/ORG-NEWS
> when all outlines are collapsed with isearch-lazy-count=t
> will show such prompts:
>
> search-invisible = t      1/4 I-search:
> search-invisible = open   1/2 (invisible 2) I-search:
> search-invisible = nil    0/0 (invisible 4) I-search:
>
> where such numbers as "1/2" show the matches that can be visited by isearch,
> and "(invisible 2)" are the rest of matches unreachable by isearch.

Here are more fixes with tests:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: isearch-lazy-highlight-invisible.patch --]
[-- Type: text/x-diff, Size: 9790 bytes --]

diff --git a/lisp/isearch.el b/lisp/isearch.el
index bc3697deb0a..529676042a2 100644
--- a/lisp/isearch.el
+++ b/lisp/isearch.el
@@ -2525,10 +2525,11 @@ isearch-delete-char
       (ding)
     (isearch-pop-state))
   ;; When going back to the hidden match, reopen it and close other overlays.
-  (when (and (eq search-invisible 'open) isearch-hide-immediately)
+  (when (and (eq isearch-invisible 'open) isearch-hide-immediately)
     (if isearch-other-end
-        (isearch-range-invisible (min (point) isearch-other-end)
-                                 (max (point) isearch-other-end))
+        (let ((search-invisible isearch-invisible))
+          (isearch-range-invisible (min (point) isearch-other-end)
+                                   (max (point) isearch-other-end)))
       (isearch-close-unnecessary-overlays (point) (point))))
   (isearch-update))
 
@@ -4008,6 +4009,7 @@ isearch-lazy-highlight-point-min
 (defvar isearch-lazy-highlight-point-max nil)
 (defvar isearch-lazy-highlight-buffer nil)
 (defvar isearch-lazy-highlight-case-fold-search nil)
+(defvar isearch-lazy-highlight-invisible nil)
 (defvar isearch-lazy-highlight-regexp nil)
 (defvar isearch-lazy-highlight-lax-whitespace nil)
 (defvar isearch-lazy-highlight-regexp-lax-whitespace nil)
@@ -4053,6 +4055,8 @@ isearch-lazy-highlight-new-loop
                             isearch-lazy-highlight-window-group))
 		 (not (eq isearch-lazy-highlight-case-fold-search
 			  isearch-case-fold-search))
+		 (not (eq isearch-lazy-highlight-invisible
+		          isearch-invisible))
 		 (not (eq isearch-lazy-highlight-regexp
 			  isearch-regexp))
 		 (not (eq isearch-lazy-highlight-regexp-function
@@ -4131,6 +4135,7 @@ isearch-lazy-highlight-new-loop
 	  isearch-lazy-highlight-wrapped      nil
 	  isearch-lazy-highlight-last-string  isearch-string
 	  isearch-lazy-highlight-case-fold-search isearch-case-fold-search
+	  isearch-lazy-highlight-invisible isearch-invisible
 	  isearch-lazy-highlight-regexp       isearch-regexp
 	  isearch-lazy-highlight-lax-whitespace   isearch-lax-whitespace
 	  isearch-lazy-highlight-regexp-lax-whitespace isearch-regexp-lax-whitespace
@@ -4180,8 +4185,10 @@ isearch-lazy-highlight-search
 	    (isearch-forward isearch-lazy-highlight-forward)
 	    ;; Count all invisible matches, but highlight only
 	    ;; matches that can be opened by visiting them later
-	    (search-invisible (or (not (null isearch-lazy-count))
-				  'can-be-opened))
+	    (search-invisible
+             (or (not (null isearch-lazy-count))
+		 (and (eq isearch-lazy-highlight-invisible 'open)
+                      'can-be-opened)))
 	    (retry t)
 	    (success nil))
 	;; Use a loop like in `isearch-search'.
@@ -4201,7 +4208,9 @@ isearch-lazy-highlight-match
   (when (or (not isearch-lazy-count)
             ;; Recheck the match that possibly was intended
             ;; for counting only, but not for highlighting
-            (let ((search-invisible 'can-be-opened))
+            (let ((search-invisible
+                   (and (eq isearch-lazy-highlight-invisible 'open)
+                        'can-be-opened)))
               (funcall isearch-filter-predicate mb me)))
     (let ((ov (make-overlay mb me)))
       (push ov isearch-lazy-highlight-overlays)
@@ -4350,9 +4359,9 @@ isearch-lazy-highlight-buffer-update
 			  ;; value `open' since then lazy-highlight
 			  ;; will open all overlays with matches.
 			  (if (not (let ((search-invisible
-					  (if (eq search-invisible 'open)
+					  (if (eq isearch-lazy-highlight-invisible 'open)
 					      'can-be-opened
-					    search-invisible)))
+					    isearch-lazy-highlight-invisible)))
 				     (funcall isearch-filter-predicate mb me)))
 			      (setq isearch-lazy-count-invisible
 				    (1+ (or isearch-lazy-count-invisible 0)))
diff --git a/lisp/replace.el b/lisp/replace.el
index c7ae77d128b..11e30e62a9e 100644
--- a/lisp/replace.el
+++ b/lisp/replace.el
@@ -2744,6 +2744,7 @@ replace-highlight
 	    (isearch-regexp-lax-whitespace
 	     replace-regexp-lax-whitespace)
 	    (isearch-case-fold-search case-fold)
+	    (isearch-invisible search-invisible)
 	    (isearch-forward (not backward))
 	    (isearch-other-end match-beg)
 	    (isearch-error nil)
diff --git a/test/lisp/isearch-tests.el b/test/lisp/isearch-tests.el
index 8cb5e5e4542..37a0ba8579c 100644
--- a/test/lisp/isearch-tests.el
+++ b/test/lisp/isearch-tests.el
@@ -38,6 +38,157 @@ isearch--test-done
   ;; Bug #21091: let `isearch-done' work without `isearch-update'.
   (isearch-done))
 
+\f
+;; Search invisible.
+
+(declare-function outline-hide-sublevels "outline")
+
+(ert-deftest isearch--test-invisible ()
+  (require 'outline)
+  (with-temp-buffer
+    (set-window-buffer nil (current-buffer))
+    (insert "\n1\n"
+            (propertize "2" 'invisible t)
+            (propertize "3" 'inhibit-isearch t)
+            "\n* h\n4\n\n")
+    (outline-mode)
+    (outline-hide-sublevels 1)
+    (goto-char (point-min))
+
+    (let ((isearch-lazy-count nil)
+          (search-invisible t)
+          (inhibit-message t))
+
+      (isearch-forward-regexp nil 1)
+      (isearch-process-search-string "[0-9]" "[0-9]")
+      (should (eq (point) 3))
+
+      (isearch-lazy-highlight-start)
+      (should (equal (seq-uniq (mapcar #'overlay-start isearch-lazy-highlight-overlays))
+                     '(2)))
+
+      (isearch-repeat-forward)
+      (should (eq (point) 5))
+      (should (get-char-property 4 'invisible))
+      (isearch-repeat-forward)
+      (should (eq (point) 12))
+      (should (get-char-property 11 'invisible))
+
+      (goto-char isearch-opoint)
+      (isearch-done t)
+
+      (isearch-forward-regexp nil 1)
+      (setq isearch-invisible nil) ;; isearch-toggle-invisible
+      (isearch-process-search-string "[0-9]" "[0-9]")
+
+      (isearch-lazy-highlight-start)
+      (should (equal (seq-uniq (mapcar #'overlay-start isearch-lazy-highlight-overlays))
+                     '(2)))
+
+      (goto-char isearch-opoint)
+      (isearch-done t)
+
+      (isearch-forward-regexp nil 1)
+      (setq isearch-invisible 'open) ;; isearch-toggle-invisible
+      (isearch-process-search-string "[0-9]" "[0-9]")
+      (should (eq (point) 3))
+
+      (isearch-lazy-highlight-start)
+      (should (equal (seq-uniq (mapcar #'overlay-start isearch-lazy-highlight-overlays))
+                     '(2 11)))
+
+      (let ((isearch-hide-immediately t))
+        (isearch-repeat-forward)
+        (should (eq (point) 12))
+        (should-not (get-char-property 11 'invisible))
+        (isearch-delete-char)
+        (should (get-char-property 11 'invisible)))
+
+      (let ((isearch-hide-immediately nil))
+        (isearch-repeat-forward)
+        (should (eq (point) 12))
+        (should-not (get-char-property 11 'invisible))
+        (isearch-delete-char)
+        (should-not (get-char-property 11 'invisible)))
+
+      (goto-char isearch-opoint)
+      (isearch-done t)
+      (isearch-clean-overlays)
+      (should (get-char-property 11 'invisible)))
+
+    (let ((isearch-lazy-count t)
+          (search-invisible t)
+          (inhibit-message t))
+
+      (isearch-forward-regexp nil 1)
+      (isearch-process-search-string "[0-9]" "[0-9]")
+      (should (eq (point) 3))
+
+      (setq isearch-lazy-count-invisible nil isearch-lazy-count-total nil)
+      (isearch-lazy-highlight-start)
+      (isearch-lazy-highlight-buffer-update)
+      (should (eq isearch-lazy-count-invisible nil))
+      (should (eq isearch-lazy-count-total 3))
+      (should (equal (seq-uniq (mapcar #'overlay-start isearch-lazy-highlight-overlays))
+                     '(2)))
+
+      (isearch-repeat-forward)
+      (should (eq (point) 5))
+      (should (get-char-property 4 'invisible))
+      (isearch-repeat-forward)
+      (should (eq (point) 12))
+      (should (get-char-property 11 'invisible))
+
+      (goto-char isearch-opoint)
+      (isearch-done t)
+
+      (isearch-forward-regexp nil 1)
+      (setq isearch-invisible nil) ;; isearch-toggle-invisible
+      (isearch-process-search-string "[0-9]" "[0-9]")
+
+      (setq isearch-lazy-count-invisible nil isearch-lazy-count-total nil)
+      (isearch-lazy-highlight-start)
+      (isearch-lazy-highlight-buffer-update)
+      (should (eq isearch-lazy-count-invisible 2))
+      (should (eq isearch-lazy-count-total 1))
+      (should (equal (seq-uniq (mapcar #'overlay-start isearch-lazy-highlight-overlays))
+                     '(2)))
+
+      (goto-char isearch-opoint)
+      (isearch-done t)
+
+      (isearch-forward-regexp nil 1)
+      (setq isearch-invisible 'open) ;; isearch-toggle-invisible
+      (isearch-process-search-string "[0-9]" "[0-9]")
+      (should (eq (point) 3))
+
+      (setq isearch-lazy-count-invisible nil isearch-lazy-count-total nil)
+      (isearch-lazy-highlight-start)
+      (isearch-lazy-highlight-buffer-update)
+      (should (eq isearch-lazy-count-invisible 1))
+      (should (eq isearch-lazy-count-total 2))
+      (should (equal (seq-uniq (mapcar #'overlay-start isearch-lazy-highlight-overlays))
+                     '(2 11)))
+
+      (let ((isearch-hide-immediately t))
+        (isearch-repeat-forward)
+        (should (eq (point) 12))
+        (should-not (get-char-property 11 'invisible))
+        (isearch-delete-char)
+        (should (get-char-property 11 'invisible)))
+
+      (let ((isearch-hide-immediately nil))
+        (isearch-repeat-forward)
+        (should (eq (point) 12))
+        (should-not (get-char-property 11 'invisible))
+        (isearch-delete-char)
+        (should-not (get-char-property 11 'invisible)))
+
+      (goto-char isearch-opoint)
+      (isearch-done t)
+      (isearch-clean-overlays)
+      (should (get-char-property 11 'invisible)))))
+
 \f
 ;; Search functions.
 

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Juri Linkov]

> Here are more fixes with tests:

I don't know if this is suitable for emacs-29.

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Juri Linkov]

>> Here are more fixes with tests:
>
> I don't know if this is suitable for emacs-29.

I guess these changes should go to master,
but a little later to avoid merge conflicts.

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Abdullah Asad]

When using isearch with isearch-lazy-count in org mode with heading
collapsed isearch gives wrong number of matches (both total & current).

Reproducing it is simple just enable isearch-lazy-count and perform
isearch in org buffer with collapsed heading.

In GNU Emacs 27.0.91 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.17)
of 2020-04-23 built on notArch
Windowing system distributor 'The X.Org Foundation

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Juri Linkov]

> When using isearch with isearch-lazy-count in org mode with heading
> collapsed isearch gives wrong number of matches (both total & current).
>
> Reproducing it is simple just enable isearch-lazy-count and perform
> isearch in org buffer with collapsed heading.

Thanks for bringing up this question, it will help to fix this feature
for the upcoming release.  I tend to agree that isearch-lazy-count
should show the number of *all* matches in the buffer, even the matches that
are invisible, and that can be opened when visited by isearch navigation.

Eli, do you agree with this patch for emacs-27.  It counts all matches, even
invisible.  And also it fixes an old bug existed in previous Emacs versions
where lazy-highlight didn't update lazy-highlighting when a hidden outline
was automatically opened at the end of the file (I noticed this bug only now).

So the condition ‘(eq search-invisible 'open)’ fixes an old bug, and
the condition ‘isearch-lazy-count’ fixes the new feature added in emacs-27:

diff --git a/lisp/isearch.el b/lisp/isearch.el
index e13a4dda83..ed1097c5ea 100644
--- a/lisp/isearch.el
+++ b/lisp/isearch.el
@@ -319,7 +319,7 @@ isearch-lazy-count
   "Show match numbers in the search prompt.
 When both this option and `isearch-lazy-highlight' are non-nil,
 show the current match number and the total number of matches
-in the buffer (or its restriction)."
+in the buffer (or its restriction), including all hidden matches."
   :type 'boolean
   :group 'lazy-count
   :group 'isearch
@@ -3869,7 +3871,8 @@ isearch-lazy-highlight-search
 	    (isearch-regexp-lax-whitespace
 	     isearch-lazy-highlight-regexp-lax-whitespace)
 	    (isearch-forward isearch-lazy-highlight-forward)
-	    (search-invisible nil)	; don't match invisible text
+            ;; don't match invisible text unless it can open or counting matches
+            (search-invisible (or (eq search-invisible 'open) isearch-lazy-count))
 	    (retry t)
 	    (success nil))
 	;; Use a loop like in `isearch-search'.

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Abdullah Asad]

> Eli, do you agree with this patch for emacs-27.
So, what is decided. Is it getting patched for emacs-27?

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Eli Zaretskii]

> From: Abdullah Asad <abdullah@net-c.com>
> Date: Wed, 29 Apr 2020 00:16:39 +0530
> Cc: 40808@debbugs.gnu.org
> 
> > Eli, do you agree with this patch for emacs-27.
> So, what is decided. Is it getting patched for emacs-27?

Yes, it's fine with me.  Thanks.

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 761 bytes --]

>> So, what is decided. Is it getting patched for emacs-27?
>
> Yes, it's fine with me.  Thanks.

Now pushed to emacs-27.

But there is still one corner case I'm worried about:
isearch-lazy-count still counts matches that can't be opened
and can't be visited, such as hidden links in org-mode.

To not count them we need to bind 'search-invisible' to 'open'
in isearch-lazy-highlight-search, but the problem is that
isearch-filter-predicate and isearch-range-invisible will
temporarily open them, whereas we need just to check
whether they can be opened.

So in the following patch I added a new variable isearch-check-overlays
that could instruct isearch-range-invisible to not open overlays
when it's non-nil that means we need only to check them, not open:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: isearch-check-overlays.patch --]
[-- Type: text/x-diff, Size: 1952 bytes --]

diff --git a/lisp/isearch.el b/lisp/isearch.el
index e13a4dda83..0ad97a092f 100644
--- a/lisp/isearch.el
+++ b/lisp/isearch.el
@@ -3535,6 +3535,7 @@ isearch-close-unnecessary-overlays
 	    (overlay-put ov 'invisible (overlay-get ov 'isearch-invisible))
 	    (overlay-put ov 'isearch-invisible nil)))))))
 
+(defvar isearch-check-overlays nil)
 
 (defun isearch-range-invisible (beg end)
   "Return t if all the text from BEG to END is invisible."
@@ -3546,7 +3547,7 @@ isearch-range-invisible
 	    (can-be-opened (eq search-invisible 'open))
 	    ;; the list of overlays that could be opened
 	    (crt-overlays nil))
-	(when (and can-be-opened isearch-hide-immediately)
+	(when (and can-be-opened isearch-hide-immediately (not isearch-check-overlays))
 	  (isearch-close-unnecessary-overlays beg end))
 	;; If the following character is currently invisible,
 	;; skip all characters with that same `invisible' property value.
@@ -3585,9 +3586,10 @@ isearch-range-invisible
 	(if (>= (point) end)
 	    (if (and can-be-opened (consp crt-overlays))
 		(progn
-		  (setq isearch-opened-overlays
-			(append isearch-opened-overlays crt-overlays))
-		  (mapc 'isearch-open-overlay-temporary crt-overlays)
+		  (unless isearch-check-overlays
+		    (setq isearch-opened-overlays
+			  (append isearch-opened-overlays crt-overlays))
+		    (mapc 'isearch-open-overlay-temporary crt-overlays))
 		  nil)
 	      (setq isearch-hidden t)))))))
 
@@ -3880,8 +3885,10 @@ isearch-lazy-highlight-search
 	  (if (or (not success)
 		  (= (point) bound) ; like (bobp) (eobp) in `isearch-search'.
 		  (= (match-beginning 0) (match-end 0))
-		  (funcall isearch-filter-predicate
-			   (match-beginning 0) (match-end 0)))
+		  (let ((search-invisible (and search-invisible 'open))
+		        (isearch-check-overlays t))
+		    (funcall isearch-filter-predicate
+			     (match-beginning 0) (match-end 0))))
 	      (setq retry nil)))
 	success)
     (error nil)))

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Abdullah Asad <abdullah@net-c.com>,  40808@debbugs.gnu.org
> Date: Wed, 29 Apr 2020 02:54:06 +0300
> 
> But there is still one corner case I'm worried about:
> isearch-lazy-count still counts matches that can't be opened
> and can't be visited, such as hidden links in org-mode.
> 
> To not count them we need to bind 'search-invisible' to 'open'
> in isearch-lazy-highlight-search, but the problem is that
> isearch-filter-predicate and isearch-range-invisible will
> temporarily open them, whereas we need just to check
> whether they can be opened.
> 
> So in the following patch I added a new variable isearch-check-overlays
> that could instruct isearch-range-invisible to not open overlays
> when it's non-nil that means we need only to check them, not open:

Fine with me, but please make this change on master, not on emacs-27.

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Juri Linkov]

>> But there is still one corner case I'm worried about:
>> isearch-lazy-count still counts matches that can't be opened
>> and can't be visited, such as hidden links in org-mode.
>>
>> To not count them we need to bind 'search-invisible' to 'open'
>> in isearch-lazy-highlight-search, but the problem is that
>> isearch-filter-predicate and isearch-range-invisible will
>> temporarily open them, whereas we need just to check
>> whether they can be opened.
>>
>> So in the following patch I added a new variable isearch-check-overlays
>> that could instruct isearch-range-invisible to not open overlays
>> when it's non-nil that means we need only to check them, not open:
>
> Fine with me, but please make this change on master, not on emacs-27.

Right decision, because org-mode is broken anyway, and this fix won't make
it better.  A test case that demonstrates the problem in org-mode:

0. emacs -Q
1. visit emacs/etc/ORG-NEWS
2. isearch for "http": C-s http C-s C-s C-s ...

it eventually stops at org links where "http" is invisible, such as

  [[https://orgmode.org/worg/library-of-babel.html][here]]

The problem is that org-mode puts the text property 'invisible'
only when the link is scrolled into view, I guess it puts the property
using font-lock.

So at least in emacs-27 now it consistently counts all matches
ignoring their visibility.

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Juri Linkov]

>> Fine with me, but please make this change on master, not on emacs-27.
>
> Right decision, because org-mode is broken anyway, and this fix won't make
> it better.  A test case that demonstrates the problem in org-mode:
>
> 0. emacs -Q
> 1. visit emacs/etc/ORG-NEWS
> 2. isearch for "http": C-s http C-s C-s C-s ...
>
> it eventually stops at org links where "http" is invisible, such as
>
>   [[https://orgmode.org/worg/library-of-babel.html][here]]
>
> The problem is that org-mode puts the text property 'invisible'
> only when the link is scrolled into view, I guess it puts the property
> using font-lock.
>
> So at least in emacs-27 now it consistently counts all matches
> ignoring their visibility.

I'm still unsure about pushing this change.  The current state of things:

- in emacs-27 and master now it counts all matches, including invisible,
  regardless of the value of search-invisible;

- the proposed change for master doesn't include in the count these
  invisible matches that can't be visited by opening overlays;

- but this fix is pointless for org-mode where visiting invisible
  matches is broken, and where its count varies after visiting
  and opening hidden overlays.

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: 40808@debbugs.gnu.org,  abdullah@net-c.com
> Date: Thu, 30 Apr 2020 23:18:08 +0300
> 
> I'm still unsure about pushing this change.  The current state of things:
> 
> - in emacs-27 and master now it counts all matches, including invisible,
>   regardless of the value of search-invisible;
> 
> - the proposed change for master doesn't include in the count these
>   invisible matches that can't be visited by opening overlays;
> 
> - but this fix is pointless for org-mode where visiting invisible
>   matches is broken, and where its count varies after visiting
>   and opening hidden overlays.

Would it be possible to fix the breakage in Org on master?

bug#40808: 27.0.91; inaccuracy in isearch-lazy-count

[Juri Linkov]

>> I'm still unsure about pushing this change.  The current state of things:
>>
>> - in emacs-27 and master now it counts all matches, including invisible,
>>   regardless of the value of search-invisible;
>>
>> - the proposed change for master doesn't include in the count these
>>   invisible matches that can't be visited by opening overlays;
>>
>> - but this fix is pointless for org-mode where visiting invisible
>>   matches is broken, and where its count varies after visiting
>>   and opening hidden overlays.
>
> Would it be possible to fix the breakage in Org on master?

We need help from Org developers to fix the breakage in Org.

Meanwhile, I reverted a part of the recent fix on emacs-27
that tried to fix a long-standing corner case of lazy-highlighting
in outlines, but whose side-effect might slow down lazy-highlighting
in collapsed outlines.  This needs more testing on master.

Whereas the fix for isearch-lazy-count still remains in the release branch.