A question about overlays and performance.

A question about overlays and performance.

[Jeff Norden]

I've been working on an improved emacs mode for CWEB files.

The elisp manual says this about overlays:

  However,... overlays generally don’t scale well...
  If you need to affect the visual appearance of many portions
  in the buffer, we recommend using text properties.

I've been using text properties, but am realizing that overlays would make
things a bit simpler.  Is there a way to predict how many overlays will be
"too many" and start to slow thing down?  Does the configuration of the
overlays play a role?

A file with 200 sections would wind up with 400 overlays, but the layout would
be simple.  There would be "section" overlays that are disjoint from each
other, and "subsection" overlays each of which is contained in a unique
section.  Each section would contain at most one subsection, so a position
would never be in more than two of the overlays.  It would also be easy to
call (overlay-recenter) each time the cursor moves to a new section.

Thanks in advance!
-Jeff

Re: A question about overlays and performance.

[Eli Zaretskii]

> From: Jeff Norden <jnorden@math.tntech.edu>
> Date: Sat, 25 Jul 2020 11:40:26 -0500
> 
> Is there a way to predict how many overlays will be "too many" and
> start to slow thing down?

Thousands.

> Does the configuration of the overlays play a role?

What do you mean by "configuration" in this context?

Re: A question about overlays and performance.

[Jeff Norden]

Thanks for the fast (nearly instantaneous!) response.
>> Is there a way to predict how many overlays will be "too many" and
>> start to slow thing down?
>
> Thousands.
That's great.  The number of overlays I'd create would certainly be less
than 1000.

>> Does the configuration of the overlays play a role?
>
> What do you mean by "configuration" in this context?

I wondered if the simple overlay layout that I described would have less
of an effect than if they were scattered about and intersected in many
different ways.  But if it takes thousands to affect performance, then
that doesn't really matter for my case.

Thanks again!
-Jeff

Re: A question about overlays and performance.

[Eli Zaretskii]

> From: Jeff Norden <jnorden@math.tntech.edu>
> Cc: emacs-devel@gnu.org
> Date: Sat, 25 Jul 2020 12:24:49 -0500
> 
> >> Does the configuration of the overlays play a role?
> >
> > What do you mean by "configuration" in this context?
> 
> I wondered if the simple overlay layout that I described would have less
> of an effect than if they were scattered about and intersected in many
> different ways.

No, not really.

Re: A question about overlays and performance.

[Ihor Radchenko]

> The elisp manual says this about overlays:

>  However,... overlays generally don’t scale well...
>  If you need to affect the visual appearance of many portions
>  in the buffer, we recommend using text properties.

Not directly related to the initial question here, but may I ask if
there are any practical guidelines to convert overlays to text
properties? I am specifically concerned about two issues when we want to
use text properties instead of overlays to hide text in buffer (pointing
at org-mode):
- text hidden via text properties is not well-handled by isearch. For
  overlays, isearch can temporary reveal the hidden text, which is not
  the case for text hidden using 'invisible text property.
- text properties are not buffer-local in indirect buffers. With
  overlays, it is possible to have define hidden text in basic and
  indirect buffer independently. However, text properties will always be
  shared and the text hidden in indirect buffer will automatically be
  hidden in the base buffer as well.

Best,
Ihor

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Jeff Norden <jnorden@math.tntech.edu>
>> Date: Sat, 25 Jul 2020 11:40:26 -0500
>> 
>> Is there a way to predict how many overlays will be "too many" and
>> start to slow thing down?
>
> Thousands.
>
>> Does the configuration of the overlays play a role?
>
> What do you mean by "configuration" in this context?
>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: A question about overlays and performance.

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 26 Jul 2020 09:33:21 +0800
> 
> Not directly related to the initial question here, but may I ask if
> there are any practical guidelines to convert overlays to text
> properties? I am specifically concerned about two issues when we want to
> use text properties instead of overlays to hide text in buffer (pointing
> at org-mode):
> - text hidden via text properties is not well-handled by isearch. For
>   overlays, isearch can temporary reveal the hidden text, which is not
>   the case for text hidden using 'invisible text property.
> - text properties are not buffer-local in indirect buffers. With
>   overlays, it is possible to have define hidden text in basic and
>   indirect buffer independently. However, text properties will always be
>   shared and the text hidden in indirect buffer will automatically be
>   hidden in the base buffer as well.

So what kind of guidelines did you want to hear regarding these
issues?

Re: A question about overlays and performance.

[Ihor Radchenko]

> So what kind of guidelines did you want to hear regarding these
> issues?

Mostly wanted to know if there is a known way to solve these issues.
If there is, it may be a good idea to describe it briefly in the manual.
If not, I was thinking about writing a patch/feature request to solve
these issues.

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Sun, 26 Jul 2020 09:33:21 +0800
>> 
>> Not directly related to the initial question here, but may I ask if
>> there are any practical guidelines to convert overlays to text
>> properties? I am specifically concerned about two issues when we want to
>> use text properties instead of overlays to hide text in buffer (pointing
>> at org-mode):
>> - text hidden via text properties is not well-handled by isearch. For
>>   overlays, isearch can temporary reveal the hidden text, which is not
>>   the case for text hidden using 'invisible text property.
>> - text properties are not buffer-local in indirect buffers. With
>>   overlays, it is possible to have define hidden text in basic and
>>   indirect buffer independently. However, text properties will always be
>>   shared and the text hidden in indirect buffer will automatically be
>>   hidden in the base buffer as well.
>
> So what kind of guidelines did you want to hear regarding these
> issues?

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: A question about overlays and performance.

[Stefan Monnier]

> - text hidden via text properties is not well-handled by isearch. For
>   overlays, isearch can temporary reveal the hidden text, which is not
>   the case for text hidden using 'invisible text property.

The recommending way to avoid this problem is to provide a patch which
extends Isearch's handling of invisible text to text-properties.
There's no reason to presume it would be hard to do.

> - text properties are not buffer-local in indirect buffers. With
>   overlays, it is possible to have define hidden text in basic and
>   indirect buffer independently. However, text properties will always be
>   shared and the text hidden in indirect buffer will automatically be
>   hidden in the base buffer as well.

And the way I recommend to avoid this problem is to stay clear of
indirect buffers (which I consider as an attractive nuisance).


        Stefan

Re: A question about overlays and performance.

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 755 bytes --]

> The recommending way to avoid this problem is to provide a patch which
> extends Isearch's handling of invisible text to text-properties.
> There's no reason to presume it would be hard to do.

See the attached.

> And the way I recommend to avoid this problem is to stay clear of
> indirect buffers (which I consider as an attractive nuisance).

It is not always possible. I am currently working on re-implementing
org-mode folding from using overlays to text-properties. However, it is
not possible to stay clear of indirect buffers in org-mode. Many
user packages and core code depend on indirect buffer functionality
(including different folding state). I will have to go ahead using my
"hacky" solution with char-property-alias-alist.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: iseach-text-properties.patch --]
[-- Type: text/x-diff, Size: 12399 bytes --]

diff --git a/lisp/isearch.el b/lisp/isearch.el
index 81e83d7950..e87f7f6756 100644
--- a/lisp/isearch.el
+++ b/lisp/isearch.el
@@ -145,6 +145,9 @@ search-invisible
 has a non-nil property `isearch-open-invisible', then incremental
 search will show the hidden text.  (This applies when using `outline.el'
 and `hideshow.el'.)
+When the value is `open-all', consider text made invisible by text
+property as well.  Text property `isearch-open-invisible' will be
+considered in the same way as for overlays.
 
 To temporarily change the value for an active incremental search,
 use \\<isearch-mode-map>\\[isearch-toggle-invisible].
@@ -942,6 +945,7 @@ isearch-new-forward
 
 ;; Accumulate here the overlays opened during searching.
 (defvar isearch-opened-overlays nil)
+(defvar isearch-opened-regions nil)
 
 ;; Non-nil if the string exists but is invisible.
 (defvar isearch-hidden nil)
@@ -1218,6 +1222,7 @@ isearch-mode
 	isearch-opoint (point)
 	search-ring-yank-pointer nil
 	isearch-opened-overlays nil
+        isearch-opened-regions nil
 	isearch-input-method-function input-method-function
 	isearch-input-method-local-p (local-variable-p 'input-method-function)
 	regexp-search-ring-yank-pointer nil
@@ -1551,6 +1556,7 @@ isearch-exit
       (let ((isearch-nonincremental t))
 	(isearch-edit-string)) ;; this calls isearch-done as well
     (isearch-done))
+  (isearch-clean-regions)
   (isearch-clean-overlays))
 
 (defun isearch-fail-pos (&optional msg)
@@ -1807,6 +1813,7 @@ isearch-cancel
 	(isearch--set-state (car isearch-cmds)))
     (goto-char isearch-opoint))
   (isearch-done t)                      ; Exit isearch..
+  (isearch-clean-regions)
   (isearch-clean-overlays)
   (signal 'quit nil))                   ; ..and pass on quit signal.
 
@@ -2271,6 +2278,7 @@ isearch-query-replace
 	(isearch-string-propertized
          (isearch-string-propertize isearch-string)))
     (isearch-done nil t)
+    (isearch-clean-regions)
     (isearch-clean-overlays)
     (if (and isearch-other-end
 	     (if backward
@@ -2376,6 +2384,7 @@ isearch--highlight-regexp-or-lines
 	;; We call `exit-recursive-edit' explicitly at the end below.
 	(isearch-recursive-edit nil))
     (isearch-done nil t)
+    (isearch-clean-regions)
     (isearch-clean-overlays))
   (require 'hi-lock nil t)
   (let ((regexp (cond ((functionp isearch-regexp-function)
@@ -2958,6 +2967,7 @@ isearch-pre-command-hook
      ;; Other characters terminate the search and are then executed normally.
      (search-exit-option
       (isearch-done)
+      (isearch-clean-regions)
       (isearch-clean-overlays)))))
 
 (defun isearch-post-command-hook ()
@@ -3312,6 +3322,22 @@ isearch-search-fun-function
 This returned function will be used by `isearch-search-string' to
 search for the first occurrence of STRING.")
 
+(defun isearch--find-text-property-region (pos prop)
+  "Find a region around POS containing PROP text property.
+Return nil when PROP is not set at POS."
+  (let* ((beg (and (get-text-property pos prop) pos))
+	 (end beg))
+    (when beg
+      (unless (or (equal beg (point-min))
+		  (not (eq (get-text-property beg prop)
+			 (get-text-property (1- beg) prop))))
+	(setq beg (previous-single-property-change pos prop nil (point-min))))
+      (unless (or (equal end (point-max))
+		  (not (eq (get-text-property end prop)
+			 (get-text-property (1+ end) prop))))
+	(setq end (next-single-property-change pos prop nil (point-max))))
+      (cons beg end))))
+
 (defun isearch-search-fun ()
   "Return the function to use for the search.
 Can be changed via `isearch-search-fun-function' for special needs."
@@ -3421,8 +3447,9 @@ isearch-search
 	  (if (or (not isearch-success)
 		  (bobp) (eobp)
 		  (= (match-beginning 0) (match-end 0))
-		  (funcall isearch-filter-predicate
-			   (match-beginning 0) (match-end 0)))
+                  (save-match-data
+		    (funcall isearch-filter-predicate
+			     (match-beginning 0) (match-end 0))))
 	      (setq retry nil)))
 	(setq isearch-just-started nil)
 	(if isearch-success
@@ -3481,6 +3508,20 @@ isearch-open-overlay-temporary
     (overlay-put ov 'isearch-invisible (overlay-get ov 'invisible))
     (overlay-put ov 'invisible nil)))
 
+;; Same as `isearch-open-overlay-temporary', but for text hidden via text
+;; properties.
+(defun isearch-open-region-temporary (hidden-region)
+  (if (not (null (get-text-property (car hidden-region) 'isearch-open-invisible-temporary)))
+      (funcall (get-text-property (car hidden-region) 'isearch-open-invisible-temporary) hidden-region nil)
+    (with-silent-modifications
+      (put-text-property (car hidden-region)
+                         (cdr hidden-region)
+                         'isearch-invisible
+                         (get-text-property (point) 'invisible))
+      (put-text-property (car hidden-region)
+                         (cdr hidden-region)
+                         'invisible
+                         nil))))
 
 ;; This is called at the end of isearch.  It will open the overlays
 ;; that contain the latest match.  Obviously in case of a C-g the
@@ -3501,6 +3542,23 @@ isearch-open-necessary-overlays
       (if fct-temp
 	  (funcall fct-temp ov t)))))
 
+;; Equivalent of `isearch-open-necessary-overlays' for text hidden via text
+;; properties.
+(defun isearch-open-necessary-region (region)
+  (let ((inside-region (and (> (point) (car region))
+                            (<= (point) (cdr region))))
+        (fct-temp (get-text-property (car region) 'isearch-open-invisible-temporary)))
+    (when (or inside-region (not fct-temp))
+      (with-silent-modifications
+        (put-text-property (car region)
+                           (cdr region)
+                           'invisible
+                           (get-text-property (car region) 'isearch-invisible))
+        (remove-text-properties (car region) (cdr region) '('isearch-invisible nil))))
+    (if inside-region
+        (funcall (get-text-property (car region) 'isearch-open-invisible) region)
+      (when fct-temp (funcall fct-temp region t)))))
+
 ;; This is called when exiting isearch. It closes the temporary
 ;; opened overlays, except the ones that contain the latest match.
 (defun isearch-clean-overlays ()
@@ -3508,6 +3566,10 @@ isearch-clean-overlays
     (mapc 'isearch-open-necessary-overlays isearch-opened-overlays)
     (setq isearch-opened-overlays nil)))
 
+(defun isearch-clean-regions ()
+  (when isearch-opened-regions
+    (mapc 'isearch-open-necessary-region isearch-opened-regions)
+    (setq isearch-opened-regions nil)))
 
 (defun isearch-intersects-p (start0 end0 start1 end1)
   "Return t if regions START0..END0 and START1..END1 intersect."
@@ -3535,6 +3597,21 @@ isearch-close-unnecessary-overlays
 	    (overlay-put ov 'invisible (overlay-get ov 'isearch-invisible))
 	    (overlay-put ov 'isearch-invisible nil)))))))
 
+(defun isearch-close-unnecessary-regions (begin end)
+  (let ((regions isearch-opened-regions))
+    (setq isearch-opened-regions nil)
+    (dolist (region regions)
+      (if (isearch-intersects-p begin end (car region) (cdr region))
+	  (push region isearch-opened-regions)
+	(let ((fct-temp (get-text-property (car region) 'isearch-open-invisible-temporary)))
+	  (if fct-temp
+	      (funcall fct-temp region t)
+            (with-silent-modifications
+              (put-text-property (car region)
+                                 (cdr region)
+                                 'invisible
+                                 (get-text-property (car region) 'isearch-invisible))
+              (remove-text-properties (car region) (cdr region) '('isearch-invisible nil)))))))))
 
 (defun isearch-range-invisible (beg end)
   "Return t if all the text from BEG to END is invisible."
@@ -3542,52 +3619,70 @@ isearch-range-invisible
     ;; Check that invisibility runs up to END.
     (save-excursion
       (goto-char beg)
-      (let (;; can-be-opened keeps track if we can open some overlays.
-	    (can-be-opened (eq search-invisible 'open))
+      (let (;; can-be-opened keeps track if we can open some hidden text.
+	    (can-be-opened (memq search-invisible '(open open-all)))
+            (region-can-be-opened (eq search-invisible 'open-all))
 	    ;; the list of overlays that could be opened
-	    (crt-overlays nil))
+	    (crt-overlays nil)
+            (crt-regions nil))
 	(when (and can-be-opened isearch-hide-immediately)
-	  (isearch-close-unnecessary-overlays beg end))
+	  (isearch-close-unnecessary-overlays beg end)
+          (when region-can-be-opened (isearch-close-unnecessary-regions beg end)))
 	;; If the following character is currently invisible,
 	;; skip all characters with that same `invisible' property value.
 	;; Do that over and over.
 	(while (and (< (point) end) (invisible-p (point)))
-	  (if (invisible-p (get-text-property (point) 'invisible))
-	      (progn
-		(goto-char (next-single-property-change (point) 'invisible
-							nil end))
-		;; if text is hidden by an `invisible' text property
-		;; we cannot open it at all.
-		(setq can-be-opened nil))
-	    (when can-be-opened
-	      (let ((overlays (overlays-at (point)))
-		    ov-list
-		    o
-		    invis-prop)
-		(while overlays
-		  (setq o (car overlays)
-			invis-prop (overlay-get o 'invisible))
-		  (if (invisible-p invis-prop)
-		      (if (overlay-get o 'isearch-open-invisible)
-			  (setq ov-list (cons o ov-list))
-			;; We found one overlay that cannot be
-			;; opened, that means the whole chunk
-			;; cannot be opened.
-			(setq can-be-opened nil)))
-		  (setq overlays (cdr overlays)))
-		(if can-be-opened
-		    ;; It makes sense to append to the open
-		    ;; overlays list only if we know that this is
-		    ;; t.
-		    (setq crt-overlays (append ov-list crt-overlays)))))
-	    (goto-char (next-overlay-change (point)))))
-	;; See if invisibility reaches up thru END.
-	(if (>= (point) end)
-	    (if (and can-be-opened (consp crt-overlays))
-		(progn
+	  (when (invisible-p (get-text-property (point) 'invisible))
+	    (if (and can-be-opened region-can-be-opened)
+                (let ((region (isearch--find-text-property-region (point) 'invisible)))
+                  ;; If the region info is passed to user-defined
+                  ;; function, tell that function what we actually
+                  ;; want to reveal.
+                  (when (< (car region) beg) (setcar region beg))
+                  (when (> (cdr region) end) (setcdr region end))
+                  (if (get-text-property (point) 'isearch-open-invisible)
+                      (push region crt-regions)
+                    (goto-char (next-single-property-change (point) 'invisible
+						            nil end))
+                    (setq can-be-opened nil)))
+	      (goto-char (next-single-property-change (point) 'invisible
+						      nil end))
+	      ;; if text is hidden by an `invisible' text property
+	      ;; we cannot open it at all.
+	      (setq can-be-opened nil)))
+	  (when can-be-opened
+	    (let ((overlays (overlays-at (point)))
+		  ov-list
+		  o
+		  invis-prop)
+	      (while overlays
+		(setq o (car overlays)
+		      invis-prop (overlay-get o 'invisible))
+		(if (invisible-p invis-prop)
+		    (if (overlay-get o 'isearch-open-invisible)
+			(setq ov-list (cons o ov-list))
+		      ;; We found one overlay that cannot be
+		      ;; opened, that means the whole chunk
+		      ;; cannot be opened.
+		      (setq can-be-opened nil)))
+		(setq overlays (cdr overlays)))
+	      (if can-be-opened
+		  ;; It makes sense to append to the open
+		  ;; overlays list only if we know that this is
+		  ;; t.
+		  (setq crt-overlays (append ov-list crt-overlays)))))
+	  (goto-char (next-single-char-property-change (point) 'invisible)))
+        ;; See if invisibility reaches up thru END.
+        (if (>= (point) end)
+	    (if (and can-be-opened (or (consp crt-overlays)
+                                       (consp crt-regions)))
+	        (progn
 		  (setq isearch-opened-overlays
-			(append isearch-opened-overlays crt-overlays))
+		        (append isearch-opened-overlays crt-overlays))
 		  (mapc 'isearch-open-overlay-temporary crt-overlays)
+                  (setq isearch-opened-regions
+                        (append isearch-opened-regions crt-regions))
+                  (mapc 'isearch-open-region-temporary crt-regions)
 		  nil)
 	      (setq isearch-hidden t)))))))
 

[-- Attachment #3: Type: text/plain, Size: 1206 bytes --]



Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> - text hidden via text properties is not well-handled by isearch. For
>>   overlays, isearch can temporary reveal the hidden text, which is not
>>   the case for text hidden using 'invisible text property.
>
> The recommending way to avoid this problem is to provide a patch which
> extends Isearch's handling of invisible text to text-properties.
> There's no reason to presume it would be hard to do.
>
>> - text properties are not buffer-local in indirect buffers. With
>>   overlays, it is possible to have define hidden text in basic and
>>   indirect buffer independently. However, text properties will always be
>>   shared and the text hidden in indirect buffer will automatically be
>>   hidden in the base buffer as well.
>
> And the way I recommend to avoid this problem is to stay clear of
> indirect buffers (which I consider as an attractive nuisance).
>
>
>         Stefan
>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: A question about overlays and performance.

[Jeff Norden]

> I am currently working on re-implementing org-mode folding from using
> overlays to text-properties.

It's interesting that you are working to change org's overlays back to
text properties, while I'm considering doing exactly the opposite for
the project that I've been playing with.  Do org buffers routinely wind
up with thousands of overlays?

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> ...
>> And the way I recommend to avoid this problem is to stay clear of
>> indirect buffers (which I consider as an attractive nuisance).

From what little I've seen of them, I'm quite inclined to agree with
Stefan's opinion about indirect buffers.  Is their main use in org to
provide multiple views of a buffer with different foldings?

I wonder if it might be possible to implement a "window-invisibility-spec",
that would override the buffer-spec when set.  It would have to be done
in C, not lisp.  You might even want to add a new field in the window
structure.  I have no real clue as to whether this is possible, feasible,
or a good idea.  But it might provide a nice way to display a single buffer
in two different windows with different levels of hidden text.  It seems
like it would be a lot simpler than using indirect buffers, but I'm
certainly no expert on this.

Regards,
-Jeff

Re: A question about overlays and performance.

[Yuan Fu]

[-- Attachment #1: Type: text/plain, Size: 384 bytes --]



> On Jul 28, 2020, at 12:43 PM, Jeff Norden <jnorden@tntech.edu> wrote:
> 
> From what little I've seen of them, I'm quite inclined to agree with
> Stefan's opinion about indirect buffers.  Is their main use in org to
> provide multiple views of a buffer with different foldings?

AFAIK org-sidebar relies on that: the sidebar showing headings is an indirect buffer.

Yuan

[-- Attachment #2: Type: text/html, Size: 3105 bytes --]

Re: A question about overlays and performance.

[Eric Abrahamsen]

Yuan Fu <casouri@gmail.com> writes:

>> On Jul 28, 2020, at 12:43 PM, Jeff Norden <jnorden@tntech.edu> wrote:
>> 
>> From what little I've seen of them, I'm quite inclined to agree with
>> Stefan's opinion about indirect buffers.  Is their main use in org to
>> provide multiple views of a buffer with different foldings?
>
> AFAIK org-sidebar relies on that: the sidebar showing headings is an indirect buffer.

An important use is the capture system: when capturing text to be
inserted in an Org buffer, the text is entered in an indirect buffer, so
that it's already in place as the user is typing. The old system used a
temporary buffer, then inserted the text on finalization.

I don't remember the whole history, but I do remember that switching to
indirect buffers solved several issues, and ended up being a pretty
clean solution to the problem. I think they still have their uses.

Re: A question about overlays and performance.

[Juri Linkov]

> I am currently working on re-implementing org-mode folding from using
> overlays to text-properties.

Any chance to fix also bug#40808? It's still open because there is
no solution for isearching invisible text in org-mode.

Here's a short test case that demonstrates the problem in org-mode:

0. emacs -Q
1. visit emacs/etc/ORG-NEWS
2. isearch for "http": C-s http C-s C-s C-s ...

it eventually stops at org links where "http" is invisible, such as

  [[https://orgmode.org/worg/library-of-babel.html][here]]

The problem is that org-mode puts the text property 'invisible'
only when the link is scrolled into view, I guess it puts the property
using font-lock.

So it seems org-mode needs to use font-lock-ensure on the line
with the next search hit before calling isearch-filter-predicate
on the found occurence.

Re: A question about overlays and performance.

[Ihor Radchenko]

> Any chance to fix also bug#40808? It's still open because there is
> no solution for isearching invisible text in org-mode.

Thanks for pointing out this bug. Searching actually works fine on my
feature branch + patched isearch (though it is actually an unintended
side-effect). However, (setq isearch-lazy-count t) leaves the last
headline open. I will need to investigate this. 

> The problem is that org-mode puts the text property 'invisible'
> only when the link is scrolled into view, I guess it puts the property
> using font-lock.
>
> So it seems org-mode needs to use font-lock-ensure on the line
> with the next search hit before calling isearch-filter-predicate
> on the found occurence.

I am currently undecided about searchability of hidden text in links. Do
I understand correctly that you prefer the hidden parts of links to be
searchable? Either way is possible. If hidden parts are searchable, it
will be even possible to show them temporarily.

Best,
Ihor


Juri Linkov <juri@linkov.net> writes:

>> I am currently working on re-implementing org-mode folding from using
>> overlays to text-properties.
>
> Any chance to fix also bug#40808? It's still open because there is
> no solution for isearching invisible text in org-mode.
>
> Here's a short test case that demonstrates the problem in org-mode:
>
> 0. emacs -Q
> 1. visit emacs/etc/ORG-NEWS
> 2. isearch for "http": C-s http C-s C-s C-s ...
>
> it eventually stops at org links where "http" is invisible, such as
>
>   [[https://orgmode.org/worg/library-of-babel.html][here]]
>
> The problem is that org-mode puts the text property 'invisible'
> only when the link is scrolled into view, I guess it puts the property
> using font-lock.
>
> So it seems org-mode needs to use font-lock-ensure on the line
> with the next search hit before calling isearch-filter-predicate
> on the found occurence.

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: A question about overlays and performance.

[Ihor Radchenko]

> It's interesting that you are working to change org's overlays back to
> text properties, while I'm considering doing exactly the opposite for
> the project that I've been playing with.

Org is using overlays to fold anything right now, same as outline-mode.
What do you mean by "exactly the opposite"?

If you need some specific feature that is lacking in text properties,
but present in overlays, let me know here or jump into the patch
discussion:
https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127740.html 

> Do org buffers routinely wind up with thousands of overlays?

It is very common when you have an org file with thousands of headlines
each containing a [folded] property drawer. I provided an example org
file in the patch discussion:
https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127740.html 

> From what little I've seen of them, I'm quite inclined to agree with
> Stefan's opinion about indirect buffers.  Is their main use in org to
> provide multiple views of a buffer with different foldings?

Another common use is narrowing to subtrees, so that you have two
buffers with different narrowing for the same org file. It is not
possible within a single buffer, as far I as know.

> I wonder if it might be possible to implement a "window-invisibility-spec",
> that would override the buffer-spec when set.  It would have to be done
> in C, not lisp.  You might even want to add a new field in the window
> structure.  I have no real clue as to whether this is possible, feasible,
> or a good idea.  But it might provide a nice way to display a single buffer
> in two different windows with different levels of hidden text.  It seems
> like it would be a lot simpler than using indirect buffers, but I'm
> certainly no expert on this.

I do not think that it is a good idea for this spefic case.

If I understand correctly, the only advantage of such approach would the
fact that you do not need to create an indirect buffer. However, there
is a problem with "window-local" approach. Anything that is
"window-local" implies that it can affect **any** buffer shown in that
window. So, we will need to handle buffer changes without adding any
extra benefit in comparison with indirect buffer approach. Not to
mention, that the ability to have independent narrow states will be
lost. 

Best,
Ihor

Jeff Norden <jnorden@tntech.edu> writes:

>> I am currently working on re-implementing org-mode folding from using
>> overlays to text-properties.
>
> It's interesting that you are working to change org's overlays back to
> text properties, while I'm considering doing exactly the opposite for
> the project that I've been playing with.  Do org buffers routinely wind
> up with thousands of overlays?
>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> ...
>>> And the way I recommend to avoid this problem is to stay clear of
>>> indirect buffers (which I consider as an attractive nuisance).
>
> From what little I've seen of them, I'm quite inclined to agree with
> Stefan's opinion about indirect buffers.  Is their main use in org to
> provide multiple views of a buffer with different foldings?
>
> I wonder if it might be possible to implement a "window-invisibility-spec",
> that would override the buffer-spec when set.  It would have to be done
> in C, not lisp.  You might even want to add a new field in the window
> structure.  I have no real clue as to whether this is possible, feasible,
> or a good idea.  But it might provide a nice way to display a single buffer
> in two different windows with different levels of hidden text.  It seems
> like it would be a lot simpler than using indirect buffers, but I'm
> certainly no expert on this.
>
> Regards,
> -Jeff

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: A question about overlays and performance.

[Jeff Norden]

>> ... inclined to agree with Stefan's opinion about indirect buffers.
> Another common use is narrowing to subtrees, so that you have two
> buffers with different narrowing for the same org file. It is not
> possible within a single buffer, as far I as know.

So the org case for indirect buffers is messier than I had imagined, which
shouldn't surprise me.  I'm a very casual user of org - just as an
alternative to outline-mode for the cycling and the intuitive use of the
tab key.

> What do you mean by "exactly the opposite"?

I was working on changing some text-property based code to use overlays
instead, but I'm having second thoughts.  Overlays simplified some things,
but not as much as I thought they would.  I also ran into a couple of snags:

1) There don't seem to be 'char-property' versions of text-property-any or
text-property-not-all.  I guess you could do this in lisp with a loop using
next-char-property-change, but that works against my original goal of
simplifying things.

2) A simple option that you have for protecting hidden text when using
text-properties is to temporarily mark it as read-only, e.g.:

  (add-text-properties beg end '(invisible t read-only t))

Of course, you need to make sure you remove both properties when you
un-hide (and take into account that you are now changing read-only text).
It also means that you can't kill or delete a region that contains hidden
text without un-hiding it first.  That may or may not be a good thing,
depending on your perspective.  There might also be other side-effects that
I'm not thinking of right now.

---

I just looked at org a bit more and discovered org-catch-invisible-edits.
It works, in part, because org re-binds the keys that usually call
delete-char and delete-backward-char.  I tried it, and found that even with
catch-invisible-edits set to 'error, if you put the cursor right before an
ellipsis and type M-d, the first word in the hidden text is silently
deleted.  I wouldn't say this is a bug or problem, since it is hard to
imagine someone doing that accidentally.  I just thought it was interesting.

Best regards,
-Jeff

Re: A question about overlays and performance.

[Juri Linkov]

>> Any chance to fix also bug#40808? It's still open because there is
>> no solution for isearching invisible text in org-mode.
>
> Thanks for pointing out this bug. Searching actually works fine on my
> feature branch + patched isearch (though it is actually an unintended
> side-effect). However, (setq isearch-lazy-count t) leaves the last
> headline open. I will need to investigate this. 

Leaving the last headline open is a known bug fixed in
https://debbugs.gnu.org/40808#8
Here it's explaned:

  "And also it fixes an old bug existed in previous Emacs versions
  where lazy-highlight didn't update lazy-highlighting when a hidden outline
  was automatically opened at the end of the file (I noticed this bug only now).
  So the condition ‘(eq search-invisible 'open)’ fixes an old bug..."

>> The problem is that org-mode puts the text property 'invisible'
>> only when the link is scrolled into view, I guess it puts the property
>> using font-lock.
>>
>> So it seems org-mode needs to use font-lock-ensure on the line
>> with the next search hit before calling isearch-filter-predicate
>> on the found occurence.
>
> I am currently undecided about searchability of hidden text in links. Do
> I understand correctly that you prefer the hidden parts of links to be
> searchable? Either way is possible. If hidden parts are searchable, it
> will be even possible to show them temporarily.

The hidden parts of links should be searchable when the value of
'search-invisible' is 't' ("Match hidden text").