unofficial mirror of emacs-orgmode@gnu.org
 help / color / Atom feed
* [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
@ 2020-04-24  6:55 Ihor Radchenko
  2020-04-24  8:02 ` Nicolas Goaziou
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-04-24  6:55 UTC (permalink / raw)
  To: emacs-orgmode

Emacs becomes very slow when opening and moving around huge org files
with many drawers. I have reported this issue last year in bug-gnu-emacs
[1] and there have been other reports on the same problem in the
internet [2]. You can easily see this problem using the attached file if
you try to move down the lines when all the headings are folded. Moving
a single line down may take over 10 seconds in the file.

According to the reply to my initial emacs bug report [1], the reasons
of performance degradation is huge number of overlays created by org in
the PROPERTY and LOGBOOK drawers. Emacs must loop over all those
overlays every time it calculates where the next visible line is
located. So, one way to improve the performance would be reducing the
number of overlays.

I have been looking into usage of overlays in the org-mode code recently
and tried to redefine org-flag-region to use text properties instead of
overlays:

#+begin_src emacs-lisp
(defun org-flag-region (from to flag spec)
  "Hide or show lines from FROM to TO, according to FLAG.
SPEC is the invisibility spec, as a symbol."
  (pcase spec
    ;; outlines must still use overlays because they rely on
    ;; 'reveal-toggle-invisible feature from reveal.el
    ;; That only works for overlays
    ('outline 
     (remove-overlays from to 'invisible spec)
     ;; Use `front-advance' since text right before to the beginning of
     ;; the overlay belongs to the visible line than to the contents.
     (when flag
       (let ((o (make-overlay from to nil 'front-advance)))
	 (overlay-put o 'evaporate t)
	 (overlay-put o 'invisible spec)
	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
    (_
     (let ((inhibit-modification-hooks t))
       (remove-text-properties from to '(invisible nil))
       ;; Use `front-advance' since text right before to the beginning of
       ;; the overlay belongs to the visible line than to the contents.
       (when flag
	 (put-text-property from to 'rear-non-sticky t)
	 (put-text-property from to 'front-sticky t)
	 (put-text-property from to 'invisible spec)
         ;; no idea if 'isearch-open-invisible is needed for text
         ;; properties 
	 ;; (overlay-put o 'isearch-open-invisible #'delete-overlay)
	 )))))
#+end_src

To my surprise, the patch did not break org to unusable state and
the performance on the sample org file [3] improved drastically. You can
try by yourself!

However, this did introduce some visual glitches with drawer display.
Though drawers can still be folded/unfolded with <tab>, they are not
folded on org-mode startup for some reason (can be fixed by running
(org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers)
are unfolded for no apparent reason sometimes. A blind guess is that it
is something to do with lack of 'isearch-open-invisible, which I am not
sure how to set via text properties.

Any thoughts about the use of text properties or about the patch
suggestion are welcome.  

Best,
Ihor

[1] https://lists.gnu.org/archive/html/bug-gnu-emacs/2019-04/msg01387.html
[2] https://www.reddit.com/r/orgmode/comments/e9p84n/scaling_org_better_to_use_more_medsize_files_or/
[3] See the attached org file in my Emacs bug report: https://lists.gnu.org/archive/html/bug-gnu-emacs/2019-04/txte6kQp35VOm.txt

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg



^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-04-24  6:55 [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers Ihor Radchenko
@ 2020-04-24  8:02 ` Nicolas Goaziou
  2020-04-25  0:29   ` stardiviner
  2020-04-26 16:04   ` Ihor Radchenko
  0 siblings, 2 replies; 55+ messages in thread
From: Nicolas Goaziou @ 2020-04-24  8:02 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> To my surprise, the patch did not break org to unusable state and
> the performance on the sample org file [3] improved drastically. You can
> try by yourself!

It is not a surprise, really. Text properties are much faster than
overlays, and very close to them features-wise. They are a bit more
complex to handle, however.

> However, this did introduce some visual glitches with drawer display.
> Though drawers can still be folded/unfolded with <tab>, they are not
> folded on org-mode startup for some reason (can be fixed by running
> (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers)
> are unfolded for no apparent reason sometimes. A blind guess is that it
> is something to do with lack of 'isearch-open-invisible, which I am not
> sure how to set via text properties.

You cannot. You may however mimic it with `cursor-sensor-functions' text
property. These assume Cursor Sensor minor mode is active, tho.
I haven't tested it, but I assume it would slow down text properties
a bit, too, but hopefully not as much as overlays.

Note there are clear advantages using text properties. For example, when
you move contents around, text properties are preserved. So there's no
more need for the `org-cycle-hide-drawer' dance, i.e., it is not
necessary anymore to re-hide drawers.

> Any thoughts about the use of text properties or about the patch
> suggestion are welcome.  

Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth
experimenting with `cursor-sensor-functions'.

We could also use text properties for property drawers, and overlays for
regular ones. This might give us a reasonable speed-up with an
acceptable feature trade-off.

Anyway, the real fix should come from Emacs itself. There are ways to
make overlays faster. These ways have already been discussed on the
Emacs devel mailing list, but no one implemented them. It is a bit sad
that we have to find workarounds for that.

Regards,

-- 
Nicolas Goaziou


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-04-24  8:02 ` Nicolas Goaziou
@ 2020-04-25  0:29   ` stardiviner
  2020-04-26 16:04   ` Ihor Radchenko
  1 sibling, 0 replies; 55+ messages in thread
From: stardiviner @ 2020-04-25  0:29 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode, Ihor Radchenko

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256


Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> To my surprise, the patch did not break org to unusable state and
>> the performance on the sample org file [3] improved drastically. You can
>> try by yourself!
>
> It is not a surprise, really. Text properties are much faster than
> overlays, and very close to them features-wise. They are a bit more
> complex to handle, however.
>
>> However, this did introduce some visual glitches with drawer display.
>> Though drawers can still be folded/unfolded with <tab>, they are not
>> folded on org-mode startup for some reason (can be fixed by running
>> (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers)
>> are unfolded for no apparent reason sometimes. A blind guess is that it
>> is something to do with lack of 'isearch-open-invisible, which I am not
>> sure how to set via text properties.
>
> You cannot. You may however mimic it with `cursor-sensor-functions' text
> property. These assume Cursor Sensor minor mode is active, tho.
> I haven't tested it, but I assume it would slow down text properties
> a bit, too, but hopefully not as much as overlays.
>
> Note there are clear advantages using text properties. For example, when
> you move contents around, text properties are preserved. So there's no
> more need for the `org-cycle-hide-drawer' dance, i.e., it is not
> necessary anymore to re-hide drawers.
>
>> Any thoughts about the use of text properties or about the patch
>> suggestion are welcome.  
>
> Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth
> experimenting with `cursor-sensor-functions'.
>
> We could also use text properties for property drawers, and overlays for
> regular ones. This might give us a reasonable speed-up with an
> acceptable feature trade-off.

That's great, making Org Mode faster will be great. (Even thought I have not
found big performance problem on Org Mode yet.) I like Thor's try.

This indeed is is an acceptable feature trade-off, if only related to
`isearch-open-invisible'.

>
> Anyway, the real fix should come from Emacs itself. There are ways to
> make overlays faster. These ways have already been discussed on the
> Emacs devel mailing list, but no one implemented them. It is a bit sad
> that we have to find workarounds for that.
>
> Regards,


- -- 
[ stardiviner ]
       I try to make every word tell the meaning what I want to express.

       Blog: https://stardiviner.github.io/
       IRC(freenode): stardiviner, Matrix: stardiviner
       GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
      
-----BEGIN PGP SIGNATURE-----

iQFIBAEBCAAyFiEE8J9lDX1nSBmJJZFAG13xyVromsMFAl6jhG0UHG51bWJjaGls
ZEBnbWFpbC5jb20ACgkQG13xyVromsPHDAf+OVnhOq5H5MYm1/RK+9xSzwAT6qc8
ajSNVNzI31q6CIesvO65GoiZ3Rpaiq/O31B9JQ1mTyXvyX81tFecKrDpsrqIc/bR
Xo3Z4dCXzCbRKD1861t4tcphtPBk+rABpl83YpXafYNDKHnp2MuWSheV0ogF7LYd
6HWCl9D351onGAHGcebXEUTvvDiqLGx5qVnrpjomH00uCj5RoSI4cpdzXydBcIYY
B6lDvsat8AHhvbPXqJc4PHOd4hPtNVehWyPfOGaAXhp/pS0y+c4cJMbHjXCwFCkj
r8bUfdK+ZyMubNiboNI9xO8EwINvZLl+C5Lt5siYs/v2mrt1+UiVrxYWTw==
=dnH4
-----END PGP SIGNATURE-----


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-04-24  8:02 ` Nicolas Goaziou
  2020-04-25  0:29   ` stardiviner
@ 2020-04-26 16:04   ` Ihor Radchenko
  2020-05-04 16:56     ` Karl Voit
                       ` (2 more replies)
  1 sibling, 3 replies; 55+ messages in thread
From: Ihor Radchenko @ 2020-04-26 16:04 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

> You cannot. You may however mimic it with `cursor-sensor-functions' text
> property. These assume Cursor Sensor minor mode is active, tho.
> I haven't tested it, but I assume it would slow down text properties
> a bit, too, but hopefully not as much as overlays.

Unfortunately, isearch sets inhibit-point-motion-hooks to non-nil
internally. Anyway, I came up with some workaround, which seems to work
(see below). Though it would be better if isearch supported hidden text
in addition to overlays.

> Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth
> experimenting with `cursor-sensor-functions'.

So far, I came up with the following partial solution searching and
showing hidden text.

;; Unfortunately isearch, sets inhibit-point-motion-hooks and we
;; cannot even use cursor-sensor-functions as a workaround
;; I used a less ideas approach with advice to isearch-search-string as
;; a workaround 

(defun org-find-text-property-region (pos prop)
  "Find a region containing PROP text property around point POS."
  (require 'org-macs) ;; org-with-point-at
  (org-with-point-at pos
    (let* ((beg (and (get-text-property pos prop) pos))
	   (end beg))
      (when beg
	(setq beg (or (previous-single-property-change pos prop)
		      beg))
	(setq end (or (next-single-property-change pos prop)
		      end))
        (unless (equal beg end)
          (cons beg end))))))

;; :FIXME: re-hide properties when point moves away
(define-advice isearch-search-string (:after (&rest _) put-overlay)
  "Reveal hidden text at point."
  (when-let ((region (org-find-text-property-region (point) 'invisible)))
    (with-silent-modifications
      (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) 'invisible)))
      (remove-text-properties (car region) (cdr region) '(invisible nil))))

;; this seems to be unstable, but I cannot figure out why
(defun org-restore-invisibility-specs (&rest _)
  ""
   (let ((pos (point-min)))
     (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-max))
       (when-let ((region (org-find-text-property-region pos 'org-invisible)))
	   (with-silent-modifications
	     (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-invisible))
	     (remove-text-properties (car region) (cdr region) '(org-invisible nil)))))))

(add-hook 'post-command-hook #'org-restore-invisibility-specs)

(defun org-flag-region (from to flag spec)
  "Hide or show lines from FROM to TO, according to FLAG.
SPEC is the invisibility spec, as a symbol."
  (pcase spec
    ('outline
     (remove-overlays from to 'invisible spec)
     ;; Use `front-advance' since text right before to the beginning of
     ;; the overlay belongs to the visible line than to the contents.
     (when flag
       (let ((o (make-overlay from to nil 'front-advance)))
	 (overlay-put o 'evaporate t)
	 (overlay-put o 'invisible spec)
	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
    (_
     (with-silent-modifications
       (remove-text-properties from to '(invisible nil))
       (when flag
	 (put-text-property from to 'invisible spec)
	 )))))

;; This normally deletes invisible text property. We do not want this now.
(defun org-unfontify-region (beg end &optional _maybe_loudly)
  "Remove fontification and activation overlays from links."
  (font-lock-default-unfontify-region beg end)
  (let* ((buffer-undo-list t)
	 (inhibit-read-only t) (inhibit-point-motion-hooks t)
	 (inhibit-modification-hooks t)
	 deactivate-mark buffer-file-name buffer-file-truename)
    (decompose-region beg end)
    (remove-text-properties beg end
			    '(mouse-face t keymap t org-linked-text t
					 ;; Do not remove invisible during fontification					 
					 ;; invisible t
                                         intangible t
					 org-emphasis t))
    (org-remove-font-lock-display-properties beg end)))

> Anyway, the real fix should come from Emacs itself. There are ways to
> make overlays faster. These ways have already been discussed on the
> Emacs devel mailing list, but no one implemented them. It is a bit sad
> that we have to find workarounds for that.

I guess that it is a very old story starting from the times when XEmacs
was a thing [1]. I recently heard about binary tree implementation of
overlays (there should be a branch in emacs git repo) [2], but there was
no update on that branch for a while. So, I do not have much hope on
Emacs implementing efficient overlay access in the near future. (And I
have problems with huge org files already).

[1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/
[2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html


Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> To my surprise, the patch did not break org to unusable state and
>> the performance on the sample org file [3] improved drastically. You can
>> try by yourself!
>
> It is not a surprise, really. Text properties are much faster than
> overlays, and very close to them features-wise. They are a bit more
> complex to handle, however.
>
>> However, this did introduce some visual glitches with drawer display.
>> Though drawers can still be folded/unfolded with <tab>, they are not
>> folded on org-mode startup for some reason (can be fixed by running
>> (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers)
>> are unfolded for no apparent reason sometimes. A blind guess is that it
>> is something to do with lack of 'isearch-open-invisible, which I am not
>> sure how to set via text properties.
>
> You cannot. You may however mimic it with `cursor-sensor-functions' text
> property. These assume Cursor Sensor minor mode is active, tho.
> I haven't tested it, but I assume it would slow down text properties
> a bit, too, but hopefully not as much as overlays.
>
> Note there are clear advantages using text properties. For example, when
> you move contents around, text properties are preserved. So there's no
> more need for the `org-cycle-hide-drawer' dance, i.e., it is not
> necessary anymore to re-hide drawers.
>
>> Any thoughts about the use of text properties or about the patch
>> suggestion are welcome.  
>
> Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth
> experimenting with `cursor-sensor-functions'.
>
> We could also use text properties for property drawers, and overlays for
> regular ones. This might give us a reasonable speed-up with an
> acceptable feature trade-off.
>
> Anyway, the real fix should come from Emacs itself. There are ways to
> make overlays faster. These ways have already been discussed on the
> Emacs devel mailing list, but no one implemented them. It is a bit sad
> that we have to find workarounds for that.
>
> Regards,
>
> -- 
> Nicolas Goaziou

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-04-26 16:04   ` Ihor Radchenko
@ 2020-05-04 16:56     ` Karl Voit
  2020-05-07  7:18       ` Karl Voit
  2020-05-09 15:43       ` Ihor Radchenko
  2020-05-07 11:04     ` Christian Heinrich
  2020-05-08 16:38     ` Nicolas Goaziou
  2 siblings, 2 replies; 55+ messages in thread
From: Karl Voit @ 2020-05-04 16:56 UTC (permalink / raw)
  To: emacs-orgmode

Hi Ihor,

* Ihor Radchenko <yantar92@gmail.com> wrote:
>
> So far, I came up with the following partial solution searching and
> showing hidden text.
>
> (defun org-find-text-property-region (pos prop)
> (define-advice isearch-search-string (:after (&rest _) put-overlay)
> (defun org-restore-invisibility-specs (&rest _)
> (add-hook 'post-command-hook #'org-restore-invisibility-specs)
> (defun org-flag-region (from to flag spec)
> (defun org-unfontify-region (beg end &optional _maybe_loudly)

After a couple of hours working with these patches, my feedback is
very positive. Besides some visual glitches when creating a new
heading with org-expiry-insinuate activated (which automatically
adds :CREATED: properties), I could not detect any side-effect so
far (will keep testing).

The visual glitch looks like that:

:PROPERTIES:X:CREATED:  [2020-05-04 Mon 18>54]
X

... with "X" being my character that symbolizes collapsed content.
The way it looked without the patch was a simple collapsed property
drawer.

To me, this is acceptable considering the huge performance gain I
got.

THANK YOU VERY MUCH! I can't remember where I had this way of
working within my large Org files[3] since ages.

>> Anyway, the real fix should come from Emacs itself. There are ways to
>> make overlays faster. These ways have already been discussed on the
>> Emacs devel mailing list, but no one implemented them. It is a bit sad
>> that we have to find workarounds for that.
>
> I guess that it is a very old story starting from the times when XEmacs
> was a thing [1]. I recently heard about binary tree implementation of
> overlays (there should be a branch in emacs git repo) [2], but there was
> no update on that branch for a while. So, I do not have much hope on
> Emacs implementing efficient overlay access in the near future. (And I
> have problems with huge org files already).

I can not express how this also reflects my personal situation.

> [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/
> [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html

[3] https://karl-voit.at/2020/05/03/current-org-files

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/



^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-04 16:56     ` Karl Voit
@ 2020-05-07  7:18       ` Karl Voit
  2020-05-09 15:43       ` Ihor Radchenko
  1 sibling, 0 replies; 55+ messages in thread
From: Karl Voit @ 2020-05-07  7:18 UTC (permalink / raw)
  To: emacs-orgmode

Hi,

* Karl Voit <devnull@Karl-Voit.at> wrote:
> Hi Ihor,
>
> * Ihor Radchenko <yantar92@gmail.com> wrote:
>>
>> So far, I came up with the following partial solution searching and
>> showing hidden text.
>>
>> (defun org-find-text-property-region (pos prop)
>> (define-advice isearch-search-string (:after (&rest _) put-overlay)
>> (defun org-restore-invisibility-specs (&rest _)
>> (add-hook 'post-command-hook #'org-restore-invisibility-specs)
>> (defun org-flag-region (from to flag spec)
>> (defun org-unfontify-region (beg end &optional _maybe_loudly)
>
> After a couple of hours working with these patches, my feedback is
> very positive. Besides some visual glitches when creating a new
> heading with org-expiry-insinuate activated (which automatically
> adds :CREATED: properties), I could not detect any side-effect so
> far (will keep testing).
>
> The visual glitch looks like that:
>
>:PROPERTIES:X:CREATED:  [2020-05-04 Mon 18>54]
> X
>
> ... with "X" being my character that symbolizes collapsed content.
> The way it looked without the patch was a simple collapsed property
> drawer.

Here some hard numbers to demonstrate the impact:

my-org-agenda: from 11-16s down to 10 -> not much of a difference

helm-org-contacts-refresh-cache: 29-59s down to 2½ -> HUGE

Emacs boot time: 50-65s down to 10 -> HUGE

Navigating the cursor in large Org files -> HUGE subjective impact

>>> Anyway, the real fix should come from Emacs itself. There are ways to
>>> make overlays faster. These ways have already been discussed on the
>>> Emacs devel mailing list, but no one implemented them. It is a bit sad
>>> that we have to find workarounds for that.
>>
>> I guess that it is a very old story starting from the times when XEmacs
>> was a thing [1]. I recently heard about binary tree implementation of
>> overlays (there should be a branch in emacs git repo) [2], but there was
>> no update on that branch for a while. So, I do not have much hope on
>> Emacs implementing efficient overlay access in the near future. (And I
>> have problems with huge org files already).
>
> I can not express how this also reflects my personal situation.
>
>> [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/
>> [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html
>
> [3] https://karl-voit.at/2020/05/03/current-org-files
>

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/



^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-04-26 16:04   ` Ihor Radchenko
  2020-05-04 16:56     ` Karl Voit
@ 2020-05-07 11:04     ` Christian Heinrich
  2020-05-09 15:46       ` Ihor Radchenko
  2020-05-08 16:38     ` Nicolas Goaziou
  2 siblings, 1 reply; 55+ messages in thread
From: Christian Heinrich @ 2020-05-07 11:04 UTC (permalink / raw)
  To: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 8181 bytes --]

Hi,

thanks for your (initial) patch! I traced another error down today and found your code by chance. I
tested it on an org-drill file that I had (with over 3500 items and hence 3500 drawers) and this
patch helps *a lot* already. (Performance broke in 4403d4685e19fb99ba9bfec2bd4ff6781c66981f when
outline-flag-region was replaced with org-flag-region, as drawers are no longer opened using
outline-show-all which I had to use anyways to deal with my huge file.)

I am not sure I understand how your follow-up code (below) needs to be incorporated. Would you mind
sending a patch file? I hope that this ends up in the master branch at some point.

Thanks again!
Christian

On Mon, 2020-04-27 at 00:04 +0800, Ihor Radchenko wrote:
> > You cannot. You may however mimic it with `cursor-sensor-functions' text
> > property. These assume Cursor Sensor minor mode is active, tho.
> > I haven't tested it, but I assume it would slow down text properties
> > a bit, too, but hopefully not as much as overlays.
> 
> Unfortunately, isearch sets inhibit-point-motion-hooks to non-nil
> internally. Anyway, I came up with some workaround, which seems to work
> (see below). Though it would be better if isearch supported hidden text
> in addition to overlays.
> 
> > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth
> > experimenting with `cursor-sensor-functions'.
> 
> So far, I came up with the following partial solution searching and
> showing hidden text.
> 
> ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we
> ;; cannot even use cursor-sensor-functions as a workaround
> ;; I used a less ideas approach with advice to isearch-search-string as
> ;; a workaround 
> 
> (defun org-find-text-property-region (pos prop)
>   "Find a region containing PROP text property around point POS."
>   (require 'org-macs) ;; org-with-point-at
>   (org-with-point-at pos
>     (let* ((beg (and (get-text-property pos prop) pos))
> 	   (end beg))
>       (when beg
> 	(setq beg (or (previous-single-property-change pos prop)
> 		      beg))
> 	(setq end (or (next-single-property-change pos prop)
> 		      end))
>         (unless (equal beg end)
>           (cons beg end))))))
> 
> ;; :FIXME: re-hide properties when point moves away
> (define-advice isearch-search-string (:after (&rest _) put-overlay)
>   "Reveal hidden text at point."
>   (when-let ((region (org-find-text-property-region (point) 'invisible)))
>     (with-silent-modifications
>       (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point)
> 'invisible)))
>       (remove-text-properties (car region) (cdr region) '(invisible nil))))
> 
> ;; this seems to be unstable, but I cannot figure out why
> (defun org-restore-invisibility-specs (&rest _)
>   ""
>    (let ((pos (point-min)))
>      (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-
> max))
>        (when-let ((region (org-find-text-property-region pos 'org-invisible)))
> 	   (with-silent-modifications
> 	     (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-
> invisible))
> 	     (remove-text-properties (car region) (cdr region) '(org-invisible nil)))))))
> 
> (add-hook 'post-command-hook #'org-restore-invisibility-specs)
> 
> (defun org-flag-region (from to flag spec)
>   "Hide or show lines from FROM to TO, according to FLAG.
> SPEC is the invisibility spec, as a symbol."
>   (pcase spec
>     ('outline
>      (remove-overlays from to 'invisible spec)
>      ;; Use `front-advance' since text right before to the beginning of
>      ;; the overlay belongs to the visible line than to the contents.
>      (when flag
>        (let ((o (make-overlay from to nil 'front-advance)))
> 	 (overlay-put o 'evaporate t)
> 	 (overlay-put o 'invisible spec)
> 	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
>     (_
>      (with-silent-modifications
>        (remove-text-properties from to '(invisible nil))
>        (when flag
> 	 (put-text-property from to 'invisible spec)
> 	 )))))
> 
> ;; This normally deletes invisible text property. We do not want this now.
> (defun org-unfontify-region (beg end &optional _maybe_loudly)
>   "Remove fontification and activation overlays from links."
>   (font-lock-default-unfontify-region beg end)
>   (let* ((buffer-undo-list t)
> 	 (inhibit-read-only t) (inhibit-point-motion-hooks t)
> 	 (inhibit-modification-hooks t)
> 	 deactivate-mark buffer-file-name buffer-file-truename)
>     (decompose-region beg end)
>     (remove-text-properties beg end
> 			    '(mouse-face t keymap t org-linked-text t
> 					 ;; Do not remove invisible during fontification			
> 		 
> 					 ;; invisible t
>                                          intangible t
> 					 org-emphasis t))
>     (org-remove-font-lock-display-properties beg end)))
> 
> > Anyway, the real fix should come from Emacs itself. There are ways to
> > make overlays faster. These ways have already been discussed on the
> > Emacs devel mailing list, but no one implemented them. It is a bit sad
> > that we have to find workarounds for that.
> 
> I guess that it is a very old story starting from the times when XEmacs
> was a thing [1]. I recently heard about binary tree implementation of
> overlays (there should be a branch in emacs git repo) [2], but there was
> no update on that branch for a while. So, I do not have much hope on
> Emacs implementing efficient overlay access in the near future. (And I
> have problems with huge org files already).
> 
> [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/
> [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html
> 
> 
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
> 
> > Hello,
> > 
> > Ihor Radchenko <yantar92@gmail.com> writes:
> > 
> > > To my surprise, the patch did not break org to unusable state and
> > > the performance on the sample org file [3] improved drastically. You can
> > > try by yourself!
> > 
> > It is not a surprise, really. Text properties are much faster than
> > overlays, and very close to them features-wise. They are a bit more
> > complex to handle, however.
> > 
> > > However, this did introduce some visual glitches with drawer display.
> > > Though drawers can still be folded/unfolded with <tab>, they are not
> > > folded on org-mode startup for some reason (can be fixed by running
> > > (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers)
> > > are unfolded for no apparent reason sometimes. A blind guess is that it
> > > is something to do with lack of 'isearch-open-invisible, which I am not
> > > sure how to set via text properties.
> > 
> > You cannot. You may however mimic it with `cursor-sensor-functions' text
> > property. These assume Cursor Sensor minor mode is active, tho.
> > I haven't tested it, but I assume it would slow down text properties
> > a bit, too, but hopefully not as much as overlays.
> > 
> > Note there are clear advantages using text properties. For example, when
> > you move contents around, text properties are preserved. So there's no
> > more need for the `org-cycle-hide-drawer' dance, i.e., it is not
> > necessary anymore to re-hide drawers.
> > 
> > > Any thoughts about the use of text properties or about the patch
> > > suggestion are welcome.  
> > 
> > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth
> > experimenting with `cursor-sensor-functions'.
> > 
> > We could also use text properties for property drawers, and overlays for
> > regular ones. This might give us a reasonable speed-up with an
> > acceptable feature trade-off.
> > 
> > Anyway, the real fix should come from Emacs itself. There are ways to
> > make overlays faster. These ways have already been discussed on the
> > Emacs devel mailing list, but no one implemented them. It is a bit sad
> > that we have to find workarounds for that.
> > 
> > Regards,
> > 
> > -- 
> > Nicolas Goaziou

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-04-26 16:04   ` Ihor Radchenko
  2020-05-04 16:56     ` Karl Voit
  2020-05-07 11:04     ` Christian Heinrich
@ 2020-05-08 16:38     ` Nicolas Goaziou
  2020-05-09 13:58       ` Nicolas Goaziou
  2020-05-09 15:40       ` Ihor Radchenko
  2 siblings, 2 replies; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-08 16:38 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we
> ;; cannot even use cursor-sensor-functions as a workaround
> ;; I used a less ideas approach with advice to isearch-search-string as
> ;; a workaround 

OK.

> (defun org-find-text-property-region (pos prop)
>   "Find a region containing PROP text property around point POS."
>   (require 'org-macs) ;; org-with-point-at
>   (org-with-point-at pos

Do we really need that since every function has a POS argument anyway?
Is it for the `widen' part?

>     (let* ((beg (and (get-text-property pos prop) pos))
> 	   (end beg))
>       (when beg
> 	(setq beg (or (previous-single-property-change pos prop)
> 		      beg))

Shouldn't fall-back be (point-min)?

> 	(setq end (or (next-single-property-change pos prop)
> 		      end))

And (point-max) here?

>         (unless (equal beg end)

Nitpick: `equal' -> =

>           (cons beg end))))))

> ;; :FIXME: re-hide properties when point moves away
> (define-advice isearch-search-string (:after (&rest _) put-overlay)
>   "Reveal hidden text at point."
>   (when-let ((region (org-find-text-property-region (point) 'invisible)))
>     (with-silent-modifications
>       (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) 'invisible)))
>       (remove-text-properties (car region) (cdr region) '(invisible nil))))

Could we use `isearch-update-post-hook' here?

Or, it seems nicer to `add-function' around `isearch-filter-predicate'
and extend isearch-filter-visible to support (i.e., stop at, and
display) invisible text through text properties.

> ;; this seems to be unstable, but I cannot figure out why
> (defun org-restore-invisibility-specs (&rest _)
>   ""
>    (let ((pos (point-min)))
>      (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-max))
>        (when-let ((region (org-find-text-property-region pos 'org-invisible)))
> 	   (with-silent-modifications
> 	     (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-invisible))
> 	     (remove-text-properties (car region) (cdr region) '(org-invisible nil)))))))

Could you use the hook above to store all visited invisible texts, and
re-hide them at the end of the search, e.g., using
`isearch-mode-end-hook'?

> (add-hook 'post-command-hook #'org-restore-invisibility-specs)

Ouch. I hope we can avoid that.

I wonder how it compares to drawers using the same invisible spec as
headlines, as it was the case before. Could you give it a try? 

I think hiding all property drawers right after opening a subtree is
fast enough.

Another option, as I already suggested, would be to use text-properties
on property drawers only. Ignoring isearch inside those sounds
tolerable, at least.

Regards,

-- 
Nicolas Goaziou


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-08 16:38     ` Nicolas Goaziou
@ 2020-05-09 13:58       ` Nicolas Goaziou
  2020-05-09 16:22         ` Ihor Radchenko
  2020-05-09 15:40       ` Ihor Radchenko
  1 sibling, 1 reply; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-09 13:58 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> I wonder how it compares to drawers using the same invisible spec as
> headlines, as it was the case before. Could you give it a try? 
>
> I think hiding all property drawers right after opening a subtree is
> fast enough.

As a follow-up, I switched property drawers (and only those) back to
using `outline' invisible spec in master branch. Hopefully, navigating
in large folded files should be faster.

Of course, this doesn't prevent us to continue exploring
text-properties. In particular, the problem is still open for regular
drawers (e.g., LOGBOOK).


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-08 16:38     ` Nicolas Goaziou
  2020-05-09 13:58       ` Nicolas Goaziou
@ 2020-05-09 15:40       ` Ihor Radchenko
  2020-05-09 16:30         ` Ihor Radchenko
  1 sibling, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-09 15:40 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 4813 bytes --]

I have prepared a patch taking into account your comments and fixing
other issues, reported by Karl Voit and found by myself.

Summary of what is done in the patch:

1. iSearching in drawers is rewritten using using
isearch-filter-predicate and isearch-mode-end-hook.
The idea is to create temporary overlays in place of drawers to make
isearch work as usual.

2. Change org-show-set-visibility to consider text properties. This
makes helm-occur open drawers.

3. Make sure (partially) that text inserted into hidden drawers is also
hidden (to avoid glitches reported by Karl Voit).
The reason why it was happening was because `insert' does not inherit
text properties by default, which means that all the inserted text is
visible by default. I have changes some instances of insert and
insert-before-markers to thair *-and-inherit versions. Still looking
into where else I need to do the replacement.

Note that "glitch" might appear in many external packages writing into
org drawers. I do not think that insert-and-inherit is often used or
even known.

Remaining problems:

1. insert-* -> insert-*-and-inherit replacement will at least need to be
done in org-table.el and probably other places

2. I found hi-lock re-opening drawers after exiting isearch for some
reason. This happens when hi-lock tries to highlight isearch matches.
Not sure about the cause.

3. There is still some visual glitch when unnecessary org-ellipsis is
shown when text was inserted into hidden property drawer, though the
inserted text itself is hidden. 

>> (defun org-find-text-property-region (pos prop)
>>   "Find a region containing PROP text property around point POS."
>>   (require 'org-macs) ;; org-with-point-at
>>   (org-with-point-at pos
>
> Do we really need that since every function has a POS argument anyway?
> Is it for the `widen' part?

Yes, it is not needed. Fixed.

>>     (let* ((beg (and (get-text-property pos prop) pos))
>> 	   (end beg))
>>       (when beg
>> 	(setq beg (or (previous-single-property-change pos prop)
>> 		      beg))
>
> Shouldn't fall-back be (point-min)?
>
>> 	(setq end (or (next-single-property-change pos prop)
>> 		      end))
>
> And (point-max) here?

No, (point-min) and (point-max) may cause problems there.
previous/next-single-property-change returns nil when called at the
beginning/end of the region with given text property. Falling back to
(point-min/max) may wrongly return too large region.

> Nitpick: `equal' -> =

Fixed.

> Or, it seems nicer to `add-function' around `isearch-filter-predicate'
> and extend isearch-filter-visible to support (i.e., stop at, and
> display) invisible text through text properties.

Done. I used
(setq-local isearch-filter-predicate #'org--isearch-filter-predicate),
which should be even cleaner. 

> I wonder how it compares to drawers using the same invisible spec as
> headlines, as it was the case before. Could you give it a try? 

> I think hiding all property drawers right after opening a subtree is
> fast enough.

I am not sure what you refer to. Just saw your relevant commit. Will
test ASAP.

Without testing, the code does not seem to change the number of
overlays. A new overlay is still created for each property drawer.
As I mentioned in the first email, the large number of overlays is what
makes Emacs slow. Citing Eli Zaretskii's reply to my Bug#354553,
explaining why Emacs becomes slow on large org file:

"... When C-n calls vertical-motion, the latter needs to find the
buffer position displayed directly below the place where you typed
C-n.  Since much of the text between these places, vertical-motion
needs to skip the invisible text as quickly as possible, because from
the user's POV that text "doesn't exist": it isn't on the screen.
However, Org makes this skipping exceedingly hard, because (1) it uses
overlays (as opposed to text properties) to hide text, and (2) it puts
an awful lot of overlays on the hidden text: there are 18400 overlays
in this file's buffer, 17500 of them between the 3rd and the 4th
heading.  Because of this, vertical-motion must examine each and every
overlay as it moves through the text, because each overlay can
potentially change the 'invisible' property of text, or it might have
a display string that needs to be displayed.  So instead of skipping
all that hidden text in one go, vertical-motion loops over those 17.5K
overlays examining the properties of each one of them.  And that takes
time."

I imagine that opening subtree will also require cycling over the
[many] overlays in the subtree.

> Another option, as I already suggested, would be to use text-properties
> on property drawers only. Ignoring isearch inside those sounds
> tolerable, at least.

Hope the patch below is a reasonable solution to isearch problem with
'invisible text properties.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: org-mode-drawertextprop.patch --]
[-- Type: text/x-diff, Size: 13892 bytes --]

diff --git a/lisp/org-clock.el b/lisp/org-clock.el
index 34179096d..463b28f47 100644
--- a/lisp/org-clock.el
+++ b/lisp/org-clock.el
@@ -1359,14 +1359,14 @@ the default behavior."
 	   (sit-for 2)
 	   (throw 'abort nil))
 	  (t
-	   (insert-before-markers "\n")
+	   (insert-before-markers-and-inherit "\n")
 	   (backward-char 1)
 	   (when (and (save-excursion
 			(end-of-line 0)
 			(org-in-item-p)))
 	     (beginning-of-line 1)
 	     (indent-line-to (- (current-indentation) 2)))
-	   (insert org-clock-string " ")
+	   (insert-and-inherit org-clock-string " ")
 	   (setq org-clock-effort (org-entry-get (point) org-effort-property))
 	   (setq org-clock-total-time (org-clock-sum-current-item
 				       (org-clock-get-sum-start)))
@@ -1658,7 +1658,7 @@ to, overriding the existing value of `org-clock-out-switch-to-state'."
 	    (if fail-quietly (throw 'exit nil) (error "Clock start time is gone")))
 	  (goto-char (match-end 0))
 	  (delete-region (point) (point-at-eol))
-	  (insert "--")
+	  (insert-and-inherit "--")
 	  (setq te (org-insert-time-stamp (or at-time now) 'with-hm 'inactive))
 	  (setq s (org-time-convert-to-integer
 		   (time-subtract
@@ -1666,7 +1666,7 @@ to, overriding the existing value of `org-clock-out-switch-to-state'."
 		    (org-time-string-to-time ts)))
 		h (floor s 3600)
 		m (floor (mod s 3600) 60))
-	  (insert " => " (format "%2d:%02d" h m))
+	  (insert-and-inherit " => " (format "%2d:%02d" h m))
 	  (move-marker org-clock-marker nil)
 	  (move-marker org-clock-hd-marker nil)
 	  ;; Possibly remove zero time clocks.  However, do not add
diff --git a/lisp/org-macs.el b/lisp/org-macs.el
index a02f713ca..4b0e23f6a 100644
--- a/lisp/org-macs.el
+++ b/lisp/org-macs.el
@@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead."
 
 
 \f
-;;; Overlays
+;;; Overlays and text properties
 
 (defun org-overlay-display (ovl text &optional face evap)
   "Make overlay OVL display TEXT with face FACE."
@@ -705,18 +705,44 @@ If DELETE is non-nil, delete all those overlays."
 	    (delete (delete-overlay ov))
 	    (t (push ov found))))))
 
+(defun org--find-text-property-region (pos prop)
+  "Find a region containing PROP text property around point POS."
+  (let* ((beg (and (get-text-property pos prop) pos))
+	 (end beg))
+    (when beg
+      ;; when beg is the first point in the region, `previous-single-property-change'
+      ;; will return nil.
+      (setq beg (or (previous-single-property-change pos prop)
+		    beg))
+      ;; when end is the last point in the region, `next-single-property-change'
+      ;; will return nil.
+      (setq end (or (next-single-property-change pos prop)
+		    end))
+      (unless (= beg end) ; this should not happen
+        (cons beg end)))))
+
 (defun org-flag-region (from to flag spec)
   "Hide or show lines from FROM to TO, according to FLAG.
 SPEC is the invisibility spec, as a symbol."
-  (remove-overlays from to 'invisible spec)
-  ;; Use `front-advance' since text right before to the beginning of
-  ;; the overlay belongs to the visible line than to the contents.
-  (when flag
-    (let ((o (make-overlay from to nil 'front-advance)))
-      (overlay-put o 'evaporate t)
-      (overlay-put o 'invisible spec)
-      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
-
+  (pcase spec
+    ('outline
+     (remove-overlays from to 'invisible spec)
+     ;; Use `front-advance' since text right before to the beginning of
+     ;; the overlay belongs to the visible line than to the contents.
+     (when flag
+       (let ((o (make-overlay from to nil 'front-advance)))
+	 (overlay-put o 'evaporate t)
+	 (overlay-put o 'invisible spec)
+	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
+    (_
+     ;; Use text properties instead of overlays for speed.
+     ;; Overlays are too slow (Emacs Bug#35453).
+     (with-silent-modifications
+       (remove-text-properties from to '(invisible nil))
+       (when flag
+	 (put-text-property from to 'rear-non-sticky nil)
+	 (put-text-property from to 'front-sticky t)
+	 (put-text-property from to 'invisible spec))))))
 
 \f
 ;;; Regexp matching
diff --git a/lisp/org.el b/lisp/org.el
index 287fe30e8..335f68a85 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.")
 (declare-function cdlatex-math-symbol "ext:cdlatex")
 (declare-function Info-goto-node "info" (nodename &optional fork strict-case))
 (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag))
+(declare-function isearch-filter-visible "isearch" (beg end))
 (declare-function org-add-archive-files "org-archive" (files))
 (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom))
 (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour))
@@ -4869,6 +4870,10 @@ The following commands are available:
   (setq-local outline-isearch-open-invisible-function
 	      (lambda (&rest _) (org-show-context 'isearch)))
 
+  ;; Make isearch search in blocks hidden via text properties
+  (setq-local isearch-filter-predicate #'org--isearch-filter-predicate)
+  (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local)
+
   ;; Setup the pcomplete hooks
   (setq-local pcomplete-command-completion-function #'org-pcomplete-initial)
   (setq-local pcomplete-command-name-function #'org-command-at-point)
@@ -5859,9 +5864,26 @@ If TAG is a number, get the corresponding match group."
 	 (inhibit-modification-hooks t)
 	 deactivate-mark buffer-file-name buffer-file-truename)
     (decompose-region beg end)
+    ;; do not remove invisible text properties specified by
+    ;; 'org-hide-block and 'org-hide-drawer (but remove  'org-link)
+    ;; this is needed to keep the drawers and blocks hidden unless
+    ;; they are toggled by user
+    ;; Note: The below may be too specific and create troubles
+    ;; if more invisibility specs are added to org in future
+    (let ((pos beg)
+	  next spec)
+      (while (< pos end)
+	(setq next (next-single-property-change pos 'invisible nil end)
+              spec (get-text-property pos 'invisible))
+        (unless (memq spec (list 'org-hide-block
+				 'org-hide-drawer))
+          (remove-text-properties pos next '(invisible t)))
+        (setq pos next)))
     (remove-text-properties beg end
 			    '(mouse-face t keymap t org-linked-text t
-					 invisible t intangible t
+					 ;; Do not remove all invisible during fontification
+					 ;; invisible t
+                                         intangible t
 					 org-emphasis t))
     (org-remove-font-lock-display-properties beg end)))
 
@@ -6677,8 +6699,13 @@ information."
     ;; expose it.
     (dolist (o (overlays-at (point)))
       (when (memq (overlay-get o 'invisible)
-		  '(org-hide-block org-hide-drawer outline))
+		  '(outline))
 	(delete-overlay o)))
+    (when (memq (get-text-property (point) 'invisible)
+		'(org-hide-block org-hide-drawer))
+      (let ((spec (get-text-property (point) 'invisible))
+	    (region (org--find-text-property-region (point) 'invisible)))
+	(org-flag-region (car region) (cdr region) nil spec)))
     (unless (org-before-first-heading-p)
       (org-with-limited-levels
        (cl-case detail
@@ -10849,8 +10876,8 @@ EXTRA is additional text that will be inserted into the notes buffer."
 	 (unless (eq org-log-note-purpose 'clock-out)
 	   (goto-char (org-log-beginning t)))
 	 ;; Make sure point is at the beginning of an empty line.
-	 (cond ((not (bolp)) (let ((inhibit-read-only t)) (insert "\n")))
-	       ((looking-at "[ \t]*\\S-") (save-excursion (insert "\n"))))
+	 (cond ((not (bolp)) (let ((inhibit-read-only t)) (insert-and-inherit "\n")))
+	       ((looking-at "[ \t]*\\S-") (save-excursion (insert-and-inherit "\n"))))
 	 ;; In an existing list, add a new item at the top level.
 	 ;; Otherwise, indent line like a regular one.
 	 (let ((itemp (org-in-item-p)))
@@ -10860,12 +10887,12 @@ EXTRA is additional text that will be inserted into the notes buffer."
 				(goto-char itemp) (org-list-struct))))
 		  (org-list-get-ind (org-list-get-top-point struct) struct)))
 	     (org-indent-line)))
-	 (insert (org-list-bullet-string "-") (pop lines))
+	 (insert-and-inherit (org-list-bullet-string "-") (pop lines))
 	 (let ((ind (org-list-item-body-column (line-beginning-position))))
 	   (dolist (line lines)
-	     (insert "\n")
+	     (insert-and-inherit "\n")
 	     (indent-line-to ind)
-	     (insert line)))
+	     (insert-and-inherit line)))
 	 (message "Note stored")
 	 (org-back-to-heading t))
 	;; Fix `buffer-undo-list' when `org-store-log-note' is called
@@ -13036,10 +13063,10 @@ decreases scheduled or deadline date by one day."
 	      (progn (delete-region (match-beginning 0) (match-end 0))
 		     (goto-char (match-beginning 0)))
 	    (goto-char end)
-	    (insert "\n")
+	    (insert-and-inherit "\n")
 	    (backward-char))
-	  (insert ":" property ":")
-	  (when value (insert " " value))
+	  (insert-and-inherit ":" property ":")
+	  (when value (insert-and-inherit " " value))
 	  (org-indent-line)))))
     (run-hook-with-args 'org-property-changed-functions property value)))
 
@@ -14177,7 +14204,7 @@ The command returns the inserted time stamp."
   (let ((fmt (funcall (if with-hm 'cdr 'car) org-time-stamp-formats))
 	stamp)
     (when inactive (setq fmt (concat "[" (substring fmt 1 -1) "]")))
-    (insert-before-markers (or pre ""))
+    (insert-before-markers-and-inherit (or pre ""))
     (when (listp extra)
       (setq extra (car extra))
       (if (and (stringp extra)
@@ -14188,8 +14215,8 @@ The command returns the inserted time stamp."
 	(setq extra nil)))
     (when extra
       (setq fmt (concat (substring fmt 0 -1) extra (substring fmt -1))))
-    (insert-before-markers (setq stamp (format-time-string fmt time)))
-    (insert-before-markers (or post ""))
+    (insert-before-markers-and-inherit (setq stamp (format-time-string fmt time)))
+    (insert-before-markers-and-inherit (or post ""))
     (setq org-last-inserted-timestamp stamp)))
 
 (defun org-toggle-time-stamp-overlays ()
@@ -20913,6 +20940,79 @@ Started from `gnus-info-find-node'."
           (t default-org-info-node))))))
 
 \f
+
+;;; Make isearch search in some text hidden via text propertoes
+
+(defvar org--isearch-overlays nil
+  "List of overlays temporarily created during isearch.
+This is used to allow searching in regions hidden via text properties.
+As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays.
+Any text hidden via text properties is not revealed even if `search-invisible'
+is set to 't.")
+
+;; Not sure if it needs to be a user option
+;; One might want to reveal hidden text in, for example, hidden parts of the links.
+;; Currently, hidden text in links is never revealed by isearch.
+(defvar org-isearch-specs '(org-hide-block
+			 org-hide-drawer)
+  "List of text invisibility specs to be searched by isearch.
+By default ([2020-05-09 Sat]), isearch does not search in hidden text,
+which was made invisible using text properties. Isearch will be forced
+to search in hidden text with any of the listed 'invisible property value.")
+
+(defun org--create-isearch-overlays (beg end)
+  "Replace text property invisibility spec by overlays between BEG and END.
+All the regions with invisibility text property spec from
+`org-isearch-specs' will be changed to use overlays instead
+of text properties. The created overlays will be stored in
+`org--isearch-overlays'."
+  (let ((pos beg))
+    (while (< pos end)
+      (when-let* ((spec (get-text-property pos 'invisible))
+		  (spec (memq spec org-isearch-specs))
+		  (region (org--find-text-property-region pos 'invisible)))
+        ;; Changing text properties is considered buffer modification.
+        ;; We do not want it here.
+	(with-silent-modifications
+          ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat]
+          ;; overlay for 'outline blocks.
+          (let ((o (make-overlay (car region) (cdr region) nil 'front-advance)))
+	    (overlay-put o 'evaporate t)
+	    (overlay-put o 'invisible spec)
+            ;; `delete-overlay' here means that spec information will be lost
+            ;; for the region. The region will remain visible.
+	    (overlay-put o 'isearch-open-invisible #'delete-overlay)
+            (push o org--isearch-overlays))
+	  (remove-text-properties (car region) (cdr region) '(invisible nil))))
+      (setq pos (next-single-property-change pos 'invisible nil end)))))
+
+(defun org--isearch-filter-predicate (beg end)
+  "Return non-nil if text between BEG and END is deemed visible by Isearch.
+This function is intended to be used as `isearch-filter-predicate'.
+Unlike `isearch-filter-visible', make text with 'invisible text property
+value listed in `org-isearch-specs' visible to Isearch."
+  (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text
+  (isearch-filter-visible beg end))
+
+(defun org--clear-isearch-overlay (ov)
+  "Convert OV region back into using text properties."
+  (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays
+    ;; Changing text properties is considered buffer modification.
+    ;; We do not want it here.
+    (with-silent-modifications
+      (put-text-property (overlay-start ov) (overlay-end ov) 'invisible spec)))
+  (when (member ov isearch-opened-overlays)
+    (setq isearch-opened-overlays (delete ov isearch-opened-overlays)))
+  (delete-overlay ov))
+
+(defun org--clear-isearch-overlays ()
+  "Convert overlays from `org--isearch-overlays' back into using text properties."
+  (when org--isearch-overlays
+    (mapc #'org--clear-isearch-overlay org--isearch-overlays)
+    (setq org--isearch-overlays nil)))
+
+\f
+
 ;;; Finish up
 
 (add-hook 'org-mode-hook     ;remove overlays when changing major mode

[-- Attachment #3: Type: text/plain, Size: 3286 bytes --]





Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we
>> ;; cannot even use cursor-sensor-functions as a workaround
>> ;; I used a less ideas approach with advice to isearch-search-string as
>> ;; a workaround 
>
> OK.
>
>> (defun org-find-text-property-region (pos prop)
>>   "Find a region containing PROP text property around point POS."
>>   (require 'org-macs) ;; org-with-point-at
>>   (org-with-point-at pos
>
> Do we really need that since every function has a POS argument anyway?
> Is it for the `widen' part?
>
>>     (let* ((beg (and (get-text-property pos prop) pos))
>> 	   (end beg))
>>       (when beg
>> 	(setq beg (or (previous-single-property-change pos prop)
>> 		      beg))
>
> Shouldn't fall-back be (point-min)?
>
>> 	(setq end (or (next-single-property-change pos prop)
>> 		      end))
>
> And (point-max) here?
>
>>         (unless (equal beg end)
>
> Nitpick: `equal' -> =
>
>>           (cons beg end))))))
>
>> ;; :FIXME: re-hide properties when point moves away
>> (define-advice isearch-search-string (:after (&rest _) put-overlay)
>>   "Reveal hidden text at point."
>>   (when-let ((region (org-find-text-property-region (point) 'invisible)))
>>     (with-silent-modifications
>>       (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) 'invisible)))
>>       (remove-text-properties (car region) (cdr region) '(invisible nil))))
>
> Could we use `isearch-update-post-hook' here?
>
> Or, it seems nicer to `add-function' around `isearch-filter-predicate'
> and extend isearch-filter-visible to support (i.e., stop at, and
> display) invisible text through text properties.
>
>> ;; this seems to be unstable, but I cannot figure out why
>> (defun org-restore-invisibility-specs (&rest _)
>>   ""
>>    (let ((pos (point-min)))
>>      (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-max))
>>        (when-let ((region (org-find-text-property-region pos 'org-invisible)))
>> 	   (with-silent-modifications
>> 	     (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-invisible))
>> 	     (remove-text-properties (car region) (cdr region) '(org-invisible nil)))))))
>
> Could you use the hook above to store all visited invisible texts, and
> re-hide them at the end of the search, e.g., using
> `isearch-mode-end-hook'?
>
>> (add-hook 'post-command-hook #'org-restore-invisibility-specs)
>
> Ouch. I hope we can avoid that.
>
> I wonder how it compares to drawers using the same invisible spec as
> headlines, as it was the case before. Could you give it a try? 
>
> I think hiding all property drawers right after opening a subtree is
> fast enough.
>
> Another option, as I already suggested, would be to use text-properties
> on property drawers only. Ignoring isearch inside those sounds
> tolerable, at least.
>
> Regards,
>
> -- 
> Nicolas Goaziou

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-04 16:56     ` Karl Voit
  2020-05-07  7:18       ` Karl Voit
@ 2020-05-09 15:43       ` Ihor Radchenko
  1 sibling, 0 replies; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-09 15:43 UTC (permalink / raw)
  To: Karl Voit, emacs-orgmode

> The visual glitch looks like that:
> 
> :PROPERTIES:X:CREATED:  [2020-05-04 Mon 18>54]
> X

Should be partially fixed in the latest patch I just sent.

OLD <<< :PROPERTIES:X:CREATED:  [2020-05-04 Mon 18>54]

NEW >>> :PROPERTIES:X X

Best,
Ihor

Karl Voit <devnull@Karl-Voit.at> writes:

> Hi Ihor,
>
> * Ihor Radchenko <yantar92@gmail.com> wrote:
>>
>> So far, I came up with the following partial solution searching and
>> showing hidden text.
>>
>> (defun org-find-text-property-region (pos prop)
>> (define-advice isearch-search-string (:after (&rest _) put-overlay)
>> (defun org-restore-invisibility-specs (&rest _)
>> (add-hook 'post-command-hook #'org-restore-invisibility-specs)
>> (defun org-flag-region (from to flag spec)
>> (defun org-unfontify-region (beg end &optional _maybe_loudly)
>
> After a couple of hours working with these patches, my feedback is
> very positive. Besides some visual glitches when creating a new
> heading with org-expiry-insinuate activated (which automatically
> adds :CREATED: properties), I could not detect any side-effect so
> far (will keep testing).
>
> The visual glitch looks like that:
>
> :PROPERTIES:X:CREATED:  [2020-05-04 Mon 18>54]
> X
>
> ... with "X" being my character that symbolizes collapsed content.
> The way it looked without the patch was a simple collapsed property
> drawer.
>
> To me, this is acceptable considering the huge performance gain I
> got.
>
> THANK YOU VERY MUCH! I can't remember where I had this way of
> working within my large Org files[3] since ages.
>
>>> Anyway, the real fix should come from Emacs itself. There are ways to
>>> make overlays faster. These ways have already been discussed on the
>>> Emacs devel mailing list, but no one implemented them. It is a bit sad
>>> that we have to find workarounds for that.
>>
>> I guess that it is a very old story starting from the times when XEmacs
>> was a thing [1]. I recently heard about binary tree implementation of
>> overlays (there should be a branch in emacs git repo) [2], but there was
>> no update on that branch for a while. So, I do not have much hope on
>> Emacs implementing efficient overlay access in the near future. (And I
>> have problems with huge org files already).
>
> I can not express how this also reflects my personal situation.
>
>> [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/
>> [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html
>
> [3] https://karl-voit.at/2020/05/03/current-org-files
>
> -- 
> get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
>        > get Memacs from https://github.com/novoid/Memacs <
> Personal Information Management > http://Karl-Voit.at/tags/pim/
> Emacs-related > http://Karl-Voit.at/tags/emacs/
>
>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-07 11:04     ` Christian Heinrich
@ 2020-05-09 15:46       ` Ihor Radchenko
  0 siblings, 0 replies; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-09 15:46 UTC (permalink / raw)
  To: Christian Heinrich, emacs-orgmode

> I am not sure I understand how your follow-up code (below) needs to be incorporated. Would you mind
> sending a patch file? I hope that this ends up in the master branch at some point.

I have sent the patch in another email.
Will appreciate any feedback.

Best,
Ihor


Christian Heinrich <christian@gladbachcity.de> writes:

> Hi,
>
> thanks for your (initial) patch! I traced another error down today and found your code by chance. I
> tested it on an org-drill file that I had (with over 3500 items and hence 3500 drawers) and this
> patch helps *a lot* already. (Performance broke in 4403d4685e19fb99ba9bfec2bd4ff6781c66981f when
> outline-flag-region was replaced with org-flag-region, as drawers are no longer opened using
> outline-show-all which I had to use anyways to deal with my huge file.)
>
> I am not sure I understand how your follow-up code (below) needs to be incorporated. Would you mind
> sending a patch file? I hope that this ends up in the master branch at some point.
>
> Thanks again!
> Christian
>
> On Mon, 2020-04-27 at 00:04 +0800, Ihor Radchenko wrote:
>> > You cannot. You may however mimic it with `cursor-sensor-functions' text
>> > property. These assume Cursor Sensor minor mode is active, tho.
>> > I haven't tested it, but I assume it would slow down text properties
>> > a bit, too, but hopefully not as much as overlays.
>> 
>> Unfortunately, isearch sets inhibit-point-motion-hooks to non-nil
>> internally. Anyway, I came up with some workaround, which seems to work
>> (see below). Though it would be better if isearch supported hidden text
>> in addition to overlays.
>> 
>> > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth
>> > experimenting with `cursor-sensor-functions'.
>> 
>> So far, I came up with the following partial solution searching and
>> showing hidden text.
>> 
>> ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we
>> ;; cannot even use cursor-sensor-functions as a workaround
>> ;; I used a less ideas approach with advice to isearch-search-string as
>> ;; a workaround 
>> 
>> (defun org-find-text-property-region (pos prop)
>>   "Find a region containing PROP text property around point POS."
>>   (require 'org-macs) ;; org-with-point-at
>>   (org-with-point-at pos
>>     (let* ((beg (and (get-text-property pos prop) pos))
>> 	   (end beg))
>>       (when beg
>> 	(setq beg (or (previous-single-property-change pos prop)
>> 		      beg))
>> 	(setq end (or (next-single-property-change pos prop)
>> 		      end))
>>         (unless (equal beg end)
>>           (cons beg end))))))
>> 
>> ;; :FIXME: re-hide properties when point moves away
>> (define-advice isearch-search-string (:after (&rest _) put-overlay)
>>   "Reveal hidden text at point."
>>   (when-let ((region (org-find-text-property-region (point) 'invisible)))
>>     (with-silent-modifications
>>       (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point)
>> 'invisible)))
>>       (remove-text-properties (car region) (cdr region) '(invisible nil))))
>> 
>> ;; this seems to be unstable, but I cannot figure out why
>> (defun org-restore-invisibility-specs (&rest _)
>>   ""
>>    (let ((pos (point-min)))
>>      (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-
>> max))
>>        (when-let ((region (org-find-text-property-region pos 'org-invisible)))
>> 	   (with-silent-modifications
>> 	     (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-
>> invisible))
>> 	     (remove-text-properties (car region) (cdr region) '(org-invisible nil)))))))
>> 
>> (add-hook 'post-command-hook #'org-restore-invisibility-specs)
>> 
>> (defun org-flag-region (from to flag spec)
>>   "Hide or show lines from FROM to TO, according to FLAG.
>> SPEC is the invisibility spec, as a symbol."
>>   (pcase spec
>>     ('outline
>>      (remove-overlays from to 'invisible spec)
>>      ;; Use `front-advance' since text right before to the beginning of
>>      ;; the overlay belongs to the visible line than to the contents.
>>      (when flag
>>        (let ((o (make-overlay from to nil 'front-advance)))
>> 	 (overlay-put o 'evaporate t)
>> 	 (overlay-put o 'invisible spec)
>> 	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
>>     (_
>>      (with-silent-modifications
>>        (remove-text-properties from to '(invisible nil))
>>        (when flag
>> 	 (put-text-property from to 'invisible spec)
>> 	 )))))
>> 
>> ;; This normally deletes invisible text property. We do not want this now.
>> (defun org-unfontify-region (beg end &optional _maybe_loudly)
>>   "Remove fontification and activation overlays from links."
>>   (font-lock-default-unfontify-region beg end)
>>   (let* ((buffer-undo-list t)
>> 	 (inhibit-read-only t) (inhibit-point-motion-hooks t)
>> 	 (inhibit-modification-hooks t)
>> 	 deactivate-mark buffer-file-name buffer-file-truename)
>>     (decompose-region beg end)
>>     (remove-text-properties beg end
>> 			    '(mouse-face t keymap t org-linked-text t
>> 					 ;; Do not remove invisible during fontification			
>> 		 
>> 					 ;; invisible t
>>                                          intangible t
>> 					 org-emphasis t))
>>     (org-remove-font-lock-display-properties beg end)))
>> 
>> > Anyway, the real fix should come from Emacs itself. There are ways to
>> > make overlays faster. These ways have already been discussed on the
>> > Emacs devel mailing list, but no one implemented them. It is a bit sad
>> > that we have to find workarounds for that.
>> 
>> I guess that it is a very old story starting from the times when XEmacs
>> was a thing [1]. I recently heard about binary tree implementation of
>> overlays (there should be a branch in emacs git repo) [2], but there was
>> no update on that branch for a while. So, I do not have much hope on
>> Emacs implementing efficient overlay access in the near future. (And I
>> have problems with huge org files already).
>> 
>> [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/
>> [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html
>> 
>> 
>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>> 
>> > Hello,
>> > 
>> > Ihor Radchenko <yantar92@gmail.com> writes:
>> > 
>> > > To my surprise, the patch did not break org to unusable state and
>> > > the performance on the sample org file [3] improved drastically. You can
>> > > try by yourself!
>> > 
>> > It is not a surprise, really. Text properties are much faster than
>> > overlays, and very close to them features-wise. They are a bit more
>> > complex to handle, however.
>> > 
>> > > However, this did introduce some visual glitches with drawer display.
>> > > Though drawers can still be folded/unfolded with <tab>, they are not
>> > > folded on org-mode startup for some reason (can be fixed by running
>> > > (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers)
>> > > are unfolded for no apparent reason sometimes. A blind guess is that it
>> > > is something to do with lack of 'isearch-open-invisible, which I am not
>> > > sure how to set via text properties.
>> > 
>> > You cannot. You may however mimic it with `cursor-sensor-functions' text
>> > property. These assume Cursor Sensor minor mode is active, tho.
>> > I haven't tested it, but I assume it would slow down text properties
>> > a bit, too, but hopefully not as much as overlays.
>> > 
>> > Note there are clear advantages using text properties. For example, when
>> > you move contents around, text properties are preserved. So there's no
>> > more need for the `org-cycle-hide-drawer' dance, i.e., it is not
>> > necessary anymore to re-hide drawers.
>> > 
>> > > Any thoughts about the use of text properties or about the patch
>> > > suggestion are welcome.  
>> > 
>> > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth
>> > experimenting with `cursor-sensor-functions'.
>> > 
>> > We could also use text properties for property drawers, and overlays for
>> > regular ones. This might give us a reasonable speed-up with an
>> > acceptable feature trade-off.
>> > 
>> > Anyway, the real fix should come from Emacs itself. There are ways to
>> > make overlays faster. These ways have already been discussed on the
>> > Emacs devel mailing list, but no one implemented them. It is a bit sad
>> > that we have to find workarounds for that.
>> > 
>> > Regards,
>> > 
>> > -- 
>> > Nicolas Goaziou

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-09 13:58       ` Nicolas Goaziou
@ 2020-05-09 16:22         ` Ihor Radchenko
  2020-05-09 17:21           ` Nicolas Goaziou
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-09 16:22 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

> As a follow-up, I switched property drawers (and only those) back to
> using `outline' invisible spec in master branch. Hopefully, navigating
> in large folded files should be faster.

Just tested the master branch.
Three observations on large org file:

1. Next/previous line on folder buffer is still terribly slow
2. Unfolding speed does not seem to be affected by the last commits - it
is still much slower than text property version. There might be some
improvement if I run Emacs for longer time though (Emacs generally
becomes slower over time).
3. <TAB> <TAB> on a headline with several levels of subheadings moves
the cursor to the end of subtree, which did not happen in the past.

Best,
Ihor

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> I wonder how it compares to drawers using the same invisible spec as
>> headlines, as it was the case before. Could you give it a try? 
>>
>> I think hiding all property drawers right after opening a subtree is
>> fast enough.
>
> As a follow-up, I switched property drawers (and only those) back to
> using `outline' invisible spec in master branch. Hopefully, navigating
> in large folded files should be faster.
>
> Of course, this doesn't prevent us to continue exploring
> text-properties. In particular, the problem is still open for regular
> drawers (e.g., LOGBOOK).

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-09 15:40       ` Ihor Radchenko
@ 2020-05-09 16:30         ` Ihor Radchenko
  2020-05-09 17:32           ` Nicolas Goaziou
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-09 16:30 UTC (permalink / raw)
  To: emacs-orgmode, Nicolas Goaziou


Note that the following commits seems to break my patch:

074ea1629 origin/master master Deprecate `org-cycle-hide-drawers'
1027e0256 Implement `org-cycle-hide-property-drawers'
8b05c06d4 Use `outline' invisibility spec for property drawers

The patch should work for commit ed0e75d24 in master.

Best,
Ihor


Ihor Radchenko <yantar92@gmail.com> writes:

> I have prepared a patch taking into account your comments and fixing
> other issues, reported by Karl Voit and found by myself.
>
> Summary of what is done in the patch:
>
> 1. iSearching in drawers is rewritten using using
> isearch-filter-predicate and isearch-mode-end-hook.
> The idea is to create temporary overlays in place of drawers to make
> isearch work as usual.
>
> 2. Change org-show-set-visibility to consider text properties. This
> makes helm-occur open drawers.
>
> 3. Make sure (partially) that text inserted into hidden drawers is also
> hidden (to avoid glitches reported by Karl Voit).
> The reason why it was happening was because `insert' does not inherit
> text properties by default, which means that all the inserted text is
> visible by default. I have changes some instances of insert and
> insert-before-markers to thair *-and-inherit versions. Still looking
> into where else I need to do the replacement.
>
> Note that "glitch" might appear in many external packages writing into
> org drawers. I do not think that insert-and-inherit is often used or
> even known.
>
> Remaining problems:
>
> 1. insert-* -> insert-*-and-inherit replacement will at least need to be
> done in org-table.el and probably other places
>
> 2. I found hi-lock re-opening drawers after exiting isearch for some
> reason. This happens when hi-lock tries to highlight isearch matches.
> Not sure about the cause.
>
> 3. There is still some visual glitch when unnecessary org-ellipsis is
> shown when text was inserted into hidden property drawer, though the
> inserted text itself is hidden. 
>
>>> (defun org-find-text-property-region (pos prop)
>>>   "Find a region containing PROP text property around point POS."
>>>   (require 'org-macs) ;; org-with-point-at
>>>   (org-with-point-at pos
>>
>> Do we really need that since every function has a POS argument anyway?
>> Is it for the `widen' part?
>
> Yes, it is not needed. Fixed.
>
>>>     (let* ((beg (and (get-text-property pos prop) pos))
>>> 	   (end beg))
>>>       (when beg
>>> 	(setq beg (or (previous-single-property-change pos prop)
>>> 		      beg))
>>
>> Shouldn't fall-back be (point-min)?
>>
>>> 	(setq end (or (next-single-property-change pos prop)
>>> 		      end))
>>
>> And (point-max) here?
>
> No, (point-min) and (point-max) may cause problems there.
> previous/next-single-property-change returns nil when called at the
> beginning/end of the region with given text property. Falling back to
> (point-min/max) may wrongly return too large region.
>
>> Nitpick: `equal' -> =
>
> Fixed.
>
>> Or, it seems nicer to `add-function' around `isearch-filter-predicate'
>> and extend isearch-filter-visible to support (i.e., stop at, and
>> display) invisible text through text properties.
>
> Done. I used
> (setq-local isearch-filter-predicate #'org--isearch-filter-predicate),
> which should be even cleaner. 
>
>> I wonder how it compares to drawers using the same invisible spec as
>> headlines, as it was the case before. Could you give it a try? 
>
>> I think hiding all property drawers right after opening a subtree is
>> fast enough.
>
> I am not sure what you refer to. Just saw your relevant commit. Will
> test ASAP.
>
> Without testing, the code does not seem to change the number of
> overlays. A new overlay is still created for each property drawer.
> As I mentioned in the first email, the large number of overlays is what
> makes Emacs slow. Citing Eli Zaretskii's reply to my Bug#354553,
> explaining why Emacs becomes slow on large org file:
>
> "... When C-n calls vertical-motion, the latter needs to find the
> buffer position displayed directly below the place where you typed
> C-n.  Since much of the text between these places, vertical-motion
> needs to skip the invisible text as quickly as possible, because from
> the user's POV that text "doesn't exist": it isn't on the screen.
> However, Org makes this skipping exceedingly hard, because (1) it uses
> overlays (as opposed to text properties) to hide text, and (2) it puts
> an awful lot of overlays on the hidden text: there are 18400 overlays
> in this file's buffer, 17500 of them between the 3rd and the 4th
> heading.  Because of this, vertical-motion must examine each and every
> overlay as it moves through the text, because each overlay can
> potentially change the 'invisible' property of text, or it might have
> a display string that needs to be displayed.  So instead of skipping
> all that hidden text in one go, vertical-motion loops over those 17.5K
> overlays examining the properties of each one of them.  And that takes
> time."
>
> I imagine that opening subtree will also require cycling over the
> [many] overlays in the subtree.
>
>> Another option, as I already suggested, would be to use text-properties
>> on property drawers only. Ignoring isearch inside those sounds
>> tolerable, at least.
>
> Hope the patch below is a reasonable solution to isearch problem with
> 'invisible text properties.
>
> Best,
> Ihor
>
> diff --git a/lisp/org-clock.el b/lisp/org-clock.el
> index 34179096d..463b28f47 100644
> --- a/lisp/org-clock.el
> +++ b/lisp/org-clock.el
> @@ -1359,14 +1359,14 @@ the default behavior."
>  	   (sit-for 2)
>  	   (throw 'abort nil))
>  	  (t
> -	   (insert-before-markers "\n")
> +	   (insert-before-markers-and-inherit "\n")
>  	   (backward-char 1)
>  	   (when (and (save-excursion
>  			(end-of-line 0)
>  			(org-in-item-p)))
>  	     (beginning-of-line 1)
>  	     (indent-line-to (- (current-indentation) 2)))
> -	   (insert org-clock-string " ")
> +	   (insert-and-inherit org-clock-string " ")
>  	   (setq org-clock-effort (org-entry-get (point) org-effort-property))
>  	   (setq org-clock-total-time (org-clock-sum-current-item
>  				       (org-clock-get-sum-start)))
> @@ -1658,7 +1658,7 @@ to, overriding the existing value of `org-clock-out-switch-to-state'."
>  	    (if fail-quietly (throw 'exit nil) (error "Clock start time is gone")))
>  	  (goto-char (match-end 0))
>  	  (delete-region (point) (point-at-eol))
> -	  (insert "--")
> +	  (insert-and-inherit "--")
>  	  (setq te (org-insert-time-stamp (or at-time now) 'with-hm 'inactive))
>  	  (setq s (org-time-convert-to-integer
>  		   (time-subtract
> @@ -1666,7 +1666,7 @@ to, overriding the existing value of `org-clock-out-switch-to-state'."
>  		    (org-time-string-to-time ts)))
>  		h (floor s 3600)
>  		m (floor (mod s 3600) 60))
> -	  (insert " => " (format "%2d:%02d" h m))
> +	  (insert-and-inherit " => " (format "%2d:%02d" h m))
>  	  (move-marker org-clock-marker nil)
>  	  (move-marker org-clock-hd-marker nil)
>  	  ;; Possibly remove zero time clocks.  However, do not add
> diff --git a/lisp/org-macs.el b/lisp/org-macs.el
> index a02f713ca..4b0e23f6a 100644
> --- a/lisp/org-macs.el
> +++ b/lisp/org-macs.el
> @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead."
>  
>  
>  \f
> -;;; Overlays
> +;;; Overlays and text properties
>  
>  (defun org-overlay-display (ovl text &optional face evap)
>    "Make overlay OVL display TEXT with face FACE."
> @@ -705,18 +705,44 @@ If DELETE is non-nil, delete all those overlays."
>  	    (delete (delete-overlay ov))
>  	    (t (push ov found))))))
>  
> +(defun org--find-text-property-region (pos prop)
> +  "Find a region containing PROP text property around point POS."
> +  (let* ((beg (and (get-text-property pos prop) pos))
> +	 (end beg))
> +    (when beg
> +      ;; when beg is the first point in the region, `previous-single-property-change'
> +      ;; will return nil.
> +      (setq beg (or (previous-single-property-change pos prop)
> +		    beg))
> +      ;; when end is the last point in the region, `next-single-property-change'
> +      ;; will return nil.
> +      (setq end (or (next-single-property-change pos prop)
> +		    end))
> +      (unless (= beg end) ; this should not happen
> +        (cons beg end)))))
> +
>  (defun org-flag-region (from to flag spec)
>    "Hide or show lines from FROM to TO, according to FLAG.
>  SPEC is the invisibility spec, as a symbol."
> -  (remove-overlays from to 'invisible spec)
> -  ;; Use `front-advance' since text right before to the beginning of
> -  ;; the overlay belongs to the visible line than to the contents.
> -  (when flag
> -    (let ((o (make-overlay from to nil 'front-advance)))
> -      (overlay-put o 'evaporate t)
> -      (overlay-put o 'invisible spec)
> -      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
> -
> +  (pcase spec
> +    ('outline
> +     (remove-overlays from to 'invisible spec)
> +     ;; Use `front-advance' since text right before to the beginning of
> +     ;; the overlay belongs to the visible line than to the contents.
> +     (when flag
> +       (let ((o (make-overlay from to nil 'front-advance)))
> +	 (overlay-put o 'evaporate t)
> +	 (overlay-put o 'invisible spec)
> +	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
> +    (_
> +     ;; Use text properties instead of overlays for speed.
> +     ;; Overlays are too slow (Emacs Bug#35453).
> +     (with-silent-modifications
> +       (remove-text-properties from to '(invisible nil))
> +       (when flag
> +	 (put-text-property from to 'rear-non-sticky nil)
> +	 (put-text-property from to 'front-sticky t)
> +	 (put-text-property from to 'invisible spec))))))
>  
>  \f
>  ;;; Regexp matching
> diff --git a/lisp/org.el b/lisp/org.el
> index 287fe30e8..335f68a85 100644
> --- a/lisp/org.el
> +++ b/lisp/org.el
> @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.")
>  (declare-function cdlatex-math-symbol "ext:cdlatex")
>  (declare-function Info-goto-node "info" (nodename &optional fork strict-case))
>  (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag))
> +(declare-function isearch-filter-visible "isearch" (beg end))
>  (declare-function org-add-archive-files "org-archive" (files))
>  (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom))
>  (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour))
> @@ -4869,6 +4870,10 @@ The following commands are available:
>    (setq-local outline-isearch-open-invisible-function
>  	      (lambda (&rest _) (org-show-context 'isearch)))
>  
> +  ;; Make isearch search in blocks hidden via text properties
> +  (setq-local isearch-filter-predicate #'org--isearch-filter-predicate)
> +  (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local)
> +
>    ;; Setup the pcomplete hooks
>    (setq-local pcomplete-command-completion-function #'org-pcomplete-initial)
>    (setq-local pcomplete-command-name-function #'org-command-at-point)
> @@ -5859,9 +5864,26 @@ If TAG is a number, get the corresponding match group."
>  	 (inhibit-modification-hooks t)
>  	 deactivate-mark buffer-file-name buffer-file-truename)
>      (decompose-region beg end)
> +    ;; do not remove invisible text properties specified by
> +    ;; 'org-hide-block and 'org-hide-drawer (but remove  'org-link)
> +    ;; this is needed to keep the drawers and blocks hidden unless
> +    ;; they are toggled by user
> +    ;; Note: The below may be too specific and create troubles
> +    ;; if more invisibility specs are added to org in future
> +    (let ((pos beg)
> +	  next spec)
> +      (while (< pos end)
> +	(setq next (next-single-property-change pos 'invisible nil end)
> +              spec (get-text-property pos 'invisible))
> +        (unless (memq spec (list 'org-hide-block
> +				 'org-hide-drawer))
> +          (remove-text-properties pos next '(invisible t)))
> +        (setq pos next)))
>      (remove-text-properties beg end
>  			    '(mouse-face t keymap t org-linked-text t
> -					 invisible t intangible t
> +					 ;; Do not remove all invisible during fontification
> +					 ;; invisible t
> +                                         intangible t
>  					 org-emphasis t))
>      (org-remove-font-lock-display-properties beg end)))
>  
> @@ -6677,8 +6699,13 @@ information."
>      ;; expose it.
>      (dolist (o (overlays-at (point)))
>        (when (memq (overlay-get o 'invisible)
> -		  '(org-hide-block org-hide-drawer outline))
> +		  '(outline))
>  	(delete-overlay o)))
> +    (when (memq (get-text-property (point) 'invisible)
> +		'(org-hide-block org-hide-drawer))
> +      (let ((spec (get-text-property (point) 'invisible))
> +	    (region (org--find-text-property-region (point) 'invisible)))
> +	(org-flag-region (car region) (cdr region) nil spec)))
>      (unless (org-before-first-heading-p)
>        (org-with-limited-levels
>         (cl-case detail
> @@ -10849,8 +10876,8 @@ EXTRA is additional text that will be inserted into the notes buffer."
>  	 (unless (eq org-log-note-purpose 'clock-out)
>  	   (goto-char (org-log-beginning t)))
>  	 ;; Make sure point is at the beginning of an empty line.
> -	 (cond ((not (bolp)) (let ((inhibit-read-only t)) (insert "\n")))
> -	       ((looking-at "[ \t]*\\S-") (save-excursion (insert "\n"))))
> +	 (cond ((not (bolp)) (let ((inhibit-read-only t)) (insert-and-inherit "\n")))
> +	       ((looking-at "[ \t]*\\S-") (save-excursion (insert-and-inherit "\n"))))
>  	 ;; In an existing list, add a new item at the top level.
>  	 ;; Otherwise, indent line like a regular one.
>  	 (let ((itemp (org-in-item-p)))
> @@ -10860,12 +10887,12 @@ EXTRA is additional text that will be inserted into the notes buffer."
>  				(goto-char itemp) (org-list-struct))))
>  		  (org-list-get-ind (org-list-get-top-point struct) struct)))
>  	     (org-indent-line)))
> -	 (insert (org-list-bullet-string "-") (pop lines))
> +	 (insert-and-inherit (org-list-bullet-string "-") (pop lines))
>  	 (let ((ind (org-list-item-body-column (line-beginning-position))))
>  	   (dolist (line lines)
> -	     (insert "\n")
> +	     (insert-and-inherit "\n")
>  	     (indent-line-to ind)
> -	     (insert line)))
> +	     (insert-and-inherit line)))
>  	 (message "Note stored")
>  	 (org-back-to-heading t))
>  	;; Fix `buffer-undo-list' when `org-store-log-note' is called
> @@ -13036,10 +13063,10 @@ decreases scheduled or deadline date by one day."
>  	      (progn (delete-region (match-beginning 0) (match-end 0))
>  		     (goto-char (match-beginning 0)))
>  	    (goto-char end)
> -	    (insert "\n")
> +	    (insert-and-inherit "\n")
>  	    (backward-char))
> -	  (insert ":" property ":")
> -	  (when value (insert " " value))
> +	  (insert-and-inherit ":" property ":")
> +	  (when value (insert-and-inherit " " value))
>  	  (org-indent-line)))))
>      (run-hook-with-args 'org-property-changed-functions property value)))
>  
> @@ -14177,7 +14204,7 @@ The command returns the inserted time stamp."
>    (let ((fmt (funcall (if with-hm 'cdr 'car) org-time-stamp-formats))
>  	stamp)
>      (when inactive (setq fmt (concat "[" (substring fmt 1 -1) "]")))
> -    (insert-before-markers (or pre ""))
> +    (insert-before-markers-and-inherit (or pre ""))
>      (when (listp extra)
>        (setq extra (car extra))
>        (if (and (stringp extra)
> @@ -14188,8 +14215,8 @@ The command returns the inserted time stamp."
>  	(setq extra nil)))
>      (when extra
>        (setq fmt (concat (substring fmt 0 -1) extra (substring fmt -1))))
> -    (insert-before-markers (setq stamp (format-time-string fmt time)))
> -    (insert-before-markers (or post ""))
> +    (insert-before-markers-and-inherit (setq stamp (format-time-string fmt time)))
> +    (insert-before-markers-and-inherit (or post ""))
>      (setq org-last-inserted-timestamp stamp)))
>  
>  (defun org-toggle-time-stamp-overlays ()
> @@ -20913,6 +20940,79 @@ Started from `gnus-info-find-node'."
>            (t default-org-info-node))))))
>  
>  \f
> +
> +;;; Make isearch search in some text hidden via text propertoes
> +
> +(defvar org--isearch-overlays nil
> +  "List of overlays temporarily created during isearch.
> +This is used to allow searching in regions hidden via text properties.
> +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays.
> +Any text hidden via text properties is not revealed even if `search-invisible'
> +is set to 't.")
> +
> +;; Not sure if it needs to be a user option
> +;; One might want to reveal hidden text in, for example, hidden parts of the links.
> +;; Currently, hidden text in links is never revealed by isearch.
> +(defvar org-isearch-specs '(org-hide-block
> +			 org-hide-drawer)
> +  "List of text invisibility specs to be searched by isearch.
> +By default ([2020-05-09 Sat]), isearch does not search in hidden text,
> +which was made invisible using text properties. Isearch will be forced
> +to search in hidden text with any of the listed 'invisible property value.")
> +
> +(defun org--create-isearch-overlays (beg end)
> +  "Replace text property invisibility spec by overlays between BEG and END.
> +All the regions with invisibility text property spec from
> +`org-isearch-specs' will be changed to use overlays instead
> +of text properties. The created overlays will be stored in
> +`org--isearch-overlays'."
> +  (let ((pos beg))
> +    (while (< pos end)
> +      (when-let* ((spec (get-text-property pos 'invisible))
> +		  (spec (memq spec org-isearch-specs))
> +		  (region (org--find-text-property-region pos 'invisible)))
> +        ;; Changing text properties is considered buffer modification.
> +        ;; We do not want it here.
> +	(with-silent-modifications
> +          ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat]
> +          ;; overlay for 'outline blocks.
> +          (let ((o (make-overlay (car region) (cdr region) nil 'front-advance)))
> +	    (overlay-put o 'evaporate t)
> +	    (overlay-put o 'invisible spec)
> +            ;; `delete-overlay' here means that spec information will be lost
> +            ;; for the region. The region will remain visible.
> +	    (overlay-put o 'isearch-open-invisible #'delete-overlay)
> +            (push o org--isearch-overlays))
> +	  (remove-text-properties (car region) (cdr region) '(invisible nil))))
> +      (setq pos (next-single-property-change pos 'invisible nil end)))))
> +
> +(defun org--isearch-filter-predicate (beg end)
> +  "Return non-nil if text between BEG and END is deemed visible by Isearch.
> +This function is intended to be used as `isearch-filter-predicate'.
> +Unlike `isearch-filter-visible', make text with 'invisible text property
> +value listed in `org-isearch-specs' visible to Isearch."
> +  (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text
> +  (isearch-filter-visible beg end))
> +
> +(defun org--clear-isearch-overlay (ov)
> +  "Convert OV region back into using text properties."
> +  (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays
> +    ;; Changing text properties is considered buffer modification.
> +    ;; We do not want it here.
> +    (with-silent-modifications
> +      (put-text-property (overlay-start ov) (overlay-end ov) 'invisible spec)))
> +  (when (member ov isearch-opened-overlays)
> +    (setq isearch-opened-overlays (delete ov isearch-opened-overlays)))
> +  (delete-overlay ov))
> +
> +(defun org--clear-isearch-overlays ()
> +  "Convert overlays from `org--isearch-overlays' back into using text properties."
> +  (when org--isearch-overlays
> +    (mapc #'org--clear-isearch-overlay org--isearch-overlays)
> +    (setq org--isearch-overlays nil)))
> +
> +\f
> +
>  ;;; Finish up
>  
>  (add-hook 'org-mode-hook     ;remove overlays when changing major mode
>
>
>
>
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Hello,
>>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>> ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we
>>> ;; cannot even use cursor-sensor-functions as a workaround
>>> ;; I used a less ideas approach with advice to isearch-search-string as
>>> ;; a workaround 
>>
>> OK.
>>
>>> (defun org-find-text-property-region (pos prop)
>>>   "Find a region containing PROP text property around point POS."
>>>   (require 'org-macs) ;; org-with-point-at
>>>   (org-with-point-at pos
>>
>> Do we really need that since every function has a POS argument anyway?
>> Is it for the `widen' part?
>>
>>>     (let* ((beg (and (get-text-property pos prop) pos))
>>> 	   (end beg))
>>>       (when beg
>>> 	(setq beg (or (previous-single-property-change pos prop)
>>> 		      beg))
>>
>> Shouldn't fall-back be (point-min)?
>>
>>> 	(setq end (or (next-single-property-change pos prop)
>>> 		      end))
>>
>> And (point-max) here?
>>
>>>         (unless (equal beg end)
>>
>> Nitpick: `equal' -> =
>>
>>>           (cons beg end))))))
>>
>>> ;; :FIXME: re-hide properties when point moves away
>>> (define-advice isearch-search-string (:after (&rest _) put-overlay)
>>>   "Reveal hidden text at point."
>>>   (when-let ((region (org-find-text-property-region (point) 'invisible)))
>>>     (with-silent-modifications
>>>       (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) 'invisible)))
>>>       (remove-text-properties (car region) (cdr region) '(invisible nil))))
>>
>> Could we use `isearch-update-post-hook' here?
>>
>> Or, it seems nicer to `add-function' around `isearch-filter-predicate'
>> and extend isearch-filter-visible to support (i.e., stop at, and
>> display) invisible text through text properties.
>>
>>> ;; this seems to be unstable, but I cannot figure out why
>>> (defun org-restore-invisibility-specs (&rest _)
>>>   ""
>>>    (let ((pos (point-min)))
>>>      (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-max))
>>>        (when-let ((region (org-find-text-property-region pos 'org-invisible)))
>>> 	   (with-silent-modifications
>>> 	     (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-invisible))
>>> 	     (remove-text-properties (car region) (cdr region) '(org-invisible nil)))))))
>>
>> Could you use the hook above to store all visited invisible texts, and
>> re-hide them at the end of the search, e.g., using
>> `isearch-mode-end-hook'?
>>
>>> (add-hook 'post-command-hook #'org-restore-invisibility-specs)
>>
>> Ouch. I hope we can avoid that.
>>
>> I wonder how it compares to drawers using the same invisible spec as
>> headlines, as it was the case before. Could you give it a try? 
>>
>> I think hiding all property drawers right after opening a subtree is
>> fast enough.
>>
>> Another option, as I already suggested, would be to use text-properties
>> on property drawers only. Ignoring isearch inside those sounds
>> tolerable, at least.
>>
>> Regards,
>>
>> -- 
>> Nicolas Goaziou
>
> -- 
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-09 16:22         ` Ihor Radchenko
@ 2020-05-09 17:21           ` Nicolas Goaziou
  2020-05-10  5:25             ` Ihor Radchenko
  0 siblings, 1 reply; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-09 17:21 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> Just tested the master branch.
> Three observations on large org file:
>
> 1. Next/previous line on folder buffer is still terribly slow

Oops, you are right. I fixed this. It should be way faster. I can
navigate in your example file without much trouble.

Please let me know how it goes.

> 2. Unfolding speed does not seem to be affected by the last commits - it
> is still much slower than text property version. There might be some
> improvement if I run Emacs for longer time though (Emacs generally
> becomes slower over time).

The last commits have nothing to do with unfolding.

I'm not pretending that overlays are faster than text properties,
either.

With the current implementation property drawers add no overhead : last
commits reduce drastically the number of overlays active in a buffer at
a given time.

> 3. <TAB> <TAB> on a headline with several levels of subheadings moves
> the cursor to the end of subtree, which did not happen in the past.

Indeed. I fixed that, too. Thank you!

Regards,

-- 
Nicolas Goaziou


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-09 16:30         ` Ihor Radchenko
@ 2020-05-09 17:32           ` Nicolas Goaziou
  2020-05-09 18:06             ` Ihor Radchenko
  0 siblings, 1 reply; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-09 17:32 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Ihor Radchenko <yantar92@gmail.com> writes:

> Note that the following commits seems to break my patch:

Unfortunately, I don't see your patch.


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-09 17:32           ` Nicolas Goaziou
@ 2020-05-09 18:06             ` Ihor Radchenko
  2020-05-10 14:59               ` Nicolas Goaziou
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-09 18:06 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

> Unfortunately, I don't see your patch.

My response to you was blocked by your mail server:

> 550 5.7.1 Reject for policy reason RULE3_2. See
> http://postmaster.gandi.net

The message landed on the mail list though:
https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127972.html

Best,
Ihor


Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Note that the following commits seems to break my patch:
>
> Unfortunately, I don't see your patch.

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-09 17:21           ` Nicolas Goaziou
@ 2020-05-10  5:25             ` Ihor Radchenko
  2020-05-10  9:47               ` Nicolas Goaziou
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-10  5:25 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

> Oops, you are right. I fixed this. It should be way faster. I can
> navigate in your example file without much trouble.
>
> Please let me know how it goes.

I tested with master + my personal config + native compilation of org,
Emacs native-comp branch, commit c984a53b4e198e31d11d7bc493dc9a686c77edae.
Did not see much improvement.
Vertical motion in the folded buffer is still quite slow.

> The last commits have nothing to do with unfolding.

Apparently I misunderstood the purpose of: 1027e0256
"Implement `org-cycle-hide-property-drawers'"

Best,
Ihor

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Just tested the master branch.
>> Three observations on large org file:
>>
>> 1. Next/previous line on folder buffer is still terribly slow
>
> Oops, you are right. I fixed this. It should be way faster. I can
> navigate in your example file without much trouble.
>
> Please let me know how it goes.
>
>> 2. Unfolding speed does not seem to be affected by the last commits - it
>> is still much slower than text property version. There might be some
>> improvement if I run Emacs for longer time though (Emacs generally
>> becomes slower over time).
>
> The last commits have nothing to do with unfolding.
>
> I'm not pretending that overlays are faster than text properties,
> either.
>
> With the current implementation property drawers add no overhead : last
> commits reduce drastically the number of overlays active in a buffer at
> a given time.
>
>> 3. <TAB> <TAB> on a headline with several levels of subheadings moves
>> the cursor to the end of subtree, which did not happen in the past.
>
> Indeed. I fixed that, too. Thank you!
>
> Regards,
>
> -- 
> Nicolas Goaziou

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-10  5:25             ` Ihor Radchenko
@ 2020-05-10  9:47               ` Nicolas Goaziou
  2020-05-10 13:29                 ` Ihor Radchenko
  0 siblings, 1 reply; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-10  9:47 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

>> Oops, you are right. I fixed this. It should be way faster. I can
>> navigate in your example file without much trouble.
>>
>> Please let me know how it goes.
>
> I tested with master + my personal config + native compilation of org,
> Emacs native-comp branch, commit c984a53b4e198e31d11d7bc493dc9a686c77edae.
> Did not see much improvement.
> Vertical motion in the folded buffer is still quite slow.

Oh! This is embarrassing. I improved speed, then broke it again in
a later commit. Sorry for wasting your time. I think I fixed it again.
Thank you for the feedback.

Could you have a look again?

> Apparently I misunderstood the purpose of: 1027e0256
> "Implement `org-cycle-hide-property-drawers'"

The function is meant to re-hide only property drawers after visibility
cycling. Its purpose is not to improve /unfolding/ speed. Unfolding is
very fast already, faster than using text properties.

Folding has roughly the same speed in both cases: most time is spent
looking for the next location to fold. However, folding with text
properties is more resilient, so you fold less often.

As a side note, your file contains 5217 headlines and 5215 property
drawers. I'll ignore the 3989 regular drawers for the time being
(although they do contribute to the slow navigation). In current master,
it means there is at most 5217 overlays in the buffer. With text
properties, the worse situation in the same.

Of course, that case happens less often with text properties. For
example, it happens in "contents" view in both cases. However, in "show
all" view, it is only a problem with overlays.

Regards,

-- 
Nicolas Goaziou


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-10  9:47               ` Nicolas Goaziou
@ 2020-05-10 13:29                 ` Ihor Radchenko
  2020-05-10 14:46                   ` Nicolas Goaziou
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-10 13:29 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

>> I tested with master + my personal config + native compilation of org,
>> Emacs native-comp branch, commit c984a53b4e198e31d11d7bc493dc9a686c77edae.
>> Did not see much improvement.
>> Vertical motion in the folded buffer is still quite slow.
>
> Oh! This is embarrassing. I improved speed, then broke it again in
> a later commit. Sorry for wasting your time. I think I fixed it again.
> Thank you for the feedback.
>
> Could you have a look again?

I still do not feel much difference, so I used elp to quantify if there
is any difference I cannot notice by myself. I tested the time to move
from to bottom of the example file with next-logical-line.

org master (7801e9236):
6(#calls)           2.852953989(total time, sec)   0.4754923315(average)
      
org e39365e32:
6           2.991771891   0.4986286485

org feature/drawertextprop:
6           0.149731379   0.0249552298

There is small improvement in speed, but it is not obvious.

> ... In current master,
> it means there is at most 5217 overlays in the buffer. With text
> properties, the worse situation in the same.

Do you mean that number of overlays is same with text properties? I feel
that I misunderstand what you want to say. 

> Of course, that case happens less often with text properties. For
> example, it happens in "contents" view in both cases. However, in "show
> all" view, it is only a problem with overlays.

I am completely lost. What do you mean by "that case"?

Best,
Ihor

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>>> Oops, you are right. I fixed this. It should be way faster. I can
>>> navigate in your example file without much trouble.
>>>
>>> Please let me know how it goes.
>>
>> I tested with master + my personal config + native compilation of org,
>> Emacs native-comp branch, commit c984a53b4e198e31d11d7bc493dc9a686c77edae.
>> Did not see much improvement.
>> Vertical motion in the folded buffer is still quite slow.
>
> Oh! This is embarrassing. I improved speed, then broke it again in
> a later commit. Sorry for wasting your time. I think I fixed it again.
> Thank you for the feedback.
>
> Could you have a look again?
>
>> Apparently I misunderstood the purpose of: 1027e0256
>> "Implement `org-cycle-hide-property-drawers'"
>
> The function is meant to re-hide only property drawers after visibility
> cycling. Its purpose is not to improve /unfolding/ speed. Unfolding is
> very fast already, faster than using text properties.
>
> Folding has roughly the same speed in both cases: most time is spent
> looking for the next location to fold. However, folding with text
> properties is more resilient, so you fold less often.
>
> As a side note, your file contains 5217 headlines and 5215 property
> drawers. I'll ignore the 3989 regular drawers for the time being
> (although they do contribute to the slow navigation). In current master,
> it means there is at most 5217 overlays in the buffer. With text
> properties, the worse situation in the same.
>
> Of course, that case happens less often with text properties. For
> example, it happens in "contents" view in both cases. However, in "show
> all" view, it is only a problem with overlays.
>
> Regards,
>
> -- 
> Nicolas Goaziou

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-10 13:29                 ` Ihor Radchenko
@ 2020-05-10 14:46                   ` Nicolas Goaziou
  2020-05-10 16:21                     ` Ihor Radchenko
  0 siblings, 1 reply; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-10 14:46 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Ihor Radchenko <yantar92@gmail.com> writes:

> I still do not feel much difference, so I used elp to quantify if there
> is any difference I cannot notice by myself. I tested the time to move
> from to bottom of the example file with next-logical-line.
>
> org master (7801e9236):
> 6(#calls)           2.852953989(total time, sec)   0.4754923315(average)
>       
> org e39365e32:
> 6           2.991771891   0.4986286485
>
> org feature/drawertextprop:
> 6           0.149731379   0.0249552298
>
> There is small improvement in speed, but it is not obvious.

I don't know how you made your test. You probably didn't
remove :LOGBOOK: lines. When headlines are fully folded, there are
8 overlays in the buffer, where there used to be 10k. It cannot be
a "small improvement".

Ah, well. It doesn't matter. At least the situation improved in some
cases, and the code is better.

>> ... In current master,
>> it means there is at most 5217 overlays in the buffer. With text
>> properties, the worse situation in the same.
>
> Do you mean that number of overlays is same with text properties? I feel
> that I misunderstand what you want to say. 

AFAIU, you still use overlays for headlines. If you activate so-called
"contents view", all headlines are visible, and are all folded. You get
5217 overlays in the buffer.

>> Of course, that case happens less often with text properties. For
>> example, it happens in "contents" view in both cases. However, in "show
>> all" view, it is only a problem with overlays.
>
> I am completely lost. What do you mean by "that case"?

I am talking about the "worse case" situation just above.

I'll comment your patch in another message.

Regards,


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-09 18:06             ` Ihor Radchenko
@ 2020-05-10 14:59               ` Nicolas Goaziou
  2020-05-10 15:15                 ` Kyle Meyer
  2020-05-10 16:30                 ` Ihor Radchenko
  0 siblings, 2 replies; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-10 14:59 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Ihor Radchenko <yantar92@gmail.com> writes:

> My response to you was blocked by your mail server:
>
>> 550 5.7.1 Reject for policy reason RULE3_2. See
>> http://postmaster.gandi.net

Aka "spam detected". Bah.

> The message landed on the mail list though:
> https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127972.html

Unfortunately, reviewing this way is not nice.

The `insert-and-inherit' issue sounds serious. We cannot reasonably
expect any library outside Org to use it. 

We could automatically extend invisible area with
`after-change-functions', i.e., if we're inserting something and both
side have the same `invisible' property, extend it. Using
`after-change-functions' sounds bad, but this kind of check shouldn't
cost much.

WDYT?

Regards,


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-10 14:59               ` Nicolas Goaziou
@ 2020-05-10 15:15                 ` Kyle Meyer
  2020-05-10 16:30                 ` Ihor Radchenko
  1 sibling, 0 replies; 55+ messages in thread
From: Kyle Meyer @ 2020-05-10 15:15 UTC (permalink / raw)
  To: emacs-orgmode; +Cc: Ihor Radchenko

Nicolas Goaziou writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> My response to you was blocked by your mail server:
>>
>>> 550 5.7.1 Reject for policy reason RULE3_2. See
>>> http://postmaster.gandi.net
>
> Aka "spam detected". Bah.
>
>> The message landed on the mail list though:
>> https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127972.html
>
> Unfortunately, reviewing this way is not nice.

It's probably not helpful at this point, but just in case: you can get
that message's mbox with

  curl -fSs https://yhetil.org/orgmode/87imh5w1zt.fsf@localhost/raw >mbox


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-10 14:46                   ` Nicolas Goaziou
@ 2020-05-10 16:21                     ` Ihor Radchenko
  2020-05-10 16:38                       ` Nicolas Goaziou
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-10 16:21 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

> I don't know how you made your test. You probably didn't
> remove :LOGBOOK: lines. When headlines are fully folded, there are
> 8 overlays in the buffer, where there used to be 10k. It cannot be
> a "small improvement".

Ouch. I did not remove :LOGBOOK: lines. I thought you referred to the
original file in "I can navigate in your example file without much
trouble."

If you want, I can test the file without :LOGBOOK: lines tomorrow.

>>> ... In current master,
>>> it means there is at most 5217 overlays in the buffer. With text
>>> properties, the worse situation in the same.
>>
>> Do you mean that number of overlays is same with text properties? I feel
>> that I misunderstand what you want to say. 
>
> AFAIU, you still use overlays for headlines. If you activate so-called
> "contents view", all headlines are visible, and are all folded. You get
> 5217 overlays in the buffer.

No, there are only 9 'outline overlays in the folded buffer if we do not
create overlays for drawers. This is because outline-hide-sublevels
called by org-overview is calling outline-flag-region on the whole
buffer thus removing all the 'outline overlays in buffer
(remove-overlays from to 'invisible 'outline) and re-creating a single
overlay for each top-level heading.

Now, thinking second time about this, using the following for
org-flag-region would achieve similar effect:

(remove-overlays from to 'invisible 'outline)
(remove-overlays from to 'invisible 'org-hide-drawer)

Now sure if it is going to break org-cycle though.
What do you think?

Best,
Ihor




Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> I still do not feel much difference, so I used elp to quantify if there
>> is any difference I cannot notice by myself. I tested the time to move
>> from to bottom of the example file with next-logical-line.
>>
>> org master (7801e9236):
>> 6(#calls)           2.852953989(total time, sec)   0.4754923315(average)
>>       
>> org e39365e32:
>> 6           2.991771891   0.4986286485
>>
>> org feature/drawertextprop:
>> 6           0.149731379   0.0249552298
>>
>> There is small improvement in speed, but it is not obvious.
>
> I don't know how you made your test. You probably didn't
> remove :LOGBOOK: lines. When headlines are fully folded, there are
> 8 overlays in the buffer, where there used to be 10k. It cannot be
> a "small improvement".
>
> Ah, well. It doesn't matter. At least the situation improved in some
> cases, and the code is better.
>
>>> ... In current master,
>>> it means there is at most 5217 overlays in the buffer. With text
>>> properties, the worse situation in the same.
>>
>> Do you mean that number of overlays is same with text properties? I feel
>> that I misunderstand what you want to say. 
>
> AFAIU, you still use overlays for headlines. If you activate so-called
> "contents view", all headlines are visible, and are all folded. You get
> 5217 overlays in the buffer.
>
>>> Of course, that case happens less often with text properties. For
>>> example, it happens in "contents" view in both cases. However, in "show
>>> all" view, it is only a problem with overlays.
>>
>> I am completely lost. What do you mean by "that case"?
>
> I am talking about the "worse case" situation just above.
>
> I'll comment your patch in another message.
>
> Regards,

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-10 14:59               ` Nicolas Goaziou
  2020-05-10 15:15                 ` Kyle Meyer
@ 2020-05-10 16:30                 ` Ihor Radchenko
  2020-05-10 19:32                   ` Nicolas Goaziou
  1 sibling, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-10 16:30 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

> Unfortunately, reviewing this way is not nice.

This should be better:
https://gist.github.com/yantar92/e37c2830d3bb6db8678b14424286c930

> The `insert-and-inherit' issue sounds serious. We cannot reasonably
> expect any library outside Org to use it. 
>
> We could automatically extend invisible area with
> `after-change-functions', i.e., if we're inserting something and both
> side have the same `invisible' property, extend it. Using
> `after-change-functions' sounds bad, but this kind of check shouldn't
> cost much.
>
> WDYT?

This might get tricky in the following case:

:PROPERTIES:
:CREATED: [2020-04-13 Mon 22:31]
<region-beginning>
:SHOWFROMDATE: 2020-05-11
:ID:       e05e3b33-71ba-4bbc-abba-8a92c565ad34
:END:

<many subtrees in between>

:PROPERTIES:
:CREATED:  [2020-04-27 Mon 13:50]
<region-end>
:ID:       b2eef49f-1c5c-4ff1-8e10-80423c8d8532
:ATTACH_DIR_INHERIT: t
:END:

If the text in the region is replaced by something else, <many subtrees
in between> should not be fully hidden. We cannot simply look at the
'invisible property before and after the changed region. 

I think that using fontification (something similar to
org-fontify-drawers) instead of after-change-functions should be faster.

Best,
Ihor


Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> My response to you was blocked by your mail server:
>>
>>> 550 5.7.1 Reject for policy reason RULE3_2. See
>>> http://postmaster.gandi.net
>
> Aka "spam detected". Bah.
>
>> The message landed on the mail list though:
>> https://www.mail-archive.com/emacs-orgmode@gnu.org/msg127972.html
>
> Unfortunately, reviewing this way is not nice.
>
> The `insert-and-inherit' issue sounds serious. We cannot reasonably
> expect any library outside Org to use it. 
>
> We could automatically extend invisible area with
> `after-change-functions', i.e., if we're inserting something and both
> side have the same `invisible' property, extend it. Using
> `after-change-functions' sounds bad, but this kind of check shouldn't
> cost much.
>
> WDYT?
>
> Regards,

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-10 16:21                     ` Ihor Radchenko
@ 2020-05-10 16:38                       ` Nicolas Goaziou
  2020-05-10 17:08                         ` Ihor Radchenko
  0 siblings, 1 reply; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-10 16:38 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Ihor Radchenko <yantar92@gmail.com> writes:

> If you want, I can test the file without :LOGBOOK: lines tomorrow.

Don't worry, it doesn't matter now.

> No, there are only 9 'outline overlays in the folded buffer if we do not
> create overlays for drawers. This is because outline-hide-sublevels
> called by org-overview is calling outline-flag-region on the whole
> buffer thus removing all the 'outline overlays in buffer
> (remove-overlays from to 'invisible 'outline) and re-creating a single
> overlay for each top-level heading.

You're talking about "overview" (org-overview), whereas I'm talking
about "contents view" (org-content). They are not the same. In the
latter, you show every headline in the buffer, so you have one overlay
per headline.

> Now, thinking second time about this, using the following for
> org-flag-region would achieve similar effect:
>
> (remove-overlays from to 'invisible 'outline)
> (remove-overlays from to 'invisible 'org-hide-drawer)
>
> Now sure if it is going to break org-cycle though.
> What do you think?

This is already the case. See first line of `org-flag-region'.

Regards,


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-10 16:38                       ` Nicolas Goaziou
@ 2020-05-10 17:08                         ` Ihor Radchenko
  2020-05-10 19:38                           ` Nicolas Goaziou
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-10 17:08 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

> You're talking about "overview" (org-overview), whereas I'm talking
> about "contents view" (org-content). They are not the same. In the
> latter, you show every headline in the buffer, so you have one overlay
> per headline.

Thanks for the explanation. I finally understand you initial note.
I was thinking about org-overview mostly because it is the case when
next/previous-line was extremely slow with many overlays jammed between
two subsequent lines. 

>> Now, thinking second time about this, using the following for
>> org-flag-region would achieve similar effect:
>>
>> (remove-overlays from to 'invisible 'outline)
>> (remove-overlays from to 'invisible 'org-hide-drawer)
>>
>> Now sure if it is going to break org-cycle though.
>> What do you think?
>
> This is already the case. See first line of `org-flag-region'.

Currently, `org-flag-region' only removes one SPEC type of overlays:

(remove-overlays from to 'invisible spec)

If we change it to 

(remove-overlays from to 'invisible spec)
(when flag
(remove-overlays from to 'invisible 'org-hide-drawer)
...
)

then all the extra drawer overlays in the flagged region will be
removed. It will require re-creating those extra overlays later if the
region is revealed again though. 

Best,
Ihor

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> If you want, I can test the file without :LOGBOOK: lines tomorrow.
>
> Don't worry, it doesn't matter now.
>
>> No, there are only 9 'outline overlays in the folded buffer if we do not
>> create overlays for drawers. This is because outline-hide-sublevels
>> called by org-overview is calling outline-flag-region on the whole
>> buffer thus removing all the 'outline overlays in buffer
>> (remove-overlays from to 'invisible 'outline) and re-creating a single
>> overlay for each top-level heading.
>
> You're talking about "overview" (org-overview), whereas I'm talking
> about "contents view" (org-content). They are not the same. In the
> latter, you show every headline in the buffer, so you have one overlay
> per headline.
>
>> Now, thinking second time about this, using the following for
>> org-flag-region would achieve similar effect:
>>
>> (remove-overlays from to 'invisible 'outline)
>> (remove-overlays from to 'invisible 'org-hide-drawer)
>>
>> Now sure if it is going to break org-cycle though.
>> What do you think?
>
> This is already the case. See first line of `org-flag-region'.
>
> Regards,

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-10 16:30                 ` Ihor Radchenko
@ 2020-05-10 19:32                   ` Nicolas Goaziou
  2020-05-12 10:03                     ` Nicolas Goaziou
  2020-05-17 15:00                     ` Ihor Radchenko
  0 siblings, 2 replies; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-10 19:32 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Ihor Radchenko <yantar92@gmail.com> writes:

> This should be better:
> https://gist.github.com/yantar92/e37c2830d3bb6db8678b14424286c930

Thank you.

> This might get tricky in the following case:
>
> :PROPERTIES:
> :CREATED: [2020-04-13 Mon 22:31]
> <region-beginning>
> :SHOWFROMDATE: 2020-05-11
> :ID:       e05e3b33-71ba-4bbc-abba-8a92c565ad34
> :END:
>
> <many subtrees in between>
>
> :PROPERTIES:
> :CREATED:  [2020-04-27 Mon 13:50]
> <region-end>
> :ID:       b2eef49f-1c5c-4ff1-8e10-80423c8d8532
> :ATTACH_DIR_INHERIT: t
> :END:
>
> If the text in the region is replaced by something else, <many subtrees
> in between> should not be fully hidden. We cannot simply look at the
> 'invisible property before and after the changed region. 

Be careful: "replaced by something else" is ambiguous. "Replacing" is an
unlikely change: you would need to do 

  (setf (buffer-substring x y) "foo")

We can safely assume this will not happen. If it does, we can accept the
subsequent glitch.

Anyway it is less confusing to think in terms of deletion and insertion.
In the case above, you probably mean "the region is deleted then
something else is inserted", or the other way. So there are two actions
going on, i.e., `after-change-functions' are called twice.

In particular the situation you foresee /cannot happen/ with an
insertion. Text is inserted at a single point. Let's assume this is in
the first drawer. Once inserted, both text before and after the new text
were part of the same drawer. Insertion introduces other problems,
though. More on this below.

It is true the deletion can produce the situation above. But in this
case, there is nothing to do, you just merged two drawers into a single
one, which stays invisible. Problem solved.

IOW, big changes like the one you describe are not an issue. I think the
"check if previous and next parts match" trick gives us roughly the same
functionality, and the same glitches, as overlays.

However, I think we can do better than that, and also fix the glitches
from overlays. Here are two of them. Write the following drawer:

  :FOO:
  bar
  :END:

fold it and delete the ":f". The overlay is still there, and you cannot
remove it with TAB any more. Or, with the same initial drawer, from
beginning of buffer, evaluate:

  (progn (re-search-forward ":END:") (replace-match ""))

This is no longer a drawer: you just removed its closing line. Yet, the
overlay is still there, and TAB is ineffective.

Here's an idea to develop that would make folding more robust, and still
fast.

Each syntactical element has a "sensitive part", a particular area that
can change the nature of the element when it is altered. Luckily drawers
(and blocks) are sturdy. For a drawer, there are three things to check:

1. the opening line must match org-drawer-regexp
2. the closing line must match org-property-end-re (case ignored)
3. between those, you must not insert text match org-property-end-re or
   org-outline-regexp-bol

Obviously, point 3 needs not be checked during deletion.

Instead of `after-change-functions', we may use `modification-hooks' for
deletions, and `insert-behind-hooks' for insertions. For example, we
might add modification-hooks property to both opening and closing line,
and `insert-behind-hooks' on all the drawer. If any of the 3 points
above is verified, we remove all properties.

Note that if we can implement something robust with text properties, we
might use them for headlines too, for another significant speed-up.

WDYT?

> I think that using fontification (something similar to
> org-fontify-drawers) instead of after-change-functions should be
> faster.

I don't think it would be faster. With `after-change-functions',
`modification-hooks' or `insert-behind-hook', we know exactly where the
change happened. Fontification is fuzzier. It is not instantaneous
either.

It is an option only if we cannot do something fast and accurate with
`after-change-functions', IMO.



^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-10 17:08                         ` Ihor Radchenko
@ 2020-05-10 19:38                           ` Nicolas Goaziou
  0 siblings, 0 replies; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-10 19:38 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Ihor Radchenko <yantar92@gmail.com> writes:

> Currently, `org-flag-region' only removes one SPEC type of overlays:
>
> (remove-overlays from to 'invisible spec)
>
> If we change it to 
>
> (remove-overlays from to 'invisible spec)
> (when flag
> (remove-overlays from to 'invisible 'org-hide-drawer)
> ...
> )
>
> then all the extra drawer overlays in the flagged region will be
> removed. It will require re-creating those extra overlays later if the
> region is revealed again though. 

Exactly. This would be equivalent to drop `org-hide-drawer' altogether,
which we did for property drawers. You have to fold again every drawer
after each visibility change.

For the record, this is the initial bug `org-hide-drawer' was trying to
solve. Back to square one.

Also, we would have the same problem with blocks.


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-10 19:32                   ` Nicolas Goaziou
@ 2020-05-12 10:03                     ` Nicolas Goaziou
  2020-05-17 15:00                     ` Ihor Radchenko
  1 sibling, 0 replies; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-12 10:03 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Completing myself,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Each syntactical element has a "sensitive part", a particular area that
> can change the nature of the element when it is altered. Luckily drawers
> (and blocks) are sturdy. For a drawer, there are three things to check:
>
> 1. the opening line must match org-drawer-regexp
> 2. the closing line must match org-property-end-re (case ignored)
> 3. between those, you must not insert text match org-property-end-re or
>    org-outline-regexp-bol
>
> Obviously, point 3 needs not be checked during deletion.

Point 3 above is inaccurate, one also needs to check that

  "^[ \t]#\\+end[:_]"

doesn't match the body, either.

> Instead of `after-change-functions', we may use `modification-hooks' for
> deletions, and `insert-behind-hooks' for insertions. For example, we
> might add modification-hooks property to both opening and closing line,
> and `insert-behind-hooks' on all the drawer. If any of the 3 points
> above is verified, we remove all properties.
>
> Note that if we can implement something robust with text properties, we
> might use them for headlines too, for another significant speed-up.

Another, less ambitious, possibility is to expand the drawer as soon as
text is inserted or removed in the invisible part. Callers (e.g.,
`org-entry-put') are then responsible to fold it again, if necessary.


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-10 19:32                   ` Nicolas Goaziou
  2020-05-12 10:03                     ` Nicolas Goaziou
@ 2020-05-17 15:00                     ` Ihor Radchenko
  2020-05-17 15:40                       ` Ihor Radchenko
  1 sibling, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-17 15:00 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 6013 bytes --]

Hi,

[All the changes below are relative to commit ed0e75d24. Later commits
make it hard to distinguish between hidden headlines and drawers. I will
need to figure out a way to merge this branch with master. It does not
seem to be trivial.]

I have finished a seemingly stable implementation of handling changes
inside drawer and block elements. For now, I did not bother with
'modification-hooks and 'insert-in-font/behind-hooks, but simply used
before/after-change-functions.

The basic idea is saving parsed org-elements before the modification
(with :begin and :end replaced by markers) and comparing them with the 
versions of the same elements after the modification.
Any valid org element can be examined in such way by an arbitrary
function (see org-track-modification-elements) [1].

For now, I have implemented tracking changes in all the drawer and block
elements. If the contents of an element is changed and the element is
hidden, the contents remains hidden unless the change was done with
self-insert-command. If the begin/end line of the element was changed in
the way that the element changes the type or extends/shrinks, the
element contents is revealed. To illustrate:

Case #1 (the element content is hidden):

:PROPERTIES:
:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
:END:

is changed to

:ROPERTIES:
:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
:END:

Text is revealed, because we have drawer in place of property-drawer

Case #2 (the element content is hidden):

:ROPERTIES:
:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
:END:

is changed to

:OPERTIES:
:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
:END:

The text remains hidden since it is still a drawer.

Case #3: (the element content is hidden):

:FOO:
bar
tmp
:END:

is changed to

:FOO:
bar
:END:
tmp
:END:

The text is revealed because the drawer contents shrank.

Case #4: (the element content is hidden in both the drawers):

:FOO:
bar
tmp
:END:
:BAR:
jjd
:END:

is changed to

:FOO:
bar
tmp
:BAR:
jjd
:END:

The text is revealed in both the drawers because the drawers are merged
into a new drawer.

> However, I think we can do better than that, and also fix the glitches
> from overlays. Here are two of them. Write the following drawer:
>
>   :FOO:
>   bar
>   :END:
>
> fold it and delete the ":f". The overlay is still there, and you cannot
> remove it with TAB any more. Or, with the same initial drawer, from
> beginning of buffer, evaluate:
>
>   (progn (re-search-forward ":END:") (replace-match ""))
>
> This is no longer a drawer: you just removed its closing line. Yet, the
> overlay is still there, and TAB is ineffective.

I think the above examples cover what you described.

Case #5 (the element content is hidden, point at <!>):

:FOO:<!>
bar
tmp
:END:

is changed (via self-insert-command) to

:FOO:a<!>
bar
tmp
:END:

The text is revealed.

This last case sounds logical and might potentially replace
org-catch-invisible-edits.

------------------------------------------------------------------------

Some potential issues with the implementation:

1. org--after-element-change-function can called many times even for
trivial operations. For example (insert "\n" ":TEST:") seems to call it
two times already. This has two implications: (1) potential performance
degradation; (2) org-element library may not be able to parse the
changed element because its intermediate modified state may not match
the element syntax. Specifically, inserting new property into
:PROPERTIES: drawer inserts a newline at some point, which makes
org-element-at-point think that it is not a 'property-drawer, but just
'drawer.

For (1), I did not really do any workaround for now. One potential way
is making use of combine-after-change-calls (info:elisp#Change Hooks).
At least, within org source code. 

For (2), I have introduced org--property-drawer-modified-re to override
org-property-drawer-re in relevant *-change-function. This seems to work
for property drawers. However, I am not sure if similar problem may
happen in some border cases with ordinary drawers or blocks. 

2. I have noticed that results of org-element-at-point and
org-element-parse-buffer are not always consistent.
In my tests, they returned different number of empty lines after drawers
(:post-blank and :end properties). I am not sure here if I did something
wrong in the code or if it is a real issue in org-element.

For now, I simply called org-element-at-point with point at :begin
property of all the elements returned by org-element-parse buffer to
make things consistent. This indeed introduced overhead, but I do not
see other way to solve the inconsistency.

3. This implementation did not directly solve the previously observed
issue with two ellipsis displayed in folded drawer after adding hidden
text inside:

:PROPERTY: ...  -->  :PROPERTY: ...  ...

For now, I just did

(org-hide-drawer-toggle 'off)
(org-hide-drawer-toggle 'hide)

to hide the second ellipsis, but I still don't understand why it is
happening. Is it some Emacs bug? I am not sure.

4. For some reason, before/after-change-functions do not seem to trigger
when adding note after todo state change. 

------------------------------------------------------------------------

Further plans:

1. Investigate the issue with log notes.
2. Try to change headings to use text properties as well.

The current version of the patch (relative to commit ed0e75d24) is
attached. 

------------------------------------------------------------------------

P.S. I have noticed an issue with hidden text on master (9bc0cc7fb) with
my personal config:

For the following .org file:

* TODO b
:PROPERTIES:
:CREATED:  [2020-05-17 Sun 22:37]
:END:

folded to

* TODO b...

Changing todo to DONE will be shown as

* DONE b
CLOSED: [2020-05-17 Sun 22:54]...:LOGBOOK:...

------------------------------------------------------------------------

[1] If one wants to track changes in two elements types, where one is
always inside the other, it is not possible now.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: featuredrawertextprop.patch --]
[-- Type: text/x-diff, Size: 17817 bytes --]

diff --git a/lisp/org-macs.el b/lisp/org-macs.el
index a02f713ca..4b0e23f6a 100644
--- a/lisp/org-macs.el
+++ b/lisp/org-macs.el
@@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead."
 
 
 \f
-;;; Overlays
+;;; Overlays and text properties
 
 (defun org-overlay-display (ovl text &optional face evap)
   "Make overlay OVL display TEXT with face FACE."
@@ -705,18 +705,44 @@ If DELETE is non-nil, delete all those overlays."
 	    (delete (delete-overlay ov))
 	    (t (push ov found))))))
 
+(defun org--find-text-property-region (pos prop)
+  "Find a region containing PROP text property around point POS."
+  (let* ((beg (and (get-text-property pos prop) pos))
+	 (end beg))
+    (when beg
+      ;; when beg is the first point in the region, `previous-single-property-change'
+      ;; will return nil.
+      (setq beg (or (previous-single-property-change pos prop)
+		    beg))
+      ;; when end is the last point in the region, `next-single-property-change'
+      ;; will return nil.
+      (setq end (or (next-single-property-change pos prop)
+		    end))
+      (unless (= beg end) ; this should not happen
+        (cons beg end)))))
+
 (defun org-flag-region (from to flag spec)
   "Hide or show lines from FROM to TO, according to FLAG.
 SPEC is the invisibility spec, as a symbol."
-  (remove-overlays from to 'invisible spec)
-  ;; Use `front-advance' since text right before to the beginning of
-  ;; the overlay belongs to the visible line than to the contents.
-  (when flag
-    (let ((o (make-overlay from to nil 'front-advance)))
-      (overlay-put o 'evaporate t)
-      (overlay-put o 'invisible spec)
-      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
-
+  (pcase spec
+    ('outline
+     (remove-overlays from to 'invisible spec)
+     ;; Use `front-advance' since text right before to the beginning of
+     ;; the overlay belongs to the visible line than to the contents.
+     (when flag
+       (let ((o (make-overlay from to nil 'front-advance)))
+	 (overlay-put o 'evaporate t)
+	 (overlay-put o 'invisible spec)
+	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
+    (_
+     ;; Use text properties instead of overlays for speed.
+     ;; Overlays are too slow (Emacs Bug#35453).
+     (with-silent-modifications
+       (remove-text-properties from to '(invisible nil))
+       (when flag
+	 (put-text-property from to 'rear-non-sticky nil)
+	 (put-text-property from to 'front-sticky t)
+	 (put-text-property from to 'invisible spec))))))
 
 \f
 ;;; Regexp matching
diff --git a/lisp/org.el b/lisp/org.el
index 96e7384f3..1bf90edae 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.")
 (declare-function cdlatex-math-symbol "ext:cdlatex")
 (declare-function Info-goto-node "info" (nodename &optional fork strict-case))
 (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag))
+(declare-function isearch-filter-visible "isearch" (beg end))
 (declare-function org-add-archive-files "org-archive" (files))
 (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom))
 (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour))
@@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.")
 
 (defvar ffap-url-regexp)
 (defvar org-element-paragraph-separate)
+(defvar org-element-all-objects)
+(defvar org-element-all-elements)
+(defvar org-element-greater-elements)
 (defvar org-indent-indentation-per-level)
 (defvar org-radio-target-regexp)
 (defvar org-target-link-regexp)
@@ -4737,6 +4741,153 @@ This is for getting out of special buffers like capture.")
 (defun org-before-change-function (_beg _end)
   "Every change indicates that a table might need an update."
   (setq org-table-may-need-update t))
+
+(defvar-local org--modified-elements nil
+  "List of unmodified versions of recently modified elements.
+
+The :begin and :end element properties contain markers instead of positions.")
+
+(defvar org--property-drawer-modified-re (concat (replace-regexp-in-string "\\$$" "\n" org-property-start-re)
+					      "\\(?:.*\n\\)*?"
+                                              (replace-regexp-in-string "^\\^" "" org-property-end-re))
+  "Matches entire property drawer, including its state during modification.
+
+This should be different from `org-property-drawer-re' because
+property drawer may contain empty or incomplete lines in the middle of
+modification.")
+
+(defun org--drawer-or-block-change-function (el)
+  "Update visibility of changed drawer/block EL.
+
+If text was added to hidden drawer/block,
+make sure that the text is also hidden, unless
+the change was done by `self-insert-command'.
+If the modification destroyed the drawer/block,
+reveal the hidden text in former drawer/block."
+  (save-match-data
+    (save-excursion
+      (save-restriction
+	(goto-char (org-element-property :begin el))
+	(let* ((newel (org-element-at-point))
+               (spec (if (string-match-p "block" (symbol-name (org-element-type el)))
+			 'org-hide-block
+                       (if (string-match-p "drawer" (symbol-name (org-element-type el)))
+			   'org-hide-drawer
+			 t))))
+	  (if (and (equal (org-element-type el) (org-element-type newel))
+		   (equal (marker-position (org-element-property :begin el))
+			  (org-element-property :begin newel))
+		   (equal (marker-position (org-element-property :end el))
+			  (org-element-property :end newel)))
+	      (when (text-property-any (marker-position (org-element-property :begin el))
+				       (marker-position (org-element-property :end el))
+                                       'invisible spec)
+		(if (memq this-command '(self-insert-command))
+		    ;; reveal if change was made by typing
+		    (org-hide-drawer-toggle 'off)
+		  ;; re-hide the inserted text
+		  ;; FIXME: opening the drawer before hiding should not be needed here
+		  (org-hide-drawer-toggle 'off) ; this is needed to avoid showing double ellipsis
+		  (org-hide-drawer-toggle 'hide)))
+            ;; The element was destroyed. Reveal everything.
+            (org-flag-region (marker-position (org-element-property :begin el))
+			  (marker-position (org-element-property :end el))
+			  nil spec)
+            (org-flag-region (org-element-property :begin newel)
+			  (org-element-property :end newel)
+			  nil spec)))))))
+
+(defvar org-track-modification-elements (list (cons 'center-block #'org--drawer-or-block-change-function)
+					      (cons 'drawer #'org--drawer-or-block-change-function)
+					      (cons 'dynamic-block #'org--drawer-or-block-change-function)
+					      (cons 'property-drawer #'org--drawer-or-block-change-function)
+					      (cons 'quote-block #'org--drawer-or-block-change-function)
+					      (cons 'special-block #'org--drawer-or-block-change-function))
+  "Alist of elements to be tracked for modifications.
+Each element of the alist is a cons of an element from
+`org-element-all-elements' and the function used to handle the
+modification.
+The function must accept a single argument - parsed element before
+modificatin with :begin and :end properties containing markers.")
+
+(defun org--find-elements-in-region (beg end elements &optional include-partial)
+  "Find all elements from ELEMENTS list in region BEG . END.
+All the listed elements must be resolvable by `org-element-at-point'.
+Include elements if they are partially inside region when INCLUDE-PARTIAL is non-nil."
+  (when include-partial
+    (org-with-point-at beg
+      (when-let ((new-beg (org-element-property :begin
+					     (org-element-lineage (org-element-at-point)
+							       elements
+							       'with-self))))
+	(setq beg new-beg))
+      (when (memq 'headline elements)
+	(when-let ((new-beg (ignore-error user-error (org-back-to-heading 'include-invisible))))
+          (setq beg new-beg))))
+    (org-with-point-at end
+      (when-let ((new-end (org-element-property :end
+					     (org-element-lineage (org-element-at-point)
+							       elements
+							       'with-self))))
+	(setq end new-end))
+      (when (memq 'headline elements)
+	(when-let ((new-end (org-with-limited-levels (outline-next-heading))))
+          (setq end (1- new-end))))))
+  (save-excursion
+    (save-restriction
+      (narrow-to-region beg end)
+      (let (has-object has-element has-greater-element granularity)
+	(dolist (el elements)
+	  (when (memq el org-element-all-objects) (setq has-object t))
+	  (when (memq el org-element-all-elements) (setq has-element t))
+	  (when (memq el org-element-greater-elements) (setq has-greater-element t)))
+	(if has-object
+	    (setq granularity 'object)
+	  (if has-greater-element
+              (setq granularity 'greater-element)
+            (if has-element
+		(setq granularity 'element)
+              (setq granularity 'headline))))
+	(org-element-map (org-element-parse-buffer granularity) elements #'identity)))))
+
+(defun org--before-element-change-function (beg end)
+  "Register upcoming element modifications in `org--modified-elements' for all elements interesting with BEG END."
+  (let ((org-property-drawer-re org--property-drawer-modified-re))
+    (save-match-data
+      (save-excursion
+	(save-restriction
+	  (dolist (el (org--find-elements-in-region beg
+						 end
+						 (mapcar #'car org-track-modification-elements)
+						 'include-partial))
+            ;; `org-element-at-point' is not consistent with results
+	    ;; of `org-element-parse-buffer' for :post-blank and :end
+            ;; Using `org-element-at-point to keep consistent
+            ;; parse results with `org--after-element-change-function'
+	    (let* ((el (org-with-point-at (org-element-property :begin el)
+			 (org-element-at-point)))
+		   (beg-marker (copy-marker (org-element-property :begin el) 't))
+		   (end-marker (copy-marker (org-element-property :end el) 't)))
+	      (when (and (marker-position beg-marker) (marker-position end-marker))
+		(org-element-put-property el :begin beg-marker)
+		(org-element-put-property el :end end-marker)
+		(add-to-list 'org--modified-elements el)))))))))
+
+;; FIXME: this function may be called many times during routine modifications
+;; The normal way to avoid this is `combine-after-change-calls' - not
+;; the case in most org primitives.
+(defun org--after-element-change-function (&rest _)
+  "Handle changed elements from `org--modified-elements'."
+  (let ((org-property-drawer-re org--property-drawer-modified-re))
+    (dolist (el org--modified-elements)
+      (save-match-data
+	(save-excursion
+          (save-restriction
+	    (let* ((type (org-element-type el))
+		   (change-func (alist-get type org-track-modification-elements)))
+	      (funcall (symbol-function change-func) el)))))))
+  (setq org--modified-elements nil))
+
 (defvar org-mode-map)
 (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param.
 (defvar org-agenda-keep-modes nil)      ; Dynamically-scoped param.
@@ -4818,6 +4969,9 @@ The following commands are available:
   ;; Activate before-change-function
   (setq-local org-table-may-need-update t)
   (add-hook 'before-change-functions 'org-before-change-function nil 'local)
+  (add-hook 'before-change-functions 'org--before-element-change-function nil 'local)
+  ;; Activate after-change-function
+  (add-hook 'after-change-functions 'org--after-element-change-function nil 'local)
   ;; Check for running clock before killing a buffer
   (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local)
   ;; Initialize macros templates.
@@ -4869,6 +5023,10 @@ The following commands are available:
   (setq-local outline-isearch-open-invisible-function
 	      (lambda (&rest _) (org-show-context 'isearch)))
 
+  ;; Make isearch search in blocks hidden via text properties
+  (setq-local isearch-filter-predicate #'org--isearch-filter-predicate)
+  (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local)
+
   ;; Setup the pcomplete hooks
   (setq-local pcomplete-command-completion-function #'org-pcomplete-initial)
   (setq-local pcomplete-command-name-function #'org-command-at-point)
@@ -5859,9 +6017,26 @@ If TAG is a number, get the corresponding match group."
 	 (inhibit-modification-hooks t)
 	 deactivate-mark buffer-file-name buffer-file-truename)
     (decompose-region beg end)
+    ;; do not remove invisible text properties specified by
+    ;; 'org-hide-block and 'org-hide-drawer (but remove  'org-link)
+    ;; this is needed to keep the drawers and blocks hidden unless
+    ;; they are toggled by user
+    ;; Note: The below may be too specific and create troubles
+    ;; if more invisibility specs are added to org in future
+    (let ((pos beg)
+	  next spec)
+      (while (< pos end)
+	(setq next (next-single-property-change pos 'invisible nil end)
+              spec (get-text-property pos 'invisible))
+        (unless (memq spec (list 'org-hide-block
+				 'org-hide-drawer))
+          (remove-text-properties pos next '(invisible t)))
+        (setq pos next)))
     (remove-text-properties beg end
 			    '(mouse-face t keymap t org-linked-text t
-					 invisible t intangible t
+					 ;; Do not remove all invisible during fontification
+					 ;; invisible t
+                                         intangible t
 					 org-emphasis t))
     (org-remove-font-lock-display-properties beg end)))
 
@@ -6666,8 +6841,13 @@ information."
     ;; expose it.
     (dolist (o (overlays-at (point)))
       (when (memq (overlay-get o 'invisible)
-		  '(org-hide-block org-hide-drawer outline))
+		  '(outline))
 	(delete-overlay o)))
+    (when (memq (get-text-property (point) 'invisible)
+		'(org-hide-block org-hide-drawer))
+      (let ((spec (get-text-property (point) 'invisible))
+	    (region (org--find-text-property-region (point) 'invisible)))
+	(org-flag-region (car region) (cdr region) nil spec)))
     (unless (org-before-first-heading-p)
       (org-with-limited-levels
        (cl-case detail
@@ -20902,6 +21082,79 @@ Started from `gnus-info-find-node'."
           (t default-org-info-node))))))
 
 \f
+
+;;; Make isearch search in some text hidden via text propertoes
+
+(defvar org--isearch-overlays nil
+  "List of overlays temporarily created during isearch.
+This is used to allow searching in regions hidden via text properties.
+As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays.
+Any text hidden via text properties is not revealed even if `search-invisible'
+is set to 't.")
+
+;; Not sure if it needs to be a user option
+;; One might want to reveal hidden text in, for example, hidden parts of the links.
+;; Currently, hidden text in links is never revealed by isearch.
+(defvar org-isearch-specs '(org-hide-block
+			 org-hide-drawer)
+  "List of text invisibility specs to be searched by isearch.
+By default ([2020-05-09 Sat]), isearch does not search in hidden text,
+which was made invisible using text properties. Isearch will be forced
+to search in hidden text with any of the listed 'invisible property value.")
+
+(defun org--create-isearch-overlays (beg end)
+  "Replace text property invisibility spec by overlays between BEG and END.
+All the regions with invisibility text property spec from
+`org-isearch-specs' will be changed to use overlays instead
+of text properties. The created overlays will be stored in
+`org--isearch-overlays'."
+  (let ((pos beg))
+    (while (< pos end)
+      (when-let* ((spec (get-text-property pos 'invisible))
+		  (spec (memq spec org-isearch-specs))
+		  (region (org--find-text-property-region pos 'invisible)))
+        ;; Changing text properties is considered buffer modification.
+        ;; We do not want it here.
+	(with-silent-modifications
+          ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat]
+          ;; overlay for 'outline blocks.
+          (let ((o (make-overlay (car region) (cdr region) nil 'front-advance)))
+	    (overlay-put o 'evaporate t)
+	    (overlay-put o 'invisible spec)
+            ;; `delete-overlay' here means that spec information will be lost
+            ;; for the region. The region will remain visible.
+	    (overlay-put o 'isearch-open-invisible #'delete-overlay)
+            (push o org--isearch-overlays))
+	  (remove-text-properties (car region) (cdr region) '(invisible nil))))
+      (setq pos (next-single-property-change pos 'invisible nil end)))))
+
+(defun org--isearch-filter-predicate (beg end)
+  "Return non-nil if text between BEG and END is deemed visible by Isearch.
+This function is intended to be used as `isearch-filter-predicate'.
+Unlike `isearch-filter-visible', make text with 'invisible text property
+value listed in `org-isearch-specs' visible to Isearch."
+  (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text
+  (isearch-filter-visible beg end))
+
+(defun org--clear-isearch-overlay (ov)
+  "Convert OV region back into using text properties."
+  (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays
+    ;; Changing text properties is considered buffer modification.
+    ;; We do not want it here.
+    (with-silent-modifications
+      (put-text-property (overlay-start ov) (overlay-end ov) 'invisible spec)))
+  (when (member ov isearch-opened-overlays)
+    (setq isearch-opened-overlays (delete ov isearch-opened-overlays)))
+  (delete-overlay ov))
+
+(defun org--clear-isearch-overlays ()
+  "Convert overlays from `org--isearch-overlays' back into using text properties."
+  (when org--isearch-overlays
+    (mapc #'org--clear-isearch-overlay org--isearch-overlays)
+    (setq org--isearch-overlays nil)))
+
+\f
+
 ;;; Finish up
 
 (add-hook 'org-mode-hook     ;remove overlays when changing major mode

[-- Attachment #3: Type: text/plain, Size: 4435 bytes --]



Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> This should be better:
>> https://gist.github.com/yantar92/e37c2830d3bb6db8678b14424286c930
>
> Thank you.
>
>> This might get tricky in the following case:
>>
>> :PROPERTIES:
>> :CREATED: [2020-04-13 Mon 22:31]
>> <region-beginning>
>> :SHOWFROMDATE: 2020-05-11
>> :ID:       e05e3b33-71ba-4bbc-abba-8a92c565ad34
>> :END:
>>
>> <many subtrees in between>
>>
>> :PROPERTIES:
>> :CREATED:  [2020-04-27 Mon 13:50]
>> <region-end>
>> :ID:       b2eef49f-1c5c-4ff1-8e10-80423c8d8532
>> :ATTACH_DIR_INHERIT: t
>> :END:
>>
>> If the text in the region is replaced by something else, <many subtrees
>> in between> should not be fully hidden. We cannot simply look at the
>> 'invisible property before and after the changed region. 
>
> Be careful: "replaced by something else" is ambiguous. "Replacing" is an
> unlikely change: you would need to do 
>
>   (setf (buffer-substring x y) "foo")
>
> We can safely assume this will not happen. If it does, we can accept the
> subsequent glitch.
>
> Anyway it is less confusing to think in terms of deletion and insertion.
> In the case above, you probably mean "the region is deleted then
> something else is inserted", or the other way. So there are two actions
> going on, i.e., `after-change-functions' are called twice.
>
> In particular the situation you foresee /cannot happen/ with an
> insertion. Text is inserted at a single point. Let's assume this is in
> the first drawer. Once inserted, both text before and after the new text
> were part of the same drawer. Insertion introduces other problems,
> though. More on this below.
>
> It is true the deletion can produce the situation above. But in this
> case, there is nothing to do, you just merged two drawers into a single
> one, which stays invisible. Problem solved.
>
> IOW, big changes like the one you describe are not an issue. I think the
> "check if previous and next parts match" trick gives us roughly the same
> functionality, and the same glitches, as overlays.
>
> However, I think we can do better than that, and also fix the glitches
> from overlays. Here are two of them. Write the following drawer:
>
>   :FOO:
>   bar
>   :END:
>
> fold it and delete the ":f". The overlay is still there, and you cannot
> remove it with TAB any more. Or, with the same initial drawer, from
> beginning of buffer, evaluate:
>
>   (progn (re-search-forward ":END:") (replace-match ""))
>
> This is no longer a drawer: you just removed its closing line. Yet, the
> overlay is still there, and TAB is ineffective.
>
> Here's an idea to develop that would make folding more robust, and still
> fast.
>
> Each syntactical element has a "sensitive part", a particular area that
> can change the nature of the element when it is altered. Luckily drawers
> (and blocks) are sturdy. For a drawer, there are three things to check:
>
> 1. the opening line must match org-drawer-regexp
> 2. the closing line must match org-property-end-re (case ignored)
> 3. between those, you must not insert text match org-property-end-re or
>    org-outline-regexp-bol
>
> Obviously, point 3 needs not be checked during deletion.
>
> Instead of `after-change-functions', we may use `modification-hooks' for
> deletions, and `insert-behind-hooks' for insertions. For example, we
> might add modification-hooks property to both opening and closing line,
> and `insert-behind-hooks' on all the drawer. If any of the 3 points
> above is verified, we remove all properties.
>
> Note that if we can implement something robust with text properties, we
> might use them for headlines too, for another significant speed-up.
>
> WDYT?
>
>> I think that using fontification (something similar to
>> org-fontify-drawers) instead of after-change-functions should be
>> faster.
>
> I don't think it would be faster. With `after-change-functions',
> `modification-hooks' or `insert-behind-hook', we know exactly where the
> change happened. Fontification is fuzzier. It is not instantaneous
> either.
>
> It is an option only if we cannot do something fast and accurate with
> `after-change-functions', IMO.
>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-17 15:00                     ` Ihor Radchenko
@ 2020-05-17 15:40                       ` Ihor Radchenko
  2020-05-18 14:35                         ` Nicolas Goaziou
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-17 15:40 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

Dear Nicolas Goaziou,

Apparently my previous email was again refused by your mail server (I
tried to add patch as attachment this time).

The patch is in
https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef

This patch is actually one commit ahead of the patch in the email, fixing
an issue when change function throws an error. I wrapped the call into
with-demoted-errors to avoid potential data loss on error in future.

Best,
Ihor



Ihor Radchenko <yantar92@gmail.com> writes:

> Hi,
>
> [All the changes below are relative to commit ed0e75d24. Later commits
> make it hard to distinguish between hidden headlines and drawers. I will
> need to figure out a way to merge this branch with master. It does not
> seem to be trivial.]
>
> I have finished a seemingly stable implementation of handling changes
> inside drawer and block elements. For now, I did not bother with
> 'modification-hooks and 'insert-in-font/behind-hooks, but simply used
> before/after-change-functions.
>
> The basic idea is saving parsed org-elements before the modification
> (with :begin and :end replaced by markers) and comparing them with the 
> versions of the same elements after the modification.
> Any valid org element can be examined in such way by an arbitrary
> function (see org-track-modification-elements) [1].
>
> For now, I have implemented tracking changes in all the drawer and block
> elements. If the contents of an element is changed and the element is
> hidden, the contents remains hidden unless the change was done with
> self-insert-command. If the begin/end line of the element was changed in
> the way that the element changes the type or extends/shrinks, the
> element contents is revealed. To illustrate:
>
> Case #1 (the element content is hidden):
>
> :PROPERTIES:
> :ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
> :END:
>
> is changed to
>
> :ROPERTIES:
> :ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
> :END:
>
> Text is revealed, because we have drawer in place of property-drawer
>
> Case #2 (the element content is hidden):
>
> :ROPERTIES:
> :ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
> :END:
>
> is changed to
>
> :OPERTIES:
> :ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
> :END:
>
> The text remains hidden since it is still a drawer.
>
> Case #3: (the element content is hidden):
>
> :FOO:
> bar
> tmp
> :END:
>
> is changed to
>
> :FOO:
> bar
> :END:
> tmp
> :END:
>
> The text is revealed because the drawer contents shrank.
>
> Case #4: (the element content is hidden in both the drawers):
>
> :FOO:
> bar
> tmp
> :END:
> :BAR:
> jjd
> :END:
>
> is changed to
>
> :FOO:
> bar
> tmp
> :BAR:
> jjd
> :END:
>
> The text is revealed in both the drawers because the drawers are merged
> into a new drawer.
>
>> However, I think we can do better than that, and also fix the glitches
>> from overlays. Here are two of them. Write the following drawer:
>>
>>   :FOO:
>>   bar
>>   :END:
>>
>> fold it and delete the ":f". The overlay is still there, and you cannot
>> remove it with TAB any more. Or, with the same initial drawer, from
>> beginning of buffer, evaluate:
>>
>>   (progn (re-search-forward ":END:") (replace-match ""))
>>
>> This is no longer a drawer: you just removed its closing line. Yet, the
>> overlay is still there, and TAB is ineffective.
>
> I think the above examples cover what you described.
>
> Case #5 (the element content is hidden, point at <!>):
>
> :FOO:<!>
> bar
> tmp
> :END:
>
> is changed (via self-insert-command) to
>
> :FOO:a<!>
> bar
> tmp
> :END:
>
> The text is revealed.
>
> This last case sounds logical and might potentially replace
> org-catch-invisible-edits.
>
> ------------------------------------------------------------------------
>
> Some potential issues with the implementation:
>
> 1. org--after-element-change-function can called many times even for
> trivial operations. For example (insert "\n" ":TEST:") seems to call it
> two times already. This has two implications: (1) potential performance
> degradation; (2) org-element library may not be able to parse the
> changed element because its intermediate modified state may not match
> the element syntax. Specifically, inserting new property into
> :PROPERTIES: drawer inserts a newline at some point, which makes
> org-element-at-point think that it is not a 'property-drawer, but just
> 'drawer.
>
> For (1), I did not really do any workaround for now. One potential way
> is making use of combine-after-change-calls (info:elisp#Change Hooks).
> At least, within org source code. 
>
> For (2), I have introduced org--property-drawer-modified-re to override
> org-property-drawer-re in relevant *-change-function. This seems to work
> for property drawers. However, I am not sure if similar problem may
> happen in some border cases with ordinary drawers or blocks. 
>
> 2. I have noticed that results of org-element-at-point and
> org-element-parse-buffer are not always consistent.
> In my tests, they returned different number of empty lines after drawers
> (:post-blank and :end properties). I am not sure here if I did something
> wrong in the code or if it is a real issue in org-element.
>
> For now, I simply called org-element-at-point with point at :begin
> property of all the elements returned by org-element-parse buffer to
> make things consistent. This indeed introduced overhead, but I do not
> see other way to solve the inconsistency.
>
> 3. This implementation did not directly solve the previously observed
> issue with two ellipsis displayed in folded drawer after adding hidden
> text inside:
>
> :PROPERTY: ...  -->  :PROPERTY: ...  ...
>
> For now, I just did
>
> (org-hide-drawer-toggle 'off)
> (org-hide-drawer-toggle 'hide)
>
> to hide the second ellipsis, but I still don't understand why it is
> happening. Is it some Emacs bug? I am not sure.
>
> 4. For some reason, before/after-change-functions do not seem to trigger
> when adding note after todo state change. 
>
> ------------------------------------------------------------------------
>
> Further plans:
>
> 1. Investigate the issue with log notes.
> 2. Try to change headings to use text properties as well.
>
> The current version of the patch (relative to commit ed0e75d24) is
> attached. 
>
> ------------------------------------------------------------------------
>
> P.S. I have noticed an issue with hidden text on master (9bc0cc7fb) with
> my personal config:
>
> For the following .org file:
>
> * TODO b
> :PROPERTIES:
> :CREATED:  [2020-05-17 Sun 22:37]
> :END:
>
> folded to
>
> * TODO b...
>
> Changing todo to DONE will be shown as
>
> * DONE b
> CLOSED: [2020-05-17 Sun 22:54]...:LOGBOOK:...
>
> ------------------------------------------------------------------------
>
> [1] If one wants to track changes in two elements types, where one is
> always inside the other, it is not possible now.
>
> Best,
> Ihor
>
> diff --git a/lisp/org-macs.el b/lisp/org-macs.el
> index a02f713ca..4b0e23f6a 100644
> --- a/lisp/org-macs.el
> +++ b/lisp/org-macs.el
> @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead."
>  
>  
>  \f
> -;;; Overlays
> +;;; Overlays and text properties
>  
>  (defun org-overlay-display (ovl text &optional face evap)
>    "Make overlay OVL display TEXT with face FACE."
> @@ -705,18 +705,44 @@ If DELETE is non-nil, delete all those overlays."
>  	    (delete (delete-overlay ov))
>  	    (t (push ov found))))))
>  
> +(defun org--find-text-property-region (pos prop)
> +  "Find a region containing PROP text property around point POS."
> +  (let* ((beg (and (get-text-property pos prop) pos))
> +	 (end beg))
> +    (when beg
> +      ;; when beg is the first point in the region, `previous-single-property-change'
> +      ;; will return nil.
> +      (setq beg (or (previous-single-property-change pos prop)
> +		    beg))
> +      ;; when end is the last point in the region, `next-single-property-change'
> +      ;; will return nil.
> +      (setq end (or (next-single-property-change pos prop)
> +		    end))
> +      (unless (= beg end) ; this should not happen
> +        (cons beg end)))))
> +
>  (defun org-flag-region (from to flag spec)
>    "Hide or show lines from FROM to TO, according to FLAG.
>  SPEC is the invisibility spec, as a symbol."
> -  (remove-overlays from to 'invisible spec)
> -  ;; Use `front-advance' since text right before to the beginning of
> -  ;; the overlay belongs to the visible line than to the contents.
> -  (when flag
> -    (let ((o (make-overlay from to nil 'front-advance)))
> -      (overlay-put o 'evaporate t)
> -      (overlay-put o 'invisible spec)
> -      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
> -
> +  (pcase spec
> +    ('outline
> +     (remove-overlays from to 'invisible spec)
> +     ;; Use `front-advance' since text right before to the beginning of
> +     ;; the overlay belongs to the visible line than to the contents.
> +     (when flag
> +       (let ((o (make-overlay from to nil 'front-advance)))
> +	 (overlay-put o 'evaporate t)
> +	 (overlay-put o 'invisible spec)
> +	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
> +    (_
> +     ;; Use text properties instead of overlays for speed.
> +     ;; Overlays are too slow (Emacs Bug#35453).
> +     (with-silent-modifications
> +       (remove-text-properties from to '(invisible nil))
> +       (when flag
> +	 (put-text-property from to 'rear-non-sticky nil)
> +	 (put-text-property from to 'front-sticky t)
> +	 (put-text-property from to 'invisible spec))))))
>  
>  \f
>  ;;; Regexp matching
> diff --git a/lisp/org.el b/lisp/org.el
> index 96e7384f3..1bf90edae 100644
> --- a/lisp/org.el
> +++ b/lisp/org.el
> @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.")
>  (declare-function cdlatex-math-symbol "ext:cdlatex")
>  (declare-function Info-goto-node "info" (nodename &optional fork strict-case))
>  (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag))
> +(declare-function isearch-filter-visible "isearch" (beg end))
>  (declare-function org-add-archive-files "org-archive" (files))
>  (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom))
>  (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour))
> @@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.")
>  
>  (defvar ffap-url-regexp)
>  (defvar org-element-paragraph-separate)
> +(defvar org-element-all-objects)
> +(defvar org-element-all-elements)
> +(defvar org-element-greater-elements)
>  (defvar org-indent-indentation-per-level)
>  (defvar org-radio-target-regexp)
>  (defvar org-target-link-regexp)
> @@ -4737,6 +4741,153 @@ This is for getting out of special buffers like capture.")
>  (defun org-before-change-function (_beg _end)
>    "Every change indicates that a table might need an update."
>    (setq org-table-may-need-update t))
> +
> +(defvar-local org--modified-elements nil
> +  "List of unmodified versions of recently modified elements.
> +
> +The :begin and :end element properties contain markers instead of positions.")
> +
> +(defvar org--property-drawer-modified-re (concat (replace-regexp-in-string "\\$$" "\n" org-property-start-re)
> +					      "\\(?:.*\n\\)*?"
> +                                              (replace-regexp-in-string "^\\^" "" org-property-end-re))
> +  "Matches entire property drawer, including its state during modification.
> +
> +This should be different from `org-property-drawer-re' because
> +property drawer may contain empty or incomplete lines in the middle of
> +modification.")
> +
> +(defun org--drawer-or-block-change-function (el)
> +  "Update visibility of changed drawer/block EL.
> +
> +If text was added to hidden drawer/block,
> +make sure that the text is also hidden, unless
> +the change was done by `self-insert-command'.
> +If the modification destroyed the drawer/block,
> +reveal the hidden text in former drawer/block."
> +  (save-match-data
> +    (save-excursion
> +      (save-restriction
> +	(goto-char (org-element-property :begin el))
> +	(let* ((newel (org-element-at-point))
> +               (spec (if (string-match-p "block" (symbol-name (org-element-type el)))
> +			 'org-hide-block
> +                       (if (string-match-p "drawer" (symbol-name (org-element-type el)))
> +			   'org-hide-drawer
> +			 t))))
> +	  (if (and (equal (org-element-type el) (org-element-type newel))
> +		   (equal (marker-position (org-element-property :begin el))
> +			  (org-element-property :begin newel))
> +		   (equal (marker-position (org-element-property :end el))
> +			  (org-element-property :end newel)))
> +	      (when (text-property-any (marker-position (org-element-property :begin el))
> +				       (marker-position (org-element-property :end el))
> +                                       'invisible spec)
> +		(if (memq this-command '(self-insert-command))
> +		    ;; reveal if change was made by typing
> +		    (org-hide-drawer-toggle 'off)
> +		  ;; re-hide the inserted text
> +		  ;; FIXME: opening the drawer before hiding should not be needed here
> +		  (org-hide-drawer-toggle 'off) ; this is needed to avoid showing double ellipsis
> +		  (org-hide-drawer-toggle 'hide)))
> +            ;; The element was destroyed. Reveal everything.
> +            (org-flag-region (marker-position (org-element-property :begin el))
> +			  (marker-position (org-element-property :end el))
> +			  nil spec)
> +            (org-flag-region (org-element-property :begin newel)
> +			  (org-element-property :end newel)
> +			  nil spec)))))))
> +
> +(defvar org-track-modification-elements (list (cons 'center-block #'org--drawer-or-block-change-function)
> +					      (cons 'drawer #'org--drawer-or-block-change-function)
> +					      (cons 'dynamic-block #'org--drawer-or-block-change-function)
> +					      (cons 'property-drawer #'org--drawer-or-block-change-function)
> +					      (cons 'quote-block #'org--drawer-or-block-change-function)
> +					      (cons 'special-block #'org--drawer-or-block-change-function))
> +  "Alist of elements to be tracked for modifications.
> +Each element of the alist is a cons of an element from
> +`org-element-all-elements' and the function used to handle the
> +modification.
> +The function must accept a single argument - parsed element before
> +modificatin with :begin and :end properties containing markers.")
> +
> +(defun org--find-elements-in-region (beg end elements &optional include-partial)
> +  "Find all elements from ELEMENTS list in region BEG . END.
> +All the listed elements must be resolvable by `org-element-at-point'.
> +Include elements if they are partially inside region when INCLUDE-PARTIAL is non-nil."
> +  (when include-partial
> +    (org-with-point-at beg
> +      (when-let ((new-beg (org-element-property :begin
> +					     (org-element-lineage (org-element-at-point)
> +							       elements
> +							       'with-self))))
> +	(setq beg new-beg))
> +      (when (memq 'headline elements)
> +	(when-let ((new-beg (ignore-error user-error (org-back-to-heading 'include-invisible))))
> +          (setq beg new-beg))))
> +    (org-with-point-at end
> +      (when-let ((new-end (org-element-property :end
> +					     (org-element-lineage (org-element-at-point)
> +							       elements
> +							       'with-self))))
> +	(setq end new-end))
> +      (when (memq 'headline elements)
> +	(when-let ((new-end (org-with-limited-levels (outline-next-heading))))
> +          (setq end (1- new-end))))))
> +  (save-excursion
> +    (save-restriction
> +      (narrow-to-region beg end)
> +      (let (has-object has-element has-greater-element granularity)
> +	(dolist (el elements)
> +	  (when (memq el org-element-all-objects) (setq has-object t))
> +	  (when (memq el org-element-all-elements) (setq has-element t))
> +	  (when (memq el org-element-greater-elements) (setq has-greater-element t)))
> +	(if has-object
> +	    (setq granularity 'object)
> +	  (if has-greater-element
> +              (setq granularity 'greater-element)
> +            (if has-element
> +		(setq granularity 'element)
> +              (setq granularity 'headline))))
> +	(org-element-map (org-element-parse-buffer granularity) elements #'identity)))))
> +
> +(defun org--before-element-change-function (beg end)
> +  "Register upcoming element modifications in `org--modified-elements' for all elements interesting with BEG END."
> +  (let ((org-property-drawer-re org--property-drawer-modified-re))
> +    (save-match-data
> +      (save-excursion
> +	(save-restriction
> +	  (dolist (el (org--find-elements-in-region beg
> +						 end
> +						 (mapcar #'car org-track-modification-elements)
> +						 'include-partial))
> +            ;; `org-element-at-point' is not consistent with results
> +	    ;; of `org-element-parse-buffer' for :post-blank and :end
> +            ;; Using `org-element-at-point to keep consistent
> +            ;; parse results with `org--after-element-change-function'
> +	    (let* ((el (org-with-point-at (org-element-property :begin el)
> +			 (org-element-at-point)))
> +		   (beg-marker (copy-marker (org-element-property :begin el) 't))
> +		   (end-marker (copy-marker (org-element-property :end el) 't)))
> +	      (when (and (marker-position beg-marker) (marker-position end-marker))
> +		(org-element-put-property el :begin beg-marker)
> +		(org-element-put-property el :end end-marker)
> +		(add-to-list 'org--modified-elements el)))))))))
> +
> +;; FIXME: this function may be called many times during routine modifications
> +;; The normal way to avoid this is `combine-after-change-calls' - not
> +;; the case in most org primitives.
> +(defun org--after-element-change-function (&rest _)
> +  "Handle changed elements from `org--modified-elements'."
> +  (let ((org-property-drawer-re org--property-drawer-modified-re))
> +    (dolist (el org--modified-elements)
> +      (save-match-data
> +	(save-excursion
> +          (save-restriction
> +	    (let* ((type (org-element-type el))
> +		   (change-func (alist-get type org-track-modification-elements)))
> +	      (funcall (symbol-function change-func) el)))))))
> +  (setq org--modified-elements nil))
> +
>  (defvar org-mode-map)
>  (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param.
>  (defvar org-agenda-keep-modes nil)      ; Dynamically-scoped param.
> @@ -4818,6 +4969,9 @@ The following commands are available:
>    ;; Activate before-change-function
>    (setq-local org-table-may-need-update t)
>    (add-hook 'before-change-functions 'org-before-change-function nil 'local)
> +  (add-hook 'before-change-functions 'org--before-element-change-function nil 'local)
> +  ;; Activate after-change-function
> +  (add-hook 'after-change-functions 'org--after-element-change-function nil 'local)
>    ;; Check for running clock before killing a buffer
>    (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local)
>    ;; Initialize macros templates.
> @@ -4869,6 +5023,10 @@ The following commands are available:
>    (setq-local outline-isearch-open-invisible-function
>  	      (lambda (&rest _) (org-show-context 'isearch)))
>  
> +  ;; Make isearch search in blocks hidden via text properties
> +  (setq-local isearch-filter-predicate #'org--isearch-filter-predicate)
> +  (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local)
> +
>    ;; Setup the pcomplete hooks
>    (setq-local pcomplete-command-completion-function #'org-pcomplete-initial)
>    (setq-local pcomplete-command-name-function #'org-command-at-point)
> @@ -5859,9 +6017,26 @@ If TAG is a number, get the corresponding match group."
>  	 (inhibit-modification-hooks t)
>  	 deactivate-mark buffer-file-name buffer-file-truename)
>      (decompose-region beg end)
> +    ;; do not remove invisible text properties specified by
> +    ;; 'org-hide-block and 'org-hide-drawer (but remove  'org-link)
> +    ;; this is needed to keep the drawers and blocks hidden unless
> +    ;; they are toggled by user
> +    ;; Note: The below may be too specific and create troubles
> +    ;; if more invisibility specs are added to org in future
> +    (let ((pos beg)
> +	  next spec)
> +      (while (< pos end)
> +	(setq next (next-single-property-change pos 'invisible nil end)
> +              spec (get-text-property pos 'invisible))
> +        (unless (memq spec (list 'org-hide-block
> +				 'org-hide-drawer))
> +          (remove-text-properties pos next '(invisible t)))
> +        (setq pos next)))
>      (remove-text-properties beg end
>  			    '(mouse-face t keymap t org-linked-text t
> -					 invisible t intangible t
> +					 ;; Do not remove all invisible during fontification
> +					 ;; invisible t
> +                                         intangible t
>  					 org-emphasis t))
>      (org-remove-font-lock-display-properties beg end)))
>  
> @@ -6666,8 +6841,13 @@ information."
>      ;; expose it.
>      (dolist (o (overlays-at (point)))
>        (when (memq (overlay-get o 'invisible)
> -		  '(org-hide-block org-hide-drawer outline))
> +		  '(outline))
>  	(delete-overlay o)))
> +    (when (memq (get-text-property (point) 'invisible)
> +		'(org-hide-block org-hide-drawer))
> +      (let ((spec (get-text-property (point) 'invisible))
> +	    (region (org--find-text-property-region (point) 'invisible)))
> +	(org-flag-region (car region) (cdr region) nil spec)))
>      (unless (org-before-first-heading-p)
>        (org-with-limited-levels
>         (cl-case detail
> @@ -20902,6 +21082,79 @@ Started from `gnus-info-find-node'."
>            (t default-org-info-node))))))
>  
>  \f
> +
> +;;; Make isearch search in some text hidden via text propertoes
> +
> +(defvar org--isearch-overlays nil
> +  "List of overlays temporarily created during isearch.
> +This is used to allow searching in regions hidden via text properties.
> +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays.
> +Any text hidden via text properties is not revealed even if `search-invisible'
> +is set to 't.")
> +
> +;; Not sure if it needs to be a user option
> +;; One might want to reveal hidden text in, for example, hidden parts of the links.
> +;; Currently, hidden text in links is never revealed by isearch.
> +(defvar org-isearch-specs '(org-hide-block
> +			 org-hide-drawer)
> +  "List of text invisibility specs to be searched by isearch.
> +By default ([2020-05-09 Sat]), isearch does not search in hidden text,
> +which was made invisible using text properties. Isearch will be forced
> +to search in hidden text with any of the listed 'invisible property value.")
> +
> +(defun org--create-isearch-overlays (beg end)
> +  "Replace text property invisibility spec by overlays between BEG and END.
> +All the regions with invisibility text property spec from
> +`org-isearch-specs' will be changed to use overlays instead
> +of text properties. The created overlays will be stored in
> +`org--isearch-overlays'."
> +  (let ((pos beg))
> +    (while (< pos end)
> +      (when-let* ((spec (get-text-property pos 'invisible))
> +		  (spec (memq spec org-isearch-specs))
> +		  (region (org--find-text-property-region pos 'invisible)))
> +        ;; Changing text properties is considered buffer modification.
> +        ;; We do not want it here.
> +	(with-silent-modifications
> +          ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat]
> +          ;; overlay for 'outline blocks.
> +          (let ((o (make-overlay (car region) (cdr region) nil 'front-advance)))
> +	    (overlay-put o 'evaporate t)
> +	    (overlay-put o 'invisible spec)
> +            ;; `delete-overlay' here means that spec information will be lost
> +            ;; for the region. The region will remain visible.
> +	    (overlay-put o 'isearch-open-invisible #'delete-overlay)
> +            (push o org--isearch-overlays))
> +	  (remove-text-properties (car region) (cdr region) '(invisible nil))))
> +      (setq pos (next-single-property-change pos 'invisible nil end)))))
> +
> +(defun org--isearch-filter-predicate (beg end)
> +  "Return non-nil if text between BEG and END is deemed visible by Isearch.
> +This function is intended to be used as `isearch-filter-predicate'.
> +Unlike `isearch-filter-visible', make text with 'invisible text property
> +value listed in `org-isearch-specs' visible to Isearch."
> +  (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text
> +  (isearch-filter-visible beg end))
> +
> +(defun org--clear-isearch-overlay (ov)
> +  "Convert OV region back into using text properties."
> +  (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays
> +    ;; Changing text properties is considered buffer modification.
> +    ;; We do not want it here.
> +    (with-silent-modifications
> +      (put-text-property (overlay-start ov) (overlay-end ov) 'invisible spec)))
> +  (when (member ov isearch-opened-overlays)
> +    (setq isearch-opened-overlays (delete ov isearch-opened-overlays)))
> +  (delete-overlay ov))
> +
> +(defun org--clear-isearch-overlays ()
> +  "Convert overlays from `org--isearch-overlays' back into using text properties."
> +  (when org--isearch-overlays
> +    (mapc #'org--clear-isearch-overlay org--isearch-overlays)
> +    (setq org--isearch-overlays nil)))
> +
> +\f
> +
>  ;;; Finish up
>  
>  (add-hook 'org-mode-hook     ;remove overlays when changing major mode
>
>
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>> This should be better:
>>> https://gist.github.com/yantar92/e37c2830d3bb6db8678b14424286c930
>>
>> Thank you.
>>
>>> This might get tricky in the following case:
>>>
>>> :PROPERTIES:
>>> :CREATED: [2020-04-13 Mon 22:31]
>>> <region-beginning>
>>> :SHOWFROMDATE: 2020-05-11
>>> :ID:       e05e3b33-71ba-4bbc-abba-8a92c565ad34
>>> :END:
>>>
>>> <many subtrees in between>
>>>
>>> :PROPERTIES:
>>> :CREATED:  [2020-04-27 Mon 13:50]
>>> <region-end>
>>> :ID:       b2eef49f-1c5c-4ff1-8e10-80423c8d8532
>>> :ATTACH_DIR_INHERIT: t
>>> :END:
>>>
>>> If the text in the region is replaced by something else, <many subtrees
>>> in between> should not be fully hidden. We cannot simply look at the
>>> 'invisible property before and after the changed region. 
>>
>> Be careful: "replaced by something else" is ambiguous. "Replacing" is an
>> unlikely change: you would need to do 
>>
>>   (setf (buffer-substring x y) "foo")
>>
>> We can safely assume this will not happen. If it does, we can accept the
>> subsequent glitch.
>>
>> Anyway it is less confusing to think in terms of deletion and insertion.
>> In the case above, you probably mean "the region is deleted then
>> something else is inserted", or the other way. So there are two actions
>> going on, i.e., `after-change-functions' are called twice.
>>
>> In particular the situation you foresee /cannot happen/ with an
>> insertion. Text is inserted at a single point. Let's assume this is in
>> the first drawer. Once inserted, both text before and after the new text
>> were part of the same drawer. Insertion introduces other problems,
>> though. More on this below.
>>
>> It is true the deletion can produce the situation above. But in this
>> case, there is nothing to do, you just merged two drawers into a single
>> one, which stays invisible. Problem solved.
>>
>> IOW, big changes like the one you describe are not an issue. I think the
>> "check if previous and next parts match" trick gives us roughly the same
>> functionality, and the same glitches, as overlays.
>>
>> However, I think we can do better than that, and also fix the glitches
>> from overlays. Here are two of them. Write the following drawer:
>>
>>   :FOO:
>>   bar
>>   :END:
>>
>> fold it and delete the ":f". The overlay is still there, and you cannot
>> remove it with TAB any more. Or, with the same initial drawer, from
>> beginning of buffer, evaluate:
>>
>>   (progn (re-search-forward ":END:") (replace-match ""))
>>
>> This is no longer a drawer: you just removed its closing line. Yet, the
>> overlay is still there, and TAB is ineffective.
>>
>> Here's an idea to develop that would make folding more robust, and still
>> fast.
>>
>> Each syntactical element has a "sensitive part", a particular area that
>> can change the nature of the element when it is altered. Luckily drawers
>> (and blocks) are sturdy. For a drawer, there are three things to check:
>>
>> 1. the opening line must match org-drawer-regexp
>> 2. the closing line must match org-property-end-re (case ignored)
>> 3. between those, you must not insert text match org-property-end-re or
>>    org-outline-regexp-bol
>>
>> Obviously, point 3 needs not be checked during deletion.
>>
>> Instead of `after-change-functions', we may use `modification-hooks' for
>> deletions, and `insert-behind-hooks' for insertions. For example, we
>> might add modification-hooks property to both opening and closing line,
>> and `insert-behind-hooks' on all the drawer. If any of the 3 points
>> above is verified, we remove all properties.
>>
>> Note that if we can implement something robust with text properties, we
>> might use them for headlines too, for another significant speed-up.
>>
>> WDYT?
>>
>>> I think that using fontification (something similar to
>>> org-fontify-drawers) instead of after-change-functions should be
>>> faster.
>>
>> I don't think it would be faster. With `after-change-functions',
>> `modification-hooks' or `insert-behind-hook', we know exactly where the
>> change happened. Fontification is fuzzier. It is not instantaneous
>> either.
>>
>> It is an option only if we cannot do something fast and accurate with
>> `after-change-functions', IMO.
>>
>
> -- 
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-17 15:40                       ` Ihor Radchenko
@ 2020-05-18 14:35                         ` Nicolas Goaziou
  2020-05-18 16:52                           ` Ihor Radchenko
  0 siblings, 1 reply; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-18 14:35 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> Apparently my previous email was again refused by your mail server (I
> tried to add patch as attachment this time).

Ah. This is annoying, for you and for me.

> The patch is in
> https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef

Thank you.

>> I have finished a seemingly stable implementation of handling changes
>> inside drawer and block elements. For now, I did not bother with
>> 'modification-hooks and 'insert-in-font/behind-hooks, but simply used
>> before/after-change-functions.
>>
>> The basic idea is saving parsed org-elements before the modification
>> (with :begin and :end replaced by markers) and comparing them with the 
>> versions of the same elements after the modification.
>> Any valid org element can be examined in such way by an arbitrary
>> function (see org-track-modification-elements) [1].

As you noticed, using Org Element is a no-go, unfortunately. Parsing an
element is a O(N) operation by the number of elements before it in
a section. In particular, it is not bounded, and not mitigated by
a cache. For large documents, it is going to be unbearably slow, too.

I don't think the solution is to use combine-after-change-calls either,
because even a single call to `org-element-at-point' can be noticeable
in a very large section. Such low-level code should avoid using the
Element library altogether, except for the initial folding part, which
is interactive.

If you use modification-hooks and al., you don't need to parse anything,
because you can store information as text properties. Therefore, once
the modification happens, you already know where you are (or, at least
where you were before the change).

The ideas I suggested about sensitive parts of elements are worth
exploring, IMO. Do you have any issue with them?

>> For (2), I have introduced org--property-drawer-modified-re to override
>> org-property-drawer-re in relevant *-change-function. This seems to work
>> for property drawers. However, I am not sure if similar problem may
>> happen in some border cases with ordinary drawers or blocks. 

I already specified what parts were "sensitive" in a previous message.

>> 2. I have noticed that results of org-element-at-point and
>> org-element-parse-buffer are not always consistent.

`org-element-at-point' is local, `org-element-parse-buffer' is global.
They are not equivalent, but is it an issue?


Regards,

-- 
Nicolas Goaziou


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-18 14:35                         ` Nicolas Goaziou
@ 2020-05-18 16:52                           ` Ihor Radchenko
  2020-05-19 13:07                             ` Nicolas Goaziou
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-18 16:52 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

> As you noticed, using Org Element is a no-go, unfortunately. Parsing an
> element is a O(N) operation by the number of elements before it in
> a section. In particular, it is not bounded, and not mitigated by
> a cache. For large documents, it is going to be unbearably slow, too.

Ouch. I thought it is faster.
What do you mean by "not mitigated by a cache"?

The reason I would like to utilise org-element parser to make tracking
modifications more robust. Using details of the syntax would make the
code fragile if any modifications are made to syntax in future.
Debugging bugs in modification functions is not easy, according to my
experience.

One possible way to avoid performance issues during modification is
running parser in advance. For example, folding an element may
as well add information about the element to its text properties.
This will not degrade performance of folding since we are already
parsing the element during folding (at least, in
org-hide-drawer-toggle).

The problem with parsing an element during folding is that we cannot
easily detect changes like below without re-parsing.

:PROPERTIES: <folded>
:CREATED: [2020-05-18 Mon]
:END: <- added line
:ID: test
:END:

or even

:PROPERTIES:
:CREATED: [2020-05-18 Mon]
:ID: test
:END: <- delete this line

:DRAWER: <folded, cannot be unfolded if we don't re-parse after deletion>
test
:END:

The re-parsing can be done via regexp, as you suggested, but I don't
like this idea, because it will end up re-implementing
org-element-*-parser. Would it be acceptable to run org-element-*-parser
in after-change-functions?

> If you use modification-hooks and al., you don't need to parse anything,
> because you can store information as text properties. Therefore, once
> the modification happens, you already know where you are (or, at least
> where you were before the change).

> The ideas I suggested about sensitive parts of elements are worth
> exploring, IMO. Do you have any issue with them?

If I understand correctly, it is not as easy.
Consider the following example:

:PROPERTIES:
:CREATED: [2020-05-18 Mon]
<region-beginning>
:ID: example
:END:

<... a lot of text, maybe containing other drawers ...>

Nullam rutrum.
Pellentesque dapibus suscipit ligula.
<region-end>
Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus.

If the region gets deleted, the modification hooks from chars inside
drawer will be called as (hook-function <region-beginning>
<region-end>). So, there is still a need to find the drawer somehow to
mark it as about to be modified (modification hooks are ran before
actual modification).

The only difference between using modification hooks and
before-change-functions is that modification hooks will trigger less
frequently. Considering the performance of org-element-at-point, it is
probably worth doing. Initially, I wanted to avoid it because setting a
single before-change-functions hook sounded cleaner than setting
modification-hooks, insert-behind-hooks, and insert-in-front-hooks.
Moreover, these text properties would be copied by default if one uses 
buffer-substring. Then, the hooks will also trigger later in the yanked
text, which may cause all kinds of bugs.

> `org-element-at-point' is local, `org-element-parse-buffer' is global.
> They are not equivalent, but is it an issue?

It was mostly an annoyance, because they returned different results on
the same element. Specifically, they returned different :post-blank and
:end properties, which does not sound right.

Best,
Ihor




Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Apparently my previous email was again refused by your mail server (I
>> tried to add patch as attachment this time).
>
> Ah. This is annoying, for you and for me.
>
>> The patch is in
>> https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef
>
> Thank you.
>
>>> I have finished a seemingly stable implementation of handling changes
>>> inside drawer and block elements. For now, I did not bother with
>>> 'modification-hooks and 'insert-in-font/behind-hooks, but simply used
>>> before/after-change-functions.
>>>
>>> The basic idea is saving parsed org-elements before the modification
>>> (with :begin and :end replaced by markers) and comparing them with the 
>>> versions of the same elements after the modification.
>>> Any valid org element can be examined in such way by an arbitrary
>>> function (see org-track-modification-elements) [1].
>
> As you noticed, using Org Element is a no-go, unfortunately. Parsing an
> element is a O(N) operation by the number of elements before it in
> a section. In particular, it is not bounded, and not mitigated by
> a cache. For large documents, it is going to be unbearably slow, too.
>
> I don't think the solution is to use combine-after-change-calls either,
> because even a single call to `org-element-at-point' can be noticeable
> in a very large section. Such low-level code should avoid using the
> Element library altogether, except for the initial folding part, which
> is interactive.
>
> If you use modification-hooks and al., you don't need to parse anything,
> because you can store information as text properties. Therefore, once
> the modification happens, you already know where you are (or, at least
> where you were before the change).
>
> The ideas I suggested about sensitive parts of elements are worth
> exploring, IMO. Do you have any issue with them?
>
>>> For (2), I have introduced org--property-drawer-modified-re to override
>>> org-property-drawer-re in relevant *-change-function. This seems to work
>>> for property drawers. However, I am not sure if similar problem may
>>> happen in some border cases with ordinary drawers or blocks. 
>
> I already specified what parts were "sensitive" in a previous message.
>
>>> 2. I have noticed that results of org-element-at-point and
>>> org-element-parse-buffer are not always consistent.
>
> `org-element-at-point' is local, `org-element-parse-buffer' is global.
> They are not equivalent, but is it an issue?
>
>
> Regards,
>
> -- 
> Nicolas Goaziou

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-18 16:52                           ` Ihor Radchenko
@ 2020-05-19 13:07                             ` Nicolas Goaziou
  2020-05-23 13:52                               ` Ihor Radchenko
  0 siblings, 1 reply; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-19 13:07 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

>> As you noticed, using Org Element is a no-go, unfortunately. Parsing an
>> element is a O(N) operation by the number of elements before it in
>> a section. In particular, it is not bounded, and not mitigated by
>> a cache. For large documents, it is going to be unbearably slow, too.
>
> Ouch. I thought it is faster.
> What do you mean by "not mitigated by a cache"?

Parsing starts from the closest headline, every time. So, if Org parses
the Nth element in the entry two times, it really parses 2N elements.

With a cache, assuming the buffer wasn't modified, Org would parse
N elements only. With a smarter cache, with fine grained cache
invalidation, it could also reduce the number of subsequent parsed
elements.

> The reason I would like to utilise org-element parser to make tracking
> modifications more robust. Using details of the syntax would make the
> code fragile if any modifications are made to syntax in future.

I don't think the code would be more fragile. Also, the syntax we're
talking about is not going to be modified anytime soon. Moreover, if
folding breaks, it is usually visible, so the bug will not be unnoticed.

This code is going to be as low-level as it can be.

> Debugging bugs in modification functions is not easy, according to my
> experience.

No, it's not. 

But this is not really related to whether you use Element or not.

> One possible way to avoid performance issues during modification is
> running parser in advance. For example, folding an element may
> as well add information about the element to its text properties.
> This will not degrade performance of folding since we are already
> parsing the element during folding (at least, in
> org-hide-drawer-toggle).

We can use this information stored at fold time. But I'm not even sure
we need it.

> The problem with parsing an element during folding is that we cannot
> easily detect changes like below without re-parsing.

Of course we can. It is only necessary to focus on changes that would
break the structure of the element. This does not entail a full parsing.

> :PROPERTIES: <folded>
> :CREATED: [2020-05-18 Mon]
> :END: <- added line
> :ID: test
> :END:
>
> or even
>
> :PROPERTIES:
> :CREATED: [2020-05-18 Mon]
> :ID: test
> :END: <- delete this line
>
> :DRAWER: <folded, cannot be unfolded if we don't re-parse after deletion>
> test
> :END:

Please have a look at the "sensitive parts" I wrote about. This takes
care of this kind of breakage.

> The re-parsing can be done via regexp, as you suggested, but I don't
> like this idea, because it will end up re-implementing
> org-element-*-parser.

You may have misunderstood my suggestion. See below.

> Would it be acceptable to run org-element-*-parser
> in after-change-functions?

I'd rather not do that. This is unnecessary consing, and matching, etc.

> If I understand correctly, it is not as easy.
> Consider the following example:
>
> :PROPERTIES:
> :CREATED: [2020-05-18 Mon]
> <region-beginning>
> :ID: example
> :END:
>
> <... a lot of text, maybe containing other drawers ...>
>
> Nullam rutrum.
> Pellentesque dapibus suscipit ligula.
> <region-end>
> Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus.
>
> If the region gets deleted, the modification hooks from chars inside
> drawer will be called as (hook-function <region-beginning>
> <region-end>). So, there is still a need to find the drawer somehow to
> mark it as about to be modified (modification hooks are ran before
> actual modification).

If we can stick with `after-change-functions' (or local equivalent),
that's better. It is more predictable than `before-change-functions' and
alike.

If it is a deletion, here is the kind of checks we could do, depending
on when they are performed.

Before actual changes :

  1. The deletion is happening within a folded drawer (unnecessary step
     in local functions).
  2. The change deleted the sensitive line ":END:".
  3. Conclusion : unfold.

Or, after actual changes :

  1. The deletion involves a drawer.
  2. Text properties indicate that the beginning of the propertized part
     of the buffer start with org-drawer-regexp, but doesn't end with
     `org-property-end-re'. A "sensitive part" disappeared!
  3. Conclusion : unfold

This is far away from parsing. IMO, a few checks cover all cases. Let me
know if you have questions about it.

Also, note that the kind of change you describe will happen perhaps
0.01% of the time. Most change are about one character, or a single
line, long.

> The only difference between using modification hooks and
> before-change-functions is that modification hooks will trigger less
> frequently. 

Exactly. Much less frequently. But extra care is required, as you noted
already.

> Considering the performance of org-element-at-point, it is
> probably worth doing. Initially, I wanted to avoid it because setting a
> single before-change-functions hook sounded cleaner than setting
> modification-hooks, insert-behind-hooks, and insert-in-front-hooks.

Well, `before-change-fuctions' and `after-change-functions' are not
clean at all: you modify an unrelated part of the buffer, but still call
those to check if a drawer needs to be unfolded somewhere.

And, more importantly, they are not meant to be used together, i.e., you
cannot assume that a single call to `before-change-functions' always
happens before calling `after-change-functions'. This can be tricky if
you want to use the former to pass information to the latter.

But I understand that they are easier to use than their local
counterparts. If you stick with (before|after)-change-functions, the
function being called needs to drop the ball very quickly if the
modification is not about folding changes. Also, I very much suggest to
stick to only `after-change-functions', if feasible (I think it is), per
above.

> Moreover, these text properties would be copied by default if one uses 
> buffer-substring. Then, the hooks will also trigger later in the yanked
> text, which may cause all kinds of bugs.

Indeed, that would be something to handle specifically. I.e.,
destructive modifications (i.e., those that unfold) could clear such
properties.

It is more work. I don't know if it is worth the trouble if we can get
out quickly of `after-change-functions' for unrelated changes.

> It was mostly an annoyance, because they returned different results on
> the same element. Specifically, they returned different :post-blank and
> :end properties, which does not sound right.

OK. If you have a reproducible recipe, I can look into it and see what
can be done.

Regards,

-- 
Nicolas Goaziou


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-19 13:07                             ` Nicolas Goaziou
@ 2020-05-23 13:52                               ` Ihor Radchenko
  2020-05-23 13:53                                 ` Ihor Radchenko
  2020-05-26  8:33                                 ` Nicolas Goaziou
  0 siblings, 2 replies; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-23 13:52 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

Hello,

[The patch itself will be provided in the following email]

I have five updates from the previous version of the patch:

1. I implemented a simplified version of element parsing to detect
changes in folded drawers or blocks. No computationally expensive calls
of org-element-at-point or org-element-parse-buffer are needed now.

2. The patch is now compatible with master (commit 2e96dc639). I
reverted the earlier change in folding drawers and blocks. Now, they are
back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would
achieve nothing when we use text properties.

3. 'invisible text property can now be nested. This is important, for
example, when text inside drawers contains fontified links (which also
use 'invisible text property to hide parts of the link). Now, the old
'invisible spec is recovered after unfolding.

4. Some outline-* function calls in org referred to outline-flag-region
implementation, which is not in sync with org-flag-region in this patch.
I have implemented their org-* versions and replaced the calls
throughout .el files. Actually, some org-* versions were already
implemented in org, but not used for some reason (or not mentioned in
the manual). I have updated the relevant sections of manual. These
changes might be relevant to org independently of this feature branch.

5. I have managed to get a working version of outline folding via text
properties. However, that approach has a big downside - folding state
cannot be different in indirect buffer when we use text properties. I
have seen packages relying on this feature of org and I do not see any
obvious way to achieve different folding state in indirect buffer while
using text properties for outline folding.

-----------------------------------------------------------------------
-----------------------------------------------------------------------

More details on the new implementation for tracking changes:

> Of course we can. It is only necessary to focus on changes that would
> break the structure of the element. This does not entail a full parsing.

I have limited parsing to matching beginning and end of a drawer/block.
The basic functions are org--get-element-region-at-point,
org--get-next-element-region-at-point, and org--find-elements-in-region.
They are simplified versions of org-element-* parsers and do not require
parsing everything from the beginning of the section.

For now, I keep everything in org.el, but those simplified parsers
probably belong to org-element.el.

> If we can stick with `after-change-functions' (or local equivalent),
> that's better. It is more predictable than `before-change-functions' and
> alike.

For now, I still used before/after-change-functions combination.
I see the following problems with using only after-change-functions: 

1. They are not guaranteed to be called after every single change:

From (elisp) Change Hooks:
"... some complex primitives call ‘before-change-functions’ once before
making changes, and then call ‘after-change-functions’ zero or more
times"

The consequence of it is a possibility that region passed to the
after-change-functions is quite big (including all the singular changes,
even if they are distant). This region may contain changed drawers as
well and unchanged drawers and needs to be parsed to determine which
drawers need to be re-folded.

> And, more importantly, they are not meant to be used together, i.e., you
> cannot assume that a single call to `before-change-functions' always
> happens before calling `after-change-functions'. This can be tricky if
> you want to use the former to pass information to the latter.

The fact that before-change-functions can be called multiple times
before after-change-functions, is trivially solved by using buffer-local
changes register (see org--modified-elements). The register is populated
by before-change-functions and cleared by after-change-functions.

> Well, `before-change-fuctions' and `after-change-functions' are not
> clean at all: you modify an unrelated part of the buffer, but still call
> those to check if a drawer needs to be unfolded somewhere.

2. As you pointed, instead of global before-change-functions, we can use
modification-hooks text property on sensitive parts of the
drawers/blocks. This would work, but I am concerned about one annoying
special case:

-------------------------------------------------------------------------
:BLAH: <inserted outside any of the existing drawers>

<some text>

:DRAWER: <folded>
Donec at pede.
:END:
-------------------------------------------------------------------------
In this example, the user would not be able to unfold the folder DRAWER
because it will technically become a part of a new giant BLAH drawer.
This may be especially annoying if <some text> is more than one screen
long and there is no easy way to identify why unfolding does not work
(with point at :DRAWER:).

Because of this scenario, limiting before-change-functions to folded
drawers is not sufficient. Any change in text may need to trigger
unfolding.

In the patch, I always register possible modifications in the
blocks/drawers intersecting with the modified region + a drawer/block
right next to the region.

-----------------------------------------------------------------------
-----------------------------------------------------------------------

More details on the nested 'invisible text property implementation.

The idea is to keep 'invisible property stack push and popping from it
as we add/remove 'invisible text property. All the work is done in
org-flag-region.

This was originally intended for folding outlines via text properties.
Since using text properties for folding outlines is not a good idea,
nested text properties have much less use. As I mentioned, they do
preserve link fontification, but I am not sure if it worth it for the
overhead to org-flag-region. Keeping this here mostly in the case if
someone has any ideas how it can be useful.

-----------------------------------------------------------------------
-----------------------------------------------------------------------

More details on replaced outline-* -> org-* function calls.

I have implemented org-* versions of the following functions:

- outline-hide-entry
- outline-hide-subtree
- outline-hide-sublevels
- outline-show-heading
- outline-show-branches

The org-* versions trivially use org-flag-region instead of
outline-flag-region.

Replaced outline-* calls where org- versions were already available:

- outline-show-entry
- outline-show-all
- outline-show-subtree

I reflected the new (including already available) functions in the
manual and removed some defalias from org-compat.el where they are not
needed. 

-----------------------------------------------------------------------
-----------------------------------------------------------------------

Further work:

1. after-change-functions use org-hide-drawer/block-toggle to
fold/unfold after modification. However, I just found that they call
org-element-at-point, which slows down modifications in folded
drawers/blocks. For example, realigning a long table inside folded
drawer takes >1sec, while it is instant in the unfolded drawer.

2. org-toggle-custom-properties is terribly slow on large org documents,
similarly to folded drawers on master. It should be relatively easy to
use text properties there instead of overlays.

3. Multiple calls to before/after-change-functions is still a problem. I
am looking into following ways to reduce this number:
 - reduce the number of elements registered as potentially modified
   + do not add duplicates to org--modified-elements
   + do not add unfolded elements to org--modified-elements
   + register after-change-function as post-command hook and remove it
     from global after-change-functions. This way, it will be called
     twice per command only.
 - determine common region containing org--modified-elements. if change
   is happening within that region, there is no need to parse
   drawers/blocks there again.

P.S.

>> It was mostly an annoyance, because they returned different results on
>> the same element. Specifically, they returned different :post-blank and
>> :end properties, which does not sound right.
>
> OK. If you have a reproducible recipe, I can look into it and see what
> can be done.

Recipe to have different (org-element-at-point) and
(org-element-parse-buffer 'element)
-------------------------------------------------------------------------
<point-min>
:PROPERTIES:
:CREATED:  [2020-05-23 Sat 02:32]
:END:


<point-max>
-------------------------------------------------------------------------


Best,
Ihor

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>>> As you noticed, using Org Element is a no-go, unfortunately. Parsing an
>>> element is a O(N) operation by the number of elements before it in
>>> a section. In particular, it is not bounded, and not mitigated by
>>> a cache. For large documents, it is going to be unbearably slow, too.
>>
>> Ouch. I thought it is faster.
>> What do you mean by "not mitigated by a cache"?
>
> Parsing starts from the closest headline, every time. So, if Org parses
> the Nth element in the entry two times, it really parses 2N elements.
>
> With a cache, assuming the buffer wasn't modified, Org would parse
> N elements only. With a smarter cache, with fine grained cache
> invalidation, it could also reduce the number of subsequent parsed
> elements.
>
>> The reason I would like to utilise org-element parser to make tracking
>> modifications more robust. Using details of the syntax would make the
>> code fragile if any modifications are made to syntax in future.
>
> I don't think the code would be more fragile. Also, the syntax we're
> talking about is not going to be modified anytime soon. Moreover, if
> folding breaks, it is usually visible, so the bug will not be unnoticed.
>
> This code is going to be as low-level as it can be.
>
>> Debugging bugs in modification functions is not easy, according to my
>> experience.
>
> No, it's not. 
>
> But this is not really related to whether you use Element or not.
>
>> One possible way to avoid performance issues during modification is
>> running parser in advance. For example, folding an element may
>> as well add information about the element to its text properties.
>> This will not degrade performance of folding since we are already
>> parsing the element during folding (at least, in
>> org-hide-drawer-toggle).
>
> We can use this information stored at fold time. But I'm not even sure
> we need it.
>
>> The problem with parsing an element during folding is that we cannot
>> easily detect changes like below without re-parsing.
>
> Of course we can. It is only necessary to focus on changes that would
> break the structure of the element. This does not entail a full parsing.
>
>> :PROPERTIES: <folded>
>> :CREATED: [2020-05-18 Mon]
>> :END: <- added line
>> :ID: test
>> :END:
>>
>> or even
>>
>> :PROPERTIES:
>> :CREATED: [2020-05-18 Mon]
>> :ID: test
>> :END: <- delete this line
>>
>> :DRAWER: <folded, cannot be unfolded if we don't re-parse after deletion>
>> test
>> :END:
>
> Please have a look at the "sensitive parts" I wrote about. This takes
> care of this kind of breakage.
>
>> The re-parsing can be done via regexp, as you suggested, but I don't
>> like this idea, because it will end up re-implementing
>> org-element-*-parser.
>
> You may have misunderstood my suggestion. See below.
>
>> Would it be acceptable to run org-element-*-parser
>> in after-change-functions?
>
> I'd rather not do that. This is unnecessary consing, and matching, etc.
>
>> If I understand correctly, it is not as easy.
>> Consider the following example:
>>
>> :PROPERTIES:
>> :CREATED: [2020-05-18 Mon]
>> <region-beginning>
>> :ID: example
>> :END:
>>
>> <... a lot of text, maybe containing other drawers ...>
>>
>> Nullam rutrum.
>> Pellentesque dapibus suscipit ligula.
>> <region-end>
>> Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus.
>>
>> If the region gets deleted, the modification hooks from chars inside
>> drawer will be called as (hook-function <region-beginning>
>> <region-end>). So, there is still a need to find the drawer somehow to
>> mark it as about to be modified (modification hooks are ran before
>> actual modification).
>
> If we can stick with `after-change-functions' (or local equivalent),
> that's better. It is more predictable than `before-change-functions' and
> alike.
>
> If it is a deletion, here is the kind of checks we could do, depending
> on when they are performed.
>
> Before actual changes :
>
>   1. The deletion is happening within a folded drawer (unnecessary step
>      in local functions).
>   2. The change deleted the sensitive line ":END:".
>   3. Conclusion : unfold.
>
> Or, after actual changes :
>
>   1. The deletion involves a drawer.
>   2. Text properties indicate that the beginning of the propertized part
>      of the buffer start with org-drawer-regexp, but doesn't end with
>      `org-property-end-re'. A "sensitive part" disappeared!
>   3. Conclusion : unfold
>
> This is far away from parsing. IMO, a few checks cover all cases. Let me
> know if you have questions about it.
>
> Also, note that the kind of change you describe will happen perhaps
> 0.01% of the time. Most change are about one character, or a single
> line, long.
>
>> The only difference between using modification hooks and
>> before-change-functions is that modification hooks will trigger less
>> frequently. 
>
> Exactly. Much less frequently. But extra care is required, as you noted
> already.
>
>> Considering the performance of org-element-at-point, it is
>> probably worth doing. Initially, I wanted to avoid it because setting a
>> single before-change-functions hook sounded cleaner than setting
>> modification-hooks, insert-behind-hooks, and insert-in-front-hooks.
>
> Well, `before-change-fuctions' and `after-change-functions' are not
> clean at all: you modify an unrelated part of the buffer, but still call
> those to check if a drawer needs to be unfolded somewhere.
>
> And, more importantly, they are not meant to be used together, i.e., you
> cannot assume that a single call to `before-change-functions' always
> happens before calling `after-change-functions'. This can be tricky if
> you want to use the former to pass information to the latter.
>
> But I understand that they are easier to use than their local
> counterparts. If you stick with (before|after)-change-functions, the
> function being called needs to drop the ball very quickly if the
> modification is not about folding changes. Also, I very much suggest to
> stick to only `after-change-functions', if feasible (I think it is), per
> above.
>
>> Moreover, these text properties would be copied by default if one uses 
>> buffer-substring. Then, the hooks will also trigger later in the yanked
>> text, which may cause all kinds of bugs.
>
> Indeed, that would be something to handle specifically. I.e.,
> destructive modifications (i.e., those that unfold) could clear such
> properties.
>
> It is more work. I don't know if it is worth the trouble if we can get
> out quickly of `after-change-functions' for unrelated changes.
>
>> It was mostly an annoyance, because they returned different results on
>> the same element. Specifically, they returned different :post-blank and
>> :end properties, which does not sound right.
>
> OK. If you have a reproducible recipe, I can look into it and see what
> can be done.
>
> Regards,
>
> -- 
> Nicolas Goaziou

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-23 13:52                               ` Ihor Radchenko
@ 2020-05-23 13:53                                 ` Ihor Radchenko
  2020-05-23 15:26                                   ` Ihor Radchenko
  2020-05-26  8:33                                 ` Nicolas Goaziou
  1 sibling, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-23 13:53 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 23 bytes --]

The patch is attached


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: featuredrawertextprop-20200523.patch --]
[-- Type: text/x-diff, Size: 45706 bytes --]

diff --git a/contrib/lisp/org-notify.el b/contrib/lisp/org-notify.el
index 9f8677871..ab470ea9b 100644
--- a/contrib/lisp/org-notify.el
+++ b/contrib/lisp/org-notify.el
@@ -246,7 +246,7 @@ seconds.  The default value for SECS is 20."
           (switch-to-buffer (find-file-noselect file))
           (org-with-wide-buffer
            (goto-char begin)
-           (outline-show-entry))
+           (org-show-entry))
           (goto-char begin)
           (search-forward "DEADLINE: <")
           (search-forward ":")
diff --git a/contrib/lisp/org-velocity.el b/contrib/lisp/org-velocity.el
index bfc4d6c3e..2312b235c 100644
--- a/contrib/lisp/org-velocity.el
+++ b/contrib/lisp/org-velocity.el
@@ -325,7 +325,7 @@ use it."
   (save-excursion
     (when narrow
       (org-narrow-to-subtree))
-    (outline-show-all)))
+    (org-show-all)))
 
 (defun org-velocity-edit-entry/inline (heading)
   "Edit entry at HEADING in the original buffer."
diff --git a/doc/org-manual.org b/doc/org-manual.org
index 96b160175..2ebe94538 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -509,11 +509,11 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
   Switch back to the startup visibility of the buffer (see [[*Initial
   visibility]]).
 
-- {{{kbd(C-u C-u C-u TAB)}}} (~outline-show-all~) ::
+- {{{kbd(C-u C-u C-u TAB)}}} (~org-show-all~) ::
 
   #+cindex: show all, command
   #+kindex: C-u C-u C-u TAB
-  #+findex: outline-show-all
+  #+findex: org-show-all
   Show all, including drawers.
 
 - {{{kbd(C-c C-r)}}} (~org-reveal~) ::
@@ -529,18 +529,18 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
   headings.  With a double prefix argument, also show the entire
   subtree of the parent.
 
-- {{{kbd(C-c C-k)}}} (~outline-show-branches~) ::
+- {{{kbd(C-c C-k)}}} (~org-show-branches~) ::
 
   #+cindex: show branches, command
   #+kindex: C-c C-k
-  #+findex: outline-show-branches
+  #+findex: org-show-branches
   Expose all the headings of the subtree, but not their bodies.
 
-- {{{kbd(C-c TAB)}}} (~outline-show-children~) ::
+- {{{kbd(C-c TAB)}}} (~org-show-children~) ::
 
   #+cindex: show children, command
   #+kindex: C-c TAB
-  #+findex: outline-show-children
+  #+findex: org-show-children
   Expose all direct children of the subtree.  With a numeric prefix
   argument {{{var(N)}}}, expose all children down to level
   {{{var(N)}}}.
@@ -7294,7 +7294,7 @@ its location in the outline tree, but behaves in the following way:
   command (see [[*Visibility Cycling]]).  You can force cycling archived
   subtrees with {{{kbd(C-TAB)}}}, or by setting the option
   ~org-cycle-open-archived-trees~.  Also normal outline commands, like
-  ~outline-show-all~, open archived subtrees.
+  ~org-show-all~, open archived subtrees.
 
 -
   #+vindex: org-sparse-tree-open-archived-trees
diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index ab13f926c..ad9244940 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -6826,7 +6826,7 @@ and stored in the variable `org-prefix-format-compiled'."
 	    (t "  %-12:c%?-12t% s")))
 	(start 0)
 	varform vars var e c f opt)
-    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)"
+    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)"
 			 s start)
       (setq var (or (cdr (assoc (match-string 4 s)
 				'(("c" . category) ("t" . time) ("l" . level) ("s" . extra)
@@ -9138,20 +9138,20 @@ if it was hidden in the outline."
      ((and (called-interactively-p 'any) (= more 1))
       (message "Remote: show with default settings"))
      ((= more 2)
-      (outline-show-entry)
+      (org-show-entry)
       (org-show-children)
       (save-excursion
 	(org-back-to-heading)
 	(run-hook-with-args 'org-cycle-hook 'children))
       (message "Remote: CHILDREN"))
      ((= more 3)
-      (outline-show-subtree)
+      (org-show-subtree)
       (save-excursion
 	(org-back-to-heading)
 	(run-hook-with-args 'org-cycle-hook 'subtree))
       (message "Remote: SUBTREE"))
      ((> more 3)
-      (outline-show-subtree)
+      (org-show-subtree)
       (message "Remote: SUBTREE AND ALL DRAWERS")))
     (select-window win)))
 
diff --git a/lisp/org-archive.el b/lisp/org-archive.el
index d3e12d17b..d864dad8a 100644
--- a/lisp/org-archive.el
+++ b/lisp/org-archive.el
@@ -330,7 +330,7 @@ direct children of this heading."
 		      (insert (if datetree-date "" "\n") heading "\n")
 		      (end-of-line 0))
 		    ;; Make the subtree visible
-		    (outline-show-subtree)
+		    (org-show-subtree)
 		    (if org-archive-reversed-order
 			(progn
 			  (org-back-to-heading t)
diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index e50a4d7c8..e656df555 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -699,7 +699,7 @@ FUN is a function called with no argument."
 			  (move-beginning-of-line 2)
 			  (org-at-heading-p t)))))
     (unwind-protect (funcall fun)
-      (when hide-body (outline-hide-entry)))))
+      (when hide-body (org-hide-entry)))))
 
 (defun org-columns-previous-allowed-value ()
   "Switch to the previous allowed value for this column."
diff --git a/lisp/org-compat.el b/lisp/org-compat.el
index 635a38dcd..8fe271896 100644
--- a/lisp/org-compat.el
+++ b/lisp/org-compat.el
@@ -139,12 +139,8 @@ This is a floating point number if the size is too large for an integer."
 ;;; Emacs < 25.1 compatibility
 
 (when (< emacs-major-version 25)
-  (defalias 'outline-hide-entry 'hide-entry)
-  (defalias 'outline-hide-sublevels 'hide-sublevels)
-  (defalias 'outline-hide-subtree 'hide-subtree)
   (defalias 'outline-show-branches 'show-branches)
   (defalias 'outline-show-children 'show-children)
-  (defalias 'outline-show-entry 'show-entry)
   (defalias 'outline-show-subtree 'show-subtree)
   (defalias 'xref-find-definitions 'find-tag)
   (defalias 'format-message 'format)
diff --git a/lisp/org-element.el b/lisp/org-element.el
index ac41b7650..2d5c8d771 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -4320,7 +4320,7 @@ element or object.  Meaningful values are `first-section',
 TYPE is the type of the current element or object.
 
 If PARENT? is non-nil, assume the next element or object will be
-located inside the current one.  "
+located inside the current one."
   (if parent?
       (pcase type
 	(`headline 'section)
diff --git a/lisp/org-keys.el b/lisp/org-keys.el
index c006e9c12..deb5d7b90 100644
--- a/lisp/org-keys.el
+++ b/lisp/org-keys.el
@@ -437,7 +437,7 @@ COMMANDS is a list of alternating OLDDEF NEWDEF command names."
   #'org-next-visible-heading)
 (define-key org-mode-map [remap outline-previous-visible-heading]
   #'org-previous-visible-heading)
-(define-key org-mode-map [remap show-children] #'org-show-children)
+(define-key org-mode-map [remap outline-show-children] #'org-show-children)
 
 ;;;; Make `C-c C-x' a prefix key
 (org-defkey org-mode-map (kbd "C-c C-x") (make-sparse-keymap))
diff --git a/lisp/org-macs.el b/lisp/org-macs.el
index a02f713ca..fa0a658f0 100644
--- a/lisp/org-macs.el
+++ b/lisp/org-macs.el
@@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead."
 
 
 \f
-;;; Overlays
+;;; Overlays and text properties
 
 (defun org-overlay-display (ovl text &optional face evap)
   "Make overlay OVL display TEXT with face FACE."
@@ -705,18 +705,99 @@ If DELETE is non-nil, delete all those overlays."
 	    (delete (delete-overlay ov))
 	    (t (push ov found))))))
 
+(defun org-remove-text-properties (start end properties &optional object)
+  "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions.
+Do not remove invisible text properties specified by 'outline,
+'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this
+is needed to keep outlines, drawers, and blocks hidden unless they are
+toggled by user.
+Note: The below may be too specific and create troubles if more
+invisibility specs are added to org in future"
+  (when (plist-member properties 'invisible)
+    (let ((pos start)
+	  next spec)
+      (while (< pos end)
+	(setq next (next-single-property-change pos 'invisible nil end)
+              spec (get-text-property pos 'invisible))
+	(unless (memq spec (list 'org-hide-block
+				 'org-hide-drawer
+				 'outline))
+          (remove-text-properties pos next '(invisible nil) object))
+	(setq pos next))))
+  (when-let ((properties-stripped (org-plist-delete properties 'invisible)))
+    (remove-text-properties start end properties-stripped object)))
+
+(defun org--find-text-property-region (pos prop)
+  "Find a region containing PROP text property around point POS."
+  (let* ((beg (and (get-text-property pos prop) pos))
+	 (end beg))
+    (when beg
+      ;; when beg is the first point in the region, `previous-single-property-change'
+      ;; will return nil.
+      (setq beg (or (previous-single-property-change pos prop)
+		    beg))
+      ;; when end is the last point in the region, `next-single-property-change'
+      ;; will return nil.
+      (setq end (or (next-single-property-change pos prop)
+		    end))
+      (unless (= beg end) ; this should not happen
+        (cons beg end)))))
+
 (defun org-flag-region (from to flag spec)
   "Hide or show lines from FROM to TO, according to FLAG.
 SPEC is the invisibility spec, as a symbol."
-  (remove-overlays from to 'invisible spec)
-  ;; Use `front-advance' since text right before to the beginning of
-  ;; the overlay belongs to the visible line than to the contents.
-  (when flag
-    (let ((o (make-overlay from to nil 'front-advance)))
-      (overlay-put o 'evaporate t)
-      (overlay-put o 'invisible spec)
-      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
-
+  (pcase spec
+    ('outline
+     (remove-overlays from to 'invisible spec)
+     ;; Use `front-advance' since text right before to the beginning of
+     ;; the overlay belongs to the visible line than to the contents.
+     (when flag
+       (let ((o (make-overlay from to nil 'front-advance)))
+	 (overlay-put o 'evaporate t)
+	 (overlay-put o 'invisible spec)
+	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
+    (_
+     ;; Use text properties instead of overlays for speed.
+     ;; Overlays are too slow (Emacs Bug#35453).
+     (with-silent-modifications
+       ;; keep a backup stack of old text properties
+       (save-excursion
+	 (goto-char from)
+	 (while (< (point) to)
+	   (let ((old-spec (get-text-property (point) 'invisible))
+		 (end (next-single-property-change (point) 'invisible nil to)))
+	     (when old-spec
+	       (alter-text-property (point) end 'org-property-stack-invisible
+				    (lambda (stack)
+				      (if (or (eq old-spec (car stack))
+					      (eq spec old-spec)
+					      (eq old-spec 'outline))
+					  stack
+					(cons old-spec stack)))))
+	     (goto-char end))))
+
+       ;; cleanup everything
+       (remove-text-properties from to '(invisible nil))
+
+       ;; Recover properties from the backup stack
+       (unless flag
+	 (save-excursion
+	   (goto-char from)
+	   (while (< (point) to)
+             (let ((stack (get-text-property (point) 'org-property-stack-invisible))
+		   (end (next-single-property-change (point) 'org-property-stack-invisible nil to)))
+               (if (not stack)
+		   (remove-text-properties (point) end '(org-property-stack-invisible nil))
+		 (put-text-property (point) end 'invisible (car stack))
+		 (alter-text-property (point) end 'org-property-stack-invisible
+				      (lambda (stack)
+					(cdr stack))))
+               (goto-char end)))))
+       
+       (when flag
+	 (put-text-property from to 'rear-non-sticky nil)
+	 (put-text-property from to 'front-sticky t)
+	 (put-text-property from to 'invisible spec))))))
 
 \f
 ;;; Regexp matching
diff --git a/lisp/org-src.el b/lisp/org-src.el
index c9eef744e..e89a1c580 100644
--- a/lisp/org-src.el
+++ b/lisp/org-src.el
@@ -523,8 +523,8 @@ Leave point in edit buffer."
 	(org-src-switch-to-buffer buffer 'edit)
 	;; Insert contents.
 	(insert contents)
-	(remove-text-properties (point-min) (point-max)
-				'(display nil invisible nil intangible nil))
+	(org-remove-text-properties (point-min) (point-max)
+				    '(display nil invisible nil intangible nil))
 	(unless preserve-ind (org-do-remove-indentation))
 	(set-buffer-modified-p nil)
 	(setq buffer-file-name nil)
diff --git a/lisp/org-table.el b/lisp/org-table.el
index 6462b99c4..75801161b 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -2001,7 +2001,7 @@ toggle `org-table-follow-field-mode'."
    (arg
     (let ((b (save-excursion (skip-chars-backward "^|") (point)))
 	  (e (save-excursion (skip-chars-forward "^|\r\n") (point))))
-      (remove-text-properties b e '(invisible t intangible t))
+      (org-remove-text-properties b e '(invisible t intangible t))
       (if (and (boundp 'font-lock-mode) font-lock-mode)
 	  (font-lock-fontify-block))))
    (t
@@ -2028,7 +2028,7 @@ toggle `org-table-follow-field-mode'."
       (setq word-wrap t)
       (goto-char (setq p (point-max)))
       (insert (org-trim field))
-      (remove-text-properties p (point-max) '(invisible t intangible t))
+      (org-remove-text-properties p (point-max) '(invisible t intangible t))
       (goto-char p)
       (setq-local org-finish-function 'org-table-finish-edit-field)
       (setq-local org-window-configuration cw)
diff --git a/lisp/org.el b/lisp/org.el
index e577dc661..360974135 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.")
 (declare-function cdlatex-math-symbol "ext:cdlatex")
 (declare-function Info-goto-node "info" (nodename &optional fork strict-case))
 (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag))
+(declare-function isearch-filter-visible "isearch" (beg end))
 (declare-function org-add-archive-files "org-archive" (files))
 (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom))
 (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour))
@@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.")
 
 (defvar ffap-url-regexp)
 (defvar org-element-paragraph-separate)
+(defvar org-element-all-objects)
+(defvar org-element-all-elements)
+(defvar org-element-greater-elements)
 (defvar org-indent-indentation-per-level)
 (defvar org-radio-target-regexp)
 (defvar org-target-link-regexp)
@@ -4734,9 +4738,381 @@ This is for getting out of special buffers like capture.")
 
 ;;;; Define the Org mode
 
+;;; Handling buffer modifications
+
 (defun org-before-change-function (_beg _end)
   "Every change indicates that a table might need an update."
   (setq org-table-may-need-update t))
+
+(defvar-local org--modified-elements nil
+  "List of elements, marked as recently modified.
+There is no guarantee that the elements in this list are fully parsed.
+Only the element type, :begin and :end properties of the elements are
+guaranteed to be available. The :begin and :end element properties
+contain markers instead of positions.")
+
+(defvar org-track-element-modification-default-sensitive-commands '(self-insert-command)
+  "List of commands triggerring element modifications unconditionally.")
+
+(defvar org--element-beginning-re-alist `((center-block . "^[ \t]*#\\+begin_center[ \t]*$")
+                                       (property-drawer . ,org-property-start-re)
+				       (drawer . ,org-drawer-regexp)
+                                       (quote-block . "^[ \t]*#\\+begin_quote[ \t]*$")
+                                       (special-block . "^[ \t]*#\\+begin_\\([^ ]+\\).*$"))
+  "Alist of regexps matching beginning of elements.
+Group 1 in the regexps (if any) contains the element type.")
+
+(defvar org--element-end-re-alist `((center-block . "^[ \t]*#\\+end_center[ \t]*$")
+				 (property-drawer . ,org-property-end-re)
+				 (drawer . ,org-property-end-re)
+				 (quote-block . "^[ \t]*#\\+end_quote[ \t]*$")
+				 (special-block . "^[ \t]*#\\+end_\\([^ ]+\\).*$"))
+  "Alist of regexps matching end of elements.
+Group 1 in the regexps (if any) contains the element type or END.")
+
+(defvar org-track-element-modifications
+  `((property-drawer  . (:after-change-function
+                         org--drawer-or-block-unfold-maybe))
+    (drawer  . (:after-change-function
+                org--drawer-or-block-unfold-maybe))
+    (center-block . (:after-change-function
+                     org--drawer-or-block-unfold-maybe))
+    (quote-block  . (:after-change-function
+                     org--drawer-or-block-unfold-maybe))
+    (special-block  . (:after-change-function
+                       org--drawer-or-block-unfold-maybe)))
+  "Alist of elements to be tracked for modifications.
+The modification is only triggered according to :sensitive-re-list and
+:sensitive-command-list (see below).
+Each element of the alist is a cons of an element symbol and plist
+defining how and when the modifications are handled.
+In case of recursive elements/duplicates, the first element from the
+list is considered.
+The plist can have the following properties:
+- :element-beginning-re   :: regex matching beginning of the element
+  (default)               :: (alist-get element org--element-beginning-re-alist)
+- :element-end-re         :: regex matching end of the element
+  (default)               :: (alist-get element org--element-end-re-alist)
+- :after-change-function  :: function called after the modification
+The function must accept a single argument - element from
+`org--modified-elements'.")
+
+(defun org--get-element-region-at-point (types)
+  "Return TYPES element at point or nil.
+If TYPES is a list, return first element at point from the list.  The
+returned value is partially parsed element only containing :begin and
+:end properties.  Only elements listed in
+org--element-beginning-re-alist and org--element-end-re-alist can be
+parsed here."
+  (catch 'exit
+    (dolist (type (if (listp types) types (list types)))
+      (let ((begin-re (alist-get type org--element-beginning-re-alist))
+	    (end-re (alist-get type org--element-end-re-alist))
+            (begin-limit (save-excursion (org-with-limited-levels
+					  (org-back-to-heading-or-point-min 'invisible-ok))
+					 (point)))
+            (end-limit (or (save-excursion (outline-next-heading))
+			   (point-max)))
+            (point (point))
+	    begin end closest-begin)
+	(when (and begin-re end-re)
+	  (save-excursion
+	    (end-of-line)
+	    (when (re-search-backward begin-re begin-limit 'noerror) (setq begin (point)))
+	    (when (re-search-forward end-re end-limit 'noerror) (setq end (point)))
+            (setq closest-begin begin)
+            ;; slurp unmatched begin-re
+	    (when (and begin end)
+              (goto-char begin)
+              (while (and (re-search-backward begin-re begin-limit 'noerror)
+			  (= end (save-excursion (re-search-forward end-re end-limit 'noerror))))
+		(setq begin (point)))
+              (when (and (>= point begin) (<= point end))
+		(throw 'exit
+		       (list type
+			     (list
+			      :begin begin
+			      :end end)))))))))))
+
+(defun org--get-next-element-region-at-point (types &optional limit previous)
+  "Return TYPES element after point or nil.
+If TYPES is a list, return first element after point from the list.
+If PREVIOUS is non-nil, return first TYPES element before point.
+Limit search by LIMIT or previous/next heading.
+The returned value is partially parsed element only containing :begin
+and :end properties.  Only elements listed in
+org--element-beginning-re-alist and org--element-end-re-alist can be
+parsed here."
+  (catch 'exit
+    (dolist (type (if (listp types) types (list types)))
+      (let* ((begin-re (alist-get type org--element-beginning-re-alist))
+	     (end-re (alist-get type org--element-end-re-alist))
+             (limit (or limit (if previous
+				  (save-excursion
+				    (org-with-limited-levels
+				     (org-back-to-heading-or-point-min 'invisible-ok)
+				     (point)))
+				(or (save-excursion (outline-next-heading))
+				    (point-max)))))
+	     begin end)
+	(when (and begin-re end-re)
+	  (save-excursion
+            (if previous
+                (when (re-search-backward begin-re limit 'noerror)
+		  (when-let ((el (org--get-element-region-at-point type)))
+		    (setq begin (org-element-property :begin el))
+		    (setq end (org-element-property :end el))))
+	      (when (re-search-forward begin-re limit 'noerror)
+                (when-let ((el (org--get-element-region-at-point type)))
+		  (setq begin (org-element-property :begin el))
+		  (setq end (org-element-property :end el))))))
+	  (when (and begin end)
+            (throw 'exit
+		   (list type
+			 (list
+			  :begin begin
+			  :end end)))))))))
+
+(defun org--find-elements-in-region (beg end elements &optional include-partial include-neighbours)
+  "Find all elements from ELEMENTS in region BEG . END.
+All the listed elements must be resolvable by
+`org--get-element-region-at-point'.
+Include elements if they are partially inside region when
+INCLUDE-PARTIAL is non-nil.
+Include preceding/subsequent neighbouring elements when no partial
+element is found at the beginning/end of the region and
+INCLUDE-NEIGHBOURS is non-nil."
+  (when include-partial
+    (org-with-point-at beg
+      (let ((new-beg (org-element-property :begin (org--get-element-region-at-point elements))))
+	(if new-beg
+	    (setq beg new-beg)
+          (when (and include-neighbours
+		     (setq new-beg (org-element-property :begin
+						      (org--get-next-element-region-at-point elements
+											  (point-min)
+											  'previous))))
+            (setq beg new-beg))))
+      (when (memq 'headline elements)
+	(when-let ((new-beg (save-excursion
+			      (org-with-limited-levels (outline-previous-heading)))))
+          (setq beg new-beg))))
+    (org-with-point-at end
+      (let ((new-end (org-element-property :end (org--get-element-region-at-point elements))))
+	(if new-end
+	    (setq end new-end)
+          (when (and include-neighbours
+		     (setq new-end (org-element-property :end
+						      (org--get-next-element-region-at-point elements (point-max)))))
+            (setq end new-end))))
+      (when (memq 'headline elements)
+	(when-let ((new-end (org-with-limited-levels (outline-next-heading))))
+          (setq end (1- new-end))))))
+  (save-excursion
+    (save-restriction
+      (narrow-to-region beg end)
+      (goto-char (point-min))
+      (let (result el)
+	(while (setq el (org--get-next-element-region-at-point elements end))
+          (push el result)
+          (goto-char (org-element-property :end el)))
+        result))))
+
+(defun org--drawer-or-block-unfold-maybe (el)
+  "Update visibility of modified folded drawer/block EL.
+If text was added to hidden drawer/block, make sure that the text is
+also hidden, unless the change was done by a command listed in
+`org-track-element-modification-default-sensitive-commands'.  If the
+modification destroyed the drawer/block, reveal the hidden text in
+former drawer/block.  If the modification shrinked/expanded the
+drawer/block beyond the hidden text, reveal the affected
+drawers/blocks as well.
+Examples:
+----------------------------------------------
+----------------------------------------------
+Case #1 (the element content is hidden):
+----------------------------------------------
+:PROPERTIES:
+:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
+:END:
+----------------------------------------------
+is changed to
+----------------------------------------------
+:ROPERTIES:
+:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
+:END:
+----------------------------------------------
+Text is revealed, because we have drawer in place of property-drawer
+----------------------------------------------
+----------------------------------------------
+Case #2 (the element content is hidden):
+----------------------------------------------
+:ROPERTIES:
+:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
+:END:
+----------------------------------------------
+is changed to
+----------------------------------------------
+:OPERTIES:
+:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
+:END:
+----------------------------------------------
+The text remains hidden since it is still a drawer.
+----------------------------------------------
+----------------------------------------------
+Case #3: (the element content is hidden):
+----------------------------------------------
+:FOO:
+bar
+tmp
+:END:
+----------------------------------------------
+is changed to
+----------------------------------------------
+:FOO:
+bar
+:END:
+tmp
+:END:
+----------------------------------------------
+The text is revealed because the drawer contents shrank.
+----------------------------------------------
+----------------------------------------------
+Case #4: (the element content is hidden in both the drawers):
+----------------------------------------------
+:FOO:
+bar
+tmp
+:END:
+:BAR:
+jjd
+:END:
+----------------------------------------------
+is changed to
+----------------------------------------------
+:FOO:
+bar
+tmp
+:BAR:
+jjd
+:END:
+----------------------------------------------
+The text is revealed in both the drawers because the drawers are merged
+into a new drawer.
+----------------------------------------------
+----------------------------------------------
+Case #5: (the element content is hidden)
+----------------------------------------------
+:test:
+Vivamus id enim.
+:end:
+----------------------------------------------
+is changed to
+----------------------------------------------
+:drawer:
+:test:
+Vivamus id enim.
+:end:
+----------------------------------------------
+The text is revealed in the drawer because the drawer expended.
+----------------------------------------------
+----------------------------------------------
+Case #6: (the element content is hidden):
+----------------------------------------------
+:test:
+Vivamus id enim.
+:end:
+----------------------------------------------
+is changed to
+----------------------------------------------
+:test:
+Vivamus id enim.
+:end:
+Nam a sapien.
+:end:
+----------------------------------------------
+The text remains hidden because drawer contents is always before the
+first :end:."
+  (save-match-data
+    (save-excursion
+      (save-restriction
+	(goto-char (org-element-property :begin el))
+	(let* ((newel (org--get-element-region-at-point
+		       (mapcar (lambda (el)
+				 (when (string-match-p (regexp-opt '("block" "drawer"))
+						       (symbol-name (car el)))
+                                   (car el)))
+                               org-track-element-modifications)))
+	       (spec (if (string-match-p "block" (symbol-name (org-element-type el)))
+			 'org-hide-block
+		       (if (string-match-p "drawer" (symbol-name (org-element-type el)))
+			   'org-hide-drawer
+			 t)))
+               (toggle-func (if (eq spec 'org-hide-drawer)
+				#'org-hide-drawer-toggle
+                              (if (eq spec 'org-hide-block)
+				  #'org-hide-block-toggle
+                                #'ignore)))) ; this should not happen
+	  (if (and (equal (org-element-type el) (org-element-type newel))
+		   (equal (marker-position (org-element-property :begin el))
+			  (org-element-property :begin newel))
+		   (equal (marker-position (org-element-property :end el))
+			  (org-element-property :end newel)))
+	      (when (text-property-any (marker-position (org-element-property :begin el))
+				       (marker-position (org-element-property :end el))
+				       'invisible spec)
+                (goto-char (org-element-property :begin newel))
+		(if (memq this-command org-track-element-modification-default-sensitive-commands)
+		    ;; reveal if change was made by typing
+		    (funcall toggle-func 'off)
+		  ;; re-hide the inserted text
+		  ;; FIXME: opening the drawer before hiding should not be needed here
+		  (funcall toggle-func 'off) ; this is needed to avoid showing double ellipsis
+		  (funcall toggle-func 'hide)))
+            ;; The element was destroyed. Reveal everything.
+            (org-flag-region (marker-position (org-element-property :begin el))
+			  (marker-position (org-element-property :end el))
+			  nil spec)
+            (when newel
+              (org-flag-region (org-element-property :begin newel)
+			    (org-element-property :end newel)
+			    nil spec))))))))
+
+(defun org--before-element-change-function (beg end)
+  "Register upcoming element modifications in `org--modified-elements' for all elements interesting with BEG END."
+  (save-match-data
+    (save-excursion
+      (save-restriction
+        (widen)
+	(dolist (el (org--find-elements-in-region beg
+					       end
+					       (mapcar #'car org-track-element-modifications)
+					       'include-partial
+                                               'include-neighbours))
+	  (let* ((beg-marker (copy-marker (org-element-property :begin el) 't))
+		 (end-marker (copy-marker (org-element-property :end el) 't)))
+	    (when (and (marker-position beg-marker) (marker-position end-marker))
+	      (org-element-put-property el :begin beg-marker)
+	      (org-element-put-property el :end end-marker)
+	      (add-to-list 'org--modified-elements el))))))))
+
+;; FIXME: this function may be called many times during routine modifications
+;; The normal way to avoid this is `combine-after-change-calls' - not
+;; the case in most org primitives.
+(defun org--after-element-change-function (&rest _)
+  "Handle changed elements from `org--modified-elements'."
+  (dolist (el org--modified-elements)
+    (save-match-data
+      (save-excursion
+        (save-restriction
+          (widen)
+	  (when-let* ((type (org-element-type el))
+		      (change-func (plist-get (alist-get type org-track-element-modifications)
+					      :after-change-function)))
+	    (with-demoted-errors
+		(funcall (symbol-function change-func) el)))))))
+  (setq org--modified-elements nil))
+
 (defvar org-mode-map)
 (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param.
 (defvar org-agenda-keep-modes nil)      ; Dynamically-scoped param.
@@ -4818,6 +5194,9 @@ The following commands are available:
   ;; Activate before-change-function
   (setq-local org-table-may-need-update t)
   (add-hook 'before-change-functions 'org-before-change-function nil 'local)
+  (add-hook 'before-change-functions 'org--before-element-change-function nil 'local)
+  ;; Activate after-change-function
+  (add-hook 'after-change-functions 'org--after-element-change-function nil 'local)
   ;; Check for running clock before killing a buffer
   (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local)
   ;; Initialize macros templates.
@@ -4869,6 +5248,10 @@ The following commands are available:
   (setq-local outline-isearch-open-invisible-function
 	      (lambda (&rest _) (org-show-context 'isearch)))
 
+  ;; Make isearch search in blocks hidden via text properties
+  (setq-local isearch-filter-predicate #'org--isearch-filter-predicate)
+  (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local)
+
   ;; Setup the pcomplete hooks
   (setq-local pcomplete-command-completion-function #'org-pcomplete-initial)
   (setq-local pcomplete-command-name-function #'org-command-at-point)
@@ -5050,8 +5433,8 @@ stacked delimiters is N.  Escaping delimiters is not possible."
 	      (when verbatim?
 		(org-remove-flyspell-overlays-in
 		 (match-beginning 0) (match-end 0))
-		(remove-text-properties (match-beginning 2) (match-end 2)
-					'(display t invisible t intangible t)))
+		(org-remove-text-properties (match-beginning 2) (match-end 2)
+					 '(display t invisible t intangible t)))
 	      (add-text-properties (match-beginning 2) (match-end 2)
 				   '(font-lock-multiline t org-emphasis t))
 	      (when (and org-hide-emphasis-markers
@@ -5166,7 +5549,7 @@ This includes angle, plain, and bracket links."
 	    (if (not (eq 'bracket style))
 		(add-text-properties start end properties)
 	      ;; Handle invisible parts in bracket links.
-	      (remove-text-properties start end '(invisible nil))
+	      (org-remove-text-properties start end '(invisible nil))
 	      (let ((hidden
 		     (append `(invisible
 			       ,(or (org-link-get-parameter type :display)
@@ -5186,8 +5569,8 @@ This includes angle, plain, and bracket links."
 (defun org-activate-code (limit)
   (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t)
     (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
-    (remove-text-properties (match-beginning 0) (match-end 0)
-			    '(display t invisible t intangible t))
+    (org-remove-text-properties (match-beginning 0) (match-end 0)
+			     '(display t invisible t intangible t))
     t))
 
 (defcustom org-src-fontify-natively t
@@ -5258,8 +5641,8 @@ by a #."
 	    (setq block-end (match-beginning 0)) ; includes the final newline.
 	    (when quoting
 	      (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline)
-	      (remove-text-properties beg end-of-endline
-				      '(display t invisible t intangible t)))
+	      (org-remove-text-properties beg end-of-endline
+				       '(display t invisible t intangible t)))
 	    (add-text-properties
 	     beg end-of-endline '(font-lock-fontified t font-lock-multiline t))
 	    (org-remove-flyspell-overlays-in beg bol-after-beginline)
@@ -5313,8 +5696,8 @@ by a #."
 	     '(font-lock-fontified t face org-document-info))))
 	 ((string-prefix-p "+caption" dc1)
 	  (org-remove-flyspell-overlays-in (match-end 2) (match-end 0))
-	  (remove-text-properties (match-beginning 0) (match-end 0)
-				  '(display t invisible t intangible t))
+	  (org-remove-text-properties (match-beginning 0) (match-end 0)
+				   '(display t invisible t intangible t))
 	  ;; Handle short captions.
 	  (save-excursion
 	    (beginning-of-line)
@@ -5336,8 +5719,8 @@ by a #."
 	   '(font-lock-fontified t face font-lock-comment-face)))
 	 (t ;; just any other in-buffer setting, but not indented
 	  (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
-	  (remove-text-properties (match-beginning 0) (match-end 0)
-				  '(display t invisible t intangible t))
+	  (org-remove-text-properties (match-beginning 0) (match-end 0)
+				   '(display t invisible t intangible t))
 	  (add-text-properties beg (match-end 0)
 			       '(font-lock-fontified t face org-meta-line))
 	  t))))))
@@ -5859,10 +6242,11 @@ If TAG is a number, get the corresponding match group."
 	 (inhibit-modification-hooks t)
 	 deactivate-mark buffer-file-name buffer-file-truename)
     (decompose-region beg end)
-    (remove-text-properties beg end
-			    '(mouse-face t keymap t org-linked-text t
-					 invisible t intangible t
-					 org-emphasis t))
+    (org-remove-text-properties beg end
+			     '(mouse-face t keymap t org-linked-text t
+					  invisible t
+                                          intangible t
+					  org-emphasis t))
     (org-remove-font-lock-display-properties beg end)))
 
 (defconst org-script-display  '(((raise -0.3) (height 0.7))
@@ -5970,6 +6354,29 @@ open and agenda-wise Org files."
 
 ;;;; Headlines visibility
 
+(defun org-hide-entry ()
+  "Hide the body directly following this heading."
+  (interactive)
+  (save-excursion
+    (outline-back-to-heading)
+    (outline-end-of-heading)
+    (org-flag-region (point) (progn (outline-next-preface) (point)) t 'outline)))
+
+(defun org-hide-subtree ()
+  "Hide everything after this heading at deeper levels."
+  (interactive)
+  (org-flag-subtree t))
+
+(defun org-hide-sublevels (levels)
+  "Hide everything but the top LEVELS levels of headers, in whole buffer.
+This also unhides the top heading-less body, if any.
+
+Interactively, the prefix argument supplies the value of LEVELS.
+When invoked without a prefix argument, LEVELS defaults to the level
+of the current heading, or to 1 if the current line is not a heading."
+  (cl-letf (((symbol-function 'outline-flag-region) #'org-flag-region))
+    (org-hide-sublevels levels)))
+
 (defun org-show-entry ()
   "Show the body directly following this heading.
 Show the heading too, if it is currently invisible."
@@ -5988,6 +6395,16 @@ Show the heading too, if it is currently invisible."
        'outline)
       (org-cycle-hide-property-drawers 'children))))
 
+(defun org-show-heading ()
+  "Show the current heading and move to its end."
+  (org-flag-region (- (point)
+ 		   (if (bobp) 0
+ 		     (if (and outline-blank-line
+                              (eq (char-before (1- (point))) ?\n))
+ 			 2 1)))
+		(progn (outline-end-of-heading) (point))
+		nil))
+
 (defun org-show-children (&optional level)
   "Show all direct subheadings of this heading.
 Prefix arg LEVEL is how many levels below the current level
@@ -6031,6 +6448,11 @@ heading to appear."
   (org-flag-region
    (point) (save-excursion (org-end-of-subtree t t)) nil 'outline))
 
+(defun org-show-branches ()
+  "Show all subheadings of this heading, but not their bodies."
+  (interactive)
+  (org-show-children 1000))
+
 ;;;; Blocks and drawers visibility
 
 (defun org--hide-wrapper-toggle (element category force no-error)
@@ -6064,8 +6486,8 @@ Return a non-nil value when toggling is successful."
 	(unless (let ((eol (line-end-position)))
 		  (and (> eol start) (/= eol end)))
 	  (let* ((spec (cond ((eq category 'block) 'org-hide-block)
-			     ((eq type 'property-drawer) 'outline)
-			     (t 'org-hide-drawer)))
+			     ((eq category 'drawer) 'org-hide-drawer)
+			     (t 'outline)))
 		 (flag
 		  (cond ((eq force 'off) nil)
 			(force t)
@@ -6158,10 +6580,7 @@ STATE should be one of the symbols listed in the docstring of
 	       (when (org-at-property-drawer-p)
 		 (let* ((case-fold-search t)
 			(end (re-search-forward org-property-end-re)))
-		   ;; Property drawers use `outline' invisibility spec
-		   ;; so they can be swallowed once we hide the
-		   ;; outline.
-		   (org-flag-region start end t 'outline)))))))))))
+		   (org-flag-region start end t 'org-hide-drawer)))))))))))
 
 ;;;; Visibility cycling
 
@@ -6536,7 +6955,7 @@ With a numeric prefix, show all headlines up to that level."
 		     (org-narrow-to-subtree)
 		     (org-content))))
 		((or "all" "showall")
-		 (outline-show-subtree))
+		 (org-show-subtree))
 		(_ nil)))
 	    (org-end-of-subtree)))))))
 
@@ -6609,7 +7028,7 @@ This function is the default value of the hook `org-cycle-hook'."
 	  (while (re-search-forward re nil t)
 	    (when (and (not (org-invisible-p))
 		       (org-invisible-p (line-end-position)))
-	      (outline-hide-entry))))
+	      (org-hide-entry))))
 	(org-cycle-hide-property-drawers 'all)
 	(org-cycle-show-empty-lines 'overview)))))
 
@@ -6683,8 +7102,13 @@ information."
     ;; expose it.
     (dolist (o (overlays-at (point)))
       (when (memq (overlay-get o 'invisible)
-		  '(org-hide-block org-hide-drawer outline))
+		  '(outline))
 	(delete-overlay o)))
+    (when (memq (get-text-property (point) 'invisible)
+		'(org-hide-block org-hide-drawer))
+      (let ((spec (get-text-property (point) 'invisible))
+	    (region (org--find-text-property-region (point) 'invisible)))
+	(org-flag-region (car region) (cdr region) nil spec)))
     (unless (org-before-first-heading-p)
       (org-with-limited-levels
        (cl-case detail
@@ -7661,7 +8085,7 @@ When REMOVE is non-nil, remove the subtree from the clipboard."
      (skip-chars-forward " \t\n\r")
      (setq beg (point))
      (when (and (org-invisible-p) visp)
-       (save-excursion (outline-show-heading)))
+       (save-excursion (org-show-heading)))
      ;; Shift if necessary.
      (unless (= shift 0)
        (save-restriction
@@ -8103,7 +8527,7 @@ function is being called interactively."
 		       (point))
 	    what "children")
       (goto-char start)
-      (outline-show-subtree)
+      (org-show-subtree)
       (outline-next-heading))
      (t
       ;; we will sort the top-level entries in this file
@@ -13150,7 +13574,7 @@ drawer is immediately hidden."
 	   (inhibit-read-only t))
        (unless (bobp) (insert "\n"))
        (insert ":PROPERTIES:\n:END:")
-       (org-flag-region (line-end-position 0) (point) t 'outline)
+       (org-flag-region (line-end-position 0) (point) t 'org-hide-drawer)
        (when (or (eobp) (= begin (point-min))) (insert "\n"))
        (org-indent-region begin (point))))))
 
@@ -17612,11 +18036,11 @@ Move point to the beginning of first heading or end of buffer."
 (defun org-show-branches-buffer ()
   "Show all branches in the buffer."
   (org-flag-above-first-heading)
-  (outline-hide-sublevels 1)
+  (org-hide-sublevels 1)
   (unless (eobp)
-    (outline-show-branches)
+    (org-show-branches)
     (while (outline-get-next-sibling)
-      (outline-show-branches)))
+      (org-show-branches)))
   (goto-char (point-min)))
 
 (defun org-kill-note-or-show-branches ()
@@ -17630,8 +18054,8 @@ Move point to the beginning of first heading or end of buffer."
 	(t
 	 (let ((beg (progn (org-back-to-heading) (point)))
 	       (end (save-excursion (org-end-of-subtree t t) (point))))
-	   (outline-hide-subtree)
-	   (outline-show-branches)
+	   (org-hide-subtree)
+	   (org-show-branches)
 	   (org-hide-archived-subtrees beg end)))))
 
 (defun org-delete-indentation (&optional arg)
@@ -17787,9 +18211,9 @@ Otherwise, call `org-show-children'.  ARG is the level to hide."
     (if (org-before-first-heading-p)
         (progn
           (org-flag-above-first-heading)
-          (outline-hide-sublevels (or arg 1))
+          (org-hide-sublevels (or arg 1))
           (goto-char (point-min)))
-      (outline-hide-subtree)
+      (org-hide-subtree)
       (org-show-children arg))))
 
 (defun org-ctrl-c-star ()
@@ -20933,6 +21357,80 @@ Started from `gnus-info-find-node'."
           (t default-org-info-node))))))
 
 \f
+
+;;; Make isearch search in some text hidden via text propertoes
+
+(defvar org--isearch-overlays nil
+  "List of overlays temporarily created during isearch.
+This is used to allow searching in regions hidden via text properties.
+As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays.
+Any text hidden via text properties is not revealed even if `search-invisible'
+is set to 't.")
+
+;; Not sure if it needs to be a user option
+;; One might want to reveal hidden text in, for example, hidden parts of the links.
+;; Currently, hidden text in links is never revealed by isearch.
+(defvar org-isearch-specs '(org-hide-block
+			 org-hide-drawer)
+  "List of text invisibility specs to be searched by isearch.
+By default ([2020-05-09 Sat]), isearch does not search in hidden text,
+which was made invisible using text properties. Isearch will be forced
+to search in hidden text with any of the listed 'invisible property value.")
+
+(defun org--create-isearch-overlays (beg end)
+  "Replace text property invisibility spec by overlays between BEG and END.
+All the regions with invisibility text property spec from
+`org-isearch-specs' will be changed to use overlays instead
+of text properties. The created overlays will be stored in
+`org--isearch-overlays'."
+  (let ((pos beg))
+    (while (< pos end)
+      (when-let* ((spec (get-text-property pos 'invisible))
+		  (spec (memq spec org-isearch-specs))
+		  (region (org--find-text-property-region pos 'invisible)))
+        (setq spec (get-text-property pos 'invisible))
+        ;; Changing text properties is considered buffer modification.
+        ;; We do not want it here.
+	(with-silent-modifications
+          ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat]
+          ;; overlay for 'outline blocks.
+          (let ((o (make-overlay (car region) (cdr region) nil 'front-advance)))
+	    (overlay-put o 'evaporate t)
+	    (overlay-put o 'invisible spec)
+            ;; `delete-overlay' here means that spec information will be lost
+            ;; for the region. The region will remain visible.
+	    (overlay-put o 'isearch-open-invisible #'delete-overlay)
+            (push o org--isearch-overlays))
+	  (org-flag-region (car region) (cdr region) nil spec)))
+      (setq pos (next-single-property-change pos 'invisible nil end)))))
+
+(defun org--isearch-filter-predicate (beg end)
+  "Return non-nil if text between BEG and END is deemed visible by Isearch.
+This function is intended to be used as `isearch-filter-predicate'.
+Unlike `isearch-filter-visible', make text with 'invisible text property
+value listed in `org-isearch-specs' visible to Isearch."
+  (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text
+  (isearch-filter-visible beg end))
+
+(defun org--clear-isearch-overlay (ov)
+  "Convert OV region back into using text properties."
+  (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays
+    ;; Changing text properties is considered buffer modification.
+    ;; We do not want it here.
+    (with-silent-modifications
+      (org-flag-region (overlay-start ov) (overlay-end ov) t spec)))
+  (when (member ov isearch-opened-overlays)
+    (setq isearch-opened-overlays (delete ov isearch-opened-overlays)))
+  (delete-overlay ov))
+
+(defun org--clear-isearch-overlays ()
+  "Convert overlays from `org--isearch-overlays' back into using text properties."
+  (when org--isearch-overlays
+    (mapc #'org--clear-isearch-overlay org--isearch-overlays)
+    (setq org--isearch-overlays nil)))
+
+\f
+
 ;;; Finish up
 
 (add-hook 'org-mode-hook     ;remove overlays when changing major mode

[-- Attachment #3: Type: text/plain, Size: 17187 bytes --]



Ihor Radchenko <yantar92@gmail.com> writes:

> Hello,
>
> [The patch itself will be provided in the following email]
>
> I have five updates from the previous version of the patch:
>
> 1. I implemented a simplified version of element parsing to detect
> changes in folded drawers or blocks. No computationally expensive calls
> of org-element-at-point or org-element-parse-buffer are needed now.
>
> 2. The patch is now compatible with master (commit 2e96dc639). I
> reverted the earlier change in folding drawers and blocks. Now, they are
> back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would
> achieve nothing when we use text properties.
>
> 3. 'invisible text property can now be nested. This is important, for
> example, when text inside drawers contains fontified links (which also
> use 'invisible text property to hide parts of the link). Now, the old
> 'invisible spec is recovered after unfolding.
>
> 4. Some outline-* function calls in org referred to outline-flag-region
> implementation, which is not in sync with org-flag-region in this patch.
> I have implemented their org-* versions and replaced the calls
> throughout .el files. Actually, some org-* versions were already
> implemented in org, but not used for some reason (or not mentioned in
> the manual). I have updated the relevant sections of manual. These
> changes might be relevant to org independently of this feature branch.
>
> 5. I have managed to get a working version of outline folding via text
> properties. However, that approach has a big downside - folding state
> cannot be different in indirect buffer when we use text properties. I
> have seen packages relying on this feature of org and I do not see any
> obvious way to achieve different folding state in indirect buffer while
> using text properties for outline folding.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the new implementation for tracking changes:
>
>> Of course we can. It is only necessary to focus on changes that would
>> break the structure of the element. This does not entail a full parsing.
>
> I have limited parsing to matching beginning and end of a drawer/block.
> The basic functions are org--get-element-region-at-point,
> org--get-next-element-region-at-point, and org--find-elements-in-region.
> They are simplified versions of org-element-* parsers and do not require
> parsing everything from the beginning of the section.
>
> For now, I keep everything in org.el, but those simplified parsers
> probably belong to org-element.el.
>
>> If we can stick with `after-change-functions' (or local equivalent),
>> that's better. It is more predictable than `before-change-functions' and
>> alike.
>
> For now, I still used before/after-change-functions combination.
> I see the following problems with using only after-change-functions: 
>
> 1. They are not guaranteed to be called after every single change:
>
> From (elisp) Change Hooks:
> "... some complex primitives call ‘before-change-functions’ once before
> making changes, and then call ‘after-change-functions’ zero or more
> times"
>
> The consequence of it is a possibility that region passed to the
> after-change-functions is quite big (including all the singular changes,
> even if they are distant). This region may contain changed drawers as
> well and unchanged drawers and needs to be parsed to determine which
> drawers need to be re-folded.
>
>> And, more importantly, they are not meant to be used together, i.e., you
>> cannot assume that a single call to `before-change-functions' always
>> happens before calling `after-change-functions'. This can be tricky if
>> you want to use the former to pass information to the latter.
>
> The fact that before-change-functions can be called multiple times
> before after-change-functions, is trivially solved by using buffer-local
> changes register (see org--modified-elements). The register is populated
> by before-change-functions and cleared by after-change-functions.
>
>> Well, `before-change-fuctions' and `after-change-functions' are not
>> clean at all: you modify an unrelated part of the buffer, but still call
>> those to check if a drawer needs to be unfolded somewhere.
>
> 2. As you pointed, instead of global before-change-functions, we can use
> modification-hooks text property on sensitive parts of the
> drawers/blocks. This would work, but I am concerned about one annoying
> special case:
>
> -------------------------------------------------------------------------
> :BLAH: <inserted outside any of the existing drawers>
>
> <some text>
>
> :DRAWER: <folded>
> Donec at pede.
> :END:
> -------------------------------------------------------------------------
> In this example, the user would not be able to unfold the folder DRAWER
> because it will technically become a part of a new giant BLAH drawer.
> This may be especially annoying if <some text> is more than one screen
> long and there is no easy way to identify why unfolding does not work
> (with point at :DRAWER:).
>
> Because of this scenario, limiting before-change-functions to folded
> drawers is not sufficient. Any change in text may need to trigger
> unfolding.
>
> In the patch, I always register possible modifications in the
> blocks/drawers intersecting with the modified region + a drawer/block
> right next to the region.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the nested 'invisible text property implementation.
>
> The idea is to keep 'invisible property stack push and popping from it
> as we add/remove 'invisible text property. All the work is done in
> org-flag-region.
>
> This was originally intended for folding outlines via text properties.
> Since using text properties for folding outlines is not a good idea,
> nested text properties have much less use. As I mentioned, they do
> preserve link fontification, but I am not sure if it worth it for the
> overhead to org-flag-region. Keeping this here mostly in the case if
> someone has any ideas how it can be useful.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on replaced outline-* -> org-* function calls.
>
> I have implemented org-* versions of the following functions:
>
> - outline-hide-entry
> - outline-hide-subtree
> - outline-hide-sublevels
> - outline-show-heading
> - outline-show-branches
>
> The org-* versions trivially use org-flag-region instead of
> outline-flag-region.
>
> Replaced outline-* calls where org- versions were already available:
>
> - outline-show-entry
> - outline-show-all
> - outline-show-subtree
>
> I reflected the new (including already available) functions in the
> manual and removed some defalias from org-compat.el where they are not
> needed. 
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> Further work:
>
> 1. after-change-functions use org-hide-drawer/block-toggle to
> fold/unfold after modification. However, I just found that they call
> org-element-at-point, which slows down modifications in folded
> drawers/blocks. For example, realigning a long table inside folded
> drawer takes >1sec, while it is instant in the unfolded drawer.
>
> 2. org-toggle-custom-properties is terribly slow on large org documents,
> similarly to folded drawers on master. It should be relatively easy to
> use text properties there instead of overlays.
>
> 3. Multiple calls to before/after-change-functions is still a problem. I
> am looking into following ways to reduce this number:
>  - reduce the number of elements registered as potentially modified
>    + do not add duplicates to org--modified-elements
>    + do not add unfolded elements to org--modified-elements
>    + register after-change-function as post-command hook and remove it
>      from global after-change-functions. This way, it will be called
>      twice per command only.
>  - determine common region containing org--modified-elements. if change
>    is happening within that region, there is no need to parse
>    drawers/blocks there again.
>
> P.S.
>
>>> It was mostly an annoyance, because they returned different results on
>>> the same element. Specifically, they returned different :post-blank and
>>> :end properties, which does not sound right.
>>
>> OK. If you have a reproducible recipe, I can look into it and see what
>> can be done.
>
> Recipe to have different (org-element-at-point) and
> (org-element-parse-buffer 'element)
> -------------------------------------------------------------------------
> <point-min>
> :PROPERTIES:
> :CREATED:  [2020-05-23 Sat 02:32]
> :END:
>
>
> <point-max>
> -------------------------------------------------------------------------
>
>
> Best,
> Ihor
>
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Hello,
>>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>>> As you noticed, using Org Element is a no-go, unfortunately. Parsing an
>>>> element is a O(N) operation by the number of elements before it in
>>>> a section. In particular, it is not bounded, and not mitigated by
>>>> a cache. For large documents, it is going to be unbearably slow, too.
>>>
>>> Ouch. I thought it is faster.
>>> What do you mean by "not mitigated by a cache"?
>>
>> Parsing starts from the closest headline, every time. So, if Org parses
>> the Nth element in the entry two times, it really parses 2N elements.
>>
>> With a cache, assuming the buffer wasn't modified, Org would parse
>> N elements only. With a smarter cache, with fine grained cache
>> invalidation, it could also reduce the number of subsequent parsed
>> elements.
>>
>>> The reason I would like to utilise org-element parser to make tracking
>>> modifications more robust. Using details of the syntax would make the
>>> code fragile if any modifications are made to syntax in future.
>>
>> I don't think the code would be more fragile. Also, the syntax we're
>> talking about is not going to be modified anytime soon. Moreover, if
>> folding breaks, it is usually visible, so the bug will not be unnoticed.
>>
>> This code is going to be as low-level as it can be.
>>
>>> Debugging bugs in modification functions is not easy, according to my
>>> experience.
>>
>> No, it's not. 
>>
>> But this is not really related to whether you use Element or not.
>>
>>> One possible way to avoid performance issues during modification is
>>> running parser in advance. For example, folding an element may
>>> as well add information about the element to its text properties.
>>> This will not degrade performance of folding since we are already
>>> parsing the element during folding (at least, in
>>> org-hide-drawer-toggle).
>>
>> We can use this information stored at fold time. But I'm not even sure
>> we need it.
>>
>>> The problem with parsing an element during folding is that we cannot
>>> easily detect changes like below without re-parsing.
>>
>> Of course we can. It is only necessary to focus on changes that would
>> break the structure of the element. This does not entail a full parsing.
>>
>>> :PROPERTIES: <folded>
>>> :CREATED: [2020-05-18 Mon]
>>> :END: <- added line
>>> :ID: test
>>> :END:
>>>
>>> or even
>>>
>>> :PROPERTIES:
>>> :CREATED: [2020-05-18 Mon]
>>> :ID: test
>>> :END: <- delete this line
>>>
>>> :DRAWER: <folded, cannot be unfolded if we don't re-parse after deletion>
>>> test
>>> :END:
>>
>> Please have a look at the "sensitive parts" I wrote about. This takes
>> care of this kind of breakage.
>>
>>> The re-parsing can be done via regexp, as you suggested, but I don't
>>> like this idea, because it will end up re-implementing
>>> org-element-*-parser.
>>
>> You may have misunderstood my suggestion. See below.
>>
>>> Would it be acceptable to run org-element-*-parser
>>> in after-change-functions?
>>
>> I'd rather not do that. This is unnecessary consing, and matching, etc.
>>
>>> If I understand correctly, it is not as easy.
>>> Consider the following example:
>>>
>>> :PROPERTIES:
>>> :CREATED: [2020-05-18 Mon]
>>> <region-beginning>
>>> :ID: example
>>> :END:
>>>
>>> <... a lot of text, maybe containing other drawers ...>
>>>
>>> Nullam rutrum.
>>> Pellentesque dapibus suscipit ligula.
>>> <region-end>
>>> Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus.
>>>
>>> If the region gets deleted, the modification hooks from chars inside
>>> drawer will be called as (hook-function <region-beginning>
>>> <region-end>). So, there is still a need to find the drawer somehow to
>>> mark it as about to be modified (modification hooks are ran before
>>> actual modification).
>>
>> If we can stick with `after-change-functions' (or local equivalent),
>> that's better. It is more predictable than `before-change-functions' and
>> alike.
>>
>> If it is a deletion, here is the kind of checks we could do, depending
>> on when they are performed.
>>
>> Before actual changes :
>>
>>   1. The deletion is happening within a folded drawer (unnecessary step
>>      in local functions).
>>   2. The change deleted the sensitive line ":END:".
>>   3. Conclusion : unfold.
>>
>> Or, after actual changes :
>>
>>   1. The deletion involves a drawer.
>>   2. Text properties indicate that the beginning of the propertized part
>>      of the buffer start with org-drawer-regexp, but doesn't end with
>>      `org-property-end-re'. A "sensitive part" disappeared!
>>   3. Conclusion : unfold
>>
>> This is far away from parsing. IMO, a few checks cover all cases. Let me
>> know if you have questions about it.
>>
>> Also, note that the kind of change you describe will happen perhaps
>> 0.01% of the time. Most change are about one character, or a single
>> line, long.
>>
>>> The only difference between using modification hooks and
>>> before-change-functions is that modification hooks will trigger less
>>> frequently. 
>>
>> Exactly. Much less frequently. But extra care is required, as you noted
>> already.
>>
>>> Considering the performance of org-element-at-point, it is
>>> probably worth doing. Initially, I wanted to avoid it because setting a
>>> single before-change-functions hook sounded cleaner than setting
>>> modification-hooks, insert-behind-hooks, and insert-in-front-hooks.
>>
>> Well, `before-change-fuctions' and `after-change-functions' are not
>> clean at all: you modify an unrelated part of the buffer, but still call
>> those to check if a drawer needs to be unfolded somewhere.
>>
>> And, more importantly, they are not meant to be used together, i.e., you
>> cannot assume that a single call to `before-change-functions' always
>> happens before calling `after-change-functions'. This can be tricky if
>> you want to use the former to pass information to the latter.
>>
>> But I understand that they are easier to use than their local
>> counterparts. If you stick with (before|after)-change-functions, the
>> function being called needs to drop the ball very quickly if the
>> modification is not about folding changes. Also, I very much suggest to
>> stick to only `after-change-functions', if feasible (I think it is), per
>> above.
>>
>>> Moreover, these text properties would be copied by default if one uses 
>>> buffer-substring. Then, the hooks will also trigger later in the yanked
>>> text, which may cause all kinds of bugs.
>>
>> Indeed, that would be something to handle specifically. I.e.,
>> destructive modifications (i.e., those that unfold) could clear such
>> properties.
>>
>> It is more work. I don't know if it is worth the trouble if we can get
>> out quickly of `after-change-functions' for unrelated changes.
>>
>>> It was mostly an annoyance, because they returned different results on
>>> the same element. Specifically, they returned different :post-blank and
>>> :end properties, which does not sound right.
>>
>> OK. If you have a reproducible recipe, I can look into it and see what
>> can be done.
>>
>> Regards,
>>
>> -- 
>> Nicolas Goaziou
>
> -- 
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-23 13:53                                 ` Ihor Radchenko
@ 2020-05-23 15:26                                   ` Ihor Radchenko
  0 siblings, 0 replies; 55+ messages in thread
From: Ihor Radchenko @ 2020-05-23 15:26 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

Github link to the patch:
https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef 


Ihor Radchenko <yantar92@gmail.com> writes:

> The patch is attached
>
> diff --git a/contrib/lisp/org-notify.el b/contrib/lisp/org-notify.el
> index 9f8677871..ab470ea9b 100644
> --- a/contrib/lisp/org-notify.el
> +++ b/contrib/lisp/org-notify.el
> @@ -246,7 +246,7 @@ seconds.  The default value for SECS is 20."
>            (switch-to-buffer (find-file-noselect file))
>            (org-with-wide-buffer
>             (goto-char begin)
> -           (outline-show-entry))
> +           (org-show-entry))
>            (goto-char begin)
>            (search-forward "DEADLINE: <")
>            (search-forward ":")
> diff --git a/contrib/lisp/org-velocity.el b/contrib/lisp/org-velocity.el
> index bfc4d6c3e..2312b235c 100644
> --- a/contrib/lisp/org-velocity.el
> +++ b/contrib/lisp/org-velocity.el
> @@ -325,7 +325,7 @@ use it."
>    (save-excursion
>      (when narrow
>        (org-narrow-to-subtree))
> -    (outline-show-all)))
> +    (org-show-all)))
>  
>  (defun org-velocity-edit-entry/inline (heading)
>    "Edit entry at HEADING in the original buffer."
> diff --git a/doc/org-manual.org b/doc/org-manual.org
> index 96b160175..2ebe94538 100644
> --- a/doc/org-manual.org
> +++ b/doc/org-manual.org
> @@ -509,11 +509,11 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
>    Switch back to the startup visibility of the buffer (see [[*Initial
>    visibility]]).
>  
> -- {{{kbd(C-u C-u C-u TAB)}}} (~outline-show-all~) ::
> +- {{{kbd(C-u C-u C-u TAB)}}} (~org-show-all~) ::
>  
>    #+cindex: show all, command
>    #+kindex: C-u C-u C-u TAB
> -  #+findex: outline-show-all
> +  #+findex: org-show-all
>    Show all, including drawers.
>  
>  - {{{kbd(C-c C-r)}}} (~org-reveal~) ::
> @@ -529,18 +529,18 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
>    headings.  With a double prefix argument, also show the entire
>    subtree of the parent.
>  
> -- {{{kbd(C-c C-k)}}} (~outline-show-branches~) ::
> +- {{{kbd(C-c C-k)}}} (~org-show-branches~) ::
>  
>    #+cindex: show branches, command
>    #+kindex: C-c C-k
> -  #+findex: outline-show-branches
> +  #+findex: org-show-branches
>    Expose all the headings of the subtree, but not their bodies.
>  
> -- {{{kbd(C-c TAB)}}} (~outline-show-children~) ::
> +- {{{kbd(C-c TAB)}}} (~org-show-children~) ::
>  
>    #+cindex: show children, command
>    #+kindex: C-c TAB
> -  #+findex: outline-show-children
> +  #+findex: org-show-children
>    Expose all direct children of the subtree.  With a numeric prefix
>    argument {{{var(N)}}}, expose all children down to level
>    {{{var(N)}}}.
> @@ -7294,7 +7294,7 @@ its location in the outline tree, but behaves in the following way:
>    command (see [[*Visibility Cycling]]).  You can force cycling archived
>    subtrees with {{{kbd(C-TAB)}}}, or by setting the option
>    ~org-cycle-open-archived-trees~.  Also normal outline commands, like
> -  ~outline-show-all~, open archived subtrees.
> +  ~org-show-all~, open archived subtrees.
>  
>  -
>    #+vindex: org-sparse-tree-open-archived-trees
> diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
> index ab13f926c..ad9244940 100644
> --- a/lisp/org-agenda.el
> +++ b/lisp/org-agenda.el
> @@ -6826,7 +6826,7 @@ and stored in the variable `org-prefix-format-compiled'."
>  	    (t "  %-12:c%?-12t% s")))
>  	(start 0)
>  	varform vars var e c f opt)
> -    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)"
> +    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)"
>  			 s start)
>        (setq var (or (cdr (assoc (match-string 4 s)
>  				'(("c" . category) ("t" . time) ("l" . level) ("s" . extra)
> @@ -9138,20 +9138,20 @@ if it was hidden in the outline."
>       ((and (called-interactively-p 'any) (= more 1))
>        (message "Remote: show with default settings"))
>       ((= more 2)
> -      (outline-show-entry)
> +      (org-show-entry)
>        (org-show-children)
>        (save-excursion
>  	(org-back-to-heading)
>  	(run-hook-with-args 'org-cycle-hook 'children))
>        (message "Remote: CHILDREN"))
>       ((= more 3)
> -      (outline-show-subtree)
> +      (org-show-subtree)
>        (save-excursion
>  	(org-back-to-heading)
>  	(run-hook-with-args 'org-cycle-hook 'subtree))
>        (message "Remote: SUBTREE"))
>       ((> more 3)
> -      (outline-show-subtree)
> +      (org-show-subtree)
>        (message "Remote: SUBTREE AND ALL DRAWERS")))
>      (select-window win)))
>  
> diff --git a/lisp/org-archive.el b/lisp/org-archive.el
> index d3e12d17b..d864dad8a 100644
> --- a/lisp/org-archive.el
> +++ b/lisp/org-archive.el
> @@ -330,7 +330,7 @@ direct children of this heading."
>  		      (insert (if datetree-date "" "\n") heading "\n")
>  		      (end-of-line 0))
>  		    ;; Make the subtree visible
> -		    (outline-show-subtree)
> +		    (org-show-subtree)
>  		    (if org-archive-reversed-order
>  			(progn
>  			  (org-back-to-heading t)
> diff --git a/lisp/org-colview.el b/lisp/org-colview.el
> index e50a4d7c8..e656df555 100644
> --- a/lisp/org-colview.el
> +++ b/lisp/org-colview.el
> @@ -699,7 +699,7 @@ FUN is a function called with no argument."
>  			  (move-beginning-of-line 2)
>  			  (org-at-heading-p t)))))
>      (unwind-protect (funcall fun)
> -      (when hide-body (outline-hide-entry)))))
> +      (when hide-body (org-hide-entry)))))
>  
>  (defun org-columns-previous-allowed-value ()
>    "Switch to the previous allowed value for this column."
> diff --git a/lisp/org-compat.el b/lisp/org-compat.el
> index 635a38dcd..8fe271896 100644
> --- a/lisp/org-compat.el
> +++ b/lisp/org-compat.el
> @@ -139,12 +139,8 @@ This is a floating point number if the size is too large for an integer."
>  ;;; Emacs < 25.1 compatibility
>  
>  (when (< emacs-major-version 25)
> -  (defalias 'outline-hide-entry 'hide-entry)
> -  (defalias 'outline-hide-sublevels 'hide-sublevels)
> -  (defalias 'outline-hide-subtree 'hide-subtree)
>    (defalias 'outline-show-branches 'show-branches)
>    (defalias 'outline-show-children 'show-children)
> -  (defalias 'outline-show-entry 'show-entry)
>    (defalias 'outline-show-subtree 'show-subtree)
>    (defalias 'xref-find-definitions 'find-tag)
>    (defalias 'format-message 'format)
> diff --git a/lisp/org-element.el b/lisp/org-element.el
> index ac41b7650..2d5c8d771 100644
> --- a/lisp/org-element.el
> +++ b/lisp/org-element.el
> @@ -4320,7 +4320,7 @@ element or object.  Meaningful values are `first-section',
>  TYPE is the type of the current element or object.
>  
>  If PARENT? is non-nil, assume the next element or object will be
> -located inside the current one.  "
> +located inside the current one."
>    (if parent?
>        (pcase type
>  	(`headline 'section)
> diff --git a/lisp/org-keys.el b/lisp/org-keys.el
> index c006e9c12..deb5d7b90 100644
> --- a/lisp/org-keys.el
> +++ b/lisp/org-keys.el
> @@ -437,7 +437,7 @@ COMMANDS is a list of alternating OLDDEF NEWDEF command names."
>    #'org-next-visible-heading)
>  (define-key org-mode-map [remap outline-previous-visible-heading]
>    #'org-previous-visible-heading)
> -(define-key org-mode-map [remap show-children] #'org-show-children)
> +(define-key org-mode-map [remap outline-show-children] #'org-show-children)
>  
>  ;;;; Make `C-c C-x' a prefix key
>  (org-defkey org-mode-map (kbd "C-c C-x") (make-sparse-keymap))
> diff --git a/lisp/org-macs.el b/lisp/org-macs.el
> index a02f713ca..fa0a658f0 100644
> --- a/lisp/org-macs.el
> +++ b/lisp/org-macs.el
> @@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead."
>  
>  
>  \f
> -;;; Overlays
> +;;; Overlays and text properties
>  
>  (defun org-overlay-display (ovl text &optional face evap)
>    "Make overlay OVL display TEXT with face FACE."
> @@ -705,18 +705,99 @@ If DELETE is non-nil, delete all those overlays."
>  	    (delete (delete-overlay ov))
>  	    (t (push ov found))))))
>  
> +(defun org-remove-text-properties (start end properties &optional object)
> +  "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions.
> +Do not remove invisible text properties specified by 'outline,
> +'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this
> +is needed to keep outlines, drawers, and blocks hidden unless they are
> +toggled by user.
> +Note: The below may be too specific and create troubles if more
> +invisibility specs are added to org in future"
> +  (when (plist-member properties 'invisible)
> +    (let ((pos start)
> +	  next spec)
> +      (while (< pos end)
> +	(setq next (next-single-property-change pos 'invisible nil end)
> +              spec (get-text-property pos 'invisible))
> +	(unless (memq spec (list 'org-hide-block
> +				 'org-hide-drawer
> +				 'outline))
> +          (remove-text-properties pos next '(invisible nil) object))
> +	(setq pos next))))
> +  (when-let ((properties-stripped (org-plist-delete properties 'invisible)))
> +    (remove-text-properties start end properties-stripped object)))
> +
> +(defun org--find-text-property-region (pos prop)
> +  "Find a region containing PROP text property around point POS."
> +  (let* ((beg (and (get-text-property pos prop) pos))
> +	 (end beg))
> +    (when beg
> +      ;; when beg is the first point in the region, `previous-single-property-change'
> +      ;; will return nil.
> +      (setq beg (or (previous-single-property-change pos prop)
> +		    beg))
> +      ;; when end is the last point in the region, `next-single-property-change'
> +      ;; will return nil.
> +      (setq end (or (next-single-property-change pos prop)
> +		    end))
> +      (unless (= beg end) ; this should not happen
> +        (cons beg end)))))
> +
>  (defun org-flag-region (from to flag spec)
>    "Hide or show lines from FROM to TO, according to FLAG.
>  SPEC is the invisibility spec, as a symbol."
> -  (remove-overlays from to 'invisible spec)
> -  ;; Use `front-advance' since text right before to the beginning of
> -  ;; the overlay belongs to the visible line than to the contents.
> -  (when flag
> -    (let ((o (make-overlay from to nil 'front-advance)))
> -      (overlay-put o 'evaporate t)
> -      (overlay-put o 'invisible spec)
> -      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
> -
> +  (pcase spec
> +    ('outline
> +     (remove-overlays from to 'invisible spec)
> +     ;; Use `front-advance' since text right before to the beginning of
> +     ;; the overlay belongs to the visible line than to the contents.
> +     (when flag
> +       (let ((o (make-overlay from to nil 'front-advance)))
> +	 (overlay-put o 'evaporate t)
> +	 (overlay-put o 'invisible spec)
> +	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
> +    (_
> +     ;; Use text properties instead of overlays for speed.
> +     ;; Overlays are too slow (Emacs Bug#35453).
> +     (with-silent-modifications
> +       ;; keep a backup stack of old text properties
> +       (save-excursion
> +	 (goto-char from)
> +	 (while (< (point) to)
> +	   (let ((old-spec (get-text-property (point) 'invisible))
> +		 (end (next-single-property-change (point) 'invisible nil to)))
> +	     (when old-spec
> +	       (alter-text-property (point) end 'org-property-stack-invisible
> +				    (lambda (stack)
> +				      (if (or (eq old-spec (car stack))
> +					      (eq spec old-spec)
> +					      (eq old-spec 'outline))
> +					  stack
> +					(cons old-spec stack)))))
> +	     (goto-char end))))
> +
> +       ;; cleanup everything
> +       (remove-text-properties from to '(invisible nil))
> +
> +       ;; Recover properties from the backup stack
> +       (unless flag
> +	 (save-excursion
> +	   (goto-char from)
> +	   (while (< (point) to)
> +             (let ((stack (get-text-property (point) 'org-property-stack-invisible))
> +		   (end (next-single-property-change (point) 'org-property-stack-invisible nil to)))
> +               (if (not stack)
> +		   (remove-text-properties (point) end '(org-property-stack-invisible nil))
> +		 (put-text-property (point) end 'invisible (car stack))
> +		 (alter-text-property (point) end 'org-property-stack-invisible
> +				      (lambda (stack)
> +					(cdr stack))))
> +               (goto-char end)))))
> +       
> +       (when flag
> +	 (put-text-property from to 'rear-non-sticky nil)
> +	 (put-text-property from to 'front-sticky t)
> +	 (put-text-property from to 'invisible spec))))))
>  
>  \f
>  ;;; Regexp matching
> diff --git a/lisp/org-src.el b/lisp/org-src.el
> index c9eef744e..e89a1c580 100644
> --- a/lisp/org-src.el
> +++ b/lisp/org-src.el
> @@ -523,8 +523,8 @@ Leave point in edit buffer."
>  	(org-src-switch-to-buffer buffer 'edit)
>  	;; Insert contents.
>  	(insert contents)
> -	(remove-text-properties (point-min) (point-max)
> -				'(display nil invisible nil intangible nil))
> +	(org-remove-text-properties (point-min) (point-max)
> +				    '(display nil invisible nil intangible nil))
>  	(unless preserve-ind (org-do-remove-indentation))
>  	(set-buffer-modified-p nil)
>  	(setq buffer-file-name nil)
> diff --git a/lisp/org-table.el b/lisp/org-table.el
> index 6462b99c4..75801161b 100644
> --- a/lisp/org-table.el
> +++ b/lisp/org-table.el
> @@ -2001,7 +2001,7 @@ toggle `org-table-follow-field-mode'."
>     (arg
>      (let ((b (save-excursion (skip-chars-backward "^|") (point)))
>  	  (e (save-excursion (skip-chars-forward "^|\r\n") (point))))
> -      (remove-text-properties b e '(invisible t intangible t))
> +      (org-remove-text-properties b e '(invisible t intangible t))
>        (if (and (boundp 'font-lock-mode) font-lock-mode)
>  	  (font-lock-fontify-block))))
>     (t
> @@ -2028,7 +2028,7 @@ toggle `org-table-follow-field-mode'."
>        (setq word-wrap t)
>        (goto-char (setq p (point-max)))
>        (insert (org-trim field))
> -      (remove-text-properties p (point-max) '(invisible t intangible t))
> +      (org-remove-text-properties p (point-max) '(invisible t intangible t))
>        (goto-char p)
>        (setq-local org-finish-function 'org-table-finish-edit-field)
>        (setq-local org-window-configuration cw)
> diff --git a/lisp/org.el b/lisp/org.el
> index e577dc661..360974135 100644
> --- a/lisp/org.el
> +++ b/lisp/org.el
> @@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.")
>  (declare-function cdlatex-math-symbol "ext:cdlatex")
>  (declare-function Info-goto-node "info" (nodename &optional fork strict-case))
>  (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag))
> +(declare-function isearch-filter-visible "isearch" (beg end))
>  (declare-function org-add-archive-files "org-archive" (files))
>  (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom))
>  (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour))
> @@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.")
>  
>  (defvar ffap-url-regexp)
>  (defvar org-element-paragraph-separate)
> +(defvar org-element-all-objects)
> +(defvar org-element-all-elements)
> +(defvar org-element-greater-elements)
>  (defvar org-indent-indentation-per-level)
>  (defvar org-radio-target-regexp)
>  (defvar org-target-link-regexp)
> @@ -4734,9 +4738,381 @@ This is for getting out of special buffers like capture.")
>  
>  ;;;; Define the Org mode
>  
> +;;; Handling buffer modifications
> +
>  (defun org-before-change-function (_beg _end)
>    "Every change indicates that a table might need an update."
>    (setq org-table-may-need-update t))
> +
> +(defvar-local org--modified-elements nil
> +  "List of elements, marked as recently modified.
> +There is no guarantee that the elements in this list are fully parsed.
> +Only the element type, :begin and :end properties of the elements are
> +guaranteed to be available. The :begin and :end element properties
> +contain markers instead of positions.")
> +
> +(defvar org-track-element-modification-default-sensitive-commands '(self-insert-command)
> +  "List of commands triggerring element modifications unconditionally.")
> +
> +(defvar org--element-beginning-re-alist `((center-block . "^[ \t]*#\\+begin_center[ \t]*$")
> +                                       (property-drawer . ,org-property-start-re)
> +				       (drawer . ,org-drawer-regexp)
> +                                       (quote-block . "^[ \t]*#\\+begin_quote[ \t]*$")
> +                                       (special-block . "^[ \t]*#\\+begin_\\([^ ]+\\).*$"))
> +  "Alist of regexps matching beginning of elements.
> +Group 1 in the regexps (if any) contains the element type.")
> +
> +(defvar org--element-end-re-alist `((center-block . "^[ \t]*#\\+end_center[ \t]*$")
> +				 (property-drawer . ,org-property-end-re)
> +				 (drawer . ,org-property-end-re)
> +				 (quote-block . "^[ \t]*#\\+end_quote[ \t]*$")
> +				 (special-block . "^[ \t]*#\\+end_\\([^ ]+\\).*$"))
> +  "Alist of regexps matching end of elements.
> +Group 1 in the regexps (if any) contains the element type or END.")
> +
> +(defvar org-track-element-modifications
> +  `((property-drawer  . (:after-change-function
> +                         org--drawer-or-block-unfold-maybe))
> +    (drawer  . (:after-change-function
> +                org--drawer-or-block-unfold-maybe))
> +    (center-block . (:after-change-function
> +                     org--drawer-or-block-unfold-maybe))
> +    (quote-block  . (:after-change-function
> +                     org--drawer-or-block-unfold-maybe))
> +    (special-block  . (:after-change-function
> +                       org--drawer-or-block-unfold-maybe)))
> +  "Alist of elements to be tracked for modifications.
> +The modification is only triggered according to :sensitive-re-list and
> +:sensitive-command-list (see below).
> +Each element of the alist is a cons of an element symbol and plist
> +defining how and when the modifications are handled.
> +In case of recursive elements/duplicates, the first element from the
> +list is considered.
> +The plist can have the following properties:
> +- :element-beginning-re   :: regex matching beginning of the element
> +  (default)               :: (alist-get element org--element-beginning-re-alist)
> +- :element-end-re         :: regex matching end of the element
> +  (default)               :: (alist-get element org--element-end-re-alist)
> +- :after-change-function  :: function called after the modification
> +The function must accept a single argument - element from
> +`org--modified-elements'.")
> +
> +(defun org--get-element-region-at-point (types)
> +  "Return TYPES element at point or nil.
> +If TYPES is a list, return first element at point from the list.  The
> +returned value is partially parsed element only containing :begin and
> +:end properties.  Only elements listed in
> +org--element-beginning-re-alist and org--element-end-re-alist can be
> +parsed here."
> +  (catch 'exit
> +    (dolist (type (if (listp types) types (list types)))
> +      (let ((begin-re (alist-get type org--element-beginning-re-alist))
> +	    (end-re (alist-get type org--element-end-re-alist))
> +            (begin-limit (save-excursion (org-with-limited-levels
> +					  (org-back-to-heading-or-point-min 'invisible-ok))
> +					 (point)))
> +            (end-limit (or (save-excursion (outline-next-heading))
> +			   (point-max)))
> +            (point (point))
> +	    begin end closest-begin)
> +	(when (and begin-re end-re)
> +	  (save-excursion
> +	    (end-of-line)
> +	    (when (re-search-backward begin-re begin-limit 'noerror) (setq begin (point)))
> +	    (when (re-search-forward end-re end-limit 'noerror) (setq end (point)))
> +            (setq closest-begin begin)
> +            ;; slurp unmatched begin-re
> +	    (when (and begin end)
> +              (goto-char begin)
> +              (while (and (re-search-backward begin-re begin-limit 'noerror)
> +			  (= end (save-excursion (re-search-forward end-re end-limit 'noerror))))
> +		(setq begin (point)))
> +              (when (and (>= point begin) (<= point end))
> +		(throw 'exit
> +		       (list type
> +			     (list
> +			      :begin begin
> +			      :end end)))))))))))
> +
> +(defun org--get-next-element-region-at-point (types &optional limit previous)
> +  "Return TYPES element after point or nil.
> +If TYPES is a list, return first element after point from the list.
> +If PREVIOUS is non-nil, return first TYPES element before point.
> +Limit search by LIMIT or previous/next heading.
> +The returned value is partially parsed element only containing :begin
> +and :end properties.  Only elements listed in
> +org--element-beginning-re-alist and org--element-end-re-alist can be
> +parsed here."
> +  (catch 'exit
> +    (dolist (type (if (listp types) types (list types)))
> +      (let* ((begin-re (alist-get type org--element-beginning-re-alist))
> +	     (end-re (alist-get type org--element-end-re-alist))
> +             (limit (or limit (if previous
> +				  (save-excursion
> +				    (org-with-limited-levels
> +				     (org-back-to-heading-or-point-min 'invisible-ok)
> +				     (point)))
> +				(or (save-excursion (outline-next-heading))
> +				    (point-max)))))
> +	     begin end)
> +	(when (and begin-re end-re)
> +	  (save-excursion
> +            (if previous
> +                (when (re-search-backward begin-re limit 'noerror)
> +		  (when-let ((el (org--get-element-region-at-point type)))
> +		    (setq begin (org-element-property :begin el))
> +		    (setq end (org-element-property :end el))))
> +	      (when (re-search-forward begin-re limit 'noerror)
> +                (when-let ((el (org--get-element-region-at-point type)))
> +		  (setq begin (org-element-property :begin el))
> +		  (setq end (org-element-property :end el))))))
> +	  (when (and begin end)
> +            (throw 'exit
> +		   (list type
> +			 (list
> +			  :begin begin
> +			  :end end)))))))))
> +
> +(defun org--find-elements-in-region (beg end elements &optional include-partial include-neighbours)
> +  "Find all elements from ELEMENTS in region BEG . END.
> +All the listed elements must be resolvable by
> +`org--get-element-region-at-point'.
> +Include elements if they are partially inside region when
> +INCLUDE-PARTIAL is non-nil.
> +Include preceding/subsequent neighbouring elements when no partial
> +element is found at the beginning/end of the region and
> +INCLUDE-NEIGHBOURS is non-nil."
> +  (when include-partial
> +    (org-with-point-at beg
> +      (let ((new-beg (org-element-property :begin (org--get-element-region-at-point elements))))
> +	(if new-beg
> +	    (setq beg new-beg)
> +          (when (and include-neighbours
> +		     (setq new-beg (org-element-property :begin
> +						      (org--get-next-element-region-at-point elements
> +											  (point-min)
> +											  'previous))))
> +            (setq beg new-beg))))
> +      (when (memq 'headline elements)
> +	(when-let ((new-beg (save-excursion
> +			      (org-with-limited-levels (outline-previous-heading)))))
> +          (setq beg new-beg))))
> +    (org-with-point-at end
> +      (let ((new-end (org-element-property :end (org--get-element-region-at-point elements))))
> +	(if new-end
> +	    (setq end new-end)
> +          (when (and include-neighbours
> +		     (setq new-end (org-element-property :end
> +						      (org--get-next-element-region-at-point elements (point-max)))))
> +            (setq end new-end))))
> +      (when (memq 'headline elements)
> +	(when-let ((new-end (org-with-limited-levels (outline-next-heading))))
> +          (setq end (1- new-end))))))
> +  (save-excursion
> +    (save-restriction
> +      (narrow-to-region beg end)
> +      (goto-char (point-min))
> +      (let (result el)
> +	(while (setq el (org--get-next-element-region-at-point elements end))
> +          (push el result)
> +          (goto-char (org-element-property :end el)))
> +        result))))
> +
> +(defun org--drawer-or-block-unfold-maybe (el)
> +  "Update visibility of modified folded drawer/block EL.
> +If text was added to hidden drawer/block, make sure that the text is
> +also hidden, unless the change was done by a command listed in
> +`org-track-element-modification-default-sensitive-commands'.  If the
> +modification destroyed the drawer/block, reveal the hidden text in
> +former drawer/block.  If the modification shrinked/expanded the
> +drawer/block beyond the hidden text, reveal the affected
> +drawers/blocks as well.
> +Examples:
> +----------------------------------------------
> +----------------------------------------------
> +Case #1 (the element content is hidden):
> +----------------------------------------------
> +:PROPERTIES:
> +:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
> +:END:
> +----------------------------------------------
> +is changed to
> +----------------------------------------------
> +:ROPERTIES:
> +:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
> +:END:
> +----------------------------------------------
> +Text is revealed, because we have drawer in place of property-drawer
> +----------------------------------------------
> +----------------------------------------------
> +Case #2 (the element content is hidden):
> +----------------------------------------------
> +:ROPERTIES:
> +:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
> +:END:
> +----------------------------------------------
> +is changed to
> +----------------------------------------------
> +:OPERTIES:
> +:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
> +:END:
> +----------------------------------------------
> +The text remains hidden since it is still a drawer.
> +----------------------------------------------
> +----------------------------------------------
> +Case #3: (the element content is hidden):
> +----------------------------------------------
> +:FOO:
> +bar
> +tmp
> +:END:
> +----------------------------------------------
> +is changed to
> +----------------------------------------------
> +:FOO:
> +bar
> +:END:
> +tmp
> +:END:
> +----------------------------------------------
> +The text is revealed because the drawer contents shrank.
> +----------------------------------------------
> +----------------------------------------------
> +Case #4: (the element content is hidden in both the drawers):
> +----------------------------------------------
> +:FOO:
> +bar
> +tmp
> +:END:
> +:BAR:
> +jjd
> +:END:
> +----------------------------------------------
> +is changed to
> +----------------------------------------------
> +:FOO:
> +bar
> +tmp
> +:BAR:
> +jjd
> +:END:
> +----------------------------------------------
> +The text is revealed in both the drawers because the drawers are merged
> +into a new drawer.
> +----------------------------------------------
> +----------------------------------------------
> +Case #5: (the element content is hidden)
> +----------------------------------------------
> +:test:
> +Vivamus id enim.
> +:end:
> +----------------------------------------------
> +is changed to
> +----------------------------------------------
> +:drawer:
> +:test:
> +Vivamus id enim.
> +:end:
> +----------------------------------------------
> +The text is revealed in the drawer because the drawer expended.
> +----------------------------------------------
> +----------------------------------------------
> +Case #6: (the element content is hidden):
> +----------------------------------------------
> +:test:
> +Vivamus id enim.
> +:end:
> +----------------------------------------------
> +is changed to
> +----------------------------------------------
> +:test:
> +Vivamus id enim.
> +:end:
> +Nam a sapien.
> +:end:
> +----------------------------------------------
> +The text remains hidden because drawer contents is always before the
> +first :end:."
> +  (save-match-data
> +    (save-excursion
> +      (save-restriction
> +	(goto-char (org-element-property :begin el))
> +	(let* ((newel (org--get-element-region-at-point
> +		       (mapcar (lambda (el)
> +				 (when (string-match-p (regexp-opt '("block" "drawer"))
> +						       (symbol-name (car el)))
> +                                   (car el)))
> +                               org-track-element-modifications)))
> +	       (spec (if (string-match-p "block" (symbol-name (org-element-type el)))
> +			 'org-hide-block
> +		       (if (string-match-p "drawer" (symbol-name (org-element-type el)))
> +			   'org-hide-drawer
> +			 t)))
> +               (toggle-func (if (eq spec 'org-hide-drawer)
> +				#'org-hide-drawer-toggle
> +                              (if (eq spec 'org-hide-block)
> +				  #'org-hide-block-toggle
> +                                #'ignore)))) ; this should not happen
> +	  (if (and (equal (org-element-type el) (org-element-type newel))
> +		   (equal (marker-position (org-element-property :begin el))
> +			  (org-element-property :begin newel))
> +		   (equal (marker-position (org-element-property :end el))
> +			  (org-element-property :end newel)))
> +	      (when (text-property-any (marker-position (org-element-property :begin el))
> +				       (marker-position (org-element-property :end el))
> +				       'invisible spec)
> +                (goto-char (org-element-property :begin newel))
> +		(if (memq this-command org-track-element-modification-default-sensitive-commands)
> +		    ;; reveal if change was made by typing
> +		    (funcall toggle-func 'off)
> +		  ;; re-hide the inserted text
> +		  ;; FIXME: opening the drawer before hiding should not be needed here
> +		  (funcall toggle-func 'off) ; this is needed to avoid showing double ellipsis
> +		  (funcall toggle-func 'hide)))
> +            ;; The element was destroyed. Reveal everything.
> +            (org-flag-region (marker-position (org-element-property :begin el))
> +			  (marker-position (org-element-property :end el))
> +			  nil spec)
> +            (when newel
> +              (org-flag-region (org-element-property :begin newel)
> +			    (org-element-property :end newel)
> +			    nil spec))))))))
> +
> +(defun org--before-element-change-function (beg end)
> +  "Register upcoming element modifications in `org--modified-elements' for all elements interesting with BEG END."
> +  (save-match-data
> +    (save-excursion
> +      (save-restriction
> +        (widen)
> +	(dolist (el (org--find-elements-in-region beg
> +					       end
> +					       (mapcar #'car org-track-element-modifications)
> +					       'include-partial
> +                                               'include-neighbours))
> +	  (let* ((beg-marker (copy-marker (org-element-property :begin el) 't))
> +		 (end-marker (copy-marker (org-element-property :end el) 't)))
> +	    (when (and (marker-position beg-marker) (marker-position end-marker))
> +	      (org-element-put-property el :begin beg-marker)
> +	      (org-element-put-property el :end end-marker)
> +	      (add-to-list 'org--modified-elements el))))))))
> +
> +;; FIXME: this function may be called many times during routine modifications
> +;; The normal way to avoid this is `combine-after-change-calls' - not
> +;; the case in most org primitives.
> +(defun org--after-element-change-function (&rest _)
> +  "Handle changed elements from `org--modified-elements'."
> +  (dolist (el org--modified-elements)
> +    (save-match-data
> +      (save-excursion
> +        (save-restriction
> +          (widen)
> +	  (when-let* ((type (org-element-type el))
> +		      (change-func (plist-get (alist-get type org-track-element-modifications)
> +					      :after-change-function)))
> +	    (with-demoted-errors
> +		(funcall (symbol-function change-func) el)))))))
> +  (setq org--modified-elements nil))
> +
>  (defvar org-mode-map)
>  (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param.
>  (defvar org-agenda-keep-modes nil)      ; Dynamically-scoped param.
> @@ -4818,6 +5194,9 @@ The following commands are available:
>    ;; Activate before-change-function
>    (setq-local org-table-may-need-update t)
>    (add-hook 'before-change-functions 'org-before-change-function nil 'local)
> +  (add-hook 'before-change-functions 'org--before-element-change-function nil 'local)
> +  ;; Activate after-change-function
> +  (add-hook 'after-change-functions 'org--after-element-change-function nil 'local)
>    ;; Check for running clock before killing a buffer
>    (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local)
>    ;; Initialize macros templates.
> @@ -4869,6 +5248,10 @@ The following commands are available:
>    (setq-local outline-isearch-open-invisible-function
>  	      (lambda (&rest _) (org-show-context 'isearch)))
>  
> +  ;; Make isearch search in blocks hidden via text properties
> +  (setq-local isearch-filter-predicate #'org--isearch-filter-predicate)
> +  (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local)
> +
>    ;; Setup the pcomplete hooks
>    (setq-local pcomplete-command-completion-function #'org-pcomplete-initial)
>    (setq-local pcomplete-command-name-function #'org-command-at-point)
> @@ -5050,8 +5433,8 @@ stacked delimiters is N.  Escaping delimiters is not possible."
>  	      (when verbatim?
>  		(org-remove-flyspell-overlays-in
>  		 (match-beginning 0) (match-end 0))
> -		(remove-text-properties (match-beginning 2) (match-end 2)
> -					'(display t invisible t intangible t)))
> +		(org-remove-text-properties (match-beginning 2) (match-end 2)
> +					 '(display t invisible t intangible t)))
>  	      (add-text-properties (match-beginning 2) (match-end 2)
>  				   '(font-lock-multiline t org-emphasis t))
>  	      (when (and org-hide-emphasis-markers
> @@ -5166,7 +5549,7 @@ This includes angle, plain, and bracket links."
>  	    (if (not (eq 'bracket style))
>  		(add-text-properties start end properties)
>  	      ;; Handle invisible parts in bracket links.
> -	      (remove-text-properties start end '(invisible nil))
> +	      (org-remove-text-properties start end '(invisible nil))
>  	      (let ((hidden
>  		     (append `(invisible
>  			       ,(or (org-link-get-parameter type :display)
> @@ -5186,8 +5569,8 @@ This includes angle, plain, and bracket links."
>  (defun org-activate-code (limit)
>    (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t)
>      (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
> -    (remove-text-properties (match-beginning 0) (match-end 0)
> -			    '(display t invisible t intangible t))
> +    (org-remove-text-properties (match-beginning 0) (match-end 0)
> +			     '(display t invisible t intangible t))
>      t))
>  
>  (defcustom org-src-fontify-natively t
> @@ -5258,8 +5641,8 @@ by a #."
>  	    (setq block-end (match-beginning 0)) ; includes the final newline.
>  	    (when quoting
>  	      (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline)
> -	      (remove-text-properties beg end-of-endline
> -				      '(display t invisible t intangible t)))
> +	      (org-remove-text-properties beg end-of-endline
> +				       '(display t invisible t intangible t)))
>  	    (add-text-properties
>  	     beg end-of-endline '(font-lock-fontified t font-lock-multiline t))
>  	    (org-remove-flyspell-overlays-in beg bol-after-beginline)
> @@ -5313,8 +5696,8 @@ by a #."
>  	     '(font-lock-fontified t face org-document-info))))
>  	 ((string-prefix-p "+caption" dc1)
>  	  (org-remove-flyspell-overlays-in (match-end 2) (match-end 0))
> -	  (remove-text-properties (match-beginning 0) (match-end 0)
> -				  '(display t invisible t intangible t))
> +	  (org-remove-text-properties (match-beginning 0) (match-end 0)
> +				   '(display t invisible t intangible t))
>  	  ;; Handle short captions.
>  	  (save-excursion
>  	    (beginning-of-line)
> @@ -5336,8 +5719,8 @@ by a #."
>  	   '(font-lock-fontified t face font-lock-comment-face)))
>  	 (t ;; just any other in-buffer setting, but not indented
>  	  (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
> -	  (remove-text-properties (match-beginning 0) (match-end 0)
> -				  '(display t invisible t intangible t))
> +	  (org-remove-text-properties (match-beginning 0) (match-end 0)
> +				   '(display t invisible t intangible t))
>  	  (add-text-properties beg (match-end 0)
>  			       '(font-lock-fontified t face org-meta-line))
>  	  t))))))
> @@ -5859,10 +6242,11 @@ If TAG is a number, get the corresponding match group."
>  	 (inhibit-modification-hooks t)
>  	 deactivate-mark buffer-file-name buffer-file-truename)
>      (decompose-region beg end)
> -    (remove-text-properties beg end
> -			    '(mouse-face t keymap t org-linked-text t
> -					 invisible t intangible t
> -					 org-emphasis t))
> +    (org-remove-text-properties beg end
> +			     '(mouse-face t keymap t org-linked-text t
> +					  invisible t
> +                                          intangible t
> +					  org-emphasis t))
>      (org-remove-font-lock-display-properties beg end)))
>  
>  (defconst org-script-display  '(((raise -0.3) (height 0.7))
> @@ -5970,6 +6354,29 @@ open and agenda-wise Org files."
>  
>  ;;;; Headlines visibility
>  
> +(defun org-hide-entry ()
> +  "Hide the body directly following this heading."
> +  (interactive)
> +  (save-excursion
> +    (outline-back-to-heading)
> +    (outline-end-of-heading)
> +    (org-flag-region (point) (progn (outline-next-preface) (point)) t 'outline)))
> +
> +(defun org-hide-subtree ()
> +  "Hide everything after this heading at deeper levels."
> +  (interactive)
> +  (org-flag-subtree t))
> +
> +(defun org-hide-sublevels (levels)
> +  "Hide everything but the top LEVELS levels of headers, in whole buffer.
> +This also unhides the top heading-less body, if any.
> +
> +Interactively, the prefix argument supplies the value of LEVELS.
> +When invoked without a prefix argument, LEVELS defaults to the level
> +of the current heading, or to 1 if the current line is not a heading."
> +  (cl-letf (((symbol-function 'outline-flag-region) #'org-flag-region))
> +    (org-hide-sublevels levels)))
> +
>  (defun org-show-entry ()
>    "Show the body directly following this heading.
>  Show the heading too, if it is currently invisible."
> @@ -5988,6 +6395,16 @@ Show the heading too, if it is currently invisible."
>         'outline)
>        (org-cycle-hide-property-drawers 'children))))
>  
> +(defun org-show-heading ()
> +  "Show the current heading and move to its end."
> +  (org-flag-region (- (point)
> + 		   (if (bobp) 0
> + 		     (if (and outline-blank-line
> +                              (eq (char-before (1- (point))) ?\n))
> + 			 2 1)))
> +		(progn (outline-end-of-heading) (point))
> +		nil))
> +
>  (defun org-show-children (&optional level)
>    "Show all direct subheadings of this heading.
>  Prefix arg LEVEL is how many levels below the current level
> @@ -6031,6 +6448,11 @@ heading to appear."
>    (org-flag-region
>     (point) (save-excursion (org-end-of-subtree t t)) nil 'outline))
>  
> +(defun org-show-branches ()
> +  "Show all subheadings of this heading, but not their bodies."
> +  (interactive)
> +  (org-show-children 1000))
> +
>  ;;;; Blocks and drawers visibility
>  
>  (defun org--hide-wrapper-toggle (element category force no-error)
> @@ -6064,8 +6486,8 @@ Return a non-nil value when toggling is successful."
>  	(unless (let ((eol (line-end-position)))
>  		  (and (> eol start) (/= eol end)))
>  	  (let* ((spec (cond ((eq category 'block) 'org-hide-block)
> -			     ((eq type 'property-drawer) 'outline)
> -			     (t 'org-hide-drawer)))
> +			     ((eq category 'drawer) 'org-hide-drawer)
> +			     (t 'outline)))
>  		 (flag
>  		  (cond ((eq force 'off) nil)
>  			(force t)
> @@ -6158,10 +6580,7 @@ STATE should be one of the symbols listed in the docstring of
>  	       (when (org-at-property-drawer-p)
>  		 (let* ((case-fold-search t)
>  			(end (re-search-forward org-property-end-re)))
> -		   ;; Property drawers use `outline' invisibility spec
> -		   ;; so they can be swallowed once we hide the
> -		   ;; outline.
> -		   (org-flag-region start end t 'outline)))))))))))
> +		   (org-flag-region start end t 'org-hide-drawer)))))))))))
>  
>  ;;;; Visibility cycling
>  
> @@ -6536,7 +6955,7 @@ With a numeric prefix, show all headlines up to that level."
>  		     (org-narrow-to-subtree)
>  		     (org-content))))
>  		((or "all" "showall")
> -		 (outline-show-subtree))
> +		 (org-show-subtree))
>  		(_ nil)))
>  	    (org-end-of-subtree)))))))
>  
> @@ -6609,7 +7028,7 @@ This function is the default value of the hook `org-cycle-hook'."
>  	  (while (re-search-forward re nil t)
>  	    (when (and (not (org-invisible-p))
>  		       (org-invisible-p (line-end-position)))
> -	      (outline-hide-entry))))
> +	      (org-hide-entry))))
>  	(org-cycle-hide-property-drawers 'all)
>  	(org-cycle-show-empty-lines 'overview)))))
>  
> @@ -6683,8 +7102,13 @@ information."
>      ;; expose it.
>      (dolist (o (overlays-at (point)))
>        (when (memq (overlay-get o 'invisible)
> -		  '(org-hide-block org-hide-drawer outline))
> +		  '(outline))
>  	(delete-overlay o)))
> +    (when (memq (get-text-property (point) 'invisible)
> +		'(org-hide-block org-hide-drawer))
> +      (let ((spec (get-text-property (point) 'invisible))
> +	    (region (org--find-text-property-region (point) 'invisible)))
> +	(org-flag-region (car region) (cdr region) nil spec)))
>      (unless (org-before-first-heading-p)
>        (org-with-limited-levels
>         (cl-case detail
> @@ -7661,7 +8085,7 @@ When REMOVE is non-nil, remove the subtree from the clipboard."
>       (skip-chars-forward " \t\n\r")
>       (setq beg (point))
>       (when (and (org-invisible-p) visp)
> -       (save-excursion (outline-show-heading)))
> +       (save-excursion (org-show-heading)))
>       ;; Shift if necessary.
>       (unless (= shift 0)
>         (save-restriction
> @@ -8103,7 +8527,7 @@ function is being called interactively."
>  		       (point))
>  	    what "children")
>        (goto-char start)
> -      (outline-show-subtree)
> +      (org-show-subtree)
>        (outline-next-heading))
>       (t
>        ;; we will sort the top-level entries in this file
> @@ -13150,7 +13574,7 @@ drawer is immediately hidden."
>  	   (inhibit-read-only t))
>         (unless (bobp) (insert "\n"))
>         (insert ":PROPERTIES:\n:END:")
> -       (org-flag-region (line-end-position 0) (point) t 'outline)
> +       (org-flag-region (line-end-position 0) (point) t 'org-hide-drawer)
>         (when (or (eobp) (= begin (point-min))) (insert "\n"))
>         (org-indent-region begin (point))))))
>  
> @@ -17612,11 +18036,11 @@ Move point to the beginning of first heading or end of buffer."
>  (defun org-show-branches-buffer ()
>    "Show all branches in the buffer."
>    (org-flag-above-first-heading)
> -  (outline-hide-sublevels 1)
> +  (org-hide-sublevels 1)
>    (unless (eobp)
> -    (outline-show-branches)
> +    (org-show-branches)
>      (while (outline-get-next-sibling)
> -      (outline-show-branches)))
> +      (org-show-branches)))
>    (goto-char (point-min)))
>  
>  (defun org-kill-note-or-show-branches ()
> @@ -17630,8 +18054,8 @@ Move point to the beginning of first heading or end of buffer."
>  	(t
>  	 (let ((beg (progn (org-back-to-heading) (point)))
>  	       (end (save-excursion (org-end-of-subtree t t) (point))))
> -	   (outline-hide-subtree)
> -	   (outline-show-branches)
> +	   (org-hide-subtree)
> +	   (org-show-branches)
>  	   (org-hide-archived-subtrees beg end)))))
>  
>  (defun org-delete-indentation (&optional arg)
> @@ -17787,9 +18211,9 @@ Otherwise, call `org-show-children'.  ARG is the level to hide."
>      (if (org-before-first-heading-p)
>          (progn
>            (org-flag-above-first-heading)
> -          (outline-hide-sublevels (or arg 1))
> +          (org-hide-sublevels (or arg 1))
>            (goto-char (point-min)))
> -      (outline-hide-subtree)
> +      (org-hide-subtree)
>        (org-show-children arg))))
>  
>  (defun org-ctrl-c-star ()
> @@ -20933,6 +21357,80 @@ Started from `gnus-info-find-node'."
>            (t default-org-info-node))))))
>  
>  \f
> +
> +;;; Make isearch search in some text hidden via text propertoes
> +
> +(defvar org--isearch-overlays nil
> +  "List of overlays temporarily created during isearch.
> +This is used to allow searching in regions hidden via text properties.
> +As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays.
> +Any text hidden via text properties is not revealed even if `search-invisible'
> +is set to 't.")
> +
> +;; Not sure if it needs to be a user option
> +;; One might want to reveal hidden text in, for example, hidden parts of the links.
> +;; Currently, hidden text in links is never revealed by isearch.
> +(defvar org-isearch-specs '(org-hide-block
> +			 org-hide-drawer)
> +  "List of text invisibility specs to be searched by isearch.
> +By default ([2020-05-09 Sat]), isearch does not search in hidden text,
> +which was made invisible using text properties. Isearch will be forced
> +to search in hidden text with any of the listed 'invisible property value.")
> +
> +(defun org--create-isearch-overlays (beg end)
> +  "Replace text property invisibility spec by overlays between BEG and END.
> +All the regions with invisibility text property spec from
> +`org-isearch-specs' will be changed to use overlays instead
> +of text properties. The created overlays will be stored in
> +`org--isearch-overlays'."
> +  (let ((pos beg))
> +    (while (< pos end)
> +      (when-let* ((spec (get-text-property pos 'invisible))
> +		  (spec (memq spec org-isearch-specs))
> +		  (region (org--find-text-property-region pos 'invisible)))
> +        (setq spec (get-text-property pos 'invisible))
> +        ;; Changing text properties is considered buffer modification.
> +        ;; We do not want it here.
> +	(with-silent-modifications
> +          ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat]
> +          ;; overlay for 'outline blocks.
> +          (let ((o (make-overlay (car region) (cdr region) nil 'front-advance)))
> +	    (overlay-put o 'evaporate t)
> +	    (overlay-put o 'invisible spec)
> +            ;; `delete-overlay' here means that spec information will be lost
> +            ;; for the region. The region will remain visible.
> +	    (overlay-put o 'isearch-open-invisible #'delete-overlay)
> +            (push o org--isearch-overlays))
> +	  (org-flag-region (car region) (cdr region) nil spec)))
> +      (setq pos (next-single-property-change pos 'invisible nil end)))))
> +
> +(defun org--isearch-filter-predicate (beg end)
> +  "Return non-nil if text between BEG and END is deemed visible by Isearch.
> +This function is intended to be used as `isearch-filter-predicate'.
> +Unlike `isearch-filter-visible', make text with 'invisible text property
> +value listed in `org-isearch-specs' visible to Isearch."
> +  (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text
> +  (isearch-filter-visible beg end))
> +
> +(defun org--clear-isearch-overlay (ov)
> +  "Convert OV region back into using text properties."
> +  (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays
> +    ;; Changing text properties is considered buffer modification.
> +    ;; We do not want it here.
> +    (with-silent-modifications
> +      (org-flag-region (overlay-start ov) (overlay-end ov) t spec)))
> +  (when (member ov isearch-opened-overlays)
> +    (setq isearch-opened-overlays (delete ov isearch-opened-overlays)))
> +  (delete-overlay ov))
> +
> +(defun org--clear-isearch-overlays ()
> +  "Convert overlays from `org--isearch-overlays' back into using text properties."
> +  (when org--isearch-overlays
> +    (mapc #'org--clear-isearch-overlay org--isearch-overlays)
> +    (setq org--isearch-overlays nil)))
> +
> +\f
> +
>  ;;; Finish up
>  
>  (add-hook 'org-mode-hook     ;remove overlays when changing major mode
>
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Hello,
>>
>> [The patch itself will be provided in the following email]
>>
>> I have five updates from the previous version of the patch:
>>
>> 1. I implemented a simplified version of element parsing to detect
>> changes in folded drawers or blocks. No computationally expensive calls
>> of org-element-at-point or org-element-parse-buffer are needed now.
>>
>> 2. The patch is now compatible with master (commit 2e96dc639). I
>> reverted the earlier change in folding drawers and blocks. Now, they are
>> back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would
>> achieve nothing when we use text properties.
>>
>> 3. 'invisible text property can now be nested. This is important, for
>> example, when text inside drawers contains fontified links (which also
>> use 'invisible text property to hide parts of the link). Now, the old
>> 'invisible spec is recovered after unfolding.
>>
>> 4. Some outline-* function calls in org referred to outline-flag-region
>> implementation, which is not in sync with org-flag-region in this patch.
>> I have implemented their org-* versions and replaced the calls
>> throughout .el files. Actually, some org-* versions were already
>> implemented in org, but not used for some reason (or not mentioned in
>> the manual). I have updated the relevant sections of manual. These
>> changes might be relevant to org independently of this feature branch.
>>
>> 5. I have managed to get a working version of outline folding via text
>> properties. However, that approach has a big downside - folding state
>> cannot be different in indirect buffer when we use text properties. I
>> have seen packages relying on this feature of org and I do not see any
>> obvious way to achieve different folding state in indirect buffer while
>> using text properties for outline folding.
>>
>> -----------------------------------------------------------------------
>> -----------------------------------------------------------------------
>>
>> More details on the new implementation for tracking changes:
>>
>>> Of course we can. It is only necessary to focus on changes that would
>>> break the structure of the element. This does not entail a full parsing.
>>
>> I have limited parsing to matching beginning and end of a drawer/block.
>> The basic functions are org--get-element-region-at-point,
>> org--get-next-element-region-at-point, and org--find-elements-in-region.
>> They are simplified versions of org-element-* parsers and do not require
>> parsing everything from the beginning of the section.
>>
>> For now, I keep everything in org.el, but those simplified parsers
>> probably belong to org-element.el.
>>
>>> If we can stick with `after-change-functions' (or local equivalent),
>>> that's better. It is more predictable than `before-change-functions' and
>>> alike.
>>
>> For now, I still used before/after-change-functions combination.
>> I see the following problems with using only after-change-functions: 
>>
>> 1. They are not guaranteed to be called after every single change:
>>
>> From (elisp) Change Hooks:
>> "... some complex primitives call ‘before-change-functions’ once before
>> making changes, and then call ‘after-change-functions’ zero or more
>> times"
>>
>> The consequence of it is a possibility that region passed to the
>> after-change-functions is quite big (including all the singular changes,
>> even if they are distant). This region may contain changed drawers as
>> well and unchanged drawers and needs to be parsed to determine which
>> drawers need to be re-folded.
>>
>>> And, more importantly, they are not meant to be used together, i.e., you
>>> cannot assume that a single call to `before-change-functions' always
>>> happens before calling `after-change-functions'. This can be tricky if
>>> you want to use the former to pass information to the latter.
>>
>> The fact that before-change-functions can be called multiple times
>> before after-change-functions, is trivially solved by using buffer-local
>> changes register (see org--modified-elements). The register is populated
>> by before-change-functions and cleared by after-change-functions.
>>
>>> Well, `before-change-fuctions' and `after-change-functions' are not
>>> clean at all: you modify an unrelated part of the buffer, but still call
>>> those to check if a drawer needs to be unfolded somewhere.
>>
>> 2. As you pointed, instead of global before-change-functions, we can use
>> modification-hooks text property on sensitive parts of the
>> drawers/blocks. This would work, but I am concerned about one annoying
>> special case:
>>
>> -------------------------------------------------------------------------
>> :BLAH: <inserted outside any of the existing drawers>
>>
>> <some text>
>>
>> :DRAWER: <folded>
>> Donec at pede.
>> :END:
>> -------------------------------------------------------------------------
>> In this example, the user would not be able to unfold the folder DRAWER
>> because it will technically become a part of a new giant BLAH drawer.
>> This may be especially annoying if <some text> is more than one screen
>> long and there is no easy way to identify why unfolding does not work
>> (with point at :DRAWER:).
>>
>> Because of this scenario, limiting before-change-functions to folded
>> drawers is not sufficient. Any change in text may need to trigger
>> unfolding.
>>
>> In the patch, I always register possible modifications in the
>> blocks/drawers intersecting with the modified region + a drawer/block
>> right next to the region.
>>
>> -----------------------------------------------------------------------
>> -----------------------------------------------------------------------
>>
>> More details on the nested 'invisible text property implementation.
>>
>> The idea is to keep 'invisible property stack push and popping from it
>> as we add/remove 'invisible text property. All the work is done in
>> org-flag-region.
>>
>> This was originally intended for folding outlines via text properties.
>> Since using text properties for folding outlines is not a good idea,
>> nested text properties have much less use. As I mentioned, they do
>> preserve link fontification, but I am not sure if it worth it for the
>> overhead to org-flag-region. Keeping this here mostly in the case if
>> someone has any ideas how it can be useful.
>>
>> -----------------------------------------------------------------------
>> -----------------------------------------------------------------------
>>
>> More details on replaced outline-* -> org-* function calls.
>>
>> I have implemented org-* versions of the following functions:
>>
>> - outline-hide-entry
>> - outline-hide-subtree
>> - outline-hide-sublevels
>> - outline-show-heading
>> - outline-show-branches
>>
>> The org-* versions trivially use org-flag-region instead of
>> outline-flag-region.
>>
>> Replaced outline-* calls where org- versions were already available:
>>
>> - outline-show-entry
>> - outline-show-all
>> - outline-show-subtree
>>
>> I reflected the new (including already available) functions in the
>> manual and removed some defalias from org-compat.el where they are not
>> needed. 
>>
>> -----------------------------------------------------------------------
>> -----------------------------------------------------------------------
>>
>> Further work:
>>
>> 1. after-change-functions use org-hide-drawer/block-toggle to
>> fold/unfold after modification. However, I just found that they call
>> org-element-at-point, which slows down modifications in folded
>> drawers/blocks. For example, realigning a long table inside folded
>> drawer takes >1sec, while it is instant in the unfolded drawer.
>>
>> 2. org-toggle-custom-properties is terribly slow on large org documents,
>> similarly to folded drawers on master. It should be relatively easy to
>> use text properties there instead of overlays.
>>
>> 3. Multiple calls to before/after-change-functions is still a problem. I
>> am looking into following ways to reduce this number:
>>  - reduce the number of elements registered as potentially modified
>>    + do not add duplicates to org--modified-elements
>>    + do not add unfolded elements to org--modified-elements
>>    + register after-change-function as post-command hook and remove it
>>      from global after-change-functions. This way, it will be called
>>      twice per command only.
>>  - determine common region containing org--modified-elements. if change
>>    is happening within that region, there is no need to parse
>>    drawers/blocks there again.
>>
>> P.S.
>>
>>>> It was mostly an annoyance, because they returned different results on
>>>> the same element. Specifically, they returned different :post-blank and
>>>> :end properties, which does not sound right.
>>>
>>> OK. If you have a reproducible recipe, I can look into it and see what
>>> can be done.
>>
>> Recipe to have different (org-element-at-point) and
>> (org-element-parse-buffer 'element)
>> -------------------------------------------------------------------------
>> <point-min>
>> :PROPERTIES:
>> :CREATED:  [2020-05-23 Sat 02:32]
>> :END:
>>
>>
>> <point-max>
>> -------------------------------------------------------------------------
>>
>>
>> Best,
>> Ihor
>>
>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>>
>>> Hello,
>>>
>>> Ihor Radchenko <yantar92@gmail.com> writes:
>>>
>>>>> As you noticed, using Org Element is a no-go, unfortunately. Parsing an
>>>>> element is a O(N) operation by the number of elements before it in
>>>>> a section. In particular, it is not bounded, and not mitigated by
>>>>> a cache. For large documents, it is going to be unbearably slow, too.
>>>>
>>>> Ouch. I thought it is faster.
>>>> What do you mean by "not mitigated by a cache"?
>>>
>>> Parsing starts from the closest headline, every time. So, if Org parses
>>> the Nth element in the entry two times, it really parses 2N elements.
>>>
>>> With a cache, assuming the buffer wasn't modified, Org would parse
>>> N elements only. With a smarter cache, with fine grained cache
>>> invalidation, it could also reduce the number of subsequent parsed
>>> elements.
>>>
>>>> The reason I would like to utilise org-element parser to make tracking
>>>> modifications more robust. Using details of the syntax would make the
>>>> code fragile if any modifications are made to syntax in future.
>>>
>>> I don't think the code would be more fragile. Also, the syntax we're
>>> talking about is not going to be modified anytime soon. Moreover, if
>>> folding breaks, it is usually visible, so the bug will not be unnoticed.
>>>
>>> This code is going to be as low-level as it can be.
>>>
>>>> Debugging bugs in modification functions is not easy, according to my
>>>> experience.
>>>
>>> No, it's not. 
>>>
>>> But this is not really related to whether you use Element or not.
>>>
>>>> One possible way to avoid performance issues during modification is
>>>> running parser in advance. For example, folding an element may
>>>> as well add information about the element to its text properties.
>>>> This will not degrade performance of folding since we are already
>>>> parsing the element during folding (at least, in
>>>> org-hide-drawer-toggle).
>>>
>>> We can use this information stored at fold time. But I'm not even sure
>>> we need it.
>>>
>>>> The problem with parsing an element during folding is that we cannot
>>>> easily detect changes like below without re-parsing.
>>>
>>> Of course we can. It is only necessary to focus on changes that would
>>> break the structure of the element. This does not entail a full parsing.
>>>
>>>> :PROPERTIES: <folded>
>>>> :CREATED: [2020-05-18 Mon]
>>>> :END: <- added line
>>>> :ID: test
>>>> :END:
>>>>
>>>> or even
>>>>
>>>> :PROPERTIES:
>>>> :CREATED: [2020-05-18 Mon]
>>>> :ID: test
>>>> :END: <- delete this line
>>>>
>>>> :DRAWER: <folded, cannot be unfolded if we don't re-parse after deletion>
>>>> test
>>>> :END:
>>>
>>> Please have a look at the "sensitive parts" I wrote about. This takes
>>> care of this kind of breakage.
>>>
>>>> The re-parsing can be done via regexp, as you suggested, but I don't
>>>> like this idea, because it will end up re-implementing
>>>> org-element-*-parser.
>>>
>>> You may have misunderstood my suggestion. See below.
>>>
>>>> Would it be acceptable to run org-element-*-parser
>>>> in after-change-functions?
>>>
>>> I'd rather not do that. This is unnecessary consing, and matching, etc.
>>>
>>>> If I understand correctly, it is not as easy.
>>>> Consider the following example:
>>>>
>>>> :PROPERTIES:
>>>> :CREATED: [2020-05-18 Mon]
>>>> <region-beginning>
>>>> :ID: example
>>>> :END:
>>>>
>>>> <... a lot of text, maybe containing other drawers ...>
>>>>
>>>> Nullam rutrum.
>>>> Pellentesque dapibus suscipit ligula.
>>>> <region-end>
>>>> Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus.
>>>>
>>>> If the region gets deleted, the modification hooks from chars inside
>>>> drawer will be called as (hook-function <region-beginning>
>>>> <region-end>). So, there is still a need to find the drawer somehow to
>>>> mark it as about to be modified (modification hooks are ran before
>>>> actual modification).
>>>
>>> If we can stick with `after-change-functions' (or local equivalent),
>>> that's better. It is more predictable than `before-change-functions' and
>>> alike.
>>>
>>> If it is a deletion, here is the kind of checks we could do, depending
>>> on when they are performed.
>>>
>>> Before actual changes :
>>>
>>>   1. The deletion is happening within a folded drawer (unnecessary step
>>>      in local functions).
>>>   2. The change deleted the sensitive line ":END:".
>>>   3. Conclusion : unfold.
>>>
>>> Or, after actual changes :
>>>
>>>   1. The deletion involves a drawer.
>>>   2. Text properties indicate that the beginning of the propertized part
>>>      of the buffer start with org-drawer-regexp, but doesn't end with
>>>      `org-property-end-re'. A "sensitive part" disappeared!
>>>   3. Conclusion : unfold
>>>
>>> This is far away from parsing. IMO, a few checks cover all cases. Let me
>>> know if you have questions about it.
>>>
>>> Also, note that the kind of change you describe will happen perhaps
>>> 0.01% of the time. Most change are about one character, or a single
>>> line, long.
>>>
>>>> The only difference between using modification hooks and
>>>> before-change-functions is that modification hooks will trigger less
>>>> frequently. 
>>>
>>> Exactly. Much less frequently. But extra care is required, as you noted
>>> already.
>>>
>>>> Considering the performance of org-element-at-point, it is
>>>> probably worth doing. Initially, I wanted to avoid it because setting a
>>>> single before-change-functions hook sounded cleaner than setting
>>>> modification-hooks, insert-behind-hooks, and insert-in-front-hooks.
>>>
>>> Well, `before-change-fuctions' and `after-change-functions' are not
>>> clean at all: you modify an unrelated part of the buffer, but still call
>>> those to check if a drawer needs to be unfolded somewhere.
>>>
>>> And, more importantly, they are not meant to be used together, i.e., you
>>> cannot assume that a single call to `before-change-functions' always
>>> happens before calling `after-change-functions'. This can be tricky if
>>> you want to use the former to pass information to the latter.
>>>
>>> But I understand that they are easier to use than their local
>>> counterparts. If you stick with (before|after)-change-functions, the
>>> function being called needs to drop the ball very quickly if the
>>> modification is not about folding changes. Also, I very much suggest to
>>> stick to only `after-change-functions', if feasible (I think it is), per
>>> above.
>>>
>>>> Moreover, these text properties would be copied by default if one uses 
>>>> buffer-substring. Then, the hooks will also trigger later in the yanked
>>>> text, which may cause all kinds of bugs.
>>>
>>> Indeed, that would be something to handle specifically. I.e.,
>>> destructive modifications (i.e., those that unfold) could clear such
>>> properties.
>>>
>>> It is more work. I don't know if it is worth the trouble if we can get
>>> out quickly of `after-change-functions' for unrelated changes.
>>>
>>>> It was mostly an annoyance, because they returned different results on
>>>> the same element. Specifically, they returned different :post-blank and
>>>> :end properties, which does not sound right.
>>>
>>> OK. If you have a reproducible recipe, I can look into it and see what
>>> can be done.
>>>
>>> Regards,
>>>
>>> -- 
>>> Nicolas Goaziou
>>
>> -- 
>> Ihor Radchenko,
>> PhD,
>> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
>> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
>> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg
>
> -- 
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-23 13:52                               ` Ihor Radchenko
  2020-05-23 13:53                                 ` Ihor Radchenko
@ 2020-05-26  8:33                                 ` Nicolas Goaziou
  2020-06-02  9:21                                   ` Ihor Radchenko
  1 sibling, 1 reply; 55+ messages in thread
From: Nicolas Goaziou @ 2020-05-26  8:33 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> I have five updates from the previous version of the patch:

Thank you.

> 1. I implemented a simplified version of element parsing to detect
> changes in folded drawers or blocks. No computationally expensive calls
> of org-element-at-point or org-element-parse-buffer are needed now.
>
> 2. The patch is now compatible with master (commit 2e96dc639). I
> reverted the earlier change in folding drawers and blocks. Now, they are
> back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would
> achieve nothing when we use text properties.
>
> 3. 'invisible text property can now be nested. This is important, for
> example, when text inside drawers contains fontified links (which also
> use 'invisible text property to hide parts of the link). Now, the old
> 'invisible spec is recovered after unfolding.

Interesting. I'm running out of time, so I cannot properly inspect the
code right now. I'll try to do that before the end of the week.

> 4. Some outline-* function calls in org referred to outline-flag-region
> implementation, which is not in sync with org-flag-region in this patch.
> I have implemented their org-* versions and replaced the calls
> throughout .el files. Actually, some org-* versions were already
> implemented in org, but not used for some reason (or not mentioned in
> the manual). I have updated the relevant sections of manual. These
> changes might be relevant to org independently of this feature branch.

Yes, we certainly want to move to org-specific versions in all cases.

> 5. I have managed to get a working version of outline folding via text
> properties. However, that approach has a big downside - folding state
> cannot be different in indirect buffer when we use text properties. I
> have seen packages relying on this feature of org and I do not see any
> obvious way to achieve different folding state in indirect buffer while
> using text properties for outline folding.

Hmm. Good point. This is a serious issue to consider. Even if we don't
use text properties for outline, this also affects drawers and blocks.

> For now, I still used before/after-change-functions combination.

You shouldn't.

> I see the following problems with using only after-change-functions: 
>
> 1. They are not guaranteed to be called after every single change:

Of course they are! See below.

> From (elisp) Change Hooks:
> "... some complex primitives call ‘before-change-functions’ once before
> making changes, and then call ‘after-change-functions’ zero or more
> times"

"zero" means there are no changes at all, so, `after-change-functions'
are not called, which is expected.

> The consequence of it is a possibility that region passed to the
> after-change-functions is quite big (including all the singular changes,
> even if they are distant). This region may contain changed drawers as
> well and unchanged drawers and needs to be parsed to determine which
> drawers need to be re-folded.

It seems you're getting it backwards. `before-change-functions' are the
functions being called with a possibly wide, imprecise, region to
handle:

    When that happens, the arguments to ‘before-change-functions’ will
    enclose a region in which the individual changes are made, but won’t
    necessarily be the minimal such region

however, after-change-functions calls are always minimal:

    and the arguments to each successive call of
    ‘after-change-functions’ will then delimit the part of text being
    changed exactly.

If you stick to `after-change-functions', there will be no such thing as
you describe.

>> And, more importantly, they are not meant to be used together, i.e., you
>> cannot assume that a single call to `before-change-functions' always
>> happens before calling `after-change-functions'. This can be tricky if
>> you want to use the former to pass information to the latter.
>
> The fact that before-change-functions can be called multiple times
> before after-change-functions, is trivially solved by using buffer-local
> changes register (see org--modified-elements).

Famous last words. Been there, done that, and it failed.

Let me quote the manual:

    In general, we advise to use either before- or the after-change
    hooks, but not both.

So, let me insist: don't do that. If you don't agree with me, let's at
least agree with Emacs developers.

> The register is populated by before-change-functions and cleared by
> after-change-functions.

You cannot expect `after-change-functions' to clear what
`before-change-functions' did. This is likely to introduce pernicious
bugs. Sorry if it sounds like FUD, but bugs in those areas are just
horrible to squash.

>> Well, `before-change-fuctions' and `after-change-functions' are not
>> clean at all: you modify an unrelated part of the buffer, but still call
>> those to check if a drawer needs to be unfolded somewhere.
>
> 2. As you pointed, instead of global before-change-functions, we can use
> modification-hooks text property on sensitive parts of the
> drawers/blocks. This would work, but I am concerned about one annoying
> special case:
>
> -------------------------------------------------------------------------
> :BLAH: <inserted outside any of the existing drawers>
>
> <some text>
>
> :DRAWER: <folded>
> Donec at pede.
> :END:
> -------------------------------------------------------------------------
> In this example, the user would not be able to unfold the folder DRAWER
> because it will technically become a part of a new giant BLAH drawer.
> This may be especially annoying if <some text> is more than one screen
> long and there is no easy way to identify why unfolding does not work
> (with point at :DRAWER:).

You shouldn't be bothered by the case you're describing here, for
multiple reasons.

First, this issue already arises in the current implementation. No one
bothered so far: this change is very unlikely to happen. If it becomes
an issue, we could make sure that `org-reveal' handles this.

But, more importantly, we actually /want it/ as a feature. Indeed, if
DRAWER is expanded every time ":BLAH:" is inserted above, then inserting
a drawer manually would unfold /all/ drawers in the section. The user is
more likely to write first ":BLAH:" (everything is unfolded) then
":END:" than ":END:", then ":BLAH:".

> Because of this scenario, limiting before-change-functions to folded
> drawers is not sufficient. Any change in text may need to trigger
> unfolding.

after-change-functions is more appropriate than before-change-functions,
and local parsing, as explained in this thread, is more efficient than
re-inventing the parser.

> In the patch, I always register possible modifications in the
> blocks/drawers intersecting with the modified region + a drawer/block
> right next to the region.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the nested 'invisible text property implementation.
>
> The idea is to keep 'invisible property stack push and popping from it
> as we add/remove 'invisible text property. All the work is done in
> org-flag-region.

This sounds like a good idea.

> This was originally intended for folding outlines via text properties.
> Since using text properties for folding outlines is not a good idea,
> nested text properties have much less use.

AFAIU, they have. You mention link fontification, but there are other
pieces that we could switch to text properties instead of overlays,
e.g., Babel hashes, narrowed table columns…

> 3. Multiple calls to before/after-change-functions is still a problem. I
> am looking into following ways to reduce this number:
>  - reduce the number of elements registered as potentially modified
>    + do not add duplicates to org--modified-elements
>    + do not add unfolded elements to org--modified-elements
>    + register after-change-function as post-command hook and remove it
>      from global after-change-functions. This way, it will be called
>      twice per command only.
>  - determine common region containing org--modified-elements. if change
>    is happening within that region, there is no need to parse
>    drawers/blocks there again.

This is over-engineering. Again, please focus on local changes, as
discussed before.

> Recipe to have different (org-element-at-point) and
> (org-element-parse-buffer 'element)
> -------------------------------------------------------------------------
> <point-min>
> :PROPERTIES:
> :CREATED:  [2020-05-23 Sat 02:32]
> :END:
>
>
> <point-max>
> -------------------------------------------------------------------------

I didn't look at this situation in particular, but there are cases where
different :post-blank values are inevitable, for example at the end of
a section.

Regards,

-- 
Nicolas Goaziou


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-05-26  8:33                                 ` Nicolas Goaziou
@ 2020-06-02  9:21                                   ` Ihor Radchenko
  2020-06-02  9:23                                     ` Ihor Radchenko
                                                       ` (2 more replies)
  0 siblings, 3 replies; 55+ messages in thread
From: Ihor Radchenko @ 2020-06-02  9:21 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

Hello,

[The patch itself will be provided in the following email]

I have three updates from the previous version of the patch:

1. I managed to implement buffer-local text properties.
   Now, outline folding also uses text properties without a need to give
   up independent folding in indirect buffers.

2. The code handling modifications in folded drawers/blocks was
   rewritten. The new code uses after-change-functions to re-hide text
   inserted in the middle of folded regions; and text properties to
   unfold folded drawers/blocks if one changes BEGIN/END line.

3. [experimental] Started working on improving memory and cpu footprint
   of the old code related to folding/unfolding. org-hide-drawer-all now
   works significantly faster because I can utilise simplified drawer
   parser, which require a lot less memory. Overall, I managed to reduce
   Emacs memory footprint after loading all my agenda_files twice. The
   loading is also noticeably faster.

-----------------------------------------------------------------------
-----------------------------------------------------------------------

More details on the buffer-local text properties:

I have found char-property-alias-alist variable that controls how Emacs
calculates text property value if the property is not set. This variable
can be buffer-local, which allows independent 'invisible states in
different buffers.

All the implementation stays in
org--get-buffer-local-text-property-symbol, which takes care about
generating unique property name and mapping it to 'invisible (or any
other) text property.

-----------------------------------------------------------------------
-----------------------------------------------------------------------

More details on the new implementation for tracking changes:

I simplified the code as suggested, without using pairs of before- and
after-change-functions.

Handling text inserted into folded/invisible region is handled by a
simple after-change function. After testing, it turned out that simple
re-hiding text based on 'invisible property of the text before/after the
inserted region works pretty well.

Modifications to BEGIN/END line of the drawers and blocks is handled via
'modification-hooks + 'insert-behind-hooks text properties (there is no
after-change-functions analogue for text properties in Emacs). The
property is applied during folding and the modification-hook function is
made aware about the drawer/block boundaries (via apply-partially
passing element containing :begin :end markers for the current
drawer/block). Passing the element boundary is important because the
'modification-hook will not directly know where it belongs to. Only the
modified region (which can be larger than the drawer) is passed to the
function. In the worst case, the region can be the whole buffer (if one
runs revert-buffer).

It turned out that adding 'modification-hook text property takes a
significant cpu time (partially, because we need to take care about
possible existing 'modification-hook value, see
org--add-to-list-text-property). For now, I decided to not clear the
modification hooks during unfolding because of poor performance.
However, this approach would lead to partial unfolding in the following
case:

:asd:
:drawer:
lksjdfksdfjl
sdfsdfsdf
:end:

If :asd: was inserted in front of folded :drawer:, changes in :drawer:
line of the new folded :asd: drawer would reveal the text between
:drawer: and :end:.

Let me know what you think on this.

> You shouldn't be bothered by the case you're describing here, for
> multiple reasons.
> 
> First, this issue already arises in the current implementation. No one
> bothered so far: this change is very unlikely to happen. If it becomes
> an issue, we could make sure that `org-reveal' handles this.
> 
> But, more importantly, we actually /want it/ as a feature. Indeed, if
> DRAWER is expanded every time ":BLAH:" is inserted above, then inserting
> a drawer manually would unfold /all/ drawers in the section. The user is
> more likely to write first ":BLAH:" (everything is unfolded) then
> ":END:" than ":END:", then ":BLAH:".

Agree. This allowed me to simplify the code significantly.

> It seems you're getting it backwards. `before-change-functions' are the
> functions being called with a possibly wide, imprecise, region to
> handle:
> 
>     When that happens, the arguments to ‘before-change-functions’ will
>     enclose a region in which the individual changes are made, but won’t
>     necessarily be the minimal such region
> 
> however, after-change-functions calls are always minimal:
> 
>     and the arguments to each successive call of
>     ‘after-change-functions’ will then delimit the part of text being
>     changed exactly.
> 
> If you stick to `after-change-functions', there will be no such thing as
> you describe.

You are right here, I missed that before-change-functions are likely to
be called on large regions. I thought that the regions are same for
before/after-change-functions, but after-change-functions could be
called more than 1 time. After second thought, your vision that it is
mostly 0 or 1 times should be the majority of cases in practice.

-----------------------------------------------------------------------
-----------------------------------------------------------------------

More details on reducing cpu and memory footprint of org buffers:

My simplified implementation of element boundary parser
(org--get-element-region-at-point) appears to be much faster and also
uses much less memory in comparison with org-element-at-point.
Moreover, not all the places where org-element-at-point is called
actually need the full parsed element. For example, org-hide-drawer-all,
org-hide-drawer-toggle, org-hide-block-toggle, and
org--hide-wrapper-toggle only need element type and some information
about the element boundaries - the information we can get from
org--get-element-region-at-point.

The following version of org-hide-drawer-all seems to work much faster
in comparison with original:

(defun org-hide-drawer-all ()
  "Fold all drawers in the current buffer."
  (save-excursion
    (goto-char (point-min))
    (while (re-search-forward org-drawer-regexp nil t)
      (when-let* ((drawer (org--get-element-region-at-point '(property-drawer drawer)))
		  (type (org-element-type drawer)))
	(org-hide-drawer-toggle t nil drawer)
	;; Make sure to skip drawer entirely or we might flag it
	;; another time when matching its ending line with
	;; `org-drawer-regexp'.
	(goto-char (org-element-property :end drawer))))))

What do you think about the idea of making use of
org--get-element-region-at-point in org code base?

-----------------------------------------------------------------------
-----------------------------------------------------------------------

Further work:

1. Look into other code using overlays. Specifically,
org-toggle-custom-properties, Babel hashes, and narrowed table columns.

Best,
Ihor

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> I have five updates from the previous version of the patch:
>
> Thank you.
>
>> 1. I implemented a simplified version of element parsing to detect
>> changes in folded drawers or blocks. No computationally expensive calls
>> of org-element-at-point or org-element-parse-buffer are needed now.
>>
>> 2. The patch is now compatible with master (commit 2e96dc639). I
>> reverted the earlier change in folding drawers and blocks. Now, they are
>> back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would
>> achieve nothing when we use text properties.
>>
>> 3. 'invisible text property can now be nested. This is important, for
>> example, when text inside drawers contains fontified links (which also
>> use 'invisible text property to hide parts of the link). Now, the old
>> 'invisible spec is recovered after unfolding.
>
> Interesting. I'm running out of time, so I cannot properly inspect the
> code right now. I'll try to do that before the end of the week.
>
>> 4. Some outline-* function calls in org referred to outline-flag-region
>> implementation, which is not in sync with org-flag-region in this patch.
>> I have implemented their org-* versions and replaced the calls
>> throughout .el files. Actually, some org-* versions were already
>> implemented in org, but not used for some reason (or not mentioned in
>> the manual). I have updated the relevant sections of manual. These
>> changes might be relevant to org independently of this feature branch.
>
> Yes, we certainly want to move to org-specific versions in all cases.
>
>> 5. I have managed to get a working version of outline folding via text
>> properties. However, that approach has a big downside - folding state
>> cannot be different in indirect buffer when we use text properties. I
>> have seen packages relying on this feature of org and I do not see any
>> obvious way to achieve different folding state in indirect buffer while
>> using text properties for outline folding.
>
> Hmm. Good point. This is a serious issue to consider. Even if we don't
> use text properties for outline, this also affects drawers and blocks.
>
>> For now, I still used before/after-change-functions combination.
>
> You shouldn't.
>
>> I see the following problems with using only after-change-functions: 
>>
>> 1. They are not guaranteed to be called after every single change:
>
> Of course they are! See below.
>
>> From (elisp) Change Hooks:
>> "... some complex primitives call ‘before-change-functions’ once before
>> making changes, and then call ‘after-change-functions’ zero or more
>> times"
>
> "zero" means there are no changes at all, so, `after-change-functions'
> are not called, which is expected.
>
>> The consequence of it is a possibility that region passed to the
>> after-change-functions is quite big (including all the singular changes,
>> even if they are distant). This region may contain changed drawers as
>> well and unchanged drawers and needs to be parsed to determine which
>> drawers need to be re-folded.
>
> It seems you're getting it backwards. `before-change-functions' are the
> functions being called with a possibly wide, imprecise, region to
> handle:
>
>     When that happens, the arguments to ‘before-change-functions’ will
>     enclose a region in which the individual changes are made, but won’t
>     necessarily be the minimal such region
>
> however, after-change-functions calls are always minimal:
>
>     and the arguments to each successive call of
>     ‘after-change-functions’ will then delimit the part of text being
>     changed exactly.
>
> If you stick to `after-change-functions', there will be no such thing as
> you describe.
>
>>> And, more importantly, they are not meant to be used together, i.e., you
>>> cannot assume that a single call to `before-change-functions' always
>>> happens before calling `after-change-functions'. This can be tricky if
>>> you want to use the former to pass information to the latter.
>>
>> The fact that before-change-functions can be called multiple times
>> before after-change-functions, is trivially solved by using buffer-local
>> changes register (see org--modified-elements).
>
> Famous last words. Been there, done that, and it failed.
>
> Let me quote the manual:
>
>     In general, we advise to use either before- or the after-change
>     hooks, but not both.
>
> So, let me insist: don't do that. If you don't agree with me, let's at
> least agree with Emacs developers.
>
>> The register is populated by before-change-functions and cleared by
>> after-change-functions.
>
> You cannot expect `after-change-functions' to clear what
> `before-change-functions' did. This is likely to introduce pernicious
> bugs. Sorry if it sounds like FUD, but bugs in those areas are just
> horrible to squash.
>
>>> Well, `before-change-fuctions' and `after-change-functions' are not
>>> clean at all: you modify an unrelated part of the buffer, but still call
>>> those to check if a drawer needs to be unfolded somewhere.
>>
>> 2. As you pointed, instead of global before-change-functions, we can use
>> modification-hooks text property on sensitive parts of the
>> drawers/blocks. This would work, but I am concerned about one annoying
>> special case:
>>
>> -------------------------------------------------------------------------
>> :BLAH: <inserted outside any of the existing drawers>
>>
>> <some text>
>>
>> :DRAWER: <folded>
>> Donec at pede.
>> :END:
>> -------------------------------------------------------------------------
>> In this example, the user would not be able to unfold the folder DRAWER
>> because it will technically become a part of a new giant BLAH drawer.
>> This may be especially annoying if <some text> is more than one screen
>> long and there is no easy way to identify why unfolding does not work
>> (with point at :DRAWER:).
>
> You shouldn't be bothered by the case you're describing here, for
> multiple reasons.
>
> First, this issue already arises in the current implementation. No one
> bothered so far: this change is very unlikely to happen. If it becomes
> an issue, we could make sure that `org-reveal' handles this.
>
> But, more importantly, we actually /want it/ as a feature. Indeed, if
> DRAWER is expanded every time ":BLAH:" is inserted above, then inserting
> a drawer manually would unfold /all/ drawers in the section. The user is
> more likely to write first ":BLAH:" (everything is unfolded) then
> ":END:" than ":END:", then ":BLAH:".
>
>> Because of this scenario, limiting before-change-functions to folded
>> drawers is not sufficient. Any change in text may need to trigger
>> unfolding.
>
> after-change-functions is more appropriate than before-change-functions,
> and local parsing, as explained in this thread, is more efficient than
> re-inventing the parser.
>
>> In the patch, I always register possible modifications in the
>> blocks/drawers intersecting with the modified region + a drawer/block
>> right next to the region.
>>
>> -----------------------------------------------------------------------
>> -----------------------------------------------------------------------
>>
>> More details on the nested 'invisible text property implementation.
>>
>> The idea is to keep 'invisible property stack push and popping from it
>> as we add/remove 'invisible text property. All the work is done in
>> org-flag-region.
>
> This sounds like a good idea.
>
>> This was originally intended for folding outlines via text properties.
>> Since using text properties for folding outlines is not a good idea,
>> nested text properties have much less use.
>
> AFAIU, they have. You mention link fontification, but there are other
> pieces that we could switch to text properties instead of overlays,
> e.g., Babel hashes, narrowed table columns…
>
>> 3. Multiple calls to before/after-change-functions is still a problem. I
>> am looking into following ways to reduce this number:
>>  - reduce the number of elements registered as potentially modified
>>    + do not add duplicates to org--modified-elements
>>    + do not add unfolded elements to org--modified-elements
>>    + register after-change-function as post-command hook and remove it
>>      from global after-change-functions. This way, it will be called
>>      twice per command only.
>>  - determine common region containing org--modified-elements. if change
>>    is happening within that region, there is no need to parse
>>    drawers/blocks there again.
>
> This is over-engineering. Again, please focus on local changes, as
> discussed before.
>
>> Recipe to have different (org-element-at-point) and
>> (org-element-parse-buffer 'element)
>> -------------------------------------------------------------------------
>> <point-min>
>> :PROPERTIES:
>> :CREATED:  [2020-05-23 Sat 02:32]
>> :END:
>>
>>
>> <point-max>
>> -------------------------------------------------------------------------
>
> I didn't look at this situation in particular, but there are cases where
> different :post-blank values are inevitable, for example at the end of
> a section.
>
> Regards,
>
> -- 
> Nicolas Goaziou

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-02  9:21                                   ` Ihor Radchenko
@ 2020-06-02  9:23                                     ` Ihor Radchenko
  2020-06-02 12:10                                       ` Bastien
  2020-06-02  9:25                                     ` Ihor Radchenko
  2020-06-05  7:26                                     ` Nicolas Goaziou
  2 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-06-02  9:23 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 44 bytes --]

The patch (against 758b039c0) is attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: featuredrawertextprop-20200602.patch --]
[-- Type: text/x-diff, Size: 45456 bytes --]

diff --git a/contrib/lisp/org-notify.el b/contrib/lisp/org-notify.el
index 9f8677871..ab470ea9b 100644
--- a/contrib/lisp/org-notify.el
+++ b/contrib/lisp/org-notify.el
@@ -246,7 +246,7 @@ seconds.  The default value for SECS is 20."
           (switch-to-buffer (find-file-noselect file))
           (org-with-wide-buffer
            (goto-char begin)
-           (outline-show-entry))
+           (org-show-entry))
           (goto-char begin)
           (search-forward "DEADLINE: <")
           (search-forward ":")
diff --git a/contrib/lisp/org-velocity.el b/contrib/lisp/org-velocity.el
index bfc4d6c3e..2312b235c 100644
--- a/contrib/lisp/org-velocity.el
+++ b/contrib/lisp/org-velocity.el
@@ -325,7 +325,7 @@ use it."
   (save-excursion
     (when narrow
       (org-narrow-to-subtree))
-    (outline-show-all)))
+    (org-show-all)))
 
 (defun org-velocity-edit-entry/inline (heading)
   "Edit entry at HEADING in the original buffer."
diff --git a/doc/org-manual.org b/doc/org-manual.org
index 92252179b..ff3e31abe 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -509,11 +509,11 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
   Switch back to the startup visibility of the buffer (see [[*Initial
   visibility]]).
 
-- {{{kbd(C-u C-u C-u TAB)}}} (~outline-show-all~) ::
+- {{{kbd(C-u C-u C-u TAB)}}} (~org-show-all~) ::
 
   #+cindex: show all, command
   #+kindex: C-u C-u C-u TAB
-  #+findex: outline-show-all
+  #+findex: org-show-all
   Show all, including drawers.
 
 - {{{kbd(C-c C-r)}}} (~org-reveal~) ::
@@ -529,18 +529,18 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
   headings.  With a double prefix argument, also show the entire
   subtree of the parent.
 
-- {{{kbd(C-c C-k)}}} (~outline-show-branches~) ::
+- {{{kbd(C-c C-k)}}} (~org-show-branches~) ::
 
   #+cindex: show branches, command
   #+kindex: C-c C-k
-  #+findex: outline-show-branches
+  #+findex: org-show-branches
   Expose all the headings of the subtree, but not their bodies.
 
-- {{{kbd(C-c TAB)}}} (~outline-show-children~) ::
+- {{{kbd(C-c TAB)}}} (~org-show-children~) ::
 
   #+cindex: show children, command
   #+kindex: C-c TAB
-  #+findex: outline-show-children
+  #+findex: org-show-children
   Expose all direct children of the subtree.  With a numeric prefix
   argument {{{var(N)}}}, expose all children down to level
   {{{var(N)}}}.
@@ -7294,7 +7294,7 @@ its location in the outline tree, but behaves in the following way:
   command (see [[*Visibility Cycling]]).  You can force cycling archived
   subtrees with {{{kbd(C-TAB)}}}, or by setting the option
   ~org-cycle-open-archived-trees~.  Also normal outline commands, like
-  ~outline-show-all~, open archived subtrees.
+  ~org-show-all~, open archived subtrees.
 
 -
   #+vindex: org-sparse-tree-open-archived-trees
diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index f07c3b801..a9c4d9eb2 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -6824,7 +6824,7 @@ and stored in the variable `org-prefix-format-compiled'."
 	    (t "  %-12:c%?-12t% s")))
 	(start 0)
 	varform vars var e c f opt)
-    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)"
+    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)"
 			 s start)
       (setq var (or (cdr (assoc (match-string 4 s)
 				'(("c" . category) ("t" . time) ("l" . level) ("s" . extra)
@@ -9136,20 +9136,20 @@ if it was hidden in the outline."
      ((and (called-interactively-p 'any) (= more 1))
       (message "Remote: show with default settings"))
      ((= more 2)
-      (outline-show-entry)
+      (org-show-entry)
       (org-show-children)
       (save-excursion
 	(org-back-to-heading)
 	(run-hook-with-args 'org-cycle-hook 'children))
       (message "Remote: CHILDREN"))
      ((= more 3)
-      (outline-show-subtree)
+      (org-show-subtree)
       (save-excursion
 	(org-back-to-heading)
 	(run-hook-with-args 'org-cycle-hook 'subtree))
       (message "Remote: SUBTREE"))
      ((> more 3)
-      (outline-show-subtree)
+      (org-show-subtree)
       (message "Remote: SUBTREE AND ALL DRAWERS")))
     (select-window win)))
 
diff --git a/lisp/org-archive.el b/lisp/org-archive.el
index d3e12d17b..d864dad8a 100644
--- a/lisp/org-archive.el
+++ b/lisp/org-archive.el
@@ -330,7 +330,7 @@ direct children of this heading."
 		      (insert (if datetree-date "" "\n") heading "\n")
 		      (end-of-line 0))
 		    ;; Make the subtree visible
-		    (outline-show-subtree)
+		    (org-show-subtree)
 		    (if org-archive-reversed-order
 			(progn
 			  (org-back-to-heading t)
diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index e50a4d7c8..e656df555 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -699,7 +699,7 @@ FUN is a function called with no argument."
 			  (move-beginning-of-line 2)
 			  (org-at-heading-p t)))))
     (unwind-protect (funcall fun)
-      (when hide-body (outline-hide-entry)))))
+      (when hide-body (org-hide-entry)))))
 
 (defun org-columns-previous-allowed-value ()
   "Switch to the previous allowed value for this column."
diff --git a/lisp/org-compat.el b/lisp/org-compat.el
index 635a38dcd..8fe271896 100644
--- a/lisp/org-compat.el
+++ b/lisp/org-compat.el
@@ -139,12 +139,8 @@ This is a floating point number if the size is too large for an integer."
 ;;; Emacs < 25.1 compatibility
 
 (when (< emacs-major-version 25)
-  (defalias 'outline-hide-entry 'hide-entry)
-  (defalias 'outline-hide-sublevels 'hide-sublevels)
-  (defalias 'outline-hide-subtree 'hide-subtree)
   (defalias 'outline-show-branches 'show-branches)
   (defalias 'outline-show-children 'show-children)
-  (defalias 'outline-show-entry 'show-entry)
   (defalias 'outline-show-subtree 'show-subtree)
   (defalias 'xref-find-definitions 'find-tag)
   (defalias 'format-message 'format)
diff --git a/lisp/org-element.el b/lisp/org-element.el
index ac41b7650..2d5c8d771 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -4320,7 +4320,7 @@ element or object.  Meaningful values are `first-section',
 TYPE is the type of the current element or object.
 
 If PARENT? is non-nil, assume the next element or object will be
-located inside the current one.  "
+located inside the current one."
   (if parent?
       (pcase type
 	(`headline 'section)
diff --git a/lisp/org-keys.el b/lisp/org-keys.el
index 37df29983..a714dec0f 100644
--- a/lisp/org-keys.el
+++ b/lisp/org-keys.el
@@ -437,7 +437,7 @@ COMMANDS is a list of alternating OLDDEF NEWDEF command names."
   #'org-next-visible-heading)
 (define-key org-mode-map [remap outline-previous-visible-heading]
   #'org-previous-visible-heading)
-(define-key org-mode-map [remap show-children] #'org-show-children)
+(define-key org-mode-map [remap outline-show-children] #'org-show-children)
 
 ;;;; Make `C-c C-x' a prefix key
 (org-defkey org-mode-map (kbd "C-c C-x") (make-sparse-keymap))
diff --git a/lisp/org-macs.el b/lisp/org-macs.el
index a02f713ca..681b5a404 100644
--- a/lisp/org-macs.el
+++ b/lisp/org-macs.el
@@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead."
 
 
 \f
-;;; Overlays
+;;; Overlays and text properties
 
 (defun org-overlay-display (ovl text &optional face evap)
   "Make overlay OVL display TEXT with face FACE."
@@ -705,26 +705,138 @@ If DELETE is non-nil, delete all those overlays."
 	    (delete (delete-overlay ov))
 	    (t (push ov found))))))
 
+(defun org-remove-text-properties (start end properties &optional object)
+  "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions.
+Do not remove invisible text properties specified by 'outline,
+'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this
+is needed to keep outlines, drawers, and blocks hidden unless they are
+toggled by user.
+Note: The below may be too specific and create troubles if more
+invisibility specs are added to org in future"
+  (when (plist-member properties 'invisible)
+    (let ((pos start)
+	  next spec)
+      (while (< pos end)
+	(setq next (next-single-property-change pos 'invisible nil end)
+              spec (get-text-property pos 'invisible))
+	(unless (memq spec (list 'org-hide-block
+				 'org-hide-drawer
+				 'outline))
+          (remove-text-properties pos next '(invisible nil) object))
+	(setq pos next))))
+  (when-let ((properties-stripped (org-plist-delete properties 'invisible)))
+    (remove-text-properties start end properties-stripped object)))
+
+(defun org--find-text-property-region (pos prop)
+  "Find a region containing PROP text property around point POS."
+  (let* ((beg (and (get-text-property pos prop) pos))
+	 (end beg))
+    (when beg
+      ;; when beg is the first point in the region, `previous-single-property-change'
+      ;; will return nil.
+      (setq beg (or (previous-single-property-change pos prop)
+		    beg))
+      ;; when end is the last point in the region, `next-single-property-change'
+      ;; will return nil.
+      (setq end (or (next-single-property-change pos prop)
+		    end))
+      (unless (= beg end) ; this should not happen
+        (cons beg end)))))
+
+(defun org--add-to-list-text-property (from to prop element)
+  "Add element to text property PROP, whos value should be a list."
+  (add-text-properties from to `(,prop ,(list element))) ; create if none
+  ;; add to existing
+  (alter-text-property from to
+		       prop
+		       (lambda (val)
+			 (if (member element val)
+                             val
+			   (cons element val)))))
+
+(defun org--remove-from-list-text-property (from to prop element)
+  "Remove ELEMENT from text propery PROP, whos value should be a list."
+  (let ((pos from))
+    (while (< pos to)
+      (when-let ((val (get-text-property pos prop)))
+	(if (equal val (list element))
+	    (remove-text-properties pos (next-single-char-property-change pos prop nil to) (list prop nil))
+	  (put-text-property pos (next-single-char-property-change pos prop nil to)
+			     prop (remove element (get-text-property pos prop)))))
+      (setq pos (next-single-char-property-change pos prop nil to)))))
+
+(defun org--get-buffer-local-text-property-symbol (prop &optional buffer)
+  "Compute unique symbol suitable to be used as buffer-local in BUFFER for PROP."
+  (let* ((buf (or buffer (current-buffer))))
+    (let ((local-prop-string (format "org--%s-buffer-local-%S" (symbol-name prop) (sxhash buf))))
+      (with-current-buffer buf
+	(unless (string-equal (symbol-name (car (alist-get prop char-property-alias-alist)))
+			      local-prop-string)
+          (let ((local-prop (make-symbol local-prop-string)))
+            ;; copy old property
+            (when-let ((old-prop (car (alist-get prop char-property-alias-alist))))
+              (org-with-wide-buffer
+               (let ((pos (point-min)))
+		 (while (< pos (point-max))
+		   (when-let (val (get-text-property pos old-prop))
+		     (put-text-property pos (next-single-char-property-change pos old-prop) local-prop val))
+		   (setq pos (next-single-char-property-change pos old-prop))))))
+	    (setq-local char-property-alias-alist
+			(cons (list prop local-prop)
+                              (remove (assq prop char-property-alias-alist)
+				      char-property-alias-alist)))))
+	(car (alist-get prop char-property-alias-alist))))))
+
 (defun org-flag-region (from to flag spec)
   "Hide or show lines from FROM to TO, according to FLAG.
 SPEC is the invisibility spec, as a symbol."
-  (remove-overlays from to 'invisible spec)
-  ;; Use `front-advance' since text right before to the beginning of
-  ;; the overlay belongs to the visible line than to the contents.
-  (when flag
-    (let ((o (make-overlay from to nil 'front-advance)))
-      (overlay-put o 'evaporate t)
-      (overlay-put o 'invisible spec)
-      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
-
+  ;; Use text properties instead of overlays for speed.
+  ;; Overlays are too slow (Emacs Bug#35453).
+  (with-silent-modifications
+    ;; keep a backup stack of old text properties
+    (save-excursion
+      (goto-char from)
+      (while (< (point) to)
+	(let ((old-spec (get-text-property (point) (org--get-buffer-local-text-property-symbol 'invisible)))
+	      (end (next-single-property-change (point) (org--get-buffer-local-text-property-symbol 'invisible) nil to)))
+	  (when old-spec
+	    (alter-text-property (point) end (org--get-buffer-local-text-property-symbol 'org-property-stack-invisible)
+				 (lambda (stack)
+				   (if (or (eq old-spec (car stack))
+					   (eq spec old-spec)
+					   (eq old-spec 'outline))
+				       stack
+				     (cons old-spec stack)))))
+	  (goto-char end))))
+
+    ;; cleanup everything
+    (remove-text-properties from to (list (org--get-buffer-local-text-property-symbol 'invisible) nil))
+    
+    ;; Recover properties from the backup stack
+    (unless flag
+      (save-excursion
+	(goto-char from)
+	(while (< (point) to)
+          (let ((stack (get-text-property (point) (org--get-buffer-local-text-property-symbol 'org-property-stack-invisible)))
+		(end (next-single-property-change (point) (org--get-buffer-local-text-property-symbol 'org-property-stack-invisible) nil to)))
+            (if (not stack)
+		(remove-text-properties (point) end '(org-property-stack-invisible nil))
+	      (put-text-property (point) end (org--get-buffer-local-text-property-symbol 'invisible) (car stack))
+	      (alter-text-property (point) end (org--get-buffer-local-text-property-symbol 'org-property-stack-invisible)
+				   (lambda (stack)
+				     (cdr stack))))
+            (goto-char end)))))
+
+    (when flag
+      (put-text-property from to (org--get-buffer-local-text-property-symbol 'invisible) spec))))
 
 \f
 ;;; Regexp matching
 
 (defsubst org-pos-in-match-range (pos n)
-  (and (match-beginning n)
-       (<= (match-beginning n) pos)
-       (>= (match-end n) pos)))
+(and (match-beginning n)
+     (<= (match-beginning n) pos)
+     (>= (match-end n) pos)))
 
 (defun org-skip-whitespace ()
   "Skip over space, tabs and newline characters."
diff --git a/lisp/org-src.el b/lisp/org-src.el
index 6f6c544dc..9e8a50044 100644
--- a/lisp/org-src.el
+++ b/lisp/org-src.el
@@ -529,8 +529,8 @@ Leave point in edit buffer."
 	(org-src-switch-to-buffer buffer 'edit)
 	;; Insert contents.
 	(insert contents)
-	(remove-text-properties (point-min) (point-max)
-				'(display nil invisible nil intangible nil))
+	(org-remove-text-properties (point-min) (point-max)
+				    '(display nil invisible nil intangible nil))
 	(unless preserve-ind (org-do-remove-indentation))
 	(set-buffer-modified-p nil)
 	(setq buffer-file-name nil)
diff --git a/lisp/org-table.el b/lisp/org-table.el
index 6462b99c4..75801161b 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -2001,7 +2001,7 @@ toggle `org-table-follow-field-mode'."
    (arg
     (let ((b (save-excursion (skip-chars-backward "^|") (point)))
 	  (e (save-excursion (skip-chars-forward "^|\r\n") (point))))
-      (remove-text-properties b e '(invisible t intangible t))
+      (org-remove-text-properties b e '(invisible t intangible t))
       (if (and (boundp 'font-lock-mode) font-lock-mode)
 	  (font-lock-fontify-block))))
    (t
@@ -2028,7 +2028,7 @@ toggle `org-table-follow-field-mode'."
       (setq word-wrap t)
       (goto-char (setq p (point-max)))
       (insert (org-trim field))
-      (remove-text-properties p (point-max) '(invisible t intangible t))
+      (org-remove-text-properties p (point-max) '(invisible t intangible t))
       (goto-char p)
       (setq-local org-finish-function 'org-table-finish-edit-field)
       (setq-local org-window-configuration cw)
diff --git a/lisp/org.el b/lisp/org.el
index f201138f1..6f5aa4b7e 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.")
 (declare-function cdlatex-math-symbol "ext:cdlatex")
 (declare-function Info-goto-node "info" (nodename &optional fork strict-case))
 (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag))
+(declare-function isearch-filter-visible "isearch" (beg end))
 (declare-function org-add-archive-files "org-archive" (files))
 (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom))
 (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour))
@@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.")
 
 (defvar ffap-url-regexp)
 (defvar org-element-paragraph-separate)
+(defvar org-element-all-objects)
+(defvar org-element-all-elements)
+(defvar org-element-greater-elements)
 (defvar org-indent-indentation-per-level)
 (defvar org-radio-target-regexp)
 (defvar org-target-link-regexp)
@@ -4734,9 +4738,174 @@ This is for getting out of special buffers like capture.")
 
 ;;;; Define the Org mode
 
+;;; Handling buffer modifications
+
 (defun org-before-change-function (_beg _end)
   "Every change indicates that a table might need an update."
   (setq org-table-may-need-update t))
+
+(defun org-after-change-function (from to len)
+  "Hide text in region if it follows and is followedby invisible text."
+  (when-let ((spec-to (get-text-property to 'invisible))
+	     (spec-from (get-text-property (max (point-min) (1- from)) 'invisible)))
+    (when (eq spec-to spec-from)
+      (org-flag-region from to 't spec-to))))
+
+
+(defvar org--element-beginning-re-alist `((center-block . "^[ \t]*#\\+begin_center[ \t]*$")
+                                       (property-drawer . ,org-property-start-re)
+				       (drawer . ,org-drawer-regexp)
+                                       (quote-block . "^[ \t]*#\\+begin_quote[ \t]*$")
+                                       (special-block . "^[ \t]*#\\+begin_\\([^ ]+\\).*$"))
+  "Alist of regexps matching beginning of elements.
+Group 1 in the regexps (if any) contains the element type.")
+
+(defvar org--element-end-re-alist `((center-block . "^[ \t]*#\\+end_center[ \t]*$")
+				 (property-drawer . ,org-property-end-re)
+				 (drawer . ,org-property-end-re)
+				 (quote-block . "^[ \t]*#\\+end_quote[ \t]*$")
+				 (special-block . "^[ \t]*#\\+end_\\([^ ]+\\).*$"))
+  "Alist of regexps matching end of elements.
+Group 1 in the regexps (if any) contains the element type or END.")
+
+(defvar org-track-element-modifications
+  `(property-drawer
+    drawer
+    center-block
+    quote-block
+    special-block)
+  "Alist of elements to be tracked for modifications.
+The modification is only triggered when beginning/end line of the element is modified.")
+
+(defun org--get-element-region-at-point (types)
+  "Return TYPES element at point or nil.
+If TYPES is a list, return first element at point from the list.  The
+returned value is partially parsed element only containing :begin and
+:end properties.  Only elements listed in
+org--element-beginning-re-alist and org--element-end-re-alist can be
+parsed here."
+  (catch 'exit
+    (dolist (type (if (listp types) types (list types)))
+      (let ((begin-re (alist-get type org--element-beginning-re-alist))
+	    (end-re (alist-get type org--element-end-re-alist))
+            (begin-limit (save-excursion (org-with-limited-levels
+					  (org-back-to-heading-or-point-min 'invisible-ok))
+					 (point)))
+            (end-limit (or (save-excursion (outline-next-heading))
+			   (point-max)))
+            (point (point))
+	    begin end)
+	(when (and begin-re end-re)
+	  (save-excursion
+	    (end-of-line)
+	    (when (re-search-backward begin-re begin-limit 'noerror) (setq begin (point)))
+	    (when (re-search-forward end-re end-limit 'noerror) (setq end (point)))
+            ;; slurp unmatched begin-re
+	    (when (and begin end)
+              (goto-char begin)
+              (while (and (re-search-backward begin-re begin-limit 'noerror)
+			  (= end (save-excursion (re-search-forward end-re end-limit 'noerror))))
+		(setq begin (point)))
+              (when (and (>= point begin) (<= point end))
+		(throw 'exit
+                       (let ((begin (copy-marker begin 't))
+			     (end (copy-marker end nil)))
+			 (list type
+			       (list
+				:begin begin
+				:post-affiliated begin
+				:contents-begin (save-excursion (goto-char begin) (copy-marker (1+ (line-end-position))
+											       't))
+				:contents-end (save-excursion (goto-char end) (copy-marker (1- (line-beginning-position))
+											   nil))
+				:end end))))))))))))
+
+(defun org--get-next-element-region-at-point (types &optional limit previous)
+  "Return TYPES element after point or nil.
+If TYPES is a list, return first element after point from the list.
+If PREVIOUS is non-nil, return first TYPES element before point.
+Limit search by LIMIT or previous/next heading.
+The returned value is partially parsed element only containing :begin
+and :end properties.  Only elements listed in
+org--element-beginning-re-alist and org--element-end-re-alist can be
+parsed here."
+  (catch 'exit
+    (dolist (type (if (listp types) types (list types)))
+      (let* ((begin-re (alist-get type org--element-beginning-re-alist))
+	     (end-re (alist-get type org--element-end-re-alist))
+             (limit (or limit (if previous
+				  (save-excursion
+				    (org-with-limited-levels
+				     (org-back-to-heading-or-point-min 'invisible-ok)
+				     (point)))
+				(or (save-excursion (outline-next-heading))
+				    (point-max)))))
+	     el)
+	(when (and begin-re end-re)
+	  (save-excursion
+            (if previous
+                (when (re-search-backward begin-re limit 'noerror)
+		  (setq el (org--get-element-region-at-point type)))
+	      (when (re-search-forward begin-re limit 'noerror)
+		(setq el (org--get-element-region-at-point type)))))
+	  (when el
+            (throw 'exit
+		   el)))))))
+
+(defun org--find-elements-in-region (beg end elements &optional include-partial include-neighbours)
+  "Find all elements from ELEMENTS in region BEG . END.
+All the listed elements must be resolvable by
+`org--get-element-region-at-point'.
+Include elements if they are partially inside region when
+INCLUDE-PARTIAL is non-nil.
+Include preceding/subsequent neighbouring elements when no partial
+element is found at the beginning/end of the region and
+INCLUDE-NEIGHBOURS is non-nil."
+  (when include-partial
+    (org-with-point-at beg
+      (let ((new-beg (org-element-property :begin (org--get-element-region-at-point elements))))
+	(if new-beg
+	    (setq beg new-beg)
+          (when (and include-neighbours
+		     (setq new-beg (org-element-property :begin
+						      (org--get-next-element-region-at-point elements
+											  (point-min)
+											  'previous))))
+            (setq beg new-beg))))
+      (when (memq 'headline elements)
+	(when-let ((new-beg (save-excursion
+			      (org-with-limited-levels (outline-previous-heading)))))
+          (setq beg new-beg))))
+    (org-with-point-at end
+      (let ((new-end (org-element-property :end (org--get-element-region-at-point elements))))
+	(if new-end
+	    (setq end new-end)
+          (when (and include-neighbours
+		     (setq new-end (org-element-property :end
+						      (org--get-next-element-region-at-point elements (point-max)))))
+            (setq end new-end))))
+      (when (memq 'headline elements)
+	(when-let ((new-end (org-with-limited-levels (outline-next-heading))))
+          (setq end (1- new-end))))))
+  (save-excursion
+    (save-restriction
+      (narrow-to-region beg end)
+      (goto-char (point-min))
+      (let (result el)
+	(while (setq el (org--get-next-element-region-at-point elements end))
+          (push el result)
+          (goto-char (org-element-property :end el)))
+        result))))
+
+(defun org--unfold-elements-in-region (el &rest _)
+  "Unfold EL element."
+  (when-let ((category (if (string-match-p "block" (symbol-name (org-element-type el)))
+			   'block
+			 (when (string-match-p "drawer" (symbol-name (org-element-type el)))
+			   'drawer))))
+    (org-with-point-at (org-element-property :begin el)
+      (org--hide-wrapper-toggle el category 'off nil))))
+
 (defvar org-mode-map)
 (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param.
 (defvar org-agenda-keep-modes nil)      ; Dynamically-scoped param.
@@ -4818,6 +4987,8 @@ The following commands are available:
   ;; Activate before-change-function
   (setq-local org-table-may-need-update t)
   (add-hook 'before-change-functions 'org-before-change-function nil 'local)
+  ;; Activate after-change-function
+  (add-hook 'after-change-functions 'org-after-change-function nil 'local)
   ;; Check for running clock before killing a buffer
   (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local)
   ;; Initialize macros templates.
@@ -4869,6 +5040,10 @@ The following commands are available:
   (setq-local outline-isearch-open-invisible-function
 	      (lambda (&rest _) (org-show-context 'isearch)))
 
+  ;; Make isearch search in blocks hidden via text properties
+  (setq-local isearch-filter-predicate #'org--isearch-filter-predicate)
+  (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local)
+
   ;; Setup the pcomplete hooks
   (setq-local pcomplete-command-completion-function #'org-pcomplete-initial)
   (setq-local pcomplete-command-name-function #'org-command-at-point)
@@ -5050,8 +5225,8 @@ stacked delimiters is N.  Escaping delimiters is not possible."
 	      (when verbatim?
 		(org-remove-flyspell-overlays-in
 		 (match-beginning 0) (match-end 0))
-		(remove-text-properties (match-beginning 2) (match-end 2)
-					'(display t invisible t intangible t)))
+		(org-remove-text-properties (match-beginning 2) (match-end 2)
+					 '(display t invisible t intangible t)))
 	      (add-text-properties (match-beginning 2) (match-end 2)
 				   '(font-lock-multiline t org-emphasis t))
 	      (when (and org-hide-emphasis-markers
@@ -5166,7 +5341,7 @@ This includes angle, plain, and bracket links."
 	    (if (not (eq 'bracket style))
 		(add-text-properties start end properties)
 	      ;; Handle invisible parts in bracket links.
-	      (remove-text-properties start end '(invisible nil))
+	      (org-remove-text-properties start end '(invisible nil))
 	      (let ((hidden
 		     (append `(invisible
 			       ,(or (org-link-get-parameter type :display)
@@ -5186,8 +5361,8 @@ This includes angle, plain, and bracket links."
 (defun org-activate-code (limit)
   (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t)
     (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
-    (remove-text-properties (match-beginning 0) (match-end 0)
-			    '(display t invisible t intangible t))
+    (org-remove-text-properties (match-beginning 0) (match-end 0)
+			     '(display t invisible t intangible t))
     t))
 
 (defcustom org-src-fontify-natively t
@@ -5258,8 +5433,8 @@ by a #."
 	    (setq block-end (match-beginning 0)) ; includes the final newline.
 	    (when quoting
 	      (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline)
-	      (remove-text-properties beg end-of-endline
-				      '(display t invisible t intangible t)))
+	      (org-remove-text-properties beg end-of-endline
+				       '(display t invisible t intangible t)))
 	    (add-text-properties
 	     beg end-of-endline '(font-lock-fontified t font-lock-multiline t))
 	    (org-remove-flyspell-overlays-in beg bol-after-beginline)
@@ -5313,9 +5488,9 @@ by a #."
 	     '(font-lock-fontified t face org-document-info))))
 	 ((string-prefix-p "+caption" dc1)
 	  (org-remove-flyspell-overlays-in (match-end 2) (match-end 0))
-	  (remove-text-properties (match-beginning 0) (match-end 0)
-				  '(display t invisible t intangible t))
-	  ;; Handle short captions
+	  (org-remove-text-properties (match-beginning 0) (match-end 0)
+				   '(display t invisible t intangible t))
+	  ;; Handle short captions.
 	  (save-excursion
 	    (beginning-of-line)
 	    (looking-at (rx (group (zero-or-more blank)
@@ -5336,8 +5511,8 @@ by a #."
 	   '(font-lock-fontified t face font-lock-comment-face)))
 	 (t ;; Just any other in-buffer setting, but not indented
 	  (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
-	  (remove-text-properties (match-beginning 0) (match-end 0)
-				  '(display t invisible t intangible t))
+	  (org-remove-text-properties (match-beginning 0) (match-end 0)
+				   '(display t invisible t intangible t))
 	  (add-text-properties beg (match-end 0)
 			       '(font-lock-fontified t face org-meta-line))
 	  t))))))
@@ -5859,10 +6034,11 @@ If TAG is a number, get the corresponding match group."
 	 (inhibit-modification-hooks t)
 	 deactivate-mark buffer-file-name buffer-file-truename)
     (decompose-region beg end)
-    (remove-text-properties beg end
-			    '(mouse-face t keymap t org-linked-text t
-					 invisible t intangible t
-					 org-emphasis t))
+    (org-remove-text-properties beg end
+			     '(mouse-face t keymap t org-linked-text t
+					  invisible t
+                                          intangible t
+					  org-emphasis t))
     (org-remove-font-lock-display-properties beg end)))
 
 (defconst org-script-display  '(((raise -0.3) (height 0.7))
@@ -5970,6 +6146,29 @@ open and agenda-wise Org files."
 
 ;;;; Headlines visibility
 
+(defun org-hide-entry ()
+  "Hide the body directly following this heading."
+  (interactive)
+  (save-excursion
+    (outline-back-to-heading)
+    (outline-end-of-heading)
+    (org-flag-region (point) (progn (outline-next-preface) (point)) t 'outline)))
+
+(defun org-hide-subtree ()
+  "Hide everything after this heading at deeper levels."
+  (interactive)
+  (org-flag-subtree t))
+
+(defun org-hide-sublevels (levels)
+  "Hide everything but the top LEVELS levels of headers, in whole buffer.
+This also unhides the top heading-less body, if any.
+
+Interactively, the prefix argument supplies the value of LEVELS.
+When invoked without a prefix argument, LEVELS defaults to the level
+of the current heading, or to 1 if the current line is not a heading."
+  (cl-letf (((symbol-function 'outline-flag-region) #'org-flag-region))
+    (org-hide-sublevels levels)))
+
 (defun org-show-entry ()
   "Show the body directly following this heading.
 Show the heading too, if it is currently invisible."
@@ -5988,6 +6187,17 @@ Show the heading too, if it is currently invisible."
        'outline)
       (org-cycle-hide-property-drawers 'children))))
 
+(defun org-show-heading ()
+  "Show the current heading and move to its end."
+  (org-flag-region (- (point)
+ 		   (if (bobp) 0
+ 		     (if (and outline-blank-line
+                              (eq (char-before (1- (point))) ?\n))
+ 			 2 1)))
+		(progn (outline-end-of-heading) (point))
+		nil
+                'outline))
+
 (defun org-show-children (&optional level)
   "Show all direct subheadings of this heading.
 Prefix arg LEVEL is how many levels below the current level
@@ -6031,6 +6241,11 @@ heading to appear."
   (org-flag-region
    (point) (save-excursion (org-end-of-subtree t t)) nil 'outline))
 
+(defun org-show-branches ()
+  "Show all subheadings of this heading, but not their bodies."
+  (interactive)
+  (org-show-children 1000))
+
 ;;;; Blocks and drawers visibility
 
 (defun org--hide-wrapper-toggle (element category force no-error)
@@ -6064,13 +6279,39 @@ Return a non-nil value when toggling is successful."
 	(unless (let ((eol (line-end-position)))
 		  (and (> eol start) (/= eol end)))
 	  (let* ((spec (cond ((eq category 'block) 'org-hide-block)
-			     ((eq type 'property-drawer) 'outline)
-			     (t 'org-hide-drawer)))
+			     ((eq category 'drawer) 'org-hide-drawer)
+			     (t 'outline)))
 		 (flag
 		  (cond ((eq force 'off) nil)
 			(force t)
 			((eq (get-char-property start 'invisible) spec) nil)
 			(t t))))
+            ;; Make beginning/end of blocks sensitive to modifications
+            ;; we never remove the hooks because modification of parts
+            ;; of blocks is practically more rare in comparison with
+            ;; folding/unfolding. Removing modification hooks would
+            ;; cost more CPU time.
+            (when flag
+              (with-silent-modifications
+		(let ((el (org--get-element-region-at-point
+                           (org-element-type element))))
+		  (unless (member (apply-partially #'org--unfold-elements-in-region el)
+				  (get-text-property (org-element-property :begin element)
+						     'modification-hooks))
+		    ;; first line
+		    (org--add-to-list-text-property (org-element-property :begin element) start
+						 'modification-hooks
+						 (apply-partially #'org--unfold-elements-in-region el))
+                    (org--add-to-list-text-property (org-element-property :begin element) start
+						 'insert-behind-hooks
+						 (apply-partially #'org--unfold-elements-in-region el))
+		    ;; last line
+		    (org--add-to-list-text-property (save-excursion (goto-char end) (line-beginning-position)) end
+						 'modification-hooks
+						 (apply-partially #'org--unfold-elements-in-region el))
+                    (org--add-to-list-text-property (save-excursion (goto-char end) (line-beginning-position)) end
+						 'insert-behind-hooks
+						 (apply-partially #'org--unfold-elements-in-region el))))))
 	    (org-flag-region start end flag spec))
 	  ;; When the block is hidden away, make sure point is left in
 	  ;; a visible part of the buffer.
@@ -6118,24 +6359,16 @@ Return a non-nil value when toggling is successful."
 
 (defun org-hide-drawer-all ()
   "Fold all drawers in the current buffer."
-  (org-show-all '(drawers))
   (save-excursion
     (goto-char (point-min))
     (while (re-search-forward org-drawer-regexp nil t)
-      (let* ((drawer (org-element-at-point))
-	     (type (org-element-type drawer)))
-	(when (memq type '(drawer property-drawer))
-	  ;; We are sure regular drawers are unfolded because of
-	  ;; `org-show-all' call above.  However, property drawers may
-	  ;; be folded, or in a folded headline.  In that case, do not
-	  ;; re-hide it.
-	  (unless (and (eq type 'property-drawer)
-		       (eq 'outline (get-char-property (point) 'invisible)))
-	    (org-hide-drawer-toggle t nil drawer))
-	  ;; Make sure to skip drawer entirely or we might flag it
-	  ;; another time when matching its ending line with
-	  ;; `org-drawer-regexp'.
-	  (goto-char (org-element-property :end drawer)))))))
+      (when-let* ((drawer (org--get-element-region-at-point '(property-drawer drawer)))
+		  (type (org-element-type drawer)))
+	(org-hide-drawer-toggle t nil drawer)
+	;; Make sure to skip drawer entirely or we might flag it
+	;; another time when matching its ending line with
+	;; `org-drawer-regexp'.
+	(goto-char (org-element-property :end drawer))))))
 
 (defun org-cycle-hide-property-drawers (state)
   "Re-hide all drawers after a visibility state change.
@@ -6150,18 +6383,16 @@ STATE should be one of the symbols listed in the docstring of
 		      (t (save-excursion (org-end-of-subtree t))))))
       (org-with-point-at beg
 	(while (re-search-forward org-property-start-re end t)
-	  (pcase (get-char-property-and-overlay (point) 'invisible)
+	  (pcase (get-char-property (point) 'invisible)
 	    ;; Do not fold already folded drawers.
-	    (`(outline . ,o) (goto-char (overlay-end o)))
+	    ('outline
+             (goto-char (min end (next-single-char-property-change (point) 'invisible))))
 	    (_
 	     (let ((start (match-end 0)))
 	       (when (org-at-property-drawer-p)
 		 (let* ((case-fold-search t)
 			(end (re-search-forward org-property-end-re)))
-		   ;; Property drawers use `outline' invisibility spec
-		   ;; so they can be swallowed once we hide the
-		   ;; outline.
-		   (org-flag-region start end t 'outline)))))))))))
+		   (org-flag-region start end t 'org-hide-drawer)))))))))))
 
 ;;;; Visibility cycling
 
@@ -6536,7 +6767,7 @@ With a numeric prefix, show all headlines up to that level."
 		     (org-narrow-to-subtree)
 		     (org-content))))
 		((or "all" "showall")
-		 (outline-show-subtree))
+		 (org-show-subtree))
 		(_ nil)))
 	    (org-end-of-subtree)))))))
 
@@ -6609,7 +6840,7 @@ This function is the default value of the hook `org-cycle-hook'."
 	  (while (re-search-forward re nil t)
 	    (when (and (not (org-invisible-p))
 		       (org-invisible-p (line-end-position)))
-	      (outline-hide-entry))))
+	      (org-hide-entry))))
 	(org-cycle-hide-property-drawers 'all)
 	(org-cycle-show-empty-lines 'overview)))))
 
@@ -6681,10 +6912,11 @@ information."
     (org-show-entry)
     ;; If point is hidden within a drawer or a block, make sure to
     ;; expose it.
-    (dolist (o (overlays-at (point)))
-      (when (memq (overlay-get o 'invisible)
-		  '(org-hide-block org-hide-drawer outline))
-	(delete-overlay o)))
+    (when (memq (get-text-property (point) 'invisible)
+		'(org-hide-block org-hide-drawer))
+      (let ((spec (get-text-property (point) 'invisible))
+	    (region (org--find-text-property-region (point) 'invisible)))
+	(org-flag-region (car region) (cdr region) nil spec)))
     (unless (org-before-first-heading-p)
       (org-with-limited-levels
        (cl-case detail
@@ -6900,9 +7132,10 @@ unconditionally."
       ;; When INVISIBLE-OK is non-nil, ensure newly created headline
       ;; is visible.
       (unless invisible-ok
-	(pcase (get-char-property-and-overlay (point) 'invisible)
-	  (`(outline . ,o)
-	   (move-overlay o (overlay-start o) (line-end-position 0)))
+	(pcase (get-char-property (point) 'invisible)
+	  ('outline
+           (let ((region (org--find-text-property-region (point) 'invisible)))
+	     (org-flag-region (line-end-position 0) (cdr region) nil 'outline)))
 	  (_ nil))))
      ;; At a headline...
      ((org-at-heading-p)
@@ -7499,7 +7732,6 @@ case."
      (setq txt (buffer-substring beg end))
      (org-save-markers-in-region beg end)
      (delete-region beg end)
-     (org-remove-empty-overlays-at beg)
      (unless (= beg (point-min)) (org-flag-region (1- beg) beg nil 'outline))
      (unless (bobp) (org-flag-region (1- (point)) (point) nil 'outline))
      (and (not (bolp)) (looking-at "\n") (forward-char 1))
@@ -7661,7 +7893,7 @@ When REMOVE is non-nil, remove the subtree from the clipboard."
      (skip-chars-forward " \t\n\r")
      (setq beg (point))
      (when (and (org-invisible-p) visp)
-       (save-excursion (outline-show-heading)))
+       (save-excursion (org-show-heading)))
      ;; Shift if necessary.
      (unless (= shift 0)
        (save-restriction
@@ -8103,7 +8335,7 @@ function is being called interactively."
 		       (point))
 	    what "children")
       (goto-char start)
-      (outline-show-subtree)
+      (org-show-subtree)
       (outline-next-heading))
      (t
       ;; we will sort the top-level entries in this file
@@ -13158,7 +13390,7 @@ drawer is immediately hidden."
 	   (inhibit-read-only t))
        (unless (bobp) (insert "\n"))
        (insert ":PROPERTIES:\n:END:")
-       (org-flag-region (line-end-position 0) (point) t 'outline)
+       (org-flag-region (line-end-position 0) (point) t 'org-hide-drawer)
        (when (or (eobp) (= begin (point-min))) (insert "\n"))
        (org-indent-region begin (point))))))
 
@@ -17621,11 +17853,11 @@ Move point to the beginning of first heading or end of buffer."
 (defun org-show-branches-buffer ()
   "Show all branches in the buffer."
   (org-flag-above-first-heading)
-  (outline-hide-sublevels 1)
+  (org-hide-sublevels 1)
   (unless (eobp)
-    (outline-show-branches)
+    (org-show-branches)
     (while (outline-get-next-sibling)
-      (outline-show-branches)))
+      (org-show-branches)))
   (goto-char (point-min)))
 
 (defun org-kill-note-or-show-branches ()
@@ -17639,8 +17871,8 @@ Move point to the beginning of first heading or end of buffer."
 	(t
 	 (let ((beg (progn (org-back-to-heading) (point)))
 	       (end (save-excursion (org-end-of-subtree t t) (point))))
-	   (outline-hide-subtree)
-	   (outline-show-branches)
+	   (org-hide-subtree)
+	   (org-show-branches)
 	   (org-hide-archived-subtrees beg end)))))
 
 (defun org-delete-indentation (&optional arg)
@@ -17796,9 +18028,9 @@ Otherwise, call `org-show-children'.  ARG is the level to hide."
     (if (org-before-first-heading-p)
         (progn
           (org-flag-above-first-heading)
-          (outline-hide-sublevels (or arg 1))
+          (org-hide-sublevels (or arg 1))
           (goto-char (point-min)))
-      (outline-hide-subtree)
+      (org-hide-subtree)
       (org-show-children arg))))
 
 (defun org-ctrl-c-star ()
@@ -20475,17 +20707,17 @@ With ARG, repeats or can move backward if negative."
       (end-of-line))
     (while (and (< arg 0) (re-search-backward regexp nil :move))
       (unless (bobp)
-	(pcase (get-char-property-and-overlay (point) 'invisible)
-	  (`(outline . ,o)
-	   (goto-char (overlay-start o))
+	(pcase (get-char-property (point) 'invisible)
+	  ('outline
+	   (goto-char (car (org--find-text-property-region (point) 'invisible)))
 	   (beginning-of-line))
 	  (_ nil)))
       (cl-incf arg))
-    (while (and (> arg 0) (re-search-forward regexp nil t))
-      (pcase (get-char-property-and-overlay (point) 'invisible)
-	(`(outline . ,o)
-	 (goto-char (overlay-end o))
-	 (skip-chars-forward " \t\n")
+    (while (and (> arg 0) (re-search-forward regexp nil :move))
+      (pcase (get-char-property (point) 'invisible)
+	('outline
+	 (goto-char (cdr (org--find-text-property-region (point) 'invisible)))
+         (skip-chars-forward " \t\n")
 	 (end-of-line))
 	(_
 	 (end-of-line)))
@@ -20943,6 +21175,80 @@ Started from `gnus-info-find-node'."
           (t default-org-info-node))))))
 
 \f
+
+;;; Make isearch search in some text hidden via text propertoes
+
+(defvar org--isearch-overlays nil
+  "List of overlays temporarily created during isearch.
+This is used to allow searching in regions hidden via text properties.
+As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays.
+Any text hidden via text properties is not revealed even if `search-invisible'
+is set to 't.")
+
+;; Not sure if it needs to be a user option
+;; One might want to reveal hidden text in, for example, hidden parts of the links.
+;; Currently, hidden text in links is never revealed by isearch.
+(defvar org-isearch-specs '(org-hide-block
+			 org-hide-drawer)
+  "List of text invisibility specs to be searched by isearch.
+By default ([2020-05-09 Sat]), isearch does not search in hidden text,
+which was made invisible using text properties. Isearch will be forced
+to search in hidden text with any of the listed 'invisible property value.")
+
+(defun org--create-isearch-overlays (beg end)
+  "Replace text property invisibility spec by overlays between BEG and END.
+All the regions with invisibility text property spec from
+`org-isearch-specs' will be changed to use overlays instead
+of text properties. The created overlays will be stored in
+`org--isearch-overlays'."
+  (let ((pos beg))
+    (while (< pos end)
+      (when-let* ((spec (get-text-property pos 'invisible))
+		  (spec (memq spec org-isearch-specs))
+		  (region (org--find-text-property-region pos 'invisible)))
+        (setq spec (get-text-property pos 'invisible))
+        ;; Changing text properties is considered buffer modification.
+        ;; We do not want it here.
+	(with-silent-modifications
+          ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat]
+          ;; overlay for 'outline blocks.
+          (let ((o (make-overlay (car region) (cdr region) nil 'front-advance)))
+	    (overlay-put o 'evaporate t)
+	    (overlay-put o 'invisible spec)
+            ;; `delete-overlay' here means that spec information will be lost
+            ;; for the region. The region will remain visible.
+	    (overlay-put o 'isearch-open-invisible #'delete-overlay)
+            (push o org--isearch-overlays))
+	  (org-flag-region (car region) (cdr region) nil spec)))
+      (setq pos (next-single-property-change pos 'invisible nil end)))))
+
+(defun org--isearch-filter-predicate (beg end)
+  "Return non-nil if text between BEG and END is deemed visible by Isearch.
+This function is intended to be used as `isearch-filter-predicate'.
+Unlike `isearch-filter-visible', make text with 'invisible text property
+value listed in `org-isearch-specs' visible to Isearch."
+  (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text
+  (isearch-filter-visible beg end))
+
+(defun org--clear-isearch-overlay (ov)
+  "Convert OV region back into using text properties."
+  (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays
+    ;; Changing text properties is considered buffer modification.
+    ;; We do not want it here.
+    (with-silent-modifications
+      (org-flag-region (overlay-start ov) (overlay-end ov) t spec)))
+  (when (member ov isearch-opened-overlays)
+    (setq isearch-opened-overlays (delete ov isearch-opened-overlays)))
+  (delete-overlay ov))
+
+(defun org--clear-isearch-overlays ()
+  "Convert overlays from `org--isearch-overlays' back into using text properties."
+  (when org--isearch-overlays
+    (mapc #'org--clear-isearch-overlay org--isearch-overlays)
+    (setq org--isearch-overlays nil)))
+
+\f
+
 ;;; Finish up
 
 (add-hook 'org-mode-hook     ;remove overlays when changing major mode

[-- Attachment #3: Type: text/plain, Size: 17692 bytes --]



Ihor Radchenko <yantar92@gmail.com> writes:

> Hello,
>
> [The patch itself will be provided in the following email]
>
> I have three updates from the previous version of the patch:
>
> 1. I managed to implement buffer-local text properties.
>    Now, outline folding also uses text properties without a need to give
>    up independent folding in indirect buffers.
>
> 2. The code handling modifications in folded drawers/blocks was
>    rewritten. The new code uses after-change-functions to re-hide text
>    inserted in the middle of folded regions; and text properties to
>    unfold folded drawers/blocks if one changes BEGIN/END line.
>
> 3. [experimental] Started working on improving memory and cpu footprint
>    of the old code related to folding/unfolding. org-hide-drawer-all now
>    works significantly faster because I can utilise simplified drawer
>    parser, which require a lot less memory. Overall, I managed to reduce
>    Emacs memory footprint after loading all my agenda_files twice. The
>    loading is also noticeably faster.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the buffer-local text properties:
>
> I have found char-property-alias-alist variable that controls how Emacs
> calculates text property value if the property is not set. This variable
> can be buffer-local, which allows independent 'invisible states in
> different buffers.
>
> All the implementation stays in
> org--get-buffer-local-text-property-symbol, which takes care about
> generating unique property name and mapping it to 'invisible (or any
> other) text property.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the new implementation for tracking changes:
>
> I simplified the code as suggested, without using pairs of before- and
> after-change-functions.
>
> Handling text inserted into folded/invisible region is handled by a
> simple after-change function. After testing, it turned out that simple
> re-hiding text based on 'invisible property of the text before/after the
> inserted region works pretty well.
>
> Modifications to BEGIN/END line of the drawers and blocks is handled via
> 'modification-hooks + 'insert-behind-hooks text properties (there is no
> after-change-functions analogue for text properties in Emacs). The
> property is applied during folding and the modification-hook function is
> made aware about the drawer/block boundaries (via apply-partially
> passing element containing :begin :end markers for the current
> drawer/block). Passing the element boundary is important because the
> 'modification-hook will not directly know where it belongs to. Only the
> modified region (which can be larger than the drawer) is passed to the
> function. In the worst case, the region can be the whole buffer (if one
> runs revert-buffer).
>
> It turned out that adding 'modification-hook text property takes a
> significant cpu time (partially, because we need to take care about
> possible existing 'modification-hook value, see
> org--add-to-list-text-property). For now, I decided to not clear the
> modification hooks during unfolding because of poor performance.
> However, this approach would lead to partial unfolding in the following
> case:
>
> :asd:
> :drawer:
> lksjdfksdfjl
> sdfsdfsdf
> :end:
>
> If :asd: was inserted in front of folded :drawer:, changes in :drawer:
> line of the new folded :asd: drawer would reveal the text between
> :drawer: and :end:.
>
> Let me know what you think on this.
>
>> You shouldn't be bothered by the case you're describing here, for
>> multiple reasons.
>> 
>> First, this issue already arises in the current implementation. No one
>> bothered so far: this change is very unlikely to happen. If it becomes
>> an issue, we could make sure that `org-reveal' handles this.
>> 
>> But, more importantly, we actually /want it/ as a feature. Indeed, if
>> DRAWER is expanded every time ":BLAH:" is inserted above, then inserting
>> a drawer manually would unfold /all/ drawers in the section. The user is
>> more likely to write first ":BLAH:" (everything is unfolded) then
>> ":END:" than ":END:", then ":BLAH:".
>
> Agree. This allowed me to simplify the code significantly.
>
>> It seems you're getting it backwards. `before-change-functions' are the
>> functions being called with a possibly wide, imprecise, region to
>> handle:
>> 
>>     When that happens, the arguments to ‘before-change-functions’ will
>>     enclose a region in which the individual changes are made, but won’t
>>     necessarily be the minimal such region
>> 
>> however, after-change-functions calls are always minimal:
>> 
>>     and the arguments to each successive call of
>>     ‘after-change-functions’ will then delimit the part of text being
>>     changed exactly.
>> 
>> If you stick to `after-change-functions', there will be no such thing as
>> you describe.
>
> You are right here, I missed that before-change-functions are likely to
> be called on large regions. I thought that the regions are same for
> before/after-change-functions, but after-change-functions could be
> called more than 1 time. After second thought, your vision that it is
> mostly 0 or 1 times should be the majority of cases in practice.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on reducing cpu and memory footprint of org buffers:
>
> My simplified implementation of element boundary parser
> (org--get-element-region-at-point) appears to be much faster and also
> uses much less memory in comparison with org-element-at-point.
> Moreover, not all the places where org-element-at-point is called
> actually need the full parsed element. For example, org-hide-drawer-all,
> org-hide-drawer-toggle, org-hide-block-toggle, and
> org--hide-wrapper-toggle only need element type and some information
> about the element boundaries - the information we can get from
> org--get-element-region-at-point.
>
> The following version of org-hide-drawer-all seems to work much faster
> in comparison with original:
>
> (defun org-hide-drawer-all ()
>   "Fold all drawers in the current buffer."
>   (save-excursion
>     (goto-char (point-min))
>     (while (re-search-forward org-drawer-regexp nil t)
>       (when-let* ((drawer (org--get-element-region-at-point '(property-drawer drawer)))
> 		  (type (org-element-type drawer)))
> 	(org-hide-drawer-toggle t nil drawer)
> 	;; Make sure to skip drawer entirely or we might flag it
> 	;; another time when matching its ending line with
> 	;; `org-drawer-regexp'.
> 	(goto-char (org-element-property :end drawer))))))
>
> What do you think about the idea of making use of
> org--get-element-region-at-point in org code base?
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> Further work:
>
> 1. Look into other code using overlays. Specifically,
> org-toggle-custom-properties, Babel hashes, and narrowed table columns.
>
> Best,
> Ihor
>
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Hello,
>>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>> I have five updates from the previous version of the patch:
>>
>> Thank you.
>>
>>> 1. I implemented a simplified version of element parsing to detect
>>> changes in folded drawers or blocks. No computationally expensive calls
>>> of org-element-at-point or org-element-parse-buffer are needed now.
>>>
>>> 2. The patch is now compatible with master (commit 2e96dc639). I
>>> reverted the earlier change in folding drawers and blocks. Now, they are
>>> back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would
>>> achieve nothing when we use text properties.
>>>
>>> 3. 'invisible text property can now be nested. This is important, for
>>> example, when text inside drawers contains fontified links (which also
>>> use 'invisible text property to hide parts of the link). Now, the old
>>> 'invisible spec is recovered after unfolding.
>>
>> Interesting. I'm running out of time, so I cannot properly inspect the
>> code right now. I'll try to do that before the end of the week.
>>
>>> 4. Some outline-* function calls in org referred to outline-flag-region
>>> implementation, which is not in sync with org-flag-region in this patch.
>>> I have implemented their org-* versions and replaced the calls
>>> throughout .el files. Actually, some org-* versions were already
>>> implemented in org, but not used for some reason (or not mentioned in
>>> the manual). I have updated the relevant sections of manual. These
>>> changes might be relevant to org independently of this feature branch.
>>
>> Yes, we certainly want to move to org-specific versions in all cases.
>>
>>> 5. I have managed to get a working version of outline folding via text
>>> properties. However, that approach has a big downside - folding state
>>> cannot be different in indirect buffer when we use text properties. I
>>> have seen packages relying on this feature of org and I do not see any
>>> obvious way to achieve different folding state in indirect buffer while
>>> using text properties for outline folding.
>>
>> Hmm. Good point. This is a serious issue to consider. Even if we don't
>> use text properties for outline, this also affects drawers and blocks.
>>
>>> For now, I still used before/after-change-functions combination.
>>
>> You shouldn't.
>>
>>> I see the following problems with using only after-change-functions: 
>>>
>>> 1. They are not guaranteed to be called after every single change:
>>
>> Of course they are! See below.
>>
>>> From (elisp) Change Hooks:
>>> "... some complex primitives call ‘before-change-functions’ once before
>>> making changes, and then call ‘after-change-functions’ zero or more
>>> times"
>>
>> "zero" means there are no changes at all, so, `after-change-functions'
>> are not called, which is expected.
>>
>>> The consequence of it is a possibility that region passed to the
>>> after-change-functions is quite big (including all the singular changes,
>>> even if they are distant). This region may contain changed drawers as
>>> well and unchanged drawers and needs to be parsed to determine which
>>> drawers need to be re-folded.
>>
>> It seems you're getting it backwards. `before-change-functions' are the
>> functions being called with a possibly wide, imprecise, region to
>> handle:
>>
>>     When that happens, the arguments to ‘before-change-functions’ will
>>     enclose a region in which the individual changes are made, but won’t
>>     necessarily be the minimal such region
>>
>> however, after-change-functions calls are always minimal:
>>
>>     and the arguments to each successive call of
>>     ‘after-change-functions’ will then delimit the part of text being
>>     changed exactly.
>>
>> If you stick to `after-change-functions', there will be no such thing as
>> you describe.
>>
>>>> And, more importantly, they are not meant to be used together, i.e., you
>>>> cannot assume that a single call to `before-change-functions' always
>>>> happens before calling `after-change-functions'. This can be tricky if
>>>> you want to use the former to pass information to the latter.
>>>
>>> The fact that before-change-functions can be called multiple times
>>> before after-change-functions, is trivially solved by using buffer-local
>>> changes register (see org--modified-elements).
>>
>> Famous last words. Been there, done that, and it failed.
>>
>> Let me quote the manual:
>>
>>     In general, we advise to use either before- or the after-change
>>     hooks, but not both.
>>
>> So, let me insist: don't do that. If you don't agree with me, let's at
>> least agree with Emacs developers.
>>
>>> The register is populated by before-change-functions and cleared by
>>> after-change-functions.
>>
>> You cannot expect `after-change-functions' to clear what
>> `before-change-functions' did. This is likely to introduce pernicious
>> bugs. Sorry if it sounds like FUD, but bugs in those areas are just
>> horrible to squash.
>>
>>>> Well, `before-change-fuctions' and `after-change-functions' are not
>>>> clean at all: you modify an unrelated part of the buffer, but still call
>>>> those to check if a drawer needs to be unfolded somewhere.
>>>
>>> 2. As you pointed, instead of global before-change-functions, we can use
>>> modification-hooks text property on sensitive parts of the
>>> drawers/blocks. This would work, but I am concerned about one annoying
>>> special case:
>>>
>>> -------------------------------------------------------------------------
>>> :BLAH: <inserted outside any of the existing drawers>
>>>
>>> <some text>
>>>
>>> :DRAWER: <folded>
>>> Donec at pede.
>>> :END:
>>> -------------------------------------------------------------------------
>>> In this example, the user would not be able to unfold the folder DRAWER
>>> because it will technically become a part of a new giant BLAH drawer.
>>> This may be especially annoying if <some text> is more than one screen
>>> long and there is no easy way to identify why unfolding does not work
>>> (with point at :DRAWER:).
>>
>> You shouldn't be bothered by the case you're describing here, for
>> multiple reasons.
>>
>> First, this issue already arises in the current implementation. No one
>> bothered so far: this change is very unlikely to happen. If it becomes
>> an issue, we could make sure that `org-reveal' handles this.
>>
>> But, more importantly, we actually /want it/ as a feature. Indeed, if
>> DRAWER is expanded every time ":BLAH:" is inserted above, then inserting
>> a drawer manually would unfold /all/ drawers in the section. The user is
>> more likely to write first ":BLAH:" (everything is unfolded) then
>> ":END:" than ":END:", then ":BLAH:".
>>
>>> Because of this scenario, limiting before-change-functions to folded
>>> drawers is not sufficient. Any change in text may need to trigger
>>> unfolding.
>>
>> after-change-functions is more appropriate than before-change-functions,
>> and local parsing, as explained in this thread, is more efficient than
>> re-inventing the parser.
>>
>>> In the patch, I always register possible modifications in the
>>> blocks/drawers intersecting with the modified region + a drawer/block
>>> right next to the region.
>>>
>>> -----------------------------------------------------------------------
>>> -----------------------------------------------------------------------
>>>
>>> More details on the nested 'invisible text property implementation.
>>>
>>> The idea is to keep 'invisible property stack push and popping from it
>>> as we add/remove 'invisible text property. All the work is done in
>>> org-flag-region.
>>
>> This sounds like a good idea.
>>
>>> This was originally intended for folding outlines via text properties.
>>> Since using text properties for folding outlines is not a good idea,
>>> nested text properties have much less use.
>>
>> AFAIU, they have. You mention link fontification, but there are other
>> pieces that we could switch to text properties instead of overlays,
>> e.g., Babel hashes, narrowed table columns…
>>
>>> 3. Multiple calls to before/after-change-functions is still a problem. I
>>> am looking into following ways to reduce this number:
>>>  - reduce the number of elements registered as potentially modified
>>>    + do not add duplicates to org--modified-elements
>>>    + do not add unfolded elements to org--modified-elements
>>>    + register after-change-function as post-command hook and remove it
>>>      from global after-change-functions. This way, it will be called
>>>      twice per command only.
>>>  - determine common region containing org--modified-elements. if change
>>>    is happening within that region, there is no need to parse
>>>    drawers/blocks there again.
>>
>> This is over-engineering. Again, please focus on local changes, as
>> discussed before.
>>
>>> Recipe to have different (org-element-at-point) and
>>> (org-element-parse-buffer 'element)
>>> -------------------------------------------------------------------------
>>> <point-min>
>>> :PROPERTIES:
>>> :CREATED:  [2020-05-23 Sat 02:32]
>>> :END:
>>>
>>>
>>> <point-max>
>>> -------------------------------------------------------------------------
>>
>> I didn't look at this situation in particular, but there are cases where
>> different :post-blank values are inevitable, for example at the end of
>> a section.
>>
>> Regards,
>>
>> -- 
>> Nicolas Goaziou
>
> -- 
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-02  9:21                                   ` Ihor Radchenko
  2020-06-02  9:23                                     ` Ihor Radchenko
@ 2020-06-02  9:25                                     ` Ihor Radchenko
  2020-06-05  7:26                                     ` Nicolas Goaziou
  2 siblings, 0 replies; 55+ messages in thread
From: Ihor Radchenko @ 2020-06-02  9:25 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

Github link to the patch:
https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef 


Ihor Radchenko <yantar92@gmail.com> writes:

> Hello,
>
> [The patch itself will be provided in the following email]
>
> I have three updates from the previous version of the patch:
>
> 1. I managed to implement buffer-local text properties.
>    Now, outline folding also uses text properties without a need to give
>    up independent folding in indirect buffers.
>
> 2. The code handling modifications in folded drawers/blocks was
>    rewritten. The new code uses after-change-functions to re-hide text
>    inserted in the middle of folded regions; and text properties to
>    unfold folded drawers/blocks if one changes BEGIN/END line.
>
> 3. [experimental] Started working on improving memory and cpu footprint
>    of the old code related to folding/unfolding. org-hide-drawer-all now
>    works significantly faster because I can utilise simplified drawer
>    parser, which require a lot less memory. Overall, I managed to reduce
>    Emacs memory footprint after loading all my agenda_files twice. The
>    loading is also noticeably faster.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the buffer-local text properties:
>
> I have found char-property-alias-alist variable that controls how Emacs
> calculates text property value if the property is not set. This variable
> can be buffer-local, which allows independent 'invisible states in
> different buffers.
>
> All the implementation stays in
> org--get-buffer-local-text-property-symbol, which takes care about
> generating unique property name and mapping it to 'invisible (or any
> other) text property.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the new implementation for tracking changes:
>
> I simplified the code as suggested, without using pairs of before- and
> after-change-functions.
>
> Handling text inserted into folded/invisible region is handled by a
> simple after-change function. After testing, it turned out that simple
> re-hiding text based on 'invisible property of the text before/after the
> inserted region works pretty well.
>
> Modifications to BEGIN/END line of the drawers and blocks is handled via
> 'modification-hooks + 'insert-behind-hooks text properties (there is no
> after-change-functions analogue for text properties in Emacs). The
> property is applied during folding and the modification-hook function is
> made aware about the drawer/block boundaries (via apply-partially
> passing element containing :begin :end markers for the current
> drawer/block). Passing the element boundary is important because the
> 'modification-hook will not directly know where it belongs to. Only the
> modified region (which can be larger than the drawer) is passed to the
> function. In the worst case, the region can be the whole buffer (if one
> runs revert-buffer).
>
> It turned out that adding 'modification-hook text property takes a
> significant cpu time (partially, because we need to take care about
> possible existing 'modification-hook value, see
> org--add-to-list-text-property). For now, I decided to not clear the
> modification hooks during unfolding because of poor performance.
> However, this approach would lead to partial unfolding in the following
> case:
>
> :asd:
> :drawer:
> lksjdfksdfjl
> sdfsdfsdf
> :end:
>
> If :asd: was inserted in front of folded :drawer:, changes in :drawer:
> line of the new folded :asd: drawer would reveal the text between
> :drawer: and :end:.
>
> Let me know what you think on this.
>
>> You shouldn't be bothered by the case you're describing here, for
>> multiple reasons.
>> 
>> First, this issue already arises in the current implementation. No one
>> bothered so far: this change is very unlikely to happen. If it becomes
>> an issue, we could make sure that `org-reveal' handles this.
>> 
>> But, more importantly, we actually /want it/ as a feature. Indeed, if
>> DRAWER is expanded every time ":BLAH:" is inserted above, then inserting
>> a drawer manually would unfold /all/ drawers in the section. The user is
>> more likely to write first ":BLAH:" (everything is unfolded) then
>> ":END:" than ":END:", then ":BLAH:".
>
> Agree. This allowed me to simplify the code significantly.
>
>> It seems you're getting it backwards. `before-change-functions' are the
>> functions being called with a possibly wide, imprecise, region to
>> handle:
>> 
>>     When that happens, the arguments to ‘before-change-functions’ will
>>     enclose a region in which the individual changes are made, but won’t
>>     necessarily be the minimal such region
>> 
>> however, after-change-functions calls are always minimal:
>> 
>>     and the arguments to each successive call of
>>     ‘after-change-functions’ will then delimit the part of text being
>>     changed exactly.
>> 
>> If you stick to `after-change-functions', there will be no such thing as
>> you describe.
>
> You are right here, I missed that before-change-functions are likely to
> be called on large regions. I thought that the regions are same for
> before/after-change-functions, but after-change-functions could be
> called more than 1 time. After second thought, your vision that it is
> mostly 0 or 1 times should be the majority of cases in practice.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on reducing cpu and memory footprint of org buffers:
>
> My simplified implementation of element boundary parser
> (org--get-element-region-at-point) appears to be much faster and also
> uses much less memory in comparison with org-element-at-point.
> Moreover, not all the places where org-element-at-point is called
> actually need the full parsed element. For example, org-hide-drawer-all,
> org-hide-drawer-toggle, org-hide-block-toggle, and
> org--hide-wrapper-toggle only need element type and some information
> about the element boundaries - the information we can get from
> org--get-element-region-at-point.
>
> The following version of org-hide-drawer-all seems to work much faster
> in comparison with original:
>
> (defun org-hide-drawer-all ()
>   "Fold all drawers in the current buffer."
>   (save-excursion
>     (goto-char (point-min))
>     (while (re-search-forward org-drawer-regexp nil t)
>       (when-let* ((drawer (org--get-element-region-at-point '(property-drawer drawer)))
> 		  (type (org-element-type drawer)))
> 	(org-hide-drawer-toggle t nil drawer)
> 	;; Make sure to skip drawer entirely or we might flag it
> 	;; another time when matching its ending line with
> 	;; `org-drawer-regexp'.
> 	(goto-char (org-element-property :end drawer))))))
>
> What do you think about the idea of making use of
> org--get-element-region-at-point in org code base?
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> Further work:
>
> 1. Look into other code using overlays. Specifically,
> org-toggle-custom-properties, Babel hashes, and narrowed table columns.
>
> Best,
> Ihor
>
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Hello,
>>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>> I have five updates from the previous version of the patch:
>>
>> Thank you.
>>
>>> 1. I implemented a simplified version of element parsing to detect
>>> changes in folded drawers or blocks. No computationally expensive calls
>>> of org-element-at-point or org-element-parse-buffer are needed now.
>>>
>>> 2. The patch is now compatible with master (commit 2e96dc639). I
>>> reverted the earlier change in folding drawers and blocks. Now, they are
>>> back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would
>>> achieve nothing when we use text properties.
>>>
>>> 3. 'invisible text property can now be nested. This is important, for
>>> example, when text inside drawers contains fontified links (which also
>>> use 'invisible text property to hide parts of the link). Now, the old
>>> 'invisible spec is recovered after unfolding.
>>
>> Interesting. I'm running out of time, so I cannot properly inspect the
>> code right now. I'll try to do that before the end of the week.
>>
>>> 4. Some outline-* function calls in org referred to outline-flag-region
>>> implementation, which is not in sync with org-flag-region in this patch.
>>> I have implemented their org-* versions and replaced the calls
>>> throughout .el files. Actually, some org-* versions were already
>>> implemented in org, but not used for some reason (or not mentioned in
>>> the manual). I have updated the relevant sections of manual. These
>>> changes might be relevant to org independently of this feature branch.
>>
>> Yes, we certainly want to move to org-specific versions in all cases.
>>
>>> 5. I have managed to get a working version of outline folding via text
>>> properties. However, that approach has a big downside - folding state
>>> cannot be different in indirect buffer when we use text properties. I
>>> have seen packages relying on this feature of org and I do not see any
>>> obvious way to achieve different folding state in indirect buffer while
>>> using text properties for outline folding.
>>
>> Hmm. Good point. This is a serious issue to consider. Even if we don't
>> use text properties for outline, this also affects drawers and blocks.
>>
>>> For now, I still used before/after-change-functions combination.
>>
>> You shouldn't.
>>
>>> I see the following problems with using only after-change-functions: 
>>>
>>> 1. They are not guaranteed to be called after every single change:
>>
>> Of course they are! See below.
>>
>>> From (elisp) Change Hooks:
>>> "... some complex primitives call ‘before-change-functions’ once before
>>> making changes, and then call ‘after-change-functions’ zero or more
>>> times"
>>
>> "zero" means there are no changes at all, so, `after-change-functions'
>> are not called, which is expected.
>>
>>> The consequence of it is a possibility that region passed to the
>>> after-change-functions is quite big (including all the singular changes,
>>> even if they are distant). This region may contain changed drawers as
>>> well and unchanged drawers and needs to be parsed to determine which
>>> drawers need to be re-folded.
>>
>> It seems you're getting it backwards. `before-change-functions' are the
>> functions being called with a possibly wide, imprecise, region to
>> handle:
>>
>>     When that happens, the arguments to ‘before-change-functions’ will
>>     enclose a region in which the individual changes are made, but won’t
>>     necessarily be the minimal such region
>>
>> however, after-change-functions calls are always minimal:
>>
>>     and the arguments to each successive call of
>>     ‘after-change-functions’ will then delimit the part of text being
>>     changed exactly.
>>
>> If you stick to `after-change-functions', there will be no such thing as
>> you describe.
>>
>>>> And, more importantly, they are not meant to be used together, i.e., you
>>>> cannot assume that a single call to `before-change-functions' always
>>>> happens before calling `after-change-functions'. This can be tricky if
>>>> you want to use the former to pass information to the latter.
>>>
>>> The fact that before-change-functions can be called multiple times
>>> before after-change-functions, is trivially solved by using buffer-local
>>> changes register (see org--modified-elements).
>>
>> Famous last words. Been there, done that, and it failed.
>>
>> Let me quote the manual:
>>
>>     In general, we advise to use either before- or the after-change
>>     hooks, but not both.
>>
>> So, let me insist: don't do that. If you don't agree with me, let's at
>> least agree with Emacs developers.
>>
>>> The register is populated by before-change-functions and cleared by
>>> after-change-functions.
>>
>> You cannot expect `after-change-functions' to clear what
>> `before-change-functions' did. This is likely to introduce pernicious
>> bugs. Sorry if it sounds like FUD, but bugs in those areas are just
>> horrible to squash.
>>
>>>> Well, `before-change-fuctions' and `after-change-functions' are not
>>>> clean at all: you modify an unrelated part of the buffer, but still call
>>>> those to check if a drawer needs to be unfolded somewhere.
>>>
>>> 2. As you pointed, instead of global before-change-functions, we can use
>>> modification-hooks text property on sensitive parts of the
>>> drawers/blocks. This would work, but I am concerned about one annoying
>>> special case:
>>>
>>> -------------------------------------------------------------------------
>>> :BLAH: <inserted outside any of the existing drawers>
>>>
>>> <some text>
>>>
>>> :DRAWER: <folded>
>>> Donec at pede.
>>> :END:
>>> -------------------------------------------------------------------------
>>> In this example, the user would not be able to unfold the folder DRAWER
>>> because it will technically become a part of a new giant BLAH drawer.
>>> This may be especially annoying if <some text> is more than one screen
>>> long and there is no easy way to identify why unfolding does not work
>>> (with point at :DRAWER:).
>>
>> You shouldn't be bothered by the case you're describing here, for
>> multiple reasons.
>>
>> First, this issue already arises in the current implementation. No one
>> bothered so far: this change is very unlikely to happen. If it becomes
>> an issue, we could make sure that `org-reveal' handles this.
>>
>> But, more importantly, we actually /want it/ as a feature. Indeed, if
>> DRAWER is expanded every time ":BLAH:" is inserted above, then inserting
>> a drawer manually would unfold /all/ drawers in the section. The user is
>> more likely to write first ":BLAH:" (everything is unfolded) then
>> ":END:" than ":END:", then ":BLAH:".
>>
>>> Because of this scenario, limiting before-change-functions to folded
>>> drawers is not sufficient. Any change in text may need to trigger
>>> unfolding.
>>
>> after-change-functions is more appropriate than before-change-functions,
>> and local parsing, as explained in this thread, is more efficient than
>> re-inventing the parser.
>>
>>> In the patch, I always register possible modifications in the
>>> blocks/drawers intersecting with the modified region + a drawer/block
>>> right next to the region.
>>>
>>> -----------------------------------------------------------------------
>>> -----------------------------------------------------------------------
>>>
>>> More details on the nested 'invisible text property implementation.
>>>
>>> The idea is to keep 'invisible property stack push and popping from it
>>> as we add/remove 'invisible text property. All the work is done in
>>> org-flag-region.
>>
>> This sounds like a good idea.
>>
>>> This was originally intended for folding outlines via text properties.
>>> Since using text properties for folding outlines is not a good idea,
>>> nested text properties have much less use.
>>
>> AFAIU, they have. You mention link fontification, but there are other
>> pieces that we could switch to text properties instead of overlays,
>> e.g., Babel hashes, narrowed table columns…
>>
>>> 3. Multiple calls to before/after-change-functions is still a problem. I
>>> am looking into following ways to reduce this number:
>>>  - reduce the number of elements registered as potentially modified
>>>    + do not add duplicates to org--modified-elements
>>>    + do not add unfolded elements to org--modified-elements
>>>    + register after-change-function as post-command hook and remove it
>>>      from global after-change-functions. This way, it will be called
>>>      twice per command only.
>>>  - determine common region containing org--modified-elements. if change
>>>    is happening within that region, there is no need to parse
>>>    drawers/blocks there again.
>>
>> This is over-engineering. Again, please focus on local changes, as
>> discussed before.
>>
>>> Recipe to have different (org-element-at-point) and
>>> (org-element-parse-buffer 'element)
>>> -------------------------------------------------------------------------
>>> <point-min>
>>> :PROPERTIES:
>>> :CREATED:  [2020-05-23 Sat 02:32]
>>> :END:
>>>
>>>
>>> <point-max>
>>> -------------------------------------------------------------------------
>>
>> I didn't look at this situation in particular, but there are cases where
>> different :post-blank values are inevitable, for example at the end of
>> a section.
>>
>> Regards,
>>
>> -- 
>> Nicolas Goaziou
>
> -- 
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-02  9:23                                     ` Ihor Radchenko
@ 2020-06-02 12:10                                       ` Bastien
  2020-06-02 13:12                                         ` Ihor Radchenko
  0 siblings, 1 reply; 55+ messages in thread
From: Bastien @ 2020-06-02 12:10 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode, Nicolas Goaziou

Hi Ihor,

Ihor Radchenko <yantar92@gmail.com> writes:

> The patch (against 758b039c0) is attached.

Thanks -- just a quick note, in case you missed the message: we are in
feature free for core functionalities, so we have time to work on this 
welcome enhancement for Org 9.5, which will give us time to properly
test it too.

-- 
 Bastien


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-02 12:10                                       ` Bastien
@ 2020-06-02 13:12                                         ` Ihor Radchenko
  2020-06-02 13:23                                           ` Bastien
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-06-02 13:12 UTC (permalink / raw)
  To: Bastien; +Cc: emacs-orgmode, Nicolas Goaziou

> Thanks -- just a quick note, in case you missed the message: we are in
> feature free for core functionalities, so we have time to work on this 
> welcome enhancement for Org 9.5, which will give us time to properly
> test it too.

I do not expect it to be merged any time soon. The patch is modifying
low-level internals. It certainly needs a careful testing under various
user configs. Not to mention that so big patch will require FSF
paperwork, unless I miss something.

Best,
Ihor

Bastien <bzg@gnu.org> writes:

> Hi Ihor,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> The patch (against 758b039c0) is attached.
>
> Thanks -- just a quick note, in case you missed the message: we are in
> feature free for core functionalities, so we have time to work on this 
> welcome enhancement for Org 9.5, which will give us time to properly
> test it too.
>
> -- 
>  Bastien

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-02 13:12                                         ` Ihor Radchenko
@ 2020-06-02 13:23                                           ` Bastien
  2020-06-02 13:30                                             ` Ihor Radchenko
  0 siblings, 1 reply; 55+ messages in thread
From: Bastien @ 2020-06-02 13:23 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode, Nicolas Goaziou

Hi Ihor,

Ihor Radchenko <yantar92@gmail.com> writes:

>> Thanks -- just a quick note, in case you missed the message: we are in
>> feature free for core functionalities, so we have time to work on this 
>> welcome enhancement for Org 9.5, which will give us time to properly
>> test it too.
>
> I do not expect it to be merged any time soon. The patch is modifying
> low-level internals. It certainly needs a careful testing under various
> user configs. 

Indeed, thanks for your patience.

> Not to mention that so big patch will require FSF
> paperwork, unless I miss something.

Oh, I thought this was already done.  Do you need to submit the form
or do you wait for the FSF confirmation?

-- 
 Bastien


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-02 13:23                                           ` Bastien
@ 2020-06-02 13:30                                             ` Ihor Radchenko
  0 siblings, 0 replies; 55+ messages in thread
From: Ihor Radchenko @ 2020-06-02 13:30 UTC (permalink / raw)
  To: Bastien; +Cc: emacs-orgmode, Nicolas Goaziou

> Oh, I thought this was already done.  Do you need to submit the form
> or do you wait for the FSF confirmation?

Need to submit.

Bastien <bzg@gnu.org> writes:

> Hi Ihor,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>>> Thanks -- just a quick note, in case you missed the message: we are in
>>> feature free for core functionalities, so we have time to work on this 
>>> welcome enhancement for Org 9.5, which will give us time to properly
>>> test it too.
>>
>> I do not expect it to be merged any time soon. The patch is modifying
>> low-level internals. It certainly needs a careful testing under various
>> user configs. 
>
> Indeed, thanks for your patience.
>
>> Not to mention that so big patch will require FSF
>> paperwork, unless I miss something.
>
> Oh, I thought this was already done.  Do you need to submit the form
> or do you wait for the FSF confirmation?
>
> -- 
>  Bastien

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-02  9:21                                   ` Ihor Radchenko
  2020-06-02  9:23                                     ` Ihor Radchenko
  2020-06-02  9:25                                     ` Ihor Radchenko
@ 2020-06-05  7:26                                     ` Nicolas Goaziou
  2020-06-05  8:18                                       ` Ihor Radchenko
  2 siblings, 1 reply; 55+ messages in thread
From: Nicolas Goaziou @ 2020-06-05  7:26 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> [The patch itself will be provided in the following email]

Thank you.

> I have found char-property-alias-alist variable that controls how Emacs
> calculates text property value if the property is not set. This variable
> can be buffer-local, which allows independent 'invisible states in
> different buffers.

Great. I didn't know about this variable!

> All the implementation stays in
> org--get-buffer-local-text-property-symbol, which takes care about
> generating unique property name and mapping it to 'invisible (or any
> other) text property.

See also `gensym'. Do we really need to use it for something else than
`invisible'? If not, the tool doesn't need to be generic.

> I simplified the code as suggested, without using pairs of before- and
> after-change-functions.

Great!

> Handling text inserted into folded/invisible region is handled by a
> simple after-change function. After testing, it turned out that simple
> re-hiding text based on 'invisible property of the text before/after the
> inserted region works pretty well.

OK, but this may not be sufficient if we want to do slightly better than
overlays in that area. This is not mandatory, though.

> Modifications to BEGIN/END line of the drawers and blocks is handled via
> 'modification-hooks + 'insert-behind-hooks text properties (there is no
> after-change-functions analogue for text properties in Emacs). The
> property is applied during folding and the modification-hook function is
> made aware about the drawer/block boundaries (via apply-partially
> passing element containing :begin :end markers for the current
> drawer/block). Passing the element boundary is important because the
> 'modification-hook will not directly know where it belongs to. Only the
> modified region (which can be larger than the drawer) is passed to the
> function. In the worst case, the region can be the whole buffer (if one
> runs revert-buffer).

As discussed before, I don't think you need to use `modification-hooks'
or `insert-behind-hooks' if you already use `after-change-functions'.

`after-change-functions' are also triggered upon text properties
changes. So, what is the use case for the other hooks?

> It turned out that adding 'modification-hook text property takes a
> significant cpu time (partially, because we need to take care about
> possible existing 'modification-hook value, see
> org--add-to-list-text-property). For now, I decided to not clear the
> modification hooks during unfolding because of poor performance.
> However, this approach would lead to partial unfolding in the following
> case:
>
> :asd:
> :drawer:
> lksjdfksdfjl
> sdfsdfsdf
> :end:
>
> If :asd: was inserted in front of folded :drawer:, changes in :drawer:
> line of the new folded :asd: drawer would reveal the text between
> :drawer: and :end:.
>
> Let me know what you think on this.

I have first to understand the use case for `modification-hook'. But
I think unfolding is the right thing to do in this situation, isn't it?

> My simplified implementation of element boundary parser
> (org--get-element-region-at-point) appears to be much faster and also
> uses much less memory in comparison with org-element-at-point.
> Moreover, not all the places where org-element-at-point is called
> actually need the full parsed element. For example, org-hide-drawer-all,
> org-hide-drawer-toggle, org-hide-block-toggle, and
> org--hide-wrapper-toggle only need element type and some information
> about the element boundaries - the information we can get from
> org--get-element-region-at-point.

[...]

> What do you think about the idea of making use of
> org--get-element-region-at-point in org code base?

`org--get-element-region-at-point' is certainly faster, but it is also
wrong, unfortunately.

Org syntax is not context-free grammar. If you try to parse it locally,
starting from anywhere, it will fail at some point. For example, your
function would choke in the following case:

    [fn:1] Def1
    #+begin_something

    [fn:2] Def2
    #+end_something

AFAIK, the only proper way to parse it is to start from a known position
in the buffer. If you have no information about the buffer, the headline
above is the position you want. With cache could help to start below.
Anyway, in this particular case, you should not use
`org--get-element-region-at-point'.

Hopefully, we don't need to parse anything. In an earlier message,
I suggested a few checks to make on the modified text in order to decide
if something should be unfolded, or not. I suggest to start from there,
and fix any shortcomings we might encounter. We're replacing overlays:
low-level is good in this area.

WDYT?


Regards,

-- 
Nicolas Goaziou


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-05  7:26                                     ` Nicolas Goaziou
@ 2020-06-05  8:18                                       ` Ihor Radchenko
  2020-06-05 13:50                                         ` Nicolas Goaziou
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-06-05  8:18 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

> See also `gensym'. Do we really need to use it for something else than
> `invisible'? If not, the tool doesn't need to be generic.

For now, I also use it for buffer-local 'invisible stack. The stack is
needed to preserve folding state of drawers/blocks inside folded
outline. Though I am thinking about replacing the stack with separate
text properties, like 'invisible-outline-buffer-local +
'invisible-drawer-buffer-local + 'invisible-block-buffer-local.
Maintaining stack takes a noticeable percentage of CPU time in profiler.

org--get-buffer-local-text-property-symbol must take care about
situation with indirect buffers. When an indirect buffer is created from
some org buffer, the old value of char-property-alias-alist is carried
over. We need to detect this case and create new buffer-local symbol,
which is unique to the newly created buffer (but not create it if the
buffer-local property is already there). Then, the new symbol must
replace the old alias in char-property-alias-alist + old folding state
must be preserved (via copying the old invisibility specs into the new
buffer-local text property). I do not see how gensym can benefit this
logic.

> OK, but this may not be sufficient if we want to do slightly better than
> overlays in that area. This is not mandatory, though.

Could you elaborate on what can be "slightly better"? 

> As discussed before, I don't think you need to use `modification-hooks'
> or `insert-behind-hooks' if you already use `after-change-functions'.
>
> `after-change-functions' are also triggered upon text properties
> changes. So, what is the use case for the other hooks?

The problem is that `after-change-functions' cannot be a text property.
Only `modification-hooks' and `insert-in-front/behind-hooks' can be a
valid text property. If we use `after-change-functions', they will
always be triggered, regardless if the change was made inside or outside
folded region.

>> :asd:
>> :drawer:
>> lksjdfksdfjl
>> sdfsdfsdf
>> :end:
>>
>> If :asd: was inserted in front of folded :drawer:, changes in :drawer:
>> line of the new folded :asd: drawer would reveal the text between
>> :drawer: and :end:.
>>
>> Let me know what you think on this.

> I have first to understand the use case for `modification-hook'. But
> I think unfolding is the right thing to do in this situation, isn't it?

That situation arises because the modification-hooks from ":drawer:"
(they are set via text properties) only have information about the
:drawer:...:end: drawer before the modifications (they were set when
:drawer: was folded last time). So, they will only unfold a part of the
new :asd: drawer. I do not see a simple way to unfold everything without
re-parsing the drawer around the changed text.

Actually, I am quite unhappy with the performance of modification-hooks
set via text properties (I am using this patch on my Emacs during this
week). It appears that setting the text properties costs a significant
CPU time in practice, even though running the hooks is pretty fast.
I will think about a way to handle modifications using global
after-change-functions.

> `org--get-element-region-at-point' is certainly faster, but it is also
> wrong, unfortunately.
>
> Org syntax is not context-free grammar. If you try to parse it locally,
> starting from anywhere, it will fail at some point. For example, your
> function would choke in the following case:
>
>     [fn:1] Def1
>     #+begin_something
>
>     [fn:2] Def2
>     #+end_something

I see. 

> AFAIK, the only proper way to parse it is to start from a known position
> in the buffer. If you have no information about the buffer, the headline
> above is the position you want. With cache could help to start below.
> Anyway, in this particular case, you should not use
> `org--get-element-region-at-point'.

OK

Best,
Ihor

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> [The patch itself will be provided in the following email]
>
> Thank you.
>
>> I have found char-property-alias-alist variable that controls how Emacs
>> calculates text property value if the property is not set. This variable
>> can be buffer-local, which allows independent 'invisible states in
>> different buffers.
>
> Great. I didn't know about this variable!
>
>> All the implementation stays in
>> org--get-buffer-local-text-property-symbol, which takes care about
>> generating unique property name and mapping it to 'invisible (or any
>> other) text property.
>
> See also `gensym'. Do we really need to use it for something else than
> `invisible'? If not, the tool doesn't need to be generic.
>
>> I simplified the code as suggested, without using pairs of before- and
>> after-change-functions.
>
> Great!
>
>> Handling text inserted into folded/invisible region is handled by a
>> simple after-change function. After testing, it turned out that simple
>> re-hiding text based on 'invisible property of the text before/after the
>> inserted region works pretty well.
>
> OK, but this may not be sufficient if we want to do slightly better than
> overlays in that area. This is not mandatory, though.
>
>> Modifications to BEGIN/END line of the drawers and blocks is handled via
>> 'modification-hooks + 'insert-behind-hooks text properties (there is no
>> after-change-functions analogue for text properties in Emacs). The
>> property is applied during folding and the modification-hook function is
>> made aware about the drawer/block boundaries (via apply-partially
>> passing element containing :begin :end markers for the current
>> drawer/block). Passing the element boundary is important because the
>> 'modification-hook will not directly know where it belongs to. Only the
>> modified region (which can be larger than the drawer) is passed to the
>> function. In the worst case, the region can be the whole buffer (if one
>> runs revert-buffer).
>
> As discussed before, I don't think you need to use `modification-hooks'
> or `insert-behind-hooks' if you already use `after-change-functions'.
>
> `after-change-functions' are also triggered upon text properties
> changes. So, what is the use case for the other hooks?
>
>> It turned out that adding 'modification-hook text property takes a
>> significant cpu time (partially, because we need to take care about
>> possible existing 'modification-hook value, see
>> org--add-to-list-text-property). For now, I decided to not clear the
>> modification hooks during unfolding because of poor performance.
>> However, this approach would lead to partial unfolding in the following
>> case:
>>
>> :asd:
>> :drawer:
>> lksjdfksdfjl
>> sdfsdfsdf
>> :end:
>>
>> If :asd: was inserted in front of folded :drawer:, changes in :drawer:
>> line of the new folded :asd: drawer would reveal the text between
>> :drawer: and :end:.
>>
>> Let me know what you think on this.
>
> I have first to understand the use case for `modification-hook'. But
> I think unfolding is the right thing to do in this situation, isn't it?
>
>> My simplified implementation of element boundary parser
>> (org--get-element-region-at-point) appears to be much faster and also
>> uses much less memory in comparison with org-element-at-point.
>> Moreover, not all the places where org-element-at-point is called
>> actually need the full parsed element. For example, org-hide-drawer-all,
>> org-hide-drawer-toggle, org-hide-block-toggle, and
>> org--hide-wrapper-toggle only need element type and some information
>> about the element boundaries - the information we can get from
>> org--get-element-region-at-point.
>
> [...]
>
>> What do you think about the idea of making use of
>> org--get-element-region-at-point in org code base?
>
> `org--get-element-region-at-point' is certainly faster, but it is also
> wrong, unfortunately.
>
> Org syntax is not context-free grammar. If you try to parse it locally,
> starting from anywhere, it will fail at some point. For example, your
> function would choke in the following case:
>
>     [fn:1] Def1
>     #+begin_something
>
>     [fn:2] Def2
>     #+end_something
>
> AFAIK, the only proper way to parse it is to start from a known position
> in the buffer. If you have no information about the buffer, the headline
> above is the position you want. With cache could help to start below.
> Anyway, in this particular case, you should not use
> `org--get-element-region-at-point'.
>
> Hopefully, we don't need to parse anything. In an earlier message,
> I suggested a few checks to make on the modified text in order to decide
> if something should be unfolded, or not. I suggest to start from there,
> and fix any shortcomings we might encounter. We're replacing overlays:
> low-level is good in this area.
>
> WDYT?
>
>
> Regards,
>
> -- 
> Nicolas Goaziou

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-05  8:18                                       ` Ihor Radchenko
@ 2020-06-05 13:50                                         ` Nicolas Goaziou
  2020-06-08  5:05                                           ` Ihor Radchenko
  0 siblings, 1 reply; 55+ messages in thread
From: Nicolas Goaziou @ 2020-06-05 13:50 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Ihor Radchenko <yantar92@gmail.com> writes:

>> See also `gensym'. Do we really need to use it for something else than
>> `invisible'? If not, the tool doesn't need to be generic.
>
> For now, I also use it for buffer-local 'invisible stack. The stack is
> needed to preserve folding state of drawers/blocks inside folded
> outline. Though I am thinking about replacing the stack with separate
> text properties, like 'invisible-outline-buffer-local +
> 'invisible-drawer-buffer-local + 'invisible-block-buffer-local.
> Maintaining stack takes a noticeable percentage of CPU time in profiler.
>
> org--get-buffer-local-text-property-symbol must take care about
> situation with indirect buffers. When an indirect buffer is created from
> some org buffer, the old value of char-property-alias-alist is carried
> over. We need to detect this case and create new buffer-local symbol,
> which is unique to the newly created buffer (but not create it if the
> buffer-local property is already there). Then, the new symbol must
> replace the old alias in char-property-alias-alist + old folding state
> must be preserved (via copying the old invisibility specs into the new
> buffer-local text property). I do not see how gensym can benefit this
> logic.

`gensym' is just a shorter, and somewhat standard way, to create a new
uninterned symbol with a given prefix. You seem to re-invent it. What
you do with that new symbol is orthogonal to that suggestion, of course.

>> OK, but this may not be sufficient if we want to do slightly better than
>> overlays in that area. This is not mandatory, though.
>
> Could you elaborate on what can be "slightly better"? 

IIRC, I gave examples of finer control of folding state after a change.
Consider this _folded_ drawer:

  :BEGIN:
  Foo
  :END:

Inserting ":END" in it should not unfold it, as it is currently the case
with overlays,

  :BEGIN
  Foo
  :END
  :END:

but a soon as the last ":" is inserted, the initial drawer could be
expanded.

  :BEGIN
  Foo
  :END:
  :END:

The latter case is not currently handled by overlays. This is what
I call "slightly better".

Also, note that this change is not related to opening and closing lines
of the initial drawer, so sticking text properties on them would not
help here.

Another case is modifying those borders, e.g.,


  :BEGIN:               :BEGIN:
  Foo         ------>   Foo  
  :END:                 :ND:

which should expand the drawer. Your implementation catches this, but
I'm pointing out that current implementation with overlays does not.
Even though that's not strictly required for compatibility with
overlays, it is a welcome slight improvement.

>> As discussed before, I don't think you need to use `modification-hooks'
>> or `insert-behind-hooks' if you already use `after-change-functions'.
>>
>> `after-change-functions' are also triggered upon text properties
>> changes. So, what is the use case for the other hooks?
>
> The problem is that `after-change-functions' cannot be a text property.
> Only `modification-hooks' and `insert-in-front/behind-hooks' can be a
> valid text property. If we use `after-change-functions', they will
> always be triggered, regardless if the change was made inside or outside
> folded region.

As discussed, text properties are local to the change, but require extra
care when moving text around. You also observed serious overhead when
using them.

OTOH, even if `a-c-f' is not local, you can quickly determine if the
change altered a folded element, so the overhead is limited, i.e.,
mostly checking for a text property at a given buffer position.

To be clear, I initially thought that text properties were a superior
choice, but I changed my mind a while ago, and I thought you had, too.
IOW, `after-change-functions' is the way to go, since you have no strong
reason to stick to text properties for this kind of function.

>>> :asd:
>>> :drawer:
>>> lksjdfksdfjl
>>> sdfsdfsdf
>>> :end:
>>>
>>> If :asd: was inserted in front of folded :drawer:, changes in :drawer:
>>> line of the new folded :asd: drawer would reveal the text between
>>> :drawer: and :end:.
>>>
>>> Let me know what you think on this.
>
>> I have first to understand the use case for `modification-hook'. But
>> I think unfolding is the right thing to do in this situation, isn't it?
>
> That situation arises because the modification-hooks from ":drawer:"
> (they are set via text properties) only have information about the
> :drawer:...:end: drawer before the modifications (they were set when
> :drawer: was folded last time). So, they will only unfold a part of the
> new :asd: drawer. I do not see a simple way to unfold everything without
> re-parsing the drawer around the changed text.

Oh! I misread your message. I withdraw what I wrote. In this case, we
don't want to unfold anything. The situation is not worse than what we
have now, and trying to fix it would have repercussions down in the
buffer, e.g., expanding drawers screen below.

As a rule of thumb, I think we can pay attention to changes in the
folded text, and its immediate surroundings (e.g., the opening line,
which is not folded), but no further.

As written above, slight changes are welcome, but let's not go overboard
and parse a whole section just to know if we can expand a drawer.

> Actually, I am quite unhappy with the performance of modification-hooks
> set via text properties (I am using this patch on my Emacs during this
> week). It appears that setting the text properties costs a significant
> CPU time in practice, even though running the hooks is pretty fast.
> I will think about a way to handle modifications using global
> after-change-functions.

That's better, IMO.

I gave you a few ideas to quickly check if a change requires expansion,
in an earlier mail. I suggest to start out from that. Let me know if you
have questions about it.


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-05 13:50                                         ` Nicolas Goaziou
@ 2020-06-08  5:05                                           ` Ihor Radchenko
  2020-06-08  5:06                                             ` Ihor Radchenko
                                                               ` (2 more replies)
  0 siblings, 3 replies; 55+ messages in thread
From: Ihor Radchenko @ 2020-06-08  5:05 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

Hello,

[The patch itself will be provided in the following email]

I have four more updates from the previous version of the patch:

1. All the code handling modifications in folded drawers/blocks is moved
   to after-change-function. It works as follows:
   - if any text is inserted in the middle of hidden region, that text
     is also hidden;
   - if BEGIN/END line of a folded drawer do not match org-drawer-regexp
     and org-property-end-re, unfold it; 
   - if org-property-end-re or new org-outline-regexp-bol is inserted in
     the middle of the drawer, unfold it;
   - the same logic for blocks.

2. The text property stack is rewritten using char-property-alias-alist.
   This is faster in comparison with previous approach, which involved
   modifying all the text properties every timer org-flag-region was
   called. 
   
3. org-toggle-custom-properties-visibility is rewritten using text
   properties. I also took a freedom to implement a new feature here.
   Now, setting new `org-custom-properties-hide-emptied-drawers' to
   non-nil will result in hiding the whole property drawer if it
   contains only org-custom-properties.

4. This patch should work against 1aa095ccf. However, the merge was not
   trivial here. Recent commits actively used the fact that drawers and
   outlines are hidden via 'outline invisibility spec, which is not the
   case in this branch. I am not confident that I did not break anything
   during the merge, especially 1aa095ccf.

-----------------------------------------------------------------------
-----------------------------------------------------------------------

More details on the new implementation for tracking changes:

> I gave you a few ideas to quickly check if a change requires expansion,
> in an earlier mail. I suggest to start out from that. Let me know if you
> have questions about it.

All the code lives in org-after-change-function. I tried to incorporate
the earlier Nicholas' suggestions, except the parts related to
intersecting blocks and drawers. I am not sure if I understand the
parsing priority of blocks vs. drawers.

-----------------------------------------------------------------------
-----------------------------------------------------------------------

More details on the text property stack:

The earlier version of the code literally used stack to save
pre-existing 'invisibility specs in org-flag-region. This was done on
every invocation of org-flag-region, which made org-flag-region
significantly slower. I re-implemented the same feature using
char-property-alias-alist. Now, different invisibility specs live in
separate text properties and can be safely modified independently. The
specs are applied according to org--invisible-spec-priority-list. A side
effect of current implementation is that char-property-alias-alist is
fully controlled by org. All the pre-existing settings for 'invisible
text property will be overwritten by org.

> `gensym' is just a shorter, and somewhat standard way, to create a new
> uninterned symbol with a given prefix. You seem to re-invent it. What
> you do with that new symbol is orthogonal to that suggestion, of course.

I do not think that `gensym' is suitable here. We don't want a new
symbol every time org--get-buffer-local-invisible-property-symbol is
called. It should return the same symbol if it is called from the same
buffer multiple times.

-----------------------------------------------------------------------
-----------------------------------------------------------------------

More details on the org-toggle-custom-properties-visibility:

The implementation showcases how to introduce new invisibility specs to
org. Apart from expected (add-to-invisibility-spec 'org-hide-custom-property) 
one also needs to add the spec into org--invisible-spec-priority-list:

(add-to-list 'org--invisible-spec-priority-list 'org-hide-custom-property)

Searching for text with the given invisibility spec is done as
follows:

(text-property-search-forward (org--get-buffer-local-invisible-property-symbol 'org-hide-custom-property) 'org-hide-custom-property t)

This last piece of code is probably not the most elegant. I am thinking
if creating some higher-level interface would be more reasonable here.
What do you think?


The new customisation `org-custom-properties-hide-emptied-drawers'
sounds logical for me since empty property drawers left after invoking
org-toggle-custom-properties-visibility are rather useless according to
my experience. If one already wants to hide parts of property drawers, I
do not see a reason to show leftover

:PROPERTIES:
:END:

-----------------------------------------------------------------------
-----------------------------------------------------------------------

More details on the merge with the latest master:

I tried my best to not break anything. However, I am not sure if I
understand all the recent commits. Could someone take a look if there is
anything suspicious in org-next-visible-heading?

Also, I have seen some optimisations making use of the fact that drawers
and headlines both use 'outline invisibility spec. This change in the
implementation details supposed to improve performance and should not be
necessary if this patch is going to be merged. Would it be possible to
refrain from abusing this particular implementation detail in the
nearest commits on master (unless really necessary)?

-----------------------------------------------------------------------
-----------------------------------------------------------------------

Further work:

I would like to finalise the current patch and work on other code using
overlays separately. This patch is already quite complicated as is. I do
not want to introduce even more potential bugs by working on things not
directly affected by this version of the patch.

Best,
Ihor


Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>>> See also `gensym'. Do we really need to use it for something else than
>>> `invisible'? If not, the tool doesn't need to be generic.
>>
>> For now, I also use it for buffer-local 'invisible stack. The stack is
>> needed to preserve folding state of drawers/blocks inside folded
>> outline. Though I am thinking about replacing the stack with separate
>> text properties, like 'invisible-outline-buffer-local +
>> 'invisible-drawer-buffer-local + 'invisible-block-buffer-local.
>> Maintaining stack takes a noticeable percentage of CPU time in profiler.
>>
>> org--get-buffer-local-text-property-symbol must take care about
>> situation with indirect buffers. When an indirect buffer is created from
>> some org buffer, the old value of char-property-alias-alist is carried
>> over. We need to detect this case and create new buffer-local symbol,
>> which is unique to the newly created buffer (but not create it if the
>> buffer-local property is already there). Then, the new symbol must
>> replace the old alias in char-property-alias-alist + old folding state
>> must be preserved (via copying the old invisibility specs into the new
>> buffer-local text property). I do not see how gensym can benefit this
>> logic.
>
> `gensym' is just a shorter, and somewhat standard way, to create a new
> uninterned symbol with a given prefix. You seem to re-invent it. What
> you do with that new symbol is orthogonal to that suggestion, of course.
>
>>> OK, but this may not be sufficient if we want to do slightly better than
>>> overlays in that area. This is not mandatory, though.
>>
>> Could you elaborate on what can be "slightly better"? 
>
> IIRC, I gave examples of finer control of folding state after a change.
> Consider this _folded_ drawer:
>
>   :BEGIN:
>   Foo
>   :END:
>
> Inserting ":END" in it should not unfold it, as it is currently the case
> with overlays,
>
>   :BEGIN
>   Foo
>   :END
>   :END:
>
> but a soon as the last ":" is inserted, the initial drawer could be
> expanded.
>
>   :BEGIN
>   Foo
>   :END:
>   :END:
>
> The latter case is not currently handled by overlays. This is what
> I call "slightly better".
>
> Also, note that this change is not related to opening and closing lines
> of the initial drawer, so sticking text properties on them would not
> help here.
>
> Another case is modifying those borders, e.g.,
>
>
>   :BEGIN:               :BEGIN:
>   Foo         ------>   Foo  
>   :END:                 :ND:
>
> which should expand the drawer. Your implementation catches this, but
> I'm pointing out that current implementation with overlays does not.
> Even though that's not strictly required for compatibility with
> overlays, it is a welcome slight improvement.
>
>>> As discussed before, I don't think you need to use `modification-hooks'
>>> or `insert-behind-hooks' if you already use `after-change-functions'.
>>>
>>> `after-change-functions' are also triggered upon text properties
>>> changes. So, what is the use case for the other hooks?
>>
>> The problem is that `after-change-functions' cannot be a text property.
>> Only `modification-hooks' and `insert-in-front/behind-hooks' can be a
>> valid text property. If we use `after-change-functions', they will
>> always be triggered, regardless if the change was made inside or outside
>> folded region.
>
> As discussed, text properties are local to the change, but require extra
> care when moving text around. You also observed serious overhead when
> using them.
>
> OTOH, even if `a-c-f' is not local, you can quickly determine if the
> change altered a folded element, so the overhead is limited, i.e.,
> mostly checking for a text property at a given buffer position.
>
> To be clear, I initially thought that text properties were a superior
> choice, but I changed my mind a while ago, and I thought you had, too.
> IOW, `after-change-functions' is the way to go, since you have no strong
> reason to stick to text properties for this kind of function.
>
>>>> :asd:
>>>> :drawer:
>>>> lksjdfksdfjl
>>>> sdfsdfsdf
>>>> :end:
>>>>
>>>> If :asd: was inserted in front of folded :drawer:, changes in :drawer:
>>>> line of the new folded :asd: drawer would reveal the text between
>>>> :drawer: and :end:.
>>>>
>>>> Let me know what you think on this.
>>
>>> I have first to understand the use case for `modification-hook'. But
>>> I think unfolding is the right thing to do in this situation, isn't it?
>>
>> That situation arises because the modification-hooks from ":drawer:"
>> (they are set via text properties) only have information about the
>> :drawer:...:end: drawer before the modifications (they were set when
>> :drawer: was folded last time). So, they will only unfold a part of the
>> new :asd: drawer. I do not see a simple way to unfold everything without
>> re-parsing the drawer around the changed text.
>
> Oh! I misread your message. I withdraw what I wrote. In this case, we
> don't want to unfold anything. The situation is not worse than what we
> have now, and trying to fix it would have repercussions down in the
> buffer, e.g., expanding drawers screen below.
>
> As a rule of thumb, I think we can pay attention to changes in the
> folded text, and its immediate surroundings (e.g., the opening line,
> which is not folded), but no further.
>
> As written above, slight changes are welcome, but let's not go overboard
> and parse a whole section just to know if we can expand a drawer.
>
>> Actually, I am quite unhappy with the performance of modification-hooks
>> set via text properties (I am using this patch on my Emacs during this
>> week). It appears that setting the text properties costs a significant
>> CPU time in practice, even though running the hooks is pretty fast.
>> I will think about a way to handle modifications using global
>> after-change-functions.
>
> That's better, IMO.
>
> I gave you a few ideas to quickly check if a change requires expansion,
> in an earlier mail. I suggest to start out from that. Let me know if you
> have questions about it.

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-08  5:05                                           ` Ihor Radchenko
@ 2020-06-08  5:06                                             ` Ihor Radchenko
  2020-06-08  5:08                                             ` Ihor Radchenko
  2020-06-10 17:14                                             ` Nicolas Goaziou
  2 siblings, 0 replies; 55+ messages in thread
From: Ihor Radchenko @ 2020-06-08  5:06 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 44 bytes --]

The patch (against 1aa095ccf) is attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: featuredrawertextprop-20200608.patch --]
[-- Type: text/x-diff, Size: 50930 bytes --]

diff --git a/contrib/lisp/org-notify.el b/contrib/lisp/org-notify.el
index 9f8677871..ab470ea9b 100644
--- a/contrib/lisp/org-notify.el
+++ b/contrib/lisp/org-notify.el
@@ -246,7 +246,7 @@ seconds.  The default value for SECS is 20."
           (switch-to-buffer (find-file-noselect file))
           (org-with-wide-buffer
            (goto-char begin)
-           (outline-show-entry))
+           (org-show-entry))
           (goto-char begin)
           (search-forward "DEADLINE: <")
           (search-forward ":")
diff --git a/contrib/lisp/org-velocity.el b/contrib/lisp/org-velocity.el
index bfc4d6c3e..2312b235c 100644
--- a/contrib/lisp/org-velocity.el
+++ b/contrib/lisp/org-velocity.el
@@ -325,7 +325,7 @@ use it."
   (save-excursion
     (when narrow
       (org-narrow-to-subtree))
-    (outline-show-all)))
+    (org-show-all)))
 
 (defun org-velocity-edit-entry/inline (heading)
   "Edit entry at HEADING in the original buffer."
diff --git a/doc/org-manual.org b/doc/org-manual.org
index efad195e1..c6f167eac 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -509,11 +509,11 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
   Switch back to the startup visibility of the buffer (see [[*Initial
   visibility]]).
 
-- {{{kbd(C-u C-u C-u TAB)}}} (~outline-show-all~) ::
+- {{{kbd(C-u C-u C-u TAB)}}} (~org-show-all~) ::
 
   #+cindex: show all, command
   #+kindex: C-u C-u C-u TAB
-  #+findex: outline-show-all
+  #+findex: org-show-all
   Show all, including drawers.
 
 - {{{kbd(C-c C-r)}}} (~org-reveal~) ::
@@ -529,18 +529,18 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
   headings.  With a double prefix argument, also show the entire
   subtree of the parent.
 
-- {{{kbd(C-c C-k)}}} (~outline-show-branches~) ::
+- {{{kbd(C-c C-k)}}} (~org-show-branches~) ::
 
   #+cindex: show branches, command
   #+kindex: C-c C-k
-  #+findex: outline-show-branches
+  #+findex: org-show-branches
   Expose all the headings of the subtree, but not their bodies.
 
-- {{{kbd(C-c TAB)}}} (~outline-show-children~) ::
+- {{{kbd(C-c TAB)}}} (~org-show-children~) ::
 
   #+cindex: show children, command
   #+kindex: C-c TAB
-  #+findex: outline-show-children
+  #+findex: org-show-children
   Expose all direct children of the subtree.  With a numeric prefix
   argument {{{var(N)}}}, expose all children down to level
   {{{var(N)}}}.
@@ -7294,7 +7294,7 @@ its location in the outline tree, but behaves in the following way:
   command (see [[*Visibility Cycling]]).  You can force cycling archived
   subtrees with {{{kbd(C-TAB)}}}, or by setting the option
   ~org-cycle-open-archived-trees~.  Also normal outline commands, like
-  ~outline-show-all~, open archived subtrees.
+  ~org-show-all~, open archived subtrees.
 
 -
   #+vindex: org-sparse-tree-open-archived-trees
diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index 9fbeb2a1e..2f121f743 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -6824,7 +6824,7 @@ and stored in the variable `org-prefix-format-compiled'."
 	    (t "  %-12:c%?-12t% s")))
 	(start 0)
 	varform vars var e c f opt)
-    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)"
+    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)"
 			 s start)
       (setq var (or (cdr (assoc (match-string 4 s)
 				'(("c" . category) ("t" . time) ("l" . level) ("s" . extra)
@@ -9136,20 +9136,20 @@ if it was hidden in the outline."
      ((and (called-interactively-p 'any) (= more 1))
       (message "Remote: show with default settings"))
      ((= more 2)
-      (outline-show-entry)
+      (org-show-entry)
       (org-show-children)
       (save-excursion
 	(org-back-to-heading)
 	(run-hook-with-args 'org-cycle-hook 'children))
       (message "Remote: CHILDREN"))
      ((= more 3)
-      (outline-show-subtree)
+      (org-show-subtree)
       (save-excursion
 	(org-back-to-heading)
 	(run-hook-with-args 'org-cycle-hook 'subtree))
       (message "Remote: SUBTREE"))
      ((> more 3)
-      (outline-show-subtree)
+      (org-show-subtree)
       (message "Remote: SUBTREE AND ALL DRAWERS")))
     (select-window win)))
 
diff --git a/lisp/org-archive.el b/lisp/org-archive.el
index d3e12d17b..d864dad8a 100644
--- a/lisp/org-archive.el
+++ b/lisp/org-archive.el
@@ -330,7 +330,7 @@ direct children of this heading."
 		      (insert (if datetree-date "" "\n") heading "\n")
 		      (end-of-line 0))
 		    ;; Make the subtree visible
-		    (outline-show-subtree)
+		    (org-show-subtree)
 		    (if org-archive-reversed-order
 			(progn
 			  (org-back-to-heading t)
diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index e50a4d7c8..e656df555 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -699,7 +699,7 @@ FUN is a function called with no argument."
 			  (move-beginning-of-line 2)
 			  (org-at-heading-p t)))))
     (unwind-protect (funcall fun)
-      (when hide-body (outline-hide-entry)))))
+      (when hide-body (org-hide-entry)))))
 
 (defun org-columns-previous-allowed-value ()
   "Switch to the previous allowed value for this column."
diff --git a/lisp/org-compat.el b/lisp/org-compat.el
index 5953f89d2..09a09472a 100644
--- a/lisp/org-compat.el
+++ b/lisp/org-compat.el
@@ -138,12 +138,8 @@ This is a floating point number if the size is too large for an integer."
 ;;; Emacs < 25.1 compatibility
 
 (when (< emacs-major-version 25)
-  (defalias 'outline-hide-entry 'hide-entry)
-  (defalias 'outline-hide-sublevels 'hide-sublevels)
-  (defalias 'outline-hide-subtree 'hide-subtree)
   (defalias 'outline-show-branches 'show-branches)
   (defalias 'outline-show-children 'show-children)
-  (defalias 'outline-show-entry 'show-entry)
   (defalias 'outline-show-subtree 'show-subtree)
   (defalias 'xref-find-definitions 'find-tag)
   (defalias 'format-message 'format)
@@ -644,7 +640,7 @@ When optional argument ELEMENT is a parsed drawer, as returned by
 When buffer positions BEG and END are provided, hide or show that
 region as a drawer without further ado."
   (declare (obsolete "use `org-hide-drawer-toggle' instead." "Org 9.4"))
-  (if (and beg end) (org-flag-region beg end flag 'outline)
+  (if (and beg end) (org-flag-region beg end flag 'org-hide-drawer)
     (let ((drawer
 	   (or element
 	       (and (save-excursion
@@ -658,7 +654,7 @@ region as a drawer without further ado."
 	   (save-excursion (goto-char (org-element-property :end drawer))
 			   (skip-chars-backward " \t\n")
 			   (line-end-position))
-	   flag 'outline)
+	   flag 'org-hide-drawer)
 	  ;; When the drawer is hidden away, make sure point lies in
 	  ;; a visible part of the buffer.
 	  (when (invisible-p (max (1- (point)) (point-min)))
diff --git a/lisp/org-element.el b/lisp/org-element.el
index ac41b7650..2d5c8d771 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -4320,7 +4320,7 @@ element or object.  Meaningful values are `first-section',
 TYPE is the type of the current element or object.
 
 If PARENT? is non-nil, assume the next element or object will be
-located inside the current one.  "
+located inside the current one."
   (if parent?
       (pcase type
 	(`headline 'section)
diff --git a/lisp/org-keys.el b/lisp/org-keys.el
index 37df29983..a714dec0f 100644
--- a/lisp/org-keys.el
+++ b/lisp/org-keys.el
@@ -437,7 +437,7 @@ COMMANDS is a list of alternating OLDDEF NEWDEF command names."
   #'org-next-visible-heading)
 (define-key org-mode-map [remap outline-previous-visible-heading]
   #'org-previous-visible-heading)
-(define-key org-mode-map [remap show-children] #'org-show-children)
+(define-key org-mode-map [remap outline-show-children] #'org-show-children)
 
 ;;;; Make `C-c C-x' a prefix key
 (org-defkey org-mode-map (kbd "C-c C-x") (make-sparse-keymap))
diff --git a/lisp/org-macs.el b/lisp/org-macs.el
index a02f713ca..b17c0cb4d 100644
--- a/lisp/org-macs.el
+++ b/lisp/org-macs.el
@@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead."
 
 
 \f
-;;; Overlays
+;;; Overlays and text properties
 
 (defun org-overlay-display (ovl text &optional face evap)
   "Make overlay OVL display TEXT with face FACE."
@@ -705,26 +705,126 @@ If DELETE is non-nil, delete all those overlays."
 	    (delete (delete-overlay ov))
 	    (t (push ov found))))))
 
+(defun org-remove-text-properties (start end properties &optional object)
+  "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions.
+Do not remove invisible text properties specified by 'outline,
+'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this
+is needed to keep outlines, drawers, and blocks hidden unless they are
+toggled by user.
+Note: The below may be too specific and create troubles if more
+invisibility specs are added to org in future"
+  (when (plist-member properties 'invisible)
+    (let ((pos start)
+	  next spec)
+      (while (< pos end)
+	(setq next (next-single-property-change pos 'invisible nil end)
+              spec (get-text-property pos 'invisible))
+	(unless (memq spec (list 'org-hide-block
+				 'org-hide-drawer
+				 'outline))
+          (remove-text-properties pos next '(invisible nil) object))
+	(setq pos next))))
+  (when-let ((properties-stripped (org-plist-delete properties 'invisible)))
+    (remove-text-properties start end properties-stripped object)))
+
+(defun org--find-text-property-region (pos prop)
+  "Find a region containing PROP text property around point POS."
+  (let* ((beg (and (get-text-property pos prop) pos))
+	 (end beg))
+    (when beg
+      ;; when beg is the first point in the region, `previous-single-property-change'
+      ;; will return nil.
+      (setq beg (or (previous-single-property-change pos prop)
+		    beg))
+      ;; when end is the last point in the region, `next-single-property-change'
+      ;; will return nil.
+      (setq end (or (next-single-property-change pos prop)
+		    end))
+      (unless (= beg end) ; this should not happen
+        (cons beg end)))))
+
+(defun org--add-to-list-text-property (from to prop element)
+  "Add element to text property PROP, whos value should be a list."
+  (add-text-properties from to `(,prop ,(list element))) ; create if none
+  ;; add to existing
+  (alter-text-property from to
+		       prop
+		       (lambda (val)
+			 (if (member element val)
+                             val
+			   (cons element val)))))
+
+(defun org--remove-from-list-text-property (from to prop element)
+  "Remove ELEMENT from text propery PROP, whos value should be a list."
+  (let ((pos from))
+    (while (< pos to)
+      (when-let ((val (get-text-property pos prop)))
+	(if (equal val (list element))
+	    (remove-text-properties pos (next-single-char-property-change pos prop nil to) (list prop nil))
+	  (put-text-property pos (next-single-char-property-change pos prop nil to)
+			     prop (remove element (get-text-property pos prop)))))
+      (setq pos (next-single-char-property-change pos prop nil to)))))
+
+(defvar org--invisible-spec-priority-list '(outline org-hide-drawer org-hide-block)
+  "Priority of invisibility specs.")
+
+(defun org--get-buffer-local-invisible-property-symbol (spec &optional buffer return-only)
+  "Return unique symbol suitable to be used as buffer-local in BUFFER for 'invisible SPEC.
+If the buffer already have buffer-local setup in `char-property-alias-alist'
+and the setup appears to be created for different buffer,
+copy the old invisibility state into new buffer-local text properties,
+unless RETURN-ONLY is non-nil."
+  (if (not (member spec org--invisible-spec-priority-list))
+      (user-error "%s should be a valid invisibility spec" spec)
+    (let* ((buf (or buffer (current-buffer))))
+      (let ((local-prop (intern (format "org--invisible-%s-buffer-local-%S"
+					(symbol-name spec)
+					;; (sxhash buf) appears to be not constant over time.
+					;; Using buffer-name is safe, since the only place where
+					;; buffer-local text property actually matters is an indirect
+					;; buffer, where the name cannot be same anyway.
+					(sxhash (buffer-name buf))))))
+        (prog1
+            local-prop
+          (unless return-only
+	    (with-current-buffer buf
+	      (unless (member local-prop (alist-get 'invisible char-property-alias-alist))
+		;; copy old property
+		(dolist (old-prop (alist-get 'invisible char-property-alias-alist))
+		  (org-with-wide-buffer
+		   (let* ((pos (point-min))
+			  (spec (seq-find (lambda (spec)
+					    (string-match-p (symbol-name spec)
+							    (symbol-name old-prop)))
+					  org--invisible-spec-priority-list))
+			  (new-prop (org--get-buffer-local-invisible-property-symbol spec nil 'return-only)))
+		     (while (< pos (point-max))
+		       (when-let (val (get-text-property pos old-prop))
+			 (put-text-property pos (next-single-char-property-change pos old-prop) new-prop val))
+		       (setq pos (next-single-char-property-change pos old-prop))))))
+		(setq-local char-property-alias-alist
+			    (cons (cons 'invisible
+					(mapcar (lambda (spec)
+						  (org--get-buffer-local-invisible-property-symbol spec nil 'return-only))
+						org--invisible-spec-priority-list))
+				  (remove (assq 'invisible char-property-alias-alist)
+					  char-property-alias-alist)))))))))))
+
 (defun org-flag-region (from to flag spec)
   "Hide or show lines from FROM to TO, according to FLAG.
 SPEC is the invisibility spec, as a symbol."
-  (remove-overlays from to 'invisible spec)
-  ;; Use `front-advance' since text right before to the beginning of
-  ;; the overlay belongs to the visible line than to the contents.
-  (when flag
-    (let ((o (make-overlay from to nil 'front-advance)))
-      (overlay-put o 'evaporate t)
-      (overlay-put o 'invisible spec)
-      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
-
+  (with-silent-modifications
+    (remove-text-properties from to (list (org--get-buffer-local-invisible-property-symbol spec) nil))
+    (when flag
+      (put-text-property from to (org--get-buffer-local-invisible-property-symbol spec) spec))))
 
 \f
 ;;; Regexp matching
 
 (defsubst org-pos-in-match-range (pos n)
-  (and (match-beginning n)
-       (<= (match-beginning n) pos)
-       (>= (match-end n) pos)))
+(and (match-beginning n)
+     (<= (match-beginning n) pos)
+     (>= (match-end n) pos)))
 
 (defun org-skip-whitespace ()
   "Skip over space, tabs and newline characters."
diff --git a/lisp/org-src.el b/lisp/org-src.el
index 6f6c544dc..9e8a50044 100644
--- a/lisp/org-src.el
+++ b/lisp/org-src.el
@@ -529,8 +529,8 @@ Leave point in edit buffer."
 	(org-src-switch-to-buffer buffer 'edit)
 	;; Insert contents.
 	(insert contents)
-	(remove-text-properties (point-min) (point-max)
-				'(display nil invisible nil intangible nil))
+	(org-remove-text-properties (point-min) (point-max)
+				    '(display nil invisible nil intangible nil))
 	(unless preserve-ind (org-do-remove-indentation))
 	(set-buffer-modified-p nil)
 	(setq buffer-file-name nil)
diff --git a/lisp/org-table.el b/lisp/org-table.el
index 6462b99c4..75801161b 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -2001,7 +2001,7 @@ toggle `org-table-follow-field-mode'."
    (arg
     (let ((b (save-excursion (skip-chars-backward "^|") (point)))
 	  (e (save-excursion (skip-chars-forward "^|\r\n") (point))))
-      (remove-text-properties b e '(invisible t intangible t))
+      (org-remove-text-properties b e '(invisible t intangible t))
       (if (and (boundp 'font-lock-mode) font-lock-mode)
 	  (font-lock-fontify-block))))
    (t
@@ -2028,7 +2028,7 @@ toggle `org-table-follow-field-mode'."
       (setq word-wrap t)
       (goto-char (setq p (point-max)))
       (insert (org-trim field))
-      (remove-text-properties p (point-max) '(invisible t intangible t))
+      (org-remove-text-properties p (point-max) '(invisible t intangible t))
       (goto-char p)
       (setq-local org-finish-function 'org-table-finish-edit-field)
       (setq-local org-window-configuration cw)
diff --git a/lisp/org.el b/lisp/org.el
index e5cea04c6..3d4a7b072 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.")
 (declare-function cdlatex-math-symbol "ext:cdlatex")
 (declare-function Info-goto-node "info" (nodename &optional fork strict-case))
 (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag))
+(declare-function isearch-filter-visible "isearch" (beg end))
 (declare-function org-add-archive-files "org-archive" (files))
 (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom))
 (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour))
@@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.")
 
 (defvar ffap-url-regexp)
 (defvar org-element-paragraph-separate)
+(defvar org-element-all-objects)
+(defvar org-element-all-elements)
+(defvar org-element-greater-elements)
 (defvar org-indent-indentation-per-level)
 (defvar org-radio-target-regexp)
 (defvar org-target-link-regexp)
@@ -4734,9 +4738,153 @@ This is for getting out of special buffers like capture.")
 
 ;;;; Define the Org mode
 
+;;; Handling buffer modifications
+
 (defun org-before-change-function (_beg _end)
   "Every change indicates that a table might need an update."
   (setq org-table-may-need-update t))
+
+(defun org-after-change-function (from to len)
+  "Process changes in folded elements.
+If a text was inserted into invisible region, hide the inserted text.
+If the beginning/end line of a folded drawer/block was changed, unfold it.
+If a valid end line was inserted in the middle of the folded drawer/block, unfold it."
+
+  ;; re-hide text inserted in the middle of a folded region
+  (dolist (spec org--invisible-spec-priority-list)
+    (when-let ((spec-to (get-text-property to (org--get-buffer-local-invisible-property-symbol spec)))
+	       (spec-from (get-text-property (max (point-min) (1- from)) (org--get-buffer-local-invisible-property-symbol spec))))
+      (when (eq spec-to spec-from)
+	(org-flag-region from to 't spec-to))))
+
+  ;; Process all the folded text between `from' and `to'
+  (org-with-wide-buffer
+
+   (if (< to from)
+       (let ((tmp from))
+	 (setq from to)
+         (setq to tmp)))
+   
+   ;; Include next/previous line into the changed region.
+   ;; This is needed to catch edits in beginning line of a folded
+   ;; element.
+   (setq to (save-excursion (goto-char to) (forward-line) (point)))
+   (setq from (save-excursion (goto-char from) (forward-line -1) (point)))
+   
+   ;; Expand the considered region to include partially present folded
+   ;; drawer/block.
+   (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))
+     (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))
+   (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-block))
+     (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))
+
+   ;; check folded drawers
+   (let ((pos from))
+     (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))
+       (setq pos (next-single-char-property-change pos
+						   (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))
+     (while (< pos to)
+       (when-let ((drawer-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))
+				     pos))
+		  (drawer-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))
+
+	 (let (unfold?)
+           ;; the line before folded text should be beginning of the drawer
+           (save-excursion
+             (goto-char drawer-begin)
+             (backward-char)
+             (beginning-of-line)
+             (unless (looking-at-p org-drawer-regexp)
+	       (setq unfold? t)))
+           ;; the last line of the folded text should be :END:
+           (save-excursion
+             (goto-char drawer-end)
+             (beginning-of-line)
+             (unless (let ((case-fold-search t)) (looking-at-p org-property-end-re))
+	       (setq unfold? t)))
+           ;; there should be no :END: anywhere in the drawer body
+           (save-excursion
+             (goto-char drawer-begin)
+             (when (save-excursion
+		     (let ((case-fold-search t))
+		       (re-search-forward org-property-end-re
+					  (max (point)
+					       (1- (save-excursion
+						     (goto-char drawer-end)
+                                                     (line-beginning-position))))
+                                          't)))
+	       (setq unfold? t)))
+           ;; there should be no new entry anywhere in the drawer body
+           (save-excursion
+             (goto-char drawer-begin)
+             (when (save-excursion
+		     (let ((case-fold-search t))
+		       (re-search-forward org-outline-regexp-bol
+					  (max (point)
+					       (1- (save-excursion
+						     (goto-char drawer-end)
+                                                     (line-beginning-position))))
+                                          't)))
+	       (setq unfold? t)))
+
+           (when unfold? (org-flag-region drawer-begin drawer-end nil 'org-hide-drawer))))
+       
+       (setq pos (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))))
+
+   ;; check folded blocks
+   (let ((pos from))
+     (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block))
+       (setq pos (next-single-char-property-change pos
+						   (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))
+     (while (< pos to)
+       (when-let ((block-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block))
+				    pos))
+		  (block-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))
+
+	 (let (unfold?)
+           ;; the line before folded text should be beginning of the block
+           (save-excursion
+             (goto-char block-begin)
+             (backward-char)
+             (beginning-of-line)
+             (unless (looking-at-p org-dblock-start-re)
+	       (setq unfold? t)))
+           ;; the last line of the folded text should be end of the block
+           (save-excursion
+             (goto-char block-end)
+             (beginning-of-line)
+             (unless (looking-at-p org-dblock-end-re)
+	       (setq unfold? t)))
+           ;; there should be no #+end anywhere in the block body
+           (save-excursion
+             (goto-char block-begin)
+             (when (save-excursion
+		     (re-search-forward org-dblock-end-re
+					(max (point)
+					     (1- (save-excursion
+						   (goto-char block-end)
+						   (line-beginning-position))))
+                                        't))
+	       (setq unfold? t)))
+           ;; there should be no new entry anywhere in the block body
+           (save-excursion
+             (goto-char block-begin)
+             (when (save-excursion
+		     (let ((case-fold-search t))
+		       (re-search-forward org-outline-regexp-bol
+					  (max (point)
+					       (1- (save-excursion
+						     (goto-char block-end)
+                                                     (line-beginning-position))))
+                                          't)))
+	       (setq unfold? t)))
+
+           (when unfold? (org-flag-region block-begin block-end nil 'org-hide-block))))
+       
+       (setq pos
+	     (next-single-char-property-change pos
+					       (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))))))
+
 (defvar org-mode-map)
 (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param.
 (defvar org-agenda-keep-modes nil)      ; Dynamically-scoped param.
@@ -4789,6 +4937,7 @@ The following commands are available:
   (org-install-agenda-files-menu)
   (when org-link-descriptive (add-to-invisibility-spec '(org-link)))
   (add-to-invisibility-spec '(org-hide-block . t))
+  (add-to-invisibility-spec '(org-hide-drawer . t))
   (setq-local outline-regexp org-outline-regexp)
   (setq-local outline-level 'org-outline-level)
   (setq bidi-paragraph-direction 'left-to-right)
@@ -4817,6 +4966,8 @@ The following commands are available:
   ;; Activate before-change-function
   (setq-local org-table-may-need-update t)
   (add-hook 'before-change-functions 'org-before-change-function nil 'local)
+  ;; Activate after-change-function
+  (add-hook 'after-change-functions 'org-after-change-function nil 'local)
   ;; Check for running clock before killing a buffer
   (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local)
   ;; Initialize macros templates.
@@ -4868,6 +5019,10 @@ The following commands are available:
   (setq-local outline-isearch-open-invisible-function
 	      (lambda (&rest _) (org-show-context 'isearch)))
 
+  ;; Make isearch search in blocks hidden via text properties
+  (setq-local isearch-filter-predicate #'org--isearch-filter-predicate)
+  (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local)
+
   ;; Setup the pcomplete hooks
   (setq-local pcomplete-command-completion-function #'org-pcomplete-initial)
   (setq-local pcomplete-command-name-function #'org-command-at-point)
@@ -5049,8 +5204,8 @@ stacked delimiters is N.  Escaping delimiters is not possible."
 	      (when verbatim?
 		(org-remove-flyspell-overlays-in
 		 (match-beginning 0) (match-end 0))
-		(remove-text-properties (match-beginning 2) (match-end 2)
-					'(display t invisible t intangible t)))
+		(org-remove-text-properties (match-beginning 2) (match-end 2)
+					 '(display t invisible t intangible t)))
 	      (add-text-properties (match-beginning 2) (match-end 2)
 				   '(font-lock-multiline t org-emphasis t))
 	      (when (and org-hide-emphasis-markers
@@ -5165,7 +5320,7 @@ This includes angle, plain, and bracket links."
 	    (if (not (eq 'bracket style))
 		(add-text-properties start end properties)
 	      ;; Handle invisible parts in bracket links.
-	      (remove-text-properties start end '(invisible nil))
+	      (org-remove-text-properties start end '(invisible nil))
 	      (let ((hidden
 		     (append `(invisible
 			       ,(or (org-link-get-parameter type :display)
@@ -5185,8 +5340,8 @@ This includes angle, plain, and bracket links."
 (defun org-activate-code (limit)
   (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t)
     (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
-    (remove-text-properties (match-beginning 0) (match-end 0)
-			    '(display t invisible t intangible t))
+    (org-remove-text-properties (match-beginning 0) (match-end 0)
+			     '(display t invisible t intangible t))
     t))
 
 (defcustom org-src-fontify-natively t
@@ -5257,8 +5412,8 @@ by a #."
 	    (setq block-end (match-beginning 0)) ; includes the final newline.
 	    (when quoting
 	      (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline)
-	      (remove-text-properties beg end-of-endline
-				      '(display t invisible t intangible t)))
+	      (org-remove-text-properties beg end-of-endline
+				       '(display t invisible t intangible t)))
 	    (add-text-properties
 	     beg end-of-endline '(font-lock-fontified t font-lock-multiline t))
 	    (org-remove-flyspell-overlays-in beg bol-after-beginline)
@@ -5312,9 +5467,9 @@ by a #."
 	     '(font-lock-fontified t face org-document-info))))
 	 ((string-prefix-p "+caption" dc1)
 	  (org-remove-flyspell-overlays-in (match-end 2) (match-end 0))
-	  (remove-text-properties (match-beginning 0) (match-end 0)
-				  '(display t invisible t intangible t))
-	  ;; Handle short captions
+	  (org-remove-text-properties (match-beginning 0) (match-end 0)
+				   '(display t invisible t intangible t))
+	  ;; Handle short captions.
 	  (save-excursion
 	    (beginning-of-line)
 	    (looking-at (rx (group (zero-or-more blank)
@@ -5335,8 +5490,8 @@ by a #."
 	   '(font-lock-fontified t face font-lock-comment-face)))
 	 (t ;; Just any other in-buffer setting, but not indented
 	  (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
-	  (remove-text-properties (match-beginning 0) (match-end 0)
-				  '(display t invisible t intangible t))
+	  (org-remove-text-properties (match-beginning 0) (match-end 0)
+				   '(display t invisible t intangible t))
 	  (add-text-properties beg (match-end 0)
 			       '(font-lock-fontified t face org-meta-line))
 	  t))))))
@@ -5721,35 +5876,59 @@ needs to be inserted at a specific position in the font-lock sequence.")
       (decompose-region (point-min) (point-max))
       (message "Entities are now displayed as plain text"))))
 
-(defvar-local org-custom-properties-overlays nil
-  "List of overlays used for custom properties.")
+(defvar-local org-custom-properties-hidden-p nil
+  "Non-nil when custom properties are hidden.")
+
+(defcustom org-custom-properties-hide-emptied-drawers nil
+  "Non-nil means that drawers containing only `org-custom-properties' will be hidden together with the properties."
+  :group 'org
+  :type '(choice
+	  (const :tag "Don't hide emptied drawers" nil)
+	  (const :tag "Hide emptied drawers" t)))
 
 (defun org-toggle-custom-properties-visibility ()
   "Display or hide properties in `org-custom-properties'."
   (interactive)
-  (if org-custom-properties-overlays
-      (progn (mapc #'delete-overlay org-custom-properties-overlays)
-	     (setq org-custom-properties-overlays nil))
+  (require 'org-macs)
+  (add-to-invisibility-spec 'org-hide-custom-property)
+  (add-to-list 'org--invisible-spec-priority-list 'org-hide-custom-property)
+  (if org-custom-properties-hidden-p
+      (let (match)
+	(setq org-custom-properties-hidden-p nil)
+	(org-with-wide-buffer
+         (goto-char (point-min))
+         (with-silent-modifications
+           (while (setq match (text-property-search-forward (org--get-buffer-local-invisible-property-symbol 'org-hide-custom-property) 'org-hide-custom-property t))
+             (org-flag-region (prop-match-beginning match)
+			      (prop-match-end match)
+			      nil 'org-hide-custom-property)))))
     (when org-custom-properties
+      (setq org-custom-properties-hidden-p t)
       (org-with-wide-buffer
-       (goto-char (point-min))
-       (let ((regexp (org-re-property (regexp-opt org-custom-properties) t t)))
+       (let* ((regexp (org-re-property (regexp-opt org-custom-properties) t t))
+	      (regexp-drawer (format "%s\n\\(?:%s\\)+\n%s"
+				     (replace-regexp-in-string "\\$$" "" org-drawer-regexp)
+                                     (replace-regexp-in-string "\\(^\\^\\|\\$$\\)" "" regexp)
+                                     (replace-regexp-in-string "^\\^" "" org-property-end-re))))
+         
+         (when org-custom-properties-hide-emptied-drawers
+	   (goto-char (point-min))
+           (while (re-search-forward regexp-drawer nil t)
+             (with-silent-modifications
+               (org-flag-region (1- (match-beginning 0)) (match-end 0) t 'org-hide-custom-property))))
+         
+         (goto-char (point-min))
 	 (while (re-search-forward regexp nil t)
 	   (let ((end (cdr (save-match-data (org-get-property-block)))))
 	     (when (and end (< (point) end))
 	       ;; Hide first custom property in current drawer.
-	       (let ((o (make-overlay (match-beginning 0) (1+ (match-end 0)))))
-		 (overlay-put o 'invisible t)
-		 (overlay-put o 'org-custom-property t)
-		 (push o org-custom-properties-overlays))
-	       ;; Hide additional custom properties in the same drawer.
-	       (while (re-search-forward regexp end t)
-		 (let ((o (make-overlay (match-beginning 0) (1+ (match-end 0)))))
-		   (overlay-put o 'invisible t)
-		   (overlay-put o 'org-custom-property t)
-		   (push o org-custom-properties-overlays)))))
-	   ;; Each entry is limited to a single property drawer.
-	   (outline-next-heading)))))))
+	       (with-silent-modifications
+		 (org-flag-region (match-beginning 0) (1+ (match-end 0)) t 'org-hide-custom-property)
+		 ;; Hide additional custom properties in the same drawer.
+		 (while (re-search-forward regexp end t)
+		   (org-flag-region (match-beginning 0) (1+ (match-end 0)) t 'org-hide-custom-property))))))
+	 ;; Each entry is limited to a single property drawer.
+	 (outline-next-heading))))))
 
 (defun org-fontify-entities (limit)
   "Find an entity to fontify."
@@ -5858,10 +6037,11 @@ If TAG is a number, get the corresponding match group."
 	 (inhibit-modification-hooks t)
 	 deactivate-mark buffer-file-name buffer-file-truename)
     (decompose-region beg end)
-    (remove-text-properties beg end
-			    '(mouse-face t keymap t org-linked-text t
-					 invisible t intangible t
-					 org-emphasis t))
+    (org-remove-text-properties beg end
+			     '(mouse-face t keymap t org-linked-text t
+					  invisible t
+                                          intangible t
+					  org-emphasis t))
     (org-remove-font-lock-display-properties beg end)))
 
 (defconst org-script-display  '(((raise -0.3) (height 0.7))
@@ -5969,6 +6149,29 @@ open and agenda-wise Org files."
 
 ;;;; Headlines visibility
 
+(defun org-hide-entry ()
+  "Hide the body directly following this heading."
+  (interactive)
+  (save-excursion
+    (outline-back-to-heading)
+    (outline-end-of-heading)
+    (org-flag-region (point) (progn (outline-next-preface) (point)) t 'outline)))
+
+(defun org-hide-subtree ()
+  "Hide everything after this heading at deeper levels."
+  (interactive)
+  (org-flag-subtree t))
+
+(defun org-hide-sublevels (levels)
+  "Hide everything but the top LEVELS levels of headers, in whole buffer.
+This also unhides the top heading-less body, if any.
+
+Interactively, the prefix argument supplies the value of LEVELS.
+When invoked without a prefix argument, LEVELS defaults to the level
+of the current heading, or to 1 if the current line is not a heading."
+  (cl-letf (((symbol-function 'outline-flag-region) #'org-flag-region))
+    (org-hide-sublevels levels)))
+
 (defun org-show-entry ()
   "Show the body directly following this heading.
 Show the heading too, if it is currently invisible."
@@ -5980,13 +6183,24 @@ Show the heading too, if it is currently invisible."
        (line-end-position 0)
        (save-excursion
 	 (if (re-search-forward
-	      (concat "[\r\n]\\(" org-outline-regexp "\\)") nil t)
+	      (concat "[\r\n]\\(" (org-get-limited-outline-regexp) "\\)") nil t)
 	     (match-beginning 1)
 	   (point-max)))
        nil
        'outline)
       (org-cycle-hide-drawers 'children))))
 
+(defun org-show-heading ()
+  "Show the current heading and move to its end."
+  (org-flag-region (- (point)
+ 		   (if (bobp) 0
+ 		     (if (and outline-blank-line
+                              (eq (char-before (1- (point))) ?\n))
+ 			 2 1)))
+		(progn (outline-end-of-heading) (point))
+		nil
+                'outline))
+
 (defun org-show-children (&optional level)
   "Show all direct subheadings of this heading.
 Prefix arg LEVEL is how many levels below the current level
@@ -6030,6 +6244,11 @@ heading to appear."
   (org-flag-region
    (point) (save-excursion (org-end-of-subtree t t)) nil 'outline))
 
+(defun org-show-branches ()
+  "Show all subheadings of this heading, but not their bodies."
+  (interactive)
+  (org-show-children 1000))
+
 ;;;; Blocks and drawers visibility
 
 (defun org--hide-wrapper-toggle (element category force no-error)
@@ -6062,7 +6281,9 @@ Return a non-nil value when toggling is successful."
 	;; at the block closing line.
 	(unless (let ((eol (line-end-position)))
 		  (and (> eol start) (/= eol end)))
-	  (let* ((spec (if (eq category 'block) 'org-hide-block 'outline))
+	  (let* ((spec (cond ((eq category 'block) 'org-hide-block)
+			     ((eq category 'drawer) 'org-hide-drawer)
+			     (t 'outline)))
 		 (flag
 		  (cond ((eq force 'off) nil)
 			(force t)
@@ -6115,24 +6336,24 @@ Return a non-nil value when toggling is successful."
 
 (defun org-hide-drawer-all ()
   "Fold all drawers in the current buffer."
+  (org-show-all '(drawers))
   (save-excursion
     (goto-char (point-min))
     (while (re-search-forward org-drawer-regexp nil t)
-      (let* ((pair (get-char-property-and-overlay (line-beginning-position)
-						  'invisible))
-	     (o (cdr-safe pair)))
-	(if (overlayp o) (goto-char (overlay-end o)) ;invisible drawer
-	  (pcase (get-char-property-and-overlay (point) 'invisible)
-	    (`(outline . ,o) (goto-char (overlay-end o))) ;already folded
-	    (_
-	     (let* ((drawer (org-element-at-point))
-		    (type (org-element-type drawer)))
-	       (when (memq type '(drawer property-drawer))
-		 (org-hide-drawer-toggle t nil drawer)
-		 ;; Make sure to skip drawer entirely or we might flag it
-		 ;; another time when matching its ending line with
-		 ;; `org-drawer-regexp'.
-		 (goto-char (org-element-property :end drawer)))))))))))
+      (let* ((drawer (org-element-at-point))
+	     (type (org-element-type drawer)))
+	(when (memq type '(drawer property-drawer))
+	  ;; We are sure regular drawers are unfolded because of
+	  ;; `org-show-all' call above.  However, property drawers may
+	  ;; be folded, or in a folded headline.  In that case, do not
+	  ;; re-hide it.
+	  (unless (and (eq type 'property-drawer)
+		       (eq 'org-hide-drawer (get-char-property (point) 'invisible)))
+	    (org-hide-drawer-toggle t nil drawer))
+	  ;; Make sure to skip drawer entirely or we might flag it
+	  ;; another time when matching its ending line with
+	  ;; `org-drawer-regexp'.
+	  (goto-char (org-element-property :end drawer)))))))
 
 (defun org-cycle-hide-drawers (state)
   "Re-hide all drawers after a visibility state change.
@@ -6147,9 +6368,10 @@ STATE should be one of the symbols listed in the docstring of
 		      (t (save-excursion (org-end-of-subtree t t))))))
       (org-with-point-at beg
 	(while (re-search-forward org-drawer-regexp end t)
-	  (pcase (get-char-property-and-overlay (point) 'invisible)
+	  (pcase (get-char-property (point) 'invisible)
 	    ;; Do not fold already folded drawers.
-	    (`(outline . ,o) (goto-char (overlay-end o)))
+	    ('outline
+             (goto-char (min end (next-single-char-property-change (point) 'invisible))))
 	    (_
 	     (let ((drawer (org-element-at-point)))
 	       (when (memq (org-element-type drawer) '(drawer property-drawer))
@@ -6172,31 +6394,13 @@ By default, the function expands headings, blocks and drawers.
 When optional argument TYPE is a list of symbols among `blocks',
 `drawers' and `headings', to only expand one specific type."
   (interactive)
-  (let ((types (or types '(blocks drawers headings))))
-    (when (memq 'blocks types)
-      (org-flag-region (point-min) (point-max) nil 'org-hide-block))
-    (cond
-     ;; Fast path.  Since headings and drawers share the same
-     ;; invisible spec, clear everything in one go.
-     ((and (memq 'headings types)
-	   (memq 'drawers types))
-      (org-flag-region (point-min) (point-max) nil 'outline))
-     ((memq 'headings types)
-      (org-flag-region (point-min) (point-max) nil 'outline)
-      (org-cycle-hide-drawers 'all))
-     ((memq 'drawers types)
-      (save-excursion
-	(goto-char (point-min))
-	(while (re-search-forward org-drawer-regexp nil t)
-	  (let* ((pair (get-char-property-and-overlay (line-beginning-position)
-						      'invisible))
-		 (o (cdr-safe pair)))
-	    (if (overlayp o) (goto-char (overlay-end o))
-	      (pcase (get-char-property-and-overlay (point) 'invisible)
-		(`(outline . ,o)
-		 (goto-char (overlay-end o))
-		 (delete-overlay o))
-		(_ nil))))))))))
+  (dolist (type (or types '(blocks drawers headings)))
+    (org-flag-region (point-min) (point-max) nil
+		     (pcase type
+		       (`blocks 'org-hide-block)
+		       (`drawers 'org-hide-drawer)
+		       (`headings 'outline)
+		       (_ (error "Invalid type: %S" type))))))
 
 ;;;###autoload
 (defun org-cycle (&optional arg)
@@ -6552,7 +6756,7 @@ With a numeric prefix, show all headlines up to that level."
 		     (org-narrow-to-subtree)
 		     (org-content))))
 		((or "all" "showall")
-		 (outline-show-subtree))
+		 (org-show-subtree))
 		(_ nil)))
 	    (org-end-of-subtree)))))))
 
@@ -6625,7 +6829,7 @@ This function is the default value of the hook `org-cycle-hook'."
 	  (while (re-search-forward re nil t)
 	    (when (and (not (org-invisible-p))
 		       (org-invisible-p (line-end-position)))
-	      (outline-hide-entry))))
+	      (org-hide-entry))))
 	(org-cycle-hide-drawers 'all)
 	(org-cycle-show-empty-lines 'overview)))))
 
@@ -6697,10 +6901,11 @@ information."
     (org-show-entry)
     ;; If point is hidden within a drawer or a block, make sure to
     ;; expose it.
-    (dolist (o (overlays-at (point)))
-      (when (memq (overlay-get o 'invisible)
-		  '(org-hide-block org-hide-drawer outline))
-	(delete-overlay o)))
+    (when (memq (get-text-property (point) 'invisible)
+		'(org-hide-block org-hide-drawer))
+      (let ((spec (get-text-property (point) 'invisible))
+	    (region (org--find-text-property-region (point) 'invisible)))
+	(org-flag-region (car region) (cdr region) nil spec)))
     (unless (org-before-first-heading-p)
       (org-with-limited-levels
        (cl-case detail
@@ -6916,9 +7121,10 @@ unconditionally."
       ;; When INVISIBLE-OK is non-nil, ensure newly created headline
       ;; is visible.
       (unless invisible-ok
-	(pcase (get-char-property-and-overlay (point) 'invisible)
-	  (`(outline . ,o)
-	   (move-overlay o (overlay-start o) (line-end-position 0)))
+	(pcase (get-char-property (point) 'invisible)
+	  ('outline
+           (let ((region (org--find-text-property-region (point) 'invisible)))
+	     (org-flag-region (line-end-position 0) (cdr region) nil 'outline)))
 	  (_ nil))))
      ;; At a headline...
      ((org-at-heading-p)
@@ -7515,7 +7721,6 @@ case."
      (setq txt (buffer-substring beg end))
      (org-save-markers-in-region beg end)
      (delete-region beg end)
-     (org-remove-empty-overlays-at beg)
      (unless (= beg (point-min)) (org-flag-region (1- beg) beg nil 'outline))
      (unless (bobp) (org-flag-region (1- (point)) (point) nil 'outline))
      (and (not (bolp)) (looking-at "\n") (forward-char 1))
@@ -7677,7 +7882,7 @@ When REMOVE is non-nil, remove the subtree from the clipboard."
      (skip-chars-forward " \t\n\r")
      (setq beg (point))
      (when (and (org-invisible-p) visp)
-       (save-excursion (outline-show-heading)))
+       (save-excursion (org-show-heading)))
      ;; Shift if necessary.
      (unless (= shift 0)
        (save-restriction
@@ -8119,7 +8324,7 @@ function is being called interactively."
 		       (point))
 	    what "children")
       (goto-char start)
-      (outline-show-subtree)
+      (org-show-subtree)
       (outline-next-heading))
      (t
       ;; we will sort the top-level entries in this file
@@ -10736,7 +10941,8 @@ narrowing."
 	     (let ((beg (point)))
 	       (insert ":" drawer ":\n:END:\n")
 	       (org-indent-region beg (point))
-	       (org-flag-region (line-end-position -1) (1- (point)) t 'outline))
+	       (org-flag-region
+		(line-end-position -1) (1- (point)) t 'org-hide-drawer))
 	     (end-of-line -1)))))
       (t
        (org-end-of-meta-data org-log-state-notes-insert-after-drawers)
@@ -13173,7 +13379,7 @@ drawer is immediately hidden."
 	   (inhibit-read-only t))
        (unless (bobp) (insert "\n"))
        (insert ":PROPERTIES:\n:END:")
-       (org-flag-region (line-end-position 0) (point) t 'outline)
+       (org-flag-region (line-end-position 0) (point) t 'org-hide-drawer)
        (when (or (eobp) (= begin (point-min))) (insert "\n"))
        (org-indent-region begin (point))))))
 
@@ -16553,7 +16759,7 @@ The detailed reaction depends on the user option `org-catch-invisible-edits'."
       (when (or invisible-at-point invisible-before-point)
 	(when (eq org-catch-invisible-edits 'error)
 	  (user-error "Editing in invisible areas is prohibited, make them visible first"))
-	(if (and org-custom-properties-overlays
+	(if (and org-custom-properties-hidden-p
 		 (y-or-n-p "Display invisible properties in this buffer? "))
 	    (org-toggle-custom-properties-visibility)
 	  ;; Make the area visible
@@ -17636,11 +17842,11 @@ Move point to the beginning of first heading or end of buffer."
 (defun org-show-branches-buffer ()
   "Show all branches in the buffer."
   (org-flag-above-first-heading)
-  (outline-hide-sublevels 1)
+  (org-hide-sublevels 1)
   (unless (eobp)
-    (outline-show-branches)
+    (org-show-branches)
     (while (outline-get-next-sibling)
-      (outline-show-branches)))
+      (org-show-branches)))
   (goto-char (point-min)))
 
 (defun org-kill-note-or-show-branches ()
@@ -17654,8 +17860,8 @@ Move point to the beginning of first heading or end of buffer."
 	(t
 	 (let ((beg (progn (org-back-to-heading) (point)))
 	       (end (save-excursion (org-end-of-subtree t t) (point))))
-	   (outline-hide-subtree)
-	   (outline-show-branches)
+	   (org-hide-subtree)
+	   (org-show-branches)
 	   (org-hide-archived-subtrees beg end)))))
 
 (defun org-delete-indentation (&optional arg)
@@ -17811,9 +18017,9 @@ Otherwise, call `org-show-children'.  ARG is the level to hide."
     (if (org-before-first-heading-p)
         (progn
           (org-flag-above-first-heading)
-          (outline-hide-sublevels (or arg 1))
+          (org-hide-sublevels (or arg 1))
           (goto-char (point-min)))
-      (outline-hide-subtree)
+      (org-hide-subtree)
       (org-show-children arg))))
 
 (defun org-ctrl-c-star ()
@@ -20489,20 +20695,20 @@ With ARG, repeats or can move backward if negative."
       (end-of-line))
     (while (and (< arg 0) (re-search-backward regexp nil :move))
       (unless (bobp)
-	(while (pcase (get-char-property-and-overlay (point) 'invisible)
-		 (`(outline . ,o)
-		  (goto-char (overlay-start o))
-		  (re-search-backward regexp nil :move))
-		 (_ nil))))
+	(pcase (get-char-property (point) 'invisible)
+	  ('outline
+	   (goto-char (car (org--find-text-property-region (point) 'invisible)))
+	   (beginning-of-line))
+	  (_ nil)))
       (cl-incf arg))
-    (while (and (> arg 0) (re-search-forward regexp nil t))
-      (while (pcase (get-char-property-and-overlay (point) 'invisible)
-	       (`(outline . ,o)
-		(goto-char (overlay-end o))
-		(re-search-forward regexp nil :move))
-	       (_
-		(end-of-line)
-		nil)))			;leave the loop
+    (while (and (> arg 0) (re-search-forward regexp nil :move))
+      (pcase (get-char-property (point) 'invisible)
+	('outline
+	 (goto-char (cdr (org--find-text-property-region (point) 'invisible)))
+         (skip-chars-forward " \t\n")
+	 (end-of-line))
+	(_
+	 (end-of-line)))
       (cl-decf arg))
     (if (> arg 0) (goto-char (point-max)) (beginning-of-line))))
 
@@ -20957,6 +21163,80 @@ Started from `gnus-info-find-node'."
           (t default-org-info-node))))))
 
 \f
+
+;;; Make isearch search in some text hidden via text propertoes
+
+(defvar org--isearch-overlays nil
+  "List of overlays temporarily created during isearch.
+This is used to allow searching in regions hidden via text properties.
+As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays.
+Any text hidden via text properties is not revealed even if `search-invisible'
+is set to 't.")
+
+;; Not sure if it needs to be a user option
+;; One might want to reveal hidden text in, for example, hidden parts of the links.
+;; Currently, hidden text in links is never revealed by isearch.
+(defvar org-isearch-specs '(org-hide-block
+			 org-hide-drawer)
+  "List of text invisibility specs to be searched by isearch.
+By default ([2020-05-09 Sat]), isearch does not search in hidden text,
+which was made invisible using text properties. Isearch will be forced
+to search in hidden text with any of the listed 'invisible property value.")
+
+(defun org--create-isearch-overlays (beg end)
+  "Replace text property invisibility spec by overlays between BEG and END.
+All the regions with invisibility text property spec from
+`org-isearch-specs' will be changed to use overlays instead
+of text properties. The created overlays will be stored in
+`org--isearch-overlays'."
+  (let ((pos beg))
+    (while (< pos end)
+      (when-let* ((spec (get-text-property pos 'invisible))
+		  (spec (memq spec org-isearch-specs))
+		  (region (org--find-text-property-region pos 'invisible)))
+        (setq spec (get-text-property pos 'invisible))
+        ;; Changing text properties is considered buffer modification.
+        ;; We do not want it here.
+	(with-silent-modifications
+          ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat]
+          ;; overlay for 'outline blocks.
+          (let ((o (make-overlay (car region) (cdr region) nil 'front-advance)))
+	    (overlay-put o 'evaporate t)
+	    (overlay-put o 'invisible spec)
+            ;; `delete-overlay' here means that spec information will be lost
+            ;; for the region. The region will remain visible.
+	    (overlay-put o 'isearch-open-invisible #'delete-overlay)
+            (push o org--isearch-overlays))
+	  (org-flag-region (car region) (cdr region) nil spec)))
+      (setq pos (next-single-property-change pos 'invisible nil end)))))
+
+(defun org--isearch-filter-predicate (beg end)
+  "Return non-nil if text between BEG and END is deemed visible by Isearch.
+This function is intended to be used as `isearch-filter-predicate'.
+Unlike `isearch-filter-visible', make text with 'invisible text property
+value listed in `org-isearch-specs' visible to Isearch."
+  (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text
+  (isearch-filter-visible beg end))
+
+(defun org--clear-isearch-overlay (ov)
+  "Convert OV region back into using text properties."
+  (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays
+    ;; Changing text properties is considered buffer modification.
+    ;; We do not want it here.
+    (with-silent-modifications
+      (org-flag-region (overlay-start ov) (overlay-end ov) t spec)))
+  (when (member ov isearch-opened-overlays)
+    (setq isearch-opened-overlays (delete ov isearch-opened-overlays)))
+  (delete-overlay ov))
+
+(defun org--clear-isearch-overlays ()
+  "Convert overlays from `org--isearch-overlays' back into using text properties."
+  (when org--isearch-overlays
+    (mapc #'org--clear-isearch-overlay org--isearch-overlays)
+    (setq org--isearch-overlays nil)))
+
+\f
+
 ;;; Finish up
 
 (add-hook 'org-mode-hook     ;remove overlays when changing major mode

[-- Attachment #3: Type: text/plain, Size: 12921 bytes --]


Ihor Radchenko <yantar92@gmail.com> writes:

> Hello,
>
> [The patch itself will be provided in the following email]
>
> I have four more updates from the previous version of the patch:
>
> 1. All the code handling modifications in folded drawers/blocks is moved
>    to after-change-function. It works as follows:
>    - if any text is inserted in the middle of hidden region, that text
>      is also hidden;
>    - if BEGIN/END line of a folded drawer do not match org-drawer-regexp
>      and org-property-end-re, unfold it; 
>    - if org-property-end-re or new org-outline-regexp-bol is inserted in
>      the middle of the drawer, unfold it;
>    - the same logic for blocks.
>
> 2. The text property stack is rewritten using char-property-alias-alist.
>    This is faster in comparison with previous approach, which involved
>    modifying all the text properties every timer org-flag-region was
>    called. 
>    
> 3. org-toggle-custom-properties-visibility is rewritten using text
>    properties. I also took a freedom to implement a new feature here.
>    Now, setting new `org-custom-properties-hide-emptied-drawers' to
>    non-nil will result in hiding the whole property drawer if it
>    contains only org-custom-properties.
>
> 4. This patch should work against 1aa095ccf. However, the merge was not
>    trivial here. Recent commits actively used the fact that drawers and
>    outlines are hidden via 'outline invisibility spec, which is not the
>    case in this branch. I am not confident that I did not break anything
>    during the merge, especially 1aa095ccf.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the new implementation for tracking changes:
>
>> I gave you a few ideas to quickly check if a change requires expansion,
>> in an earlier mail. I suggest to start out from that. Let me know if you
>> have questions about it.
>
> All the code lives in org-after-change-function. I tried to incorporate
> the earlier Nicholas' suggestions, except the parts related to
> intersecting blocks and drawers. I am not sure if I understand the
> parsing priority of blocks vs. drawers.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the text property stack:
>
> The earlier version of the code literally used stack to save
> pre-existing 'invisibility specs in org-flag-region. This was done on
> every invocation of org-flag-region, which made org-flag-region
> significantly slower. I re-implemented the same feature using
> char-property-alias-alist. Now, different invisibility specs live in
> separate text properties and can be safely modified independently. The
> specs are applied according to org--invisible-spec-priority-list. A side
> effect of current implementation is that char-property-alias-alist is
> fully controlled by org. All the pre-existing settings for 'invisible
> text property will be overwritten by org.
>
>> `gensym' is just a shorter, and somewhat standard way, to create a new
>> uninterned symbol with a given prefix. You seem to re-invent it. What
>> you do with that new symbol is orthogonal to that suggestion, of course.
>
> I do not think that `gensym' is suitable here. We don't want a new
> symbol every time org--get-buffer-local-invisible-property-symbol is
> called. It should return the same symbol if it is called from the same
> buffer multiple times.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the org-toggle-custom-properties-visibility:
>
> The implementation showcases how to introduce new invisibility specs to
> org. Apart from expected (add-to-invisibility-spec 'org-hide-custom-property) 
> one also needs to add the spec into org--invisible-spec-priority-list:
>
> (add-to-list 'org--invisible-spec-priority-list 'org-hide-custom-property)
>
> Searching for text with the given invisibility spec is done as
> follows:
>
> (text-property-search-forward (org--get-buffer-local-invisible-property-symbol 'org-hide-custom-property) 'org-hide-custom-property t)
>
> This last piece of code is probably not the most elegant. I am thinking
> if creating some higher-level interface would be more reasonable here.
> What do you think?
>
>
> The new customisation `org-custom-properties-hide-emptied-drawers'
> sounds logical for me since empty property drawers left after invoking
> org-toggle-custom-properties-visibility are rather useless according to
> my experience. If one already wants to hide parts of property drawers, I
> do not see a reason to show leftover
>
> :PROPERTIES:
> :END:
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the merge with the latest master:
>
> I tried my best to not break anything. However, I am not sure if I
> understand all the recent commits. Could someone take a look if there is
> anything suspicious in org-next-visible-heading?
>
> Also, I have seen some optimisations making use of the fact that drawers
> and headlines both use 'outline invisibility spec. This change in the
> implementation details supposed to improve performance and should not be
> necessary if this patch is going to be merged. Would it be possible to
> refrain from abusing this particular implementation detail in the
> nearest commits on master (unless really necessary)?
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> Further work:
>
> I would like to finalise the current patch and work on other code using
> overlays separately. This patch is already quite complicated as is. I do
> not want to introduce even more potential bugs by working on things not
> directly affected by this version of the patch.
>
> Best,
> Ihor
>
>
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>>> See also `gensym'. Do we really need to use it for something else than
>>>> `invisible'? If not, the tool doesn't need to be generic.
>>>
>>> For now, I also use it for buffer-local 'invisible stack. The stack is
>>> needed to preserve folding state of drawers/blocks inside folded
>>> outline. Though I am thinking about replacing the stack with separate
>>> text properties, like 'invisible-outline-buffer-local +
>>> 'invisible-drawer-buffer-local + 'invisible-block-buffer-local.
>>> Maintaining stack takes a noticeable percentage of CPU time in profiler.
>>>
>>> org--get-buffer-local-text-property-symbol must take care about
>>> situation with indirect buffers. When an indirect buffer is created from
>>> some org buffer, the old value of char-property-alias-alist is carried
>>> over. We need to detect this case and create new buffer-local symbol,
>>> which is unique to the newly created buffer (but not create it if the
>>> buffer-local property is already there). Then, the new symbol must
>>> replace the old alias in char-property-alias-alist + old folding state
>>> must be preserved (via copying the old invisibility specs into the new
>>> buffer-local text property). I do not see how gensym can benefit this
>>> logic.
>>
>> `gensym' is just a shorter, and somewhat standard way, to create a new
>> uninterned symbol with a given prefix. You seem to re-invent it. What
>> you do with that new symbol is orthogonal to that suggestion, of course.
>>
>>>> OK, but this may not be sufficient if we want to do slightly better than
>>>> overlays in that area. This is not mandatory, though.
>>>
>>> Could you elaborate on what can be "slightly better"? 
>>
>> IIRC, I gave examples of finer control of folding state after a change.
>> Consider this _folded_ drawer:
>>
>>   :BEGIN:
>>   Foo
>>   :END:
>>
>> Inserting ":END" in it should not unfold it, as it is currently the case
>> with overlays,
>>
>>   :BEGIN
>>   Foo
>>   :END
>>   :END:
>>
>> but a soon as the last ":" is inserted, the initial drawer could be
>> expanded.
>>
>>   :BEGIN
>>   Foo
>>   :END:
>>   :END:
>>
>> The latter case is not currently handled by overlays. This is what
>> I call "slightly better".
>>
>> Also, note that this change is not related to opening and closing lines
>> of the initial drawer, so sticking text properties on them would not
>> help here.
>>
>> Another case is modifying those borders, e.g.,
>>
>>
>>   :BEGIN:               :BEGIN:
>>   Foo         ------>   Foo  
>>   :END:                 :ND:
>>
>> which should expand the drawer. Your implementation catches this, but
>> I'm pointing out that current implementation with overlays does not.
>> Even though that's not strictly required for compatibility with
>> overlays, it is a welcome slight improvement.
>>
>>>> As discussed before, I don't think you need to use `modification-hooks'
>>>> or `insert-behind-hooks' if you already use `after-change-functions'.
>>>>
>>>> `after-change-functions' are also triggered upon text properties
>>>> changes. So, what is the use case for the other hooks?
>>>
>>> The problem is that `after-change-functions' cannot be a text property.
>>> Only `modification-hooks' and `insert-in-front/behind-hooks' can be a
>>> valid text property. If we use `after-change-functions', they will
>>> always be triggered, regardless if the change was made inside or outside
>>> folded region.
>>
>> As discussed, text properties are local to the change, but require extra
>> care when moving text around. You also observed serious overhead when
>> using them.
>>
>> OTOH, even if `a-c-f' is not local, you can quickly determine if the
>> change altered a folded element, so the overhead is limited, i.e.,
>> mostly checking for a text property at a given buffer position.
>>
>> To be clear, I initially thought that text properties were a superior
>> choice, but I changed my mind a while ago, and I thought you had, too.
>> IOW, `after-change-functions' is the way to go, since you have no strong
>> reason to stick to text properties for this kind of function.
>>
>>>>> :asd:
>>>>> :drawer:
>>>>> lksjdfksdfjl
>>>>> sdfsdfsdf
>>>>> :end:
>>>>>
>>>>> If :asd: was inserted in front of folded :drawer:, changes in :drawer:
>>>>> line of the new folded :asd: drawer would reveal the text between
>>>>> :drawer: and :end:.
>>>>>
>>>>> Let me know what you think on this.
>>>
>>>> I have first to understand the use case for `modification-hook'. But
>>>> I think unfolding is the right thing to do in this situation, isn't it?
>>>
>>> That situation arises because the modification-hooks from ":drawer:"
>>> (they are set via text properties) only have information about the
>>> :drawer:...:end: drawer before the modifications (they were set when
>>> :drawer: was folded last time). So, they will only unfold a part of the
>>> new :asd: drawer. I do not see a simple way to unfold everything without
>>> re-parsing the drawer around the changed text.
>>
>> Oh! I misread your message. I withdraw what I wrote. In this case, we
>> don't want to unfold anything. The situation is not worse than what we
>> have now, and trying to fix it would have repercussions down in the
>> buffer, e.g., expanding drawers screen below.
>>
>> As a rule of thumb, I think we can pay attention to changes in the
>> folded text, and its immediate surroundings (e.g., the opening line,
>> which is not folded), but no further.
>>
>> As written above, slight changes are welcome, but let's not go overboard
>> and parse a whole section just to know if we can expand a drawer.
>>
>>> Actually, I am quite unhappy with the performance of modification-hooks
>>> set via text properties (I am using this patch on my Emacs during this
>>> week). It appears that setting the text properties costs a significant
>>> CPU time in practice, even though running the hooks is pretty fast.
>>> I will think about a way to handle modifications using global
>>> after-change-functions.
>>
>> That's better, IMO.
>>
>> I gave you a few ideas to quickly check if a change requires expansion,
>> in an earlier mail. I suggest to start out from that. Let me know if you
>> have questions about it.
>
> -- 
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-08  5:05                                           ` Ihor Radchenko
  2020-06-08  5:06                                             ` Ihor Radchenko
@ 2020-06-08  5:08                                             ` Ihor Radchenko
  2020-06-10 17:14                                             ` Nicolas Goaziou
  2 siblings, 0 replies; 55+ messages in thread
From: Ihor Radchenko @ 2020-06-08  5:08 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

Github link to the patch:
https://gist.github.com/yantar92/6447754415457927293acda43a7fcaef 

Ihor Radchenko <yantar92@gmail.com> writes:

> Hello,
>
> [The patch itself will be provided in the following email]
>
> I have four more updates from the previous version of the patch:
>
> 1. All the code handling modifications in folded drawers/blocks is moved
>    to after-change-function. It works as follows:
>    - if any text is inserted in the middle of hidden region, that text
>      is also hidden;
>    - if BEGIN/END line of a folded drawer do not match org-drawer-regexp
>      and org-property-end-re, unfold it; 
>    - if org-property-end-re or new org-outline-regexp-bol is inserted in
>      the middle of the drawer, unfold it;
>    - the same logic for blocks.
>
> 2. The text property stack is rewritten using char-property-alias-alist.
>    This is faster in comparison with previous approach, which involved
>    modifying all the text properties every timer org-flag-region was
>    called. 
>    
> 3. org-toggle-custom-properties-visibility is rewritten using text
>    properties. I also took a freedom to implement a new feature here.
>    Now, setting new `org-custom-properties-hide-emptied-drawers' to
>    non-nil will result in hiding the whole property drawer if it
>    contains only org-custom-properties.
>
> 4. This patch should work against 1aa095ccf. However, the merge was not
>    trivial here. Recent commits actively used the fact that drawers and
>    outlines are hidden via 'outline invisibility spec, which is not the
>    case in this branch. I am not confident that I did not break anything
>    during the merge, especially 1aa095ccf.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the new implementation for tracking changes:
>
>> I gave you a few ideas to quickly check if a change requires expansion,
>> in an earlier mail. I suggest to start out from that. Let me know if you
>> have questions about it.
>
> All the code lives in org-after-change-function. I tried to incorporate
> the earlier Nicholas' suggestions, except the parts related to
> intersecting blocks and drawers. I am not sure if I understand the
> parsing priority of blocks vs. drawers.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the text property stack:
>
> The earlier version of the code literally used stack to save
> pre-existing 'invisibility specs in org-flag-region. This was done on
> every invocation of org-flag-region, which made org-flag-region
> significantly slower. I re-implemented the same feature using
> char-property-alias-alist. Now, different invisibility specs live in
> separate text properties and can be safely modified independently. The
> specs are applied according to org--invisible-spec-priority-list. A side
> effect of current implementation is that char-property-alias-alist is
> fully controlled by org. All the pre-existing settings for 'invisible
> text property will be overwritten by org.
>
>> `gensym' is just a shorter, and somewhat standard way, to create a new
>> uninterned symbol with a given prefix. You seem to re-invent it. What
>> you do with that new symbol is orthogonal to that suggestion, of course.
>
> I do not think that `gensym' is suitable here. We don't want a new
> symbol every time org--get-buffer-local-invisible-property-symbol is
> called. It should return the same symbol if it is called from the same
> buffer multiple times.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the org-toggle-custom-properties-visibility:
>
> The implementation showcases how to introduce new invisibility specs to
> org. Apart from expected (add-to-invisibility-spec 'org-hide-custom-property) 
> one also needs to add the spec into org--invisible-spec-priority-list:
>
> (add-to-list 'org--invisible-spec-priority-list 'org-hide-custom-property)
>
> Searching for text with the given invisibility spec is done as
> follows:
>
> (text-property-search-forward (org--get-buffer-local-invisible-property-symbol 'org-hide-custom-property) 'org-hide-custom-property t)
>
> This last piece of code is probably not the most elegant. I am thinking
> if creating some higher-level interface would be more reasonable here.
> What do you think?
>
>
> The new customisation `org-custom-properties-hide-emptied-drawers'
> sounds logical for me since empty property drawers left after invoking
> org-toggle-custom-properties-visibility are rather useless according to
> my experience. If one already wants to hide parts of property drawers, I
> do not see a reason to show leftover
>
> :PROPERTIES:
> :END:
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the merge with the latest master:
>
> I tried my best to not break anything. However, I am not sure if I
> understand all the recent commits. Could someone take a look if there is
> anything suspicious in org-next-visible-heading?
>
> Also, I have seen some optimisations making use of the fact that drawers
> and headlines both use 'outline invisibility spec. This change in the
> implementation details supposed to improve performance and should not be
> necessary if this patch is going to be merged. Would it be possible to
> refrain from abusing this particular implementation detail in the
> nearest commits on master (unless really necessary)?
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> Further work:
>
> I would like to finalise the current patch and work on other code using
> overlays separately. This patch is already quite complicated as is. I do
> not want to introduce even more potential bugs by working on things not
> directly affected by this version of the patch.
>
> Best,
> Ihor
>
>
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>>> See also `gensym'. Do we really need to use it for something else than
>>>> `invisible'? If not, the tool doesn't need to be generic.
>>>
>>> For now, I also use it for buffer-local 'invisible stack. The stack is
>>> needed to preserve folding state of drawers/blocks inside folded
>>> outline. Though I am thinking about replacing the stack with separate
>>> text properties, like 'invisible-outline-buffer-local +
>>> 'invisible-drawer-buffer-local + 'invisible-block-buffer-local.
>>> Maintaining stack takes a noticeable percentage of CPU time in profiler.
>>>
>>> org--get-buffer-local-text-property-symbol must take care about
>>> situation with indirect buffers. When an indirect buffer is created from
>>> some org buffer, the old value of char-property-alias-alist is carried
>>> over. We need to detect this case and create new buffer-local symbol,
>>> which is unique to the newly created buffer (but not create it if the
>>> buffer-local property is already there). Then, the new symbol must
>>> replace the old alias in char-property-alias-alist + old folding state
>>> must be preserved (via copying the old invisibility specs into the new
>>> buffer-local text property). I do not see how gensym can benefit this
>>> logic.
>>
>> `gensym' is just a shorter, and somewhat standard way, to create a new
>> uninterned symbol with a given prefix. You seem to re-invent it. What
>> you do with that new symbol is orthogonal to that suggestion, of course.
>>
>>>> OK, but this may not be sufficient if we want to do slightly better than
>>>> overlays in that area. This is not mandatory, though.
>>>
>>> Could you elaborate on what can be "slightly better"? 
>>
>> IIRC, I gave examples of finer control of folding state after a change.
>> Consider this _folded_ drawer:
>>
>>   :BEGIN:
>>   Foo
>>   :END:
>>
>> Inserting ":END" in it should not unfold it, as it is currently the case
>> with overlays,
>>
>>   :BEGIN
>>   Foo
>>   :END
>>   :END:
>>
>> but a soon as the last ":" is inserted, the initial drawer could be
>> expanded.
>>
>>   :BEGIN
>>   Foo
>>   :END:
>>   :END:
>>
>> The latter case is not currently handled by overlays. This is what
>> I call "slightly better".
>>
>> Also, note that this change is not related to opening and closing lines
>> of the initial drawer, so sticking text properties on them would not
>> help here.
>>
>> Another case is modifying those borders, e.g.,
>>
>>
>>   :BEGIN:               :BEGIN:
>>   Foo         ------>   Foo  
>>   :END:                 :ND:
>>
>> which should expand the drawer. Your implementation catches this, but
>> I'm pointing out that current implementation with overlays does not.
>> Even though that's not strictly required for compatibility with
>> overlays, it is a welcome slight improvement.
>>
>>>> As discussed before, I don't think you need to use `modification-hooks'
>>>> or `insert-behind-hooks' if you already use `after-change-functions'.
>>>>
>>>> `after-change-functions' are also triggered upon text properties
>>>> changes. So, what is the use case for the other hooks?
>>>
>>> The problem is that `after-change-functions' cannot be a text property.
>>> Only `modification-hooks' and `insert-in-front/behind-hooks' can be a
>>> valid text property. If we use `after-change-functions', they will
>>> always be triggered, regardless if the change was made inside or outside
>>> folded region.
>>
>> As discussed, text properties are local to the change, but require extra
>> care when moving text around. You also observed serious overhead when
>> using them.
>>
>> OTOH, even if `a-c-f' is not local, you can quickly determine if the
>> change altered a folded element, so the overhead is limited, i.e.,
>> mostly checking for a text property at a given buffer position.
>>
>> To be clear, I initially thought that text properties were a superior
>> choice, but I changed my mind a while ago, and I thought you had, too.
>> IOW, `after-change-functions' is the way to go, since you have no strong
>> reason to stick to text properties for this kind of function.
>>
>>>>> :asd:
>>>>> :drawer:
>>>>> lksjdfksdfjl
>>>>> sdfsdfsdf
>>>>> :end:
>>>>>
>>>>> If :asd: was inserted in front of folded :drawer:, changes in :drawer:
>>>>> line of the new folded :asd: drawer would reveal the text between
>>>>> :drawer: and :end:.
>>>>>
>>>>> Let me know what you think on this.
>>>
>>>> I have first to understand the use case for `modification-hook'. But
>>>> I think unfolding is the right thing to do in this situation, isn't it?
>>>
>>> That situation arises because the modification-hooks from ":drawer:"
>>> (they are set via text properties) only have information about the
>>> :drawer:...:end: drawer before the modifications (they were set when
>>> :drawer: was folded last time). So, they will only unfold a part of the
>>> new :asd: drawer. I do not see a simple way to unfold everything without
>>> re-parsing the drawer around the changed text.
>>
>> Oh! I misread your message. I withdraw what I wrote. In this case, we
>> don't want to unfold anything. The situation is not worse than what we
>> have now, and trying to fix it would have repercussions down in the
>> buffer, e.g., expanding drawers screen below.
>>
>> As a rule of thumb, I think we can pay attention to changes in the
>> folded text, and its immediate surroundings (e.g., the opening line,
>> which is not folded), but no further.
>>
>> As written above, slight changes are welcome, but let's not go overboard
>> and parse a whole section just to know if we can expand a drawer.
>>
>>> Actually, I am quite unhappy with the performance of modification-hooks
>>> set via text properties (I am using this patch on my Emacs during this
>>> week). It appears that setting the text properties costs a significant
>>> CPU time in practice, even though running the hooks is pretty fast.
>>> I will think about a way to handle modifications using global
>>> after-change-functions.
>>
>> That's better, IMO.
>>
>> I gave you a few ideas to quickly check if a change requires expansion,
>> in an earlier mail. I suggest to start out from that. Let me know if you
>> have questions about it.
>
> -- 
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-08  5:05                                           ` Ihor Radchenko
  2020-06-08  5:06                                             ` Ihor Radchenko
  2020-06-08  5:08                                             ` Ihor Radchenko
@ 2020-06-10 17:14                                             ` Nicolas Goaziou
  2020-06-21  9:52                                               ` Ihor Radchenko
  2 siblings, 1 reply; 55+ messages in thread
From: Nicolas Goaziou @ 2020-06-10 17:14 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> [The patch itself will be provided in the following email]

Thank you! I'll first make some generic remarks, then comment the diff
in more details.

> I have four more updates from the previous version of the patch:
>
> 1. All the code handling modifications in folded drawers/blocks is moved
>    to after-change-function. It works as follows:
>    - if any text is inserted in the middle of hidden region, that text
>      is also hidden;
>    - if BEGIN/END line of a folded drawer do not match org-drawer-regexp
>      and org-property-end-re, unfold it; 
>    - if org-property-end-re or new org-outline-regexp-bol is inserted in
>      the middle of the drawer, unfold it;
>    - the same logic for blocks.

This sounds good, barring a minor error in the regexp for blocks, and
missing optimizations. More on this in the detailed comments.

> 2. The text property stack is rewritten using char-property-alias-alist.
>    This is faster in comparison with previous approach, which involved
>    modifying all the text properties every timer org-flag-region was
>    called. 

I'll need information about this, as I'm not sure to fully understand
all the consequences of this. But more importantly, this needs to be
copiously documented somewhere for future hackers.

> 3. org-toggle-custom-properties-visibility is rewritten using text
>    properties. I also took a freedom to implement a new feature here.
>    Now, setting new `org-custom-properties-hide-emptied-drawers' to
>    non-nil will result in hiding the whole property drawer if it
>    contains only org-custom-properties.

I don't think this is a good idea. AFAIR, we always refused to hide
completely anything, including empty drawers. The reason is that if the
drawer is completely hidden, you cannot expand it easily, or even know
there is one.

In any case, this change shouldn't belong to this patch set, and should
be discussed separately.

> 4. This patch should work against 1aa095ccf. However, the merge was not
>    trivial here. Recent commits actively used the fact that drawers and
>    outlines are hidden via 'outline invisibility spec, which is not the
>    case in this branch. I am not confident that I did not break anything
>    during the merge, especially 1aa095ccf.

[...]

> Also, I have seen some optimisations making use of the fact that drawers
> and headlines both use 'outline invisibility spec. This change in the
> implementation details supposed to improve performance and should not be
> necessary if this patch is going to be merged. Would it be possible to
> refrain from abusing this particular implementation detail in the
> nearest commits on master (unless really necessary)?

To be clear, I didn't intend to make your life miserable.

However, I had to fix regression on drawers visibility before Org 9.4
release. Also, merging invisibility properties for drawers and outline
was easier for me. So, I had the opportunity to kill two birds with one
stone. 

As a reminder, Org 9.4 is about to be released, but Org 9.5 will take
months to go out. So, even though I hope your changes will land into
Org, there is no reason for us to refrain from improving (actually
fixing a regression in) 9.4 release. Hopefully, once 9.4 is out, such
changes are not expected to happen anymore.

I hope you understand.

> I would like to finalise the current patch and work on other code using
> overlays separately. This patch is already quite complicated as is. I do
> not want to introduce even more potential bugs by working on things not
> directly affected by this version of the patch.

The patch is technically mostly good, but needs more work for
integration into Org.

First, it includes a few unrelated changes that should be removed (e.g.,
white space fixes in unrelated parts of the code). Also, as written
above, the changes about `org-custom-properties-hide-emptied-drawers'
should be removed for the time being.

Once done, I think we should move (or copy, first) _all_ folding-related
functions into a new "org-fold.el" library. Functions and variables
included there should have a proper "org-fold-" prefix. More on this in
the detailed report.

The functions `org-find-text-property-region',
`org-add-to-list-text-property', and
`org-remove-from-list-text-property' can be left in "org-macs.el", since
they do not directly depend on the `invisible' property. Note the last
two functions I mentioned are not used throughout your patch. They might
be removed.

This first patch can coexist with overlay folding since functions in
both mechanisms are named differently.

Then, another patch can integrate "org-fold.el" into Org folding. I also
suggest to move the Outline -> Org transition to yet another patch.
I think there's more work to do on this part.

Now, into the details of your patch. The first remarks are:

1. we still support Emacs 24.4 (and probably Emacs 24.3, but I'm not
   sure), so some functions cannot be used.

2. we don't use "subr-x.el" in the code base. In particular, it would be
   nice to replace `when-let' with `when' + `let'. This change costs
   only one loc.

3. Some docstrings need more work. In particular, Emacs documentation
   expects all arguments to be explained in the docstring, if possible
   in the order in which they appear. There are exceptions, though. For
   example, in a function like `org-remove-text-properties', you can
   mention arguments are simply the same as in `remove-text-properties'.

4. Some refactorization is needed in some places. I mentioned them in
   the report below.

5. I didn't dive much into the Isearch code so far. I tested it a bit
   and seems to work nicely. I noticed one bug though. In the following
   document:

       #+begin: foo
       :FOO:
       bar
       :END:
       #+end
       bar

   when both the drawer and the block are folded (i.e., you fold the
   drawer first, then the block), searching for "bar" first find the
   last one, then overwraps and find the first one.

6. Since we're rewriting folding code, we might as well rename folding
   properties: org-hide-drawer -> org-fold-drawer, outline ->
   org-fold-headline…

Now, here are more comments about the code.

-----

> +(defun org-remove-text-properties (start end properties &optional object)

IMO, this generic name doesn't match the specialized nature of the
function. It doesn't belong to "org-macs.el", but to the new "Org Fold" library.

> +  "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions.

Line is too long. Suggestion:

   Remove text properties except folding-related ones.

> +Do not remove invisible text properties specified by 'outline,
> +'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this
> +is needed to keep outlines, drawers, and blocks hidden unless they are
> +toggled by user.

Said properties should be moved into a defconst, e.g.,
`org-fold-properties', then:

  Remove text properties as in `remove-text-properties'.  See the
  function for the description of the arguments.

  However, do not remove invisible text properties defined in
  `org-fold-properties'. Those are required to keep headlines, drawers
  and blocks folded.

> +Note: The below may be too specific and create troubles if more
> +invisibility specs are added to org in future"

You can remove the note. If you think the note is important, it should
put a comment in the code instead.

> +  (when (plist-member properties 'invisible)
> +    (let ((pos start)
> +	  next spec)
> +      (while (< pos end)
> +	(setq next (next-single-property-change pos 'invisible nil end)
> +              spec (get-text-property pos 'invisible))
> +	(unless (memq spec (list 'org-hide-block
> +				 'org-hide-drawer
> +				 'outline))

The (list ...) should be moved outside the `while' loop. Better, this
should be a constant defined somewhere. I also suggest to move
`outline' to `org-outline' since we differ from Outline mode.

> +          (remove-text-properties pos next '(invisible nil) object))
> +	(setq pos next))))
> +  (when-let ((properties-stripped (org-plist-delete properties 'invisible)))

Typo here. There should a single pair of parenthesis, but see above
about `when-let'.

> +    (remove-text-properties start end properties-stripped object)))
> +
> +(defun org--find-text-property-region (pos prop)

I think this is a function useful enough to have a name without double
dashes. It can be left in "org-macs.el". It would be nice to have
a wrapper for `invisible' property in "org-fold.el", tho.

> +  "Find a region containing PROP text property around point POS."

Reverse the order of arguments in the docstring:

  Find a region around POS containing PROP text property.

> +  (let* ((beg (and (get-text-property pos prop) pos))
> +	 (end beg))
> +    (when beg

BEG can only be nil if arguments are wrong. In this case, you can
throw an error (assuming this is no longer an internal function):

  (unless beg (user-error "..."))

> +      ;; when beg is the first point in the region, `previous-single-property-change'
> +      ;; will return nil.

when -> When

> +      (setq beg (or (previous-single-property-change pos prop)
> +		    beg))
> +      ;; when end is the last point in the region, `next-single-property-change'
> +      ;; will return nil.

Ditto.

> +      (setq end (or (next-single-property-change pos prop)
> +		    end))
> +      (unless (= beg end) ; this should not happen

I assume this will be the case in an empty buffer. Anyway, (1 . 1)
sounds more regular than a nil return value, not specified in the
docstring. IOW, I suggest to remove this check.

> +        (cons beg end)))))
> +
> +(defun org--add-to-list-text-property (from to prop element)
> +  "Add element to text property PROP, whos value should be a list."

The docstring is incomplete. All arguments need to be described. Also,
I suggest:

  Append ELEMENT to the value of text property PROP.

> +  (add-text-properties from to `(,prop ,(list element))) ; create if none

Here, you are resetting all the properties before adding anything,
aren't you? IOW, there might be a bug there.

> +  ;; add to existing
> +  (alter-text-property from to
> +		       prop
> +		       (lambda (val)
> +			 (if (member element val)
> +                             val
> +			   (cons element val)))))

> +(defun org--remove-from-list-text-property (from to prop element)
> +  "Remove ELEMENT from text propery PROP, whos value should be a list."

The docstring needs to be improved.

> +  (let ((pos from))
> +    (while (< pos to)
> +      (when-let ((val (get-text-property pos prop)))
> +	(if (equal val (list element))

(list element) needs to be moved out of the `while' loop.

> +	    (remove-text-properties pos (next-single-char-property-change pos prop nil to) (list prop nil))
> +	  (put-text-property pos (next-single-char-property-change pos prop nil to)
> +			     prop (remove element (get-text-property pos prop)))))

If we specialize the function, `remove' -> `remq'

> +      (setq pos (next-single-char-property-change pos prop nil to)))))

Please factor out `next-single-char-property-change'.

Note that `org--remove-from-list-text-property' and
`org--add-to-list-text-property' do not seem to be used throughout
your patch.

> +(defvar org--invisible-spec-priority-list '(outline org-hide-drawer org-hide-block)
> +  "Priority of invisibility specs.")

This should be the constant I wrote about earlier. Note that those are
not "specs", just properties. I suggest to rename it.

> +(defun org--get-buffer-local-invisible-property-symbol (spec &optional buffer return-only)

This name is waaaaaaay too long.

> +  "Return unique symbol suitable to be used as buffer-local in BUFFER for 'invisible SPEC.

Maybe:


  Return a unique symbol suitable for `invisible' property.

Then:

  Return value is meant to be used as a buffer-local variable in
  current buffer, or BUFFER if this is non-nil.

> +If the buffer already have buffer-local setup in `char-property-alias-alist'
> +and the setup appears to be created for different buffer,
> +copy the old invisibility state into new buffer-local text properties,
> +unless RETURN-ONLY is non-nil."
> +  (if (not (member spec org--invisible-spec-priority-list))
> +      (user-error "%s should be a valid invisibility spec" spec)

No need to waste an indentation level for that:

  (unless (member …)
   (user-error "%S should be …" spec))

Also, this is a property, not a "spec".

> +    (let* ((buf (or buffer (current-buffer))))
> +      (let ((local-prop (intern (format "org--invisible-%s-buffer-local-%S"

This clearly needs a shorter name. In particular, "buffer-local" can be removed.

> +					(symbol-name spec)
> +					;; (sxhash buf) appears to be not constant over time.
> +					;; Using buffer-name is safe, since the only place where
> +					;; buffer-local text property actually matters is an indirect
> +					;; buffer, where the name cannot be same anyway.
> +					(sxhash (buffer-name buf))))))


> +        (prog1
> +            local-prop

Please move LOCAL-PROP after the (unless return-only ...) sexp.

> +          (unless return-only
> +	    (with-current-buffer buf
> +	      (unless (member local-prop (alist-get 'invisible char-property-alias-alist))
> +		;; copy old property

"Copy old property."

> +		(dolist (old-prop (alist-get 'invisible char-property-alias-alist))

We cannot use `alist-get', which was added in Emacs 25.3 only.

> +		  (org-with-wide-buffer
> +		   (let* ((pos (point-min))
> +			  (spec (seq-find (lambda (spec)
> +					    (string-match-p (symbol-name spec)
> +							    (symbol-name old-prop)))
> +					  org--invisible-spec-priority-list))

Likewise, we cannot use `seq-find'.

> +			  (new-prop (org--get-buffer-local-invisible-property-symbol spec nil 'return-only)))
> +		     (while (< pos (point-max))
> +		       (when-let (val (get-text-property pos old-prop))
> +			 (put-text-property pos (next-single-char-property-change pos old-prop) new-prop val))
> +		       (setq pos (next-single-char-property-change pos old-prop))))))
> +		(setq-local char-property-alias-alist
> +			    (cons (cons 'invisible
> +					(mapcar (lambda (spec)
> +						  (org--get-buffer-local-invisible-property-symbol spec nil 'return-only))
> +						org--invisible-spec-priority-list))
> +				  (remove (assq 'invisible char-property-alias-alist)
> +					  char-property-alias-alist)))))))))))

This begs for explainations in the docstring or as comments. In
particular, just by reading the code, I have no clue about how this is
going to be used, how it is going to solve issues with indirect
buffers, with invisibility stacking, etc.

I don't mind if there are more comment lines than lines of code in
that area.

> -  (remove-overlays from to 'invisible spec)
> -  ;; Use `front-advance' since text right before to the beginning of
> -  ;; the overlay belongs to the visible line than to the contents.
> -  (when flag
> -    (let ((o (make-overlay from to nil 'front-advance)))
> -      (overlay-put o 'evaporate t)
> -      (overlay-put o 'invisible spec)
> -      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
> -
> +  (with-silent-modifications
> +    (remove-text-properties from to (list (org--get-buffer-local-invisible-property-symbol spec) nil))
> +    (when flag
> +      (put-text-property from to (org--get-buffer-local-invisible-property-symbol spec) spec))))

I don't think there is a need for `remove-text-properties' in every
case. Also, (org--get-buffer-local-invisible-property-symbol spec)
should be factored out. 

I suggest:

  (with-silent-modifications
    (let ((property (org--get-buffer-local-invisible-property-symbol spec)))
      (if flag
          (put-text-property from to property spec)
        (remove-text-properties from to (list property nil)))))

> +(defun org-after-change-function (from to len)

This is a terrible name. Org may add different functions in a-c-f,
they cannot all be called like this. Assuming the "org-fold" prefix,
it could be:

  org-fold--fix-folded-region

> +  "Process changes in folded elements.
> +If a text was inserted into invisible region, hide the inserted text.
> +If the beginning/end line of a folded drawer/block was changed, unfold it.
> +If a valid end line was inserted in the middle of the folded drawer/block, unfold it."

Nitpick: please do not skip lines amidst a function. Empty lines are
used to separate functions, so this is distracting. 

If a part of the function should stand out, a comment explaining what
the part is doing is enough.

> +  ;; re-hide text inserted in the middle of a folded region

  Re-hide … folded region.

> +  (dolist (spec org--invisible-spec-priority-list)
> +    (when-let ((spec-to (get-text-property to (org--get-buffer-local-invisible-property-symbol spec)))
> +	       (spec-from (get-text-property (max (point-min) (1- from)) (org--get-buffer-local-invisible-property-symbol spec))))
> +      (when (eq spec-to spec-from)
> +	(org-flag-region from to 't spec-to))))

This part should first check if we're really after an insertion, e.g.,
if FROM is different from TO, and exit early if that's not the case.

Also, no need to quote t.

> +  ;; Process all the folded text between `from' and `to'
> +  (org-with-wide-buffer
> +
> +   (if (< to from)
> +       (let ((tmp from))
> +	 (setq from to)
> +         (setq to tmp)))

I'm surprised you need to do that. Did you encounter a case where
a-c-f was called with boundaries in reverse order?

> +   ;; Include next/previous line into the changed region.
> +   ;; This is needed to catch edits in beginning line of a folded
> +   ;; element.
> +   (setq to (save-excursion (goto-char to) (forward-line) (point)))

(forward-line) (point)  ---> (line-beginning-position 2)

> +   (setq from (save-excursion (goto-char from) (forward-line -1) (point)))

(forward-line -1) (point)  ---> (line-beginning-position 0)

Anyway, I have the feeling this is not a good idea to extend it now,
without first checking that we are in a folded drawer or block. It may
also catch unwanted parts, e.g., a folded drawer ending on the line
above.

What about first finding the whole region with property

  (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)

then extending the initial part to include the drawer opening? I don't
think we need to extend past the ending part, because drawer closing
line is always included in the invisible part of the drawer.

> +   ;; Expand the considered region to include partially present folded
> +   ;; drawer/block.
> +   (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))
> +     (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))
> +   (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-block))
> +     (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))

Please factor out (org--get-buffer-local-invisible-property-symbol
XXX), this is difficult to read.

> +   ;; check folded drawers

  Check folded drawers.

> +   (let ((pos from))
> +     (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))
> +       (setq pos (next-single-char-property-change pos
> +						   (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))
> +     (while (< pos to)
> +       (when-let ((drawer-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))
> +				     pos))
> +		  (drawer-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))
> +
> +	 (let (unfold?)
> +           ;; the line before folded text should be beginning of the drawer
> +           (save-excursion
> +             (goto-char drawer-begin)
> +             (backward-char)

Why `backward-char'?

> +             (beginning-of-line)
> +             (unless (looking-at-p org-drawer-regexp)

   looking-at-p ---> looking-at

However, you must wrap this function within `save-match-data'.

> +	       (setq unfold? t)))
> +           ;; the last line of the folded text should be :END:
> +           (save-excursion
> +             (goto-char drawer-end)
> +             (beginning-of-line)
> +             (unless (let ((case-fold-search t)) (looking-at-p org-property-end-re))
> +	       (setq unfold? t)))
> +           ;; there should be no :END: anywhere in the drawer body
> +           (save-excursion
> +             (goto-char drawer-begin)
> +             (when (save-excursion
> +		     (let ((case-fold-search t))
> +		       (re-search-forward org-property-end-re
> +					  (max (point)
> +					       (1- (save-excursion
> +						     (goto-char drawer-end)
> +                                                     (line-beginning-position))))
> +                                          't)))

>  (max (point) 
>       (save-excursion (goto-char drawer-end) (line-end-position 0))

> +	       (setq unfold? t)))
> +           ;; there should be no new entry anywhere in the drawer body
> +           (save-excursion
> +             (goto-char drawer-begin)
> +             (when (save-excursion
> +		     (let ((case-fold-search t))
> +		       (re-search-forward org-outline-regexp-bol
> +					  (max (point)
> +					       (1- (save-excursion
> +						     (goto-char drawer-end)
> +                                                     (line-beginning-position))))
> +                                          't)))
> +	       (setq unfold? t)))

In the phase above, you need to bail out as soon as unfold? is non-nil:

 (catch :exit
  ...
  (throw :exit (setq unfold? t))
  ...)

Also last two checks should be lumped together, with an appropriate
regexp.

Finally, I have the feeling we're missing out some early exits when
nothing is folded around point (e.g., most of the case).

> +
> +           (when unfold? (org-flag-region drawer-begin drawer-end nil 'org-hide-drawer))))
> +       
> +       (setq pos (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))))
> +
> +   ;; check folded blocks
> +   (let ((pos from))
> +     (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block))
> +       (setq pos (next-single-char-property-change pos
> +						   (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))
> +     (while (< pos to)
> +       (when-let ((block-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block))
> +				    pos))
> +		  (block-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))
> +
> +	 (let (unfold?)
> +           ;; the line before folded text should be beginning of the block
> +           (save-excursion
> +             (goto-char block-begin)
> +             (backward-char)
> +             (beginning-of-line)
> +             (unless (looking-at-p org-dblock-start-re)
> +	       (setq unfold? t)))
> +           ;; the last line of the folded text should be end of the block
> +           (save-excursion
> +             (goto-char block-end)
> +             (beginning-of-line)
> +             (unless (looking-at-p org-dblock-end-re)
> +	       (setq unfold? t)))
> +           ;; there should be no #+end anywhere in the block body
> +           (save-excursion
> +             (goto-char block-begin)
> +             (when (save-excursion
> +		     (re-search-forward org-dblock-end-re
> +					(max (point)
> +					     (1- (save-excursion
> +						   (goto-char block-end)
> +						   (line-beginning-position))))
> +                                        't))
> +	       (setq unfold? t)))
> +           ;; there should be no new entry anywhere in the block body
> +           (save-excursion
> +             (goto-char block-begin)
> +             (when (save-excursion
> +		     (let ((case-fold-search t))
> +		       (re-search-forward org-outline-regexp-bol
> +					  (max (point)
> +					       (1- (save-excursion
> +						     (goto-char block-end)
> +                                                     (line-beginning-position))))
> +                                          't)))
> +	       (setq unfold? t)))
> +
> +           (when unfold? (org-flag-region block-begin block-end nil 'org-hide-block))))
> +       
> +       (setq pos
> +	     (next-single-char-property-change pos
> +					       (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))))))

See remarks above. The parts related to drawers and blocks are so
similar they should be factorized out.

Also `org-dblock-start-re' and `org-dblock-end-re' are not regexps we
want here. The correct regexps would be:

  (rx bol
      (zero-or-more (any " " "\t"))
      "#+begin"
      (or ":" 
          (seq "_" 
              (group (one-or-more (not (syntax whitespace)))))))

and closing line should match match-group 1 from the regexp above, e.g.:

  (concat (rx bol (zero-or-more (any " " "\t")) "#+end")
          (if block-type
              (concat "_"
                      (regexp-quote block-type)
                      (rx (zero-or-more (any " " "\t")) eol))
            (rx (opt ":") (zero-or-more (any " " "\t")) eol)))

assuming `block-type' is the type of the block, or nil, i.e.,
(match-string 1) in the previous regexp.

> -	  (pcase (get-char-property-and-overlay (point) 'invisible)
> +	  (pcase (get-char-property (point) 'invisible)
> 	    ;; Do not fold already folded drawers.
> -	    (`(outline . ,o) (goto-char (overlay-end o)))
> +	    ('outline

'outline --> `outline
 
>      (end-of-line))
>    (while (and (< arg 0) (re-search-backward regexp nil :move))
>      (unless (bobp)
> -	(while (pcase (get-char-property-and-overlay (point) 'invisible)
> -		 (`(outline . ,o)
> -		  (goto-char (overlay-start o))
> -		  (re-search-backward regexp nil :move))
> -		 (_ nil))))
> +	(pcase (get-char-property (point) 'invisible)
> +	  ('outline
> +	   (goto-char (car (org--find-text-property-region (point) 'invisible)))
> +	   (beginning-of-line))
> +	  (_ nil)))

Does this move to the beginning of the widest invisible part around
point? If that's not the case, we need a function in "org-fold.el"
doing just that. Or we need to nest `while' loops as it was the case
in the code you reverted.

-----

Regards,

-- 
Nicolas Goaziou


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-10 17:14                                             ` Nicolas Goaziou
@ 2020-06-21  9:52                                               ` Ihor Radchenko
  2020-06-21 15:01                                                 ` Nicolas Goaziou
  0 siblings, 1 reply; 55+ messages in thread
From: Ihor Radchenko @ 2020-06-21  9:52 UTC (permalink / raw)
  To: Nicolas Goaziou; +Cc: emacs-orgmode

> Once done, I think we should move (or copy, first) _all_ folding-related
> functions into a new "org-fold.el" library. Functions and variables
> included there should have a proper "org-fold-" prefix. More on this in
> the detailed report.

I am currently working on org-fold.el. However, I am not sure if it is
acceptable to move some of the existing functions from org.el to
org-fold.el.

Specifically, functions from the following sections of org.el might be
moved to org-fold.el:
> ;;; Visibility (headlines, blocks, drawers)
> ;;;; Reveal point location
> ;;;; Visibility cycling

Should I do it?

Best,
Ihor

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> [The patch itself will be provided in the following email]
>
> Thank you! I'll first make some generic remarks, then comment the diff
> in more details.
>
>> I have four more updates from the previous version of the patch:
>>
>> 1. All the code handling modifications in folded drawers/blocks is moved
>>    to after-change-function. It works as follows:
>>    - if any text is inserted in the middle of hidden region, that text
>>      is also hidden;
>>    - if BEGIN/END line of a folded drawer do not match org-drawer-regexp
>>      and org-property-end-re, unfold it; 
>>    - if org-property-end-re or new org-outline-regexp-bol is inserted in
>>      the middle of the drawer, unfold it;
>>    - the same logic for blocks.
>
> This sounds good, barring a minor error in the regexp for blocks, and
> missing optimizations. More on this in the detailed comments.
>
>> 2. The text property stack is rewritten using char-property-alias-alist.
>>    This is faster in comparison with previous approach, which involved
>>    modifying all the text properties every timer org-flag-region was
>>    called. 
>
> I'll need information about this, as I'm not sure to fully understand
> all the consequences of this. But more importantly, this needs to be
> copiously documented somewhere for future hackers.
>
>> 3. org-toggle-custom-properties-visibility is rewritten using text
>>    properties. I also took a freedom to implement a new feature here.
>>    Now, setting new `org-custom-properties-hide-emptied-drawers' to
>>    non-nil will result in hiding the whole property drawer if it
>>    contains only org-custom-properties.
>
> I don't think this is a good idea. AFAIR, we always refused to hide
> completely anything, including empty drawers. The reason is that if the
> drawer is completely hidden, you cannot expand it easily, or even know
> there is one.
>
> In any case, this change shouldn't belong to this patch set, and should
> be discussed separately.
>
>> 4. This patch should work against 1aa095ccf. However, the merge was not
>>    trivial here. Recent commits actively used the fact that drawers and
>>    outlines are hidden via 'outline invisibility spec, which is not the
>>    case in this branch. I am not confident that I did not break anything
>>    during the merge, especially 1aa095ccf.
>
> [...]
>
>> Also, I have seen some optimisations making use of the fact that drawers
>> and headlines both use 'outline invisibility spec. This change in the
>> implementation details supposed to improve performance and should not be
>> necessary if this patch is going to be merged. Would it be possible to
>> refrain from abusing this particular implementation detail in the
>> nearest commits on master (unless really necessary)?
>
> To be clear, I didn't intend to make your life miserable.
>
> However, I had to fix regression on drawers visibility before Org 9.4
> release. Also, merging invisibility properties for drawers and outline
> was easier for me. So, I had the opportunity to kill two birds with one
> stone. 
>
> As a reminder, Org 9.4 is about to be released, but Org 9.5 will take
> months to go out. So, even though I hope your changes will land into
> Org, there is no reason for us to refrain from improving (actually
> fixing a regression in) 9.4 release. Hopefully, once 9.4 is out, such
> changes are not expected to happen anymore.
>
> I hope you understand.
>
>> I would like to finalise the current patch and work on other code using
>> overlays separately. This patch is already quite complicated as is. I do
>> not want to introduce even more potential bugs by working on things not
>> directly affected by this version of the patch.
>
> The patch is technically mostly good, but needs more work for
> integration into Org.
>
> First, it includes a few unrelated changes that should be removed (e.g.,
> white space fixes in unrelated parts of the code). Also, as written
> above, the changes about `org-custom-properties-hide-emptied-drawers'
> should be removed for the time being.
>
> Once done, I think we should move (or copy, first) _all_ folding-related
> functions into a new "org-fold.el" library. Functions and variables
> included there should have a proper "org-fold-" prefix. More on this in
> the detailed report.
>
> The functions `org-find-text-property-region',
> `org-add-to-list-text-property', and
> `org-remove-from-list-text-property' can be left in "org-macs.el", since
> they do not directly depend on the `invisible' property. Note the last
> two functions I mentioned are not used throughout your patch. They might
> be removed.
>
> This first patch can coexist with overlay folding since functions in
> both mechanisms are named differently.
>
> Then, another patch can integrate "org-fold.el" into Org folding. I also
> suggest to move the Outline -> Org transition to yet another patch.
> I think there's more work to do on this part.
>
> Now, into the details of your patch. The first remarks are:
>
> 1. we still support Emacs 24.4 (and probably Emacs 24.3, but I'm not
>    sure), so some functions cannot be used.
>
> 2. we don't use "subr-x.el" in the code base. In particular, it would be
>    nice to replace `when-let' with `when' + `let'. This change costs
>    only one loc.
>
> 3. Some docstrings need more work. In particular, Emacs documentation
>    expects all arguments to be explained in the docstring, if possible
>    in the order in which they appear. There are exceptions, though. For
>    example, in a function like `org-remove-text-properties', you can
>    mention arguments are simply the same as in `remove-text-properties'.
>
> 4. Some refactorization is needed in some places. I mentioned them in
>    the report below.
>
> 5. I didn't dive much into the Isearch code so far. I tested it a bit
>    and seems to work nicely. I noticed one bug though. In the following
>    document:
>
>        #+begin: foo
>        :FOO:
>        bar
>        :END:
>        #+end
>        bar
>
>    when both the drawer and the block are folded (i.e., you fold the
>    drawer first, then the block), searching for "bar" first find the
>    last one, then overwraps and find the first one.
>
> 6. Since we're rewriting folding code, we might as well rename folding
>    properties: org-hide-drawer -> org-fold-drawer, outline ->
>    org-fold-headline…
>
> Now, here are more comments about the code.
>
> -----
>
>> +(defun org-remove-text-properties (start end properties &optional object)
>
> IMO, this generic name doesn't match the specialized nature of the
> function. It doesn't belong to "org-macs.el", but to the new "Org Fold" library.
>
>> +  "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions.
>
> Line is too long. Suggestion:
>
>    Remove text properties except folding-related ones.
>
>> +Do not remove invisible text properties specified by 'outline,
>> +'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this
>> +is needed to keep outlines, drawers, and blocks hidden unless they are
>> +toggled by user.
>
> Said properties should be moved into a defconst, e.g.,
> `org-fold-properties', then:
>
>   Remove text properties as in `remove-text-properties'.  See the
>   function for the description of the arguments.
>
>   However, do not remove invisible text properties defined in
>   `org-fold-properties'. Those are required to keep headlines, drawers
>   and blocks folded.
>
>> +Note: The below may be too specific and create troubles if more
>> +invisibility specs are added to org in future"
>
> You can remove the note. If you think the note is important, it should
> put a comment in the code instead.
>
>> +  (when (plist-member properties 'invisible)
>> +    (let ((pos start)
>> +	  next spec)
>> +      (while (< pos end)
>> +	(setq next (next-single-property-change pos 'invisible nil end)
>> +              spec (get-text-property pos 'invisible))
>> +	(unless (memq spec (list 'org-hide-block
>> +				 'org-hide-drawer
>> +				 'outline))
>
> The (list ...) should be moved outside the `while' loop. Better, this
> should be a constant defined somewhere. I also suggest to move
> `outline' to `org-outline' since we differ from Outline mode.
>
>> +          (remove-text-properties pos next '(invisible nil) object))
>> +	(setq pos next))))
>> +  (when-let ((properties-stripped (org-plist-delete properties 'invisible)))
>
> Typo here. There should a single pair of parenthesis, but see above
> about `when-let'.
>
>> +    (remove-text-properties start end properties-stripped object)))
>> +
>> +(defun org--find-text-property-region (pos prop)
>
> I think this is a function useful enough to have a name without double
> dashes. It can be left in "org-macs.el". It would be nice to have
> a wrapper for `invisible' property in "org-fold.el", tho.
>
>> +  "Find a region containing PROP text property around point POS."
>
> Reverse the order of arguments in the docstring:
>
>   Find a region around POS containing PROP text property.
>
>> +  (let* ((beg (and (get-text-property pos prop) pos))
>> +	 (end beg))
>> +    (when beg
>
> BEG can only be nil if arguments are wrong. In this case, you can
> throw an error (assuming this is no longer an internal function):
>
>   (unless beg (user-error "..."))
>
>> +      ;; when beg is the first point in the region, `previous-single-property-change'
>> +      ;; will return nil.
>
> when -> When
>
>> +      (setq beg (or (previous-single-property-change pos prop)
>> +		    beg))
>> +      ;; when end is the last point in the region, `next-single-property-change'
>> +      ;; will return nil.
>
> Ditto.
>
>> +      (setq end (or (next-single-property-change pos prop)
>> +		    end))
>> +      (unless (= beg end) ; this should not happen
>
> I assume this will be the case in an empty buffer. Anyway, (1 . 1)
> sounds more regular than a nil return value, not specified in the
> docstring. IOW, I suggest to remove this check.
>
>> +        (cons beg end)))))
>> +
>> +(defun org--add-to-list-text-property (from to prop element)
>> +  "Add element to text property PROP, whos value should be a list."
>
> The docstring is incomplete. All arguments need to be described. Also,
> I suggest:
>
>   Append ELEMENT to the value of text property PROP.
>
>> +  (add-text-properties from to `(,prop ,(list element))) ; create if none
>
> Here, you are resetting all the properties before adding anything,
> aren't you? IOW, there might be a bug there.
>
>> +  ;; add to existing
>> +  (alter-text-property from to
>> +		       prop
>> +		       (lambda (val)
>> +			 (if (member element val)
>> +                             val
>> +			   (cons element val)))))
>
>> +(defun org--remove-from-list-text-property (from to prop element)
>> +  "Remove ELEMENT from text propery PROP, whos value should be a list."
>
> The docstring needs to be improved.
>
>> +  (let ((pos from))
>> +    (while (< pos to)
>> +      (when-let ((val (get-text-property pos prop)))
>> +	(if (equal val (list element))
>
> (list element) needs to be moved out of the `while' loop.
>
>> +	    (remove-text-properties pos (next-single-char-property-change pos prop nil to) (list prop nil))
>> +	  (put-text-property pos (next-single-char-property-change pos prop nil to)
>> +			     prop (remove element (get-text-property pos prop)))))
>
> If we specialize the function, `remove' -> `remq'
>
>> +      (setq pos (next-single-char-property-change pos prop nil to)))))
>
> Please factor out `next-single-char-property-change'.
>
> Note that `org--remove-from-list-text-property' and
> `org--add-to-list-text-property' do not seem to be used throughout
> your patch.
>
>> +(defvar org--invisible-spec-priority-list '(outline org-hide-drawer org-hide-block)
>> +  "Priority of invisibility specs.")
>
> This should be the constant I wrote about earlier. Note that those are
> not "specs", just properties. I suggest to rename it.
>
>> +(defun org--get-buffer-local-invisible-property-symbol (spec &optional buffer return-only)
>
> This name is waaaaaaay too long.
>
>> +  "Return unique symbol suitable to be used as buffer-local in BUFFER for 'invisible SPEC.
>
> Maybe:
>
>
>   Return a unique symbol suitable for `invisible' property.
>
> Then:
>
>   Return value is meant to be used as a buffer-local variable in
>   current buffer, or BUFFER if this is non-nil.
>
>> +If the buffer already have buffer-local setup in `char-property-alias-alist'
>> +and the setup appears to be created for different buffer,
>> +copy the old invisibility state into new buffer-local text properties,
>> +unless RETURN-ONLY is non-nil."
>> +  (if (not (member spec org--invisible-spec-priority-list))
>> +      (user-error "%s should be a valid invisibility spec" spec)
>
> No need to waste an indentation level for that:
>
>   (unless (member …)
>    (user-error "%S should be …" spec))
>
> Also, this is a property, not a "spec".
>
>> +    (let* ((buf (or buffer (current-buffer))))
>> +      (let ((local-prop (intern (format "org--invisible-%s-buffer-local-%S"
>
> This clearly needs a shorter name. In particular, "buffer-local" can be removed.
>
>> +					(symbol-name spec)
>> +					;; (sxhash buf) appears to be not constant over time.
>> +					;; Using buffer-name is safe, since the only place where
>> +					;; buffer-local text property actually matters is an indirect
>> +					;; buffer, where the name cannot be same anyway.
>> +					(sxhash (buffer-name buf))))))
>
>
>> +        (prog1
>> +            local-prop
>
> Please move LOCAL-PROP after the (unless return-only ...) sexp.
>
>> +          (unless return-only
>> +	    (with-current-buffer buf
>> +	      (unless (member local-prop (alist-get 'invisible char-property-alias-alist))
>> +		;; copy old property
>
> "Copy old property."
>
>> +		(dolist (old-prop (alist-get 'invisible char-property-alias-alist))
>
> We cannot use `alist-get', which was added in Emacs 25.3 only.
>
>> +		  (org-with-wide-buffer
>> +		   (let* ((pos (point-min))
>> +			  (spec (seq-find (lambda (spec)
>> +					    (string-match-p (symbol-name spec)
>> +							    (symbol-name old-prop)))
>> +					  org--invisible-spec-priority-list))
>
> Likewise, we cannot use `seq-find'.
>
>> +			  (new-prop (org--get-buffer-local-invisible-property-symbol spec nil 'return-only)))
>> +		     (while (< pos (point-max))
>> +		       (when-let (val (get-text-property pos old-prop))
>> +			 (put-text-property pos (next-single-char-property-change pos old-prop) new-prop val))
>> +		       (setq pos (next-single-char-property-change pos old-prop))))))
>> +		(setq-local char-property-alias-alist
>> +			    (cons (cons 'invisible
>> +					(mapcar (lambda (spec)
>> +						  (org--get-buffer-local-invisible-property-symbol spec nil 'return-only))
>> +						org--invisible-spec-priority-list))
>> +				  (remove (assq 'invisible char-property-alias-alist)
>> +					  char-property-alias-alist)))))))))))
>
> This begs for explainations in the docstring or as comments. In
> particular, just by reading the code, I have no clue about how this is
> going to be used, how it is going to solve issues with indirect
> buffers, with invisibility stacking, etc.
>
> I don't mind if there are more comment lines than lines of code in
> that area.
>
>> -  (remove-overlays from to 'invisible spec)
>> -  ;; Use `front-advance' since text right before to the beginning of
>> -  ;; the overlay belongs to the visible line than to the contents.
>> -  (when flag
>> -    (let ((o (make-overlay from to nil 'front-advance)))
>> -      (overlay-put o 'evaporate t)
>> -      (overlay-put o 'invisible spec)
>> -      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
>> -
>> +  (with-silent-modifications
>> +    (remove-text-properties from to (list (org--get-buffer-local-invisible-property-symbol spec) nil))
>> +    (when flag
>> +      (put-text-property from to (org--get-buffer-local-invisible-property-symbol spec) spec))))
>
> I don't think there is a need for `remove-text-properties' in every
> case. Also, (org--get-buffer-local-invisible-property-symbol spec)
> should be factored out. 
>
> I suggest:
>
>   (with-silent-modifications
>     (let ((property (org--get-buffer-local-invisible-property-symbol spec)))
>       (if flag
>           (put-text-property from to property spec)
>         (remove-text-properties from to (list property nil)))))
>
>> +(defun org-after-change-function (from to len)
>
> This is a terrible name. Org may add different functions in a-c-f,
> they cannot all be called like this. Assuming the "org-fold" prefix,
> it could be:
>
>   org-fold--fix-folded-region
>
>> +  "Process changes in folded elements.
>> +If a text was inserted into invisible region, hide the inserted text.
>> +If the beginning/end line of a folded drawer/block was changed, unfold it.
>> +If a valid end line was inserted in the middle of the folded drawer/block, unfold it."
>
> Nitpick: please do not skip lines amidst a function. Empty lines are
> used to separate functions, so this is distracting. 
>
> If a part of the function should stand out, a comment explaining what
> the part is doing is enough.
>
>> +  ;; re-hide text inserted in the middle of a folded region
>
>   Re-hide … folded region.
>
>> +  (dolist (spec org--invisible-spec-priority-list)
>> +    (when-let ((spec-to (get-text-property to (org--get-buffer-local-invisible-property-symbol spec)))
>> +	       (spec-from (get-text-property (max (point-min) (1- from)) (org--get-buffer-local-invisible-property-symbol spec))))
>> +      (when (eq spec-to spec-from)
>> +	(org-flag-region from to 't spec-to))))
>
> This part should first check if we're really after an insertion, e.g.,
> if FROM is different from TO, and exit early if that's not the case.
>
> Also, no need to quote t.
>
>> +  ;; Process all the folded text between `from' and `to'
>> +  (org-with-wide-buffer
>> +
>> +   (if (< to from)
>> +       (let ((tmp from))
>> +	 (setq from to)
>> +         (setq to tmp)))
>
> I'm surprised you need to do that. Did you encounter a case where
> a-c-f was called with boundaries in reverse order?
>
>> +   ;; Include next/previous line into the changed region.
>> +   ;; This is needed to catch edits in beginning line of a folded
>> +   ;; element.
>> +   (setq to (save-excursion (goto-char to) (forward-line) (point)))
>
> (forward-line) (point)  ---> (line-beginning-position 2)
>
>> +   (setq from (save-excursion (goto-char from) (forward-line -1) (point)))
>
> (forward-line -1) (point)  ---> (line-beginning-position 0)
>
> Anyway, I have the feeling this is not a good idea to extend it now,
> without first checking that we are in a folded drawer or block. It may
> also catch unwanted parts, e.g., a folded drawer ending on the line
> above.
>
> What about first finding the whole region with property
>
>   (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)
>
> then extending the initial part to include the drawer opening? I don't
> think we need to extend past the ending part, because drawer closing
> line is always included in the invisible part of the drawer.
>
>> +   ;; Expand the considered region to include partially present folded
>> +   ;; drawer/block.
>> +   (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))
>> +     (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))
>> +   (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-block))
>> +     (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))
>
> Please factor out (org--get-buffer-local-invisible-property-symbol
> XXX), this is difficult to read.
>
>> +   ;; check folded drawers
>
>   Check folded drawers.
>
>> +   (let ((pos from))
>> +     (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))
>> +       (setq pos (next-single-char-property-change pos
>> +						   (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))
>> +     (while (< pos to)
>> +       (when-let ((drawer-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))
>> +				     pos))
>> +		  (drawer-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))
>> +
>> +	 (let (unfold?)
>> +           ;; the line before folded text should be beginning of the drawer
>> +           (save-excursion
>> +             (goto-char drawer-begin)
>> +             (backward-char)
>
> Why `backward-char'?
>
>> +             (beginning-of-line)
>> +             (unless (looking-at-p org-drawer-regexp)
>
>    looking-at-p ---> looking-at
>
> However, you must wrap this function within `save-match-data'.
>
>> +	       (setq unfold? t)))
>> +           ;; the last line of the folded text should be :END:
>> +           (save-excursion
>> +             (goto-char drawer-end)
>> +             (beginning-of-line)
>> +             (unless (let ((case-fold-search t)) (looking-at-p org-property-end-re))
>> +	       (setq unfold? t)))
>> +           ;; there should be no :END: anywhere in the drawer body
>> +           (save-excursion
>> +             (goto-char drawer-begin)
>> +             (when (save-excursion
>> +		     (let ((case-fold-search t))
>> +		       (re-search-forward org-property-end-re
>> +					  (max (point)
>> +					       (1- (save-excursion
>> +						     (goto-char drawer-end)
>> +                                                     (line-beginning-position))))
>> +                                          't)))
>
>>  (max (point) 
>>       (save-excursion (goto-char drawer-end) (line-end-position 0))
>
>> +	       (setq unfold? t)))
>> +           ;; there should be no new entry anywhere in the drawer body
>> +           (save-excursion
>> +             (goto-char drawer-begin)
>> +             (when (save-excursion
>> +		     (let ((case-fold-search t))
>> +		       (re-search-forward org-outline-regexp-bol
>> +					  (max (point)
>> +					       (1- (save-excursion
>> +						     (goto-char drawer-end)
>> +                                                     (line-beginning-position))))
>> +                                          't)))
>> +	       (setq unfold? t)))
>
> In the phase above, you need to bail out as soon as unfold? is non-nil:
>
>  (catch :exit
>   ...
>   (throw :exit (setq unfold? t))
>   ...)
>
> Also last two checks should be lumped together, with an appropriate
> regexp.
>
> Finally, I have the feeling we're missing out some early exits when
> nothing is folded around point (e.g., most of the case).
>
>> +
>> +           (when unfold? (org-flag-region drawer-begin drawer-end nil 'org-hide-drawer))))
>> +       
>> +       (setq pos (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))))
>> +
>> +   ;; check folded blocks
>> +   (let ((pos from))
>> +     (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block))
>> +       (setq pos (next-single-char-property-change pos
>> +						   (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))
>> +     (while (< pos to)
>> +       (when-let ((block-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block))
>> +				    pos))
>> +		  (block-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))
>> +
>> +	 (let (unfold?)
>> +           ;; the line before folded text should be beginning of the block
>> +           (save-excursion
>> +             (goto-char block-begin)
>> +             (backward-char)
>> +             (beginning-of-line)
>> +             (unless (looking-at-p org-dblock-start-re)
>> +	       (setq unfold? t)))
>> +           ;; the last line of the folded text should be end of the block
>> +           (save-excursion
>> +             (goto-char block-end)
>> +             (beginning-of-line)
>> +             (unless (looking-at-p org-dblock-end-re)
>> +	       (setq unfold? t)))
>> +           ;; there should be no #+end anywhere in the block body
>> +           (save-excursion
>> +             (goto-char block-begin)
>> +             (when (save-excursion
>> +		     (re-search-forward org-dblock-end-re
>> +					(max (point)
>> +					     (1- (save-excursion
>> +						   (goto-char block-end)
>> +						   (line-beginning-position))))
>> +                                        't))
>> +	       (setq unfold? t)))
>> +           ;; there should be no new entry anywhere in the block body
>> +           (save-excursion
>> +             (goto-char block-begin)
>> +             (when (save-excursion
>> +		     (let ((case-fold-search t))
>> +		       (re-search-forward org-outline-regexp-bol
>> +					  (max (point)
>> +					       (1- (save-excursion
>> +						     (goto-char block-end)
>> +                                                     (line-beginning-position))))
>> +                                          't)))
>> +	       (setq unfold? t)))
>> +
>> +           (when unfold? (org-flag-region block-begin block-end nil 'org-hide-block))))
>> +       
>> +       (setq pos
>> +	     (next-single-char-property-change pos
>> +					       (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))))))
>
> See remarks above. The parts related to drawers and blocks are so
> similar they should be factorized out.
>
> Also `org-dblock-start-re' and `org-dblock-end-re' are not regexps we
> want here. The correct regexps would be:
>
>   (rx bol
>       (zero-or-more (any " " "\t"))
>       "#+begin"
>       (or ":" 
>           (seq "_" 
>               (group (one-or-more (not (syntax whitespace)))))))
>
> and closing line should match match-group 1 from the regexp above, e.g.:
>
>   (concat (rx bol (zero-or-more (any " " "\t")) "#+end")
>           (if block-type
>               (concat "_"
>                       (regexp-quote block-type)
>                       (rx (zero-or-more (any " " "\t")) eol))
>             (rx (opt ":") (zero-or-more (any " " "\t")) eol)))
>
> assuming `block-type' is the type of the block, or nil, i.e.,
> (match-string 1) in the previous regexp.
>
>> -	  (pcase (get-char-property-and-overlay (point) 'invisible)
>> +	  (pcase (get-char-property (point) 'invisible)
>> 	    ;; Do not fold already folded drawers.
>> -	    (`(outline . ,o) (goto-char (overlay-end o)))
>> +	    ('outline
>
> 'outline --> `outline
>  
>>      (end-of-line))
>>    (while (and (< arg 0) (re-search-backward regexp nil :move))
>>      (unless (bobp)
>> -	(while (pcase (get-char-property-and-overlay (point) 'invisible)
>> -		 (`(outline . ,o)
>> -		  (goto-char (overlay-start o))
>> -		  (re-search-backward regexp nil :move))
>> -		 (_ nil))))
>> +	(pcase (get-char-property (point) 'invisible)
>> +	  ('outline
>> +	   (goto-char (car (org--find-text-property-region (point) 'invisible)))
>> +	   (beginning-of-line))
>> +	  (_ nil)))
>
> Does this move to the beginning of the widest invisible part around
> point? If that's not the case, we need a function in "org-fold.el"
> doing just that. Or we need to nest `while' loops as it was the case
> in the code you reverted.
>
> -----
>
> Regards,
>
> -- 
> Nicolas Goaziou

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
  2020-06-21  9:52                                               ` Ihor Radchenko
@ 2020-06-21 15:01                                                 ` Nicolas Goaziou
  0 siblings, 0 replies; 55+ messages in thread
From: Nicolas Goaziou @ 2020-06-21 15:01 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> I am currently working on org-fold.el. However, I am not sure if it is
> acceptable to move some of the existing functions from org.el to
> org-fold.el.
>
> Specifically, functions from the following sections of org.el might be
> moved to org-fold.el:
>> ;;; Visibility (headlines, blocks, drawers)
>> ;;;; Reveal point location
>> ;;;; Visibility cycling
>
> Should I do it?

That makes sense, yes.

Note that you can first copy and rename most functions to make the
transition easier. As a second step, we can plug new functions into the
main system.

Regards,
-- 
Nicolas Goaziou


^ permalink raw reply	[flat|nested] 55+ messages in thread

end of thread, back to index

Thread overview: 55+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-24  6:55 [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers Ihor Radchenko
2020-04-24  8:02 ` Nicolas Goaziou
2020-04-25  0:29   ` stardiviner
2020-04-26 16:04   ` Ihor Radchenko
2020-05-04 16:56     ` Karl Voit
2020-05-07  7:18       ` Karl Voit
2020-05-09 15:43       ` Ihor Radchenko
2020-05-07 11:04     ` Christian Heinrich
2020-05-09 15:46       ` Ihor Radchenko
2020-05-08 16:38     ` Nicolas Goaziou
2020-05-09 13:58       ` Nicolas Goaziou
2020-05-09 16:22         ` Ihor Radchenko
2020-05-09 17:21           ` Nicolas Goaziou
2020-05-10  5:25             ` Ihor Radchenko
2020-05-10  9:47               ` Nicolas Goaziou
2020-05-10 13:29                 ` Ihor Radchenko
2020-05-10 14:46                   ` Nicolas Goaziou
2020-05-10 16:21                     ` Ihor Radchenko
2020-05-10 16:38                       ` Nicolas Goaziou
2020-05-10 17:08                         ` Ihor Radchenko
2020-05-10 19:38                           ` Nicolas Goaziou
2020-05-09 15:40       ` Ihor Radchenko
2020-05-09 16:30         ` Ihor Radchenko
2020-05-09 17:32           ` Nicolas Goaziou
2020-05-09 18:06             ` Ihor Radchenko
2020-05-10 14:59               ` Nicolas Goaziou
2020-05-10 15:15                 ` Kyle Meyer
2020-05-10 16:30                 ` Ihor Radchenko
2020-05-10 19:32                   ` Nicolas Goaziou
2020-05-12 10:03                     ` Nicolas Goaziou
2020-05-17 15:00                     ` Ihor Radchenko
2020-05-17 15:40                       ` Ihor Radchenko
2020-05-18 14:35                         ` Nicolas Goaziou
2020-05-18 16:52                           ` Ihor Radchenko
2020-05-19 13:07                             ` Nicolas Goaziou
2020-05-23 13:52                               ` Ihor Radchenko
2020-05-23 13:53                                 ` Ihor Radchenko
2020-05-23 15:26                                   ` Ihor Radchenko
2020-05-26  8:33                                 ` Nicolas Goaziou
2020-06-02  9:21                                   ` Ihor Radchenko
2020-06-02  9:23                                     ` Ihor Radchenko
2020-06-02 12:10                                       ` Bastien
2020-06-02 13:12                                         ` Ihor Radchenko
2020-06-02 13:23                                           ` Bastien
2020-06-02 13:30                                             ` Ihor Radchenko
2020-06-02  9:25                                     ` Ihor Radchenko
2020-06-05  7:26                                     ` Nicolas Goaziou
2020-06-05  8:18                                       ` Ihor Radchenko
2020-06-05 13:50                                         ` Nicolas Goaziou
2020-06-08  5:05                                           ` Ihor Radchenko
2020-06-08  5:06                                             ` Ihor Radchenko
2020-06-08  5:08                                             ` Ihor Radchenko
2020-06-10 17:14                                             ` Nicolas Goaziou
2020-06-21  9:52                                               ` Ihor Radchenko
2020-06-21 15:01                                                 ` Nicolas Goaziou

unofficial mirror of emacs-orgmode@gnu.org

Archives are clonable:
	git clone --mirror https://yhetil.org/orgmode/0 orgmode/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 orgmode orgmode/ https://yhetil.org/orgmode \
		emacs-orgmode@gnu.org
	public-inbox-index orgmode

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.emacs.orgmode
	nntp://news.gmane.io/gmane.emacs.orgmode


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git