Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers

[Ihor Radchenko <yantar92@gmail.com>]

> You cannot. You may however mimic it with `cursor-sensor-functions' text
> property. These assume Cursor Sensor minor mode is active, tho.
> I haven't tested it, but I assume it would slow down text properties
> a bit, too, but hopefully not as much as overlays.

Unfortunately, isearch sets inhibit-point-motion-hooks to non-nil
internally. Anyway, I came up with some workaround, which seems to work
(see below). Though it would be better if isearch supported hidden text
in addition to overlays.

> Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth
> experimenting with `cursor-sensor-functions'.

So far, I came up with the following partial solution searching and
showing hidden text.

;; Unfortunately isearch, sets inhibit-point-motion-hooks and we
;; cannot even use cursor-sensor-functions as a workaround
;; I used a less ideas approach with advice to isearch-search-string as
;; a workaround 

(defun org-find-text-property-region (pos prop)
  "Find a region containing PROP text property around point POS."
  (require 'org-macs) ;; org-with-point-at
  (org-with-point-at pos
    (let* ((beg (and (get-text-property pos prop) pos))
	   (end beg))
      (when beg
	(setq beg (or (previous-single-property-change pos prop)
		      beg))
	(setq end (or (next-single-property-change pos prop)
		      end))
        (unless (equal beg end)
          (cons beg end))))))

;; :FIXME: re-hide properties when point moves away
(define-advice isearch-search-string (:after (&rest _) put-overlay)
  "Reveal hidden text at point."
  (when-let ((region (org-find-text-property-region (point) 'invisible)))
    (with-silent-modifications
      (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point) 'invisible)))
      (remove-text-properties (car region) (cdr region) '(invisible nil))))

;; this seems to be unstable, but I cannot figure out why
(defun org-restore-invisibility-specs (&rest _)
  ""
   (let ((pos (point-min)))
     (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-max))
       (when-let ((region (org-find-text-property-region pos 'org-invisible)))
	   (with-silent-modifications
	     (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-invisible))
	     (remove-text-properties (car region) (cdr region) '(org-invisible nil)))))))

(add-hook 'post-command-hook #'org-restore-invisibility-specs)

(defun org-flag-region (from to flag spec)
  "Hide or show lines from FROM to TO, according to FLAG.
SPEC is the invisibility spec, as a symbol."
  (pcase spec
    ('outline
     (remove-overlays from to 'invisible spec)
     ;; Use `front-advance' since text right before to the beginning of
     ;; the overlay belongs to the visible line than to the contents.
     (when flag
       (let ((o (make-overlay from to nil 'front-advance)))
	 (overlay-put o 'evaporate t)
	 (overlay-put o 'invisible spec)
	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
    (_
     (with-silent-modifications
       (remove-text-properties from to '(invisible nil))
       (when flag
	 (put-text-property from to 'invisible spec)
	 )))))

;; This normally deletes invisible text property. We do not want this now.
(defun org-unfontify-region (beg end &optional _maybe_loudly)
  "Remove fontification and activation overlays from links."
  (font-lock-default-unfontify-region beg end)
  (let* ((buffer-undo-list t)
	 (inhibit-read-only t) (inhibit-point-motion-hooks t)
	 (inhibit-modification-hooks t)
	 deactivate-mark buffer-file-name buffer-file-truename)
    (decompose-region beg end)
    (remove-text-properties beg end
			    '(mouse-face t keymap t org-linked-text t
					 ;; Do not remove invisible during fontification					 
					 ;; invisible t
                                         intangible t
					 org-emphasis t))
    (org-remove-font-lock-display-properties beg end)))

> Anyway, the real fix should come from Emacs itself. There are ways to
> make overlays faster. These ways have already been discussed on the
> Emacs devel mailing list, but no one implemented them. It is a bit sad
> that we have to find workarounds for that.

I guess that it is a very old story starting from the times when XEmacs
was a thing [1]. I recently heard about binary tree implementation of
overlays (there should be a branch in emacs git repo) [2], but there was
no update on that branch for a while. So, I do not have much hope on
Emacs implementing efficient overlay access in the near future. (And I
have problems with huge org files already).

[1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/
[2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html


Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> To my surprise, the patch did not break org to unusable state and
>> the performance on the sample org file [3] improved drastically. You can
>> try by yourself!
>
> It is not a surprise, really. Text properties are much faster than
> overlays, and very close to them features-wise. They are a bit more
> complex to handle, however.
>
>> However, this did introduce some visual glitches with drawer display.
>> Though drawers can still be folded/unfolded with <tab>, they are not
>> folded on org-mode startup for some reason (can be fixed by running
>> (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers)
>> are unfolded for no apparent reason sometimes. A blind guess is that it
>> is something to do with lack of 'isearch-open-invisible, which I am not
>> sure how to set via text properties.
>
> You cannot. You may however mimic it with `cursor-sensor-functions' text
> property. These assume Cursor Sensor minor mode is active, tho.
> I haven't tested it, but I assume it would slow down text properties
> a bit, too, but hopefully not as much as overlays.
>
> Note there are clear advantages using text properties. For example, when
> you move contents around, text properties are preserved. So there's no
> more need for the `org-cycle-hide-drawer' dance, i.e., it is not
> necessary anymore to re-hide drawers.
>
>> Any thoughts about the use of text properties or about the patch
>> suggestion are welcome.  
>
> Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth
> experimenting with `cursor-sensor-functions'.
>
> We could also use text properties for property drawers, and overlays for
> regular ones. This might give us a reasonable speed-up with an
> acceptable feature trade-off.
>
> Anyway, the real fix should come from Emacs itself. There are ways to
> make overlays faster. These ways have already been discussed on the
> Emacs devel mailing list, but no one implemented them. It is a bit sad
> that we have to find workarounds for that.
>
> Regards,
>
> -- 
> Nicolas Goaziou

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg