Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers

[Ihor Radchenko <yantar92@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 23 bytes --]

The patch is attached


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: featuredrawertextprop-20200523.patch --]
[-- Type: text/x-diff, Size: 45706 bytes --]

diff --git a/contrib/lisp/org-notify.el b/contrib/lisp/org-notify.el
index 9f8677871..ab470ea9b 100644
--- a/contrib/lisp/org-notify.el
+++ b/contrib/lisp/org-notify.el
@@ -246,7 +246,7 @@ seconds.  The default value for SECS is 20."
           (switch-to-buffer (find-file-noselect file))
           (org-with-wide-buffer
            (goto-char begin)
-           (outline-show-entry))
+           (org-show-entry))
           (goto-char begin)
           (search-forward "DEADLINE: <")
           (search-forward ":")
diff --git a/contrib/lisp/org-velocity.el b/contrib/lisp/org-velocity.el
index bfc4d6c3e..2312b235c 100644
--- a/contrib/lisp/org-velocity.el
+++ b/contrib/lisp/org-velocity.el
@@ -325,7 +325,7 @@ use it."
   (save-excursion
     (when narrow
       (org-narrow-to-subtree))
-    (outline-show-all)))
+    (org-show-all)))
 
 (defun org-velocity-edit-entry/inline (heading)
   "Edit entry at HEADING in the original buffer."
diff --git a/doc/org-manual.org b/doc/org-manual.org
index 96b160175..2ebe94538 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -509,11 +509,11 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
   Switch back to the startup visibility of the buffer (see [[*Initial
   visibility]]).
 
-- {{{kbd(C-u C-u C-u TAB)}}} (~outline-show-all~) ::
+- {{{kbd(C-u C-u C-u TAB)}}} (~org-show-all~) ::
 
   #+cindex: show all, command
   #+kindex: C-u C-u C-u TAB
-  #+findex: outline-show-all
+  #+findex: org-show-all
   Show all, including drawers.
 
 - {{{kbd(C-c C-r)}}} (~org-reveal~) ::
@@ -529,18 +529,18 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
   headings.  With a double prefix argument, also show the entire
   subtree of the parent.
 
-- {{{kbd(C-c C-k)}}} (~outline-show-branches~) ::
+- {{{kbd(C-c C-k)}}} (~org-show-branches~) ::
 
   #+cindex: show branches, command
   #+kindex: C-c C-k
-  #+findex: outline-show-branches
+  #+findex: org-show-branches
   Expose all the headings of the subtree, but not their bodies.
 
-- {{{kbd(C-c TAB)}}} (~outline-show-children~) ::
+- {{{kbd(C-c TAB)}}} (~org-show-children~) ::
 
   #+cindex: show children, command
   #+kindex: C-c TAB
-  #+findex: outline-show-children
+  #+findex: org-show-children
   Expose all direct children of the subtree.  With a numeric prefix
   argument {{{var(N)}}}, expose all children down to level
   {{{var(N)}}}.
@@ -7294,7 +7294,7 @@ its location in the outline tree, but behaves in the following way:
   command (see [[*Visibility Cycling]]).  You can force cycling archived
   subtrees with {{{kbd(C-TAB)}}}, or by setting the option
   ~org-cycle-open-archived-trees~.  Also normal outline commands, like
-  ~outline-show-all~, open archived subtrees.
+  ~org-show-all~, open archived subtrees.
 
 -
   #+vindex: org-sparse-tree-open-archived-trees
diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index ab13f926c..ad9244940 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -6826,7 +6826,7 @@ and stored in the variable `org-prefix-format-compiled'."
 	    (t "  %-12:c%?-12t% s")))
 	(start 0)
 	varform vars var e c f opt)
-    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)"
+    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)"
 			 s start)
       (setq var (or (cdr (assoc (match-string 4 s)
 				'(("c" . category) ("t" . time) ("l" . level) ("s" . extra)
@@ -9138,20 +9138,20 @@ if it was hidden in the outline."
      ((and (called-interactively-p 'any) (= more 1))
       (message "Remote: show with default settings"))
      ((= more 2)
-      (outline-show-entry)
+      (org-show-entry)
       (org-show-children)
       (save-excursion
 	(org-back-to-heading)
 	(run-hook-with-args 'org-cycle-hook 'children))
       (message "Remote: CHILDREN"))
      ((= more 3)
-      (outline-show-subtree)
+      (org-show-subtree)
       (save-excursion
 	(org-back-to-heading)
 	(run-hook-with-args 'org-cycle-hook 'subtree))
       (message "Remote: SUBTREE"))
      ((> more 3)
-      (outline-show-subtree)
+      (org-show-subtree)
       (message "Remote: SUBTREE AND ALL DRAWERS")))
     (select-window win)))
 
diff --git a/lisp/org-archive.el b/lisp/org-archive.el
index d3e12d17b..d864dad8a 100644
--- a/lisp/org-archive.el
+++ b/lisp/org-archive.el
@@ -330,7 +330,7 @@ direct children of this heading."
 		      (insert (if datetree-date "" "\n") heading "\n")
 		      (end-of-line 0))
 		    ;; Make the subtree visible
-		    (outline-show-subtree)
+		    (org-show-subtree)
 		    (if org-archive-reversed-order
 			(progn
 			  (org-back-to-heading t)
diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index e50a4d7c8..e656df555 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -699,7 +699,7 @@ FUN is a function called with no argument."
 			  (move-beginning-of-line 2)
 			  (org-at-heading-p t)))))
     (unwind-protect (funcall fun)
-      (when hide-body (outline-hide-entry)))))
+      (when hide-body (org-hide-entry)))))
 
 (defun org-columns-previous-allowed-value ()
   "Switch to the previous allowed value for this column."
diff --git a/lisp/org-compat.el b/lisp/org-compat.el
index 635a38dcd..8fe271896 100644
--- a/lisp/org-compat.el
+++ b/lisp/org-compat.el
@@ -139,12 +139,8 @@ This is a floating point number if the size is too large for an integer."
 ;;; Emacs < 25.1 compatibility
 
 (when (< emacs-major-version 25)
-  (defalias 'outline-hide-entry 'hide-entry)
-  (defalias 'outline-hide-sublevels 'hide-sublevels)
-  (defalias 'outline-hide-subtree 'hide-subtree)
   (defalias 'outline-show-branches 'show-branches)
   (defalias 'outline-show-children 'show-children)
-  (defalias 'outline-show-entry 'show-entry)
   (defalias 'outline-show-subtree 'show-subtree)
   (defalias 'xref-find-definitions 'find-tag)
   (defalias 'format-message 'format)
diff --git a/lisp/org-element.el b/lisp/org-element.el
index ac41b7650..2d5c8d771 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -4320,7 +4320,7 @@ element or object.  Meaningful values are `first-section',
 TYPE is the type of the current element or object.
 
 If PARENT? is non-nil, assume the next element or object will be
-located inside the current one.  "
+located inside the current one."
   (if parent?
       (pcase type
 	(`headline 'section)
diff --git a/lisp/org-keys.el b/lisp/org-keys.el
index c006e9c12..deb5d7b90 100644
--- a/lisp/org-keys.el
+++ b/lisp/org-keys.el
@@ -437,7 +437,7 @@ COMMANDS is a list of alternating OLDDEF NEWDEF command names."
   #'org-next-visible-heading)
 (define-key org-mode-map [remap outline-previous-visible-heading]
   #'org-previous-visible-heading)
-(define-key org-mode-map [remap show-children] #'org-show-children)
+(define-key org-mode-map [remap outline-show-children] #'org-show-children)
 
 ;;;; Make `C-c C-x' a prefix key
 (org-defkey org-mode-map (kbd "C-c C-x") (make-sparse-keymap))
diff --git a/lisp/org-macs.el b/lisp/org-macs.el
index a02f713ca..fa0a658f0 100644
--- a/lisp/org-macs.el
+++ b/lisp/org-macs.el
@@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead."
 
 
 \f
-;;; Overlays
+;;; Overlays and text properties
 
 (defun org-overlay-display (ovl text &optional face evap)
   "Make overlay OVL display TEXT with face FACE."
@@ -705,18 +705,99 @@ If DELETE is non-nil, delete all those overlays."
 	    (delete (delete-overlay ov))
 	    (t (push ov found))))))
 
+(defun org-remove-text-properties (start end properties &optional object)
+  "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions.
+Do not remove invisible text properties specified by 'outline,
+'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this
+is needed to keep outlines, drawers, and blocks hidden unless they are
+toggled by user.
+Note: The below may be too specific and create troubles if more
+invisibility specs are added to org in future"
+  (when (plist-member properties 'invisible)
+    (let ((pos start)
+	  next spec)
+      (while (< pos end)
+	(setq next (next-single-property-change pos 'invisible nil end)
+              spec (get-text-property pos 'invisible))
+	(unless (memq spec (list 'org-hide-block
+				 'org-hide-drawer
+				 'outline))
+          (remove-text-properties pos next '(invisible nil) object))
+	(setq pos next))))
+  (when-let ((properties-stripped (org-plist-delete properties 'invisible)))
+    (remove-text-properties start end properties-stripped object)))
+
+(defun org--find-text-property-region (pos prop)
+  "Find a region containing PROP text property around point POS."
+  (let* ((beg (and (get-text-property pos prop) pos))
+	 (end beg))
+    (when beg
+      ;; when beg is the first point in the region, `previous-single-property-change'
+      ;; will return nil.
+      (setq beg (or (previous-single-property-change pos prop)
+		    beg))
+      ;; when end is the last point in the region, `next-single-property-change'
+      ;; will return nil.
+      (setq end (or (next-single-property-change pos prop)
+		    end))
+      (unless (= beg end) ; this should not happen
+        (cons beg end)))))
+
 (defun org-flag-region (from to flag spec)
   "Hide or show lines from FROM to TO, according to FLAG.
 SPEC is the invisibility spec, as a symbol."
-  (remove-overlays from to 'invisible spec)
-  ;; Use `front-advance' since text right before to the beginning of
-  ;; the overlay belongs to the visible line than to the contents.
-  (when flag
-    (let ((o (make-overlay from to nil 'front-advance)))
-      (overlay-put o 'evaporate t)
-      (overlay-put o 'invisible spec)
-      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
-
+  (pcase spec
+    ('outline
+     (remove-overlays from to 'invisible spec)
+     ;; Use `front-advance' since text right before to the beginning of
+     ;; the overlay belongs to the visible line than to the contents.
+     (when flag
+       (let ((o (make-overlay from to nil 'front-advance)))
+	 (overlay-put o 'evaporate t)
+	 (overlay-put o 'invisible spec)
+	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
+    (_
+     ;; Use text properties instead of overlays for speed.
+     ;; Overlays are too slow (Emacs Bug#35453).
+     (with-silent-modifications
+       ;; keep a backup stack of old text properties
+       (save-excursion
+	 (goto-char from)
+	 (while (< (point) to)
+	   (let ((old-spec (get-text-property (point) 'invisible))
+		 (end (next-single-property-change (point) 'invisible nil to)))
+	     (when old-spec
+	       (alter-text-property (point) end 'org-property-stack-invisible
+				    (lambda (stack)
+				      (if (or (eq old-spec (car stack))
+					      (eq spec old-spec)
+					      (eq old-spec 'outline))
+					  stack
+					(cons old-spec stack)))))
+	     (goto-char end))))
+
+       ;; cleanup everything
+       (remove-text-properties from to '(invisible nil))
+
+       ;; Recover properties from the backup stack
+       (unless flag
+	 (save-excursion
+	   (goto-char from)
+	   (while (< (point) to)
+             (let ((stack (get-text-property (point) 'org-property-stack-invisible))
+		   (end (next-single-property-change (point) 'org-property-stack-invisible nil to)))
+               (if (not stack)
+		   (remove-text-properties (point) end '(org-property-stack-invisible nil))
+		 (put-text-property (point) end 'invisible (car stack))
+		 (alter-text-property (point) end 'org-property-stack-invisible
+				      (lambda (stack)
+					(cdr stack))))
+               (goto-char end)))))
+       
+       (when flag
+	 (put-text-property from to 'rear-non-sticky nil)
+	 (put-text-property from to 'front-sticky t)
+	 (put-text-property from to 'invisible spec))))))
 
 \f
 ;;; Regexp matching
diff --git a/lisp/org-src.el b/lisp/org-src.el
index c9eef744e..e89a1c580 100644
--- a/lisp/org-src.el
+++ b/lisp/org-src.el
@@ -523,8 +523,8 @@ Leave point in edit buffer."
 	(org-src-switch-to-buffer buffer 'edit)
 	;; Insert contents.
 	(insert contents)
-	(remove-text-properties (point-min) (point-max)
-				'(display nil invisible nil intangible nil))
+	(org-remove-text-properties (point-min) (point-max)
+				    '(display nil invisible nil intangible nil))
 	(unless preserve-ind (org-do-remove-indentation))
 	(set-buffer-modified-p nil)
 	(setq buffer-file-name nil)
diff --git a/lisp/org-table.el b/lisp/org-table.el
index 6462b99c4..75801161b 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -2001,7 +2001,7 @@ toggle `org-table-follow-field-mode'."
    (arg
     (let ((b (save-excursion (skip-chars-backward "^|") (point)))
 	  (e (save-excursion (skip-chars-forward "^|\r\n") (point))))
-      (remove-text-properties b e '(invisible t intangible t))
+      (org-remove-text-properties b e '(invisible t intangible t))
       (if (and (boundp 'font-lock-mode) font-lock-mode)
 	  (font-lock-fontify-block))))
    (t
@@ -2028,7 +2028,7 @@ toggle `org-table-follow-field-mode'."
       (setq word-wrap t)
       (goto-char (setq p (point-max)))
       (insert (org-trim field))
-      (remove-text-properties p (point-max) '(invisible t intangible t))
+      (org-remove-text-properties p (point-max) '(invisible t intangible t))
       (goto-char p)
       (setq-local org-finish-function 'org-table-finish-edit-field)
       (setq-local org-window-configuration cw)
diff --git a/lisp/org.el b/lisp/org.el
index e577dc661..360974135 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.")
 (declare-function cdlatex-math-symbol "ext:cdlatex")
 (declare-function Info-goto-node "info" (nodename &optional fork strict-case))
 (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag))
+(declare-function isearch-filter-visible "isearch" (beg end))
 (declare-function org-add-archive-files "org-archive" (files))
 (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom))
 (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour))
@@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.")
 
 (defvar ffap-url-regexp)
 (defvar org-element-paragraph-separate)
+(defvar org-element-all-objects)
+(defvar org-element-all-elements)
+(defvar org-element-greater-elements)
 (defvar org-indent-indentation-per-level)
 (defvar org-radio-target-regexp)
 (defvar org-target-link-regexp)
@@ -4734,9 +4738,381 @@ This is for getting out of special buffers like capture.")
 
 ;;;; Define the Org mode
 
+;;; Handling buffer modifications
+
 (defun org-before-change-function (_beg _end)
   "Every change indicates that a table might need an update."
   (setq org-table-may-need-update t))
+
+(defvar-local org--modified-elements nil
+  "List of elements, marked as recently modified.
+There is no guarantee that the elements in this list are fully parsed.
+Only the element type, :begin and :end properties of the elements are
+guaranteed to be available. The :begin and :end element properties
+contain markers instead of positions.")
+
+(defvar org-track-element-modification-default-sensitive-commands '(self-insert-command)
+  "List of commands triggerring element modifications unconditionally.")
+
+(defvar org--element-beginning-re-alist `((center-block . "^[ \t]*#\\+begin_center[ \t]*$")
+                                       (property-drawer . ,org-property-start-re)
+				       (drawer . ,org-drawer-regexp)
+                                       (quote-block . "^[ \t]*#\\+begin_quote[ \t]*$")
+                                       (special-block . "^[ \t]*#\\+begin_\\([^ ]+\\).*$"))
+  "Alist of regexps matching beginning of elements.
+Group 1 in the regexps (if any) contains the element type.")
+
+(defvar org--element-end-re-alist `((center-block . "^[ \t]*#\\+end_center[ \t]*$")
+				 (property-drawer . ,org-property-end-re)
+				 (drawer . ,org-property-end-re)
+				 (quote-block . "^[ \t]*#\\+end_quote[ \t]*$")
+				 (special-block . "^[ \t]*#\\+end_\\([^ ]+\\).*$"))
+  "Alist of regexps matching end of elements.
+Group 1 in the regexps (if any) contains the element type or END.")
+
+(defvar org-track-element-modifications
+  `((property-drawer  . (:after-change-function
+                         org--drawer-or-block-unfold-maybe))
+    (drawer  . (:after-change-function
+                org--drawer-or-block-unfold-maybe))
+    (center-block . (:after-change-function
+                     org--drawer-or-block-unfold-maybe))
+    (quote-block  . (:after-change-function
+                     org--drawer-or-block-unfold-maybe))
+    (special-block  . (:after-change-function
+                       org--drawer-or-block-unfold-maybe)))
+  "Alist of elements to be tracked for modifications.
+The modification is only triggered according to :sensitive-re-list and
+:sensitive-command-list (see below).
+Each element of the alist is a cons of an element symbol and plist
+defining how and when the modifications are handled.
+In case of recursive elements/duplicates, the first element from the
+list is considered.
+The plist can have the following properties:
+- :element-beginning-re   :: regex matching beginning of the element
+  (default)               :: (alist-get element org--element-beginning-re-alist)
+- :element-end-re         :: regex matching end of the element
+  (default)               :: (alist-get element org--element-end-re-alist)
+- :after-change-function  :: function called after the modification
+The function must accept a single argument - element from
+`org--modified-elements'.")
+
+(defun org--get-element-region-at-point (types)
+  "Return TYPES element at point or nil.
+If TYPES is a list, return first element at point from the list.  The
+returned value is partially parsed element only containing :begin and
+:end properties.  Only elements listed in
+org--element-beginning-re-alist and org--element-end-re-alist can be
+parsed here."
+  (catch 'exit
+    (dolist (type (if (listp types) types (list types)))
+      (let ((begin-re (alist-get type org--element-beginning-re-alist))
+	    (end-re (alist-get type org--element-end-re-alist))
+            (begin-limit (save-excursion (org-with-limited-levels
+					  (org-back-to-heading-or-point-min 'invisible-ok))
+					 (point)))
+            (end-limit (or (save-excursion (outline-next-heading))
+			   (point-max)))
+            (point (point))
+	    begin end closest-begin)
+	(when (and begin-re end-re)
+	  (save-excursion
+	    (end-of-line)
+	    (when (re-search-backward begin-re begin-limit 'noerror) (setq begin (point)))
+	    (when (re-search-forward end-re end-limit 'noerror) (setq end (point)))
+            (setq closest-begin begin)
+            ;; slurp unmatched begin-re
+	    (when (and begin end)
+              (goto-char begin)
+              (while (and (re-search-backward begin-re begin-limit 'noerror)
+			  (= end (save-excursion (re-search-forward end-re end-limit 'noerror))))
+		(setq begin (point)))
+              (when (and (>= point begin) (<= point end))
+		(throw 'exit
+		       (list type
+			     (list
+			      :begin begin
+			      :end end)))))))))))
+
+(defun org--get-next-element-region-at-point (types &optional limit previous)
+  "Return TYPES element after point or nil.
+If TYPES is a list, return first element after point from the list.
+If PREVIOUS is non-nil, return first TYPES element before point.
+Limit search by LIMIT or previous/next heading.
+The returned value is partially parsed element only containing :begin
+and :end properties.  Only elements listed in
+org--element-beginning-re-alist and org--element-end-re-alist can be
+parsed here."
+  (catch 'exit
+    (dolist (type (if (listp types) types (list types)))
+      (let* ((begin-re (alist-get type org--element-beginning-re-alist))
+	     (end-re (alist-get type org--element-end-re-alist))
+             (limit (or limit (if previous
+				  (save-excursion
+				    (org-with-limited-levels
+				     (org-back-to-heading-or-point-min 'invisible-ok)
+				     (point)))
+				(or (save-excursion (outline-next-heading))
+				    (point-max)))))
+	     begin end)
+	(when (and begin-re end-re)
+	  (save-excursion
+            (if previous
+                (when (re-search-backward begin-re limit 'noerror)
+		  (when-let ((el (org--get-element-region-at-point type)))
+		    (setq begin (org-element-property :begin el))
+		    (setq end (org-element-property :end el))))
+	      (when (re-search-forward begin-re limit 'noerror)
+                (when-let ((el (org--get-element-region-at-point type)))
+		  (setq begin (org-element-property :begin el))
+		  (setq end (org-element-property :end el))))))
+	  (when (and begin end)
+            (throw 'exit
+		   (list type
+			 (list
+			  :begin begin
+			  :end end)))))))))
+
+(defun org--find-elements-in-region (beg end elements &optional include-partial include-neighbours)
+  "Find all elements from ELEMENTS in region BEG . END.
+All the listed elements must be resolvable by
+`org--get-element-region-at-point'.
+Include elements if they are partially inside region when
+INCLUDE-PARTIAL is non-nil.
+Include preceding/subsequent neighbouring elements when no partial
+element is found at the beginning/end of the region and
+INCLUDE-NEIGHBOURS is non-nil."
+  (when include-partial
+    (org-with-point-at beg
+      (let ((new-beg (org-element-property :begin (org--get-element-region-at-point elements))))
+	(if new-beg
+	    (setq beg new-beg)
+          (when (and include-neighbours
+		     (setq new-beg (org-element-property :begin
+						      (org--get-next-element-region-at-point elements
+											  (point-min)
+											  'previous))))
+            (setq beg new-beg))))
+      (when (memq 'headline elements)
+	(when-let ((new-beg (save-excursion
+			      (org-with-limited-levels (outline-previous-heading)))))
+          (setq beg new-beg))))
+    (org-with-point-at end
+      (let ((new-end (org-element-property :end (org--get-element-region-at-point elements))))
+	(if new-end
+	    (setq end new-end)
+          (when (and include-neighbours
+		     (setq new-end (org-element-property :end
+						      (org--get-next-element-region-at-point elements (point-max)))))
+            (setq end new-end))))
+      (when (memq 'headline elements)
+	(when-let ((new-end (org-with-limited-levels (outline-next-heading))))
+          (setq end (1- new-end))))))
+  (save-excursion
+    (save-restriction
+      (narrow-to-region beg end)
+      (goto-char (point-min))
+      (let (result el)
+	(while (setq el (org--get-next-element-region-at-point elements end))
+          (push el result)
+          (goto-char (org-element-property :end el)))
+        result))))
+
+(defun org--drawer-or-block-unfold-maybe (el)
+  "Update visibility of modified folded drawer/block EL.
+If text was added to hidden drawer/block, make sure that the text is
+also hidden, unless the change was done by a command listed in
+`org-track-element-modification-default-sensitive-commands'.  If the
+modification destroyed the drawer/block, reveal the hidden text in
+former drawer/block.  If the modification shrinked/expanded the
+drawer/block beyond the hidden text, reveal the affected
+drawers/blocks as well.
+Examples:
+----------------------------------------------
+----------------------------------------------
+Case #1 (the element content is hidden):
+----------------------------------------------
+:PROPERTIES:
+:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
+:END:
+----------------------------------------------
+is changed to
+----------------------------------------------
+:ROPERTIES:
+:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
+:END:
+----------------------------------------------
+Text is revealed, because we have drawer in place of property-drawer
+----------------------------------------------
+----------------------------------------------
+Case #2 (the element content is hidden):
+----------------------------------------------
+:ROPERTIES:
+:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
+:END:
+----------------------------------------------
+is changed to
+----------------------------------------------
+:OPERTIES:
+:ID:       279e797c-f4a7-47bb-80f6-e72ac6f3ec55
+:END:
+----------------------------------------------
+The text remains hidden since it is still a drawer.
+----------------------------------------------
+----------------------------------------------
+Case #3: (the element content is hidden):
+----------------------------------------------
+:FOO:
+bar
+tmp
+:END:
+----------------------------------------------
+is changed to
+----------------------------------------------
+:FOO:
+bar
+:END:
+tmp
+:END:
+----------------------------------------------
+The text is revealed because the drawer contents shrank.
+----------------------------------------------
+----------------------------------------------
+Case #4: (the element content is hidden in both the drawers):
+----------------------------------------------
+:FOO:
+bar
+tmp
+:END:
+:BAR:
+jjd
+:END:
+----------------------------------------------
+is changed to
+----------------------------------------------
+:FOO:
+bar
+tmp
+:BAR:
+jjd
+:END:
+----------------------------------------------
+The text is revealed in both the drawers because the drawers are merged
+into a new drawer.
+----------------------------------------------
+----------------------------------------------
+Case #5: (the element content is hidden)
+----------------------------------------------
+:test:
+Vivamus id enim.
+:end:
+----------------------------------------------
+is changed to
+----------------------------------------------
+:drawer:
+:test:
+Vivamus id enim.
+:end:
+----------------------------------------------
+The text is revealed in the drawer because the drawer expended.
+----------------------------------------------
+----------------------------------------------
+Case #6: (the element content is hidden):
+----------------------------------------------
+:test:
+Vivamus id enim.
+:end:
+----------------------------------------------
+is changed to
+----------------------------------------------
+:test:
+Vivamus id enim.
+:end:
+Nam a sapien.
+:end:
+----------------------------------------------
+The text remains hidden because drawer contents is always before the
+first :end:."
+  (save-match-data
+    (save-excursion
+      (save-restriction
+	(goto-char (org-element-property :begin el))
+	(let* ((newel (org--get-element-region-at-point
+		       (mapcar (lambda (el)
+				 (when (string-match-p (regexp-opt '("block" "drawer"))
+						       (symbol-name (car el)))
+                                   (car el)))
+                               org-track-element-modifications)))
+	       (spec (if (string-match-p "block" (symbol-name (org-element-type el)))
+			 'org-hide-block
+		       (if (string-match-p "drawer" (symbol-name (org-element-type el)))
+			   'org-hide-drawer
+			 t)))
+               (toggle-func (if (eq spec 'org-hide-drawer)
+				#'org-hide-drawer-toggle
+                              (if (eq spec 'org-hide-block)
+				  #'org-hide-block-toggle
+                                #'ignore)))) ; this should not happen
+	  (if (and (equal (org-element-type el) (org-element-type newel))
+		   (equal (marker-position (org-element-property :begin el))
+			  (org-element-property :begin newel))
+		   (equal (marker-position (org-element-property :end el))
+			  (org-element-property :end newel)))
+	      (when (text-property-any (marker-position (org-element-property :begin el))
+				       (marker-position (org-element-property :end el))
+				       'invisible spec)
+                (goto-char (org-element-property :begin newel))
+		(if (memq this-command org-track-element-modification-default-sensitive-commands)
+		    ;; reveal if change was made by typing
+		    (funcall toggle-func 'off)
+		  ;; re-hide the inserted text
+		  ;; FIXME: opening the drawer before hiding should not be needed here
+		  (funcall toggle-func 'off) ; this is needed to avoid showing double ellipsis
+		  (funcall toggle-func 'hide)))
+            ;; The element was destroyed. Reveal everything.
+            (org-flag-region (marker-position (org-element-property :begin el))
+			  (marker-position (org-element-property :end el))
+			  nil spec)
+            (when newel
+              (org-flag-region (org-element-property :begin newel)
+			    (org-element-property :end newel)
+			    nil spec))))))))
+
+(defun org--before-element-change-function (beg end)
+  "Register upcoming element modifications in `org--modified-elements' for all elements interesting with BEG END."
+  (save-match-data
+    (save-excursion
+      (save-restriction
+        (widen)
+	(dolist (el (org--find-elements-in-region beg
+					       end
+					       (mapcar #'car org-track-element-modifications)
+					       'include-partial
+                                               'include-neighbours))
+	  (let* ((beg-marker (copy-marker (org-element-property :begin el) 't))
+		 (end-marker (copy-marker (org-element-property :end el) 't)))
+	    (when (and (marker-position beg-marker) (marker-position end-marker))
+	      (org-element-put-property el :begin beg-marker)
+	      (org-element-put-property el :end end-marker)
+	      (add-to-list 'org--modified-elements el))))))))
+
+;; FIXME: this function may be called many times during routine modifications
+;; The normal way to avoid this is `combine-after-change-calls' - not
+;; the case in most org primitives.
+(defun org--after-element-change-function (&rest _)
+  "Handle changed elements from `org--modified-elements'."
+  (dolist (el org--modified-elements)
+    (save-match-data
+      (save-excursion
+        (save-restriction
+          (widen)
+	  (when-let* ((type (org-element-type el))
+		      (change-func (plist-get (alist-get type org-track-element-modifications)
+					      :after-change-function)))
+	    (with-demoted-errors
+		(funcall (symbol-function change-func) el)))))))
+  (setq org--modified-elements nil))
+
 (defvar org-mode-map)
 (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param.
 (defvar org-agenda-keep-modes nil)      ; Dynamically-scoped param.
@@ -4818,6 +5194,9 @@ The following commands are available:
   ;; Activate before-change-function
   (setq-local org-table-may-need-update t)
   (add-hook 'before-change-functions 'org-before-change-function nil 'local)
+  (add-hook 'before-change-functions 'org--before-element-change-function nil 'local)
+  ;; Activate after-change-function
+  (add-hook 'after-change-functions 'org--after-element-change-function nil 'local)
   ;; Check for running clock before killing a buffer
   (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local)
   ;; Initialize macros templates.
@@ -4869,6 +5248,10 @@ The following commands are available:
   (setq-local outline-isearch-open-invisible-function
 	      (lambda (&rest _) (org-show-context 'isearch)))
 
+  ;; Make isearch search in blocks hidden via text properties
+  (setq-local isearch-filter-predicate #'org--isearch-filter-predicate)
+  (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local)
+
   ;; Setup the pcomplete hooks
   (setq-local pcomplete-command-completion-function #'org-pcomplete-initial)
   (setq-local pcomplete-command-name-function #'org-command-at-point)
@@ -5050,8 +5433,8 @@ stacked delimiters is N.  Escaping delimiters is not possible."
 	      (when verbatim?
 		(org-remove-flyspell-overlays-in
 		 (match-beginning 0) (match-end 0))
-		(remove-text-properties (match-beginning 2) (match-end 2)
-					'(display t invisible t intangible t)))
+		(org-remove-text-properties (match-beginning 2) (match-end 2)
+					 '(display t invisible t intangible t)))
 	      (add-text-properties (match-beginning 2) (match-end 2)
 				   '(font-lock-multiline t org-emphasis t))
 	      (when (and org-hide-emphasis-markers
@@ -5166,7 +5549,7 @@ This includes angle, plain, and bracket links."
 	    (if (not (eq 'bracket style))
 		(add-text-properties start end properties)
 	      ;; Handle invisible parts in bracket links.
-	      (remove-text-properties start end '(invisible nil))
+	      (org-remove-text-properties start end '(invisible nil))
 	      (let ((hidden
 		     (append `(invisible
 			       ,(or (org-link-get-parameter type :display)
@@ -5186,8 +5569,8 @@ This includes angle, plain, and bracket links."
 (defun org-activate-code (limit)
   (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t)
     (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
-    (remove-text-properties (match-beginning 0) (match-end 0)
-			    '(display t invisible t intangible t))
+    (org-remove-text-properties (match-beginning 0) (match-end 0)
+			     '(display t invisible t intangible t))
     t))
 
 (defcustom org-src-fontify-natively t
@@ -5258,8 +5641,8 @@ by a #."
 	    (setq block-end (match-beginning 0)) ; includes the final newline.
 	    (when quoting
 	      (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline)
-	      (remove-text-properties beg end-of-endline
-				      '(display t invisible t intangible t)))
+	      (org-remove-text-properties beg end-of-endline
+				       '(display t invisible t intangible t)))
 	    (add-text-properties
 	     beg end-of-endline '(font-lock-fontified t font-lock-multiline t))
 	    (org-remove-flyspell-overlays-in beg bol-after-beginline)
@@ -5313,8 +5696,8 @@ by a #."
 	     '(font-lock-fontified t face org-document-info))))
 	 ((string-prefix-p "+caption" dc1)
 	  (org-remove-flyspell-overlays-in (match-end 2) (match-end 0))
-	  (remove-text-properties (match-beginning 0) (match-end 0)
-				  '(display t invisible t intangible t))
+	  (org-remove-text-properties (match-beginning 0) (match-end 0)
+				   '(display t invisible t intangible t))
 	  ;; Handle short captions.
 	  (save-excursion
 	    (beginning-of-line)
@@ -5336,8 +5719,8 @@ by a #."
 	   '(font-lock-fontified t face font-lock-comment-face)))
 	 (t ;; just any other in-buffer setting, but not indented
 	  (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
-	  (remove-text-properties (match-beginning 0) (match-end 0)
-				  '(display t invisible t intangible t))
+	  (org-remove-text-properties (match-beginning 0) (match-end 0)
+				   '(display t invisible t intangible t))
 	  (add-text-properties beg (match-end 0)
 			       '(font-lock-fontified t face org-meta-line))
 	  t))))))
@@ -5859,10 +6242,11 @@ If TAG is a number, get the corresponding match group."
 	 (inhibit-modification-hooks t)
 	 deactivate-mark buffer-file-name buffer-file-truename)
     (decompose-region beg end)
-    (remove-text-properties beg end
-			    '(mouse-face t keymap t org-linked-text t
-					 invisible t intangible t
-					 org-emphasis t))
+    (org-remove-text-properties beg end
+			     '(mouse-face t keymap t org-linked-text t
+					  invisible t
+                                          intangible t
+					  org-emphasis t))
     (org-remove-font-lock-display-properties beg end)))
 
 (defconst org-script-display  '(((raise -0.3) (height 0.7))
@@ -5970,6 +6354,29 @@ open and agenda-wise Org files."
 
 ;;;; Headlines visibility
 
+(defun org-hide-entry ()
+  "Hide the body directly following this heading."
+  (interactive)
+  (save-excursion
+    (outline-back-to-heading)
+    (outline-end-of-heading)
+    (org-flag-region (point) (progn (outline-next-preface) (point)) t 'outline)))
+
+(defun org-hide-subtree ()
+  "Hide everything after this heading at deeper levels."
+  (interactive)
+  (org-flag-subtree t))
+
+(defun org-hide-sublevels (levels)
+  "Hide everything but the top LEVELS levels of headers, in whole buffer.
+This also unhides the top heading-less body, if any.
+
+Interactively, the prefix argument supplies the value of LEVELS.
+When invoked without a prefix argument, LEVELS defaults to the level
+of the current heading, or to 1 if the current line is not a heading."
+  (cl-letf (((symbol-function 'outline-flag-region) #'org-flag-region))
+    (org-hide-sublevels levels)))
+
 (defun org-show-entry ()
   "Show the body directly following this heading.
 Show the heading too, if it is currently invisible."
@@ -5988,6 +6395,16 @@ Show the heading too, if it is currently invisible."
        'outline)
       (org-cycle-hide-property-drawers 'children))))
 
+(defun org-show-heading ()
+  "Show the current heading and move to its end."
+  (org-flag-region (- (point)
+ 		   (if (bobp) 0
+ 		     (if (and outline-blank-line
+                              (eq (char-before (1- (point))) ?\n))
+ 			 2 1)))
+		(progn (outline-end-of-heading) (point))
+		nil))
+
 (defun org-show-children (&optional level)
   "Show all direct subheadings of this heading.
 Prefix arg LEVEL is how many levels below the current level
@@ -6031,6 +6448,11 @@ heading to appear."
   (org-flag-region
    (point) (save-excursion (org-end-of-subtree t t)) nil 'outline))
 
+(defun org-show-branches ()
+  "Show all subheadings of this heading, but not their bodies."
+  (interactive)
+  (org-show-children 1000))
+
 ;;;; Blocks and drawers visibility
 
 (defun org--hide-wrapper-toggle (element category force no-error)
@@ -6064,8 +6486,8 @@ Return a non-nil value when toggling is successful."
 	(unless (let ((eol (line-end-position)))
 		  (and (> eol start) (/= eol end)))
 	  (let* ((spec (cond ((eq category 'block) 'org-hide-block)
-			     ((eq type 'property-drawer) 'outline)
-			     (t 'org-hide-drawer)))
+			     ((eq category 'drawer) 'org-hide-drawer)
+			     (t 'outline)))
 		 (flag
 		  (cond ((eq force 'off) nil)
 			(force t)
@@ -6158,10 +6580,7 @@ STATE should be one of the symbols listed in the docstring of
 	       (when (org-at-property-drawer-p)
 		 (let* ((case-fold-search t)
 			(end (re-search-forward org-property-end-re)))
-		   ;; Property drawers use `outline' invisibility spec
-		   ;; so they can be swallowed once we hide the
-		   ;; outline.
-		   (org-flag-region start end t 'outline)))))))))))
+		   (org-flag-region start end t 'org-hide-drawer)))))))))))
 
 ;;;; Visibility cycling
 
@@ -6536,7 +6955,7 @@ With a numeric prefix, show all headlines up to that level."
 		     (org-narrow-to-subtree)
 		     (org-content))))
 		((or "all" "showall")
-		 (outline-show-subtree))
+		 (org-show-subtree))
 		(_ nil)))
 	    (org-end-of-subtree)))))))
 
@@ -6609,7 +7028,7 @@ This function is the default value of the hook `org-cycle-hook'."
 	  (while (re-search-forward re nil t)
 	    (when (and (not (org-invisible-p))
 		       (org-invisible-p (line-end-position)))
-	      (outline-hide-entry))))
+	      (org-hide-entry))))
 	(org-cycle-hide-property-drawers 'all)
 	(org-cycle-show-empty-lines 'overview)))))
 
@@ -6683,8 +7102,13 @@ information."
     ;; expose it.
     (dolist (o (overlays-at (point)))
       (when (memq (overlay-get o 'invisible)
-		  '(org-hide-block org-hide-drawer outline))
+		  '(outline))
 	(delete-overlay o)))
+    (when (memq (get-text-property (point) 'invisible)
+		'(org-hide-block org-hide-drawer))
+      (let ((spec (get-text-property (point) 'invisible))
+	    (region (org--find-text-property-region (point) 'invisible)))
+	(org-flag-region (car region) (cdr region) nil spec)))
     (unless (org-before-first-heading-p)
       (org-with-limited-levels
        (cl-case detail
@@ -7661,7 +8085,7 @@ When REMOVE is non-nil, remove the subtree from the clipboard."
      (skip-chars-forward " \t\n\r")
      (setq beg (point))
      (when (and (org-invisible-p) visp)
-       (save-excursion (outline-show-heading)))
+       (save-excursion (org-show-heading)))
      ;; Shift if necessary.
      (unless (= shift 0)
        (save-restriction
@@ -8103,7 +8527,7 @@ function is being called interactively."
 		       (point))
 	    what "children")
       (goto-char start)
-      (outline-show-subtree)
+      (org-show-subtree)
       (outline-next-heading))
      (t
       ;; we will sort the top-level entries in this file
@@ -13150,7 +13574,7 @@ drawer is immediately hidden."
 	   (inhibit-read-only t))
        (unless (bobp) (insert "\n"))
        (insert ":PROPERTIES:\n:END:")
-       (org-flag-region (line-end-position 0) (point) t 'outline)
+       (org-flag-region (line-end-position 0) (point) t 'org-hide-drawer)
        (when (or (eobp) (= begin (point-min))) (insert "\n"))
        (org-indent-region begin (point))))))
 
@@ -17612,11 +18036,11 @@ Move point to the beginning of first heading or end of buffer."
 (defun org-show-branches-buffer ()
   "Show all branches in the buffer."
   (org-flag-above-first-heading)
-  (outline-hide-sublevels 1)
+  (org-hide-sublevels 1)
   (unless (eobp)
-    (outline-show-branches)
+    (org-show-branches)
     (while (outline-get-next-sibling)
-      (outline-show-branches)))
+      (org-show-branches)))
   (goto-char (point-min)))
 
 (defun org-kill-note-or-show-branches ()
@@ -17630,8 +18054,8 @@ Move point to the beginning of first heading or end of buffer."
 	(t
 	 (let ((beg (progn (org-back-to-heading) (point)))
 	       (end (save-excursion (org-end-of-subtree t t) (point))))
-	   (outline-hide-subtree)
-	   (outline-show-branches)
+	   (org-hide-subtree)
+	   (org-show-branches)
 	   (org-hide-archived-subtrees beg end)))))
 
 (defun org-delete-indentation (&optional arg)
@@ -17787,9 +18211,9 @@ Otherwise, call `org-show-children'.  ARG is the level to hide."
     (if (org-before-first-heading-p)
         (progn
           (org-flag-above-first-heading)
-          (outline-hide-sublevels (or arg 1))
+          (org-hide-sublevels (or arg 1))
           (goto-char (point-min)))
-      (outline-hide-subtree)
+      (org-hide-subtree)
       (org-show-children arg))))
 
 (defun org-ctrl-c-star ()
@@ -20933,6 +21357,80 @@ Started from `gnus-info-find-node'."
           (t default-org-info-node))))))
 
 \f
+
+;;; Make isearch search in some text hidden via text propertoes
+
+(defvar org--isearch-overlays nil
+  "List of overlays temporarily created during isearch.
+This is used to allow searching in regions hidden via text properties.
+As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays.
+Any text hidden via text properties is not revealed even if `search-invisible'
+is set to 't.")
+
+;; Not sure if it needs to be a user option
+;; One might want to reveal hidden text in, for example, hidden parts of the links.
+;; Currently, hidden text in links is never revealed by isearch.
+(defvar org-isearch-specs '(org-hide-block
+			 org-hide-drawer)
+  "List of text invisibility specs to be searched by isearch.
+By default ([2020-05-09 Sat]), isearch does not search in hidden text,
+which was made invisible using text properties. Isearch will be forced
+to search in hidden text with any of the listed 'invisible property value.")
+
+(defun org--create-isearch-overlays (beg end)
+  "Replace text property invisibility spec by overlays between BEG and END.
+All the regions with invisibility text property spec from
+`org-isearch-specs' will be changed to use overlays instead
+of text properties. The created overlays will be stored in
+`org--isearch-overlays'."
+  (let ((pos beg))
+    (while (< pos end)
+      (when-let* ((spec (get-text-property pos 'invisible))
+		  (spec (memq spec org-isearch-specs))
+		  (region (org--find-text-property-region pos 'invisible)))
+        (setq spec (get-text-property pos 'invisible))
+        ;; Changing text properties is considered buffer modification.
+        ;; We do not want it here.
+	(with-silent-modifications
+          ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat]
+          ;; overlay for 'outline blocks.
+          (let ((o (make-overlay (car region) (cdr region) nil 'front-advance)))
+	    (overlay-put o 'evaporate t)
+	    (overlay-put o 'invisible spec)
+            ;; `delete-overlay' here means that spec information will be lost
+            ;; for the region. The region will remain visible.
+	    (overlay-put o 'isearch-open-invisible #'delete-overlay)
+            (push o org--isearch-overlays))
+	  (org-flag-region (car region) (cdr region) nil spec)))
+      (setq pos (next-single-property-change pos 'invisible nil end)))))
+
+(defun org--isearch-filter-predicate (beg end)
+  "Return non-nil if text between BEG and END is deemed visible by Isearch.
+This function is intended to be used as `isearch-filter-predicate'.
+Unlike `isearch-filter-visible', make text with 'invisible text property
+value listed in `org-isearch-specs' visible to Isearch."
+  (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text
+  (isearch-filter-visible beg end))
+
+(defun org--clear-isearch-overlay (ov)
+  "Convert OV region back into using text properties."
+  (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays
+    ;; Changing text properties is considered buffer modification.
+    ;; We do not want it here.
+    (with-silent-modifications
+      (org-flag-region (overlay-start ov) (overlay-end ov) t spec)))
+  (when (member ov isearch-opened-overlays)
+    (setq isearch-opened-overlays (delete ov isearch-opened-overlays)))
+  (delete-overlay ov))
+
+(defun org--clear-isearch-overlays ()
+  "Convert overlays from `org--isearch-overlays' back into using text properties."
+  (when org--isearch-overlays
+    (mapc #'org--clear-isearch-overlay org--isearch-overlays)
+    (setq org--isearch-overlays nil)))
+
+\f
+
 ;;; Finish up
 
 (add-hook 'org-mode-hook     ;remove overlays when changing major mode

[-- Attachment #3: Type: text/plain, Size: 17187 bytes --]



Ihor Radchenko <yantar92@gmail.com> writes:

> Hello,
>
> [The patch itself will be provided in the following email]
>
> I have five updates from the previous version of the patch:
>
> 1. I implemented a simplified version of element parsing to detect
> changes in folded drawers or blocks. No computationally expensive calls
> of org-element-at-point or org-element-parse-buffer are needed now.
>
> 2. The patch is now compatible with master (commit 2e96dc639). I
> reverted the earlier change in folding drawers and blocks. Now, they are
> back to using 'org-hide-block and 'org-hide-drawer. Using 'outline would
> achieve nothing when we use text properties.
>
> 3. 'invisible text property can now be nested. This is important, for
> example, when text inside drawers contains fontified links (which also
> use 'invisible text property to hide parts of the link). Now, the old
> 'invisible spec is recovered after unfolding.
>
> 4. Some outline-* function calls in org referred to outline-flag-region
> implementation, which is not in sync with org-flag-region in this patch.
> I have implemented their org-* versions and replaced the calls
> throughout .el files. Actually, some org-* versions were already
> implemented in org, but not used for some reason (or not mentioned in
> the manual). I have updated the relevant sections of manual. These
> changes might be relevant to org independently of this feature branch.
>
> 5. I have managed to get a working version of outline folding via text
> properties. However, that approach has a big downside - folding state
> cannot be different in indirect buffer when we use text properties. I
> have seen packages relying on this feature of org and I do not see any
> obvious way to achieve different folding state in indirect buffer while
> using text properties for outline folding.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the new implementation for tracking changes:
>
>> Of course we can. It is only necessary to focus on changes that would
>> break the structure of the element. This does not entail a full parsing.
>
> I have limited parsing to matching beginning and end of a drawer/block.
> The basic functions are org--get-element-region-at-point,
> org--get-next-element-region-at-point, and org--find-elements-in-region.
> They are simplified versions of org-element-* parsers and do not require
> parsing everything from the beginning of the section.
>
> For now, I keep everything in org.el, but those simplified parsers
> probably belong to org-element.el.
>
>> If we can stick with `after-change-functions' (or local equivalent),
>> that's better. It is more predictable than `before-change-functions' and
>> alike.
>
> For now, I still used before/after-change-functions combination.
> I see the following problems with using only after-change-functions: 
>
> 1. They are not guaranteed to be called after every single change:
>
> From (elisp) Change Hooks:
> "... some complex primitives call ‘before-change-functions’ once before
> making changes, and then call ‘after-change-functions’ zero or more
> times"
>
> The consequence of it is a possibility that region passed to the
> after-change-functions is quite big (including all the singular changes,
> even if they are distant). This region may contain changed drawers as
> well and unchanged drawers and needs to be parsed to determine which
> drawers need to be re-folded.
>
>> And, more importantly, they are not meant to be used together, i.e., you
>> cannot assume that a single call to `before-change-functions' always
>> happens before calling `after-change-functions'. This can be tricky if
>> you want to use the former to pass information to the latter.
>
> The fact that before-change-functions can be called multiple times
> before after-change-functions, is trivially solved by using buffer-local
> changes register (see org--modified-elements). The register is populated
> by before-change-functions and cleared by after-change-functions.
>
>> Well, `before-change-fuctions' and `after-change-functions' are not
>> clean at all: you modify an unrelated part of the buffer, but still call
>> those to check if a drawer needs to be unfolded somewhere.
>
> 2. As you pointed, instead of global before-change-functions, we can use
> modification-hooks text property on sensitive parts of the
> drawers/blocks. This would work, but I am concerned about one annoying
> special case:
>
> -------------------------------------------------------------------------
> :BLAH: <inserted outside any of the existing drawers>
>
> <some text>
>
> :DRAWER: <folded>
> Donec at pede.
> :END:
> -------------------------------------------------------------------------
> In this example, the user would not be able to unfold the folder DRAWER
> because it will technically become a part of a new giant BLAH drawer.
> This may be especially annoying if <some text> is more than one screen
> long and there is no easy way to identify why unfolding does not work
> (with point at :DRAWER:).
>
> Because of this scenario, limiting before-change-functions to folded
> drawers is not sufficient. Any change in text may need to trigger
> unfolding.
>
> In the patch, I always register possible modifications in the
> blocks/drawers intersecting with the modified region + a drawer/block
> right next to the region.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the nested 'invisible text property implementation.
>
> The idea is to keep 'invisible property stack push and popping from it
> as we add/remove 'invisible text property. All the work is done in
> org-flag-region.
>
> This was originally intended for folding outlines via text properties.
> Since using text properties for folding outlines is not a good idea,
> nested text properties have much less use. As I mentioned, they do
> preserve link fontification, but I am not sure if it worth it for the
> overhead to org-flag-region. Keeping this here mostly in the case if
> someone has any ideas how it can be useful.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on replaced outline-* -> org-* function calls.
>
> I have implemented org-* versions of the following functions:
>
> - outline-hide-entry
> - outline-hide-subtree
> - outline-hide-sublevels
> - outline-show-heading
> - outline-show-branches
>
> The org-* versions trivially use org-flag-region instead of
> outline-flag-region.
>
> Replaced outline-* calls where org- versions were already available:
>
> - outline-show-entry
> - outline-show-all
> - outline-show-subtree
>
> I reflected the new (including already available) functions in the
> manual and removed some defalias from org-compat.el where they are not
> needed. 
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> Further work:
>
> 1. after-change-functions use org-hide-drawer/block-toggle to
> fold/unfold after modification. However, I just found that they call
> org-element-at-point, which slows down modifications in folded
> drawers/blocks. For example, realigning a long table inside folded
> drawer takes >1sec, while it is instant in the unfolded drawer.
>
> 2. org-toggle-custom-properties is terribly slow on large org documents,
> similarly to folded drawers on master. It should be relatively easy to
> use text properties there instead of overlays.
>
> 3. Multiple calls to before/after-change-functions is still a problem. I
> am looking into following ways to reduce this number:
>  - reduce the number of elements registered as potentially modified
>    + do not add duplicates to org--modified-elements
>    + do not add unfolded elements to org--modified-elements
>    + register after-change-function as post-command hook and remove it
>      from global after-change-functions. This way, it will be called
>      twice per command only.
>  - determine common region containing org--modified-elements. if change
>    is happening within that region, there is no need to parse
>    drawers/blocks there again.
>
> P.S.
>
>>> It was mostly an annoyance, because they returned different results on
>>> the same element. Specifically, they returned different :post-blank and
>>> :end properties, which does not sound right.
>>
>> OK. If you have a reproducible recipe, I can look into it and see what
>> can be done.
>
> Recipe to have different (org-element-at-point) and
> (org-element-parse-buffer 'element)
> -------------------------------------------------------------------------
> <point-min>
> :PROPERTIES:
> :CREATED:  [2020-05-23 Sat 02:32]
> :END:
>
>
> <point-max>
> -------------------------------------------------------------------------
>
>
> Best,
> Ihor
>
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Hello,
>>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>>> As you noticed, using Org Element is a no-go, unfortunately. Parsing an
>>>> element is a O(N) operation by the number of elements before it in
>>>> a section. In particular, it is not bounded, and not mitigated by
>>>> a cache. For large documents, it is going to be unbearably slow, too.
>>>
>>> Ouch. I thought it is faster.
>>> What do you mean by "not mitigated by a cache"?
>>
>> Parsing starts from the closest headline, every time. So, if Org parses
>> the Nth element in the entry two times, it really parses 2N elements.
>>
>> With a cache, assuming the buffer wasn't modified, Org would parse
>> N elements only. With a smarter cache, with fine grained cache
>> invalidation, it could also reduce the number of subsequent parsed
>> elements.
>>
>>> The reason I would like to utilise org-element parser to make tracking
>>> modifications more robust. Using details of the syntax would make the
>>> code fragile if any modifications are made to syntax in future.
>>
>> I don't think the code would be more fragile. Also, the syntax we're
>> talking about is not going to be modified anytime soon. Moreover, if
>> folding breaks, it is usually visible, so the bug will not be unnoticed.
>>
>> This code is going to be as low-level as it can be.
>>
>>> Debugging bugs in modification functions is not easy, according to my
>>> experience.
>>
>> No, it's not. 
>>
>> But this is not really related to whether you use Element or not.
>>
>>> One possible way to avoid performance issues during modification is
>>> running parser in advance. For example, folding an element may
>>> as well add information about the element to its text properties.
>>> This will not degrade performance of folding since we are already
>>> parsing the element during folding (at least, in
>>> org-hide-drawer-toggle).
>>
>> We can use this information stored at fold time. But I'm not even sure
>> we need it.
>>
>>> The problem with parsing an element during folding is that we cannot
>>> easily detect changes like below without re-parsing.
>>
>> Of course we can. It is only necessary to focus on changes that would
>> break the structure of the element. This does not entail a full parsing.
>>
>>> :PROPERTIES: <folded>
>>> :CREATED: [2020-05-18 Mon]
>>> :END: <- added line
>>> :ID: test
>>> :END:
>>>
>>> or even
>>>
>>> :PROPERTIES:
>>> :CREATED: [2020-05-18 Mon]
>>> :ID: test
>>> :END: <- delete this line
>>>
>>> :DRAWER: <folded, cannot be unfolded if we don't re-parse after deletion>
>>> test
>>> :END:
>>
>> Please have a look at the "sensitive parts" I wrote about. This takes
>> care of this kind of breakage.
>>
>>> The re-parsing can be done via regexp, as you suggested, but I don't
>>> like this idea, because it will end up re-implementing
>>> org-element-*-parser.
>>
>> You may have misunderstood my suggestion. See below.
>>
>>> Would it be acceptable to run org-element-*-parser
>>> in after-change-functions?
>>
>> I'd rather not do that. This is unnecessary consing, and matching, etc.
>>
>>> If I understand correctly, it is not as easy.
>>> Consider the following example:
>>>
>>> :PROPERTIES:
>>> :CREATED: [2020-05-18 Mon]
>>> <region-beginning>
>>> :ID: example
>>> :END:
>>>
>>> <... a lot of text, maybe containing other drawers ...>
>>>
>>> Nullam rutrum.
>>> Pellentesque dapibus suscipit ligula.
>>> <region-end>
>>> Proin quam nisl, tincidunt et, mattis eget, convallis nec, purus.
>>>
>>> If the region gets deleted, the modification hooks from chars inside
>>> drawer will be called as (hook-function <region-beginning>
>>> <region-end>). So, there is still a need to find the drawer somehow to
>>> mark it as about to be modified (modification hooks are ran before
>>> actual modification).
>>
>> If we can stick with `after-change-functions' (or local equivalent),
>> that's better. It is more predictable than `before-change-functions' and
>> alike.
>>
>> If it is a deletion, here is the kind of checks we could do, depending
>> on when they are performed.
>>
>> Before actual changes :
>>
>>   1. The deletion is happening within a folded drawer (unnecessary step
>>      in local functions).
>>   2. The change deleted the sensitive line ":END:".
>>   3. Conclusion : unfold.
>>
>> Or, after actual changes :
>>
>>   1. The deletion involves a drawer.
>>   2. Text properties indicate that the beginning of the propertized part
>>      of the buffer start with org-drawer-regexp, but doesn't end with
>>      `org-property-end-re'. A "sensitive part" disappeared!
>>   3. Conclusion : unfold
>>
>> This is far away from parsing. IMO, a few checks cover all cases. Let me
>> know if you have questions about it.
>>
>> Also, note that the kind of change you describe will happen perhaps
>> 0.01% of the time. Most change are about one character, or a single
>> line, long.
>>
>>> The only difference between using modification hooks and
>>> before-change-functions is that modification hooks will trigger less
>>> frequently. 
>>
>> Exactly. Much less frequently. But extra care is required, as you noted
>> already.
>>
>>> Considering the performance of org-element-at-point, it is
>>> probably worth doing. Initially, I wanted to avoid it because setting a
>>> single before-change-functions hook sounded cleaner than setting
>>> modification-hooks, insert-behind-hooks, and insert-in-front-hooks.
>>
>> Well, `before-change-fuctions' and `after-change-functions' are not
>> clean at all: you modify an unrelated part of the buffer, but still call
>> those to check if a drawer needs to be unfolded somewhere.
>>
>> And, more importantly, they are not meant to be used together, i.e., you
>> cannot assume that a single call to `before-change-functions' always
>> happens before calling `after-change-functions'. This can be tricky if
>> you want to use the former to pass information to the latter.
>>
>> But I understand that they are easier to use than their local
>> counterparts. If you stick with (before|after)-change-functions, the
>> function being called needs to drop the ball very quickly if the
>> modification is not about folding changes. Also, I very much suggest to
>> stick to only `after-change-functions', if feasible (I think it is), per
>> above.
>>
>>> Moreover, these text properties would be copied by default if one uses 
>>> buffer-substring. Then, the hooks will also trigger later in the yanked
>>> text, which may cause all kinds of bugs.
>>
>> Indeed, that would be something to handle specifically. I.e.,
>> destructive modifications (i.e., those that unfold) could clear such
>> properties.
>>
>> It is more work. I don't know if it is worth the trouble if we can get
>> out quickly of `after-change-functions' for unrelated changes.
>>
>>> It was mostly an annoyance, because they returned different results on
>>> the same element. Specifically, they returned different :post-blank and
>>> :end properties, which does not sound right.
>>
>> OK. If you have a reproducible recipe, I can look into it and see what
>> can be done.
>>
>> Regards,
>>
>> -- 
>> Nicolas Goaziou
>
> -- 
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg