Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers

[Ihor Radchenko <yantar92@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 44 bytes --]

The patch (against 1aa095ccf) is attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: featuredrawertextprop-20200608.patch --]
[-- Type: text/x-diff, Size: 50930 bytes --]

diff --git a/contrib/lisp/org-notify.el b/contrib/lisp/org-notify.el
index 9f8677871..ab470ea9b 100644
--- a/contrib/lisp/org-notify.el
+++ b/contrib/lisp/org-notify.el
@@ -246,7 +246,7 @@ seconds.  The default value for SECS is 20."
           (switch-to-buffer (find-file-noselect file))
           (org-with-wide-buffer
            (goto-char begin)
-           (outline-show-entry))
+           (org-show-entry))
           (goto-char begin)
           (search-forward "DEADLINE: <")
           (search-forward ":")
diff --git a/contrib/lisp/org-velocity.el b/contrib/lisp/org-velocity.el
index bfc4d6c3e..2312b235c 100644
--- a/contrib/lisp/org-velocity.el
+++ b/contrib/lisp/org-velocity.el
@@ -325,7 +325,7 @@ use it."
   (save-excursion
     (when narrow
       (org-narrow-to-subtree))
-    (outline-show-all)))
+    (org-show-all)))
 
 (defun org-velocity-edit-entry/inline (heading)
   "Edit entry at HEADING in the original buffer."
diff --git a/doc/org-manual.org b/doc/org-manual.org
index efad195e1..c6f167eac 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -509,11 +509,11 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
   Switch back to the startup visibility of the buffer (see [[*Initial
   visibility]]).
 
-- {{{kbd(C-u C-u C-u TAB)}}} (~outline-show-all~) ::
+- {{{kbd(C-u C-u C-u TAB)}}} (~org-show-all~) ::
 
   #+cindex: show all, command
   #+kindex: C-u C-u C-u TAB
-  #+findex: outline-show-all
+  #+findex: org-show-all
   Show all, including drawers.
 
 - {{{kbd(C-c C-r)}}} (~org-reveal~) ::
@@ -529,18 +529,18 @@ Org uses just two commands, bound to {{{kbd(TAB)}}} and
   headings.  With a double prefix argument, also show the entire
   subtree of the parent.
 
-- {{{kbd(C-c C-k)}}} (~outline-show-branches~) ::
+- {{{kbd(C-c C-k)}}} (~org-show-branches~) ::
 
   #+cindex: show branches, command
   #+kindex: C-c C-k
-  #+findex: outline-show-branches
+  #+findex: org-show-branches
   Expose all the headings of the subtree, but not their bodies.
 
-- {{{kbd(C-c TAB)}}} (~outline-show-children~) ::
+- {{{kbd(C-c TAB)}}} (~org-show-children~) ::
 
   #+cindex: show children, command
   #+kindex: C-c TAB
-  #+findex: outline-show-children
+  #+findex: org-show-children
   Expose all direct children of the subtree.  With a numeric prefix
   argument {{{var(N)}}}, expose all children down to level
   {{{var(N)}}}.
@@ -7294,7 +7294,7 @@ its location in the outline tree, but behaves in the following way:
   command (see [[*Visibility Cycling]]).  You can force cycling archived
   subtrees with {{{kbd(C-TAB)}}}, or by setting the option
   ~org-cycle-open-archived-trees~.  Also normal outline commands, like
-  ~outline-show-all~, open archived subtrees.
+  ~org-show-all~, open archived subtrees.
 
 -
   #+vindex: org-sparse-tree-open-archived-trees
diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index 9fbeb2a1e..2f121f743 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -6824,7 +6824,7 @@ and stored in the variable `org-prefix-format-compiled'."
 	    (t "  %-12:c%?-12t% s")))
 	(start 0)
 	varform vars var e c f opt)
-    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)"
+    (while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)"
 			 s start)
       (setq var (or (cdr (assoc (match-string 4 s)
 				'(("c" . category) ("t" . time) ("l" . level) ("s" . extra)
@@ -9136,20 +9136,20 @@ if it was hidden in the outline."
      ((and (called-interactively-p 'any) (= more 1))
       (message "Remote: show with default settings"))
      ((= more 2)
-      (outline-show-entry)
+      (org-show-entry)
       (org-show-children)
       (save-excursion
 	(org-back-to-heading)
 	(run-hook-with-args 'org-cycle-hook 'children))
       (message "Remote: CHILDREN"))
      ((= more 3)
-      (outline-show-subtree)
+      (org-show-subtree)
       (save-excursion
 	(org-back-to-heading)
 	(run-hook-with-args 'org-cycle-hook 'subtree))
       (message "Remote: SUBTREE"))
      ((> more 3)
-      (outline-show-subtree)
+      (org-show-subtree)
       (message "Remote: SUBTREE AND ALL DRAWERS")))
     (select-window win)))
 
diff --git a/lisp/org-archive.el b/lisp/org-archive.el
index d3e12d17b..d864dad8a 100644
--- a/lisp/org-archive.el
+++ b/lisp/org-archive.el
@@ -330,7 +330,7 @@ direct children of this heading."
 		      (insert (if datetree-date "" "\n") heading "\n")
 		      (end-of-line 0))
 		    ;; Make the subtree visible
-		    (outline-show-subtree)
+		    (org-show-subtree)
 		    (if org-archive-reversed-order
 			(progn
 			  (org-back-to-heading t)
diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index e50a4d7c8..e656df555 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -699,7 +699,7 @@ FUN is a function called with no argument."
 			  (move-beginning-of-line 2)
 			  (org-at-heading-p t)))))
     (unwind-protect (funcall fun)
-      (when hide-body (outline-hide-entry)))))
+      (when hide-body (org-hide-entry)))))
 
 (defun org-columns-previous-allowed-value ()
   "Switch to the previous allowed value for this column."
diff --git a/lisp/org-compat.el b/lisp/org-compat.el
index 5953f89d2..09a09472a 100644
--- a/lisp/org-compat.el
+++ b/lisp/org-compat.el
@@ -138,12 +138,8 @@ This is a floating point number if the size is too large for an integer."
 ;;; Emacs < 25.1 compatibility
 
 (when (< emacs-major-version 25)
-  (defalias 'outline-hide-entry 'hide-entry)
-  (defalias 'outline-hide-sublevels 'hide-sublevels)
-  (defalias 'outline-hide-subtree 'hide-subtree)
   (defalias 'outline-show-branches 'show-branches)
   (defalias 'outline-show-children 'show-children)
-  (defalias 'outline-show-entry 'show-entry)
   (defalias 'outline-show-subtree 'show-subtree)
   (defalias 'xref-find-definitions 'find-tag)
   (defalias 'format-message 'format)
@@ -644,7 +640,7 @@ When optional argument ELEMENT is a parsed drawer, as returned by
 When buffer positions BEG and END are provided, hide or show that
 region as a drawer without further ado."
   (declare (obsolete "use `org-hide-drawer-toggle' instead." "Org 9.4"))
-  (if (and beg end) (org-flag-region beg end flag 'outline)
+  (if (and beg end) (org-flag-region beg end flag 'org-hide-drawer)
     (let ((drawer
 	   (or element
 	       (and (save-excursion
@@ -658,7 +654,7 @@ region as a drawer without further ado."
 	   (save-excursion (goto-char (org-element-property :end drawer))
 			   (skip-chars-backward " \t\n")
 			   (line-end-position))
-	   flag 'outline)
+	   flag 'org-hide-drawer)
 	  ;; When the drawer is hidden away, make sure point lies in
 	  ;; a visible part of the buffer.
 	  (when (invisible-p (max (1- (point)) (point-min)))
diff --git a/lisp/org-element.el b/lisp/org-element.el
index ac41b7650..2d5c8d771 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -4320,7 +4320,7 @@ element or object.  Meaningful values are `first-section',
 TYPE is the type of the current element or object.
 
 If PARENT? is non-nil, assume the next element or object will be
-located inside the current one.  "
+located inside the current one."
   (if parent?
       (pcase type
 	(`headline 'section)
diff --git a/lisp/org-keys.el b/lisp/org-keys.el
index 37df29983..a714dec0f 100644
--- a/lisp/org-keys.el
+++ b/lisp/org-keys.el
@@ -437,7 +437,7 @@ COMMANDS is a list of alternating OLDDEF NEWDEF command names."
   #'org-next-visible-heading)
 (define-key org-mode-map [remap outline-previous-visible-heading]
   #'org-previous-visible-heading)
-(define-key org-mode-map [remap show-children] #'org-show-children)
+(define-key org-mode-map [remap outline-show-children] #'org-show-children)
 
 ;;;; Make `C-c C-x' a prefix key
 (org-defkey org-mode-map (kbd "C-c C-x") (make-sparse-keymap))
diff --git a/lisp/org-macs.el b/lisp/org-macs.el
index a02f713ca..b17c0cb4d 100644
--- a/lisp/org-macs.el
+++ b/lisp/org-macs.el
@@ -682,7 +682,7 @@ When NEXT is non-nil, check the next line instead."
 
 
 \f
-;;; Overlays
+;;; Overlays and text properties
 
 (defun org-overlay-display (ovl text &optional face evap)
   "Make overlay OVL display TEXT with face FACE."
@@ -705,26 +705,126 @@ If DELETE is non-nil, delete all those overlays."
 	    (delete (delete-overlay ov))
 	    (t (push ov found))))))
 
+(defun org-remove-text-properties (start end properties &optional object)
+  "Remove text properties as in `remove-text-properties', but keep 'invisibility specs for folded regions.
+Do not remove invisible text properties specified by 'outline,
+'org-hide-block, and 'org-hide-drawer (but remove i.e. 'org-link) this
+is needed to keep outlines, drawers, and blocks hidden unless they are
+toggled by user.
+Note: The below may be too specific and create troubles if more
+invisibility specs are added to org in future"
+  (when (plist-member properties 'invisible)
+    (let ((pos start)
+	  next spec)
+      (while (< pos end)
+	(setq next (next-single-property-change pos 'invisible nil end)
+              spec (get-text-property pos 'invisible))
+	(unless (memq spec (list 'org-hide-block
+				 'org-hide-drawer
+				 'outline))
+          (remove-text-properties pos next '(invisible nil) object))
+	(setq pos next))))
+  (when-let ((properties-stripped (org-plist-delete properties 'invisible)))
+    (remove-text-properties start end properties-stripped object)))
+
+(defun org--find-text-property-region (pos prop)
+  "Find a region containing PROP text property around point POS."
+  (let* ((beg (and (get-text-property pos prop) pos))
+	 (end beg))
+    (when beg
+      ;; when beg is the first point in the region, `previous-single-property-change'
+      ;; will return nil.
+      (setq beg (or (previous-single-property-change pos prop)
+		    beg))
+      ;; when end is the last point in the region, `next-single-property-change'
+      ;; will return nil.
+      (setq end (or (next-single-property-change pos prop)
+		    end))
+      (unless (= beg end) ; this should not happen
+        (cons beg end)))))
+
+(defun org--add-to-list-text-property (from to prop element)
+  "Add element to text property PROP, whos value should be a list."
+  (add-text-properties from to `(,prop ,(list element))) ; create if none
+  ;; add to existing
+  (alter-text-property from to
+		       prop
+		       (lambda (val)
+			 (if (member element val)
+                             val
+			   (cons element val)))))
+
+(defun org--remove-from-list-text-property (from to prop element)
+  "Remove ELEMENT from text propery PROP, whos value should be a list."
+  (let ((pos from))
+    (while (< pos to)
+      (when-let ((val (get-text-property pos prop)))
+	(if (equal val (list element))
+	    (remove-text-properties pos (next-single-char-property-change pos prop nil to) (list prop nil))
+	  (put-text-property pos (next-single-char-property-change pos prop nil to)
+			     prop (remove element (get-text-property pos prop)))))
+      (setq pos (next-single-char-property-change pos prop nil to)))))
+
+(defvar org--invisible-spec-priority-list '(outline org-hide-drawer org-hide-block)
+  "Priority of invisibility specs.")
+
+(defun org--get-buffer-local-invisible-property-symbol (spec &optional buffer return-only)
+  "Return unique symbol suitable to be used as buffer-local in BUFFER for 'invisible SPEC.
+If the buffer already have buffer-local setup in `char-property-alias-alist'
+and the setup appears to be created for different buffer,
+copy the old invisibility state into new buffer-local text properties,
+unless RETURN-ONLY is non-nil."
+  (if (not (member spec org--invisible-spec-priority-list))
+      (user-error "%s should be a valid invisibility spec" spec)
+    (let* ((buf (or buffer (current-buffer))))
+      (let ((local-prop (intern (format "org--invisible-%s-buffer-local-%S"
+					(symbol-name spec)
+					;; (sxhash buf) appears to be not constant over time.
+					;; Using buffer-name is safe, since the only place where
+					;; buffer-local text property actually matters is an indirect
+					;; buffer, where the name cannot be same anyway.
+					(sxhash (buffer-name buf))))))
+        (prog1
+            local-prop
+          (unless return-only
+	    (with-current-buffer buf
+	      (unless (member local-prop (alist-get 'invisible char-property-alias-alist))
+		;; copy old property
+		(dolist (old-prop (alist-get 'invisible char-property-alias-alist))
+		  (org-with-wide-buffer
+		   (let* ((pos (point-min))
+			  (spec (seq-find (lambda (spec)
+					    (string-match-p (symbol-name spec)
+							    (symbol-name old-prop)))
+					  org--invisible-spec-priority-list))
+			  (new-prop (org--get-buffer-local-invisible-property-symbol spec nil 'return-only)))
+		     (while (< pos (point-max))
+		       (when-let (val (get-text-property pos old-prop))
+			 (put-text-property pos (next-single-char-property-change pos old-prop) new-prop val))
+		       (setq pos (next-single-char-property-change pos old-prop))))))
+		(setq-local char-property-alias-alist
+			    (cons (cons 'invisible
+					(mapcar (lambda (spec)
+						  (org--get-buffer-local-invisible-property-symbol spec nil 'return-only))
+						org--invisible-spec-priority-list))
+				  (remove (assq 'invisible char-property-alias-alist)
+					  char-property-alias-alist)))))))))))
+
 (defun org-flag-region (from to flag spec)
   "Hide or show lines from FROM to TO, according to FLAG.
 SPEC is the invisibility spec, as a symbol."
-  (remove-overlays from to 'invisible spec)
-  ;; Use `front-advance' since text right before to the beginning of
-  ;; the overlay belongs to the visible line than to the contents.
-  (when flag
-    (let ((o (make-overlay from to nil 'front-advance)))
-      (overlay-put o 'evaporate t)
-      (overlay-put o 'invisible spec)
-      (overlay-put o 'isearch-open-invisible #'delete-overlay))))
-
+  (with-silent-modifications
+    (remove-text-properties from to (list (org--get-buffer-local-invisible-property-symbol spec) nil))
+    (when flag
+      (put-text-property from to (org--get-buffer-local-invisible-property-symbol spec) spec))))
 
 \f
 ;;; Regexp matching
 
 (defsubst org-pos-in-match-range (pos n)
-  (and (match-beginning n)
-       (<= (match-beginning n) pos)
-       (>= (match-end n) pos)))
+(and (match-beginning n)
+     (<= (match-beginning n) pos)
+     (>= (match-end n) pos)))
 
 (defun org-skip-whitespace ()
   "Skip over space, tabs and newline characters."
diff --git a/lisp/org-src.el b/lisp/org-src.el
index 6f6c544dc..9e8a50044 100644
--- a/lisp/org-src.el
+++ b/lisp/org-src.el
@@ -529,8 +529,8 @@ Leave point in edit buffer."
 	(org-src-switch-to-buffer buffer 'edit)
 	;; Insert contents.
 	(insert contents)
-	(remove-text-properties (point-min) (point-max)
-				'(display nil invisible nil intangible nil))
+	(org-remove-text-properties (point-min) (point-max)
+				    '(display nil invisible nil intangible nil))
 	(unless preserve-ind (org-do-remove-indentation))
 	(set-buffer-modified-p nil)
 	(setq buffer-file-name nil)
diff --git a/lisp/org-table.el b/lisp/org-table.el
index 6462b99c4..75801161b 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -2001,7 +2001,7 @@ toggle `org-table-follow-field-mode'."
    (arg
     (let ((b (save-excursion (skip-chars-backward "^|") (point)))
 	  (e (save-excursion (skip-chars-forward "^|\r\n") (point))))
-      (remove-text-properties b e '(invisible t intangible t))
+      (org-remove-text-properties b e '(invisible t intangible t))
       (if (and (boundp 'font-lock-mode) font-lock-mode)
 	  (font-lock-fontify-block))))
    (t
@@ -2028,7 +2028,7 @@ toggle `org-table-follow-field-mode'."
       (setq word-wrap t)
       (goto-char (setq p (point-max)))
       (insert (org-trim field))
-      (remove-text-properties p (point-max) '(invisible t intangible t))
+      (org-remove-text-properties p (point-max) '(invisible t intangible t))
       (goto-char p)
       (setq-local org-finish-function 'org-table-finish-edit-field)
       (setq-local org-window-configuration cw)
diff --git a/lisp/org.el b/lisp/org.el
index e5cea04c6..3d4a7b072 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -114,6 +114,7 @@ Stars are put in group 1 and the trimmed body in group 2.")
 (declare-function cdlatex-math-symbol "ext:cdlatex")
 (declare-function Info-goto-node "info" (nodename &optional fork strict-case))
 (declare-function isearch-no-upper-case-p "isearch" (string regexp-flag))
+(declare-function isearch-filter-visible "isearch" (beg end))
 (declare-function org-add-archive-files "org-archive" (files))
 (declare-function org-agenda-entry-get-agenda-timestamp "org-agenda" (pom))
 (declare-function org-agenda-list "org-agenda" (&optional arg start-day span with-hour))
@@ -192,6 +193,9 @@ Stars are put in group 1 and the trimmed body in group 2.")
 
 (defvar ffap-url-regexp)
 (defvar org-element-paragraph-separate)
+(defvar org-element-all-objects)
+(defvar org-element-all-elements)
+(defvar org-element-greater-elements)
 (defvar org-indent-indentation-per-level)
 (defvar org-radio-target-regexp)
 (defvar org-target-link-regexp)
@@ -4734,9 +4738,153 @@ This is for getting out of special buffers like capture.")
 
 ;;;; Define the Org mode
 
+;;; Handling buffer modifications
+
 (defun org-before-change-function (_beg _end)
   "Every change indicates that a table might need an update."
   (setq org-table-may-need-update t))
+
+(defun org-after-change-function (from to len)
+  "Process changes in folded elements.
+If a text was inserted into invisible region, hide the inserted text.
+If the beginning/end line of a folded drawer/block was changed, unfold it.
+If a valid end line was inserted in the middle of the folded drawer/block, unfold it."
+
+  ;; re-hide text inserted in the middle of a folded region
+  (dolist (spec org--invisible-spec-priority-list)
+    (when-let ((spec-to (get-text-property to (org--get-buffer-local-invisible-property-symbol spec)))
+	       (spec-from (get-text-property (max (point-min) (1- from)) (org--get-buffer-local-invisible-property-symbol spec))))
+      (when (eq spec-to spec-from)
+	(org-flag-region from to 't spec-to))))
+
+  ;; Process all the folded text between `from' and `to'
+  (org-with-wide-buffer
+
+   (if (< to from)
+       (let ((tmp from))
+	 (setq from to)
+         (setq to tmp)))
+   
+   ;; Include next/previous line into the changed region.
+   ;; This is needed to catch edits in beginning line of a folded
+   ;; element.
+   (setq to (save-excursion (goto-char to) (forward-line) (point)))
+   (setq from (save-excursion (goto-char from) (forward-line -1) (point)))
+   
+   ;; Expand the considered region to include partially present folded
+   ;; drawer/block.
+   (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))
+     (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))
+   (when (get-text-property from (org--get-buffer-local-invisible-property-symbol 'org-hide-block))
+     (setq from (previous-single-char-property-change from (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))
+
+   ;; check folded drawers
+   (let ((pos from))
+     (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))
+       (setq pos (next-single-char-property-change pos
+						   (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))
+     (while (< pos to)
+       (when-let ((drawer-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))
+				     pos))
+		  (drawer-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer))))
+
+	 (let (unfold?)
+           ;; the line before folded text should be beginning of the drawer
+           (save-excursion
+             (goto-char drawer-begin)
+             (backward-char)
+             (beginning-of-line)
+             (unless (looking-at-p org-drawer-regexp)
+	       (setq unfold? t)))
+           ;; the last line of the folded text should be :END:
+           (save-excursion
+             (goto-char drawer-end)
+             (beginning-of-line)
+             (unless (let ((case-fold-search t)) (looking-at-p org-property-end-re))
+	       (setq unfold? t)))
+           ;; there should be no :END: anywhere in the drawer body
+           (save-excursion
+             (goto-char drawer-begin)
+             (when (save-excursion
+		     (let ((case-fold-search t))
+		       (re-search-forward org-property-end-re
+					  (max (point)
+					       (1- (save-excursion
+						     (goto-char drawer-end)
+                                                     (line-beginning-position))))
+                                          't)))
+	       (setq unfold? t)))
+           ;; there should be no new entry anywhere in the drawer body
+           (save-excursion
+             (goto-char drawer-begin)
+             (when (save-excursion
+		     (let ((case-fold-search t))
+		       (re-search-forward org-outline-regexp-bol
+					  (max (point)
+					       (1- (save-excursion
+						     (goto-char drawer-end)
+                                                     (line-beginning-position))))
+                                          't)))
+	       (setq unfold? t)))
+
+           (when unfold? (org-flag-region drawer-begin drawer-end nil 'org-hide-drawer))))
+       
+       (setq pos (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-drawer)))))
+
+   ;; check folded blocks
+   (let ((pos from))
+     (unless (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block))
+       (setq pos (next-single-char-property-change pos
+						   (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))
+     (while (< pos to)
+       (when-let ((block-begin (and (get-text-property pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block))
+				    pos))
+		  (block-end (next-single-char-property-change pos (org--get-buffer-local-invisible-property-symbol 'org-hide-block))))
+
+	 (let (unfold?)
+           ;; the line before folded text should be beginning of the block
+           (save-excursion
+             (goto-char block-begin)
+             (backward-char)
+             (beginning-of-line)
+             (unless (looking-at-p org-dblock-start-re)
+	       (setq unfold? t)))
+           ;; the last line of the folded text should be end of the block
+           (save-excursion
+             (goto-char block-end)
+             (beginning-of-line)
+             (unless (looking-at-p org-dblock-end-re)
+	       (setq unfold? t)))
+           ;; there should be no #+end anywhere in the block body
+           (save-excursion
+             (goto-char block-begin)
+             (when (save-excursion
+		     (re-search-forward org-dblock-end-re
+					(max (point)
+					     (1- (save-excursion
+						   (goto-char block-end)
+						   (line-beginning-position))))
+                                        't))
+	       (setq unfold? t)))
+           ;; there should be no new entry anywhere in the block body
+           (save-excursion
+             (goto-char block-begin)
+             (when (save-excursion
+		     (let ((case-fold-search t))
+		       (re-search-forward org-outline-regexp-bol
+					  (max (point)
+					       (1- (save-excursion
+						     (goto-char block-end)
+                                                     (line-beginning-position))))
+                                          't)))
+	       (setq unfold? t)))
+
+           (when unfold? (org-flag-region block-begin block-end nil 'org-hide-block))))
+       
+       (setq pos
+	     (next-single-char-property-change pos
+					       (org--get-buffer-local-invisible-property-symbol 'org-hide-block)))))))
+
 (defvar org-mode-map)
 (defvar org-inhibit-startup-visibility-stuff nil) ; Dynamically-scoped param.
 (defvar org-agenda-keep-modes nil)      ; Dynamically-scoped param.
@@ -4789,6 +4937,7 @@ The following commands are available:
   (org-install-agenda-files-menu)
   (when org-link-descriptive (add-to-invisibility-spec '(org-link)))
   (add-to-invisibility-spec '(org-hide-block . t))
+  (add-to-invisibility-spec '(org-hide-drawer . t))
   (setq-local outline-regexp org-outline-regexp)
   (setq-local outline-level 'org-outline-level)
   (setq bidi-paragraph-direction 'left-to-right)
@@ -4817,6 +4966,8 @@ The following commands are available:
   ;; Activate before-change-function
   (setq-local org-table-may-need-update t)
   (add-hook 'before-change-functions 'org-before-change-function nil 'local)
+  ;; Activate after-change-function
+  (add-hook 'after-change-functions 'org-after-change-function nil 'local)
   ;; Check for running clock before killing a buffer
   (add-hook 'kill-buffer-hook 'org-check-running-clock nil 'local)
   ;; Initialize macros templates.
@@ -4868,6 +5019,10 @@ The following commands are available:
   (setq-local outline-isearch-open-invisible-function
 	      (lambda (&rest _) (org-show-context 'isearch)))
 
+  ;; Make isearch search in blocks hidden via text properties
+  (setq-local isearch-filter-predicate #'org--isearch-filter-predicate)
+  (add-hook 'isearch-mode-end-hook #'org--clear-isearch-overlays nil 'local)
+
   ;; Setup the pcomplete hooks
   (setq-local pcomplete-command-completion-function #'org-pcomplete-initial)
   (setq-local pcomplete-command-name-function #'org-command-at-point)
@@ -5049,8 +5204,8 @@ stacked delimiters is N.  Escaping delimiters is not possible."
 	      (when verbatim?
 		(org-remove-flyspell-overlays-in
 		 (match-beginning 0) (match-end 0))
-		(remove-text-properties (match-beginning 2) (match-end 2)
-					'(display t invisible t intangible t)))
+		(org-remove-text-properties (match-beginning 2) (match-end 2)
+					 '(display t invisible t intangible t)))
 	      (add-text-properties (match-beginning 2) (match-end 2)
 				   '(font-lock-multiline t org-emphasis t))
 	      (when (and org-hide-emphasis-markers
@@ -5165,7 +5320,7 @@ This includes angle, plain, and bracket links."
 	    (if (not (eq 'bracket style))
 		(add-text-properties start end properties)
 	      ;; Handle invisible parts in bracket links.
-	      (remove-text-properties start end '(invisible nil))
+	      (org-remove-text-properties start end '(invisible nil))
 	      (let ((hidden
 		     (append `(invisible
 			       ,(or (org-link-get-parameter type :display)
@@ -5185,8 +5340,8 @@ This includes angle, plain, and bracket links."
 (defun org-activate-code (limit)
   (when (re-search-forward "^[ \t]*\\(:\\(?: .*\\|$\\)\n?\\)" limit t)
     (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
-    (remove-text-properties (match-beginning 0) (match-end 0)
-			    '(display t invisible t intangible t))
+    (org-remove-text-properties (match-beginning 0) (match-end 0)
+			     '(display t invisible t intangible t))
     t))
 
 (defcustom org-src-fontify-natively t
@@ -5257,8 +5412,8 @@ by a #."
 	    (setq block-end (match-beginning 0)) ; includes the final newline.
 	    (when quoting
 	      (org-remove-flyspell-overlays-in bol-after-beginline nl-before-endline)
-	      (remove-text-properties beg end-of-endline
-				      '(display t invisible t intangible t)))
+	      (org-remove-text-properties beg end-of-endline
+				       '(display t invisible t intangible t)))
 	    (add-text-properties
 	     beg end-of-endline '(font-lock-fontified t font-lock-multiline t))
 	    (org-remove-flyspell-overlays-in beg bol-after-beginline)
@@ -5312,9 +5467,9 @@ by a #."
 	     '(font-lock-fontified t face org-document-info))))
 	 ((string-prefix-p "+caption" dc1)
 	  (org-remove-flyspell-overlays-in (match-end 2) (match-end 0))
-	  (remove-text-properties (match-beginning 0) (match-end 0)
-				  '(display t invisible t intangible t))
-	  ;; Handle short captions
+	  (org-remove-text-properties (match-beginning 0) (match-end 0)
+				   '(display t invisible t intangible t))
+	  ;; Handle short captions.
 	  (save-excursion
 	    (beginning-of-line)
 	    (looking-at (rx (group (zero-or-more blank)
@@ -5335,8 +5490,8 @@ by a #."
 	   '(font-lock-fontified t face font-lock-comment-face)))
 	 (t ;; Just any other in-buffer setting, but not indented
 	  (org-remove-flyspell-overlays-in (match-beginning 0) (match-end 0))
-	  (remove-text-properties (match-beginning 0) (match-end 0)
-				  '(display t invisible t intangible t))
+	  (org-remove-text-properties (match-beginning 0) (match-end 0)
+				   '(display t invisible t intangible t))
 	  (add-text-properties beg (match-end 0)
 			       '(font-lock-fontified t face org-meta-line))
 	  t))))))
@@ -5721,35 +5876,59 @@ needs to be inserted at a specific position in the font-lock sequence.")
       (decompose-region (point-min) (point-max))
       (message "Entities are now displayed as plain text"))))
 
-(defvar-local org-custom-properties-overlays nil
-  "List of overlays used for custom properties.")
+(defvar-local org-custom-properties-hidden-p nil
+  "Non-nil when custom properties are hidden.")
+
+(defcustom org-custom-properties-hide-emptied-drawers nil
+  "Non-nil means that drawers containing only `org-custom-properties' will be hidden together with the properties."
+  :group 'org
+  :type '(choice
+	  (const :tag "Don't hide emptied drawers" nil)
+	  (const :tag "Hide emptied drawers" t)))
 
 (defun org-toggle-custom-properties-visibility ()
   "Display or hide properties in `org-custom-properties'."
   (interactive)
-  (if org-custom-properties-overlays
-      (progn (mapc #'delete-overlay org-custom-properties-overlays)
-	     (setq org-custom-properties-overlays nil))
+  (require 'org-macs)
+  (add-to-invisibility-spec 'org-hide-custom-property)
+  (add-to-list 'org--invisible-spec-priority-list 'org-hide-custom-property)
+  (if org-custom-properties-hidden-p
+      (let (match)
+	(setq org-custom-properties-hidden-p nil)
+	(org-with-wide-buffer
+         (goto-char (point-min))
+         (with-silent-modifications
+           (while (setq match (text-property-search-forward (org--get-buffer-local-invisible-property-symbol 'org-hide-custom-property) 'org-hide-custom-property t))
+             (org-flag-region (prop-match-beginning match)
+			      (prop-match-end match)
+			      nil 'org-hide-custom-property)))))
     (when org-custom-properties
+      (setq org-custom-properties-hidden-p t)
       (org-with-wide-buffer
-       (goto-char (point-min))
-       (let ((regexp (org-re-property (regexp-opt org-custom-properties) t t)))
+       (let* ((regexp (org-re-property (regexp-opt org-custom-properties) t t))
+	      (regexp-drawer (format "%s\n\\(?:%s\\)+\n%s"
+				     (replace-regexp-in-string "\\$$" "" org-drawer-regexp)
+                                     (replace-regexp-in-string "\\(^\\^\\|\\$$\\)" "" regexp)
+                                     (replace-regexp-in-string "^\\^" "" org-property-end-re))))
+         
+         (when org-custom-properties-hide-emptied-drawers
+	   (goto-char (point-min))
+           (while (re-search-forward regexp-drawer nil t)
+             (with-silent-modifications
+               (org-flag-region (1- (match-beginning 0)) (match-end 0) t 'org-hide-custom-property))))
+         
+         (goto-char (point-min))
 	 (while (re-search-forward regexp nil t)
 	   (let ((end (cdr (save-match-data (org-get-property-block)))))
 	     (when (and end (< (point) end))
 	       ;; Hide first custom property in current drawer.
-	       (let ((o (make-overlay (match-beginning 0) (1+ (match-end 0)))))
-		 (overlay-put o 'invisible t)
-		 (overlay-put o 'org-custom-property t)
-		 (push o org-custom-properties-overlays))
-	       ;; Hide additional custom properties in the same drawer.
-	       (while (re-search-forward regexp end t)
-		 (let ((o (make-overlay (match-beginning 0) (1+ (match-end 0)))))
-		   (overlay-put o 'invisible t)
-		   (overlay-put o 'org-custom-property t)
-		   (push o org-custom-properties-overlays)))))
-	   ;; Each entry is limited to a single property drawer.
-	   (outline-next-heading)))))))
+	       (with-silent-modifications
+		 (org-flag-region (match-beginning 0) (1+ (match-end 0)) t 'org-hide-custom-property)
+		 ;; Hide additional custom properties in the same drawer.
+		 (while (re-search-forward regexp end t)
+		   (org-flag-region (match-beginning 0) (1+ (match-end 0)) t 'org-hide-custom-property))))))
+	 ;; Each entry is limited to a single property drawer.
+	 (outline-next-heading))))))
 
 (defun org-fontify-entities (limit)
   "Find an entity to fontify."
@@ -5858,10 +6037,11 @@ If TAG is a number, get the corresponding match group."
 	 (inhibit-modification-hooks t)
 	 deactivate-mark buffer-file-name buffer-file-truename)
     (decompose-region beg end)
-    (remove-text-properties beg end
-			    '(mouse-face t keymap t org-linked-text t
-					 invisible t intangible t
-					 org-emphasis t))
+    (org-remove-text-properties beg end
+			     '(mouse-face t keymap t org-linked-text t
+					  invisible t
+                                          intangible t
+					  org-emphasis t))
     (org-remove-font-lock-display-properties beg end)))
 
 (defconst org-script-display  '(((raise -0.3) (height 0.7))
@@ -5969,6 +6149,29 @@ open and agenda-wise Org files."
 
 ;;;; Headlines visibility
 
+(defun org-hide-entry ()
+  "Hide the body directly following this heading."
+  (interactive)
+  (save-excursion
+    (outline-back-to-heading)
+    (outline-end-of-heading)
+    (org-flag-region (point) (progn (outline-next-preface) (point)) t 'outline)))
+
+(defun org-hide-subtree ()
+  "Hide everything after this heading at deeper levels."
+  (interactive)
+  (org-flag-subtree t))
+
+(defun org-hide-sublevels (levels)
+  "Hide everything but the top LEVELS levels of headers, in whole buffer.
+This also unhides the top heading-less body, if any.
+
+Interactively, the prefix argument supplies the value of LEVELS.
+When invoked without a prefix argument, LEVELS defaults to the level
+of the current heading, or to 1 if the current line is not a heading."
+  (cl-letf (((symbol-function 'outline-flag-region) #'org-flag-region))
+    (org-hide-sublevels levels)))
+
 (defun org-show-entry ()
   "Show the body directly following this heading.
 Show the heading too, if it is currently invisible."
@@ -5980,13 +6183,24 @@ Show the heading too, if it is currently invisible."
        (line-end-position 0)
        (save-excursion
 	 (if (re-search-forward
-	      (concat "[\r\n]\\(" org-outline-regexp "\\)") nil t)
+	      (concat "[\r\n]\\(" (org-get-limited-outline-regexp) "\\)") nil t)
 	     (match-beginning 1)
 	   (point-max)))
        nil
        'outline)
       (org-cycle-hide-drawers 'children))))
 
+(defun org-show-heading ()
+  "Show the current heading and move to its end."
+  (org-flag-region (- (point)
+ 		   (if (bobp) 0
+ 		     (if (and outline-blank-line
+                              (eq (char-before (1- (point))) ?\n))
+ 			 2 1)))
+		(progn (outline-end-of-heading) (point))
+		nil
+                'outline))
+
 (defun org-show-children (&optional level)
   "Show all direct subheadings of this heading.
 Prefix arg LEVEL is how many levels below the current level
@@ -6030,6 +6244,11 @@ heading to appear."
   (org-flag-region
    (point) (save-excursion (org-end-of-subtree t t)) nil 'outline))
 
+(defun org-show-branches ()
+  "Show all subheadings of this heading, but not their bodies."
+  (interactive)
+  (org-show-children 1000))
+
 ;;;; Blocks and drawers visibility
 
 (defun org--hide-wrapper-toggle (element category force no-error)
@@ -6062,7 +6281,9 @@ Return a non-nil value when toggling is successful."
 	;; at the block closing line.
 	(unless (let ((eol (line-end-position)))
 		  (and (> eol start) (/= eol end)))
-	  (let* ((spec (if (eq category 'block) 'org-hide-block 'outline))
+	  (let* ((spec (cond ((eq category 'block) 'org-hide-block)
+			     ((eq category 'drawer) 'org-hide-drawer)
+			     (t 'outline)))
 		 (flag
 		  (cond ((eq force 'off) nil)
 			(force t)
@@ -6115,24 +6336,24 @@ Return a non-nil value when toggling is successful."
 
 (defun org-hide-drawer-all ()
   "Fold all drawers in the current buffer."
+  (org-show-all '(drawers))
   (save-excursion
     (goto-char (point-min))
     (while (re-search-forward org-drawer-regexp nil t)
-      (let* ((pair (get-char-property-and-overlay (line-beginning-position)
-						  'invisible))
-	     (o (cdr-safe pair)))
-	(if (overlayp o) (goto-char (overlay-end o)) ;invisible drawer
-	  (pcase (get-char-property-and-overlay (point) 'invisible)
-	    (`(outline . ,o) (goto-char (overlay-end o))) ;already folded
-	    (_
-	     (let* ((drawer (org-element-at-point))
-		    (type (org-element-type drawer)))
-	       (when (memq type '(drawer property-drawer))
-		 (org-hide-drawer-toggle t nil drawer)
-		 ;; Make sure to skip drawer entirely or we might flag it
-		 ;; another time when matching its ending line with
-		 ;; `org-drawer-regexp'.
-		 (goto-char (org-element-property :end drawer)))))))))))
+      (let* ((drawer (org-element-at-point))
+	     (type (org-element-type drawer)))
+	(when (memq type '(drawer property-drawer))
+	  ;; We are sure regular drawers are unfolded because of
+	  ;; `org-show-all' call above.  However, property drawers may
+	  ;; be folded, or in a folded headline.  In that case, do not
+	  ;; re-hide it.
+	  (unless (and (eq type 'property-drawer)
+		       (eq 'org-hide-drawer (get-char-property (point) 'invisible)))
+	    (org-hide-drawer-toggle t nil drawer))
+	  ;; Make sure to skip drawer entirely or we might flag it
+	  ;; another time when matching its ending line with
+	  ;; `org-drawer-regexp'.
+	  (goto-char (org-element-property :end drawer)))))))
 
 (defun org-cycle-hide-drawers (state)
   "Re-hide all drawers after a visibility state change.
@@ -6147,9 +6368,10 @@ STATE should be one of the symbols listed in the docstring of
 		      (t (save-excursion (org-end-of-subtree t t))))))
       (org-with-point-at beg
 	(while (re-search-forward org-drawer-regexp end t)
-	  (pcase (get-char-property-and-overlay (point) 'invisible)
+	  (pcase (get-char-property (point) 'invisible)
 	    ;; Do not fold already folded drawers.
-	    (`(outline . ,o) (goto-char (overlay-end o)))
+	    ('outline
+             (goto-char (min end (next-single-char-property-change (point) 'invisible))))
 	    (_
 	     (let ((drawer (org-element-at-point)))
 	       (when (memq (org-element-type drawer) '(drawer property-drawer))
@@ -6172,31 +6394,13 @@ By default, the function expands headings, blocks and drawers.
 When optional argument TYPE is a list of symbols among `blocks',
 `drawers' and `headings', to only expand one specific type."
   (interactive)
-  (let ((types (or types '(blocks drawers headings))))
-    (when (memq 'blocks types)
-      (org-flag-region (point-min) (point-max) nil 'org-hide-block))
-    (cond
-     ;; Fast path.  Since headings and drawers share the same
-     ;; invisible spec, clear everything in one go.
-     ((and (memq 'headings types)
-	   (memq 'drawers types))
-      (org-flag-region (point-min) (point-max) nil 'outline))
-     ((memq 'headings types)
-      (org-flag-region (point-min) (point-max) nil 'outline)
-      (org-cycle-hide-drawers 'all))
-     ((memq 'drawers types)
-      (save-excursion
-	(goto-char (point-min))
-	(while (re-search-forward org-drawer-regexp nil t)
-	  (let* ((pair (get-char-property-and-overlay (line-beginning-position)
-						      'invisible))
-		 (o (cdr-safe pair)))
-	    (if (overlayp o) (goto-char (overlay-end o))
-	      (pcase (get-char-property-and-overlay (point) 'invisible)
-		(`(outline . ,o)
-		 (goto-char (overlay-end o))
-		 (delete-overlay o))
-		(_ nil))))))))))
+  (dolist (type (or types '(blocks drawers headings)))
+    (org-flag-region (point-min) (point-max) nil
+		     (pcase type
+		       (`blocks 'org-hide-block)
+		       (`drawers 'org-hide-drawer)
+		       (`headings 'outline)
+		       (_ (error "Invalid type: %S" type))))))
 
 ;;;###autoload
 (defun org-cycle (&optional arg)
@@ -6552,7 +6756,7 @@ With a numeric prefix, show all headlines up to that level."
 		     (org-narrow-to-subtree)
 		     (org-content))))
 		((or "all" "showall")
-		 (outline-show-subtree))
+		 (org-show-subtree))
 		(_ nil)))
 	    (org-end-of-subtree)))))))
 
@@ -6625,7 +6829,7 @@ This function is the default value of the hook `org-cycle-hook'."
 	  (while (re-search-forward re nil t)
 	    (when (and (not (org-invisible-p))
 		       (org-invisible-p (line-end-position)))
-	      (outline-hide-entry))))
+	      (org-hide-entry))))
 	(org-cycle-hide-drawers 'all)
 	(org-cycle-show-empty-lines 'overview)))))
 
@@ -6697,10 +6901,11 @@ information."
     (org-show-entry)
     ;; If point is hidden within a drawer or a block, make sure to
     ;; expose it.
-    (dolist (o (overlays-at (point)))
-      (when (memq (overlay-get o 'invisible)
-		  '(org-hide-block org-hide-drawer outline))
-	(delete-overlay o)))
+    (when (memq (get-text-property (point) 'invisible)
+		'(org-hide-block org-hide-drawer))
+      (let ((spec (get-text-property (point) 'invisible))
+	    (region (org--find-text-property-region (point) 'invisible)))
+	(org-flag-region (car region) (cdr region) nil spec)))
     (unless (org-before-first-heading-p)
       (org-with-limited-levels
        (cl-case detail
@@ -6916,9 +7121,10 @@ unconditionally."
       ;; When INVISIBLE-OK is non-nil, ensure newly created headline
       ;; is visible.
       (unless invisible-ok
-	(pcase (get-char-property-and-overlay (point) 'invisible)
-	  (`(outline . ,o)
-	   (move-overlay o (overlay-start o) (line-end-position 0)))
+	(pcase (get-char-property (point) 'invisible)
+	  ('outline
+           (let ((region (org--find-text-property-region (point) 'invisible)))
+	     (org-flag-region (line-end-position 0) (cdr region) nil 'outline)))
 	  (_ nil))))
      ;; At a headline...
      ((org-at-heading-p)
@@ -7515,7 +7721,6 @@ case."
      (setq txt (buffer-substring beg end))
      (org-save-markers-in-region beg end)
      (delete-region beg end)
-     (org-remove-empty-overlays-at beg)
      (unless (= beg (point-min)) (org-flag-region (1- beg) beg nil 'outline))
      (unless (bobp) (org-flag-region (1- (point)) (point) nil 'outline))
      (and (not (bolp)) (looking-at "\n") (forward-char 1))
@@ -7677,7 +7882,7 @@ When REMOVE is non-nil, remove the subtree from the clipboard."
      (skip-chars-forward " \t\n\r")
      (setq beg (point))
      (when (and (org-invisible-p) visp)
-       (save-excursion (outline-show-heading)))
+       (save-excursion (org-show-heading)))
      ;; Shift if necessary.
      (unless (= shift 0)
        (save-restriction
@@ -8119,7 +8324,7 @@ function is being called interactively."
 		       (point))
 	    what "children")
       (goto-char start)
-      (outline-show-subtree)
+      (org-show-subtree)
       (outline-next-heading))
      (t
       ;; we will sort the top-level entries in this file
@@ -10736,7 +10941,8 @@ narrowing."
 	     (let ((beg (point)))
 	       (insert ":" drawer ":\n:END:\n")
 	       (org-indent-region beg (point))
-	       (org-flag-region (line-end-position -1) (1- (point)) t 'outline))
+	       (org-flag-region
+		(line-end-position -1) (1- (point)) t 'org-hide-drawer))
 	     (end-of-line -1)))))
       (t
        (org-end-of-meta-data org-log-state-notes-insert-after-drawers)
@@ -13173,7 +13379,7 @@ drawer is immediately hidden."
 	   (inhibit-read-only t))
        (unless (bobp) (insert "\n"))
        (insert ":PROPERTIES:\n:END:")
-       (org-flag-region (line-end-position 0) (point) t 'outline)
+       (org-flag-region (line-end-position 0) (point) t 'org-hide-drawer)
        (when (or (eobp) (= begin (point-min))) (insert "\n"))
        (org-indent-region begin (point))))))
 
@@ -16553,7 +16759,7 @@ The detailed reaction depends on the user option `org-catch-invisible-edits'."
       (when (or invisible-at-point invisible-before-point)
 	(when (eq org-catch-invisible-edits 'error)
 	  (user-error "Editing in invisible areas is prohibited, make them visible first"))
-	(if (and org-custom-properties-overlays
+	(if (and org-custom-properties-hidden-p
 		 (y-or-n-p "Display invisible properties in this buffer? "))
 	    (org-toggle-custom-properties-visibility)
 	  ;; Make the area visible
@@ -17636,11 +17842,11 @@ Move point to the beginning of first heading or end of buffer."
 (defun org-show-branches-buffer ()
   "Show all branches in the buffer."
   (org-flag-above-first-heading)
-  (outline-hide-sublevels 1)
+  (org-hide-sublevels 1)
   (unless (eobp)
-    (outline-show-branches)
+    (org-show-branches)
     (while (outline-get-next-sibling)
-      (outline-show-branches)))
+      (org-show-branches)))
   (goto-char (point-min)))
 
 (defun org-kill-note-or-show-branches ()
@@ -17654,8 +17860,8 @@ Move point to the beginning of first heading or end of buffer."
 	(t
 	 (let ((beg (progn (org-back-to-heading) (point)))
 	       (end (save-excursion (org-end-of-subtree t t) (point))))
-	   (outline-hide-subtree)
-	   (outline-show-branches)
+	   (org-hide-subtree)
+	   (org-show-branches)
 	   (org-hide-archived-subtrees beg end)))))
 
 (defun org-delete-indentation (&optional arg)
@@ -17811,9 +18017,9 @@ Otherwise, call `org-show-children'.  ARG is the level to hide."
     (if (org-before-first-heading-p)
         (progn
           (org-flag-above-first-heading)
-          (outline-hide-sublevels (or arg 1))
+          (org-hide-sublevels (or arg 1))
           (goto-char (point-min)))
-      (outline-hide-subtree)
+      (org-hide-subtree)
       (org-show-children arg))))
 
 (defun org-ctrl-c-star ()
@@ -20489,20 +20695,20 @@ With ARG, repeats or can move backward if negative."
       (end-of-line))
     (while (and (< arg 0) (re-search-backward regexp nil :move))
       (unless (bobp)
-	(while (pcase (get-char-property-and-overlay (point) 'invisible)
-		 (`(outline . ,o)
-		  (goto-char (overlay-start o))
-		  (re-search-backward regexp nil :move))
-		 (_ nil))))
+	(pcase (get-char-property (point) 'invisible)
+	  ('outline
+	   (goto-char (car (org--find-text-property-region (point) 'invisible)))
+	   (beginning-of-line))
+	  (_ nil)))
       (cl-incf arg))
-    (while (and (> arg 0) (re-search-forward regexp nil t))
-      (while (pcase (get-char-property-and-overlay (point) 'invisible)
-	       (`(outline . ,o)
-		(goto-char (overlay-end o))
-		(re-search-forward regexp nil :move))
-	       (_
-		(end-of-line)
-		nil)))			;leave the loop
+    (while (and (> arg 0) (re-search-forward regexp nil :move))
+      (pcase (get-char-property (point) 'invisible)
+	('outline
+	 (goto-char (cdr (org--find-text-property-region (point) 'invisible)))
+         (skip-chars-forward " \t\n")
+	 (end-of-line))
+	(_
+	 (end-of-line)))
       (cl-decf arg))
     (if (> arg 0) (goto-char (point-max)) (beginning-of-line))))
 
@@ -20957,6 +21163,80 @@ Started from `gnus-info-find-node'."
           (t default-org-info-node))))))
 
 \f
+
+;;; Make isearch search in some text hidden via text propertoes
+
+(defvar org--isearch-overlays nil
+  "List of overlays temporarily created during isearch.
+This is used to allow searching in regions hidden via text properties.
+As for [2020-05-09 Sat], Isearch only has special handling of hidden overlays.
+Any text hidden via text properties is not revealed even if `search-invisible'
+is set to 't.")
+
+;; Not sure if it needs to be a user option
+;; One might want to reveal hidden text in, for example, hidden parts of the links.
+;; Currently, hidden text in links is never revealed by isearch.
+(defvar org-isearch-specs '(org-hide-block
+			 org-hide-drawer)
+  "List of text invisibility specs to be searched by isearch.
+By default ([2020-05-09 Sat]), isearch does not search in hidden text,
+which was made invisible using text properties. Isearch will be forced
+to search in hidden text with any of the listed 'invisible property value.")
+
+(defun org--create-isearch-overlays (beg end)
+  "Replace text property invisibility spec by overlays between BEG and END.
+All the regions with invisibility text property spec from
+`org-isearch-specs' will be changed to use overlays instead
+of text properties. The created overlays will be stored in
+`org--isearch-overlays'."
+  (let ((pos beg))
+    (while (< pos end)
+      (when-let* ((spec (get-text-property pos 'invisible))
+		  (spec (memq spec org-isearch-specs))
+		  (region (org--find-text-property-region pos 'invisible)))
+        (setq spec (get-text-property pos 'invisible))
+        ;; Changing text properties is considered buffer modification.
+        ;; We do not want it here.
+	(with-silent-modifications
+          ;; The overlay is modelled after `org-flag-region' [2020-05-09 Sat]
+          ;; overlay for 'outline blocks.
+          (let ((o (make-overlay (car region) (cdr region) nil 'front-advance)))
+	    (overlay-put o 'evaporate t)
+	    (overlay-put o 'invisible spec)
+            ;; `delete-overlay' here means that spec information will be lost
+            ;; for the region. The region will remain visible.
+	    (overlay-put o 'isearch-open-invisible #'delete-overlay)
+            (push o org--isearch-overlays))
+	  (org-flag-region (car region) (cdr region) nil spec)))
+      (setq pos (next-single-property-change pos 'invisible nil end)))))
+
+(defun org--isearch-filter-predicate (beg end)
+  "Return non-nil if text between BEG and END is deemed visible by Isearch.
+This function is intended to be used as `isearch-filter-predicate'.
+Unlike `isearch-filter-visible', make text with 'invisible text property
+value listed in `org-isearch-specs' visible to Isearch."
+  (org--create-isearch-overlays beg end) ;; trick isearch by creating overlays in place of invisible text
+  (isearch-filter-visible beg end))
+
+(defun org--clear-isearch-overlay (ov)
+  "Convert OV region back into using text properties."
+  (when-let ((spec (overlay-get ov 'invisible))) ;; ignore deleted overlays
+    ;; Changing text properties is considered buffer modification.
+    ;; We do not want it here.
+    (with-silent-modifications
+      (org-flag-region (overlay-start ov) (overlay-end ov) t spec)))
+  (when (member ov isearch-opened-overlays)
+    (setq isearch-opened-overlays (delete ov isearch-opened-overlays)))
+  (delete-overlay ov))
+
+(defun org--clear-isearch-overlays ()
+  "Convert overlays from `org--isearch-overlays' back into using text properties."
+  (when org--isearch-overlays
+    (mapc #'org--clear-isearch-overlay org--isearch-overlays)
+    (setq org--isearch-overlays nil)))
+
+\f
+
 ;;; Finish up
 
 (add-hook 'org-mode-hook     ;remove overlays when changing major mode

[-- Attachment #3: Type: text/plain, Size: 12921 bytes --]


Ihor Radchenko <yantar92@gmail.com> writes:

> Hello,
>
> [The patch itself will be provided in the following email]
>
> I have four more updates from the previous version of the patch:
>
> 1. All the code handling modifications in folded drawers/blocks is moved
>    to after-change-function. It works as follows:
>    - if any text is inserted in the middle of hidden region, that text
>      is also hidden;
>    - if BEGIN/END line of a folded drawer do not match org-drawer-regexp
>      and org-property-end-re, unfold it; 
>    - if org-property-end-re or new org-outline-regexp-bol is inserted in
>      the middle of the drawer, unfold it;
>    - the same logic for blocks.
>
> 2. The text property stack is rewritten using char-property-alias-alist.
>    This is faster in comparison with previous approach, which involved
>    modifying all the text properties every timer org-flag-region was
>    called. 
>    
> 3. org-toggle-custom-properties-visibility is rewritten using text
>    properties. I also took a freedom to implement a new feature here.
>    Now, setting new `org-custom-properties-hide-emptied-drawers' to
>    non-nil will result in hiding the whole property drawer if it
>    contains only org-custom-properties.
>
> 4. This patch should work against 1aa095ccf. However, the merge was not
>    trivial here. Recent commits actively used the fact that drawers and
>    outlines are hidden via 'outline invisibility spec, which is not the
>    case in this branch. I am not confident that I did not break anything
>    during the merge, especially 1aa095ccf.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the new implementation for tracking changes:
>
>> I gave you a few ideas to quickly check if a change requires expansion,
>> in an earlier mail. I suggest to start out from that. Let me know if you
>> have questions about it.
>
> All the code lives in org-after-change-function. I tried to incorporate
> the earlier Nicholas' suggestions, except the parts related to
> intersecting blocks and drawers. I am not sure if I understand the
> parsing priority of blocks vs. drawers.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the text property stack:
>
> The earlier version of the code literally used stack to save
> pre-existing 'invisibility specs in org-flag-region. This was done on
> every invocation of org-flag-region, which made org-flag-region
> significantly slower. I re-implemented the same feature using
> char-property-alias-alist. Now, different invisibility specs live in
> separate text properties and can be safely modified independently. The
> specs are applied according to org--invisible-spec-priority-list. A side
> effect of current implementation is that char-property-alias-alist is
> fully controlled by org. All the pre-existing settings for 'invisible
> text property will be overwritten by org.
>
>> `gensym' is just a shorter, and somewhat standard way, to create a new
>> uninterned symbol with a given prefix. You seem to re-invent it. What
>> you do with that new symbol is orthogonal to that suggestion, of course.
>
> I do not think that `gensym' is suitable here. We don't want a new
> symbol every time org--get-buffer-local-invisible-property-symbol is
> called. It should return the same symbol if it is called from the same
> buffer multiple times.
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the org-toggle-custom-properties-visibility:
>
> The implementation showcases how to introduce new invisibility specs to
> org. Apart from expected (add-to-invisibility-spec 'org-hide-custom-property) 
> one also needs to add the spec into org--invisible-spec-priority-list:
>
> (add-to-list 'org--invisible-spec-priority-list 'org-hide-custom-property)
>
> Searching for text with the given invisibility spec is done as
> follows:
>
> (text-property-search-forward (org--get-buffer-local-invisible-property-symbol 'org-hide-custom-property) 'org-hide-custom-property t)
>
> This last piece of code is probably not the most elegant. I am thinking
> if creating some higher-level interface would be more reasonable here.
> What do you think?
>
>
> The new customisation `org-custom-properties-hide-emptied-drawers'
> sounds logical for me since empty property drawers left after invoking
> org-toggle-custom-properties-visibility are rather useless according to
> my experience. If one already wants to hide parts of property drawers, I
> do not see a reason to show leftover
>
> :PROPERTIES:
> :END:
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> More details on the merge with the latest master:
>
> I tried my best to not break anything. However, I am not sure if I
> understand all the recent commits. Could someone take a look if there is
> anything suspicious in org-next-visible-heading?
>
> Also, I have seen some optimisations making use of the fact that drawers
> and headlines both use 'outline invisibility spec. This change in the
> implementation details supposed to improve performance and should not be
> necessary if this patch is going to be merged. Would it be possible to
> refrain from abusing this particular implementation detail in the
> nearest commits on master (unless really necessary)?
>
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
>
> Further work:
>
> I would like to finalise the current patch and work on other code using
> overlays separately. This patch is already quite complicated as is. I do
> not want to introduce even more potential bugs by working on things not
> directly affected by this version of the patch.
>
> Best,
> Ihor
>
>
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>>> See also `gensym'. Do we really need to use it for something else than
>>>> `invisible'? If not, the tool doesn't need to be generic.
>>>
>>> For now, I also use it for buffer-local 'invisible stack. The stack is
>>> needed to preserve folding state of drawers/blocks inside folded
>>> outline. Though I am thinking about replacing the stack with separate
>>> text properties, like 'invisible-outline-buffer-local +
>>> 'invisible-drawer-buffer-local + 'invisible-block-buffer-local.
>>> Maintaining stack takes a noticeable percentage of CPU time in profiler.
>>>
>>> org--get-buffer-local-text-property-symbol must take care about
>>> situation with indirect buffers. When an indirect buffer is created from
>>> some org buffer, the old value of char-property-alias-alist is carried
>>> over. We need to detect this case and create new buffer-local symbol,
>>> which is unique to the newly created buffer (but not create it if the
>>> buffer-local property is already there). Then, the new symbol must
>>> replace the old alias in char-property-alias-alist + old folding state
>>> must be preserved (via copying the old invisibility specs into the new
>>> buffer-local text property). I do not see how gensym can benefit this
>>> logic.
>>
>> `gensym' is just a shorter, and somewhat standard way, to create a new
>> uninterned symbol with a given prefix. You seem to re-invent it. What
>> you do with that new symbol is orthogonal to that suggestion, of course.
>>
>>>> OK, but this may not be sufficient if we want to do slightly better than
>>>> overlays in that area. This is not mandatory, though.
>>>
>>> Could you elaborate on what can be "slightly better"? 
>>
>> IIRC, I gave examples of finer control of folding state after a change.
>> Consider this _folded_ drawer:
>>
>>   :BEGIN:
>>   Foo
>>   :END:
>>
>> Inserting ":END" in it should not unfold it, as it is currently the case
>> with overlays,
>>
>>   :BEGIN
>>   Foo
>>   :END
>>   :END:
>>
>> but a soon as the last ":" is inserted, the initial drawer could be
>> expanded.
>>
>>   :BEGIN
>>   Foo
>>   :END:
>>   :END:
>>
>> The latter case is not currently handled by overlays. This is what
>> I call "slightly better".
>>
>> Also, note that this change is not related to opening and closing lines
>> of the initial drawer, so sticking text properties on them would not
>> help here.
>>
>> Another case is modifying those borders, e.g.,
>>
>>
>>   :BEGIN:               :BEGIN:
>>   Foo         ------>   Foo  
>>   :END:                 :ND:
>>
>> which should expand the drawer. Your implementation catches this, but
>> I'm pointing out that current implementation with overlays does not.
>> Even though that's not strictly required for compatibility with
>> overlays, it is a welcome slight improvement.
>>
>>>> As discussed before, I don't think you need to use `modification-hooks'
>>>> or `insert-behind-hooks' if you already use `after-change-functions'.
>>>>
>>>> `after-change-functions' are also triggered upon text properties
>>>> changes. So, what is the use case for the other hooks?
>>>
>>> The problem is that `after-change-functions' cannot be a text property.
>>> Only `modification-hooks' and `insert-in-front/behind-hooks' can be a
>>> valid text property. If we use `after-change-functions', they will
>>> always be triggered, regardless if the change was made inside or outside
>>> folded region.
>>
>> As discussed, text properties are local to the change, but require extra
>> care when moving text around. You also observed serious overhead when
>> using them.
>>
>> OTOH, even if `a-c-f' is not local, you can quickly determine if the
>> change altered a folded element, so the overhead is limited, i.e.,
>> mostly checking for a text property at a given buffer position.
>>
>> To be clear, I initially thought that text properties were a superior
>> choice, but I changed my mind a while ago, and I thought you had, too.
>> IOW, `after-change-functions' is the way to go, since you have no strong
>> reason to stick to text properties for this kind of function.
>>
>>>>> :asd:
>>>>> :drawer:
>>>>> lksjdfksdfjl
>>>>> sdfsdfsdf
>>>>> :end:
>>>>>
>>>>> If :asd: was inserted in front of folded :drawer:, changes in :drawer:
>>>>> line of the new folded :asd: drawer would reveal the text between
>>>>> :drawer: and :end:.
>>>>>
>>>>> Let me know what you think on this.
>>>
>>>> I have first to understand the use case for `modification-hook'. But
>>>> I think unfolding is the right thing to do in this situation, isn't it?
>>>
>>> That situation arises because the modification-hooks from ":drawer:"
>>> (they are set via text properties) only have information about the
>>> :drawer:...:end: drawer before the modifications (they were set when
>>> :drawer: was folded last time). So, they will only unfold a part of the
>>> new :asd: drawer. I do not see a simple way to unfold everything without
>>> re-parsing the drawer around the changed text.
>>
>> Oh! I misread your message. I withdraw what I wrote. In this case, we
>> don't want to unfold anything. The situation is not worse than what we
>> have now, and trying to fix it would have repercussions down in the
>> buffer, e.g., expanding drawers screen below.
>>
>> As a rule of thumb, I think we can pay attention to changes in the
>> folded text, and its immediate surroundings (e.g., the opening line,
>> which is not folded), but no further.
>>
>> As written above, slight changes are welcome, but let's not go overboard
>> and parse a whole section just to know if we can expand a drawer.
>>
>>> Actually, I am quite unhappy with the performance of modification-hooks
>>> set via text properties (I am using this patch on my Emacs during this
>>> week). It appears that setting the text properties costs a significant
>>> CPU time in practice, even though running the hooks is pretty fast.
>>> I will think about a way to handle modifications using global
>>> after-change-functions.
>>
>> That's better, IMO.
>>
>> I gave you a few ideas to quickly check if a change requires expansion,
>> in an earlier mail. I suggest to start out from that. Let me know if you
>> have questions about it.
>
> -- 
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg