[BUG] org-goto slows down org-set-property

[BUG] org-goto slows down org-set-property

[Maxim Nikulin]

It is rather annoying that C-c C-x p `org-set-property' may be quite 
slow if a file is large enough. (I do not consider it huge however.) 
There is noticeable delay before the prompt for property name appears in 
minibuffer. Accidentally I have noticed that sometimes it is fast.

Problem appears after first jump using C-u C-c C-j `org-goto'. Actually 
a jump is not required, it is enough to get jump (refile) targets.

I hope, the following numbers are convincing enough. Second attempt to 
get list of properties is much slower than the first one.

#+begin_src elisp
   (require 'org-refile)
   (let ((org-refile-targets '((nil :maxlevel . 6)))
	(org-refile-use-outline-path 'file)
	(org-refile-use-cache t)
	(iters 10))
     (cons
      (list (org-version))
      (mapcar (lambda (f)
	       (append (list f) (benchmark-run iters (funcall f))))
	     (list #'org-buffer-property-keys
		   #'org-refile-get-targets
		   #'org-buffer-property-keys))))
#+end_src

git master:

#+RESULTS:
| 9.4.6                    |          |    |                     |
| org-buffer-property-keys | 0.065695 |  0 |                 0.0 |
| org-refile-get-targets   | 1.836473 | 12 | 0.39498671199999985 |
| org-buffer-property-keys | 0.563227 |  0 |                 0.0 |

elpa-org system package for Ubuntu-20.04 is even more slower:

#+RESULTS:
| 9.3.1                    |             |    |             |
| org-buffer-property-keys | 0.242574778 |  0 |         0.0 |
| org-refile-get-targets   | 2.669112479 | 13 | 0.285175928 |
| org-buffer-property-keys | 1.231223427 |  0 |         0.0 |

The effect is noticeable for the doc/org-manual.org file from org 
sources though absolute numbers are well below delay that makes 
interaction incomfortable.

In the configuration above effect of `org-refile-use-cache' is crucial, 
contribution of `org-refile-use-outline-path' is noticeable but not so 
important.

My expectation is that neither `org-refile-use-cache' nor `org-goto' 
command should make adding property slower.

Re: [BUG] org-goto slows down org-set-property

[Maxim Nikulin]

On 10/07/2021 22:47, Maxim Nikulin wrote:
> It is rather annoying that C-c C-x p `org-set-property' may be quite 
> slow if a file is large enough. (I do not consider it huge however.) 
> There is noticeable delay before the prompt for property name appears in 
> minibuffer. Accidentally I have noticed that sometimes it is fast.
> 
> Problem appears after first jump using C-u C-c C-j `org-goto'. Actually 
> a jump is not required, it is enough to get jump (refile) targets.

It is strange. Merely 3500 markers created by org-refile-cache in 2.2Mb 
file ruin performance of regexps in that buffer. I would expect either 
no influence of markers at all since search is a read-only operation or 
some negligible penalty proportional to fraction of markers in relation 
to total amount of characters.

Look at the numbers for nm-test that is just some regexp searches from 
`org-buffer-property-keys'. Performance is restored after clearing of 
the cache.

#+begin_src elisp
   (defun nm-tst ()
     (org-with-wide-buffer
      (goto-char (point-min))
      (while (re-search-forward org-property-start-re nil t)
        (org-get-property-block)
        (outline-next-heading))
      nil))

   (org-refile-cache-clear)

   (let ((org-refile-targets '((nil :maxlevel . 6)))
	(org-refile-use-outline-path 'file)
	(org-refile-use-cache t)
	(iters 10))
     (cons
      (list (org-version))
      (mapcar (lambda (f)
	       (append (list f) (benchmark-run iters (funcall f))))
	     (list #'nm-tst ; #'org-buffer-property-keys
		   #'org-refile-get-targets
		   #'nm-tst ; #'org-buffer-property-keys
		   #'org-refile-cache-clear
		   #'nm-tst))))
#+end_src

#+RESULTS:
| 9.4.6                  |             |   |                    |
| nm-tst                 | 0.044981942 | 0 |                0.0 |
| org-refile-get-targets | 1.748572638 | 8 | 0.3876601379999993 |
| nm-tst                 | 0.393022201 | 0 |                0.0 |
| org-refile-cache-clear | 0.001787078 | 0 |                0.0 |
| nm-tst                 | 0.035731803 | 0 |                0.0 |

It seems, it is better to drop org-refile-cache completely. It has some 
design problems making it rather fragile. At the same time a patch can 
be applied that improves performance of `org-refile-get-targets' by a 
factor of at least 2 for (org-refile-use-outline-path 'file).

Emacs-26.3

Re: [BUG] org-goto slows down org-set-property

[Ihor Radchenko]

Maxim Nikulin <manikulin@gmail.com> writes:

> It seems, it is better to drop org-refile-cache completely. It has some 
> design problems making it rather fragile. At the same time a patch can 
> be applied that improves performance of `org-refile-get-targets' by a 
> factor of at least 2 for (org-refile-use-outline-path 'file).

I tend agree. I got rid of org-refile-cache some time ago in favour of
org-ql. The performance is much better subjectively, which indicates
that alternative caching mechanisms could be much better.

Your test on my 12M chars 18k headings org file (Emacs master, Org mode
master). I cannot use org-set-property even without org-refile-cache :(
 
#+RESULTS[e3fd8ffd9dd1a1d1cb8cf2ab3927bf647e944dd8]:
| 9.4.6                  |                    |   |                    |
| nm-tst                 | 13.638543295000002 | 4 | 1.9903503830000204 |
| org-refile-get-targets |        5.355179174 | 2 |   0.97954138099999 |
| nm-tst                 |       22.599429478 | 4 | 2.0785714209999924 |
| org-refile-cache-clear |        0.003903494 | 0 |                0.0 |
| nm-tst                 |       13.969909913 | 4 |   2.06943181500003 |

Best,
Ihor

Re: [BUG] org-goto slows down org-set-property

[Maxim Nikulin]

On 11/07/2021 18:49, Ihor Radchenko wrote:
> 
> Your test on my 12M chars 18k headings org file (Emacs master, Org mode
> master). I cannot use org-set-property even without org-refile-cache :(
>   
> | nm-tst                 |       13.969909913 | 4 |   2.06943181500003 |

Certainly 13 seconds is no go especially for interactive usage. I use 
`org-set-property' to add CUSTOM_ID when necessary. I do not like purely 
random ID. Do you have an alternative for adding properties? 
Unfortunately `org-set-property-and-value' C-c C-x P does not allow to 
set even fixed list of options for completion.

Re: [BUG] org-goto slows down org-set-property

[Ihor Radchenko]

Maxim Nikulin <manikulin@gmail.com> writes:

> Certainly 13 seconds is no go especially for interactive usage.

Yeah. So, I do not use it for now until me or someone else fixes the
issue.

> Do you have an alternative for adding properties? 
> Unfortunately `org-set-property-and-value' C-c C-x P does not allow to 
> set even fixed list of options for completion.

For now, I just edit the properties drawer manually or use custom
setters utilising org-set-property.

You can always generate a list of options using `completing-read'.
Example:

(completing-read
 "Category: "
 (org-ql-select (org-agenda-files t t)
 '(property "CATEGORY")
 :action (lambda () (substring-no-properties (org-entry-get (point) "CATEGORY")))))

> I use 
> `org-set-property' to add CUSTOM_ID when necessary. I do not like purely 
> random ID.

You may want to customise `org-id-method'. Also, I often use
`org-capture-ref-generate-key-human-readable' from my org-capture-ref
package [1].

[1] https://github.com/yantar92/org-capture-ref

Best,
Ihor

Re: [BUG] org-goto slows down org-set-property

[Maxim Nikulin]

On 11/07/2021 21:20, Ihor Radchenko wrote:
> Maxim Nikulin writes > | nm-tst                 |       22.599429478 | 4 | 2.0785714209999924 |
> | org-refile-cache-clear |        0.003903494 | 0 |                0.0 |
> | nm-tst                 |       13.969909913 | 4 |   2.06943181500003 |
> 
>> Certainly 13 seconds is no go especially for interactive usage.
> 
> Yeah. So, I do not use it for now until me or someone else fixes the
> issue.

Ihor, did you get that numbers using emacs -Q? If not, I suspect you may 
have twice more markers in the buffer than `org-refile-get-targets' 
usually generates. If my guess is correct it may be necessary to file a 
bug that org-ql or some other package abuses markers by generation to 
much of them.

P.S. Maybe generating of custom IDs automatically is a viable idea. 
Before I have not considered such variant believing that they would be 
noticeably longer and would have weaker associations.

Re: [BUG] org-goto slows down org-set-property

[Ihor Radchenko]

Maxim Nikulin <manikulin@gmail.com> writes:

> Ihor, did you get that numbers using emacs -Q? If not, I suspect you may 
> have twice more markers in the buffer than `org-refile-get-targets' 
> usually generates. If my guess is correct it may be necessary to file a 
> bug that org-ql or some other package abuses markers by generation to 
> much of them.

With emacs -Q things are twice better, though still not good enough:

#+RESULTS[b9f5ebaf91a21d8da95b8840c70f6d9aeb2ab3ae]:
| 9.4.4                  |                    |    |                    |
| nm-tst                 |  5.572397445999999 | 38 | 1.0039881250000002 |
| org-refile-get-targets |       13.029634993 | 53 | 1.3204672779999997 |
| nm-tst                 | 24.142289434000002 | 25 | 0.7106094290000002 |
| org-refile-cache-clear |         0.00502349 |  0 |                0.0 |
| nm-tst                 |        5.043277087 | 38 | 0.8292470489999997 |

org-ql might be one of the reasons of slowdown. Without it, the nm-tst
after clearing the cache is slightly faster (was 13 sec):

#+RESULTS[e5ee2bee7b348265a19834d5ca2697618a9bf261]:
| 9.4.6                  |                   |    |                    |
| nm-tst                 |      10.106726732 | 34 | 3.0993166650000004 |
| org-refile-get-targets | 5.721277742000001 | 19 | 1.6862241439999996 |
| nm-tst                 |      22.056917422 | 33 |  3.000191514999999 |
| org-refile-cache-clear |       0.001498722 |  0 |                0.0 |
| nm-tst                 |      10.066160733 | 34 |        3.203417302 |

However, markers are also created by agendas for example. The last
result is also on fresh Emacs. Fresh Emacs tends to be faster in general.

Do you know a good way to find out all the markers associated with a buffer?

> P.S. Maybe generating of custom IDs automatically is a viable idea. 
> Before I have not considered such variant believing that they would be 
> noticeably longer and would have weaker associations.

Some examples of autogenerated IDs using bibtex-autokey:
du-2021-recen-studies-in-mechan-proper-e4d
Youtube-veritasium-2021-you-cant-prove-every-thats-true-346
schuh-2005-quant-insig-into-disloc-nucleat-a1d
Karl-Voit-voit-2020-draft-workf-advan-projec-manag

You can judge for yourself if it is too long or not. Also, the length
can be customised as described in bibtex-generate-autokey docstring.

Best,
Ihor

Re: [BUG] org-goto slows down org-set-property

[Maxim Nikulin]

On 11/07/2021 23:00, Ihor Radchenko wrote:
> Maxim Nikulin writes:
> 
>> Ihor, did you get that numbers using emacs -Q? If not, I suspect you may
>> have twice more markers in the buffer than `org-refile-get-targets'
>> usually generates. If my guess is correct it may be necessary to file a
>> bug that org-ql or some other package abuses markers by generation to
>> much of them.
> 
> With emacs -Q things are twice better, though still not good enough:
...
> org-ql might be one of the reasons of slowdown. Without it, the nm-tst
> after clearing the cache is slightly faster (was 13 sec):

13.8 - 10.1 = 3.7 that is comparable with "emacs -Q" that is 5.3. So 
contribution of org-ql is not major one, but it is still significant.

> However, markers are also created by agendas for example. The last
> result is also on fresh Emacs. Fresh Emacs tends to be faster in general.
> 
> Do you know a good way to find out all the markers associated with a buffer?

I was going to ask you, but forgot to do it. Perhaps it is impossible by 
design.

(length org-agenda-markers)

Re: [BUG] org-goto slows down org-set-property

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 1649 bytes --]

This link (
https://stackoverflow.com/questions/36926513/are-there-elisp-functions-that-list-markers-in-a-given-buffer)
suggests there is not a good way to find all the markers associated with a
buffer.

John

-----------------------------------
Professor John Kitchin (he/him/his)
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu



On Mon, Jul 12, 2021 at 11:01 AM Maxim Nikulin <manikulin@gmail.com> wrote:

> On 11/07/2021 23:00, Ihor Radchenko wrote:
> > Maxim Nikulin writes:
> >
> >> Ihor, did you get that numbers using emacs -Q? If not, I suspect you may
> >> have twice more markers in the buffer than `org-refile-get-targets'
> >> usually generates. If my guess is correct it may be necessary to file a
> >> bug that org-ql or some other package abuses markers by generation to
> >> much of them.
> >
> > With emacs -Q things are twice better, though still not good enough:
> ...
> > org-ql might be one of the reasons of slowdown. Without it, the nm-tst
> > after clearing the cache is slightly faster (was 13 sec):
>
> 13.8 - 10.1 = 3.7 that is comparable with "emacs -Q" that is 5.3. So
> contribution of org-ql is not major one, but it is still significant.
>
> > However, markers are also created by agendas for example. The last
> > result is also on fresh Emacs. Fresh Emacs tends to be faster in general.
> >
> > Do you know a good way to find out all the markers associated with a
> buffer?
>
> I was going to ask you, but forgot to do it. Perhaps it is impossible by
> design.
>
> (length org-agenda-markers)
>
>
>

[-- Attachment #2: Type: text/html, Size: 2503 bytes --]

Re: [BUG] org-goto slows down org-set-property

[Ihor Radchenko]

Maxim Nikulin <manikulin@gmail.com> writes:

> It is strange. Merely 3500 markers created by org-refile-cache in 2.2Mb 
> file ruin performance of regexps in that buffer. I would expect either 
> no influence of markers at all since search is a read-only operation or 
> some negligible penalty proportional to fraction of markers in relation 
> to total amount of characters.
>
> Look at the numbers for nm-test that is just some regexp searches from 
> `org-buffer-property-keys'. Performance is restored after clearing of 
> the cache.
> ...

This is no longer reproducible on the latest Emacs master, after a
couple of improvements in the marker handling.

Canceled.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] org-goto slows down org-set-property

[Ihor Radchenko]

Maxim Nikulin <manikulin@gmail.com> writes:

> It is rather annoying that C-c C-x p `org-set-property' may be quite 
> slow if a file is large enough. (I do not consider it huge however.) 
> There is noticeable delay before the prompt for property name appears in 
> minibuffer. Accidentally I have noticed that sometimes it is fast.
>
> Problem appears after first jump using C-u C-c C-j `org-goto'. Actually 
> a jump is not required, it is enough to get jump (refile) targets.
>
> I hope, the following numbers are convincing enough. Second attempt to 
> get list of properties is much slower than the first one.
> ...

This is no longer a problem in the newer Emacs versions.
Canceled.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>