Typing latency

Typing latency

[Sébastien Miquel]

[-- Attachment #1: Type: text/plain, Size: 1759 bytes --]

I often get some unpleasant latency when editing org-mode
buffers. My use case combines a lot of special blocks and LaTeX
fragments, and the fontification seems to slow things down.

I've done some measurements using typometer
([[https://pavelfatin.com/typometer/]]). The numbers aren't very accurate
since it depends on external factors, such as how long the emacs
session has run.

AFAIU, the elisp profiler is useless in this case (fontification being
slow). At least my runs don't seem to report anything relevant.

The results are compiled below. In one of my use case (I've attached a
picture of what it looks like), I get up to 85 ms of delay, which is
very unpleasant.

| Mode                   | Content                    | Latency |
|------------------------+----------------------------+---------|
| Fundamental mode       | Generic                    | 25 ms |
| Org, w/o font locking  | Generic                    | 30 ms |
| Org-mode               | None                       | 30 ms |
| Org-mode               | Special blocks             | 47 ms |
| Org-mode               | LaTeX fragments            | 40 ms |
| Org-mode, new session  | Fragments + special blocks | 65 ms |
| Org-mode, long session | Fragments + special blocks | 80 ms |


  1. Is anyone experiencing something similar ?
  2. Are you aware of any better way to measure typing latency in emacs ?
  3. Is there any thing that can be done from the org-mode/elisp side of 
things ?
  4. Would offloading the fontification to ~tree-sitter~ be a possible 
way forward ?

Regards,

-- 
Sébastien Miquel


[-- Attachment #2: Slow.png --]
[-- Type: image/png, Size: 181899 bytes --]

Re: Typing latency

[Russell Adams]

On Mon, Feb 08, 2021 at 07:24:21PM +0100, Sébastien Miquel wrote:
> I often get some unpleasant latency when editing org-mode
> buffers. My use case combines a lot of special blocks and LaTeX
> fragments, and the fontification seems to slow things down.

My only suggestion is have you looked at disabling flyspell?


------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Typing latency

[Ihor Radchenko]

Sébastien Miquel <sebastien.miquel@posteo.eu> writes:

> AFAIU, the elisp profiler is useless in this case (fontification being
> slow). At least my runs don't seem to report anything relevant.

You can try font-lock-profiler package
(https://github.com/Lindydancer/font-lock-profiler)

Best,
Ihor

Re: Typing latency

[Eric S Fraga]

On Monday,  8 Feb 2021 at 19:24, Sébastien Miquel wrote:
>  1. Is anyone experiencing something similar ?

Yes, particularly since I moved to using exwm as my window manager, so
not necessarily org related.  For instance, I find that having doc-view
buffers active, my whole Emacs shows latency.  This is most visible in
org mode buffers but probably because that's where I spend most of my
time.

What Emacs version are you using?

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.4-213-g49364f

Re: Typing latency

[Sébastien Miquel]

Russell Adams writes:
> My only suggestion is have you looked at disabling flyspell?
Yes, thank you. Flyspell can cause some delay on its own aswell,
but in this situation/configuration it doesn't change the results.

-- 
Sébastien Miquel

Re: Typing latency

[Sébastien Miquel]

Eric S Fraga writes:
> What Emacs version are you using?
This is using the latest ative-comp git branch.

-- 
Sébastien Miquel

Re: Typing latency

[Christian Thäter]

On 2021-02-09 09:38, Eric S Fraga wrote:

>On Monday,  8 Feb 2021 at 19:24, Sébastien Miquel wrote:
>>  1. Is anyone experiencing something similar ?
>
>Yes, particularly since I moved to using exwm as my window manager, so
>not necessarily org related.  For instance, I find that having doc-view
>buffers active, my whole Emacs shows latency.  This is most visible in
>org mode buffers but probably because that's where I spend most of my
>time.
>
>What Emacs version are you using?


Apropos, try out the native-comp Emacs branch.
https://akrl.sdf.org/gccemacs.html That should give a decent
performance boost. 

	Christian 

Re: Typing latency

[Sébastien Miquel]

Ihor Radchenko writes:
> You can try font-lock-profiler package
> (https://github.com/Lindydancer/font-lock-profiler)
This is useful, thank you.

It reports that most of the time is spent in org-do-latex-and-related, 
and some 20% in something related to the priority faces (despite the 
lack of priority cookies).

I doubt that those two account for the whole of the slowdown, but I'll 
be checking it out.

-- 
Sébastien Miquel

Re: Typing latency

[Ihor Radchenko]

Sébastien Miquel <sebastien.miquel@posteo.eu> writes:

> It reports that most of the time is spent in org-do-latex-and-related, 
> and some 20% in something related to the priority faces (despite the 
> lack of priority cookies).

I also had issues with priority faces. It is related to sub-optimal
regexp used to detect priority cookies.

I have the following in my config to speed things up:

(setq org-priority-regexp "^\\*+.*\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)")

For the latex fontification, I never had issues, but you might play with
org-highlight-latex-and-related.

Hope it helps.

Best,
Ihor

Re: Typing latency

[Sébastien Miquel]

Ihor Radchenko writes:
> I have the following in my config to speed things up:
>
> (setq org-priority-regexp "^\\*+.*\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)")
>
> For the latex fontification, I never had issues, but you might play with
> org-highlight-latex-and-related.
>
> Hope it helps.
Thanks, I've set it so aswell. It does speed up the initial 
fontification of the buffer.

I've looked into org-do-latex-and-related, but even replacing it with a 
dud doesn't fix
the delay, which must be caused by the already fontified parts.
I guess the issue must be wih emacs' internals.

-- 
Sébastien Miquel

Re: Typing latency

[Ihor Radchenko]

Sébastien Miquel <sebastien.miquel@posteo.eu> writes:

> the delay, which must be caused by the already fontified parts.
> I guess the issue must be wih emacs' internals.

I had some issues with performance when displaying unicode symbols in
the past for certain (large) fonts. That time, the following helped to
improve Emacs performance drastically:

(setq inhibit-compacting-font-caches t)

That is the only thing I can think about in regard to the Emacs display
engine slowness.

Best,
Ihor

Re: Typing latency

[Timothy]

Ihor Radchenko <yantar92@gmail.com> writes:

> I also had issues with priority faces. It is related to sub-optimal
> regexp used to detect priority cookies.
>
> I have the following in my config to speed things up:
>
> (setq org-priority-regexp "^\\*+.*\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)")

If this doesn't produce regressions, is this worth patching into Org?

--
Timothy.

Re: Typing latency

[Ihor Radchenko]

Timothy <tecosaur@gmail.com> writes:
>> (setq org-priority-regexp "^\\*+.*\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)")
>
> If this doesn't produce regressions, is this worth patching into Org?

I believe so, but I am not sure about regressions. Current master allow
placing priority cookies anywhere, while my version only limits priority
to headline.

Best,
Ihor

Re: Typing latency

[Eric S Fraga]

On Tuesday,  9 Feb 2021 at 23:33, Ihor Radchenko wrote:
> I have the following in my config to speed things up:
>
> (setq org-priority-regexp "^\\*+.*\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)")

Should this not be the default setting?  Reading the documentation,
priority cookies must be in a headline with a TODO item.  The current
default setting is not anchored to headlines so does seem to be both
expensive and not consistent with the documentation.  Your suggestion is
more consistent.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.4-213-g49364f

Re: Typing latency

[Samuel Wales]

i have a note that in org 8.0+ they have to be after the todo kw.
however, i /always/ interpret this to mean after the todo kw position
i.e. after where the todo kw would go if it is not there.  thus "*
#[A] take out garbage" is legit.  "* #[A] NEXT take out garbage" is
wrong.

"* NEXT take out #[A] garbage" feels wrong to me as an assignment of
priority to that entry.  org maint seem to agree with me.


On 2/10/21, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
> On Tuesday,  9 Feb 2021 at 23:33, Ihor Radchenko wrote:
>> I have the following in my config to speed things up:
>>
>> (setq org-priority-regexp "^\\*+.*\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)")
>
> Should this not be the default setting?  Reading the documentation,
> priority cookies must be in a headline with a TODO item.  The current
> default setting is not anchored to headlines so does seem to be both
> expensive and not consistent with the documentation.  Your suggestion is
> more consistent.
>
> --
> : Eric S Fraga via Emacs 28.0.50, Org release_9.4.4-213-g49364f
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: Typing latency

[Eric S Fraga]

On Tuesday,  9 Feb 2021 at 15:23, Christian Thäter wrote:
> Apropos, try out the native-comp Emacs branch.

Thank you.  I've been watching the development of this branch with
interest!  I'm too busy at the moment with work to play around with it
but I will soon.

Emacs, so long as I remember to close certain windows (mostly doc-view
but also teams, when running under exwm, sometimes), is fast enough at
the moment for normal work.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.4-213-g49364f

Re: Typing latency

[Maxim Nikulin]

On 09/02/2021 01:24, Sébastien Miquel wrote:
> I often get some unpleasant latency when editing org-mode
> 
>   1. Is anyone experiencing something similar ?

It is not namely typing latency, but I have noticed lags while moving 
over collapsed headings with "up" key (with "down" it is not so 
apparent) e.g. in overview view. It has happened after linux upgrade, 
emacs version changed from 25.2 to 25.3, system package with org mode 
updated as well. The org file has significant size: 50k lines, 2Mb. I 
have created a LXC container to compare performance with older emacs. It 
is quite strange. With org version from git, emacs version is not really 
important, flyspell mode is irrelevant.

It seems, it depends on loading order. From earlier experiments the 
following line survived in my init.el just below of custom-set-variables:

    (require 'org-protocol)

as soon as I commented it out (independently if it is added to 
org-modules), moving over headers has become significantly more 
responsive. A side effect: with that line, "#+STARTUP: overview" was not 
necessary. Sorry, I do not have any numbers from profiler.

On the other hand, removing of that line does not have so great effect 
on org mode 9.3.1 from the system elpa-org package. Latency is still 
noticeable.

Re: Typing latency

[Ihor Radchenko]

Maxim Nikulin <manikulin@gmail.com> writes:

> It is not namely typing latency, but I have noticed lags while moving 
> over collapsed headings with "up" key (with "down" it is not so 
> apparent) e.g. in overview view. It has happened after linux upgrade, 
> emacs version changed from 25.2 to 25.3, system package with org mode 
> updated as well. The org file has significant size: 50k lines, 2Mb. I 
> have created a LXC container to compare performance with older emacs. It 
> is quite strange. With org version from git, emacs version is not really 
> important, flyspell mode is irrelevant.

You can try feature/org-fold branch aiming to address issues with
performance on large files https://github.com/yantar92/org

Best,
Ihor

Re: Typing latency

[Maxim Nikulin]

On 11/02/2021 23:59, Ihor Radchenko wrote:
> Maxim Nikulin writes:
> 
>> It is not namely typing latency, but I have noticed lags while moving
>> over collapsed headings with "up" key (with "down" it is not so
>> apparent) e.g. in overview view. It has happened after linux upgrade,
>> emacs version changed from 25.2 to 25.3, system package with org mode
>> updated as well. The org file has significant size: 50k lines, 2Mb. I
>> have created a LXC container to compare performance with older emacs. It
>> is quite strange. With org version from git, emacs version is not really
>> important, flyspell mode is irrelevant.
> 
> You can try feature/org-fold branch aiming to address issues with
> performance on large files https://github.com/yantar92/org

Ihor, I have seen the thread with discussion of your branch. This case I 
was surprised the simple move to next or previous currently displayed 
line could be slow. I would expect that emacs has information (in some 
internal structures) what position in the buffer is related to the same 
x coordinate on the previous or next (visual) visible line. Ideally it 
would not depend on how much text is hidden by overlays, properties, 
etc. and should work instantly.

Update to my results. I suspected some problem with loader and I have 
realized what actually had happened. I believed that -L (--directory) 
emacs command line options have precedence and used the following command

     emacs -L ~/org-mode/lisp test-file.org

actually -L processed after init.el file, so (require 'org-protocol) in 
the init file loaded some files from org-9.1.6 (elpa-org system package) 
then remaining files were loaded from git HEAD org version. Even 
org-submit-bug-report did not worked, I tried it to get summary of 
actual configuration. I do not run emacs in such way routinely, so the 
problem was specific to my tests.

My current impression that in both cases of emacs-25.2 and 26.3, with 
org from git moving cursor over collapsed headers works faster than with 
org-9.1.6 or org-9.3.1 from elpa-org package. In all cases 
line-move-visual dominates in profiler reports.

So slow down I have noticed is likely related to growth of the file size 
rather than to system upgrade. I just mostly navigate through the file 
using C-u C-c C-j, so I just was not using cursor keys in overview tree 
display for some time.

So strange observation, due to that I sent previous message, is 
explained: incorrect usage of command line option during tests. My 
complains related to performance is not related to emacs upgrade, org 
master branch HEAD from git works better. Maybe later I will try 
suggestions how to improve performance or I will just split my file into 
smaller parts.

Sorry for the noise.

Re: Typing latency

[Timothy]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Tuesday,  9 Feb 2021 at 23:33, Ihor Radchenko wrote:
>> I have the following in my config to speed things up:
>>
>> (setq org-priority-regexp "^\\*+.*\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)")
>
> Should this not be the default setting?  Reading the documentation,
> priority cookies must be in a headline with a TODO item.  The current
> default setting is not anchored to headlines so does seem to be both
> expensive and not consistent with the documentation.  Your suggestion is
> more consistent.

Did this end up being patched into Org?

--
Timothy

Re: Typing latency

[Eric S Fraga]

On Sunday, 14 Feb 2021 at 12:39, Timothy wrote:
> Did this end up being patched into Org?

I don't think anybody has proposed a patch...

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.4-213-g49364f

Re: Typing latency

[Timothy]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Sunday, 14 Feb 2021 at 12:39, Timothy wrote:
>> Did this end up being patched into Org?
>
> I don't think anybody has proposed a patch...

By patch "patch" I really mean just someone with commit access doing something like:

#+begin_src diff
org-priority-regexp
- (defvar org-priority-regexp ".*?\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)"
+ (defvar org-priority-regexp "^\\*+.*\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)"
#+end_src

Since this thread seems to suggest it's (1) faster (2) more correct.

--
Timothy

Re: Typing latency

[Eric S Fraga]

On Sunday, 14 Feb 2021 at 20:45, Timothy wrote:
> By patch "patch" I really mean just someone with commit access doing
> something like:

Hi Timothy,

Sure but the best way is if you prepare a patch with a commit message
that describes the change and post it here.  Instructions on Worg on how
to prepare a patch.  I'm sure somebody will get around to applying it
unless there are issues raised.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.4-213-g49364f

Re: Typing latency

[Sébastien Miquel]

Timothy writes:
> #+begin_src diff
> org-priority-regexp
> - (defvar org-priority-regexp ".*?\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)"
> + (defvar org-priority-regexp "^\\*+.*\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)"
> #+end_src
>
> Since this thread seems to suggest it's (1) faster (2) more correct.
If anyone's considering writing a patch, note that this breaks
~org-get-priority~ and ~org-agenda-fontify-priorities~ (used in the
agenda). Best to keep both regexps around.

In a 1k lines org file, it reduces the time to fontify the whole
buffer by 0.1 second. I do not know how much of a change this makes
wrt to typing latency.

-- 
Sébastien Miquel

[PATCH] ~org-font-lock-add-priority-faces~: ensure priority cookies are in a headline

[Sébastien Miquel]

[-- Attachment #1: Type: text/plain, Size: 273 bytes --]

Priority cookies are always in a headline.

The attached patch speeds up fontification of a 1k lines buffer by 0.1
second.

Note that the variable org-priority-regexp can't be modified since
it is used in the agenda and in org-get-priority.

Regards,

--
Sébastien Miquel

[-- Attachment #2: 0001-org.el-org-font-lock-add-priority-faces-Speed-up-reg.patch --]
[-- Type: text/x-patch, Size: 966 bytes --]

From a348a3834b79608a20bfd4e28815ae3995c7eb5a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?S=C3=A9bastien=20Miquel?= <sebastien.miquel@posteo.eu>
Date: Fri, 26 Feb 2021 18:02:32 +0100
Subject: [PATCH] org.el (org-font-lock-add-priority-faces): Speed up regexp

* org.el (org-font-lock-add-priority-faces): Speed up regexp.

Only fontify priority cookies in headlines.

TINYCHANGE
---
 lisp/org.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/org.el b/lisp/org.el
index 00596564f..8c976213d 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -5855,7 +5855,7 @@ If TAG is a number, get the corresponding match group."
 
 (defun org-font-lock-add-priority-faces (limit)
   "Add the special priority faces."
-  (while (re-search-forward org-priority-regexp limit t)
+  (while (re-search-forward (concat "^\\*+" org-priority-regexp) limit t)
     (let ((beg (match-beginning 1))
 	  (end (1+ (match-end 2))))
       (add-face-text-property
-- 
2.30.1

Re: [PATCH] ~org-font-lock-add-priority-faces~: ensure priority cookies are in a headline

[Kyle Meyer]

Sébastien Miquel writes:

> Priority cookies are always in a headline.
>
> The attached patch speeds up fontification of a 1k lines buffer by 0.1
> second.

Thank.  Pushed (a03b4656c), adding this...

> Note that the variable org-priority-regexp can't be modified since
> it is used in the agenda and in org-get-priority.

... bit to the commit message, and also adding a trailer to credit Ihor
for the initial suggestion.