C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Max Nikulin]

Another case of

Warning (emacs): org-element--cache: Unregistered buffer modifications 
detected. Resetting.
If this warning appears regularly, please report it to Org mode mailing 
list (M-x org-submit-bug-report).
The buffer is: c-star.org
  Current command: nil
  Backtrace:
nil

Org main, Emacs-26.3

c-star.org file:
---- >8 ----
Ф Put cursor on this line and hit =C-c C-*= to make it a heading
---- 8< ----

Notice a Cyrillic letter that causes a problem with `org-ctrl-c-star'.

Re: C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> Org main, Emacs-26.3
>
> c-star.org file:
> ---- >8 ----
> Ф Put cursor on this line and hit =C-c C-*= to make it a heading
> ---- 8< ----
>
> Notice a Cyrillic letter that causes a problem with `org-ctrl-c-star'.

Confirmed
However, I can only trigger the warning up to Emacs 27. Emacs 28 and
later has no issue.

Need to investigate further.

Best,
Ihor

Re: C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Ihor Radchenko]

Ihor Radchenko <yantar92@gmail.com> writes:

> Confirmed
> However, I can only trigger the warning up to Emacs 27. Emacs 28 and
> later has no issue.
>
> Need to investigate further.

Fixed via c3f457375. This is another case when Emacs internals
(replace-match) increase buffer-chars-modified-tick. Eventually, this
warning should be disabled and we need to make a patch for Emacs to
trigger cache updates regardless of inhibit-modification-hooks. See
https://lists.gnu.org/archive/html/bug-gnu-emacs/2021-11/msg01069.html

Best,
Ihor

[BUG] C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Max Nikulin]

On 14/11/2021 14:59, Ihor Radchenko wrote:
> Ihor Radchenko writes:
> 
>> Confirmed
>> However, I can only trigger the warning up to Emacs 27. Emacs 28 and
>> later has no issue.
>>
>> Need to investigate further.
> 
> Fixed via c3f457375. This is another case when Emacs internals
> (replace-match) increase buffer-chars-modified-tick. Eventually, this
> warning should be disabled and we need to make a patch for Emacs to
> trigger cache updates regardless of inhibit-modification-hooks. See
> https://lists.gnu.org/archive/html/bug-gnu-emacs/2021-11/msg01069.html

Ihor, I am sorry, but I am still getting the error

LANG=en_US.UTF-8 emacs -Q -L ~/src/emacs/org-mode/lisp/ new.org

C-\ russian-computer RET
A
C-*

Latin "A" to type Cyrillic "Ф"

Org 58db28db04 + Emacs-26.3

Re: [BUG] C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> Ihor, I am sorry, but I am still getting the error
>
> LANG=en_US.UTF-8 emacs -Q -L ~/src/emacs/org-mode/lisp/ new.org
>
> C-\ russian-computer RET
> A
> C-*
>
> Latin "A" to type Cyrillic "Ф"

Apparently, C-* on
> Ф Put cursor on this line and hit =C-c C-*= to make it a heading
and just
> Ф
are producing different buffer-chars-modified-tick/buffer-modified-tick
signatures. I have added this kind of edit and the one from your other
report [1] to the exception list. The commit is 28d47cfb8 (main).

Given the growing number of such "special" cases, I also changed to code
to reset cache all the time but for "self-insert-command" in
russian-computer. Just to be safe. The warning will not be shown though.

[1] https://orgmode.org/list/sn5uaa$ehd$1@ciao.gmane.io

Best,
Ihor

Re: [BUG] C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Max Nikulin]

On 21/11/2021 15:35, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> C-\ russian-computer RET
>> A
>> C-*
>>
>> Latin "A" to type Cyrillic "Ф"
> 
> Apparently, C-* on
>> Ф Put cursor on this line and hit =C-c C-*= to make it a heading
> and just
>> Ф
> are producing different buffer-chars-modified-tick/buffer-modified-tick
> signatures. I have added this kind of edit and the one from your other
> report [1] to the exception list. The commit is 28d47cfb8 (main).

Ihor, thank you for your work related to such issues. I had a hope to 
thank you for the fix, but I faced a warning again in a bit modified 
scenario. This time it is soft indent mode.

---- >8 ----
#+startup: indent
Do not forget to activate settings above by =C-c C-c=

* H
Try =C-c C-*= on this line
---- 8< ----

Warning (org-element-cache): org-element--cache: Unregistered buffer 
modifications detected. Resetting.
If this warning appears regularly, please report it to Org mode mailing 
list (M-x org-submit-bug-report).
The buffer is: new.org
  Current command: (nil 312 320)
  Chars modified: 312
  Buffer modified: 320
  Backtrace:
nil

First letter of new heading must be a capital one, though it can be 
Latin. Converting top-level "H" heading by C-c C-* does not cause such 
warning.

> Given the growing number of such "special" cases, I also changed to code
> to reset cache all the time but for "self-insert-command" in
> russian-computer. Just to be safe. The warning will not be shown though.
> 
> [1] https://orgmode.org/list/sn5uaa$ehd$1@ciao.gmane.io

I am not an active user of main branch (I was merely hunting for another 
bug), so I can not estimate performance penalty for large files due to 
continuous cache resetting. I do not follow emacs-devel mail list last 
weeks. Have you managed to negotiate with Eli concerning changes 
required in Emacs code? I mean some followups of

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51766
#51766 29.0.50; Return value of buffer-chars-modified-tick changes when 
buffer text is not yet changed before inserting a character for 
non-latin input methods

Re: [BUG] C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> Ihor, thank you for your work related to such issues. I had a hope to 
> thank you for the fix, but I faced a warning again in a bit modified 
> scenario. This time it is soft indent mode.
> ...
> First letter of new heading must be a capital one, though it can be 
> Latin. Converting top-level "H" heading by C-c C-* does not cause such 
> warning.

Well... I added yet another exception on main. Note that this special
case is also just in older Emacs versions.

> I am not an active user of main branch (I was merely hunting for another 
> bug), so I can not estimate performance penalty for large files due to 
> continuous cache resetting. I do not follow emacs-devel mail list last 
> weeks. Have you managed to negotiate with Eli concerning changes 
> required in Emacs code? I mean some followups of ...

The conclusion from that discussion is that someone needs to prepare
upstream patch for Emacs. I may do it in future, but it does not feel
like high priority because fixes for Emacs master will not solve the
problem in older Emacs versions. (and I secretly hope that this kind of
patch will be implemented by someone else as a part of tree-sitter
integration).

As for the performance, the last series of special cases you reported
(thanks again for doing this!) should not happen too frequently. I
cannot imagine users spamming C-c C-* all the time. The most problematic
is the case triggered by self-insert-command, but it will not trigger
cache reset. The worst impact we can see with C-c C-* issue is ~1.5-2x
slower agenda (or other full-text search queries) if cache is reset
between agenda updates. It is still much better than not using cache.

Actually, growing cache too much makes Emacs garbage collector perform
pretty poorly (at least when you have a file so large that cache has
300-400k elements). Reducing cache size from time to time may be even
beneficial for performance. I do not know the realistic cutoff values
though. We may want to implement access frequency-based cache element
removal.

Best,
Ihor

Re: [BUG] C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Max Nikulin]

On 30/11/2021 19:54, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> Ihor, thank you for your work related to such issues. I had a hope to
>> thank you for the fix, but I faced a warning again in a bit modified
>> scenario. This time it is soft indent mode.
>> ...
>> First letter of new heading must be a capital one, though it can be
>> Latin. Converting top-level "H" heading by C-c C-* does not cause such
>> warning.
> 
> Well... I added yet another exception on main. Note that this special
> case is also just in older Emacs versions.

Ihor, have you pushed the change? I still can reproduce the issue with 
Emacs-26.3

Next Ubuntu LTS is expected in April, upgrades will be enabled about 
September. As fallback (for the case of some intrusive changes that 
might be added by Canonical) I consider Debian and it has ~2 years 
release cycle as well.

> (and I secretly hope that this kind of
> patch will be implemented by someone else as a part of tree-sitter
> integration).

A piece of friendly trolling: it would be tree-sitter module for Org syntax.

> The most problematic
> is the case triggered by self-insert-command, but it will not trigger
> cache reset.

I mean namely this case of just typing text somewhere in a large file.

Re: [BUG] C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> Well... I added yet another exception on main. Note that this special
>> case is also just in older Emacs versions.
>
> Ihor, have you pushed the change? I still can reproduce the issue with 
> Emacs-26.3

Oops. I fixed Emacs 27, but apparently not Emacs 26. Should be fixed
now.

>> (and I secretly hope that this kind of
>> patch will be implemented by someone else as a part of tree-sitter
>> integration).
>
> A piece of friendly trolling: it would be tree-sitter module for Org syntax.

tree-sitter is comparable with org-element. org-element
parser is fairly fast and also uses incremental parsing ;)

tree-sitter vs. org-element on 15M Org file
org-element-parse-buffer
(16.090262757 1 0.7365683609999962)

org-element-parse-buffer 'element granularity
(7.688000744 0 0.0)
8sec

tree-sitter via https://github.com/milisims/tree-sitter-org
parsed down to 58% of the buffer in 5.3sec and exited with error
extrapolates to ~9sec

Racket's brack via https://github.com/tgbugs/laundry
failed to finish parsing in reasonable time. Cancelled at 10m11.436s

Clojure parser via https://github.com/200ok-ch/org-parser
failed to finish parsing with java.lang.OutOfMemoryError: GC overhead limit exceeded
Running time 8m28.078s

So, tree-sitter may be faster, but not that much faster and we will have
communication overheads when using tree-sitter module.

>> The most problematic
>> is the case triggered by self-insert-command, but it will not trigger
>> cache reset.
>
> I mean namely this case of just typing text somewhere in a large file.

My data is for 15M file. Indeed, things will be worse for even larger
files, but let's hope that people's Org files are not yet that large and
we can wait until Emacs 28 is released.

Best,
Ihor

Re: [BUG] C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Max Nikulin]

On 02/12/2021 08:48, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>>> Well... I added yet another exception on main. Note that this special
>>> case is also just in older Emacs versions.
>>
>> Ihor, have you pushed the change? I still can reproduce the issue with
>> Emacs-26.3
> 
> Oops. I fixed Emacs 27, but apparently not Emacs 26. Should be fixed
> now.

Unfortunately currently it fails in Emacs-26.3 event without "#+startup: 
indent":

Warning (org-element-cache): org-element--cache: Unregistered buffer 
modifications detected. Resetting.
If this warning appears regularly, please report it to Org mode mailing 
list (M-x org-submit-bug-report).
The buffer is: new.org
  Current command: (nil 163 167)
  Chars modified: 163
  Buffer modified: 167
  Backtrace:
nil

Org: 521c534101

> tree-sitter is comparable with org-element. org-element
> parser is fairly fast and also uses incremental parsing ;)
> 
> tree-sitter vs. org-element on 15M Org file
> org-element-parse-buffer
> (16.090262757 1 0.7365683609999962)

I hope, Timothy will fix bugs in his new fast parser. Anyway thank you 
for sharing these results. I have not looked closely at various parsers, 
I only created a rudimentary one that understands just headings and 
links to call it from my browser extension (no more than proof of concept).

Re: [BUG] C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> Unfortunately currently it fails in Emacs-26.3 event without "#+startup: 
> indent":
>
> Warning (org-element-cache): org-element--cache: Unregistered buffer 
> modifications detected. Resetting.
> If this warning appears regularly, please report it to Org mode mailing 
> list (M-x org-submit-bug-report).
> The buffer is: new.org
>   Current command: (nil 163 167)
>   Chars modified: 163
>   Buffer modified: 167
>   Backtrace:
> nil

I am unable to reproduce on my side also using Emacs 26.3. Are you sure
that you are running a clean Emacs?

Best,
Ihor

Re: [BUG] C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Max Nikulin]

On 03/12/2021 11:36, Ihor Radchenko wrote:
> Max Nikulin <manikulin@gmail.com> writes:
> 
>> Unfortunately currently it fails in Emacs-26.3 event without "#+startup:
>> indent":
>>
>> Warning (org-element-cache): org-element--cache: Unregistered buffer
>> modifications detected. Resetting.
>> If this warning appears regularly, please report it to Org mode mailing
>> list (M-x org-submit-bug-report).
>> The buffer is: new.org
>>    Current command: (nil 163 167)
>>    Chars modified: 163
>>    Buffer modified: 167
>>    Backtrace:
>> nil
> 
> I am unable to reproduce on my side also using Emacs 26.3. Are you sure
> that you are running a clean Emacs?

make clean
make autoloads
git log -1 --format=short
commit 8ca254696b002f5079e31ac12a1ad87c003f673d (HEAD -> main, origin/main)
Merge: 521c53410 0acd2fed2
Author: Kyle Meyer <kyle@kyleam.com>

     Merge branch 'bugfix'

git diff HEAD
# nothing
LANG=en_US.UTF-8 emacs -Q -L ~/src/org-mode/lisp/ ~/examples/org/new2.org &

---- >8 ----
* A
B
---- 8< ----

C-c C-* while cursor is at the end of second line

Warning (org-element-cache): org-element--cache: Unregistered buffer 
modifications detected. Resetting.
If this warning appears regularly, please report it to Org mode mailing 
list (M-x org-submit-bug-report).
The buffer is: new2.org
  Current command: (nil 26 30)
  Chars modified: 26
  Buffer modified: 30
  Backtrace:
nil

Re: [BUG] C-c C-* causes "org-element--cache: Unregistered buffer modifications detected."

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> ---- >8 ----
> * A
> B
> ---- 8< ----
>
> C-c C-* while cursor is at the end of second line

Thanks! I was able to reproduce. And it is bad news.

>   Current command: (nil 26 30)
>   Chars modified: 26
>   Buffer modified: 30

This has the same footprint with
(let ((inhibit-modification-hooks t))
     (insert "* This is going to break the cache\n"))

So, I pushed a change that suppresses the warning completely for
Emacs <28 and silently resets the cache. I see no other way around it.

Fortunately, all the problems you reported in this thread do not happen
in newer Emacs versions.

Best,
Ihor