bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Eli Zaretskii <eliz@gnu.org>]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: 51766@debbugs.gnu.org
> Date: Sat, 13 Nov 2021 17:10:11 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > "Such change" being what exactly? the situation where
> > buffer-chars-modified-tick changes between post-command-hook and the
> > following pre-command-hook? or something else?
> 
> The former.
> 
> > So what exactly is the problem with these hooks when non-latin input
> > methods are used?  Or what am I missing?
> 
> There is no problem with the hooks in your example. However, consider
> the following:
> 
> (let ((inhibit-modification-hooks t))
>   (insert "Insertion that will never trigger before/after-change-functions"))

So the problem is that Org uses the modification hooks as the primary
mechanism (with which quail presents no problem), and
buffer-chars-modified-tick as the fallback, and that when some code
inhibits the modification hooks, then the primary mechanism cannot
work and quail breaks the fallback?

> The quail's insertion+deletion itself is not a problem for Org cache -
> it does not really alter the buffer text and cannot break the cache. The
> problem is that it cannot be distinguished from the first example - both
> cases will trigger buffer-chars-modified-tick increase.

You didn't answer my question regarding buffer-modified-tick: it can
explain to Org why buffer-chars-modified-tick jumped unexpectedly, and
thus (hopefully) resolve this situation.  If that helps, you could
perhaps turn the table and use buffer-chars-modified-tick is the
primary method of discovering changes, not as fallback.

> > Perhaps Org developers should ask for infrastructure changes that will
> > allow Org to maintain such a cache reliably and not too expensively?
> > It sounds like Org currently applies all kinds of heuristics based on
> > assumptions about how the internals work and using hooks and features
> > that were never designed to support this kind of caching.  Jumping
> > through hoops in Lisp trying to implement something that might be much
> > easier or even trivial in C is not the best way of getting such jobs
> > done.
> >
> > So perhaps someone could describe on emacs-devel what does Org need to
> > maintain this cache, and we could then see how to provide those
> > features to Org.
> 
> I am one of the Org developers.
> 
> The only assumption I had it that Emacs does not frequently change
> buffer text without triggering modification hooks. Clearly, the
> assumption was wrong.

I meant the assumptions about what buffer-chars-modified-tick does and
what its value means.

> Ideally, a way to track _all_ buffer modifications regardless of
> inhibit-modification-hooks would be useful.

But Org is not interested in just any moidification, AFAIU.  It is
only interested in modifications that change the buffer text.  Isn't
that true?  Or what else is Org interested in for this purpose.

> Alternatively, Emacs could support language parsers.

I meant support on the low level, where changes to buffer text are
considered and indicated.  As I indicate below, the integration of
tree-sitter simply uses the existing change indications, so I'm not
sure how would a parser support help you in this matter.

> Org cache
> implements editing syntax tree generated by Org element parser. It is
> very similar to what tree-sitter editing API does: https://tree-sitter.github.io/tree-sitter/using-parsers#editing
> 
> Native support for storing, modifying, and querying syntax trees using
> efficient data structures could be a great addition to Emacs from Org's
> perspective. Though it is not an easy feature to implement.
> 
> AFAIR, something similar to my last suggestion has been already
> proposed: tree-sitter support. I can also propose the first idea about
> reliable buffer change tracking if you think that it is something
> reasonable.

The Emacs code related to tree-sitter already uses change indications,
and AFAIR didn't require any changes to the existing infrastructure.
I wonder why Org cannot settle with what we have, if your needs are
similar enough to those of tree-sitter.