bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko <yantar92@gmail.com>]

Eli Zaretskii <eliz@gnu.org> writes:

>> (let ((inhibit-modification-hooks t))
>>   (insert "Insertion that will never trigger before/after-change-functions"))
>
> So the problem is that Org uses the modification hooks as the primary
> mechanism (with which quail presents no problem), and
> buffer-chars-modified-tick as the fallback, and that when some code
> inhibits the modification hooks, then the primary mechanism cannot
> work and quail breaks the fallback?

Not exactly. Org uses modification hooks as the only mechanism to
process buffer changes because Org needs to know the region where the
buffer text changes. buffer-chars-modified-tick is used for error
detection - when buffer text is changed, but Org modification hooks are
not called for some reason. quail triggers false positives during the error
detection.

>> The quail's insertion+deletion itself is not a problem for Org cache -
>> it does not really alter the buffer text and cannot break the cache. The
>> problem is that it cannot be distinguished from the first example - both
>> cases will trigger buffer-chars-modified-tick increase.
>
> You didn't answer my question regarding buffer-modified-tick: it can
> explain to Org why buffer-chars-modified-tick jumped unexpectedly, and
> thus (hopefully) resolve this situation.  If that helps, you could
> perhaps turn the table and use buffer-chars-modified-tick is the
> primary method of discovering changes, not as fallback.
> ...
>> The only assumption I had it that Emacs does not frequently change
>> buffer text without triggering modification hooks. Clearly, the
>> assumption was wrong.
>
> I meant the assumptions about what buffer-chars-modified-tick does and
> what its value means.

It seems that we have some misunderstanding here. Org does not care
about the value of buffer-chars-modified-tick - just whether
buffer-chars-modified-tick is changed or not (see the above).

Even if Org used the value of buffer-chars-modified-tick, it would not
be useful. There is no information about buffer region where the edits
happened if we just look at buffer-chars-modified-tick. AFAIK, only
after-change-functions have access to the changed region bounds.

>> Ideally, a way to track _all_ buffer modifications regardless of
>> inhibit-modification-hooks would be useful.
>
> But Org is not interested in just any moidification, AFAIU.  It is
> only interested in modifications that change the buffer text.  Isn't
> that true?  Or what else is Org interested in for this purpose.

You are right. Org is interested in modifications that change buffer
text. Also, Org is interested to be not affected by
inhibit-modification-hooks. Making sure that Org modification hooks run
for every modification is automatically making sure that no
modifications that do change buffer text are missed. I thought that it
can be the simplest approach to fix the issue.

>> Alternatively, Emacs could support language parsers.
>
> I meant support on the low level, where changes to buffer text are
> considered and indicated.  As I indicate below, the integration of
> tree-sitter simply uses the existing change indications, so I'm not
> sure how would a parser support help you in this matter.

I probably went too far with my suggestion here. I meant not only
handling changes, but also adding API for working with syntax trees using
C-level functions. It is out of scope of this bug report.

> The Emacs code related to tree-sitter already uses change indications,
> and AFAIR didn't require any changes to the existing infrastructure.
> I wonder why Org cannot settle with what we have, if your needs are
> similar enough to those of tree-sitter.

I just checked https://github.com/casouri/emacs/blob/ts/src/insdel.c
That ts Emacs branch directly modifies C-level Emacs buffer editing
primitives (see the calls to ts_record_change). So, it is not affected
by inhibit-modification-hooks.

Best,
Ihor