bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko <yantar92@gmail.com>]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> (let ((inhibit-modification-hooks t))
>>   (insert "Insertion that will never trigger before/after-change-functions"))
>
> This is a severely broken piece of code.  I don't think anyone should
> try and handle this in any other way than by screaming bloody murder
> when it detects the consequences.

>> (defun org-element--before-change-function (...)
>>  (unless (eq org-element-chars-modified-tick (buffer-chars-modified-tick))
>>      ;; Buffer has been changed without calling after-change-function
>>      ;; and we have no way to determine which part of buffer has been changed.
>>  ))
>
> So this `unless` is intended to detect the case where we should scream
> bloody murder, right?
>
> Why do you need it?  AFAICT it should only be needed for debugging
> purposes until the offender is found and shamed publicly.
> [ I have a weird feeling that I might be one of the offenders.  ]
>
>> Ideally, a way to track _all_ buffer modifications regardless of
>> inhibit-modification-hooks would be useful.

Well. This was also my assumption. I initially implemented the checking
code to detect internal errors in Org.

Then we received 12 bug reports on this. The offenders were identified
in some cases. They were:
- polymode, valign, and ox-hugo
- Doom (unrelated to this particular issue, but Doom is let-binding major-mode...)
- Emacs quail and also replace-match in some cases (because of
  false-positives caused by changing buffer-chars-modified-tick)

ox-hugo and polymode fixed the issue.

valign relies on disabling modification hooks because it is otherwise
difficult to figure out pixel width of a string in current buffer:
https://github.com/casouri/valign/issues/30

In other cases, the offenders were hard to identify because of remote
debugging difficulty.

What I want to say is that the problem is more widespread than you may
think. And the consequences of missed modifications can cause very real
data corruption in Org files (some editing Org commands are relying on
cache being valid; if syntax boundaries are incorrect, the editing can
mess up the text). It must be avoided at all costs, regardless of the
recommended Elisp programming practices.

On top of the misbehaving third-party code, there is an issue described
in bug#46982. This discussion reminded me about
clone-indirect-buffer-hook, but it is only executed by
`clone-indirect-buffer', not by `make-indirect-buffer'. The latter is
used ubiquitously. See
https://github.com/search?q=make-indirect-buffer&type=code
Again, some unsuspecting users can experience data corruption just
because someone carelessly uses `make-indirect-buffer' in the code.

> I don't think this *can* exist: if we add a mechanism which ignores
> `inhibit-modification-hooks` it will still need some way to ignore some
> changes so we'll need another `inhibit-<foo>` variable to "silence"
> those changes and we'll be back at square one.
>
> I think the better way to proceed is to figure out why/when
> significant changes are made while `inhibit-modification-hooks` is
> non-nil, since that's the origin of your problems, AFAICT.

See the above. There is real-world code doing things that make
Emacs ignore after-change-functions.

Combined with the issue revealed in this bug report, I am left with no
Emacs tools to handle the problematic buffer modifications.

Best,
Ihor