bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko]

According to buffer-chars-modified-tick docstring:
"By comparing the values returned by two individual calls of
buffer-chars-modified-tick, you can tell whether a character change
occurred in that buffer in between these calls"

However, the return value can change even when no visible change is made
to buffer text.

Steps to reproduce:
1. emacs -Q
2. Evaluate the following code:

(defun print-tick-before ()
  (when (eq this-command 'self-insert-command)
  (warn "Tick before: %S" (buffer-chars-modified-tick))))
(defun print-tick-after ()
    (when (eq this-command 'self-insert-command)
  (warn "Tick after: %S" (buffer-chars-modified-tick))))
(add-hook 'pre-command-hook #'print-tick-before)
(add-hook 'post-command-hook #'print-tick-after)

3. Insert a latin symbol ?a twice. The warning buffer will print
something like

Warning (emacs): Tick before: 1698 Disable showing Disable logging
Warning (emacs): Tick after: 1699 Disable showing Disable logging
Warning (emacs): Tick before: 1699 Disable showing Disable logging
Warning (emacs): Tick after: 1702 Disable showing Disable logging

Note that second and third line show the same buffer-chars-modified-tick
value.

4. Change input method (C-\) to russian-computer or i.e. arabic
5. Insert a non-latin symbol ?ф twice. The warning buffer will print
something like

Warning (emacs): Tick before: 1706 Disable showing Disable logging
Warning (emacs): Tick after: 1707 Disable showing Disable logging
Warning (emacs): Tick before: 1711 Disable showing Disable logging
Warning (emacs): Tick after: 1712 Disable showing Disable logging

Note that second and third line _do not_ show the same
buffer-chars-modified-tick value even though buffer text has not been
changed between the two self-insert commands

Expected behaviour: return value of buffer-chars-modified-tick does not
change when no changes in buffer text are made.

This issue causes breakage in latest version of Org. See
https://list.orgmode.org/87sfw2luhj.fsf@localhost/T/#you

Best,
Ihor


In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, cairo version 1.16.0)
 of 2021-10-30 built on localhost
Repository revision: c3499b8ddc357544a58917bfd3846f88caf5d97c
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12013000
System Description: Gentoo/Linux

Configured using:
 'configure --prefix=/usr --build=x86_64-pc-linux-gnu
 --host=x86_64-pc-linux-gnu --mandir=/usr/share/man
 --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc
 --localstatedir=/var/lib --datarootdir=/usr/share
 --disable-silent-rules --docdir=/usr/share/doc/emacs-29.0.9999
 --htmldir=/usr/share/doc/emacs-29.0.9999/html --libdir=/usr/lib64
 --program-suffix=-emacs-29-vcs --includedir=/usr/include/emacs-29-vcs
 --infodir=/usr/share/info/emacs-29-vcs --localstatedir=/var
 --enable-locallisppath=/etc/emacs:/usr/share/emacs/site-lisp
 --without-compress-install --without-hesiod --without-pop
 --with-file-notification=inotify --with-pdumper --enable-acl
 --with-dbus --with-modules --without-gameuser --with-libgmp
 --without-gpm --with-native-compilation --with-json --without-kerberos
 --without-kerberos5 --without-lcms2 --with-xml2 --without-mailutils
 --with-selinux --with-gnutls --without-libsystemd --with-threads
 --with-wide-int --with-zlib --with-sound=oss --with-x --without-ns
 --without-gconf --without-gsettings --without-toolkit-scroll-bars
 --with-gif --with-jpeg --with-png --with-rsvg --with-tiff --with-xpm
 --with-imagemagick --with-xft --with-cairo --with-harfbuzz
 --without-libotf --without-m17n-flt --with-x-toolkit=no
 --with-dumping=pdumper 'CFLAGS=-march=native -pipe -O2' CPPFLAGS=
 'LDFLAGS=-Wl,-O1 -Wl,--as-needed''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS HARFBUZZ IMAGEMAGICK JPEG
JSON LIBSELINUX LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY OLDXMENU
PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF WEBP X11 XDBE XIM XPM ZLIB

Important settings:
  value of $LC_COLLATE: C
  value of $LANG: en_US.utf8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  pdf-occur-global-minor-mode: t
  TeX-PDF-mode: t
  org-edna-mode: t
  eros-mode: t
  which-key-mode: t
  diredfl-global-mode: t
  dired-async-mode: t
  winner-mode: t
  recentf-mode: t
  helm-adaptive-mode: t
  helm-mode: t
  helm--remap-mouse-mode: t
  async-bytecomp-package-mode: t
  eval-sexp-fu-flash-mode: t
  el-patch-use-package-mode: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  shell-dirtrack-mode: t
  unpackaged/magit-log-date-headers-mode: t
  hl-todo-mode: t
  pretty-symbols-mode: t
  company-mode: t
  persistent-scratch-autosave-mode: t
  savehist-mode: t
  boon-mode: t
  boon-local-mode: t
  global-hl-line-mode: t
  global-page-break-lines-mode: t
  page-break-lines-mode: t
  shackle-mode: t
  gcmh-mode: t
  override-global-mode: t
  straight-use-package-mode: t
  straight-package-neutering-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  global-prettify-symbols-mode: t
  prettify-symbols-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  window-divider-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Load-path shadows:
/home/yantar92/.emacs.d/straight/build/helm-org/helm-org hides /home/yantar92/.emacs.d/elpa/helm-org-20210324.1927/helm-org
/home/yantar92/.emacs.d/straight/build/helm-org/helm-org-autoloads hides /home/yantar92/.emacs.d/elpa/helm-org-20210324.1927/helm-org-autoloads
/home/yantar92/.emacs.d/straight/build/helm/helm-x-files hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-x-files
/home/yantar92/.emacs.d/straight/build/helm/helm-utils hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-utils
/home/yantar92/.emacs.d/straight/build/helm/helm-types hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-types
/home/yantar92/.emacs.d/straight/build/helm/helm-tags hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-tags
/home/yantar92/.emacs.d/straight/build/helm/helm-sys hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-sys
/home/yantar92/.emacs.d/straight/build/helm/helm-shell hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-shell
/home/yantar92/.emacs.d/straight/build/helm/helm-semantic hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-semantic
/home/yantar92/.emacs.d/straight/build/helm/helm-ring hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-ring
/home/yantar92/.emacs.d/straight/build/helm/helm-regexp hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-regexp
/home/yantar92/.emacs.d/straight/build/helm/helm-occur hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-occur
/home/yantar92/.emacs.d/straight/build/helm/helm-net hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-net
/home/yantar92/.emacs.d/straight/build/helm/helm-mode hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-mode
/home/yantar92/.emacs.d/straight/build/helm/helm-misc hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-misc
/home/yantar92/.emacs.d/straight/build/helm/helm-man hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-man
/home/yantar92/.emacs.d/straight/build/helm/helm-locate hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-locate
/home/yantar92/.emacs.d/straight/build/helm/helm-info hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-info
/home/yantar92/.emacs.d/straight/build/helm/helm-imenu hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-imenu
/home/yantar92/.emacs.d/straight/build/helm/helm-id-utils hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-id-utils
/home/yantar92/.emacs.d/straight/build/helm/helm-help hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-help
/home/yantar92/.emacs.d/straight/build/helm/helm-grep hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-grep
/home/yantar92/.emacs.d/straight/build/helm/helm-global-bindings hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-global-bindings
/home/yantar92/.emacs.d/straight/build/helm/helm-for-files hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-for-files
/home/yantar92/.emacs.d/straight/build/helm/helm-font hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-font
/home/yantar92/.emacs.d/straight/build/helm/helm-find hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-find
/home/yantar92/.emacs.d/straight/build/helm/helm-files hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-files
/home/yantar92/.emacs.d/straight/build/helm/helm-fd hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-fd
/home/yantar92/.emacs.d/straight/build/helm/helm-external hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-external
/home/yantar92/.emacs.d/straight/build/helm/helm-eval hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-eval
/home/yantar92/.emacs.d/straight/build/helm/helm-eshell hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-eshell
/home/yantar92/.emacs.d/straight/build/helm/helm-epa hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-epa
/home/yantar92/.emacs.d/straight/build/helm/helm-elisp hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-elisp
/home/yantar92/.emacs.d/straight/build/helm/helm-elisp-package hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-elisp-package
/home/yantar92/.emacs.d/straight/build/helm/helm-easymenu hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-easymenu
/home/yantar92/.emacs.d/straight/build/helm/helm-dabbrev hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-dabbrev
/home/yantar92/.emacs.d/straight/build/helm/helm-config hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-config
/home/yantar92/.emacs.d/straight/build/helm/helm-command hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-command
/home/yantar92/.emacs.d/straight/build/helm/helm-comint hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-comint
/home/yantar92/.emacs.d/straight/build/helm/helm-color hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-color
/home/yantar92/.emacs.d/straight/build/helm/helm-buffers hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-buffers
/home/yantar92/.emacs.d/straight/build/helm/helm-bookmark hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-bookmark
/home/yantar92/.emacs.d/straight/build/helm/helm-adaptive hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-adaptive
/home/yantar92/.emacs.d/straight/build/helm/helm-autoloads hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-autoloads
/home/yantar92/.emacs.d/straight/build/helm/helm-pkg hides /home/yantar92/.emacs.d/elpa/helm-20210827.1619/helm-pkg
/home/yantar92/.emacs.d/straight/build/helm-core/helm hides /home/yantar92/.emacs.d/elpa/helm-core-20210822.952/helm
/home/yantar92/.emacs.d/straight/build/helm-core/helm-source hides /home/yantar92/.emacs.d/elpa/helm-core-20210822.952/helm-source
/home/yantar92/.emacs.d/straight/build/helm-core/helm-multi-match hides /home/yantar92/.emacs.d/elpa/helm-core-20210822.952/helm-multi-match
/home/yantar92/.emacs.d/straight/build/helm-core/helm-lib hides /home/yantar92/.emacs.d/elpa/helm-core-20210822.952/helm-lib
/home/yantar92/.emacs.d/straight/build/helm-core/helm-core-autoloads hides /home/yantar92/.emacs.d/elpa/helm-core-20210822.952/helm-core-autoloads
/home/yantar92/.emacs.d/straight/build/helm-core/helm-core-pkg hides /home/yantar92/.emacs.d/elpa/helm-core-20210822.952/helm-core-pkg
/home/yantar92/.emacs.d/straight/build/async/smtpmail-async hides /home/yantar92/.emacs.d/elpa/async-20210823.528/smtpmail-async
/home/yantar92/.emacs.d/straight/build/async/dired-async hides /home/yantar92/.emacs.d/elpa/async-20210823.528/dired-async
/home/yantar92/.emacs.d/straight/build/async/async hides /home/yantar92/.emacs.d/elpa/async-20210823.528/async
/home/yantar92/.emacs.d/straight/build/async/async-bytecomp hides /home/yantar92/.emacs.d/elpa/async-20210823.528/async-bytecomp
/home/yantar92/.emacs.d/straight/build/async/async-autoloads hides /home/yantar92/.emacs.d/elpa/async-20210823.528/async-autoloads
/home/yantar92/.emacs.d/straight/build/org-ql/org-ql hides /home/yantar92/.emacs.d/elpa/org-ql-20210713.233/org-ql
/home/yantar92/.emacs.d/straight/build/org-ql/org-ql-view hides /home/yantar92/.emacs.d/elpa/org-ql-20210713.233/org-ql-view
/home/yantar92/.emacs.d/straight/build/org-ql/org-ql-search hides /home/yantar92/.emacs.d/elpa/org-ql-20210713.233/org-ql-search
/home/yantar92/.emacs.d/straight/build/org-ql/org-ql-autoloads hides /home/yantar92/.emacs.d/elpa/org-ql-20210713.233/org-ql-autoloads
/home/yantar92/.emacs.d/straight/build/f/f hides /home/yantar92/.emacs.d/elpa/f-20210624.1103/f
/home/yantar92/.emacs.d/straight/build/f/f-autoloads hides /home/yantar92/.emacs.d/elpa/f-20210624.1103/f-autoloads
/home/yantar92/.emacs.d/straight/build/org-super-agenda/org-super-agenda hides /home/yantar92/.emacs.d/elpa/org-super-agenda-20201211.918/org-super-agenda
/home/yantar92/.emacs.d/straight/build/org-super-agenda/org-super-agenda-autoloads hides /home/yantar92/.emacs.d/elpa/org-super-agenda-20201211.918/org-super-agenda-autoloads
/home/yantar92/.emacs.d/straight/build/ht/ht hides /home/yantar92/.emacs.d/elpa/ht-20210119.741/ht
/home/yantar92/.emacs.d/straight/build/ht/ht-autoloads hides /home/yantar92/.emacs.d/elpa/ht-20210119.741/ht-autoloads
/home/yantar92/.emacs.d/straight/build/ov/ov hides /home/yantar92/.emacs.d/elpa/ov-20200326.1042/ov
/home/yantar92/.emacs.d/straight/build/ov/ov-autoloads hides /home/yantar92/.emacs.d/elpa/ov-20200326.1042/ov-autoloads
/home/yantar92/.emacs.d/straight/build/peg/peg hides /home/yantar92/.emacs.d/elpa/peg-1.0/peg
/home/yantar92/.emacs.d/straight/build/peg/peg-tests hides /home/yantar92/.emacs.d/elpa/peg-1.0/peg-tests
/home/yantar92/.emacs.d/straight/build/peg/peg-autoloads hides /home/yantar92/.emacs.d/elpa/peg-1.0/peg-autoloads
/home/yantar92/.emacs.d/straight/build/popup/popup hides /home/yantar92/.emacs.d/elpa/popup-20210625.400/popup
/home/yantar92/.emacs.d/straight/build/popup/popup-autoloads hides /home/yantar92/.emacs.d/elpa/popup-20210625.400/popup-autoloads
/home/yantar92/.emacs.d/straight/build/transient/transient hides /home/yantar92/.emacs.d/elpa/transient-20210819.2118/transient
/home/yantar92/.emacs.d/straight/build/transient/transient-autoloads hides /home/yantar92/.emacs.d/elpa/transient-20210819.2118/transient-autoloads
/home/yantar92/.emacs.d/straight/build/ts/ts hides /home/yantar92/.emacs.d/elpa/ts-20210813.1617/ts
/home/yantar92/.emacs.d/straight/build/ts/ts-autoloads hides /home/yantar92/.emacs.d/elpa/ts-20210813.1617/ts-autoloads
/home/yantar92/.emacs.d/straight/build/s/s hides /home/yantar92/.emacs.d/elpa/s-20210616.619/s
/home/yantar92/.emacs.d/straight/build/s/s-autoloads hides /home/yantar92/.emacs.d/elpa/s-20210616.619/s-autoloads
/home/yantar92/.emacs.d/straight/build/dash/dash hides /home/yantar92/.emacs.d/elpa/dash-20210826.1149/dash
/home/yantar92/.emacs.d/straight/build/dash/dash-autoloads hides /home/yantar92/.emacs.d/elpa/dash-20210826.1149/dash-autoloads
/usr/share/emacs/site-lisp/cmake-mode hides /usr/share/emacs/site-lisp/cmake/cmake-mode
/home/yantar92/.emacs.d/straight/build/dash/dash hides /usr/share/emacs/site-lisp/dash/dash
/usr/share/emacs/site-lisp/desktop-entry-mode hides /usr/share/emacs/site-lisp/desktop-file-utils/desktop-entry-mode
/home/yantar92/.emacs.d/straight/build/f/f hides /usr/share/emacs/site-lisp/f/f
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-lib hides /usr/share/emacs/site-lisp/notmuch/notmuch-lib
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-compat hides /usr/share/emacs/site-lisp/notmuch/notmuch-compat
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-parser hides /usr/share/emacs/site-lisp/notmuch/notmuch-parser
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch hides /usr/share/emacs/site-lisp/notmuch/notmuch
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-query hides /usr/share/emacs/site-lisp/notmuch/notmuch-query
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-show hides /usr/share/emacs/site-lisp/notmuch/notmuch-show
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-tree hides /usr/share/emacs/site-lisp/notmuch/notmuch-tree
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-wash hides /usr/share/emacs/site-lisp/notmuch/notmuch-wash
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-hello hides /usr/share/emacs/site-lisp/notmuch/notmuch-hello
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-mua hides /usr/share/emacs/site-lisp/notmuch/notmuch-mua
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-address hides /usr/share/emacs/site-lisp/notmuch/notmuch-address
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-maildir-fcc hides /usr/share/emacs/site-lisp/notmuch/notmuch-maildir-fcc
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-message hides /usr/share/emacs/site-lisp/notmuch/notmuch-message
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-crypto hides /usr/share/emacs/site-lisp/notmuch/notmuch-crypto
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-tag hides /usr/share/emacs/site-lisp/notmuch/notmuch-tag
/home/yantar92/.emacs.d/straight/build/notmuch/coolj hides /usr/share/emacs/site-lisp/notmuch/coolj
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-print hides /usr/share/emacs/site-lisp/notmuch/notmuch-print
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-jump hides /usr/share/emacs/site-lisp/notmuch/notmuch-jump
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-company hides /usr/share/emacs/site-lisp/notmuch/notmuch-company
/home/yantar92/.emacs.d/straight/build/notmuch/notmuch-draft hides /usr/share/emacs/site-lisp/notmuch/notmuch-draft
/home/yantar92/.emacs.d/straight/build/s/s hides /usr/share/emacs/site-lisp/s/s
/home/yantar92/.emacs.d/straight/build/with-editor/with-editor hides /usr/share/emacs/site-lisp/with-editor/with-editor
/home/yantar92/.emacs.d/straight/build/transient/transient hides /usr/share/emacs/29.0.50/lisp/transient
/home/yantar92/.emacs.d/straight/build/org/ob-C hides /usr/share/emacs/29.0.50/lisp/org/ob-C
/home/yantar92/.emacs.d/straight/build/org/ob-R hides /usr/share/emacs/29.0.50/lisp/org/ob-R
/home/yantar92/.emacs.d/straight/build/org/ob-awk hides /usr/share/emacs/29.0.50/lisp/org/ob-awk
/home/yantar92/.emacs.d/straight/build/org/ob-calc hides /usr/share/emacs/29.0.50/lisp/org/ob-calc
/home/yantar92/.emacs.d/straight/build/org/ob-clojure hides /usr/share/emacs/29.0.50/lisp/org/ob-clojure
/home/yantar92/.emacs.d/straight/build/org/ob-comint hides /usr/share/emacs/29.0.50/lisp/org/ob-comint
/home/yantar92/.emacs.d/straight/build/org/ob-core hides /usr/share/emacs/29.0.50/lisp/org/ob-core
/home/yantar92/.emacs.d/straight/build/org/ob-css hides /usr/share/emacs/29.0.50/lisp/org/ob-css
/home/yantar92/.emacs.d/straight/build/org/ob-ditaa hides /usr/share/emacs/29.0.50/lisp/org/ob-ditaa
/home/yantar92/.emacs.d/straight/build/org/ob-dot hides /usr/share/emacs/29.0.50/lisp/org/ob-dot
/home/yantar92/.emacs.d/straight/build/org/ob-emacs-lisp hides /usr/share/emacs/29.0.50/lisp/org/ob-emacs-lisp
/home/yantar92/.emacs.d/straight/build/org/ob-eshell hides /usr/share/emacs/29.0.50/lisp/org/ob-eshell
/home/yantar92/.emacs.d/straight/build/org/ob-eval hides /usr/share/emacs/29.0.50/lisp/org/ob-eval
/home/yantar92/.emacs.d/straight/build/org/ob-exp hides /usr/share/emacs/29.0.50/lisp/org/ob-exp
/home/yantar92/.emacs.d/straight/build/org/ob-forth hides /usr/share/emacs/29.0.50/lisp/org/ob-forth
/home/yantar92/.emacs.d/straight/build/org/ob-fortran hides /usr/share/emacs/29.0.50/lisp/org/ob-fortran
/home/yantar92/.emacs.d/straight/build/org/ob-gnuplot hides /usr/share/emacs/29.0.50/lisp/org/ob-gnuplot
/home/yantar92/.emacs.d/straight/build/org/ob-groovy hides /usr/share/emacs/29.0.50/lisp/org/ob-groovy
/home/yantar92/.emacs.d/straight/build/org/ob-haskell hides /usr/share/emacs/29.0.50/lisp/org/ob-haskell
/home/yantar92/.emacs.d/straight/build/org/ob-java hides /usr/share/emacs/29.0.50/lisp/org/ob-java
/home/yantar92/.emacs.d/straight/build/org/ob-js hides /usr/share/emacs/29.0.50/lisp/org/ob-js
/home/yantar92/.emacs.d/straight/build/org/ob-julia hides /usr/share/emacs/29.0.50/lisp/org/ob-julia
/home/yantar92/.emacs.d/straight/build/org/ob-latex hides /usr/share/emacs/29.0.50/lisp/org/ob-latex
/home/yantar92/.emacs.d/straight/build/org/ob-lilypond hides /usr/share/emacs/29.0.50/lisp/org/ob-lilypond
/home/yantar92/.emacs.d/straight/build/org/ob-lisp hides /usr/share/emacs/29.0.50/lisp/org/ob-lisp
/home/yantar92/.emacs.d/straight/build/org/ob-lob hides /usr/share/emacs/29.0.50/lisp/org/ob-lob
/home/yantar92/.emacs.d/straight/build/org/ob-lua hides /usr/share/emacs/29.0.50/lisp/org/ob-lua
/home/yantar92/.emacs.d/straight/build/org/ob-makefile hides /usr/share/emacs/29.0.50/lisp/org/ob-makefile
/home/yantar92/.emacs.d/straight/build/org/ob-matlab hides /usr/share/emacs/29.0.50/lisp/org/ob-matlab
/home/yantar92/.emacs.d/straight/build/org/ob-maxima hides /usr/share/emacs/29.0.50/lisp/org/ob-maxima
/home/yantar92/.emacs.d/straight/build/org/ob-ocaml hides /usr/share/emacs/29.0.50/lisp/org/ob-ocaml
/home/yantar92/.emacs.d/straight/build/org/ob-octave hides /usr/share/emacs/29.0.50/lisp/org/ob-octave
/home/yantar92/.emacs.d/straight/build/org/ob-org hides /usr/share/emacs/29.0.50/lisp/org/ob-org
/home/yantar92/.emacs.d/straight/build/org/ob-perl hides /usr/share/emacs/29.0.50/lisp/org/ob-perl
/home/yantar92/.emacs.d/straight/build/org/ob-plantuml hides /usr/share/emacs/29.0.50/lisp/org/ob-plantuml
/home/yantar92/.emacs.d/straight/build/org/ob-processing hides /usr/share/emacs/29.0.50/lisp/org/ob-processing
/home/yantar92/.emacs.d/straight/build/org/ob-python hides /usr/share/emacs/29.0.50/lisp/org/ob-python
/home/yantar92/.emacs.d/straight/build/org/ob-ref hides /usr/share/emacs/29.0.50/lisp/org/ob-ref
/home/yantar92/.emacs.d/straight/build/org/ob-ruby hides /usr/share/emacs/29.0.50/lisp/org/ob-ruby
/home/yantar92/.emacs.d/straight/build/org/ob-sass hides /usr/share/emacs/29.0.50/lisp/org/ob-sass
/home/yantar92/.emacs.d/straight/build/org/ob-scheme hides /usr/share/emacs/29.0.50/lisp/org/ob-scheme
/home/yantar92/.emacs.d/straight/build/org/ob-screen hides /usr/share/emacs/29.0.50/lisp/org/ob-screen
/home/yantar92/.emacs.d/straight/build/org/ob-sed hides /usr/share/emacs/29.0.50/lisp/org/ob-sed
/home/yantar92/.emacs.d/straight/build/org/ob-shell hides /usr/share/emacs/29.0.50/lisp/org/ob-shell
/home/yantar92/.emacs.d/straight/build/org/ob-sql hides /usr/share/emacs/29.0.50/lisp/org/ob-sql
/home/yantar92/.emacs.d/straight/build/org/ob-sqlite hides /usr/share/emacs/29.0.50/lisp/org/ob-sqlite
/home/yantar92/.emacs.d/straight/build/org/ob-table hides /usr/share/emacs/29.0.50/lisp/org/ob-table
/home/yantar92/.emacs.d/straight/build/org/ob-tangle hides /usr/share/emacs/29.0.50/lisp/org/ob-tangle
/home/yantar92/.emacs.d/straight/build/org/ob hides /usr/share/emacs/29.0.50/lisp/org/ob
/home/yantar92/.emacs.d/straight/build/org/oc-basic hides /usr/share/emacs/29.0.50/lisp/org/oc-basic
/home/yantar92/.emacs.d/straight/build/org/oc-biblatex hides /usr/share/emacs/29.0.50/lisp/org/oc-biblatex
/home/yantar92/.emacs.d/straight/build/org/oc-csl hides /usr/share/emacs/29.0.50/lisp/org/oc-csl
/home/yantar92/.emacs.d/straight/build/org/oc-natbib hides /usr/share/emacs/29.0.50/lisp/org/oc-natbib
/home/yantar92/.emacs.d/straight/build/org/oc hides /usr/share/emacs/29.0.50/lisp/org/oc
/home/yantar92/.emacs.d/straight/build/org/ol-bbdb hides /usr/share/emacs/29.0.50/lisp/org/ol-bbdb
/home/yantar92/.emacs.d/straight/build/org/ol-bibtex hides /usr/share/emacs/29.0.50/lisp/org/ol-bibtex
/home/yantar92/.emacs.d/straight/build/org/ol-docview hides /usr/share/emacs/29.0.50/lisp/org/ol-docview
/home/yantar92/.emacs.d/straight/build/org/ol-doi hides /usr/share/emacs/29.0.50/lisp/org/ol-doi
/home/yantar92/.emacs.d/straight/build/org/ol-eshell hides /usr/share/emacs/29.0.50/lisp/org/ol-eshell
/home/yantar92/.emacs.d/straight/build/org/ol-eww hides /usr/share/emacs/29.0.50/lisp/org/ol-eww
/home/yantar92/.emacs.d/straight/build/org/ol-gnus hides /usr/share/emacs/29.0.50/lisp/org/ol-gnus
/home/yantar92/.emacs.d/straight/build/org/ol-info hides /usr/share/emacs/29.0.50/lisp/org/ol-info
/home/yantar92/.emacs.d/straight/build/org/ol-irc hides /usr/share/emacs/29.0.50/lisp/org/ol-irc
/home/yantar92/.emacs.d/straight/build/org/ol-man hides /usr/share/emacs/29.0.50/lisp/org/ol-man
/home/yantar92/.emacs.d/straight/build/org/ol-mhe hides /usr/share/emacs/29.0.50/lisp/org/ol-mhe
/home/yantar92/.emacs.d/straight/build/org/ol-rmail hides /usr/share/emacs/29.0.50/lisp/org/ol-rmail
/home/yantar92/.emacs.d/straight/build/org/ol-w3m hides /usr/share/emacs/29.0.50/lisp/org/ol-w3m
/home/yantar92/.emacs.d/straight/build/org/ol hides /usr/share/emacs/29.0.50/lisp/org/ol
/home/yantar92/.emacs.d/straight/build/org/org-agenda hides /usr/share/emacs/29.0.50/lisp/org/org-agenda
/home/yantar92/.emacs.d/straight/build/org/org-archive hides /usr/share/emacs/29.0.50/lisp/org/org-archive
/home/yantar92/.emacs.d/straight/build/org/org-attach-git hides /usr/share/emacs/29.0.50/lisp/org/org-attach-git
/home/yantar92/.emacs.d/straight/build/org/org-attach hides /usr/share/emacs/29.0.50/lisp/org/org-attach
/home/yantar92/.emacs.d/straight/build/org/org-capture hides /usr/share/emacs/29.0.50/lisp/org/org-capture
/home/yantar92/.emacs.d/straight/build/org/org-clock hides /usr/share/emacs/29.0.50/lisp/org/org-clock
/home/yantar92/.emacs.d/straight/build/org/org-colview hides /usr/share/emacs/29.0.50/lisp/org/org-colview
/home/yantar92/.emacs.d/straight/build/org/org-compat hides /usr/share/emacs/29.0.50/lisp/org/org-compat
/home/yantar92/.emacs.d/straight/build/org/org-crypt hides /usr/share/emacs/29.0.50/lisp/org/org-crypt
/home/yantar92/.emacs.d/straight/build/org/org-ctags hides /usr/share/emacs/29.0.50/lisp/org/org-ctags
/home/yantar92/.emacs.d/straight/build/org/org-datetree hides /usr/share/emacs/29.0.50/lisp/org/org-datetree
/home/yantar92/.emacs.d/straight/build/org/org-duration hides /usr/share/emacs/29.0.50/lisp/org/org-duration
/home/yantar92/.emacs.d/straight/build/org/org-element hides /usr/share/emacs/29.0.50/lisp/org/org-element
/home/yantar92/.emacs.d/straight/build/org/org-entities hides /usr/share/emacs/29.0.50/lisp/org/org-entities
/home/yantar92/.emacs.d/straight/build/org/org-faces hides /usr/share/emacs/29.0.50/lisp/org/org-faces
/home/yantar92/.emacs.d/straight/build/org/org-feed hides /usr/share/emacs/29.0.50/lisp/org/org-feed
/home/yantar92/.emacs.d/straight/build/org/org-footnote hides /usr/share/emacs/29.0.50/lisp/org/org-footnote
/home/yantar92/.emacs.d/straight/build/org/org-goto hides /usr/share/emacs/29.0.50/lisp/org/org-goto
/home/yantar92/.emacs.d/straight/build/org/org-habit hides /usr/share/emacs/29.0.50/lisp/org/org-habit
/home/yantar92/.emacs.d/straight/build/org/org-id hides /usr/share/emacs/29.0.50/lisp/org/org-id
/home/yantar92/.emacs.d/straight/build/org/org-indent hides /usr/share/emacs/29.0.50/lisp/org/org-indent
/home/yantar92/.emacs.d/straight/build/org/org-inlinetask hides /usr/share/emacs/29.0.50/lisp/org/org-inlinetask
/home/yantar92/.emacs.d/straight/build/org/org-install hides /usr/share/emacs/29.0.50/lisp/org/org-install
/home/yantar92/.emacs.d/straight/build/org/org-keys hides /usr/share/emacs/29.0.50/lisp/org/org-keys
/home/yantar92/.emacs.d/straight/build/org/org-lint hides /usr/share/emacs/29.0.50/lisp/org/org-lint
/home/yantar92/.emacs.d/straight/build/org/org-list hides /usr/share/emacs/29.0.50/lisp/org/org-list
/home/yantar92/.emacs.d/straight/build/org/org-macro hides /usr/share/emacs/29.0.50/lisp/org/org-macro
/home/yantar92/.emacs.d/straight/build/org/org-macs hides /usr/share/emacs/29.0.50/lisp/org/org-macs
/home/yantar92/.emacs.d/straight/build/org/org-mobile hides /usr/share/emacs/29.0.50/lisp/org/org-mobile
/home/yantar92/.emacs.d/straight/build/org/org-mouse hides /usr/share/emacs/29.0.50/lisp/org/org-mouse
/home/yantar92/.emacs.d/straight/build/org/org-num hides /usr/share/emacs/29.0.50/lisp/org/org-num
/home/yantar92/.emacs.d/straight/build/org/org-pcomplete hides /usr/share/emacs/29.0.50/lisp/org/org-pcomplete
/home/yantar92/.emacs.d/straight/build/org/org-plot hides /usr/share/emacs/29.0.50/lisp/org/org-plot
/home/yantar92/.emacs.d/straight/build/org/org-protocol hides /usr/share/emacs/29.0.50/lisp/org/org-protocol
/home/yantar92/.emacs.d/straight/build/org/org-refile hides /usr/share/emacs/29.0.50/lisp/org/org-refile
/home/yantar92/.emacs.d/straight/build/org/org-src hides /usr/share/emacs/29.0.50/lisp/org/org-src
/home/yantar92/.emacs.d/straight/build/org/org-table hides /usr/share/emacs/29.0.50/lisp/org/org-table
/home/yantar92/.emacs.d/straight/build/org/org-tempo hides /usr/share/emacs/29.0.50/lisp/org/org-tempo
/home/yantar92/.emacs.d/straight/build/org/org-timer hides /usr/share/emacs/29.0.50/lisp/org/org-timer
/home/yantar92/.emacs.d/straight/build/org/org-version hides /usr/share/emacs/29.0.50/lisp/org/org-version
/home/yantar92/.emacs.d/straight/build/org/org hides /usr/share/emacs/29.0.50/lisp/org/org
/home/yantar92/.emacs.d/straight/build/org/ox-ascii hides /usr/share/emacs/29.0.50/lisp/org/ox-ascii
/home/yantar92/.emacs.d/straight/build/org/ox-beamer hides /usr/share/emacs/29.0.50/lisp/org/ox-beamer
/home/yantar92/.emacs.d/straight/build/org/ox-html hides /usr/share/emacs/29.0.50/lisp/org/ox-html
/home/yantar92/.emacs.d/straight/build/org/ox-icalendar hides /usr/share/emacs/29.0.50/lisp/org/ox-icalendar
/home/yantar92/.emacs.d/straight/build/org/ox-koma-letter hides /usr/share/emacs/29.0.50/lisp/org/ox-koma-letter
/home/yantar92/.emacs.d/straight/build/org/ox-latex hides /usr/share/emacs/29.0.50/lisp/org/ox-latex
/home/yantar92/.emacs.d/straight/build/org/ox-man hides /usr/share/emacs/29.0.50/lisp/org/ox-man
/home/yantar92/.emacs.d/straight/build/org/ox-md hides /usr/share/emacs/29.0.50/lisp/org/ox-md
/home/yantar92/.emacs.d/straight/build/org/ox-odt hides /usr/share/emacs/29.0.50/lisp/org/ox-odt
/home/yantar92/.emacs.d/straight/build/org/ox-org hides /usr/share/emacs/29.0.50/lisp/org/ox-org
/home/yantar92/.emacs.d/straight/build/org/ox-publish hides /usr/share/emacs/29.0.50/lisp/org/ox-publish
/home/yantar92/.emacs.d/straight/build/org/ox-texinfo hides /usr/share/emacs/29.0.50/lisp/org/ox-texinfo
/home/yantar92/.emacs.d/straight/build/org/ox hides /usr/share/emacs/29.0.50/lisp/org/ox
/home/yantar92/.emacs.d/straight/build/org/org-loaddefs hides /usr/share/emacs/29.0.50/lisp/org/org-loaddefs
/home/yantar92/.emacs.d/straight/build/let-alist/let-alist hides /usr/share/emacs/29.0.50/lisp/emacs-lisp/let-alist
/home/yantar92/.emacs.d/straight/build/map/map hides /usr/share/emacs/29.0.50/lisp/emacs-lisp/map

Features:
(shadow emacsbug shell-pop shortdoc cl-print pyim-dhashcache
org-pdftools org-noter pdf-view-restore pdf-sync pdf-annot facemenu
pdf-outline pdf-links pdf-history pdf-occur tablist tablist-filter
semantic/wisent/comp semantic/wisent semantic/wisent/wisent
semantic/util-modes semantic/util semantic semantic/tag semantic/lex
semantic/fw mode-local cedet pdf-isearch pdf-misc pdf-tools pdf-view
pdf-cache pdf-info tq pdf-util pdf-macs magit-extras helm-imenu
dired-open helm-ring qrencode helm-command helm-elisp helm-eval
all-the-icons-dired dired-filter dired-hide-dotfiles network-stream
url-cache qp thai-util thai-word helm-font avy mule-util cal-move
ledger-mode ledger-check ledger-texi ledger-test ledger-sort
ledger-report ledger-reconcile ledger-occur ledger-fonts ledger-fontify
ledger-state ledger-complete ledger-schedule ledger-init ledger-xact
ledger-post ledger-exec ledger-navigate eshell esh-cmd esh-ext esh-opt
esh-proc esh-io esh-arg esh-module esh-groups esh-util ledger-context
ledger-commodities ledger-regex ox-org tabify elfeed-link cus-edit
cus-start cus-load w3m-form w3m-symbol w3m-bookmark w3m w3m-hist w3m-fb
bookmark-w3m w3m-ems w3m-favicon w3m-image tab-line w3m-proc w3m-util
mm-archive org-datetree org-learn latex latex-flymake flymake-proc
flymake tex-ispell tex-style tex sendmail boon-moves er-basic-expansions
expand-region-core expand-region-custom sort footnote mail-extr
boon-main boon-arguments multiple-cursors mc-separate-operations
rectangular-region-mode mc-mark-pop mc-edit-lines
mc-hide-unmatched-lines-mode mc-mark-more mc-cycle-cursors
multiple-cursors-core boon-regs boon-utils boon-hl misearch
multi-isearch org-duration cal-iso ffap org-table-sticky-header
org-appear ol-eww eww mm-url ol-rmail ol-mhe ol-irc ol-info ol-gnus
nnselect gnus-search eieio-opt speedbar ezimage dframe ol-docview
doc-view jka-compr ol-bbdb ol-w3m ol-doi org-link-doi tramp-archive
tramp-gvfs helm-x-files org-crypt helm-notmuch helm-notmuch-autoloads
ol-notmuch org-eldoc org-appear-autoloads doom-themes-ext-org
doom-themes doom-themes-base doom-themes-autoloads
org-table-sticky-header-autoloads posframe ob-async ob-async-autoloads
ob-latex ob-dot ob-calc calc-store calc-trail ob-gnuplot ob-ditaa ob-C
cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs ob-python python tramp-sh ob-perl ob-org ob-shell
ob-mathematica org-tempo tempo org-archive ox-md ox-beamer ox-extra doct
ya-org-capture ya-org-capture-autoloads doct-autoloads
org-capture-pop-frame org-capture-pop-frame-autoloads org-protocol
org-analyzer-autoloads pomidor-autoloads alert-autoloads log4e-autoloads
gntp-autoloads org-clock org-autosort org-autosort-autoloads
helm-org-contacts helm-org-contacts-autoloads org-contacts gnus-art
mm-uu mml2015 gnus-sum gnus-group gnus-undo gnus-start gnus-dbus
gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int
gnus-range gnus-win gnus nnheader helm-org-ql helm-org
helm-org-ql-autoloads helm-org-autoloads org-ql-search org-ql-view ov
org-super-agenda org-ql peg org-ql-autoloads peg-autoloads ov-autoloads
org-super-agenda-autoloads map-autoloads org-quick-peek
org-quick-peek-autoloads calfw-org calfw-org-autoloads calfw holidays
hol-loaddefs calfw-autoloads org-attach cdlatex texmathp
cdlatex-autoloads helm-recoll eieio-compat helm-for-files helm-bookmark
helm-info helm-external helm-recoll-autoloads org-capture-ref
org-ref-url-utils org-ref org-ref-helm-bibtex org-ref-helm helm-bibtex
helm-net helm-config org-ref-core reftex-cite reftex reftex-loaddefs
reftex-vars org-ref-glossary org-ref-bibtex org-ref-citeproc key-chord
doi-utils org-ref-utils org-ref-pdf ol-bibtex htmlize bibtex-completion
biblio biblio-download biblio-dissemin biblio-ieee biblio-hal
biblio-dblp biblio-crossref biblio-arxiv timezone biblio-doi biblio-core
ido parsebib org-ref-autoloads key-chord-autoloads ivy-autoloads
helm-bibtex-autoloads bibtex-completion-autoloads biblio-autoloads
biblio-core-autoloads parsebib-autoloads htmlize-autoloads
scimax-inkscape scimax-inkscape-autoloads org-pdftools-autoloads
org-noter-autoloads org-capture org-checklist org-habit org-edna
org-edna-autoloads org-inlinetask org-drill persist org-drill-autoloads
persist-autoloads speed-type speed-type-autoloads notmuch-calendar-x
notmuch-calendar-x-autoloads notmuch notmuch-tree notmuch-jump
notmuch-hello notmuch-show notmuch-print notmuch-crypto notmuch-mua
notmuch-message notmuch-draft notmuch-maildir-fcc notmuch-address
notmuch-company notmuch-parser notmuch-wash coolj notmuch-query
goto-addr icalendar diary-lib diary-loaddefs notmuch-tag notmuch-lib
notmuch-version notmuch-compat mm-view mml-smime smime dig w3m-load
w3m-autoloads notmuch-autoloads elfeed-score elfeed-score-maint
elfeed-score-scoring elfeed-score-serde elfeed-score-rule-stats
elfeed-org elfeed-org-autoloads quick-peek quick-peek-autoloads
elfeed-show elfeed-search vc-mtn vc-hg vc-bzr vc-src vc-sccs vc-svn
vc-cvs vc-rcs vc hideshow display-fill-column-indicator eros
flycheck-tip error-tip notifications dbus flycheck-tip-autoloads
flycheck rainbow-delimiters highlight-numbers parent-mode easy-escape
yasnippet-snippets-autoloads yasnippet-snippets yasnippet elfeed-csv
elfeed elfeed-curl elfeed-log elfeed-db elfeed-lib url-queue xml-query
elfeed-score-rules elfeed-score-log elfeed-score-autoloads
elfeed-autoloads qrencode-el-autoloads keycast keycast-autoloads
gif-screencast gif-screencast-autoloads yaml-mode yaml-mode-autoloads
mingus libmpdee cl mingus-autoloads libmpdee-autoloads calctex calc-sel
calc-ext calctex-autoloads calc calc-loaddefs rect calc-macs
shell-pop-autoloads eterm-256color-autoloads xterm-color-autoloads vterm
term ehelp vterm-module term/xterm xterm vterm-autoloads ereader xml+
view shr kinsoku svg dom picture ereader-autoloads xml+-autoloads
diffpdf diffpdf-autoloads pdf-view-restore-autoloads pdf-tools-autoloads
tablist-autoloads wolfram-mode wolfram-mode-autoloads
ledger-mode-autoloads auctex-autoloads tex-site ebuild-mode skeleton
sh-script smie executable ebuild-mode-autoloads lua-mode
lua-mode-autoloads gnuplot-autoloads eros-autoloads nameless lisp-mnt
nameless-autoloads paredit paredit-autoloads which-key
which-key-autoloads helm-descbinds helm-descbinds-autoloads elisp-demos
elisp-demos-autoloads helpful edebug info-look help-fns radix-tree
elisp-refs helpful-autoloads elisp-refs-autoloads tldr tldr-autoloads
macrostep macrostep-autoloads font-lock-profiler
font-lock-profiler-autoloads font-lock-studio font-lock-studio-autoloads
memory-usage memory-usage-autoloads bug-hunter bug-hunter-autoloads
lorem-ipsum lorem-ipsum-autoloads debug backtrace yasnippet-autoloads
move-text move-text-autoloads aggressive-indent
aggressive-indent-autoloads visual-regexp-autoloads magit-bookmark
bookmark pp helm-bm helm-bm-autoloads bm bm-autoloads helm-dash
dash-docs use-package-dash-docs xml helm-dash-autoloads
dash-docs-autoloads disk-usage disk-usage-autoloads
dired-git-info-autoloads dired-hide-dotfiles-autoloads
dired-filter-autoloads diredfl diredfl-autoloads
all-the-icons-dired-autoloads dired-async dired-open-autoloads
dired-avfs dired-avfs-autoloads dired-narrow-autoloads dired-hacks-utils
dired-hacks-utils-autoloads dired+ image-dired image-mode exif
image-file image-converter dired-x dired-aux dired+-autoloads winner
windower emacs-windower-autoloads goggles pulse skip-buffers-mode
recentf tree-widget wid-edit helm-icons treemacs-icons treemacs-themes
treemacs-core-utils treemacs-logging treemacs-customization pfuture
inline helm-adaptive helm-mode helm-files tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat ls-lisp helm-buffers helm-occur
helm-tags helm-locate helm-grep helm-regexp helm-utils helm-help
helm-types helm async-bytecomp helm-global-bindings helm-source
helm-multi-match helm-lib async eval-sexp-fu eval-sexp-fu-autoloads
goggles-autoloads easy-escape-autoloads highlight-numbers-autoloads
parent-mode-autoloads rainbow-delimiters-autoloads highlight-parentheses
highlight-parentheses-autoloads flycheck-autoloads pkg-info-autoloads
epl-autoloads langtool compile langtool-autoloads el-patch
el-patch-autoloads flyspell ispell hi-lock ediff ediff-merg ediff-mult
ediff-wind ediff-diff ediff-help ediff-init ediff-util browse-at-remote
vc-git vc-dispatcher f browse-at-remote-autoloads forge-list
forge-commands forge-semi forge-bitbucket buck forge-gogs gogs
forge-gitea gtea forge-gitlab glab forge-github ghub-graphql treepy
gsexp ghub let-alist gnutls forge-notify forge-revnote forge-pullreq
forge-issue forge-topic yaml parse-time bug-reference forge-post
markdown-mode thingatpt forge-repo forge forge-core forge-db closql
emacsql-sqlite emacsql emacsql-compiler url-http url-auth url-gw nsm
magit-submodule magit-obsolete magit-blame magit-stash magit-reflog
magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-tag
magit-merge magit-branch magit-reset magit-files magit-refs magit-status
magit magit-repos magit-apply magit-wip magit-log which-func imenu
magit-diff git-commit log-edit message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail
rmail-loaddefs text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr
mailabbrev mail-utils gmm-utils mailheader pcvs-util add-log magit-core
magit-autorevert magit-margin magit-transient magit-process with-editor
shell magit-mode transient magit-git magit-section magit-utils crm
forge-autoloads yaml-autoloads markdown-mode-autoloads ghub-autoloads
treepy-autoloads let-alist-autoloads closql-autoloads
emacsql-sqlite-autoloads emacsql-autoloads unpackaged smerge-mode
diff-mode diff ox-odt rng-loc rng-uri rng-parse rng-match rng-dt
rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok nxml-util ox-latex
ox-icalendar org-agenda ox-html table ox-ascii ox-publish ox org-element
org-persist xdg avl-tree ibuf-ext ibuffer ibuffer-loaddefs use-package
use-package-ensure use-package-delight ts ts-autoloads
unpackaged-autoloads magit-autoloads magit-section-autoloads
git-commit-autoloads with-editor-autoloads transient-autoloads
autorevert filenotify disp-table hl-todo pretty-symbols company-oddmuse
company-keywords company-etags etags fileloop generator xref project
company-gtags company-dabbrev-code company-dabbrev company-files
company-clang company-capf company-cmake company-semantic
company-template company-bbdb company persistent-scratch
persistent-scratch-autoloads savehist backup-walker-autoloads
company-autoloads helm-icons-autoloads treemacs-autoloads cfrs-autoloads
posframe-autoloads pfuture-autoloads ace-window-autoloads avy-autoloads
f-autoloads helm-autoloads helm-core-autoloads popup-autoloads
face-remap pyim pyim-hacks pyim-probe pyim-cregexp xr pyim-process
pyim-cstring pyim-autoselector pyim-punctuation pyim-outcome
pyim-indicator pyim-preview pyim-magic pyim-candidates pyim-codes
pyim-imobjs pyim-pinyin pyim-pymap pyim-entered pyim-dcache pyim-dict
pyim-page popup pyim-scheme pyim-common pyim-autoloads xr-autoloads
async-autoloads reverse-im quail reverse-im-autoloads hydra lv
boon-qwerty color olivetti straight-x boon boon-keys boon-core
boon-loaddefs boon-autoloads pcre2el-autoloads
multiple-cursors-autoloads expand-region-autoloads meta-functions org-id
org-refile meta-functions-autoloads hl-line memoize memoize-autoloads
info-colors info-colors-autoloads hl-todo-autoloads latex-pretty-symbols
latex-pretty-symbols-autoloads pretty-symbols-autoloads page-break-lines
page-break-lines-autoloads edmacro kmacro adaptive-wrap
adaptive-wrap-autoloads olivetti-autoloads shackle trace
shackle-autoloads all-the-icons all-the-icons-faces data-material
data-weathericons data-octicons data-fileicons data-faicons
data-alltheicons all-the-icons-autoloads org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-footnote org-src ob-comint org-pcomplete
pcomplete comint ansi-color ring org-list org-faces org-entities
noutline outline org-version ob-emacs-lisp ob-core ob-eval org-cycle
org-table oc-basic bibtex iso8601 time-date ol org-fold org-fold-core
org-keys oc org-compat advice org-macs org-loaddefs format-spec
find-func cal-menu calendar cal-loaddefs modus-vivendi-theme
modus-operandi-theme modus-themes modus-themes-autoloads gcmh
gcmh-autoloads use-package-diminish s s-autoloads ht dash ht-autoloads
dash-autoloads pcase asoc asoc.el-autoloads no-littering
no-littering-autoloads hydra-autoloads lv-autoloads finder-inf
use-package-bind-key org-contrib-autoloads comp comp-cstr warnings rx
bind-key easy-mmode diminish diminish-autoloads use-package-core
use-package-autoloads bind-key-autoloads straight-autoloads cl-extra
help-mode straight server site-gentoo helm-easymenu info package
browse-url url url-proxy url-privacy url-expand url-methods url-history
url-cookie url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json map url-vars seq gv subr-x byte-opt bytecomp
byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
button loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting font-render-setting cairo x multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 6986171 2237182)
 (symbols 48 104486 98)
 (strings 32 752636 136785)
 (string-bytes 1 25502273)
 (vectors 16 561998)
 (vector-slots 8 24848211 822896)
 (floats 8 65913 13311)
 (intervals 56 655756 11352)
 (buffers 992 175))

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Date: Thu, 11 Nov 2021 21:56:46 +0800
> 
> According to buffer-chars-modified-tick docstring:
> "By comparing the values returned by two individual calls of
> buffer-chars-modified-tick, you can tell whether a character change
> occurred in that buffer in between these calls"
> 
> However, the return value can change even when no visible change is made
> to buffer text.
> 
> Steps to reproduce:
> 1. emacs -Q
> 2. Evaluate the following code:
> 
> (defun print-tick-before ()
>   (when (eq this-command 'self-insert-command)
>   (warn "Tick before: %S" (buffer-chars-modified-tick))))
> (defun print-tick-after ()
>     (when (eq this-command 'self-insert-command)
>   (warn "Tick after: %S" (buffer-chars-modified-tick))))
> (add-hook 'pre-command-hook #'print-tick-before)
> (add-hook 'post-command-hook #'print-tick-after)
> 
> 3. Insert a latin symbol ?a twice. The warning buffer will print
> something like
> 
> Warning (emacs): Tick before: 1698 Disable showing Disable logging
> Warning (emacs): Tick after: 1699 Disable showing Disable logging
> Warning (emacs): Tick before: 1699 Disable showing Disable logging
> Warning (emacs): Tick after: 1702 Disable showing Disable logging
> 
> Note that second and third line show the same buffer-chars-modified-tick
> value.
> 
> 4. Change input method (C-\) to russian-computer or i.e. arabic
> 5. Insert a non-latin symbol ?ф twice. The warning buffer will print
> something like
> 
> Warning (emacs): Tick before: 1706 Disable showing Disable logging
> Warning (emacs): Tick after: 1707 Disable showing Disable logging
> Warning (emacs): Tick before: 1711 Disable showing Disable logging
> Warning (emacs): Tick after: 1712 Disable showing Disable logging
> 
> Note that second and third line _do not_ show the same
> buffer-chars-modified-tick value even though buffer text has not been
> changed between the two self-insert commands
> 
> Expected behaviour: return value of buffer-chars-modified-tick does not
> change when no changes in buffer text are made.

How do you know there was no changes in the buffer?  You call your
function from pre/post-command-hook, but why is it guaranteed that
there was no change in the buffer between post-command-hook and the
following pre-command-hook?

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> How do you know there was no changes in the buffer?  You call your
> function from pre/post-command-hook, but why is it guaranteed that
> there was no change in the buffer between post-command-hook and the
> following pre-command-hook?

I also tested the bug by setting debug-on-entry for self-insert-command.
Try the following:
1. emacs -Q
2. M-\ russian-computer <RET>
3. M-\
4. M-: (buffer-chars-modified-tick) <RET>. Note the return value.
5. M-x debug-on-entry <RET> self-insert-command <RET>
6. Insert ?a
7. The debugger window appears. ?a is not yet inserted
8. In the debugger window: e M-p <RET> (call
   buffer-chars-modified-tick). The return value should be the same with
   4.
9. Continue execution of self-insert-command in the debugger (c). The ?a
   is inserted
10. Repeat 8. The return value correctly changes. Note the return value.
11. Continue execution to exit the debugger window (c)
12. Switch to russian input method (M-\)
13. Run M-: M-p <RET>. Note the return value. It is same with 10.
14. Run M-: (buffer-hash) <RET>. "(buffer-hash)" should by yanked to
    avoid triggering debugger. Note that hash value.
13. Type ?ф (?a on qwerty keyboard)
14. The debugger appears again. ?ф is _not_ yet inserted
15. Repeat 8. The return value is different from 10 even though the
    buffer text is not changed (and it can be confirmed if you run
    e (buffer-hash) <RET>)

Of course, there might be some kind of invisible change in buffer. I.e.
text is added and immediately deleted from the buffer without redisplay.
However, even if there is any change like that, before-change-functions
and after-change-functions are not triggered. That would be another bug
then.

Best,
Ihor

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: 51766@debbugs.gnu.org
> Date: Thu, 11 Nov 2021 23:50:31 +0800
> 
> Of course, there might be some kind of invisible change in buffer. I.e.
> text is added and immediately deleted from the buffer without redisplay.

That's exactly what happens: quail.el deletes the inserted character
and then reinserts it (for reasons unrelated to this issue).  So the
count of the changes is not equal to the number of characters actually
inserted.  I see no problem here, since the documentation never
promises that the difference between the values returned by successive
calls to buffer-chars-modified-tick will be exactly equal to the
number of inserted or deleted characters.

So if Org relies on such an equality, it's a bug in Org (but I didn't
look at the relevant Org code, and don't have a clear idea of how
exactly it uses the above function for whatever it is caching).

> However, even if there is any change like that, before-change-functions
> and after-change-functions are not triggered. That would be another bug
> then.

quail.el inhibit buffer modifications in places, since otherwise you'd
have too many of them.  It wants to pretend that just one character
was inserted.

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> That's exactly what happens: quail.el deletes the inserted character
> and then reinserts it (for reasons unrelated to this issue).  So the
> count of the changes is not equal to the number of characters actually
> inserted.  I see no problem here, since the documentation never
> promises that the difference between the values returned by successive
> calls to buffer-chars-modified-tick will be exactly equal to the
> number of inserted or deleted characters.

I agree that Emacs does not break any promises here. Though it is
unfortunate. Feel free to close this bug report.

> So if Org relies on such an equality, it's a bug in Org (but I didn't
> look at the relevant Org code, and don't have a clear idea of how
> exactly it uses the above function for whatever it is caching).

Let me explain a little (hoping that you might have some idea about
alternative solutions without using buffer-chars-modified-tick).

Org has a caching mechanism (org-element-cache) that keeps parsed buffer
representation in memory and updates it on the fly as the buffer
changes. To make the mechanism work, Org must keep track of all the
changes in buffer and update the affected Org elements in memory.
Naturally, this is done using before/after-change-functions.

However, some third-party code carelessly uses
inhibit-modification-hooks and some edits may be missed by element
cache. If we just ignore the possibility of such edits, cache can be
broken badly. So, there is currently a control code that detects if
buffer has been changed outside the Org's change functions. The control
code uses buffer-chars-modified-tick.

The behaviour of quail.el makes the control code useless -
buffer-chars-modified-tick can no longer be reliably used to detect
unfavourable "stealthy" changes. AFAIK, the only alternative way to
detect the changes is buffer-hash/secure-hash. But calculating hash is
very too slow when I try to put it into before/after-change-functions. I
do not know any fast (as fast as buffer-chars-modified-tick) way to
detect buffer changes.

> quail.el inhibit buffer modifications in places, since otherwise you'd
> have too many of them.  It wants to pretend that just one character
> was inserted.

I understand the idea behind suppressing the modification hooks by
quail. Though it would be helpful if before-change-functions were called
before inserting+deleting a character by quail is done.

Best,
Ihor

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: 51766@debbugs.gnu.org
> Date: Fri, 12 Nov 2021 20:06:41 +0800
> 
> Org has a caching mechanism (org-element-cache) that keeps parsed buffer
> representation in memory and updates it on the fly as the buffer
> changes. To make the mechanism work, Org must keep track of all the
> changes in buffer and update the affected Org elements in memory.
> Naturally, this is done using before/after-change-functions.
> 
> However, some third-party code carelessly uses
> inhibit-modification-hooks and some edits may be missed by element
> cache. If we just ignore the possibility of such edits, cache can be
> broken badly. So, there is currently a control code that detects if
> buffer has been changed outside the Org's change functions. The control
> code uses buffer-chars-modified-tick.
> 
> The behaviour of quail.el makes the control code useless -
> buffer-chars-modified-tick can no longer be reliably used to detect
> unfavourable "stealthy" changes.

This last part I don't think I understand: why does quail's behavior
make the control code useless?  The value returned by
buffer-chars-modified-tick still increases in your recipe, so what
exactly is the aspect of that behavior that makes the control code
useless?  I think some additional details here are missing from your
description which could explain the issue.

> > quail.el inhibit buffer modifications in places, since otherwise you'd
> > have too many of them.  It wants to pretend that just one character
> > was inserted.
> 
> I understand the idea behind suppressing the modification hooks by
> quail. Though it would be helpful if before-change-functions were called
> before inserting+deleting a character by quail is done.

I don't understand this, either.  Are you saying that inserting a
character via an input method doesn't call buffer-modification hooks
even once?  If the hooks are called, then what exactly is the problem
with the hooks in this scenario?

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> This last part I don't think I understand: why does quail's behavior
> make the control code useless?  The value returned by
> buffer-chars-modified-tick still increases in your recipe, so what
> exactly is the aspect of that behavior that makes the control code
> useless?  I think some additional details here are missing from your
> description which could explain the issue.

The control code makes sure that all the changes made in buffer are
processed by org-element-cache. It means that
org-element--after-change-function saves the buffer-chars-modified-tick
and the next org-element--before-change-function checks if the saved
value is unchanged. If the saved value is changed, the buffer has been
changed after org-element--after-change-function, but before next
org-element--before-change-function. Such change may be arbitrary and
the whole cache is potentially obsolete.

In code, the described roughly looks like:

(defun org-element--after-change-function (...)
 (setq org-element-chars-modified-tick (buffer-chars-modified-tick))
 (org-element-cache-submit-request ...))

(defun org-element--before-change-function (...)
 (unless (eq org-element-chars-modified-tick (buffer-chars-modified-tick))
     ;; Buffer has been changed without calling after-change-function
     ;; and we have no way to determine which part of buffer has been changed.
 ))

quail changes the buffer after org-element--after-change-function call,
but before org-element--before-change-function. So, all Org can see is
that something has been changed in buffer, but there is no way to tell
what it was. Org cannot distinguish between harmless buffer edits by
quail (they do not change buffer text) and other kinds of "silent"
changes.

> I don't understand this, either.  Are you saying that inserting a
> character via an input method doesn't call buffer-modification hooks
> even once?  If the hooks are called, then what exactly is the problem
> with the hooks in this scenario?

The hooks are called, but after quail already triggered
buffer-chars-modified-tick increase. If quail called
before-change-functions before buffer-chars-modified-tick increases, it
would be useful for my scenario. Though I am not sure how feasible it
is. Just an idea.

Best,
Ihor

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: 51766@debbugs.gnu.org
> Date: Fri, 12 Nov 2021 20:53:33 +0800
> 
> The control code makes sure that all the changes made in buffer are
> processed by org-element-cache. It means that
> org-element--after-change-function saves the buffer-chars-modified-tick
> and the next org-element--before-change-function checks if the saved
> value is unchanged. If the saved value is changed, the buffer has been
> changed after org-element--after-change-function, but before next
> org-element--before-change-function. Such change may be arbitrary and
> the whole cache is potentially obsolete.
> 
> In code, the described roughly looks like:
> 
> (defun org-element--after-change-function (...)
>  (setq org-element-chars-modified-tick (buffer-chars-modified-tick))
>  (org-element-cache-submit-request ...))
> 
> (defun org-element--before-change-function (...)
>  (unless (eq org-element-chars-modified-tick (buffer-chars-modified-tick))
>      ;; Buffer has been changed without calling after-change-function
>      ;; and we have no way to determine which part of buffer has been changed.
>  ))
> 
> quail changes the buffer after org-element--after-change-function call,
> but before org-element--before-change-function. So, all Org can see is
> that something has been changed in buffer, but there is no way to tell
> what it was. Org cannot distinguish between harmless buffer edits by
> quail (they do not change buffer text) and other kinds of "silent"
> changes.

OK, but why does this invalidate what Org does?  All it means, AFAIU,
is that in some cases Org will do unnecessary processing.  Those cases
are probably not too frequent.

IOW, why invalidating the cache unnecessarily is such a big deal?

> > I don't understand this, either.  Are you saying that inserting a
> > character via an input method doesn't call buffer-modification hooks
> > even once?  If the hooks are called, then what exactly is the problem
> > with the hooks in this scenario?
> 
> The hooks are called, but after quail already triggered
> buffer-chars-modified-tick increase. If quail called
> before-change-functions before buffer-chars-modified-tick increases, it
> would be useful for my scenario. Though I am not sure how feasible it
> is. Just an idea.

Would it help if Org looked at both buffer-modified-tick and
buffer-chars-modified-tick?

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> quail changes the buffer after org-element--after-change-function call,
>> but before org-element--before-change-function. So, all Org can see is
>> that something has been changed in buffer, but there is no way to tell
>> what it was. Org cannot distinguish between harmless buffer edits by
>> quail (they do not change buffer text) and other kinds of "silent"
>> changes.
>
> OK, but why does this invalidate what Org does?  All it means, AFAIU,
> is that in some cases Org will do unnecessary processing.  Those cases
> are probably not too frequent.

Normally, buffer changes under inhibit-modification-hooks are not
frequent indeed. But not with quail + non-latin input methods. Every
single self-insert-command triggers such change.

> IOW, why invalidating the cache unnecessarily is such a big deal?

It is critical for Org to know which part of buffer was changed (i.e.
beg, end, length that are normally passed as arguments of
after-change-functions). org-element-cache can contain >100k elements
for especially large buffers. Manually checking which elements are
changed without knowing the changed region is inefficient. Clearing the
cache is not too much of a big deal, but causes slowdown when user runs
a query on Org buffers (i.e. in agenda or sparse trees) - the buffer has
to be re-parsed. When every edit triggers cache invalidation (that's
what happens when user uses non-latin input method), the slowdown is
pretty much guaranteed. Moreover, too frequent cache resets increase the
load on Emacs' garbage collector (cache size is typically a multiple of
buffer size). Overloading garbage collector leads to overall Emacs
slowdown.

>> The hooks are called, but after quail already triggered
>> buffer-chars-modified-tick increase. If quail called
>> before-change-functions before buffer-chars-modified-tick increases, it
>> would be useful for my scenario. Though I am not sure how feasible it
>> is. Just an idea.
>
> Would it help if Org looked at both buffer-modified-tick and
> buffer-chars-modified-tick?

When buffer-chars-modified-tick is changed, buffer-modified-tick is also
changed. AFAIU, buffer-chars-modified-tick registers a subset of buffer
modifications that actually change buffer text. buffer-modified-tick
also registers text property changes. So, I do not see how
buffer-modified-tick can help.

Best,
Ihor

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: 51766@debbugs.gnu.org
> Date: Fri, 12 Nov 2021 21:39:54 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> quail changes the buffer after org-element--after-change-function call,
> >> but before org-element--before-change-function. So, all Org can see is
> >> that something has been changed in buffer, but there is no way to tell
> >> what it was. Org cannot distinguish between harmless buffer edits by
> >> quail (they do not change buffer text) and other kinds of "silent"
> >> changes.
> >
> > OK, but why does this invalidate what Org does?  All it means, AFAIU,
> > is that in some cases Org will do unnecessary processing.  Those cases
> > are probably not too frequent.
> 
> Normally, buffer changes under inhibit-modification-hooks are not
> frequent indeed. But not with quail + non-latin input methods. Every
> single self-insert-command triggers such change.

"Such change" being what exactly? the situation where
buffer-chars-modified-tick changes between post-command-hook and the
following pre-command-hook? or something else?

I'm asking because I don't think I see the problem you are describing.
With the following code:

  (defun my-before (beg end)
    (message "buf %s beg %s end %s" (current-buffer) beg end))
  (defun my-after (beg end len)
    (message "buf %s beg %s end %s len %s" (current-buffer) beg end len))
  (add-hook 'before-change-functions 'my-before)
  (add-hook 'after-change-functions 'my-after)

if I activate the chinese-py input method, then inserting any
character via the input method produces a single call to my-before and
a single call to my-after, and with the expected values, for example:

  buf *scratch* beg 440 end 440
  buf *scratch* beg 440 end 441 len 0

So what exactly is the problem with these hooks when non-latin input
methods are used?  Or what am I missing?

> > IOW, why invalidating the cache unnecessarily is such a big deal?
> 
> It is critical for Org to know which part of buffer was changed (i.e.
> beg, end, length that are normally passed as arguments of
> after-change-functions). org-element-cache can contain >100k elements
> for especially large buffers. Manually checking which elements are
> changed without knowing the changed region is inefficient. Clearing the
> cache is not too much of a big deal, but causes slowdown when user runs
> a query on Org buffers (i.e. in agenda or sparse trees) - the buffer has
> to be re-parsed. When every edit triggers cache invalidation (that's
> what happens when user uses non-latin input method), the slowdown is
> pretty much guaranteed. Moreover, too frequent cache resets increase the
> load on Emacs' garbage collector (cache size is typically a multiple of
> buffer size). Overloading garbage collector leads to overall Emacs
> slowdown.

Perhaps Org developers should ask for infrastructure changes that will
allow Org to maintain such a cache reliably and not too expensively?
It sounds like Org currently applies all kinds of heuristics based on
assumptions about how the internals work and using hooks and features
that were never designed to support this kind of caching.  Jumping
through hoops in Lisp trying to implement something that might be much
easier or even trivial in C is not the best way of getting such jobs
done.

So perhaps someone could describe on emacs-devel what does Org need to
maintain this cache, and we could then see how to provide those
features to Org.

> >> The hooks are called, but after quail already triggered
> >> buffer-chars-modified-tick increase. If quail called
> >> before-change-functions before buffer-chars-modified-tick increases, it
> >> would be useful for my scenario. Though I am not sure how feasible it
> >> is. Just an idea.
> >
> > Would it help if Org looked at both buffer-modified-tick and
> > buffer-chars-modified-tick?
> 
> When buffer-chars-modified-tick is changed, buffer-modified-tick is also
> changed. AFAIU, buffer-chars-modified-tick registers a subset of buffer
> modifications that actually change buffer text. buffer-modified-tick
> also registers text property changes. So, I do not see how
> buffer-modified-tick can help.

If you look at the implementation, you will see that when Emacs
decides that the buffer's chars-modified-tick needs to be increased,
it simply assigns to it the value of the buffer's modified-tick at
that moment.  So by tracking the value of buffer-modified-tick you
could perhaps explain why buffer-chars-modified-tick jumps by more
than you expected.

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> "Such change" being what exactly? the situation where
> buffer-chars-modified-tick changes between post-command-hook and the
> following pre-command-hook? or something else?

The former.

> So what exactly is the problem with these hooks when non-latin input
> methods are used?  Or what am I missing?

There is no problem with the hooks in your example. However, consider
the following:

(let ((inhibit-modification-hooks t))
  (insert "Insertion that will never trigger before/after-change-functions"))

Org cache is bound to track all the changes in buffer. Any missed change
will lead to cache corruption. So, situations like the above must be
tracked somehow. This tracking can be done using
buffer-chars-modified-tick or buffer-hash/secure-hash. The latter is too
slow.

If I understand your earlier explanation correctly, quail for non-latin
input (I tested with russian-computer) does something like

(let ((inhibit-modification-hooks t))
  (insert ?char)
  (backward-delete-char)) ;; This increases buffer-chars-modified-tick
(insert ?translated_char_according_to_input_method)

The change hooks will only be called for the last insertion. However,
the first insertion+deletion will change buffer-chars-modified-tick.

The quail's insertion+deletion itself is not a problem for Org cache -
it does not really alter the buffer text and cannot break the cache. The
problem is that it cannot be distinguished from the first example - both
cases will trigger buffer-chars-modified-tick increase.

> Perhaps Org developers should ask for infrastructure changes that will
> allow Org to maintain such a cache reliably and not too expensively?
> It sounds like Org currently applies all kinds of heuristics based on
> assumptions about how the internals work and using hooks and features
> that were never designed to support this kind of caching.  Jumping
> through hoops in Lisp trying to implement something that might be much
> easier or even trivial in C is not the best way of getting such jobs
> done.
>
> So perhaps someone could describe on emacs-devel what does Org need to
> maintain this cache, and we could then see how to provide those
> features to Org.

I am one of the Org developers.

The only assumption I had it that Emacs does not frequently change
buffer text without triggering modification hooks. Clearly, the
assumption was wrong.

Ideally, a way to track _all_ buffer modifications regardless of
inhibit-modification-hooks would be useful. Currently, Org has to work
around the possibilities that text can be inserted without triggering
modification hooks: (1) when
inhibit-modification-hooks/before-change-functions/after-change-functions
are let-bound to nil; (2) when changes are made in indirect buffer with
different buffer-local values of before/after-change-functions.

Alternatively, Emacs could support language parsers. Org cache
implements editing syntax tree generated by Org element parser. It is
very similar to what tree-sitter editing API does: https://tree-sitter.github.io/tree-sitter/using-parsers#editing

Native support for storing, modifying, and querying syntax trees using
efficient data structures could be a great addition to Emacs from Org's
perspective. Though it is not an easy feature to implement.

AFAIR, something similar to my last suggestion has been already
proposed: tree-sitter support. I can also propose the first idea about
reliable buffer change tracking if you think that it is something
reasonable.

Best,
Ihor

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: 51766@debbugs.gnu.org
> Date: Sat, 13 Nov 2021 17:10:11 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > "Such change" being what exactly? the situation where
> > buffer-chars-modified-tick changes between post-command-hook and the
> > following pre-command-hook? or something else?
> 
> The former.
> 
> > So what exactly is the problem with these hooks when non-latin input
> > methods are used?  Or what am I missing?
> 
> There is no problem with the hooks in your example. However, consider
> the following:
> 
> (let ((inhibit-modification-hooks t))
>   (insert "Insertion that will never trigger before/after-change-functions"))

So the problem is that Org uses the modification hooks as the primary
mechanism (with which quail presents no problem), and
buffer-chars-modified-tick as the fallback, and that when some code
inhibits the modification hooks, then the primary mechanism cannot
work and quail breaks the fallback?

> The quail's insertion+deletion itself is not a problem for Org cache -
> it does not really alter the buffer text and cannot break the cache. The
> problem is that it cannot be distinguished from the first example - both
> cases will trigger buffer-chars-modified-tick increase.

You didn't answer my question regarding buffer-modified-tick: it can
explain to Org why buffer-chars-modified-tick jumped unexpectedly, and
thus (hopefully) resolve this situation.  If that helps, you could
perhaps turn the table and use buffer-chars-modified-tick is the
primary method of discovering changes, not as fallback.

> > Perhaps Org developers should ask for infrastructure changes that will
> > allow Org to maintain such a cache reliably and not too expensively?
> > It sounds like Org currently applies all kinds of heuristics based on
> > assumptions about how the internals work and using hooks and features
> > that were never designed to support this kind of caching.  Jumping
> > through hoops in Lisp trying to implement something that might be much
> > easier or even trivial in C is not the best way of getting such jobs
> > done.
> >
> > So perhaps someone could describe on emacs-devel what does Org need to
> > maintain this cache, and we could then see how to provide those
> > features to Org.
> 
> I am one of the Org developers.
> 
> The only assumption I had it that Emacs does not frequently change
> buffer text without triggering modification hooks. Clearly, the
> assumption was wrong.

I meant the assumptions about what buffer-chars-modified-tick does and
what its value means.

> Ideally, a way to track _all_ buffer modifications regardless of
> inhibit-modification-hooks would be useful.

But Org is not interested in just any moidification, AFAIU.  It is
only interested in modifications that change the buffer text.  Isn't
that true?  Or what else is Org interested in for this purpose.

> Alternatively, Emacs could support language parsers.

I meant support on the low level, where changes to buffer text are
considered and indicated.  As I indicate below, the integration of
tree-sitter simply uses the existing change indications, so I'm not
sure how would a parser support help you in this matter.

> Org cache
> implements editing syntax tree generated by Org element parser. It is
> very similar to what tree-sitter editing API does: https://tree-sitter.github.io/tree-sitter/using-parsers#editing
> 
> Native support for storing, modifying, and querying syntax trees using
> efficient data structures could be a great addition to Emacs from Org's
> perspective. Though it is not an easy feature to implement.
> 
> AFAIR, something similar to my last suggestion has been already
> proposed: tree-sitter support. I can also propose the first idea about
> reliable buffer change tracking if you think that it is something
> reasonable.

The Emacs code related to tree-sitter already uses change indications,
and AFAIR didn't require any changes to the existing infrastructure.
I wonder why Org cannot settle with what we have, if your needs are
similar enough to those of tree-sitter.

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> (let ((inhibit-modification-hooks t))
>>   (insert "Insertion that will never trigger before/after-change-functions"))
>
> So the problem is that Org uses the modification hooks as the primary
> mechanism (with which quail presents no problem), and
> buffer-chars-modified-tick as the fallback, and that when some code
> inhibits the modification hooks, then the primary mechanism cannot
> work and quail breaks the fallback?

Not exactly. Org uses modification hooks as the only mechanism to
process buffer changes because Org needs to know the region where the
buffer text changes. buffer-chars-modified-tick is used for error
detection - when buffer text is changed, but Org modification hooks are
not called for some reason. quail triggers false positives during the error
detection.

>> The quail's insertion+deletion itself is not a problem for Org cache -
>> it does not really alter the buffer text and cannot break the cache. The
>> problem is that it cannot be distinguished from the first example - both
>> cases will trigger buffer-chars-modified-tick increase.
>
> You didn't answer my question regarding buffer-modified-tick: it can
> explain to Org why buffer-chars-modified-tick jumped unexpectedly, and
> thus (hopefully) resolve this situation.  If that helps, you could
> perhaps turn the table and use buffer-chars-modified-tick is the
> primary method of discovering changes, not as fallback.
> ...
>> The only assumption I had it that Emacs does not frequently change
>> buffer text without triggering modification hooks. Clearly, the
>> assumption was wrong.
>
> I meant the assumptions about what buffer-chars-modified-tick does and
> what its value means.

It seems that we have some misunderstanding here. Org does not care
about the value of buffer-chars-modified-tick - just whether
buffer-chars-modified-tick is changed or not (see the above).

Even if Org used the value of buffer-chars-modified-tick, it would not
be useful. There is no information about buffer region where the edits
happened if we just look at buffer-chars-modified-tick. AFAIK, only
after-change-functions have access to the changed region bounds.

>> Ideally, a way to track _all_ buffer modifications regardless of
>> inhibit-modification-hooks would be useful.
>
> But Org is not interested in just any moidification, AFAIU.  It is
> only interested in modifications that change the buffer text.  Isn't
> that true?  Or what else is Org interested in for this purpose.

You are right. Org is interested in modifications that change buffer
text. Also, Org is interested to be not affected by
inhibit-modification-hooks. Making sure that Org modification hooks run
for every modification is automatically making sure that no
modifications that do change buffer text are missed. I thought that it
can be the simplest approach to fix the issue.

>> Alternatively, Emacs could support language parsers.
>
> I meant support on the low level, where changes to buffer text are
> considered and indicated.  As I indicate below, the integration of
> tree-sitter simply uses the existing change indications, so I'm not
> sure how would a parser support help you in this matter.

I probably went too far with my suggestion here. I meant not only
handling changes, but also adding API for working with syntax trees using
C-level functions. It is out of scope of this bug report.

> The Emacs code related to tree-sitter already uses change indications,
> and AFAIR didn't require any changes to the existing infrastructure.
> I wonder why Org cannot settle with what we have, if your needs are
> similar enough to those of tree-sitter.

I just checked https://github.com/casouri/emacs/blob/ts/src/insdel.c
That ts Emacs branch directly modifies C-level Emacs buffer editing
primitives (see the calls to ts_record_change). So, it is not affected
by inhibit-modification-hooks.

Best,
Ihor

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: 51766@debbugs.gnu.org
> Date: Sat, 13 Nov 2021 19:29:36 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> (let ((inhibit-modification-hooks t))
> >>   (insert "Insertion that will never trigger before/after-change-functions"))
> >
> > So the problem is that Org uses the modification hooks as the primary
> > mechanism (with which quail presents no problem), and
> > buffer-chars-modified-tick as the fallback, and that when some code
> > inhibits the modification hooks, then the primary mechanism cannot
> > work and quail breaks the fallback?
> 
> Not exactly. Org uses modification hooks as the only mechanism to
> process buffer changes because Org needs to know the region where the
> buffer text changes. buffer-chars-modified-tick is used for error
> detection - when buffer text is changed, but Org modification hooks are
> not called for some reason. quail triggers false positives during the error
> detection.

Is that any different from what I said, which is that you need error
detection only when the modification hooks are not called?  And that
the quail behavior is only the issue when using this error detection,
i.e. when the modification hooks are not called?

> >> The quail's insertion+deletion itself is not a problem for Org cache -
> >> it does not really alter the buffer text and cannot break the cache. The
> >> problem is that it cannot be distinguished from the first example - both
> >> cases will trigger buffer-chars-modified-tick increase.
> >
> > You didn't answer my question regarding buffer-modified-tick: it can
> > explain to Org why buffer-chars-modified-tick jumped unexpectedly, and
> > thus (hopefully) resolve this situation.  If that helps, you could
> > perhaps turn the table and use buffer-chars-modified-tick is the
> > primary method of discovering changes, not as fallback.
> > ...
> >> The only assumption I had it that Emacs does not frequently change
> >> buffer text without triggering modification hooks. Clearly, the
> >> assumption was wrong.
> >
> > I meant the assumptions about what buffer-chars-modified-tick does and
> > what its value means.
> 
> It seems that we have some misunderstanding here. Org does not care
> about the value of buffer-chars-modified-tick - just whether
> buffer-chars-modified-tick is changed or not (see the above).

But if buffer-modified-tick completely explains the change in
buffer-chars-modified-tick, you can conclude that
buffer-chars-modified-tick didn't change for your purposes, and then
all's well, no?

> Even if Org used the value of buffer-chars-modified-tick, it would not
> be useful. There is no information about buffer region where the edits
> happened if we just look at buffer-chars-modified-tick. AFAIK, only
> after-change-functions have access to the changed region bounds.

So what does Org do if the modification hooks were not called, and
buffer-chars-modified-tick says the text was changed?

> > But Org is not interested in just any moidification, AFAIU.  It is
> > only interested in modifications that change the buffer text.  Isn't
> > that true?  Or what else is Org interested in for this purpose.
> 
> You are right. Org is interested in modifications that change buffer
> text. Also, Org is interested to be not affected by
> inhibit-modification-hooks.

Then maybe this is the missing infrastructure you'd like to see
implemented.

> > The Emacs code related to tree-sitter already uses change indications,
> > and AFAIR didn't require any changes to the existing infrastructure.
> > I wonder why Org cannot settle with what we have, if your needs are
> > similar enough to those of tree-sitter.
> 
> I just checked https://github.com/casouri/emacs/blob/ts/src/insdel.c
> That ts Emacs branch directly modifies C-level Emacs buffer editing
> primitives (see the calls to ts_record_change). So, it is not affected
> by inhibit-modification-hooks.

That's what I meant by "existing infrastructure".

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Not exactly. Org uses modification hooks as the only mechanism to
>> process buffer changes because Org needs to know the region where the
>> buffer text changes. buffer-chars-modified-tick is used for error
>> detection - when buffer text is changed, but Org modification hooks are
>> not called for some reason. quail triggers false positives during the error
>> detection.
>
> Is that any different from what I said, which is that you need error
> detection only when the modification hooks are not called?  And that
> the quail behavior is only the issue when using this error detection,
> i.e. when the modification hooks are not called?

Your understanding is correct.

>> It seems that we have some misunderstanding here. Org does not care
>> about the value of buffer-chars-modified-tick - just whether
>> buffer-chars-modified-tick is changed or not (see the above).
>
> But if buffer-modified-tick completely explains the change in
> buffer-chars-modified-tick, you can conclude that
> buffer-chars-modified-tick didn't change for your purposes, and then
> all's well, no?

I looked into it again and tried to play with non-cyrillic input looking
at the values of buffer-chars-modified-tick and buffer-modified-tick.
You are right, there seems to be a special relation between the values
when I use non-latin input method
(buffer-chars-modified-tick=buffer-modified-tick). Thanks!

However, I am not sure if equality of the chars-modified-tick and
modified-tick is unique to non-changing edits. Can test in the wild
though.

> So what does Org do if the modification hooks were not called, and
> buffer-chars-modified-tick says the text was changed?

The cache is potentially invalid, so it is dropped altogether by
org-element-cache-reset.

>> > But Org is not interested in just any moidification, AFAIU.  It is
>> > only interested in modifications that change the buffer text.  Isn't
>> > that true?  Or what else is Org interested in for this purpose.
>> 
>> You are right. Org is interested in modifications that change buffer
>> text. Also, Org is interested to be not affected by
>> inhibit-modification-hooks.
>
> Then maybe this is the missing infrastructure you'd like to see
> implemented.

Yes, I think. In practical terms, it may something like a new pair of
hooks: before/after-change-no-inhibit-functions. The hooks work exactly
like before/after-change-functions, but cannot be suppressed by
let-binding inhibit-modification-hooks and
before/after-change-functions. If necessary they can still be explicitly
let-bound to nil, but it should be discouraged. WDYT?

Best,
Ihor

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: 51766@debbugs.gnu.org
> Date: Sat, 13 Nov 2021 22:43:17 +0800
> 
> > But if buffer-modified-tick completely explains the change in
> > buffer-chars-modified-tick, you can conclude that
> > buffer-chars-modified-tick didn't change for your purposes, and then
> > all's well, no?
> 
> I looked into it again and tried to play with non-cyrillic input looking
> at the values of buffer-chars-modified-tick and buffer-modified-tick.
> You are right, there seems to be a special relation between the values
> when I use non-latin input method
> (buffer-chars-modified-tick=buffer-modified-tick). Thanks!

That's what the implementation does: it copies the value from the
latter to the former.

> However, I am not sure if equality of the chars-modified-tick and
> modified-tick is unique to non-changing edits. Can test in the wild
> though.

I'd be surprised if the relation were violated.  But weirder things
have happened, so...

> > Then maybe this is the missing infrastructure you'd like to see
> > implemented.
> 
> Yes, I think. In practical terms, it may something like a new pair of
> hooks: before/after-change-no-inhibit-functions. The hooks work exactly
> like before/after-change-functions, but cannot be suppressed by
> let-binding inhibit-modification-hooks and
> before/after-change-functions. If necessary they can still be explicitly
> let-bound to nil, but it should be discouraged. WDYT?

Something like that, yes.  Just with shorter names ;-)

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> (let ((inhibit-modification-hooks t))
>   (insert "Insertion that will never trigger before/after-change-functions"))

This is a severely broken piece of code.  I don't think anyone should
try and handle this in any other way than by screaming bloody murder
when it detects the consequences.

> (defun org-element--after-change-function (...)
>  (setq org-element-chars-modified-tick (buffer-chars-modified-tick))
>  (org-element-cache-submit-request ...))
>
> (defun org-element--before-change-function (...)
>  (unless (eq org-element-chars-modified-tick (buffer-chars-modified-tick))
>      ;; Buffer has been changed without calling after-change-function
>      ;; and we have no way to determine which part of buffer has been changed.
>  ))

So this `unless` is intended to detect the case where we should scream
bloody murder, right?

Why do you need it?  AFAICT it should only be needed for debugging
purposes until the offender is found and shamed publicly.
[ I have a weird feeling that I might be one of the offenders.  ]

> Ideally, a way to track _all_ buffer modifications regardless of
> inhibit-modification-hooks would be useful.

I don't think this *can* exist: if we add a mechanism which ignores
`inhibit-modification-hooks` it will still need some way to ignore some
changes so we'll need another `inhibit-<foo>` variable to "silence"
those changes and we'll be back at square one.

I think the better way to proceed is to figure out why/when
significant changes are made while `inhibit-modification-hooks` is
non-nil, since that's the origin of your problems, AFAICT.


        Stefan

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  51766@debbugs.gnu.org
> Date: Thu, 16 Jun 2022 22:54:22 -0400
> 
> I think the better way to proceed is to figure out why/when
> significant changes are made while `inhibit-modification-hooks` is
> non-nil, since that's the origin of your problems, AFAICT.

I thought that was clear from the rest of the discussion: it's quail's
input methods that cause the issue, because quail tries to pretend
that just one character was inserted, when in fact the user could type
several characters.

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> (let ((inhibit-modification-hooks t))
>>   (insert "Insertion that will never trigger before/after-change-functions"))
>
> This is a severely broken piece of code.  I don't think anyone should
> try and handle this in any other way than by screaming bloody murder
> when it detects the consequences.

>> (defun org-element--before-change-function (...)
>>  (unless (eq org-element-chars-modified-tick (buffer-chars-modified-tick))
>>      ;; Buffer has been changed without calling after-change-function
>>      ;; and we have no way to determine which part of buffer has been changed.
>>  ))
>
> So this `unless` is intended to detect the case where we should scream
> bloody murder, right?
>
> Why do you need it?  AFAICT it should only be needed for debugging
> purposes until the offender is found and shamed publicly.
> [ I have a weird feeling that I might be one of the offenders.  ]
>
>> Ideally, a way to track _all_ buffer modifications regardless of
>> inhibit-modification-hooks would be useful.

Well. This was also my assumption. I initially implemented the checking
code to detect internal errors in Org.

Then we received 12 bug reports on this. The offenders were identified
in some cases. They were:
- polymode, valign, and ox-hugo
- Doom (unrelated to this particular issue, but Doom is let-binding major-mode...)
- Emacs quail and also replace-match in some cases (because of
  false-positives caused by changing buffer-chars-modified-tick)

ox-hugo and polymode fixed the issue.

valign relies on disabling modification hooks because it is otherwise
difficult to figure out pixel width of a string in current buffer:
https://github.com/casouri/valign/issues/30

In other cases, the offenders were hard to identify because of remote
debugging difficulty.

What I want to say is that the problem is more widespread than you may
think. And the consequences of missed modifications can cause very real
data corruption in Org files (some editing Org commands are relying on
cache being valid; if syntax boundaries are incorrect, the editing can
mess up the text). It must be avoided at all costs, regardless of the
recommended Elisp programming practices.

On top of the misbehaving third-party code, there is an issue described
in bug#46982. This discussion reminded me about
clone-indirect-buffer-hook, but it is only executed by
`clone-indirect-buffer', not by `make-indirect-buffer'. The latter is
used ubiquitously. See
https://github.com/search?q=make-indirect-buffer&type=code
Again, some unsuspecting users can experience data corruption just
because someone carelessly uses `make-indirect-buffer' in the code.

> I don't think this *can* exist: if we add a mechanism which ignores
> `inhibit-modification-hooks` it will still need some way to ignore some
> changes so we'll need another `inhibit-<foo>` variable to "silence"
> those changes and we'll be back at square one.
>
> I think the better way to proceed is to figure out why/when
> significant changes are made while `inhibit-modification-hooks` is
> non-nil, since that's the origin of your problems, AFAICT.

See the above. There is real-world code doing things that make
Emacs ignore after-change-functions.

Combined with the issue revealed in this bug report, I am left with no
Emacs tools to handle the problematic buffer modifications.

Best,
Ihor

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  51766@debbugs.gnu.org
> Date: Fri, 17 Jun 2022 18:05:36 +0800
> 
> Combined with the issue revealed in this bug report, I am left with no
> Emacs tools to handle the problematic buffer modifications.

You aren't supposed to try to do that in Lisp.  I suggest to describe
a generalization of the use cases you are aware of which you think
need this, and then we could think about implementing some or all of
it in C.

> valign relies on disabling modification hooks because it is otherwise
> difficult to figure out pixel width of a string in current buffer:
> https://github.com/casouri/valign/issues/30

That discussion is very short and lacking in detail, but up front, why
doesn't valign use the primitives we provide for determining the pixel
width of a string?

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii [2022-06-17 08:36:37] wrote:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  51766@debbugs.gnu.org
>> Date: Thu, 16 Jun 2022 22:54:22 -0400
>> 
>> I think the better way to proceed is to figure out why/when
>> significant changes are made while `inhibit-modification-hooks` is
>> non-nil, since that's the origin of your problems, AFAICT.
>
> I thought that was clear from the rest of the discussion: it's quail's
> input methods that cause the issue, because quail tries to pretend
> that just one character was inserted, when in fact the user could type
> several characters.

AFAIK in the case of Quail the char-modified-ticks changes (so there's
some insertions/deletions going on) while `inhibit-modification-hooks`
is set, but the state of the buffer at the next
`before-change-functions` is correct, e.g. the buffer-hash is unchanged.

IOW in the cse of Quail the Org mode code doesn't need to flush the
whole parser's state, which means that the code that flushes the parser
state when char-modified-ticks is modified silently was written to
defend against *other* problems.


        Stefan

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Well.  This was also my assumption.  I initially implemented the checking
> code to detect internal errors in Org.
>
> Then we received 12 bug reports on this. The offenders were identified
> in some cases. They were:
> - polymode, valign, and ox-hugo
> - Doom (unrelated to this particular issue, but Doom is let-binding major-mode...)
> - Emacs quail and also replace-match in some cases (because of
>   false-positives caused by changing buffer-chars-modified-tick)

IIUC Quail and replace-match shouldn't be in the above list: they're
caught by your debugging check but they're false positives.

> valign relies on disabling modification hooks because it is otherwise
> difficult to figure out pixel width of a string in current buffer:
> https://github.com/casouri/valign/issues/30

AFAIK this is *also* a false positive: the buffer is only temporarily
modified within the `with-silent-modifications` and reverted to its
previous state before the end of that macro, so there's no need to flush
your parser's state.

>> I don't think this *can* exist: if we add a mechanism which ignores
>> `inhibit-modification-hooks` it will still need some way to ignore some
>> changes so we'll need another `inhibit-<foo>` variable to "silence"
>> those changes and we'll be back at square one.
>>
>> I think the better way to proceed is to figure out why/when
>> significant changes are made while `inhibit-modification-hooks` is
>> non-nil, since that's the origin of your problems, AFAICT.
>
> See the above. There is real-world code doing things that make
> Emacs ignore after-change-functions.

I don't see how this relates to what I'm saying: what I'm saying is that
for the same reason there's code that has very valid reasons to inhibit
`after-change-functions`, there will be code that has very valid reasons
to inhibit some new `after-really-every-change-functions`, and then
there will inevitably also be code that abuses this.

The only real solution is to "push back" and get those abuses fixed.

One thing you could do, for example is replace your char-modified-tick
check with one based on buffer-size: it won't catch all cases, but it
won't suffer from false positives, so you can really scream bloody
murder when it happens.


        Stefan

bug#51766: string-pixel-width limitations (was: bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> valign relies on disabling modification hooks because it is otherwise
>> difficult to figure out pixel width of a string in current buffer:
>> https://github.com/casouri/valign/issues/30
>
> That discussion is very short and lacking in detail, but up front, why
> doesn't valign use the primitives we provide for determining the pixel
> width of a string?

Because string width in different buffers may be different depending on
the fontification, frame font size, face remapping,
wrap-prefix/line-prefix string properties (AFAIK, the built-in
string-pixel-width will return incorrect value on string with such
properties), invisibility specs in the buffer, line numbers mode, etc
We have implemented a number of workarounds in org-string-width on main,
but I am not 100% sure that I covered all the edge cases.

The most accurate way to measure the real string width inside current
buffer is actually inserting it and requesting the measurement.

Best,
Ihor

bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> See the above. There is real-world code doing things that make
>> Emacs ignore after-change-functions.
>
> I don't see how this relates to what I'm saying: what I'm saying is that
> for the same reason there's code that has very valid reasons to inhibit
> `after-change-functions`, there will be code that has very valid reasons
> to inhibit some new `after-really-every-change-functions`, and then
> there will inevitably also be code that abuses this.
>
> The only real solution is to "push back" and get those abuses fixed.
>
> One thing you could do, for example is replace your char-modified-tick
> check with one based on buffer-size: it won't catch all cases, but it
> won't suffer from false positives, so you can really scream bloody
> murder when it happens.

Checking the buffer-size is a great idea. Thanks!
It should be reliable enough for Org purposes.

Best,
Ihor

bug#51766: string-pixel-width limitations (was: bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: monnier@iro.umontreal.ca,  51766@debbugs.gnu.org
> Date: Tue, 21 Jun 2022 12:13:09 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That discussion is very short and lacking in detail, but up front, why
> > doesn't valign use the primitives we provide for determining the pixel
> > width of a string?
> 
> Because string width in different buffers may be different depending on
> the fontification, frame font size, face remapping,
> wrap-prefix/line-prefix string properties (AFAIK, the built-in
> string-pixel-width will return incorrect value on string with such
> properties), invisibility specs in the buffer, line numbers mode, etc
> We have implemented a number of workarounds in org-string-width on main,
> but I am not 100% sure that I covered all the edge cases.

If you need such high accuracy, may I suggest window-text-pixel-size?

bug#51766: string-pixel-width limitations (was: bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > That discussion is very short and lacking in detail, but up front, why
>> > doesn't valign use the primitives we provide for determining the pixel
>> > width of a string?
>> 
>> Because string width in different buffers may be different depending on
>> the fontification, frame font size, face remapping,
>> wrap-prefix/line-prefix string properties (AFAIK, the built-in
>> string-pixel-width will return incorrect value on string with such
>> properties), invisibility specs in the buffer, line numbers mode, etc
>> We have implemented a number of workarounds in org-string-width on main,
>> but I am not 100% sure that I covered all the edge cases.
>
> If you need such high accuracy, may I suggest window-text-pixel-size?

window-text-pixel-size suffers from the same issues with
wrap-prefix/line-prefix and line numbers mode.

Also, in order to use it in current buffer on not-yet-inserted string,
you need to insert it. That where the issue in valign originated from.

Best,
Ihor

bug#51766: string-pixel-width limitations (was: bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: monnier@iro.umontreal.ca,  51766@debbugs.gnu.org
> Date: Tue, 21 Jun 2022 19:00:34 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > That discussion is very short and lacking in detail, but up front, why
> >> > doesn't valign use the primitives we provide for determining the pixel
> >> > width of a string?
> >> 
> >> Because string width in different buffers may be different depending on
> >> the fontification, frame font size, face remapping,
> >> wrap-prefix/line-prefix string properties (AFAIK, the built-in
> >> string-pixel-width will return incorrect value on string with such
> >> properties), invisibility specs in the buffer, line numbers mode, etc
> >> We have implemented a number of workarounds in org-string-width on main,
> >> but I am not 100% sure that I covered all the edge cases.
> >
> > If you need such high accuracy, may I suggest window-text-pixel-size?
> 
> window-text-pixel-size suffers from the same issues with
> wrap-prefix/line-prefix and line numbers mode.

What issue are those?

> Also, in order to use it in current buffer on not-yet-inserted string,
> you need to insert it. That where the issue in valign originated from.

The usual method is to use a temporary buffer.

bug#51766: string-pixel-width limitations (was: bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> Because string width in different buffers may be different depending on
>> >> the fontification, frame font size, face remapping,
>> >> wrap-prefix/line-prefix string properties (AFAIK, the built-in
>> >> string-pixel-width will return incorrect value on string with such
>> >> properties), invisibility specs in the buffer, line numbers mode, etc
>> >> We have implemented a number of workarounds in org-string-width on main,
>> >> but I am not 100% sure that I covered all the edge cases.
>> >
>> > If you need such high accuracy, may I suggest window-text-pixel-size?
>> 
>> window-text-pixel-size suffers from the same issues with
>> wrap-prefix/line-prefix and line numbers mode.
>
> What issue are those?

The length of line-prefix is added to the return value of
window-text-pixel-size, which makes it wrong when the intention is
measuring the actual string width.

Not that window-text-pixel-size is misbehaving here - line-prefix does
need to be included if the intention is to query the actual full width
of text in buffer. The same goes for line numbers mode - line numbers
are a part of window size.

However, using window-text-pixel-size becomes awkward as the means to
measure expected string width when it is going to be inserted into
current buffer.

>> Also, in order to use it in current buffer on not-yet-inserted string,
>> you need to insert it. That where the issue in valign originated from.
>
> The usual method is to use a temporary buffer.

Yes, and it is not accurate because temporary buffer may not have the
same local environment for face remapping, invisibility specs,
char-property-alias, etc

To show all the trickery, let me share org-string-width I had to
implement for the purposes of Org mode. I did not want this function to
be complex and every single extra LOC there is fixing some edge case,
test failure, or bug report:

(defun org-string-width (string &optional pixels)
  "Return width of STRING when displayed in the current buffer.
Return width in pixels when PIXELS is non-nil."
  (if (and (version< emacs-version "28") (not pixels))
      ;; FIXME: Fallback to old limited version, because
      ;; `window-pixel-width' is buggy in older Emacs.
      (org--string-width-1 string)
    ;; Wrap/line prefix will make `window-text-pizel-size' return too
    ;; large value including the prefix.
    (remove-text-properties 0 (length string)
                            '(wrap-prefix t line-prefix t)
                            string)
    ;; Face should be removed to make sure that all the string symbols
    ;; are using default face with constant width.  Constant char width
    ;; is critical to get right string width from pixel width (not needed
    ;; when PIXELS are requested though).
    (unless pixels
      (remove-text-properties 0 (length string) '(face t) string))
    (let (;; We need to remove the folds to make sure that folded table
          ;; alignment is not messed up.
          (current-invisibility-spec
           (or (and (not (listp buffer-invisibility-spec))
                    buffer-invisibility-spec)
               (let (result)
                 (dolist (el buffer-invisibility-spec)
                   (unless (or (memq el
                                     '(org-fold-drawer
                                       org-fold-block
                                       org-fold-outline))
                               (and (listp el)
                                    (memq (car el)
                                          '(org-fold-drawer
                                            org-fold-block
                                            org-fold-outline))))
                     (push el result)))
                 result)))
          (current-char-property-alias-alist char-property-alias-alist))
      (with-temp-buffer
        (setq-local display-line-numbers nil)
        (setq-local buffer-invisibility-spec
                    (if (listp current-invisibility-spec)
                        (mapcar (lambda (el)
                                  ;; Consider elipsis to have 0 width.
                                  ;; It is what Emacs 28+ does, but we have
                                  ;; to force it in earlier Emacs versions.
                                  (if (and (consp el) (cdr el))
                                      (list (car el))
                                    el))
                                current-invisibility-spec)
                      current-invisibility-spec))
        (setq-local char-property-alias-alist
                    current-char-property-alias-alist)
        (let (pixel-width symbol-width)
          (with-silent-modifications
            (setf (buffer-string) string)
            (setq pixel-width
                  (if (get-buffer-window (current-buffer))
                      (car (window-text-pixel-size
                            nil (line-beginning-position) (point-max)))
                    (set-window-buffer nil (current-buffer))
                    (car (window-text-pixel-size
                          nil (line-beginning-position) (point-max)))))
            (unless pixels
              (setf (buffer-string) "a")
              (setq symbol-width
                    (if (get-buffer-window (current-buffer))
                        (car (window-text-pixel-size
                              nil (line-beginning-position) (point-max)))
                      (set-window-buffer nil (current-buffer))
                      (car (window-text-pixel-size
                            nil (line-beginning-position) (point-max)))))))
          (if pixels
              pixel-width
            (/ pixel-width symbol-width)))))))

Best,
Ihor

bug#51766: string-pixel-width limitations (was: bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: monnier@iro.umontreal.ca,  51766@debbugs.gnu.org
> Date: Tue, 21 Jun 2022 20:39:10 +0800
> 
> >> > If you need such high accuracy, may I suggest window-text-pixel-size?
> >> 
> >> window-text-pixel-size suffers from the same issues with
> >> wrap-prefix/line-prefix and line numbers mode.
> >
> > What issue are those?
> [...]
> To show all the trickery, let me share org-string-width I had to
> implement for the purposes of Org mode. I did not want this function to
> be complex and every single extra LOC there is fixing some edge case,
> test failure, or bug report:

Ah, so you do use window-text-pixel-size...  Then we are in violent
agreement.

Anyway, the beginning of this sub-thread, specifically about valign,
was in the context of Lisp programs that do buffer modifications under
with-silent-modifications or equivalent, and valign seems to do that
because it just needs to measure the pixel width of a string, and it
does that by inserting the string and then removing it.  So in that
case, the "buffer modifications" are indeed null and void, and Org
shouldn't be bothered by such "modifications", because the buffer
really remains unmodified.  Right?

bug#51766: string-pixel-width limitations (was: bug#51766: 29.0.50; Return value of buffer-chars-modified-tick changes when buffer text is not yet changed before inserting a character for non-latin input methods)

[Ihor Radchenko]

Note that I changed the topic in this particular branch. My aim here is
to make you aware about the issues with current Emacs tools to measure
pixel-precise string width. Maybe, things can be improved on Emacs side
in this regard.

The below is going back to the initial topic.

Eli Zaretskii <eliz@gnu.org> writes:

> Anyway, the beginning of this sub-thread, specifically about valign,
> was in the context of Lisp programs that do buffer modifications under
> with-silent-modifications or equivalent, and valign seems to do that
> because it just needs to measure the pixel width of a string, and it
> does that by inserting the string and then removing it.  So in that
> case, the "buffer modifications" are indeed null and void, and Org
> shouldn't be bothered by such "modifications", because the buffer
> really remains unmodified.  Right?

In short, you are right. To clarify the problem on my side goes like:

1. Org has a real issue with bad third-party code inhibiting
   before/after-change function + modifications in indirect buffers not
   always triggering before/after-change
2. Because the issue is critical and can cause data corruption, we
   cannot just ignore it
3. The first attempt to detect "stealthy" modifications was using
   buffer-chars-modified-tick
4. But this method is not reliable because (a) quail does some legit
   edits under inhibit-modification-hooks; (b) some other code, like
   valign also does legit edits under inhibit-modification-hooks
   These buffer modifications are harmless from Org perspective.
5. However, We end up with numerous false-positives using (3) and I am
   clueless how to reliably detect or work around harmful "stealthy"
   edits

   - The suggestion to compare buffer size is helpful, but not 100%
     reliable
   - My other idea to request before/after-change function variants are
     too specific to the problem at hand and may be not good for Emacs
     in a whole

In any case, bug#51766 should not be considered a bug because quail does
modify the buffer and changes in buffer-chars-modified-tick are legit.

Best,
Ihor
   

bug#51766: string-pixel-width limitations

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> 1. Org has a real issue with bad third-party code inhibiting
>    before/after-change function + modifications in indirect buffers not
>    always triggering before/after-change
> 2. Because the issue is critical and can cause data corruption, we
>    cannot just ignore it

The bug is not yours.  You don't have to ignore it, but you don't have
to fix it either.

>    - The suggestion to compare buffer size is helpful, but not 100%
>      reliable

To the extent that this just helps to detect other people's bug, I don't
think it needs to be 100%.


        Stefan