bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Mitchell]

[-- Attachment #1: Type: text/plain, Size: 24147 bytes --]

I recently upgraded from emacs 28.2 to 29.3. When I resumed working in a
large 11MB org-mode file that never gave me problems with speed on 28.2, I
noticed a severe slowdown when editing the buffer after invoking
`counsel-outline`. It took a while for me to isolate the issue, but it
seems to be caused by the many markers that `counsel-outline` creates in
the buffer. After invoking the command, every subsequent edit to the buffer
renders much slower on screen than it did in 28.2.

This problem is not only caused by `counsel-outline`, but other functions
that create a lot of markers in the buffer, like `org-refile` if
`org-refile-use-cache` is set to `t`. I tried upgrading my system to emacs
30.0.50, which I've used to generate this bug report, but the issue
persists.

Here are the steps to reproduce the issue:

1. Start emacs 29.3 (or 30.0.50) with emacs -q.
2. Paste the minimal config at
https://gist.github.com/kings2u/30b7a22dfa38fe0d5dc18adaf0e5e9de into the
scratch buffer and eval-buffer. (Note on my system I'm using the `counsel`
version current as of counsel-20240502.743, but any recent version will
reproduce the bug).
3. Open the following big file in org-mode:
https://gist.github.com/kings2u/c123a30de7e507a153a9500f03f08a9e
4. `goto-line` 35.
5. `M-x counsel-outline` and simply press enter to navigate point to the
headline at line 34.
6. Go to the end of the line and press `enter` to start typing on a new
line at line 35.
7. Start typing `n` `space`, or `t` `space` several times very fast and
observe how long it takes for the abbrev expansions (defined in the minimal
config above) to show up on the screen.

To verify this slowdown doesn’t occur before invoking `counsel-outline`,
when all the markers in the buffer were created, you can try the steps
again beginning from Step 1 but after Step 4, jump to Step 7 and see how
fast the abbrevs expand. You can also repeat all Steps 1 through 7 above
with emacs -q using emacs 28.2 and see that no slowdown occurs at Step 7.

Using emacs 29+, when I use `profiler-start` (CPU) between Steps 4 and 5,
and then `profiler-stop` after Step 7, I get a very big entry (e.g., 70%)
for `redisplay_internal (C function)` in the `profiler-report`. Using the
`profiler` at the same points using emacs 28.2, I don’t get a big entry for
`redisplay_internal (C function)`.

This slowdown during editing only seems noticeable with larger files, but
they don’t have to be 11MB like the first test file. I still notice a
half-second difference at least between emacs 29+ and emacs 28.2 when in
Step 3 I instead use the 5MB file at
https://gist.github.com/kings2u/7f607bd1069fbe92d611de70b39b936c. It wasn’t
until I reduced the test file to about 2.5MB that I saw emacs 29+
performing edits at the same speed as emacs 28.2 in Step 7 (see this
smallest file at:
https://gist.github.com/kings2u/2f76d30cea4f12c1bb90fe4f3a90e36e).

I’ve tried many different setups to avoid the problem, including different
versions of org-mode, but the only thing that solves it is going back to
emacs 28.2, which apparently handles markers more efficiently than 29+. The
only way to solve the problem while staying in emacs 29.3 that I’ve found
is to modify `counsel-outline` to not generate markers. (This solution was
suggested at https://github.com/abo-abo/swiper/pull/1700. The specific
modifications I use can be found at
https://gist.github.com/kings2u/5a8acbf0986f0848be66169d2dba7260). For what
it’s worth, I have installed emacs 29.3+ on multiple different machines and
observed the exact same behavior.

My big 11MB org file is such an integral part of my workflow that I've gone
back to emacs 28.2 for now with the hopes that the current version of emacs
will go back to handling markers efficiently. Please let me know if I can
help further in diagnosing the problem.

Thank you for all you do!


In GNU Emacs 30.0.50 (build 1, x86_64-w64-mingw32) of 2024-05-25 built
 on MITCHELL-16G-TO
Repository revision: 57dc1ed665d72bc58befa4853fa479b770fe4fcc
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 10.0.19045
System Description: Microsoft Windows 10 Home (v10.0.2009.19045.4412)

Configured using:
 'configure --prefix=/c/emacs --with-modules --without-dbus
 --with-native-compilation --without-compress-install --with-tree-sitter
 --with-mailutils --without-gconf --without-gsettings 'CFLAGS=-O2
 -mtune=native -march=native -fomit-frame-pointer''

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  global-tab-line-mode: t
  tab-line-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  abbrev-mode: t

Load-path shadows:
c:/Users/Mitchell/Emacs/.emacs.d/elpa/transient-20240509.1849/transient
hides c:/emacs/share/emacs/30.0.50/lisp/transient
c:/Users/Mitchell/Emacs/.emacs.d/elpa/bind-key-20230203.2004/bind-key hides
c:/emacs/share/emacs/30.0.50/lisp/bind-key
c:/Users/Mitchell/Emacs/.emacs.d/elpa/use-package-20230426.2324/use-package
hides c:/emacs/share/emacs/30.0.50/lisp/use-package/use-package
c:/Users/Mitchell/Emacs/.emacs.d/elpa/use-package-20230426.2324/use-package-lint
hides c:/emacs/share/emacs/30.0.50/lisp/use-package/use-package-lint
c:/Users/Mitchell/Emacs/.emacs.d/elpa/use-package-20230426.2324/use-package-jump
hides c:/emacs/share/emacs/30.0.50/lisp/use-package/use-package-jump
c:/Users/Mitchell/Emacs/.emacs.d/elpa/use-package-20230426.2324/use-package-ensure
hides c:/emacs/share/emacs/30.0.50/lisp/use-package/use-package-ensure
c:/Users/Mitchell/Emacs/.emacs.d/elpa/use-package-20230426.2324/use-package-diminish
hides c:/emacs/share/emacs/30.0.50/lisp/use-package/use-package-diminish
c:/Users/Mitchell/Emacs/.emacs.d/elpa/use-package-20230426.2324/use-package-delight
hides c:/emacs/share/emacs/30.0.50/lisp/use-package/use-package-delight
c:/Users/Mitchell/Emacs/.emacs.d/elpa/use-package-20230426.2324/use-package-core
hides c:/emacs/share/emacs/30.0.50/lisp/use-package/use-package-core
c:/Users/Mitchell/Emacs/.emacs.d/elpa/use-package-20230426.2324/use-package-bind-key
hides c:/emacs/share/emacs/30.0.50/lisp/use-package/use-package-bind-key
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox-texinfo hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox-texinfo
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox-publish hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox-publish
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox-org hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox-org
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox-odt hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox-odt
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox-md hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox-md
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox-man hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox-man
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox-latex hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox-latex
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox-koma-letter hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox-koma-letter
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox-icalendar hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox-icalendar
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox-html hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox-html
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox-beamer hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox-beamer
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ox-ascii hides
c:/emacs/share/emacs/30.0.50/lisp/org/ox-ascii
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org hides
c:/emacs/share/emacs/30.0.50/lisp/org/org
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-version hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-version
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-timer hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-timer
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-tempo hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-tempo
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-table hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-table
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-src hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-src
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-refile hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-refile
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-protocol hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-protocol
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-plot hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-plot
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-persist hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-persist
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-pcomplete hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-pcomplete
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-num hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-num
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-mouse hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-mouse
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-mobile hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-mobile
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-macs hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-macs
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-macro hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-macro
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-loaddefs hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-loaddefs
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-list hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-list
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-lint hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-lint
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-keys hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-keys
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-inlinetask hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-inlinetask
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-indent hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-indent
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-id hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-id
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-habit hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-habit
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-goto hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-goto
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-footnote hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-footnote
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-fold hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-fold
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-fold-core hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-fold-core
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-feed hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-feed
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-faces hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-faces
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-entities hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-entities
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-element hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-element
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-duration hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-duration
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-datetree hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-datetree
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-cycle hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-cycle
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-ctags hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-ctags
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-crypt hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-crypt
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-compat hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-compat
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-colview hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-colview
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-clock hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-clock
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-capture hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-capture
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-attach hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-attach
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-attach-git hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-attach-git
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-archive hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-archive
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/org-agenda hides
c:/emacs/share/emacs/30.0.50/lisp/org/org-agenda
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-w3m hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-w3m
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-rmail hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-rmail
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-mhe hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-mhe
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-man hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-man
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-irc hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-irc
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-info hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-info
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-gnus hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-gnus
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-eww hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-eww
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-eshell hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-eshell
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-doi hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-doi
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-docview hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-docview
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-bibtex hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-bibtex
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ol-bbdb hides
c:/emacs/share/emacs/30.0.50/lisp/org/ol-bbdb
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/oc hides
c:/emacs/share/emacs/30.0.50/lisp/org/oc
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/oc-natbib hides
c:/emacs/share/emacs/30.0.50/lisp/org/oc-natbib
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/oc-csl hides
c:/emacs/share/emacs/30.0.50/lisp/org/oc-csl
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/oc-bibtex hides
c:/emacs/share/emacs/30.0.50/lisp/org/oc-bibtex
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/oc-biblatex hides
c:/emacs/share/emacs/30.0.50/lisp/org/oc-biblatex
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/oc-basic hides
c:/emacs/share/emacs/30.0.50/lisp/org/oc-basic
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-tangle hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-tangle
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-table hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-table
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-sqlite hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-sqlite
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-sql hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-sql
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-shell hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-shell
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-sed hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-sed
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-screen hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-screen
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-scheme hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-scheme
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-sass hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-sass
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-ruby hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-ruby
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-ref hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-ref
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-R hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-R
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-python hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-python
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-processing hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-processing
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-plantuml hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-plantuml
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-perl hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-perl
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-org hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-org
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-octave hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-octave
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-ocaml hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-ocaml
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-maxima hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-maxima
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-matlab hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-matlab
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-makefile hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-makefile
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-lua hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-lua
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-lob hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-lob
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-lisp hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-lisp
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-lilypond hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-lilypond
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-latex hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-latex
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-julia hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-julia
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-js hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-js
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-java hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-java
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-haskell hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-haskell
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-groovy hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-groovy
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-gnuplot hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-gnuplot
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-fortran hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-fortran
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-forth hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-forth
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-exp hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-exp
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-eval hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-eval
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-eshell hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-eshell
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-emacs-lisp hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-emacs-lisp
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-dot hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-dot
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-ditaa hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-ditaa
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-css hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-css
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-core hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-core
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-comint hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-comint
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-clojure hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-clojure
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-calc hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-calc
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-C hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-C
c:/Users/Mitchell/Emacs/.emacs.d/elpa/org-9.6.30/ob-awk hides
c:/emacs/share/emacs/30.0.50/lisp/org/ob-awk

Features:
(shadow sort mail-extr emacsbug message yank-media puny rfc822 mml
mml-sec epa derived gnus-util mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mail-utils org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-src ob-comint org-pcomplete pcomplete
org-list org-footnote org-faces org-entities time-date noutline outline
ob-emacs-lisp ob-core ob-eval org-cycle org-table ol rx org-fold
org-fold-core org-keys oc org-loaddefs find-func cal-menu calendar
cal-loaddefs org-version org-compat org-macs format-spec counsel xdg
xref project dired dired-loaddefs compile text-property-search comint
ansi-osc ansi-color swiper ivy delsel ring ivy-faces ivy-overlay colir
color modus-vivendi-theme modus-themes cl-extra help-mode tab-line
use-package use-package-ensure use-package-delight use-package-diminish
use-package-bind-key bind-key use-package-core finder-inf easy-mmode epg
rfc6068 epg-config gnu-elpa-keyring-update pdf-tools-autoloads
tablist-autoloads info package browse-url url url-proxy url-privacy
url-expand url-methods url-history url-cookie generate-lisp-file
url-domsuf url-util mailcap url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs password-cache json subr-x map byte-opt gv
bytecomp byte-compile url-vars cus-edit pp cus-start cus-load icons
wid-edit cl-loaddefs cl-lib rmc iso-transl tooltip cconv eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
dos-w32 ls-lisp disp-table term/w32-win w32-win w32-vars term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs theme-loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads
w32notify w32 lcms2 multi-tty move-toolbar make-network-process
native-compile emacs)

Memory information:
((conses 16 476739 29195) (symbols 48 26493 0)
 (strings 32 133104 6737) (string-bytes 1 3784773) (vectors 16 33874)
 (vector-slots 8 430476 21000) (floats 8 269 15) (intervals 56 554 0)
 (buffers 992 12))

[-- Attachment #2: Type: text/html, Size: 25166 bytes --]

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> From: Mitchell <mitchellahren@gmail.com>
> Date: Tue, 18 Jun 2024 23:25:18 -0600
> 
> I recently upgraded from emacs 28.2 to 29.3. When I resumed working in a large 11MB org-mode file that
> never gave me problems with speed on 28.2, I noticed a severe slowdown when editing the buffer after
> invoking `counsel-outline`. It took a while for me to isolate the issue, but it seems to be caused by the many
> markers that `counsel-outline` creates in the buffer. After invoking the command, every subsequent edit to the
> buffer renders much slower on screen than it did in 28.2.
> 
> This problem is not only caused by `counsel-outline`, but other functions that create a lot of markers in the
> buffer, like `org-refile` if `org-refile-use-cache` is set to `t`. I tried upgrading my system to emacs 30.0.50,
> which I've used to generate this bug report, but the issue persists. 

I cannot reproduce this on my system, with the current master branch,
i.e. Emacs 30.  I _can_ reproduce this in Emacs 29.3, where indeed
expanding any of the two abbrevs takes about 0.5 sec to show on the
screen.  But in Emacs 30 it's instantaneous, even though I used an
unoptimized build of Emacs 30.  Maybe the reason is that Emacs 30 now
uses Org 9.7.4, whereas you have a local installation of 9.6.30?

Why did you decide the problem was due to markers?  I see a single
change in the implementation of markers in Emacs 29 as compared to
Emacs 28, and no changes at all in Emacs 30 (except one that affects
the Android, I think).

> Using emacs 29+, when I use `profiler-start` (CPU) between Steps 4 and 5, and then `profiler-stop` after Step
> 7, I get a very big entry (e.g., 70%) for `redisplay_internal (C function)` in the `profiler-report`. Using the
> `profiler` at the same points using emacs 28.2, I don’t get a big entry for `redisplay_internal (C function)`.

This doesn't necessarily mean the problem is due to the markers (but
doesn't refute that, either).

I see 3400 markers created by counsel-outline in this case, which is
not too many, IMO.

> Here are the steps to reproduce the issue:
> 
> 1. Start emacs 29.3 (or 30.0.50) with emacs -q.
> 2. Paste the minimal config at https://gist.github.com/kings2u/30b7a22dfa38fe0d5dc18adaf0e5e9de into the
> scratch buffer and eval-buffer. (Note on my system I'm using the `counsel` version current as of
> counsel-20240502.743, but any recent version will reproduce the bug).
> 3. Open the following big file in org-mode:
> https://gist.github.com/kings2u/c123a30de7e507a153a9500f03f08a9e
> 4. `goto-line` 35.
> 5. `M-x counsel-outline` and simply press enter to navigate point to the headline at line 34.
> 6. Go to the end of the line and press `enter` to start typing on a new line at line 35.
> 7. Start typing `n` `space`, or `t` `space` several times very fast and observe how long it takes for the abbrev
> expansions (defined in the minimal config above) to show up on the screen.

I didn't want to install all those packages (counsel is just the tip
of the iceberg, it wants you to install ivy and swiper and whatnot),
so I simply unpacked their tarballs from ELPA, and did this instead of
steps 1 and 2 above:

  (add-to-list 'load-path "~/foo/ivy-0.14.2")
  (add-to-list 'load-path "~/foo/swiper-0.14.2")
  (setq-default abbrev-mode t)
  (setq save-abbrevs nil)
  (add-hook 'minibuffer-setup-hook (lambda () (setq-local abbrev-mode nil)))
  (define-abbrev global-abbrev-table "n" "and")
  (define-abbrev global-abbrev-table "t" "the")
  (define-abbrev global-abbrev-table "th" "that")

  M-x load-file RET ~/foo/counsel-0.14.2/counsel.el RET
  M-x eval-region RET

Then I did all the rest of steps.  As mentioned, I do see the slow
responses in Emacs 29.3, but not in the latest master branch of the
Emacs Git repository.

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> I cannot reproduce this on my system, with the current master branch,
> i.e. Emacs 30.  I _can_ reproduce this in Emacs 29.3, where indeed
> expanding any of the two abbrevs takes about 0.5 sec to show on the
> screen.  But in Emacs 30 it's instantaneous, even though I used an
> unoptimized build of Emacs 30.  Maybe the reason is that Emacs 30 now
> uses Org 9.7.4, whereas you have a local installation of 9.6.30?

What if you add
(setq org-fold-core-style 'text-properties) to your reproducer?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Mitchell <mitchellahren@gmail.com>, 71644@debbugs.gnu.org
> Date: Thu, 20 Jun 2024 13:49:10 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I cannot reproduce this on my system, with the current master branch,
> > i.e. Emacs 30.  I _can_ reproduce this in Emacs 29.3, where indeed
> > expanding any of the two abbrevs takes about 0.5 sec to show on the
> > screen.  But in Emacs 30 it's instantaneous, even though I used an
> > unoptimized build of Emacs 30.  Maybe the reason is that Emacs 30 now
> > uses Org 9.7.4, whereas you have a local installation of 9.6.30?
> 
> What if you add
> (setq org-fold-core-style 'text-properties) to your reproducer?

Then abbrevs don't work (not sure I understand why), so I cannot run
the recipe.

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> Cc: mitchellahren@gmail.com, 71644@debbugs.gnu.org
> Date: Thu, 20 Jun 2024 19:11:28 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Ihor Radchenko <yantar92@posteo.net>
> > Cc: Mitchell <mitchellahren@gmail.com>, 71644@debbugs.gnu.org
> > Date: Thu, 20 Jun 2024 13:49:10 +0000
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > > I cannot reproduce this on my system, with the current master branch,
> > > i.e. Emacs 30.  I _can_ reproduce this in Emacs 29.3, where indeed
> > > expanding any of the two abbrevs takes about 0.5 sec to show on the
> > > screen.  But in Emacs 30 it's instantaneous, even though I used an
> > > unoptimized build of Emacs 30.  Maybe the reason is that Emacs 30 now
> > > uses Org 9.7.4, whereas you have a local installation of 9.6.30?
> > 
> > What if you add
> > (setq org-fold-core-style 'text-properties) to your reproducer?
> 
> Then abbrevs don't work (not sure I understand why), so I cannot run
> the recipe.

Sorry, that was a cockpit error.  Now that I fixed that, I can report
that the setting you suggested doesn't change anything: I see no
delays in Emacs 30.

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Mitchell]

[-- Attachment #1: Type: text/plain, Size: 6025 bytes --]

Thank you for the response, Eli.

> Maybe the reason is that Emacs 30 now
> uses Org 9.7.4, whereas you have a local installation of 9.6.30?

Upgrading to Org 9.7.4 and reinstalling counsel did seem to improve the
abbrev expansion speed slightly on my Emacs 30.0.50 build, at least at
first. It’s still not nearly as snappy as 28.2 the longer I edit the
buffer. I will work on a reliable recipe to recreate this slowness.

> Why did you decide the problem was due to markers?  I see a single
> change in the implementation of markers in Emacs 29 as compared to
> Emacs 28, and no changes at all in Emacs 30 (except one that affects
> the Android, I think).

I thought it was markers because (1) overwriting a few functions in
counsel.el to be the versions at
https://gist.github.com/kings2u/5a8acbf0986f0848be66169d2dba7260 so that
counsel-outline cleaned up the markers it created caused the bug to go
away, and (2) the bug also exists when instead of counsel-outline at Step 5
I used org-refile (with `org-refile-use-cache` is set to `t`), and
`org-refile-use-cache` seems to create many markers in the buffer. But
maybe counsel-outline and org-refile are both doing something else that
cause the bug. (FWIW I did also notice that the abbrev expansions were
slightly quicker in Step 7 after using org-refile than after
counsel-outline...)

> I _can_ reproduce this in Emacs 29.3, where indeed
> expanding any of the two abbrevs takes about 0.5 sec to show on the
> screen.

Abbrev expansions on my machine take longer. When I re-create the bug with
counsel-outline, the abbrevs take over a second each to expand. Maybe your
hardware is faster than mine?

> I see 3400 markers created by counsel-outline in this case, which is
> not too many, IMO.

Interesting, maybe it isn’t markers after all. Can I ask how you count
markers? I couldn’t find anything in the docs or online.

On Wed, Jun 19, 2024 at 6:57 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Mitchell <mitchellahren@gmail.com>
> > Date: Tue, 18 Jun 2024 23:25:18 -0600
> >
> > I recently upgraded from emacs 28.2 to 29.3. When I resumed working in a
> large 11MB org-mode file that
> > never gave me problems with speed on 28.2, I noticed a severe slowdown
> when editing the buffer after
> > invoking `counsel-outline`. It took a while for me to isolate the issue,
> but it seems to be caused by the many
> > markers that `counsel-outline` creates in the buffer. After invoking the
> command, every subsequent edit to the
> > buffer renders much slower on screen than it did in 28.2.
> >
> > This problem is not only caused by `counsel-outline`, but other
> functions that create a lot of markers in the
> > buffer, like `org-refile` if `org-refile-use-cache` is set to `t`. I
> tried upgrading my system to emacs 30.0.50,
> > which I've used to generate this bug report, but the issue persists.
>
> I cannot reproduce this on my system, with the current master branch,
> i.e. Emacs 30.  I _can_ reproduce this in Emacs 29.3, where indeed
> expanding any of the two abbrevs takes about 0.5 sec to show on the
> screen.  But in Emacs 30 it's instantaneous, even though I used an
> unoptimized build of Emacs 30.  Maybe the reason is that Emacs 30 now
> uses Org 9.7.4, whereas you have a local installation of 9.6.30?
>
> Why did you decide the problem was due to markers?  I see a single
> change in the implementation of markers in Emacs 29 as compared to
> Emacs 28, and no changes at all in Emacs 30 (except one that affects
> the Android, I think).
>
> > Using emacs 29+, when I use `profiler-start` (CPU) between Steps 4 and
> 5, and then `profiler-stop` after Step
> > 7, I get a very big entry (e.g., 70%) for `redisplay_internal (C
> function)` in the `profiler-report`. Using the
> > `profiler` at the same points using emacs 28.2, I don’t get a big entry
> for `redisplay_internal (C function)`.
>
> This doesn't necessarily mean the problem is due to the markers (but
> doesn't refute that, either).
>
> I see 3400 markers created by counsel-outline in this case, which is
> not too many, IMO.
>
> > Here are the steps to reproduce the issue:
> >
> > 1. Start emacs 29.3 (or 30.0.50) with emacs -q.
> > 2. Paste the minimal config at
> https://gist.github.com/kings2u/30b7a22dfa38fe0d5dc18adaf0e5e9de into the
> > scratch buffer and eval-buffer. (Note on my system I'm using the
> `counsel` version current as of
> > counsel-20240502.743, but any recent version will reproduce the bug).
> > 3. Open the following big file in org-mode:
> > https://gist.github.com/kings2u/c123a30de7e507a153a9500f03f08a9e
> > 4. `goto-line` 35.
> > 5. `M-x counsel-outline` and simply press enter to navigate point to the
> headline at line 34.
> > 6. Go to the end of the line and press `enter` to start typing on a new
> line at line 35.
> > 7. Start typing `n` `space`, or `t` `space` several times very fast and
> observe how long it takes for the abbrev
> > expansions (defined in the minimal config above) to show up on the
> screen.
>
> I didn't want to install all those packages (counsel is just the tip
> of the iceberg, it wants you to install ivy and swiper and whatnot),
> so I simply unpacked their tarballs from ELPA, and did this instead of
> steps 1 and 2 above:
>
>   (add-to-list 'load-path "~/foo/ivy-0.14.2")
>   (add-to-list 'load-path "~/foo/swiper-0.14.2")
>   (setq-default abbrev-mode t)
>   (setq save-abbrevs nil)
>   (add-hook 'minibuffer-setup-hook (lambda () (setq-local abbrev-mode
> nil)))
>   (define-abbrev global-abbrev-table "n" "and")
>   (define-abbrev global-abbrev-table "t" "the")
>   (define-abbrev global-abbrev-table "th" "that")
>
>   M-x load-file RET ~/foo/counsel-0.14.2/counsel.el RET
>   M-x eval-region RET
>
> Then I did all the rest of steps.  As mentioned, I do see the slow
> responses in Emacs 29.3, but not in the latest master branch of the
> Emacs Git repository.
>

[-- Attachment #2: Type: text/html, Size: 7372 bytes --]

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> From: Mitchell <mitchellahren@gmail.com>
> Date: Thu, 20 Jun 2024 12:57:58 -0600
> Cc: 71644@debbugs.gnu.org
> 
> > Why did you decide the problem was due to markers?  I see a single
> > change in the implementation of markers in Emacs 29 as compared to
> > Emacs 28, and no changes at all in Emacs 30 (except one that affects
> > the Android, I think). 
> 
> I thought it was markers because (1) overwriting a few functions in counsel.el to be the versions at
> https://gist.github.com/kings2u/5a8acbf0986f0848be66169d2dba7260 so that counsel-outline cleaned up the
> markers it created caused the bug to go away, and (2) the bug also exists when instead of counsel-outline at
> Step 5 I used org-refile (with `org-refile-use-cache` is set to `t`), and `org-refile-use-cache` seems to create
> many markers in the buffer. But maybe counsel-outline and org-refile are both doing something else that
> cause the bug. (FWIW I did also notice that the abbrev expansions were slightly quicker in Step 7 after using
> org-refile than after counsel-outline...)

If you remove all the non-ASCII characters from the Org file, does the
slowdown go away?

> > I _can_ reproduce this in Emacs 29.3, where indeed
> > expanding any of the two abbrevs takes about 0.5 sec to show on the
> > screen.  
> 
> Abbrev expansions on my machine take longer. When I re-create the bug with counsel-outline, the abbrevs
> take over a second each to expand. Maybe your hardware is faster than mine?

Could be, but twice slower sounds too much to be explained by CPU
speed.

> > I see 3400 markers created by counsel-outline in this case, which is
> > not too many, IMO. 
> 
> Interesting, maybe it isn’t markers after all. Can I ask how you count markers? I couldn’t find anything in the
> docs or online.

There's a function count_markers in Emacs, it just isn't compiled
unless you compile with -DDEBUG_MARKERS=1.  So I compiled it and
called it from GDB.

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Mitchell]

[-- Attachment #1: Type: text/plain, Size: 2820 bytes --]

> If you remove all the non-ASCII characters from the Org file, does the
> slowdown go away?

Eli, that solved it! The new test file is at
https://gist.github.com/kings2u/2ef0e145f2b42d0a13605b0dc9b6e6e2. I
replaced every non-ASCII character with an "a" so the file still has the
same number of total characters, and in my emacs 30.0 50 build (as of
2024-05-25), doing Steps 1 to Step 7 gives me abbrev expansions that are as
lighting fast as in emacs 28.2!

So what now? Do you think you can solve what’s going on in the backend so
bigger buffers with markers and non-ASCII characters don’t exhibit this
slowness in the latest emacs?

Thank you again!

On Thu, Jun 20, 2024 at 1:04 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Mitchell <mitchellahren@gmail.com>
> > Date: Thu, 20 Jun 2024 12:57:58 -0600
> > Cc: 71644@debbugs.gnu.org
> >
> > > Why did you decide the problem was due to markers?  I see a single
> > > change in the implementation of markers in Emacs 29 as compared to
> > > Emacs 28, and no changes at all in Emacs 30 (except one that affects
> > > the Android, I think).
> >
> > I thought it was markers because (1) overwriting a few functions in
> counsel.el to be the versions at
> > https://gist.github.com/kings2u/5a8acbf0986f0848be66169d2dba7260 so
> that counsel-outline cleaned up the
> > markers it created caused the bug to go away, and (2) the bug also
> exists when instead of counsel-outline at
> > Step 5 I used org-refile (with `org-refile-use-cache` is set to `t`),
> and `org-refile-use-cache` seems to create
> > many markers in the buffer. But maybe counsel-outline and org-refile are
> both doing something else that
> > cause the bug. (FWIW I did also notice that the abbrev expansions were
> slightly quicker in Step 7 after using
> > org-refile than after counsel-outline...)
>
> If you remove all the non-ASCII characters from the Org file, does the
> slowdown go away?
>
> > > I _can_ reproduce this in Emacs 29.3, where indeed
> > > expanding any of the two abbrevs takes about 0.5 sec to show on the
> > > screen.
> >
> > Abbrev expansions on my machine take longer. When I re-create the bug
> with counsel-outline, the abbrevs
> > take over a second each to expand. Maybe your hardware is faster than
> mine?
>
> Could be, but twice slower sounds too much to be explained by CPU
> speed.
>
> > > I see 3400 markers created by counsel-outline in this case, which is
> > > not too many, IMO.
> >
> > Interesting, maybe it isn’t markers after all. Can I ask how you count
> markers? I couldn’t find anything in the
> > docs or online.
>
> There's a function count_markers in Emacs, it just isn't compiled
> unless you compile with -DDEBUG_MARKERS=1.  So I compiled it and
> called it from GDB.
>

[-- Attachment #2: Type: text/html, Size: 3995 bytes --]

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

Mitchell <mitchellahren@gmail.com> writes:

>> If you remove all the non-ASCII characters from the Org file, does the
>> slowdown go away?
>
> Eli, that solved it! The new test file is at
> https://gist.github.com/kings2u/2ef0e145f2b42d0a13605b0dc9b6e6e2. I
> replaced every non-ASCII character with an "a" so the file still has the
> same number of total characters, and in my emacs 30.0 50 build (as of
> 2024-05-25), doing Steps 1 to Step 7 gives me abbrev expansions that are as
> lighting fast as in emacs 28.2!

Hmm. Then, does the attached patch help?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-src-marker.c-buf_bytepos_to_charpos-Limit-marker-sea.patch --]
[-- Type: text/x-patch, Size: 1180 bytes --]

From 0bafd288faee8cae33fe4a122f6e3ac73ec10d60 Mon Sep 17 00:00:00 2001
Message-ID: <0bafd288faee8cae33fe4a122f6e3ac73ec10d60.1718950719.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sun, 23 Apr 2023 21:31:46 +0200
Subject: [PATCH] * src/marker.c (buf_bytepos_to_charpos): Limit marker search

Limit searching across buffer markers to first 50 markers and thus
avoid performance scaling with the number of markers.

I got 5x `re-search-forward' speed improvement in my setup with this
dumb change.
---
 src/marker.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/marker.c b/src/marker.c
index f016bf9c088..4d7d6621513 100644
--- a/src/marker.c
+++ b/src/marker.c
@@ -354,8 +354,10 @@ buf_bytepos_to_charpos (struct buffer *b, ptrdiff_t bytepos)
   if (b == cached_buffer && BUF_MODIFF (b) == cached_modiff)
     CONSIDER (cached_bytepos, cached_charpos);
 
-  for (tail = BUF_MARKERS (b); tail; tail = tail->next)
+  int i = 0;
+  for (tail = BUF_MARKERS (b); tail && i < 50; tail = tail->next)
     {
+      i++;
       CONSIDER (tail->bytepos, tail->charpos);
 
       /* If we are down to a range of 50 chars,
-- 
2.45.1


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>From 0bafd288faee8cae33fe4a122f6e3ac73ec10d60 Mon Sep 17 00:00:00 2001
> Message-ID: <0bafd288faee8cae33fe4a122f6e3ac73ec10d60.1718950719.git.yantar92@posteo.net>
> From: Ihor Radchenko <yantar92@posteo.net>
> Date: Sun, 23 Apr 2023 21:31:46 +0200
> Subject: [PATCH] * src/marker.c (buf_bytepos_to_charpos): Limit marker search
>
> Limit searching across buffer markers to first 50 markers and thus
> avoid performance scaling with the number of markers.
>
> I got 5x `re-search-forward' speed improvement in my setup with this
> dumb change.

Hmm... of course, there'd likely be other circumstances where it would
make it 5x slower.  E.g. in a large buffer, doing a forward search from
the middle of the buffer when the first 50 markers happen to all be near
the beginning of the buffer will mean that we always use the "slow path"
which scans all the bytes between PT and the position of interest to
count the number of chars therein.

BTW, we already stop after at most N/50 markers where N is the smallest
distance between the position of interest and point/bob/eob/gap (oh and
the position of the last conversion).  So it seems that in your test,
this distance N is >2500.  Also when that distance is >5000 we create
a new marker, so next time around that position should be at the
beginning of the marker-list.  So I wonder what happens in your test:
why do we jump "very far" (more than 2500) between each call to the
conversion function?  Also, maybe we should increase
BYTECHAR_DISTANCE_INCREMENT?  ]

This byte_to_char and char_to_byte conversion business is a real PITA.
[ In retrospect I wish we'd stuck to the Emacs-20.1 design where
  chars=bytes.  I know it introduced a lot of breakage (and it'd be even
  worse now), but it is The Right Thing™ if you can ignore backward
  compatibility.  ]

Using markers as a cheap cache of conversions was a cute hack but we
really need to replace it.

Some options that come to mind:

- Keep the tradition of abusing an existing data structure, and stash
  the bytepos info inside the overlay tree or the text properties.
  This way the conversion is bounded by O(log BUFFERSIZE).

- Use a dedicated data-structure.  E.g. a pair of array-with-a-gap
  (one indexed by BYTEPOS/STEP the other indexed by CHARPOS/STEP, where
  STEP would be a large enough constant to make those arrays cheap yet
  small enough that the remaining scan is cheap).
  This way the conversion is O(STEP), i.e. "constant-time".

BTW, among my various local hacks, I've been using the hack below, which
aims to randomize the order in our markers-list, so as to minimize the
risk that we have to wade through lots of markers all clumped around the
same position.  I don't think it does a good job of it, but maybe we can
improve the execution of this idea, tho it still doesn't help if there's
no GC involved.

BTW, if/when we use some other data-structure to convert bytes<->chars,
then we could presumably get rid of our markers-list and stash markers
inside our overlay tree (basically represent them as degenerate overlays
with beg==end and no properties).


        Stefan


diff --git a/src/alloc.c b/src/alloc.c
index 9304e4e42bb..07409e7cfc3 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -7836,10 +7514,22 @@ sweep_symbols (void)
 unchain_dead_markers (struct buffer *buffer)
 {
   struct Lisp_Marker *this, **prev = &BUF_MARKERS (buffer);
+  /* In order to try and avoid worst case behaviors in buf_charpos_to_bytepos
+     we try and randomize the order of markers here.  */
+  unsigned i = 4;
 
   while ((this = *prev))
     if (vectorlike_marked_p (&this->header))
-      prev = &this->next;
+      {
+        if (!shuffle_markers || i++ % 8)
+          prev = &this->next;
+        else
+          { /* Move this one to front, just to shuffle things a bit.  */
+            *prev = this->next;
+            this->next = BUF_MARKERS (buffer);
+            BUF_MARKERS (buffer) = this;
+          }
+      }
     else
       {
         this->buffer = NULL;

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Gerd Möllmann]

Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> [ In retrospect I wish we'd stuck to the Emacs-20.1 design where
>   chars=bytes.  I know it introduced a lot of breakage (and it'd be even
>   worse now), but it is The Right Thing™ if you can ignore backward
>   compatibility.  ]

(Wasn't that pre-20? I remember having a ton of problems with the "new"
redisplay when I ported it 19 to 20.)

+1

(And make character a type.)

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I got 5x `re-search-forward' speed improvement in my setup with this
>> dumb change.
>
> Hmm... of course, there'd likely be other circumstances where it would
> make it 5x slower.  E.g. in a large buffer, doing a forward search from
> the middle of the buffer when the first 50 markers happen to all be near
> the beginning of the buffer will mean that we always use the "slow path"
> which scans all the bytes between PT and the position of interest to
> count the number of chars therein.

Yup. I _do not_ propose my patch to be merged.
(BTW, we should probably merge this bug and bug#63040 where I first
shared this patch - just to demonstrate the problem and discuss possible
solutions)

> BTW, we already stop after at most N/50 markers where N is the smallest
> distance between the position of interest and point/bob/eob/gap (oh and
> the position of the last conversion).  So it seems that in your test,
> this distance N is >2500.  Also when that distance is >5000 we create
> a new marker, so next time around that position should be at the
> beginning of the marker-list.  So I wonder what happens in your test:
> why do we jump "very far" (more than 2500) between each call to the
> conversion function?  Also, maybe we should increase
> BYTECHAR_DISTANCE_INCREMENT?  ]

It is indeed another option.
Also, from bug#63040

    Another idea could be moving the cache markers into a separate
    array, so that we can examine them without mixing with all other
    buffer markers.

> Using markers as a cheap cache of conversions was a cute hack but we
> really need to replace it.
>
> Some options that come to mind:
>
> - Keep the tradition of abusing an existing data structure, and stash
>   the bytepos info inside the overlay tree or the text properties.
>   This way the conversion is bounded by O(log BUFFERSIZE).

For overlay tree, it might be even better to stash all the markers in
Emacs into itree structure. For now, every operation involving
adjusting/searching markers scales linearly - not ideal.

> - Use a dedicated data-structure.  E.g. a pair of array-with-a-gap
>   (one indexed by BYTEPOS/STEP the other indexed by CHARPOS/STEP, where
>   STEP would be a large enough constant to make those arrays cheap yet
>   small enough that the remaining scan is cheap).
>   This way the conversion is O(STEP), i.e. "constant-time".

I think that it will be less efficient compared to using a tree-like
structure (if we can manage to use it). Will it be easier?

> BTW, among my various local hacks, I've been using the hack below, which
> aims to randomize the order in our markers-list, so as to minimize the
> risk that we have to wade through lots of markers all clumped around the
> same position.  I don't think it does a good job of it, but maybe we can
> improve the execution of this idea, tho it still doesn't help if there's
> no GC involved.

I am not sure if I believe that this approach can yield practical gains.
AFAIU, the problem with the slowdown we are discussing here is markers
that are all around the same position. It's rather too many markers in
general, spaced not far from each other.

> BTW, if/when we use some other data-structure to convert bytes<->chars,
> then we could presumably get rid of our markers-list and stash markers
> inside our overlay tree (basically represent them as degenerate overlays
> with beg==end and no properties).

I am wondering why it is impossible to stash markers inside overlay tree
without doing anything special about bytes<->chars conversion (other
than changing the linear loop with itree query).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> (BTW, we should probably merge this bug and bug#63040 where I first
> shared this patch - just to demonstrate the problem and discuss possible
> solutions)

Ah, thanks, I'll take a look at that bug.

>     Another idea could be moving the cache markers into a separate
>     array, so that we can examine them without mixing with all other
>     buffer markers.

🙂

>> Using markers as a cheap cache of conversions was a cute hack but we
>> really need to replace it.
>>
>> Some options that come to mind:
>>
>> - Keep the tradition of abusing an existing data structure, and stash
>>   the bytepos info inside the overlay tree or the text properties.
>>   This way the conversion is bounded by O(log BUFFERSIZE).
>
> For overlay tree, it might be even better to stash all the markers in
> Emacs into itree structure.

Yes, I don't really distinguish the itree structure and the overlay
tree, but indeed we could have a separate itree for the markers.

> For now, every operation involving
> adjusting/searching markers scales linearly - not ideal.

Adjusting only happens upon insertion, and while it's definitely not
ideal, it's surprising how little it bites in practice.
And if we move to an array-with-gap it'll bite even much less.

As for searching markers, AFAIK the only time we do that is for ....
.... the bytes<->chars conversion 🙂

[ Admittedly, we also do a search when we delete a marker, but that'd be
  easy to optimize away in the common case if we cared about it.  ]

IIRC, XEmacs doesn't use a linked-list of markers but an array-with-gap
of markers instead.  Not sure if they keep it sorted, but if they do, such
an array-with-gap can even be binary-searched, so it's quite efficient.

>> - Use a dedicated data-structure.  E.g. a pair of array-with-a-gap
>>   (one indexed by BYTEPOS/STEP the other indexed by CHARPOS/STEP, where
>>   STEP would be a large enough constant to make those arrays cheap yet
>>   small enough that the remaining scan is cheap).
>>   This way the conversion is O(STEP), i.e. "constant-time".
> I think that it will be less efficient compared to using a tree-like
> structure (if we can manage to use it). Will it be easier?

Given we've survived with a *really* poor data-structure until now,
I suspect that we don't need to worry about choosing the most efficient
option. 🙂

>> BTW, among my various local hacks, I've been using the hack below, which
>> aims to randomize the order in our markers-list, so as to minimize the
>> risk that we have to wade through lots of markers all clumped around the
>> same position.  I don't think it does a good job of it, but maybe we can
>> improve the execution of this idea, tho it still doesn't help if there's
>> no GC involved.
> I am not sure if I believe that this approach can yield practical gains.

Neither am I.  Another idea I had in the same vicinity (and hence
arguably just as unconvincing) was that in buf_*pos_to_*pos, when we
exit the

    for (tail = BUF_MARKERS (b); tail; tail = tail->next)

loop, we could move the markers we just skipped to the end of the list,
based on the idea that we've just found them to be useless at the head.
Doing it efficiently requires keeping a pointer to the end of the list, tho.

> AFAIU, the problem with the slowdown we are discussing here is markers
> that are all around the same position. It's rather too many markers in
> general, spaced not far from each other.

Sounds about right.  But if we could keep them nicely randomized (which
was my original goal with my quick hack), then the total number of
markers wouldn't matter that much because the first N markers in the
list would still be (probabilistically) nicely spread over the
whole buffer.

>> BTW, if/when we use some other data-structure to convert bytes<->chars,
>> then we could presumably get rid of our markers-list and stash markers
>> inside our overlay tree (basically represent them as degenerate overlays
>> with beg==end and no properties).
> I am wondering why it is impossible to stash markers inside overlay tree
> without doing anything special about bytes<->chars conversion (other
> than changing the linear loop with itree query).

I think the answer is that it's not not possible.


        Stefan

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

One other thing I notice: If I use Emacs-29 but with Org-9.5.5 (i.e., the
version included in Emacs-28.2) the recipe doesn't show the slowdown.

There's still a good chance that the slow behavior is due to
chars<->bytes conversion (there's no doubt that this code's performance
can be quite poor), but I suspect the reason for the change of behavior
between Emacs-28 and Emacs-29 lies in Org rather than in Emacs's C code.
If someone more familiar with the changes that occurred in Org around
that time could try and track down the specific change that triggers the
problem, that would be appreciated.


        Stefan

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

On June 25, 2024 6:07:32 AM GMT+03:00, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> One other thing I notice: If I use Emacs-29 but with Org-9.5.5 (i.e., the
> version included in Emacs-28.2) the recipe doesn't show the slowdown.
> 
> There's still a good chance that the slow behavior is due to
> chars<->bytes conversion (there's no doubt that this code's performance
> can be quite poor), but I suspect the reason for the change of behavior
> between Emacs-28 and Emacs-29 lies in Org rather than in Emacs's C code.
> If someone more familiar with the changes that occurred in Org around
> that time could try and track down the specific change that triggers the
> problem, that would be appreciated.
> 
> 
>         Stefan
> 
> 

Yes, I think it is very important to find out which changes in Org triggered this issue in Emacs 29.

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> One other thing I notice: If I use Emacs-29 but with Org-9.5.5 (i.e., the
> version included in Emacs-28.2) the recipe doesn't show the slowdown.
> ...
> If someone more familiar with the changes that occurred in Org around
> that time could try and track down the specific change that triggers the
> problem, that would be appreciated.

The biggest suspect is the fact that Org switched from overlay-based
folding to text property-based between Org 9.5 and Org 9.6.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Mitchell <mitchellahren@gmail.com>, Eli Zaretskii <eliz@gnu.org>,
>  71644@debbugs.gnu.org
> Date: Tue, 25 Jun 2024 09:30:00 +0000
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > One other thing I notice: If I use Emacs-29 but with Org-9.5.5 (i.e., the
> > version included in Emacs-28.2) the recipe doesn't show the slowdown.
> > ...
> > If someone more familiar with the changes that occurred in Org around
> > that time could try and track down the specific change that triggers the
> > problem, that would be appreciated.
> 
> The biggest suspect is the fact that Org switched from overlay-based
> folding to text property-based between Org 9.5 and Org 9.6.

But overlays aren't supposed to be faster than text properties, so I'm
unsure why this change would have such an effect.

However, it is important to establish the facts first, and try to
explain them later.  So if you could come up with a change to try on
top of Org 9.6 that would change it back to use overlays for folding,
we could try that and see if the slowdown goes away, and take it from
there.

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> ...  So if you could come up with a change to try on
> top of Org 9.6 that would change it back to use overlays for folding,
> we could try that and see if the slowdown goes away, and take it from
> there.

(setq org-fold-core-style 'overlays) ; before Org is loaded

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: monnier@iro.umontreal.ca, mitchellahren@gmail.com, 71644@debbugs.gnu.org
> Date: Tue, 25 Jun 2024 13:50:21 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > ...  So if you could come up with a change to try on
> > top of Org 9.6 that would change it back to use overlays for folding,
> > we could try that and see if the slowdown goes away, and take it from
> > there.
> 
> (setq org-fold-core-style 'overlays) ; before Org is loaded

Thanks.  Mitchell and Stefan, can you try this?

P.S. For some reason, Stefan's email server rejects my mails "due to
abuse", I hope it could be fixed soon.

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> P.S. For some reason, Stefan's email server rejects my mails "due to
> abuse", I hope it could be fixed soon.

Same for me.

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Mitchell]

[-- Attachment #1: Type: text/plain, Size: 679 bytes --]

 > > Eli Zaretskii <eliz@gnu.org> writes:
> >
> > > ...  So if you could come up with a change to try on
> > > top of Org 9.6 that would change it back to use overlays for folding,
> > > we could try that and see if the slowdown goes away, and take it from
> > > there.
> >
> > (setq org-fold-core-style 'overlays) ; before Org is loaded
>
>    Thanks.  Mitchell and Stefan, can you try this?

I tested setting org-fold-core-style to 'overlays before loading org and
unfortunately it did not help the slowness in the test org buffer. These
are the two lines I tried in separate tests:

(setq org-fold-core-style 'overlays)
(customize-set-variable 'org-fold-core-style 'overlays)

[-- Attachment #2: Type: text/html, Size: 1007 bytes --]

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> From: Mitchell <mitchellahren@gmail.com>
> Date: Tue, 25 Jun 2024 21:53:42 -0600
> Cc: Ihor Radchenko <yantar92@posteo.net>, monnier@iro.umontreal.ca, 71644@debbugs.gnu.org
> 
> > > Eli Zaretskii <eliz@gnu.org> writes:
> > >
> > > > ...  So if you could come up with a change to try on
> > > > top of Org 9.6 that would change it back to use overlays for folding,
> > > > we could try that and see if the slowdown goes away, and take it from
> > > > there.
> > >
> > > (setq org-fold-core-style 'overlays) ; before Org is loaded
> > 
> >    Thanks.  Mitchell and Stefan, can you try this?
>  
> I tested setting org-fold-core-style to 'overlays before loading org and unfortunately it did not help the
> slowness in the test org buffer. These are the two lines I tried in separate tests:
> 
> (setq org-fold-core-style 'overlays)
> (customize-set-variable 'org-fold-core-style 'overlays)

Thanks, I guess it means this theory eats dust.

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> One other thing I notice: If I use Emacs-29 but with Org-9.5.5 (i.e., the
> version included in Emacs-28.2) the recipe doesn't show the slowdown.

One more thing: when I use Mitchell's recipe, sometimes buffer edits are
immediate and sometimes they're not (I mean, when following the recipe
exactly, it's not immediate, but if I do things slightly differently and
sometimes after doing other(?) things, it's immediate again).

When I'm in the "state" where editing is not
immediate, it seems to apply only to edits done right before or right
after a header: edits elsewhere (e.g. within some paragraph) are
still "instant".

Oh, and typing text at the end of a header seems to get me out of that
state, somehow (and I don't know how to get back into that state, tho
it sometimes happen).

BTW, my test case is now a large (26MB, 640kLines) Org file consisting of a long
repetition of:

    * asdgasgasdfgsdéadfgasdgafg
    
    asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh
    asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh
    asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh
    asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh
    asdfadfkh asdfadfkh
    
    ** sf na;l bnsdkh bsadlfv asdl fkbasldfv alsfkg v
    
    asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh
    asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh
    asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh
    asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh asdfadfkh
    asdfadfkh asdfadfkh

I instrumented the bytes<->chars routines to try and keep track of what
happens there.  This gives me a histogram showing how many times the
conversion routines consulted N markers as well as another histogram
telling me how many times those routines scanned between M*100B and
(M+1)*100B of text.
Oh and it gives me the number of markers in the buffer as well (the
histogram is not buffer-local, but there's basically only one buffer in
that session).

The result are below (followed by the actual code).
There are several odd things in there:

- The markers histogram is weirdly flat.
- The markers histogram seems to have a threshold around 800
  I can't explain.  It seems directly related to the buffer size, tho:
  with a file half the size, I get half as many markers (40k) and
  a threshold in the histogram around 400.
- The number of times we consider exactly 1 marker is probably "too
  high" because the C code happens to look at the first marker before
  checking if it's necessary.

(80044 :markers
 ((0 16) (1 3290720) (2 28912) (3 34427) (4 30479) (5 71089) (6 29858)
  (7 29855) (8 29953) (9 30156) (10 30251) (11 30185) (12 30999)
  (13 30270) (14 30530) (15 30332) (16 30175) (17 30278) (18 30187)
  (19 30316) (20 29698) (21 29760) (22 29674) (23 29695) (24 29682)
  (25 29689) (26 29699) (27 29835) (28 30483) (29 30455) (30 30444)
  (31 30313) (32 29666) (33 29696) (34 29739) (35 29781) (36 29694)
  (37 29726) (38 29722) (39 29729) (40 29721) (41 29800) (42 29846)
  (43 29751) (44 29775) (45 29780) (46 29782) (47 29788) (48 29869)
  (49 29891) (50 29811) (51 29823) (52 29834) (53 29817) (54 29833)
  (55 29901) (56 29927) (57 29854) (58 29836) (59 29856) (60 29843)
  (61 29835) (62 29918) (63 29928) (64 29851) (65 29858) (66 29857)
  (67 29859) (68 29854) (69 29934) (70 29907) (71 29859) (72 29848)
  (73 29859) (74 29859) (75 29875) (76 29916) (77 29904) (78 29865)
  (79 29840) (80 29867) (81 29857) (82 29853) (83 29948) (84 29887)
  (85 29861) (86 29852) (87 29850) (88 29866) (89 29880) (90 29935)
  (91 29881) (92 29862) (93 29843) (94 29858) (95 29857) (96 29892)
  (97 29931) (98 29878) (99 29856) (100 29848) (101 29855) (102 29856)
  (103 29904) (104 29931) (105 29866) (106 29867) (107 29849)
  (108 29843) (109 29876) (110 29906) (111 29917) (112 29859)
  (113 29862) (114 29851) (115 29852) (116 29867) (117 29910)
  (118 29922) (119 29848) (120 29862) (121 29858) (122 29837)
  (123 29873) (124 29922) (125 29910) (126 29852) (127 29864)
  (128 29852) (129 29852) (130 29866) (131 29916) (132 29928)
  (133 29833) (134 29863) (135 29864) (136 29844) (137 29864)
  (138 29923) (139 29923) (140 29842) (141 29854) (142 29864)
  (143 29844) (144 29882) (145 29907) (146 29921) (147 29856)
  (148 29838) (149 29864) (150 29873) (151 29854) (152 29925)
  (153 29906) (154 29844) (155 29854) (156 29858) (157 29866)
  (158 29885) (159 29917) (160 29888) (161 29860) (162 29831)
  (163 29865) (164 29874) (165 29888) (166 29907) (167 29887)
  (168 29862) (169 29844) (170 29850) (171 29880) (172 29887)
  (173 29929) (174 29861) (175 29864) (176 29849) (177 29843)
  (178 29880) (179 29909) (180 29916) (181 29853) (182 29864)
  (183 29852) (184 29840) (185 29898) (186 29894) (187 29919)
  (188 29845) (189 29864) (190 29858) (191 29850) (192 29880)
  (193 29908) (194 29914) (195 29842) (196 29860) (197 29860)
  (198 29854) (199 29891) (200 29903) (201 29931) (202 29847)
  (203 29865) (204 29874) (205 29860) (206 29892) (207 29913)
  (208 29921) (209 29860) (210 29850) (211 29872) (212 29877)
  (213 29888) (214 29924) (215 29945) (216 29875) (217 29884)
  (218 29892) (219 29903) (220 29892) (221 29960) (222 29931)
  (223 29884) (224 29877) (225 29898) (226 29919) (227 29927)
  (228 29973) (229 29964) (230 29925) (231 29902) (232 29923)
  (233 29936) (234 29929) (235 29999) (236 29948) (237 29920)
  (238 29913) (239 29917) (240 29938) (241 29948) (242 29994)
  (243 29930) (244 29928) (245 29893) (246 29461) (247 29304)
  (248 29906) (249 29983) (250 29929) (251 29922) (252 29909)
  (253 29917) (254 29930) (255 29947) (256 30002) (257 29913)
  (258 29928) (259 29919) (260 29908) (261 29927) (262 29978)
  (263 29977) (264 29914) (265 29921) (266 29921) (267 29914)
  (268 29924) (269 29963) (270 29998) (271 29903) (272 29921)
  (273 29928) (274 29903) (275 29931) (276 29978) (277 29978)
  (278 29908) (279 29921) (280 29923) (281 29926) (282 29920)
  (283 29969) (284 29987) (285 29899) (286 29919) (287 29930)
  (288 29922) (289 29918) (290 29978) (291 29976) (292 29911)
  (293 29911) (294 29930) (295 29920) (296 29918) (297 29979)
  (298 29977) (299 29918) (300 29904) (301 29928) (302 29929)
  (303 29920) (304 29986) (305 29961) (306 29916) (307 29911)
  (308 29921) (309 29932) (310 29931) (311 29991) (312 29942)
  (313 29924) (314 29902) (315 29922) (316 29940) (317 29934)
  (318 29982) (319 29940) (320 29926) (321 29913) (322 29921)
  (323 29928) (324 29936) (325 30000) (326 29920) (327 29928)
  (328 29915) (329 29915) (330 29930) (331 29959) (332 29983)
  (333 29918) (334 29924) (335 29920) (336 29913) (337 29927)
  (338 29960) (339 29991) (340 29911) (341 29922) (342 29928)
  (343 29907) (344 29931) (345 29966) (346 29984) (347 29909)
  (348 29921) (349 29924) (350 29919) (351 29925) (352 29965)
  (353 29989) (354 29901) (355 29921) (356 29930) (357 29917)
  (358 29917) (359 29979) (360 29981) (361 29913) (362 29909)
  (363 29934) (364 29914) (365 29920) (366 29972) (367 29984)
  (368 29908) (369 29914) (370 29930) (371 29931) (372 29914)
  (373 29990) (374 29959) (375 29914) (376 29913) (377 29927)
  (378 29926) (379 29932) (380 29968) (381 29964) (382 29921)
  (383 29908) (384 29925) (385 29938) (386 29911) (387 29995)
  (388 29946) (389 29920) (390 29913) (391 29925) (392 29928)
  (393 29940) (394 29990) (395 29928) (396 29928) (397 29907)
  (398 29919) (399 29938) (400 29948) (401 29979) (402 29929)
  (403 29922) (404 29917) (405 29917) (406 29932) (407 29941)
  (408 29998) (409 29915) (410 29930) (411 29923) (412 29912)
  (413 29927) (414 29966) (415 29979) (416 29912) (417 29927)
  (418 29925) (419 29916) (420 29920) (421 29955) (422 30000)
  (423 29901) (424 29927) (425 29932) (426 29907) (427 29921)
  (428 29974) (429 29978) (430 29910) (431 29923) (432 29929)
  (433 29922) (434 29918) (435 29965) (436 29987) (437 29899)
  (438 29923) (439 29932) (440 29926) (441 29912) (442 29974)
  (443 29978) (444 29909) (445 29919) (446 29930) (447 29922)
  (448 29914) (449 29973) (450 29979) (451 29920) (452 29908)
  (453 29932) (454 29929) (455 29908) (456 29988) (457 29959)
  (458 29922) (459 29915) (460 29923) (461 29928) (462 29923)
  (463 29993) (464 29940) (465 29928) (466 29908) (467 29926)
  (468 29930) (469 29930) (470 29982) (471 29940) (472 29930)
  (473 29919) (474 29917) (475 29926) (476 29934) (477 29998)
  (478 29920) (479 29932) (480 29917) (481 29921) (482 29924)
  (483 29953) (484 29985) (485 29916) (486 29932) (487 29920)
  (488 29915) (489 29923) (490 29908) (491 29377) (492 28865)
  (493 29336) (494 29924) (495 29905) (496 29917) (497 29966)
  (498 29982) (499 29907) (500 29927) (501 29922) (502 29913)
  (503 29915) (504 29965) (505 29987) (506 29903) (507 29923)
  (508 29928) (509 29909) (510 29911) (511 29975) (512 29981)
  (513 29915) (514 29911) (515 29926) (516 29914) (517 29912)
  (518 29968) (519 29984) (520 29910) (521 29912) (522 29932)
  (523 29921) (524 29908) (525 29990) (526 29955) (527 29922)
  (528 29909) (529 29923) (530 29920) (531 29926) (532 29966)
  (533 29962) (534 29927) (535 29906) (536 29923) (537 29924)
  (538 29911) (539 29993) (540 29944) (541 29926) (542 29911)
  (543 29921) (544 29920) (545 29936) (546 29988) (547 29930)
  (548 29930) (549 29905) (550 29917) (551 29926) (552 29944)
  (553 29979) (554 29931) (555 29924) (556 29911) (557 29915)
  (558 29924) (559 29937) (560 29998) (561 29917) (562 29928)
  (563 29925) (564 29902) (565 29921) (566 29964) (567 29977)
  (568 29918) (569 29925) (570 29921) (571 29910) (572 29914)
  (573 29953) (574 29998) (575 29907) (576 29925) (577 29930)
  (578 29897) (579 29917) (580 29972) (581 29976) (582 29916)
  (583 29921) (584 29927) (585 29912) (586 29914) (587 29963)
  (588 29989) (589 29903) (590 29919) (591 29930) (592 29914)
  (593 29908) (594 29974) (595 29980) (596 29911) (597 29913)
  (598 29928) (599 29912) (600 29912) (601 29973) (602 29981)
  (603 29918) (604 29910) (605 29922) (606 29923) (607 29906)
  (608 29986) (609 29963) (610 29922) (611 29911) (612 29917)
  (613 29922) (614 29921) (615 29991) (616 29944) (617 29930)
  (618 29904) (619 29916) (620 29926) (621 29928) (622 29980)
  (623 29948) (624 29926) (625 29917) (626 29907) (627 29922)
  (628 29932) (629 30000) (630 29924) (631 29928) (632 29915)
  (633 29907) (634 29922) (635 29953) (636 29987) (637 29918)
  (638 29926) (639 29918) (640 29905) (641 29921) (642 29954)
  (643 29995) (644 29911) (645 29928) (646 29922) (647 29901)
  (648 29917) (649 29966) (650 29986) (651 29915) (652 29921)
  (653 29920) (654 29909) (655 29915) (656 29965) (657 29991)
  (658 29907) (659 29923) (660 29924) (661 29905) (662 29911)
  (663 29975) (664 29989) (665 29913) (666 29911) (667 29924)
  (668 29910) (669 29912) (670 29972) (671 29988) (672 29908)
  (673 29914) (674 29924) (675 29921) (676 29908) (677 29994)
  (678 29959) (679 29916) (680 29911) (681 29919) (682 29920)
  (683 29926) (684 29972) (685 29964) (686 29927) (687 29902)
  (688 29919) (689 29924) (690 29911) (691 29997) (692 29952)
  (693 29920) (694 29909) (695 29917) (696 29920) (697 29936)
  (698 29992) (699 29932) (700 29930) (701 29903) (702 29913)
  (703 29926) (704 29944) (705 29987) (706 29929) (707 29924)
  (708 29909) (709 29911) (710 29924) (711 29941) (712 30002)
  (713 29915) (714 29930) (715 29917) (716 29902) (717 29921)
  (718 29970) (719 29979) (720 29914) (721 29925) (722 29917)
  (723 29910) (724 29914) (725 29959) (726 30000) (727 29907)
  (728 29921) (729 29926) (730 29897) (731 29917) (732 29976)
  (733 29982) (734 29908) (735 29845) (736 29275) (737 28476)
  (738 28898) (739 29693) (740 29989) (741 29897) (742 29915)
  (743 29924) (744 29912) (745 29908) (746 29980) (747 29974)
  (748 29907) (749 29909) (750 29922) (751 29912) (752 29914)
  (753 29975) (754 29975) (755 29914) (756 29902) (757 29920)
  (758 29921) (759 29910) (760 29988) (761 29957) (762 29914)
  (763 29907) (764 29915) (765 29920) (766 29925) (767 29993)
  (768 29938) (769 29922) (770 29898) (771 29916) (772 29924)
  (773 29930) (774 29984) (775 29938) (776 29922) (777 29911)
  (778 29907) (779 29920) (780 29934) (781 30000) (782 29918)
  (783 29924) (784 29909) (785 29907) (786 29920) (787 29959)
  (788 29981) (789 29914) (790 29795) (791 21494) (792 8509) (793 804)
  (794 804) (795 800) (796 800) (797 800) (798 717) (799 323)
  (800 200) (801 202) (802 200) (803 200) (804 172) (805 26) (5533 1)
  (5548 1) (26958 1) (26974 1) (48384 1) (48400 1))
 :distance
 ((0 26807330) (1 5518) (2 941) (3 160056) (4 2) (5 87) (6 25) (8 2)
  (9 2) (10 7) (11 1) (12 22) (13 1) (15 7) (28 1) (30 4) (32 5)
  (35 7) (39 3) (40 1) (42 5) (45 4) (48 2) (49 3) (51 1) (52 1)
  (54 22) (55 7) (82 1) (2762 1) (2772 1) (13478 1) (13485 1)
  (24191 1) (24198 1)))


diff --git a/lisp/subr.el b/lisp/subr.el
index d9df8d1a458..1e9c768fdfe 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -6968,6 +6968,22 @@ internal--format-docstring-line
     (error "Unable to fill string containing newline: %S" string))
   (internal--fill-string-single-line (apply #'format string objects)))
 
+(defun bytechar--stats ()
+  (unless (hash-table-p internal--bytechar-marker-counts)
+    (setq internal--bytechar-marker-counts (make-hash-table)))
+  (unless (hash-table-p internal--bytechar-distance-counts)
+    (setq internal--bytechar-distance-counts (make-hash-table)))
+  (require 'map)
+  (list (internal--count-markers)
+        :markers
+        (mapcar (lambda (x) (list (car x) (cdr x)))
+             (sort (map-into internal--bytechar-marker-counts 'alist)
+                   #'car-less-than-car))
+        :distance
+        (mapcar (lambda (x) (list (car x) (cdr x)))
+             (sort (map-into internal--bytechar-distance-counts 'alist)
+              #'car-less-than-car))))
+
 (defun json-available-p ()
   "Return non-nil if Emacs has libjansson support."
   (and (fboundp 'json--available-p)
diff --git a/src/marker.c b/src/marker.c
index d220dd82692..623065de3f7 100644
--- a/src/marker.c
+++ b/src/marker.c
@@ -133,6 +133,28 @@ CHECK_MARKER (Lisp_Object x)
   CHECK_TYPE (MARKERP (x), Qmarkerp, x);
 }
 
+static void
+bytechar_stats (int marker_count, ptrdiff_t distance)
+{
+  if (HASH_TABLE_P (bytechar_marker_counts))
+    {
+      Lisp_Object key = make_fixnum (marker_count);
+      Lisp_Object oldcount = Fgethash (key, bytechar_marker_counts, Qnil);
+      Lisp_Object newcount
+	= make_fixnum (FIXNUMP (oldcount) ? 1 + XFIXNUM (oldcount) : 1);
+      Fputhash (key, newcount, bytechar_marker_counts);
+    }
+
+  if (HASH_TABLE_P (bytechar_distance_counts))
+    {
+      Lisp_Object key = make_fixnum (distance / 100);
+      Lisp_Object oldcount = Fgethash (key, bytechar_distance_counts, Qnil);
+      Lisp_Object newcount
+	= make_fixnum (FIXNUMP (oldcount) ? 1 + XFIXNUM (oldcount) : 1);
+      Fputhash (key, newcount, bytechar_distance_counts);
+    }
+}
+
 /* When converting bytes from/to chars, we look through the list of
    markers to try and find a good starting point (since markers keep
    track of both bytepos and charpos at the same time).
@@ -164,6 +186,7 @@ buf_charpos_to_bytepos (struct buffer *b, ptrdiff_t charpos)
   ptrdiff_t best_above, best_above_byte;
   ptrdiff_t best_below, best_below_byte;
   ptrdiff_t distance = BYTECHAR_DISTANCE_INITIAL;
+  int marker_count = 0;
 
   eassert (BUF_BEG (b) <= charpos && charpos <= BUF_Z (b));
 
@@ -198,6 +221,7 @@ buf_charpos_to_bytepos (struct buffer *b, ptrdiff_t charpos)
 
   for (tail = BUF_MARKERS (b); tail; tail = tail->next)
     {
+      marker_count++;
       CONSIDER (tail->charpos, tail->bytepos);
 
       /* If we are down to a range of 50 chars,
@@ -215,6 +239,10 @@ buf_charpos_to_bytepos (struct buffer *b, ptrdiff_t charpos)
      Scan, counting characters, from whichever one is closer.  */
 
   eassert (best_below <= charpos && charpos <= best_above);
+  if (!NILP (bytechar_marker_counts))
+    bytechar_stats (marker_count,
+		    min (charpos - best_below, best_above - charpos));
+
   if (charpos - best_below < best_above - charpos)
     {
       bool record = charpos - best_below > 5000;
@@ -321,6 +349,7 @@ buf_bytepos_to_charpos (struct buffer *b, ptrdiff_t bytepos)
   ptrdiff_t best_above, best_above_byte;
   ptrdiff_t best_below, best_below_byte;
   ptrdiff_t distance = BYTECHAR_DISTANCE_INITIAL;
+  int marker_count = 0;
 
   eassert (BUF_BEG_BYTE (b) <= bytepos && bytepos <= BUF_Z_BYTE (b));
 
@@ -350,6 +379,7 @@ buf_bytepos_to_charpos (struct buffer *b, ptrdiff_t bytepos)
 
   for (tail = BUF_MARKERS (b); tail; tail = tail->next)
     {
+      marker_count++;
       CONSIDER (tail->bytepos, tail->charpos);
 
       /* If we are down to a range of 50 chars,
@@ -365,6 +395,9 @@ buf_bytepos_to_charpos (struct buffer *b, ptrdiff_t bytepos)
   /* We get here if we did not exactly hit one of the known places.
      We have one known above and one known below.
      Scan, counting characters, from whichever one is closer.  */
+  if (!NILP (bytechar_marker_counts))
+    bytechar_stats (marker_count,
+		    min (bytepos - best_below_byte, best_above_byte - bytepos));
 
   if (bytepos - best_below_byte < best_above_byte - bytepos)
     {
@@ -759,22 +792,23 @@ DEFUN ("set-marker-insertion-type", Fset_marker_insertion_type,
   return type;
 }
 
-#ifdef MARKER_DEBUG
-
 /* For debugging -- count the markers in buffer BUF.  */
 
-int
-count_markers (struct buffer *buf)
+DEFUN ("internal--count-markers", Fcount_markers, Scount_markers, 0, 0, 0,
+       doc: /* Return the number of markers in the current buffer.  */)
+  (void)
 {
   int total = 0;
   struct Lisp_Marker *tail;
 
-  for (tail = BUF_MARKERS (buf); tail; tail = tail->next)
+  for (tail = BUF_MARKERS (current_buffer); tail; tail = tail->next)
     total++;
 
-  return total;
+  return make_fixnum (total);
 }
 
+#ifdef MARKER_DEBUG
+
 /* For debugging -- recompute the bytepos corresponding
    to CHARPOS in the simplest, most reliable way.  */
 
@@ -804,4 +838,18 @@ syms_of_marker (void)
   defsubr (&Scopy_marker);
   defsubr (&Smarker_insertion_type);
   defsubr (&Sset_marker_insertion_type);
+  defsubr (&Scount_markers);
+
+  DEFVAR_LISP ("internal--bytechar-marker-counts",
+	       bytechar_marker_counts, doc: /* Hihi */);
+  bytechar_marker_counts = Qnil;
+
+  DEFVAR_LISP ("internal--bytechar-distance-counts",
+	       bytechar_distance_counts, doc: /* Hoho */);
+  bytechar_distance_counts = Qnil;
+
+  DEFVAR_INT ("internal--bytechar-distance-increment",
+              bytechar_distance_increment,
+              doc: /* Haha */);
+  bytechar_distance_increment = 50;
 }

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Mitchell <mitchellahren@gmail.com>,  Eli Zaretskii <eliz@gnu.org>,
>   71644@debbugs.gnu.org
> Date: Tue, 25 Jun 2024 16:54:44 -0400
> 
> I instrumented the bytes<->chars routines to try and keep track of what
> happens there.  This gives me a histogram showing how many times the
> conversion routines consulted N markers as well as another histogram
> telling me how many times those routines scanned between M*100B and
> (M+1)*100B of text.
> Oh and it gives me the number of markers in the buffer as well (the
> histogram is not buffer-local, but there's basically only one buffer in
> that session).
> 
> The result are below (followed by the actual code).
> There are several odd things in there:
> 
> - The markers histogram is weirdly flat.
> - The markers histogram seems to have a threshold around 800
>   I can't explain.  It seems directly related to the buffer size, tho:
>   with a file half the size, I get half as many markers (40k) and
>   a threshold in the histogram around 400.
> - The number of times we consider exactly 1 marker is probably "too
>   high" because the C code happens to look at the first marker before
>   checking if it's necessary.

The fact that it's related to buffer size is probably because your
test buffer is basically N >> 1 repetitions of the same pattern, so
whatever Org does it does that N times, where N is directly
proportional to the buffer size.

Questions:

How many markers are there in this buffer during the experiment where
you produced the histogram you've posted?  My guess is 48400 or
thereabouts?

Also, what were the buffer positions where you tried your editing
operations?  In particular, where those positions distributed evenly
over the buffer?

And questions to Ihor: what are the reasons Org creates these markers?
In particular, is there any way to know or guess whether these markers
will be distributed more-or-less evenly over the buffer text, or
clustered near some specific positions?  Also, does Org move these
markers frequently, or do they stay put once created?

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> And questions to Ihor: what are the reasons Org creates these markers?
> In particular, is there any way to know or guess whether these markers
> will be distributed more-or-less evenly over the buffer text, or
> clustered near some specific positions?  Also, does Org move these
> markers frequently, or do they stay put once created?

In this particular case, with the reproducer we are using here, Org does
not create markers. `councel-outline' does - for every headline in
buffer with marker pointing to the beginning of each headline.
These markers are distributed as headlines do.

Org mode may also create markers in some scenarios. Also at headlines,
but not necessarily _all_ headlines. org-agenda creates markers for
every headline that is displayed in agenda (matches agenda search
criterion). org-refile creates markers for each top level heading (by
default). org-goto creates markers for each heading down to level 5.

Org mode does not move these markers frequently, except markers created
by agenda, when a heading is killed/yanked via
org-cut-special/org-paste-subtree or org-refile commands.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> - The markers histogram is weirdly flat.
>> - The markers histogram seems to have a threshold around 800
>>   I can't explain.  It seems directly related to the buffer size, tho:
>>   with a file half the size, I get half as many markers (40k) and
>>   a threshold in the histogram around 400.
>> - The number of times we consider exactly 1 marker is probably "too
>>   high" because the C code happens to look at the first marker before
>>   checking if it's necessary.
>
> The fact that it's related to buffer size is probably because your
> test buffer is basically N >> 1 repetitions of the same pattern, so
> whatever Org does it does that N times, where N is directly
> proportional to the buffer size.

Yes, definitely.  The threshold is at about N/100 where N is the number
of markers/headings.  But why 100?

> How many markers are there in this buffer during the experiment where
> you produced the histogram you've posted?  My guess is 48400 or
> thereabouts?

It's the first number in the list (the one before `:markers`): 80044

> Also, what were the buffer positions where you tried your editing
> operations?  In particular, where those positions distributed evenly
> over the buffer?

I focused on edits on the first copy and on the last copy.

> And questions to Ihor: what are the reasons Org creates these markers?
> In particular, is there any way to know or guess whether these markers
> will be distributed more-or-less evenly over the buffer text, or
> clustered near some specific positions?  Also, does Org move these
> markers frequently, or do they stay put once created?

They're created by `consult-outline` rather than by Org.
There's one marker per heading, AFAICT, and so they're very
evenly spread (and presumably ordered last-to-first in the `markers`
list).


        Stefan

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Yes, definitely.  The threshold is at about N/100 where N is the number
> of markers/headings.  But why 100?

Maybe because

bytepos - best_below_byte > 5000
and distance += BYTECHAR_DISTANCE_INCREMENT;
(BYTECHAR_DISTANCE_INCREMENT = 50)

Then, (/ 5000 50) ; -> 100.

In other words,
for (tail = BUF_MARKERS (b); tail; tail = tail->next)
loop is roughly bound to (min number-of-markers 100)-ish
repetitions. So, once the number of markers exceeds 100, there is no
more scaling with the marker number.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Yes, definitely.  The threshold is at about N/100 where N is the number
>> of markers/headings.  But why 100?
>
> Maybe because
>
> bytepos - best_below_byte > 5000
> and distance += BYTECHAR_DISTANCE_INCREMENT;
> (BYTECHAR_DISTANCE_INCREMENT = 50)
>
> Then, (/ 5000 50) ; -> 100.
>
> In other words,
> for (tail = BUF_MARKERS (b); tail; tail = tail->next)
> loop is roughly bound to (min number-of-markers 100)-ish
> repetitions. So, once the number of markers exceeds 100, there is no
> more scaling with the marker number.

I'm sorry, I don't understand what you're trying to say:

The histogram is showing the number of iterations of this loop,
and the threshold is not at 100 but at 800 (a.k.a number of markers /
100).


        Stefan

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Mitchell]

[-- Attachment #1: Type: text/plain, Size: 412 bytes --]

For what it’s worth, I tried the same steps 1 through 7 while in text-mode,
not org, and the slowdown did not occur. The default value of
outline-regexp in a text-mode buffer is "[* ]+", so it’s basically the same
as the regex for an org-mode heading. I also tried Steps 1 to 7 in
outline-mode and didn’t see a slowdown.

Should I direct the org-mode devs to this bug report at tracker.orgmode.org?

[-- Attachment #2: Type: text/html, Size: 547 bytes --]

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Yup. I _do not_ propose my patch to be merged.
> (BTW, we should probably merge this bug and bug#63040 where I first
> shared this patch - just to demonstrate the problem and discuss possible
> solutions)

FWIW, I'm not sure it's exactly the same problem yet.


        Stefan

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> From: Mitchell <mitchellahren@gmail.com>
> Date: Thu, 20 Jun 2024 20:46:51 -0600
> Cc: 71644@debbugs.gnu.org
> 
> > If you remove all the non-ASCII characters from the Org file, does the
> > slowdown go away?
> 
> Eli, that solved it! The new test file is at
> https://gist.github.com/kings2u/2ef0e145f2b42d0a13605b0dc9b6e6e2. I replaced every non-ASCII character
> with an "a" so the file still has the same number of total characters, and in my emacs 30.0 50 build (as of
> 2024-05-25), doing Steps 1 to Step 7 gives me abbrev expansions that are as lighting fast as in emacs 28.2!
> 
> So what now? Do you think you can solve what’s going on in the backend so bigger buffers with markers and
> non-ASCII characters don’t exhibit this slowness in the latest emacs? 

The fact that the problem goes away when you remove non-ASCII
characters is a pretty convincing argument in favor of your theory
that markers are involved.  Because Emacs consults the buffer's
markers whenever it needs to convert character positions to byte
positions or vice versa, which is done _a_lot_.

The only significant change I see in markers code that could be
related to converting character to byte positions is this:

  commit 8783700b23e70874c4996908bf02c010ae6f3fe1
  Author:     Stefan Monnier <monnier@iro.umontreal.ca>
  AuthorDate: Tue Aug 2 10:38:53 2022 -0400
  Commit:     Stefan Monnier <monnier@iro.umontreal.ca>
  CommitDate: Tue Aug 2 13:06:51 2022 -0400

      * src/xdisp.c (redisplay_window): Use BEG rather than hard coding 1

It changed the comparison operator in two places in marker.c.

Curiously, the log message doesn't even mention the change in
marker.c, which could be a sign that this change was not intended to
be installed.  Stefan, did you intend to install it, and if so, do you
have any comments about this bug report?

I'm a bit confused by the fact that I don't see the slowdown on my
machine, but maybe there are other factors at work here that hide the
regression.

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>   commit 8783700b23e70874c4996908bf02c010ae6f3fe1
>   Author:     Stefan Monnier <monnier@iro.umontreal.ca>
>   AuthorDate: Tue Aug 2 10:38:53 2022 -0400
>   Commit:     Stefan Monnier <monnier@iro.umontreal.ca>
>   CommitDate: Tue Aug 2 13:06:51 2022 -0400
>
>       * src/xdisp.c (redisplay_window): Use BEG rather than hard coding 1
>
> It changed the comparison operator in two places in marker.c.
>
> Curiously, the log message doesn't even mention the change in
> marker.c, which could be a sign that this change was not intended to
> be installed.  Stefan, did you intend to install it, and if so, do you
> have any comments about this bug report?

Hmm... can't remember why/how it ended up in the above commit.
Looks like an oversight.  But the change should be harmless: the
`eassert` should make sure that the comparison gives the same answer
either way (and AFAICT if/when the new comparison gives a different
answer from the old code, the old code will loop until it segfaults).

> I'm a bit confused by the fact that I don't see the slowdown on my
> machine, but maybe there are other factors at work here that hide
> the regression.

The byte<->char conversion code is affected by many unrelated moving
parts, so it can be difficult to come up with a reproducible recipe.


        Stefan

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Mitchell <mitchellahren@gmail.com>,  71644@debbugs.gnu.org
> Date: Fri, 21 Jun 2024 17:06:31 -0400
> 
> >   commit 8783700b23e70874c4996908bf02c010ae6f3fe1
> >   Author:     Stefan Monnier <monnier@iro.umontreal.ca>
> >   AuthorDate: Tue Aug 2 10:38:53 2022 -0400
> >   Commit:     Stefan Monnier <monnier@iro.umontreal.ca>
> >   CommitDate: Tue Aug 2 13:06:51 2022 -0400
> >
> >       * src/xdisp.c (redisplay_window): Use BEG rather than hard coding 1
> >
> > It changed the comparison operator in two places in marker.c.
> >
> > Curiously, the log message doesn't even mention the change in
> > marker.c, which could be a sign that this change was not intended to
> > be installed.  Stefan, did you intend to install it, and if so, do you
> > have any comments about this bug report?
> 
> Hmm... can't remember why/how it ended up in the above commit.
> Looks like an oversight.  But the change should be harmless: the
> `eassert` should make sure that the comparison gives the same answer
> either way (and AFAICT if/when the new comparison gives a different
> answer from the old code, the old code will loop until it segfaults).

Mitchell, can you try reverting that change, and see if that affects
performance in your case?

> > I'm a bit confused by the fact that I don't see the slowdown on my
> > machine, but maybe there are other factors at work here that hide
> > the regression.
> 
> The byte<->char conversion code is affected by many unrelated moving
> parts, so it can be difficult to come up with a reproducible recipe.

But in this case, we did that in "emacs -Q" and with a specific test
file that was posted and used.  What other variables can affect this
recipe?  Do you see the slow response on your machine, for example,
when running this recipe?

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Mitchell]

[-- Attachment #1: Type: text/plain, Size: 1932 bytes --]

On Sat, Jun 22, 2024 at 12:58 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > > Cc: Mitchell <mitchellahren@gmail.com>,  71644@debbugs.gnu.org
> > > Date: Fri, 21 Jun 2024 17:06:31 -0400
> > >
> > > >   commit 8783700b23e70874c4996908bf02c010ae6f3fe1
> > > >   Author:     Stefan Monnier <monnier@iro.umontreal.ca>
> > > >   AuthorDate: Tue Aug 2 10:38:53 2022 -0400
> > > >   Commit:     Stefan Monnier <monnier@iro.umontreal.ca>
> > > >   CommitDate: Tue Aug 2 13:06:51 2022 -0400
> > > >
> > > >       * src/xdisp.c (redisplay_window): Use BEG rather than hard
> coding 1
> > > >
> > > > It changed the comparison operator in two places in marker.c.
> > > >
> > > > Curiously, the log message doesn't even mention the change in
> > > > marker.c, which could be a sign that this change was not intended to
> > > > be installed.  Stefan, did you intend to install it, and if so, do
> you
> > > > have any comments about this bug report?
> > >
> > > Hmm... can't remember why/how it ended up in the above commit.
> > > Looks like an oversight.  But the change should be harmless: the
> > > `eassert` should make sure that the comparison gives the same answer
> > > either way (and AFAICT if/when the new comparison gives a different
> > > answer from the old code, the old code will loop until it segfaults).
> >
> > Mitchell, can you try reverting that change, and see if that affects
> > performance in your case?
>

Eli, after Ihor and Stefan were able to reproduce it now, would it still be
helpful for me to do this and report back? I’m more than happy to if it
would help in any way.

Also should I be replying to the renamed thread from now on? (i.e.,
"chars==bytes (was: bug#71644: 30.0.50; Severe slowdown in larger files
with markers beginning in emacs 29+") This is my first bug report, so I’m
not sure the etiquette, haha.

[-- Attachment #2: Type: text/html, Size: 3068 bytes --]

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> From: Mitchell <mitchellahren@gmail.com>
> Date: Sat, 22 Jun 2024 12:03:05 -0600
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, 71644@debbugs.gnu.org
> 
> On Sat, Jun 22, 2024 at 12:58 AM Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  > > From: Stefan Monnier <monnier@iro.umontreal.ca>
>  > > Cc: Mitchell <mitchellahren@gmail.com>,  71644@debbugs.gnu.org
>  > > Date: Fri, 21 Jun 2024 17:06:31 -0400
>  > > 
>  > > >   commit 8783700b23e70874c4996908bf02c010ae6f3fe1
>  > > >   Author:     Stefan Monnier <monnier@iro.umontreal.ca>
>  > > >   AuthorDate: Tue Aug 2 10:38:53 2022 -0400
>  > > >   Commit:     Stefan Monnier <monnier@iro.umontreal.ca>
>  > > >   CommitDate: Tue Aug 2 13:06:51 2022 -0400
>  > > >
>  > > >       * src/xdisp.c (redisplay_window): Use BEG rather than hard coding 1
>  > > >
>  > > > It changed the comparison operator in two places in marker.c.
>  > > >
>  > > > Curiously, the log message doesn't even mention the change in
>  > > > marker.c, which could be a sign that this change was not intended to
>  > > > be installed.  Stefan, did you intend to install it, and if so, do you
>  > > > have any comments about this bug report?
>  > > 
>  > > Hmm... can't remember why/how it ended up in the above commit.
>  > > Looks like an oversight.  But the change should be harmless: the
>  > > `eassert` should make sure that the comparison gives the same answer
>  > > either way (and AFAICT if/when the new comparison gives a different
>  > > answer from the old code, the old code will loop until it segfaults).
>  > 
>  > Mitchell, can you try reverting that change, and see if that affects
>  > performance in your case?
> 
> Eli, after Ihor and Stefan were able to reproduce it now, would it still be helpful for me to do this and report
> back? I’m more than happy to if it would help in any way. 

Yes, because we still don't understand well why performance regressed
from Emacs 28 to Emacs 29.

> Also should I be replying to the renamed thread from now on? (i.e., "chars==bytes (was: bug#71644: 30.0.50;
> Severe slowdown in larger files with markers beginning in emacs 29+") This is my first bug report, so I’m not
> sure the etiquette, haha.

No, please keep replying with the original Subject.  That other thread
was a side issue.

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Mitchell]

[-- Attachment #1: Type: text/plain, Size: 2542 bytes --]

I am not a programmer by trade. I did previously manage to build the latest
version of emacs on Windows as explained at
https://readingworldmagazine.com/emacs/2022-02-24-compiling-emacs-29-from-source-on-windows/
, but after several hours trying to compile a version of emacs rolled back
to just before that commit, I haven’t had luck. I git-cloned emacs master
to my machine, used `git checkout 8783700b23e70874c4996908bf02c010ae6f3fe1^`
to narrow down to the parent of the commit in question, ran `./autogen.sh`
and then `./configure`, and finally tried `make`, but it keeps raising
errors. ChatGPT diagnoses the errors like this:

Warnings:
    calloc Argument Order: Multiple warnings about the incorrect order of
arguments in calloc calls. The first argument should specify the number of
elements, and the second argument should specify the size of each element.
For example:
           c
           newstate = (re_dfastate_t *) calloc(1, sizeof(re_dfastate_t));
    These warnings alone should not cause the build to fail but should be
corrected for code correctness and stability.
Errors in sysdep.c:
    Implicit Declaration of waitpid:
             sysdep.c:472:13: error: implicit declaration of function
'waitpid' [-Wimplicit-function-declaration]
            472 | pid = waitpid(child, status, options);
This error indicates that waitpid is being used without including the
proper header file that declares it.
Undeclared Identifier WNOHANG:
           sysdep.c:518:43: error: 'WNOHANG' undeclared (first use in this
function)
           518 | return get_child_status(child, status, WNOHANG | options,
0);
This suggests that WNOHANG is not defined, likely because the appropriate
headers are not included.
Control Reaches End of Non-Void Function:
           sysdep.c:519:1: warning: control reaches end of non-void
function [-Wreturn-type]
    519 | }
    This means the function is expected to return a value but doesn't in
all code paths.
Error in print.c:
    Storing the Address of Local Variable:
        process.c:7419:53: error: storing the address of local variable
'buf' in 'current_thread->stack_top' [-Wdangling-pointer=]
This indicates a potential issue with a dangling pointer, where the address
of a local variable is being assigned to a global or long-lived structure.

If you have any advice I’m happy to try compiling it again. Or perhaps Ihor
or Stefan would have more luck rolling back emacs to just before that
commit to confirm that’s the issue? Sorry!

[-- Attachment #2: Type: text/html, Size: 2944 bytes --]

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> From: Mitchell <mitchellahren@gmail.com>
> Date: Mon, 24 Jun 2024 01:09:29 -0600
> Cc: monnier@iro.umontreal.ca, 71644@debbugs.gnu.org
> 
> I am not a programmer by trade. I did previously manage to build the latest version of emacs on Windows as
> explained at
> https://readingworldmagazine.com/emacs/2022-02-24-compiling-emacs-29-from-source-on-windows/ , but
> after several hours trying to compile a version of emacs rolled back to just before that commit, I haven’t had
> luck. I git-cloned emacs master to my machine, used `git checkout
> 8783700b23e70874c4996908bf02c010ae6f3fe1^` to narrow down to the parent of the commit in question,
> ran `./autogen.sh` and then `./configure`, and finally tried `make`, but it keeps raising errors. ChatGPT
> diagnoses the errors like this: 

Sorry, I haven't realized you don't have a fully operational build
environment for Emacs.  The errors you show mean that your development
environment is broken.

So please forget this.

Maybe someone else could try reverting that change?  Stefan, could you
do that, since you see the regression wrt Emacs 28 on your system?

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Maybe someone else could try reverting that change?  Stefan, could you
> do that, since you see the regression wrt Emacs 28 on your system?

Removing that patch made no difference in my tests on Emacs-29.
[ And adding it to Emacs-28 also made no difference, in my tests.  ]


        Stefan

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Ihor Radchenko]

Mitchell <mitchellahren@gmail.com> writes:

> Here are the steps to reproduce the issue:
> ...

For the record, I am able to reproduce the problem using the latest
Emacs master.

And the problem does disappear with my patch that affects the marker
loop in src/marker.c (buf_bytepos_to_charpos)

So, markers appear to be the culprit on my system.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> Cc: 71644@debbugs.gnu.org
> From: Ihor Radchenko <yantar92@posteo.net>
> Date: Sat, 22 Jun 2024 13:53:08 +0000
> 
> Mitchell <mitchellahren@gmail.com> writes:
> 
> > Here are the steps to reproduce the issue:
> > ...
> 
> For the record, I am able to reproduce the problem using the latest
> Emacs master.
> 
> And the problem does disappear with my patch that affects the marker
> loop in src/marker.c (buf_bytepos_to_charpos)
> 
> So, markers appear to be the culprit on my system.

Not the markers, but conversion of character positions to buffer
positions.  Which uses markers, but that's an implementation detail...

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Eli Zaretskii]

> Cc: mitchellahren@gmail.com, 71644@debbugs.gnu.org
> Date: Sat, 22 Jun 2024 17:12:47 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Not the markers, but conversion of character positions to buffer
> positions.                                                ^^^^^^
  ^^^^^^^^^
Sorry, I meant byte positions, of course.

bug#71644: 30.0.50; Severe slowdown in larger files with markers beginning in emacs 29+

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> 1. Start emacs 29.3 (or 30.0.50) with emacs -q.
> 2. Paste the minimal config at
> https://gist.github.com/kings2u/30b7a22dfa38fe0d5dc18adaf0e5e9de into the
> scratch buffer and eval-buffer. (Note on my system I'm using the `counsel`
> version current as of counsel-20240502.743, but any recent version will
> reproduce the bug).

I used the code below:

    (customize-set-variable 'large-file-warning-threshold nil)
    (package-install 'counsel)
    (package-activate 'counsel)

> 7. Start typing `n` `space`, or `t` `space` several times very fast and
> observe how long it takes for the abbrev expansions (defined in the minimal
> config above) to show up on the screen.

Without running `counsel-outline`, the insertion is "immediate", whereas
after `counsel-outline` it takes about two seconds for me.

> Using emacs 29+, when I use `profiler-start` (CPU) between Steps 4 and 5,
> and then `profiler-stop` after Step 7, I get a very big entry (e.g., 70%)
> for `redisplay_internal (C function)` in the `profiler-report`. Using the
> `profiler` at the same points using emacs 28.2, I don’t get a big entry for
> `redisplay_internal (C function)`.

[ I haven't been able to get a meaningful profiler report yet.
  E.g. I get 82% of the time given to `read-from-minibuffer`, which is
  presumably the time spent typing `M-x profiler-report RET` or some
  such, even though I typed enough `n SPC` repetitions that it should
  dwarf that.  Not sure what's going on.  ]

Also I notice that if I do `n` and then wait, it takes the same
amount of time to see the result as if I do `n n n n`
and then wait.  So it seems that the command themselves execute
"instantly", and the delay occurs later (jit-lock? redisplay? ...).

I also notice that step 6 is important: if I just go straight to line 35
and type text, there's no delay.
Other thing I noticed

   M-g M-g 60000 RET
   n n n n n
   M-g M-g 35 RET
   n n n n n

there's no delays typing the `n` at line 60000 but there is a 2s delay
at line 35.

The fact that the absence of non-ASCII chars eliminates the problem
strongly points the finger to the bytes<->chars conversion which relies
on markers, indeed, but it's still very puzzling: line 35 is about 1.5kB
from the beginning of the buffer, so the conversion should be simple
and fast. there.  IOW, if it's a slow bytes<->chars conversion it's
probably a conversion taking place elsewhere than near point.

AFAICT the most significant change w.r.t markers in Emacs-29 is that
markers aren't used internally for overlays any more, so there are
usually *fewer* markers in Emacs-29 buffers than in Emacs-28 buffers
(which can be both favorable and detrimental to bytes<->chars
conversion, depending on details).  But in any case, (overlays-in
(point-min) (point-max)) tells me there's only one overlay and even 0 in
Emacs-28, so that can't be the reason for the change.


        Stefan "puzzled"