bug#58687: 29.0.50; Enabling pp-use-max-width dramatically slows down formatting of large sexps like org-persist--index

bug#58687: 29.0.50; Enabling pp-use-max-width dramatically slows down formatting of large sexps like org-persist--index

[Michael Eliachevitch]

[-- Attachment #1: Type: text/plain, Size: 5332 bytes --]

I set the new setting `pp-use-max-width' to t to fold the output of interactive commands  like `pp-eval-last-sexp'. However, I found out that this increased the time that kill-emacs took to run by 30 seconds. By profiling I found that this is because I use the latest org-version (9.5.5-gcb1359) on the main branch with persistent caching enabled and in `kill-emacs-hook' it then saves the `org-persist--index' to a file via `org-persist-write:index'. The index sexp can be quite large when one has many org files and when org uses pp on it that takes a long time.

What I wasn't aware when I customized pp-use-max-width is that it's used by other packages to format lisp code and this might slow down these operations quiet significantly, as I just wanted to set only for my custom pretty-printing purposes of usually small sexps.

My suggestion is to put a note into the emacs-news and the variable docstring that it can have significant performance penalties on large sexps. If the performance can be improved that would be also nice, but not sure if that's possible. I assume the authors are aware of the downsides, but it then should be documented well at least.

I attached a file with the value of my `org-persist--index` expression and a file with a benchmark where I run pp on it, which took me 25s when running it with emacs -Q. Until recently my org persist index was much longer, but I pruned it a bit back when I wasn't aware what exactly caused the slowdown.

I had reported this first on the org-mode mailing list at https://lists.gnu.org/archive/html/emacs-orgmode/2022-10/msg00734.html.

Best regards,
Michael Eliachevitch

--
In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.34, cairo version 1.17.6) of 2022-10-20 built on e490
Repository revision: f61db42fc580fb671016c77be942506d9081ac2c
Repository branch: master
System Description: Arch Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
 --with-modules --without-libotf --without-m17n-flt --without-gconf
 --enable-link-time-optimization --with-native-compilation
 --with-xinput2 --with-pgtk --without-xaw3d --with-sound=alsa
 --with-xwidgets --without-gpm --without-compress-install
 '--program-transform-name=s/\([ec]tags\)/\1.emacs/'
 'CFLAGS=-march=native -mtune=generic -O3 -pipe -fno-plt -fexceptions
 -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
 -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK
PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS WEBP XIM
XWIDGETS GTK3 ZLIB

Important settings:
  value of $XMODIFIERS: @im=fcitx
  locale-coding-system: nil

Major mode: ELisp/d

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils time-date edebug debug
backtrace find-func benchmark pp vc-git diff-mode easy-mmode
vc-dispatcher cl-loaddefs comp comp-cstr warnings icons subr-x rx cl-seq
cl-macs gv cl-extra help-mode bytecomp byte-compile cconv cl-lib rmc
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win
pgtk-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list
replace newcomment text-mode lisp-mode prog-mode register page tab-bar
menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse
jit-lock font-lock syntax font-core term/tty-colors frame minibuffer
nadvice seq simple cl-generic indonesian philippine cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop
case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads xwidget-internal dbusbind inotify dynamic-setting
system-font-setting font-render-setting cairo gtk pgtk lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 92875 7823)
 (symbols 48 8353 1)
 (strings 32 23567 2442)
 (string-bytes 1 712274)
 (vectors 16 17653)
 (vector-slots 8 356365 12834)
 (floats 8 30 53)
 (intervals 56 434 3)
 (buffers 1000 15))


[-- Attachment #2: org-persist-index.el --]
[-- Type: text/plain, Size: 15036 bytes --]

((:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "40/15aa22-87f7-47a7-a2d4-c911234c0cbc" :associated
  (:hash "57b5678236c2a9be9559b1a111b26eeb" :file "/home/michael/org/roam/20210331114711-support_vector_mashine.org" :inode 360063)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "48/be5c27-7287-4744-b440-1529ae74d243" :associated
  (:hash "8093afef7c3f2138f68dd75307688e52" :file "/home/michael/org/roam/20210721102303-continuum_suppression.org" :inode 360323)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "d8/3ec555-87f6-4ff2-a2d4-b2e581ab0e08" :associated
  (:hash "3cf1d3e7f757a4770b522749d48a9fea" :file "/home/michael/org/roam/20210929174243-superkekb.org" :inode 358863)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "4a/8f0169-05c8-4715-bf34-67703759e669" :associated
  (:hash "dd1c342a277169bbedd1de404e2e295e" :file "/home/michael/org/roam/journal/2021-11-26.org" :inode 358640)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "61/c2b5cf-7625-4fed-b817-3ca5779c6461" :associated
  (:hash "c30502418aba98af28f9d04ab69b7e3b" :file "/home/michael/org/roam/20201028235900-max_welsch.org" :inode 359176)
  :expiry 30 :last-access 1665751762.0571246 :last-access-hr "2022-10-14T14:49:22+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "5f/a39411-e67d-4e42-99f6-61979afccf47" :associated
  (:hash "4d4b05e99bbe33de47f2f14ac4d88546" :file "/home/michael/org/roam/20211002114419-expansion_des_raumes.org" :inode 358861)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "8a/148ed3-4fe0-4708-8acc-6efdc57efe3f" :associated
  (:hash "0c10f424cb2bf36e6ba9304c017ef41a" :file "/home/michael/org/roam/20220824152922-effort.org" :inode 360072)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "23/12f9fe-2740-4a80-9818-ad064da54f25" :associated
  (:hash "6401e21d9a24812a083f13b5c9019f26" :file "/home/michael/org/roam/20220116211118-interception_tools.org" :inode 360472)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "91/095a9e-9954-40ad-a985-e1997c607a49" :associated
  (:hash "77ba74e6aa78b5b9a2eb04524775cd1d" :file "/home/michael/org/roam/20201104153018-janina.org" :inode 358946)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "d8/89dfd7-8d3e-4a99-a156-26826061434b" :associated
  (:hash "31e73df707b382b344e30ec3e7823d64" :file "/home/michael/org/clock.org" :inode 358069)
  :expiry 3 :last-access 1666304677.3369315 :last-access-hr "2022-10-21T00:24:37+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "44/a9740a-22db-41db-9a1c-307958c6932d" :associated
  (:hash "68c3e2e50713d630394673886cb11bd7" :file "/home/michael/blog/content-org/blog.org" :inode 353724)
  :expiry 30 :last-access 1665501728.8860543 :last-access-hr "2022-10-11T17:22:08+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "d5/7a088d-daf2-4b5f-89b4-53dd620a04aa" :associated
  (:hash "0306b42a733aedb706ec5bd1777ab54f" :file "/home/michael/org/roam/20210405235900-tenside.org" :inode 360069)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "8d/e479fa-98ab-4d7f-bc60-42dc54cec4c0" :associated
  (:hash "fd304b811f165936a279fb6b742e4a43" :file "/home/michael/org/roam/ref/thebellecollaborationPreciseDeterminationCKM2017.org" :inode 359691)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "52/061733-000e-47d5-b7c0-c9772df9a669" :associated
  (:hash "efd7ad53e443ed89c6028ed281b547d0" :file "/home/michael/org/roam/journal/2022-02-09.org" :inode 358673)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "a9/50a965-eb81-4236-ae95-43a06f2139c2" :associated
  (:hash "2543ebe12fa66005245696443b01c006" :file "/home/michael/org/roam/20201027182903-templatefitter.org" :inode 359135)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "33/a629f8-534f-40bf-98a5-d5b7cc25dbc0" :associated
  (:hash "230337252e613be224db0d894bcbbb78" :file "/home/michael/org/roam/20211014104529-a_fb.org" :inode 360364)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "c3/5d43d0-eaa6-435b-8af2-e77fe52d5787" :associated
  (:hash "c7133d0a0f285a3db98b47a3daf6be4d" :file "/home/michael/org/roam/20210105145348-staat.org" :inode 360036)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "40/ea9187-8f3a-4f40-a0ea-716fba67015c" :associated
  (:hash "789bba672930f723b2a715f7e21e8d8e" :file "/home/michael/org/roam/20221006125743-großbrittanien.org" :inode 2131510)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "ba/0fce57-afb2-44c0-9d53-70e1cd6cd708" :associated
  (:hash "8c32a94fbca69e72f286c7ba1411e816" :file "/home/michael/.emacs.d/etc/elfeed/feeds.org" :inode 362775)
  :expiry 30 :last-access 1665751762.0688756 :last-access-hr "2022-10-14T14:49:22+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "58/31ced8-062e-4a40-9a2b-f475b9a62d7d" :associated
  (:hash "5d33ffc716f1dafdd357d117a8f964fd" :file "/home/michael/.emacs.d/var/treemacs/persist.org" :inode 3059083)
  :expiry 30 :last-access 1666298484.2475097 :last-access-hr "2022-10-20T22:41:24+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "af/1b8808-f155-4891-a7bf-ac7c73bca041" :associated
  (:hash "85ab77d863ac443ff1925e23548b0d3f" :file "/home/michael/org/roam/20220215235900-decision_making.org" :inode 360505)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "f6/60a896-73be-4e16-a7ef-4d14e6f6ca0e" :associated
  (:hash "c1c7254bee6a076ca857847e6d83feb8" :file "/home/michael/org/roam/ref/abdesselamMeasurementCKMMatrix2018.org" :inode 359692)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "d4/e7aeb6-a4b3-42b7-8cbd-639eb01c432e" :associated
  (:hash "336104ea2aba5581ea5fb6bf2dd297d2" :file "/home/michael/org/roam/references/ferlewiczRevisitingFitsEll2021.org" :inode 360376)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "81/9afc0f-714e-439a-9ec3-3bc3e5d74626" :associated
  (:hash "dfc8a8a18023f1809168faeb9fc75d9d" :file "/home/michael/org/roam/20210305191300-laufen.org" :inode 360138)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "cb/26c4ae-ef44-47b0-9763-b9b152d4a64f" :associated
  (:hash "8b6179aede2b32bb9b38bace6ffc0d52" :file "/home/michael/org/roam/journal/2021-10-17.org" :inode 358391)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "d4/7857bd-0623-4299-b147-7fc40b514d79" :associated
  (:hash "6a9288b3fd59f54f046bb4982b3bae0e" :file "/home/michael/org/roam/20210727164202-allgemeine_relativitatstheorie.org" :inode 359082)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "f5/d0a1dd-9adf-43fb-bbdf-1b9f5953805f" :associated
  (:hash "063d9a48030b6ab806234630e9e5c62a" :file "/home/michael/org/roam/20220121140806-systematiken_in_meiner_analyse.org" :inode 360478)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "93/00e296-5f3b-4dd6-9279-f9adb486f314" :associated
  (:hash "97c28d67e853bfba6b3ed328c24a5e9a" :file "/home/michael/org/roam/20220811133340-vertrag.org" :inode 359171)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "a2/6669ce-b654-4b04-9e49-b26d40271c6d" :associated
  (:hash "9329d34a7033257dd13ac5a5b264f237" :file "/home/michael/org/roam/web/20220905140331-what_s_virtual_memory_computerphile_youtube.org" :inode 359601)
  :expiry 30)
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "6c/1b2b2c-82b2-49c3-bc4c-8fba6f96e99a" :associated
  (:hash "13c9936c0b80d610cb741132ab6d2887" :file "/home/michael/org/roam/bücher/buecher.org" :inode 359840)
  :expiry 30 :last-access 1665134302.1608012 :last-access-hr "2022-10-07T11:18:22+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "ec/9fcc17-9cab-46eb-9ecc-c6322d9ae14f" :associated
  (:hash "ec66c29b47c8f188c3aba5bb137d8d1d" :file "/home/michael/org/roam/20211215141559-dpg_tagung_heidelberg_2022.org" :inode 360394)
  :expiry 30 :last-access 1665756769.3734863 :last-access-hr "2022-10-14T16:12:49+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "c5/e125eb-20c4-4248-860a-572a66e22b29" :associated
  (:hash "1257d8f84ec48d056586716878b221f4" :file "/home/michael/org/roam/journal/2021-09-14.org" :inode 358240)
  :expiry 30 :last-access 1665756769.376799 :last-access-hr "2022-10-14T16:12:49+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "76/7ddbaf-2fc5-40be-a117-ff8b5ed16256" :associated
  (:hash "493b033bf2108f51ebc37bfc80479c3a" :file "/home/michael/org/waage.org" :inode 357907)
  :expiry 30 :last-access 1665756769.3804295 :last-access-hr "2022-10-14T16:12:49+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "15/489bd0-ece7-425f-8d71-74a43e15a9c6" :associated
  (:hash "4994954ffdcd956233b713431976a762" :file "/home/michael/org/sport.org" :inode 358072)
  :expiry 30 :last-access 1665756769.3844242 :last-access-hr "2022-10-14T16:12:49+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "c2/644553-b85d-44a7-acb5-ade4e051d1b6" :associated
  (:hash "c32f6ae7e1fb2a81fd1961d59994bd40" :file "/home/michael/org/organizer.org" :inode 358068)
  :expiry 30 :last-access 1665756769.3883088 :last-access-hr "2022-10-14T16:12:49+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "db/6a4d58-5f7e-438a-9ab2-27bd8b064f08" :associated
  (:hash "0e039cdc585f370c879cf80e70f9e70b" :file "/home/michael/org/notes.org" :inode 358074)
  :expiry 30 :last-access 1665756769.3921976 :last-access-hr "2022-10-14T16:12:49+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "78/50a761-6b71-4a4d-a272-bcfa4db988fb" :associated
  (:hash "5b6ca65396519632c8abcd8a9a185dd6" :file "/home/michael/org/medikamente.org" :inode 358058)
  :expiry 30 :last-access 1665756769.395902 :last-access-hr "2022-10-14T16:12:49+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "44/0b0587-ca52-4bbb-8756-241175216edd" :associated
  (:hash "d3e0d11106f6b49fd331925fdad6ccc7" :file "/home/michael/org/kontakte.org" :inode 357867)
  :expiry 30 :last-access 1665756769.399829 :last-access-hr "2022-10-14T16:12:49+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "44/056f64-b843-4bc0-9555-70b170224a8a" :associated
  (:hash "b672eb802ec10cdb5227c34afa96de8d" :file "/home/michael/org/kaufen.org" :inode 2677171)
  :expiry 30 :last-access 1665756769.403763 :last-access-hr "2022-10-14T16:12:49+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "ae/eb0d79-6144-44e5-8452-82cd0f05dbe3" :associated
  (:hash "98e58f6e8253f1ae827eabeedd6ccdb2" :file "/home/michael/org/habits.org" :inode 1643626)
  :expiry 30 :last-access 1665756769.407665 :last-access-hr "2022-10-14T16:12:49+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "e6/0f34a6-7c5d-4a8a-b5ed-0bf844483559" :associated
  (:hash "1dd1823e47a04a2d0ad19ec21f801bb0" :file "/home/michael/org/calendar.org" :inode 358034)
  :expiry 30 :last-access 1665756769.411561 :last-access-hr "2022-10-14T16:12:49+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "ce/dfe116-6563-4a7b-8c7f-e065c38d6365" :associated
  (:hash "601c17471acb222cf12adef1492691da" :file "/home/michael/org/roam/journal/2022-09-12.org" :inode 701031)
  :expiry 3 :last-access 1666304677.3428493 :last-access-hr "2022-10-21T00:24:37+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "33/f3af90-7344-477e-a943-fded6ff14fa4" :associated
  (:hash "3984beca12e81a4342aa8db3af7673e6" :file "/home/michael/org/archive.org_archive" :inode 357866)
  :expiry 3 :last-access 1666304677.353239 :last-access-hr "2022-10-21T00:24:37+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "d1/86f70a-c6b4-4f38-9610-4a231396c27a" :associated
  (:hash "79000244264d75f5a31734dd91ff01af" :file "/home/michael/org/roam/journal/2022-08-24.org" :inode 358385)
  :expiry 3 :last-access 1666304677.8559783 :last-access-hr "2022-10-21T00:24:37+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "33/5c04cd-d96d-4ee3-a41c-34dd23e2ffa3" :associated
  (:hash "45b163593cbec35a17b7faa2f45dec84" :file "/home/michael/org/roam/journal/2022-06-20.org" :inode 358370)
  :expiry 3 :last-access 1666304677.859343 :last-access-hr "2022-10-21T00:24:37+0200")
 (:container
  ((elisp org-element--headline-cache)
   (elisp org-element--cache))
  :persist-file "63/5cd43b-0753-4f05-b9e8-72fc510d418d" :associated
  (:hash "25b71687d558a10152565ce983f61269" :file "/home/michael/org/phd.org" :inode 358033)
  :expiry 3 :last-access 1666304677.862949 :last-access-hr "2022-10-21T00:24:37+0200")
 (:container
  ((index "2.5"))
  :persist-file "85/7529d0-72fe-411e-b18d-7422afbfad12" :associated nil :expiry never :last-access 1666310369.8001707 :last-access-hr "2022-10-21T01:59:29+0200"))

[-- Attachment #3: benchmark-pp-on-org-persist-index.el --]
[-- Type: text/plain, Size: 307 bytes --]

;; straight bootstrapping code
(require 'pp)

(let ((pp-use-max-width t)
      (pp-max-width 80)
      (org-persist--index
       (with-current-buffer (find-file-noselect "org-persist-index.el")
         (goto-char (point-min))
         (read (current-buffer)))))
  (benchmark-run (pp org-persist--index)))

bug#58687: 29.0.50; Enabling pp-use-max-width dramatically slows down formatting of large sexps like org-persist--index

[Ihor Radchenko]

Michael Eliachevitch <m.eliachevitch@posteo.de> writes:

> I attached a file with the value of my `org-persist--index` expression and a file with a benchmark where I run pp on it, which took me 25s when running it with emacs -Q. Until recently my org persist index was much longer, but I pruned it a bit back when I wasn't aware what exactly caused the slowdown.

I'd like to bump this bug.

If it helps, here are the steps to reproduce:

1. Save the attached files to same folder
2. Open benchmark-pp-on-org-persist-index.el
3. M-x eval-buffer

Observed: (19.617422592 2 1.0353073050000035)
Expected: `pp' not taking ~20sec to write 15k of elisp data.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#58687: 29.0.50; Enabling pp-use-max-width dramatically slows down formatting of large sexps like org-persist--index

[Eli Zaretskii]

> Cc: 58687@debbugs.gnu.org
> From: Ihor Radchenko <yantar92@posteo.net>
> Date: Thu, 12 Jan 2023 11:27:35 +0000
> 
> Michael Eliachevitch <m.eliachevitch@posteo.de> writes:
> 
> > I attached a file with the value of my `org-persist--index` expression and a file with a benchmark where I run pp on it, which took me 25s when running it with emacs -Q. Until recently my org persist index was much longer, but I pruned it a bit back when I wasn't aware what exactly caused the slowdown.
> 
> I'd like to bump this bug.

Why?

> 1. Save the attached files to same folder
> 2. Open benchmark-pp-on-org-persist-index.el
> 3. M-x eval-buffer
> 
> Observed: (19.617422592 2 1.0353073050000035)
> Expected: `pp' not taking ~20sec to write 15k of elisp data.

Did you look at what pp.el does when pp-use-max-width is non-nil?  I
show a profile below, to make that clear.

The "regular" pp (when pp-use-max-width is nil) finishes almost
instantaneously in this case.

       40048  85% - command-execute
       40048  85%  - call-interactively
       40046  85%   - funcall-interactively
       40046  85%    - execute-extended-command
       40044  85%     - command-execute
       40044  85%      - call-interactively
       40044  85%       - funcall-interactively
       40044  85%        - eval-buffer
       40043  85%         - let
       40042  85%          - benchmark-call
       40041  85%           - #<lambda 0x5461c05f>
       40041  85%            - pp
       40041  85%             - pp-to-string
       40041  85%              - pp-emacs-lisp-code
       40028  85%               - pp--insert-lisp
       40028  85%                - pp--format-list
       40028  85%                 - pp--insert
       38710  82%                  - pp--insert-lisp
       38710  82%                   - pp--format-list
       38708  82%                    - pp--insert
       20781  44%                     - pp--insert-lisp
       20780  44%                      - pp--format-list
       20780  44%                       - pp--insert
       13619  28%                        - pp--indent-buffer
       13603  28%                         - lisp-indent-line
       12595  26%                          - lisp-ppss
        9763  20%                           - syntax-ppss
           6   0%                              #<compiled -0x163d6f700e816ab6>
           1   0%                              syntax-propertize

bug#58687: 29.0.50; Enabling pp-use-max-width dramatically slows down formatting of large sexps like org-persist--index

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > I attached a file with the value of my `org-persist--index` expression and a file with a benchmark where I run pp on it, which took me 25s when running it with emacs -Q. Until recently my org persist index was much longer, but I pruned it a bit back when I wasn't aware what exactly caused the slowdown.
>> 
>> I'd like to bump this bug.
>
> Why?

Because, I'd like to know if there are chances that it could be fixed.
The original report demonstrates how this issue slows down Emacs quit
time dramatically. I should either let-bind `pp-use-max-with' to nil in
org-persist.el to work around the problem or wait for the fix.

>> Observed: (19.617422592 2 1.0353073050000035)
>> Expected: `pp' not taking ~20sec to write 15k of elisp data.
>
> Did you look at what pp.el does when pp-use-max-width is non-nil?  I
> show a profile below, to make that clear.
>
> The "regular" pp (when pp-use-max-width is nil) finishes almost
> instantaneously in this case.
> ...
>        40041  85%            - pp
>        40041  85%             - pp-to-string
>        40041  85%              - pp-emacs-lisp-code
>        40028  85%               - pp--insert-lisp
>        40028  85%                - pp--format-list
>        40028  85%                 - pp--insert
>        38710  82%                  - pp--insert-lisp
>        38710  82%                   - pp--format-list

So, the current `pp' implementation is re-parsing from bob for every
nested list inside sexp. This is quadratic scaling, and, as the repro
demonstrates, the time goes up very quickly. Is the new option usable
at all in practice?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#58687: 29.0.50; Enabling pp-use-max-width dramatically slows down formatting of large sexps like org-persist--index

[Visuwesh]

[வியாழன் ஜனவரி 12, 2023] Ihor Radchenko wrote:

> [...]
> So, the current `pp' implementation is re-parsing from bob for every
> nested list inside sexp. This is quadratic scaling, and, as the repro
> demonstrates, the time goes up very quickly. Is the new option usable
> at all in practice?

IME, unfortunately not.  Like the OP, I saw the announcement of the new
user option, got excited and enabled it right away... only to be hit by
bugs more often than not, add to it the slowness, it wasn't a fun
experience.
Personally, I always thought it would be best if the user facing
commands like pp-eval-sexp and friends alone respected the user option.

bug#58687: 29.0.50; Enabling pp-use-max-width dramatically slows down formatting of large sexps like org-persist--index

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: m.eliachevitch@posteo.de, 58687@debbugs.gnu.org
> Date: Thu, 12 Jan 2023 16:19:37 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >        40041  85%            - pp
> >        40041  85%             - pp-to-string
> >        40041  85%              - pp-emacs-lisp-code
> >        40028  85%               - pp--insert-lisp
> >        40028  85%                - pp--format-list
> >        40028  85%                 - pp--insert
> >        38710  82%                  - pp--insert-lisp
> >        38710  82%                   - pp--format-list
> 
> So, the current `pp' implementation is re-parsing from bob for every
> nested list inside sexp.

Not "the current 'pp'", but the implementation for this optional
behavior.

> This is quadratic scaling, and, as the repro demonstrates, the time
> goes up very quickly. Is the new option usable at all in practice?

Maybe only for small code fragments.

bug#58687: Fwd: Re: bug#58687: 29.0.50; Enabling pp-use-max-width dramatically slows down formatting of large sexps like org-persist--index

[Michael Eliachevitch]

Sorry for the noise, but I need to resend (i.e. re-cc) this email to 58687@debbugs.gnu.org. I had sent it already to that list earlier, but my e-mail provider had blocked it because I recently enabled the "TLS-sending guarantee" it had blocked 58687@debbugs.gnu.org, therefore I disabled that setting again. Sorry for the noise.

To: Eli Zaretskii <eliz@gnu.org>
Cc: Ihor Radchenko <yantar92@posteo.net>, 58687@debbugs.gnu.org

On 2023-01-12 at 23:22 +01, Michael Eliachevitch <m.eliachevitch@posteo.de> wrote:

> On 2023-01-12 at 18:39 +02, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> Not "the current 'pp'", but the implementation for this optional
>> behavior.
>
> This is optional behavior, but I would prefer if the performance impact of
> enabling this optional behavior would be documented, e.g. in the variable
> docstring and NEWS.29. It's disabled by default and will only be used by those
> like me and Visuwesh who the documentation and news, so I'm fine with it being
> almost unusably slow if that's documented. Though better performance if possible
> would be nice of course.
>
> On 2023-01-12 at 22:03 +0530, Visuwesh <visuweshm@gmail.com> wrote:
>
>> Personally, I always thought it would be best if the user facing
>> commands like pp-eval-sexp and friends alone respected the user option.
>
> Sounds like a good idea. I definetly didn't expect that me personally setting
> this option for myself would affect how lisp objects are serialized to disk in
> external packages. Not sure if I would only enable it for interactive commands
> and the like, in the emacs-world we are all hackers and hard to say what is
> user-facing. People might write their own functions using pp on small s-exps,
> and wonder why this setting isn't doing anything. Not sure what's the best
> approach there
>
> Maybe pp isn't meant to be used for doing anything that's not meant primarily
> for human eyes, like serialization of lisp objects, maybe it's an error on
> package maintainers that use it that way, but at least that could also somehow
> be communicated more clearly to them.


--
Michael Eliachevitch
Public PGP Key: https://keyoxide.org/hkp/546908c782383ad0e7d894ec1b8f95c8125dce31

bug#58687: 29.0.50; Enabling pp-use-max-width dramatically slows down formatting of large sexps like org-persist--index

[Eli Zaretskii]

> From: Michael Eliachevitch <m.eliachevitch@posteo.de>
> Cc: Ihor Radchenko <yantar92@posteo.net>, 58687@debbugs.gnu.org
> Date: Thu, 12 Jan 2023 22:22:20 +0000
> 
> On 2023-01-12 at 18:39 +02, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Not "the current 'pp'", but the implementation for this optional
> > behavior.
> 
> This is optional behavior, but I would prefer if the performance impact of enabling this optional behavior would be documented, e.g. in the variable docstring and NEWS.29.

Done.

> On 2023-01-12 at 22:03 +0530, Visuwesh <visuweshm@gmail.com> wrote:
> 
> > Personally, I always thought it would be best if the user facing
> > commands like pp-eval-sexp and friends alone respected the user option.
> 
> Sounds like a good idea.

Also done.  (Most of the functions already honored this option, but 2
of them didn't.)

bug#58687: 29.0.50; Enabling pp-use-max-width dramatically slows down formatting of large sexps like org-persist--index

[Ihor Radchenko]

Michael Eliachevitch <m.eliachevitch@posteo.de> writes:

>> Personally, I always thought it would be best if the user facing
>> commands like pp-eval-sexp and friends alone respected the user option.
>
> Sounds like a good idea. I definetly didn't expect that me personally setting this option for myself would affect how lisp objects are serialized to disk in external packages. Not sure if I would only enable it for interactive commands and the like, in the emacs-world we are all hackers and hard to say what is user-facing. People might write their own functions using pp on small s-exps, and wonder why this setting isn't doing anything. Not sure what's the best approach there
>
> Maybe pp isn't meant to be used for doing anything that's not meant primarily for human eyes, like serialization of lisp objects, maybe it's an error on package maintainers that use it that way, but at least that could also somehow be communicated more clearly to them.

In org-persist, `pp' is used because "index" file might be something
users may want to check manually. Following the notion that Elisp data
should be something consumable by humans, if possible.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>