31.0.50; Org Persist file formatting

31.0.50; Org Persist file formatting

[Michael Mauger]

This bug has been difficult to reproduce but several people have
encountered it, or symptoms of it. Most recently, there has been this
thread on Reddit:
https://www.reddit.com/r/emacs/comments/1drxsz6/tangled_0_code_blocks_from_filename_problem/

Unfortunately, replicating on emacs -Q has proven impossible since -Q
disables some of the implicated features. My Emacs config is contained 
in Org Babel files that are invoked with `org-babel-tangle-file' and have
demonstrated the error multiple times.

After some poking around I started getting failure when I tried to
reopen an Org file or ran `normal-mode' on it. Shutting down emacs and
restarting cleared the issue. When the issue arose when reopening the
file within a session, the backtrace was pointing to `plistp' failing in
`org-persist-read'. When I looked at the persist index there were
entries like:

     (:container
      ((elisp org-element--headline-cache) (elisp org-element--cache)
       (version "2.3"))
      :persist-file "49/605653-361a-49a4-a000-47db8d522096" :associated
      (:hash "aa1176747642fdac45aaf96095a88367" :file
    	 "/home/michael/Projects/my-config/emacs/my-org.org" :inode
    	 2554490)
      :expiry 30 :last-access 1719791738.4428508 :last-access-hr
      "2024-06-30T19:55:38-0400" ...)

(Note: the "..." in the last entry; sometimes it showed up as "\...".)

The failure during tangling occurred with `org-get-heading' expecting
`stringp' but getting `nil'. That seemed to be tracked down to
`org-complex-heading-regexp' not being set yet. If an error had aborted
out of the `org-persist-read' it appears we could have half of org-mode
up but not everything properly initialized.

Looking at the code in `org-persist.el' in
`org-persist--write-elisp-file (file data &optional no-circular pp)'
around line 486 it handles calling `pp' or `prin1'. The code for `pp'
uses `pp-use-max-width' (as part of a bug fix #58687) which has since
been superseded in Emacs 30.

It looks like the `org-persist-write' code ought to be updated and the
`org-persist-read' code made more robust so that it does not just shut
down processing of Org files entirely. That is if the entry is not a
plist, act as though no entry exists and proceed as if it were a new
org file. 

By deleting the persist index file, the calls to `org-babel-tangle-file'
succeed and I proceed without error. If an entry in the persist index
has the elided entries, then the tangling fails.

I'm not entirely sure that this is the recipe but I have seen some 
weirdness around the `org-persist' feature but fixes other than deleting
persistent storage seem to be inconsistent.

Please feel free to reach out if you'd like me to verify or try some other 
recipe.

In GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.42, cairo version 1.18.0) of 2024-06-29 built on michael-laptop
Repository revision: fcd6403d8ebf7ce6e61ba17971d4fdaaaaecf06c
Repository branch: master
System Description: Fedora Linux 40 (Workstation Edition)
Package: Org mode version 9.7.5 (release_9.7.5-9-ga091ca @ /usr/local/share/emacs/31.0.50/lisp/org/)

Configured using:
 'configure --with-pgtk --with-native-compilation
 --with-file-notification=yes --with-wide-int --with-tree-sitter
 --with-mailutils 'CFLAGS= -O0 -g''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY
PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Help

Minor modes in effect:
  server-mode: t
  org-popup-posframe-mode: t
  transient-posframe-mode: t
  treemacs-filewatch-mode: t
  treemacs-follow-mode: t
  treemacs-git-mode: t
  treemacs-fringe-indicator-mode: t
  global-org-modern-mode: t
  repeat-mode: t
  global-dash-fontify-mode: t
  global-diff-hl-mode: t
  global-flycheck-mode: t
  company-posframe-mode: t
  recentf-mode: t
  global-company-mode: t
  company-mode: t
  savehist-mode: t
  vertico-posframe-mode: t
  vertico-mode: t
  which-key-posframe-mode: t
  which-key-mode: t
  which-function-mode: t
  electric-pair-mode: t
  delete-selection-mode: t
  auto-insert-mode: t
  save-place-mode: t
  my-region-size-mode: t
  override-global-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-layout-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  global-prettify-symbols-mode: t
  tab-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  isearch-fold-quotes-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/michael/.config/emacs/elpa/modus-themes-4.4.0/theme-loaddefs hides /usr/local/share/emacs/31.0.50/lisp/theme-loaddefs

Features:
(shadow sort mail-extr emacsbug shortdoc cl-print cursor-sensor misearch
multi-isearch add-log descr-text server network-stream url-http url-gw
nsm url-cache url-auth flyspell ispell org-auto-tangle async face-remap
org-popup-posframe vc-hg vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs
em-unix em-term term disp-table ehelp em-script em-prompt em-pred em-ls
em-glob em-extpipe em-cmpl em-dirs em-basic em-banner home-config
geiser-guile info-look transient-posframe transient geiser-debug
geiser-repl geiser-image geiser-capf geiser-doc geiser-menu
geiser-autodoc geiser-edit geiser-completion geiser-eval
geiser-connection tq geiser-syntax scheme geiser-log geiser-popup eglot
external-completion jsonrpc flymake ert debug warnings em-alias fortune
xkcd let-alist treemacs-tab-bar treemacs-tags treemacs
treemacs-header-line treemacs-compatibility treemacs-mode
treemacs-interface treemacs-persistence treemacs-filewatch-mode
treemacs-follow-mode treemacs-rendering treemacs-annotations
treemacs-async treemacs-workspaces treemacs-dom treemacs-visuals
treemacs-fringe-indicator treemacs-scope pulse treemacs-faces
treemacs-macros treemacs-icons treemacs-themes treemacs-core-utils
treemacs-logging treemacs-customization pfuture ht s org-indent em-hist
esh-mode esh-var time-stamp org-modern ob-sqlite ob-sql ob-shell shell
ob-sed ob-screen ob-scheme geiser-impl help-fns radix-tree geiser-custom
geiser-base geiser ob-sass ob-ruby ob-python python ob-plantuml
ob-processing ob-perl ob-org ob-octave ob-ocaml ob-maxima ob-makefile
ob-lua ob-lisp ob-lilypond ob-latex ob-julia ob-js ob-java ob-haskell
ob-groovy ob-gnuplot ox-odt rng-loc rng-uri rng-parse rng-match rng-dt
rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok nxml-util ox-latex
ox-icalendar org-agenda ox-html table ox-ascii ox-publish ox ob-fortran
ob-forth ob-eshell eshell esh-cmd esh-ext esh-opt esh-proc esh-io
esh-arg esh-module esh-module-loaddefs esh-util files-x ob-dot ob-ditaa
ob-css ob-clojure ob-calc calc-store calc-trail calc-ext calc
calc-loaddefs calc-macs ob-R ob-C cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs ob-awk cus-edit pp
cus-load flylisp highlight-quoted ruler-mode annotate fic-mode num3-mode
display-line-numbers vc-git hl-line goto-addr image-file image-converter
oc-basic org-element org-persist org-id org-refile org-element-ast
inline avl-tree ol-eww eww xdg url-queue mm-url ol-rmail ol-mhe ol-irc
ol-info ol-gnus nnselect gnus-art mm-uu mml2015 mm-view mml-smime smime
gnutls dig gnus-sum shr pixel-fill kinsoku url-file svg dom gnus-group
gnus-undo gnus-start gnus-dbus dbus compile comp-run comp-common xml
gnus-cloud nnimap nnmail mail-source utf7 nnoo parse-time gnus-spec
gnus-int gnus-range message sendmail yank-media puny rfc822 mml mml-sec
epa derived epg rfc6068 epg-config mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev gmm-utils
mailheader gnus-win gnus nnheader gnus-util text-property-search
mail-utils range mm-util mail-prsvr ol-docview doc-view filenotify
jka-compr image-mode exif dired dired-loaddefs ol-bibtex bibtex iso8601
ol-bbdb ol-w3m ol-doi org-link-doi sql view repeat dash smerge-mode diff
diff-hl log-view pcvs-util vc-dir ewoc vc vc-dispatcher diff-mode
track-changes flycheck lisp-mnt company-posframe company-oddmuse
company-keywords company-gtags company-dabbrev-code company-files
company-clang company-cmake company-semantic company-template
company-bbdb company-capf company-etags etags fileloop generator xref
recentf tree-widget wid-edit project company-dabbrev company orderless
savehist vertico-posframe vertico-multiform vertico compat
which-key-posframe which-key which-func imenu posframe
modus-vivendi-tinted-theme modus-themes ffap edmacro kmacro elec-pair
delsel autoinsert saveplace color my-region-size-mode rect avoid
diminish cl-extra backtrace help-mode use-package use-package-ensure
use-package-delight use-package-diminish use-package-bind-key bind-key
easy-mmode use-package-core finder-inf pcase org ob ob-tangle ob-ref
ob-lob ob-table ob-exp org-macro org-src sh-script smie treesit
executable ob-comint org-pcomplete pcomplete comint ansi-osc ansi-color
ring org-list org-footnote org-faces org-entities time-date noutline
outline ob-emacs-lisp ob-core ob-eval org-cycle org-table ol rx org-fold
org-fold-core org-keys oc org-loaddefs thingatpt find-func cal-menu
calendar cal-loaddefs org-version org-compat org-macs format-spec
annotate-autoloads company-posframe-autoloads company-autoloads
csv-mode-autoloads diff-hl-autoloads diminish-autoloads
dired-narrow-autoloads dired-hacks-utils-autoloads
expand-region-autoloads fic-mode-autoloads flycheck-autoloads
flylisp-autoloads flyspell-correct-autoloads geiser-guile-autoloads
geiser-autoloads greenbar-autoloads highlight-quoted-autoloads
markdown-mode-autoloads modus-themes-autoloads num3-mode-autoloads
orderless-autoloads org-auto-tangle-autoloads async-autoloads
org-bullets-autoloads org-modern-autoloads org-popup-posframe-autoloads
org-present-autoloads paredit-everywhere-autoloads paredit-autoloads
shelisp-autoloads svg-lib-autoloads transient-posframe-autoloads
treemacs-icons-dired-autoloads treemacs-magit-autoloads magit-autoloads
magit-section-autoloads git-commit-autoloads treemacs-tab-bar-autoloads
treemacs-autoloads cfrs-autoloads ht-autoloads hydra-autoloads
lv-autoloads pfuture-autoloads ace-window-autoloads avy-autoloads
s-autoloads dash-autoloads vertico-posframe-autoloads vertico-autoloads
which-key-posframe-autoloads posframe-autoloads info
with-editor-autoloads xkcd-autoloads package browse-url url url-proxy
url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs icons password-cache json
subr-x map byte-opt gv bytecomp byte-compile url-vars cl-loaddefs cl-lib
rmc iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win
term/common-win touch-screen pgtk-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo gtk pgtk
multi-tty move-toolbar make-network-process native-compile emacs)

Memory information:
((conses 16 1007060 160070) (symbols 48 68318 3)
 (strings 32 333909 18654) (string-bytes 1 9147096) (vectors 16 94245)
 (vector-slots 8 1975518 114459) (floats 8 836 202)
 (intervals 56 5983 1627) (buffers 992 19))

-- 
MICHAEL@MAUGER.COM // FSF and SFConservancy member // GNU Emacs sql.el maintainer

Re: 31.0.50; Org Persist file formatting

[Ihor Radchenko]

Michael Mauger <michael.mauger@protonmail.com> writes:

> This bug has been difficult to reproduce but several people have
> encountered it, or symptoms of it. Most recently, there has been this
> thread on Reddit:
> https://www.reddit.com/r/emacs/comments/1drxsz6/tangled_0_code_blocks_from_filename_problem/
>
> Unfortunately, replicating on emacs -Q has proven impossible since -Q
> disables some of the implicated features. My Emacs config is contained 
> in Org Babel files that are invoked with `org-babel-tangle-file' and have
> demonstrated the error multiple times.

Thanks for reporting!

> After some poking around I started getting failure when I tried to
> reopen an Org file or ran `normal-mode' on it. Shutting down emacs and
> restarting cleared the issue. When the issue arose when reopening the
> file within a session, the backtrace was pointing to `plistp' failing in
> `org-persist-read'. When I looked at the persist index there were
> entries like:
>
>      (:container
>       ((elisp org-element--headline-cache) (elisp org-element--cache)
>        (version "2.3"))
>       :persist-file "49/605653-361a-49a4-a000-47db8d522096" :associated
>       (:hash "aa1176747642fdac45aaf96095a88367" :file
>     	 "/home/michael/Projects/my-config/emacs/my-org.org" :inode
>     	 2554490)
>       :expiry 30 :last-access 1719791738.4428508 :last-access-hr
>       "2024-06-30T19:55:38-0400" ...)
>
> (Note: the "..." in the last entry; sometimes it showed up as "\...".)

This is curious. May it be that you have non-default value of
`pp-default-function'?

> The failure during tangling occurred with `org-get-heading' expecting
> `stringp' but getting `nil'. That seemed to be tracked down to
> `org-complex-heading-regexp' not being set yet. If an error had aborted
> out of the `org-persist-read' it appears we could have half of org-mode
> up but not everything properly initialized.

Sounds right.

> Looking at the code in `org-persist.el' in
> `org-persist--write-elisp-file (file data &optional no-circular pp)'
> around line 486 it handles calling `pp' or `prin1'. The code for `pp'
> uses `pp-use-max-width' (as part of a bug fix #58687) which has since
> been superseded in Emacs 30.

Yes, it should be fixed. I did not know that they quickly obsoleted that
variable.

> It looks like the `org-persist-write' code ought to be updated and the
> `org-persist-read' code made more robust so that it does not just shut
> down processing of Org files entirely. That is if the entry is not a
> plist, act as though no entry exists and proceed as if it were a new
> org file. 

Yes. The case when the contents of index file is a valid Elisp data, but
not the data we expect is currently not covered. It should be.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: 31.0.50; Org Persist file formatting

[Michael Mauger]

On Monday, July 8th, 2024 at 11:45 AM, Ihor Radchenko <yantar92@posteo.net> wrote:

> Michael Mauger michael.mauger@protonmail.com writes:
> 
> > This bug has been difficult to reproduce but several people have
> > encountered it, or symptoms of it. Most recently, there has been this
> > thread on Reddit:
> > https://www.reddit.com/r/emacs/comments/1drxsz6/tangled_0_code_blocks_from_filename_problem/
> > 

Just encountered it again. :( The recipe is still not obvious but I'm trying to find one... 
(This time got the "\..." version)

> > After some poking around I started getting failure when I tried to
> > reopen an Org file or ran `normal-mode' on it. Shutting down emacs 
> > and restarting cleared the issue. When the issue arose when reopening 
> > the file within a session, the backtrace was pointing to` plistp' failing in
> > `org-persist-read'. When I looked at the persist index there were
> > entries like:
> > 
> > (:container
> > ((elisp org-element--headline-cache) (elisp org-element--cache)
> > (version "2.3"))
> > :persist-file "49/605653-361a-49a4-a000-47db8d522096" :associated
> > (:hash "aa1176747642fdac45aaf96095a88367" :file
> > "/home/michael/Projects/my-config/emacs/my-org.org" :inode
> > 2554490)
> > :expiry 30 :last-access 1719791738.4428508 :last-access-hr
> > "2024-06-30T19:55:38-0400" ...)
> > 
> > (Note: the "..." in the last entry; sometimes it showed up as "\...".)
> 
> 
> This is curious. May it be that you have non-default value of
> `pp-default-function'?
> 

I don't touch `pp-default-function' and I've confirmed that it is 
set to `pp-fill' in my instance.

> --
> Ihor Radchenko // yantar92,
> Org mode contributor,

Thanks for your attention.

Re: 31.0.50; Org Persist file formatting

[Ihor Radchenko]

Michael Mauger <michael.mauger@protonmail.com> writes:

>> > (Note: the "..." in the last entry; sometimes it showed up as "\...".)
>> 
>> 
>> This is curious. May it be that you have non-default value of
>> `pp-default-function'?
>> 
>
> I don't touch `pp-default-function' and I've confirmed that it is 
> set to `pp-fill' in my instance.

Then, what if you re-define `org-persist-write:index' to

(defun org-persist-write:index (container _)
  "Write index CONTAINER."
  (org-persist--get-collection container)
  (unless (file-exists-p org-persist-directory)
    (condition-case nil
        (make-directory org-persist-directory 'parent)
      (t
       (warn "Failed to create org-persist storage in %s."
             org-persist-directory)
       (org-persist--check-write-access org-persist-directory))))
  (when (file-exists-p org-persist-directory)
    (let ((index-file
           (org-file-name-concat org-persist-directory org-persist-index-file)))
      (org-persist--merge-index-with-disk)
      ;; CHANGED: disable pretty-printing
      (org-persist--write-elisp-file index-file org-persist--index t nil)
      (setq org-persist--index-age
            (file-attribute-modification-time (file-attributes index-file)))
      index-file)))

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: 31.0.50; Org Persist file formatting

[Michael Mauger]

Sorry for not getting back to you sooner. This change has seemed to work, but I haven't been playing with my Org Babel config much lately. I'll let you know if I see it again.

Thanks for your help.

--
MICHAEL@MAUGER.COM // FSF and SFConservancy member // GNU Emacs sql.el maintainer


On Monday, July 8th, 2024 at 1:39 PM, Ihor Radchenko <yantar92@posteo.net> wrote:

> Michael Mauger michael.mauger@protonmail.com writes:
> 
> > > > (Note: the "..." in the last entry; sometimes it showed up as "\...".)
> > > 
> > > This is curious. May it be that you have non-default value of
> > > `pp-default-function'?
> > 
> > I don't touch `pp-default-function' and I've confirmed that it is set to` pp-fill' in my instance.
> 
> 
> Then, what if you re-define `org-persist-write:index' to
> 
> (defun org-persist-write:index (container _)
> "Write index CONTAINER."
> (org-persist--get-collection container)
> (unless (file-exists-p org-persist-directory)
> (condition-case nil
> (make-directory org-persist-directory 'parent)
> (t
> (warn "Failed to create org-persist storage in %s."
> org-persist-directory)
> (org-persist--check-write-access org-persist-directory))))
> (when (file-exists-p org-persist-directory)
> (let ((index-file
> (org-file-name-concat org-persist-directory org-persist-index-file)))
> (org-persist--merge-index-with-disk)
> ;; CHANGED: disable pretty-printing
> (org-persist--write-elisp-file index-file org-persist--index t nil)
> (setq org-persist--index-age
> (file-attribute-modification-time (file-attributes index-file)))
> index-file)))
> 
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at https://orgmode.org/.
> 
> Support Org development at https://liberapay.com/org-mode,
> 
> or support my work at https://liberapay.com/yantar92