bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[N. Jackson]

In the Emacs 30 pretest I have been getting the following warning
every few days:

  Warning (emacs): Emacs reader failed to read data in
  "/home/nlj/.cache/org-persist/gc-lock.eld". The error was: "End of
  file during parsing"

The `gc-lock' part suggests this might have something to do with
garbage collection, whereas `org-persist' suggests Org mode, but
I could find nothing in the Org manual about org-persist or about
gc-lock.

I don't know how to reproduce this.  The warning pops up seemingly
at random, often when the only visible window [before the *Warnings*
window appears] is NOT in Org mode.


In GNU Emacs 30.0.93 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.43, cairo version 1.18.0) of 2024-12-20 built on fedora
Windowing system distributor 'The X.Org Foundation', version 11.0.12014000
System Description: Fedora Linux 40 (Xfce)

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ
JPEG LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS WEBP X11 XDBE XIM XINPUT2 XPM GTK3
ZLIB

Important settings:
  value of $LANG: en_CA.utf8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Text

Minor modes in effect:
  TeX-PDF-mode: t
  flyspell-mode: t
  recentf-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  savehist-mode: t
  save-place-mode: t
  electric-pair-mode: t
  display-time-mode: t
  display-battery-mode: t
  desktop-save-mode: t
  delete-selection-mode: t
  cua-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  global-visual-line-mode: t
  visual-line-mode: t
  transient-mark-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  abbrev-mode: t

Load-path shadows:
None found.

Features:
(shadow sort bbdb-message mail-extr emacsbug message puny rfc822 mml
mml-sec epa epg rfc6068 epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils mule-util cdlatex
reftex reftex-loaddefs reftex-vars dired-aux dired dired-loaddefs
emacs-news-mode bug-reference display-fill-column-indicator
display-line-numbers tex-mode font-latex latexenc preview
latex-mode-expansions latex edmacro latex-flymake flymake warnings
tex-ispell tex-style tex texmathp auctex yank-media oc-basic bibtex
iso8601 org-habit vc-git diff-mode track-changes easy-mmode
vc-dispatcher flyspell ispell kmacro mines derived cookie1 gamegrid
transpar expand-region text-mode-expansions the-org-mode-expansions
python-el-fgallina-expansions er-basic-expansions expand-region-core
expand-region-custom hydra advice lv compile text-property-search
org-clock comp-run comp-common org-agenda org-element org-persist
xdg org-id org-element-ast inline avl-tree generator org-refile org
org-macro org-pcomplete org-list org-footnote org-faces org-entities
time-date noutline outline ob-shell shell pcomplete ob-R ob-python
python project compat ob-plantuml ob-org ob-gnuplot ob-ditaa ob-calc
calc-store calc-trail calc-ext calc calc-loaddefs rect calc-macs
ob-awk ob-dot ob-maxima ob ob-tangle org-src sh-script smie treesit
executable ob-ref ob-lob ob-table ob-exp ob-comint comint ansi-osc
ansi-color ring ob-emacs-lisp ob-core ob-eval org-cycle org-table
org-keys oc org-loaddefs thingatpt find-func ol org-fold
org-fold-core org-compat org-version org-macs bbdb-anniv diary-lib
diary-loaddefs cal-menu calendar cal-loaddefs bbdb-com crm
mailabbrev bbdb bbdb-site timezone recentf tree-widget cus-edit pp
wid-edit ido format-spec modus-vivendi-theme modus-themes
yasnippet-classic-snippets cl-extra yasnippet help-mode savehist
saveplace company pcase elec-pair time battery dbus xml desktop
frameset delsel cua-base cus-load ace-window-autoloads
auctex-autoloads tex-site avy-autoloads bbdb-autoloads
cdlatex-autoloads company-autoloads csv-mode-autoloads
debbugs-autoloads ess-autoloads expand-region-autoloads
geiser-autoloads info orderless-autoloads rx sql-indent-autoloads
yasnippet-autoloads package browse-url url url-proxy url-privacy
url-expand url-methods url-history url-cookie generate-lisp-file
url-domsuf url-util mailcap url-handlers url-parse auth-source
cl-seq eieio eieio-core cl-macs icons password-cache json subr-x map
byte-opt gv bytecomp byte-compile url-vars cl-loaddefs cl-lib rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd touch-screen tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock
syntax font-core term/tty-colors frame minibuffer nadvice seq simple
cl-generic indonesian philippine cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded
button loaddefs theme-loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads
dbusbind inotify dynamic-setting system-font-setting
font-render-setting cairo gtk x-toolkit xinput2 x multi-tty
move-toolbar make-network-process native-compile emacs)

Memory information:
((conses 16 874702 109492) (symbols 48 33757 0) (strings 32 141824 6883)
 (string-bytes 1 4270475) (vectors 16 88842)
 (vector-slots 8 1784278 36799) (floats 8 367 91)
 (intervals 56 24167 1981) (buffers 984 43))

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Eli Zaretskii]

> From: "N. Jackson" <njackson@posteo.net>
> Date: Mon, 30 Dec 2024 18:48:31 +0000
> 
> In the Emacs 30 pretest I have been getting the following warning
> every few days:
> 
>   Warning (emacs): Emacs reader failed to read data in
>   "/home/nlj/.cache/org-persist/gc-lock.eld". The error was: "End of
>   file during parsing"
> 
> The `gc-lock' part suggests this might have something to do with
> garbage collection, whereas `org-persist' suggests Org mode, but
> I could find nothing in the Org manual about org-persist or about
> gc-lock.

Does the file exist?  If so, what is its content (assuming you can
post it here)?

> I don't know how to reproduce this.  The warning pops up seemingly
> at random, often when the only visible window [before the *Warnings*
> window appears] is NOT in Org mode.

Ihor, any ideas or suggestions?  Should this be reported to the Org
list first?

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Eli Zaretskii]

> From: "N. Jackson" <njackson@posteo.net>
> Cc: Ihor Radchenko <yantar92@posteo.net>,  75209@debbugs.gnu.org
> Date: Mon, 30 Dec 2024 19:53:42 +0000
> 
> At 21:31 +0200 on Monday 2024-12-30, Eli Zaretskii wrote:
> 
> >> From: "N. Jackson" <njackson@posteo.net>
> >> Date: Mon, 30 Dec 2024 18:48:31 +0000
> >> 
> >>   Warning (emacs): Emacs reader failed to read data in
> >>   "/home/nlj/.cache/org-persist/gc-lock.eld". The error was: "End
> >>   of file during parsing"
> 
> > Does the file exist?  If so, what is its content (assuming you can
> > post it here)?
> 
> It exists and currently has the following contents:
> 
>   ;;   -*- mode: lisp-data; -*-
>   (((26482 57035 301257 992000) 26482 60639 74163 973000) ((26482 62694 821331 522000) 26482 62698 583212 450000))

This one seems okay.  I guess we need to wait for the warning and see
then?

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Pip Cet via Bug reports for GNU Emacs, the Swiss army knife of text editors]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> From: "N. Jackson" <njackson@posteo.net>
>> Cc: Ihor Radchenko <yantar92@posteo.net>,  75209@debbugs.gnu.org
>> Date: Mon, 30 Dec 2024 19:53:42 +0000
>>
>> At 21:31 +0200 on Monday 2024-12-30, Eli Zaretskii wrote:
>>
>> >> From: "N. Jackson" <njackson@posteo.net>
>> >> Date: Mon, 30 Dec 2024 18:48:31 +0000
>> >>
>> >>   Warning (emacs): Emacs reader failed to read data in
>> >>   "/home/nlj/.cache/org-persist/gc-lock.eld". The error was: "End
>> >>   of file during parsing"
>>
>> > Does the file exist?  If so, what is its content (assuming you can
>> > post it here)?
>>
>> It exists and currently has the following contents:
>>
>>   ;;   -*- mode: lisp-data; -*-
>>   (((26482 57035 301257 992000) 26482 60639 74163 973000) ((26482 62694 821331 522000) 26482 62698 583212 450000))
>
> This one seems okay.  I guess we need to wait for the warning and see
> then?

I'm assuming that the resolution was that the file was read before we
finished writing it.  I've run into the same issue a number of times
(interrupting and resuming Emacs builds leads to build failures, "make
bootstrap" makes them go away).

Can we consider modifying the .elc format to have a footer indicating
that the file is complete?  Ideally, it would also indicate the checksum
of the file as well as the fact that it is complete, but this would have
a performance impact which might be significant in some cases (very
large .elc files; of course, we could simply modify the footer to
indicate a "too large to checksum" condition has occurred, if the file
is large).

It's tempting to put this information in the header, the way we do for
pdumps (they are first written to start with "!UMPEDGNUEMACS", then the
last thing pdumper does is to rewrite the first character to be "D"),
but using a footer is more reliable: it detects truncation (or
modification) for whatever reason, and makes fewer assumptions about
data atomicity.

While we're in there, let's indicate in the ELC header whether the
special circumstances of native compilation applied to the compilation
process of this file.  This is particularly important if we use
benchmarks defined in .elc files: using the wrong compiled version would
lead to unreliable benchmark results, and be somewhat difficult to
detect otherwise.  (I'm assuming it is still the case that
native-compiling a Lisp file leaves behind user-visible .elc artifacts.
If that has been fixed, please ignore this paragraph).

But, please, no timestamp.  Let's keep things reproducible where we can,
and not leak sensitive information by accident.

It may be necessary to bump the produced ELC version code for this.

The equivalent issues are less urgent, but ultimately identical, for
pdumper files (apparently, we don't detect truncation or modification)
and object files produced during the build (it's the job of the make
implementation and the compiler to avoid truncated .o files, but if they
don't do that, we might want to write x.o.tmp first, then rename it, in
the usual fashion of Makefiles).

Note that it is, of course, possible to usefully modify .elc (and .pdmp)
files after creation, so we shouldn't make detected modifications an
unconditional error.

Pip

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Eli Zaretskii]

> Date: Wed, 01 Jan 2025 17:41:57 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: "N. Jackson" <njackson@posteo.net>, yantar92@posteo.net, 75209@debbugs.gnu.org
> 
> I'm assuming that the resolution was that the file was read before we
> finished writing it.  I've run into the same issue a number of times
> (interrupting and resuming Emacs builds leads to build failures, "make
> bootstrap" makes them go away).
> 
> Can we consider modifying the .elc format to have a footer indicating
> that the file is complete?

The .elc file is supposed to be created only when the compilation is
complete and successful.  If you look at byte-compile-file, you will
see that we first compile the Lisp code, then write the produced
bytecode to a temporary file, and only after that we rename the
temporary file into the target .elc file.  Renaming a file is an
atomic operation on Posix filesystems, so it either completely
succeeds or completely fails.

We only write directly to the target file if that file's directory is
unwritable.

So I don't understand why you see incomplete .elc files when you
interrupt the build.  What happens in my case is that I see those
temporary files left around, but I don't think I've ever saw an
incomplete .elc file after interrupting the build.

Is it likely that the directory where you build Emacs is not writable
by your user?  That's the only way I could explain what you see.  Or
maybe there's some other factor at work here, in which case we should
find out what that factor is, before we consider how to fix it.

In any case, I think this is a separate issue, so I'd prefer to have a
separate bug report for it.

Thanks.

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Pip Cet via Bug reports for GNU Emacs, the Swiss army knife of text editors]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> Date: Wed, 01 Jan 2025 17:41:57 +0000
>> From: Pip Cet <pipcet@protonmail.com>
>> Cc: "N. Jackson" <njackson@posteo.net>, yantar92@posteo.net, 75209@debbugs.gnu.org
>>
>> I'm assuming that the resolution was that the file was read before we
>> finished writing it.  I've run into the same issue a number of times
>> (interrupting and resuming Emacs builds leads to build failures, "make
>> bootstrap" makes them go away).
>>
>> Can we consider modifying the .elc format to have a footer indicating
>> that the file is complete?
>
> The .elc file is supposed to be created only when the compilation is
> complete and successful.  If you look at byte-compile-file, you will

You're right.  Sorry for the noise.  The most likely explanation is I
missed a "Pure Lisp storage overflowed" message which "explained" it,
because I just tried and that's what happened.

There's a bug in pin_string which assumes no purespace overflow, and
corrupts bytecode after one, so it's entirely possible that an .elc file
was truncated.  Probably not worth fixing at this point.

I'll try interrupting a few more builds when no-purespace is merged.

Pip

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I don't know how to reproduce this.  The warning pops up seemingly
>> at random, often when the only visible window [before the *Warnings*
>> window appears] is NOT in Org mode.
>
> Ihor, any ideas or suggestions?  Should this be reported to the Org
> list first?

Sounds like Emacs being killed by force in the middle of writing to that
file. Or, alternatively, C-g in the middle of writing.

If that's kill -9 or similar, I would not call this unexpected.

gc-lock.eld is a file used to flag that cache dir is being worked on by
multiple emacs instances. GC here refers to garbage-collecting cache data.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Ihor Radchenko]

"N. Jackson" <njackson@posteo.net> writes:

>> Or, alternatively, C-g in the middle of writing.
>
> I use C-g very frequently.  I type `M-x' and then realise I want to
> do `C-h f' first instead, so I do `C-g' to exit from the M-x prompt.
> Or I do an isearch and then change my mind (or find and read
> whatever I was looking for) then I do `C-g' (twice, I think) to exit
> the isearch and get back to where I started.  Usages like that.
> From my (probably naive) point of view, if that messes up Org Mode,
> then Org Mode is doing something wrong.

That should not be a problem then.
Reading/writing GC file is done using timer and, AFAIK, Emacs should not
run timers while you are running a command.

>> gc-lock.eld is a file used to flag that cache dir is being worked
>> on by multiple emacs instances. GC here refers to
>> garbage-collecting cache data.
>
> I do run multiple (two) instances of Emacs.  One is my normal
> session where I use Org quite heavily.  The other is my Gnus session
> in which I never open an Org file and never (as far as I know) use
> any Org features.

Gnus may load Org. (AFAIU, it does it when viewing gnus articles)

Another possible scenario is two Org instances writing to the same file
at the same time.
If it is what is happening in your case, your problem may be similar to 
https://list.orgmode.org/orgmode/CAMJKaZxA_VmLdFP_u1rNiF2s0X2kVivjT31jEM_r3BYCHri1PQ@mail.gmail.com/

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Eli Zaretskii <eliz@gnu.org>, 75209@debbugs.gnu.org
> Date: Tue, 31 Dec 2024 19:02:35 +0000
> 
> "N. Jackson" <njackson@posteo.net> writes:
> 
> >> Or, alternatively, C-g in the middle of writing.
> >
> > I use C-g very frequently.  I type `M-x' and then realise I want to
> > do `C-h f' first instead, so I do `C-g' to exit from the M-x prompt.
> > Or I do an isearch and then change my mind (or find and read
> > whatever I was looking for) then I do `C-g' (twice, I think) to exit
> > the isearch and get back to where I started.  Usages like that.
> > From my (probably naive) point of view, if that messes up Org Mode,
> > then Org Mode is doing something wrong.
> 
> That should not be a problem then.
> Reading/writing GC file is done using timer and, AFAIK, Emacs should not
> run timers while you are running a command.

If this happens while the user types some command, then timers could
fire during that typing, since people rarely type fast enough to not
let timers run.

But all this is not relevant, because Emacs binds inhibit-quit to a
non-nil value while it runs the timer function.  So, unless the timer
in question somehow forcibly resets inhibit-quit to nil, C-g should
not be able to interrupt a timer.

> >> gc-lock.eld is a file used to flag that cache dir is being worked
> >> on by multiple emacs instances. GC here refers to
> >> garbage-collecting cache data.
> >
> > I do run multiple (two) instances of Emacs.  One is my normal
> > session where I use Org quite heavily.  The other is my Gnus session
> > in which I never open an Org file and never (as far as I know) use
> > any Org features.
> 
> Gnus may load Org. (AFAIU, it does it when viewing gnus articles)
> 
> Another possible scenario is two Org instances writing to the same file
> at the same time.
> If it is what is happening in your case, your problem may be similar to 
> https://list.orgmode.org/orgmode/CAMJKaZxA_VmLdFP_u1rNiF2s0X2kVivjT31jEM_r3BYCHri1PQ@mail.gmail.com/

Can't Org prevent more than one session writing to this file?  We have
file locks which can be used here, I think.

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Another possible scenario is two Org instances writing to the same file
>> at the same time.
>> If it is what is happening in your case, your problem may be similar to 
>> https://list.orgmode.org/orgmode/CAMJKaZxA_VmLdFP_u1rNiF2s0X2kVivjT31jEM_r3BYCHri1PQ@mail.gmail.com/
>
> Can't Org prevent more than one session writing to this file?  We have
> file locks which can be used here, I think.

That's exactly the idea I am trying in the linked thread to address the
issue.

It is not the biggest problem there though. The problem is when there is
a race between Emacs processes writing to the same file one after
another (without any locking). Contents of the file may then become
unexpected compared to other Emacs session.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: njackson@posteo.net, 75209@debbugs.gnu.org
> Date: Wed, 01 Jan 2025 09:42:28 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Another possible scenario is two Org instances writing to the same file
> >> at the same time.
> >> If it is what is happening in your case, your problem may be similar to 
> >> https://list.orgmode.org/orgmode/CAMJKaZxA_VmLdFP_u1rNiF2s0X2kVivjT31jEM_r3BYCHri1PQ@mail.gmail.com/
> >
> > Can't Org prevent more than one session writing to this file?  We have
> > file locks which can be used here, I think.
> 
> That's exactly the idea I am trying in the linked thread to address the
> issue.
> 
> It is not the biggest problem there though. The problem is when there is
> a race between Emacs processes writing to the same file one after
> another (without any locking). Contents of the file may then become
> unexpected compared to other Emacs session.

Is the gc-lock.eld file supposed to be a singleton across all the
Emacs sessions?  Earlier you said:

> gc-lock.eld is a file used to flag that cache dir is being worked
> on by multiple emacs instances. GC here refers to
> garbage-collecting cache data.

Can you tell more about the purpose and use of this file?  What is
written to it, and how is it supposed to be used after being written?
And what bad things happen when the Lisp readers errors out because it
is unable to read the data for some reason?

I'm asking because I'd like to think about, and then suggest, some
suitable solutions, but I don't want to suggest nonsensical ones.

Thanks.

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> gc-lock.eld is a file used to flag that cache dir is being worked
>> on by multiple emacs instances. GC here refers to
>> garbage-collecting cache data.
>
> Can you tell more about the purpose and use of this file?  What is
> written to it, and how is it supposed to be used after being written?
> And what bad things happen when the Lisp readers errors out because it
> is unable to read the data for some reason?

Let me then describe briefly what org-persist does.

In the nutshell, it is cache manager.
The main cache data consists of:
1. index describing everything stored in the cache and its expiry
   settings
2. cache data stored in individual files. Each file in the cache is
   mentioned in the index file

From time to time (before quitting Emacs), org-persist needs to do some
"garbage collection" and remove cache files that are expired or
unreferenced from index to avoid cache growing infinitely.

The GC process works well, and helps keeping the cache directory
clean. However, there are problems when multiple Emacs processes are
running simultaneously.

Consider Emacs A loading cache index into memory and doing nothing.
Then, Emacs B also loads the cache index, but adds data to the cache.
If Emacs A is closed while Emacs B is running (and Emacs B not yet
updating cache index on disk), it also performs garbage
collection. However, Emacs A has no knowledge about cache data written
by Emacs B and may "garabge collect" this data. We do not want that.

"gc-lock.eld" keeps track of the running Emacs processes - every Emacs
process regularly write to "gc-lock.eld", putting a record in the form
of (before-init-time . <last known time that Emacs is running>). If
there are no known recently running Emacs processes (apart from
current), garbage collection process is suppressed to avoid removing
cache data from other Emacsen.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: njackson@posteo.net, 75209@debbugs.gnu.org
> Date: Thu, 02 Jan 2025 17:28:02 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Can you tell more about the purpose and use of this file?  What is
> > written to it, and how is it supposed to be used after being written?
> > And what bad things happen when the Lisp readers errors out because it
> > is unable to read the data for some reason?
> 
> Let me then describe briefly what org-persist does.
> 
> In the nutshell, it is cache manager.
> The main cache data consists of:
> 1. index describing everything stored in the cache and its expiry
>    settings
> 2. cache data stored in individual files. Each file in the cache is
>    mentioned in the index file
> 
> >From time to time (before quitting Emacs), org-persist needs to do some
> "garbage collection" and remove cache files that are expired or
> unreferenced from index to avoid cache growing infinitely.
> 
> The GC process works well, and helps keeping the cache directory
> clean. However, there are problems when multiple Emacs processes are
> running simultaneously.
> 
> Consider Emacs A loading cache index into memory and doing nothing.
> Then, Emacs B also loads the cache index, but adds data to the cache.
> If Emacs A is closed while Emacs B is running (and Emacs B not yet
> updating cache index on disk), it also performs garbage
> collection. However, Emacs A has no knowledge about cache data written
> by Emacs B and may "garabge collect" this data. We do not want that.

Thanks.  I think I still don't have a clear idea of the usage of these
caches.  Are the caches supposed to be common to all Emacs sessions?
E.g., when a cache changes by one session, are other sessions supposed
to know about the change?  If the cache is for a single session, then
why are several session allowed to write to the cache simultaneously?
And if the cache is common to all sessions, then perhaps reading the
index before writing it should avoid several sessions step on each
other's toes?

> "gc-lock.eld" keeps track of the running Emacs processes - every Emacs
> process regularly write to "gc-lock.eld", putting a record in the form
> of (before-init-time . <last known time that Emacs is running>). If
> there are no known recently running Emacs processes (apart from
> current), garbage collection process is suppressed to avoid removing
> cache data from other Emacsen.

One way of rewriting a file atomically is to write the stuff to a
temporary file, then rename it to the target name.  If Org doesn't
already do that, maybe you should try doing that (together with
reading the file before updating it)?

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks.  I think I still don't have a clear idea of the usage of these
> caches.  Are the caches supposed to be common to all Emacs sessions?

Yes.

> E.g., when a cache changes by one session, are other sessions supposed
> to know about the change?

Usually yes, except for cache entries that are supposed to live until
the end of Emacs session.

> And if the cache is common to all sessions, then perhaps reading the
> index before writing it should avoid several sessions step on each
> other's toes?

You are right. The only problem is short-living caches that should be
cleared at the end of Emacs session that created it.

> One way of rewriting a file atomically is to write the stuff to a
> temporary file, then rename it to the target name.  If Org doesn't
> already do that, maybe you should try doing that (together with
> reading the file before updating it)?

Org uses `with-temp-file'. Is there an alternative built-in and more
robust way to write string to file?

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: njackson@posteo.net, 75209@debbugs.gnu.org
> Date: Sun, 05 Jan 2025 10:03:49 +0000
> 
> > And if the cache is common to all sessions, then perhaps reading the
> > index before writing it should avoid several sessions step on each
> > other's toes?
> 
> You are right. The only problem is short-living caches that should be
> cleared at the end of Emacs session that created it.

Does this mean you have ideas for solving this problem by reading the
file before it is written?  Or does this mean you already read the
file before writing to it?

> > One way of rewriting a file atomically is to write the stuff to a
> > temporary file, then rename it to the target name.  If Org doesn't
> > already do that, maybe you should try doing that (together with
> > reading the file before updating it)?
> 
> Org uses `with-temp-file'. Is there an alternative built-in and more
> robust way to write string to file?

Writing to a file is not atomic.  If you instead write to a temporary
file, then rename it to the final file name, the renaming is atomic on
Posix filesystems.

This would mean you can still use with-temp-file, but with a temporary
file name as its argument, and you need to add a single rename-file
call afterwards.

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> You are right. The only problem is short-living caches that should be
>> cleared at the end of Emacs session that created it.
>
> Does this mean you have ideas for solving this problem by reading the
> file before it is written?  Or does this mean you already read the
> file before writing to it?

The problem is that not every piece of cached data is stored in the
index on disk. Some caches should just live for the duration of Emacs
session that creates them.

Yet, other parallel Emacs sessions should not "GC" those transient
caches.

>> Org uses `with-temp-file'. Is there an alternative built-in and more
>> robust way to write string to file?
>
> Writing to a file is not atomic.  If you instead write to a temporary
> file, then rename it to the final file name, the renaming is atomic on
> Posix filesystems.
>
> This would mean you can still use with-temp-file, but with a temporary
> file name as its argument, and you need to add a single rename-file
> call afterwards.

That's fine. I just thought that there is some existing function doing
exactly this. If not, I can do it manually, of course.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[N. Jackson]

At 19:02 +0000 on Tuesday 2024-12-31, Ihor Radchenko wrote:
>
> Gnus may load Org. (AFAIU, it does it when viewing gnus articles)

Yes, I suppose it might.  [When I read an email with Org markup in
Gnus (source blocks, for example), the email gets nicely fontified.
Gnus might be using Org to do that rather than rolling its own
parser.]

> Another possible scenario is two Org instances writing to the same
> file at the same time.

I don't think that's the case here.

Certainly not the same user-owned Org Mode data file.  (I'm careful
not to open my Org mode files in my Gnus instance of Emacs [and
consequently I have to live with the inconvenience of not being able
to Capture directly from email or News].)

However now that I've learned that Org has internal files that it
writes to, it seems quite possible that two instances of Org might
write to one of those files at the same time.

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[N. Jackson]

At 18:48 +0000 on Monday 2024-12-30, N. Jackson wrote:
>
> In the Emacs 30 pretest I have been getting the following warning
> every few days:
>
>   Warning (emacs): Emacs reader failed to read data in
>   "/home/nlj/.cache/org-persist/gc-lock.eld". The error was: "End of
>   file during parsing"

When I woke my system from suspend this morning and switched to the
Emacs session running Gnus, the warning had popped up in that
session -- possibly during suspend or resume.

I immediately looked at gc-lock.eld which contained this:

  ;;   -*- mode: lisp-data; -*-
  (((26485 51608 866710 15000) 26486 34929 321731 426000))

(which I guess is again unremarkable).

I checked my buffer list to be absolutely certain that I hadn't
opened any Org Mode files and the only buffers were *Group*,
*scratch*, *Messages*, .newsrc-dribble, and *Warnings*.

The session has one frame and in that frame there was just the Gnus
Group buffer open (until the *Warnings* buffer popped up below it).

With Gnus sitting idle with just it's Group buffer open, I can't see
how it would use any Org Mode features -- or even _do_ anything at
all -- and as I said, no Org Mode (user) files were open in the
session.

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[N. Jackson]

The following might be "normal", in which case I apologise for the
noise, but it seems odd to me and it might have some bearing on the
bug.

Running list-timers shows:

Idle                Next  Repeat Function
       -1d 15h 43m 30.2s      1h org-persist--refresh-gc-lock
               2.4s           1m battery-update-handler
               2.4s           5m savehist-autosave
               4.2s            - undo-auto--boundary-timer
              50.1s           1m display-time-event-handler
   *           0.1s            t show-paren-function
   *           0.5s            t #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_9>
   *           0.5s      :repeat blink-cursor-start
   *          30.0s            - desktop-auto-save

I see nothing in the manuals about what it means for a
relative timer to be negative.  (Or is org-persist--refresh-gc-lock
running on a timer set with an absolute time that list-timers is
merely displaying as a relative time?)  And it seems odd that this
time is before this Emacs session started (emacs-uptime shows 1 day,
1 hour, 16 minutes, 40 seconds).

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[Eli Zaretskii]

> From: "N. Jackson" <njackson@posteo.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Ihor Radchenko <yantar92@posteo.net>
> Date: Sun, 05 Jan 2025 14:18:14 +0000
> 
> The following might be "normal", in which case I apologise for the
> noise, but it seems odd to me and it might have some bearing on the
> bug.
> 
> Running list-timers shows:
> 
> Idle                Next  Repeat Function
>        -1d 15h 43m 30.2s      1h org-persist--refresh-gc-lock
>                2.4s           1m battery-update-handler
>                2.4s           5m savehist-autosave
>                4.2s            - undo-auto--boundary-timer
>               50.1s           1m display-time-event-handler
>    *           0.1s            t show-paren-function
>    *           0.5s            t #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_9>
>    *           0.5s      :repeat blink-cursor-start
>    *          30.0s            - desktop-auto-save
> 
> I see nothing in the manuals about what it means for a
> relative timer to be negative.  (Or is org-persist--refresh-gc-lock
> running on a timer set with an absolute time that list-timers is
> merely displaying as a relative time?)  And it seems odd that this
> time is before this Emacs session started (emacs-uptime shows 1 day,
> 1 hour, 16 minutes, 40 seconds).

This timer is disabled.  See bug#39824 for some related discussions,
in particular

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39824#53

bug#75209: 30.0.93; Emacs reader failed to read data in "/home/nlj/.cache/org-persist/gc-lock.eld"

[N. Jackson]

At 19:21 +0200 on Sunday 2025-01-05, Eli Zaretskii wrote:

>> From: "N. Jackson" <njackson@posteo.net>
>> 
>> Running list-timers shows:
>> 
>> Idle                Next  Repeat Function
>>        -1d 15h 43m 30.2s      1h org-persist--refresh-gc-lock
>
> This timer is disabled.  See bug#39824 for some related discussions,
> in particular
>
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39824#53

I have now read that bug report and I admit I don't fully understand
it[1].

IIUC, the user's timer had encountered an error, a backtrace had
been produced, and the user had said `q' to the debugger, leaving
the timer in a broken state.

Here, like in that bug report, the broken timer has `t' in the first
element:

  [t 26490 7240 117604 3600 org-persist--refresh-gc-lock nil nil 17000 nil]

However, I have seen no error.  (If I were presented with a
backtrace, I would almost certainly make a copy of the buffer and
then hit `c' rather than `q', but in fact I haven't seen a backtrace
in a long time.  Indeed, debug-on-error is nil.  I have seen no
error messages in this run of Emacs and there are no errors (or
anything else unexpected in Messages).)

I'm guessing (wildly) that what happened is this:

1.  I woke my system from suspend.

2.  All timers in both my instances of Emacs ran roughly
    simultaneously.

3.  Org Mode's locking mechanisms are not working properly when two
    copies of org-persist--refresh-gc-lock run at essentially the 
    same time, and it failed in one instance of Emacs.

4.  Org Mode (or something else) caught the failure and reported

      Warning (emacs): Emacs reader failed to read data in
      "/home/nlj/.cache/org-persist/gc-lock.eld". The error was:
      "End of file during parsing"

    and the running of the timer was aborted, leaving it in a broken
    state.

I think this wild conjecture would explain why sometimes (but by no
means always) I see this warning when I resume from suspend; why I
rarely see the warning at other times; and why sometimes I see the
warning in my regular Emacs session and sometimes in the instance in
which I'm running Gnus.


(One other observation: IIUC, it says in bug#39824 that the broken
timer moves farther into the past, but here my broken timer is
counting forward (it is now due in negative 1 day and 5 hours) so
presumably in a day or so it will no longer be negative.  Will it
then start running again I wonder?  If this behaviour is expected,
perhaps it should be mentioned in the documentation.  It seems a bit
peculiar to me.)

[1] I don't understand why bug#39824 was closed as Not A Bug when
the mystery of how the timers got in an incoherent state wasn't
fully clarified.  (But maybe it was well understood and the
mechanism was too trivial to record.) [And I don't think, just
because a timer fails once, that one necessarily wants that timer
disabled (because the problem might be transient).  Also it seems to
me that if a timer is going to be disabled then that should be done
explicitly rather than as a side-effect of an abort.]  But that is
all irrelevant here.