Re: Please document the caching and its user options

Re: Please document the caching and its user options

[Ihor Radchenko]

CCing emacs-devel as I'd like to upgrade this discussion to Emacs-wide
context.

Eli Zaretskii <eliz@gnu.org> writes:

>> ... I wanted to know what is being cached, why, and in what file/directory.
>> 
> >  ...
>> Would it be possible for Emacs to define a framework for cache/var/data
>> locations? Such framework would not only be useful in the context of
>> this discussion, but also to tackle the issue with packages sprinkling
>> things randomly into .emacs.d or ~/ (see
>> https://github.com/emacscollective/no-littering/)
>
> I think Emacs already provides all the framework for caching that is
> needed.  Caching simply means you write some data to file, and all the
> building blocks of that already exist, for quite some time, actually.
> The only thing that is application dependent is the data to be cached
> and how to serialize that, but that cannot be usefully generalized.

I was referring to some kind of global option that defines cache
directory, data directory, etc. Something akin XDG.

Then, Org can place cache inside that directory rather than trying to
cook up something independently.

Also, caching is not as simple, because caches may contain sensitive
data. (see
https://list.orgmode.org/orgmode/CAM9ALR8fuSu0YWS1SehRw7sYxprJFX-r2juXd_DgvCYVKQc95Q@mail.gmail.com/)
Some users may want to move caches to read-restricted location
or even to location dependent on where the cache is originating from
(separate caches depending on whether default-directory is from
encrypted volume, remote mount, etc)

Finally, we got several requests to have caches cleared up upon exiting
Emacs, which is also something that should be better managed centrally,
by Emacs, for all possible kinds of cache/history data.

>> > multisession is an optional package, it is neither preloaded nor
>> > turned on by default in Emacs.
>> 
>> It is used by default in emoji.el (C-x 8 e r)
>
> Which is also optional.  And a minor feature at that.

It is just for now.
TRAMP (by no means a minor feature), has the following TODO item in
tramp-cache.el:

;;; TODO:
;;
;; * Use multisession.el, starting with Emacs 29.1.

>> (Also, should we open some kind of bug report to track documenting
>> multisession in the manual?)
>
> I don't mind, but it sounds like an exaggeration to me.

I kind of agree, if we talk about the current state of affairs. But, I'd
like to discuss this in the context I elaborated on above - more
centralized cache management.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Please document the caching and its user options

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: emacs-orgmode@gnu.org, emacs-devel@gnu.org, Michael Albinus
>  <michael.albinus@gmx.de>
> Date: Sat, 15 Jun 2024 14:13:03 +0000
> 
> CCing emacs-devel as I'd like to upgrade this discussion to Emacs-wide
> context.
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> ... I wanted to know what is being cached, why, and in what file/directory.
> >> 
> > >  ...
> >> Would it be possible for Emacs to define a framework for cache/var/data
> >> locations? Such framework would not only be useful in the context of
> >> this discussion, but also to tackle the issue with packages sprinkling
> >> things randomly into .emacs.d or ~/ (see
> >> https://github.com/emacscollective/no-littering/)
> >
> > I think Emacs already provides all the framework for caching that is
> > needed.  Caching simply means you write some data to file, and all the
> > building blocks of that already exist, for quite some time, actually.
> > The only thing that is application dependent is the data to be cached
> > and how to serialize that, but that cannot be usefully generalized.
> 
> I was referring to some kind of global option that defines cache
> directory, data directory, etc. Something akin XDG.

We already have xdg-cache-home (and a few others in xdg.el).  Is that
what you meant?

> Also, caching is not as simple, because caches may contain sensitive
> data. (see
> https://list.orgmode.org/orgmode/CAM9ALR8fuSu0YWS1SehRw7sYxprJFX-r2juXd_DgvCYVKQc95Q@mail.gmail.com/)
> Some users may want to move caches to read-restricted location
> or even to location dependent on where the cache is originating from
> (separate caches depending on whether default-directory is from
> encrypted volume, remote mount, etc)

AFAIK, Emacs has APIs for at least some of that, but whether to use
them is up to the application, I think.

> Finally, we got several requests to have caches cleared up upon exiting
> Emacs, which is also something that should be better managed centrally,
> by Emacs, for all possible kinds of cache/history data.

Deleting files in a directory, recursively if needed, is already
available.  is that what you meant?

> >> > multisession is an optional package, it is neither preloaded nor
> >> > turned on by default in Emacs.
> >> 
> >> It is used by default in emoji.el (C-x 8 e r)
> >
> > Which is also optional.  And a minor feature at that.
> 
> It is just for now.
> TRAMP (by no means a minor feature), has the following TODO item in
> tramp-cache.el:
> 
> ;;; TODO:
> ;;
> ;; * Use multisession.el, starting with Emacs 29.1.

How far are you prepared to go just to make a point?

> >> (Also, should we open some kind of bug report to track documenting
> >> multisession in the manual?)
> >
> > I don't mind, but it sounds like an exaggeration to me.
> 
> I kind of agree, if we talk about the current state of affairs. But, I'd
> like to discuss this in the context I elaborated on above - more
> centralized cache management.

Can we first fix the problems for which I started this thread?  The
more general issues should be subjects of separate discussions, IMO.

Re: Please document the caching and its user options

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I was referring to some kind of global option that defines cache
>> directory, data directory, etc. Something akin XDG.
>
> We already have xdg-cache-home (and a few others in xdg.el).  Is that
> what you meant?

Yes, except that `xdg-cache-home' is limited:

1. It cannot be customized by users
2. It may sometimes return nil
3. It is limited to XDG - not all the Emacs platforms

What I had in mind is a new custom option for cache dir (defaulting to
OS-specific cache like XDG on Linux or something equivalent on Windows)
+ a new API function like `system-cache-home' that will be guaranteed to
return some kind of meaningful dir.

>> Also, caching is not as simple, because caches may contain sensitive
>> data. (see
>> https://list.orgmode.org/orgmode/CAM9ALR8fuSu0YWS1SehRw7sYxprJFX-r2juXd_DgvCYVKQc95Q@mail.gmail.com/)
>> Some users may want to move caches to read-restricted location
>> or even to location dependent on where the cache is originating from
>> (separate caches depending on whether default-directory is from
>> encrypted volume, remote mount, etc)
>
> AFAIK, Emacs has APIs for at least some of that, but whether to use
> them is up to the application, I think.

What are those APIs?

>> Finally, we got several requests to have caches cleared up upon exiting
>> Emacs, which is also something that should be better managed centrally,
>> by Emacs, for all possible kinds of cache/history data.
>
> Deleting files in a directory, recursively if needed, is already
> available.  is that what you meant?

No. I mean a new user option like `clear-caches-on-exit' that will work
across all the packages. Then, concerned users may set it to non-nil to
delete *all* the caches upon exiting Emacs.

Having to set this for each specific package (with some packages not
documenting that they use cache, or users not expecting that cache may
be used and not reading _all_ the docs carefully enough) is not ideal,
IMHO.

> Can we first fix the problems for which I started this thread?  The
> more general issues should be subjects of separate discussions, IMO.

If there is a global Emacs-wide customization how to handle caches,
there will be no need to document it in Org mode manual. So, I would
like to see if introducing such global customization is feasible before
making non-trivial changes to Org manual. (I am not even sure where to
document these things in the manual yet; they seem way too generic wrt
Org mode's scope)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Please document the caching and its user options

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: emacs-orgmode@gnu.org, emacs-devel@gnu.org, michael.albinus@gmx.de
> Date: Sun, 16 Jun 2024 09:05:02 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I was referring to some kind of global option that defines cache
> >> directory, data directory, etc. Something akin XDG.
> >
> > We already have xdg-cache-home (and a few others in xdg.el).  Is that
> > what you meant?
> 
> Yes, except that `xdg-cache-home' is limited:
> 
> 1. It cannot be customized by users

Of course it can: just make the default value of a defcustom be
derived by xdg-cache-home, and users can then customize the option to
a different value if they want.

> 2. It may sometimes return nil

The fallback is well-known.

> 3. It is limited to XDG - not all the Emacs platforms

No, it's supported on all platforms, even if XDG isn't.

> What I had in mind is a new custom option for cache dir (defaulting to
> OS-specific cache like XDG on Linux or something equivalent on Windows)
> + a new API function like `system-cache-home' that will be guaranteed to
> return some kind of meaningful dir.

Using xdg-cache-home and its fallbacks is a de-facto standard of
solving this in Emacs, and it supports all the platforms.  Even
startup.el uses it (albeit by customized code, to avoid interfering
with user customizations) when looking for init files and suchlikes.

So I think you raise a problem that is already solved in Emacs.

> >> Also, caching is not as simple, because caches may contain sensitive
> >> data. (see
> >> https://list.orgmode.org/orgmode/CAM9ALR8fuSu0YWS1SehRw7sYxprJFX-r2juXd_DgvCYVKQc95Q@mail.gmail.com/)
> >> Some users may want to move caches to read-restricted location
> >> or even to location dependent on where the cache is originating from
> >> (separate caches depending on whether default-directory is from
> >> encrypted volume, remote mount, etc)
> >
> > AFAIK, Emacs has APIs for at least some of that, but whether to use
> > them is up to the application, I think.
> 
> What are those APIs?

Making files and directories readable only by the owner, for example:
set-file-modes and with-file-modes.  All the other Lisp programs in
Emacs use that, so why would Org need something special?

> >> Finally, we got several requests to have caches cleared up upon exiting
> >> Emacs, which is also something that should be better managed centrally,
> >> by Emacs, for all possible kinds of cache/history data.
> >
> > Deleting files in a directory, recursively if needed, is already
> > available.  is that what you meant?
> 
> No. I mean a new user option like `clear-caches-on-exit' that will work
> across all the packages.

Having a single option for all the caches makes little sense to me.
This must be a per-cache setting.

However, users on XDG platforms can have that via XDG system-wide
settings.

> Having to set this for each specific package (with some packages not
> documenting that they use cache, or users not expecting that cache may
> be used and not reading _all_ the docs carefully enough) is not ideal,
> IMHO.

I cannot disagree more.  Each cache has its own logic for when it is a
good time to empty the cache.

> > Can we first fix the problems for which I started this thread?  The
> > more general issues should be subjects of separate discussions, IMO.
> 
> If there is a global Emacs-wide customization how to handle caches,
> there will be no need to document it in Org mode manual.

I respectfully ask the Org developers to solve this particular issue
first, without waiting for some hypothetical general Emacs feature,
which may or may not materialize.

> like to see if introducing such global customization is feasible before
> making non-trivial changes to Org manual. (I am not even sure where to
> document these things in the manual yet; they seem way too generic wrt
> Org mode's scope)

A new chapter should be fine, if no existing chapter is relevant.

TIA

Re: Please document the caching and its user options

[Björn Bidar]

Eli Zaretskii <eliz@gnu.org> writes:

>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> I was referring to some kind of global option that defines cache
>> >> directory, data directory, etc. Something akin XDG.
>> >
>> > We already have xdg-cache-home (and a few others in xdg.el).  Is that
>> > what you meant?
>> 
>> Yes, except that `xdg-cache-home' is limited:
>> 
>> 1. It cannot be customized by users
>
> Of course it can: just make the default value of a defcustom be
> derived by xdg-cache-home, and users can then customize the option to
> a different value if they want.

That and it can be overridden just like any XDG Directory variable using
environment variables.