Incorporating caching into defgroup/defcustom/defvar for Emacs 25

Incorporating caching into defgroup/defcustom/defvar for Emacs 25

[Sean Allred]

Hi all,

Since this is a long email (sorry!), an executive summary: many Emacs applications use data that is persistent from session-to-session. The mechanism for doing so varies from application-to-application, but there are a few packages on the ELPAs that promise this, notably cache, pcache, and the new stash.el (my own creation for sx.el, both on github). I’ve discovered a lot of overlap in the way I’ve designed stash.el and the way the defgroup/defcustom system already works, so I’m curious to see how open you all are about introducing caching functionality baked into Emacs. defgroup and defcustom aren’t so difficult, but defvar (being a C function) introduces concerns that I’ve yet to truly research / understand.

One of the reasons that makes Emacs so awesome (or at least interesting) is the variety and number of full-blown application written in/for Emacs. Gnus and Org are prime examples, and *nothing* would be the same (for me) without Ido. All of these applications use on-disk data to keep information from session to session. This need is hardly limited to Emacs internally, too; Artur and I have been at work on sx.el (on github), another moderately complex Emacs application that uses (and outright needs) on-disk data. Most veritable applications use on-disk data, and I can only guess that they all do so with an in-house mechanic. The use-case is there.

There are a few caching libraries on the ELPAs at this time: cache, pcache, and stash (my own creation for sx.el). In my honest opinion,

- `cache’ is too simple. For one, it doesn’t even save its data! It merely provides a means for cached data to expire - a necessary utility for full support.
- `pcache’ is too complex (for my taste). It’s not immediately apparent where/how the data is stored. (Disclaimer: I have absolutely zero experience with eieio, its sole dependency.) As an added :(, it’s not available on the GNU ELPA.
- `stash’ is, I think, a good compromise between the two, though some filling out has to be done on the featureset. For example, it currently does not support cache timeouts. SX hasn’t needed it yet, but I’m certain other applications would. I’m, uhhh, I’m working on it :)

Something I’ve noticed in writing stash.el (and discussing it with Artur a bit) is that I’m duplicating functionality. To see this, let me explain (briefly) how stash.el works:

- &optional Programmer defines an ‘application’. This application implies a subdirectory within the over-arching `stash-directory’ where data is saved at some programmer-defined interval.
- Programmer defines a `stash’, optionally belonging to an application, with an associated filename. This filename is where this `stash’ is saved to disk as interpreted by `prin1-to-string'. (A `stash’ is just a variable with a few symbol-properties as metadata.)
- ...
  From this point on, the programmer just sets the variable’s value normally (with set/setq/etc.). On an idle-timer (set up by defining the application - there is a ’null’ application set to a minute’s idle time), the entire application is written to disk.

There are a few flaws in this approach (for example, not every variable has changed from the time it was last saved), but it’s working out pretty well. The most glaring flaw in this design is the duplication of the group-variable association already available (or conceptually so) in the defgroup/defcustom/defvar system. Given its generally applicability, I think caching in this (or similar) fashion would be a good addition to the system via keyword arguments to those three macros:

    (defgroup sx nil
      "Customization group for the `sx' package."
      :prefix "sx-"
      :tag "SX"
      :group ‘applications)
     ; => Would prepare to create a folder called “sx” in `customize-cache-directory’
    
    (defcustom sx-init-hook nil
      "Hook run when SX initializes.
    Run after `sx-init--internal-hook'."
      :group 'sx
      :type ‘hook
      :cache “init-hook.el”)
     ; => Would prepare to create a file called “init-hook.el” in `customize-cache-directory’/sx
    
    (defvar sx-variable :group ’sx :cache t :expire 60)
     ; => same as above, but basename defaults to “sx-variable” set to expire after one minute of inactivity

It’s this last that I’m most concerned about: `defvar’ is defined in the C source code. I haven’t taken the slightest peek at Emacs’ C sources, but I’d reckon there wouldn’t be a simple analog from the approach I took in stash.el. I’m just not sure. Perhaps the functionality can be moved to a Lisp defvar*.

I see a conceptual overlap with Customize’s own managed `custom-set-variables’ form, but they aren’t quite the same thing. It’s worth pointing out the similarity, but I’m not sure what to make of it.

I know this is a long email, so many thanks for reading it through. I’d really like to know interested thoughts on the topic. I’m perfectly willing and able to implement the functionality (or at least do most of the legwork) pending the necessary FSF paperwork, but it would be a non-trivial undertaking in an already busy schedule. Moreover, I’m sure there are multiple considerations I’ve missed :)

Best,
Sean Allred

Re: Incorporating caching into defgroup/defcustom/defvar for Emacs 25

[Sean Allred]

Actually, I’ve just read a message from Artur on the matter:

> The way I see it, emacs has a great interface for persistent customizable options (defcustom) and it has defvar for internal non-persistent variables. But it doesn't have anything for internal persistent variables. You can call custom-save-variable on an internal variable, but then it's up to you to check the value on initialization, and it's also a little awkward to use this for very large variables (since they'll clog up the user's custom-file, which is probably also the init file). That's why several packages (ido, bookmark, etc) end up manually writing/reading variables to/from disk. 
> 
> Thus, there's clearly the need for stashes, persistent internal variables. These are not customizable variables, because we already have defcustom for that.
> However, similar to a defcustom, a defstash also belongs to a group. This grouping is not used for customization (stashes don't show up in the customize interface), but it is used to group stashes in the file system. Each group corresponds to a subdirectory, and all stashes from that group go in that directory. This would be specified by passing the :stash-directory keyword to defgroup. This keyword (and one or two others) would be the only changes necessary outside the stash.el file. 
> 
> Obviously, groups with no stashes will not have a directory. Also, groups with only one or two stashes may opt to not use a subdirectory. 


Specifically, his point about defcustom is a good one. Such variables shouldn’t be cached in the same sense. stash.el was and is designed as a patch for the lack of “internal persistent variables”. The additional keyword(s?) to defgroup and a defvar* macro would suffice.

Best,
Sean

Re: Incorporating caching into defgroup/defcustom/defvar for Emacs 25

[Stephen J. Turnbull]

Sean Allred writes:

 > Specifically, his point about defcustom is a good one. Such
 > variables shouldn’t be cached in the same sense.

Why not?  Custom provides a "reset" functionality if the user *wants*
to reset.  It's not clear to me that a user wouldn't want changes to
customizable variables to persist.

 > stash.el was and is designed as a patch for the lack of “internal
 > persistent variables”. The additional keyword(s?) to defgroup and a
 > defvar* macro would suffice.

What does stash provide that a session manager such as desktop.el
doesn't?  That wasn't clear to me from your long email.

Re: Incorporating caching into defgroup/defcustom/defvar for Emacs 25

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 2189 bytes --]

On 2 Feb 2015 01:07, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
>
> Sean Allred writes:
>
>  > Specifically, his point about defcustom is a good one. Such
>  > variables shouldn’t be cached in the same sense.
>
> Why not?  Custom provides a "reset" functionality if the user *wants*
> to reset.  It's not clear to me that a user wouldn't want changes to
> customizable variables to persist.

I wasn't saying defcustoms shouldn't be persistent, I was just saying they
already have a great mechanism for that so they don't need to use this
caching mechanism that is being suggested for internal variables.

Similarly, internal variables can't use the defcustom mechanism (at least,
not without modifications) because (1) they'll show up in the customize
interface and (2) they'll clutter up the user's init file.

>
>  > stash.el was and is designed as a patch for the lack of “internal
>  > persistent variables”. The additional keyword(s?) to defgroup and a
>  > defvar* macro would suffice.
>
> What does stash provide that a session manager such as desktop.el
> doesn't?  That wasn't clear to me from your long email.

With stash, a package can specify that a global variable is supposed to be
persistent, and this variable will automatically be saved/loaded from disk.

I understand desktop.el provides similar functionality, but, IIUC,
desktop.el is more of a user interface. By that I mean sessions are only
saved/loaded if desktop-mode is enabled, and desktop-mode might just be
disabled because the user doesn't want to restore open buffers/files. Some
packages need a way to store persistent data regardless of desktop-mode
(see abbrevs and bookmarks, for instance), and so far they've all been
doing that manually.

Stash provides a simple abstraction for a variable that will be persistent
accross sessions. You just declare a variable to be persistent (Sean
suggested a `defvar*' macro, but I prefer something more specific like
`defstash') and its value will be saved to disk during a session and loaded
from disk at the start of each session (very similar to what desktop.el
does, but without relying on desktop-mode).

[-- Attachment #2: Type: text/html, Size: 2554 bytes --]

Re: Incorporating caching into defgroup/defcustom/defvar for Emacs 25

[Stephen J. Turnbull]

Artur Malabarba writes:
 > On 2 Feb 2015 01:07, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:

 > > Why not?  Custom provides a "reset" functionality if the user *wants*
 > > to reset.  It's not clear to me that a user wouldn't want changes to
 > > customizable variables to persist.
 > 
 > I wasn't saying defcustoms shouldn't be persistent, I was just
 > saying they already have a great mechanism for that so they don't
 > need to use this caching mechanism that is being suggested for
 > internal variables.

Yes, they might.  A user who is testing out a new custom setting might
like the setting to persist without explicitly saving it if Emacs
crashes or he runs out of memory and needs to stop and restart Emacs,
or something like that.

 > > What does stash provide that a session manager such as desktop.el
 > > doesn't?  That wasn't clear to me from your long email.
 > 
 > With stash, a package can specify that a global variable is supposed to be
 > persistent, and this variable will automatically be saved/loaded from disk.
 > 
 > I understand desktop.el provides similar functionality, but, IIUC,
 > desktop.el is more of a user interface. By that I mean sessions are
 > only saved/loaded if desktop-mode is enabled,

The people I know who use desktop generally enable it in init.el.  The
only way to disable it AFAIK is -q.

 > Stash provides a simple abstraction for a variable that will be
 > persistent accross sessions. You just declare a variable to be
 > persistent (Sean suggested a `defvar*' macro,

Ah, OK, so the important thing is that the programmer controls it, not
the user.

Re: Incorporating caching into defgroup/defcustom/defvar for Emacs 25

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Artur Malabarba writes:
>  > On 2 Feb 2015 01:07, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
>
>  > > Why not?  Custom provides a "reset" functionality if the user *wants*
>  > > to reset.  It's not clear to me that a user wouldn't want changes to
>  > > customizable variables to persist.
>  > 
>  > I wasn't saying defcustoms shouldn't be persistent, I was just
>  > saying they already have a great mechanism for that so they don't
>  > need to use this caching mechanism that is being suggested for
>  > internal variables.
>
> Yes, they might.  A user who is testing out a new custom setting might
> like the setting to persist without explicitly saving it if Emacs
> crashes or he runs out of memory and needs to stop and restart Emacs,
> or something like that.

Personally, I very much prefer it if my experiments do not follow me
around for the rest of my life.  That is, I _like_ to be able to restart
Emacs and have it return to a well-known state.

-- 
David Kastrup

Re: Incorporating caching into defgroup/defcustom/defvar for Emacs 25

[Artur Malabarba]

2015-02-02 13:03 GMT+00:00 Stephen J. Turnbull <stephen@xemacs.org>:
> Artur Malabarba writes:
>  > On 2 Feb 2015 01:07, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
>
>  > > Why not?  Custom provides a "reset" functionality if the user *wants*
>  > > to reset.  It's not clear to me that a user wouldn't want changes to
>  > > customizable variables to persist.
>  >
>  > I wasn't saying defcustoms shouldn't be persistent, I was just
>  > saying they already have a great mechanism for that so they don't
>  > need to use this caching mechanism that is being suggested for
>  > internal variables.
>
> Yes, they might.  A user who is testing out a new custom setting might
> like the setting to persist without explicitly saving it if Emacs
> crashes or he runs out of memory and needs to stop and restart Emacs,
> or something like that.
>
>  > > What does stash provide that a session manager such as desktop.el
>  > > doesn't?  That wasn't clear to me from your long email.
>  >
>  > With stash, a package can specify that a global variable is supposed to be
>  > persistent, and this variable will automatically be saved/loaded from disk.
>  >
>  > I understand desktop.el provides similar functionality, but, IIUC,
>  > desktop.el is more of a user interface. By that I mean sessions are
>  > only saved/loaded if desktop-mode is enabled,
>
> The people I know who use desktop generally enable it in init.el.  The
> only way to disable it AFAIK is -q.
>
>  > Stash provides a simple abstraction for a variable that will be
>  > persistent accross sessions. You just declare a variable to be
>  > persistent (Sean suggested a `defvar*' macro,
>
> Ah, OK, so the important thing is that the programmer controls it, not
> the user.

Precisely.

RE: Incorporating caching into defgroup/defcustom/defvar for Emacs 25

[Drew Adams]

> sessions are only saved/loaded if desktop-mode is enabled

Only saved/loaded _automatically_ if `desktop-mode' is enabled.
Or if other code explicitly does the same thing automatically.

Desktop files can be saved/loaded at any time, by code, users,
bookmarks, etc.

Wrt automatically caching selected variables, another way to do
that is to create a variables-list bookmark and automatically
update it (e.g. on exit from Emacs or whatever).

(Var-list bookmarks are available with Bookmark+, and could be
trivially added to vanilla Emacs.  Likewise, desktop bookmarks.)

Caveat: I haven't been following this thread.  Just pointing
out that `desktop-mode' is not a requirement for saving/loading
desktop files.

Re: Incorporating caching into defgroup/defcustom/defvar for Emacs 25

[Stefan Monnier]

I'd welcome such a mechanism, and clearly various packages roll their
own, so it would be great to introduce something that can be used by
those packages (e.g. desktop.el and savehist.el could make use of such
a thing, obviously).

But I don't see much benefit from trying to link it to Custom (at most,
maybe Custom could be changed to make use of it internally).
E.g. I don't see the benefit of

     :group 'sx
over
     :dir "sx"

Also, based on the desktop.el and Gnus use cases, it seems important to
be able to save a state which is not in a variable (e.g. the frame
configuration), and it's also important that those persistent vars don't
be reloaded eagerly at init time, but later on when the corresponding
package is loaded/activated.


        Stefan

Re: Incorporating caching into defgroup/defcustom/defvar for Emacs 25

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

> Also, based on the desktop.el and Gnus use cases, it seems important to
> be able to save a state which is not in a variable (e.g. the frame
> configuration),

I'll look into these specific use cases, but ultimately everything can be
(and probably already is) stored in variables.

Also, I don't know if that's what you meant, but I see now it would be
useful to have an :init property on stash variables (similar to the one in
defcustoms), which would only be called if the stash was loaded from disk.
This could be used to restore frame configuration, for instance.

> and it's also important that those persistent vars don't
> be reloaded eagerly at init time, but later on when the corresponding
> package is loaded/activated.

Yes, that's how it works.

[-- Attachment #2: Type: text/html, Size: 911 bytes --]

Re: Incorporating caching into defgroup/defcustom/defvar for Emacs 25

[Stefan Monnier]

> I'll look into these specific use cases, but ultimately everything can be
> (and probably already is) stored in variables.

In the case of desktop.el the state is kept in a whole bunch of
different places.  Same for savehist.el where the state is spread over
several variables.

> This could be used to restore frame configuration, for instance.

That's right: for states that aren't stored in variables, we clearly
need both a getter and a setter.


        Stefan