Gnus: caching message headers?

Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

Hi list,

I'm trying to configure Gnus to access my email through IMAP.

I'm too accustomed to traditional MUAs to switch to the "Gnus way" in one 
step, so I have:

(setq gnus-permanently-visible-groups ".*")
(setq gnus-parameters
    '((".*"
       (gnus-show-threads nil)
       (gnus-use-scoring nil)
       (gnus-article-sort-functions '(gnus-article-sort-by-number))
       (display . all))))

I would like to have, in the *Summary*, a complete list of the emails 
contained in a folder when I hit RET on its label.  Each time I do this 
however, Gnus asks me how many articles I want to retrieve, and issues a 
"UID FETCH 1:N (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS 
(...)])" IMAP command, which can take quite some time to complete when N 
is large.

Is there a way to convince Gnus to cache the result of that command 
(without caching all emails), and to issue a command only for the new 
UIDs?  Caching the result of that command should not eat too much disk 
space.

I apologize in advance if that question is naïve, or if I missed some part 
of the manual.

Gregory

Re: Gnus: caching message headers?

[Eric Abrahamsen]

Gregory Heytings via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> Hi list,
>
> I'm trying to configure Gnus to access my email through IMAP.
>
> I'm too accustomed to traditional MUAs to switch to the "Gnus way" in
> one step, so I have:
>
> (setq gnus-permanently-visible-groups ".*")
> (setq gnus-parameters
>    '((".*"
>       (gnus-show-threads nil)
>       (gnus-use-scoring nil)
>       (gnus-article-sort-functions '(gnus-article-sort-by-number))
>       (display . all))))
>
> I would like to have, in the *Summary*, a complete list of the emails
> contained in a folder when I hit RET on its label.  Each time I do
> this however, Gnus asks me how many articles I want to retrieve, and
> issues a "UID FETCH 1:N (UID RFC822.SIZE BODYSTRUCTURE
> BODY.PEEK[HEADER.FIELDS (...)])" IMAP command, which can take quite
> some time to complete when N is large.
>
> Is there a way to convince Gnus to cache the result of that command
> (without caching all emails), and to issue a command only for the new
> UIDs?  Caching the result of that command should not eat too much disk
> space.

No, not at the moment. You can avoid the prompt by customizing the value
of `gnus-large-newsgroup', but it's still going to retrieve all the
headers for the group.

There's currently no way around this, as Gnus only lets you have one
active Summary buffer at a time, which means all the old data is dumped
every time you switch groups. In the back of my head I have some ideas
for removing this restriction, but it will take a long time to get
there.

Eric

Re: Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

Hi Eric,

Many thanks for your answer.

>> I would like to have, in the *Summary*, a complete list of the emails 
>> contained in a folder when I hit RET on its label.  Each time I do this 
>> however, Gnus asks me how many articles I want to retrieve, and issues 
>> a "UID FETCH 1:N (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS 
>> (...)])" IMAP command, which can take quite some time to complete when 
>> N is large.
>>
>> Is there a way to convince Gnus to cache the result of that command 
>> (without caching all emails), and to issue a command only for the new 
>> UIDs?  Caching the result of that command should not eat too much disk 
>> space.
>
> No, not at the moment. You can avoid the prompt by customizing the value 
> of `gnus-large-newsgroup', but it's still going to retrieve all the 
> headers for the group.
>
> There's currently no way around this, as Gnus only lets you have one 
> active Summary buffer at a time, which means all the old data is dumped 
> every time you switch groups. In the back of my head I have some ideas 
> for removing this restriction, but it will take a long time to get 
> there.
>

I'm clearly not an expert, but would it not be enough to save and retrieve 
the contents of nntp-server-buffer in nnimap-retrieve-headers?  Or are you 
thinking about a more generic solution that would work with all backends?

Re: Gnus: caching message headers?

[Eric Abrahamsen]

Gregory Heytings via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> Hi Eric,
>
> Many thanks for your answer.
>
>>> I would like to have, in the *Summary*, a complete list of the
>>> emails contained in a folder when I hit RET on its label.  Each
>>> time I do this however, Gnus asks me how many articles I want to
>>> retrieve, and issues a "UID FETCH 1:N (UID RFC822.SIZE
>>> BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (...)])" IMAP command, which
>>> can take quite some time to complete when N is large.
>>>
>>> Is there a way to convince Gnus to cache the result of that command
>>> (without caching all emails), and to issue a command only for the
>>> new UIDs?  Caching the result of that command should not eat too
>>> much disk space.
>>
>> No, not at the moment. You can avoid the prompt by customizing the
>> value of `gnus-large-newsgroup', but it's still going to retrieve
>> all the headers for the group.
>>
>> There's currently no way around this, as Gnus only lets you have one
>> active Summary buffer at a time, which means all the old data is
>> dumped every time you switch groups. In the back of my head I have
>> some ideas for removing this restriction, but it will take a long
>> time to get there.
>>
>
> I'm clearly not an expert, but would it not be enough to save and
> retrieve the contents of nntp-server-buffer in
> nnimap-retrieve-headers?  Or are you thinking about a more generic
> solution that would work with all backends?

We could do something ad-hoc for nnimap, but yes I'm thinking of
something more generic. All the header data (what's used to create the
Summary display) is held in variables that are local to the Summary
buffer, so in principle there's no reason we couldn't just leave the
local data in place when we leave the buffer. There are plenty of
obstacles to making it work correctly, but in principle I don't see why not.

Re: Gnus: caching message headers?

[wgreenhouse]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Gregory Heytings via Users list for the GNU Emacs text editor
> <help-gnu-emacs@gnu.org> writes:
>
>> Hi Eric,
>>
>> Many thanks for your answer.
>>
>>>> I would like to have, in the *Summary*, a complete list of the
>>>> emails contained in a folder when I hit RET on its label.  Each
>>>> time I do this however, Gnus asks me how many articles I want to
>>>> retrieve, and issues a "UID FETCH 1:N (UID RFC822.SIZE
>>>> BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (...)])" IMAP command, which
>>>> can take quite some time to complete when N is large.
>>>>
>>>> Is there a way to convince Gnus to cache the result of that command
>>>> (without caching all emails), and to issue a command only for the
>>>> new UIDs?  Caching the result of that command should not eat too
>>>> much disk space.

[...]

> We could do something ad-hoc for nnimap, but yes I'm thinking of
> something more generic. All the header data (what's used to create the
> Summary display) is held in variables that are local to the Summary
> buffer, so in principle there's no reason we couldn't just leave the
> local data in place when we leave the buffer. There are plenty of
> obstacles to making it work correctly, but in principle I don't see why not.

The `agent' functionality might help here, in that it (among other
things) creates an on-disk cache of headers seen in a group handled by
the agent, so that it only needs to download headers again for articles
that have not been seen yet. See (info "(gnus) Agent as Cache") for more
on this.

--wgreenhouse

Re: Gnus: caching message headers?

[Eric S Fraga]

On Monday,  7 Sep 2020 at 13:49, wgreenhouse wrote:
> The `agent' functionality might help here, in that it (among other

It will indeed and I second this suggestion!

I used to use the agent functionality for off-line reading of my emails
while commuting on train.  Now, with hotspot availability through my
phone, I've not used it for this purpose in several years.  However,
I've kept the groups under agent control for efficient handling of large
groups (news and imap) even if I don't necessarily download all the
articles.

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.3.7 on Debian bullseye/sid

Re: Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

>
> The `agent' functionality might help here, in that it (among other 
> things) creates an on-disk cache of headers seen in a group handled by 
> the agent, so that it only needs to download headers again for articles 
> that have not been seen yet. See (info "(gnus) Agent as Cache") for more 
> on this.
>

That did the trick, thank you.  I had some difficulty to set it up, but 
now it seems to work.  Is there a way to tell gnus-agent to create only a 
cache of headers, that is, to not store a local copy of the emails that 
have been read?

Gregory

Re: Gnus: caching message headers?

[Stefan Monnier]

> it seems to work.  Is there a way to tell gnus-agent to create only a cache
> of headers, that is, to not store a local copy of the emails that have
> been read?

Interesting.  IMO nowadays MUAs should work by systematically
maintaining a full local copy of your mailboxes/folders.

[ And ideally, the IMAP protocol would be replaced by something like the
  Git protocol for that, so something like `git pull`, and `git push` would
  be used to fetch new mail and update flags, delete messages on the
  remote server, ...  ]


-- Stefan

Re: Gnus: caching message headers?

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> it seems to work.  Is there a way to tell gnus-agent to create only a cache
>> of headers, that is, to not store a local copy of the emails that have
>> been read?
>
> Interesting.  IMO nowadays MUAs should work by systematically
> maintaining a full local copy of your mailboxes/folders.

Setting `gnus-use-cache' to t will save everything. I can't remember if
there are any knobs for limiting the total disk space used, etc.

> [ And ideally, the IMAP protocol would be replaced by something like the
>   Git protocol for that, so something like `git pull`, and `git push` would
>   be used to fetch new mail and update flags, delete messages on the
>   remote server, ...  ]

Now this is just crazy talk :) But have you looked a JMAP?

https://jmap.io/spec.html

Re: Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

>
>> Is there a way to tell gnus-agent to create only a cache of headers, 
>> that is, to not store a local copy of the emails that have been read?
>
> Interesting.  IMO nowadays MUAs should work by systematically 
> maintaining a full local copy of your mailboxes/folders.
>

I (respectfully!) disagree.  This means that each MUA has to reinvent what 
already exists on the server, that is, to reinvent its own IMAP-like 
implementation, to store emails and to locate information in emails.  It 
is also a waste of resources, given that only a very small fraction of the 
emails are read more than once.  A "pure IMAP" MUA only has to query the 
server through the IMAP protocol, and only manages (1) a list of folders 
and (2) a list of email summaries in each folder.  In fact it could even 
manage nothing, and query the list of folders each time it is started, and 
create a summary of the N most recent emails each time a folder is opened.

Re: Gnus: caching message headers?

[Stefan Monnier]

>>> Is there a way to tell gnus-agent to create only a cache of headers, that
>>> is, to not store a local copy of the emails that have been read?
>> Interesting.  IMO nowadays MUAs should work by systematically maintaining
>> a full local copy of your mailboxes/folders.
> I (respectfully!) disagree.  This means that each MUA has to reinvent what
> already exists on the server, that is, to reinvent its own IMAP-like
> implementation, to store emails and to locate information in emails.

If your MUA caches some email messages locally across sessions, then it
probably already has all that complexity ;-)


        Stefan

Re: Gnus: caching message headers?

[wgreenhouse]

Gregory Heytings <ghe@sdf.org> writes:

[...]

> That did the trick, thank you.  I had some difficulty to set it up,
> but now it seems to work.  Is there a way to tell gnus-agent to create
> only a cache of headers, that is, to not store a local copy of the
> emails that have been read?
>
> Gregory

I'm really not sure. It might be that you could set agent parameters in
such a way that no articles were eligible for downloading. But I'm not
sure the "agent cache" obeys the parameters; I have a feeling it just
downloads every message you read unconditionally.

Re: Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

>
>> That did the trick, thank you.  I had some difficulty to set it up, but 
>> now it seems to work.  Is there a way to tell gnus-agent to create only 
>> a cache of headers, that is, to not store a local copy of the emails 
>> that have been read?
>
> I'm really not sure. It might be that you could set agent parameters in 
> such a way that no articles were eligible for downloading. But I'm not 
> sure the "agent cache" obeys the parameters; I have a feeling it just 
> downloads every message you read unconditionally.
>

A simple and direct way to do what I wanted is:

(defun gnus-agent-do-not-store-article (fun &rest args) "Do nothing") 
(advice-add 'gnus-agent-store-article :around #'gnus-agent-do-not-store-article)

Re: Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

>> I'm clearly not an expert, but would it not be enough to save and 
>> retrieve the contents of nntp-server-buffer in nnimap-retrieve-headers? 
>> Or are you thinking about a more generic solution that would work with 
>> all backends?
>
> We could do something ad-hoc for nnimap, but yes I'm thinking of 
> something more generic. All the header data (what's used to create the 
> Summary display) is held in variables that are local to the Summary 
> buffer, so in principle there's no reason we couldn't just leave the 
> local data in place when we leave the buffer. There are plenty of 
> obstacles to making it work correctly, but in principle I don't see why 
> not.
>

I would be happy with an ad-hoc solution for nnimap, which I suppose is 
the most used backend these days (but I could be wrong).  In fact I wasn't 
thinking of just leaving the local data in place when the summary buffer 
is left (which could eat a lot of memory), but instead to store them 
permanently in a cache on disk, and to read that file again instead of 
doing a "UID FETCH" on the server.

Re: Gnus: caching message headers?

[Eric Abrahamsen]

Gregory Heytings via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

>>> I'm clearly not an expert, but would it not be enough to save and
>>> retrieve the contents of nntp-server-buffer in
>>> nnimap-retrieve-headers? Or are you thinking about a more generic
>>> solution that would work with all backends?
>>
>> We could do something ad-hoc for nnimap, but yes I'm thinking of
>> something more generic. All the header data (what's used to create
>> the Summary display) is held in variables that are local to the
>> Summary buffer, so in principle there's no reason we couldn't just
>> leave the local data in place when we leave the buffer. There are
>> plenty of obstacles to making it work correctly, but in principle I
>> don't see why not.
>>
>
> I would be happy with an ad-hoc solution for nnimap, which I suppose
> is the most used backend these days (but I could be wrong).  In fact I
> wasn't thinking of just leaving the local data in place when the
> summary buffer is left (which could eat a lot of memory), but instead
> to store them permanently in a cache on disk, and to read that file
> again instead of doing a "UID FETCH" on the server.

Check out wgreenhouse's suggestion!

Multiple summary buffers (was: Gnus: caching message headers?)

[Tim Landscheidt]

Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> […]

> There's currently no way around this, as Gnus only lets you have one
> active Summary buffer at a time, which means all the old data is dumped
> every time you switch groups. In the back of my head I have some ideas
> for removing this restriction, but it will take a long time to get
> there.

> […]

Just to clarify: I've been using several *Summary* buffers
(for /different/ groups) all the time.  Is this not supposed
to work?  Or did you mean "Gnus only lets you have one ac-
tive Summary buffer per group at a time?"

Tim

Re: Multiple summary buffers

[Eric Abrahamsen]

Tim Landscheidt <tim@tim-landscheidt.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> […]
>
>> There's currently no way around this, as Gnus only lets you have one
>> active Summary buffer at a time, which means all the old data is dumped
>> every time you switch groups. In the back of my head I have some ideas
>> for removing this restriction, but it will take a long time to get
>> there.
>
>> […]
>
> Just to clarify: I've been using several *Summary* buffers
> (for /different/ groups) all the time.  Is this not supposed
> to work?  Or did you mean "Gnus only lets you have one ac-
> tive Summary buffer per group at a time?"

Yes, that wasn't phrased quite right. It's certainly possible to simply
leave the Summary buffers active, but it's awkward, and the buffers
don't sync with the server without "/ N" and/or "M-g", both of which
effectively redraw the buffer anyway. So I'm looking at ways to update
the Summary buffer "in place", either automatically or on an interactive
command.