Gnus: caching message headers?

Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

Hi list,

I'm trying to configure Gnus to access my email through IMAP.

I'm too accustomed to traditional MUAs to switch to the "Gnus way" in one 
step, so I have:

(setq gnus-permanently-visible-groups ".*")
(setq gnus-parameters
    '((".*"
       (gnus-show-threads nil)
       (gnus-use-scoring nil)
       (gnus-article-sort-functions '(gnus-article-sort-by-number))
       (display . all))))

I would like to have, in the *Summary*, a complete list of the emails 
contained in a folder when I hit RET on its label.  Each time I do this 
however, Gnus asks me how many articles I want to retrieve, and issues a 
"UID FETCH 1:N (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS 
(...)])" IMAP command, which can take quite some time to complete when N 
is large.

Is there a way to convince Gnus to cache the result of that command 
(without caching all emails), and to issue a command only for the new 
UIDs?  Caching the result of that command should not eat too much disk 
space.

I apologize in advance if that question is naïve, or if I missed some part 
of the manual.

Gregory

Re: Gnus: caching message headers?

[Eric Abrahamsen]

Gregory Heytings via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> Hi list,
>
> I'm trying to configure Gnus to access my email through IMAP.
>
> I'm too accustomed to traditional MUAs to switch to the "Gnus way" in
> one step, so I have:
>
> (setq gnus-permanently-visible-groups ".*")
> (setq gnus-parameters
>    '((".*"
>       (gnus-show-threads nil)
>       (gnus-use-scoring nil)
>       (gnus-article-sort-functions '(gnus-article-sort-by-number))
>       (display . all))))
>
> I would like to have, in the *Summary*, a complete list of the emails
> contained in a folder when I hit RET on its label.  Each time I do
> this however, Gnus asks me how many articles I want to retrieve, and
> issues a "UID FETCH 1:N (UID RFC822.SIZE BODYSTRUCTURE
> BODY.PEEK[HEADER.FIELDS (...)])" IMAP command, which can take quite
> some time to complete when N is large.
>
> Is there a way to convince Gnus to cache the result of that command
> (without caching all emails), and to issue a command only for the new
> UIDs?  Caching the result of that command should not eat too much disk
> space.

No, not at the moment. You can avoid the prompt by customizing the value
of `gnus-large-newsgroup', but it's still going to retrieve all the
headers for the group.

There's currently no way around this, as Gnus only lets you have one
active Summary buffer at a time, which means all the old data is dumped
every time you switch groups. In the back of my head I have some ideas
for removing this restriction, but it will take a long time to get
there.

Eric

Re: Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

Hi Eric,

Many thanks for your answer.

>> I would like to have, in the *Summary*, a complete list of the emails 
>> contained in a folder when I hit RET on its label.  Each time I do this 
>> however, Gnus asks me how many articles I want to retrieve, and issues 
>> a "UID FETCH 1:N (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS 
>> (...)])" IMAP command, which can take quite some time to complete when 
>> N is large.
>>
>> Is there a way to convince Gnus to cache the result of that command 
>> (without caching all emails), and to issue a command only for the new 
>> UIDs?  Caching the result of that command should not eat too much disk 
>> space.
>
> No, not at the moment. You can avoid the prompt by customizing the value 
> of `gnus-large-newsgroup', but it's still going to retrieve all the 
> headers for the group.
>
> There's currently no way around this, as Gnus only lets you have one 
> active Summary buffer at a time, which means all the old data is dumped 
> every time you switch groups. In the back of my head I have some ideas 
> for removing this restriction, but it will take a long time to get 
> there.
>

I'm clearly not an expert, but would it not be enough to save and retrieve 
the contents of nntp-server-buffer in nnimap-retrieve-headers?  Or are you 
thinking about a more generic solution that would work with all backends?

Re: Gnus: caching message headers?

[Eric Abrahamsen]

Gregory Heytings via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> Hi Eric,
>
> Many thanks for your answer.
>
>>> I would like to have, in the *Summary*, a complete list of the
>>> emails contained in a folder when I hit RET on its label.  Each
>>> time I do this however, Gnus asks me how many articles I want to
>>> retrieve, and issues a "UID FETCH 1:N (UID RFC822.SIZE
>>> BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (...)])" IMAP command, which
>>> can take quite some time to complete when N is large.
>>>
>>> Is there a way to convince Gnus to cache the result of that command
>>> (without caching all emails), and to issue a command only for the
>>> new UIDs?  Caching the result of that command should not eat too
>>> much disk space.
>>
>> No, not at the moment. You can avoid the prompt by customizing the
>> value of `gnus-large-newsgroup', but it's still going to retrieve
>> all the headers for the group.
>>
>> There's currently no way around this, as Gnus only lets you have one
>> active Summary buffer at a time, which means all the old data is
>> dumped every time you switch groups. In the back of my head I have
>> some ideas for removing this restriction, but it will take a long
>> time to get there.
>>
>
> I'm clearly not an expert, but would it not be enough to save and
> retrieve the contents of nntp-server-buffer in
> nnimap-retrieve-headers?  Or are you thinking about a more generic
> solution that would work with all backends?

We could do something ad-hoc for nnimap, but yes I'm thinking of
something more generic. All the header data (what's used to create the
Summary display) is held in variables that are local to the Summary
buffer, so in principle there's no reason we couldn't just leave the
local data in place when we leave the buffer. There are plenty of
obstacles to making it work correctly, but in principle I don't see why not.

Re: Gnus: caching message headers?

[wgreenhouse]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Gregory Heytings via Users list for the GNU Emacs text editor
> <help-gnu-emacs@gnu.org> writes:
>
>> Hi Eric,
>>
>> Many thanks for your answer.
>>
>>>> I would like to have, in the *Summary*, a complete list of the
>>>> emails contained in a folder when I hit RET on its label.  Each
>>>> time I do this however, Gnus asks me how many articles I want to
>>>> retrieve, and issues a "UID FETCH 1:N (UID RFC822.SIZE
>>>> BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (...)])" IMAP command, which
>>>> can take quite some time to complete when N is large.
>>>>
>>>> Is there a way to convince Gnus to cache the result of that command
>>>> (without caching all emails), and to issue a command only for the
>>>> new UIDs?  Caching the result of that command should not eat too
>>>> much disk space.

[...]

> We could do something ad-hoc for nnimap, but yes I'm thinking of
> something more generic. All the header data (what's used to create the
> Summary display) is held in variables that are local to the Summary
> buffer, so in principle there's no reason we couldn't just leave the
> local data in place when we leave the buffer. There are plenty of
> obstacles to making it work correctly, but in principle I don't see why not.

The `agent' functionality might help here, in that it (among other
things) creates an on-disk cache of headers seen in a group handled by
the agent, so that it only needs to download headers again for articles
that have not been seen yet. See (info "(gnus) Agent as Cache") for more
on this.

--wgreenhouse

Re: Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

>> I'm clearly not an expert, but would it not be enough to save and 
>> retrieve the contents of nntp-server-buffer in nnimap-retrieve-headers? 
>> Or are you thinking about a more generic solution that would work with 
>> all backends?
>
> We could do something ad-hoc for nnimap, but yes I'm thinking of 
> something more generic. All the header data (what's used to create the 
> Summary display) is held in variables that are local to the Summary 
> buffer, so in principle there's no reason we couldn't just leave the 
> local data in place when we leave the buffer. There are plenty of 
> obstacles to making it work correctly, but in principle I don't see why 
> not.
>

I would be happy with an ad-hoc solution for nnimap, which I suppose is 
the most used backend these days (but I could be wrong).  In fact I wasn't 
thinking of just leaving the local data in place when the summary buffer 
is left (which could eat a lot of memory), but instead to store them 
permanently in a cache on disk, and to read that file again instead of 
doing a "UID FETCH" on the server.

Re: Gnus: caching message headers?

[Eric Abrahamsen]

Gregory Heytings via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

>>> I'm clearly not an expert, but would it not be enough to save and
>>> retrieve the contents of nntp-server-buffer in
>>> nnimap-retrieve-headers? Or are you thinking about a more generic
>>> solution that would work with all backends?
>>
>> We could do something ad-hoc for nnimap, but yes I'm thinking of
>> something more generic. All the header data (what's used to create
>> the Summary display) is held in variables that are local to the
>> Summary buffer, so in principle there's no reason we couldn't just
>> leave the local data in place when we leave the buffer. There are
>> plenty of obstacles to making it work correctly, but in principle I
>> don't see why not.
>>
>
> I would be happy with an ad-hoc solution for nnimap, which I suppose
> is the most used backend these days (but I could be wrong).  In fact I
> wasn't thinking of just leaving the local data in place when the
> summary buffer is left (which could eat a lot of memory), but instead
> to store them permanently in a cache on disk, and to read that file
> again instead of doing a "UID FETCH" on the server.

Check out wgreenhouse's suggestion!

Multiple summary buffers (was: Gnus: caching message headers?)

[Tim Landscheidt]

Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> […]

> There's currently no way around this, as Gnus only lets you have one
> active Summary buffer at a time, which means all the old data is dumped
> every time you switch groups. In the back of my head I have some ideas
> for removing this restriction, but it will take a long time to get
> there.

> […]

Just to clarify: I've been using several *Summary* buffers
(for /different/ groups) all the time.  Is this not supposed
to work?  Or did you mean "Gnus only lets you have one ac-
tive Summary buffer per group at a time?"

Tim

Re: Multiple summary buffers

[Eric Abrahamsen]

Tim Landscheidt <tim@tim-landscheidt.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> […]
>
>> There's currently no way around this, as Gnus only lets you have one
>> active Summary buffer at a time, which means all the old data is dumped
>> every time you switch groups. In the back of my head I have some ideas
>> for removing this restriction, but it will take a long time to get
>> there.
>
>> […]
>
> Just to clarify: I've been using several *Summary* buffers
> (for /different/ groups) all the time.  Is this not supposed
> to work?  Or did you mean "Gnus only lets you have one ac-
> tive Summary buffer per group at a time?"

Yes, that wasn't phrased quite right. It's certainly possible to simply
leave the Summary buffers active, but it's awkward, and the buffers
don't sync with the server without "/ N" and/or "M-g", both of which
effectively redraw the buffer anyway. So I'm looking at ways to update
the Summary buffer "in place", either automatically or on an interactive
command.

Re: Gnus: caching message headers?

[Eric S Fraga]

On Monday,  7 Sep 2020 at 13:49, wgreenhouse wrote:
> The `agent' functionality might help here, in that it (among other

It will indeed and I second this suggestion!

I used to use the agent functionality for off-line reading of my emails
while commuting on train.  Now, with hotspot availability through my
phone, I've not used it for this purpose in several years.  However,
I've kept the groups under agent control for efficient handling of large
groups (news and imap) even if I don't necessarily download all the
articles.

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.3.7 on Debian bullseye/sid

Re: Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

>
> The `agent' functionality might help here, in that it (among other 
> things) creates an on-disk cache of headers seen in a group handled by 
> the agent, so that it only needs to download headers again for articles 
> that have not been seen yet. See (info "(gnus) Agent as Cache") for more 
> on this.
>

That did the trick, thank you.  I had some difficulty to set it up, but 
now it seems to work.  Is there a way to tell gnus-agent to create only a 
cache of headers, that is, to not store a local copy of the emails that 
have been read?

Gregory

Re: Gnus: caching message headers?

[Stefan Monnier]

> it seems to work.  Is there a way to tell gnus-agent to create only a cache
> of headers, that is, to not store a local copy of the emails that have
> been read?

Interesting.  IMO nowadays MUAs should work by systematically
maintaining a full local copy of your mailboxes/folders.

[ And ideally, the IMAP protocol would be replaced by something like the
  Git protocol for that, so something like `git pull`, and `git push` would
  be used to fetch new mail and update flags, delete messages on the
  remote server, ...  ]


-- Stefan

Re: Gnus: caching message headers?

[wgreenhouse]

Gregory Heytings <ghe@sdf.org> writes:

[...]

> That did the trick, thank you.  I had some difficulty to set it up,
> but now it seems to work.  Is there a way to tell gnus-agent to create
> only a cache of headers, that is, to not store a local copy of the
> emails that have been read?
>
> Gregory

I'm really not sure. It might be that you could set agent parameters in
such a way that no articles were eligible for downloading. But I'm not
sure the "agent cache" obeys the parameters; I have a feeling it just
downloads every message you read unconditionally.

Re: Gnus: caching message headers?

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> it seems to work.  Is there a way to tell gnus-agent to create only a cache
>> of headers, that is, to not store a local copy of the emails that have
>> been read?
>
> Interesting.  IMO nowadays MUAs should work by systematically
> maintaining a full local copy of your mailboxes/folders.

Setting `gnus-use-cache' to t will save everything. I can't remember if
there are any knobs for limiting the total disk space used, etc.

> [ And ideally, the IMAP protocol would be replaced by something like the
>   Git protocol for that, so something like `git pull`, and `git push` would
>   be used to fetch new mail and update flags, delete messages on the
>   remote server, ...  ]

Now this is just crazy talk :) But have you looked a JMAP?

https://jmap.io/spec.html

Re: Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

>
>> That did the trick, thank you.  I had some difficulty to set it up, but 
>> now it seems to work.  Is there a way to tell gnus-agent to create only 
>> a cache of headers, that is, to not store a local copy of the emails 
>> that have been read?
>
> I'm really not sure. It might be that you could set agent parameters in 
> such a way that no articles were eligible for downloading. But I'm not 
> sure the "agent cache" obeys the parameters; I have a feeling it just 
> downloads every message you read unconditionally.
>

A simple and direct way to do what I wanted is:

(defun gnus-agent-do-not-store-article (fun &rest args) "Do nothing") 
(advice-add 'gnus-agent-store-article :around #'gnus-agent-do-not-store-article)

Re: Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

>
>> Is there a way to tell gnus-agent to create only a cache of headers, 
>> that is, to not store a local copy of the emails that have been read?
>
> Interesting.  IMO nowadays MUAs should work by systematically 
> maintaining a full local copy of your mailboxes/folders.
>

I (respectfully!) disagree.  This means that each MUA has to reinvent what 
already exists on the server, that is, to reinvent its own IMAP-like 
implementation, to store emails and to locate information in emails.  It 
is also a waste of resources, given that only a very small fraction of the 
emails are read more than once.  A "pure IMAP" MUA only has to query the 
server through the IMAP protocol, and only manages (1) a list of folders 
and (2) a list of email summaries in each folder.  In fact it could even 
manage nothing, and query the list of folders each time it is started, and 
create a summary of the N most recent emails each time a folder is opened.

Re: Gnus: caching message headers?

[Stefan Monnier]

>>> Is there a way to tell gnus-agent to create only a cache of headers, that
>>> is, to not store a local copy of the emails that have been read?
>> Interesting.  IMO nowadays MUAs should work by systematically maintaining
>> a full local copy of your mailboxes/folders.
> I (respectfully!) disagree.  This means that each MUA has to reinvent what
> already exists on the server, that is, to reinvent its own IMAP-like
> implementation, to store emails and to locate information in emails.

If your MUA caches some email messages locally across sessions, then it
probably already has all that complexity ;-)


        Stefan

Re: Gnus: caching message headers?

[Ozhap]

> Is there a way to tell gnus-agent to create only a cache of headers,
> that is, to not store a local copy of the emails that have been read?

I don't think it does. From info '(gnus)Agent Caveats':

> “If I read an article while plugged, do they get entered into the Agent?”
> 
>      *No*.  If you want this behavior, add
>      ‘gnus-agent-fetch-selected-article’ to ‘gnus-select-article-hook’.

--
Ozhap

Re: Gnus: caching message headers?

[Gregory Heytings via Users list for the GNU Emacs text editor]

>
>> Is there a way to tell gnus-agent to create only a cache of headers, 
>> that is, to not store a local copy of the emails that have been read?
>
> I don't think it does. From info '(gnus)Agent Caveats':
>
>> “If I read an article while plugged, do they get entered into the 
>> Agent?”
>>
>> *No*.  If you want this behavior, add 
>> ‘gnus-agent-fetch-selected-article’ to ‘gnus-select-article-hook’.
>

That's what the doc says, and I'm not sure what it means exactly, but with 
the current version of Gnus, and with `gnus-select-article-hook' set to 
nil (the default), a local copy of the emails is indeed entered into the 
agent cache, by `gnus-agent-store-article'.  The backtrace is

gnus-agent-store-article
gnus-request-article-this-buffer 
gnus-article-prepare
gnus-summary-display-article 
gnus-summary-select-article

Re: Gnus: caching message headers?

[Ozhap]

Gregory Heytings <ghe@sdf.org> writes:

>>> Is there a way to tell gnus-agent to create only a cache of headers, 
>>> that is, to not store a local copy of the emails that have been read?
>>
>> I don't think it does. From info '(gnus)Agent Caveats':
>>
>>> “If I read an article while plugged, do they get entered into the 
>>> Agent?”
>>>
>>> *No*.  If you want this behavior, add 
>>> ‘gnus-agent-fetch-selected-article’ to ‘gnus-select-article-hook’.
>>
>
> That's what the doc says, and I'm not sure what it means exactly, but with 
> the current version of Gnus, and with `gnus-select-article-hook' set to 
> nil (the default), a local copy of the emails is indeed entered into the 
> agent cache, by `gnus-agent-store-article'.  The backtrace is

You should reset the default agent category predicate from 'short' to
'false' if you havent. See '(gnus)Category Syntax' for details.

Used to work properly for me when I had it set up like that. I'm using 26.3.

--
Ozhap

Re: Gnus: caching message headers?

[Ozhap]

Ozhap <ozhap@vollbio.de> writes:

> Gregory Heytings <ghe@sdf.org> writes:
>
>>>> Is there a way to tell gnus-agent to create only a cache of headers, 
>>>> that is, to not store a local copy of the emails that have been read?
>>>
>>> I don't think it does. From info '(gnus)Agent Caveats':
>>>
>>>> “If I read an article while plugged, do they get entered into the 
>>>> Agent?”
>>>>
>>>> *No*.  If you want this behavior, add 
>>>> ‘gnus-agent-fetch-selected-article’ to ‘gnus-select-article-hook’.
>>>
>>
>> That's what the doc says, and I'm not sure what it means exactly, but with 
>> the current version of Gnus, and with `gnus-select-article-hook' set to 
>> nil (the default), a local copy of the emails is indeed entered into the 
>> agent cache, by `gnus-agent-store-article'.  The backtrace is
>
> You should reset the default agent category predicate from 'short' to
> 'false' if you havent. See '(gnus)Category Syntax' for details.

Scratch that. I guess it's only for gnus-agent-fetch-session.

> Used to work properly for me when I had it set up like that. I'm using 26.3.

In fact I was once trying to get the opposite behaviour to your desired
one - to store read emails, which was not happening by default. And
finally achieved it with the above option.

--
Ozhap

Re: Gnus: caching message headers?

[wgreenhouse]

Gregory Heytings via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

[...]

> That's what the doc says, and I'm not sure what it means exactly, but
> with the current version of Gnus, and with `gnus-select-article-hook'
> set to nil (the default), a local copy of the emails is indeed entered
> into the agent cache, by `gnus-agent-store-article'.  The backtrace is
>
> gnus-agent-store-article
> gnus-request-article-this-buffer gnus-article-prepare
> gnus-summary-display-article gnus-summary-select-article

That Info page could be more explicit that `gnus-agent-cache' means that
articles *will* be added to the agent as they are read.

Re: Gnus: caching message headers?

[Ozhap]

>> “If I read an article while plugged, do they get entered into the 
>> Agent?”
>>
>> *No*.  If you want this behavior, add 
>> ‘gnus-agent-fetch-selected-article’ to ‘gnus-select-article-hook’.

> That Info page could be more explicit that `gnus-agent-cache' means that
> articles *will* be added to the agent as they are read.

That's how it stands now, but in info '(gnus)Agent as Cache' it says:

"..Gnus normally only downloads *headers* once, and stores them in the
Agent"

"Articles are *not* cached in the Agent by default though (that would
potentially consume lots of disk space), but if you have *already
downloaded an article into the Agent*, Gnus will not download the
article from the server again but use the locally stored copy instead."

I did some digging, and this behaviour was changed due to bug#8502 where
Lars concludes (erroneously, on the face of it):

> Let's see...  caching...  so if you're online (in an agentised group)
> and select an article, you want the article to be saved in the Agent
> directory?  That sounds eminently reasonable, and I thought that was
> supposed to happen, but I can't find any code for doing that...  hm...
> Nope.

Ref:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=8502

--
Ozhap

Re: Gnus: caching message headers?

[Eric Abrahamsen]

Ozhap <ozhap@vollbio.de> writes:

>>> “If I read an article while plugged, do they get entered into the 
>>> Agent?”
>>>
>>> *No*.  If you want this behavior, add 
>>> ‘gnus-agent-fetch-selected-article’ to ‘gnus-select-article-hook’.
>
>> That Info page could be more explicit that `gnus-agent-cache' means that
>> articles *will* be added to the agent as they are read.
>
> That's how it stands now, but in info '(gnus)Agent as Cache' it says:
>
> "..Gnus normally only downloads *headers* once, and stores them in the
> Agent"
>
> "Articles are *not* cached in the Agent by default though (that would
> potentially consume lots of disk space), but if you have *already
> downloaded an article into the Agent*, Gnus will not download the
> article from the server again but use the locally stored copy instead."
>
> I did some digging, and this behaviour was changed due to bug#8502 where
> Lars concludes (erroneously, on the face of it):
>
>> Let's see...  caching...  so if you're online (in an agentised group)
>> and select an article, you want the article to be saved in the Agent
>> directory?  That sounds eminently reasonable, and I thought that was
>> supposed to happen, but I can't find any code for doing that...  hm...
>> Nope.
>
> Ref:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=8502

If I were you I would do this up as a proper bug report. Lars is now an
Emacs maintainer, so he'll be obliged to look at it :)

One way or another there is a mismatch between documentation,
customization options, and actual observed behavior. I think if you can
make it clear what that mismatch is, we stand a good chance of getting
it fixed.

Re: Gnus: caching message headers?

[Ozhap]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Ozhap <ozhap@vollbio.de> writes:
>
>>>> “If I read an article while plugged, do they get entered into the 
>>>> Agent?”
>>>>
>>>> *No*.  If you want this behavior, add 
>>>> ‘gnus-agent-fetch-selected-article’ to ‘gnus-select-article-hook’.
>>
>>> That Info page could be more explicit that `gnus-agent-cache' means that
>>> articles *will* be added to the agent as they are read.
>>
>> That's how it stands now, but in info '(gnus)Agent as Cache' it says:
>>
>> "..Gnus normally only downloads *headers* once, and stores them in the
>> Agent"
>>
>> "Articles are *not* cached in the Agent by default though (that would
>> potentially consume lots of disk space), but if you have *already
>> downloaded an article into the Agent*, Gnus will not download the
>> article from the server again but use the locally stored copy instead."
>>
>> I did some digging, and this behaviour was changed due to bug#8502 where
>> Lars concludes (erroneously, on the face of it):
>>
>>> Let's see...  caching...  so if you're online (in an agentised group)
>>> and select an article, you want the article to be saved in the Agent
>>> directory?  That sounds eminently reasonable, and I thought that was
>>> supposed to happen, but I can't find any code for doing that...  hm...
>>> Nope.
>>
>> Ref:
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=8502
>
> If I were you I would do this up as a proper bug report. Lars is now an
> Emacs maintainer, so he'll be obliged to look at it :)
>
> One way or another there is a mismatch between documentation,
> customization options, and actual observed behavior. I think if you can
> make it clear what that mismatch is, we stand a good chance of getting
> it fixed.

Here you go. I think I did good :)
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43356

--
Ozhap

Re: Gnus: caching message headers?

[Eric Abrahamsen]

Ozhap <ozhap@vollbio.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Ozhap <ozhap@vollbio.de> writes:
>>
>>>>> “If I read an article while plugged, do they get entered into the 
>>>>> Agent?”
>>>>>
>>>>> *No*.  If you want this behavior, add 
>>>>> ‘gnus-agent-fetch-selected-article’ to ‘gnus-select-article-hook’.
>>>
>>>> That Info page could be more explicit that `gnus-agent-cache' means that
>>>> articles *will* be added to the agent as they are read.
>>>
>>> That's how it stands now, but in info '(gnus)Agent as Cache' it says:
>>>
>>> "..Gnus normally only downloads *headers* once, and stores them in the
>>> Agent"
>>>
>>> "Articles are *not* cached in the Agent by default though (that would
>>> potentially consume lots of disk space), but if you have *already
>>> downloaded an article into the Agent*, Gnus will not download the
>>> article from the server again but use the locally stored copy instead."
>>>
>>> I did some digging, and this behaviour was changed due to bug#8502 where
>>> Lars concludes (erroneously, on the face of it):
>>>
>>>> Let's see...  caching...  so if you're online (in an agentised group)
>>>> and select an article, you want the article to be saved in the Agent
>>>> directory?  That sounds eminently reasonable, and I thought that was
>>>> supposed to happen, but I can't find any code for doing that...  hm...
>>>> Nope.
>>>
>>> Ref:
>>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=8502
>>
>> If I were you I would do this up as a proper bug report. Lars is now an
>> Emacs maintainer, so he'll be obliged to look at it :)
>>
>> One way or another there is a mismatch between documentation,
>> customization options, and actual observed behavior. I think if you can
>> make it clear what that mismatch is, we stand a good chance of getting
>> it fixed.
>
> Here you go. I think I did good :)
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43356

I saw it, I agree it's good!

Re: Gnus: caching message headers?

[Madhu]

* Eric Abrahamsen <87lfheefq4.fsf@ericabrahamsen.net> :
Wrote on Sat, 12 Sep 2020 16:29:39 -0700:

> Ozhap <ozhap@vollbio.de> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Ozhap <ozhap@vollbio.de> writes:
>>>
>>>>>> "If I read an article while plugged, do they get entered into the
>>>>>> Agent?"
>>>>>>
>>>>>> *No*.  If you want this behavior, add
>>>>>> `gnus-agent-fetch-selected-article' to `gnus-select-article-hook'.
>>>>
>>>>> That Info page could be more explicit that `gnus-agent-cache' means that
>>>>> articles *will* be added to the agent as they are read.
>>>>
>>>> That's how it stands now, but in info '(gnus)Agent as Cache' it says:
>>>>
>>>> "..Gnus normally only downloads *headers* once, and stores them in the
>>>> Agent"
>>>>
>>>> "Articles are *not* cached in the Agent by default though (that
>>>> would potentially consume lots of disk space), but if you have
>>>> *already downloaded an article into the Agent*, Gnus will not
>>>> download the article from the server again but use the locally
>>>> stored copy instead."
>>>>
>>>> I did some digging, and this behaviour was changed due to bug#8502
>>>> where Lars concludes (erroneously, on the face of it):
>>>>
>>>>> Let's see...  caching...  so if you're online (in an agentised
>>>>> group) and select an article, you want the article to be saved in
>>>>> the Agent directory?  That sounds eminently reasonable, and I
>>>>> thought that was supposed to happen, but I can't find any code for
>>>>> doing that...  hm...  Nope.
>>>>
>>>> Ref:
>>>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=8502
>>>
>>> If I were you I would do this up as a proper bug report. Lars is now
>>> an Emacs maintainer, so he'll be obliged to look at it :)
>>>
>>> One way or another there is a mismatch between documentation,
>>> customization options, and actual observed behavior. I think if you
>>> can make it clear what that mismatch is, we stand a good chance of
>>> getting it fixed.
>>
>> Here you go. I think I did good :)
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43356
>
> I saw it, I agree it's good!

Lars followed the instructions like a good boy and and has obsoleted
gnus-agent-store-article and removed that functionality.

| commit def34a20766ea5179afd5e5ed090a2b86fcccb5e
| Author:     Lars Ingebrigtsen <larsi@gnus.org>
| AuthorDate: Sun Sep 13 16:46:11 2020 +0200
|
|     "If I read an article while plugged, do they get entered into the
|     Agent?"
|
|          *No*. If you want this behavior, add
|          `gnus-agent-fetch-selected-article' to
|          `gnus-select-article-hook'.
|


However the overhead of calling gnus-agent-fetch-selected-article is too
much in my 100s of (NNTP) news groups and the several thousands of
articles with complicated reading history. "Yuuge lists of gnus cons
data structures".  The overhead is not acceptable.

The earlier behaviour was exactly what I wanted for NNTP (I dont use
gnus for mail) and I have been using the feature on selective servers
ever since it was added, relying on gnus to back up my memory (If I read
an article gnus can help me remember where I read it). I don't give a
shit about gmail users and I'd suggest reverting it unobsoleting
gnus-agent-store-article and yet another switch for those who want to
keep the original very useful behaviour code path. For those who prefer
to control their own data instead of selling it (along with their access
patterns and their souls) to google.

Re: Gnus: caching message headers?

[Ozhap]

> However the overhead of calling gnus-agent-fetch-selected-article is too
> much

Whoops! Omitting the relevant xkcd :) I should learn to curb my
enthusiasm...

Perhaps you could try advising gnus-request-article to call
gnus-agent-store-article after, when it has a non-nil 3rd argument (it
doesn't seem to be used with it anywhere else).

But probably there should be a new hook. Or of course, reverting it and
rewriting the manual instead.

--
Ozhap

Re: Gnus: caching message headers?

[Ozhap]

> But probably there should be a new hook. Or of course, reverting it and
> rewriting the manual instead.

An easier way would be to use gnus-agent-store-article instead of
gnus-agent-fetch-articles (which is designed for many messages) in
gnus-agent-fetch-selected-article. Please open a bug report so that we
can fix it there.

(A related problem seems to be the download-mark not being updated in
the previous arrangement)

--
Ozhap

Re: Gnus: caching message headers?

[Ozhap]

Ozhap wrote:

>> But probably there should be a new hook. Or of course, reverting it and
>> rewriting the manual instead.
>
> An easier way would be to use gnus-agent-store-article instead of
> gnus-agent-fetch-articles (which is designed for many messages) in
> gnus-agent-fetch-selected-article. Please open a bug report so that we
> can fix it there.

That should read: Check for stored article as in
gnus-agent-store-article in -agent-fetch-selected-article, before
fetching. Well, the previous code (or a new hook) seems better :|