bug#38011: 27.0.50; [PATCH] WIP on allowing Gnus backends to return header data directly

[Eric Abrahamsen <eric@ericabrahamsen.net>]

On 01/03/21 08:45 AM, Lars Ingebrigtsen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Sure thing. It's in girzel/gnus-headers now. I made a few more sneaky
>> last minute changes, so yes... testing is in order.
>
> I'm now running this branch, and good news: No breakages so far.  :-)

Great!

>> It gets complicated because the cache and the agent need to mix their
>> saved headers into whatever newly-fetched headers we get from the
>> server. So instead of having them call `gnus-retrieve-headers' and
>> mixing their cached text into the nntp-server-buffer, they now call
>> `gnus-fetch-headers' on the server, which actually returns real headers.
>
> Yes, that was the main bit I was unsure of.  The braiding stuff tries to
> be efficient and avoid parsing things twice (or more) -- you usually get
> a bunch of headers from the NNTP server, and then you have even more
> headers in the agent/cache, and it stitches them all together as text,
> and parses the resulting mess.  (If I remember correctly; it's been at
> least a decade since I looked at that code.)
>
> Are headers parsed more than once now and then merged?

I certainly hope they're not parsed more than once, but it would
definitely be good to have more eyes on that part of the code. For
instance, `gnus-cache-retrieve-headers' finds cached articles in the
group, then uses (gnus-sorted-difference articles cached) to find
uncached articles. The uncached articles are sent to
`gnus-fetch-headers', and the cached articles are parsed from the cache
file using:

(gnus-get-newsgroup-headers-xover
 (gnus-sorted-difference
 cached uncached-articles))

Then the two sets of headers are merged and sorted. The agent does
something similar. I don't think the parsing should do any duplicate
work, though it's possible that adding the newly-received headers back
into the cache file is not optimal: it appends the new headers at the end
of the buffer, then re-sorts the whole buffer. 

Crud, I just realized that the agent will re-sort its cache file using gnus-agent-check-overview-buffer