bug#38011: 27.0.50; [PATCH] WIP on allowing Gnus backends to return header data directly

[Eric Abrahamsen <eric@ericabrahamsen.net>]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I revisit this every few months, and have to completely relearn all the
>> code each time. With any luck that means that I've looked over these
>> diffs sufficiently to have caught more bugs.
>>
>> At any rate, I think this is finally ready to go.
>
> Congratulations!
>
> That's a big patch, and skimming it, I'm not quite sure I understand it
> all.  Could you put this on a branch so that we can get a bit of testing
> before merging it?

Sure thing. It's in girzel/gnus-headers now. I made a few more sneaky
last minute changes, so yes... testing is in order.

The basic principle is simple: it gives backends the option of parsing
their own article headers, rather than writing text into the
nntp-server-buffer to get parsed later. In this sense, the diff on
`gnus-fetch-headers' is all there is to it. None of the backends
actually do this right now.

It gets complicated because the cache and the agent need to mix their
saved headers into whatever newly-fetched headers we get from the
server. So instead of having them call `gnus-retrieve-headers' and
mixing their cached text into the nntp-server-buffer, they now call
`gnus-fetch-headers' on the server, which actually returns real headers.

That means they also need to be responsible for extracting real headers
from their own cache files (rather than letting that happen further down
the line). In this patch both do that with
`gnus-get-newsgroup-headers-xover' (which efficiently parses only a
subset of [potentially very many] cached headers), then merge/sort those
headers with what came back from the server.

A few points of contention:

1. I'm not sure there's a real difference between
   `gnus-agent-fetch-headers' and `gnus-agent-retrieve-headers' anymore.
   Both return actual headers. It would take a quiet afternoon of
   staring at the code to know for sure.
2. The agent and cache are now using `gnus-get-newsgroup-headers-xover'
   to parse their cached headers, which does its own dependency
   building. This means that `gnus-fetch-headers' has to be careful not
   to double-register headers in the dependency table. It also means
   that the agent and cache have to reach waaaaaay back to find a
   reference to the `gnus-dependency-table', which they're both doing
   with a call to `buffer-local-value', which feels gross and fragile.
   In general I would much prefer to build the dependency table in one
   place, preferably after all the headers have been retrieved,
   preferably in `gnus-select-newsgroup'. Another option would be to not
   use the higher-level `gnus-get-newsgroup-headers-xover', but instead
   to scan the cache files for article numbers and use the lower-level
   `nnheader-parse-nov', which isn't concerned with dependencies.
3. In general it took many extra brain cycles (of which I do not have a
   surplus) to follow the code flow. I would love it if
   `gnus-retrieve-headers' -- instead of calling one of
   `gnus-(cache|agent)-retrieve-headers' and expecting them to re-call
   `gnus-retrieve-headers' multiple times with various global variables
   set -- instead called everything consecutively, dumping the article
   list into each function by turn -- cache, agent, server -- and
   filtering the list according to which headers we get back. But that's
   another patch for another time.