From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eric Abrahamsen Newsgroups: gmane.emacs.bugs Subject: bug#38011: 27.0.50; [PATCH] WIP on allowing Gnus backends to return header data directly Date: Sun, 03 Jan 2021 13:38:40 -0800 Message-ID: <87turxk8f3.fsf@ericabrahamsen.net> References: <87y2x0a9bl.fsf@ericabrahamsen.net> <87a79f3cu3.fsf@gnus.org> <8736f7l9rd.fsf@ericabrahamsen.net> <87eeyqfi4u.fsf@gnus.org> <87mud75l3i.fsf@ericabrahamsen.net> <87k18a3wtc.fsf@gnus.org> <871ruiqc19.fsf@ericabrahamsen.net> <87imnu2foy.fsf@gnus.org> <87tv27rlcz.fsf@ericabrahamsen.net> <87mu6t37c5.fsf@gnus.org> <878scvluw2.fsf@ericabrahamsen.net> <87a6xb5sa7.fsf@gnus.org> <87h7rikcsq.fsf@ericabrahamsen.net> <87zh1sowlf.fsf@ericabrahamsen.net> <874kjzangc.fsf@gnus.org> <87ble7kqsn.fsf@ericabrahamsen.net> <877doua2g5.fsf@gnus.org> <87y2h9kd88.fsf@ericabrahamsen.net> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="36211"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: 38011@debbugs.gnu.org To: Lars Ingebrigtsen Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sun Jan 03 22:39:20 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kwB5r-0009KS-Pm for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 03 Jan 2021 22:39:20 +0100 Original-Received: from localhost ([::1]:55452 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kwB5p-0004zP-4J for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 03 Jan 2021 16:39:18 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:44198) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kwB5a-0004zH-Ja for bug-gnu-emacs@gnu.org; Sun, 03 Jan 2021 16:39:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:55415) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kwB5a-0004zo-CD for bug-gnu-emacs@gnu.org; Sun, 03 Jan 2021 16:39:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kwB5a-00075i-9Z for bug-gnu-emacs@gnu.org; Sun, 03 Jan 2021 16:39:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Eric Abrahamsen Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sun, 03 Jan 2021 21:39:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 38011 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 38011-submit@debbugs.gnu.org id=B38011.160970993527244 (code B ref 38011); Sun, 03 Jan 2021 21:39:02 +0000 Original-Received: (at 38011) by debbugs.gnu.org; 3 Jan 2021 21:38:55 +0000 Original-Received: from localhost ([127.0.0.1]:38728 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kwB5T-00075M-1c for submit@debbugs.gnu.org; Sun, 03 Jan 2021 16:38:55 -0500 Original-Received: from ericabrahamsen.net ([52.70.2.18]:50548 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kwB5R-00075A-8m for 38011@debbugs.gnu.org; Sun, 03 Jan 2021 16:38:54 -0500 Original-Received: from localhost (c-73-254-86-141.hsd1.wa.comcast.net [73.254.86.141]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id DE8FCFA099; Sun, 3 Jan 2021 21:38:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericabrahamsen.net; s=mail; t=1609709927; bh=fbcqJrqjc+P2PwvZDUqChU8ezoWpOPaMKtwJbr+SeQ8=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=DAIHsPf8quaNj2eSRk0lEzGt2yT2w4rDf7+CjfoZ//x5gpPFiDxaFk9QaoJmIivjb PV0Y6/m+8QdF76fG4N9vgWr3zPtPGvBKdAERQlWk8kkAJloUztFPqpyMqcawaHXFjK YVonvsSqBXqOYhcfPc2jZo1oBtn8+y5v+Q/IELtU= In-Reply-To: <87y2h9kd88.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Sun, 03 Jan 2021 11:54:47 -0800") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:197277 Archived-At: Eric Abrahamsen writes: > On 01/03/21 08:45 AM, Lars Ingebrigtsen wrote: >> Eric Abrahamsen writes: >> >>> Sure thing. It's in girzel/gnus-headers now. I made a few more sneaky >>> last minute changes, so yes... testing is in order. >> >> I'm now running this branch, and good news: No breakages so far. :-) > > Great! > >>> It gets complicated because the cache and the agent need to mix their >>> saved headers into whatever newly-fetched headers we get from the >>> server. So instead of having them call `gnus-retrieve-headers' and >>> mixing their cached text into the nntp-server-buffer, they now call >>> `gnus-fetch-headers' on the server, which actually returns real headers. >> >> Yes, that was the main bit I was unsure of. The braiding stuff tries to >> be efficient and avoid parsing things twice (or more) -- you usually get >> a bunch of headers from the NNTP server, and then you have even more >> headers in the agent/cache, and it stitches them all together as text, >> and parses the resulting mess. (If I remember correctly; it's been at >> least a decade since I looked at that code.) >> >> Are headers parsed more than once now and then merged? > > I certainly hope they're not parsed more than once, but it would > definitely be good to have more eyes on that part of the code. For > instance, `gnus-cache-retrieve-headers' finds cached articles in the > group, then uses (gnus-sorted-difference articles cached) to find > uncached articles. The uncached articles are sent to > `gnus-fetch-headers', and the cached articles are parsed from the cache > file using: > > (gnus-get-newsgroup-headers-xover > (gnus-sorted-difference > cached uncached-articles)) > > Then the two sets of headers are merged and sorted. The agent does > something similar. I don't think the parsing should do any duplicate > work, though it's possible that adding the newly-received headers back > into the cache file is not optimal: it appends the new headers at the end > of the buffer, then re-sorts the whole buffer. > > Crud, I just realized that the agent will re-sort its cache file using > `gnus-agent-check-overview-buffer', but gnus-cache doesn't do anything > similar. So the write process will have to insert the new headers into > the buffer in sorted order. Oh hang on, we're not supposed to be adding any new headers to the gnus-cache cache files here at all. So that chunk of code can just be removed. Just when you thought you had a handle on what was happening...