From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eric Abrahamsen Newsgroups: gmane.emacs.bugs Subject: bug#38011: 27.0.50; [PATCH] WIP on allowing Gnus backends to return header data directly Date: Sun, 03 Jan 2021 11:54:47 -0800 Message-ID: <87y2h9kd88.fsf@ericabrahamsen.net> References: <87y2x0a9bl.fsf@ericabrahamsen.net> <87a79f3cu3.fsf@gnus.org> <8736f7l9rd.fsf@ericabrahamsen.net> <87eeyqfi4u.fsf@gnus.org> <87mud75l3i.fsf@ericabrahamsen.net> <87k18a3wtc.fsf@gnus.org> <871ruiqc19.fsf@ericabrahamsen.net> <87imnu2foy.fsf@gnus.org> <87tv27rlcz.fsf@ericabrahamsen.net> <87mu6t37c5.fsf@gnus.org> <878scvluw2.fsf@ericabrahamsen.net> <87a6xb5sa7.fsf@gnus.org> <87h7rikcsq.fsf@ericabrahamsen.net> <87zh1sowlf.fsf@ericabrahamsen.net> <874kjzangc.fsf@gnus.org> <87ble7kqsn.fsf@ericabrahamsen.net> <877doua2g5.fsf@gnus.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="30542"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: 38011@debbugs.gnu.org To: Lars Ingebrigtsen Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sun Jan 03 20:55:31 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kw9TP-0007qu-Ky for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 03 Jan 2021 20:55:31 +0100 Original-Received: from localhost ([::1]:33470 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kw9TO-0002EW-OA for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 03 Jan 2021 14:55:30 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:57254) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kw9Sw-0002Cb-Ki for bug-gnu-emacs@gnu.org; Sun, 03 Jan 2021 14:55:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:55336) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kw9Sw-00030l-E3 for bug-gnu-emacs@gnu.org; Sun, 03 Jan 2021 14:55:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kw9Sw-0002YJ-DU for bug-gnu-emacs@gnu.org; Sun, 03 Jan 2021 14:55:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Eric Abrahamsen Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sun, 03 Jan 2021 19:55:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 38011 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 38011-submit@debbugs.gnu.org id=B38011.16097036979797 (code B ref 38011); Sun, 03 Jan 2021 19:55:02 +0000 Original-Received: (at 38011) by debbugs.gnu.org; 3 Jan 2021 19:54:57 +0000 Original-Received: from localhost ([127.0.0.1]:38649 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kw9Sq-0002Xx-MR for submit@debbugs.gnu.org; Sun, 03 Jan 2021 14:54:56 -0500 Original-Received: from ericabrahamsen.net ([52.70.2.18]:46456 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kw9So-0002Xk-SJ for 38011@debbugs.gnu.org; Sun, 03 Jan 2021 14:54:55 -0500 Original-Received: from localhost (c-73-254-86-141.hsd1.wa.comcast.net [73.254.86.141]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id 42CC0FA099; Sun, 3 Jan 2021 19:54:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericabrahamsen.net; s=mail; t=1609703689; bh=cGkoAjOpy+9s1CDI7easJwHR+D5fGFJP6WR4BJz9mQ4=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=ErwFZVozBkH9I/+jfxaEHWT6L/DzcDpS5kH9ag2q7dR8AbvBed3duRV9fb+Shtl5r +KFxl125KT+rg7XO1/3VNVxCCEwOJhkb5RU76UQA7zttGFXSzb+oD3ASERoMrnljMS TmcNtsv5CEn5xD2o8uq3rLBaGyg9DgvUgqMEbqcQ= In-Reply-To: <877doua2g5.fsf@gnus.org> (Lars Ingebrigtsen's message of "Sun, 03 Jan 2021 08:45:30 +0100") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:197266 Archived-At: On 01/03/21 08:45 AM, Lars Ingebrigtsen wrote: > Eric Abrahamsen writes: > >> Sure thing. It's in girzel/gnus-headers now. I made a few more sneaky >> last minute changes, so yes... testing is in order. > > I'm now running this branch, and good news: No breakages so far. :-) Great! >> It gets complicated because the cache and the agent need to mix their >> saved headers into whatever newly-fetched headers we get from the >> server. So instead of having them call `gnus-retrieve-headers' and >> mixing their cached text into the nntp-server-buffer, they now call >> `gnus-fetch-headers' on the server, which actually returns real headers. > > Yes, that was the main bit I was unsure of. The braiding stuff tries to > be efficient and avoid parsing things twice (or more) -- you usually get > a bunch of headers from the NNTP server, and then you have even more > headers in the agent/cache, and it stitches them all together as text, > and parses the resulting mess. (If I remember correctly; it's been at > least a decade since I looked at that code.) > > Are headers parsed more than once now and then merged? I certainly hope they're not parsed more than once, but it would definitely be good to have more eyes on that part of the code. For instance, `gnus-cache-retrieve-headers' finds cached articles in the group, then uses (gnus-sorted-difference articles cached) to find uncached articles. The uncached articles are sent to `gnus-fetch-headers', and the cached articles are parsed from the cache file using: (gnus-get-newsgroup-headers-xover (gnus-sorted-difference cached uncached-articles)) Then the two sets of headers are merged and sorted. The agent does something similar. I don't think the parsing should do any duplicate work, though it's possible that adding the newly-received headers back into the cache file is not optimal: it appends the new headers at the end of the buffer, then re-sorts the whole buffer. Crud, I just realized that the agent will re-sort its cache file using `gnus-agent-check-overview-buffer', but gnus-cache doesn't do anything similar. So the write process will have to insert the new headers into the buffer in sorted order. Eric