From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Eric Abrahamsen Newsgroups: gmane.emacs.bugs Subject: bug#38011: 27.0.50; [PATCH] WIP on allowing Gnus backends to return header data directly Date: Fri, 01 Nov 2019 13:52:23 -0700 Message-ID: <87wocjjp4o.fsf@ericabrahamsen.net> References: <87y2x0a9bl.fsf@ericabrahamsen.net> <87a79f3cu3.fsf@gnus.org> <8736f7l9rd.fsf@ericabrahamsen.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="267236"; mail-complaints-to="usenet@blaine.gmane.org" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) Cc: Andrew Cohen , 38011@debbugs.gnu.org To: Lars Ingebrigtsen Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Fri Nov 01 21:53:16 2019 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1iQdv0-0017Me-Jm for geb-bug-gnu-emacs@m.gmane.org; Fri, 01 Nov 2019 21:53:14 +0100 Original-Received: from localhost ([::1]:42856 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iQduz-0007ea-Gv for geb-bug-gnu-emacs@m.gmane.org; Fri, 01 Nov 2019 16:53:13 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:49837) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iQdup-0007aJ-GK for bug-gnu-emacs@gnu.org; Fri, 01 Nov 2019 16:53:05 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iQdun-0007il-UN for bug-gnu-emacs@gnu.org; Fri, 01 Nov 2019 16:53:03 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:48518) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iQdun-0007i7-OM for bug-gnu-emacs@gnu.org; Fri, 01 Nov 2019 16:53:01 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1iQdun-0008FB-L4 for bug-gnu-emacs@gnu.org; Fri, 01 Nov 2019 16:53:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Eric Abrahamsen Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Fri, 01 Nov 2019 20:53:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 38011 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 38011-submit@debbugs.gnu.org id=B38011.157264155531645 (code B ref 38011); Fri, 01 Nov 2019 20:53:01 +0000 Original-Received: (at 38011) by debbugs.gnu.org; 1 Nov 2019 20:52:35 +0000 Original-Received: from localhost ([127.0.0.1]:57339 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iQduM-0008EK-IO for submit@debbugs.gnu.org; Fri, 01 Nov 2019 16:52:35 -0400 Original-Received: from ericabrahamsen.net ([52.70.2.18]:44928 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iQduJ-0008E0-Ss for 38011@debbugs.gnu.org; Fri, 01 Nov 2019 16:52:32 -0400 Original-Received: from localhost (c-73-254-86-141.hsd1.wa.comcast.net [73.254.86.141]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id EEE61FA028; Fri, 1 Nov 2019 20:52:24 +0000 (UTC) In-Reply-To: <8736f7l9rd.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Fri, 01 Nov 2019 11:41:26 -0700") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:170682 Archived-At: --=-=-= Content-Type: text/plain Eric Abrahamsen writes: > Lars Ingebrigtsen writes: > >> Eric Abrahamsen writes: >> >>> This patch provides the possibility for backends to return their own >>> headers (ie a list of vectors), though it doesn't actually change any of >>> the backends to do that -- that will be another patch. >> >> Great! >> >>> I have one question at this stage: the 'nov or 'headers value gets >>> stored into the `gnus-headers-retrieved-by' variable. That variable is >>> later checked in a couple of places like so: Okay, here's the patch as it stands -- I'll do some testing first. Andy, I'm copying you in because this touches nnir.el, and nnselect.el will need to be edited accordingly. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-WIP-on-allowing-Gnus-backends-to-return-headers-dire.patch >From b184e82d0551fe52a9dcf025b355da7faafc68b2 Mon Sep 17 00:00:00 2001 From: Eric Abrahamsen Date: Thu, 31 Oct 2019 14:14:44 -0700 Subject: [PATCH] WIP on allowing Gnus backends to return headers directly * lisp/gnus/gnus-sum.el (gnus-fetch-headers): Allow the gnus-retrieve-headers backend function to directly return a list of headers, instead of inserting data to parse in the nntp-server-buffer. * lisp/gnus/nnir.el (nnir-retrieve-headers): Handle the same case when nnir calls the "real" backend function. --- lisp/gnus/gnus-sum.el | 55 +++++++++++++++++++++++++++++-------------- lisp/gnus/gnus.el | 9 ++++++- lisp/gnus/nnir.el | 35 ++++++++++++++++----------- 3 files changed, 66 insertions(+), 33 deletions(-) diff --git a/lisp/gnus/gnus-sum.el b/lisp/gnus/gnus-sum.el index f21bc7584e..142c50cac0 100644 --- a/lisp/gnus/gnus-sum.el +++ b/lisp/gnus/gnus-sum.el @@ -5629,25 +5629,44 @@ gnus-summary-remove-list-identifiers (setf (mail-header-subject header) subject)))))) (defun gnus-fetch-headers (articles &optional limit force-new dependencies) - "Fetch headers of ARTICLES." + "Fetch headers of ARTICLES. +This calls the `gnus-retrieve-headers' function of the current +group's backend server. The server can do one of two things: + +1. Write the headers for ARTICLES into the + `nntp-server-buffer' (the current buffer) in a parseable format, or +2. Return the headers directly as a list of vectors. + +In the first case, `gnus-retrieve-headers' returns a symbol +value, either `nov' or `headers'. This value determines which +parsing function is used to read the headers. It is also stored +into the variable `gnus-headers-retrieved-by', which is consulted +later when possibly building full threads." (gnus-message 7 "Fetching headers for %s..." gnus-newsgroup-name) - (prog1 - (if (eq 'nov - (setq gnus-headers-retrieved-by - (gnus-retrieve-headers - articles gnus-newsgroup-name - (or limit - ;; We might want to fetch old headers, but - ;; not if there is only 1 article. - (and (or (and - (not (eq gnus-fetch-old-headers 'some)) - (not (numberp gnus-fetch-old-headers))) - (> (length articles) 1)) - gnus-fetch-old-headers))))) - (gnus-get-newsgroup-headers-xover - articles force-new dependencies gnus-newsgroup-name t) - (gnus-get-newsgroup-headers dependencies force-new)) - (gnus-message 7 "Fetching headers for %s...done" gnus-newsgroup-name))) + (let ((res (setq gnus-headers-retrieved-by + (gnus-retrieve-headers + articles gnus-newsgroup-name + (or limit + ;; We might want to fetch old headers, but + ;; not if there is only 1 article. + (and (or (and + (not (eq gnus-fetch-old-headers 'some)) + (not (numberp gnus-fetch-old-headers))) + (> (length articles) 1)) + gnus-fetch-old-headers)))))) + (prog1 + (pcase res + ('nov + (gnus-get-newsgroup-headers-xover + articles force-new dependencies gnus-newsgroup-name t)) + ;; For now, assume that any backend returning its own + ;; headers takes some effort to do so, so return `headers'. + ((pred listp) + (setq gnus-headers-retrieved-by 'headers) + res) + ;; 'headers is the other likely value. + (_ (gnus-get-newsgroup-headers dependencies force-new))) + (gnus-message 7 "Fetching headers for %s...done" gnus-newsgroup-name)))) (defun gnus-select-newsgroup (group &optional read-all select-articles) "Select newsgroup GROUP. diff --git a/lisp/gnus/gnus.el b/lisp/gnus/gnus.el index 0673ac15f6..a5b2891477 100644 --- a/lisp/gnus/gnus.el +++ b/lisp/gnus/gnus.el @@ -2387,7 +2387,14 @@ gnus-article-unpropagated-mark-lists such as a mark that says whether an article is stored in the cache \(which doesn't make sense in a standalone back end).") -(defvar gnus-headers-retrieved-by nil) +(defvar gnus-headers-retrieved-by nil + "Holds the return value of `gnus-retrieve-headers'. +This is either the symbol `nov' or the symbol `headers'. This +value is checked during the summary creation process, when +building threads. A value of `nov' indicates that header +retrieval is relatively cheap and threading is encouraged to +include more old articles. A value of `headers' indciates that +retrieval is expensive and should be minimized.") (defvar gnus-article-reply nil) (defvar gnus-override-method nil) (defvar gnus-opened-servers nil) diff --git a/lisp/gnus/nnir.el b/lisp/gnus/nnir.el index 7cb2d1615a..48af6067f0 100644 --- a/lisp/gnus/nnir.el +++ b/lisp/gnus/nnir.el @@ -719,22 +719,29 @@ nnir-retrieve-headers (setq parsefunc 'nnheader-parse-nov)) ('headers (setq parsefunc 'nnheader-parse-head)) + ((pred listp) + (setq headers gnus-headers-retrieved-by)) (_ (error "Unknown header type %s while requesting articles \ of group %s" gnus-headers-retrieved-by artgroup))) - (goto-char (point-min)) - (while (not (eobp)) - (let* ((novitem (funcall parsefunc)) - (artno (and novitem - (mail-header-number novitem))) - (art (car (rassq artno articleids)))) - (when art - (setf (mail-header-number novitem) art) - (push novitem headers)) - (forward-line 1))))) - (setq headers - (sort headers - (lambda (x y) - (< (mail-header-number x) (mail-header-number y))))) + (unless headers + (goto-char (point-min)) + (while (not (eobp)) + (push (funcall parsefunc) headers) + (forward-line 1))) + (setq headers + (sort + (delq nil + (mapcar + (lambda (novitem) + (let* ((artno (and novitem + (mail-header-number novitem))) + (art (car-safe (rassq artno articleids)))) + (when art + (setf (mail-header-number novitem) art) + novitem))) + headers)) + (lambda (x y) + (< (mail-header-number x) (mail-header-number y))))))) (erase-buffer) (mapc 'nnheader-insert-nov headers) 'nov))) -- 2.23.0 --=-=-=--