bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi,

Since sometimes now, in Gnus summary, calling 'A T'
(gnus-summary-refer-thread) is really slow (30 seconds to 1 minute) and
it was instantaneous before.  I have bisected it to the following patch:

bf986c1faf53f3abd260f72cb36d9143afac353d is the first bad commit
commit bf986c1faf53f3abd260f72cb36d9143afac353d
Author: Andrew G Cohen <cohen@andy.bu.edu>
Date:   Tue Nov 22 15:39:01 2022 +0800
    Improve gnus thread-referral
    
    Allow thread referral to use search whenever possible, displaying the
    results in the current summary buffer if possible and a new nnselect
    buffer if not.
    
    * lisp/gnus/nnimap.el (nnimap-request-thread): Obsolete function.
    * lisp/gnus/gnus-search.el (gnus-search-thread): Allow detailed
    specification of how/where to search. Add found articles to the
    current summary buffer if possible, or create a new ephemeral nnselect
    group if not.
    * lisp/gnus/gnus-sum.el (gnus-refer-thread-use-search): Allow a list
    of servers and groups to search.
    (gnus-summary-refer-thread): Find thread-related articles by using a
    backend-specific method, gnus-search, or retrieving nearby headers in
    the current group.
    * lisp/gnus/nnselect.el (nnselect-search-thread): Obsolete function.
    (nnselect-request-thread): Allow thread referral from nnselect groups.
    * doc/misc/gnus.texi (Finding the Parent): Document changes to thread
    referral.


In GNU Emacs 30.0.50 (build 1, x86_64-unknown-openbsd7.3, cairo version
 1.17.8) of 2023-06-02 built on computer
Repository revision: bf986c1faf53f3abd260f72cb36d9143afac353d
Repository branch: HEAD
Windowing system distributor 'The X.Org Foundation', version 11.0.12101006
System Description: OpenBSD computer 7.3 GENERIC.MP#1125 amd64

Configured using:
 'configure --prefix=/home/manuel/emacs --bindir=/home/manuel/bin
 --with-x-toolkit=no --without-sound --without-compress-install
 CPPFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBOTF LIBXML2 MODULES NOTIFY KQUEUE OLDXMENU PDUMPER PNG RSVG
SQLITE3 THREADS TIFF TREE_SITTER WEBP X11 XDBE XIM XINPUT2 XPM ZLIB

Important settings:
  value of $LC_ALL: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  gnus-topic-mode: t
  display-time-mode: t
  display-battery-mode: t
  server-mode: t
  gnus-undo-mode: t
  shell-dirtrack-mode: t
  override-global-mode: t
  repeat-mode: t
  desktop-save-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/manuel/.emacs.d/elpa/ef-themes-1.0.2/theme-loaddefs hides /home/manuel/emacs/share/emacs/30.0.50/lisp/theme-loaddefs

Features:
(shadow shortdoc help-fns radix-tree magit-extras emacsbug pulse
face-remap magit-submodule magit-obsolete magit-blame magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log
which-func magit-diff smerge-mode diff git-commit log-edit add-log
magit-core magit-autorevert magit-margin magit-transient magit-process
with-editor magit-mode transient magit-git magit-section magit-utils
dash nnmaildir gnus-search sort gnus-cite mail-extr textsec uni-scripts
idna-mapping ucs-normalize uni-confusable textsec-check gnus-async
gnus-bcklg qp gnus-ml gnus-topic mm-archive url-cache utf-7 imap rfc2104
nndoc nndraft nnmh network-stream nnfolder nnml gnus-agent gnus-srvr
gnus-score score-mode nnvirtual nntp gnus-cache nnrss org-agenda
mule-util org-indent org-element org-persist org-id org-refile avl-tree
oc-basic ol-eww ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect
ol-docview ol-bibtex bibtex ol-bbdb ol-w3m ol-doi org-link-doi org ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-src ob-comint
org-pcomplete org-list org-footnote org-faces org-entities ob-emacs-lisp
ob-core ob-eval org-cycle org-table ol org-fold org-fold-core org-keys
oc org-loaddefs org-version org-compat org-macs css-mode sh-script smie
treesit executable vc-cvs vc-rcs log-view pcvs-util pascal eww xdg
url-queue mm-url vc-git bug-reference paredit imenu doc-view jka-compr
image-mode exif gnus-dired view vc-hg diff-mode vc-dispatcher vc-svn
warnings rng-xsd xsd-regexp rng-cmpct rng-nxml rng-valid rng-loc rng-uri
rng-parse nxml-parse rng-match rng-dt rng-util rng-pttrn nxml-ns
nxml-mode nxml-outln nxml-rap sgml-mode facemenu nxml-util nxml-enc
xmltok time battery cus-load exwm-randr xcb-randr exwm-config ido exwm
exwm-input xcb-keysyms xcb-xkb exwm-manage exwm-floating xcb-cursor
xcb-render exwm-layout exwm-workspace exwm-core xcb-ewmh xcb-icccm xcb
xcb-xproto xcb-types xcb-debug server modus-operandi-theme modus-themes
zone speed-type url-http url-auth url-gw nsm compat ytdious mingus
libmpdee reporter edebug debug backtrace detached-init detached
autorevert filenotify transmission color calc-bin calc-ext calc
calc-loaddefs rect calc-macs supercite regi ebdb-message ebdb-gnus
gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime gnutls dig
gnus-sum shr pixel-fill kinsoku url-file svg dom gnus-group gnus-undo
gnus-start gnus-dbus gnus-cloud nnimap nnmail mail-source utf7 nnoo
gnus-spec gnus-int gnus-range message sendmail yank-media puny rfc822
mml mml-sec epa epg rfc6068 epg-config mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 ietf-drums gmm-utils mailheader
gnus-win gnus nnheader gnus-util mail-utils range mm-util mail-prsvr
wid-edit ebdb-mua ebdb-com crm ebdb-format ebdb mailabbrev eieio-opt
speedbar ezimage dframe find-func eieio-base pcase timezone
visual-basic-mode cl web-mode derived disp-table erlang-start
smart-tabs-mode skeleton cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs slime-asdf grep slime-tramp
tramp rx tramp-loaddefs trampver tramp-integration files-x tramp-compat
shell pcomplete parse-time iso8601 time-date ls-lisp format-spec
slime-fancy slime-indentation slime-cl-indent cl-indent
slime-trace-dialog slime-fontifying-fu slime-package-fu slime-references
slime-compiler-notes-tree advice slime-scratch slime-presentations
bridge slime-macrostep macrostep slime-mdot-fu slime-enclosing-context
slime-fuzzy slime-fancy-trace slime-fancy-inspector slime-c-p-c
slime-editing-commands slime-autodoc slime-repl slime-parse slime
apropos compile text-property-search etags fileloop generator xref
project arc-mode archive-mode noutline outline icons pp comint ansi-osc
ansi-color ring hyperspec thingatpt slime-autoloads edmacro kmacro
use-package-bind-key bind-key appt diary-lib diary-loaddefs cal-menu
calendar cal-loaddefs dired-x dired-aux dired dired-loaddefs
notifications dbus xml cl-extra help-mode use-package-core repeat
easy-mmode desktop frameset speed-type-autoloads osm-autoloads
ebdb-autoloads magit-autoloads debbugs-autoloads git-commit-autoloads
finder-inf rust-mode-autoloads magit-section-autoloads
ef-themes-autoloads with-editor-autoloads compat-autoloads
dash-autoloads ytdious-autoloads transmission-autoloads
paredit-autoloads exwm-autoloads hyperbole-autoloads detached-autoloads
info package browse-url url url-proxy url-privacy url-expand url-methods
url-history url-cookie generate-lisp-file url-domsuf url-util mailcap
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
password-cache json subr-x map byte-opt gv bytecomp byte-compile
url-vars cl-loaddefs cl-lib rmc iso-transl tooltip cconv eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind kqueue lcms2
dynamic-setting system-font-setting font-render-setting cairo xinput2 x
multi-tty make-network-process emacs)

Memory information:
((conses 16 984879 633622)
 (symbols 48 62657 9)
 (strings 32 304460 91596)
 (string-bytes 1 9630727)
 (vectors 16 189334)
 (vector-slots 8 3198927 648189)
 (floats 8 710 1136)
 (intervals 56 14286 482)
 (buffers 984 91))

-- 
Manuel Giraud

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 72 bytes --]

Hi,

FWIW, the attached patch fixes the issue for me.
-- 
Manuel Giraud

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-gnus-summary-refer-thread-slowness.patch --]
[-- Type: text/x-patch, Size: 785 bytes --]

From 4242e4f5a828af94036c3d4f6e21726dbff4dc48 Mon Sep 17 00:00:00 2001
From: Manuel Giraud <manuel@ledu-giraud.fr>
Date: Fri, 2 Jun 2023 17:23:15 +0200
Subject: [PATCH] Fix 'gnus-summary-refer-thread' slowness

---
 lisp/gnus/gnus-sum.el | 1 -
 1 file changed, 1 deletion(-)

diff --git a/lisp/gnus/gnus-sum.el b/lisp/gnus/gnus-sum.el
index 4effaa981ec..725f5a1a5c9 100644
--- a/lisp/gnus/gnus-sum.el
+++ b/lisp/gnus/gnus-sum.el
@@ -9029,7 +9029,6 @@ gnus-summary-refer-thread
          (id (mail-header-id header))
          (gnus-inhibit-demon t)
          (gnus-summary-ignore-duplicates t)
-         (gnus-read-all-available-headers t)
          (gnus-refer-thread-use-search
           (if (or (null limit) (numberp limit))
               gnus-refer-thread-use-search
-- 
2.40.0

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Andrew Cohen]

Dear Manuel:

>>>>> "MG" == Manuel Giraud <manuel@ledu-giraud.fr> writes:

    MG> Hi, Since sometimes now, in Gnus summary, calling 'A T'
    MG> (gnus-summary-refer-thread) is really slow (30 seconds to 1
    MG> minute) and it was instantaneous before.

Thanks for the bug report, and thanks for finding what is causing the
slowdown. The variable 'gnus-read-all-available-headers causes gnus to
parse all the headers it has available. There are some cases in which
this is needed to adequately find the referring thread, but others where
it is unnecessary. We can almost certainly move this to restrict to
those cases where it is really necessary and hopefully fix the
problem. We just need to know a bit more about which cases are slow (for
example, for me with mostly imap groups, there is no slowness).

Can you tell me (for circumstances where the referral is slow):

1. What kind of group and the backend used  in the group from which you
did the referral (imap, nntp, etc)
2. What search engine are you using (imap, notmuch, etc)
3. When the thread referral happens which groups are getting searched?
(In particular, what is the value of
'gnus-refer-thread-use-search? And is the search on the full server or
is it restricted to a single group?). And can you tell if the same set of
groups is being searched before and after the slowdown appeared?

Thanks for helping finding and fixing this.

Best,
Andy




-- 
Andrew Cohen

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andrew Cohen <cohen@bu.edu> writes:

[...]

> Can you tell me (for circumstances where the referral is slow):

Hi Andrew,

> 1. What kind of group and the backend used  in the group from which you
> did the referral (imap, nntp, etc)

I'm retrieving my mail via IMAP into a local nnml backend.  This is in
such group that I witness the slowdown.  I have not tested on any other
backend.

> 2. What search engine are you using (imap, notmuch, etc)

I have setup notmuch onto the default nnml directory and set
'gnus-select-method' as follow:
--8<---------------cut here---------------start------------->8---
(setq gnus-select-method '(nnml "" (gnus-search-engine gnus-search-notmuch)))
--8<---------------cut here---------------end--------------->8---

> 3. When the thread referral happens which groups are getting searched?
> (In particular, what is the value of
> 'gnus-refer-thread-use-search? And is the search on the full server or
> is it restricted to a single group?). And can you tell if the same set of
> groups is being searched before and after the slowdown appeared?

I have let 'gnus-refer-thread-use-search' to its default NIL value.  My
search was restricted to a single (mail) group and, previously, I was
doing the same thread referral search on the same group.

Thanks and hope this helps.
-- 
Manuel Giraud

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Eli Zaretskii]

Ping!  Can we revive this and solve the issue, please?

> Cc: 63842@debbugs.gnu.org, cohen@andy.bu.edu
> Date: Sat, 03 Jun 2023 15:23:17 +0200
> From:  Manuel Giraud via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Andrew Cohen <cohen@bu.edu> writes:
> 
> [...]
> 
> > Can you tell me (for circumstances where the referral is slow):
> 
> Hi Andrew,
> 
> > 1. What kind of group and the backend used  in the group from which you
> > did the referral (imap, nntp, etc)
> 
> I'm retrieving my mail via IMAP into a local nnml backend.  This is in
> such group that I witness the slowdown.  I have not tested on any other
> backend.
> 
> > 2. What search engine are you using (imap, notmuch, etc)
> 
> I have setup notmuch onto the default nnml directory and set
> 'gnus-select-method' as follow:
> --8<---------------cut here---------------start------------->8---
> (setq gnus-select-method '(nnml "" (gnus-search-engine gnus-search-notmuch)))
> --8<---------------cut here---------------end--------------->8---
> 
> > 3. When the thread referral happens which groups are getting searched?
> > (In particular, what is the value of
> > 'gnus-refer-thread-use-search? And is the search on the full server or
> > is it restricted to a single group?). And can you tell if the same set of
> > groups is being searched before and after the slowdown appeared?
> 
> I have let 'gnus-refer-thread-use-search' to its default NIL value.  My
> search was restricted to a single (mail) group and, previously, I was
> doing the same thread referral search on the same group.
> 
> Thanks and hope this helps.
> -- 
> Manuel Giraud
> 
> 
> 
> 

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Ping!  Can we revive this and solve the issue, please?

I'm trying to investigate it.
-- 
Manuel Giraud

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 687 bytes --]

Hi,

So here is the crux of this issue.  When using
'gnus-summary-refer-thread' in a nnml group, Emacs ends up calling
'gnus-get-newsgroup-headers-xover' (via 'gnus-fetch-headers').  AFAIU in
this function when 'gnus-read-all-available-headers' is t, Emacs will
parse *all* of the " *nntp*" buffer content.  In my case, this buffer is
quite big (about 50k lines and 23MiB) hence the slowness.

BTW, I also have examples where 'gnus-summary-refer-thread' gives me
some false positives (eg., not the same thread but part of the subject
matching)

I don't know how to proceed except from unsetting
'gnus-read-all-available-headers' for nnml backends (see attached
patch).
-- 
Manuel Giraud

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Do-not-read-all-headers-for-nnml-groups.patch --]
[-- Type: text/x-patch, Size: 917 bytes --]

From 170cc2f96ebc606c25d85e345f9f9196fe6a7f33 Mon Sep 17 00:00:00 2001
From: Manuel Giraud <manuel@ledu-giraud.fr>
Date: Fri, 16 Jun 2023 12:20:32 +0200
Subject: [PATCH] Do not read all headers for nnml groups

---
 lisp/gnus/gnus-sum.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/gnus/gnus-sum.el b/lisp/gnus/gnus-sum.el
index 4effaa981ec..14da8735e88 100644
--- a/lisp/gnus/gnus-sum.el
+++ b/lisp/gnus/gnus-sum.el
@@ -9029,7 +9029,8 @@ gnus-summary-refer-thread
          (id (mail-header-id header))
          (gnus-inhibit-demon t)
          (gnus-summary-ignore-duplicates t)
-         (gnus-read-all-available-headers t)
+         (gnus-read-all-available-headers
+          (not (eq (car (gnus-find-method-for-group group)) 'nnml)))
          (gnus-refer-thread-use-search
           (if (or (null limit) (numberp limit))
               gnus-refer-thread-use-search
-- 
2.40.0

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Andrew Cohen]

Sorry, I have gotten busy with other things at the moment.

>>>>> "MG" == Manuel Giraud <manuel@ledu-giraud.fr> writes:

    MG> Hi, So here is the crux of this issue.  When using
    MG> 'gnus-summary-refer-thread' in a nnml group, Emacs ends up
    MG> calling 'gnus-get-newsgroup-headers-xover' (via
    MG> 'gnus-fetch-headers').  AFAIU in this function when
    MG> 'gnus-read-all-available-headers' is t, Emacs will parse *all*
    MG> of the " *nntp*" buffer content.  In my case, this buffer is
    MG> quite big (about 50k lines and 23MiB) hence the slowness.

Thanks for continuing to debug this. I am confused---why is the nntp
buffer so full? The search routine should populate the buffer only with
the headers of the articles found in the search (I am assuming that this
list of found articles is not 50K lines long).  Maybe the search is not
working properly? Can you step through gnus-summary-refer-thread and
in the conditional that retrieves the new headers can you tell me which
branch of the conditional is chosen (there are three possibilities:
'gnus-request-thread, 'gnus-search-thread, and the clause with the
comment "Otherwise just retrieve some headers").

    MG>BTW, I also have examples where 'gnus-summary-refer-thread' gives me
    MG>some false positives (eg., not the same thread but part of the subject
    MG>matching)

This is probably by design: in the olden days many mailers were broken
and didn't handle the references header properly (I don't know if this
is still the case). So by default gnus tries to use information from the
subject header to help gather loose threads, which can result in
articles not actually part of the thread being included. You can check
if this is the reason for what you are seeing by setting

(setq gnus-summary-thread-gathering-function
                      'gnus-gather-threads-by-references)

and seeing if this makes a difference.

Best,
Andy


-- 
Andrew Cohen

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andrew Cohen <cohen@bu.edu> writes:

> Sorry, I have gotten busy with other things at the moment.

Hi Andrew and you don't need to be sorry for this :-)

>>>>>> "MG" == Manuel Giraud <manuel@ledu-giraud.fr> writes:
>
>     MG> Hi, So here is the crux of this issue.  When using
>     MG> 'gnus-summary-refer-thread' in a nnml group, Emacs ends up
>     MG> calling 'gnus-get-newsgroup-headers-xover' (via
>     MG> 'gnus-fetch-headers').  AFAIU in this function when
>     MG> 'gnus-read-all-available-headers' is t, Emacs will parse *all*
>     MG> of the " *nntp*" buffer content.  In my case, this buffer is
>     MG> quite big (about 50k lines and 23MiB) hence the slowness.
>
> Thanks for continuing to debug this. I am confused---why is the nntp
> buffer so full?

I think in a nnml group the nntp buffer is populated with the content of
the ".overview" file.  In this particular group, I have thousands of
messages and I think that explains the size of this file.

> The search routine should populate the buffer only with the headers of
> the articles found in the search (I am assuming that this list of
> found articles is not 50K lines long).  Maybe the search is not
> working properly?

As we are talking about 'gnus-summary-refer-thread', I guess that it is
expected that the nntp buffer is filled with this content.  A regular
query (with 'G G') is still fast, so I think my search engine is set up
properly.

> Can you step through gnus-summary-refer-thread and in the conditional
> that retrieves the new headers can you tell me which branch of the
> conditional is chosen (there are three possibilities:
> 'gnus-request-thread, 'gnus-search-thread, and the clause with the
> comment "Otherwise just retrieve some headers").

In my case, Emacs is using the 'gnus-search-thread' branch and ends up
calling 'gnus-get-newsgroup-headers-xover' which is the function that
parses all the nntp buffer content.

>     MG>BTW, I also have examples where 'gnus-summary-refer-thread' gives me
>     MG>some false positives (eg., not the same thread but part of the subject
>     MG>matching)
>
> This is probably by design: in the olden days many mailers were broken
> and didn't handle the references header properly (I don't know if this
> is still the case). So by default gnus tries to use information from the
> subject header to help gather loose threads, which can result in
> articles not actually part of the thread being included. You can check
> if this is the reason for what you are seeing by setting
>
> (setq gnus-summary-thread-gathering-function
>                       'gnus-gather-threads-by-references)
>
> and seeing if this makes a difference.

Ok, thanks for the explanation and FWIW, my
'gnus-summary-thread-gathering-function' is already set to
'gnus-gather-threads-by-references.

Best regards,
-- 
Manuel Giraud

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Andrew Cohen]

OK, I think I understand the problem.

Before the change that Manuel identified as the culprit of the slowdown,
 thread referral resulted in the creation of a new ephemeral group to
 hold the search results. One of the major features of the changes was
 to try to add the articles from the search to the existing summary
 buffer rather than create a new one (this is an important improvement
 for a variety of reasons; for example, changes made in the additional
 ephemeral buffer would be overriden when exiting the originating
 buffer).

So what Manuel is seeing: previously the new ephemeral group parses only
the headers for the articles in the thread, while in the current
implementation where the newly found articles are simply added to the
existing summary buffer, all the headers in the original buffer are
parsed.

I need to figure out the right way to fix this. I think parsing the full
set of headers is the right thing, but since it is slow that creates a
problem. In the meantime, Manuel can you try setting
'gnus-refer-thread-use-search to t and see if this resolves your
problem? (This should effectively restore the old behavior of creating a
new ephemeral group to hold the search result).

Best,
Andy



>>>>> "MG" == Manuel Giraud <manuel@ledu-giraud.fr> writes:

    MG> Andrew Cohen <cohen@bu.edu> writes:
    >> Sorry, I have gotten busy with other things at the moment.

    MG> Hi Andrew and you don't need to be sorry for this :-)

    >>>>>>> "MG" == Manuel Giraud <manuel@ledu-giraud.fr> writes:
    >> 
    MG> Hi, So here is the crux of this issue.  When using
    MG> 'gnus-summary-refer-thread' in a nnml group, Emacs ends up
    MG> calling 'gnus-get-newsgroup-headers-xover' (via
    MG> 'gnus-fetch-headers').  AFAIU in this function when
    MG> 'gnus-read-all-available-headers' is t, Emacs will parse *all*
    MG> of the " *nntp*" buffer content.  In my case, this buffer is
    MG> quite big (about 50k lines and 23MiB) hence the slowness.
    >> 
    >> Thanks for continuing to debug this. I am confused---why is the
    >> nntp buffer so full?

    MG> I think in a nnml group the nntp buffer is populated with the
    MG> content of the ".overview" file.  In this particular group, I
    MG> have thousands of messages and I think that explains the size of
    MG> this file.

    >> The search routine should populate the buffer only with the
    >> headers of the articles found in the search (I am assuming that
    >> this list of found articles is not 50K lines long).  Maybe the
    >> search is not working properly?

    MG> As we are talking about 'gnus-summary-refer-thread', I guess
    MG> that it is expected that the nntp buffer is filled with this
    MG> content.  A regular query (with 'G G') is still fast, so I think
    MG> my search engine is set up properly.

    >> Can you step through gnus-summary-refer-thread and in the
    >> conditional that retrieves the new headers can you tell me which
    >> branch of the conditional is chosen (there are three
    >> possibilities: 'gnus-request-thread, 'gnus-search-thread, and the
    >> clause with the comment "Otherwise just retrieve some headers").

    MG> In my case, Emacs is using the 'gnus-search-thread' branch and
    MG> ends up calling 'gnus-get-newsgroup-headers-xover' which is the
    MG> function that parses all the nntp buffer content.

    MG> BTW, I also have examples where 'gnus-summary-refer-thread'
    MG> gives me some false positives (eg., not the same thread but part
    MG> of the subject matching)
    >> 
    >> This is probably by design: in the olden days many mailers were
    >> broken and didn't handle the references header properly (I don't
    >> know if this is still the case). So by default gnus tries to use
    >> information from the subject header to help gather loose threads,
    >> which can result in articles not actually part of the thread
    >> being included. You can check if this is the reason for what you
    >> are seeing by setting
    >> 
    >> (setq gnus-summary-thread-gathering-function
    >> 'gnus-gather-threads-by-references)
    >> 
    >> and seeing if this makes a difference.

    MG> Ok, thanks for the explanation and FWIW, my
    MG> 'gnus-summary-thread-gathering-function' is already set to
    MG> 'gnus-gather-threads-by-references.

    MG> Best regards, -- Manuel Giraud

-- 
Andrew Cohen
Director, HKUST Jockey Club Institute for Advanced Study
Lam Woo Foundation Professor and Chair Professor of Physics
The Hong Kong University of Science and Technology

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andrew Cohen <cohen@bu.edu> writes:

> OK, I think I understand the problem.
>
> Before the change that Manuel identified as the culprit of the slowdown,
>  thread referral resulted in the creation of a new ephemeral group to
>  hold the search results. One of the major features of the changes was
>  to try to add the articles from the search to the existing summary
>  buffer rather than create a new one (this is an important improvement
>  for a variety of reasons; for example, changes made in the additional
>  ephemeral buffer would be overriden when exiting the originating
>  buffer).
>
> So what Manuel is seeing: previously the new ephemeral group parses only
> the headers for the articles in the thread, while in the current
> implementation where the newly found articles are simply added to the
> existing summary buffer, all the headers in the original buffer are
> parsed.
>
> I need to figure out the right way to fix this. I think parsing the full
> set of headers is the right thing, but since it is slow that creates a
> problem. In the meantime, Manuel can you try setting
> 'gnus-refer-thread-use-search to t and see if this resolves your
> problem? (This should effectively restore the old behavior of creating a
> new ephemeral group to hold the search result).

So I have tested with 'gnus-refer-thread-use-search' set to t and it
works (and is fast) as it was before.  But, as you said, this creates a
new nnselect group.

From my tiny testing set, it does not seems to me that parsing all the
headers is the way to go: the call to 'gnus-search-run-query' in
gnus-search.el line 2206 directly returns direcly the correct set of
messages and the call to 'gnus-get-newsgroup-headers-xover' later looks
like some "deep" search (eg. on subject content).

Best regards,
-- 
Manuel Giraud

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Andrew Cohen]

>>>>> "MG" == Manuel Giraud <manuel@ledu-giraud.fr> writes:

    MG> Andrew Cohen <cohen@bu.edu> writes:
    >> OK, I think I understand the problem.

[...]

    MG> From my tiny testing set, it does not seems to me that parsing
    MG> all the headers is the way to go: the call to
    MG> 'gnus-search-run-query' in gnus-search.el line 2206 directly
    MG> returns direcly the correct set of messages and the call to
    MG> 'gnus-get-newsgroup-headers-xover' later looks like some "deep"
    MG> search (eg. on subject content).

OK, I understand it now.  This isn't really about searching, or subject
content (the fact that Manuel sees some articles not in the thread but
with similar subject remains a bit of a mystery). To get everything
right, any articles in the thread that need to be added to the summary
buffer have to be added to the dependencies hash. In the case of
searching we know exactly which articles need to be added so we have no
need for 'gnus-read-all-available-headers to be non-nil: the "found"
articles are each added to the hash. The only case where
'gnus-read-all-available-headers needs to be non-nil is when we don't
know exactly which articles are part of the thread in which case we have
to parse a larger set. This is what happens in the 3rd case in the
conditional (the "t" clause) in 'gnus-summary-refer-thread. That is,
this variable is only relevant in those cases where we don't have a
configured search engine and just retrieve a lot of headers and hope for
the best. So the setting of 'gnus-read-all-available-headers is just in
the wrong place. 

Thanks to Manuel for figuring out the error. I'll push the fix shortly.

Best,
Andy

-- 
Andrew Cohen

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andrew Cohen <cohen@bu.edu> writes:

>>>>>> "MG" == Manuel Giraud <manuel@ledu-giraud.fr> writes:
>
>     MG> Andrew Cohen <cohen@bu.edu> writes:
>     >> OK, I think I understand the problem.
>
> [...]
>
>     MG> From my tiny testing set, it does not seems to me that parsing
>     MG> all the headers is the way to go: the call to
>     MG> 'gnus-search-run-query' in gnus-search.el line 2206 directly
>     MG> returns direcly the correct set of messages and the call to
>     MG> 'gnus-get-newsgroup-headers-xover' later looks like some "deep"
>     MG> search (eg. on subject content).
>
> OK, I understand it now.  This isn't really about searching, or subject
> content (the fact that Manuel sees some articles not in the thread but
> with similar subject remains a bit of a mystery).

It is a mistery to me too.  If 'gnus-get-newsgroup-headers-xover' job is
to gather messages from the same thread, I think there is something
wrong here.  I'm still searching.

> To get everything right, any articles in the thread that need to be
> added to the summary buffer have to be added to the dependencies
> hash. In the case of searching we know exactly which articles need to
> be added so we have no need for 'gnus-read-all-available-headers to be
> non-nil: the "found" articles are each added to the hash. The only
> case where 'gnus-read-all-available-headers needs to be non-nil is
> when we don't know exactly which articles are part of the thread in
> which case we have to parse a larger set. This is what happens in the
> 3rd case in the conditional (the "t" clause) in
> 'gnus-summary-refer-thread. That is, this variable is only relevant in
> those cases where we don't have a configured search engine and just
> retrieve a lot of headers and hope for the best. So the setting of
> 'gnus-read-all-available-headers is just in the wrong place.

Thanks.  I don't know for other backends/search engines but a binding of
'gnus-read-all-available-headers' to t elsewhere will sure fix my issue.
-- 
Manuel Giraud

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Manuel Giraud <manuel@ledu-giraud.fr> writes:

[...]

>> To get everything right, any articles in the thread that need to be
>> added to the summary buffer have to be added to the dependencies
>> hash. In the case of searching we know exactly which articles need to
>> be added so we have no need for 'gnus-read-all-available-headers to be
>> non-nil: the "found" articles are each added to the hash. The only
>> case where 'gnus-read-all-available-headers needs to be non-nil is
>> when we don't know exactly which articles are part of the thread in
>> which case we have to parse a larger set. This is what happens in the
>> 3rd case in the conditional (the "t" clause) in
>> 'gnus-summary-refer-thread. That is, this variable is only relevant in
>> those cases where we don't have a configured search engine and just
>> retrieve a lot of headers and hope for the best. So the setting of
>> 'gnus-read-all-available-headers is just in the wrong place.
>
> Thanks.  I don't know for other backends/search engines but a binding of
> 'gnus-read-all-available-headers' to t elsewhere will sure fix my
> issue.

Hi Andrew,

I have seen your patch and it works as expected.
'gnus-summary-refer-thread' is as fast as before for me.  Thanks.
-- 
Manuel Giraud

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Eli Zaretskii]

> From: Manuel Giraud <manuel@ledu-giraud.fr>
> Cc: Eli Zaretskii <eliz@gnu.org>,  63842@debbugs.gnu.org,  cohen@andy.bu.edu
> Date: Fri, 23 Jun 2023 12:00:45 +0200
> 
> Manuel Giraud <manuel@ledu-giraud.fr> writes:
> 
> [...]
> 
> >> To get everything right, any articles in the thread that need to be
> >> added to the summary buffer have to be added to the dependencies
> >> hash. In the case of searching we know exactly which articles need to
> >> be added so we have no need for 'gnus-read-all-available-headers to be
> >> non-nil: the "found" articles are each added to the hash. The only
> >> case where 'gnus-read-all-available-headers needs to be non-nil is
> >> when we don't know exactly which articles are part of the thread in
> >> which case we have to parse a larger set. This is what happens in the
> >> 3rd case in the conditional (the "t" clause) in
> >> 'gnus-summary-refer-thread. That is, this variable is only relevant in
> >> those cases where we don't have a configured search engine and just
> >> retrieve a lot of headers and hope for the best. So the setting of
> >> 'gnus-read-all-available-headers is just in the wrong place.
> >
> > Thanks.  I don't know for other backends/search engines but a binding of
> > 'gnus-read-all-available-headers' to t elsewhere will sure fix my
> > issue.
> 
> Hi Andrew,
> 
> I have seen your patch and it works as expected.
> 'gnus-summary-refer-thread' is as fast as before for me.  Thanks.

Thanks.  can the fix be installed and the bug closed then, please?

bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Eli Zaretskii]

> From: Andrew Cohen <cohen@bu.edu>
> Date: Sat, 01 Jul 2023 16:34:39 +0800
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
>     >> From: Manuel Giraud <manuel@ledu-giraud.fr> Cc: Eli Zaretskii
>     >> <eliz@gnu.org>, 63842@debbugs.gnu.org, cohen@andy.bu.edu Date:
>     >> Fri, 23 Jun 2023 12:00:45 +0200
>     >> 
>     >> Manuel Giraud <manuel@ledu-giraud.fr> writes:
>     >> 
>     >> [...]
>     >> 
>     >> >> To get everything right, any articles in the thread that need
>     >> to be >> added to the summary buffer have to be added to the
>     >> dependencies >> hash. In the case of searching we know exactly
>     >> which articles need to >> be added so we have no need for
>     >> 'gnus-read-all-available-headers to be >> non-nil: the "found"
>     >> articles are each added to the hash. The only >> case where
>     >> 'gnus-read-all-available-headers needs to be non-nil is >> when
>     >> we don't know exactly which articles are part of the thread in >>
>     >> which case we have to parse a larger set. This is what happens in
>     >> the >> 3rd case in the conditional (the "t" clause) in >>
>     >> 'gnus-summary-refer-thread. That is, this variable is only
>     >> relevant in >> those cases where we don't have a configured
>     >> search engine and just >> retrieve a lot of headers and hope for
>     >> the best. So the setting of >> 'gnus-read-all-available-headers
>     >> is just in the wrong place.
>     >> >
>     >> > Thanks.  I don't know for other backends/search engines but a
>     >> binding of > 'gnus-read-all-available-headers' to t elsewhere
>     >> will sure fix my > issue.
>     >> 
>     >> Hi Andrew,
>     >> 
>     >> I have seen your patch and it works as expected.
>     >> 'gnus-summary-refer-thread' is as fast as before for me.  Thanks.
> 
>     EZ> Thanks.  can the fix be installed and the bug closed then,
>     EZ> please?
> 
> Already installed (commit 1e13610b75718e7904f8af181fb73571639e1211). Bug
> can be closed.

Thanks, done.