bug#68249: 30.0.50; Rmail summaries much slower

bug#68249: 30.0.50; Rmail summaries much slower

[Richard Stallman]

In master from Jan 1, making Rmail summaries has become far
slower than it was on July 2.  I am calling rmail-summary-by-topic
and have around 7000 messages in the RMAIL file.  It takes many secnds
to finish, now.

Can anyone suggest a way to investigate the cause of the slowdown?


In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 2.24.32, cairo version 1.16.0) of 2024-01-02 built on freetop
Repository revision: 1c369263db826a02e01d4209c7b6b8617971c37c
Repository branch: master
System Description: Trisquel GNU/Linux Nabia (10.0)

Configured using:
 'configure 'CFLAGS=-O0 -g' --with-gnutls=ifavailable'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GPM GSETTINGS HARFBUZZ JPEG JSON LIBOTF
LIBSELINUX LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XINPUT2
XPM GTK2 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Help

Minor modes in effect:
  gpm-mouse-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  isearch-fold-quotes-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: linux
  auto-encryption-mode: t
  auto-compression-mode: t
  abbrev-mode: t

Load-path shadows:
None found.

Features:
(shadow emacsbug shortdoc comp-common rmail-spam-filter rmailedit
rmailsort undigest kmacro cl-extra parse-time iso8601 vc-cvs vc-rcs
log-view easy-mmode pcvs-util mhtml-mode css-mode eww xdg url-queue
mm-url gnus nnheader range wid-edit color js c-ts-common cc-mode
cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs sgml-mode facemenu format-spec battery dbus diff
help-fns radix-tree mule-util display-line-numbers grep compile etags
fileloop generator xref quail epa-mail sh-script rx smie treesit
executable shell pcomplete comint ansi-osc ring tabify imenu man
ansi-color rect thingatpt help-mode novice project misearch
multi-isearch ispell textsec uni-scripts idna-mapping ucs-normalize
uni-confusable textsec-check shr pixel-fill kinsoku url-file svg xml
dom dabbrev mailalias rmailout rmailkwd qp rmailmm message sendmail
yank-media puny rfc822 mml mml-sec epa epg epg-config gnus-util
text-property-search time-date mm-decode mm-bodies mm-encode
mailabbrev gmm-utils mailheader mail-parse rfc2231 files-x dired-aux
dired dired-loaddefs t-mouse term/linux view derived disp-table advice
rmailsum rmail rfc6068 rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils finder-inf osm-autoloads info compat-autoloads package
browse-url url url-proxy url-privacy url-expand url-methods
url-history url-cookie generate-lisp-file url-domsuf url-util mailcap
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
password-cache json subr-x map byte-opt gv bytecomp byte-compile
url-vars cl-loaddefs cl-lib rmc iso-transl tooltip cconv eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode
mwheel term/x-win x-win term/common-win x-dnd touch-screen tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice
seq simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese
eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese composite emoji-zwj charscript charprop
case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable
backquote threads dbusbind inotify dynamic-setting system-font-setting
font-render-setting cairo gtk x-toolkit xinput2 x multi-tty
move-toolbar make-network-process emacs)

Memory information:
((conses 16 533533 109378) (symbols 48 18701 4)
 (strings 32 94791 7168) (string-bytes 1 3149144) (vectors 16 70370)
 (vector-slots 8 1635953 197499) (floats 8 246 373)
 (intervals 56 43446 492) (buffers 976 64))
[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#68249: 30.0.50; Rmail summaries much slower

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Thu, 04 Jan 2024 23:23:37 -0500
> 
> 
> In master from Jan 1, making Rmail summaries has become far
> slower than it was on July 2.  I am calling rmail-summary-by-topic
> and have around 7000 messages in the RMAIL file.  It takes many secnds
> to finish, now.
> 
> Can anyone suggest a way to investigate the cause of the slowdown?

Before invoking rmail-summary-by-topic, type

  M-x profiler-start RET RET

Then, after rmail-summary-by-topic finishes, type

  M-x profiler-report RET

In the buffer presented by the last command, type "C-u RET" on every
line that has a "+" before it, to fully expand the profile.  Then look
at the profile to find the hot spots.

The profile is more detailed if you load rmailsum.el, not
rmailsum.elc.

bug#68249: 30.0.50; Rmail summaries much slower

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Thanks.  Here's what I got:

        1573  54% + command-execute
        1098  38%   Automatic GC
         159   5% + timer-event-handler
          25   0% + ...
          19   0%   redisplay_internal (C function)
           8   0% + rmail-summary-rmail-update

I am not sure what that implies, since nothing in a loop
to check messages appears in it.

Should I try this in the previous version for comparison?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#68249: 30.0.50; Rmail summaries much slower

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: 68249@debbugs.gnu.org
> Date: Sat, 06 Jan 2024 23:31:45 -0500
> 
>         1573  54% + command-execute
>         1098  38%   Automatic GC
>          159   5% + timer-event-handler
>           25   0% + ...
>           19   0%   redisplay_internal (C function)
>            8   0% + rmail-summary-rmail-update
> 
> I am not sure what that implies, since nothing in a loop
> to check messages appears in it.

You didn't expand the profile.  All those lines which have a "+"
before the function's name are "folded", and you can "unfold" them by
going to the line and pressing RET.  "C-u RET" will unfold _all_ such
folded lines, and you will see the detailed profile of
command-execute, which I think where the problem happens, and where
you will find the loop that checks messages (or a function that runs
such a loop).

> Should I try this in the previous version for comparison?

That could help, yes.  But before that, I think a fully expanded
("unfolded") profile of the current version is needed.

It is also strange that GC takes such a large proportion of time.  It
is possible that the slowdown is because some subroutine makes much
more garbage than the previous implementation did.

bug#68249: 30.0.50; Rmail summaries much slower

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > From: Richard Stallman <rms@gnu.org>
  > > Cc: 68249@debbugs.gnu.org
  > > Date: Sat, 06 Jan 2024 23:31:45 -0500
  > > 
  > >         1573  54% + command-execute
  > >         1098  38%   Automatic GC
  > >          159   5% + timer-event-handler
  > >           25   0% + ...
  > >           19   0%   redisplay_internal (C function)
  > >            8   0% + rmail-summary-rmail-update
  > > 
  > > I am not sure what that implies, since nothing in a loop
  > > to check messages appears in it.

  > You didn't expand the profile.  All those lines which have a "+"
  > before the function's name are "folded", and you can "unfold" them by
  > going to the line and pressing RET.  "C-u RET" will unfold _all_ such
  > folded lines, and you will see the detailed profile of
  > command-execute, which I think where the problem happens, and where
  > you will find the loop that checks messages (or a function that runs

Thank you.  I just saw this message today.
I found the cause of the problem a few days later, and fixed it.

I will note this down for later.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#68249: 30.0.50; Rmail summaries much slower

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

> I found the cause of the problem a few days later, and fixed it.

I'm therefore closing this bug report.  Let me know if that's wrong, and
I'll reopen it.  Thanks.

bug#68249: 30.0.50; Rmail summaries much slower

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: 68249@debbugs.gnu.org
> Date: Tue, 23 Jan 2024 23:16:13 -0500
> 
> I found the cause of the problem a few days later, and fixed it.

You mean, you found a problem that was not a bug in Emacs?  Or was the
problem in Emacs, but it was fixed by your change from 3 weeks ago in
rmail-epa-decrypt-1?

bug#68249: 30.0.50; Rmail summaries much slower

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You mean, you found a problem that was not a bug in Emacs?  Or was the
  > problem in Emacs, but it was fixed by your change from 3 weeks ago in
  > rmail-epa-decrypt-1?

I was using that patch privately, but forgot to copy it into the newer
sources.  After I copied it in, the slowness went away.
I decided it was time to install that patch.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)