bug#73473: 31.0.50; Minibuffer completions include nonsense prefix candidates

bug#73473: 31.0.50; Minibuffer completions include nonsense prefix candidates

[Jordan Ellis Coppard via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hello,


I've noticed that my completions have slowed down A LOT on recent Emacs 
builds, it turns out nonsense completion candidates (which I am calling 
"prefix candidates") are being included as... completion candidates. By 
my rough estimate this results in about... 10,000 extra completion 
candidates when invoking `C-h o` (with my personal Emacs configuration).

To reproduce, using `emacs -Q` of course:

1. Open Emacs.
2. C-h o foo TAB

Observe that completion candidate `footnote-` is listed and that there 
are 80 completion candidates.

If I do the same on the currently released Emacs or another (older) 
build of Emacs (30.0.50) there are 79 (not 80) completion candidates and 
`footnote-` is not listed as one.

This seems to be occurring for almost every unique prefix possible, 
compounding as time goes on. So later on `f-` is listed as a completion 
candidate, then also `go-` and so on. These are not valid completion 
candidates. `footnote-` is not a symbol. Over time the huge increase in 
completion candidate volume (and perhaps what is causing this behind the 
scenes) results in such a slowdown that I can visibly see Emacs crawl to 
1 FPS when I type in minibuffer completion, when the exact same init.el 
on said older 30.0.50 is faster.

This is reproduceable on `emacs -Q` as above, (and the useful bug report 
probably ends right here) but as an aside: my configuration is minimal, 
my Emacs starts up in about 0.3s and I keep things minimal because I do 
not want Emacs to ever take longer than instantaneously to respond to 
keystrokes (if it has to do some long-running job, or something which 
takes time that's different since feedback that such a thing has kicked 
off would/should still be instant).

Tangentially to this, it would be nice if completions could be decoupled 
from the UI (i.e. they could stream in, with an API to let the user know 
when the list is complete) because even on 30.0.50 I can often times 
type faster than completion can handle which drives me insane (UI 
hitching)... but that's an aside.


--- diagnostic info ---


In GNU Emacs 31.0.50 (build 1, aarch64-apple-darwin23.5.0, NS
  appkit-2487.60 Version 14.5 (Build 23F79)) of 2024-09-16 built on
  yote.local
Windowing system distributor 'Apple', version 10.3.2487
System Description:  macOS 14.5

Configured using:
  'configure --prefix=/opt/local --disable-silent-rules --without-dbus
  --without-gconf --without-libotf --without-m17n-flt --with-libgmp
  --with-gnutls --with-xml2 --with-modules --with-sqlite3 --with-webp
  --infodir /opt/local/share/info/emacs --with-native-compilation=no
  --with-ns --with-lcms2 --without-harfbuzz --without-imagemagick
  --without-xaw3d --with-rsvg --with-xwidgets
  --with-native-compilation=aot --with-tree-sitter 'CFLAGS=-pipe -Os
  -Wno-attributes
  -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -arch
  arm64' 'CPPFLAGS=-I/opt/local/include
  -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk'
  'LDFLAGS=-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-no_pie
  -Wl,-rpath /opt/local/lib/gcc14 -Wl,-rpath /opt/local/lib
  -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk
  -arch arm64''

Configured features:
ACL GIF GLIB GMP GNUTLS JPEG LCMS2 LIBXML2 MODULES NATIVE_COMP NOTIFY
KQUEUE NS PDUMPER PNG RSVG SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS
TREE_SITTER WEBP XIM XWIDGETS ZLIB

Important settings:
   value of $LC_ALL: en_US.UTF-8
   value of $LC_CTYPE: en_US.UTF-8
   value of $LANG: en_AU.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   show-paren-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   minibuffer-regexp-mode: t
   line-number-mode: t
   indent-tabs-mode: t
   transient-mark-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/ns-win ns-win ucs-normalize mule-util term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads xwidget-internal kqueue cocoa
ns lcms2 multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 48976 9047) (symbols 48 5284 0) (strings 32 13815 1840)
  (string-bytes 1 469136) (vectors 16 9182)
  (vector-slots 8 130405 8531) (floats 8 22 13) (intervals 56 279 5)
  (buffers 992 10))

bug#73473: 31.0.50; Minibuffer completions include nonsense prefix candidates

[Eli Zaretskii]

merge 73473 72787
thanks

> Date: Wed, 25 Sep 2024 19:54:59 +0900
> From:  Jordan Ellis Coppard via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> I've noticed that my completions have slowed down A LOT on recent Emacs 
> builds, it turns out nonsense completion candidates (which I am calling 
> "prefix candidates") are being included as... completion candidates. By 
> my rough estimate this results in about... 10,000 extra completion 
> candidates when invoking `C-h o` (with my personal Emacs configuration).
> 
> To reproduce, using `emacs -Q` of course:
> 
> 1. Open Emacs.
> 2. C-h o foo TAB
> 
> Observe that completion candidate `footnote-` is listed and that there 
> are 80 completion candidates.
> 
> If I do the same on the currently released Emacs or another (older) 
> build of Emacs (30.0.50) there are 79 (not 80) completion candidates and 
> `footnote-` is not listed as one.
> 
> This seems to be occurring for almost every unique prefix possible, 
> compounding as time goes on. So later on `f-` is listed as a completion 
> candidate, then also `go-` and so on. These are not valid completion 
> candidates. `footnote-` is not a symbol. Over time the huge increase in 
> completion candidate volume (and perhaps what is causing this behind the 
> scenes) results in such a slowdown that I can visibly see Emacs crawl to 
> 1 FPS when I type in minibuffer completion, when the exact same init.el 
> on said older 30.0.50 is faster.
> 
> This is reproduceable on `emacs -Q` as above, (and the useful bug report 
> probably ends right here) but as an aside: my configuration is minimal, 
> my Emacs starts up in about 0.3s and I keep things minimal because I do 
> not want Emacs to ever take longer than instantaneously to respond to 
> keystrokes (if it has to do some long-running job, or something which 
> takes time that's different since feedback that such a thing has kicked 
> off would/should still be instant).

Thanks, this is a known bug#72787.  I hope someone will be able to
look into it soon.

> Tangentially to this, it would be nice if completions could be decoupled 
> from the UI (i.e. they could stream in, with an API to let the user know 
> when the list is complete) because even on 30.0.50 I can often times 
> type faster than completion can handle which drives me insane (UI 
> hitching)... but that's an aside.

Please file a separate feature request for this, if you think we
should change the design so radically.

bug#73473: 31.0.50; Minibuffer completions include nonsense prefix candidates

[Eshel Yaron via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> merge 73473 72787
> thanks
>
>> Date: Wed, 25 Sep 2024 19:54:59 +0900
>> From:  Jordan Ellis Coppard via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
[...]
>> 1. Open Emacs.
>> 2. C-h o foo TAB
>> 
>> Observe that completion candidate `footnote-` is listed and that there 
>> are 80 completion candidates.
[...]
> Thanks, this is a known bug#72787.  I hope someone will be able to
> look into it soon.

FWIW, I've been using a change along the following lines:
(I don't completely understand why the existing code does what it does,
so I'm not sure that this fix is the right thing to do, but it makes
sense to me and it seems to solve this issue.)

diff --git a/lisp/help-fns.el b/lisp/help-fns.el
index e1daa8977f0..d2539e6074c 100644
--- a/lisp/help-fns.el
+++ b/lisp/help-fns.el
@@ -209,13 +209,9 @@ help--symbol-completion-table
     (when help-enable-completion-autoload
       (let ((prefixes (radix-tree-prefixes (help-definition-prefixes) string)))
         (help--load-prefixes prefixes)))
-    (let ((prefix-completions
-           (and help-enable-completion-autoload
-                (mapcar #'intern (all-completions string definition-prefixes)))))
-      (complete-with-action action obarray string
-                            (if pred (lambda (sym)
-                                       (or (funcall pred sym)
-                                           (memq sym prefix-completions))))))))
+    (when help-enable-completion-autoload
+      (mapc #'intern (all-completions string definition-prefixes)))
+    (complete-with-action action obarray string pred)))
 
 (defvar describe-function-orig-buffer nil
   "Buffer that was current when `describe-function' was invoked.

bug#73473: 31.0.50; Minibuffer completions include nonsense prefix candidates

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> I've noticed that my completions have slowed down A LOT on recent Emacs
>> builds, it turns out nonsense completion candidates (which I am calling
>> "prefix candidates") are being included as... completion candidates.
[...]
>> To reproduce, using `emacs -Q` of course:
>>
>> 1. Open Emacs.
>> 2. C-h o foo TAB
[...]
>> If I do the same on the currently released Emacs or another (older)
>> build of Emacs (30.0.50) there are 79 (not 80) completion candidates and
>> `footnote-` is not listed as one.

This was done on-purpose: the `footnote-` prefix is included as a "stand
in" for the not-yet-loaded definitions of `footnote.el`.  The idea is
that if you're looking for, say, `PREFIX-SUFFIX` in package PREFIX you
can get there by doing

    C-h o PREF TAB

which completes to `PREFIX-`, and then

    SUF TAB

(which now works because `PREFIX.el` was loaded under the hood).

AFAICT, there is a performance bug here in that just doing the
`C-h o PREF TAB` which completes to `PREFIX-` already loads `PREFIX.el`
which is a bit over-enthusiatic (I intended it to load `PREFIX.el`
only when you hit TAB while you already have `PREFIX-` in your
minibuffer).

>> This seems to be occurring for almost every unique prefix possible, 
>> compounding as time goes on. So later on `f-` is listed as a completion 
>> candidate, then also `go-` and so on. These are not valid completion 
>> candidates. `footnote-` is not a symbol. Over time the huge increase in 
>> completion candidate volume (and perhaps what is causing this behind the 
>> scenes) results in such a slowdown that I can visibly see Emacs crawl to 
>> 1 FPS when I type in minibuffer completion, when the exact same init.el 
>> on said older 30.0.50 is faster.

It's probably not directly linked to the number of symbols defined, but
rather to side-effects introduced by some of the packages that were
loaded (which I'd argue are probably themselves (performance) bugs in
those packages).


        Stefan

bug#73473: 31.0.50; Minibuffer completions include nonsense prefix candidates

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 433 bytes --]

> AFAICT, there is a performance bug here in that just doing the
> `C-h o PREF TAB` which completes to `PREFIX-` already loads `PREFIX.el`
> which is a bit over-enthusiatic (I intended it to load `PREFIX.el`
> only when you hit TAB while you already have `PREFIX-` in your
> minibuffer).

The patch below seems to fix this performance bug in my test.
Can you see if it avoids the major slowdown you're experiencing?


        Stefan

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: completion-autoload.patch --]
[-- Type: text/x-diff, Size: 941 bytes --]

diff --git a/lisp/help-fns.el b/lisp/help-fns.el
index e1daa8977f0..a9634745282 100644
--- a/lisp/help-fns.el
+++ b/lisp/help-fns.el
@@ -206,9 +206,12 @@ help--symbol-completion-table
         ,@(when completions-detailed
             '((affixation-function . help--symbol-completion-table-affixation)))
         (category . symbol-help))
-    (when help-enable-completion-autoload
+    (when (and help-enable-completion-autoload
+               (memq action '(nil t lambda)))
       (let ((prefixes (radix-tree-prefixes (help-definition-prefixes) string)))
-        (help--load-prefixes prefixes)))
+        ;; Don't load FOO.el during `test-completion' of `FOO-'.
+        (unless (and (eq action 'lambda) (assoc string prefixes))
+          (help--load-prefixes prefixes))))
     (let ((prefix-completions
            (and help-enable-completion-autoload
                 (mapcar #'intern (all-completions string definition-prefixes)))))