bug#65250: 30.0.50; "C-h f" is much slower on the master branch

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Eli Zaretskii]

To reproduce:

  emacs -Q
  C-h f dictionary-search RET

(almost any other function will do, I think).

This takes a whopping 6.6 sec of CPU on master, vs 2.4 sec on the
emacs-29 branch.  These are both unoptimized builds, but even so,
6.6 seconds of CPU time for looking up a doc string of a function
is too much, I think.

The patch below fixes the problem in a build without
native-compilation, but won't help in a build with native-compilation.
I wonder why comp-function-type-spec is so expensive?



In GNU Emacs 30.0.50 (build 417, i686-pc-mingw32) of 2023-08-12 built on
 HOME-C4E4A596F7
Repository revision: d3dae88e6cc8118c875957ba0347be9599014b34
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)

Configured using:
 'configure -C --prefix=/d/usr --with-wide-int
 --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NOTIFY
W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel dos-w32
ls-lisp disp-table term/w32-win w32-win w32-vars term/common-win
touch-screen tool-bar dnd fontset image regexp-opt fringe tabulated-list
replace newcomment text-mode lisp-mode prog-mode register page tab-bar
menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse
jit-lock font-lock syntax font-core term/tty-colors frame minibuffer
nadvice seq simple cl-generic indonesian philippine cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop
case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads w32notify w32 lcms2 multi-tty move-toolbar make-network-process
emacs)

Memory information:
((conses 16 42844 17019) (symbols 48 6336 0) (strings 16 16702 3269)
 (string-bytes 1 401689) (vectors 16 9351)
 (vector-slots 8 147820 16098) (floats 8 23 25) (intervals 40 276 95)
 (buffers 896 10))

Here's the patch I promised:

diff --git a/lisp/help-fns.el b/lisp/help-fns.el
index fc8f431..bedc5a9 100644
--- a/lisp/help-fns.el
+++ b/lisp/help-fns.el
@@ -715,7 +715,8 @@ help-fns--signature
           (unless (and (symbolp function)
                        (get function 'reader-construct))
             (insert high-usage "\n")
-            (when-let* ((res (comp-function-type-spec function))
+            (when-let* ((res (and (native-comp-available-p)
+                                  (comp-function-type-spec function)))
                         (type-spec (car res))
                         (kind (cdr res)))
               (insert (format

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

> To reproduce:
>
>   emacs -Q
>   C-h f dictionary-search RET
>
> (almost any other function will do, I think).
>
> This takes a whopping 6.6 sec of CPU on master, vs 2.4 sec on the
> emacs-29 branch.  These are both unoptimized builds, but even so,
> 6.6 seconds of CPU time for looking up a doc string of a function
> is too much, I think.
>
> The patch below fixes the problem in a build without
> native-compilation, but won't help in a build with native-compilation.
> I wonder why comp-function-type-spec is so expensive?

Hi Eli,

can't look at this now, will do on Monday and report.

Best Regards

  Andrea

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: 65250@debbugs.gnu.org
> Date: Sat, 12 Aug 2023 12:57:27 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > To reproduce:
> >
> >   emacs -Q
> >   C-h f dictionary-search RET
> >
> > (almost any other function will do, I think).
> >
> > This takes a whopping 6.6 sec of CPU on master, vs 2.4 sec on the
> > emacs-29 branch.  These are both unoptimized builds, but even so,
> > 6.6 seconds of CPU time for looking up a doc string of a function
> > is too much, I think.
> >
> > The patch below fixes the problem in a build without
> > native-compilation, but won't help in a build with native-compilation.
> > I wonder why comp-function-type-spec is so expensive?
> 
> Hi Eli,
> 
> can't look at this now, will do on Monday and report.

Thanks, this is on master, so not very urgent.

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

> To reproduce:
>
>   emacs -Q
>   C-h f dictionary-search RET
>
> (almost any other function will do, I think).
>
> This takes a whopping 6.6 sec of CPU on master, vs 2.4 sec on the
> emacs-29 branch.  These are both unoptimized builds, but even so,
> 6.6 seconds of CPU time for looking up a doc string of a function
> is too much, I think.
>
> The patch below fixes the problem in a build without
> native-compilation, but won't help in a build with native-compilation.
> I wonder why comp-function-type-spec is so expensive?

Hi Eli,

I'm failing to reproduce it, this is what I did on latest master:

~~~

CFLAGS='-g3 -O0' make bootstrap  -j16

./src/emacs -Q

M-: (require 'dictionary)

C-h f dictionary-search RET

~~~

The result is pretty much istantaneous on my machine.

I guess I'm doing something different from what you did?

Thanks

  Andrea

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: 65250@debbugs.gnu.org
> Date: Mon, 14 Aug 2023 04:59:18 -0400
> 
> I'm failing to reproduce it, this is what I did on latest master:
> 
> ~~~
> 
> CFLAGS='-g3 -O0' make bootstrap  -j16
> 
> ./src/emacs -Q
> 
> M-: (require 'dictionary)
> 
> C-h f dictionary-search RET
> 
> ~~~
> 
> The result is pretty much istantaneous on my machine.
> 
> I guess I'm doing something different from what you did?

My configuration was included with the report:

   'configure -C --prefix=/d/usr --with-wide-int
   --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''

Perhaps --enable-checking makes the difference?

If even that doesn't show the problem, just time the above and compare
with Emacs 29: it's possible that the command is much faster on your
system, but the question is it significantly slower than Emacs 29?

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Eli Zaretskii]

> Cc: 65250@debbugs.gnu.org
> Date: Mon, 14 Aug 2023 15:19:16 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
>    'configure -C --prefix=/d/usr --with-wide-int
>    --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''
> 
> Perhaps --enable-checking makes the difference?
> 
> If even that doesn't show the problem, just time the above and compare
> with Emacs 29: it's possible that the command is much faster on your
> system, but the question is it significantly slower than Emacs 29?

It sounds like the problem is the packages that Emacs needs to load
when comp-function-type-spec is called.  If I set force-load-messages
to t before invoking "C-h f", I see this in the *Messages* buffer:

  Loading help-fns...
  Loading cl-lib...
  Loading cl-loaddefs...done
  Loading cl-lib...done
  Loading help-mode...done
  Loading radix-tree...done
  Loading help-fns...done
  Loading thingatpt...done
  Loading dictionary...
  Loading dictionary-connection...done
  Loading external-completion...done
  Loading dictionary...done
  Loading lisp/emacs-lisp/comp.el (source)...
  Loading bytecomp...done
  Loading cl-extra...done
  Loading cl-macs...
  Loading gv...done
  Loading cl-macs...done
  Loading cl-seq...done
  Loading rx...done
  Loading subr-x...done
  Loading warnings...
  Loading icons...done
  Loading warnings...done
  Loading lisp/emacs-lisp/comp-cstr.el (source)...
  Loading pcase...done
  Loading lisp/emacs-lisp/comp-cstr.el (source)...done
  Loading derived...done
  Loading lisp/emacs-lisp/comp.el (source)...done
  Loading shortdoc...
  Loading text-property-search...done
  Loading shortdoc...done

Note the loading of comp.el and comp-cstr.el -- we load their source
files, not the *.elc files.  That's because in a build without native
compilation these two files are not byte-compiled.  I think loading of
these files, especially of comp.el, in source form is what slows down
the command.

I'm guessing your build was with native compilation?  Because in such
a build the "C-h f" command is indeed fast, especially after the
requisite *.el files are all native-compiled (i.e. starting from the
second Emacs invocation after the build).

So I think the patch I presented in my original report is exactly what
is needed here: the problem only happens in builds without
native-compilation, and in that case there's no reason whatsoever to
call comp-function-type-spec.  (And builds from a release tarball will
not see that problem, since the tarball comes with byte-compiled
comp.el and comp-cstr.el.)

Do you agree?

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: 65250@debbugs.gnu.org
>> Date: Mon, 14 Aug 2023 15:19:16 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>>    'configure -C --prefix=/d/usr --with-wide-int
>>    --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''
>> 
>> Perhaps --enable-checking makes the difference?
>> 
>> If even that doesn't show the problem, just time the above and compare
>> with Emacs 29: it's possible that the command is much faster on your
>> system, but the question is it significantly slower than Emacs 29?
>
> It sounds like the problem is the packages that Emacs needs to load
> when comp-function-type-spec is called.  If I set force-load-messages
> to t before invoking "C-h f", I see this in the *Messages* buffer:
>
>   Loading help-fns...
>   Loading cl-lib...
>   Loading cl-loaddefs...done
>   Loading cl-lib...done
>   Loading help-mode...done
>   Loading radix-tree...done
>   Loading help-fns...done
>   Loading thingatpt...done
>   Loading dictionary...
>   Loading dictionary-connection...done
>   Loading external-completion...done
>   Loading dictionary...done
>   Loading lisp/emacs-lisp/comp.el (source)...
>   Loading bytecomp...done
>   Loading cl-extra...done
>   Loading cl-macs...
>   Loading gv...done
>   Loading cl-macs...done
>   Loading cl-seq...done
>   Loading rx...done
>   Loading subr-x...done
>   Loading warnings...
>   Loading icons...done
>   Loading warnings...done
>   Loading lisp/emacs-lisp/comp-cstr.el (source)...
>   Loading pcase...done
>   Loading lisp/emacs-lisp/comp-cstr.el (source)...done
>   Loading derived...done
>   Loading lisp/emacs-lisp/comp.el (source)...done
>   Loading shortdoc...
>   Loading text-property-search...done
>   Loading shortdoc...done
>
> Note the loading of comp.el and comp-cstr.el -- we load their source
> files, not the *.elc files.  That's because in a build without native
> compilation these two files are not byte-compiled.  I think loading of
> these files, especially of comp.el, in source form is what slows down
> the command.

Maybe, comp-cstr.el might have even a bigger part.

> I'm guessing your build was with native compilation?

Yes, I was experimenting just now (and failing to reproduce) with

CFLAGS='-O0 -gdwarf-4 -g3' ./configure --without-x
--with-native-compilation=yes --prefix='/home/andcor03'
--enable-checking=yes,glyphs --with-wide-int

> Because in such
> a build the "C-h f" command is indeed fast, especially after the
> requisite *.el files are all native-compiled (i.e. starting from the
> second Emacs invocation after the build).

Ah right! now it's all clear!

> So I think the patch I presented in my original report is exactly what
> is needed here: the problem only happens in builds without
> native-compilation, and in that case there's no reason whatsoever to
> call comp-function-type-spec.  (And builds from a release tarball will
> not see that problem, since the tarball comes with byte-compiled
> comp.el and comp-cstr.el.)
>
> Do you agree?

I certainly do.  Thanks for the anylysis and the patch!

  Andrea

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: 65250@debbugs.gnu.org
> Date: Mon, 14 Aug 2023 10:51:46 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So I think the patch I presented in my original report is exactly what
> > is needed here: the problem only happens in builds without
> > native-compilation, and in that case there's no reason whatsoever to
> > call comp-function-type-spec.  (And builds from a release tarball will
> > not see that problem, since the tarball comes with byte-compiled
> > comp.el and comp-cstr.el.)
> >
> > Do you agree?
> 
> I certainly do.  Thanks for the anylysis and the patch!

Installed.

Btw, why aren't comp.el and comp-cstr.el byte-compiled in a build
without native-compilation?  That's probably a bug in itself: we
generally byte-compile all the *.el files, even those that are not
relevant to the configuration being built.

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: 65250@debbugs.gnu.org
>> Date: Mon, 14 Aug 2023 10:51:46 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > So I think the patch I presented in my original report is exactly what
>> > is needed here: the problem only happens in builds without
>> > native-compilation, and in that case there's no reason whatsoever to
>> > call comp-function-type-spec.  (And builds from a release tarball will
>> > not see that problem, since the tarball comes with byte-compiled
>> > comp.el and comp-cstr.el.)
>> >
>> > Do you agree?
>> 
>> I certainly do.  Thanks for the anylysis and the patch!
>
> Installed.
>
> Btw, why aren't comp.el and comp-cstr.el byte-compiled in a build
> without native-compilation?  That's probably a bug in itself: we
> generally byte-compile all the *.el files, even those that are not
> relevant to the configuration being built.

That's a good question, I'll have a look probably tomorrow.

Thanks

  Andrea

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <acorallo@gnu.org>
>>> Cc: 65250@debbugs.gnu.org
>>> Date: Mon, 14 Aug 2023 10:51:46 -0400
>>>
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>> > So I think the patch I presented in my original report is exactly what
>>> > is needed here: the problem only happens in builds without
>>> > native-compilation, and in that case there's no reason whatsoever to
>>> > call comp-function-type-spec.  (And builds from a release tarball will
>>> > not see that problem, since the tarball comes with byte-compiled
>>> > comp.el and comp-cstr.el.)
>>> >
>>> > Do you agree?
>>>
>>> I certainly do.  Thanks for the anylysis and the patch!
>>
>> Installed.
>>
>> Btw, why aren't comp.el and comp-cstr.el byte-compiled in a build
>> without native-compilation?  That's probably a bug in itself: we
>> generally byte-compile all the *.el files, even those that are not
>> relevant to the configuration being built.
>
> That's a good question, I'll have a look probably tomorrow.

Thinking better about it I guess the original rational behind was me
trying to minimize the impact of the original native-comp patch on
Emacs.

The following patch should do what we want, I'll test it as soon as I've
time before pushing it.

diff --git a/lisp/Makefile.in b/lisp/Makefile.in
index 5af2168a827..c4dd1e7a1f3 100644
--- a/lisp/Makefile.in
+++ b/lisp/Makefile.in
@@ -351,11 +351,7 @@ .PHONY:
 # TARGETS is set dynamically in the recursive call from 'compile-main'.
 # Do not build comp.el unless necessary not to exceed max-lisp-eval-depth
 # in normal builds.
-ifneq ($(HAVE_NATIVE_COMP),yes)
-compile-targets: $(filter-out ./emacs-lisp/comp-cstr.elc,$(filter-out ./emacs-lisp/comp.elc,$(TARGETS)))
-else
 compile-targets: $(TARGETS)
-endif

 # Compile all the Elisp files that need it.  Beware: it approximates
 # 'no-byte-compile', so watch out for false-positives!

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: Andrea Corallo <acorallo@gnu.org>
>>>> Cc: 65250@debbugs.gnu.org
>>>> Date: Mon, 14 Aug 2023 10:51:46 -0400
>>>>
>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>>
>>>> > So I think the patch I presented in my original report is exactly what
>>>> > is needed here: the problem only happens in builds without
>>>> > native-compilation, and in that case there's no reason whatsoever to
>>>> > call comp-function-type-spec.  (And builds from a release tarball will
>>>> > not see that problem, since the tarball comes with byte-compiled
>>>> > comp.el and comp-cstr.el.)
>>>> >
>>>> > Do you agree?
>>>>
>>>> I certainly do.  Thanks for the anylysis and the patch!
>>>
>>> Installed.
>>>
>>> Btw, why aren't comp.el and comp-cstr.el byte-compiled in a build
>>> without native-compilation?  That's probably a bug in itself: we
>>> generally byte-compile all the *.el files, even those that are not
>>> relevant to the configuration being built.
>>
>> That's a good question, I'll have a look probably tomorrow.
>
> Thinking better about it I guess the original rational behind was me
> trying to minimize the impact of the original native-comp patch on
> Emacs.
>
> The following patch should do what we want, I'll test it as soon as I've
> time before pushing it.
>
> diff --git a/lisp/Makefile.in b/lisp/Makefile.in
> index 5af2168a827..c4dd1e7a1f3 100644
> --- a/lisp/Makefile.in
> +++ b/lisp/Makefile.in
> @@ -351,11 +351,7 @@ .PHONY:
>  # TARGETS is set dynamically in the recursive call from 'compile-main'.
>  # Do not build comp.el unless necessary not to exceed max-lisp-eval-depth
>  # in normal builds.
> -ifneq ($(HAVE_NATIVE_COMP),yes)
> -compile-targets: $(filter-out ./emacs-lisp/comp-cstr.elc,$(filter-out ./emacs-lisp/comp.elc,$(TARGETS)))
> -else
>  compile-targets: $(TARGETS)
> -endif
>
>  # Compile all the Elisp files that need it.  Beware: it approximates
>  # 'no-byte-compile', so watch out for false-positives!

Okay I pushed 2eaf1e3efca to always byte-compile native comp related
files also on non native builds.

Also thinking better about it, I think if this fix is sufficient to make
C-h f sufficiently fast in your test conditions we should revert
545f95d1a32.

Before installing it, even in non native builds, C-h f was showing the
type for functions listed in `comp-known-type-specifiers'.  I completely
forgot about this aspect reviewing the patch the other day apologies.

WDYT

  Andrea

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: 65250@debbugs.gnu.org
> Date: Wed, 16 Aug 2023 04:37:36 -0400
> 
> Okay I pushed 2eaf1e3efca to always byte-compile native comp related
> files also on non native builds.

Thanks.

> Also thinking better about it, I think if this fix is sufficient to make
> C-h f sufficiently fast in your test conditions we should revert
> 545f95d1a32.

Done.

> Before installing it, even in non native builds, C-h f was showing the
> type for functions listed in `comp-known-type-specifiers'.

How can I see this capability?

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: 65250@debbugs.gnu.org
>> Date: Wed, 16 Aug 2023 04:37:36 -0400
>> 
>> Okay I pushed 2eaf1e3efca to always byte-compile native comp related
>> files also on non native builds.
>
> Thanks.
>
>> Also thinking better about it, I think if this fix is sufficient to make
>> C-h f sufficiently fast in your test conditions we should revert
>> 545f95d1a32.
>
> Done.
>
>> Before installing it, even in non native builds, C-h f was showing the
>> type for functions listed in `comp-known-type-specifiers'.
>
> How can I see this capability?

Should be sufficient say 'C-h f +'.

(+ or any function in comp-known-type-specifiers).

Best Regards

  Andrea

bug#65250: 30.0.50; "C-h f" is much slower on the master branch

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: 65250@debbugs.gnu.org
> Date: Wed, 16 Aug 2023 09:55:19 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Before installing it, even in non native builds, C-h f was showing the
> >> type for functions listed in `comp-known-type-specifiers'.
> >
> > How can I see this capability?
> 
> Should be sufficient say 'C-h f +'.

OK, works here in a build without native compilation.

So I think we can now close this bug.