bug#54437: 28.0.92; command-modes returns nil for native compiled functions

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Knut Anders Hatlen]

With Emacs built with --with-native-compilation, command-modes appear to
return nil for native compiled functions:

(progn
  (load "help-mode")
  (command-modes 'help-view-source))
=> nil

When the function is interpreted or byte-compiled, it returns the
expected result:

(progn
  (load "help-mode.el")
  (command-modes 'help-view-source))
=> (help-mode)

(progn
  (load "help-mode.elc")
  (command-modes 'help-view-source))
=> (help-mode)



In GNU Emacs 28.0.92 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.16.0)
 of 2022-03-16 built on dell
Repository revision: 62e830c3d9f3a74c65309d1f8f18f45a7f065a9f
Repository branch: emacs-28
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux 11 (bullseye)

Configured using:
 'configure --with-native-compilation --with-json --with-xml2
 --with-modules 'CFLAGS=-O2 -flto -march=native' LDFLAGS=-flto
 --prefix=/usr/local/stow/emacs'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBOTF LIBSELINUX LIBXML2 M17N_FLT MODULES NATIVE_COMP NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LANG: nn_NO.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa epg rfc6068 epg-config gnus-util rmail
rmail-loaddefs auth-source eieio eieio-core eieio-loaddefs
password-cache json map text-property-search time-date mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
misearch multi-isearch derived jka-compr help-fns radix-tree cl-print
debug backtrace comp comp-cstr warnings subr-x rx cl-seq cl-macs
cl-extra seq byte-opt gv bytecomp byte-compile cconv find-func help-mode
cl-loaddefs cl-lib iso-transl tooltip eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 95067 5497)
 (symbols 48 8235 3)
 (strings 32 23907 1513)
 (string-bytes 1 813611)
 (vectors 16 18711)
 (vector-slots 8 356569 13037)
 (floats 8 29 34)
 (intervals 56 294 0)
 (buffers 992 14))

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Lars Ingebrigtsen]

Knut Anders Hatlen <kahatlen@gmail.com> writes:

> With Emacs built with --with-native-compilation, command-modes appear to
> return nil for native compiled functions:
>
> (progn
>   (load "help-mode")
>   (command-modes 'help-view-source))
> => nil

I'm unable to reproduce this in emacs-28 with nativecomp (on
Debian/bookworm) from "emacs -Q".

Is this with a clean build?  I.e., "git clean -xf", then build Emacs,
and "emacs -Q"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I'm unable to reproduce this in emacs-28 with nativecomp (on
> Debian/bookworm) from "emacs -Q".
>
> Is this with a clean build?  I.e., "git clean -xf", then build Emacs,
> and "emacs -Q"?

Hm.  I tried that now, and I can now reproduce the problem.  Weird.
I'll debug further.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Eli Zaretskii]

> From: Knut Anders Hatlen <kahatlen@gmail.com>
> Date: Thu, 17 Mar 2022 15:53:18 +0100
> 
> 
> With Emacs built with --with-native-compilation, command-modes appear to
> return nil for native compiled functions:
> 
> (progn
>   (load "help-mode")
>   (command-modes 'help-view-source))
> => nil
> 
> When the function is interpreted or byte-compiled, it returns the
> expected result:
> 
> (progn
>   (load "help-mode.el")
>   (command-modes 'help-view-source))
> => (help-mode)
> 
> (progn
>   (load "help-mode.elc")
>   (command-modes 'help-view-source))
> => (help-mode)

Obviously, command-modes doesn't consider natively-compiled functions,
but even if it did, it sounds like the information is lost at some
point, even with loading *.el and *.elc files:

  (progn
    (load "help-mode.elc")
    (interactive-form 'help-view-source))
   => (interactive nil)

Shouldn't interactive-form return the 3rd element as well?

Andrea, can you please look at this?  This is about the Emacs 28
pretest, so it's quite urgent.  TIA.

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>   (progn
>     (load "help-mode.elc")
>     (interactive-form 'help-view-source))
>    => (interactive nil)
>
> Shouldn't interactive-form return the 3rd element as well?

No, `interactive-form' is backwards-compatible (and doesn't have the
modes).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Andrea, can you please look at this?  This is about the Emacs 28
> pretest, so it's quite urgent.  TIA.

And I think I've just about fixed this now, but it needs more testing.
And it's quite invasive, so perhaps it's too late for emacs-28.  We'll
see.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Knut Anders Hatlen <kahatlen@gmail.com>,  Andrea Corallo <akrl@sdf.org>,
>   54437@debbugs.gnu.org
> Date: Thu, 17 Mar 2022 18:58:55 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Andrea, can you please look at this?  This is about the Emacs 28
> > pretest, so it's quite urgent.  TIA.
> 
> And I think I've just about fixed this now, but it needs more testing.
> And it's quite invasive, so perhaps it's too late for emacs-28.  We'll
> see.

Did you push?

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Did you push?

No, I'm still running through various setups.  I'm on holday(ish) and
this laptop is rather slow when doing a bunch of builds in different
configurations.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> No, I'm still running through various setups.  I'm on holday(ish) and
> this laptop is rather slow when doing a bunch of builds in different
> configurations.

Something was wonky, and I wasn't sure what, but I think there's an
error in our nativecomp builds: If make a change to a header file that
results in a fingerprint change, say "make" won't build the .eln files
now required in the native-lisp/ directory.  Which made me think I was
doing something wrong, but apparently this is just the way it is.

The patch works, and applying it does not seem to make previous
installations bug out.  I think.  But I have to do a few more test
builds, because I'm still not sure of just what I'm seeing.  So no patch
today.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Knut Anders Hatlen <kahatlen@gmail.com>
>> Date: Thu, 17 Mar 2022 15:53:18 +0100
>> 
>> 
>> With Emacs built with --with-native-compilation, command-modes appear to
>> return nil for native compiled functions:
>> 
>> (progn
>>   (load "help-mode")
>>   (command-modes 'help-view-source))
>> => nil
>> 
>> When the function is interpreted or byte-compiled, it returns the
>> expected result:
>> 
>> (progn
>>   (load "help-mode.el")
>>   (command-modes 'help-view-source))
>> => (help-mode)
>> 
>> (progn
>>   (load "help-mode.elc")
>>   (command-modes 'help-view-source))
>> => (help-mode)
>
> Obviously, command-modes doesn't consider natively-compiled functions,
> but even if it did, it sounds like the information is lost at some
> point, even with loading *.el and *.elc files:
>
>   (progn
>     (load "help-mode.elc")
>     (interactive-form 'help-view-source))
>    => (interactive nil)
>
> Shouldn't interactive-form return the 3rd element as well?
>
> Andrea, can you please look at this?  This is about the Emacs 28
> pretest, so it's quite urgent.  TIA.

Hi Eli & all,

I'm reading this mail now while catching a plane, I won't be able to
actively look at it before the weekend.  If no one finds a solution
sooner I'll look into it as soon as I can.

Best Regards

  Andrea

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Hm.  I tried that now, and I can now reproduce the problem.  Weird.
> I'll debug further.

I've now pushed a fix for this to Emacs 29.  A less invasive fix for
Emacs 28 might be nice, but I couldn't think of one, so I think we might
just have to say that command-modes + nativecomp just doesn't work in
Emacs 28.1.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 19 Mar 2022 15:13:07 +0100
> Cc: 54437@debbugs.gnu.org
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Hm.  I tried that now, and I can now reproduce the problem.  Weird.
> > I'll debug further.
> 
> I've now pushed a fix for this to Emacs 29.  A less invasive fix for
> Emacs 28 might be nice, but I couldn't think of one, so I think we might
> just have to say that command-modes + nativecomp just doesn't work in
> Emacs 28.1.

Thanks.

Andrea, any thoughts?

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Sat, 19 Mar 2022 15:13:07 +0100
>> Cc: 54437@debbugs.gnu.org
>> 
>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>> 
>> > Hm.  I tried that now, and I can now reproduce the problem.  Weird.
>> > I'll debug further.
>> 
>> I've now pushed a fix for this to Emacs 29.  A less invasive fix for
>> Emacs 28 might be nice, but I couldn't think of one, so I think we might
>> just have to say that command-modes + nativecomp just doesn't work in
>> Emacs 28.1.
>
> Thanks.
>
> Andrea, any thoughts?

Just had a look, the fix LGTM (thanks Lars!).

I just have a question for Lars, couldn't we just use and hash table to
map function-name -> modes?  Maybe this way the fix would be
considerably less invasive?

Thanks

  Andrea

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> I just have a question for Lars, couldn't we just use and hash table to
> map function-name -> modes?  Maybe this way the fix would be
> considerably less invasive?

Something would have to keep the table updated.  Did you have something
particular in mind?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Knut Anders Hatlen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Hm.  I tried that now, and I can now reproduce the problem.  Weird.
>> I'll debug further.
>
> I've now pushed a fix for this to Emacs 29.  A less invasive fix for
> Emacs 28 might be nice, but I couldn't think of one, so I think we might
> just have to say that command-modes + nativecomp just doesn't work in
> Emacs 28.1.

Thank you, Lars! It works on Emacs 29 now, as far as I can tell. The
reason I filed this bug was that setting read-extended-command-predicate
to command-completion-default-include-p did not work as I expected. Now
it does.

-- 
Knut Anders

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> I just have a question for Lars, couldn't we just use and hash table to
>> map function-name -> modes?  Maybe this way the fix would be
>> considerably less invasive?
>
> Something would have to keep the table updated.

Yes, we'd still need to store the information in each .eln and update
the value of this new global hash table each time (probably in
comp--register-subr).  So some C should be modified anyway even if
(maybe?) less, I doubt is doable in the realease branch at this point
anyway.

  Andrea

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Yes, we'd still need to store the information in each .eln and update
> the value of this new global hash table each time (probably in
> comp--register-subr).  So some C should be modified anyway even if
> (maybe?) less, I doubt is doable in the realease branch at this point
> anyway.

There's also the problem with stale entries -- if you're re-loading an
.eln with different functions with different command modes, you'd need
some way to nix out the old ones (for instance, if a command goes from
having modes to not).  Or you'd have to register all the commands in the
hash table, which seems awkward.

I think if we want this in Emacs 28, it's probably safer to just
cherry-pick the commit to master, as invasive as that is.  On the other
hand, the command-modes stuff is new functionality in Emacs 28, so just
documenting that it doesn't work with native-comp (yet) would also be
fine, and then perhaps do an Emacs 28.2 later with the patch.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Yes, we'd still need to store the information in each .eln and update
>> the value of this new global hash table each time (probably in
>> comp--register-subr).  So some C should be modified anyway even if
>> (maybe?) less, I doubt is doable in the realease branch at this point
>> anyway.
>
> There's also the problem with stale entries -- if you're re-loading an
> .eln with different functions with different command modes, you'd need
> some way to nix out the old ones (for instance, if a command goes from
> having modes to not).  Or you'd have to register all the commands in the
> hash table, which seems awkward.

In comp--register-subr I'd add the entry into the hash table if the
function being loaded is connected to some mode or I'd clean the entry
in case the new function being defined has no mode but an entry is
already present.

But that said I agree with you that given the resulting patch would not
be trivial and we have to touch some C anyway having two codes to solve
the same issue doesn't sound like a good idea.

> I think if we want this in Emacs 28, it's probably safer to just
> cherry-pick the commit to master, as invasive as that is.  On the other
> hand, the command-modes stuff is new functionality in Emacs 28, so just
> documenting that it doesn't work with native-comp (yet) would also be
> fine, and then perhaps do an Emacs 28.2 later with the patch.

Agree

Thanks

  Andrea

bug#54437: 28.0.92; command-modes returns nil for native compiled functions

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>> I think if we want this in Emacs 28, it's probably safer to just
>> cherry-pick the commit to master, as invasive as that is.  On the other
>> hand, the command-modes stuff is new functionality in Emacs 28, so just
>> documenting that it doesn't work with native-comp (yet) would also be
>> fine, and then perhaps do an Emacs 28.2 later with the patch.
>
> Agree

I've now done this in emacs-28.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no