bug#66998: 29.1; Regression for recursive keyboard macros + minibuffers in (I think) Emacs 28

bug#66998: 29.1; Regression for recursive keyboard macros + minibuffers in (I think) Emacs 28

[Morgon Kanter]

[-- Attachment #1: Type: text/plain, Size: 7256 bytes --]

I believe there is a regression, but possibly intentional, caused by this
patch:
https://github.com/emacs-mirror/emacs/commit/203e61ff837128b397eb313a5bb1b703f0eae0ec

This affects minibuffers created when (kbd-macro-query t) is called as
part of the hook that runs when the (read-from-minibuffer) function is
called. You get the error message "Not in most nested command loop". For
example, this code here:
https://www.emacswiki.org/emacs/KeyboardMacros#h5o-5

Or, pasting the code in question:

    (defun my-macro-query (arg)
      "Prompt for input using minibuffer during kbd macro execution.
    With prefix argument, allows you to select what prompt string to use.
    If the input is non-empty, it is inserted at point."
      (interactive "P")
      (let* ((prompt (if arg (read-from-minibuffer "PROMPT: ") "Input: "))
             (input (minibuffer-with-setup-hook (lambda () (kbd-macro-query
t))
                      (read-from-minibuffer prompt))))
        (unless (string= "" input) (insert input))))
    (global-set-key (kbd "C-x Q") 'my-macro-query)

If you attempt to start a keyboard macro via F3, then attempt to read a
minibuffer with the above code via C-x Q, upon pressing ENTER to close
the minibuffer, you get the following error message:
"Not in most nested command loop"

You won't be able to close out the minibuffer, the only way I found to
proceed was to C-] or multiple escapes, which canceled the keyboard
macro creation. As a result, it appears we can't use the above method to
read and set variables during keyboard macro creation. I'm not sure if
this is intentional or not, or if there's a replacement for the above or
not. But it appears to be a regression from before that series of patches.


In GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.38,
cairo version 1.17.8)
Windowing system distributor 'Microsoft Corporation', version 11.0.12010000
System Description: Arch Linux

Configured using:
 'configure --with-x-toolkit=gtk3 --with-native-compilation=aot
 --sysconfdir=/etc --prefix=/usr --libexecdir=/usr/lib
 --with-tree-sitter --localstatedir=/var --with-cairo
 --disable-build-details --with-harfbuzz --with-libsystemd
 --with-modules 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt
 -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
 -fstack-clash-protection -fcf-protection -g
 -ffile-prefix-map=/build/emacs/src=/usr/src/debug/emacs -flto=auto'
 'LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto'
 'CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions
 -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
 -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS -g
 -ffile-prefix-map=/build/emacs/src=/usr/src/debug/emacs -flto=auto''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 M17N_FLT MODULES NATIVE_COMP NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: ELisp/d

Minor modes in effect:
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  shell-dirtrack-mode: t
  marginalia-mode: t
  override-global-mode: t
  vertico-mode: t
  which-key-mode: t
  server-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/surgo/.emacs.d/elpa/transient-20231103.2312/transient hides
/usr/share/emacs/29.1/lisp/transient
/home/surgo/.emacs.d/elpa/seq-2.24/seq hides
/usr/share/emacs/29.1/lisp/emacs-lisp/seq

Features:
(shadow sort mail-extr emacsbug minibuf-eldef thingatpt misearch
multi-isearch cl-print debug backtrace find-func mule-util shortdoc
help-fns radix-tree macros dired-aux cc-styles cc-align cc-engine
cc-vars cc-defs magit-submodule magit-blame magit-stash magit-reflog
magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-tag
magit-merge magit-branch magit-reset magit-files magit-refs magit-status
magit magit-repos magit-apply magit-wip magit-log which-func imenu
magit-diff smerge-mode diff diff-mode git-commit log-edit message
sendmail yank-media puny dired dired-loaddefs rfc822 mml mml-sec epa
derived epg rfc6068 epg-config gnus-util text-property-search time-date
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils mailheader
pcvs-util add-log magit-core magit-autorevert autorevert filenotify
magit-margin magit-transient magit-process with-editor comp comp-cstr
warnings icons rx shell pcomplete comint ansi-osc ring ansi-color
magit-mode transient magit-git magit-base magit-section format-spec
cursor-sensor crm dash marginalia edmacro kmacro use-package-bind-key
bind-key easy-mmode vertico compat diminish which-key cl-extra help-mode
use-package-diminish use-package-core tango-dark-theme server
magit-autoloads pcase git-commit-autoloads magit-section-autoloads
dash-autoloads marginalia-autoloads projectile-autoloads
transient-autoloads vertico-autoloads with-editor-autoloads info
compat-autoloads seq-autoloads package browse-url url url-proxy
url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs password-cache json subr-x
map byte-opt gv bytecomp byte-compile url-vars cl-loaddefs cl-lib rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify lcms2 dynamic-setting system-font-setting
font-render-setting cairo move-toolbar gtk x-toolkit xinput2 x multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 241723 51274)
 (symbols 48 17022 0)
 (strings 32 64370 2962)
 (string-bytes 1 2928754)
 (vectors 16 41856)
 (vector-slots 8 758358 34977)
 (floats 8 161 322)
 (intervals 56 1802 323)
 (buffers 984 17))

[-- Attachment #2: Type: text/html, Size: 8134 bytes --]

bug#66998: Further information

[Morgon Kanter]

[-- Attachment #1: Type: text/plain, Size: 317 bytes --]

Some additional information --

It works the same whether you use kbd-macro-query or recursive-edit in the
hook. Adding exit-recursive-edit to the minibuffer-exit hook doesn't help.

To actually finish typing in the minibuffer, it appears like you need to do
M-C-c, and then you can press enter to finish.

-- Morgon

[-- Attachment #2: Type: text/html, Size: 415 bytes --]

bug#66998: 29.1; Regression for recursive keyboard macros + minibuffers in (I think) Emacs 28

[Eli Zaretskii]

> From: Morgon Kanter <morgon.kanter@gmail.com>
> Date: Tue, 7 Nov 2023 22:29:22 -0500
> 
> I believe there is a regression, but possibly intentional, caused by this patch:
> https://github.com/emacs-mirror/emacs/commit/203e61ff837128b397eb313a5bb1b703f0eae0ec
> 
> This affects minibuffers created when (kbd-macro-query t) is called as
> part of the hook that runs when the (read-from-minibuffer) function is
> called. You get the error message "Not in most nested command loop". For
> example, this code here:
> https://www.emacswiki.org/emacs/KeyboardMacros#h5o-5
> 
> Or, pasting the code in question:
> 
>     (defun my-macro-query (arg)
>       "Prompt for input using minibuffer during kbd macro execution.
>     With prefix argument, allows you to select what prompt string to use.
>     If the input is non-empty, it is inserted at point."
>       (interactive "P")
>       (let* ((prompt (if arg (read-from-minibuffer "PROMPT: ") "Input: "))
>              (input (minibuffer-with-setup-hook (lambda () (kbd-macro-query t))
>                       (read-from-minibuffer prompt))))
>         (unless (string= "" input) (insert input))))
>     (global-set-key (kbd "C-x Q") 'my-macro-query)
> 
> If you attempt to start a keyboard macro via F3, then attempt to read a
> minibuffer with the above code via C-x Q, upon pressing ENTER to close
> the minibuffer, you get the following error message:
> "Not in most nested command loop"
> 
> You won't be able to close out the minibuffer, the only way I found to
> proceed was to C-] or multiple escapes, which canceled the keyboard
> macro creation. As a result, it appears we can't use the above method to
> read and set variables during keyboard macro creation. I'm not sure if
> this is intentional or not, or if there's a replacement for the above or
> not. But it appears to be a regression from before that series of patches.

Alan, can you please look into this?

bug#66998: 29.1; Regression for recursive keyboard macros + minibuffers in (I think) Emacs 28

[Alan Mackenzie]

Hello, Eli.

On Wed, Nov 08, 2023 at 14:36:56 +0200, Eli Zaretskii wrote:
> > From: Morgon Kanter <morgon.kanter@gmail.com>
> > Date: Tue, 7 Nov 2023 22:29:22 -0500

> > I believe there is a regression, but possibly intentional, caused by this patch:
> > https://github.com/emacs-mirror/emacs/commit/203e61ff837128b397eb313a5bb1b703f0eae0ec

> > This affects minibuffers created when (kbd-macro-query t) is called as
> > part of the hook that runs when the (read-from-minibuffer) function is
> > called. You get the error message "Not in most nested command loop". For
> > example, this code here:
> > https://www.emacswiki.org/emacs/KeyboardMacros#h5o-5

> > Or, pasting the code in question:

> >     (defun my-macro-query (arg)
> >       "Prompt for input using minibuffer during kbd macro execution.
> >     With prefix argument, allows you to select what prompt string to use.
> >     If the input is non-empty, it is inserted at point."
> >       (interactive "P")
> >       (let* ((prompt (if arg (read-from-minibuffer "PROMPT: ") "Input: "))
> >              (input (minibuffer-with-setup-hook (lambda () (kbd-macro-query t))
> >                       (read-from-minibuffer prompt))))
> >         (unless (string= "" input) (insert input))))
> >     (global-set-key (kbd "C-x Q") 'my-macro-query)

> > If you attempt to start a keyboard macro via F3, then attempt to read a
> > minibuffer with the above code via C-x Q, upon pressing ENTER to close
> > the minibuffer, you get the following error message:
> > "Not in most nested command loop"

> > You won't be able to close out the minibuffer, the only way I found to
> > proceed was to C-] or multiple escapes, which canceled the keyboard
> > macro creation. As a result, it appears we can't use the above method to
> > read and set variables during keyboard macro creation. I'm not sure if
> > this is intentional or not, or if there's a replacement for the above or
> > not. But it appears to be a regression from before that series of patches.

> Alan, can you please look into this?

Will do.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66998: 29.1; Regression for recursive keyboard macros + minibuffers in (I think) Emacs 28

[Alan Mackenzie]

Hello, Morgon.

Thanks for taking the trouble to report this.

[ Unfortunately, your post, in HTML, has got corrupted somewhere,
possibly in my mail user agent, thus is difficult to read. ]

On Tue, Nov 07, 2023 at 22:29:22 -0500, Morgon Kanter wrote:
>    I believe there is a regression, but possibly intentional, caused by
>    this patch:
>    [1]https://github.com/emacs-mirror/emacs/commit/203e61ff837128b397eb313
>    a5bb1b703f0eae0ec
>    This affects minibuffers created when (kbd-macro-query t) is called as
>    part of the hook that runs when the (read-from-minibuffer) function is
>    called. You get the error message "Not in most nested command loop".

>    For example, this code here:
>    [2]https://www.emacswiki.org/emacs/KeyboardMacros#h5o-5
>    Or, pasting the code in question:
>    Â  Â  (defun my-macro-query (arg)
>    Â  Â  Â  "Prompt for input using minibuffer during kbd macro execution.
>    Â  Â  With prefix argument, allows you to select what prompt string to
>    use.
>    Â  Â  If the input is non-empty, it is inserted at point."
>    Â  Â  Â  (interactive "P")
>    Â  Â  Â  (let* ((prompt (if arg (read-from-minibuffer "PROMPT: ")
>    "Input: "))
>    Â  Â  Â  Â  Â  Â  Â (input (minibuffer-with-setup-hook (lambda ()
>    (kbd-macro-query t))
>    Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  (read-from-minibuffer prompt))))
>    Â  Â  Â  Â  (unless (string= "" input) (insert input))))
>    Â  Â  (global-set-key (kbd "C-x Q") 'my-macro-query)

>    If you attempt to start a keyboard macro via F3, then attempt to read a
>    minibuffer with the above code via C-x Q, upon pressing ENTER to close
>    the minibuffer, you get the following error message:
>    "Not in most nested command loop"

>    You won't be able to close out the minibuffer, the only way I found
>    to proceed was to C-] or multiple escapes, which canceled the
>    keyboard macro creation. As a result, it appears we can't use the
>    above method to read and set variables during keyboard macro
>    creation. I'm not sure if this is intentional or not, or if there's
>    a replacement for the above or not. But it appears to be a
>    regression from before that series of patches.

As you say in your later post, you can terminate the recursive edit with
C-M-c.  I'm not sure there's actually a bug, here.  While in the
recursive edit, the minibuffer "belongs" to the outer editing level,
and this outer level expects the recursive edit to be closed before its
minibuffer input can be terminated.

So I think the error message "Not in most nested command loop" is
correct, even if its not very clear in this context.

What are you actually trying to achieve in your real Lisp code with this
recursive edit?  At first acquaintance, it looks rather unusual.

>    In GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.38,
>    cairo version 1.17.8)
>    Windowing system distributor 'Microsoft Corporation', version
>    11.0.12010000
>    System Description: Arch Linux

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66998: 29.1; Regression for recursive keyboard macros + minibuffers in (I think) Emacs 28

[Morgon Kanter]

Hi Alan,

tl;dr: you're right, not a bug, just user error :-)

Trying this one more time, I rediscovered how to turn on "plain text
mode". So I hope this one doesn't get garbled HTML.

First, this was the original code that got garbled. It should be
visible in the mailing list archive in a web browser. Pasted again
here:

> (defun config:macro-query (arg)
>   "Prompt for input using minibuffer during kbd macro execution.
> With prefix argument, allows you to select what prompt string to use.
> If the input is non-empty, it is inserted at point."
>   (interactive "P")
>   (let* ((prompt (if arg (read-from-minibuffer "PROMPT: ") "Input: "))
>          (input (minibuffer-with-setup-hook (lambda () (kbd-macro-query t))
>                   (read-from-minibuffer prompt))))

Your intuition was totally right. This isn't really a bug, and
probably not a regression in behavior either. Use of C-M-c to exit the
recursive edit before the minibuffer works as expected. The only
"problem" is that you need to press C-M-c to terminate the minibuffer,
rather than RET. That's a bit awkward and weird, but it's livable. I
could probably temporarily rebind RET to make it more ergonomic. But
the truth is that from Emacs's perspective this isn't even something
that *should* be fixed -- you *should* be exiting the recursive edit
before you exit the minibuffer, in that order!

So this, at least, is WAI and this bug should be closed.

> So I think the error message "Not in most nested command loop" is
> correct, even if its not very clear in this context.
>
> What are you actually trying to achieve in your real Lisp code with this
> recursive edit?  At first acquaintance, it looks rather unusual.

What I am trying to achieve is the ability to prompt the user as part
of a keyboard macro, and receive input which the macro will then do
something with. Importantly, this input could be different every time
the keyboard macro is run. Ordinarily if you were to prompt the user
for input, all those actions would be considered part of the keyboard
macro and simply re-run every time. So you need to invoke the
recursive edit to make it work.

bug#66998: 29.1; Regression for recursive keyboard macros + minibuffers in (I think) Emacs 28

[Alan Mackenzie]

Hello, Morgon.

On Thu, Nov 09, 2023 at 13:41:26 -0500, Morgon Kanter wrote:
> Hi Alan,

> tl;dr: you're right, not a bug, just user error :-)

> Trying this one more time, I rediscovered how to turn on "plain text
> mode". So I hope this one doesn't get garbled HTML.

Thanks, appreciated!

> First, this was the original code that got garbled. It should be
> visible in the mailing list archive in a web browser. Pasted again
> here:

> > (defun config:macro-query (arg)
> >   "Prompt for input using minibuffer during kbd macro execution.
> > With prefix argument, allows you to select what prompt string to use.
> > If the input is non-empty, it is inserted at point."
> >   (interactive "P")
> >   (let* ((prompt (if arg (read-from-minibuffer "PROMPT: ") "Input: "))
> >          (input (minibuffer-with-setup-hook (lambda () (kbd-macro-query t))
> >                   (read-from-minibuffer prompt))))

> Your intuition was totally right. This isn't really a bug, and
> probably not a regression in behavior either. Use of C-M-c to exit the
> recursive edit before the minibuffer works as expected. The only
> "problem" is that you need to press C-M-c to terminate the minibuffer,
> rather than RET. That's a bit awkward and weird, but it's livable. I
> could probably temporarily rebind RET to make it more ergonomic. But
> the truth is that from Emacs's perspective this isn't even something
> that *should* be fixed -- you *should* be exiting the recursive edit
> before you exit the minibuffer, in that order!

It should be possible in Emacs to do what you want to do.  I've not been
able to come up with any clean way to do this, even after sleeping on
it.

It seems there is a deficiency in Emacs's keyboard macro handling.  I
think we need a new interactive command called something like
interpolate-kbd-macro, which would take one argument, a function to run.
This function would take no arguments and return a list of key
sequences.  These key sequences, rather than being inserted into the
keyboard macro, would instead be looked up in the current keymaps, and
their commands (e.g.  self-insert-command) would get run as part of the
current keyboard macro invocation.

Or something like that.  What do you think?

> So this, at least, is WAI and this bug should be closed.

WAI?  That's a new one on me!

Possibly a new bug should be opened to implement my suggestion above.

> > So I think the error message "Not in most nested command loop" is
> > correct, even if its not very clear in this context.

> > What are you actually trying to achieve in your real Lisp code with this
> > recursive edit?  At first acquaintance, it looks rather unusual.

> What I am trying to achieve is the ability to prompt the user as part
> of a keyboard macro, and receive input which the macro will then do
> something with. Importantly, this input could be different every time
> the keyboard macro is run. Ordinarily if you were to prompt the user
> for input, all those actions would be considered part of the keyboard
> macro and simply re-run every time. So you need to invoke the
> recursive edit to make it work.

OK, thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#66998: 29.1; Regression for recursive keyboard macros + minibuffers in (I think) Emacs 28

[Morgon Kanter]

> > Your intuition was totally right. This isn't really a bug, and
> > probably not a regression in behavior either. Use of C-M-c to exit the
> > recursive edit before the minibuffer works as expected. The only
> > "problem" is that you need to press C-M-c to terminate the minibuffer,
> > rather than RET. That's a bit awkward and weird, but it's livable. I
> > could probably temporarily rebind RET to make it more ergonomic. But
> > the truth is that from Emacs's perspective this isn't even something
> > that *should* be fixed -- you *should* be exiting the recursive edit
> > before you exit the minibuffer, in that order!
>
> It should be possible in Emacs to do what you want to do.  I've not been
> able to come up with any clean way to do this, even after sleeping on
> it.
>
> It seems there is a deficiency in Emacs's keyboard macro handling.  I
> think we need a new interactive command called something like
> interpolate-kbd-macro, which would take one argument, a function to run.
> This function would take no arguments and return a list of key
> sequences.  These key sequences, rather than being inserted into the
> keyboard macro, would instead be looked up in the current keymaps, and
> their commands (e.g.  self-insert-command) would get run as part of the
> current keyboard macro invocation.
>
> Or something like that.  What do you think?

I ended up advising recursive-edit. I set up a transient keymap that
remaps anything that calls minibuffer-exit to exit-recursive-edit.
Then the advice, which fires after recursive-edit completes, undoes
the transient keymap. As far as I can tell, this works out pretty
well.

Your idea does sound cleaner overall than my solution. It's worth
thinking if there's anything else that would be interesting for
enhancing keyboard macros and what's the most general thing we can do
as well. Personally the largest issue I had, which I wasn't able to
completely solve, was keeping things local to a given kmacro. There's
no way I can tell to have some kind of local state for a kmacro. So
the most useful enhancement for me, I think, would be to have some
kind of explicitly local-to-kmacro variable. I imagine just some
specially named variable that could be set when the kmacro is defined,
and referenced when the kmacro is running. So if you wanted to you
could even turn it into an obarray and have a kmacro-local namespace.

> > So this, at least, is WAI and this bug should be closed.
>
> WAI?  That's a new one on me!
>
> Possibly a new bug should be opened to implement my suggestion above.

For reference, here's what I ended up doing with the advice:
https://pastebin.com/wG9L7xRb

Cheers,
-- Morgon