Native compilation of Keyboard Macros

Native compilation of Keyboard Macros

[Philip Kaludercic]

Is it possible to use native compilation to speed up the execution of
keyboard macros (or is this already being done)?  I think that there are
certain situations where a macro is executed many times over (C-x C-k r,
C-0 C-x e) that can slow down Emacs noticeably, where JITing could help.

Re: Native compilation of Keyboard Macros

[Stefan Monnier]

Philip Kaludercic [2022-09-16 18:56:19] wrote:
> Is it possible to use native compilation to speed up the execution of
> keyboard macros (or is this already being done)?  I think that there are
> certain situations where a macro is executed many times over (C-x C-k r,
> C-0 C-x e) that can slow down Emacs noticeably, where JITing could help.

I think it's somewhere between difficult and impossible.
The problem is not in the *native* part but in the transformation of the
keyboard macro into code: a keyboard macro is defined as a sequence of
events, and each one of those events can cause execution of a different
command every time it's repeated.

Also, I think that if `C-0 C-x e` is too slow, it's usually either:

- The few commands that are executed just take a lot of time
  (native-compilation of those commands can help but that's not specific
  to the case where they're run via a keyboard macro).
- The `C-x 0 C-x e` looping itself is costly, but most of that is
  execution of the repl (the lookup through keymaps, the pre/post
  command hooks, processing `interactive`, etc...) most of which is
  already coded in C.  In that case, turning the keyboard macro into
  "plain" ELisp code would bring a significant performance increase,
  regardless if it's (native) compiled, but it's almost impossible to do
  in general.

So, I think you should focus on the problem of turning a keyboard macro
into code.  We know it's impossible in general, so it has to be
"user-visible" (i.e. the user specifically asks for it and is made aware
that the result is a chunk of code which will not always run the same
commands as the keyboard macro).

You could have a "run kmacro and turn *this* execution into ELisp code".
IIUC some people tried to do that already, but I can't remember where
I saw that and my search-fu is weak right now.


        Stefan

Re: Native compilation of Keyboard Macros

[Juri Linkov]

> So, I think you should focus on the problem of turning a keyboard macro
> into code.  We know it's impossible in general, so it has to be
> "user-visible" (i.e. the user specifically asks for it and is made aware
> that the result is a chunk of code which will not always run the same
> commands as the keyboard macro).
>
> You could have a "run kmacro and turn *this* execution into ELisp code".
> IIUC some people tried to do that already, but I can't remember where
> I saw that and my search-fu is weak right now.

The last executed commands are already remembered at different levels,
so the hard task is to decide at what level to turn them to code:

1. at level of complex command execution, such as from the history
   retrieved by `repeat-complex-command` (C-x ESC ESC), for example:

   (rgrep "macro" "* .*" "emacs/lisp/")

2. at level of commands corresponding to all typed keys,
   such as retrieved by `view-lossage` (C-h l),
   for example, for the same as above:

   M-x                ;; execute-extended-command
   r                  ;; self-insert-command
   g                  ;; self-insert-command
   r                  ;; self-insert-command
   <tab>              ;; minibuffer-complete
   <return>           ;; minibuffer-complete-and-exit
   m                  ;; self-insert-command
   a                  ;; self-insert-command
   c                  ;; self-insert-command
   r                  ;; self-insert-command
   o                  ;; self-insert-command
   <return>           ;; exit-minibuffer
   <return>           ;; exit-minibuffer
   <return>           ;; minibuffer-complete-and-exit

Re: Native compilation of Keyboard Macros

[Philip Kaludercic]

Juri Linkov <juri@linkov.net> writes:

>> So, I think you should focus on the problem of turning a keyboard macro
>> into code.  We know it's impossible in general, so it has to be
>> "user-visible" (i.e. the user specifically asks for it and is made aware
>> that the result is a chunk of code which will not always run the same
>> commands as the keyboard macro).
>>
>> You could have a "run kmacro and turn *this* execution into ELisp code".
>> IIUC some people tried to do that already, but I can't remember where
>> I saw that and my search-fu is weak right now.
>
> The last executed commands are already remembered at different levels,
> so the hard task is to decide at what level to turn them to code:
>
> 1. at level of complex command execution, such as from the history
>    retrieved by `repeat-complex-command` (C-x ESC ESC), for example:
>
>    (rgrep "macro" "* .*" "emacs/lisp/")
>
> 2. at level of commands corresponding to all typed keys,
>    such as retrieved by `view-lossage` (C-h l),
>    for example, for the same as above:
>
>    M-x                ;; execute-extended-command
>    r                  ;; self-insert-command
>    g                  ;; self-insert-command
>    r                  ;; self-insert-command
>    <tab>              ;; minibuffer-complete
>    <return>           ;; minibuffer-complete-and-exit
>    m                  ;; self-insert-command
>    a                  ;; self-insert-command
>    c                  ;; self-insert-command
>    r                  ;; self-insert-command
>    o                  ;; self-insert-command
>    <return>           ;; exit-minibuffer
>    <return>           ;; exit-minibuffer
>    <return>           ;; minibuffer-complete-and-exit

Intuitively I would expect the first level to make more sense, though I
see that when considering commands like `self-insert-command' you
couldn't just deduce all the command invocations and wrap them in a
`progn' body.  At the same time, what is there to compile on the second
level?  You still need to look up all the commands and invoke them.  As
Stefan said, it is perhaps necessary to either detect what commands can
be safely converted into repeat-complex-command'-esque representations
(and what commands can be assisted/transformed), and only optimise if
that is possible.

Re: Native compilation of Keyboard Macros

[Juri Linkov]

>> 1. at level of complex command execution, such as from the history
>>    retrieved by `repeat-complex-command` (C-x ESC ESC), for example:
>>
>>    (rgrep "macro" "* .*" "emacs/lisp/")
>>
>> 2. at level of commands corresponding to all typed keys,
>>    such as retrieved by `view-lossage` (C-h l),
>>    for example, for the same as above:
>>
>>    M-x                ;; execute-extended-command
>>    r                  ;; self-insert-command
>>    g                  ;; self-insert-command
>>    r                  ;; self-insert-command
>>    <tab>              ;; minibuffer-complete
>
> Intuitively I would expect the first level to make more sense, though I
> see that when considering commands like `self-insert-command' you
> couldn't just deduce all the command invocations and wrap them in a
> `progn' body.  At the same time, what is there to compile on the second
> level?  You still need to look up all the commands and invoke them.  As
> Stefan said, it is perhaps necessary to either detect what commands can
> be safely converted into repeat-complex-command'-esque representations
> (and what commands can be assisted/transformed), and only optimise if
> that is possible.

Keyboard macros are executed on the keys level, so even the decision
what complex command to execute might depend on the position of that
command in the completion list of `M-x', e.g.:

      M-x        ;; execute-extended-command
      r          ;; self-insert-command
      <tab>      ;; minibuffer-complete
      <kp-prior> ;; switch-to-completions
      <tab>      ;; next-completion
      <tab>      ;; next-completion
      <tab>      ;; next-completion
      <tab>      ;; next-completion
      <return>   ;; choose-completion

Re: Native compilation of Keyboard Macros

[Philip Kaludercic]

Juri Linkov <juri@linkov.net> writes:

>>> 1. at level of complex command execution, such as from the history
>>>    retrieved by `repeat-complex-command` (C-x ESC ESC), for example:
>>>
>>>    (rgrep "macro" "* .*" "emacs/lisp/")
>>>
>>> 2. at level of commands corresponding to all typed keys,
>>>    such as retrieved by `view-lossage` (C-h l),
>>>    for example, for the same as above:
>>>
>>>    M-x                ;; execute-extended-command
>>>    r                  ;; self-insert-command
>>>    g                  ;; self-insert-command
>>>    r                  ;; self-insert-command
>>>    <tab>              ;; minibuffer-complete
>>
>> Intuitively I would expect the first level to make more sense, though I
>> see that when considering commands like `self-insert-command' you
>> couldn't just deduce all the command invocations and wrap them in a
>> `progn' body.  At the same time, what is there to compile on the second
>> level?  You still need to look up all the commands and invoke them.  As
>> Stefan said, it is perhaps necessary to either detect what commands can
>> be safely converted into repeat-complex-command'-esque representations
>> (and what commands can be assisted/transformed), and only optimise if
>> that is possible.
>
> Keyboard macros are executed on the keys level, so even the decision
> what complex command to execute might depend on the position of that
> command in the completion list of `M-x', e.g.:
>
>       M-x        ;; execute-extended-command
>       r          ;; self-insert-command
>       <tab>      ;; minibuffer-complete
>       <kp-prior> ;; switch-to-completions
>       <tab>      ;; next-completion
>       <tab>      ;; next-completion
>       <tab>      ;; next-completion
>       <tab>      ;; next-completion
>       <return>   ;; choose-completion

OK, but wouldn't that exclude the possibility of it being compiled,
*unless* there were some way to translate the rest of the input into
something else.

Re: Native compilation of Keyboard Macros

[Stefan Monnier]

> OK, but wouldn't that exclude the possibility of it being compiled,
> *unless* there were some way to translate the rest of the input into
> something else.

You can't "transparently" compile them, no.  But the user can ask
explicitly for a kind of "snapshot": compile them to the code
executed for a particular run of that keyboard macro.

In any case, as for all optimization problems, I'd start by
investigating your very concrete example to try and figure exactly where
the time is spent.  Else, you may end up missing the important spot.


        Stefan

Re: Native compilation of Keyboard Macros

[Visuwesh]

[வெள்ளி செப்டம்பர் 16, 2022] Stefan Monnier wrote:

> You could have a "run kmacro and turn *this* execution into ELisp code".
> IIUC some people tried to do that already, but I can't remember where
> I saw that and my search-fu is weak right now.

Are you thinking of https://github.com/Silex/elmacro perhaps?

Re: Native compilation of Keyboard Macros

[Stefan Monnier]

>> You could have a "run kmacro and turn *this* execution into ELisp code".
>> IIUC some people tried to do that already, but I can't remember where
>> I saw that and my search-fu is weak right now.
>
> Are you thinking of https://github.com/Silex/elmacro perhaps?

Yes, thanks!


        Stefan

Re: Native compilation of Keyboard Macros

[Óscar Fuentes]

Philip Kaludercic <philipk@posteo.net> writes:

> Is it possible to use native compilation to speed up the execution of
> keyboard macros (or is this already being done)?

It would be a big surprise to me if that would achieve a perceptible
improvement.

Re: Native compilation of Keyboard Macros

[Philip Kaludercic]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Is it possible to use native compilation to speed up the execution of
>> keyboard macros (or is this already being done)?
>
> It would be a big surprise to me if that would achieve a perceptible
> improvement.

I don't know either, but when repeating a simple macro over and over
again it took me long enough to wonder if native compilation would help.

(Specifically what I was doing was going through a text file that I
wanted to convert into HTML file by adding some fluff to each line.  The
file was over a megabyte of plain text so it took a while.  I guess it
would be faster to use pipe the buffer contents through AWK or something
similar).

Re: Native compilation of Keyboard Macros

[Óscar Fuentes]

Philip Kaludercic <philipk@posteo.net> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> Is it possible to use native compilation to speed up the execution of
>>> keyboard macros (or is this already being done)?
>>
>> It would be a big surprise to me if that would achieve a perceptible
>> improvement.
>
> I don't know either, but when repeating a simple macro over and over
> again it took me long enough to wonder if native compilation would help.

The key point is not the "over-and-over", but where the real work is
done, which most likely is already compiled (either by nativecomp or by
the C compiler.)