recording-elisp.el - try recording commands as elisp code

recording-elisp.el - try recording commands as elisp code

[yzhh]

Hi,

This recording.el is maybe some useful results from our discussion on
recording kbd macro as elisp results. I'm now posting it not because it is
perfectly usable, but to trigger some ideas on improvements.

I started from a patch Juri gave to me. In his code there are modifications
in the C code and in kmacro.el and isearch.el. I recognize that the C code
patch is a must to get command args, and left them as is. 

But I think it is better to implement the recording outside other elisp
source. So I pulled his code out of those el files and wrote
recording-elisp.el. I also made further modifications to handle isearch
commands better.

If anybody is interested to give it a try. it is quite simple: M-x
start-recording-commands, do something, M-x end-recording-commands, and
then M-x insert-recorded-commands into current buffer.

Currently it is just a proof o f concept patch. It does not handle multiple
query replace, multiple cursor movement, etc. But it do compress multiple
character input into one insert, and gives quite short code for a fairly
long isearch session.

I really need help from some old-hand on elisp to push it further. It's even
better if some experienced developer want to take it over.

The patch is here:

Index: src/keyboard.c
===================================================================
--- src/keyboard.c      (revision 4)
+++ src/keyboard.c      (revision 9)
@@ -371,6 +371,9 @@
    command is stored in this-original-command.  It is nil otherwise.  */
 Lisp_Object Vthis_original_command;
 
+/* The arguments of the command being executed by the command loop.  */
+Lisp_Object Vthis_command_args;
+
 /* The value of point when the last command was started.  */
 int last_point_position;
 
@@ -1765,6 +1768,7 @@
 
       Vthis_command = cmd;
       real_this_command = cmd;
+      Vthis_command_args = Qnil;
       /* Note that the value cell will never directly contain nil
         if the symbol is a local variable.  */
       if (!NILP (Vpre_command_hook) && !NILP (Vrun_hooks))
@@ -1798,6 +1802,7 @@
                    = window_display_table (XWINDOW (selected_window));
                  lose = FETCH_CHAR (PT_BYTE);
                  SET_PT (PT + 1);
+                 Vthis_command_args = Fcons (Vcurrent_prefix_arg, Qnil);
                  if (! NILP (Vpost_command_hook))
                    /* Put this before calling adjust_point_for_property
                       so it will only get called once in any case.  */
@@ -1838,6 +1843,7 @@
                    = window_display_table (XWINDOW (selected_window));
                  SET_PT (PT - 1);
                  lose = FETCH_CHAR (PT_BYTE);
+                 Vthis_command_args = Fcons (Vcurrent_prefix_arg, Qnil);
                  if (! NILP (Vpost_command_hook))
                    goto directly_done;
                  if (current_buffer == prev_buffer
@@ -1905,6 +1911,7 @@
                  if (value == 2)
                    nonundocount = 0;
 
+                 Vthis_command_args = Fcons (make_number (c), Fcons
(Vcurrent_prefix_arg, Qnil));
                  if (! NILP (Vpost_command_hook))
                    /* Put this before calling adjust_point_for_property
                       so it will only get called once in any case.  */
@@ -3183,7 +3190,8 @@
 #endif
       
       last_input_char = c;
-      Fcommand_execute (tem, Qnil, Fvector (1, &last_input_char), Qt);
+      Fcommand_execute (tem, Qnil, /* Qt, */
+                     Fvector (1, &last_input_char), Qt);
 
       if (CONSP (c) && EQ (XCAR (c), Qselect_window) && !end_time)
        /* We stopped being idle for this event; undo that.  This
@@ -12004,6 +12012,10 @@
 result of looking up the original command in the active keymaps.  */);
   Vthis_original_command = Qnil;
 
+  DEFVAR_LISP ("this-command-args", &Vthis_command_args,
+              doc: /* Arguments of the command being executed.  */);
+  Vthis_command_args = Qnil;
+
   DEFVAR_INT ("auto-save-interval", &auto_save_interval,
              doc: /* *Number of input events between auto-saves.
 Zero means disable autosaving due to number of characters typed.  */);
Index: src/callint.c
===================================================================
--- src/callint.c       (revision 4)
+++ src/callint.c       (revision 9)
@@ -42,6 +42,7 @@
 
 extern Lisp_Object Vhistory_length;
 extern Lisp_Object Vthis_original_command, real_this_command;
+extern Lisp_Object Vthis_command_args;
 
 Lisp_Object Vcommand_debug_status, Qcommand_debug_status;
 Lisp_Object Qenable_recursive_minibuffers;
@@ -364,6 +365,9 @@
             and turn them into things we can eval.  */
          values = quotify_args (Fcopy_sequence (specs));
          fix_command (input, values);
+
+         Vthis_command_args = values;
+
          Vcommand_history
            = Fcons (Fcons (function, values), Vcommand_history);
 
@@ -808,6 +812,9 @@
          else
            visargs[i] = quotify_arg (args[i]);
        }
+
+      Vthis_command_args = XCDR (Flist (count + 1, visargs));
+
       Vcommand_history = Fcons (Flist (count + 1, visargs),
                                Vcommand_history);
       /* Don't keep command history around forever.  */
@@ -832,6 +839,8 @@
   Vthis_original_command = save_this_original_command;
   real_this_command= save_real_this_command;
   current_kboard->Vlast_command = save_last_command;
+  if (NILP (Vthis_command_args))
+    Vthis_command_args = XCDR (Flist (count + 1, args));
 
   {
     Lisp_Object val;
Index: lisp/recording-elisp.el
===================================================================
--- lisp/recording-elisp.el     (revision 0)
+++ lisp/recording-elisp.el     (revision 9)
@@ -0,0 +1,174 @@
+;;; recording-elisp.el --- recording emacs commands
+
+;;; Commentary:
+
+;; This recording-elisp.el provides the user interface to record a
+;; sequence of commands executed (in response to a sequence of user
+;; interaction) by emacs, and to provide the recorded commands in the
+;; form of emacs-lisp code. 
+
+;; To start recording commands, use M-x start-recording-commands.  To
+;; end, use M-x end-recording-commands.  Then M-x
+;; insert-recorded-commands will insert the recorded commands into
+;; current buffer at current point.
+
+;; Note: the output of insert-recorded-commands will not always be the
+;; same as the sequence of commands actually recorded.  Post
+;; processing of the commands are performed to make the output more
+;; readable and useful for further editing.  But the functionality of
+;; the resulting code is kept the same as the original commands,
+;; whenever possible and reasonable.
+
+(defvar recorded-commands nil
+  "List of recorded commands executed during macro definition.
+Each command is represented as a form to evaluate.")
+
+(defvar recording-commands nil
+  "Bool value indicating whether commands are being recorded.")
+
+(defun recording-isearch-push-state-function ()
+  "Function to be set as value of isearch-push-state-fuction.
+It returns value of isearch-regexp when called, which is then
+Stored in isearch state stack. We need this value to correctly
+recognize isearch operations given (only) its state stack."
+  isearch-regexp)
+  
+(defun start-recording-commands ()
+  "Start recording commands executed by emacs."
+  (interactive)
+  (add-hook 'post-command-hook 'recording-command t)
+  ; Set isearch-push-state-fuction only when it is not already set.
+  (if (not isearch-push-state-function)
+      (setq
isearch-push-state-function 'recording-isearch-push-state-function))
+  (setq recorded-commands nil)
+  (setq recording-commands t))
+
+(defun end-recording-commands ()
+  "End recording commands."
+  (interactive)
+  (remove-hook 'post-command-hook 'recording-command)
+  ; Clear isearch-push-state-fuction only when it has not been
+  ; set during the recording.
+  (if (eq
isearch-push-state-function 'recording-isearch-push-state-function)
+      (setq isearch-push-state-function nil))
+  (setq recording-commands nil))
+
+(defun recording-command ()
+  "This function is added to post-command-hook to perform the recording."
+  (unless (active-minibuffer-window)
+  (cond 
+   ((eq this-command 'isearch-exit)
+    (setq this-command-args isearch-cmds))) ; a fake isearch-exit command
+  
+  (setq recorded-commands (cons (cons this-command this-command-args)
+                               recorded-commands))))
+
+(defun insert-recorded-commands ()
+  (interactive)
+  (insert (pp-to-string
+          `(defun recorded-commands ()
+             "Commands recorded during the last 'recording commands' cycle."
+             (interactive)
+             ,@(convert-recorded-commands recorded-commands)))))
+
+(defun convert-recorded-commands (commands)
+  (let ((cmds commands) cmd name ret)
+    (while cmds
+      (setq cmd (car cmds))
+      (setq name (car cmd))
+      (cond
+       ;; skip next commands
+       ((memq name '(start-recording-commands
+                    universal-argument universal-argument-other-key
+                    digit-argument
+                    isearch-forward isearch-backward
+                    isearch-forward-regexp isearch-backward-regexp
+                    isearch-printing-char isearch-other-control-char
+                    isearch-repeat-forward isearch-repeat-backward
+                    isearch-delete-char isearch-del-char
+                    isearch-yank-word-or-char isearch-yank-char
+                    isearch-yank-line isearch-yank-kill
+                    isearch-edit-string 
+                    isearch-query-replace isearch-query-replace-regexp
+                    isearch-ring-advance isearch-ring-retreat
+                    isearch-complete 
+                    isearch-toggle-input-method
+                    isearch-toggle-specified-input-method
+                    isearch-abort
+                    )))
+
+       ((eq name 'isearch-exit) 
+       (setq ret (append (convert-isearch-cmds (cdr cmd)) ret)))
+
+       ((eq name 'self-insert-command)
+       (let* ((prev-cmd (if ret (car ret) nil))
+              (prev-cmd-name (if prev-cmd (car prev-cmd) nil)))
+         (if (eq prev-cmd-name 'insert)
+           (setq ret (cons
+                      `(insert ,(concat (nth 1 prev-cmd) (string (nth 1 cmd))))
+                      (cdr ret)))
+         (push `(insert ,(string (nth 1 cmd))) ret))))
+
+       (t (push cmd ret)))
+
+      (setq cmds (cdr cmds)))
+    ret))
+
+(defun convert-isearch-cmds (isearch-state-stack)
+  (let ((isch-state-list (reverse isearch-state-stack))
+       (result '()))
+    (when (and isch-state-list
+              (cdr isch-state-list))
+      (let ((prev-state nil)
+           (this-state (car isch-state-list))
+           isch-string regexp forward)
+       (setq isch-state-list (cdr isch-state-list))     
+       (while isch-state-list
+         (setq prev-state this-state)
+         (setq this-state (car isch-state-list))
+         (setq isch-state-list (cdr isch-state-list))
+         (cond 
+          ; 1. Ignore no-movement steps.
+          ((eq (isearch-point-state this-state)
+               (isearch-point-state prev-state)))
+
+          ; 2. The point moved but barrier didn't, real incremental
+          ; search happened.  We only record the last one of such a 
+          ; series, in 3.  Here we just pass.
+          ((eq (isearch-barrier-state this-state)
+               (isearch-barrier-state prev-state)))
+
+          ; 3. Other cases: repeated search or search with modified
+          ; string.  We record the previous-state, this-state may get
+          ; ignored on next loop, in 1 or 2.
+          (t
+           (setq regexp (isearch-pop-fun-state prev-state))
+           (setq forward (isearch-forward-state prev-state))
+           (setq isch-string (isearch-string-state prev-state))
+           (setq result 
+                 (append result 
+                         `(,(make-search-command
+                             forward regexp isch-string)))))))
+ 
+       ;; Recording the last state is always necessary.
+       (setq result 
+             (append result
+                     `(,(make-search-command
+                          (isearch-forward-state this-state)
+                          (isearch-pop-fun-state this-state) ; regexp flag
+                          (isearch-string-state this-state)))))))
+    result))
+        
+(defun make-search-command (forward regexp search-string)
+  "Return a quoted search command. The args sets the search type and
string."
+  (cond (forward
+        (cond (regexp
+               `(search-forward-regexp ,search-string))
+              (t
+               `(search-forward ,search-string))))
+       (t
+        (cond (regexp
+               `(search-backward-regexp ,search-string))
+              (t
+               `(search-backward ,search-string))))))
+

-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Juri Linkov]

> This recording.el is maybe some useful results from our discussion on
> recording kbd macro as elisp results. I'm now posting it not because it is
> perfectly usable, but to trigger some ideas on improvements.

Thanks.

> I started from a patch Juri gave to me. In his code there are modifications
> in the C code and in kmacro.el and isearch.el. I recognize that the C code
> patch is a must to get command args, and left them as is.

I think I should install my C patch that adds a new variable this-command-args.
This will allow you to continue freely implementing this feature.

> But I think it is better to implement the recording outside other elisp
> source. So I pulled his code out of those el files and wrote
> recording-elisp.el. I also made further modifications to handle isearch
> commands better.

I suggest you to give a shorter name to this package (e.g. record-lisp.el)
and to use the same prefix for all function and variable names.

> If anybody is interested to give it a try. it is quite simple: M-x
> start-recording-commands, do something, M-x end-recording-commands, and
> then M-x insert-recorded-commands into current buffer.

It's fine that you implement separate commands to record Lisp commands
with their arguments since actually this feature is not dependent
on keyboard macros.  But I think `C-x (' and `C-x )' still should record
Lisp commands too, because these keybindings are convenient to start and
end recording of command sequences, even though they record different
things: one feature records a keyboard macro, and another - Lisp code.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Juri Linkov wrote:

>Thanks.

I owe you a thank you for give it a start.

> I think I should install my C patch that adds a new variable
> this-command-args. This will allow you to continue freely implementing
> this feature.

Do you mean you'll make the C patch available on CVS? I don't catch the
exact meaning of the word 'install' here.

> I suggest you to give a shorter name to this package (e.g. record-lisp.el)
> and to use the same prefix for all function and variable names.

Yes I thought about this. But "record-lisp-" looks too long as a prefix
while "record-" is too general a word that can be mistaken for other
things. I've not come up with a good one yet.

Any suggestion are welcome here regarding the names.

> It's fine that you implement separate commands to record Lisp commands
> with their arguments since actually this feature is not dependent
> on keyboard macros.  But I think `C-x (' and `C-x )' still should record
> Lisp commands too, because these keybindings are convenient to start and
> end recording of command sequences, even though they record different
> things: one feature records a keyboard macro, and another - Lisp code.

I've not written code to bind keys to these recording commands. I'd leave
decision about the default binding to the emacs developers. Currently using
seperate keys is easier. Doing both together would need some tricky code.
Recording a replayed kbd macro is a partial solution for the moment.

-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Juri Linkov]

>> I think I should install my C patch that adds a new variable
>> this-command-args. This will allow you to continue freely implementing
>> this feature.
>
> Do you mean you'll make the C patch available on CVS? I don't catch the
> exact meaning of the word 'install' here.

Yes, by `install' I meant to check in the C patch into CVS.

>> I suggest you to give a shorter name to this package (e.g. record-lisp.el)
>> and to use the same prefix for all function and variable names.
>
> Yes I thought about this. But "record-lisp-" looks too long as a prefix
> while "record-" is too general a word that can be mistaken for other
> things. I've not come up with a good one yet.
>
> Any suggestion are welcome here regarding the names.

For instance, `reclisp-' is a short prefix.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Juri Linkov wrote:

> Yes, by `install' I meant to check in the C patch into CVS.

That's good news. Please notify me when that happens. 

BTW, I'd like to know the names of emacs developers who have commit
permission to the CVS, to have an idea who I'm talking to.  I haven't find
such information on emacs site.

> For instance, `reclisp-' is a short prefix.

Looks good, I'll take this, and 'record-lisp.el'.

-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Reiner Steib]

On Sat, Nov 03 2007, yzhh wrote:

> BTW, I'd like to know the names of emacs developers who have commit
> permission to the CVS, to have an idea who I'm talking to.  I haven't find
> such information on emacs site.

<http://savannah.gnu.org/projects/emacs/>, -> Membership Info -> View
Members: <http://savannah.gnu.org/project/memberlist.php?group=emacs>

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Reiner Steib wrote:

> <http://savannah.gnu.org/projects/emacs/>, -> Membership Info -> View
> Members: <http://savannah.gnu.org/project/memberlist.php?group=emacs>

Thank you very much.
-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

Is it a good start.  Can it work for replaying a keyboard macro?

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Richard Stallman wrote:

> Is it a good start.  Can it work for replaying a keyboard macro?

Yes. M-x start-recording-commands, C-x e, M-x stop-recording-commands does
the trick.

-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

    > Is it a good start.  Can it work for replaying a keyboard macro?

    Yes. M-x start-recording-commands, C-x e, M-x stop-recording-commands does
    the trick.

That combination needs to be made more convenient.

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Richard Stallman wrote:

>     Yes. M-x start-recording-commands, C-x e, M-x stop-recording-commands
>     does the trick.
> 
> That combination needs to be made more convenient.
I'll write a command for this after I understand kbd macro replaying
commands.
-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

A question:

To get good isearch recording I used isearch internal information (a stack
of intermidiate states left on isearch-exit). This looks a bad practice
since it's prone to isearch implementation changes. 

But the situation is: it cannot be done without knowledge about it's
internals. Recording every isearch command does not work because many of
them do not have args (i.e. they've not been designed to be recorded). And
any attempt to transform them to be shorter relies on isearch
implementation. The "isearch state stack " is just another kind of interal
knowledge and I choose it for simplicity.

It's quite contradictory. If I give up relying on internals the recorded
result looks bad. If I insist this style there are other isearch-alike
commands waiting - I simply can not and should not handle them all.

-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Juri Linkov]

> To get good isearch recording I used isearch internal information (a stack
> of intermidiate states left on isearch-exit). This looks a bad practice
> since it's prone to isearch implementation changes.
>
> But the situation is: it cannot be done without knowledge about it's
> internals. Recording every isearch command does not work because many of
> them do not have args (i.e. they've not been designed to be recorded). And
> any attempt to transform them to be shorter relies on isearch
> implementation. The "isearch state stack " is just another kind of interal
> knowledge and I choose it for simplicity.
>
> It's quite contradictory. If I give up relying on internals the recorded
> result looks bad. If I insist this style there are other isearch-alike
> commands waiting - I simply can not and should not handle them all.

I think it is enough to convert the isearch exit event with the
last search string to the calls of search-forward, search-backward,
re-search-forward and re-search-backward.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Juri Linkov wrote:

> I think it is enough to convert the isearch exit event with the
> last search string to the calls of search-forward, search-backward,
> re-search-forward and re-search-backward.

Maybe you are right, since the recorded code is meant to be modified. And
there are word-searches, isearch-case-fold-search that should be
considered.

Can we setup a vote for a policy about this (balance between simplicity and
correctness)? 

-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

It was suggested:

    > I think it is enough to convert the isearch exit event with the
    > last search string to the calls of search-forward, search-backward,
    > re-search-forward and re-search-backward.

That is a good approach for the simple cases.  But if the user yanks
into the search, it is important to at least try to reproduce the
effect of the yanking.  Even if it can't work right in all cases, at
least it should work right in simple cases.

Likewise, if a macro does a search with C-s C-s (using the default
from before the macro), it is far better to generate a program that
will use the default, rather than one which uses a constant.

The same goes for minibuffers.  When someone uses completion in a
minibuffer, there are many cases where using the end result as a
constant is best, and perhaps that is always good; but there could be
some cases where it is important to repeat the completion operation.
(I am not sure.)  And yanking into the minibuffer ought to be repeated.

As regards isearch-case-fold-search, I think it is ok to ignore that.
The generated code should simply call a search function, which will
obey case-fold-search.  I think this will produce the most useful Lisp
code, in terms of practical use.

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Richard Stallman wrote:

> It was suggested:
> 
>     > I think it is enough to convert the isearch exit event with the
>     > last search string to the calls of search-forward, search-backward,
>     > re-search-forward and re-search-backward.
> 
> That is a good approach for the simple cases.  But if the user yanks
> into the search, it is important to at least try to reproduce the
> effect of the yanking.  Even if it can't work right in all cases, at
> least it should work right in simple cases.

In the case of iserach. I think it might be a good approach to output a
constant string search when the yanked string is not killed during the
recording, and to output a yank-in-the-search when the killing is also
recorded.

However, two things I mentioned above are difficult to do right:
1.  "recognize whether a string is killed during the recording". There are
quite a few sources for yanking, including the whole kill-ring. Tracking
them all is asking for trouble.
2. "output a yank-in-the-search". That is actually to require reproducing
the whole isearch session, since the yanking can happen anytime during
isearch, and can be surrounded by other search string editing. That's
really really difficult - the required information is NOT ALL available
from recorded isearch commands and the isearch state stack.

My current implementation does neither. To illustrate, let's see what we get
for this isearch session (3 searches are all successful):
C-s abc M-y(yank "def" from kill-ring) C-s RET

Juri's approach get: (search-forward "abcdef")
My implementation get: (search-forward "abc") (search-forward "abcdef"
(search-forward "abcdef")
rms's approach want: (isearch-forward) (isearch-append-string "abc")
(isearch-yank-kill) (isearch-repeat-forward) (isearch-exit)

Well, isearch-append-string is imagined, the most close thing we have is
isearch-printing-char (WITH NO ARGS). AFAIK, this and
other "lack-of-support" reason rendered rms's approach impossible for now.

What do we really want here?

> Likewise, if a macro does a search with C-s C-s (using the default
> from before the macro), it is far better to generate a program that
> will use the default, rather than one which uses a constant.

That might be "kbd macro" thinking. For recorded lisp code, it's better to
modify it to receive an arg "interactively", than to receive it from
previous prepared "runtime environment" (in this case, the latest search
string).

If you're talking about kill-and-yank during the recording, that's
reasonable. Then my analysis above and bellow apply.

> The same goes for minibuffers.  When someone uses completion in a
> minibuffer, there are many cases where using the end result as a
> constant is best, and perhaps that is always good; but there could be
> some cases where it is important to repeat the completion operation.
> (I am not sure.)  And yanking into the minibuffer ought to be repeated.

We may provide an option "recording minibuffer editing on/off", or even
provide commands to switch it during the recording. But the real challenge
is not recording but replaying.  Example: M-x previ TAB line RET
The recorded commands:
    (execute-extended-command nil)
    (insert "previ")
    (minibuffer-complete)
    (insert "line")
    (previous-line 1 1)
can not be replayed directly, because running the first command will prompt
the user for input instead of continue running. This is a foundmental
difference between kbd macro and lisp code. And I don't known how to
do "call-interactively without promting the user" from code. 

Another issue is with recognizing (previous-line ...) as "to be deleted" to
avoid doing it twice. We'll have to put some flag on this recorded command
at recording time, and this flag depends on a previous
(execute-extendied-command nil) plus 'leaving minibuffer' event.

> As regards isearch-case-fold-search, I think it is ok to ignore that.
> The generated code should simply call a search function, which will
> obey case-fold-search.  I think this will produce the most useful Lisp
> code, in terms of practical use.

If one toggles case-fold during isearch, I believe he does it for a reason
and we'd better record it. Currently the generated code looks like this:
(defun recorded-command ()
  (interactive)
  (let
      ((case-fold case-fold-search))    ; save case-fold
    (setq case-fold-search t)           ; set to recorded value
    ...
    (search-forward "something") 
    (setq case-fold-search (not case-fold-search))      ; user toggles case-fold
    (search-forward "SOMETHING")                        ; last isearch string
    (setq case-fold-search (not case-fold-search))      ; toggle it back
    ...
    (setq case-fold-search case-fold))) ; restore case-fold

It easy to do without the "case-fold" lines, though. Or another option can
be used to turn this on/off.

-- 
   regards,
yzhh

RE: recording-elisp.el - try recording commands as elisp code

[Drew Adams]

I have not followed all the details in this thread. I have the impression
that the discussion now is mostly about how to do the best thing when
dealing with various interactive dialogs (isearch, completion, recursive
minibuffers,...). I suspect there is no one simple answer here.

There seem to be several viable interpretations of TRT, depending on the
context and user goals and preferences. If so, then, once the discussion has
settled down a bit and there is some agreement on TRT in general, please
consider also letting the user have a say interactively, perhaps by toggling
an option or two on the fly (during macro recording). The user's choice
could affect both the macro itself (replay) and the recorded Lisp code.

That is, assuming that there is no simple one-size-fits-all solution, let's
avoid hard-coding a smart, reasonable algorithm that fits many situations,
if we could (easily) also provide that default behavior plus alternative
behaviors that a user might prefer, depending on the situation. IOW, instead
of hard-coding TRT, let the user decide TRT on the spot (if possible).

IOW, consider promoting some of the design discussion you are having now to
interactive user choices. Just a suggestion to keep in mind. I have no idea
how feasible this is.

RE: recording-elisp.el - try recording commands as elisp code

[yzhh]

Drew Adams wrote:

> IOW, consider promoting some of the design discussion you are having now
> to interactive user choices. Just a suggestion to keep in mind. I have no
> idea how feasible this is.

I agree with you on offering choices to users, both custom options and
recording time interactions. So we are supposed to discuss what choices to
offer.

Allow me repeat them here (to make it clearer):
1. three choices for recording one isearch session:
    a. Record the last isearch item as a constant string search
    b. Record all successful isearch item as constant string searches
    c. Record all interactions, (especially) including yanking into the
search string.
2. two choices for recording minibuffer interactions:
    a. Totally ignore them, recording only the final command (this is for
M-x interactions, I haven't thought clearly about others)
    b. Record all interactions so that we replay everything including
yanking into the minibuffer.
3. two choices for recording case-fold-search value:
    a. Totally ignore.
    b. Record (and replay) case-fold value in-use at start of recording, and
every toggle-case-fold command during that.

Please note that I have also expressed the "maybe" infeasibility of choices
1.c and 2.b (given currently available commands). I would deeply appreciate
if anyone suggest solutions.

-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

If you offer user choices, I think the only way to make that easy to use
is to generate output in one way (the default), then let the user point at
various places in the generated Lisp code and give commands that say
"generate this function call using a different method".

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Richard Stallman wrote:

> If you offer user choices, I think the only way to make that easy to use
> is to generate output in one way (the default), then let the user point at
> various places in the generated Lisp code and give commands that say
> "generate this function call using a different method".

Users should love this if done properly. One complication is "generate this
sequence of function calls using a different method." I mean we have to
design an intuitive way to indicate the user where he has choices, and
better show what choices he has.

I'd like to discuss this later when I've done a usable default version.
-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

    Users should love this if done properly. One complication is "generate this
    sequence of function calls using a different method." I mean we have to
    design an intuitive way to indicate the user where he has choices, and
    better show what choices he has.

Perhaps you can use `face' text properties to show the places where there is a
choice of strategies.  Then  commands such as C-c C-c and Mouse-3 could
be used to select among them.

Re: recording-elisp.el - try recording commands as elisp code

[Mathias Dahl]

> If you offer user choices, I think the only way to make that easy to use
> is to generate output in one way (the default), then let the user point at
> various places in the generated Lisp code and give commands that say
> "generate this function call using a different method".

Slightly off-topic, but this got me wondering if it would be useful
with a command similar to `checkdoc', something that would scan
through the code for various "mistakes", suggesting changes. For
example suggesting `replace-string' to be replaced by the loop
discussed earlier. Does something like this already exist? I would
imagine this would be useful to beginning elispers.

Re: recording-elisp.el - try recording commands as elisp code

[Juri Linkov]

> Slightly off-topic, but this got me wondering if it would be useful
> with a command similar to `checkdoc', something that would scan
> through the code for various "mistakes", suggesting changes. For
> example suggesting `replace-string' to be replaced by the loop
> discussed earlier. Does something like this already exist? I would
> imagine this would be useful to beginning elispers.

The byte-compiler already warns about using some interactive functions
from the list `byte-compile-interactive-only-functions' that includes
`replace-string' and `replace-regexp'.

Another task would be to write a converter that will fix existing Lisp
code that uses interactive functions and replace them with non-interactive
constructions.  Maybe some part of record-lisp.el will do this.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

    Slightly off-topic, but this got me wondering if it would be useful
    with a command similar to `checkdoc', something that would scan
    through the code for various "mistakes", suggesting changes. For
    example suggesting `replace-string' to be replaced by the loop
    discussed earlier. Does something like this already exist? I would
    imagine this would be useful to beginning elispers.

I don't know of anything like that interface, but it might be worth a
try.  It could be useful as an interface for various features.

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

    However, two things I mentioned above are difficult to do right:
    1.  "recognize whether a string is killed during the recording". There are
    quite a few sources for yanking, including the whole kill-ring. Tracking
    them all is asking for trouble.

You're being too perfectionist.
Please stop asking whether you can "track them all",
and instead work on tracking common cases.
I am sure you will succeed.  The only question is how far
you can carry it.

    rms's approach want: (isearch-forward) (isearch-append-string "abc")
    (isearch-yank-kill) (isearch-repeat-forward) (isearch-exit)

    Well, isearch-append-string is imagined, the most close thing we have is
    isearch-printing-char (WITH NO ARGS). AFAIK, this and
    other "lack-of-support" reason rendered rms's approach impossible for now.

We can add isearch-append-string for this.

    > Likewise, if a macro does a search with C-s C-s (using the default
    > from before the macro), it is far better to generate a program that
    > will use the default, rather than one which uses a constant.

    That might be "kbd macro" thinking. For recorded lisp code, it's better to
    modify it to receive an arg "interactively", than to receive it from
    previous prepared "runtime environment" (in this case, the latest search
    string).

That is a good idea.  Please do it that way.

The same argument would apply to yanking something that was not
killed within the same macro.

    We may provide an option "recording minibuffer editing on/off", or even
    provide commands to switch it during the recording. But the real challenge
    is not recording but replaying.  Example: M-x previ TAB line RET
    The recorded commands:
	(execute-extended-command nil)
	(insert "previ")
	(minibuffer-complete)
	(insert "line")
	(previous-line 1 1)
    can not be replayed directly, because running the first command will prompt
    the user for input instead of continue running.

The code to set up an argument in the minibuffer needs to be converted
into computing the argument to pass to a function.

However, mere completion in most cases can be treated as a constant.
Thus, if the user types M-x que SPC SPC SPC RET, there is no need
for your code to go through the routine of completing `que'.
You should just generate a call to `query-replace-region'.

The only possible exceptions to that statement are cases where the
effect of the completion is very dynamic.  I don't know whether any
such cases exist; I mention the possibility because I am not yet
convinced there are none.

Thus, the issue that complicates minibuffer args is not completion.
It is yanking.

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Richard Stallman wrote:

> You're being too perfectionist.

I am, from time to time. :) I prefer generic solution over ad hoc ones.

> Please stop asking whether you can "track them all",
> and instead work on tracking common cases.
> I am sure you will succeed.  The only question is how far
> you can carry it.

My progress will be slow, but I'll stick to this for a long time. My major
concern here is maintainability of the code. When details relating to other
emacs packages accumulate in one place, I could lose control of it. I do
want it last long.

Since your "common case" policy looks practical and the code is now far from
out-of-control, I'll do it that way.

> We can add isearch-append-string for this.

I'm still investigating work-arounds. I'd like to hear from isearch
developers about their idea on recording isearch, if that's appropriate.

> That is a good idea.  Please do it that way.
> 
> The same argument would apply to yanking something that was not
> killed within the same macro.

It depends on the above "yank sources tracking". So it'll be implemented
after that.

> The code to set up an argument in the minibuffer needs to be converted
> into computing the argument to pass to a function.
> 
> However, mere completion in most cases can be treated as a constant.
> ...
> The only possible exceptions to that statement are cases where the
> effect of the completion is very dynamic. ...
> ...
> Thus, the issue that complicates minibuffer args is not completion.
> It is yanking.

So let's apply the "common case" policy here and use completed string
constants for now. Later when yanking is dealt with, we can experiment on
completion.

-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Stefan Monnier]

> My progress will be slow, but I'll stick to this for a long time.  My major
> concern here is maintainability of the code.  When details relating to other
> emacs packages accumulate in one place, I could lose control of it.  I do
> want it last long.

It's always a source of problems, but before you accumulate a few solutions
for a few packages, it's often difficult to figure out what's the right way
to provide a generic functionality which packages can then use (so that the
details relating to each package get moved to the relevant package).


        Stefan

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Stefan Monnier wrote:

> It's always a source of problems, but before you accumulate a few
> solutions for a few packages, it's often difficult to figure out what's
> the right way to provide a generic functionality which packages can then
> use (so that the details relating to each package get moved to the
> relevant package).

Good reasoning. Thank you Stefan.

-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

    It's always a source of problems, but before you accumulate a few solutions
    for a few packages, it's often difficult to figure out what's the right way
    to provide a generic functionality which packages can then use (so that the
    details relating to each package get moved to the relevant package).

I find that implementing specific solutions for a few uses is often
the best way to figure out what general functionality is needed.
Theoretical thinking, not guided by practical experience, often leads
to solving the wrong problem.

Re: recording-elisp.el - try recording commands as elisp code

[Stefan Monnier]

>     It's always a source of problems, but before you accumulate a few
>     solutions for a few packages, it's often difficult to figure out
>     what's the right way to provide a generic functionality which packages
>     can then use (so that the details relating to each package get moved
>     to the relevant package).

> I find that implementing specific solutions for a few uses is often
> the best way to figure out what general functionality is needed.
> Theoretical thinking, not guided by practical experience, often leads
> to solving the wrong problem.

Great minds think alike ;-)


        Stefan

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

    > As regards isearch-case-fold-search, I think it is ok to ignore that.
    > The generated code should simply call a search function, which will
    > obey case-fold-search.  I think this will produce the most useful Lisp
    > code, in terms of practical use.

    If one toggles case-fold during isearch, I believe he does it for a reason
    and we'd better record it. Currently the generated code looks like this:

I agree.  I was talking about the ordinary case where you do an isearch
without specifying whether it is case-sensitive.

Re: recording-elisp.el - try recording commands as elisp code

[Juri Linkov]

> Juri's approach get: (search-forward "abcdef")
> My implementation get: (search-forward "abc") (search-forward "abcdef"
> (search-forward "abcdef")
> rms's approach want: (isearch-forward) (isearch-append-string "abc")
> (isearch-yank-kill) (isearch-repeat-forward) (isearch-exit)
>
> What do we really want here?

Another case that might help us to decide to what level to record
Lisp code is converting M-% (`query-replace').  There are three basic
levels of "interactivity":

1. convert it to the call to `query-replace', e.g.

    (query-replace "from" "to" nil
                   (if (and transient-mark-mode mark-active) (region-beginning))
                   (if (and transient-mark-mode mark-active) (region-end)))

   That's how it is recorded now to `command-history'.

2. convert M-% to the call to `replace-string';

3. but the doc-string of this command says:

    This function is usually the wrong thing to use in a Lisp program.
    What you probably want is a loop like this:
      (while (search-forward FROM-STRING nil t)
        (replace-match TO-STRING nil t))
    which will run faster and will not set the mark or print anything.

So according to this guideline a Lisp program should use a loop with
search-forward and replace-match.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Juri Linkov wrote:

> 3. but the doc-string of this command says:
> 
>     This function is usually the wrong thing to use in a Lisp program.
>     What you probably want is a loop like this:
>       (while (search-forward FROM-STRING nil t)
>         (replace-match TO-STRING nil t))
>     which will run faster and will not set the mark or print anything.
> 
> So according to this guideline a Lisp program should use a loop with
> search-forward and replace-match.
 
Thanks for this. I haven't get to query-replace yet. What I have in mind is
to record also the y/n/a/q choices of the user.
-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

    Thanks for this. I haven't get to query-replace yet. What I have in mind is
    to record also the y/n/a/q choices of the user.

That approach is valid, but I suspect users will not want it very often.

Meanwhile, the best way to implement that approach would be to generate
a series of search function calls and replacements, with no loop.

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

    Another case that might help us to decide to what level to record
    Lisp code is converting M-% (`query-replace').  There are three basic
    levels of "interactivity":

    1. convert it to the call to `query-replace', e.g.

That is the only reasonable thing to do with M-%.

    2. convert M-% to the call to `replace-string';

That's just no good.  If the user wanted to replace with no queries he
would have run `replace-string' himself.

    3. but the doc-string of this command says:

	This function is usually the wrong thing to use in a Lisp program.
	What you probably want is a loop like this:
	  (while (search-forward FROM-STRING nil t)
	    (replace-match TO-STRING nil t))
	which will run faster and will not set the mark or print anything.

    So according to this guideline a Lisp program should use a loop with
    search-forward and replace-match.

If the user runs M-x replace-string, replacing it with a loop like this
is a good idea.

Re: recording-elisp.el - try recording commands as elisp code

[Juri Linkov]

>     1. convert it to the call to `query-replace', e.g.
>
> That is the only reasonable thing to do with M-%.
>
>     2. convert M-% to the call to `replace-string';
>
> That's just no good.  If the user wanted to replace with no queries he
> would have run `replace-string' himself.

I always use `M-% from RET to RET !' to replace all matches with
no queries because this is very convenient user interface.  So Lisp
recording should take care of this and convert the key `!' to the loop
with search-forward and replace-match.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

    > That's just no good.  If the user wanted to replace with no queries he
    > would have run `replace-string' himself.

    I always use `M-% from RET to RET !' to replace all matches with
    no queries because this is very convenient user interface.  So Lisp
    recording should take care of this and convert the key `!' to the loop
    with search-forward and replace-match.

I agree, as regards that special case.

Re: recording-elisp.el - try recording commands as elisp code

[Stefan Monnier]

> It's quite contradictory. If I give up relying on internals the recorded
> result looks bad. If I insist this style there are other isearch-alike
> commands waiting - I simply can not and should not handle them all.

I suspect that this kind of problem is going to re-appear in various
other forms for other commands.  I think the right way to do it is to
provide "hooks" in record-lisp which isearch can then use to explain
what it does.  E.g. record-lisp could check the last command's
`record-lisp-cleanup-function' symbol and if set call it to clean up the
last few commands.  Isearch could then set this property on
`isearch-exit'.  Well, to tell you the truth, I have no idea what I'm
talking about, so the suggestion probably isn't workable as is, but
hopefully it gives you an idea of how to cleanly interface the
two packages.


        Stefan

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Stefan Monnier wrote:

> I suspect that this kind of problem is going to re-appear in various
> other forms for other commands.  I think the right way to do it is to
> provide "hooks" in record-lisp which isearch can then use to explain
> what it does.  E.g. record-lisp could check the last command's
> `record-lisp-cleanup-function' symbol and if set call it to clean up the
> last few commands.  Isearch could then set this property on
> `isearch-exit'.  Well, to tell you the truth, I have no idea what I'm
> talking about, so the suggestion probably isn't workable as is, but
> hopefully it gives you an idea of how to cleanly interface the
> two packages.

I saw your post later than I took actions. Your suggestion is similar to
what I did:

I've reconstructed the code so that the output procedure uses an alist to
lookup a "special treatment" function for certain commands. The function
takes the original recorded commands and already-have output command list,
and returns a modified output.

I implemented 3 such functions: one for simply ignoring a command, one for
cumulating consecutive self-insert-command, and another for isearch-exit.
More to come.

In this way I think we can put some special treatment of commonly used
commands here. And other not-very-common commands can be taken care of by
the corresponding package developer or the user. I'm I right?

-- 
   regards,
yzhh

Re: recording-elisp.el - try recording commands as elisp code

[Richard Stallman]

    To get good isearch recording I used isearch internal information (a stack
    of intermidiate states left on isearch-exit). This looks a bad practice
    since it's prone to isearch implementation changes. 

Isearch is very important, so it is ok if this interacts with some
isearch internals.

Re: recording-elisp.el - try recording commands as elisp code

[yzhh]

Richard Stallman wrote:

> Isearch is very important, so it is ok if this interacts with some
> isearch internals.

Your opinion is quite different from Juri's. A vote?

-- 
   regards,
yzhh