general perform-replace REPLACEMENTS arg for regexp query-replacement?

general perform-replace REPLACEMENTS arg for regexp query-replacement?

[Drew Adams]

See 2008-08-16, thread "Then how to replace all my $$... $$ to \[ \] ..." in
help-gnu-emacs. The subject line says it all. The answer, provided by Reiner
Steib, was to do this:

 M-: (replace-string "$$" '("\\\[" "\\\]"))

or this:

 M-: (replace-regexp "\\$\\$" '("\\\[" "\\\]"))

This works too, and is quite handy:

 M-: (query-replace-regexp "\\$\\$" '("\\\\[" "\\\\]"))

But AFAICT, you cannot use `query-replace-regexp' interactively to do this. You
cannot, for instance, do this to get the same effect:

 C-M-% RET \$\$ RET \, '("\\\\[" "\\\\]")

That just replaces "$$" by the string "(\\[ \\])". 

Reiner also said:

 "However, IMHO this neat feature should be available
  interactively (maybe it is?) and the documentation should
  not be hidden in (info "(elisp)Search and Replace") under
  the internal function `perform-replace'."

I agree with Reiner's suggestions: please make this available to
query-replacement, and please document it more prominently, even if you don't
make it available interactively. Users should be made aware of this feature,
preferably in the Emacs manual's discussion of text replacement.

[See also my mail of today, subject "Emacs manual nodes about text
replacement".]

RE: general perform-replace REPLACEMENTS arg for regexpquery-replacement?

[Drew Adams]

>  M-: (query-replace-regexp "\\$\\$" '("\\\\[" "\\\\]"))
> 
> But AFAICT, you cannot use `query-replace-regexp' 
> interactively to do this. You
> cannot, for instance, do this to get the same effect:
> 
>  C-M-% RET \$\$ RET \, '("\\\\[" "\\\\]")
> 
> That just replaces "$$" by the string "(\\[ \\])".

What's needed, I think, is some new char sequnce, e.g. `\@', that signifies that
what follows it should be taken as a list of replacement strings, not converted
to a single parenthesized replacement string as `\,' does.

Whether such a construct should be available at all levels recursively (as is
the case for `\,') or just at the top level, I don't know. But I suspect only
top level makes sense, given what `perform-replace' expects. 

Whether such a construct should be robust enough to treat not just a literal
list of regexp strings but an arbitrary sexp that evals to such a list, I don't
know.

Here is a naive definition that lets `C-M-%' work for a literal list of strings
at top level, such as in the example above (with `\,' replaced by `\@'):

 C-M-% RET \$\$ RET \@("\\\\[" "\\\\]")

The only change to the existing code is to add the outermost `if' and the first
`if' clause. The rest is unchanged.

(defun query-replace-compile-replacement (to regexp-flag)
  "Maybe convert a regexp replacement TO to Lisp.
Returns a list suitable for `perform-replace' if necessary,
the original string if not."
  (if (and regexp-flag
	   (string-match
            "\\(\\`\\|[^\\]\\)\\(\\\\\\\\\\)*\\\\\
@\\((\"[^\"].*\"\\s-*\"[^\"].*\")\\)" to))
      (let ((lst (car (read-from-string
                       (substring
                        to (match-beginning 3) (match-end 3))))))
        (mapcar
         (lambda (x)
           (query-replace-compile-replacement x regexp-flag))
         lst))
    (if (and regexp-flag
             (string-match
              "\\(\\`\\|[^\\]\\)\\(\\\\\\\\\\)*\\\\[,#]" to))
        (let (pos list char)
          (while
              (progn
                (setq pos (match-end 0))
                (push (substring to 0 (- pos 2)) list)
                (setq char (aref to (1- pos))
                      to (substring to pos))
                (cond
                  ((eq char ?\#)
                   (push '(number-to-string replace-count) list))
                  ((eq char ?\,)
                   (setq pos (read-from-string to))
                   (push `(replace-quote ,(car pos)) list)
                   (let ((end
                          ;; Swallow a space after a symbol
                          ;; if there is a space.
                          (if (and
                               (or (symbolp (car pos))
                                   ;; Swallow a space after 'foo
                                   ;; but not after (quote foo).
                                   (and (eq (car-safe (car pos))
                                            'quote)
                                        (not (= ?\(
                                                (aref to 0)))))
                               (eq (string-match " " to (cdr pos))
                                   (cdr pos)))
                              (1+ (cdr pos))
                            (cdr pos))))
                     (setq to (substring to end)))))
                (string-match
                 "\\(\\`\\|[^\\]\\)\\(\\\\\\\\\\)*\\\\[,#]" to)))
          (setq to (nreverse (delete "" (cons to list))))
          (replace-match-string-symbols to)
          (cons 'replace-eval-replacement
                (if (cdr to)
                    (cons 'concat to)
                  (car to)))))
    to))

Perhaps this is good enough as is? Or perhaps someone more familiar with the
existing code can improve it. Some possible improvements were mentioned above.

If people try this and think it is OK as is, I can submit a patch.

Re: general perform-replace REPLACEMENTS arg for regexpquery-replacement?

[Juri Linkov]

> Here is a naive definition that lets `C-M-%' work for a literal list of strings
> at top level, such as in the example above (with `\,' replaced by `\@'):
>
>  C-M-% RET \$\$ RET \@("\\\\[" "\\\\]")

Then by analogy with the Lisp backquote syntax it should be `\,@':

  C-M-% \$\$ RET \,@("\\\\[" "\\\\]")

-- 
Juri Linkov
http://www.jurta.org/emacs/

RE: general perform-replace REPLACEMENTS arg for regexpquery-replacement?

[Drew Adams]

> > Here is a naive definition that lets `C-M-%' work for a 
> > literal list of strings at top level, such as in the
> > example above (with `\,' replaced by `\@'):
> >
> >  C-M-% RET \$\$ RET \@("\\\\[" "\\\\]")
> 
> Then by analogy with the Lisp backquote syntax it should be `\,@':
> 
>   C-M-% \$\$ RET \,@("\\\\[" "\\\\]")

OK, I have no problem with that (actually, I started with that). However, it's
not that analogous. This is not really about splicing a list.

Actually, after sending the mail and realizing that the sequence would always be
not just \@ but \@(, I thought (and think) that perhaps simply \( would be
better. This is about passing a list. So I'd propose that.

 C-M-% RET abc RET \("<xyz>" "</xyz>")

WDOT?

Re: general perform-replace REPLACEMENTS arg for regexpquery-replacement?

[Juri Linkov]

> Actually, after sending the mail and realizing that the sequence would always be
> not just \@ but \@(, I thought (and think) that perhaps simply \( would be
> better. This is about passing a list. So I'd propose that.
>
>  C-M-% RET abc RET \("<xyz>" "</xyz>")
>
> WDOT?

Then we can perfectly use just the current syntax `\,' without inventing
a new one, e.g.

  C-M-% RET abc RET \,'("<xyz>" "</xyz>")

The only place currently preventing this is prin1-to-string that turns
a list into a string.  We could change it to keep the list intact.

-- 
Juri Linkov
http://www.jurta.org/emacs/

RE: general perform-replace REPLACEMENTS arg for regexpquery-replacement?

[Drew Adams]

> > Actually, after sending the mail and realizing that the 
> > sequence would always be not just \@ but \@(, I thought
> > (and think) that perhaps simply \( would be
> > better. This is about passing a list. So I'd propose that.
> >
> >  C-M-% RET abc RET \("<xyz>" "</xyz>")
> >
> > WDOT?
> 
> Then we can perfectly use just the current syntax `\,' 
> without inventing a new one, e.g.
> 
>   C-M-% RET abc RET \,'("<xyz>" "</xyz>")

Have you tried it? That was the first thing I tried.

The problem is that \, converts the sexp that follows it to a string.
 
> The only place currently preventing this is prin1-to-string that turns
> a list into a string.  We could change it to keep the list intact.

Yes, but then you lose the ability to get the current behavior: replace by the
string, "(\"zzz\" \"aaa\")".

IMO, two different syntaxes are needed to separate the two different user
intentions: a sexp that evals to a string "(...)" and a sexp that evals to a
list of strings to pass to `perform-replace'.

For the second case, I think, as I said, that there is no need to provide for a
sexp that evals to the list; it is enough to provide for a literal list. IMO.

Re: general perform-replace REPLACEMENTS arg for regexpquery-replacement?

[Juri Linkov]

>> The only place currently preventing this is prin1-to-string that turns
>> a list into a string.  We could change it to keep the list intact.
>
> Yes, but then you lose the ability to get the current behavior: replace by the
> string, "(\"zzz\" \"aaa\")".

When you want to replace by the string, you don't need `\,'.

> IMO, two different syntaxes are needed to separate the two different user
> intentions: a sexp that evals to a string "(...)" and a sexp that evals to a
> list of strings to pass to `perform-replace'.

There two syntaxes naturally are `\,(' and `\,'('.

-- 
Juri Linkov
http://www.jurta.org/emacs/

RE: general perform-replace REPLACEMENTS arg for regexpquery-replacement?

[Drew Adams]

> >> The only place currently preventing this is 
> >> prin1-to-string that turns a list into a string.
> >> We could change it to keep the list intact.
> >
> > Yes, but then you lose the ability to get the current 
> > behavior: replace by the string, "(\"zzz\" \"aaa\")".
> 
> When you want to replace by the string, you don't need `\,'.

What if it's \,(list "zzz" "aaa") or some hairy sexp that evals to the same list
("zzz" "aaa")? You snipped the part about the string being the result of Lisp
evaluation. Currently \, evals the sexp that follows it and then
`prin1-to-string's it to come up with a replacement string.

Is that a useful feature: being able to replace text by the print representation
of a Lisp value that is the result of evaluation? If so, shouldn't it still be
allowed in case the result of evaluation is a list of strings?

I'm not sure it's super important as a feature, but why not keep it? Why lose
that just to favor a particular syntax choice?

I don't feel strongly about that existing feature, and I don't feel strongly
about which syntax we use. I am in favor of the new feature I proposed. Beyond
that, I would point out that, IIUC, what you suggest means losing some current
functionality. That's all.

> > IMO, two different syntaxes are needed to separate the two 
> > different user intentions: a sexp that evals to a string "(...)"
> > and a sexp that evals to a list of strings to pass to
> > `perform-replace'.
> 
> There two syntaxes naturally are `\,(' and `\,'('.

Perhaps we're miscommunicating. \, evals the sexp that follows it. In your first
case, the sexp is presumably a function application or (). In your second case,
the sexp is a quoted list.

What should \, then do with the result of evaluation? Currently \, just uses
prin1 to convert the result to a string to use as replacement. Would you have it
do something different depending on the type of evaluation result?

Would you have it always pass the result on to `perform-replace', as is,
whenever it is a list of strings, but otherwise convert it to a string (as now)?
That is, would you treat \,(list "a" "b") and \,'("a" "b") the same (pass two
replacement strings to perform-replace) but different from \,(list 2 3) (pass
the single string "(2 3)" to perform-replace)?

That doesn't sound too reasonable to me.

Or are you suggesting to explicitly look for the ( and '( as part of the \,
syntax? Do you mean that \,'( would always pass the list that is quoted to
`perform-replace', and never convert it to a string for replacement? If so,
would you check whether the list elements were strings? Or would you always
convert the elements in the quoted list to strings to be sure, so that \,'(2 3)
would pass the list of replacement strings ("2" "3")?

Just what you're proposing is not clear. If it's something like what I've
guessed above, then I'd say:

(1) The expressions will be less readable than if we more clearly separate the
two syntax possibilities, instead of trying to use \, to introduce both.

(2) Its use will be error-prone, because of the similarity in appearance.

Maybe you can give a concrete example or two, to make clear what you're saying.

I'm open on (1) the syntax to use and (2) whether or not to keep the current
possibility of evaling to a Lisp value and then prin1-ing that to a replacement
string - including in the case where the eval result is a list of strings.

I'd prefer a clean separation of syntax: \, for Lisp evaluation and
`prin1-to-string' conversion (always), and something else (dissimilar visually)
for passing a list of strings to `perform-replace'. 

I proposed \@ and later \( for the latter, but I don't much care what we pick. I
don't however favor something that starts, as does Lisp evaluation, with \, -
too confusing for users, too complex as code, and seemingly a loss of some
current functionality. And I don't see any advantage from such a choice.