bug#50136: 28.0.50; A problem with rx-let expansion

bug#50136: 28.0.50; A problem with rx-let expansion

[Michael Heerdegen]

Hello,

I'm referring to this paragraph in rx.el:

;; FIXME: Consider adding extensions in Lisp macro style, where
;; arguments are passed unevaluated to code that returns the rx form
;; to use.
;; [...]
;; While this would permit more powerful extensions, it's unclear just
;; how often they would be used in practice.  Let's wait until there is
;; demand for it.

Ok - here is!  I would find that approach much more natural than the
current one.  Look:

Already the first `rx-let' pseudo macro I tried hit a problem (bug) with
the current approach:

#+begin_src emacs-lisp
(rx-let ((scatter (string)
                  (regex (mapconcat #'string (string-to-list string) ".*"))))
  (rx (scatter "abc"))) => useless error message
#+end_src

while the following version (the only difference is the argument name)
works as expected:

#+begin_src emacs-lisp
(rx-let ((scatter (s) (regex (mapconcat #'string (string-to-list s) ".*"))))
  (rx (scatter "abc"))) ==> "a.*b.*c"
#+end_src

Seems the function form #'string gets replaced and ends as #'"abc" in
the first version because the function name accidentally collides with
the argument name.

Personally I would be more happy with a thing called `rx-macrolet' than
with an extended `rx-let' to support this additional macro-like kind of
syntax.

TIA,

Michael.

bug#50136: 28.0.50; A problem with rx-let expansion

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> #+begin_src emacs-lisp
> (rx-let ((scatter (s) (regex (mapconcat #'string (string-to-list s) ".*"))))
>   (rx (scatter "abc"))) ==> "a.*b.*c"
> #+end_src

AFAIU, the current approach doesn't allow to extend this with an
optional CHARS argument to specify with chars are allowed "in between"
(instead of "any").  The "&optional" keyword makes it barf.

And I can't actually "look" at the arguments to provide a conditional
replacement.  All I can do is to use `regex' to eval a constant
expression I specify (at run-time).  This seems a quite limiting
approach.  Something more macro-like would be appropriate IMO.

Michael.

bug#50136: 28.0.50; A problem with rx-let expansion

[Mattias Engdegård]

20 aug. 2021 kl. 15.59 skrev Michael Heerdegen <michael_heerdegen@web.de>:

> (rx-let ((scatter (string)
>                  (regex (mapconcat #'string (string-to-list string) ".*"))))
>  (rx (scatter "abc"))) => useless error message

Yes, that's an unfortunate consequence of the current substitution mechanism (validating Stefan's premonitions to some extent).
However the syndrome is entirely avoidable and does not make the facility less powerful.

> Personally I would be more happy with a thing called `rx-macrolet' than
> with an extended `rx-let' to support this additional macro-like kind of
> syntax.

Noted (in the honest sense)!

> AFAIU, the current approach doesn't allow to extend this with an
> optional CHARS argument to specify with chars are allowed "in between"
> (instead of "any").  The "&optional" keyword makes it barf.

That's right. What would be the preferred semantics? For example, what value would you expect an absent optional parameter to take? `nil` isn't very useful inside rx forms, so it would only make sense inside embedded Lisp expressions.

> And I can't actually "look" at the arguments to provide a conditional
> replacement.  All I can do is to use `regex' to eval a constant
> expression I specify (at run-time).  This seems a quite limiting
> approach.  Something more macro-like would be appropriate IMO.

Actually it's the rx `eval` form that you are looking for if you want arbitrary compile-time computation. `regexp` and `literal` are explicitly made for run-time expressions.

bug#50136: 28.0.50; A problem with rx-let expansion

[Michael Heerdegen]

Mattias Engdegård <mattiase@acm.org> writes:

> > AFAIU, the current approach doesn't allow to extend this with an
> > optional CHARS argument to specify with chars are allowed "in between"
> > (instead of "any").  The "&optional" keyword makes it barf.
>
> That's right. What would be the preferred semantics? For example, what
> value would you expect an absent optional parameter to take?

I guess it's not very useful for the current mechanism.  With a
macro-like thing, the expansion would just not use the value if it's nil.  The
expander would check whether the argument was provided.

> Actually it's the rx `eval` form that you are looking for if you want
> arbitrary compile-time computation. `regexp` and `literal` are
> explicitly made for run-time expressions.

Yes, I think you are right.


Michael.

bug#50136: 28.0.50; A problem with rx-let expansion

[Mattias Engdegård]

20 aug. 2021 kl. 19.21 skrev Michael Heerdegen <michael_heerdegen@web.de>:

> I guess it's not very useful for the current mechanism.  With a
> macro-like thing, the expansion would just not use the value if it's nil.  The
> expander would check whether the argument was provided.

Yes. It was (and is) a trade-off between the simpler "template-like" semantics currently used and the programmatic "execute code" semantics of Lisp macros. The idea was that the simpler semantics would be simpler and sufficient for most uses, and `eval` forms could always be employed in other cases.

I'm still fretting over having made the wrong call, and that at least I should have omitted parametrised rx definitions until better understood.

If you just have one optional argument, try &rest -- at least its default-case semantics are crystal clear (no arguments inserted). Obviously, you could also use separate rx macros for different parameter lists:

(rx-let ((delimited (grp delim) (seq delim (group-n grp (* (not delim)) delim)))
         (double-quoted (grp) (delimited grp ?\")))

etc.

bug#50136: 28.0.50; A problem with rx-let expansion

[Michael Heerdegen]

Mattias Engdegård <mattiase@acm.org> writes:

> Yes. It was (and is) a trade-off between the simpler "template-like"
> semantics currently used and the programmatic "execute code" semantics
> of Lisp macros. The idea was that the simpler semantics would be
> simpler and sufficient for most uses, and `eval` forms could always be
> employed in other cases.

Ah ok, now my picture gets somewhat clearer.  The semantics I wish for
are already possible.  I need to use `eval' on the one side, and &rest
on the other.  Since REST from &rest rest is spliced in, I can't use
REST like a variable (and e.g. test whether any args have been provided,
that is impossible, since in that case it expands to nothing); instead I
can use something like

  (let ((args (list rest))) ...)

inside rx `eval' and use that as argument list variable.

I think I can live with that.

Maybe it would be good to improve the docstring to say the things that I
missed clearer? - Say explicitly that this mechanism is different from
macros, instead it's just a simple substitution mechanism, and add a
simple example to the docstring as a reference, even if we already have
examples in the manual.

Maybe also tell that using `eval' you can still have a macro-like
behavior in the end.

Michael.

bug#50136: 28.0.50; A problem with rx-let expansion

[Mattias Engdegård]

21 aug. 2021 kl. 13.45 skrev Michael Heerdegen <michael_heerdegen@web.de>:

>  (let ((args (list rest))) ...)
> 
> inside rx `eval' and use that as argument list variable.

Or just put all the actual code in a plain function:

(eval-when-compile
 (defun expand-my-rx-thing (x y &optional z &rest r) ...))

(rx-define my-rx-thing (x y &rest more) (eval (expand-my-rx-thing x y more)))

and use &optional and &rest arguments as you are used to, without any risks of substitution accidents like "string" in the example you showed earlier. The function would effectively work exactly like a macro of the sort you requested.

> Maybe it would be good to improve the docstring to say the things that I
> missed clearer? - Say explicitly that this mechanism is different from
> macros, instead it's just a simple substitution mechanism, and add a
> simple example to the docstring as a reference, even if we already have
> examples in the manual.

I'll see what can be done. It's not really Emacs tradition to have examples in doc strings but maybe they can be improved a bit as you say.

> Maybe also tell that using `eval' you can still have a macro-like
> behavior in the end.

Not a bad idea!

bug#50136: 28.0.50; A problem with rx-let expansion

[Michael Heerdegen]

Mattias Engdegård <mattiase@acm.org> writes:

> Or just put all the actual code in a plain function:
>
> (eval-when-compile
>  (defun expand-my-rx-thing (x y &optional z &rest r) ...))
>
> (rx-define my-rx-thing (x y &rest more) (eval (expand-my-rx-thing x y
> more)))
>
> and use &optional and &rest arguments as you are used to, without any
> risks of substitution accidents like "string" in the example you
> showed earlier. The function would effectively work exactly like a
> macro of the sort you requested.

A good idea, I like it!

> > [... talking about docstring tweaks ...]
>
> I'll see what can be done. It's not really Emacs tradition to have
> examples in doc strings but maybe they can be improved a bit as you
> say.

I you find a way to get along without examples, all the better.  I
suggested to use an example because it might be the simplest way to
explain how things work, and because the way things work may come
unexpected for some (like me).  But providing some more details more
explicitly might be as good as well.


Regards,

Michael.

bug#50136: 28.0.50; A problem with rx-let expansion

[Mattias Engdegård]

23 aug. 2021 kl. 12.45 skrev Michael Heerdegen <michael_heerdegen@web.de>:

>> (rx-define my-rx-thing (x y &rest more) (eval (expand-my-rx-thing x y more)))

> A good idea, I like it!

The variant

 (rx-define my-rx-thing (&rest more) (eval (expand-my-rx-thing more)))

would also work, at the cost of slightly worse error messages if you pass too few arguments to my-rx-thing.

bug#50136: 28.0.50; A problem with rx-let expansion

[Mattias Engdegård]

[-- Attachment #1: Type: text/plain, Size: 537 bytes --]

23 aug. 2021 kl. 12.45 skrev Michael Heerdegen <michael_heerdegen@web.de>:

> I you find a way to get along without examples, all the better.  I
> suggested to use an example because it might be the simplest way to
> explain how things work, and because the way things work may come
> unexpected for some (like me).  But providing some more details more
> explicitly might be as good as well.

Here is my proposed change to the manual. I didn't change the doc strings since they already refer to that Info node.
Good enough?


[-- Attachment #2: 0001-Add-example-of-advanced-user-defined-Rx-form-to-manu.patch --]
[-- Type: application/octet-stream, Size: 1680 bytes --]

From cb1daffa5f226278723b4a96a698883d7010947a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Mattias=20Engdeg=C3=A5rd?= <mattiase@acm.org>
Date: Mon, 23 Aug 2021 17:02:51 +0200
Subject: [PATCH] Add example of advanced user-defined Rx form to manual

* doc/lispref/searching.texi (Extending Rx): Add example illustrating
how to define a user-defined Rx form that performs computation,
from a discussion with Michael Herdeegen (bug#50136).
---
 doc/lispref/searching.texi | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/doc/lispref/searching.texi b/doc/lispref/searching.texi
index 4d5ae3cb43..a0935eee2b 100644
--- a/doc/lispref/searching.texi
+++ b/doc/lispref/searching.texi
@@ -1649,6 +1649,25 @@ Extending Rx
 Since the definition is global, it is recommended to give @var{name} a
 package prefix to avoid name clashes with definitions elsewhere, as is
 usual when naming non-local variables and functions.
+
+User-defined forms themselves only perform simple template
+substitution.  For arbitrary computations, use them together with with
+the @code{rx} forms @code{eval}, @code{regexp} or @code{literal}.
+Example:
+
+@example
+@group
+(defun n-tuple-rx (n element)
+  `(seq "<"
+        (group-n 1 ,element)
+        ,@@(mapcar (lambda (i) `(seq ?, (group-n ,i ,element)))
+                  (number-sequence 2 n))
+        ">"))
+(rx-define n-tuple (n element) (eval (n-tuple-rx n 'element)))
+(rx (n-tuple 3 (+ (in "0-9"))))
+  @result{} "<\\(?1:[0-9]+\\),\\(?2:[0-9]+\\),\\(?3:[0-9]+\\)>"
+@end group
+@end example
 @end defmac
 
 @defmac rx-let (bindings@dots{}) body@dots{}
-- 
2.21.1 (Apple Git-122.3)

bug#50136: 28.0.50; A problem with rx-let expansion

[Michael Heerdegen]

Mattias Engdegård <mattiase@acm.org> writes:

> Here is my proposed change to the manual. I didn't change the doc
> strings since they already refer to that Info node.

Maybe at least one sentence about that this mechanism just performs
trivial substitution?

> +User-defined forms themselves only perform simple template
> +substitution.

I find the term "user-defined" distracting, because anybody (including
library and package developers) is "affected".  Apart from that, ok for
me.

>  For arbitrary computations, use them together with with
> +the @code{rx} forms @code{eval}, @code{regexp} or @code{literal}.
> +Example:
> +
> +@example
> +@group
> +(defun n-tuple-rx (n element)
> +  `(seq "<"
> +        (group-n 1 ,element)
> +        ,@@(mapcar (lambda (i) `(seq ?, (group-n ,i ,element)))
> +                  (number-sequence 2 n))
> +        ">"))
> +(rx-define n-tuple (n element) (eval (n-tuple-rx n 'element)))
> +(rx (n-tuple 3 (+ (in "0-9"))))
> +  @result{} "<\\(?1:[0-9]+\\),\\(?2:[0-9]+\\),\\(?3:[0-9]+\\)>"
> +@end group
> +@end example
>  @end defmac

Didn't try the example, but it looks good.


I have one more thing, however, once we are here: In the docstring of
`rx', near the end:

| (eval EXPR)    Match the rx sexp from evaluating EXPR at compile time.

can we say "expansion time" instead of "compile time"?


Regards,

Michael.

bug#50136: 28.0.50; A problem with rx-let expansion

[Mattias Engdegård]

23 aug. 2021 kl. 18.59 skrev Michael Heerdegen <michael_heerdegen@web.de>:

> Maybe at least one sentence about that this mechanism just performs
> trivial substitution?

I didn't do that now, because I couldn't see how it could be interpreted otherwise (and there's the manual link).

> I find the term "user-defined" distracting, because anybody (including
> library and package developers) is "affected".  Apart from that, ok for
> me.

Thank you, changed.

> Didn't try the example, but it looks good.

Thanks for looking at it. Good examples are always hard to write!

> | (eval EXPR)    Match the rx sexp from evaluating EXPR at compile time.
> 
> can we say "expansion time" instead of "compile time"?

We can, and now do!

Closing; I think we're done here (complain if not).

bug#50136: 28.0.50; A problem with rx-let expansion

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Actually it's the rx `eval` form that you are looking for if you want
  > > arbitrary compile-time computation. `regexp` and `literal` are
  > > explicitly made for run-time expressions.

If there are variables which can hold an rx expression, we need to make
them as unsafe file variable bindings.  Or else have special code
to study them and see if evaluation constructs are used.

Has this been done?

Are these constructs an insecurity?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)