rx and rx-to-string handle (regexp VAR) differently

rx and rx-to-string handle (regexp VAR) differently

[Kévin Le Gouguec]

Hello Emacs!

I am using (rx-define NAME (regexp STRING-VAR)) to give friendlier names
to regular expressions that I have lying around, written out manually
without rx.  To my surprise, rx accepts NAME, but rx-to-string does not.

For example:

  (setq foo-regexp "fo+")
  (rx-define foo (regexp foo-regexp))
  (message "(rx foo): %s"
           (rx foo))
  (message "(rx-to-string '(seq foo) t): %s"
           (rx-to-string '(seq foo) t))

This yields:

  (rx foo): fo+
  rx--translate-regexp: rx ‘regexp’ form with non-string argument

The backtrace from rx-to-string suggests that the symbol foo-regexp is
not substituted for its value:

  Debugger entered--Lisp error: (error "rx ‘regexp’ form with non-string argument")
    signal(error ("rx ‘regexp’ form with non-string argument"))
    error("rx `regexp' form with non-string argument")
⇒   rx--translate-regexp((foo-regexp))
⇒   rx--translate-form((regexp foo-regexp))
⇒   rx--translate((regexp foo-regexp))
⇒   rx--translate-symbol(foo)
⇒   rx--translate(foo)
    mapcar(rx--translate (foo))
    rx--translate-seq((foo))
    rx--translate-form((seq foo))
    rx--translate((seq foo))
    rx-to-string((seq foo) t)
    (message "(rx-to-string '(seq foo) t): %s" (rx-to-string '(seq foo) t))

I can circumvent this by running rx-define through eval:

  (setq foo-regexp "fo+")
  (eval `(rx-define foo (regexp ,foo-regexp)))
  (message "(rx foo): %s"
           (rx foo))
  (message "(rx-to-string '(seq foo) t): %s"
           (rx-to-string '(seq foo) t))

AFAIU this ensures that foo-regexp is substituted for its value before
rx-define is called, so both rx and rx-to-string end up processing a
(regexp "fo+") form.  This does yield:

  (rx foo): fo+
  (rx-to-string ’(seq foo) t): fo+

Is there a bug to file here, or have I missed something in the
documentation that explains why rx can work with (regexp VAR) but
rx-to-string cannot?  "(elisp) Rx Constructs" says:

> ‘(regexp EXPR)’
> ‘(regex EXPR)’
>      Match the string regexp that is the result from evaluating the Lisp
>      expression EXPR.  The evaluation takes place at call time, in the
>      current lexical environment.

Re(-re)ⁿ-reading that last sentence makes me tentatively conclude that
foo-regexp is not in rx-to-string's "current lexical environment", but
somehow is in rx's?  "(elisp) Lexical Binding" makes me doubt that
interpretation:

>    Here is how lexical binding works.  Each binding construct defines a
> “lexical environment”, specifying the variables that are bound within
> the construct and their local values.  When the Lisp evaluator wants the
> current value of a variable, it looks first in the lexical environment;
> if the variable is not specified in there, it looks in the symbol’s
> value cell, where the dynamic value is stored.

I don't see anything in there that suggests that things would be
different for rx or rx-to-string; AFAIU in those (message …) forms,
there is no "binding construct" at play (assuming this designates let &
friends), so in both cases the "dynamic value" of foo-regexp should be
retrieved, which I'd expect to be the value set by the (setq …) form?

(FWIW, using defvar instead of setq yields the same results)


Anyhoo; for now, I have my (eval …) workaround (which admittedly is not
perfect, since (rx foo) will not be updated if I change foo-regexp; it's
good enough for my purposes though), so I am mainly bringing this up
because I figure either (a) there's a bug to be reported, or(/and) (b)
I'm about to get schooled on some aspect of ELisp I don't fully grasp.
TBH at this stage (b) sounds more interesting.


At any rate, thanks for your time 🙏


(If that matters: I can reproduce this with 28.2, 29.0.90 from ≈1 month
ago, 30.0.50 from ≈3 weeks ago)

Re: rx and rx-to-string handle (regexp VAR) differently

[Platon Pronko]

> Is there a bug to file here, or have I missed something in the
> documentation that explains why rx can work with (regexp VAR) but
> rx-to-string cannot?

rx-to-string docs explicitly say: "The arguments to ‘literal’ and ‘regexp’ forms inside FORM must be constant strings."

As to "why it was done like this?", I guess that's because (rx) is a macro and (rx-to-string) is a function. So (rx-to-string) tries to evaluate everything in a self-contained way, without accessing outside environment (this might prevent certain error cases when rx form is created in one place, then stored and evaluated in a different context).
But (rx) is a macro, and thus it can be free to expand to whatever it likes - so it sets (rx--delayed-evaluation t) and outputs an expression that references outside variables.

> Re(-re)ⁿ-reading that last sentence makes me tentatively conclude that
> foo-regexp is not in rx-to-string's "current lexical environment", but
> somehow is in rx's?

In this case it actually is in rx-to-string's current lexical environment, but it just doesn't care - it checks if expression is a literal string and throws an error otherwise.
And in rx's case it also is in the current lexical environment, however the symbol evaluation happens only after the macro was expanded - (rx) itself simply returns foo-regexp as-is:

(macroexpand-1 '(rx foo)) ;; output: foo-regexp

-- 
Best regards,
Platon Pronko
PGP 2A62D77A7A2CB94E

Re: rx and rx-to-string handle (regexp VAR) differently

[Michael Heerdegen]

Platon Pronko <platon7pronko@gmail.com> writes:

> rx-to-string docs explicitly say: "The arguments to ‘literal’ and
> ‘regexp’ forms inside FORM must be constant strings."

Is this a correct way to get what Kévin wants?

#+begin_src emacs-lisp
  (setq foo-regexp "fo+")
  (rx-define foo (eval `(regexp ,foo-regexp)))
#+end_src

This also works in both cases:

#+begin_src emacs-lisp
  (setq foo-regexp '(regexp "fo+"))
  (rx-define foo (eval foo-regexp))
#+end_src


Michael.

Re: rx and rx-to-string handle (regexp VAR) differently

[Kévin Le Gouguec]

Platon Pronko <platon7pronko@gmail.com> writes:

>> Is there a bug to file here, or have I missed something in the
>> documentation that explains why rx can work with (regexp VAR) but
>> rx-to-string cannot?
>
> rx-to-string docs explicitly say: "The arguments to ‘literal’ and ‘regexp’ forms
> inside FORM must be constant strings."

Drats.  So they do.  Second sentence of the docstring no less; and a
completely separate paragraph in "(elisp) Rx Functions".

I have no words 🤦

> As to "why it was done like this?", I guess that's because (rx) is a macro and
> (rx-to-string) is a function. So (rx-to-string) tries to evaluate everything in
> a self-contained way, without accessing outside environment (this might prevent
> certain error cases when rx form is created in one place, then stored and
> evaluated in a different context).
> But (rx) is a macro, and thus it can be free to expand to whatever it likes - so
> it sets (rx--delayed-evaluation t) and outputs an expression that references
> outside variables.
>
>> Re(-re)ⁿ-reading that last sentence makes me tentatively conclude that
>> foo-regexp is not in rx-to-string's "current lexical environment", but
>> somehow is in rx's?
>
> In this case it actually is in rx-to-string's current lexical environment, but
> it just doesn't care - it checks if expression is a literal string and throws an
> error otherwise.
> And in rx's case it also is in the current lexical environment, however the
> symbol evaluation happens only after the macro was expanded - (rx) itself simply
> returns foo-regexp as-is:
>
> (macroexpand-1 '(rx foo)) ;; output: foo-regexp

Thanks for the perspective, much appreciated!

Re: rx and rx-to-string handle (regexp VAR) differently

[Eli Zaretskii]

> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
> Date: Thu, 08 Jun 2023 23:44:07 +0200
> 
> Is there a bug to file here, or have I missed something in the
> documentation that explains why rx can work with (regexp VAR) but
> rx-to-string cannot?

FWIW, "Extending Rx" in the ELisp manual says:

   • Definitions are only ever expanded in calls to ‘rx’ or
     ‘rx-to-string’, not merely by their presence in definition macros.

So I think this ought to be supported, and you should file a bug:
either the implementation or the documentation should be fixed.

Re: rx and rx-to-string handle (regexp VAR) differently

[Kévin Le Gouguec]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Platon Pronko <platon7pronko@gmail.com> writes:
>
>> rx-to-string docs explicitly say: "The arguments to ‘literal’ and
>> ‘regexp’ forms inside FORM must be constant strings."
>
> Is this a correct way to get what Kévin wants?
>
> #+begin_src emacs-lisp
>   (setq foo-regexp "fo+")
>   (rx-define foo (eval `(regexp ,foo-regexp)))
> #+end_src

Ah, nice catch!  I had started messing around with rx's 'eval', but
figured I should sort out my understanding of 'regexp' before throwing
more indirection at the problem.  It does seem like it fits my original
use-case better:

  (eval-and-compile
    (put 'foo 'rx-definition
         '((eval
            `(regexp ,foo-regexp))))
    'foo)

AFAICT (from that expansion and some cursory testing) that does allow
for (rx-to-string '(seq foo)) to change whenever foo-regexp changes.

> This also works in both cases:
>
> #+begin_src emacs-lisp
>   (setq foo-regexp '(regexp "fo+"))
>   (rx-define foo (eval foo-regexp))
> #+end_src

Noted.  Thanks a bunch for these hints!

Re: rx and rx-to-string handle (regexp VAR) differently

[Kévin Le Gouguec]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
>> Date: Thu, 08 Jun 2023 23:44:07 +0200
>> 
>> Is there a bug to file here, or have I missed something in the
>> documentation that explains why rx can work with (regexp VAR) but
>> rx-to-string cannot?
>
> FWIW, "Extending Rx" in the ELisp manual says:
>
>    • Definitions are only ever expanded in calls to ‘rx’ or
>      ‘rx-to-string’, not merely by their presence in definition macros.
>
> So I think this ought to be supported, and you should file a bug:
> either the implementation or the documentation should be fixed.

ACK; will do.

I see that the note in rx-to-string's docstring re. only supporting
literal strings for 'regexp' and 'literal' dates from 2019-06-25
"Support (rx (and (regexp EXPR) (literal EXPR))) (Bug#36237)"
(b59ffd2290f):

   (defun rx-to-string (form &optional no-group)
     "Parse and produce code for regular expression FORM.
   FORM is a regular expression in sexp form.
  -NO-GROUP non-nil means don't put shy groups around the result."
  +NO-GROUP non-nil means don't put shy groups around the result.
  +
  +In contrast to the `rx' macro, subforms `literal' and `regexp'
  +will not accept non-string arguments, i.e., (literal STRING)
  +becomes just a more verbose version of STRING."

I might also give that thread a read to see what prompted this
limitation.