`eval'ing form in the current lexical environment

`eval'ing form in the current lexical environment

[Thuna]

How would I `eval' a form as though it were where the `(eval ...)' form
was?  That is, is there an argument to LEXICAL I can pass such that
  (... (eval form LEXICAL) ...)
becomes identical to
  (... <value-of-form> ...)

Let me state my problem as well, in case anyone knows of a solution to
that: I have a macro which modifies the form fed to it, macroexpanding
it as necessary.  Functions are handled specifically, in that its
arguments are computed and the call to the function replaced by
  (func arg1 arg2...)
If, however, `func' is then redefined as a macro, and uses one of its
args without evaluating it, then this would understandably break.  At
the same time, the process the original form went through in my macro is
one that this now-macro form needs to go through as well, so I expand
into a form which *during evaluation* calls this process on the literal
form which is fed to it and evals it.  This all looks something like
  (defmacro (lambda-list head &rest args)
    `(defun ,lambda-list
       ...
       (if (macrop #',head)
           (eval (<processor> '(,head ,@args)))
         (,head ,@argsyms))))
and it is absolutely expected for the forms in ARGS to be aware of and
reference the variables established in LAMBDA-LIST, but because of the
above problem that is not the case (as far as I could tell).

Re: `eval'ing form in the current lexical environment

[Stefan Monnier via Users list for the GNU Emacs text editor]

> How would I `eval' a form as though it were where the `(eval ...)' form
> was?  That is, is there an argument to LEXICAL I can pass such that
>   (... (eval form LEXICAL) ...)
> becomes identical to
>   (... <value-of-form> ...)

No.  If you know the relevant set of variables, you can do

    (eval form `((VAR1 . ,VAL1) (VAR2 . ,VAL2) ...))

but why not just put `form` in there?
Why exactly do you need `eval` in the generated code?

>   (defmacro (lambda-list head &rest args)
>     `(defun ,lambda-list
>        ...
>        (if (macrop #',head)
>            (eval (<processor> '(,head ,@args)))
>          (,head ,@argsyms))))

I'm afraid I had trouble following your explanation (not fault of
yours: it's just difficult to describe in prose what a code does,
especially when it itself manipulates code).
Could you show an example that illustrates why you need `eval` in there?

My naive self would tell you to use

    (defmacro (lambda-list head &rest args)
      `(defun ,lambda-list
         ...
         (if (macrop #',head)
             ,(<processor> `(,head ,@args))
           (,head ,@argsyms))))

Or maybe your "procesor" needs to traverse the code, and what you really
need is to macro-expand it beforehand, something like:

    (defmacro (lambda-list head &rest args)
      (let* ((code (macroexpand-all (... `(,head ,@args) ...)
                                    macroexpand-all-environment))
             (processed (<processor> code)))
        `(defun ,lambda-list
           ... ,code ...)))


- Stefan

Re: `eval'ing form in the current lexical environment

[Thuna]

[ I appear to have forgotten to CC the list, sorry about that.  Below's
what I have previously sent.  Please only reply to this mail. ]

Hmm, I now realize that I misspoke in my original post.  What is doing
this is not a macro but a function, so that was why I went to `eval'.
This form is being generated from within <processor> itself.

Now, your suggestion did bring up an interesting point for me in that I
could simply make a wrapper macro for this function and put that in, but
there does appear to be a problem with this approach in this specific
situation which... will be even more difficult to explain.

Below are two versions of me trying to explain the problem, one with
context and one with me attempting to reduce it down to the bare
essentials.  I'll leave the former in in case I leave out some important
details in the latter, but I have to be honest: it is extremely
convoluted.  Please only read it if the first explanation doesn't make
sense or if you wanted to take a look at the code and find yourself
confused as to what's happening in it.

--- Reduced down to essentials ---

The function (which I accidentally called a macro) we are talking about
is the <processor> itself, that is, it looks something like

  (defun <processor> (form)
    ;; `(,head ,@args) == form
    `(...
      (if (macrop #',head)
          (eval (<processor> `(,head ,@args)))
        ...
        `(,head ,@argsyms))))

and there is a different macro which expands into the defun by doing
 
  (defmacro (name lambda-list &rest body)
    `(defun ,name ,lambda-list
       ,(<processor> `(progn ,@body))))

<processor> recursively expands and modifies FORM, and returns a version
of it which is "processed", and then the definition of NAME is set to be
that processed body.

All macros are expanded and processed, thus `(,head ,@args) above is
/necessarily/ a function call... that is, unless it is later redefined
as a macro.

The goal with (if (macrop #',head) ...) is to check for this exact case,
an in case it DID happen, fix it by processing the form again, calling
that processed version instead of what was supposed to happen.

This however can only during the runtime of NAME, because that is when
we are concerned with `head' being a macro, so the call to <processor>
must happen within NAME and not within the top-level <processor>.
Because we are calling <processor> during NAME, we obtain a form which
needs to be evaluated.

--- With context ---

Let me provide you with the actual code we are discussing.  I intend to
bring this up in emacs-devel at some point anyways so hopefully it'll be
worth the read: https://git.sr.ht/~thuna/tcl-values

Quick rundown of what the code does: This is a library which simulates
true Common Lisp style multiple values.  The way this is achieved by
having `cl-values' set a global variable which holds the values ONLY IF
an "expecting multiple values" flag is set.  Any function which supports
multiple values must therefore ensure that this flag is suppressed (the
term used in the library for this is "sanitization") everywhere in the
code except for in the forms which are "returning" (in that, their
values is what the function will/may return).

While the function author can manually sanitize their function, this is
a cumbursome and tedious process so the function `tcl-values--sanitize'
is provided to sanitize any forms fed to it, which is called from
`tcl-defun' before defining the function.  This is the nebulous
<processor> that we are talking about.

The specific branch in which this eval exists is the basic call to (func
arg1 arg2...).  It's purpose is - in case `func' is later redefined as a
macro - to preserve the sanitization of the function by going through
the form and re-sanitize it.  This needs to happen at runtime because
that's when we are attempting to detect if `func' was redefined or not.

Now, I looked into defining a macro which simply called
`tcl-values--sanitize', but this led to two problems:

1. This resolution happens at macroexpansion time, due to it being
`macroexpand-all'ed.

2. The branch which calls the macro is the same branch we feed into the
macro, thus the expansion of the macro contains exactly the same macro,
resulting in infinite recursion.

There *might* be some trick I can use to get rid of the second problem,
but I cannot think of one that would allow me to fix the first one,
unless I can somehow prevent macroexpand-all from expanding the
resulting macro.

Re: `eval'ing form in the current lexical environment

[Stefan Monnier]

> All macros are expanded and processed, thus `(,head ,@args) above is
> /necessarily/ a function call... that is, unless it is later redefined
> as a macro.

I think this is the part I don't understand: why do you worry about
`head` being a function (or undefined) during macro-expansion but
a macro at run-time?

If the code is compiled, such a change leads to an error and we blame
the change, not the compiler.

E.g. M-: (disassemble '(lambda () (foobar 5))) RET
turns into the byte-code:

     byte code:
       args: nil
     0       constant  foobar
     1       constant  5
     2       call      1
     3       return

and this `call` just signals an error if `foobar` is not a function at
run-time.

So, I think what I'm saying is that you should probably not try to
handle this case at all.


        Stefan

Re: `eval'ing form in the current lexical environment

[Thuna]

Aha, I wasn't aware that that situation was considered illegal.
Alright, that'll definitely make my job a whole lot easier, thanks for
the heads up!

Re: `eval'ing form in the current lexical environment

[Stefan Monnier]

> Aha, I wasn't aware that that situation was considered illegal.

It's pretty much incompatible with the notion of compilation:
Usually, when you compile code like

    (foo (bar))

under the assumption that `foo` is a function, the code you generate
evaluates (bar) before doing anything else.  So by the time it looks at
`foo` it's too late: if `foo` turns out to be a macro you'd have to
"undo" the evaluation of (bar).

IOW, you're forced to do something like what you suggested, i.e. replace
*every* function call with

    (if (is-a-macro 'FUN)
        (handle-the-corner-case 'FUN 'ARGS)
      (FUN . ARGS))

so the above ends up compiled as if you had written:

    (if (is-a-macro 'foo)
        (handle-the-corner-case 'foo '(bar))
      (foo (if (is-a-macro 'bar)
               (handle-the-corner-case 'bar nil)
             (bar))))

which kills all hopes of generating good-quality code.

You can admittedly do better by "watching" the definition of functions
and keeping dependencies, so when a function gets redefined as a macro
you can throw away all the compiled code which depends on it (and
recompile it later as needed).

Nevertheless, it's costly and not considered worth the trouble.

The reverse is accepted, OTOH: you can take a macro and later turn it
into a function.  Code compiled when it was defined as a macro will
simply keep using the old definition until it gets recompiled.


        Stefan

Re: `eval'ing form in the current lexical environment

[Thuna]

Yeah, I knew about how macro-redefined-as-function was handled but I
(mistakenly) figured the reverse would be handled by expanding it during
evaluation.

Now, what we are talking about only applies when the code is compiled,
so interpreted functions /will/ break, but given I don't have a working
fix I am more then willing to call it the programmer's problem and leave
it be.