Using the content of a dynamic variable in a macro

Using the content of a dynamic variable in a macro

[Damien Cassou]

I tried to send the mail below to help-gnu-emacs@gnu.org (without being
subscribed) but it never reached the archive:
https://lists.gnu.org/archive/html/help-gnu-emacs/2023-02/threads.html.

Hi,

I'm wondering why the code below works but won't compile.

foo.el:
  (defvar foo-var '((message "hello world")))
  
  (defmacro foo-macro ()
    `(progn
       ,@foo-var))
  
  (defun foo-fun ()
    (foo-macro))


$ emacs --batch -l foo.el --eval '(foo-fun)'
hello world

$ emacs --batch --eval '(find-file "foo.el")' --eval '(emacs-lisp-byte-compile)'
In toplevel form:
foo.el:32:1: Error: Symbol’s value as variable is void: foo-var

Why isn't the compiler aware of the foo-var variable?

-- 
Damien Cassou

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

Re: Using the content of a dynamic variable in a macro

[Eli Zaretskii]

> From: Damien Cassou <damien@cassou.me>
> Date: Sat, 25 Feb 2023 09:34:07 +0100
> 
> I'm wondering why the code below works but won't compile.
> 
> foo.el:
>   (defvar foo-var '((message "hello world")))
>   
>   (defmacro foo-macro ()
>     `(progn
>        ,@foo-var))
>   
>   (defun foo-fun ()
>     (foo-macro))
> 
> 
> $ emacs --batch -l foo.el --eval '(foo-fun)'
> hello world
> 
> $ emacs --batch --eval '(find-file "foo.el")' --eval '(emacs-lisp-byte-compile)'
> In toplevel form:
> foo.el:32:1: Error: Symbol’s value as variable is void: foo-var
> 
> Why isn't the compiler aware of the foo-var variable?

Because lexical-binding is turned on?

Re: Using the content of a dynamic variable in a macro

[Damien Cassou]

Eli Zaretskii <eliz@gnu.org> writes:
>> Why isn't the compiler aware of the foo-var variable?
>
> Because lexical-binding is turned on?

yes it is but I miss the deep understanding to fully understand.

-- 
Damien Cassou

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

Re: Using the content of a dynamic variable in a macro

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

On Sat, Feb 25, 2023, 5:19 AM Damien Cassou <damien@cassou.me> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
> >> Why isn't the compiler aware of the foo-var variable?
> >
> > Because lexical-binding is turned on?
>
> yes it is but I miss the deep understanding to fully understand.
>

Lexical binding isn't the issue - the variable is global in scope.
The issue is that when you explicitly run the byte-compiler in batch mode,
the "defvar" expression is only compiled, not evaluated, while the defmacro
is evaluated, and the application of the macro function is evaluated during
compilation.
Hence you need "eval-when-compile" per Tomas's response.
The reason it works in interactive mode is that the defvar is evaluated,
not compiled, and the macro function is evaluated in the run-time
environment.

Lynn

[-- Attachment #2: Type: text/html, Size: 1389 bytes --]

Re: Using the content of a dynamic variable in a macro

[Richard Stallman]

  > Lexical binding isn't the issue - the variable is global in scope.
  > The issue is that when you explicitly run the byte-compiler in batch mode,
  > the "defvar" expression is only compiled, not evaluated, while the defmacro
  > is evaluated, and the application of the macro function is evaluated during
  > compilation.

Should we change this to evaluate defvar in batch mode?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Using the content of a dynamic variable in a macro

[Basil Contovounesios]

Richard Stallman [2023-02-26 22:24 -0500] wrote:

>   > Lexical binding isn't the issue - the variable is global in scope.
>   > The issue is that when you explicitly run the byte-compiler in batch mode,
>   > the "defvar" expression is only compiled, not evaluated, while the defmacro
>   > is evaluated, and the application of the macro function is evaluated during
>   > compilation.
>
> Should we change this to evaluate defvar in batch mode?

The byte-compiler avoids evaluating the defvar also in non-batch mode.

So the question is how much of the code that it is compiling should the
byte-compiler evaluate.

We may need to reuse or possibly extend the notions of Lisp code safety
we have before the byte-compiler can start evaluating arbitrary
expressions.  I'm not sure it's worth the complexity.

-- 
Basil

Re: Using the content of a dynamic variable in a macro

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1726 bytes --]

On Mon, Feb 27, 2023, 2:05 PM Basil Contovounesios <contovob@tcd.ie> wrote:

> Richard Stallman [2023-02-26 22:24 -0500] wrote:
>
> >   > Lexical binding isn't the issue - the variable is global in scope.
> >   > The issue is that when you explicitly run the byte-compiler in batch
> mode,
> >   > the "defvar" expression is only compiled, not evaluated, while the
> defmacro
> >   > is evaluated, and the application of the macro function is evaluated
> during
> >   > compilation.
> >
> > Should we change this to evaluate defvar in batch mode?
>
> The byte-compiler avoids evaluating the defvar also in non-batch mode.
>
> So the question is how much of the code that it is compiling should the
> byte-compiler evaluate.
>
> We may need to reuse or possibly extend the notions of Lisp code safety
> we have before the byte-compiler can start evaluating arbitrary
> expressions.  I'm not sure it's worth the complexity.
>

I didn't word my response very precisely.  I mentioned batch mode because
the user was byte compiling in a separate batch process, so it seemed
clearer to say that the compile time and evaluation time environments must
be distinct in that case.
The difference I described above happens any time a file is compiled as a
unit and not evaluated.
One reason to not make defvar (and defun) take effect in the compile-time
environment is that newcomers to across may mistakenly believe that
changing the value of a variable should be able to dynamically change the
code produced by instances already loaded.  In other words, believing that
macro expansion can be dynamic, when in fact the result of the expansion is
a constant embedded in the compiled file (modulo optimization by the byte
compiler).

Lynn

[-- Attachment #2: Type: text/html, Size: 2484 bytes --]

Re: Using the content of a dynamic variable in a macro

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1392 bytes --]

On Sat, Feb 25, 2023 at 09:34:07AM +0100, Damien Cassou wrote:
> I tried to send the mail below to help-gnu-emacs@gnu.org (without being
> subscribed) but it never reached the archive:
> https://lists.gnu.org/archive/html/help-gnu-emacs/2023-02/threads.html.
> 
> Hi,
> 
> I'm wondering why the code below works but won't compile.
> 
> foo.el:
>   (defvar foo-var '((message "hello world")))
>   
>   (defmacro foo-macro ()
>     `(progn
>        ,@foo-var))
>   
>   (defun foo-fun ()
>     (foo-macro))
> 
> 
> $ emacs --batch -l foo.el --eval '(foo-fun)'
> hello world
> 
> $ emacs --batch --eval '(find-file "foo.el")' --eval '(emacs-lisp-byte-compile)'
> In toplevel form:
> foo.el:32:1: Error: Symbol’s value as variable is void: foo-var
> 
> Why isn't the compiler aware of the foo-var variable?

Because the (dynamic) variable comes into existence later,
at run time (at compile time the (defvar ...) hasn't yet
been executed).

You might try to wrap your defvar in an (eval-when-compile ...)
form. Other more knowledgeable folks around here might chime
in whether (or when) this may be a good idea or not.

But the main takeaway is that compile time (and thus macro
expansion time) and run time are kind of different "worlds".

(See also eval-and-compile, but in this case you'll be fine
with eval-when-compile, I think)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Using the content of a dynamic variable in a macro

[Basil Contovounesios]

Damien Cassou [2023-02-25 09:34 +0100] wrote:

> I'm wondering why the code below works but won't compile.
>
> foo.el:
>   (defvar foo-var '((message "hello world")))
>   
>   (defmacro foo-macro ()
>     `(progn
>        ,@foo-var))
>   
>   (defun foo-fun ()
>     (foo-macro))
>
>
> $ emacs --batch -l foo.el --eval '(foo-fun)'
> hello world
>
> $ emacs --batch --eval '(find-file "foo.el")' --eval '(emacs-lisp-byte-compile)'
> In toplevel form:
> foo.el:32:1: Error: Symbol’s value as variable is void: foo-var
>
> Why isn't the compiler aware of the foo-var variable?

[ Lynn and Tomas have already answered this, but just to put it all
  together and cite the manual: ]

The reason is that the byte-compiler generally avoids evaluating the
source it is compiling, with the exception of forms like 'require' and
'eval-and-compile'.

But at the same time, the byte-compiler needs to expand all macro calls.

So in foo-fun it tries to expand the call to foo-macro, which inspects
the value of foo-var, but foo-var's initial value has not yet been
evaluated (by the byte-compiler, that is).

To load a definition into the byte-compiler, it needs to be placed in
either eval-when-compile (for things specific to the byte-compilation
process, such as loading macro definitions from another library),
eval-and-compile (for things that are needed at load time as well, such
as defining a missing function or modifying load-path), or a separate
library that is then 'require'd.

'require' is probably the least likely to confuse the reader, but is
often overkill.  In the example foo-var is only ever used during
macroexpansion, so eval-when-compile is okay - the variable will be
completely omitted from the .elc, but no load/run time code uses it
anyway.

But IME this is a rare use-case for global variables, which are more
often also present in the resulting .elc - so eval-and-compile may be
safer or more appropriate depending on the context.  It is also closer
to 'require'.

This stuff is briefly touched on under "(elisp) Compilation Functions":

--8<---------------cut here---------------start------------->8---
   Be careful when writing macro calls in files that you intend to
byte-compile.  Since macro calls are expanded when they are compiled,
the macros need to be loaded into Emacs or the byte compiler will not do
the right thing.  The usual way to handle this is with ‘require’ forms
which specify the files containing the needed macro definitions (see
Named Features).  Normally, the byte compiler does not evaluate the
code that it is compiling, but it handles ‘require’ forms specially, by
loading the specified libraries.  To avoid loading the macro definition
files when someone _runs_ the compiled program, write
‘eval-when-compile’ around the ‘require’ calls (see Eval During
Compile).  For more details, see Compiling Macros.
--8<---------------cut here---------------end--------------->8---

HTH,

-- 
Basil