Macro Expansion Inconsistency

Macro Expansion Inconsistency

[Alexander Shukaev]

Hello everyone,

Just wasted a couple of hours with the following macro:

(defmacro test
    (name)
  `(let* ((name             ',name)
          (symbol           (intern (concat "some" "-" (symbol-name
name)))))
     ,symbol))

Running

(test xxx)

reports error: (void-variable symbol).

What?

Refactoring to

(defmacro test
    (name)
  `(let* ((name             ',name))
     ,(intern (concat "some" "-" (symbol-name name)))))

and running

(test xxx)

works as expected.

Could anybody clarify what's going on? Thanks.

Regards,
Alexander

Re: Macro Expansion Inconsistency

[Joost Kremers]

Alexander Shukaev wrote:
> (defmacro test
>     (name)
>   `(let* ((name             ',name)
>           (symbol           (intern (concat "some" "-" (symbol-name name)))))
>      ,symbol))
>
> Running
>
> (test xxx)
>
> reports error: (void-variable symbol).
>
> What?
>
> Refactoring to
>
> (defmacro test
>     (name)
>   `(let* ((name             ',name))
>      ,(intern (concat "some" "-" (symbol-name name)))))
>
> and running
>
> (test xxx)
>
> works as expected.
>
> Could anybody clarify what's going on? Thanks.

Just consider when exactly the different parts of your macro are
evaluated. When is something that is quoted in the definition evaluated?
And when is something that is comma-escaped evaluated? Then look
carefully at what is quoted and what is comma-escaped. You'll see that
in the first version, something doesn't match.


-- 
Joost Kremers                                   joostkremers@fastmail.fm
Selbst in die Unterwelt dringt durch Spalten Licht
EN:SiS(9)

Re: Macro Expansion Inconsistency

[Alexander Shukaev]

>
> Just consider when exactly the different parts of your macro are
> evaluated. When is something that is quoted in the definition evaluated?
> And when is something that is comma-escaped evaluated? Then look
> carefully at what is quoted and what is comma-escaped. You'll see that
> in the first version, something doesn't match.


If I would want to think more myself I would not ask for help. Could you
just explain if you know?

Re: Macro Expansion Inconsistency

[John Mastro]

Alexander Shukaev <haroogan@gmail.com> wrote:
> > Just consider when exactly the different parts of your macro are
> > evaluated. When is something that is quoted in the definition evaluated?
> > And when is something that is comma-escaped evaluated? Then look
> > carefully at what is quoted and what is comma-escaped. You'll see that
> > in the first version, something doesn't match.
>
>
> If I would want to think more myself I would not ask for help. Could you
> just explain if you know?

    (defmacro test (name)
      `(let* ((name ',name)
              (symbol (intern (concat "some" "-" (symbol-name name)))))
         ,symbol))
         ^^^^^^^

The comma operator causes something to be evaluated when the macro is
expanded, but the `symbol' binding doesn't exist until runtime (at which
time it exists where the macro expanded, not within the macro's body).

This is another version that will work (notice the lack of quoting):

    (defmacro test (name)
      (let ((symbol (intern (concat "some" "-" (symbol-name name)))))
        symbol))

---
john

Re: Macro Expansion Inconsistency

[Alexander Shukaev]

>
>     (defmacro test (name)
>       `(let* ((name ',name)
>               (symbol (intern (concat "some" "-" (symbol-name name)))))
>          ,symbol))
>          ^^^^^^^
>
> The comma operator causes something to be evaluated when the macro is
> expanded, but the `symbol' binding doesn't exist until runtime (at which
> time it exists where the macro expanded, not within the macro's body).
>
> This is another version that will work (notice the lack of quoting):
>
>     (defmacro test (name)
>       (let ((symbol (intern (concat "some" "-" (symbol-name name)))))
>         symbol))
>

Yes, I get it, but then how do I do something like

(defmacro test
    (name)
  `(let* ((name             ',name))
     (defvar ,(intern (concat "some" "-" (symbol-name name))))))

but using variable `symbol'?

Re: Macro Expansion Inconsistency

[Alexander Shukaev]

I think I found it. It seems like one has to use

(eval `(defvar ,symbol ...

in this case.

Re: Macro Expansion Inconsistency

[John Mastro]

Alexander Shukaev <haroogan@gmail.com> wrote:
> Yes, I get it, but then how do I do something like
>
> (defmacro test
>     (name)
>   `(let* ((name             ',name))
>      (defvar ,(intern (concat "some" "-" (symbol-name name))))))
>
> but using variable `symbol'?

Does this work? If not, can you clarify why the `symbol' variable
needs to be around at evaluation time?

    (defmacro test (name)
      (let ((symbol (intern (concat "some" "-" (symbol-name name)))))
        `(defvar ,symbol)))

-- 
john

Re: Macro Expansion Inconsistency

[Stefan Monnier]

> (defmacro test (name)
>   `(let* ((name             ',name)
>           (symbol           (intern (concat "some" "-" (symbol-name name)))))
>      ,symbol))

You have 2 macros here: the one you define (test) and the one you use
(backquote).  The problem is in the one you *use*.
I.e. you can reproduce your problem by running the following:

   `(let* ((name             ',"hello")
           (symbol           (intern (concat "some" "-" (symbol-name "hello")))))
      ,symbol))

I.e. the `symbol' variable should be bound in `let' that is executed
during the macro expansion whereas you instead return a form that binds
it (so this `let' binding of `symbol' would only happen later when the
expanded code is executed, which here doesn't happen since you first
try to get the value of `symbol' which signal an error since there is
no such variable at that time).


        Stefan

Re: Macro Expansion Inconsistency

[Alexander Shukaev]

I currently have the following macro baked:

(defmacro bm-define-flag
    (name index character foreground)
  (let* ((symbol           (intern (format "bm-%s-flag"
                                           (symbol-name name))))
         (character-symbol (intern (format "%s-character"
                                           (symbol-name symbol)))))
    (put symbol 'index     index)
    (put symbol 'character character-symbol)
    (eval `(defcustom ,character-symbol
             ,character
             "Character for flag."
             :tag "BM Flag Character"
             :group 'buffer-manager
             :type 'character))
    (eval `(defface ,symbol
             `((t :foreground ,foreground
                  :weight     bold))
             "Face for flag."
             :tag "BM Flag Face"
             :group 'buffer-manager-faces))))

and running

(bm-define-flag xxx 777 ?$ "yellow")

gives the expected results, i.e. all the data structures are populated
properly. The only problem is that `defface' seems to be returning
`,symbol' itself (would be `bm-xxx-flag' in the above test case) and that
causes the error: (void-variable bm-xxx-flag) in `eval', i.e. in

eval(bm-xxx-flag nil)

How to overcome it? Do you have any further recommendations on this macro?

P.S. It's my first somewhat serious macro in Emacs Lisp ever, so I'd love
to learn more from pros until I do something dumb.

Re: Macro Expansion Inconsistency

[Rusi]

On Wednesday, December 17, 2014 7:49:21 AM UTC+5:30, Alexander Shukaev wrote:
> How to overcome it? Do you have any further recommendations on this macro?
> 
> P.S. It's my first somewhat serious macro in Emacs Lisp ever, so I'd love
> to learn more from pros until I do something dumb.

Macros are super powerful ... and super hard.

My recommendations?
1. [Just an imperative version of Joost suggestons]:
use macroexpand to debug them ie see (info "(elisp)expansion")
ie if form is a macro-call you can debug it with
first using (macroexpand form)
and if that looks sane then
(eval (macroexpand form))
However...

2. Lisp IS Eval... its evaling all the time so you rarely need to write 'eval'
Macros are double eval-layer So you doubly dont need eval

Re: Macro Expansion Inconsistency

[John Mastro]

Hi Alexander,

Alexander Shukaev <haroogan@gmail.com> wrote:
> John, please, have a look at my latest post. It will give you the whole idea
> of what's going on. Thanks for your time.

You're very close. I think the below will do what you're looking for,
without using `eval'.

The problem you were having with `foreground' is because it's within two
backquotes. You can think of each unquote as peeling away a single level
of quoting.

The `progn' is necessary because the macro, like other functions, can
only return a single value. Since we need to emit seval forms, we wrap
them up in a single `progn'. This wasn't necessary in yours since you
weren't actually emitting any forms from the macro, but it is necessary
in this way of doing it.

I moved the calls to `put' within the backquote just for the example of
how it's sometimes necessary to quote the result of what you've just
unquoted.

Let me know if anything's unclear or if I missed something.

    (defmacro bm-define-flag2 (name index character foreground)
      (let* ((symbol
              (intern (format "bm-%s-flag" (symbol-name name))))
             (character-symbol
              (intern (format "%s-character" (symbol-name symbol)))))
        `(progn
           (put ',symbol 'index ,index)
           (put ',symbol 'character ',character-symbol)
           (defcustom ,character-symbol
             ,character
             :tag "BM Flag Character"
             :group 'buffer-manager
             :type 'character)
           (defface ,symbol
             `((t :foreground ,,foreground
                  :weight bold))
             "Face for flag."
             :tag "BM Flag Face"
             :group 'buffer-manager-faces))))

-- 
john

Re: Macro Expansion Inconsistency

[John Mastro]

John Mastro <john.b.mastro@gmail.com> wrote:
> The problem you were having with `foreground' is because it's within two
> backquotes. You can think of each unquote as peeling away a single level
> of quoting.

Actually, in this case another fix would be to remove the second level
of quoting - it's not needed there. Then you would only need a single
unquote for `foreground'.

-- 
john

Re: Macro Expansion Inconsistency

[Nicolas Richard]

Rusi <rustompmody@gmail.com> writes:
> use macroexpand to debug them ie see (info "(elisp)expansion")

pp-macroexpand-last-sexp is nice too.

-- 
Nicolas Richard

Re: Macro Expansion Inconsistency

[Nicolas Richard]

Alexander Shukaev <haroogan@gmail.com> writes:

> I currently have the following macro baked:
>
> (defmacro bm-define-flag
>     (name index character foreground)
>   (let* ((symbol           (intern (format "bm-%s-flag"
>                                            (symbol-name name))))
>          (character-symbol (intern (format "%s-character"
>                                            (symbol-name symbol)))))
>     (put symbol 'index     index)
>     (put symbol 'character character-symbol)
>     (eval `(defcustom ,character-symbol
>              ,character
>              "Character for flag."
>              :tag "BM Flag Character"
>              :group 'buffer-manager
>              :type 'character))
>     (eval `(defface ,symbol
>              `((t :foreground ,foreground
>                   :weight     bold))
>              "Face for flag."
>              :tag "BM Flag Face"
>              :group 'buffer-manager-faces))))
> [...]
> causes the error: (void-variable bm-xxx-flag) in `eval'
> [...]
> How to overcome it?

Quick'n'very-dirty : add nil or t (or anything that can be evaluated
twice safely) at the end of your macro definition.

> Do you have any further recommendations on this macro?

It's awful :) But don't worry : it's easy to write awful things, and
it's good that you post about it here.

In your case I'd point out these problems :
(i) it uses explicit eval for no good reason
(ii) it doesn't return code (that's the error you see)
(iii) it relies on dynamical binding (because of ',foreground' in a
second layer of quasi-quoting, which is only evalled by the explicit
eval.).

I think this works (lightly tested) :

(defmacro bm-define-flag (name index character foreground)
  (let* ((symbol           (intern (format "bm-%s-flag"
                                           (symbol-name name))))
         (character-symbol (intern (format "%s-character"
                                           (symbol-name symbol)))))
    (put symbol 'index     index)
    (put symbol 'character character-symbol)
    `(progn ;; this is the code we're going to return after expansion.
            ;; Emacs will eval the whole progn form.
       (defcustom ,character-symbol
         ,character
         "Character for flag."
         :tag "BM Flag Character"
         :group 'buffer-manager
         :type 'character)
       (defface ,symbol
         '((t :foreground ,foreground ;; the comma can be used,
                                      ;; even after normal quoting
               :weight     bold))
         "Face for flag."
         :tag "BM Flag Face"
         :group 'buffer-manager-faces))))


If you'd like to read more about macros, I wrote about them at 
http://lists.gnu.org/archive/html/help-gnu-emacs/2014-09/msg00473.html
(there's a typo in the first paragraph, apply
`sed s/evaluation time/expansion time/' please)

HTH,

-- 
Nicolas Richard

Re: Macro Expansion Inconsistency

[Alexander Shukaev]

Thank you very much, John and Nicolas. I think I finally understand how
macros expand in Emacs Lisp better now. I'm sure it's not a coincidence
that you both came up with almost identical solutions. It suggests that the
style of returning forms for evaluation from macro is probably the best way
to go in most cases when `let' is involved. So the final tested solution is:

(defmacro bm-define-flag
    (name index character foreground)
  (let* ((symbol           (intern (format "bm-%s-flag"
                                           (symbol-name name))))
         (character-symbol (intern (format "%s-character"
                                           (symbol-name symbol)))))
    (put symbol 'index     index)
    (put symbol 'character character-symbol)
    `(progn
       (defcustom ,character-symbol
         ,character
         "Character for flag."
         :tag "BM Flag Character"
         :group 'buffer-manager
         :type 'character)
       (defface ,symbol
         '((t :foreground ,foreground
              :weight     bold))
         "Face for flag."
         :tag "BM Flag Face"
         :group 'buffer-manager-faces))))

(put 'bm-define-flag 'lisp-indent-function 'defun)

As a final exam for myself, considering what I've learned, I've implemented
a collective definition macro as well:

(defmacro bm-define-flags
    (name index character foreground &rest ...)
  (let ((name       `,name)
        (index      `,index)
        (character  `,character)
        (foreground `,foreground)
        (...        `(,@...))
        (body       '()))
    (while name
      (push `(bm-define-flag ,name ,index ,character ,foreground) body)
      (setq name       (pop ...)
            index      (pop ...)
            character  (pop ...)
            foreground (pop ...)))
    `(progn ,@body)))

(put 'bm-define-flags 'lisp-indent-function 'defun)

How do you like this one? Now I simply write:

(bm-define-flags
  delete    0 ?D "#FF0000"
  open      0 ?O "#00FFFF"
  previous  0 ?P nil
  modified  1 ?M "#FFFF00"
  write     1 ?W "#00FF00"
  read-only 2 ? "#FF0000")

Looks absolutely amazing ;)

Kind regards,
Alexander

Re: Macro Expansion Inconsistency

[Stefan Monnier]

> (defmacro bm-define-flag
>     (name index character foreground)
[...]
> (put 'bm-define-flag 'lisp-indent-function 'defun)

This is old-style.  The current style (since Emacs-22, IIRC) is:

   (defmacro bm-define-flag
       (name index character foreground)
     (declare (indent defun))
     ...)


-- Stefan

Re: Macro Expansion Inconsistency

[Nicolas Richard]

Le 17/12/2014 15:03, Alexander Shukaev a écrit :
> Thank you very much, John and Nicolas. I think I finally understand
> how macros expand in Emacs Lisp better now. I'm sure it's not a
> coincidence that you both came up with almost identical solutions. It
> suggests that the style of returning forms for evaluation from macro
> is probably the best way to go in most cases when `let' is involved.

I hadn't seen John's answer. His solution is better than mine because
the (put ...) forms should indeed be included in the progn. (otherwise
they get executed at expansion time, and I think that'll not be good if
the code is byte-compiled.)

> As a final exam for myself, considering what I've learned, I've implemented a collective definition macro as well:
> 
> (defmacro bm-define-flags
>     (name index character foreground &rest ...)
>   (let ((name       `,name)
>         (index      `,index)
>         (character  `,character)
>         (foreground `,foreground)
>         (...        `(,@...))

I think the above bindings are unnecessary.

>         (body       '()))
>     (while name
>       (push `(bm-define-flag ,name ,index ,character ,foreground) body)
>       (setq name       (pop ...)
>             index      (pop ...)
>             character  (pop ...)
>             foreground (pop ...)))
>     `(progn ,@body)))

Looks ok to me (but I'm not experienced).

> (put 'bm-define-flags 'lisp-indent-function 'defun)

I don't think that's useful for bm-define-flags. Btw you can also use a
(declare ...) form in your macro definition to achieve this goal.

-- 
Nicolas.

Re: Macro Expansion Inconsistency

[Alexander Shukaev]

>
> This is old-style.  The current style (since Emacs-22, IIRC) is:
>
>    (defmacro bm-define-flag
>        (name index character foreground)
>      (declare (indent defun))
>      ...)


​Sigh, Emacs is so big to be able to know these details on time
... Thank you for pointing this out.
​

Re: Macro Expansion Inconsistency

[Alexander Shukaev]

>
> > (defmacro bm-define-flags
> >     (name index character foreground &rest ...)
> >   (let ((name       `,name)
> >         (index      `,index)
> >         (character  `,character)
> >         (foreground `,foreground)
> >
> ​​
> (...        `(,@...))
>
> I think the above bindings are unnecessary.


Feels like

​
(...        `(,@...))

is absolutely necessary here anyway or?
​

Re: Macro Expansion Inconsistency

[Nicolas Richard]

Alexander Shukaev <haroogan@gmail.com> writes:
> Feels like
>
> ​
> (...        `(,@...))
>
> is absolutely necessary here anyway or?

Seems superfluous to me.

(let ((... '(foo bar baz)))
     (equal ...   `(,@...)))
 => t

-- 
Nicolas Richard

Re: Macro Expansion Inconsistency

[Alexander Shukaev]

>
> Seems superfluous to me.
>
> (let ((... '(foo bar baz)))
>      (equal ...   `(,@...)))
>  => t


​Ah, yes, you are of course right! Thank you for your feedback.​

Re: Macro Expansion Inconsistency

[Barry Margolin]

In article <mailman.16290.1418825041.1147.help-gnu-emacs@gnu.org>,
 Alexander Shukaev <haroogan@gmail.com> wrote:

> Thank you very much, John and Nicolas. I think I finally understand how
> macros expand in Emacs Lisp better now. I'm sure it's not a coincidence
> that you both came up with almost identical solutions. It suggests that the
> style of returning forms for evaluation from macro is probably the best way
> to go in most cases when `let' is involved.

Not just "probably the best" -- this is the ONLY way to go in macros. 
The whole point of macros is to produce new code to be evaluated in 
place of the original code.

The important thing to remember about macros is that they run when 
you're COMPILING the code, not when you run it. These may be 
approximately the same time if you evaluate the calling code (e.g. by 
typing it into the *scratch* buffer, loading the source file, M-x 
eval-region). This is why calling eval seems to work when you're testing 
the macro interactively.

But if you byte-compile the file, it will be run then, not when you 
later load the .elc file. This will usually be a totally different Emacs 
session, so it should be obvious why the results of calling eval won't 
be available then.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Macro Expansion Inconsistency

[Alexander Shukaev]

>
> Not just "probably the best" -- this is the ONLY way to go in macros.
> The whole point of macros is to produce new code to be evaluated in
> place of the original code.
>
> The important thing to remember about macros is that they run when
> you're COMPILING the code, not when you run it. These may be
> approximately the same time if you evaluate the calling code (e.g. by
> typing it into the *scratch* buffer, loading the source file, M-x
> eval-region). This is why calling eval seems to work when you're testing
> the macro interactively.
>
> But if you byte-compile the file, it will be run then, not when you
> later load the .elc file. This will usually be a totally different Emacs
> session, so it should be obvious why the results of calling eval won't
> be available then.


​Nice explanation, it indeed does make sense. Thank you.​