[native-comp] Small, new bug with macros?

[native-comp] Small, new bug with macros?

[Adam Porter]

Hi Andrea,

I've been testing the latest Docker image with the deferred
compilation.  I noticed what may be a small, new bug.  To reproduce,
you could use this code:

  (setq package-archives '(("gnu" . "https://elpa.gnu.org/packages/")
                           ("melpa" . "https://melpa.org/packages/")))

  (package-initialize)

  (package-install 'quelpa-use-package)

  (use-package bufler
    :quelpa
    (bufler :fetcher github :repo "alphapapa/bufler.el"
            :files (:defaults (:exclude "helm-bufler.el"))))

In the *Async-native-compile-log* buffer, I see:

  Compiling /home/me/tmp/src/emacs/native-comp/.emacs.d/elpa/bufler-20200405.1230/bufler.el...
  Symbol's function definition is void: bufler-defauto-group

bufler-defauto-group is a macro, not a function, and it is defined
before it is called, so I don't understand why the error is happening.
The package loads and works fine in its byte-compiled form, and the
error didn't happen when I was using an earlier Docker image from a
few weeks ago.

Thanks,
Adam

Re: [native-comp] Small, new bug with macros?

[Andrea Corallo]

Adam Porter <adam@alphapapa.net> writes:

> Hi Andrea,
>
> I've been testing the latest Docker image with the deferred
> compilation.  I noticed what may be a small, new bug.  To reproduce,
> you could use this code:
>
>   (setq package-archives '(("gnu" . "https://elpa.gnu.org/packages/")
>                            ("melpa" . "https://melpa.org/packages/")))
>
>   (package-initialize)
>
>   (package-install 'quelpa-use-package)
>
>   (use-package bufler
>     :quelpa
>     (bufler :fetcher github :repo "alphapapa/bufler.el"
>             :files (:defaults (:exclude "helm-bufler.el"))))
>
> In the *Async-native-compile-log* buffer, I see:
>
>   Compiling /home/me/tmp/src/emacs/native-comp/.emacs.d/elpa/bufler-20200405.1230/bufler.el...
>   Symbol's function definition is void: bufler-defauto-group
>
> bufler-defauto-group is a macro, not a function, and it is defined
> before it is called, so I don't understand why the error is happening.
> The package loads and works fine in its byte-compiled form, and the
> error didn't happen when I was using an earlier Docker image from a
> few weeks ago.
>
> Thanks,
> Adam

Hi Adam,

This looks quite similar to
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40126 isn't?

I had no time to look into this but if it's the same issue is good we
have a smaller reproducert to debug on.

Thanks for reporting it.

Andrea

-- 
akrl@sdf.org

Re: [native-comp] Small, new bug with macros?

[Adam Porter]

Andrea Corallo <akrl@sdf.org> writes:

> Adam Porter <adam@alphapapa.net> writes:
>
>> Hi Andrea,
>>
>> I've been testing the latest Docker image with the deferred
>> compilation.  I noticed what may be a small, new bug.  To reproduce,
>> you could use this code:
>>
>>   (setq package-archives '(("gnu" . "https://elpa.gnu.org/packages/")
>>                            ("melpa" . "https://melpa.org/packages/")))
>>
>>   (package-initialize)
>>
>>   (package-install 'quelpa-use-package)
>>
>>   (use-package bufler
>>     :quelpa
>>     (bufler :fetcher github :repo "alphapapa/bufler.el"
>>             :files (:defaults (:exclude "helm-bufler.el"))))
>>
>> In the *Async-native-compile-log* buffer, I see:
>>
>>   Compiling /home/me/tmp/src/emacs/native-comp/.emacs.d/elpa/bufler-20200405.1230/bufler.el...
>>   Symbol's function definition is void: bufler-defauto-group
>>
>> bufler-defauto-group is a macro, not a function, and it is defined
>> before it is called, so I don't understand why the error is happening.
>> The package loads and works fine in its byte-compiled form, and the
>> error didn't happen when I was using an earlier Docker image from a
>> few weeks ago.
>>
>> Thanks,
>> Adam
>
> Hi Adam,
>
> This looks quite similar to
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40126 isn't?
>
> I had no time to look into this but if it's the same issue is good we
> have a smaller reproducert to debug on.

Yes, thanks.

Re: [native-comp] Small, new bug with macros?

[Andrea Corallo]

Hi Adam,

I've looked into the issue.  I think the code in bufler.el generting the
error reduces to something like:

#+BEGIN_SRC elisp
;;; bufler.el  -*- lexical-binding: t; -*-

(defmacro bufler-defauto-group ())

(eval-and-compile
  (if 1
      (bufler-defauto-group)
    (bufler-defauto-group)))
#+BEGIN_END

I think the compiler error is correct because `bufler-defauto-group' is
not defined in the compilation environment.  You should get the same
error from a fresh Emacs instance just byte-compiling.

The fix should be to wrap its definition within `eval-and-compile' too.

Because the native compiler spawn sub-processes for async compilations
it is a little more severe into highlight this tricky issues into the
code.

Changing topic I have no idea why this instead go through.  To me looks
like a bug, isn't?

#+BEGIN_SRC elisp
;;; bufler.el  -*- lexical-binding: t; -*-

(defmacro bufler-defauto-group ())

 (eval-and-compile
   (bufler-defauto-group))
#+BEGIN_END

Thanks

  Andrea

--
akrl@sdf.org

Expansion of macros at compile time in eval-and-compile bodies (was: Re: [native-comp] Small, new bug with macros?)

[Adam Porter]

Hi Andrea,

Andrea Corallo <akrl@sdf.org> writes:

> Hi Adam,
>
> I've looked into the issue.  I think the code in bufler.el generting
> the error reduces to something like:
>
> #+BEGIN_SRC elisp
> ;;; bufler.el  -*- lexical-binding: t; -*-
>
> (defmacro bufler-defauto-group ())
>
> (eval-and-compile
>   (if 1
>       (bufler-defauto-group)
>     (bufler-defauto-group)))
> #+BEGIN_END
>
> I think the compiler error is correct because `bufler-defauto-group'
> is not defined in the compilation environment.  You should get the
> same error from a fresh Emacs instance just byte-compiling.

Yes, you seem to be right about that.  If I run, on that file:

  emacs --batch --file bufler.el -f emacs-lisp-byte-compile

I get the same error:

  Symbol’s function definition is void: bufler-defauto-group

> The fix should be to wrap its definition within `eval-and-compile'
> too.

Yes, that seems to fix it.  But I don't understand why.

There must be something I'm misunderstanding about how loading and
compilation work.  In the Emacs 26.3 Elisp manual, in section 14.3, it
says:

  In order for compilation of macro calls to work, the macros must
  already be defined in Lisp when the calls to them are compiled.  The
  compiler has a special feature to help you do this: if a file being
  compiled contains a ‘defmacro’ form, the macro is defined temporarily
  for the rest of the compilation of that file.

That would seem to imply that the error shouldn't happen, because the
macro is defined in the same file before it is called.

The next paragraph says:

  Byte-compiling a file also executes any ‘require’ calls at top-level
  in the file, so you can ensure that necessary macro definitions are
  available during compilation by requiring the files that define them

Then in section 17.5 of that manual, in the listing for
eval-and-compile, it says:

  This form marks BODY to be evaluated both when you compile the
  containing code and when you run it (whether compiled or not).

  You can get a similar result by putting BODY in a separate file and
  referring to that file with ‘require’.  That method is preferable
  when BODY is large.  Effectively ‘require’ is automatically
  ‘eval-and-compile’, the package is loaded both when compiling and
  executing.

Does that mean that the body of an eval-and-compile form is evaluated in
a different environment than the one defined by the file containing the
eval-and-compile form?  If so, putting the macro definition in an
eval-and-compile form wouldn't seem to imply that a macro defined in it
would be available in subsequent eval-and-compile forms.  Does that mean
there's some kind of parallel environment for bodies of any
eval-and-compile forms in a file?

Maybe I'm just missing something obvious, but I wonder if the manual is
missing an explanation of this finer point.  It does say, "Most uses of
‘eval-and-compile’ are fairly sophisticated."

> Because the native compiler spawn sub-processes for async compilations
> it is a little more severe into highlight this tricky issues into the
> code.

Maybe so.  But as long as it has the same behavior as the byte-compiler,
that seems correct, right?

> Changing topic I have no idea why this instead go through.  To me looks
> like a bug, isn't?
>
> #+BEGIN_SRC elisp
> ;;; bufler.el  -*- lexical-binding: t; -*-
>
> (defmacro bufler-defauto-group ())
>
>  (eval-and-compile
>    (bufler-defauto-group))
> #+BEGIN_END

You mean that byte-compiling that file doesn't produce an error, right?
And, yes, you're right, it doesn't.  That confuses me as well.  I don't
understand what's going on here.

In case this does indicate a need to improve the manual or fix a minor
bug, I'm moving this to a new thread.

I have a sense of déjà vu about this; I think a similar issue came up
several weeks ago, which I also asked you about.  So thanks for your
help here, and my apologies if I'm using your time on an issue that
isn't your problem.

Re: Expansion of macros at compile time in eval-and-compile bodies

[Stefan Monnier]

>   In order for compilation of macro calls to work, the macros must
>   already be defined in Lisp when the calls to them are compiled.  The
>   compiler has a special feature to help you do this: if a file being
>   compiled contains a ‘defmacro’ form, the macro is defined temporarily
>   for the rest of the compilation of that file.
>
> That would seem to imply that the error shouldn't happen, because the
> macro is defined in the same file before it is called.

The "temporarily" part is too vague to precisely describe what happens.
What really happens is that the compiler remembers the macros it has
compiled in the current file so that when it seen a call to them it can
expand it.  But it's not defined globally, so any macro-expansion that
is not performed directly by the byte-compiler doesn't actually see it.

Whether the provided example should or should not work depends on how we
think macro-expansion within `eval-and-compile` should behave.
Maybe Common-Lisp's `eval-when` specifies this more precisely, and that
could be a motivation to make the behavior of `eval-and-compile` more
precise as well (and that might also motivate changing its behavior in
corner cases like the above, tho not necessarily).

>   You can get a similar result by putting BODY in a separate file and
>   referring to that file with ‘require’.

Clearly, if you put BODY into a separate file and `require` it, BODY
won't be macro-expanded directly by the compiler that compiles the
`require` (it should ideally be macro-expanded in a separate compilation
phase before compiling the file that `require`s).

> Does that mean that the body of an eval-and-compile form is evaluated in
> a different environment than the one defined by the file containing the
> eval-and-compile form?

Pretty much.  tho I don't think any code relies on that detail.

> If so, putting the macro definition in an
> eval-and-compile form wouldn't seem to imply that a macro defined in it
> would be available in subsequent eval-and-compile forms.

Yes, it does, because the two forms contained within those two
`eval-and-compile` will be executed in sequence within the same Emacs
session, so the first will globally define the macro, which will thus be
defined when macro-expanding the second.

> Maybe I'm just missing something obvious, but I wonder if the manual is
> missing an explanation of this finer point.

A lof of such finer points are purposefully not documented because the
exact behavior will depend on circumstances (we want to leave some
freedom to the implementation).

IOW, the user of those macros should avoid relying on those
finer points.

>> Changing topic I have no idea why this instead go through.  To me looks
>> like a bug, isn't?
>>
>> #+BEGIN_SRC elisp
>> ;;; bufler.el  -*- lexical-binding: t; -*-
>>
>> (defmacro bufler-defauto-group ())
>>
>>  (eval-and-compile
>>    (bufler-defauto-group))
>> #+BEGIN_END

I think it's a side-effect of `byte-compile-recurse-toplevel`, which
tries to implement the behavior described in CLHS section
3.2.3.1, "Processing of Top Level Forms".

Maybe this could be considered as a bug, but we could also just say that
it's one of those finer points on which one shouldn't rely.


        Stefan