bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Eli Zaretskii]

From: eliz@HOME-C4E4A596F7.i-did-not-set--mail-host-address--so-tickle-me
--text follows this line--
I made some local modifications in comp.el, and used that modified
comp.el for a while (by loading it manually at run time) to debug some
problem.  Then I undid those local modifications of comp.el (by saying
"git checkout") and said "make" to rebuild Emacs.  And I see this:

    ELC      emacs-lisp/comp.elc
  Warning: Eager macro-expansion skipped due to cycle:
      => (load "comp.el") => (macroexpand-all (defalias 'comp-add-call-cstr  )) => (macroexpand (comp-loop-insn-in-block  )) => (load "comp.el")

Why does this happen? could it be that while comp.el was modified it
got native-compiled, and now that inconsistent comp.eln gets in the
way?  Anything I can check to find out the culprit?

In GNU Emacs 28.0.50 (build 26, i686-pc-mingw32)
 of 2021-03-09 built on HOME-C4E4A596F7
Repository revision: 79c83f79c5b618cb9ef5eca7be2245f15ff54626
Repository branch: feature/native-comp
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)

Configured using:
 'configure -C --prefix=/d/usr --with-wide-int --with-native-compilation
 --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
iso-transl tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win
w32-vars term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face pcase macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads w32notify
w32 lcms2 multi-tty make-network-process nativecomp emacs)

Memory information:
((conses 16 74162 8424)
 (symbols 48 7913 0)
 (strings 16 22637 2689)
 (string-bytes 1 661551)
 (vectors 16 14091)
 (vector-slots 8 278584 12025)
 (floats 8 24 40)
 (intervals 40 255 60)
 (buffers 888 10))

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> From: eliz@HOME-C4E4A596F7.i-did-not-set--mail-host-address--so-tickle-me
> --text follows this line--
> I made some local modifications in comp.el, and used that modified
> comp.el for a while (by loading it manually at run time) to debug some
> problem.  Then I undid those local modifications of comp.el (by saying
> "git checkout") and said "make" to rebuild Emacs.  And I see this:
>
>     ELC      emacs-lisp/comp.elc
>   Warning: Eager macro-expansion skipped due to cycle:
>       => (load "comp.el") => (macroexpand-all (defalias 'comp-add-call-cstr  )) => (macroexpand (comp-loop-insn-in-block  )) => (load "comp.el")
>
> Why does this happen?

I'm not sure, but I can reproduce it.

I guess (just a guess) it might be because `comp-add-call-cstr' is using
a macro `comp-loop-insn-in-block' that is expanding with a
`cl-symbol-macrolet' inside? (no idea why this should be a problem).

> could it be that while comp.el was modified it
> got native-compiled, and now that inconsistent comp.eln gets in the
> way?

I think this has not to do specifically with native compilation.

I see it goes away removing '(setq load-prefer-newer t)' from the
invocation tho.

Perhaps somebody already more into the macro expansion business might
have some suggestion.

Thanks

  Andrea

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: 47049@debbugs.gnu.org
> Date: Wed, 10 Mar 2021 16:05:10 +0000
> 
> > I made some local modifications in comp.el, and used that modified
> > comp.el for a while (by loading it manually at run time) to debug some
> > problem.  Then I undid those local modifications of comp.el (by saying
> > "git checkout") and said "make" to rebuild Emacs.  And I see this:
> >
> >     ELC      emacs-lisp/comp.elc
> >   Warning: Eager macro-expansion skipped due to cycle:
> >       => (load "comp.el") => (macroexpand-all (defalias 'comp-add-call-cstr  )) => (macroexpand (comp-loop-insn-in-block  )) => (load "comp.el")
> >
> > Why does this happen?
> 
> I'm not sure, but I can reproduce it.
> 
> I guess (just a guess) it might be because `comp-add-call-cstr' is using
> a macro `comp-loop-insn-in-block' that is expanding with a
> `cl-symbol-macrolet' inside? (no idea why this should be a problem).
> 
> > could it be that while comp.el was modified it
> > got native-compiled, and now that inconsistent comp.eln gets in the
> > way?
> 
> I think this has not to do specifically with native compilation.
> 
> I see it goes away removing '(setq load-prefer-newer t)' from the
> invocation tho.
> 
> Perhaps somebody already more into the macro expansion business might
> have some suggestion.

Stefan, can you help, please?

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Stefan Monnier]

>> > I made some local modifications in comp.el, and used that modified
>> > comp.el for a while (by loading it manually at run time) to debug some
>> > problem.  Then I undid those local modifications of comp.el (by saying
>> > "git checkout") and said "make" to rebuild Emacs.  And I see this:
>> >
>> >     ELC      emacs-lisp/comp.elc
>> >   Warning: Eager macro-expansion skipped due to cycle:
>> >       => (load "comp.el") => (macroexpand-all (defalias 'comp-add-call-cstr  )) => (macroexpand (comp-loop-insn-in-block  )) => (load "comp.el")
>> >
>> > Why does this happen?
>> 
>> I'm not sure, but I can reproduce it.
>> 
>> I guess (just a guess) it might be because `comp-add-call-cstr' is using
>> a macro `comp-loop-insn-in-block' that is expanding with a
>> `cl-symbol-macrolet' inside? (no idea why this should be a problem).
>> 
>> > could it be that while comp.el was modified it
>> > got native-compiled, and now that inconsistent comp.eln gets in the
>> > way?
>> 
>> I think this has not to do specifically with native compilation.
>> 
>> I see it goes away removing '(setq load-prefer-newer t)' from the
>> invocation tho.
>> 
>> Perhaps somebody already more into the macro expansion business might
>> have some suggestion.
>
> Stefan, can you help, please?

Nothing jumps at me by looking at the code.  I'll have to look at the
actual complete stacktrace I think.


        Stefan

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> > I made some local modifications in comp.el, and used that modified
>>> > comp.el for a while (by loading it manually at run time) to debug some
>>> > problem.  Then I undid those local modifications of comp.el (by saying
>>> > "git checkout") and said "make" to rebuild Emacs.  And I see this:
>>> >
>>> >     ELC      emacs-lisp/comp.elc
>>> >   Warning: Eager macro-expansion skipped due to cycle:
>>> >       => (load "comp.el") => (macroexpand-all (defalias 'comp-add-call-cstr  )) => (macroexpand (comp-loop-insn-in-block  )) => (load "comp.el")
>>> >
>>> > Why does this happen?
>>>
>>> I'm not sure, but I can reproduce it.
>>>
>>> I guess (just a guess) it might be because `comp-add-call-cstr' is using
>>> a macro `comp-loop-insn-in-block' that is expanding with a
>>> `cl-symbol-macrolet' inside? (no idea why this should be a problem).
>>>
>>> > could it be that while comp.el was modified it
>>> > got native-compiled, and now that inconsistent comp.eln gets in the
>>> > way?
>>>
>>> I think this has not to do specifically with native compilation.
>>>
>>> I see it goes away removing '(setq load-prefer-newer t)' from the
>>> invocation tho.
>>>
>>> Perhaps somebody already more into the macro expansion business might
>>> have some suggestion.
>>
>> Stefan, can you help, please?
>
> Nothing jumps at me by looking at the code.  I'll have to look at the
> actual complete stacktrace I think.

Cool didn't know was so easy to generate backtraces to debug this kind
of issues.  At the bottom I attached the backtrace I produced and had a
look into.

IIUC while *loading* "comp.el" we try to advice `macroexpand' and
consequentially we try to install a trampoline (was already compiled)
for `macroexpand', this is supposed to be done by
`comp-subr-trampoline-install' requiring "comp.el" to be loaded again!

Assuming my analysis is correct I'm not sure what's the best work around
for this condition.  WDYT?

  Andrea

==============
$ touch lisp/emacs-lisp/comp.el 
$ ./src/emacs -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t macroexp--debug-eager t)' -l comp -f batch-byte-native-compile-for-bootstrap lisp/emacs-lisp/comp.el

Debugger entered: (eager-macroexp-cycle)
  internal-macroexpand-for-load((require 'bytecomp) nil)
  load-with-code-conversion("/home/andcor03/emacs2/lisp/emacs-lisp/comp.el" "/home/andcor03/emacs2/lisp/emacs-lisp/comp.el" nil t)
  comp-subr-trampoline-install(macroexpand)
  #f(compiled-function (v1102) #<bytecode -0xbf6b4b02ba1e1b9>)(#f(advice-wrapper :around #<subr macroexpand> cl--sm-macroexpand))
  #f(compiled-function (gv--val) #<bytecode 0x9f608bbba7cb3c2>)(#f(advice-wrapper :around #<subr macroexpand> cl--sm-macroexpand))
  advice--add-function(:around (#f(compiled-function () #<bytecode 0x32e14019df7e91>) . #f(compiled-function (gv--val) #<bytecode 0x9f608bbba7cb3c2>)) cl--sm-macroexpand nil)
  advice-add(macroexpand :around cl--sm-macroexpand)
  #f(compiled-function (arg1 &rest rest) "Make symbol macro definitions.\nWithin the body FORMs, references to the variable NAME will be replaced\nby EXPANSION, and (setq NAME ...) will act like (setf EXPANSION ...)." #<bytecode 0x242ccdd625656f5>)(((insn (car insn-cell))) (let ((insn-cell (comp-block-insns bb))) (while insn-cell (when-let ((match (pcase insn (... ...) (... ...)))) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = (comp-lambda-list-gen (comp-cstr-f-args cstr-f)) for arg in args for cstr = (funcall gen) for target = (comp-cond-cstrs-target-mvar arg insn bb) unless (comp-cstr-p cstr) do (sign\
al 'native-ice (list "Incoherent type specifier for function" f)) when (and target (not ...) (or ... ...)) do (comp-emit-call-cstr target insn-cell cstr)))) (setf insn-cell (cdr insn-cell)))))
  macroexpand((comp-loop-insn-in-block bb (when-let ((match (pcase insn (`... (when-let ... ...)) (`... (when-let ... ...))))) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = (comp-lambda-list-gen (comp-cstr-f-args cstr-f)) for arg in args for cstr = (funcall gen) for target = (comp-cond-cstrs-target-mvar arg insn bb) unless (comp-cstr-p cstr) do (signal 'native-ice (list "Incoherent type specifier for function" f)) when (and target (not (equal comp-cstr-t cstr)) (or (null lhs) (not ...))) do (comp-emit-call-cstr target insn-cell cstr))))) nil)
  macroexp-macroexpand((comp-loop-insn-in-block bb (when-let ((match (pcase insn (`... (when-let ... ...)) (`... (when-let ... ...))))) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = (comp-lambda-list-gen (comp-cstr-f-args cstr-f)) for arg in args for cstr = (funcall gen) for target = (comp-cond-cstrs-target-mvar arg insn bb) unless (comp-cstr-p cstr) do (signal 'native-ice (list "Incoherent type specifier for function" f)) when (and target (not (equal comp-cstr-t cstr)) (or (null lhs) (not ...))) do (comp-emit-call-cstr target insn-cell cstr))))) nil)
  macroexp--expand-all((comp-loop-insn-in-block bb (when-let ((match (pcase insn (`... (when-let ... ...)) (`... (when-let ... ...))))) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = (comp-lambda-list-gen (comp-cstr-f-args cstr-f)) for arg in args for cstr = (funcall gen) for target = (comp-cond-cstrs-target-mvar arg insn bb) unless (comp-cstr-p cstr) do (signal 'native-ice (list "Incoherent type specifier for function" f)) when (and target (not (equal comp-cstr-t cstr)) (or (null lhs) (not ...))) do (comp-emit-call-cstr target insn-cell cstr))))))
  macroexp--all-forms((lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let ((match (pcase insn (... ...) (... ...)))) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = (comp-lambda-list-gen (comp-cstr-f-args cstr-f)) for arg in args for cstr = (funcall gen) for target = (comp-cond-cstrs-target-mvar arg insn bb) unless (comp-cstr-p cstr) do (signal 'native-ice (list "Incoherent type specifier for function" f)) when (and target (not ...) (or ... ...)) do (comp-emit-call-cstr target insn-cell cstr)))))) 2)
  macroexp--expand-all((lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let ((match (pcase insn (... ...) (... ...)))) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = (comp-lambda-list-gen (comp-cstr-f-args cstr-f)) for arg in args for cstr = (funcall gen) for target = (comp-cond-cstrs-target-mvar arg insn bb) unless (comp-cstr-p cstr) do (signal 'native-ice (list "Incoherent type specifier for function" f)) when (and target (not ...) (or ... ...)) do (comp-emit-call-cstr target insn-cell cstr)))))))
  macroexp--all-forms((maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let ((match (pcase insn ... ...))) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = (comp-lambda-list-gen ...) for arg in args for cstr = (funcall gen) for target = (comp-cond-cstrs-target-mvar arg insn bb) unless (comp-cstr-p cstr) do (signal ... ...) when (and target ... ...) do (comp-emit-call-cstr target insn-cell cstr)))))) (comp-func-blocks comp-func)) 1)
  macroexp--expand-all((maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let ((match (pcase insn ... ...))) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = (comp-lambda-list-gen ...) for arg in args for cstr = (funcall gen) for target = (comp-cond-cstrs-target-mvar arg insn bb) unless (comp-cstr-p cstr) do (signal ... ...) when (and target ... ...) do (comp-emit-call-cstr target insn-cell cstr)))))) (comp-func-blocks comp-func)))
  macroexpand-all((maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let ((match (pcase insn ... ...))) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = (comp-lambda-list-gen ...) for arg in args for cstr = (funcall gen) for target = (comp-cond-cstrs-target-mvar arg insn bb) unless (comp-cstr-p cstr) do (signal ... ...) when (and target ... ...) do (comp-emit-call-cstr target insn-cell cstr)))))) (comp-func-blocks comp-func)) nil)
  cl--block-wrapper--cmacro((cl--block-wrapper (catch '--cl-block---cl-finish---- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let (...) (cl-multiple-value-bind ... match ...)))) (comp-func-blocks comp-func)))) (catch '--cl-block---cl-finish---- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let ((match ...)) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = ... for arg in args for cstr = ... for target = ... unless ... do ... when ... do ...))))) (comp-func-blocks comp-func))))
  apply(cl--block-wrapper--cmacro (cl--block-wrapper (catch '--cl-block---cl-finish---- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let (...) (cl-multiple-value-bind ... match ...)))) (comp-func-blocks comp-func)))) (catch '--cl-block---cl-finish---- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let ((match ...)) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = ... for arg in args for cstr = ... for target = ... unless ... do ... when ... do ...))))) (comp-func-blocks comp-func))))
  macroexp--compiler-macro(cl--block-wrapper--cmacro (cl--block-wrapper (catch '--cl-block---cl-finish---- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let (...) (cl-multiple-value-bind ... match ...)))) (comp-func-blocks comp-func)))))
  macroexp--expand-all((cl-block --cl-finish-- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let ((match ...)) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = ... for arg in args for cstr = ... for target = ... unless ... do ... when ... do ...))))) (comp-func-blocks comp-func))))
  macroexp--all-forms((progn (cl-block --cl-finish-- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let (...) (cl-multiple-value-bind ... match ...)))) (comp-func-blocks comp-func))) nil) 1)
  macroexp--expand-all((progn (cl-block --cl-finish-- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let (...) (cl-multiple-value-bind ... match ...)))) (comp-func-blocks comp-func))) nil))
  macroexpand-all((progn (cl-block --cl-finish-- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let (...) (cl-multiple-value-bind ... match ...)))) (comp-func-blocks comp-func))) nil) nil)
  cl--block-wrapper--cmacro((cl--block-wrapper (catch '--cl-block-nil-- (cl-block --cl-finish-- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let ... ...))) (comp-func-blocks comp-func))) nil)) (catch '--cl-block-nil-- (cl-block --cl-finish-- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let (...) (cl-multiple-value-bind ... match ...)))) (comp-func-blocks comp-func))) nil))
  apply(cl--block-wrapper--cmacro (cl--block-wrapper (catch '--cl-block-nil-- (cl-block --cl-finish-- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let ... ...))) (comp-func-blocks comp-func))) nil)) (catch '--cl-block-nil-- (cl-block --cl-finish-- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let (...) (cl-multiple-value-bind ... match ...)))) (comp-func-blocks comp-func))) nil))
  macroexp--compiler-macro(cl--block-wrapper--cmacro (cl--block-wrapper (catch '--cl-block-nil-- (cl-block --cl-finish-- (maphash (lambda (--cl-var-- bb) (comp-loop-insn-in-block bb (when-let ... ...))) (comp-func-blocks comp-func))) nil)))
  macroexp--expand-all((cl-loop for bb being each hash-value of (comp-func-blocks comp-func) do (comp-loop-insn-in-block bb (when-let ((match (pcase insn (... ...) (... ...)))) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = (comp-lambda-list-gen (comp-cstr-f-args cstr-f)) for arg in args for cstr = (funcall gen) for target = (comp-cond-cstrs-target-mvar arg insn bb) unless (comp-cstr-p cstr) do (signal 'native-ice (list "Incoherent type specifier for function" f)) when (and target (not ...) (or ... ...)) do (comp-emit-call-cstr target insn-cell cstr)))))))
  macroexp--all-forms((lambda nil "Add args assumptions for each function of which th..." (cl-loop for bb being each hash-value of (comp-func-blocks comp-func) do (comp-loop-insn-in-block bb (when-let ((match (pcase insn ... ...))) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = (comp-lambda-list-gen ...) for arg in args for cstr = (funcall gen) for target = (comp-cond-cstrs-target-mvar arg insn bb) unless (comp-cstr-p cstr) do (signal ... ...) when (and target ... ...) do (comp-emit-call-cstr target insn-cell cstr))))))) 2)
  macroexp--expand-all(#'(lambda nil "Add args assumptions for each function of which th..." (cl-loop for bb being each hash-value of (comp-func-blocks comp-func) do (comp-loop-insn-in-block bb (when-let ((match ...)) (cl-multiple-value-bind (f cstr-f lhs args) match (cl-loop with gen = ... for arg in args for cstr = ... for target = ... unless ... do ... when ... do ...)))))))
  macroexp--all-forms((defalias 'comp-add-call-cstr #'(lambda nil "Add args assumptions for each function of which th..." (cl-loop for bb being each hash-value of (comp-func-blocks comp-func) do (comp-loop-insn-in-block bb (when-let (...) (cl-multiple-value-bind ... match ...)))))) 1)
  macroexp--expand-all((defalias 'comp-add-call-cstr #'(lambda nil "Add args assumptions for each function of which th..." (cl-loop for bb being each hash-value of (comp-func-blocks comp-func) do (comp-loop-insn-in-block bb (when-let (...) (cl-multiple-value-bind ... match ...)))))))
  macroexpand-all((defalias 'comp-add-call-cstr #'(lambda nil "Add args assumptions for each function of which th..." (cl-loop for bb being each hash-value of (comp-func-blocks comp-func) do (comp-loop-insn-in-block bb (when-let (...) (cl-multiple-value-bind ... match ...)))))))
  internal-macroexpand-for-load((defalias 'comp-add-call-cstr #'(lambda nil "Add args assumptions for each function of which th..." (cl-loop for bb being each hash-value of (comp-func-blocks comp-func) do (comp-loop-insn-in-block bb (when-let (...) (cl-multiple-value-bind ... match ...)))))) t)
  load-with-code-conversion("/home/andcor03/emacs2/lisp/emacs-lisp/comp.el" "/home/andcor03/emacs2/lisp/emacs-lisp/comp.el" nil t)
  command-line-1(("--eval" "(setq load-prefer-newer t macroexp--debug-eager t)" "-l" "comp" "-f" "batch-byte-native-compile-for-bootstrap" "lisp/emacs-lisp/comp.el"))
  command-line()
  normal-top-level()
==============

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>> > I made some local modifications in comp.el, and used that modified
>>>> > comp.el for a while (by loading it manually at run time) to debug some
>>>> > problem.  Then I undid those local modifications of comp.el (by saying
>>>> > "git checkout") and said "make" to rebuild Emacs.  And I see this:
>>>> >
>>>> >     ELC      emacs-lisp/comp.elc
>>>> >   Warning: Eager macro-expansion skipped due to cycle:
>>>> >       => (load "comp.el") => (macroexpand-all (defalias 'comp-add-call-cstr  )) => (macroexpand (comp-loop-insn-in-block  )) => (load "comp.el")
>>>> >
>>>> > Why does this happen?
>>>>
>>>> I'm not sure, but I can reproduce it.
>>>>
>>>> I guess (just a guess) it might be because `comp-add-call-cstr' is using
>>>> a macro `comp-loop-insn-in-block' that is expanding with a
>>>> `cl-symbol-macrolet' inside? (no idea why this should be a problem).
>>>>
>>>> > could it be that while comp.el was modified it
>>>> > got native-compiled, and now that inconsistent comp.eln gets in the
>>>> > way?
>>>>
>>>> I think this has not to do specifically with native compilation.
>>>>
>>>> I see it goes away removing '(setq load-prefer-newer t)' from the
>>>> invocation tho.
>>>>
>>>> Perhaps somebody already more into the macro expansion business might
>>>> have some suggestion.
>>>
>>> Stefan, can you help, please?
>>
>> Nothing jumps at me by looking at the code.  I'll have to look at the
>> actual complete stacktrace I think.
>
> Cool didn't know was so easy to generate backtraces to debug this kind
> of issues.  At the bottom I attached the backtrace I produced and had a
> look into.
>
> IIUC while *loading* "comp.el" we try to advice `macroexpand' and
> consequentially we try to install a trampoline (was already compiled)
> for `macroexpand', this is supposed to be done by
> `comp-subr-trampoline-install' requiring "comp.el" to be loaded again!
>
> Assuming my analysis is correct I'm not sure what's the best work around
> for this condition.  WDYT?

Apparently moving the definition of `comp-subr-trampoline-install'
before the other code in comp.el solves the issue.  Not sure if is the
best way to fix this tho.

  Andrea

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Stefan Monnier]

>>>> > I made some local modifications in comp.el, and used that modified
>>>> > comp.el for a while (by loading it manually at run time) to debug some
>>>> > problem.  Then I undid those local modifications of comp.el (by saying
>>>> > "git checkout") and said "make" to rebuild Emacs.  And I see this:
>>>> >
>>>> >     ELC      emacs-lisp/comp.elc
>>>> >   Warning: Eager macro-expansion skipped due to cycle:
>>>> >       => (load "comp.el") => (macroexpand-all (defalias 'comp-add-call-cstr  )) => (macroexpand (comp-loop-insn-in-block  )) => (load "comp.el")
>>>> >
>>>> > Why does this happen?
>>>>
>>>> I'm not sure, but I can reproduce it.
>>>>
>>>> I guess (just a guess) it might be because `comp-add-call-cstr' is using
>>>> a macro `comp-loop-insn-in-block' that is expanding with a
>>>> `cl-symbol-macrolet' inside? (no idea why this should be a problem).
>>>>
>>>> > could it be that while comp.el was modified it
>>>> > got native-compiled, and now that inconsistent comp.eln gets in the
>>>> > way?
>>>>
>>>> I think this has not to do specifically with native compilation.
>>>>
>>>> I see it goes away removing '(setq load-prefer-newer t)' from the
>>>> invocation tho.
>>>>
>>>> Perhaps somebody already more into the macro expansion business might
>>>> have some suggestion.
>>>
>>> Stefan, can you help, please?
>>
>> Nothing jumps at me by looking at the code.  I'll have to look at the
>> actual complete stacktrace I think.
>
> Cool didn't know was so easy to generate backtraces to debug this kind
> of issues.  At the bottom I attached the backtrace I produced and had a
> look into.
>
> IIUC while *loading* "comp.el" we try to advice `macroexpand' and
> consequentially we try to install a trampoline (was already compiled)
> for `macroexpand', this is supposed to be done by
> `comp-subr-trampoline-install' requiring "comp.el" to be loaded again!
>
> Assuming my analysis is correct I'm not sure what's the best work around
> for this condition.  WDYT?

Does the circularity cause problems later on?
[ As you can see above, this cyclic expansion problem only results in
  a warning rather than an error: the eager macroexpansion fails, but
  that just means we fallback on lazy macroexpansion instead.  ]

If so, then let's first focus on solving that problem.

And if not, then it's probably a good idea to understand why the
problem doesn't show up when the macroexpansion happens lazily:
is it because we just don't go through that cl-symbol-macrolet while
compiling the trampoline?

Maybe you can guess my opinion on the best solution: pre-compile all the
trampoline we may need (and ideally only one trampoline per
number-of-arguments, more or less)  ;-)

But maybe a simpler solution is to move the definition of
`comp-subr-trampoline-install` earlier in the file (before the first
use of `comp-loop-insn-in-block`)?


        Stefan

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>>> > I made some local modifications in comp.el, and used that modified
>>>>> > comp.el for a while (by loading it manually at run time) to debug some
>>>>> > problem.  Then I undid those local modifications of comp.el (by saying
>>>>> > "git checkout") and said "make" to rebuild Emacs.  And I see this:
>>>>> >
>>>>> >     ELC      emacs-lisp/comp.elc
>>>>> >   Warning: Eager macro-expansion skipped due to cycle:
>>>>> >       => (load "comp.el") => (macroexpand-all (defalias 'comp-add-call-cstr  )) => (macroexpand (comp-loop-insn-in-block  )) => (load "comp.el")
>>>>> >
>>>>> > Why does this happen?
>>>>>
>>>>> I'm not sure, but I can reproduce it.
>>>>>
>>>>> I guess (just a guess) it might be because `comp-add-call-cstr' is using
>>>>> a macro `comp-loop-insn-in-block' that is expanding with a
>>>>> `cl-symbol-macrolet' inside? (no idea why this should be a problem).
>>>>>
>>>>> > could it be that while comp.el was modified it
>>>>> > got native-compiled, and now that inconsistent comp.eln gets in the
>>>>> > way?
>>>>>
>>>>> I think this has not to do specifically with native compilation.
>>>>>
>>>>> I see it goes away removing '(setq load-prefer-newer t)' from the
>>>>> invocation tho.
>>>>>
>>>>> Perhaps somebody already more into the macro expansion business might
>>>>> have some suggestion.
>>>>
>>>> Stefan, can you help, please?
>>>
>>> Nothing jumps at me by looking at the code.  I'll have to look at the
>>> actual complete stacktrace I think.
>>
>> Cool didn't know was so easy to generate backtraces to debug this kind
>> of issues.  At the bottom I attached the backtrace I produced and had a
>> look into.
>>
>> IIUC while *loading* "comp.el" we try to advice `macroexpand' and
>> consequentially we try to install a trampoline (was already compiled)
>> for `macroexpand', this is supposed to be done by
>> `comp-subr-trampoline-install' requiring "comp.el" to be loaded again!
>>
>> Assuming my analysis is correct I'm not sure what's the best work around
>> for this condition.  WDYT?
>
> Does the circularity cause problems later on?
> [ As you can see above, this cyclic expansion problem only results in
>   a warning rather than an error: the eager macroexpansion fails, but
>   that just means we fallback on lazy macroexpansion instead.  ]

I'm not aware of any functional problem except the aesthetic of the
warning emitted.

> If so, then let's first focus on solving that problem.
>
> And if not, then it's probably a good idea to understand why the
> problem doesn't show up when the macroexpansion happens lazily:
> is it because we just don't go through that cl-symbol-macrolet while
> compiling the trampoline?

Yes I think that's the case.

> Maybe you can guess my opinion on the best solution: pre-compile all the
> trampoline we may need (and ideally only one trampoline per
> number-of-arguments, more or less)  ;-)

:)

> But maybe a simpler solution is to move the definition of
> `comp-subr-trampoline-install` earlier in the file (before the first
> use of `comp-loop-insn-in-block`)?

Yep that's what I attemped and reported in the other mail, I guess our
mails crossed over the ocean.

Thanks!

  Andrea

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Stefan Monnier]

> Yep that's what I attemped and reported in the other mail, I guess our
> mails crossed over the ocean.

I think it was *under* the ocean...


        Stefan

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Yep that's what I attemped and reported in the other mail, I guess our
>> mails crossed over the ocean.
>
> I think it was *under* the ocean...

Right! I pushed through the underside of the ocean d018584814 that is
fixing this for me.

Eli please have a look if it works for you.

Thanks!

  Andrea

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, 47049@debbugs.gnu.org
> Date: Fri, 12 Mar 2021 21:37:23 +0000
> 
> Right! I pushed through the underside of the ocean d018584814 that is
> fixing this for me.
> 
> Eli please have a look if it works for you.

Looks like it's fixed, closing.

Thanks.

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: 47049@debbugs.gnu.org
> Date: Wed, 10 Mar 2021 16:05:10 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >     ELC      emacs-lisp/comp.elc
> >   Warning: Eager macro-expansion skipped due to cycle:
> >       => (load "comp.el") => (macroexpand-all (defalias 'comp-add-call-cstr  )) => (macroexpand (comp-loop-insn-in-block  )) => (load "comp.el")
> >
> > Why does this happen?
> 
> I'm not sure, but I can reproduce it.

Could this be somehow related to the crashes I see, which you cannot
reproduce?  IOW, does this message mean comp.el is somehow incorrectly
compiled?

bug#47049: [feature/native] macro-expansion cycle when compiling comp.el

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: 47049@debbugs.gnu.org
>> Date: Wed, 10 Mar 2021 16:05:10 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >     ELC      emacs-lisp/comp.elc
>> >   Warning: Eager macro-expansion skipped due to cycle:
>> >       => (load "comp.el") => (macroexpand-all (defalias 'comp-add-call-cstr  )) => (macroexpand (comp-loop-insn-in-block  )) => (load "comp.el")
>> >
>> > Why does this happen?
>> 
>> I'm not sure, but I can reproduce it.
>
> Could this be somehow related to the crashes I see, which you cannot
> reproduce?  IOW, does this message mean comp.el is somehow incorrectly
> compiled?

I don't know for sure but at the same time I don't think so.

I also see `comp-loop-insn-in-block' is used in optimizing code in
comp.el that compiling dynamic scope is not executed.

Thanks

  Andrea