bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Juanma Barranquero]

./temacs --batch  -l loadup --temacs=pbootstrap
Loading loadup.el (source)...
dump mode: pbootstrap
Using load-path (d:/Devel/emacs/repo/trunk/lisp
d:/Devel/emacs/repo/trunk/lisp/emacs-lisp
d:/Devel/emacs/repo/trunk/lisp/progmodes
d:/Devel/emacs/repo/trunk/lisp/language
d:/Devel/emacs/repo/trunk/lisp/international
d:/Devel/emacs/repo/trunk/lisp/textmodes
d:/Devel/emacs/repo/trunk/lisp/vc)
Loading emacs-lisp/byte-run (source)...
Loading emacs-lisp/backquote (source)...
Loading subr (source)...
Loading version (source)...
Loading widget (source)...
Loading custom (source)...
Loading emacs-lisp/map-ynp (source)...
Loading international/mule (source)...
Loading international/mule-conf (source)...
Loading env (source)...
Loading format (source)...
Loading bindings (source)...
Loading window (source)...
Loading d:/Devel/emacs/repo/trunk/lisp/files.el (source)...
[...etc...]
Loading d:/Devel/emacs/repo/trunk/lisp/emacs-lisp/cl-generic.el (source)...
Eager macro-expansion failure: (error "Variable binding depth exceeds
max-specpdl-size")
Eager macro-expansion failure: (error "Variable binding depth exceeds
max-specpdl-size")
Eager macro-expansion failure: (error "Variable binding depth exceeds
max-specpdl-size")
Eager macro-expansion failure: (error "Variable binding depth exceeds
max-specpdl-size")
Compiler-macro error for cl--block-wrapper: (error "Variable binding
depth exceeds max-specpdl-size")
Compiler-macro error for cl--block-wrapper: (error "Variable binding
depth exceeds max-specpdl-size")
Compiler-macro error for cl--block-wrapper: (error "Variable binding
depth exceeds max-specpdl-size")
Eager macro-expansion failure: (error "Variable binding depth exceeds
max-specpdl-size")
Loading d:/Devel/emacs/repo/trunk/lisp/frame.el (source)...



In GNU Emacs 27.0.50 (build 1, x86_64-w64-mingw32)
 of 2019-06-15 built on ODIEFAST
Repository revision: d957164ca3c6271989c66017551093d34a197ecf
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 10.0.18362
System Description: Microsoft Windows 10 Home (v10.0.1903.18362.175)

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark saved where search started
Mark set
Saved text from "./temacs --batch  -l loadup --temacs=pbo"
Making completion list... [2 times]
Quit
Type C-x 1 to delete the help window.


Configured using:
 'configure --prefix=/d/Devel/emacs/repo/trunk --with-modules
 --enable-checking=yes
 --enable-locallisppath=%emacs_dir%/../site-lisp:%emacs_dir%/share/emacs/@VER@/site-lisp:%emacs_dir%/share/emacs/site-lisp
 'CFLAGS=-Og -ggdb3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY W32NOTIFY ACL GNUTLS LIBXML2
HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: ESN
  locale-coding-system: cp1252

Major mode: Fundamental

Minor modes in effect:
  server-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml mml-sec password-cache epa derived epg epg-config
gnus-util rmail rmail-loaddefs text-property-search time-date seq
byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils cl-extra bug-reference
thingatpt help-fns radix-tree help-mode easymenu misearch multi-isearch
server elec-pair pcase subr-x cl-loaddefs cl-lib mule-util tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel dos-w32
ls-lisp disp-table term/w32-win w32-win w32-vars term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core term/tty-colors frame cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads w32notify w32
lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 55691 10449)
 (symbols 48 6410 1)
 (strings 32 19891 1902)
 (string-bytes 1 574925)
 (vectors 16 9851)
 (vector-slots 8 122682 10260)
 (floats 8 23 331)
 (intervals 56 347 26)
 (buffers 992 16))

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Sat, 15 Jun 2019 02:53:18 +0200
> 
> Loading d:/Devel/emacs/repo/trunk/lisp/files.el (source)...
> [...etc...]
> Loading d:/Devel/emacs/repo/trunk/lisp/emacs-lisp/cl-generic.el (source)...
> Eager macro-expansion failure: (error "Variable binding depth exceeds
> max-specpdl-size")
> Eager macro-expansion failure: (error "Variable binding depth exceeds
> max-specpdl-size")
> Eager macro-expansion failure: (error "Variable binding depth exceeds
> max-specpdl-size")
> Eager macro-expansion failure: (error "Variable binding depth exceeds
> max-specpdl-size")
> Compiler-macro error for cl--block-wrapper: (error "Variable binding
> depth exceeds max-specpdl-size")
> Compiler-macro error for cl--block-wrapper: (error "Variable binding
> depth exceeds max-specpdl-size")
> Compiler-macro error for cl--block-wrapper: (error "Variable binding
> depth exceeds max-specpdl-size")
> Eager macro-expansion failure: (error "Variable binding depth exceeds
> max-specpdl-size")
> Loading d:/Devel/emacs/repo/trunk/lisp/frame.el (source)...

Did you try to increase max-specpdl-size?

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Juanma Barranquero]

On Sat, Jun 15, 2019 at 8:29 AM Eli Zaretskii <eliz@gnu.org> wrote:

> Did you try to increase max-specpdl-size?

Nope. Not sure where or how. I have a distant memory of doing so in the past...

Bear with me, I'm just warming up after a long while.

TIA,

    Juanma

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Sat, 15 Jun 2019 12:00:48 +0200
> Cc: 36216@debbugs.gnu.org
> 
> On Sat, Jun 15, 2019 at 8:29 AM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Did you try to increase max-specpdl-size?
> 
> Nope. Not sure where or how. I have a distant memory of doing so in the past...
> 
> Bear with me, I'm just warming up after a long while.

Well, does the same command fails in the same way when invoked from
the command line?  If so, just invoke it with the appropriate --eval
argument that increases the value of max-specpdl-size.

If that doesn't work, and you must re-bootstrap to recreate the
problem, then add that --eval to the command in the Makefile which
triggers these messages, and re-run bootstrap.

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Juanma Barranquero]

> Well, does the same command fails in the same way when invoked from
> the command line?

Once I remove the .elc files, any invocation of

./temacs --batch

(with or without explicitly loading loadup.el or passing the --temacs
arg) produces the same error, because it automatically loads
loadup.el.

> If so, just invoke it with the appropriate --eval
> argument that increases the value of max-specpdl-size.

AFAICS, setting max-specpdl-size in an --eval doesn't work, because
temacs loads loadup.el before processing the --eval. If I instrument
loadup.el to show its value, I get:

$ ./temacs --eval "(setq max-specpdl-size 1450)" --batch
Loading loadup.el (source)...
> (loadup.el) max-specpdl-size = 1300
dump mode: nil

Obviously, the problem disappears if I bind max-specpdl-size to a
bigger value around the load of cl-generic, or if I set it explicitly
in the conditional code at the beginning of loadup.el that also sets
max-lisp-eval-depth

(if (or (member dump-mode '("bootstrap" "pbootstrap"))
;; FIXME this is irritatingly fragile.
        (and (stringp (nth 4 command-line-args))
             (string-match "^unidata-gen\\(\\.elc?\\)?$"
                           (nth 4 command-line-args)))
        (member (nth 7 command-line-args) '("unidata-gen-file"
                                            "unidata-gen-charprop"))
        (null dump-mode))
    (progn
      [...etc etc...]
      (setq max-specpdl-size 1450)   ;;; <=== THIS WORKS
      ;; During bootstrapping the byte-compiler is run interpreted
      ;; when compiling itself, which uses a lot more stack
      ;; than usual.
      (setq max-lisp-eval-depth 2200)))

I wonder if it wouldn't just make sense to borrow the same trick
loadup.el uses with pcase.el to disable eager macroexpansion, i.e.,
something like

diff --git i/lisp/loadup.el w/lisp/loadup.el
index 67e8aa7d40..9f08b19043 100644
--- i/lisp/loadup.el
+++ w/lisp/loadup.el
@@ -246,7 +246,10 @@
 (load "language/cham")

 (load "indent")
-(load "emacs-lisp/cl-generic")
+(if (byte-code-function-p (symbol-function 'macroexpand-all))
+    (load "emacs-lisp/cl-generic")
+  (let ((macroexp--pending-eager-loads '(skip)))
+    (load "emacs-lisp/cl-generic")))
 (load "frame")
 (load "startup")
 (load "term/tty-colors")

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Juanma Barranquero]

On Sun, Jun 16, 2019 at 7:57 AM Juanma Barranquero <lekktu@gmail.com> wrote:

> I wonder if it wouldn't just make sense to borrow the same trick
> loadup.el uses with pcase.el to disable eager macroexpansion, i.e.,
> something like

The idea above doesn't work, it breaks bootstraping while compiling startup.el.

So, back to the beginning. This fixes the reported problem:


diff --git i/lisp/loadup.el w/lisp/loadup.el
index 67e8aa7d40..ca0babd6ed 100644
--- i/lisp/loadup.el
+++ w/lisp/loadup.el
@@ -105,4 +105,8 @@
       ;; We'll probably overflow the pure space.
       (setq purify-flag nil)
+      ;; Set max-specpdl-size to a larger value to avoid a
+      ;; macroexpansion error when loading the cl-generic.el
+      ;; source file (bug#36216).
+      (setq max-specpdl-size 1450)
       ;; Value of max-lisp-eval-depth when compiling initially.

       ;; During bootstrapping the byte-compiler is run interpreted

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Sun, 16 Jun 2019 14:01:22 +0200
> Cc: 36216@debbugs.gnu.org
> 
> diff --git i/lisp/loadup.el w/lisp/loadup.el
> index 67e8aa7d40..ca0babd6ed 100644
> --- i/lisp/loadup.el
> +++ w/lisp/loadup.el
> @@ -105,4 +105,8 @@
>        ;; We'll probably overflow the pure space.
>        (setq purify-flag nil)
> +      ;; Set max-specpdl-size to a larger value to avoid a
> +      ;; macroexpansion error when loading the cl-generic.el
> +      ;; source file (bug#36216).
> +      (setq max-specpdl-size 1450)
>        ;; Value of max-lisp-eval-depth when compiling initially.
> 
>        ;; During bootstrapping the byte-compiler is run interpreted

Since you've established that the problem is not infinite recursion,
any such solution is OK.  But maybe we simply should bump the default
value of that variable?

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Juanma Barranquero]

> But maybe we simply should bump the default value of that variable?

That's certainly OK to me.

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Sun, 16 Jun 2019 16:17:33 +0200
> Cc: 36216@debbugs.gnu.org
> 
> > But maybe we simply should bump the default value of that variable?
> 
> That's certainly OK to me.

Any objections to bumping max-specpld-size from 1300 to 1500?

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Stefan Monnier]

Earlier, Juanma said:
> I wonder if it wouldn't just make sense to borrow the same trick
> loadup.el uses with pcase.el to disable eager macroexpansion, i.e.,
> something like

That pcase situation is different (not only because pcase is not
pre-loaded, contrary to cl-generic, but also because this trick is used
to break a circular dependency, rather than to minimize stack depth).

Eli said:
> Any objections to bumping max-specpld-size from 1300 to 1500?

We've been increasing the stack size bit by bit for a long time now
(and IIRC max-specpld-size is not allocated on the C stack, so we can
go crazy without any risk of crashing on a small C stack), so:
No objection from me.

I do wonder what caused the increase of stack depth, tho: I can't
remember offhand anything recent that would explain it in this part of
the code.


        Stefan

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Juanma Barranquero]

> That pcase situation is different (not only because pcase is not
> pre-loaded, contrary to cl-generic, but also because this trick is used
> to break a circular dependency, rather than to minimize stack depth).

Still, the trick of skipping the macro-expansion sort of worked with
cl-generic, on normal compilations. Broke bootstrapping. And I accept
it was a bit gross.

> We've been increasing the stack size bit by bit for a long time now
> (and IIRC max-specpld-size is not allocated on the C stack, so we can
> go crazy without any risk of crashing on a small C stack), so:
> No objection from me.

Ok, I'll commit it.

> I do wonder what caused the increase of stack depth, tho: I can't
> remember offhand anything recent that would explain it in this part of
> the code.

The problem does not happen (at least on Windows) on the release branch.

In this case, with max-specpdl-size = 1300 it was working, now it
requires on the order of 1435 or so, which is at least a 10% increment
(likely a bit more, assuming it didn't need exactly 1300 before).
That's not very big, but certainly significant.

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Juanma Barranquero]

Sorry, didn't notice John in the Cc: and just pushed the commit.

Commited now as 4156dd384a1b6d999bdbce30507bfab77e49ab4e

Not closing the bug, waiting for John's answer.

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Juanma Barranquero]

> Not closing the bug, waiting for John's answer.

No more comments received, so closing it.

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Stefan Monnier]

> In this case, with max-specpdl-size = 1300 it was working, now it
> requires on the order of 1435 or so, which is at least a 10% increment
> (likely a bit more, assuming it didn't need exactly 1300 before).
> That's not very big, but certainly significant.

Exactly, 10% is quite significant.  It can easily happen for perfectly
legitimate reasons, but there's a chance it's an undesirable side-effect
of some change which has a more global impact.


        Stefan


PS: When I tried it here, `git bisect` pointed the finger at:

    commit afdc20d73c8588e5a744ecf7bffaf4401a557d20 (HEAD, refs/bisect/bad)
    Author: Mattias Engdegård <mattiase@acm.org>
    Date:   Wed May 15 22:44:00 2019 +0200
    
        Allow zero-argument rx `or' and `seq' forms
        
        Make the rx `or' and `seq' forms accept zero arguments to produce a
        never-matching regexp and an empty string, respectively.
        
        * lisp/emacs-lisp/rx.el: Require cl-extra.
        (rx-constituents, rx-or): Permit zero args.
        (rx): Amend doc string for `or' and `seq'.
        * test/lisp/emacs-lisp/rx-tests.el (rx-or, rx-seq): Test the change.
        * etc/NEWS (Changes in Specialized Modes and Packages): Mention the change.

which suggests the 10% increase is localized: I guess we "require
cl-extra" right when we happen to be (near) the deepest recursion and
since it's not yet byte-compiled it causes yet more recursive
eager-macroexpansion.

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Lars Ingebrigtsen]

Juanma Barranquero <lekktu@gmail.com> writes:

> Loading d:/Devel/emacs/repo/trunk/lisp/emacs-lisp/cl-generic.el (source)...
> Eager macro-expansion failure: (error "Variable binding depth exceeds
> max-specpdl-size")

Yeah, this happens for me too when bootstrapping:

Loading /home/larsi/src/emacs/trunk/lisp/emacs-lisp/cl-generic.el (source)...
Eager macro-expansion failure: (error "Variable binding depth exceeds max-specpdl-size")
Eager macro-expansion failure: (error "Variable binding depth exceeds max-specpdl-size")

But nothing seems to actually, like, fail, so I wondered whether this
error message was just wrong, or whether the error was subsequently
fixed when Emacs got to a more properly byte-compiled state...  (Because
the stack is usually much deeper when we're running uncompiled.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#36216: 27.0.50; Variable binding depth exceeds max-specpld-size during bootstrap

[Juanma Barranquero]

On Sat, Jun 15, 2019 at 1:22 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:

> But nothing seems to actually, like, fail, so I wondered whether this
> error message was just wrong, or whether the error was subsequently
> fixed when Emacs got to a more properly byte-compiled state...  (Because
> the stack is usually much deeper when we're running uncompiled.)

It isn't wrong, in the sense that the eager macroexpansion is failing.
It doesn't have lasting consequences, but surely the error must be
avoided somehow.

>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no