bug#31549: 25.3; bytecompile fails with eval-when-compile

bug#31549: 25.3; bytecompile fails with eval-when-compile

[ynyaaa]

(byte-compile `(eval-when-compile (list ,@(make-list 2047 0))))

Evaluating the form above, emacs (with -Q option) reports
  Error: Memory exhausted--use C-x s then exit and restart Emacs
in *Compile-Log* buffer.
The returned value of the form is t.




In GNU Emacs 25.3.1 (x86_64-w64-mingw32)
 of 2017-09-27 built on LAPHROAIG
Windowing system distributor 'Microsoft Corp.', version 6.3.9600
Configured using:
 'configure --without-dbus --without-compress-install 'CFLAGS=-O2
 -static -g3' PKG_CONFIG_PATH=/mingw64/lib/pkgconfig'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS

Important settings:
  value of $LANG: JPN
  locale-coding-system: cp932

Major mode: Emacs-Lisp

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util help-fns mail-prsvr mail-utils
warnings byte-opt bytecomp byte-compile cl-extra help-mode easymenu
cl-loaddefs pcase cl-lib cconv time-date mule-util japan-util tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
dos-w32 ls-lisp disp-table w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame
cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote w32notify w32 multi-tty
make-network-process emacs)

Memory information:
((conses 16 96577 5188)
 (symbols 56 20186 0)
 (miscs 48 54 117)
 (strings 32 17426 3747)
 (string-bytes 1 488163)
 (vectors 16 13751)
 (vector-slots 8 513163 14478)
 (floats 8 165 49)
 (intervals 56 237 13)
 (buffers 976 19))

bug#31549: 25.3; bytecompile fails with eval-when-compile

[Noam Postavsky]

ynyaaa@gmail.com writes:

> (byte-compile `(eval-when-compile (list ,@(make-list 2047 0))))
>
> Evaluating the form above, emacs (with -Q option) reports
>   Error: Memory exhausted--use C-x s then exit and restart Emacs
> in *Compile-Log* buffer.
> The returned value of the form is t.

The error is coming from exec_byte_code:

  if (MAX_ALLOCA / word_size <= XFASTINT (maxdepth))
    memory_full (SIZE_MAX);

It's more like a Lisp stack overflow than a memory exhausted situation
though, hardly calls for restarting Emacs.  Perhaps the byte compiler
should refuse to compile such a large expression?

I can't reproduce in Emacs 26, but only because MAX_ALLOCA is bigger, I
think.

bug#31549: 25.3; bytecompile fails with eval-when-compile

[Eli Zaretskii]

> Resent-Sender: help-debbugs@gnu.org
> From: Noam Postavsky <npostavs@gmail.com>
> Date: Tue, 22 May 2018 19:33:03 -0400
> Cc: 31549@debbugs.gnu.org
> 
> > (byte-compile `(eval-when-compile (list ,@(make-list 2047 0))))
> >
> > Evaluating the form above, emacs (with -Q option) reports
> >   Error: Memory exhausted--use C-x s then exit and restart Emacs
> > in *Compile-Log* buffer.
> > The returned value of the form is t.
> 
> The error is coming from exec_byte_code:
> 
>   if (MAX_ALLOCA / word_size <= XFASTINT (maxdepth))
>     memory_full (SIZE_MAX);

You mean, in expansion of SAFE_ALLOCA_LISP_EXTRA?

> It's more like a Lisp stack overflow than a memory exhausted situation
> though, hardly calls for restarting Emacs.  Perhaps the byte compiler
> should refuse to compile such a large expression?

How about simply signaling a special error instead of memory_full?
Something like

  error ("Lisp stack overflow");

bug#31549: 25.3; bytecompile fails with eval-when-compile

[Noam Postavsky]

fixed 31549 26.1
quit

Eli Zaretskii <eliz@gnu.org> writes:

>> The error is coming from exec_byte_code:
>> 
>>   if (MAX_ALLOCA / word_size <= XFASTINT (maxdepth))
>>     memory_full (SIZE_MAX);
>
> You mean, in expansion of SAFE_ALLOCA_LISP_EXTRA?

Oh, sorry, I missed that the code has been changed quite a bit in Emacs
26.  I said:

    I can't reproduce in Emacs 26, but only because MAX_ALLOCA is
    bigger, I think.

But v26 exec_byte_code will allocate with malloc if needed, so the bug
is fixed (and MAX_ALLOCA is the same size).

> How about simply signaling a special error instead of memory_full?
> Something like
>
>   error ("Lisp stack overflow");

Still might be worth considering changing the error, although stack
overflow doesn't describe it correctly anymore.  As far as I can tell,
SAFE_ALLOCA_LISP_EXTRA will only signal memory_full if the number of
bytes to allocate won't fit in a ptrdiff_t variable.  That should be
pretty much impossible, barring some strange bugs.  So maybe:

--- i/src/lisp.h
+++ w/src/lisp.h
@@ -4662,7 +4662,7 @@ egetenv (const char *var)
     if (INT_MULTIPLY_WRAPV (nelt, word_size, &alloca_nbytes)   \
 	|| INT_ADD_WRAPV (alloca_nbytes, extra, &alloca_nbytes) \
 	|| SIZE_MAX < alloca_nbytes)			       \
-      memory_full (SIZE_MAX);				       \
+      error ("Oversize allocation (0x%lX)", (size_t) alloca_nbytes);    \
     else if (alloca_nbytes <= sa_avail)			       \
       (buf) = AVAIL_ALLOCA (alloca_nbytes);		       \
     else						       \

bug#31549: 25.3; bytecompile fails with eval-when-compile

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: ynyaaa@gmail.com,  31549@debbugs.gnu.org
> Date: Wed, 23 May 2018 18:34:06 -0400
> 
> > How about simply signaling a special error instead of memory_full?
> > Something like
> >
> >   error ("Lisp stack overflow");
> 
> Still might be worth considering changing the error, although stack
> overflow doesn't describe it correctly anymore.  As far as I can tell,
> SAFE_ALLOCA_LISP_EXTRA will only signal memory_full if the number of
> bytes to allocate won't fit in a ptrdiff_t variable.  That should be
> pretty much impossible, barring some strange bugs.  So maybe:
> 
> --- i/src/lisp.h
> +++ w/src/lisp.h
> @@ -4662,7 +4662,7 @@ egetenv (const char *var)
>      if (INT_MULTIPLY_WRAPV (nelt, word_size, &alloca_nbytes)   \
>  	|| INT_ADD_WRAPV (alloca_nbytes, extra, &alloca_nbytes) \
>  	|| SIZE_MAX < alloca_nbytes)			       \
> -      memory_full (SIZE_MAX);				       \
> +      error ("Oversize allocation (0x%lX)", (size_t) alloca_nbytes);    \
>      else if (alloca_nbytes <= sa_avail)			       \
>        (buf) = AVAIL_ALLOCA (alloca_nbytes);		       \
>      else						       \

I agree that memory_full is suboptimal here, but "Oversize allocation"
with a number is too technical to be useful to the programmer who
bumps into this problem.  We need some text which will indicate that
the program is too "complex" (a better word is needed here) and should
be simplified.  Can you come up with something along those lines?

bug#31549: 25.3; bytecompile fails with eval-when-compile

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

>> --- i/src/lisp.h
>> +++ w/src/lisp.h
>> @@ -4662,7 +4662,7 @@ egetenv (const char *var)
>>      if (INT_MULTIPLY_WRAPV (nelt, word_size, &alloca_nbytes)   \
>>  	|| INT_ADD_WRAPV (alloca_nbytes, extra, &alloca_nbytes) \
>>  	|| SIZE_MAX < alloca_nbytes)			       \
>> -      memory_full (SIZE_MAX);				       \
>> +      error ("Oversize allocation (0x%lX)", (size_t) alloca_nbytes);    \
>>      else if (alloca_nbytes <= sa_avail)			       \
>>        (buf) = AVAIL_ALLOCA (alloca_nbytes);		       \
>>      else						       \
>
> I agree that memory_full is suboptimal here, but "Oversize allocation"
> with a number is too technical to be useful to the programmer who
> bumps into this problem.  We need some text which will indicate that
> the program is too "complex" (a better word is needed here) and should
> be simplified.  Can you come up with something along those lines?

Sorry, if my initial response confused things, but I'm fairly certain
now that there is no way to trigger this error by compiling a Lisp
program in Emacs 26.  It would have to require a stack depth of 2^63 (or
2^31 on 32 bit builds), I imagine actual memory exhaustion would happen
first.

Actually, even though memory_full probably isn't correct, maybe we
should just leave it.  Triggering this error probably indicates some bug
in Emacs, so the first thing to do after hitting it would be to set a
breakpoint in gdb; this is a bit more convenient to do with memory_full
than Fsignal or error: fewer false positives.

bug#31549: 25.3; bytecompile fails with eval-when-compile

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: ynyaaa@gmail.com,  31549@debbugs.gnu.org
> Date: Thu, 24 May 2018 17:18:17 -0400
> 
> Sorry, if my initial response confused things

Seeking the truth doesn't always work in linear ways ;-)

> I'm fairly certain now that there is no way to trigger this error by
> compiling a Lisp program in Emacs 26.  It would have to require a
> stack depth of 2^63 (or 2^31 on 32 bit builds), I imagine actual
> memory exhaustion would happen first.
> 
> Actually, even though memory_full probably isn't correct, maybe we
> should just leave it.  Triggering this error probably indicates some bug
> in Emacs, so the first thing to do after hitting it would be to set a
> breakpoint in gdb; this is a bit more convenient to do with memory_full
> than Fsignal or error: fewer false positives.

Fine by me, but do we understand what change(s) between 25.3 and 26.1
fixed this problem?  If not, maybe we should try to understand that?

bug#31549: 25.3; bytecompile fails with eval-when-compile

[Noam Postavsky]

close 31549
quit

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Noam Postavsky <npostavs@gmail.com>
>> Cc: ynyaaa@gmail.com,  31549@debbugs.gnu.org
>> Date: Thu, 24 May 2018 17:18:17 -0400
>> 
>> Sorry, if my initial response confused things
>
> Seeking the truth doesn't always work in linear ways ;-)
>
>> I'm fairly certain now that there is no way to trigger this error by
>> compiling a Lisp program in Emacs 26.  It would have to require a
>> stack depth of 2^63 (or 2^31 on 32 bit builds), I imagine actual
>> memory exhaustion would happen first.
>> 
>> Actually, even though memory_full probably isn't correct, maybe we
>> should just leave it.  Triggering this error probably indicates some bug
>> in Emacs, so the first thing to do after hitting it would be to set a
>> breakpoint in gdb; this is a bit more convenient to do with memory_full
>> than Fsignal or error: fewer false positives.
>
> Fine by me, but do we understand what change(s) between 25.3 and 26.1
> fixed this problem?  If not, maybe we should try to understand that?

Ah, here it is, seems pretty straightforward.

[1: cb71a119f7]: 2016-08-09 01:31:22 -0700
  Remove arbitrary limit on bytecode maxdepth
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=cb71a119f7231984e010cc28ef33854721036a0f

bug#31549: 25.3; bytecompile fails with eval-when-compile

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: ynyaaa@gmail.com,  31549@debbugs.gnu.org
> Date: Sun, 27 May 2018 11:09:48 -0400
> 
> > Fine by me, but do we understand what change(s) between 25.3 and 26.1
> > fixed this problem?  If not, maybe we should try to understand that?
> 
> Ah, here it is, seems pretty straightforward.
> 
> [1: cb71a119f7]: 2016-08-09 01:31:22 -0700
>   Remove arbitrary limit on bytecode maxdepth

Right, thanks.  Case closed.