bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Basil L. Contovounesios via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 31 bytes --]

Given this sample input file:


[-- Attachment #2: my.el --]
[-- Type: application/emacs-lisp, Size: 429 bytes --]

[-- Attachment #3: Type: text/plain, Size: 240 bytes --]


Byte-compiling it as follows hangs indefinitely:
  
  emacs -Q -batch -f batch-byte-compile my.el

While it's hanging, might as well poke it:

  pkill -f 'batch-byte-compile my\.el' -SIGUSR2  

To which it yields the following backtrace:


[-- Attachment #4: backtrace.txt --]
[-- Type: text/plain, Size: 7392 bytes --]

Debugger entered--Lisp error: (quit)
  byte-compile--first-symbol-with-pos((t . #1))
  byte-compile--first-symbol-with-pos('(t . #2))
  byte-compile--first-symbol-with-pos((progn (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '(t . #5)))
  byte-compile--first-symbol-with-pos(((my-cycle) (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) (progn (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '(t . #8)) (setcar value-0 (progn (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '(t . #12))) (unwind-protect (setcar value-0 (progn (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '(t . #15))) :fun-body (internal-make-closure nil (form-description-1 value-0) nil (setcar (internal-get-closed-var 0) (list '(should ...) :form '(progn ... ...) :value (car-safe (internal-get-closed-var 1)))) (ert--signal-should-execution (car-safe (internal-get-closed-var 0))))) (let ((form-description-1 (list nil))) (unwind-protect (setcar value-0 (progn (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '(t . #19))) :fun-body (internal-make-closure nil (form-description-1 value-0) nil (setcar (internal-get-closed-var 0) (list '... :form '... :value (car-safe ...))) (ert--signal-should-execution (car-safe (internal-get-closed-var 0)))))) (let ((value-0 (list (gensym "ert-form-evaluation-aborted-")))) (let ((form-description-1 (list nil))) (unwind-protect (setcar value-0 (progn (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '...)) :fun-body (internal-make-closure nil (form-description-1 value-0) nil (setcar (internal-get-closed-var 0) (list ... :form ... :value ...)) (ert--signal-should-execution (car-safe ...))))) (car-safe value-0)) #'(lambda nil (let ((value-0 (list ...))) (let ((form-description-1 ...)) (if (unwind-protect ... :fun-body ...) nil (ert-fail ...))) (car-safe value-0))) (record 'ert-test 'my-test nil #'(lambda nil (let ((value-0 ...)) (let (...) (if ... nil ...)) (car-safe value-0))) nil :passed nil "/tmp/my.el") (ert-set-test 'my-test (record 'ert-test 'my-test nil #'(lambda nil (let (...) (let ... ...) (car-safe value-0))) nil :passed nil "/tmp/my.el")) (ert-deftest my-test nil (should (my-identity (my-cycle))))))
  byte-compile--warning-source-offset()
  byte-compile-log-warning("Oh dear" t :warning)
  byte-compile-warn("%s" "Oh dear")
  apply(byte-compile-warn "%s" "Oh dear")
  byte-compile-warn-x((my-cycle) "%s" "Oh dear")
  #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)()
  #f(compiled-function (form) #<bytecode -0x384fa88d0504449>)((macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)))
  byte-compile-form((macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) t)
  byte-compile-body(((macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '(t . #4)) nil)
  byte-compile-progn((progn (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '(t . #5)))
  byte-compile-form((progn (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '(t . #5)))
  byte-compile-two-args((setcar value-0 (progn (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '(t . #8))))
  byte-compile-form((setcar value-0 (progn (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '(t . #8))) t)
  byte-compile-unwind-protect((unwind-protect (setcar value-0 (progn (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '(t . #10))) :fun-body (internal-make-closure nil (form-description-1 value-0) nil (setcar (internal-get-closed-var 0) (list '(should (my-identity ...)) :form '(progn (macroexp--funcall-if-compiled ...) (my-cycle)) :value (car-safe (internal-get-closed-var 1)))) (ert--signal-should-execution (car-safe (internal-get-closed-var 0))))))
  byte-compile-form((unwind-protect (setcar value-0 (progn (macroexp--funcall-if-compiled #f(compiled-function () #<bytecode 0xe3f7c0047f6bb37>)) '(t . #10))) :fun-body (internal-make-closure nil (form-description-1 value-0) nil (setcar (internal-get-closed-var 0) (list '(should (my-identity ...)) :form '(progn (macroexp--funcall-if-compiled ...) (my-cycle)) :value (car-safe (internal-get-closed-var 1)))) (ert--signal-should-execution (car-safe (internal-get-closed-var 0))))) t)
...
  byte-compile-form((record 'ert-test 'my-test nil #'(lambda nil (let ((value-0 (list ...))) (let ((form-description-1 ...)) (if (unwind-protect ... :fun-body ...) nil (ert-fail ...))) (car-safe value-0))) nil :passed nil "/tmp/my.el"))
  mapc(byte-compile-form ('my-test (record 'ert-test 'my-test nil #'(lambda nil (let ((value-0 ...)) (let (...) (if ... nil ...)) (car-safe value-0))) nil :passed nil "/tmp/my.el")))
  byte-compile-normal-call((ert-set-test 'my-test (record 'ert-test 'my-test nil #'(lambda nil (let ((value-0 ...)) (let (...) (if ... nil ...)) (car-safe value-0))) nil :passed nil "/tmp/my.el")))
  byte-compile-form((ert-set-test 'my-test (record 'ert-test 'my-test nil #'(lambda nil (let ((value-0 ...)) (let (...) (if ... nil ...)) (car-safe value-0))) nil :passed nil "/tmp/my.el")) t)
  byte-compile-keep-pending((ert-set-test 'my-test (record 'ert-test 'my-test nil #'(lambda nil (let ((value-0 ...)) (let (...) (if ... nil ...)) (car-safe value-0))) nil ':passed 'nil "/tmp/my.el")))
  byte-compile-file-form((ert-set-test 'my-test (record 'ert-test 'my-test nil #'(lambda nil (let ((value-0 ...)) (let (...) (if ... nil ...)) (car-safe value-0))) nil ':passed 'nil "/tmp/my.el")))
  #f(compiled-function (form) #<bytecode -0x1079b636bff336ff>)((ert-set-test 'my-test (record 'ert-test 'my-test nil #'(lambda nil (let ((value-0 ...)) (let (form-description-1) (if ... nil ...)) value-0)) nil ':passed 'nil "/tmp/my.el")))
  byte-compile-recurse-toplevel((ert-set-test 'my-test (record 'ert-test 'my-test nil #'(lambda nil (let ((value-0 ...)) (let (form-description-1) (if ... nil ...)) value-0)) nil ':passed 'nil "/tmp/my.el")) #f(compiled-function (form) #<bytecode -0x1079b636bff336ff>))
  #f(compiled-function (subform) #<bytecode -0x88ecb93aa9a0d82>)((ert-set-test 'my-test (record 'ert-test 'my-test nil #'(lambda nil (let ((value-0 ...)) (let (form-description-1) (if ... nil ...)) value-0)) nil ':passed 'nil "/tmp/my.el")))
  mapcar(#f(compiled-function (subform) #<bytecode -0x88ecb93aa9a0d82>) ((ert-set-test 'my-test (record 'ert-test 'my-test nil #'(lambda nil (let (...) (let ... ...) value-0)) nil ':passed 'nil "/tmp/my.el")) 'my-test))
  byte-compile-recurse-toplevel((ert-deftest my-test nil (should (my-identity (my-cycle)))) #f(compiled-function (form) #<bytecode -0x1079b636bff336ff>))
  byte-compile-toplevel-file-form((ert-deftest my-test nil (should (my-identity (my-cycle)))))
  #f(compiled-function (inbuffer) #<bytecode 0x1c4a7e9ef683d101>)(#<buffer  *Compiler Input*>)
  byte-compile-from-buffer(#<buffer  *Compiler Input*>)
  byte-compile-file("my.el")
  batch-byte-compile-file("my.el")
  batch-byte-compile()
  command-line-1(("-f" "batch-byte-compile" "my.el"))
  command-line()
  normal-top-level()

[-- Attachment #5: Type: text/plain, Size: 3659 bytes --]


I.e. the 'pure' compile-time form has a cycle, and the DFS of
byte-compile--first-symbol-with-pos gets lost.  Paraphrasing Stefan,
it's acceptable if the compiler mishandles circular source code, but it
should be able to handle circular data.

I don't know why, but I can reproduce the hang only when the form is inside
ert-deftest+should, and not inside a plain defun.

Perhaps byte-compile--first-symbol-with-pos needs to employ something
like byte-run--ssp-seen?  Or does ERT somehow come into play?

Thanks,

-- 
Basil

In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.16.0, Xaw3d scroll bars) of 2022-10-18 built on tia
Repository revision: c2b79d9148f21c3717f1eaa2d37b837b0922b94c
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101004
System Description: Debian GNU/Linux bookworm/sid

Configured using:
 'configure 'CFLAGS=-Og -ggdb3' -C --prefix=/home/blc/.local
 --enable-checking=structs --with-file-notification=yes
 --with-x-toolkit=lucid --with-x'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP X11 XAW3D XDBE XIM XINPUT2 XPM LUCID ZLIB

Important settings:
  value of $LANG: en_IE.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo x-toolkit
xinput2 x multi-tty make-network-process emacs)

Memory information:
((conses 16 36538 7652)
 (symbols 48 5112 0)
 (strings 32 13853 1864)
 (string-bytes 1 382346)
 (vectors 16 9260)
 (vector-slots 8 147212 11667)
 (floats 8 23 25)
 (intervals 56 242 0)
 (buffers 1000 10))

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Perhaps byte-compile--first-symbol-with-pos needs to employ something
> like byte-run--ssp-seen?  Or does ERT somehow come into play?

I just added in `master` an arbitrary limit to the loop inside
`byte-compile--first-symbol-with-pos` to make sure it can't inf-loop.

Can you confirm it fixes your original problem?


        Stefan

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Alan Mackenzie]

Hello, Basil.

On Tue, Oct 18, 2022 at 02:24:02 +0300, Basil L. Contovounesios wrote:
> Given this sample input file:

[ .... ]


> Byte-compiling it as follows hangs indefinitely:
  
>   emacs -Q -batch -f batch-byte-compile my.el

> While it's hanging, might as well poke it:

>   pkill -f 'batch-byte-compile my\.el' -SIGUSR2  

> To which it yields the following backtrace:

[ .... ]

> I.e. the 'pure' compile-time form has a cycle, and the DFS of
> byte-compile--first-symbol-with-pos gets lost.

What's a DFS?

> Paraphrasing Stefan, it's acceptable if the compiler mishandles
> circular source code, but it should be able to handle circular data.

The problem here seems to be feeding macroexp-warn-and-return with data
as the FORM argument.  FORM is intended only to be an executable lisp
form.  The doc string, perhaps, could be clearer on this.

> I don't know why, but I can reproduce the hang only when the form is inside
> ert-deftest+should, and not inside a plain defun.

There is a huge concentration of "advanced" features inside those ~20
lines of Lisp code.  There's eval-and-compile, nconc, a compiler-macro,
and ert.  ;-)

> Perhaps byte-compile--first-symbol-with-pos needs to employ something
> like byte-run--ssp-seen?  Or does ERT somehow come into play?

It wouldn't be difficult, just tedious, to add checking for circular
lists into byte-compile--first-symbol-with-pos.  But a circular list is
an invalid argument for FORM in macorexp-warn-and-return, see above.
There are surely lots of places in Emacs where feeding a circular list as
an argument to a function will cause a hang.

At the moment, I'm not in favour of doing anything here.  I don't think
there's a bug.

> Thanks,

> -- 
> Basil

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[dick]

AM> But a circular list is an invalid argument

This betrays a basic misunderstanding of what compilers do.

Would that it were so convenient that compiler writers could admit
non-terminating compilation based on the content of user programs.

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Basil L. Contovounesios via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Monnier [2022-10-18 10:51 -0400] wrote:

>> Perhaps byte-compile--first-symbol-with-pos needs to employ something
>> like byte-run--ssp-seen?  Or does ERT somehow come into play?
>
> I just added in `master` an arbitrary limit to the loop inside
> `byte-compile--first-symbol-with-pos` to make sure it can't inf-loop.
>
> Can you confirm it fixes your original problem?

Can confirm, thanks,

-- 
Basil

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Basil L. Contovounesios via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Alan Mackenzie [2022-10-18 15:01 +0000] wrote:

> On Tue, Oct 18, 2022 at 02:24:02 +0300, Basil L. Contovounesios wrote:
>
>> I.e. the 'pure' compile-time form has a cycle, and the DFS of
>> byte-compile--first-symbol-with-pos gets lost.
>
> What's a DFS?

Depth-First Search.

>> Paraphrasing Stefan, it's acceptable if the compiler mishandles
>> circular source code, but it should be able to handle circular data.
>
> The problem here seems to be feeding macroexp-warn-and-return with data
> as the FORM argument.

That's not the problem, because it's just for illustrative purposes.
Instead of 'arg' being passed unchanged as the FORM argument, it could
just as well have been `(my-frobnicate ,arg).

> FORM is intended only to be an executable lisp form.

`(my-frobnicate ,arg) fits that bill, right?  The end result is the
same: macroexp-warn-and-return is fed a form containing a cycle, without
any of the code that gives rise to the form being circular itself.

>> I don't know why, but I can reproduce the hang only when the form is inside
>> ert-deftest+should, and not inside a plain defun.
>
> There is a huge concentration of "advanced" features inside those ~20
> lines of Lisp code.

Maybe when the compiler detects sufficiently advanced Lisp it should

  (signal 'indistinguishable-from-magic (list form))

>  There's eval-and-compile,

That's just a shortcut: one could equivalently put my-cycle and
my-identity in a separate file and 'require' it.  The compiler just
needs to know that my-cycle is pure and my-identity has a
compiler-macro before reaching their call sites.

> nconc, a compiler-macro, and ert.  ;-)

You forgot 'pure'.  We're spoilt for choice!

>> Perhaps byte-compile--first-symbol-with-pos needs to employ something
>> like byte-run--ssp-seen?  Or does ERT somehow come into play?
>
> It wouldn't be difficult, just tedious, to add checking for circular
> lists into byte-compile--first-symbol-with-pos.  But a circular list is
> an invalid argument for FORM in macorexp-warn-and-return, see above.

How else should a compiler-macro safely warn about a problematic
function argument?

> There are surely lots of places in Emacs where feeding a circular list as
> an argument to a function will cause a hang.

Sure, but a subset of those places should reasonably be expected to not
hang...

> At the moment, I'm not in favour of doing anything here.  I don't think
> there's a bug.

...for instance, when dealing with compile-time constants in real-world
Elisp.

Thanks,

-- 
Basil

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Alan Mackenzie]

Hello, Basil.

On Tue, Oct 18, 2022 at 19:34:08 +0300, Basil L. Contovounesios wrote:
> Alan Mackenzie [2022-10-18 15:01 +0000] wrote:

> > On Tue, Oct 18, 2022 at 02:24:02 +0300, Basil L. Contovounesios wrote:

> >> I.e. the 'pure' compile-time form has a cycle, and the DFS of
> >> byte-compile--first-symbol-with-pos gets lost.

> > What's a DFS?

> Depth-First Search.

Thanks!

> >> Paraphrasing Stefan, it's acceptable if the compiler mishandles
> >> circular source code, but it should be able to handle circular data.

> > The problem here seems to be feeding macroexp-warn-and-return with data
> > as the FORM argument.

> That's not the problem, because it's just for illustrative purposes.
> Instead of 'arg' being passed unchanged as the FORM argument, it could
> just as well have been `(my-frobnicate ,arg).

That would not loop, since there is a symbol with position there.

> > FORM is intended only to be an executable lisp form.

> `(my-frobnicate ,arg) fits that bill, right?  The end result is the
> same: macroexp-warn-and-return is fed a form containing a cycle, without
> any of the code that gives rise to the form being circular itself.

This would not loop.

> >> I don't know why, but I can reproduce the hang only when the form is inside
> >> ert-deftest+should, and not inside a plain defun.

> > There is a huge concentration of "advanced" features inside those ~20
> > lines of Lisp code.

> Maybe when the compiler detects sufficiently advanced Lisp it should

>   (signal 'indistinguishable-from-magic (list form))

> >  There's eval-and-compile,

> That's just a shortcut: one could equivalently put my-cycle and
> my-identity in a separate file and 'require' it.  The compiler just
> needs to know that my-cycle is pure and my-identity has a
> compiler-macro before reaching their call sites.

> > nconc, a compiler-macro, and ert.  ;-)

> You forgot 'pure'.  We're spoilt for choice!

:-)

> >> Perhaps byte-compile--first-symbol-with-pos needs to employ something
> >> like byte-run--ssp-seen?  Or does ERT somehow come into play?

> > It wouldn't be difficult, just tedious, to add checking for circular
> > lists into byte-compile--first-symbol-with-pos.  But a circular list is
> > an invalid argument for FORM in macorexp-warn-and-return, see above.

> How else should a compiler-macro safely warn about a problematic
> function argument?

by calling the normal byte compiler warning facilities.  If there is a
symbol somewhere in the FORM passed to them, it won't loop, even if it is
a circular list.

You called macroexp-warn-and-return from outside of the byte compiler.
From within the byte compiler, it cannot generate a loop (see above).

> > There are surely lots of places in Emacs where feeding a circular
> > list as an argument to a function will cause a hang.

> Sure, but a subset of those places should reasonably be expected to not
> hang...

Your circular list containing an ordinary symbol (i.e. without position)
is not going to arise in the byte compiler.

> > At the moment, I'm not in favour of doing anything here.  I don't think
> > there's a bug.

> ...for instance, when dealing with compile-time constants in real-world
> Elisp.

Maybe you could construct an example of a circular list without a symbol
from code being compiled.  Maybe you could cause a warning from a correct
call of macroexp-warn-and-return to loop.  But I'd be surprised.

> Thanks,

> -- 
> Basil

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[dick]

AM> a correct call of macroexp-warn-and-return

Your notions of correctness appear to coincide with those which do not
require you to make changes.  Mine do as well.

Your defense is a novel one, however.  Someone approaches you asserting
"Some cars don't stop with your brake calipers," and you refute by
showing him cars which do.

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> That's not the problem, because it's just for illustrative purposes.
>> Instead of 'arg' being passed unchanged as the FORM argument, it could
>> just as well have been `(my-frobnicate ,arg).
>
> That would not loop, since there is a symbol with position there.

BTE, the real-world use case is a compiler macro which takes the list of
args and tucks the name of another function in front.  So the generated
code, passed to `macroexp-warn-and-return` has the shape

    (somefun somearg1 somearg2)

where `somefun` does *not* come from the source but from the
compiler-macro instead and hence doesn't have any sympos, and then
`somearg1` was an expression of the form

   (quote #1=(1 2 3 . #1#))

So, there was no cycle in the code part, the cycle is only inside the
data embedded in the code.


        Stefan

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Basil L. Contovounesios [2022-10-18 19:33:55] wrote:
> Stefan Monnier [2022-10-18 10:51 -0400] wrote:
>>> Perhaps byte-compile--first-symbol-with-pos needs to employ something
>>> like byte-run--ssp-seen?  Or does ERT somehow come into play?
>>
>> I just added in `master` an arbitrary limit to the loop inside
>> `byte-compile--first-symbol-with-pos` to make sure it can't inf-loop.
>>
>> Can you confirm it fixes your original problem?
>
> Can confirm, thanks,

Thanks, closing,


        Stefan

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Alan Mackenzie]

On Tue, Oct 18, 2022 at 15:19:35 -0400, Stefan Monnier wrote:
> >> That's not the problem, because it's just for illustrative purposes.
> >> Instead of 'arg' being passed unchanged as the FORM argument, it could
> >> just as well have been `(my-frobnicate ,arg).

> > That would not loop, since there is a symbol with position there.

> BTE, the real-world use case is a compiler macro which takes the list of
> args and tucks the name of another function in front.  So the generated
> code, passed to `macroexp-warn-and-return` has the shape

>     (somefun somearg1 somearg2)

> where `somefun` does *not* come from the source but from the
> compiler-macro instead and hence doesn't have any sympos, and then
> `somearg1` was an expression of the form

>    (quote #1=(1 2 3 . #1#))

> So, there was no cycle in the code part, the cycle is only inside the
> data embedded in the code.

I think somebody who writes something like that _deserves_ to go into an
infinite loop.  Have you ever seen anything at all like this in real
life?

You have fixed this "problem" by arbitrarily limiting the size of valid
form stacks that can be searched, and the depth to which they can be
searched.  This limit, 10, seems too small.  There can easily be form
stacks with more than 10 elements.  Do you have any objection if I
change that 10 to something closer to infinity?  Something which will
encompass any form stack likely to be encountered in practice?  I would
suggest the number 300.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> where `somefun` does *not* come from the source but from the
>> compiler-macro instead and hence doesn't have any sympos, and then
>> `somearg1` was an expression of the form
>
>>    (quote #1=(1 2 3 . #1#))
>
>> So, there was no cycle in the code part, the cycle is only inside the
>> data embedded in the code.
>
> I think somebody who writes something like that _deserves_ to go into an
> infinite loop.

Circular data is not very frequent, but it's not rare either (as you
found out during the development of sympos, as evidenced by
`byte-run--ssp-seen`).

`byte-compile--first-symbol-with-pos` didn't bump into it until now
simply because it usually gets "lucky" and finds a sympos before getting
to a cyclic embedded data.

> Have you ever seen anything at all like this in real life?

Of course, and so have you :-)

> You have fixed this "problem" by arbitrarily limiting the size of
> valid form stacks that can be searched, and the depth to which they
> can be searched.  This limit, 10, seems too small.  There can easily
> be form stacks with more than 10 elements.  Do you have any objection
> if I change that 10 to something closer to infinity?

Be my guest.

> Something which will encompass any form stack likely to be
> encountered in practice?  I would suggest the number 300.

Fine by me.  Tho once we get into this range, 666 sounds a lot
more enticing. :-)


        Stefan

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Basil L. Contovounesios via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

Alan Mackenzie [2022-10-18 17:06 +0000] wrote:
> On Tue, Oct 18, 2022 at 19:34:08 +0300, Basil L. Contovounesios wrote:
>> Alan Mackenzie [2022-10-18 15:01 +0000] wrote:
>> > The problem here seems to be feeding macroexp-warn-and-return with data
>> > as the FORM argument.
>> That's not the problem, because it's just for illustrative purposes.
>> Instead of 'arg' being passed unchanged as the FORM argument, it could
>> just as well have been `(my-frobnicate ,arg).
>
> That would not loop, since there is a symbol with position there.
>
>> > FORM is intended only to be an executable lisp form.
>> `(my-frobnicate ,arg) fits that bill, right?  The end result is the
>> same: macroexp-warn-and-return is fed a form containing a cycle, without
>> any of the code that gives rise to the form being circular itself.
>
> This would not loop.

And yet:


[-- Attachment #2: my.el --]
[-- Type: application/emacs-lisp, Size: 330 bytes --]

[-- Attachment #3: my-tests.el --]
[-- Type: application/emacs-lisp, Size: 165 bytes --]

[-- Attachment #4: Type: text/plain, Size: 2342 bytes --]


Followed by:

  emacs -Q -L . -batch -f batch-byte-compile my.el
  emacs -Q -L . -batch -f batch-byte-compile my-tests.el

The latter hangs prior to Stefan's limit on recursion.

>> >> Perhaps byte-compile--first-symbol-with-pos needs to employ something
>> >> like byte-run--ssp-seen?  Or does ERT somehow come into play?
>> > It wouldn't be difficult, just tedious, to add checking for circular
>> > lists into byte-compile--first-symbol-with-pos.  But a circular list is
>> > an invalid argument for FORM in macorexp-warn-and-return, see above.
>> How else should a compiler-macro safely warn about a problematic
>> function argument?
>
> by calling the normal byte compiler warning facilities.

Could you please point me to them?

> If there is a symbol somewhere in the FORM passed to them, it won't
> loop, even if it is a circular list.

See the case of my-identity above.

> You called macroexp-warn-and-return from outside of the byte compiler.
> From within the byte compiler, it cannot generate a loop (see above).

What counts as outside of the byte compiler?  Do you mean that
macroexp-warn-and-return should be disallowed in declare forms or macro
definitions?  Or that a compiler-macro should not pass forms derived
from the function's arguments to macroexp-warn-and-return?  AFAIA there
is no precedent for such a restriction, and in fact I see the opposite
trend in recent history:

https://git.sv.gnu.org/cgit/emacs.git/commit/?id=d52c929e31
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=52d5771e0a
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=5ee4209f30
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=96926fa6eb
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=fffa53ff1a
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=bd40ec5d57
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=13d6e8fa54
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=155ddde4dd

> Maybe you could construct an example of a circular list without a symbol
> from code being compiled.  Maybe you could cause a warning from a correct
> call of macroexp-warn-and-return to loop.  But I'd be surprised.

You have repeatedly asserted that macroexp-warn-and-return will not loop
if used correctly, but it is not clear to me from your description what
correct use of macroexp-warn-and-return entails.  Could you please
explain?

Thanks,

-- 
Basil

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Basil L. Contovounesios via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Alan Mackenzie [2022-10-19 08:52 +0000] wrote:

> On Tue, Oct 18, 2022 at 15:19:35 -0400, Stefan Monnier wrote:
>> >> That's not the problem, because it's just for illustrative purposes.
>> >> Instead of 'arg' being passed unchanged as the FORM argument, it could
>> >> just as well have been `(my-frobnicate ,arg).
>
>> > That would not loop, since there is a symbol with position there.
>
>> BTE, the real-world use case is a compiler macro which takes the list of
>> args and tucks the name of another function in front.  So the generated
>> code, passed to `macroexp-warn-and-return` has the shape
>
>>     (somefun somearg1 somearg2)
>
>> where `somefun` does *not* come from the source but from the
>> compiler-macro instead and hence doesn't have any sympos, and then
>> `somearg1` was an expression of the form
>
>>    (quote #1=(1 2 3 . #1#))
>
>> So, there was no cycle in the code part, the cycle is only inside the
>> data embedded in the code.
>
> I think somebody who writes something like that _deserves_ to go into an
> infinite loop.  Have you ever seen anything at all like this in real
> life?

See some of the recent commits listed in my other message.

-- 
Basil

bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

[Mattias Engdegård]

19 okt. 2022 kl. 14.48 skrev Stefan Monnier <monnier@iro.umontreal.ca>:

> Fine by me.  Tho once we get into this range, 666 sounds a lot
> more enticing.

Since this code is only run when reporting errors, quadratic behaviour is perhaps not a disaster and we can replace the depth fuel with a list of values seen and thus detect loops in arbitrary data structures. (Or just run with 666.)