bug#68075: 30.0.50; New special form `handler-bind`

[Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

Package: Emacs
Version: 30.0.50


I have now pushed to `scratch/handler-bind` a set of patches I'd like to
commit to `master`.  They add the new special form `handler-bind`, which
provides a functionality a bit halfway between `condition-case` and
`signal-hook-function`.

Among other things, this allows us to fix bug#11218, bug#65267, and
bug#67196.  It also makes it possible to move code out of
`signal_or_quit`.

The complete log can be found below, but the oneliners are:

    New special form `handler-bind`
    (eval-expression): Fix bug#67196
    ert.el: Use `handler-bind` to record backtraces
    Fix ert-tests.el for the new `handler-bind` code
    Use handler-bind to repair bytecomp-tests
    emacs-module-tests.el (mod-test-non-local-exit-signal-test): Repair test
    startup.el: Use `handler-bind` to implement `--debug-init`
    Move batch backtrace code to `top_level_2`
    (macroexp--with-extended-form-stack): Use plain `let`
    eval.c: Add new var `lisp-eval-depth-reserve`
    (signal_or_quit): Preserve error object identity
    (backtrace-on-redisplay-error): Use `handler-bind`

As you can see, the first commit adds the new feature, and subsequent
ones basically make use of it in various different places.

Among those commits, I'll note the introduction of a new variable
`lisp-eval-depth-reserve`, which lets us control how much Emacs can grow
`max-lisp-eval-depth` to run the debugger.  This is not indispensable,
but seemed like a good idea.  I also subtly changed the way errors
are built such that we can rely on the error object being `eq` to itself
as it moves up from the call to `signal` up to its `condition-case`
handler, which should make it possible to keep extra info about it
in an `eq` hashtable, for example.

Comments, objections?


        Stefan


commit a4efbe4c499623b33882a770c896ebfd31459ce9
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Mon Dec 25 22:32:17 2023 -0500

    New special form `handler-bind`
    
    AFAIK, this provides the same semantics as Common Lisp's `handler-bind`,
    modulo the differences about how error objects and conditions are
    represented.
    
    * lisp/subr.el (handler-bind): New macro.
    
    * src/eval.c (pop_handler): New function.
    (Fhandler_Bind_1): New function.
    (signal_or_quit): Handle new handlertypes `HANDLER` and `SKIP_CONDITIONS`.
    (find_handler_clause): Simplify.
    (syms_of_eval): Defsubr `Fhandler_bind_1`.
    
    * doc/lispref/control.texi (Handling Errors): Add `handler-bind`.
    
    * test/src/eval-tests.el (eval-tests--handler-bind): New test.
    
    * lisp/emacs-lisp/lisp-mode.el (lisp-font-lock-keywords):
    Move 'handler-bind' from CL-only to generic Lisp.
    (handler-bind): Remove indentation setting, it now lives in the macro
    definition.

commit 19f1d2a9f5111a2f06003f45f3af2a39c7029047
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Mon Dec 18 23:47:56 2023 -0500

    (eval-expression): Fix bug#67196
    
    * lisp/simple.el (eval-expression--debug): New function.
    (eval-expression): Use it together with `handler-bind` instead of
    let-binding `debug-on-error`.

commit ae21819496a7f003c4d4e185204533660197daaa
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Mon Dec 18 23:57:45 2023 -0500

    ert.el: Use `handler-bind` to record backtraces
    
    * lisp/emacs-lisp/ert.el (ert--should-signal-hook): Delete function.
    (ert--expand-should-1): Don't bind `signal-hook-function`.
    (ert--test-execution-info): Remove `next-debugger` slot.
    (ert--run-test-debugger): Adjust to new calling convention.
    Pass the `:backtrace-base` info to the debugger.
    (ert--run-test-internal): Use `handler-bind` rather than let-binding
    `debugger` and `debug-on-error`.
    
    * lisp/emacs-lisp/ert-x.el (ert-remote-temporary-file-directory): Don't
    use `defconst` if it's not meant to stay constant (e.g. we let-bind it
    in tramp-tests.el).

commit 89a298b3d2f86e75750617ef7e301372ea5aa46f
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Thu Dec 28 00:46:36 2023 -0500

    Fix ert-tests.el for the new `handler-bind` code
    
    Now that `ert.el` uses `handler-bind` instead of `debugger`, some
    details of the behavior have changed.  More specifically,
    three tests are now broken, but these basically tested the failure
    of ERT's machinery to record errors when ERT was run within
    a `condition-case`.
    AFAICT, these tests do not check for a behavior that we want,
    so rather than "fix" them, I disabled them.
    
    * test/lisp/emacs-lisp/ert-tests.el (ert-test-error-debug)
    (ert-test-fail-debug-with-condition-case)
    (ert-test-should-failure-debugging): Comment out.
    (ert-test-with-demoted-errors): It now passes.  Bug#11218 is fixed!

commit 1c1d2eb3e389bb5e397cd8e1568e4e3129067912
Author: Mattias Engdegård <mattiase@acm.org>
Date:   Wed Dec 27 11:32:49 2023 +0100

    Use handler-bind to repair bytecomp-tests
    
    * test/lisp/emacs-lisp/bytecomp-tests.el
    (bytecomp-tests--error-frame, bytecomp--byte-op-error-backtrace):
    Make test pass again and simplify, using handler-bind instead
    of the previous debugger hack.

commit dcf7508c947359866151171a840d99d939c35cdf
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Thu Dec 28 00:49:39 2023 -0500

    emacs-module-tests.el (mod-test-non-local-exit-signal-test): Repair test
    
    That test relied on `debugger` and `debug-on-signal` in a way that
    doesn't work with the new ERT code.
    
    * test/src/emacs-module-tests.el (mod-test-non-local-exit-signal-test):
    Use `handler-bind` rather than the debugger.

commit 917596160c1c831b3a41b320a0e357e5161cb4c8
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Tue Dec 19 19:46:47 2023 -0500

    startup.el: Use `handler-bind` to implement `--debug-init`
    
    This provides a more reliable fix for bug#65267 since we don't
    touch `debug-on-error` nor `debug-ignore-errors` any more.
    
    * lisp/startup.el (startup--debug): New function.
    (startup--load-user-init-file): Use it and `handler-bind` instead of
    let-binding `debug-on-error`.

commit 6a57b9151b149a445daecc211d8f58010e605768
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Wed Dec 20 23:31:39 2023 -0500

    Move batch backtrace code to `top_level_2`
    
    Move ad-hoc code meant to ease debugging of bootstrap (and batch mode)
    to `top_level_2` so it doesn't pollute `signal_or_quit`.
    
    * src/lisp.h (pop_handler, push_handler_bind): Declare.
    * src/keyboard.c (top_level_2): Setup an error handler to call
    `debug-early` when noninteractive.
    * src/eval.c (pop_handler): Not static any more.
    (signal_or_quit): Remove special case for noninteractive use.
    (push_handler_bind): New function, extracted from `Fhandler_bind_1`.
    (Fhandler_bind_1): Use it.
    (syms_of_eval): Declare `Qdebug_early__handler`.
    * lisp/emacs-lisp/debug-early.el (debug-early-backtrace): Weed out
    frames below `debug-early`.
    (debug-early--handler): New function.

commit 634bf61947606d1b8344ba090fdd0fc098fb5eb6
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Mon Dec 25 23:55:53 2023 -0500

    (macroexp--with-extended-form-stack): Use plain `let`
    
    `macroexp--with-extended-form-stack` used manual push/pop so that upon
    non-local exits the "deeper" value is kept, so the error handler gets
    to know what was the deeper value, so as to be able to compute more
    precise error locations.
    Replace this with a `handler-bind` which catches that "deeper" value
    more explicitly.
    
    * lisp/emacs-lisp/bytecomp.el (bytecomp--displaying-warnings):
    Use `handler-bind` to catch the value of `byte-compile-form-stack`
    at the time of the error.  Also consolidate the duplicated code.
    
    * lisp/emacs-lisp/macroexp.el (macroexp--with-extended-form-stack):
    Use a plain dynbound let-rebinding.

commit c89b234405f8fa6c52f83104a46a4a2c3121198f
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Tue Dec 26 23:56:09 2023 -0500

    eval.c: Add new var `lisp-eval-depth-reserve`
    
    Rather than blindly increase `max-lisp-eval-depth` when entering the
    debugger or running `signal-hook-function`, use this new "reserve"
    to keep track of how much we have grown the stack for "debugger"
    purposes so that for example recursive calls to `signal-hook-function`
    can't eat up the whole C stack.
    
    * src/eval.c (max_ensure_room): Rewrite.
    (restore_stack_limits): Move before `max_ensure_room`.  Rewrite.
    (call_debugger, signal_or_quit): Adjust calls accordingly.
    Also grow `max-lisp-eval-depth` for `hander-bind` handlers.
    (init_eval_once): Don't initialize `max_lisp_eval_depth` here.
    (syms_of_eval): Initialize it here instead.
    Add new var `lisp-eval-depth-reserve`.
    
    * doc/lispref/eval.texi (Eval): Add `lisp-eval-depth-reserve`.

commit b925152bffce30abbd48361af6858cd45b785d84
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Wed Dec 27 15:06:32 2023 -0500

    (signal_or_quit): Preserve error object identity
    
    Make sure we build the (ERROR-SYMBOL . ERROR-DATA) object only once
    when signaling an error, so that its `eq` identity can be used.
    It also gets us a tiny bit closer to having real "error objects"
    like in most other current programming languages.
    
    * src/eval.c (maybe_call_debugger): Change arglist to receive the error
    object instead of receiving the signal and the data separately.
    (signal_or_quit): Build the error object right at the beginning so it
    stays `eq` to itself.
    Rename the `keyboard_quit` arg to `continuable` so say what it does
    rather than what it's used for.
    (signal_quit_p): Change arg to be the error object rather than just the
    error-symbol.
    
    * src/keyboard.c (cmd_error_internal, menu_item_eval_property_1):
    Adjust calls to `signal_quit_p` accordingly.
    
    * test/src/eval-tests.el (eval-tests--error-id): New test.

commit 26b7078705ae5b9226c99e370740ab9a4063f20f
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Mon Dec 25 21:41:08 2023 -0500

    (backtrace-on-redisplay-error): Use `handler-bind`
    
    Reimplement `backtrace-on-redisplay-error` using `push_handler_bind`.
    This moves the code from `signal_or_quit` to `xdisp.c` and
    `debug-early.el`.
    
    * lisp/emacs-lisp/debug-early.el (debug-early-backtrace):
    Add `base` arg to strip "internal" frames.
    (debug--early): New function, extracted from `debug-early`.
    (debug-early, debug-early--handler): Use it.
    (debug-early--muted): New function, extracted (translated) from
    `signal_or_quit`; trim the buffer to a max of 10 backtraces.
    
    * src/xdisp.c (funcall_with_backtraces): New function.
    (dsafe_calln): Use it.
    (syms_of_xdisp): Defsym `Qdebug_early__muted`.
    
    * src/eval.c (redisplay_deep_handler): Delete var.
    (init_eval, internal_condition_case_n): Don't set it any more.
    (backtrace_yet): Delete var.
    (signal_or_quit): Remove special case for `backtrace_on_redisplay_error`.
    * src/keyboard.c (command_loop_1): Don't set `backtrace_yet` any more.
    * src/lisp.h (backtrace_yet): Don't declare.