bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Joost Kremers]

Running with `debug-on-error' set to `t', I get the following 
error when triggering an(other) error:

Starting from `emacs -Q`, run `M-x ielm RET` to get an Elisp 
prompt. Then:

```
*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (mapconcat #'identity '(("config") ("value")) " ")
*** Eval error ***  Wrong type argument: characterp, "config"
```

So far, so good. Now do `M-x toggle-debug-on-error` and repeat the 
input:

``` 
ELISP> (mapconcat #'identity '(("config") ("value")) " ")
*** Eval error ***  Symbol’s value as variable is void: 
    debugger-outer-match-data
ELISP> 
```

The debug window doesn't pop up.

The output from `report-emacs-bug':

In GNU Emacs 27.0.90 (build 2, x86_64-pc-linux-gnu, GTK+ Version 
3.22.30)
 of 2020-03-11 built on IdeaPad
Repository revision: 1bc3fa0bd02cb167ae82b65fc56f95651d2bea16
Repository branch: emacs-27
Windowing system distributor 'The X.Org Foundation', version 
11.0.11906000
System Description: elementary OS 5.1.2 Hera

Recent messages:
Contacting host: elpa.gnu.org:443 [2 times]
Contacting host: melpa.org:443 [2 times]
Contacting host: orgmode.org:443
Package refresh done
scroll-up-command: End of buffer
Mark set
Debug on Error enabled globally
Entering debugger...
End of buffer [3 times]
Mark set
next-line: End of buffer
Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS GSETTINGS GLIB NOTIFY 
INOTIFY
GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS PDUMPER 
LCMS2 GMP

Important settings:
  value of $LC_MONETARY: en_IE.UTF-8
  value of $LC_NUMERIC: en_IE.UTF-8
  value of $LC_TIME: en_IE.UTF-8
  value of $LANG: en_GB.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: IELM

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug sendmail debug backtrace help-mode
find-func cus-start cus-load ielm pp comint ansi-color ring 
finder-inf
mm-archive message dired dired-loaddefs format-spec rfc822 mml 
mml-sec
epa derived gnus-util rmail rmail-loaddefs text-property-search
time-date mailabbrev gmm-utils mailheader mm-decode mm-bodies 
mm-encode
mail-utils gnutls network-stream url-http mail-parse rfc2231 
rfc2047
rfc2045 mm-util ietf-drums mail-prsvr url-gw nsm rmc puny 
url-cache
url-auth url url-proxy url-privacy url-expand url-methods 
url-history
url-cookie url-domsuf url-util mailcap epg epg-config package 
easymenu
browse-url url-handlers url-parse auth-source cl-seq eieio 
eieio-core
cl-macs eieio-loaddefs password-cache json subr-x map url-vars seq
byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib tooltip 
eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel 
term/x-win
x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt 
fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu 
font-core
term/tty-colors frame minibuffer cl-generic cham georgian 
utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese 
eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic 
indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice 
loaddefs
button faces cus-face macroexp files text-properties overlay sha1 
md5
base64 format env code-pages mule custom widget 
hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit 
x
multi-tty make-network-process emacs)

Memory information:
((conses 16 206582 84875)
 (symbols 48 14217 2)
 (strings 32 71046 30324)
 (string-bytes 1 1763859)
 (vectors 16 18700)
 (vector-slots 8 262878 17906)
 (floats 8 25 354)
 (intervals 56 280 0)
 (buffers 1000 13))



-- 
Joost Kremers
Life has its moments

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Michael Heerdegen]

Joost Kremers <joostkremers@fastmail.fm> writes:

> *** Eval error ***  Symbol’s value as variable is void:
> debugger-outer-match-data

Another defvar with a missing init value so the variable is not globally
special?  Or is this case intentional for some reason?

I guess someone with a bride knowledge of Emacs should look at all
locations of such defvars and see whether an init value should be added.

Michael.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Robert Pluim]

>>>>> On Tue, 17 Mar 2020 02:01:22 +0100, Michael Heerdegen <michael_heerdegen@web.de> said:

    Michael> Joost Kremers <joostkremers@fastmail.fm> writes:
    >> *** Eval error ***  Symbol’s value as variable is void:
    >> debugger-outer-match-data

    Michael> Another defvar with a missing init value so the variable is not globally
    Michael> special?  Or is this case intentional for some reason?

    Michael> I guess someone with a bride knowledge of Emacs should look at all
    Michael> locations of such defvars and see whether an init value should be added.

All 2970 of them? Donʼt let me stop you :-)

Robert

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Dmitry Gutov]

On 17.03.2020 3:01, Michael Heerdegen wrote:
> Another defvar with a missing init value so the variable is not globally
> special?  Or is this case intentional for some reason?
> 
> I guess someone with a bride knowledge of Emacs should look at all
> locations of such defvars and see whether an init value should be added.

It doesn't look like this bug should be solved this way. Having "... is 
unbound" is better because it points at some function forgetting to 
let-bind this variable (someone should find out which and to which values).

Whereas if it had an init value, that would mask such problems because 
nil is a somewhat valid value.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Noam Postavsky]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Joost Kremers <joostkremers@fastmail.fm> writes:
>
>> *** Eval error ***  Symbol’s value as variable is void:
>> debugger-outer-match-data
>
> Another defvar with a missing init value so the variable is not globally
> special?  Or is this case intentional for some reason?

It's something more complicated than that I think.  There are some extra
errors that get raised during the handling of the initial inciting
error, that somehow ends in inhibit-debugger being left set to t.  So
some spec/let-binding is getting snuck out of incorrectly perhaps?

It seems to be ielm specific, but I guess ielm just happens to be doing
something tricky enough to trigger the real bug which lies elsewhere.
As a workaround, you can (setq inhibit-debugger nil) after the first
failure happens.

I can't trigger the debugger-outer-match-data error in Emacs 26, but the
debugger doesn't seem to trigger from ielm at all in that version.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Michael Heerdegen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> It doesn't look like this bug should be solved this way. Having
> "... is unbound" is better because it points at some function
> forgetting to let-bind this variable (someone should find out which
> and to which values).
>
> Whereas if it had an init value, that would mask such problems because
> nil is a somewhat valid value.

If there are lots of cases like this under the cited thousands then I
guess we need a way to declare a variable special without assigning an
init value.

I don't like the (defvar var) syntax we have now.  It's different than
the (defvar var value) syntax in more than one way, and it's too
easily overseeable that it doesn't make a variable special.

Michael.

bug#40088: 27.0.90; Symbol?s value as variable is void: debugger-outer-match-data

[Drew Adams]

> > It doesn't look like this bug should be solved this way. Having
> > "... is unbound" is better because it points at some function
> > forgetting to let-bind this variable (someone should find out which
> > and to which values).
> >
> > Whereas if it had an init value, that would mask such problems because
> > nil is a somewhat valid value.
> 
> If there are lots of cases like this under the cited thousands then I
> guess we need a way to declare a variable special without assigning an
> init value.
> 
> I don't like the (defvar var) syntax we have now.  It's different than
> the (defvar var value) syntax in more than one way, and it's too
> easily overseeable that it doesn't make a variable special.

FWIW, I agree with this.

I was going to say something similar a
while back, but chickened out. ;-)

Not sure what a good alternative would be,
or whether it would be considered too late.

Common Lisp handles it differently, but its
approach can also be complicated, as Stefan
knows.

bug#40088: 27.0.90; Symbol?s value as variable is void: debugger-outer-match-data

[Noam Postavsky]

Drew Adams <drew.adams@oracle.com> writes:

>> I don't like the (defvar var) syntax we have now.  It's different than
>> the (defvar var value) syntax in more than one way, and it's too
>> easily overseeable that it doesn't make a variable special.
>
> FWIW, I agree with this.

Can you guys take this to emacs-devel please?  It's not really relevant
to this bug.

bug#40088: 27.0.90; Symbol?s value as variable is void: debugger-outer-match-data

[Drew Adams]

> > FWIW, I agree with this.
> 
> Can you guys take this to emacs-devel please?  It's not really relevant
> to this bug.

Sorry.  I agree; thanks.

I won't take it there, but if I have
something to add to whatever ends up
there I might add it.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Dmitry Gutov]

On 18.03.2020 2:06, Michael Heerdegen wrote:
> If there are lots of cases like this under the cited thousands then I
> guess we need a way to declare a variable special without assigning an
> init value.

It's already "special" where it's used (in one file). "value as variable 
is void" is what happens when it doesn't have an init value, and nobody 
bound it to anything.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Michael Heerdegen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> It's already "special" where it's used (in one file). "value as
> variable is void" is what happens when it doesn't have an init value,
> and nobody bound it to anything.

But some uses that "bind it to anything" might create lexical bindings
after the last changes - that's my concern.

Michael.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Dmitry Gutov]

On 19.03.2020 00:24, Michael Heerdegen wrote:
> But some uses that "bind it to anything" might create lexical bindings
> after the last changes - that's my concern.

Byte-compiler warns about unused lexical bindings.

bug#40088: 27.0.90; Symbol?s value as variable is void: debugger-outer-match-data

[Michael Heerdegen]

Noam Postavsky <npostavs@gmail.com> writes:

> Can you guys take this to emacs-devel please?  It's not really
> relevant to this bug.

When I wrote that I thought it would have.  But I see that my thoughts,
although I often seem to make good points, are also too often a
distraction for people really working on the bugs.  I hope to do better
in the future.  Thanks for the hint.

I'll start a new thread in emacs-dev.

Thanks,

Michael.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Michael Heerdegen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 19.03.2020 00:24, Michael Heerdegen wrote:
> > But some uses that "bind it to anything" might create lexical bindings
> > after the last changes - that's my concern.
>
> Byte-compiler warns about unused lexical bindings.

Yes, but the issues possibly introduced by Stefan's change are not
necessarily covered by byte compilation.  Like *ielm*, it uses `eval'
etc.  Sorry if I'm already polluting the bug report again, no doubt Noam
can estimate much better if there could be more cases like this.

Michael.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Noam Postavsky]

Noam Postavsky <npostavs@gmail.com> writes:

>>> *** Eval error ***  Symbol’s value as variable is void:
>>> debugger-outer-match-data

A simpler reproducer (although it only reproduces once per-session) is

    (make-local-variable 'xx)
    (let ((xx 1))
      (backtrace--locals 1))

In Emacs 26, this gives

Debugger entered--Lisp error: (void-variable xx)
  backtrace--locals(1)
  (let ((xx 1)) (backtrace--locals 1))
  ...
  eval-defun(nil)
  ...

The error shouldn't happen, but it's not as severe as in Emacs 27.  I
think because in v27, the debugger display code got more fancy, and
recursively triggers the problem.

Since the void-variable is getting triggered from
backtrace_eval_unrewind, I tried the patch below, which almost seems to
fix the problem, but after continuing from the debugger there's a
segfault in GC, so it's definitely not the Right Thing.  I think this is
a bit over my head.  Ccing Stefan, any ideas?

diff --git i/src/data.c w/src/data.c
index b153068846..5ce5e360ab 100644
--- i/src/data.c
+++ w/src/data.c
@@ -1573,7 +1573,7 @@ notify_variable_watchers (Lisp_Object symbol,
 /* Return the default value of SYMBOL, but don't check for voidness.
    Return Qunbound if it is void.  */
 
-static Lisp_Object
+Lisp_Object
 default_value (Lisp_Object symbol)
 {
   struct Lisp_Symbol *sym;
diff --git i/src/eval.c w/src/eval.c
index 4559a0e1f6..78a787c4ff 100644
--- i/src/eval.c
+++ w/src/eval.c
@@ -3816,7 +3816,7 @@ backtrace_eval_unrewind (int distance)
 	  {
 	    Lisp_Object sym = specpdl_symbol (tmp);
 	    Lisp_Object old_value = specpdl_old_value (tmp);
-	    set_specpdl_old_value (tmp, Fdefault_value (sym));
+	    set_specpdl_old_value (tmp, default_value (sym));
 	    Fset_default (sym, old_value);
 	  }
 	  break;
@@ -3832,7 +3832,7 @@ backtrace_eval_unrewind (int distance)
 	    if (!NILP (Flocal_variable_p (symbol, where)))
 	      {
 		set_specpdl_old_value
-		  (tmp, Fbuffer_local_value (symbol, where));
+		  (tmp, buffer_local_value (symbol, where));
                 set_internal (symbol, old_value, where, SET_INTERNAL_UNBIND);
 	      }
 	  }

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Stefan Monnier]

> Since the void-variable is getting triggered from
> backtrace_eval_unrewind, I tried the patch below, which almost seems
> to fix the problem,

Indeed, the patch looks good (except for the removal of `static` which
seems like a spurious artifact).

> but after continuing from the debugger there's a
> segfault in GC, so it's definitely not the Right Thing.

Maybe it's not directly related to your patch (your patch just reveals
another latent bug)?


        Stefan

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

tags 40088 + patch
quit

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Since the void-variable is getting triggered from
>> backtrace_eval_unrewind, I tried the patch below, which almost seems
>> to fix the problem,
>
> Indeed, the patch looks good (except for the removal of `static` which
> seems like a spurious artifact).

No, the removal is on purpose, because I'm calling default_value from
eval.c, and it's defined in data.c.  But, maybe you thought that because
I didn't add the corresponding declaration in a header file, and...

>> but after continuing from the debugger there's a
>> segfault in GC, so it's definitely not the Right Thing.

... the segfault happens because I forgot to declare default_value in
lisp.h, so its return value got converted to int.  *Forehead slap*

So here's the proper patch.  Eli, is it okay for emacs-27?  As I
mentioned earlier, the underlying bug was present in Emacs 26 and
earlier, but it's only surfaced in Emacs 27 because of debug.el
enhancements.


[-- Attachment #2: patch --]
[-- Type: text/plain, Size: 2139 bytes --]

From 9afae526cf1959f5922dffb9254578dfe12ee4be Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Fri, 20 Mar 2020 04:07:39 -0400
Subject: [PATCH] Don't signal during backtrace unrewind (Bug#40088)

* src/data.c (default_value): Make non-static.
* src/lisp.h: Declare it.
* src/eval.c (backtrace_eval_unrewind): Call default_value and
buffer_local_value instead of Fdefault_value and Fbuffer_local_value,
respectively, so that unrewinding to unbound variables will not signal
an error.
---
 src/data.c | 2 +-
 src/eval.c | 4 ++--
 src/lisp.h | 1 +
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/src/data.c b/src/data.c
index b153068846..5ce5e360ab 100644
--- a/src/data.c
+++ b/src/data.c
@@ -1573,7 +1573,7 @@ notify_variable_watchers (Lisp_Object symbol,
 /* Return the default value of SYMBOL, but don't check for voidness.
    Return Qunbound if it is void.  */
 
-static Lisp_Object
+Lisp_Object
 default_value (Lisp_Object symbol)
 {
   struct Lisp_Symbol *sym;
diff --git a/src/eval.c b/src/eval.c
index 4559a0e1f6..78a787c4ff 100644
--- a/src/eval.c
+++ b/src/eval.c
@@ -3816,7 +3816,7 @@ backtrace_eval_unrewind (int distance)
 	  {
 	    Lisp_Object sym = specpdl_symbol (tmp);
 	    Lisp_Object old_value = specpdl_old_value (tmp);
-	    set_specpdl_old_value (tmp, Fdefault_value (sym));
+	    set_specpdl_old_value (tmp, default_value (sym));
 	    Fset_default (sym, old_value);
 	  }
 	  break;
@@ -3832,7 +3832,7 @@ backtrace_eval_unrewind (int distance)
 	    if (!NILP (Flocal_variable_p (symbol, where)))
 	      {
 		set_specpdl_old_value
-		  (tmp, Fbuffer_local_value (symbol, where));
+		  (tmp, buffer_local_value (symbol, where));
                 set_internal (symbol, old_value, where, SET_INTERNAL_UNBIND);
 	      }
 	  }
diff --git a/src/lisp.h b/src/lisp.h
index 8674fe11a6..92294ac1d3 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -594,6 +594,7 @@ #define ENUM_BF(TYPE) enum TYPE
 
 /* Defined in data.c.  */
 extern AVOID wrong_type_argument (Lisp_Object, Lisp_Object);
+extern Lisp_Object default_value (Lisp_Object symbol);
 
 
 /* Defined in emacs.c.  */
-- 
2.11.0

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Date: Fri, 20 Mar 2020 04:37:49 -0400
> Cc: Michael Heerdegen <michael_heerdegen@web.de>,
>  Joost Kremers <joostkremers@fastmail.fm>, 40088@debbugs.gnu.org
> 
> So here's the proper patch.  Eli, is it okay for emacs-27?  As I
> mentioned earlier, the underlying bug was present in Emacs 26 and
> earlier, but it's only surfaced in Emacs 27 because of debug.el
> enhancements.

I don't think I understand what problem are you trying to solve, let
alone why it's important to solve it in Emacs 27.  At least the
original issue doesn't sound important to me, but maybe I'm missing
something.

I also don't think I understand what is your analysis of the issue;
perhaps the commit log message could include more detailed
explanation.

Thanks.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Stefan Monnier]

> So here's the proper patch.

LGTM.

> Subject: [PATCH] Don't signal during backtrace unrewind (Bug#40088)

I don't think "[PATCH]" should be in the subject.


        Stefan

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Robert Pluim]

>>>>> On Fri, 20 Mar 2020 09:20:12 -0400, Stefan Monnier <monnier@iro.umontreal.ca> said:

    >> So here's the proper patch.
    Stefan> LGTM.

    >> Subject: [PATCH] Don't signal during backtrace unrewind (Bug#40088)

    Stefan> I don't think "[PATCH]" should be in the subject.

Why not? 'git am' strips that.

Robert

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Stefan Monnier]

>     Stefan> I don't think "[PATCH]" should be in the subject.
> Why not? 'git am' strips that.

Really?  Then, I guess it's harmless.


        Stefan

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't think I understand what problem are you trying to solve, let
> alone why it's important to solve it in Emacs 27.  At least the
> original issue doesn't sound important to me, but maybe I'm missing
> something.

> I also don't think I understand what is your analysis of the issue;
> perhaps the commit log message could include more detailed
> explanation.

Yes, sorry, my analysis previously wasn't really more than 'raising a
signal while messing with the pdl stack seems bad'.  I've debugged a bit
more to see what's happening.

The underlying cause is in backtrace_eval_unrewind, which is used to
temporarily reverse let-bindings (it's called with a positive argument
to reverse bindings, and then a negative argument to re-apply them) by
backtrace--locals and backtrace-eval.  For the SPECPDL_LET_DEFAULT and
SPECPDL_LET_LOCAL cases (which occur for let-bindings on buffer-local
variables), the code calls Fdefault_value and Fbuffer_local_value on the
symbol.

For symbols which are unbound at top-level, the first (with positive
argument) call to backtrace_eval_unrewind will set the symbol's value to
unbound (putting the current value in the specpdl's "old value" slot).
On the second (with negative argument) call, backtrace_eval_unrewind
attempts to retrieve the symbol's value with Fdefault_value or
Fbuffer_local_value, but that raises a void-variable signal.  This
interrupts the restoration of the let-bindings, so any other variables
more recent on the stack will now have the wrong value.

ielm happens to have some buffer-local variables which are unbound at
top-level, which is why it can trigger this problem.  But any situation
with void variables with buffer-local let-bindings can also trigger it.

Now, all of the above is true also for Emacs 26 and earlier.  The main
difference in Emacs 27 is that debug.el calls backtrace--locals up
front, during setup.  In Emacs 26, it's only called when calling
backtrace-toggle-locals.  This means there would be hardly other
variables on the stack to get wrong value.  Also, errors in v26 ielm
evaluation don't raise the debugger (because ielm-eval-input used
conditiion-case rather than condition-case-unless-debug).

So the effects of this bug are newly significant in Emacs 27.  The fact
that it strikes when the debugger opens, means that it can multiply the
difficulty of analysing other bugs.  And the fix looks quite small and
safe enough to qualify for emacs-27.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: michael_heerdegen@web.de,  joostkremers@fastmail.fm,
>   40088@debbugs.gnu.org,  monnier@iro.umontreal.ca
> Date: Sat, 21 Mar 2020 19:47:07 -0400
> 
> The underlying cause is in backtrace_eval_unrewind, which is used to
> temporarily reverse let-bindings (it's called with a positive argument
> to reverse bindings, and then a negative argument to re-apply them) by
> backtrace--locals and backtrace-eval.  For the SPECPDL_LET_DEFAULT and
> SPECPDL_LET_LOCAL cases (which occur for let-bindings on buffer-local
> variables), the code calls Fdefault_value and Fbuffer_local_value on the
> symbol.
> 
> For symbols which are unbound at top-level, the first (with positive
> argument) call to backtrace_eval_unrewind will set the symbol's value to
> unbound (putting the current value in the specpdl's "old value" slot).
> On the second (with negative argument) call, backtrace_eval_unrewind
> attempts to retrieve the symbol's value with Fdefault_value or
> Fbuffer_local_value, but that raises a void-variable signal.  This
> interrupts the restoration of the let-bindings, so any other variables
> more recent on the stack will now have the wrong value.
> 
> ielm happens to have some buffer-local variables which are unbound at
> top-level, which is why it can trigger this problem.  But any situation
> with void variables with buffer-local let-bindings can also trigger it.
> 
> Now, all of the above is true also for Emacs 26 and earlier.

It sounds like a very basic error.  How come we never saw it before?
Is it just luck?

> So the effects of this bug are newly significant in Emacs 27.  The fact
> that it strikes when the debugger opens, means that it can multiply the
> difficulty of analysing other bugs.  And the fix looks quite small and
> safe enough to qualify for emacs-27.

I'm okay with making the change on the emacs-27 branch, but please put
some of the description above in the commit log message, including why
calling default_value etc. avoids the problem (which might not be
obvious, unless one remembers by heart what Fdefault_value does).

Thanks.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Noam Postavsky]

tags 40088 fixed
close 40088 27.1
quit

Eli Zaretskii <eliz@gnu.org> writes:

> It sounds like a very basic error.  How come we never saw it before?
> Is it just luck?

Like I said, the debug.el code changed in Emacs 27 so that the bug can
be triggered more often.

    The main difference in Emacs 27 is that debug.el calls backtrace--locals
    up front, during setup.  In Emacs 26, it's only called when calling
    backtrace-toggle-locals.

> I'm okay with making the change on the emacs-27 branch, but please put
> some of the description above in the commit log message, including why
> calling default_value etc. avoids the problem (which might not be
> obvious, unless one remembers by heart what Fdefault_value does).

Done and pushed.

[1: 9ab85f087f]: 2020-03-22 23:06:31 -0400
  Fix cl-concatenate (Bug#40180)
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=9ab85f087f7db38168dcf07d24f51ecd2c583f8a

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: michael_heerdegen@web.de,  joostkremers@fastmail.fm,
>   40088@debbugs.gnu.org,  monnier@iro.umontreal.ca
> Date: Sun, 22 Mar 2020 23:17:06 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It sounds like a very basic error.  How come we never saw it before?
> > Is it just luck?
> 
> Like I said, the debug.el code changed in Emacs 27 so that the bug can
> be triggered more often.

Is this code used only in debug.el?

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Noam Postavsky <npostavs@gmail.com>
>> Cc: michael_heerdegen@web.de,  joostkremers@fastmail.fm,
>>   40088@debbugs.gnu.org,  monnier@iro.umontreal.ca
>> Date: Sun, 22 Mar 2020 23:17:06 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > It sounds like a very basic error.  How come we never saw it before?
>> > Is it just luck?
>> 
>> Like I said, the debug.el code changed in Emacs 27 so that the bug can
>> be triggered more often.
>
> Is this code used only in debug.el?

Actually, the call to backtrace--locals moved from debug.el to the new
file backtrace.el (via backtrace-get-frames).  edebug.el and ert.el call
backtrace-get-frames too.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: michael_heerdegen@web.de,  joostkremers@fastmail.fm,
>   40088@debbugs.gnu.org,  monnier@iro.umontreal.ca
> Date: Sun, 22 Mar 2020 23:40:49 -0400
> 
> >> > It sounds like a very basic error.  How come we never saw it before?
> >> > Is it just luck?
> >> 
> >> Like I said, the debug.el code changed in Emacs 27 so that the bug can
> >> be triggered more often.
> >
> > Is this code used only in debug.el?
> 
> Actually, the call to backtrace--locals moved from debug.el to the new
> file backtrace.el (via backtrace-get-frames).  edebug.el and ert.el call
> backtrace-get-frames too.

So only backtrace-related functions can bump into this?  The root
cause sounded much more general to me.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

> So only backtrace-related functions can bump into this?  The root
> cause sounded much more general to me.

Yes, it's backtrace-related functions.  The function with the problem,
backtrace_eval_unrewind, is accessible to Lisp only via backtrace--locals
and backtrace-eval.

bug#40088: 27.0.90; Symbol’s value as variable is void: debugger-outer-match-data

[Joost Kremers]

On Sun, Mar 22 2020, Eli Zaretskii wrote:
> It sounds like a very basic error.  How come we never saw it 
> before?
> Is it just luck?

Well, as Noam mentioned, the recipe I provided doesn't produce an 
explicit error on Emacs 26, it just keeps the backtrace buffer 
from appearing. I've always found `toggle-debug-on-error` a bit of 
hit-and-miss, sometimes it would display a backtrace when an error 
occurs, sometimes not. That might have been this bug in action, I 
don't know. (I do use IELM a lot, so who knows.)

I've never been able to come up with a reliable recipe to 
reproduce the issue, however, so I never reported it.

-- 
Joost Kremers
Life has its moments