bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Sven Joachim]

Today I've built and installed an emacs-snapshot package for Debian, and
I'm seeing this:

,----
| % emacs -Q
| Wrong type argument: keymapp, ("DEAD" . 35215396)
| % echo $?
| 255
`----

Unfortunately this is a Heisenbug, I'm not able to reproduce it under
gdb.  It even depends on the exact contents of argv[0], i.e. running
"/usr/bin/emacs -Q" or "emacs-snapshot -Q" does not show the error.

A similar issue has been reported against Debian's emacs23 package, see
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=550170.  The most
valuable message there is probably
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=550170#26.


In GNU Emacs 23.1.91.1 (i486-pc-linux-gnu, GTK+ Version 2.18.5)
 of 2010-01-12 on turtle, modified by Debian
 (emacs-snapshot package, version 1:20100112-1)
Windowing system distributor `The X.Org Foundation', version 11.0.10703902
configured using `configure  '--build' 'i486-linux-gnu' '--host' 'i486-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs-snapshot:/etc/emacs:/usr/local/share/emacs/23.1.91/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.1.91/site-lisp:/usr/share/emacs/site-lisp' '--with-x=yes' '--with-x-toolkit=gtk' 'build_alias=i486-linux-gnu' 'host_alias=i486-linux-gnu' 'CFLAGS=-DDEBIAN -DSITELOAD_PURESIZE_EXTRA=5000 -g -O2' 'LDFLAGS=-g -Wl,--as-needed' 'CPPFLAGS=''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: C
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: de_DE.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Sven Joachim]

On 2010-01-12 15:58 +0100, Sven Joachim wrote:

> Today I've built and installed an emacs-snapshot package for Debian, and
> I'm seeing this:
>
> ,----
> | % emacs -Q
> | Wrong type argument: keymapp, ("DEAD" . 35215396)
> | % echo $?
> | 255
> `----
>
> Unfortunately this is a Heisenbug, I'm not able to reproduce it under
> gdb.

I now managed to attach gdb to a running process and get a backtrace
after setting a breakpoint on Fsignal:

,----
| (gdb) bt
| #0  Fsignal (error_symbol=138366426, data=138990086) at eval.c:1629
| #1  0x0818cfa8 in xsignal (error_symbol=138366426, data=138990086) at eval.c:1729
| #2  0x0818d377 in xsignal2 (error_symbol=138366426, arg1=138358370, arg2=140861614) at eval.c:1753
| #3  0x0817b0a1 in wrong_type_argument (predicate=138358370, value=140861614) at data.c:118
| #4  0x081324a5 in get_keymap (object=140861614, error=1, autoload=1) at keymap.c:307
| #5  0x08132d32 in keymap_parent (keymap=140861614, autoload=1) at keymap.c:321
| #6  0x08132dc9 in Fkeymap_parent (keymap=140861614) at keymap.c:341
| #7  0x0818cb44 in Ffuncall (nargs=2, args=0xffa14ac8) at eval.c:3024
| #8  0x081c4bd1 in Fbyte_code (bytestr=137365729, vector=137365749, maxdepth=16) at bytecode.c:679
| #9  0x0818e944 in funcall_lambda (fun=<value optimized out>, nargs=<value optimized out>, arg_vector=0x848d216) at eval.c:3211
| #10 0x0818c953 in Ffuncall (nargs=2, args=0xffa14c30) at eval.c:3081
| #11 0x081c4bd1 in Fbyte_code (bytestr=136627065, vector=136627085, maxdepth=20) at bytecode.c:679
| #12 0x0818e944 in funcall_lambda (fun=<value optimized out>, nargs=<value optimized out>, arg_vector=0x848d216) at eval.c:3211
| #13 0x0818c953 in Ffuncall (nargs=2, args=0xffa14db0) at eval.c:3081
| #14 0x081c4bd1 in Fbyte_code (bytestr=137074713, vector=137074733, maxdepth=24) at bytecode.c:679
| #15 0x0818e944 in funcall_lambda (fun=<value optimized out>, nargs=<value optimized out>, arg_vector=0x848d216) at eval.c:3211
| #16 0x0818c953 in Ffuncall (nargs=2, args=0xffa14f30) at eval.c:3081
| #17 0x081c4bd1 in Fbyte_code (bytestr=137071977, vector=137071997, maxdepth=24) at bytecode.c:679
| #18 0x0818e944 in funcall_lambda (fun=<value optimized out>, nargs=<value optimized out>, arg_vector=0x848d216) at eval.c:3211
| #19 0x0818c953 in Ffuncall (nargs=1, args=0xffa150b0) at eval.c:3081
| #20 0x081c4bd1 in Fbyte_code (bytestr=136675297, vector=136675317, maxdepth=28) at bytecode.c:679
| #21 0x0818e944 in funcall_lambda (fun=<value optimized out>, nargs=<value optimized out>, arg_vector=0x848d216) at eval.c:3211
| #22 0x0818c953 in Ffuncall (nargs=1, args=0xffa15230) at eval.c:3081
| #23 0x081c4bd1 in Fbyte_code (bytestr=136672241, vector=136672261, maxdepth=24) at bytecode.c:679
| #24 0x0818e944 in funcall_lambda (fun=<value optimized out>, nargs=<value optimized out>, arg_vector=0x848d216) at eval.c:3211
| #25 0x0818eb43 in apply_lambda (fun=136672221, args=138328562, eval_flag=1) at eval.c:3135
| #26 0x0818e224 in Feval (form=138570382) at eval.c:2406
| #27 0x08123cf3 in top_level_2 () at keyboard.c:1369
| #28 0x0818be91 in internal_condition_case (bfun=0x8123ce0 <top_level_2>, handlers=138366378, hfun=0x8128a00 <cmd_error>) at eval.c:1490
| #29 0x081287b5 in top_level_1 () at keyboard.c:1377
| #30 0x0818bf71 in internal_catch (tag=138363450, func=0x8128750 <top_level_1>, arg=138328562) at eval.c:1226
| #31 0x08128831 in command_loop () at keyboard.c:1332
| #32 0x08128bea in recursive_edit_1 () at keyboard.c:954
| #33 0x08128d12 in Frecursive_edit () at keyboard.c:1016
| #34 0x0811d3d8 in main (argc=<value optimized out>, argv=<value optimized out>) at emacs.c:1833
| 
| Lisp Backtrace:
| "keymap-parent" (0xffa14acc)
| "x-setup-function-keys" (0xffa14c34)
| "x-create-frame-with-faces" (0xffa14db4)
| "make-frame" (0xffa14f34)
| "frame-initialize" (0xffa150b4)
| "command-line" (0xffa15234)
| "normal-top-level" (0xffa15330)
| (gdb)
`----

Any advice how to proceed would be appreciated.

Sven

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Stefan Monnier]

>> Today I've built and installed an emacs-snapshot package for Debian, and
>> I'm seeing this:
>> 
>> ,----
>> | % emacs -Q
>> | Wrong type argument: keymapp, ("DEAD" . 35215396)
>> | % echo $?
>> | 255
>> `----

This cons cell is almost for sure on the free-list.  I.e. the memory
management code thought it was unused and put it back on the free list
for reuse (and luckily it hasn't been reused yet).

This implies that a backtrace probably won't be sufficient to track it
down :-(

>> Unfortunately this is a Heisenbug, I'm not able to reproduce it under
>> gdb.

> I now managed to attach gdb to a running process and get a backtrace
> after setting a breakpoint on Fsignal:

> | Lisp Backtrace:
> | "keymap-parent" (0xffa14acc)
> | "x-setup-function-keys" (0xffa14c34)
> | "x-create-frame-with-faces" (0xffa14db4)
> | "make-frame" (0xffa14f34)
> | "frame-initialize" (0xffa150b4)
> | "command-line" (0xffa15234)
> | "normal-top-level" (0xffa15330)
> | (gdb)

OK, so if we trust your backtrace (if you compiled with optimizations,
there's a good chance that some parts of the backtrace aren't to be
trusted, tho the Lisp part is less likely to be invented), the above
implies that we're looking at the (keymap-parent local-function-key-map)
call in x-win.el's x-setup-function-keys.

> ,----
> | (gdb) bt
> | #0  Fsignal (error_symbol=138366426, data=138990086) at eval.c:1629
> | #1  0x0818cfa8 in xsignal (error_symbol=138366426, data=138990086) at eval.c:1729
> | #2  0x0818d377 in xsignal2 (error_symbol=138366426, arg1=138358370, arg2=140861614) at eval.c:1753
> | #3  0x0817b0a1 in wrong_type_argument (predicate=138358370, value=140861614) at data.c:118
> | #4  0x081324a5 in get_keymap (object=140861614, error=1, autoload=1) at keymap.c:307
> | #5  0x08132d32 in keymap_parent (keymap=140861614, autoload=1) at keymap.c:321
> | #6  0x08132dc9 in Fkeymap_parent (keymap=140861614) at keymap.c:341

And this seems to indicate that the free cons cell ("DEAD" . 35215396)
is actually the one we got from local-function-key-map.  As for how that
could happen...


        Stefan

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> | #4  0x081324a5 in get_keymap (object=140861614, error=1, autoload=1) at keymap.c:307
>> | #5  0x08132d32 in keymap_parent (keymap=140861614, autoload=1) at keymap.c:321
>> | #6  0x08132dc9 in Fkeymap_parent (keymap=140861614) at keymap.c:341
>> ...
>> | Lisp Backtrace:
>> | "keymap-parent" (0xffa14acc)
>> | "x-setup-function-keys" (0xffa14c34)
>> | "x-create-frame-with-faces" (0xffa14db4)
>> | "make-frame" (0xffa14f34)
>> | "frame-initialize" (0xffa150b4)
>> | "command-line" (0xffa15234)
>> | "normal-top-level" (0xffa15330)
>> | (gdb)
>
> OK, so if we trust your backtrace (if you compiled with optimizations,
> there's a good chance that some parts of the backtrace aren't to be
> trusted, tho the Lisp part is less likely to be invented), the above
> implies that we're looking at the (keymap-parent local-function-key-map)
> call in x-win.el's x-setup-function-keys.

Sven, could you verify this?  In GDB, do

  f 5
  p keymap
  p current_kboard->Vlocal_function_key_map

The two values should be the same.

Even if that's the case, I'm not sure why Vlocal_function_key_map is
getting garbage-collected, tho.

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Chong Yidong]

Chong Yidong <cyd@stupidchicken.com> writes:

> Even if that's the case, I'm not sure why Vlocal_function_key_map is
> getting garbage-collected, tho.

OK, I see one place where this could happen.  In xterm.c:10207:

    terminal->kboard = (KBOARD *) xmalloc (sizeof (KBOARD));
    ...
    if (!EQ (XSYMBOL (Qvendor_specific_keysyms)->function, Qunbound))
      {
        char *vendor = ServerVendor (dpy);
        /* Temporarily hide the partially initialized terminal */
        terminal_list = terminal->next_terminal;
        UNBLOCK_INPUT;
        terminal->kboard->Vsystem_key_alist
          = call1 (Qvendor_specific_keysyms,
                   vendor ? build_string (vendor) : empty_unibyte_string);
        BLOCK_INPUT;
        ...
      }

    terminal->kboard->next_kboard = all_kboards;
    all_kboards = terminal->kboard;

It's possible that garbage-collection occurs during the call1, when the
keyboard has not yet been put on the all_kboards linked list.  In that
case, it will not be protected from garbage collection.

Sven, does the following patch fix the bug?


*** src/xterm.c	2010-01-09 04:16:32 +0000
--- src/xterm.c	2010-01-13 17:27:27 +0000
***************
*** 10210,10215 ****
--- 10210,10216 ----
  	if (!EQ (XSYMBOL (Qvendor_specific_keysyms)->function, Qunbound))
  	  {
  	    char *vendor = ServerVendor (dpy);
+ 	    int count = inhibit_garbage_collection ();
  	    /* Temporarily hide the partially initialized terminal */
  	    terminal_list = terminal->next_terminal;
  	    UNBLOCK_INPUT;
***************
*** 10217,10222 ****
--- 10218,10224 ----
  	      = call1 (Qvendor_specific_keysyms,
  		       vendor ? build_string (vendor) : empty_unibyte_string);
  	    BLOCK_INPUT;
+ 	    unbind_to (count, Qnil);
  	    terminal->next_terminal = terminal_list;
   	    terminal_list = terminal;
  	  }

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Stefan Monnier]

>> Even if that's the case, I'm not sure why Vlocal_function_key_map is
>> getting garbage-collected, tho.

> OK, I see one place where this could happen.  In xterm.c:10207:

terminal-> kboard = (KBOARD *) xmalloc (sizeof (KBOARD));
>     ...
>     if (!EQ (XSYMBOL (Qvendor_specific_keysyms)->function, Qunbound))
>       {
>         char *vendor = ServerVendor (dpy);
>         /* Temporarily hide the partially initialized terminal */
>         terminal_list = terminal->next_terminal;
>         UNBLOCK_INPUT;
terminal-> kboard->Vsystem_key_alist
>           = call1 (Qvendor_specific_keysyms,
>                    vendor ? build_string (vendor) : empty_unibyte_string);
>         BLOCK_INPUT;
>         ...
>       }

Indeed, that looks risky.  Why don't we add this new kboard to the
all_kboards list before calling Qvendor_specific_keysyms?


        Stefan

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Even if that's the case, I'm not sure why Vlocal_function_key_map is
>>> getting garbage-collected, tho.
>
>> OK, I see one place where this could happen.  In xterm.c:10207:
>
> terminal-> kboard = (KBOARD *) xmalloc (sizeof (KBOARD));
>>     ...
>>     if (!EQ (XSYMBOL (Qvendor_specific_keysyms)->function, Qunbound))
>>       {
>>         char *vendor = ServerVendor (dpy);
>>         /* Temporarily hide the partially initialized terminal */
>>         terminal_list = terminal->next_terminal;
>>         UNBLOCK_INPUT;
> terminal-> kboard->Vsystem_key_alist
>>           = call1 (Qvendor_specific_keysyms,
>>                    vendor ? build_string (vendor) : empty_unibyte_string);
>>         BLOCK_INPUT;
>>         ...
>>       }
>
> Indeed, that looks risky.  Why don't we add this new kboard to the
> all_kboards list before calling Qvendor_specific_keysyms?

We'd still have to protect the terminal object.  I've checked in a fix
that uses inhibit_garbage_collection, but if you'd like to replace this
with a more elegant fix, go ahead.

Since it appears to fix the problem (as far as Sven can tell), I'll
close this bug.  I'll forward the patch to the Debian bts too.

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Stefan Monnier]

>> Indeed, that looks risky.  Why don't we add this new kboard to the
>> all_kboards list before calling Qvendor_specific_keysyms?
> We'd still have to protect the terminal object.

Why?  It's a normal Lisp object, so it should be protected by the usual
GCPRO or stack marking, no?
[ Oddly enough, mark_terminal doesn't traverse the terminal's kboard.  ]

As for my original question: can anyone think of a reason not to place
the new kboard in all_kboards earlier?


        Stefan

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Indeed, that looks risky.  Why don't we add this new kboard to the
>>> all_kboards list before calling Qvendor_specific_keysyms?
>> We'd still have to protect the terminal object.
>
> Why?  It's a normal Lisp object, so it should be protected by the usual
> GCPRO or stack marking, no?
> [ Oddly enough, mark_terminal doesn't traverse the terminal's kboard.  ]

But the terminal object is removed from the terminal list before the
call1 (this was before my latest patch):

 if (!EQ (XSYMBOL (Qvendor_specific_keysyms)->function, Qunbound))
   {
     char *vendor = ServerVendor (dpy);
     terminal_list = terminal->next_terminal;
     UNBLOCK_INPUT;
     terminal->kboard->Vsystem_key_alist
       = call1 (Qvendor_specific_keysyms,
                vendor ? build_string (vendor) : empty_unibyte_string);
     ...

This means that mark_terminal won't be able to reach the newly-allocated
terminal object, so its contents could get gc'ed.  I haven't checked
that there is anything in the terminal object that is actually in danger
of being gc'ed, but given some could hypothetically be added in the
future, it seems better to pre-emptively turn off gc during this
function call.

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Stefan Monnier]

>>>> Indeed, that looks risky.  Why don't we add this new kboard to the
>>>> all_kboards list before calling Qvendor_specific_keysyms?
>>> We'd still have to protect the terminal object.
>> 
>> Why?  It's a normal Lisp object, so it should be protected by the usual
>> GCPRO or stack marking, no?
>> [ Oddly enough, mark_terminal doesn't traverse the terminal's kboard.  ]

> But the terminal object is removed from the terminal list before the
> call1 (this was before my latest patch):

But that's OK because the `terminal' variable is still on the stack so
the stack marking should see it and mark the corresponding terminal,
like any other Lisp object.

We should add a corresponding GCPRO, to make sure it also works without
conservative stack marking.


        Stefan

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Sven Joachim]

On 2010-01-13 18:27 +0100, Chong Yidong wrote:

> Chong Yidong <cyd@stupidchicken.com> writes:
>
>> Even if that's the case, I'm not sure why Vlocal_function_key_map is
>> getting garbage-collected, tho.
>
> OK, I see one place where this could happen.  In xterm.c:10207:
>
>     terminal->kboard = (KBOARD *) xmalloc (sizeof (KBOARD));
>     ...
>     if (!EQ (XSYMBOL (Qvendor_specific_keysyms)->function, Qunbound))
>       {
>         char *vendor = ServerVendor (dpy);
>         /* Temporarily hide the partially initialized terminal */
>         terminal_list = terminal->next_terminal;
>         UNBLOCK_INPUT;
>         terminal->kboard->Vsystem_key_alist
>           = call1 (Qvendor_specific_keysyms,
>                    vendor ? build_string (vendor) : empty_unibyte_string);
>         BLOCK_INPUT;
>         ...
>       }
>
>     terminal->kboard->next_kboard = all_kboards;
>     all_kboards = terminal->kboard;
>
> It's possible that garbage-collection occurs during the call1, when the
> keyboard has not yet been put on the all_kboards linked list.  In that
> case, it will not be protected from garbage collection.
>
> Sven, does the following patch fix the bug?

Apparently, at least I can no longer reproduce the bug.

Sven

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Sven Joachim]

On 2010-01-13 18:11 +0100, Chong Yidong wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>> OK, so if we trust your backtrace (if you compiled with optimizations,
>> there's a good chance that some parts of the backtrace aren't to be
>> trusted, tho the Lisp part is less likely to be invented), the above
>> implies that we're looking at the (keymap-parent local-function-key-map)
>> call in x-win.el's x-setup-function-keys.
>
> Sven, could you verify this?  In GDB, do
>
>   f 5
>   p keymap
>   p current_kboard->Vlocal_function_key_map
>
> The two values should be the same.

Yes, they are.

Sven

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Chong Yidong]

> | % emacs -Q
> | Wrong type argument: keymapp, ("DEAD" . 35215396)
>
> Unfortunately this is a Heisenbug, I'm not able to reproduce it under
> gdb.  It even depends on the exact contents of argv[0], i.e. running
> "/usr/bin/emacs -Q" or "emacs-snapshot -Q" does not show the error.
>
> A similar issue has been reported against Debian's emacs23 package, see
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=550170.  The most
> valuable message there is probably
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=550170#26.

Pretty bizaare.  I can't reproduce this on my machine, but according to
the Debian bug report, it seems to be related to environment variables
somehow.  Do you see the bug if you start Emacs with an empty
environment like

 env -i DISPLAY=":0.0" HOME=/home/cyd /home/cyd/emacs/src/emacs

(replacing cyd with your username)?

bug#5365: 23.1.91; Wrong type argument: keymapp, ("DEAD" . 35215396)

[Sven Joachim]

On 2010-01-12 20:08 +0100, Chong Yidong wrote:

>> | % emacs -Q
>> | Wrong type argument: keymapp, ("DEAD" . 35215396)
>>
>> Unfortunately this is a Heisenbug, I'm not able to reproduce it under
>> gdb.  It even depends on the exact contents of argv[0], i.e. running
>> "/usr/bin/emacs -Q" or "emacs-snapshot -Q" does not show the error.
>>
>> A similar issue has been reported against Debian's emacs23 package, see
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=550170.  The most
>> valuable message there is probably
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=550170#26.
>
> Pretty bizaare.  I can't reproduce this on my machine, but according to
> the Debian bug report, it seems to be related to environment variables
> somehow.

Apparently, though I don't see the bug with Debian's emacs23 package in
the same environment.

>  Do you see the bug if you start Emacs with an empty
> environment like
>
>  env -i DISPLAY=":0.0" HOME=/home/cyd /home/cyd/emacs/src/emacs
>
> (replacing cyd with your username)?

In that case, Emacs starts normally.

Sven