bug#15688: 24.3.50; clear-temporary-overlay-map

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

I found that the crashes in GC result from trying to mark
clear-temporary-overlay-map.  Its function cell is
a vectorlike which pr reports as #<invalid-lisp-object ADDR>.

The presence of such an object in the function cell is a bug,
and it seems to me that for GC to try to mark the contents
of a vectorlike with such a type is a second bug.




In GNU Emacs 24.3.50.11 (mips64el-unknown-linux-gnu, GTK+ Version 2.20.1)
 of 2013-10-03 on chiefs-gnewsense
Bzr revision: 114502 monnier@iro.umontreal.ca-20131002233348-j245zww7t0dfmeng
System Description:	gNewSense GNU/Linux 3.0 (parkes)

Configured using:
 `configure 'CFLAGS=-g -O0''

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: RMAIL

Minor modes in effect:
  shell-dirtrack-mode: t
  gpm-mouse-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Recent input:
r a . x TAB RET d d d u x 1 r C-u C-p C-u C-p C-p C-p 
C-p ESC f C-f C-f C-k C-x o C-u C-n C-n ESC f C-f C-f 
C-@ C-e ESC w C-x o C-y C-a C-u C-n C-u C-n C-n C-n 
C-n C-o W h a t SPC f o r m a t SPC d o SPC t h e y 
SPC ESC DEL ESC DEL w i l l SPC b e SPC u s e d SPC 
f o r SPC t h e SPC s t r e a m i n g ? C-c C-c d d 
d d x SPC C-s [ 2 3 ] C-s C-a C-n W C-c C-c ESC v d 
d d x T C-c C-c d d d d d d d d d d x C-x b i n o u 
t RET ESC x g r e p RET C-g C-x b TAB RET i x m a i 
l / l a p t o p TAB RET C-l ESC - ESC s b u n n i e 
RET ESC - ESC s RET C-@ C-e ESC w C-x b R TAB 1 RET 
C-x k RET C-x b 1 r DEL DEL C-g 1 r L a SPC d i r e 
c c C-\ i o ' n SPC q u e SPC t e n g o SPC e s SPC 
C-y . RET C-c C-c C-x k RET RET y e s RET r C-_ C-@ 
C-p ESC w C-x o r C-y C-c C-c d d d ESC x r e p o r 
t SPC e m a s SPC DEL DEL c DEL a c s SPC b u g RE
T

Recent messages:
Mark set
Sending...
Wrote /home/rms/outgoing/out-92
Sending...done
Please answer yes or no.
Undo!
Mark set [2 times]
Sending...
Wrote /home/rms/outgoing/out-93
Sending...done

Load-path shadows:
/home/rms/emacs-bzr/trunk/lisp/net/shr-color hides /home/rms/emacs-bzr/trunk/lisp/gnus/shr-color
/home/rms/emacs-bzr/trunk/lisp/net/shr hides /home/rms/emacs-bzr/trunk/lisp/gnus/shr

Features:

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Stefan Monnier]

> I found that the crashes in GC result from trying to mark
> clear-temporary-overlay-map.  Its function cell is
> a vectorlike which pr reports as #<invalid-lisp-object ADDR>.

The most likely explanation is that it used to be a byte-code-function-p
object but was somehow GC'd too early.


        Stefan

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

        [ To any NSA and FBI agents reading my email: please consider
        [ whether defending the US Constitution against all enemies,
        [ foreign or domestic, requires you to follow Snowden's example.

    > I found that the crashes in GC result from trying to mark
    > clear-temporary-overlay-map.  Its function cell is
    > a vectorlike which pr reports as #<invalid-lisp-object ADDR>.

    The most likely explanation is that it used to be a byte-code-function-p
    object but was somehow GC'd too early.

If that symbol is interned, why would its function definition get
GC'd at all?

Similar crashes have occurred several times.  This is the first time
I analyzed it to see what data it is operating on.  I will do it again
next time.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Stefan Monnier]

>     The most likely explanation is that it used to be a byte-code-function-p
>     object but was somehow GC'd too early.
> If that symbol is interned, why would its function definition get
> GC'd at all?

It's not interned.  It's "make-symbol"d in set-temporary-overlay-map.


        Stefan

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

        [ To any NSA and FBI agents reading my email: please consider
        [ whether defending the US Constitution against all enemies,
        [ foreign or domestic, requires you to follow Snowden's example.

    > If that symbol is interned, why would its function definition get
    > GC'd at all?

    It's not interned.  It's "make-symbol"d in set-temporary-overlay-map.

Ok, but apparently the symbol does get marked -- so how did its function
definition get gc'd?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Stefan Monnier]

[ Richard, please send all email related to this issue to this
  15688@debbugs.gnu.org address, otherwise the thread is nightmarish
  to follow.  ]

>> If that symbol is interned, why would its function definition get
>> GC'd at all?
>     It's not interned.  It's "make-symbol"d in set-temporary-overlay-map.
> Ok, but apparently the symbol does get marked -- so how did its function
> definition get gc'd?

I have no idea.
From the value you show it seems that the function cell's value is
not Vdead.  I tend to assume that it's always the case, but just to make
double sure: Does your build set GC_MARK_STACK?


        Stefan

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

      I tend to assume that it's always the case, but just to make
    double sure: Does your build set GC_MARK_STACK?

How can I check that?  Where would it be set?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Eli Zaretskii]

> Date: Thu, 28 Nov 2013 14:05:11 -0500
> From: Richard Stallman <rms@gnu.org>
> Cc: 15688@debbugs.gnu.org
> 
>       I tend to assume that it's always the case, but just to make
>     double sure: Does your build set GC_MARK_STACK?
> 
> How can I check that?  Where would it be set?

It's a macro set in src/lisp.h.  You can check it with #warning or
some such.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Stefan Monnier]

> I found that the crashes in GC result from trying to mark
> clear-temporary-overlay-map.  Its function cell is
> a vectorlike which pr reports as #<invalid-lisp-object ADDR>.

I'm looking again at this bug but can't find a backtrace.
Could you show us the backtrace?


        Stefan

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I'm looking again at this bug but can't find a backtrace.
    Could you show us the backtrace?

It was mark_object called from mark_vectorlike called from mark_object
called from mark_object (marking that symbol).  I could not see why it
was marking that symbol.  ISTR it was called from mark_memory
and I didn't know how to see where the pointer came from.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Dmitry Antipov]

On 01/24/2014 03:09 AM, Richard Stallman wrote:

> It was mark_object called from mark_vectorlike called from mark_object
> called from mark_object (marking that symbol).  I could not see why it
> was marking that symbol.  ISTR it was called from mark_memory
> and I didn't know how to see where the pointer came from.

I suppose that this is a kind of heisenbug and so it may be hard to
reproduce, but what about the recipe?

And, if there is a binary and core, I can build cross-gdb and look inside.

Dmitry

bug#15688: 24.3.50; clear-temporary-overlay-map

[Stefan Monnier]

> It was mark_object called from mark_vectorlike called from mark_object
> called from mark_object (marking that symbol).  I could not see why it
> was marking that symbol.  ISTR it was called from mark_memory
> and I didn't know how to see where the pointer came from.

Please tell us exactly what value is used for GC_MARK_STACK; as Eli said:

   It's a macro set in src/lisp.h.  You can check it with #warning or
   some such

This said, since you have mark_memory in your backtrace, that indicates
that GC_MARK_STACK is probably set to GC_MAKE_GCPROS_NOOPS, as usual.

Maybe complete backtrace would help, tho.
Also, it'd be useful to know what were the last few keys (the "lossage").


        Stefan

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I suppose that this is a kind of heisenbug and so it may be hard to
    reproduce, but what about the recipe?

I don't have one, sorry.

    And, if there is a binary and core, I can build cross-gdb and look inside.

Is there a command I can use to tell GDB to make a core file?

    Maybe complete backtrace would help, tho.
    Also, it'd be useful to know what were the last few keys (the "lossage").

I will try to remember to send this next time.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Please tell us exactly what value is used for GC_MARK_STACK; as Eli said:

       It's a macro set in src/lisp.h.  You can check it with #warning or
       some such

    This said, since you have mark_memory in your backtrace, that indicates
    that GC_MARK_STACK is probably set to GC_MAKE_GCPROS_NOOPS, as usual.

I don't see a way to examine the value, but the only definition I see
in lisp.h is

#ifndef GC_MARK_STACK
#define GC_MARK_STACK GC_MAKE_GCPROS_NOOPS
#endif

I suggest adding

enum gc_mark_stack { gc_mark_stack_value = GC_MARK_STACK; }; in lisp.h
or some source file.  Then it will be possible to view
gc_mark_stack_value in GDB.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Eli Zaretskii]

> Date: Fri, 24 Jan 2014 09:55:39 -0500
> From: Richard Stallman <rms@gnu.org>
> Cc: 15688@debbugs.gnu.org
> 
> Is there a command I can use to tell GDB to make a core file?

Yes, "gcore".

bug#15688: 24.3.50; clear-temporary-overlay-map

[Eli Zaretskii]

> Date: Fri, 24 Jan 2014 09:55:41 -0500
> From: Richard Stallman <rms@gnu.org>
> Cc: 15688@debbugs.gnu.org
> 
> I suggest adding
> 
> enum gc_mark_stack { gc_mark_stack_value = GC_MARK_STACK; }; in lisp.h
> or some source file.  Then it will be possible to view
> gc_mark_stack_value in GDB.

I think GDB can display values of macros, just use it in the "print"
command.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> I think GDB can display values of macros, just use it in the "print"
> command.

You need to compile with -g3 to make the macros available to the
debugger.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#15688: 24.3.50; clear-temporary-overlay-map

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: rms@gnu.org,  dmantipov@yandex.ru,  15688@debbugs.gnu.org
> Date: Fri, 24 Jan 2014 17:10:50 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think GDB can display values of macros, just use it in the "print"
> > command.
> 
> You need to compile with -g3 to make the macros available to the
> debugger.

Right, thanks for pointing that out.  Though I think that configure
already sets that up, when GCC is used.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Yes, "gcore".

Does that alter the running process?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I think GDB can display values of macros, just use it in the "print"
    command.

I tried that.  I think my version of GDB is somewhat old.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Eli Zaretskii]

> Date: Sat, 25 Jan 2014 08:26:06 -0500
> From: Richard Stallman <rms@gnu.org>
> CC: dmantipov@yandex.ru, 15688@debbugs.gnu.org
> 
>     Yes, "gcore".
> 
> Does that alter the running process?

No.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Eli Zaretskii]

> Date: Sat, 25 Jan 2014 08:26:10 -0500
> From: Richard Stallman <rms@gnu.org>
> CC: dmantipov@yandex.ru, 15688@debbugs.gnu.org
> 
>     I think GDB can display values of macros, just use it in the "print"
>     command.
> 
> I tried that.  I think my version of GDB is somewhat old.

What version is that?  GDB supports macros since v5.3.

Does the "macro expand" command work?

Also, what is your version of GCC?

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    What version is that?  GDB supports macros since v5.3.

Mine is newer than that, so I guess its that I didn't use -g3.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Eli Zaretskii]

> Date: Sat, 25 Jan 2014 18:01:15 -0500
> From: Richard Stallman <rms@gnu.org>
> CC: dmantipov@yandex.ru, 15688@debbugs.gnu.org
> 
> Mine is newer than that, so I guess its that I didn't use -g3.

That would be strange, since configure arranges for that when GCC is
used as the compiler.  It is more probable that your GCC is old (and
has bugs in this area), which is why I asked about its version.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> That would be strange, since configure arranges for that when GCC is
> used as the compiler.

Unless you have set CFLAGS yourself.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#15688: 24.3.50; clear-temporary-overlay-map

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: rms@gnu.org,  dmantipov@yandex.ru,  15688@debbugs.gnu.org
> Date: Sun, 26 Jan 2014 10:11:45 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That would be strange, since configure arranges for that when GCC is
> > used as the compiler.
> 
> Unless you have set CFLAGS yourself.

Right.  Richard, did you?

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I have GCC 4.4.5.  Anyway, I probably specified CFLAGS="-O0 -g"
explicitly in configuration, which I've done for many years.  If it is now
unnecessary to add -g, I didn't know.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Eli Zaretskii]

> Date: Sun, 26 Jan 2014 15:05:54 -0500
> From: Richard Stallman <rms@gnu.org>
> CC: dmantipov@yandex.ru, 15688@debbugs.gnu.org
> 
> I have GCC 4.4.5.  Anyway, I probably specified CFLAGS="-O0 -g"
> explicitly in configuration, which I've done for many years.  If it is now
> unnecessary to add -g, I didn't know.

I suggest to switch to CFLAGS="-O0 -g3", then the macro information
will be recorded in debug info.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I got more information about this bug.  The crash occurs when
mark_stack calls mark_memory.  pp is pointing at a stack slot
in Fgarbage_collect.  As far as I can tell by looking at the code
of Fgarbage_collect with x/i, that slot is never actually used -- it
contains junk left over from previous function calls.

I am surprised this unoptimized compilation makes a stack slot that is
not actually used.  Maybe it is used in some unobvious way.  It is
16(sp).

This suggests that the problem cannot be fixed by trying to prevent a
pointer to clear-temporary-map from being found and marked.

I wrote a change in the Lisp code to fix this problem,
then ran without it to get more info about the problem.
I think I should install that fix soon, but I can hold
off longer if you want.  Please respond.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It crashed again.  This time the slot pointed to was again in
Fgarbage_collect, but it was 80(sp).  That's where the variable
message_p (char) is stored.  The value of message_p is 0,
but it happens that the low 8 bits of the address of clear-temporary-map
are zero, so storing the value of message_p into that byte didn't
alter the pointer in the word.

What seems fishy here is that clear-temporary-map still points to
that vectorlike even though the vectorlike has been freed.
Maybe that's where the real bug is.  Why didn't this code

#if GC_MARK_STACK
		symbol_free_list->function = Vdead;
#endif

prevent the problem?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

bug#15688: 24.3.50; clear-temporary-overlay-map

[Stefan Monnier]

> Why didn't this code

> #if GC_MARK_STACK
>          symbol_free_list->function = Vdead;
> #endif

> prevent the problem?

Indeed, that seems to be the source of the problem: if the symbol has
never been found to be dead, it should not point to a dead vector, and
if it has ever been found to be dead then its `function' cell should
hold Vdead.


        Stefan

bug#15688: 24.3.50; clear-temporary-overlay-map

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Would you like to write some code to detect when that vector becomes dead?
I could put a breakpoint there and investigate.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.