Post-mortem debugging and abort

Post-mortem debugging and abort

[David Kastrup]

Hi,

I just wanted to report that the declaration of "abort" in glibc is not
going to be changed to be more compatible with debugging.

<URL:http://sourceware.org/bugzilla/show_bug.cgi?id=6522> shows my
report and its resolution.

So it remains the Emacs developers' duty to either use the
-fno-crossjumping option when compiling (as specified in etc/DEBUG in
the Emacs distribution) or locally edit /usr/include/stdlib.h to remove
the noreturn attribute from abort.  The latter is, of course, giving
yourself a non-standard development system, but if you want to do any
post-mortem debugging (which usually goes through abort), it should
definitely help for all projects one actually wants to do post-mortem
debugging on for failed assertions and similar.

-- 
David Kastrup

Re: Post-mortem debugging and abort

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Fri, 13 Jun 2008 17:02:20 +0200
> 
> 
> I just wanted to report that the declaration of "abort" in glibc is not
> going to be changed to be more compatible with debugging.
> 
> <URL:http://sourceware.org/bugzilla/show_bug.cgi?id=6522> shows my
> report and its resolution.

Sigh.  I concluded a long time ago that it is useless to ask GCC and
glibc developers to cater to debugging needs.  At best, you are
ignored; but more often you are flamed back into silence.  I'm sorry
that you, David, wasted your effort as well on this.

Re: Post-mortem debugging and abort

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Fri, 13 Jun 2008 17:02:20 +0200
>> 
>> 
>> I just wanted to report that the declaration of "abort" in glibc is not
>> going to be changed to be more compatible with debugging.
>> 
>> <URL:http://sourceware.org/bugzilla/show_bug.cgi?id=6522> shows my
>> report and its resolution.
>
> Sigh.  I concluded a long time ago that it is useless to ask GCC and
> glibc developers to cater to debugging needs.  At best, you are
> ignored; but more often you are flamed back into silence.

I was a bit amused about "stop keeping to reopen".  I reopened once,
giving detailed reasons and citing additional manpage data.  When the
thing was basically closed without looking at the argument after a month
of silence.  So it could not have been all that painful.

> I'm sorry that you, David, wasted your effort as well on this.

Well, the whole difference between exit (or maybe _exit?) and abort is
the core dump, and that is why abort rather than exit is also called on
failed assertions.  So debugging is the whole point of abort and
declaring it in a way that precludes debugging is a bit pointless.  So
that's the reason why I considered it worth a try, even though I know
that GCC/glibc consider debugging a second class citizen.  But as long
as we are talking about what is pretty much a pure debugging facility...

Well, no point preaching to the choir here.  I am pretty sure that not
just Emacs developers will be tripped up by this, but it is here that I
have had quite a bit of anecdotal evidence about people wasting a lot of
time unnecessarily.

-- 
David Kastrup

Re: Post-mortem debugging and abort

[Tom Tromey]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

Eli> Sigh.  I concluded a long time ago that it is useless to ask GCC and
Eli> glibc developers to cater to debugging needs.  At best, you are
Eli> ignored; but more often you are flamed back into silence.  I'm sorry
Eli> that you, David, wasted your effort as well on this.

FWIW on the GCC side there is a renewed awareness of the needs of
debuggers.  There are a couple competing proposals for better
debuginfo; they are both being presented at the GCC Summit next week.

So, I encourage folks to report GCC bugs in this area.  I can't
guarantee that nobody will be rude -- though this seems to happen less
often nowadays.

Tom

Re: Post-mortem debugging and abort

[Lennart Borgman (gmail)]

David Kastrup wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> From: David Kastrup <dak@gnu.org>
>>> Date: Fri, 13 Jun 2008 17:02:20 +0200
>>>
>>>
>>> I just wanted to report that the declaration of "abort" in glibc is not
>>> going to be changed to be more compatible with debugging.
>>>
>>> <URL:http://sourceware.org/bugzilla/show_bug.cgi?id=6522> shows my
>>> report and its resolution.
>> Sigh.  I concluded a long time ago that it is useless to ask GCC and
>> glibc developers to cater to debugging needs.  At best, you are
>> ignored; but more often you are flamed back into silence.
> 
> I was a bit amused about "stop keeping to reopen".  I reopened once,
> giving detailed reasons and citing additional manpage data.  When the
> thing was basically closed without looking at the argument after a month
> of silence.  So it could not have been all that painful.
> 
>> I'm sorry that you, David, wasted your effort as well on this.

The discussion of the bug looked like a very difficult thing ;-)

I do not understand the details, but the only thing that looked like a 
response at all was that UD said there was a workaround, "just add gcc 
flag". Just from the non-friendly tone I would expect this to be a half 
lie - half true.

What did he mean and why does it not work as you would like it to?

Re: Post-mortem debugging and abort

[David Kastrup]

"Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:

> David Kastrup wrote:
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: David Kastrup <dak@gnu.org>
>>>> Date: Fri, 13 Jun 2008 17:02:20 +0200
>>>>
>>>>
>>>> I just wanted to report that the declaration of "abort" in glibc is not
>>>> going to be changed to be more compatible with debugging.
>>>>
>>>> <URL:http://sourceware.org/bugzilla/show_bug.cgi?id=6522> shows my
>>>> report and its resolution.
>>> Sigh.  I concluded a long time ago that it is useless to ask GCC and
>>> glibc developers to cater to debugging needs.  At best, you are
>>> ignored; but more often you are flamed back into silence.
>>
>> I was a bit amused about "stop keeping to reopen".  I reopened once,
>> giving detailed reasons and citing additional manpage data.  When the
>> thing was basically closed without looking at the argument after a month
>> of silence.  So it could not have been all that painful.
>>
>>> I'm sorry that you, David, wasted your effort as well on this.
>
> The discussion of the bug looked like a very difficult thing ;-)
>
> I do not understand the details, but the only thing that looked like a
> response at all was that UD said there was a workaround, "just add gcc
> flag". Just from the non-friendly tone I would expect this to be a
> half lie - half true.
>
> What did he mean and why does it not work as you would like it to?

He means that if you want to have useful post-mortem debugging (namely,
seeing where an assertion went wrong or abort() had been called), you
need to add -fno-crossjumping to the compiler flags.  I found this out
the hard way, by searching for days at the wrong place in Emacs sources.
Naturally, there is no information whatsoever in the gcc or glibc docs
about the importance of this flag for getting usable tracebacks.  There
have been at least two occasions where other Emacs developers have also
spent days of debugging in the wrong place because of this.  I wrote
down in the file etc/DEBUG in Emacs that one should use this flag when
trying to track down the cause of failed assertions.

But of course, people tend not to read such things before being written,
and of course, failed assertions and calls of abort are not exactly
specific to Emacs.

When you don't use the flag, there will tend to be just a single call to
abort compiled into the code and everybody jumps to that call without
bothering to clean up the stack or bothering what happens after the
call.  So at the time the debugger gets to see the stack frame, the
information which abort call in the source code is responsible is lost,
and the stack frame and data on it can only be usefully interpreted when
this information is not lost.

Basically, this will bite you whenever you debug failed assertions and
have not used -O0 for compiling.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Post-mortem debugging and abort

[David Kastrup]

Tom Tromey <tromey@redhat.com> writes:

>>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:
>
> Eli> Sigh.  I concluded a long time ago that it is useless to ask GCC and
> Eli> glibc developers to cater to debugging needs.  At best, you are
> Eli> ignored; but more often you are flamed back into silence.  I'm sorry
> Eli> that you, David, wasted your effort as well on this.
>
> FWIW on the GCC side there is a renewed awareness of the needs of
> debuggers.  There are a couple competing proposals for better
> debuginfo; they are both being presented at the GCC Summit next week.
>
> So, I encourage folks to report GCC bugs in this area.  I can't
> guarantee that nobody will be rude -- though this seems to happen less
> often nowadays.

No static debuginfo can help reconstructing a _jump_ history when
several places in the code jump to the same abort call.  Only _call_
histories can be reconstructed with good debug info.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Post-mortem debugging and abort

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 13 Jun 2008 18:48:34 +0200
> 
> Well, the whole difference between exit (or maybe _exit?) and abort is
> the core dump, and that is why abort rather than exit is also called on
> failed assertions.  So debugging is the whole point of abort and
> declaring it in a way that precludes debugging is a bit pointless.

And even if they disagree about this, GCC and glibc could have behaved
differently under -g.

Re: Post-mortem debugging and abort

[Eli Zaretskii]

> Cc: David Kastrup <dak@gnu.org>, emacs-devel@gnu.org
> From: Tom Tromey <tromey@redhat.com>
> Date: Fri, 13 Jun 2008 11:04:11 -0600
> 
> FWIW on the GCC side there is a renewed awareness of the needs of
> debuggers.  There are a couple competing proposals for better
> debuginfo; they are both being presented at the GCC Summit next week.

Thanks for the heads-up.

Where can I see the details of these proposals and the relevant
discussions on the GCC-related mailing lists?

Re: Post-mortem debugging and abort

[Tom Tromey]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

Eli> Where can I see the details of these proposals and the relevant
Eli> discussions on the GCC-related mailing lists?

http://gcc.gnu.org/svn.html#devbranches

You want the var-tracking-assignments branch and the var-mappings-branch.
There are discussions of these in the archives of the gcc and
gcc-patches lists.

Tom

Re: Post-mortem debugging and abort

[Richard M Stallman]

      So debugging is the whole point of abort and
    declaring it in a way that precludes debugging is a bit pointless.

I agree with you.  I will take a look at the URL you sent.

Re: Post-mortem debugging and abort

[Richard M Stallman]

Perhaps the GCC manual should say more about the importance of
-fno-cross-jump for debugging.  If you write a patch to it,
then if I like it, I will send it to the GCC maintainers.

Re: Post-mortem debugging and abort

[Lennart Borgman (gmail)]

Richard M Stallman wrote:
> Perhaps the GCC manual should say more about the importance of
> -fno-cross-jump for debugging.  If you write a patch to it,
> then if I like it, I will send it to the GCC maintainers.

Wouldn't it also be good to add some of the thoughts David had to 
etc/DEBUG? That could save time for other people and also be used as a 
source when trying to discuss this with the GCC maintainers.

Re: Post-mortem debugging and abort

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 13 Jun 2008 18:48:34 +0200
>> 
>> Well, the whole difference between exit (or maybe _exit?) and abort is
>> the core dump, and that is why abort rather than exit is also called on
>> failed assertions.  So debugging is the whole point of abort and
>> declaring it in a way that precludes debugging is a bit pointless.
>
> And even if they disagree about this, GCC and glibc could have behaved
> differently under -g.

Actually, I don't agree here: if you are hunting down a compiler bug,
you don't want to have the bug disappear as soon as you ask for
debugging symbols.  Not changing the generated code just because debug
info is output is a reasonable choice in my opinion.

One might want to have an independent code generator option -fdebug
which switches off just those optimizations that are notably awful when
debugging.

Anyway, we can discuss this on this list until we are blue in the face.
Won't change a thing.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Post-mortem debugging and abort

[David Kastrup]

"Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:

> Richard M Stallman wrote:
>> Perhaps the GCC manual should say more about the importance of
>> -fno-cross-jump for debugging.  If you write a patch to it,
>> then if I like it, I will send it to the GCC maintainers.
>
> Wouldn't it also be good to add some of the thoughts David had to
> etc/DEBUG? That could save time for other people and also be used as a
> source when trying to discuss this with the GCC maintainers.

etc/DEBUG contains

    ** When you are trying to analyze failed assertions, it will be
    essential to compile Emacs either completely without optimizations or
    at least (when using GCC) with the -fno-crossjumping option.  Failure
    to do so may make the compiler recycle the same abort call for all
    assertions in a given function, rendering the stack backtrace useless
    for identifying the specific failed assertion.

It is a bit too weak in that the compiler nowadays will recycle the same
abort call for all assertions in a given _compilation unit_ I think.  So
the stack frame might not even belong at all to the function that called
abort just by proxy.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Post-mortem debugging and abort

[Lennart Borgman (gmail)]

David Kastrup wrote:
> "Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:
> 
>> Richard M Stallman wrote:
>>> Perhaps the GCC manual should say more about the importance of
>>> -fno-cross-jump for debugging.  If you write a patch to it,
>>> then if I like it, I will send it to the GCC maintainers.
>> Wouldn't it also be good to add some of the thoughts David had to
>> etc/DEBUG? That could save time for other people and also be used as a
>> source when trying to discuss this with the GCC maintainers.
> 
> etc/DEBUG contains
> 
>     ** When you are trying to analyze failed assertions, it will be
>     essential to compile Emacs either completely without optimizations or
>     at least (when using GCC) with the -fno-crossjumping option.  Failure
>     to do so may make the compiler recycle the same abort call for all
>     assertions in a given function, rendering the stack backtrace useless
>     for identifying the specific failed assertion.


"failure to do so may unfortunately currently make the compiler ..."

Re: Post-mortem debugging and abort

[Richard M Stallman]

    Wouldn't it also be good to add some of the thoughts David had to 
    etc/DEBUG? 

I think it would be useful to do that.