* The emacs_backtrace "feature"
@ 2012-09-21 9:49 Eli Zaretskii
2012-09-21 9:58 ` Andreas Schwab
` (4 more replies)
0 siblings, 5 replies; 43+ messages in thread
From: Eli Zaretskii @ 2012-09-21 9:49 UTC (permalink / raw)
To: emacs-devel
Based on my experience, I expect this "feature" to be hated, by users
and Emacs maintainers alike.
My experience is based on years of working with the DJGPP development
environment. DJGPP (www.delorie.com/djgpp/) is a Posix-compliant
development environment, based on ported GNU tools and an
independently written standard C library, for developing 32-bit
protected-mode programs that run on MS-DOS and compatible systems. In
particular, the MS-DOS build of Emacs uses DJGPP.
In DJGPP, displaying the backtrace on fatal errors is the default,
because core files are not supported. So, when a DJGPP-compiled
program crashes, it displays a register dump and a backtrace. Here's
a typical example (I deliberately truncated the backtrace at the end,
which was much longer in reality):
Exiting due to signal SIGABRT
Raised at eip=0012f2a6
eax=002ee7fc ebx=00000120 ecx=00000000 edx=00000000 esi=003a533d edi=002f4cc0
ebp=002ee8a8 esp=002ee7f8 program=H:\test\emacs-djgpp\emacs\src\temacs.exe
cs: sel=0257 base=02c30000 limit=0104ffff
ds: sel=025f base=02c30000 limit=0104ffff
es: sel=025f base=02c30000 limit=0104ffff
fs: sel=022f base=0001d580 limit=0000ffff
gs: sel=027f base=00000000 limit=0010ffff
ss: sel=025f base=02c30000 limit=0104ffff
App stack: [002eed94..002d5d94] Exceptn stack: [002d5c68..002d3d28]
Call frame traceback EIPs:
0x0012f1c4
0x0012f2a6
0x00118377
0x0011191d
0x00068cff
0x00068c41
A companion utility program captures the addresses and the executable
file name from the screen, and adds the corresponding function name
plus offset to each line (if the executable was not stripped), and
also the source file/line information, if that info is found.
Example:
Call frame traceback EIPs:
0x0001039f execute_builtin+191, file c:/djgpp/gnu/bash-2.03/execute_cmd.c, line 2878
0x00010840 execute_builtin_or_function+176, file c:/djgpp/gnu/bash-2.03/execute_cmd.c, line 3173
0x0001011b execute_simple_command+659, file c:/djgpp/gnu/bash-2.03/execute_cmd.c, line 2745
0x0000de00 execute_command_internal+1876, file c:/djgpp/gnu/bash-2.03/execute_cmd.c, line 824
0x0000d459 execute_command+69, file c:/djgpp/gnu/bash-2.03/execute_cmd.c, line 314
As nice as this looks, it has several disadvantages:
. Many real-life backtraces are long and quickly scroll off the
screen. If you didn't make a point of setting up very large screen
buffers of your shell windows, or redirect standard error to a
file, you'll lose precious information. Since these precautions
are only taken when one expects a crash, guess how many times these
measures are in place when they are needed.
. Many calls to emacs_backtrace in the current sources limit the
number of backtrace frames to 10, but that is an arbitrary
limitation which will be too small in most, if not all, situations.
Check out the crash backtraces posted to the bug tracker. As an
extreme (but quite frequent) data point, crashes in GC tend to have
many hundreds, and sometimes many thousands, of frames in them. In
reality, there's no way of knowing how many frames will be there,
and how many of them will be needed to get enough useful
information for finding the problem. I predict that more often
than not we will be looking at useless backtraces, while users who
reported those backtraces will rightfully expect us to find the bug
and fix it.
. The backtrace is written to the standard error file handle. Is
that handle always guaranteed to be available and connected to a
screen or a disk file that the user can find afterwards? E.g., if
Emacs is invoked from an environment which redirects that handle to
the null device, the information will be lost. (On MS-Windows, GUI
applications launched by clicking a desktop icon have this handle
closed, so anything written to it disappears without a trace; I
don't know if Posix desktops have something similar.)
. Last, but not least, even if the drawbacks described above are not
an issue in some particular crash report, using the limited
information it provides can be quite difficult, especially if the
crash happened in a binary compiled by a different compiler version
than yours, let alone on an architecture different from the one
used by the person who tries to get some sense out of it. Here's
an example of what emacs_backtrace will produce (slightly edited
from what you see on
http://linux.die.net/man/3/backtrace_symbols_fd):
Backtrace:
./emacs(myfunc4+0x5c) [0x80487f0]
./emacs [0x8048871]
./emacs(myfunc3+0x21) [0x8048894]
./emacs(myfunc2+0x1a) [0x804888d]
./emacs(myfunc1+0x1a) [0x804888d]
./emacs(main+0x65) [0x80488fb]
/lib/libc.so.6(__libc_start_main+0xdc) [0xb7e38f9c]
./emacs [0x8048711]
It doesn't even show the source line info, like DJGPP did.
Translating myfunc1+0x1a etc. into source-level info is not an easy
task, unless you are lucky and there's only one place where it
calls myfunc2. If not, you are left with guesswork. Making sense
of the backtrace without being able to get at the corresponding
source lines is not for the faint at heart. More often than not,
the Emacs maintainers will be tempted to ignore such a report, and
ask for a GDB backtrace instead.
So given all of the above, I'm asking why do we want this feature?
Why not use the good old core dump files? They have all the
information that is needed for debugging the crash, while the above
falls short of that mark by a large measure. It seems like a step
backward. I always thought that the lack of core files in DJGPP was a
serious limitations, so I'm amazed to see modern environments actually
_wanting_ that limited debug feature in favor of core dumps and real
debuggability. Until now, the only uses I saw for the 'backtrace'
function were when a debugger couldn't be used at all, or the core
file couldn't be produced due to system-level requirements, such as
limited disk space or some stringent time constraints. But here we do
that voluntarily and by default. Why?
Having said all that, I'm not really interested in disputing these
points. I wanted to communicate my own, mostly negative, experience
of many years using a similar feature. If more information is
required, in particular about DJGPP and how it created and used the
backtraces, I will gladly provide answers to any questions.
Otherwise, I guess we will find soon enough whether this is a great
feature or not.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-21 9:49 The emacs_backtrace "feature" Eli Zaretskii
@ 2012-09-21 9:58 ` Andreas Schwab
2012-09-21 10:17 ` Eli Zaretskii
2012-09-21 19:29 ` Florian Weimer
2012-09-21 12:24 ` Óscar Fuentes
` (3 subsequent siblings)
4 siblings, 2 replies; 43+ messages in thread
From: Andreas Schwab @ 2012-09-21 9:58 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
Eli Zaretskii <eliz@gnu.org> writes:
> As nice as this looks, it has several disadvantages:
>
> . Many real-life backtraces are long and quickly scroll off the
> screen.
That's not a real problem. The output of programs that are started from
the desktop is typically redirected to a file, so nothing gets lost.
> . The backtrace is written to the standard error file handle. Is
> that handle always guaranteed to be available and connected to a
> screen or a disk file that the user can find afterwards?
See above.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-21 9:49 The emacs_backtrace "feature" Eli Zaretskii
2012-09-21 9:58 ` Andreas Schwab
@ 2012-09-21 12:24 ` Óscar Fuentes
2012-09-21 16:35 ` Stefan Monnier
2012-09-21 16:49 ` Stephen J. Turnbull
` (2 subsequent siblings)
4 siblings, 1 reply; 43+ messages in thread
From: Óscar Fuentes @ 2012-09-21 12:24 UTC (permalink / raw)
To: emacs-devel
Eli Zaretskii <eliz@gnu.org> writes:
> Based on my experience, I expect this "feature" to be hated, by users
> and Emacs maintainers alike.
FWIW: I have no problem about attaching a backtrace to a bug report. A
core dump file? Never.
[snip]
^ permalink raw reply [flat|nested] 43+ messages in thread
* The emacs_backtrace "feature"
2012-09-21 9:49 The emacs_backtrace "feature" Eli Zaretskii
2012-09-21 9:58 ` Andreas Schwab
2012-09-21 12:24 ` Óscar Fuentes
@ 2012-09-21 16:49 ` Stephen J. Turnbull
2012-09-21 17:07 ` Tom Tromey
2012-09-21 17:41 ` Andreas Schwab
2012-09-21 19:37 ` Paul Eggert
2012-09-21 21:40 ` Richard Stallman
4 siblings, 2 replies; 43+ messages in thread
From: Stephen J. Turnbull @ 2012-09-21 16:49 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
Eli Zaretskii writes:
> If you didn't make a point of setting up [for a crash], you'll
> lose precious information.
[...]
> Why not use the good old core dump files? They have all the
> information that is needed for debugging the crash,
Assuming that they exist. Sadly, on modern GNU systems (as
distributed by the major distros) they usually don't ... unless the
user makes a point of setting up for a crash. As you say, people
usually don't.
> But here we do that voluntarily and by default. Why?
Because modern GNU systems (as distributed by the major distros)
usually *do* come with installable "debug" packages that contain the
symbol table that gets stripped out by default. I agree, trying to
find the Corresponding Source for "find_file_internal + 0xDEAD" is
pretty darn annoying, but I've debugged a lot of issues with just the
most recently called function name.
What bugs me a lot more is inability to get at variable values,
especially actual parameters. But thanks to the wonders of modern
GCC, the value of whatever you're interested in is <variable optimized
out>, so you also get that feature for free with <drum roll>modern GNU
systems as distributed by the major distros. (I hear that really
recent GDB does a much better job of disentangling most optimizations,
but guess what isn't installed on <drum roll>....)
> Otherwise, I guess we will find soon enough whether this is a great
> feature or not.
I agree that core dumps are better (but habitually running under gdb
is even better yet :-). But it's nice to get at least some
information out of random users who for some strange reason don't
expect Emacs to crash. :-)
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-21 16:49 ` Stephen J. Turnbull
@ 2012-09-21 17:07 ` Tom Tromey
2012-09-21 17:41 ` Andreas Schwab
1 sibling, 0 replies; 43+ messages in thread
From: Tom Tromey @ 2012-09-21 17:07 UTC (permalink / raw)
To: Stephen J. Turnbull; +Cc: Eli Zaretskii, emacs-devel
Stephen> (I hear that really
Stephen> recent GDB does a much better job of disentangling most optimizations,
Stephen> but guess what isn't installed on <drum roll>....)
gdb is really at the mercy of the compiler here.
It doesn't do any disentangling of its own.
What has changed is that GCC is now somewhat better at generating
debuginfo when optimizing; read about the "VTA" project for details.
Tom
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-21 16:49 ` Stephen J. Turnbull
2012-09-21 17:07 ` Tom Tromey
@ 2012-09-21 17:41 ` Andreas Schwab
1 sibling, 0 replies; 43+ messages in thread
From: Andreas Schwab @ 2012-09-21 17:41 UTC (permalink / raw)
To: Stephen J. Turnbull; +Cc: Eli Zaretskii, emacs-devel
"Stephen J. Turnbull" <stephen@xemacs.org> writes:
> What bugs me a lot more is inability to get at variable values,
> especially actual parameters.
That has always been the case, but nowadays the compilers are much
better at telling you about that fact. (What does it help if the
debugger lies to you?)
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-21 9:49 The emacs_backtrace "feature" Eli Zaretskii
` (2 preceding siblings ...)
2012-09-21 16:49 ` Stephen J. Turnbull
@ 2012-09-21 19:37 ` Paul Eggert
2012-09-22 12:05 ` Richard Stallman
` (2 more replies)
2012-09-21 21:40 ` Richard Stallman
4 siblings, 3 replies; 43+ messages in thread
From: Paul Eggert @ 2012-09-21 19:37 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
I agree with a lot of points that Eli raised: backtraces can be a
pain. Still, in my experience these backtraces, limited as they
are, are typically an improvement over not having backtraces.
Going through some of Eli's comments:
> . Many real-life backtraces are long and quickly scroll off the
> screen.
That's less of a problem these days. On GNU and POSIXish
platforms most people run Emacs in an environment where backtraces
are sent to a file, or to a window or a console that has history
that is typically long enough. It's not perfect, but it's
typically better than not having a backtrace.
> . Many calls to emacs_backtrace in the current sources limit the
> number of backtrace frames to 10, but that is an arbitrary
> limitation which will be too small in most, if not all, situations.
We can change that number if needed. 10 was enough for my use
cases. Of course no number will be ideal in all cases. Still, 10
is almost invariably better than 0.
> . The backtrace is written to the standard error file handle. Is
> that handle always guaranteed to be available
No. But typically it is, and overall we're better off.
> . using the limited information it provides can be quite
> difficult, especially if the crash happened in a binary
> compiled by a different compiler version than yours
I'd say it's more than just "difficult". It's typically impossible.
A backtrace makes sense only if you have access to the environment
that generated the crash, or a pretty-close clone of that
environment. In this respect it's no better than a core dump.
Core dumps are better than backtraces if they're available, but
modern GNUish distributions often disable them, alas, and they're
a pain to send via email, and it's nice to have a bit more info
than 'Fatal error 27' when Emacs crashes.
The main point of these backtraces is to help diagnose Emacs
problems after a crash in which a core dump is not available and
where the bug cannot easily be reproduced. This situation is
becoming increasinly common, and for this situation a backtrace,
even with its limitations, is invaluable. Anyone with access to
the system where the crash occurred can use GDB or addr2line to
get the source line numbers etc. for the crash. This is info
that's available in no other way.
Come to think of it, the addr2line stuff should be documented. I
gave it a shot and installed the following as trunk bzr 110125.
* trouble.texi (Crashing): Document addr2line.
=== modified file 'doc/emacs/trouble.texi'
--- doc/emacs/trouble.texi 2012-09-04 18:29:04 +0000
+++ doc/emacs/trouble.texi 2012-09-21 19:27:37 +0000
@@ -308,13 +308,26 @@
@noindent
The number @samp{11} is the system signal number that corresponds to
-the problem, a segmentation fault here. The hexadecimal program
-addresses can be useful in debugging sessions. For example, the GDB
-command @samp{list *0x509af6} prints the source-code lines
-corresponding to the @samp{emacs[0x509af6]} entry in the backtrace.
-
-The three dots at the end indicate that Emacs suppressed further
-backtrace entries, in the interest of brevity.
+the problem, a segmentation fault here. The three dots at the end
+indicate that Emacs suppressed further backtrace entries, in the
+interest of brevity.
+
+The hexadecimal program addresses can be useful in debugging sessions.
+For example, the GDB command @samp{list *0x509af6} prints the
+source-code lines corresponding to the @samp{emacs[0x509af6]} entry in
+the backtrace. Or, if your system has @command{addr2line}, the
+following shell command outputs a backtrace with source-code line
+numbers:
+
+@example
+sed -n 's/.*\[\(.*\)]$/\1/p' @var{backtrace} |
+ addr2line -Cfip -e @var{bindir}/emacs
+@end example
+
+@noindent
+Here, @var{backtrace} is the name of a text file containing a copy of
+the backtrace, and @var{bindir} is the name of the directory that
+contains the Emacs executable.
@node After a Crash
@subsection Recovery After a Crash
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-21 19:37 ` Paul Eggert
@ 2012-09-22 12:05 ` Richard Stallman
2012-09-22 13:54 ` Christoph Scholtes
2012-09-24 15:55 ` Steinar Bang
2012-09-22 12:41 ` Eli Zaretskii
2012-09-29 19:31 ` Juri Linkov
2 siblings, 2 replies; 43+ messages in thread
From: Richard Stallman @ 2012-09-22 12:05 UTC (permalink / raw)
To: Paul Eggert; +Cc: eliz, emacs-devel
Core dumps are better than backtraces if they're available, but
modern GNUish distributions often disable them, alas, and they're
a pain to send via email, and it's nice to have a bit more info
than 'Fatal error 27' when Emacs crashes.
If you get a core dump, you can run GDB and make a backtrace
and mail that.
--
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
Use Ekiga or an ordinary phone call
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-22 12:05 ` Richard Stallman
@ 2012-09-22 13:54 ` Christoph Scholtes
2012-09-22 14:23 ` covici
2012-09-24 15:55 ` Steinar Bang
1 sibling, 1 reply; 43+ messages in thread
From: Christoph Scholtes @ 2012-09-22 13:54 UTC (permalink / raw)
To: rms; +Cc: eliz, Paul Eggert, emacs-devel
On 9/22/2012 6:05 AM, Richard Stallman wrote:
> Core dumps are better than backtraces if they're available, but
> modern GNUish distributions often disable them, alas, and they're
> a pain to send via email, and it's nice to have a bit more info
> than 'Fatal error 27' when Emacs crashes.
>
> If you get a core dump, you can run GDB and make a backtrace
> and mail that.
If you get a core dump, you _have_ to run GDB and make a backtrace. Some
users might not know how nor care to know how to generate a backtrace
from a core dump. I believe this is independent of the platform they are
running on.
I think an automated way of generating backtraces is a good idea. The
more we can automate the process of gathering information in case of a
crash, the more likely users are to submit that information.
What do you think about the following:
Check if gdb is available (on startup, or when the crash happens).
If it is, run something like 'gdb -batch-silent -ex 'thread apply all bt
full'' and save backtrace to a file that can be attached to bug report.
If it isn't, fall back to the backtrace() function and generate at least
something.
Or we ask the user what she wants to do:
1. Create backtrace (with or without gdb) and attach to bug report
2. Attach debugger
3. Nothing
This could be an option, too, like debug-on-error:
create-backtrace-on-error or something like that.
Christoph
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-22 13:54 ` Christoph Scholtes
@ 2012-09-22 14:23 ` covici
0 siblings, 0 replies; 43+ messages in thread
From: covici @ 2012-09-22 14:23 UTC (permalink / raw)
To: Christoph Scholtes; +Cc: eliz, Paul Eggert, rms, emacs-devel
Christoph Scholtes <cschol2112@gmail.com> wrote:
> On 9/22/2012 6:05 AM, Richard Stallman wrote:
> > Core dumps are better than backtraces if they're available, but
> > modern GNUish distributions often disable them, alas, and they're
> > a pain to send via email, and it's nice to have a bit more info
> > than 'Fatal error 27' when Emacs crashes.
> >
> > If you get a core dump, you can run GDB and make a backtrace
> > and mail that.
>
> If you get a core dump, you _have_ to run GDB and make a
> backtrace. Some users might not know how nor care to know how to
> generate a backtrace from a core dump. I believe this is independent
> of the platform they are running on.
>
> I think an automated way of generating backtraces is a good idea. The
> more we can automate the process of gathering information in case of a
> crash, the more likely users are to submit that information.
>
> What do you think about the following:
>
> Check if gdb is available (on startup, or when the crash happens).
> If it is, run something like 'gdb -batch-silent -ex 'thread apply all
> bt full'' and save backtrace to a file that can be attached to bug
> report.
> If it isn't, fall back to the backtrace() function and generate at
> least something.
>
> Or we ask the user what she wants to do:
>
> 1. Create backtrace (with or without gdb) and attach to bug report
> 2. Attach debugger
> 3. Nothing
>
> This could be an option, too, like debug-on-error:
> create-backtrace-on-error or something like that.
The problem is that crashes happen in a way which precludes automation
of the backtrace. The only thing is to give the user instructions as to
what to do with the core dump, even as to how to make sure he can get a
core dump at all.
--
Your life is like a penny. You're going to lose it. The question is:
How do
you spend it?
John Covici
covici@ccs.covici.com
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-22 12:05 ` Richard Stallman
2012-09-22 13:54 ` Christoph Scholtes
@ 2012-09-24 15:55 ` Steinar Bang
2012-09-24 17:29 ` Paul Eggert
1 sibling, 1 reply; 43+ messages in thread
From: Steinar Bang @ 2012-09-24 15:55 UTC (permalink / raw)
To: emacs-devel
>>>>> Richard Stallman <rms@gnu.org>:
>> Core dumps are better than backtraces if they're available, but
>> modern GNUish distributions often disable them, alas, and they're a
>> pain to send via email, and it's nice to have a bit more info than
>> 'Fatal error 27' when Emacs crashes.
> If you get a core dump, you can run GDB and make a backtrace
> and mail that.
Can you get a backtrace even if the core is from an executable that was
compiled without debug information?
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-24 15:55 ` Steinar Bang
@ 2012-09-24 17:29 ` Paul Eggert
0 siblings, 0 replies; 43+ messages in thread
From: Paul Eggert @ 2012-09-24 17:29 UTC (permalink / raw)
To: emacs-devel; +Cc: Lars Ingebrigtsen, Steinar Bang
On 09/24/2012 08:55 AM, Steinar Bang wrote:
> Can you get a backtrace even if the core is from an executable that was
> compiled without debug information?
Yes. Debug info doesn't affect the backtrace.
It's more useful, though, to get a backtrace from an executable
built with debug information, just as it's more useful to get
a core dump that way. One *can* analyze either one without the
debug information, but it's no picnic.
On 09/24/2012 07:27 AM, Lars Ingebrigtsen wrote:
> "let's just spew out reams of nearly meaningless data to the terminal"
It's normally just ten lines, maybe 300 bytes total. It's more than
that (500 lines) only if you build emacs with --enable-checking and then
an assertion fails, under the assumption that people who are debugging
prefer to have more debug info. In practice these numbers worked OK
for my use cases; we can change them as further experience suggests.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-21 19:37 ` Paul Eggert
2012-09-22 12:05 ` Richard Stallman
@ 2012-09-22 12:41 ` Eli Zaretskii
2012-09-22 19:28 ` Paul Eggert
2012-09-23 13:05 ` Stephen J. Turnbull
2012-09-29 19:31 ` Juri Linkov
2 siblings, 2 replies; 43+ messages in thread
From: Eli Zaretskii @ 2012-09-22 12:41 UTC (permalink / raw)
To: Paul Eggert; +Cc: emacs-devel
> Date: Fri, 21 Sep 2012 12:37:30 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: emacs-devel@gnu.org
>
> Come to think of it, the addr2line stuff should be documented. I
> gave it a shot and installed the following as trunk bzr 110125.
It strikes me that we should tell the users how to enable core files,
in the same section of the manual.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-22 12:41 ` Eli Zaretskii
@ 2012-09-22 19:28 ` Paul Eggert
2012-09-23 13:05 ` Stephen J. Turnbull
1 sibling, 0 replies; 43+ messages in thread
From: Paul Eggert @ 2012-09-22 19:28 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
On 09/22/2012 05:41 AM, Eli Zaretskii wrote:
> It strikes me that we should tell the users how to enable core files,
> in the same section of the manual.
Good suggestion, thanks, I did that in trunk bzr 110147.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-22 12:41 ` Eli Zaretskii
2012-09-22 19:28 ` Paul Eggert
@ 2012-09-23 13:05 ` Stephen J. Turnbull
2012-09-23 14:19 ` Andreas Schwab
1 sibling, 1 reply; 43+ messages in thread
From: Stephen J. Turnbull @ 2012-09-23 13:05 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Paul Eggert, emacs-devel
Eli Zaretskii writes:
> It strikes me that we should tell the users how to enable core files,
> in the same section of the manual.
It strikes me that you could add it to the crash report, since you're
intercepting fatal errors anyway.
Something like:
On Unix-like systems, you can enable core files with "ulimit -c unlimited."
This is useful because the first useful thing the user is going to do
(assuming emacs isn't totally bollixed) is to fire up another emacs to
make the bug report. May as well have core files enabled for that
instance, too. :-)
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-23 13:05 ` Stephen J. Turnbull
@ 2012-09-23 14:19 ` Andreas Schwab
2012-09-23 19:30 ` Stephen J. Turnbull
0 siblings, 1 reply; 43+ messages in thread
From: Andreas Schwab @ 2012-09-23 14:19 UTC (permalink / raw)
To: Stephen J. Turnbull; +Cc: Eli Zaretskii, Paul Eggert, emacs-devel
"Stephen J. Turnbull" <stephen@xemacs.org> writes:
> Eli Zaretskii writes:
>
> > It strikes me that we should tell the users how to enable core files,
> > in the same section of the manual.
>
> It strikes me that you could add it to the crash report, since you're
> intercepting fatal errors anyway.
>
> Something like:
>
> On Unix-like systems, you can enable core files with "ulimit -c unlimited."
If it's so important to get core dumps then Emacs could just enable them
itself.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-23 14:19 ` Andreas Schwab
@ 2012-09-23 19:30 ` Stephen J. Turnbull
2012-09-23 21:01 ` Nix
0 siblings, 1 reply; 43+ messages in thread
From: Stephen J. Turnbull @ 2012-09-23 19:30 UTC (permalink / raw)
To: Andreas Schwab; +Cc: Eli Zaretskii, Paul Eggert, emacs-devel
Andreas Schwab writes:
> If it's so important to get core dumps then Emacs could just enable them
> itself.
And distros will likely just turn them right off again.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-23 19:30 ` Stephen J. Turnbull
@ 2012-09-23 21:01 ` Nix
2012-09-24 5:00 ` Stephen J. Turnbull
0 siblings, 1 reply; 43+ messages in thread
From: Nix @ 2012-09-23 21:01 UTC (permalink / raw)
To: Stephen J. Turnbull
Cc: Eli Zaretskii, Paul Eggert, Andreas Schwab, emacs-devel
On 23 Sep 2012, Stephen J. Turnbull verbalised:
> Andreas Schwab writes:
>
> > If it's so important to get core dumps then Emacs could just enable them
> > itself.
>
> And distros will likely just turn them right off again.
Those that aren't redirecting them through a piped program (which is
most of them) -- in which case the coredump ulimit has no effect at all,
except if consulted by that program.
--
NULL && (void)
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-23 21:01 ` Nix
@ 2012-09-24 5:00 ` Stephen J. Turnbull
2012-09-24 9:29 ` Nix
0 siblings, 1 reply; 43+ messages in thread
From: Stephen J. Turnbull @ 2012-09-24 5:00 UTC (permalink / raw)
To: Nix; +Cc: Eli Zaretskii, Paul Eggert, Andreas Schwab, emacs-devel
Nix writes:
> Those [distros] that aren't redirecting [core dumps] through a
> piped program (which is most of them)
By "most of them", I take it you mean "Red Hat derivatives" or
something like that, as neither my Debian system nor my Gentoo system
does this (unless /proc/sys/kernel/core_pattern lies to unprivileged
processes: it's "core" in both of them according to cat(1)). Elderly
BSD-ish systems (at least, I assume Darwin is relatively elderly, the
man page is dated 2008) don't support piping or even renaming core
files (according to the man page).
I stand by my original suggestion: documenting the ulimit command
takes up only one line, and may be useful to many users if their crash
is easily reproducible -- even if they don't know how to do that yet,
they're likely to accidentally trigger it again in the process of
attempting to complete the work interrupted by the first crash.
Steve
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-24 5:00 ` Stephen J. Turnbull
@ 2012-09-24 9:29 ` Nix
2012-09-24 23:55 ` Stephen J. Turnbull
0 siblings, 1 reply; 43+ messages in thread
From: Nix @ 2012-09-24 9:29 UTC (permalink / raw)
To: Stephen J. Turnbull
Cc: Eli Zaretskii, Paul Eggert, Andreas Schwab, emacs-devel
On 24 Sep 2012, Stephen J. Turnbull verbalised:
> Nix writes:
>
> > Those [distros] that aren't redirecting [core dumps] through a
> > piped program (which is most of them)
>
> By "most of them", I take it you mean "Red Hat derivatives" or
> something like that, as neither my Debian system nor my Gentoo system
> does this
Ubuntu does, RH and all derivatives do. That covers a huge proportion of
distros right there. I suspect that all distros will eventually start
doing it because it means they can convert core files into compressed
coredumps, saving lots of room (which is not the same as gzipping them).
It also means you don't end up with corefiles scattered across the disk
because you can put them in one central place if you want to, though
that's not why RH or Ubuntu do it.
> I stand by my original suggestion: documenting the ulimit command
> takes up only one line, and may be useful to many users if their crash
> is easily reproducible
Oh, I agree. It just might not do anything. :)
--
NULL && (void)
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-21 19:37 ` Paul Eggert
2012-09-22 12:05 ` Richard Stallman
2012-09-22 12:41 ` Eli Zaretskii
@ 2012-09-29 19:31 ` Juri Linkov
2012-09-29 20:01 ` Eli Zaretskii
2 siblings, 1 reply; 43+ messages in thread
From: Juri Linkov @ 2012-09-29 19:31 UTC (permalink / raw)
To: Paul Eggert; +Cc: Eli Zaretskii, emacs-devel
>> . Many calls to emacs_backtrace in the current sources limit the
>> number of backtrace frames to 10, but that is an arbitrary
>> limitation which will be too small in most, if not all, situations.
>
> We can change that number if needed. 10 was enough for my use
> cases. Of course no number will be ideal in all cases. Still, 10
> is almost invariably better than 0.
10 is too small number. I tried to visit an .ico file, and Emacs crashed
in the ImageMagick library. I found the backtrace in ~/.xsession-errors:
Backtrace:
/usr/local/src/emacs/trunk/src/emacs[0x4f1bdb]
/usr/local/src/emacs/trunk/src/emacs[0x4d70dd]
/usr/local/src/emacs/trunk/src/emacs[0x4f1b4e]
/usr/local/src/emacs/trunk/src/emacs[0x4f28c3]
/lib/libpthread.so.0(+0xfb40)[0x7fa162357b40]
/lib/libc.so.6(gsignal+0x35)[0x7fa16136eba5]
/lib/libc.so.6(abort+0x180)[0x7fa1613726b0]
/lib/libc.so.6(__assert_fail+0xf1)[0x7fa161367a71]
/usr/lib/libMagickCore.so.3(DestroyLinkedList+0xea)[0x7fa1628939ba]
/usr/lib/libMagickCore.so.3(DestroyExceptionInfo+0x52)[0x7fa16287bdc2]
/usr/lib/libMagickWand.so.3(DestroyPixelWand+0x5b)[0x7fa162cee84b]
...
Running it through `addr2line' helps to see source-code lines of
crash-processing functions:
emacs_backtrace at /usr/local/src/emacs/trunk/src/sysdep.c:2034
terminate_due_to_signal at /usr/local/src/emacs/trunk/src/emacs.c:314
handle_fatal_signal at /usr/local/src/emacs/trunk/src/sysdep.c:1552
deliver_thread_signal at /usr/local/src/emacs/trunk/src/sysdep.c:1537
(inlined by) deliver_fatal_thread_signal at /usr/local/src/emacs/trunk/src/sysdep.c:1564
But the most interesting part of the backtrace with information
about Emacs image functions is truncated.
Displaying more lines might help to understand whether the bug is caused
by the old version of ImageMagick or by a bug in src/image.c.
Would be better to increase the number of lines to 24 or 32
or any other round number.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-29 19:31 ` Juri Linkov
@ 2012-09-29 20:01 ` Eli Zaretskii
2012-09-29 20:51 ` Paul Eggert
2012-09-30 1:48 ` The emacs_backtrace "feature" Stefan Monnier
0 siblings, 2 replies; 43+ messages in thread
From: Eli Zaretskii @ 2012-09-29 20:01 UTC (permalink / raw)
To: Juri Linkov; +Cc: eggert, emacs-devel
> From: Juri Linkov <juri@jurta.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> Date: Sat, 29 Sep 2012 22:31:27 +0300
>
> Would be better to increase the number of lines to 24 or 32
> or any other round number.
Instead of guessing, I'd suggest a small research: collect all the
backtraces reported to the bug tracker, and build the historgram of
their sizes. Then decide based on that.
I would also suggest some heuristics, e.g. if the crash is in GC, the
number will have to be much larger.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-29 20:01 ` Eli Zaretskii
@ 2012-09-29 20:51 ` Paul Eggert
2012-09-29 21:19 ` Eli Zaretskii
2012-09-30 1:48 ` The emacs_backtrace "feature" Stefan Monnier
1 sibling, 1 reply; 43+ messages in thread
From: Paul Eggert @ 2012-09-29 20:51 UTC (permalink / raw)
To: emacs-devel; +Cc: Juri Linkov
On 09/29/2012 01:01 PM, Eli Zaretskii wrote:
> I'd suggest a small research
It'd be nice to do that, but until someone has the time,
how about if we just bump the handle_fatal_signal
backtrace number from 10 to 24? It sounds like that would
have addressed Juri's problem.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-29 20:51 ` Paul Eggert
@ 2012-09-29 21:19 ` Eli Zaretskii
2012-09-30 1:30 ` Paul Eggert
0 siblings, 1 reply; 43+ messages in thread
From: Eli Zaretskii @ 2012-09-29 21:19 UTC (permalink / raw)
To: Paul Eggert; +Cc: juri, emacs-devel
> Date: Sat, 29 Sep 2012 13:51:08 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: Juri Linkov <juri@jurta.org>
>
> On 09/29/2012 01:01 PM, Eli Zaretskii wrote:
> > I'd suggest a small research
>
> It'd be nice to do that, but until someone has the time,
> how about if we just bump the handle_fatal_signal
> backtrace number from 10 to 24? It sounds like that would
> have addressed Juri's problem.
A quick look at several backtraces in the tracker indicates that
something like 40 will cover most cases.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-29 21:19 ` Eli Zaretskii
@ 2012-09-30 1:30 ` Paul Eggert
2012-09-30 9:44 ` messages-buffer-max-lines (was: The emacs_backtrace "feature") Juri Linkov
0 siblings, 1 reply; 43+ messages in thread
From: Paul Eggert @ 2012-09-30 1:30 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: juri, emacs-devel
On 09/29/2012 02:19 PM, Eli Zaretskii wrote:
> something like 40 will cover most cases.
OK, thanks, I did that in trunk bzr 110259.
^ permalink raw reply [flat|nested] 43+ messages in thread
* messages-buffer-max-lines (was: The emacs_backtrace "feature")
2012-09-30 1:30 ` Paul Eggert
@ 2012-09-30 9:44 ` Juri Linkov
2012-10-01 5:51 ` messages-buffer-max-lines Paul Eggert
0 siblings, 1 reply; 43+ messages in thread
From: Juri Linkov @ 2012-09-30 9:44 UTC (permalink / raw)
To: Paul Eggert; +Cc: Eli Zaretskii, emacs-devel
>> something like 40 will cover most cases.
>
> OK, thanks, I did that in trunk bzr 110259.
Thanks, 40 should be enough. There is a similar problem with
`messages-buffer-max-lines' (`message-log-max') whose default value
100 is too small. Often while debugging a problem in `emacs -Q'
it truncates lines in the *Messages* buffer, so I have to set it
to a higher number and redo all tests from the beginning.
Actually I see no reason to truncate *Messages* at all, but at least
if it should be a number, please increase it to a number one order of
magnitude higher (e.g. 1000) that won't cause frequent problems.
PS: Previous discussion 5 years ago:
http://lists.gnu.org/archive/html/emacs-devel/2007-04/msg00276.html
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-29 20:01 ` Eli Zaretskii
2012-09-29 20:51 ` Paul Eggert
@ 2012-09-30 1:48 ` Stefan Monnier
2012-09-30 7:14 ` Eli Zaretskii
2012-10-01 2:54 ` Chong Yidong
1 sibling, 2 replies; 43+ messages in thread
From: Stefan Monnier @ 2012-09-30 1:48 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Juri Linkov, eggert, emacs-devel
>> Would be better to increase the number of lines to 24 or 32
>> or any other round number.
> Instead of guessing, I'd suggest a small research: collect all the
> backtraces reported to the bug tracker, and build the historgram of
> their sizes. Then decide based on that.
> I would also suggest some heuristics, e.g. if the crash is in GC, the
> number will have to be much larger.
I really dislike the idea of writing those backtraces into a file (when
RMS mentioned it, I was shocked), but the more it goes, the more I think
it would make sense for the non-ENABLE_CHECKING builds. Tho maybe just
not showing the backtraces at all would work just as well in those cases.
I still would prefer going back to having the easserts in macros rather
than in functions (so that the assertion itself immediately gives you
the line-number, without having to go through the addr2line rigmarole).
Stefan
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-30 1:48 ` The emacs_backtrace "feature" Stefan Monnier
@ 2012-09-30 7:14 ` Eli Zaretskii
2012-09-30 7:24 ` Paul Eggert
2012-10-01 2:54 ` Chong Yidong
1 sibling, 1 reply; 43+ messages in thread
From: Eli Zaretskii @ 2012-09-30 7:14 UTC (permalink / raw)
To: Stefan Monnier; +Cc: juri, eggert, emacs-devel
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Juri Linkov <juri@jurta.org>, eggert@cs.ucla.edu, emacs-devel@gnu.org
> Date: Sat, 29 Sep 2012 21:48:39 -0400
>
> I still would prefer going back to having the easserts in macros rather
> than in functions (so that the assertion itself immediately gives you
> the line-number, without having to go through the addr2line rigmarole).
Sorry, I don't follow: the eassert macro includes the file name and
the line number in the message it prints, together with the condition
that was violated.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-30 7:14 ` Eli Zaretskii
@ 2012-09-30 7:24 ` Paul Eggert
0 siblings, 0 replies; 43+ messages in thread
From: Paul Eggert @ 2012-09-30 7:24 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
On 09/30/2012 12:14 AM, Eli Zaretskii wrote:
> Sorry, I don't follow: the eassert macro includes the file name and
> the line number in the message it prints
Stefan's talking about the macro (or function) F that invokes eassert,
not eassert itself. If F is a macro, eassert outputs the line
number of F's caller. If F is a function, it outputs F's line number.
If F is small, he'd often rather see the former.
Since the complaint is about the addr2line hassle I expect the
problem could be addressed by doing the addr2line work inside Emacs.
In the long run that would be better than forcing the use of macros
rather than functions purely because of crash debug output.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-30 1:48 ` The emacs_backtrace "feature" Stefan Monnier
2012-09-30 7:14 ` Eli Zaretskii
@ 2012-10-01 2:54 ` Chong Yidong
2012-10-01 11:43 ` Lars Magne Ingebrigtsen
1 sibling, 1 reply; 43+ messages in thread
From: Chong Yidong @ 2012-10-01 2:54 UTC (permalink / raw)
To: Stefan Monnier; +Cc: Juri Linkov, Eli Zaretskii, eggert, emacs-devel
Stefan Monnier <monnier@iro.umontreal.ca> writes:
> I really dislike the idea of writing those backtraces into a file (when
> RMS mentioned it, I was shocked), but the more it goes, the more I think
> it would make sense for the non-ENABLE_CHECKING builds. Tho maybe just
> not showing the backtraces at all would work just as well in those cases.
The user often wouldn't even be aware of the existence of these files.
Littering the filesystem with these backtrace files is kind of
obnoxious.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-21 9:49 The emacs_backtrace "feature" Eli Zaretskii
` (3 preceding siblings ...)
2012-09-21 19:37 ` Paul Eggert
@ 2012-09-21 21:40 ` Richard Stallman
2012-09-22 6:53 ` Stephen J. Turnbull
2012-09-24 14:27 ` Lars Ingebrigtsen
4 siblings, 2 replies; 43+ messages in thread
From: Richard Stallman @ 2012-09-21 21:40 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
I tend to agree with you, and perhaps the backtrace should
be directed to a specific disk file rather than to standard error.
Andreas Schwab <schwab@linux-m68k.org> wrote:
That's not a real problem. The output of programs that are started from
the desktop is typically redirected to a file, so nothing gets lost.
That statement seems very surprising to me. Is it true?
Where does Gnome redirect the error output from Emacs?
--
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
Use Ekiga or an ordinary phone call
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-21 21:40 ` Richard Stallman
@ 2012-09-22 6:53 ` Stephen J. Turnbull
2012-09-24 14:27 ` Lars Ingebrigtsen
1 sibling, 0 replies; 43+ messages in thread
From: Stephen J. Turnbull @ 2012-09-22 6:53 UTC (permalink / raw)
To: rms; +Cc: Eli Zaretskii, emacs-devel
Richard Stallman writes:
> Where does Gnome redirect the error output from Emacs?
I believe it's done at a lower level than GNOME. Here's what Gentoo's
xsession does:
----------------------------------------------------------------
# redirect errors to a file in user's home directory if we can
for errfile in "$HOME/.xsession-errors" "${TMPDIR-/tmp}/xses-$USER" "/tmp/xses-$USER"
do
if ( cp /dev/null "$errfile" 2> /dev/null )
then
chmod 600 "$errfile"
exec > "$errfile" 2>&1
break
fi
done
----------------------------------------------------------------
Indeed, various boring messages (SCIM startup, in particular) are
captured there.
My window manager is Fluxbox, whose session-starter has code to do the
same, to ~/.fluxbox-errors.
Debian's Xsession script does something similar.
Of course this doesn't help much if you usually start apps from the
command line. I typically start them using M-! or a shell buffer, so
the output ends up in a buffer rather than the session manager's
xsession-errors.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: The emacs_backtrace "feature"
2012-09-21 21:40 ` Richard Stallman
2012-09-22 6:53 ` Stephen J. Turnbull
@ 2012-09-24 14:27 ` Lars Ingebrigtsen
1 sibling, 0 replies; 43+ messages in thread
From: Lars Ingebrigtsen @ 2012-09-24 14:27 UTC (permalink / raw)
To: rms; +Cc: Eli Zaretskii, emacs-devel
Richard Stallman <rms@gnu.org> writes:
> I tend to agree with you, and perhaps the backtrace should
> be directed to a specific disk file rather than to standard error.
I agree. The current "let's just spew out reams of nearly meaningless
data to the terminal" methodology isn't very useful.
--
(domestic pets only, the antidote for overdose, milk.)
http://lars.ingebrigtsen.no * Lars Magne Ingebrigtsen
^ permalink raw reply [flat|nested] 43+ messages in thread
end of thread, other threads:[~2012-10-01 11:43 UTC | newest]
Thread overview: 43+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-09-21 9:49 The emacs_backtrace "feature" Eli Zaretskii
2012-09-21 9:58 ` Andreas Schwab
2012-09-21 10:17 ` Eli Zaretskii
2012-09-21 12:00 ` Andreas Schwab
2012-09-21 19:29 ` Florian Weimer
2012-09-21 19:45 ` Paul Eggert
2012-09-24 15:52 ` Steinar Bang
2012-09-21 12:24 ` Óscar Fuentes
2012-09-21 16:35 ` Stefan Monnier
2012-09-21 16:49 ` Stephen J. Turnbull
2012-09-21 17:07 ` Tom Tromey
2012-09-21 17:41 ` Andreas Schwab
2012-09-21 19:37 ` Paul Eggert
2012-09-22 12:05 ` Richard Stallman
2012-09-22 13:54 ` Christoph Scholtes
2012-09-22 14:23 ` covici
2012-09-24 15:55 ` Steinar Bang
2012-09-24 17:29 ` Paul Eggert
2012-09-22 12:41 ` Eli Zaretskii
2012-09-22 19:28 ` Paul Eggert
2012-09-23 13:05 ` Stephen J. Turnbull
2012-09-23 14:19 ` Andreas Schwab
2012-09-23 19:30 ` Stephen J. Turnbull
2012-09-23 21:01 ` Nix
2012-09-24 5:00 ` Stephen J. Turnbull
2012-09-24 9:29 ` Nix
2012-09-24 23:55 ` Stephen J. Turnbull
2012-09-29 19:31 ` Juri Linkov
2012-09-29 20:01 ` Eli Zaretskii
2012-09-29 20:51 ` Paul Eggert
2012-09-29 21:19 ` Eli Zaretskii
2012-09-30 1:30 ` Paul Eggert
2012-09-30 9:44 ` messages-buffer-max-lines (was: The emacs_backtrace "feature") Juri Linkov
2012-10-01 5:51 ` messages-buffer-max-lines Paul Eggert
2012-10-01 6:50 ` messages-buffer-max-lines Chong Yidong
2012-09-30 1:48 ` The emacs_backtrace "feature" Stefan Monnier
2012-09-30 7:14 ` Eli Zaretskii
2012-09-30 7:24 ` Paul Eggert
2012-10-01 2:54 ` Chong Yidong
2012-10-01 11:43 ` Lars Magne Ingebrigtsen
2012-09-21 21:40 ` Richard Stallman
2012-09-22 6:53 ` Stephen J. Turnbull
2012-09-24 14:27 ` Lars Ingebrigtsen
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.