Re: Crash in handling tar files (Was: GDB debugger mode for Emacs in ELPA)

Re: Crash in handling tar files (Was: GDB debugger mode for Emacs in ELPA)

[filerz-emacs]

Hello,
 Do we really need a different allocation method for M$? Looks like the REL_ALLOC is a pool based allocator. Could someone comment on benefits seen in using this pool based allocation? If this is used only in M$ platform, we could create multiple heaps at startup and use them. Once the heap becomes highly fragmented (we can use the low fragmentation heap), we can create a new heap, wait till all references to the fragmented heap are freed and destroy the fragmented heap. Since Emacs is single threaded, we can create a heap with no serialization, that will further speedup the allocations on M$.

If someone confirms that REL_ALLOC is primarily for M$ or feel it might be worth exploring the above, I am willing to give it a try (as I have done that in my day job).

----- Original Message ----
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Does anyone see this problem on platforms other than MS-Windows?

I do not see it on GNU/Linux platform.
 
> > I saw another crash with the following steps:
> > 1. Repeat #1, #2 from the previous scenario
> > 2. Open the tar file, you will see the file listing
> > 3. Position the cursor on one of the listed files and hit enter (open the file 
> in the archive)
> > You see a crash. Stack trace follows for 2nd scenario:
> 
> It doesn't crash for me with these steps.  Maybe here the exact
> contents of the tarball does matter, at least the file names in it
> (look at the Lisp backtrace).

Even the second scenario (without loading the 'tar-mode') does not happen on GNU/Linux (as REL_ALLOC stuff is not used there I guess)

I could reproduce by creating a tar file with 'src/w32.c' as "tar cvf crash.tar src/w32.c". Here is the backtrace:

(gdb) bt
#0  0x0119eaf3 in re_search_2 (bufp=0x1376420, str1=0x388a160 "ostfix modifiers\"\n \"quail/latin-alt\")\n(register-input-method\n \"latin-3-alt-postfix\" \"Latin-3\" 'quail-use-package\n \"3<\" \"Latin-3 char
#1  0x0114fb2f in search_buffer (string=44205683, pos=513, pos_byte=513, lim=127920, lim_byte=127920, n=1, RE=1, trt=53274628, inverse_trt=53274116, posix=0) at search.c:1190
#2  0x0114f495 in search_command (string=44205683, bound=1023360, noerror=43722801, count=43722753, direction=1, RE=1, posix=0) at search.c:972
#3  0x011533c2 in Fre_search_forward (regexp=44205683, bound=1023360, noerror=43722801, count=43722753) at search.c:2306
#4  0x01023f1a in Ffuncall (nargs=4, args=0x82ebf0) at eval.c:3052
#5  0x011171ee in Fbyte_code (bytestr=18836419, vector=18836436, maxdepth=32) at bytecode.c:678
#6  0x010245fb in funcall_lambda (fun=18836372, nargs=2, arg_vector=0x82ef04) at eval.c:3229
#7  0x010240cf in Ffuncall (nargs=3, args=0x82ef00) at eval.c:3088
#8  0x011171ee in Fbyte_code (bytestr=18837043, vector=18837060, maxdepth=48) at bytecode.c:678
#9  0x010245fb in funcall_lambda (fun=18836996, nargs=2, arg_vector=0x82f224) at eval.c:3229
#10 0x010240cf in Ffuncall (nargs=3, args=0x82f220) at eval.c:3088
#11 0x011171ee in Fbyte_code (bytestr=18837715, vector=18837732, maxdepth=32) at bytecode.c:678
#12 0x010245fb in funcall_lambda (fun=18837668, nargs=2, arg_vector=0x82f534) at eval.c:3229
#13 0x010240cf in Ffuncall (nargs=3, args=0x82f530) at eval.c:3088
#14 0x011171ee in Fbyte_code (bytestr=53331011, vector=45840388, maxdepth=48) at bytecode.c:678
#15 0x010245fb in funcall_lambda (fun=49166660, nargs=0, arg_vector=0x82f884) at eval.c:3229
#16 0x010240cf in Ffuncall (nargs=1, args=0x82f880) at eval.c:3088
#17 0x01023950 in apply1 (fn=48517289, arg=43722753) at eval.c:2783
#18 0x0111a380 in Fcall_interactively (function=48517289, record_flag=43722753, keys=43756292) at callint.c:389
#19 0x01023ee3 in Ffuncall (nargs=4, args=0x82fb18) at eval.c:3048
#20 0x01023a52 in call3 (fn=43917313, arg1=48517289, arg2=43722753, arg3=43722753) at eval.c:2868
#21 0x01015344 in Fcommand_execute (cmd=48517289, record_flag=43722753, keys=43722753, special=43722753) at keyboard.c:10420
#22 0x01007823 in command_loop_1 () at keyboard.c:1910
#23 0x01021a66 in internal_condition_case (bfun=0x10060cf <command_loop_1>, handlers=43786457, hfun=0x1005acc <cmd_error>) at eval.c:1511
#24 0x01005e18 in command_loop_2 () at keyboard.c:1367
#25 0x01021576 in internal_catch (tag=43782529, func=0x1005df8 <command_loop_2>, arg=43722753) at eval.c:1247
#26 0x01005dcf in command_loop () at keyboard.c:1346
#27 0x010056d2 in recursive_edit_1 () at keyboard.c:955
#28 0x01005840 in Frecursive_edit () at keyboard.c:1017
#29 0x010027dc in main (argc=2, argv=0xa34290) at emacs.c:1762

Lisp Backtrace:
"re-search-forward" (0x82ebf4)
"auto-coding-regexp-alist-lookup" (0x82ef04)
"find-auto-coding" (0x82f224)
"set-auto-coding" (0x82f534)
"tar-extract" (0x82f884)
"call-interactively" (0x82fb1c)

-dhruva



      Unlimited freedom, unlimited storage. Get it now, on http://help.yahoo.com/l/in/yahoo/mail/yahoomail/tools/tools-08.html/

Re: Crash in handling tar files (Was: GDB debugger mode for Emacs in ELPA)

[Eli Zaretskii]

> Date: Mon, 30 Jun 2008 09:25:04 +0530 (IST)
> From: filerz-emacs@yahoo.com
> Cc: emacs-devel@gnu.org
> 
>  Do we really need a different allocation method for M$?

How do you mean ``different''?  REL_ALLOC is used not only on
MS-Windows.

> Looks like the REL_ALLOC is a pool based allocator. Could someone
> comment on benefits seen in using this pool based allocation?

REL_ALLOC (actually, ralloc.c) is a relocatable memory allocator.  Its
benefit is that, unlike most (maybe all?) malloc implementations, it
can recover from memory fragmentation by relocating some of the Lisp
objects (strings and buffers, IIRC) in order to bring small fragmented
chunks of free memory into larger and less scattered ones.  Thus, when
large memory buffers are requested by alloc.c, ralloc.c can serve
those requests for longer time without asking more core from the OS.

Without ralloc.c, Emacs's memory footprint tends to grow much faster
and does not always level out.  You can try it: just recompile Emacs
without REL_ALLOC defined, and run it in a long, real-life editing
session; then compare the memory footprints.

> If this is used only in M$ platform, we could create multiple heaps at startup and use them. Once the heap becomes highly fragmented (we can use the low fragmentation heap), we can create a new heap, wait till all references to the fragmented heap are freed and destroy the fragmented heap. Since Emacs is single threaded, we can create a heap with no serialization, that will further speedup the allocations on M$.

Do we really want to develop a new memory allocation scheme from
scratch?  ralloc.c is used on many platforms for many years (GNU/Linux
stopped using it because they use mmap and Doug Lea's allocator (again
IIRC), which is even better than ralloc.c).  There's no reason to
believe ralloc.c causes any real trouble, certainly not because of
some petty bug in tar-mode.el or thereabouts!

> Even the second scenario (without loading the 'tar-mode') does not happen on GNU/Linux (as REL_ALLOC stuff is not used there I guess)

You are assuming that this is ralloc.c's fault.  But it's much more
probable that it's some memory corruption elsewhere, and if that is
the case, other memory allocation mechanisms will see it as well.

> I could reproduce by creating a tar file with 'src/w32.c' as "tar cvf crash.tar src/w32.c". Here is the backtrace:
> 
> (gdb) bt
> #0  0x0119eaf3 in re_search_2 (bufp=0x1376420, str1=0x388a160 "ostfix modifiers\"\n \"quail/latin-alt\")\n(register-input-method\n \"latin-3-alt-postfix\" \"Latin-3\" 'quail-use-package\n \"3<\" \"Latin-3 char

This is strange text!  What is your value of auto-coding-regexp-alist?
Do you have any idea why auto-coding-regexp-alist-lookup ended up
looking at the string pointed to by str1 above?

Re: Crash in handling tar files (Was: GDB debugger mode for Emacs in ELPA)

[filerz-emacs]

Hi,

----- Original Message ----
> From: Eli Zaretskii <eliz@gnu.org>
> > (gdb) bt
> > #0  0x0119eaf3 in re_search_2 (bufp=0x1376420, str1=0x388a160 "ostfix 
> modifiers\"\n \"quail/latin-alt\")\n(register-input-method\n 
> \"latin-3-alt-postfix\" \"Latin-3\" 'quail-use-package\n \"3<\" \"Latin-3 char
> 
> This is strange text!  What is your value of auto-coding-regexp-alist?

(("^BABYL OPTIONS:[     ]*-\\*-[     ]*rmail[     ]*-\\*-" . no-conversion) ("\\`þÿ" . utf-16be-with-signature) ("\\`ÿþ" . utf-16le-with-signature) ("\\`" . utf-8) ("\\`;ELC

I am running 'emacs -q --no-site-file' on WXP built with MinGW and see the above value

> Do you have any idea why auto-coding-regexp-alist-lookup ended up
> looking at the string pointed to by str1 above?

Not a clue!

-dhruva



      Bring your gang together. Do your thing. Find your favourite Yahoo! group at http://in.promos.yahoo.com/groups/

Re: Crash in handling tar files (Was: GDB debugger mode for Emacs in ELPA)

[Eli Zaretskii]

> Date: Tue, 1 Jul 2008 15:37:38 +0530 (IST)
> From: filerz-emacs@yahoo.com
> Cc: emacs-devel@gnu.org
> 
> > From: Eli Zaretskii <eliz@gnu.org>
> > > (gdb) bt
> > > #0  0x0119eaf3 in re_search_2 (bufp=0x1376420, str1=0x388a160 "ostfix 
> > modifiers\"\n \"quail/latin-alt\")\n(register-input-method\n 
> > \"latin-3-alt-postfix\" \"Latin-3\" 'quail-use-package\n \"3<\" \"Latin-3 char
> > 
> > This is strange text!  What is your value of auto-coding-regexp-alist?
> 
> (("^BABYL OPTIONS:[     ]*-\\*-[     ]*rmail[     ]*-\\*-" . no-conversion) ("\\`þÿ" . utf-16be-with-signature) ("\\`ÿþ" . utf-16le-with-signature) ("\\`" . utf-8) ("\\`;ELC
> 
> I am running 'emacs -q --no-site-file' on WXP built with MinGW and see the above value
> 
> > Do you have any idea why auto-coding-regexp-alist-lookup ended up
> > looking at the string pointed to by str1 above?
> 
> Not a clue!

In that case, my guess is that this is the result of some memory
mess-up, perhaps a GC with un-GCPRO'ed variables.  That could explain
the strange text we see in regex search initiated by
auto-coding-regexp-alist-lookup, since GC could relocate strings.

Unfortunately, I will be traveling for 3 weeks starting tomorrow, so I
probably won't have time to debug this.  Maybe someone beats me to it;
if not, I hope to look into this in 3 weeks time.

Re: Crash in handling tar files (Was: GDB debugger mode for Emacs in ELPA)

[filerz-emacs]

Hello,

----- Original Message ----
> From: Angelo Graziosi <angelo.graziosi@alice.it>

> Could the crash discussed in this thread be related to that, [1], we 
> discussed longly and which still happens on Cygwin?

From the trace, it does appear quite similar. I am now looking into the code. The problem seems to be that a call to realloc is not able to locate the pointer as allocated by it earlier. I did debug and stepped through the code. I am planning to log all allocations, re-allocations and free to see what is happening.

-dhruva



      Did you know? You can CHAT without downloading messenger. Go to http://in.messenger.yahoo.com/webmessengerpromo.php/

Re: Crash in handling tar files (Was: GDB debugger mode for Emacs in ELPA)

[Angelo Graziosi]

Eli,

Could the crash discussed in this thread be related to that, [1], we 
discussed longly and which still happens on Cygwin?


    Angelo

---
[1] http://lists.gnu.org/archive/html/emacs-devel/2008-06/msg00839.html

Re: Crash in handling tar files (Was: GDB debugger mode for Emacs in ELPA)

[David Robinow]

On Fri, Jun 27, 2008 at 8:26 AM, Angelo Graziosi
<angelo.graziosi@alice.it> wrote:
> Could the crash discussed in this thread be related to that, [1], we
> discussed longly and which still happens on Cygwin?
Certainly. There's been no evidence that this is related to Cygwin.
My emacs crashes on every tar file I've tried to open, on 3 different machines.
Windows XP with MSVC.
Windows XP with MINGW
Vista with MINGW.

Re: Crash in handling tar files (Was: GDB debugger mode for Emacs in ELPA)

[Eli Zaretskii]

> Date: Fri, 27 Jun 2008 14:26:05 +0200
> From: Angelo Graziosi <angelo.graziosi@alice.it>
> CC: emacs-devel@gnu.org
> 
> Could the crash discussed in this thread be related to that, [1], we 
> discussed longly and which still happens on Cygwin?

It could, but I won't know that until I reproduce it on my machine.
Also, knowing that it's related doesn't help a bit in debugging it
(except when time comes to test the fix).

Crash in handling tar files (Was: GDB debugger mode for Emacs in ELPA)

[filerz-emacs]

Hi,

----- Original Message ----
> From: Eli Zaretskii <eliz@gnu.org>

> Which tarball causes this crash, and where can I find it?  Also, would
> the OP please post a detailed recipe to reproduce this crash, starting
> with "emacs -Q"?

Steps to reproduce:
1. Create a small tar file with 2 or 3 files in it
2. emacs -q
3. M-x load-library tar-mode
4. Open the tar file
You see the crash (with earlier posted stack trace)

I saw another crash with the following steps:
1. Repeat #1, #2 from the previous scenario
2. Open the tar file, you will see the file listing
3. Position the cursor on one of the listed files and hit enter (open the file in the archive)
You see a crash. Stack trace follows for 2nd scenario:

(gdb) bt
#0  0x0119ea03 in re_search_2 (bufp=0x1377928, str1=0x389f1b0 "ix\" \"Latin-3\" 'quail-use-package\n \"3<\" \"Latin-3 character input method with postfix modifiers\
#1  0x0114fabf in search_buffer (string=44210019, pos=513, pos_byte=513, lim=33830, lim_byte=33830, n=1, RE=1, trt=53475332, inverse_trt=53474820, posix=0) at searc
#2  0x0114f425 in search_command (string=44210019, bound=270640, noerror=43726897, count=43726849, direction=1, RE=1, posix=0) at search.c:972
#3  0x01153352 in Fre_search_forward (regexp=44210019, bound=270640, noerror=43726897, count=43726849) at search.c:2306
#4  0x01023f1a in Ffuncall (nargs=4, args=0x82ebf0) at eval.c:3052
#5  0x011171ee in Fbyte_code (bytestr=18840531, vector=18840548, maxdepth=32) at bytecode.c:678
#6  0x010245fb in funcall_lambda (fun=18840484, nargs=2, arg_vector=0x82ef04) at eval.c:3229
#7  0x010240cf in Ffuncall (nargs=3, args=0x82ef00) at eval.c:3088
#8  0x011171ee in Fbyte_code (bytestr=18841155, vector=18841172, maxdepth=48) at bytecode.c:678
#9  0x010245fb in funcall_lambda (fun=18841108, nargs=2, arg_vector=0x82f224) at eval.c:3229
#10 0x010240cf in Ffuncall (nargs=3, args=0x82f220) at eval.c:3088
#11 0x011171ee in Fbyte_code (bytestr=18841827, vector=18841844, maxdepth=32) at bytecode.c:678
#12 0x010245fb in funcall_lambda (fun=18841780, nargs=2, arg_vector=0x82f534) at eval.c:3229
#13 0x010240cf in Ffuncall (nargs=3, args=0x82f530) at eval.c:3088
#14 0x011171ee in Fbyte_code (bytestr=53331763, vector=45444612, maxdepth=48) at bytecode.c:678
#15 0x010245fb in funcall_lambda (fun=49306564, nargs=0, arg_vector=0x82f884) at eval.c:3229
#16 0x010240cf in Ffuncall (nargs=1, args=0x82f880) at eval.c:3088
#17 0x01023950 in apply1 (fn=48507577, arg=43726849) at eval.c:2783
#18 0x0111a380 in Fcall_interactively (function=48507577, record_flag=43726849, keys=43760388) at callint.c:389
#19 0x01023ee3 in Ffuncall (nargs=4, args=0x82fb18) at eval.c:3048
#20 0x01023a52 in call3 (fn=43921409, arg1=48507577, arg2=43726849, arg3=43726849) at eval.c:2868
#21 0x01015344 in Fcommand_execute (cmd=48507577, record_flag=43726849, keys=43726849, special=43726849) at keyboard.c:10420
#22 0x01007823 in command_loop_1 () at keyboard.c:1910
#23 0x01021a66 in internal_condition_case (bfun=0x10060cf <command_loop_1>, handlers=43790553, hfun=0x1005acc <cmd_error>) at eval.c:1511
#24 0x01005e18 in command_loop_2 () at keyboard.c:1367
#25 0x01021576 in internal_catch (tag=43786625, func=0x1005df8 <command_loop_2>, arg=43726849) at eval.c:1247
#26 0x01005dcf in command_loop () at keyboard.c:1346
#27 0x010056d2 in recursive_edit_1 () at keyboard..c:955
#28 0x01005840 in Frecursive_edit () at keyboard.c:1017
#29 0x010027dc in main (argc=2, argv=0xa34128) at emacs.c:1762

Lisp Backtrace:
"re-search-forward" (0x82ebf4)
"auto-coding-regexp-alist-lookup" (0x82ef04)
"find-auto-coding" (0x82f224)
"set-auto-coding" (0x82f534)
"tar-extract" (0x82f884)
"call-interactively" (0x82fb1c)

-dhruva



      From Chandigarh to Chennai - find friends all over India. Go to http://in.promos.yahoo.com/groups/citygroups/

Re: Crash in handling tar files (Was: GDB debugger mode for Emacs in ELPA)

[Eli Zaretskii]

> Date: Fri, 27 Jun 2008 15:18:21 +0530 (IST)
> From: filerz-emacs@yahoo.com
> Cc: emacs-devel@gnu.org
> 
> Steps to reproduce:
> 1. Create a small tar file with 2 or 3 files in it
> 2. emacs -q
> 3. M-x load-library tar-mode
> 4. Open the tar file
> You see the crash (with earlier posted stack trace)

Confirmed.

Curiously enough, if you don't load tar-mode manually, it doesn't
crash (which will probably go a long way towards explaining why no one
noticed this before).

Does anyone see this problem on platforms other than MS-Windows?

> I saw another crash with the following steps:
> 1. Repeat #1, #2 from the previous scenario
> 2. Open the tar file, you will see the file listing
> 3. Position the cursor on one of the listed files and hit enter (open the file in the archive)
> You see a crash. Stack trace follows for 2nd scenario:

It doesn't crash for me with these steps.  Maybe here the exact
contents of the tarball does matter, at least the file names in it
(look at the Lisp backtrace).