bug#8884: wide-int crash [was Re: [PATCH updated] Support for filesystem watching (inotify)]

bug#8884: wide-int crash [was Re: [PATCH updated] Support for filesystem watching (inotify)]

[Paul Eggert]

On 07/06/11 12:14, Glenn Morris wrote:

> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8884

Thanks for the heads-up.  Although the nearby code has a portability bug
regardless of whether --with-wide-int is used, --with-wide-int
is more likely to tickle the bug.  The bug could well explain the
symptoms Peter observed.

I committed the following patch into the trunk.  Peter,
can you please check whether it solves your problem?  If not,
can you please compile with -g and without -O, and use GDB to report
the following values at the point of the crash: buffer_local_flags,
idx, offset, sizeof (struct buffer).  Thanks.

=== modified file 'src/ChangeLog'
--- src/ChangeLog	2011-07-05 09:51:56 +0000
+++ src/ChangeLog	2011-07-06 22:22:32 +0000
@@ -1,3 +1,15 @@
+2011-07-06  Paul Eggert  <eggert@cs.ucla.edu>
+
+	Remove unportable assumption about struct layout (Bug#8884).
+	* alloc.c (mark_buffer):
+	* buffer.c (reset_buffer_local_variables, Fbuffer_local_variables)
+	(clone_per_buffer_values): Don't assume that
+	sizeof (struct buffer) is a multiple of sizeof (Lisp_Object).
+	This isn't true in general, and it's particularly not true
+	if Emacs is configured with --with-wide-int.
+	* buffer.h (FIRST_FIELD_PER_BUFFER, LAST_FIELD_PER_BUFFER):
+	New macros, used in the buffer.c change.
+
 2011-07-05  Jan Djärv  <jan.h.d@swipnet.se>
 
 	* xsettings.c: Use both GConf and GSettings if both are available.

=== modified file 'src/alloc.c'
--- src/alloc.c	2011-06-24 21:25:22 +0000
+++ src/alloc.c	2011-07-06 22:22:32 +0000
@@ -5619,7 +5619,8 @@
   /* buffer-local Lisp variables start at `undo_list',
      tho only the ones from `name' on are GC'd normally.  */
   for (ptr = &buffer->BUFFER_INTERNAL_FIELD (name);
-       (char *)ptr < (char *)buffer + sizeof (struct buffer);
+       ptr <= &PER_BUFFER_VALUE (buffer,
+				 PER_BUFFER_VAR_OFFSET (LAST_FIELD_PER_BUFFER));
        ptr++)
     mark_object (*ptr);
 

=== modified file 'src/buffer.c'
--- src/buffer.c	2011-07-04 15:32:22 +0000
+++ src/buffer.c	2011-07-06 22:22:32 +0000
@@ -471,8 +471,8 @@
 
   /* buffer-local Lisp variables start at `undo_list',
      tho only the ones from `name' on are GC'd normally.  */
-  for (offset = PER_BUFFER_VAR_OFFSET (undo_list);
-       offset < sizeof *to;
+  for (offset = PER_BUFFER_VAR_OFFSET (FIRST_FIELD_PER_BUFFER);
+       offset <= PER_BUFFER_VAR_OFFSET (LAST_FIELD_PER_BUFFER);
        offset += sizeof (Lisp_Object))
     {
       Lisp_Object obj;
@@ -830,8 +830,8 @@
 
   /* buffer-local Lisp variables start at `undo_list',
      tho only the ones from `name' on are GC'd normally.  */
-  for (offset = PER_BUFFER_VAR_OFFSET (undo_list);
-       offset < sizeof *b;
+  for (offset = PER_BUFFER_VAR_OFFSET (FIRST_FIELD_PER_BUFFER);
+       offset <= PER_BUFFER_VAR_OFFSET (LAST_FIELD_PER_BUFFER);
        offset += sizeof (Lisp_Object))
     {
       int idx = PER_BUFFER_IDX (offset);
@@ -1055,8 +1055,8 @@
 
     /* buffer-local Lisp variables start at `undo_list',
        tho only the ones from `name' on are GC'd normally.  */
-    for (offset = PER_BUFFER_VAR_OFFSET (undo_list);
-	 offset < sizeof (struct buffer);
+    for (offset = PER_BUFFER_VAR_OFFSET (FIRST_FIELD_PER_BUFFER);
+	 offset <= PER_BUFFER_VAR_OFFSET (LAST_FIELD_PER_BUFFER);
 	 /* sizeof EMACS_INT == sizeof Lisp_Object */
 	 offset += (sizeof (EMACS_INT)))
       {

=== modified file 'src/buffer.h'
--- src/buffer.h	2011-06-21 21:32:10 +0000
+++ src/buffer.h	2011-07-06 21:53:56 +0000
@@ -612,6 +612,7 @@
   /* Everything from here down must be a Lisp_Object.  */
   /* buffer-local Lisp variables start at `undo_list',
      tho only the ones from `name' on are GC'd normally.  */
+  #define FIRST_FIELD_PER_BUFFER undo_list
 
   /* Changes in the buffer are recorded here for undo.
      t means don't record anything.
@@ -846,6 +847,9 @@
      t means to use hollow box cursor.
      See `cursor-type' for other values.  */
   Lisp_Object BUFFER_INTERNAL_FIELD (cursor_in_non_selected_windows);
+
+  /* This must be the last field in the above list.  */
+  #define LAST_FIELD_PER_BUFFER cursor_in_non_selected_windows
 };
 
 \f

bug#8884: wide-int crash

[Glenn Morris]

Paul Eggert wrote:

> I committed the following patch into the trunk.  Peter,
> can you please check whether it solves your problem?  If not,
> can you please compile with -g and without -O, and use GDB to report
> the following values at the point of the crash: buffer_local_flags,
> idx, offset, sizeof (struct buffer).

Since no reply was received I am closing this under the presumption that
it was fixed.

bug#8884: 24.0.50; temacs crashes and dumps core when configured --with-wide-int

[Peter Dyballa]

Mac OS X 10.5.8, PPC, PowerPC 7447A (7450), GCC 4.5.2. Configuration  
with:

	CFLAGS="-g -H -pipe -fPIC -fno-common -mcpu=7450 -mtune=7450 - 
maltivec -faltivec -mabi=altivec -Os -mfused-madd -mmultiple -ftree- 
vectorize"
	LDFLAGS="-Wl,-dead_strip_dylibs -Wl,-bind_at_load -Wl,-t"


I'm trying again with MacPorts instead of Fink.

--
Greetings

   Pete

There is no national science just as there is no national  
multiplication table; what is national is no longer science.
				– Anton Checov

bug#8884: 24.0.50; temacs crashes and dumps core when configured --with-wide-int

[Peter Dyballa]

I removed all optimisation CFLAGS ("-g -H -pipe -fPIC -O0" were left),  
even -fPIC, and all LDFLAGS. The result is always the same, crash and  
core dump:

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x000000001075ad60
Crashed Thread:  0

Thread 0 Crashed:
0   temacs                        	0x00241084  
reset_buffer_local_variables + 2192 (buffer.c:833)
1   temacs                        	0x00249d9c  
Fkill_all_local_variables + 112 (buffer.c:2462)
2   temacs                        	0x00270ad4 get_minibuffer + 876  
(minibuf.c:807)
3   temacs                        	0x00255d60 init_buffer + 1032  
(buffer.c:5137)
4   temacs                        	0x001f33b0 main + 5616 (emacs.c:1444)
5   temacs                        	0x00002a3c start + 64

--
Greetings

   Pete

Sending unsolicited commercial eMail to this account incurs a fee of €  
500 per message and acknowledges the legality of this contract.

bug#8884: closed (Re: bug#8884: wide-int crash)

[Peter Dyballa]

Am 07.10.2011 um 09:31 schrieb GNU bug Tracking System:

> Since no reply was received I am closing this under the presumption that
> it was fixed.

Paul,

I'm sorry, I was too occupied with other things! Now that GNU Emacs 24.0.90 compiles well it also runs well with this wide-int patch. I compiled with -O0 and -Os with GCC 4.5.3, the original compiler. I'll try now with Apple modified GCC version 4.0 and 4.2 and their -fast optimiser switch. For theses tests I'll use the new optimised GNU Emacs 24.0.90.

--
Greetings

  Pete

One person with a belief is a social power equal to ninety-nine who have only interests.
				– John Stuart Mill

bug#8884: closed (Re: bug#8884: wide-int crash)

[Peter Dyballa]

Hello!

I managed to create quite large test files on an external disk:

-rw-r--r--  1 pete pete 11936487768 17. Okt 15:26 Halde.tar.lzo
-rw-r--r--  1 pete pete  5990912000 17. Okt 19:40 SW.tar

On my 64-bit intel-based (Sandy Bridge) system GNU Emacs 24.0.90 quite  
easily loads SW.tar (it allocates some GB of swap space) and also  
started to allocate even more swap space when I tried to view a small  
file in the archive – as expected!

On my 32-bit PowerPC 7447A based system GNU Emacs 24.0.90 (started  
with -Q) fails to open the same file (I created it on this system):

	File SW.tar is large (5713MB), really open? (y or n)  y
	byte-code: Maximum buffer size exceeded
	Auto-saving...

The *Buffers List* buffer has:

	. * *Messages*            2320  Fundamental
	  * *unsent mail to bug-: 4855  Message		  ~/Mail/drafts/ 
*message*-20111017-233119
	 %  Leo                   1025  Dired by name	  /Volumes/Leo/
	    *scratch*              191  Lisp Interaction
	 %  *Bug Help*             279  Help
	    SW.tar                   0  Fundamental

The unsent mail contains:

	In GNU Emacs 24.0.90.1 (powerpc-apple-darwin9.8.0, X toolkit, Xaw3d  
scroll bars)
	 of 2011-10-17 on Latsche.fritz.box
	Windowing system distributor `The X.Org Foundation', version  
11.0.11100000
	configured using `configure  '--without-sound' '--without-dbus' '-- 
without-pop' '--without-gconf' '--without-gpm' '--without-gsettings'  
'--without-gif' '--without-jpeg' '--without-png' '--without-rsvg' '-- 
without-tiff' '--without-xpm' '--with-wide-int' '--with-x- 
toolkit=athena' '--x-libraries=/usr/X11/lib' '--x-includes=/usr/X11/ 
include' '--enable-locallisppath=/Library/Application Support/Emacs/ 
calendar24:/Library/Application Support/Emacs' 'CFLAGS=-g -H -pipe - 
fPIC -fno-common -mcpu=7450 -mtune=7450 -maltivec -faltivec - 
mabi=altivec -Os -mfused-madd -mmultiple -ftree-vectorize -mpowerpc- 
gfxopt' 'LDFLAGS=-Wl,-dead_strip_dylibs -Wl,-bind_at_load -Wl,-t'  
'CC=gcc-4' 'CPP=cpp-4' 'PKG_CONFIG_PATH=/sw/lib/xft2/lib/pkgconfig:/sw/ 
share/pkgconfig:/sw/lib/pkgconfig:/usr/X11/lib/pkgconfig:/usr/X11/ 
share/pkgconfig:/usr/lib/pkgconfig''
	
	Important settings:
	  value of $LC_ALL: nil
	  value of $LC_COLLATE: nil
	  value of $LC_CTYPE: de_DE.UTF-8
	  value of $LC_MESSAGES: nil
	  value of $LC_MONETARY: nil
	  value of $LC_NUMERIC: nil
	  value of $LC_TIME: nil
	  value of $LANG: de_DE.UTF-8
	  value of $XMODIFIERS: nil
	  locale-coding-system: utf-8-unix
	  default enable-multibyte-characters: t
	
	Major mode: Lisp Interaction
	
	Minor modes in effect:
	  tooltip-mode: t
	  mouse-wheel-mode: t
	  tool-bar-mode: t
	  menu-bar-mode: t
	  file-name-shadow-mode: t
	  global-font-lock-mode: t
	  font-lock-mode: t
	  blink-cursor-mode: t
	  auto-composition-mode: t
	  auto-encryption-mode: t
	  auto-compression-mode: t
	  line-number-mode: t
	  transient-mark-mode: t

"gcc-4" is: gcc-4 (GCC) 4.5.3.


--
Greetings

              ~  O
   Pete       ~~_\\_/%
              ~  O  o

bug#8884: closed (Re: bug#8884: wide-int crash)

[Paul Eggert]

On 10/17/11 15:24, Peter Dyballa wrote:
> On my 32-bit PowerPC 7447A based system GNU Emacs 24.0.90 (started with -Q) fails to open the same file (I created it on this system):
> 
>     File SW.tar is large (5713MB), really open? (y or n)  y
>     byte-code: Maximum buffer size exceeded
>     Auto-saving...

Perhaps I'm missing something, but
this is the behavior I expect.
SW.tar is too large to fit into that system's memory.
(A 32-bit machine can't access more than 4 GiB of memory.

bug#8884: closed (Re: bug#8884: wide-int crash)

[Peter Dyballa]

Am 18.10.2011 um 07:42 schrieb Paul Eggert:

> Perhaps I'm missing something, but
> this is the behavior I expect.
> SW.tar is too large to fit into that system's memory.
> (A 32-bit machine can't access more than 4 GiB of memory.

It rather looks that *I* am missing some understanding... I thought that the use of wide ints on a 32-bit architecture was meant to give it the ability to handle larger files. What is the intention of their introduction then? Increasing the size of the binary? The Emacs NEWS/Changelog only tells:

	** There is a new configure option --with-wide-int.
	With it, Emacs integers typically have 62 bits, even on 32-bit machines.

From buffers.texi this text is derived:

	   A buffer's size cannot be larger than some maximum, which is defined
	by the largest buffer position representable by the "Emacs integer"
	data type.  This is because Emacs tracks buffer positions using that
	data type.  For 32-bit machines, the largest buffer size is 256
	megabytes.

4 GiB is 16 times that value, quite a bit more. Maths with large numbers, greater than 2**28, works. 

--
Greetings

  Pete

Increase the size of your bike by at least *five* inches!

bug#8884: closed (Re: bug#8884: wide-int crash)

[Eli Zaretskii]

> From: Peter Dyballa <Peter_Dyballa@Freenet.DE>
> Date: Tue, 18 Oct 2011 11:14:10 +0200
> Cc: 8884@debbugs.gnu.org
> 
> > (A 32-bit machine can't access more than 4 GiB of memory.
> 
> It rather looks that *I* am missing some understanding... I thought that the use of wide ints on a 32-bit architecture was meant to give it the ability to handle larger files.

It was, and it does.  See below.

> What is the intention of their introduction then? Increasing the size of the binary?

:-)


> 	** There is a new configure option --with-wide-int.
> 	With it, Emacs integers typically have 62 bits, even on 32-bit machines.

Perhaps the NEWS entry could be improved.

> From buffers.texi this text is derived:
> 
> 	   A buffer's size cannot be larger than some maximum, which is defined
> 	by the largest buffer position representable by the "Emacs integer"
> 	data type.  This is because Emacs tracks buffer positions using that
> 	data type.  For 32-bit machines, the largest buffer size is 256
> 	megabytes.
> 
> 4 GiB is 16 times that value, quite a bit more. Maths with large numbers, greater than 2**28, works. 

Exactly.  So with this option, you can:

  . visit files or have buffers up to 4GB

  . use integer arithmetics up to 62-bit values

None of these is possible without this option.  The first of these two
advantages is the more important one: Emacs can now access the largest
files supported by the OS, instead of having a more-or-less arbitrary
limit below that.

Sounds better now?

bug#8884: closed (Re: bug#8884: wide-int crash)

[Stefan Monnier]

>> 4 GiB is 16 times that value, quite a bit more. Maths with large
>> numbers, greater than 2**28, works.
> Exactly.  So with this option, you can:
>   . visit files or have buffers up to 4GB

Actually, it will usually be less than 4GB, it depends on the amount of
space the OS gives us (it can be as little as 2GB).  E.g. you definitely
won't be able to open two files of 2.5GB each.

The current buffers.texi is fairly clear:

     A buffer's size cannot be larger than some maximum, which is defined
   by the largest buffer position representable by the @dfn{Emacs
   integer} data type.  This is because Emacs tracks buffer positions
   using that data type.  For typical 64-bit machines, the maximum buffer size
   enforced by the data types is @math{2^61 - 2} bytes, or about 2 EiB.
   For typical 32-bit machines, the maximum is @math{2^29 - 2} bytes, or
   about 512 MiB.  Buffer sizes are also limited by the size of Emacs's
   virtual memory.

So w.r.t accessing large files, --with-wide-int pushes the limit from
512MB to "a bit less than 4GB".  That's one of the reasons why I'm not
convinced it's worth the trouble, tho admittedly, the zone between 512MB
and 4GB might contain a fair number of mbox files, so it can be useful
for Rmail users.

> None of these is possible without this option.  The first of these two
> advantages is the more important one: Emacs can now access the largest
> files supported by the OS, instead of having a more-or-less arbitrary
> limit below that.

Of course, it's actually not "the largest files supported by the OS",
since most OSes nowadays happily handle files larger than 4GB, even on
32bit systems.


        Stefan

bug#8884: closed (Re: bug#8884: wide-int crash)

[Paul Eggert]

On 10/18/11 03:55, Eli Zaretskii wrote:
> Perhaps the NEWS entry could be improved.

I made the following change, to try to do that:

--- etc/NEWS	2011-10-18 06:52:32 +0000
+++ etc/NEWS	2011-10-18 15:34:06 +0000
@@ -51,6 +51,8 @@ This is not a new feature; only the conf
 ---
 ** There is a new configure option --with-wide-int.
 With it, Emacs integers typically have 62 bits, even on 32-bit machines.
+On 32-bit hosts, this raises the limit on buffer sizes from 512 MiB to
+a bit less than 4 GiB, with the exact limit set by the operating system.
 
 ---
 ** New translation of the Emacs Tutorial in Hebrew is available.

bug#8884: closed (Re: bug#8884: wide-int crash)

[Andreas Schwab]

Paul Eggert <eggert@cs.ucla.edu> writes:

> --- etc/NEWS	2011-10-18 06:52:32 +0000
> +++ etc/NEWS	2011-10-18 15:34:06 +0000
> @@ -51,6 +51,8 @@ This is not a new feature; only the conf
>  ---
>  ** There is a new configure option --with-wide-int.
>  With it, Emacs integers typically have 62 bits, even on 32-bit machines.
> +On 32-bit hosts, this raises the limit on buffer sizes from 512 MiB to
> +a bit less than 4 GiB, with the exact limit set by the operating system.

"a bit less than 4 GB" is an exaggeration, "almost 2GB" is much closer
to the truth.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#8884: closed (Re: bug#8884: wide-int crash)

[Paul Eggert]

On 10/18/11 08:45, Andreas Schwab wrote:
> "a bit less than 4 GB" is an exaggeration, "almost 2GB" is much closer
> to the truth.

Whoops!  Thanks for catching that; I forgot about the ptrdiff_t limit.
I installed this instead:

--- etc/NEWS	2011-10-18 06:52:32 +0000
+++ etc/NEWS	2011-10-18 15:55:20 +0000
@@ -51,6 +51,8 @@
 ---
 ** There is a new configure option --with-wide-int.
 With it, Emacs integers typically have 62 bits, even on 32-bit machines.
+On 32-bit hosts, this raises the limit on buffer sizes from about 512 MiB
+to about 2 GiB.
 
 ---
 ** New translation of the Emacs Tutorial in Hebrew is available.

bug#8884: closed (Re: bug#8884: wide-int crash)

[Peter Dyballa]

Am 18.10.2011 um 17:45 schrieb Andreas Schwab:

> "a bit less than 4 GB" is an exaggeration, "almost 2GB" is much closer
> to the truth.

With Mac OS X 10.5.8 and a PowerPC 7447A CPU GNU Emacs 24.0.90 launched with -Q can open (tar) files up to 2,047 MB. Ls reports that 2,147,450,880 is OK and 2,148,024,320 is too much (2,048 MB). 2**31 is 2,147,483,648.

The TAR file was created by an i386 tar (GNU tar) 1.26 on an external FireWire disk I attached either on my fast 64-bit intel based Mac or on my slow 32-bit G4, where I performed the tests. In dired-mode I opened the directory and read the file in with v. The MB numbers where reported by GNU Emacs' routine which asked me to confirm that I really wanted to open such a large file.

While I opened too small files I killed the tar-mode buffer, ejected the disk, created a larger file, attached the disk again, and opened from freshened dired listing (in my PATH only a gls exists which accepts --dired) the tar file with v. This produced failures like these:

	emacs(84226) malloc: *** mmap(size=2113650688) failed (error code=12)
	*** error: can't allocate region
	*** set a breakpoint in malloc_error_break to debug
	
	emacs(84357) malloc: *** mmap(size=2126585856) failed (error code=12)
	*** error: can't allocate region
	*** set a breakpoint in malloc_error_break to debug
	
	emacs(84410) malloc: *** mmap(size=2128408576) failed (error code=12)
	*** error: can't allocate region
	*** set a breakpoint in malloc_error_break to debug
	
	emacs(84517) malloc: *** mmap(size=2136588288) failed (error code=12)
	*** error: can't allocate region
	*** set a breakpoint in malloc_error_break to debug
	
	emacs(84576) malloc: *** mmap(size=2140327936) failed (error code=12)
	*** error: can't allocate region
	*** set a breakpoint in malloc_error_break to debug

Is this expected behaviour? I would have thought GNU Emacs could clear all for the file allocated memory... There were still 8 or 9 GB disk space available for swap/vm to grow, so this could not have caused the failures. (I killed GNU Emacs and launched a new one for each new test.) Or should I send a new bug report?

(I can, some rather rainy time, repeat the tests on Mac OS X 10.4.11...)

--
Greetings

  Pete

Work is the curse of the drinking class.
				– Oscar Wilde

bug#8884: closed (Re: bug#8884: wide-int crash)

[Paul Eggert]

On 10/18/11 14:41, Peter Dyballa wrote:

> With Mac OS X 10.5.8 and a PowerPC 7447A CPU GNU Emacs 24.0.90
> launched with -Q can open (tar) files up to 2,047 MB. Ls reports
> that 2,147,450,880 is OK and 2,148,024,320 is too much (2,048
> MB). 2**31 is 2,147,483,648.

Yes, that sounds about right.  There's a ~2**31 byte buffer size
limit due to PTRDIFF_MAX, and Emacs itself enforces this limit.

> 	emacs(84226) malloc: *** mmap(size=2113650688) failed (error code=12)
> 	*** error: can't allocate region
> 	*** set a breakpoint in malloc_error_break to debug

I'm not familiar with this error format, but I expect it's some
Mac OS X thing.  Possibly Mac OS malloc stops working around 2 GiB.
Or perhaps your applications have a system-imposed limit
(try the shell command "ulimit -a").

> I would have thought GNU Emacs could clear all for the file
> allocated memory... There were still 8 or 9 GB disk space available
> for swap/vm to grow,

A single 32-bit process can't possibly allocate more than 4 GiB of
address space, so as long as there's plenty of swap space (which
there appears to be), it's an address-space limit you're running
into, not a swap-space limit.

bug#8884: closed (Re: bug#8884: wide-int crash)

[Peter Dyballa]

Am 19.10.2011 um 03:47 schrieb Paul Eggert:

> I'm not familiar with this error format, but I expect it's some
> Mac OS X thing.

Likely. Google seems to indicate this.

> Or perhaps your applications have a system-imposed limit
> (try the shell command "ulimit -a").

Not likely:

	core file size          (blocks, -c) unlimited
	data seg size           (kbytes, -d) unlimited
	file size               (blocks, -f) unlimited
	max locked memory       (kbytes, -l) unlimited
	max memory size         (kbytes, -m) unlimited
	open files                      (-n) 256
	pipe size            (512 bytes, -p) 1
	stack size              (kbytes, -s) 8192
	cpu time               (seconds, -t) unlimited
	max user processes              (-u) 266
	virtual memory          (kbytes, -v) unlimited


> A single 32-bit process can't possibly allocate more than 4 GiB of
> address space, so as long as there's plenty of swap space (which
> there appears to be), it's an address-space limit you're running
> into, not a swap-space limit.


It seems the limit starts to be turned on around 3 GiB... From here it  
succeeds or does not succeed to load a file greater than 10 MiB. It  
even plays a role whether I load the 1 GiB or the 2 GiB file first...

--
Greetings

   Pete

A common mistake that people make when trying to design something  
completely foolproof is to underestimate the ingenuity of complete  
fools.
	- Douglas Adams, »Mostly Harmless«

bug#8884: closed (Re: bug#8884: wide-int crash)

[Stefan Monnier]

> While I opened too small files I killed the tar-mode buffer, ejected the
> disk, created a larger file, attached the disk again, and opened from
> freshened dired listing (in my PATH only a gls exists which accepts --dired)
> the tar file with v. This produced failures like these:

> 	emacs(84226) malloc: *** mmap(size=2113650688) failed (error code=12)
> 	*** error: can't allocate region
> 	*** set a breakpoint in malloc_error_break to debug
	
It may simply be due to fragmentation: while there may still be more
than 2GB of unused memory, there may not be a single contiguous unused
chunk of 2GB.


        Stefan

bug#8884: closed (Re: bug#8884: wide-int crash)

[Peter Dyballa]

Am 19.10.2011 um 05:43 schrieb Stefan Monnier:

> It may simply be due to fragmentation: while there may still be more
> than 2GB of unused memory, there may not be a single contiguous unused
> chunk of 2GB.

Not likely. VM is on a dedicated disk volume of almost 12 GB. It  
happened that I used (faulty) CPAN/Perl related tools that allocated 7  
or 8 GB until I could kill them.

--
Greetings

   Pete

One cannot live by television, video games, top ten CDs, and dumb  
movies alone.
				– Amiri Baraka, 1999

bug#8884: closed (Re: bug#8884: wide-int crash)

[Stefan Monnier]

>> It may simply be due to fragmentation: while there may still be more
>> than 2GB of unused memory, there may not be a single contiguous unused
>> chunk of 2GB.
> Not likely.

Why so?

> VM is on a dedicated disk volume of almost 12 GB.

Irrelevant.  The fragmentation is in the process's address space
(limited at most to 4GB).  Think of the following scenario:

1- start Emacs:
   30MB used, + (4096 - 30)MB free.
2- load 2GB file:
   30MB + 2048MB + 2018MB free
3- look at the file, causing some allocation:
   30MB + 2048MB + 1MB + 2017MB free
4- delete the large buffer
   30MB + 2048MB free + 1MB + 2017MB free
5- run a few minor commands, e.g. visit some small files.
   32MB + 2046MB free + 1MB + 2017MB free
6- try to load 2G file again
   Oops there's no hole large enough to put the 2048MB file.


-- Stefan

bug#8884: closed (Re: bug#8884: wide-int crash)

[Peter Dyballa]

Am 19.10.2011 um 16:44 schrieb Stefan Monnier:

> Think of the following scenario:


Why did it work better, without that failure, when I killed previous  
GNU Emacs from last test and launched a new one for new test? Then the  
OS would have allocated VM in a similar manner. But I never hit that  
situation.

I'm trying to find a means to observe whether GNU Emacs really frees  
the once allocated memory...

--
Greetings

   Pete

Bake pizza not war!

bug#8884: closed (Re: bug#8884: wide-int crash)

[Stefan Monnier]

>> Think of the following scenario:
> Why did it work better, without that failure, when I killed previous GNU
> Emacs from last test and launched a new one for new test? Then the OS would
> have allocated VM in a similar manner. But I never hit that situation.

No, again the VM is irrelevant.  When a start a new Emacs process, its
address space is completely clean (and in "30MB used and 4066MB free",
even if you only have 100MB of space left in your swap space).

You really need to read up on how virtual memory and process memory works.


        Stefan

bug#8884: closed (Re: bug#8884: wide-int crash)

[Peter Dyballa]

Am 19.10.2011 um 20:00 schrieb Stefan Monnier:

> You really need to read up on how virtual memory and process memory  
> works.

Those in GNU Emacs? Experiments seem to indicate that its behaviour is  
not so easy to predict...

In Mac OS X the swap files are created in a normal formatted file  
system (HFS+ or HFSX). The first file has 64 MiB, the second as well,  
the third 128 MiB, the fourth one 256 MiB, the fifth one 512 MiB, and  
all the others following have each 1 GiB size. It always succeeds to  
allocate a big chunk of disk space.

--
Greetings

   Pete

Don't force it; get a larger hammer.
				– Anthony's Law of Force

bug#8884: closed (Re: bug#8884: wide-int crash)

[Stefan Monnier]

>> You really need to read up on how virtual memory and process memory works.
> Those in GNU Emacs?

No, the general concepts as they're used in Unixy (and Windows)
operating systems.


        Stefan