bug#11236: 24.1.50; Maximum buffer size exceeded

bug#11236: 24.1.50; Maximum buffer size exceeded

[Peter Dyballa]

Hello!

When I try 'make bootstrap' on PPC Mac OS X 10.5.8 (Leopard) for a  
GTK2 or Athena/libXaw3d X client or the NS variant the build process  
each time ends here:

	Loading loadup.el (source)...
	Using load-path (.../emacs-24.1.50/lisp .../emacs-24.1.50/lisp/emacs- 
lisp .../emacs-24.1.50/lisp/language .../emacs-24.1.50/lisp/ 
international .../emacs-24.1.50/lisp/textmodes)
	Loading emacs-lisp/byte-run (source)...
	Loading emacs-lisp/backquote (source)...
	Loading subr (source)...
	Loading version.el (source)...
	Loading widget (source)...
	Loading custom (source)...
	Loading emacs-lisp/map-ynp (source)...
	Loading international/mule (source)...
	Loading international/mule-conf (source)...
	Loading env (source)...
	Loading format (source)...
	Loading bindings (source)...
	Loading cus-start (source)...
	Loading window (source)...
	Loading .../emacs-24.1.50/lisp/files.el (source)...
	Maximum buffer size exceeded
	make[2]: *** [bootstrap-emacs] Error 1
	make[1]: *** [src] Error 2
	make: *** [bootstrap] Error 2

I am now at

	revno: 107887
	branch nick: trunk
	timestamp: Fri 2012-04-13 06:17:25 -0400

I tried to configure with and without --with-wide-int, GCC used is  
Apple's version 4.2. The configuration is performed like for example  
this (this time to use Apple's LLVM GCC 4.2):

	time nice +11 env LANG=C PATH=/Developer/usr/bin:/sw/bin:$PATH ./ 
configure --without-sound --without-dbus --without-pop --without-gconf  
--without-gpm --without-gsettings --without-selinux --without-gif -- 
without-jpeg --without-png --without-rsvg --without-tiff --without-xpm  
--with-x --x-libraries=/usr/X11/lib --x-includes=/usr/X11/include -- 
with-x-toolkit=athena --enable-locallisppath=/Library/Application\  
Support/Emacs/calendar24:/Library/Application\ Support/Emacs CFLAGS="- 
g -H -pipe -fPIC -fno-common -fast -pthread -mfused-madd -mmultiple - 
ftree-vectorize -mpowerpc-gfxopt -maltivec -faltivec -mabi=altivec - 
mcpu=7450 -mtune=7450" LDFLAGS="-Wl,-dead_strip_dylibs -Wl,- 
bind_at_load -Wl,-t" PKG_CONFIG_PATH=/sw/lib/xft2/lib/pkgconfig:/sw/ 
share/pkgconfig:/sw/lib/pkgconfig:/usr/X11/lib/pkgconfig:/usr/X11/ 
share/pkgconfig:/usr/lib/pkgconfig

--
Greetings

   Pete

Email is a wonderful thing for people whose role in life is to be on  
top of things. But not for me; my role is to be on the bottom of  
things. What I do takes long hours of studying and uninterruptible  
concentration.
				– Donald Knuth

bug#11236: 24.1.50; Maximum buffer size exceeded

[Eli Zaretskii]

> From: Peter Dyballa <Peter_Dyballa@Freenet.DE>
> Date: Fri, 13 Apr 2012 16:04:29 +0200
> 
> When I try 'make bootstrap' on PPC Mac OS X 10.5.8 (Leopard) for a  
> GTK2 or Athena/libXaw3d X client or the NS variant the build process  
> each time ends here:
> 
> 	Loading loadup.el (source)...
> 	Using load-path (.../emacs-24.1.50/lisp .../emacs-24.1.50/lisp/emacs- 
> lisp .../emacs-24.1.50/lisp/language .../emacs-24.1.50/lisp/ 
> international .../emacs-24.1.50/lisp/textmodes)
> 	Loading emacs-lisp/byte-run (source)...
> 	Loading emacs-lisp/backquote (source)...
> 	Loading subr (source)...
> 	Loading version.el (source)...
> 	Loading widget (source)...
> 	Loading custom (source)...
> 	Loading emacs-lisp/map-ynp (source)...
> 	Loading international/mule (source)...
> 	Loading international/mule-conf (source)...
> 	Loading env (source)...
> 	Loading format (source)...
> 	Loading bindings (source)...
> 	Loading cus-start (source)...
> 	Loading window (source)...
> 	Loading .../emacs-24.1.50/lisp/files.el (source)...
> 	Maximum buffer size exceeded
> 	make[2]: *** [bootstrap-emacs] Error 1
> 	make[1]: *** [src] Error 2
> 	make: *** [bootstrap] Error 2

Can you run under a debugger, but a breakpoint inside buffer_overflow,
and when it breaks, look around in make_gap_larger and see why exactly
this overflow happens?  E.g., what are the values of current_size and
of nbytes_added?  Please compile without optimizations, to make
debugging reliable.

bug#11236: 24.1.50; Maximum buffer size exceeded

[Peter Dyballa]

Am 13.04.2012 um 17:31 schrieb Eli Zaretskii:

> Please compile without optimizations, to make debugging reliable.


Compiling with -O0 temacs survives loading lisp/files.el...

Will add --with-wide-int and just 'make' for the next round.

--
Greetings

   Pete

For some reason, this fortune reminds everyone of Marvin Zelkowitz.

bug#11236: 24.1.50; Maximum buffer size exceeded

[Eli Zaretskii]

> Cc: 11236@debbugs.gnu.org
> From: Peter Dyballa <Peter_Dyballa@Freenet.DE>
> Date: Sat, 14 Apr 2012 12:57:51 +0200
> 
> 
> Am 13.04.2012 um 17:31 schrieb Eli Zaretskii:
> 
> > Please compile without optimizations, to make debugging reliable.
> 
> 
> Compiling with -O0 temacs survives loading lisp/files.el...

I feared this is what will happen.  It could be a compiler bug, but to
make sure it isn't an Emacs bug, I would suggest to look at the
machine code emitted by the compiler in an optimized build at the
point of the fatal error.

bug#11236: 24.1.50; Maximum buffer size exceeded

[Peter Dyballa]

Am 14.04.2012 um 13:31 schrieb Eli Zaretskii:

> It could be a compiler bug, but to
> make sure it isn't an Emacs bug, I would suggest to look at the
> machine code emitted by the compiler in an optimized build at the
> point of the fatal error.

I decided to perform something I understand: I modified the function  
buffer_overflow () to report where it was called. It happens in src/ 
fileio.c on line #3424, so buf_growth_max < likely_growth is true.

--
Greetings

   Pete

"A TRUE Klingon warrior does not comment his code."

bug#11236: 24.1.50; Maximum buffer size exceeded

[Eli Zaretskii]

> Cc: 11236@debbugs.gnu.org
> From: Peter Dyballa <Peter_Dyballa@Freenet.DE>
> Date: Mon, 16 Apr 2012 16:59:47 +0200
> 
> 
> Am 14.04.2012 um 13:31 schrieb Eli Zaretskii:
> 
> > It could be a compiler bug, but to
> > make sure it isn't an Emacs bug, I would suggest to look at the
> > machine code emitted by the compiler in an optimized build at the
> > point of the fatal error.
> 
> I decided to perform something I understand: I modified the function  
> buffer_overflow () to report where it was called. It happens in src/ 
> fileio.c on line #3424, so buf_growth_max < likely_growth is true.

Can you show all the other variables involved in the calculations
leading to line 3424 of fileio.c?

bug#11236: 24.1.50; Maximum buffer size exceeded

[Peter Dyballa]

Am 16.04.2012 um 18:54 schrieb Eli Zaretskii:

>> I decided to perform something I understand: I modified the function
>> buffer_overflow () to report where it was called. It happens in src/
>> fileio.c on line #3424, so buf_growth_max < likely_growth is true.
>
> Can you show all the other variables involved in the calculations
> leading to line 3424 of fileio.c?


The GUD documentation mentions a few key bindings – C-x C-a C-p can  
(cleverly) print values!

	(gdb) pr buf_growth_max
	The history is empty.
	buf_growth_max < likely_growth = $1 = 1
	buf_growth_max = $2 = 2147483646
	likely_growth = $3 = 5425793530331136
	likely_end = $4 = 5425793530331136
	beg_offset = $5 = 0
	BUF_BYTES_MAX = No symbol "BUF_BYTES_MAX" in current context.
	buf_bytes = $6 = 1
	not_regular = $7 = 0
	end_offset = $8 = 5425793530331136
	st.st_size = $9 = 5425793530331136

The values of likely_growth, likely_end, end_offset, and st.st_size  
are the same, ≈ 5·10^15 – this is an unlikely value for a 50 GB  
partition on an 80 GB disk...

I configured --with-wide-int. When GDB hits the breakpoint it prints  
out:

	Breakpoint 2, Finsert_file_contents (filename=-9223372036828660272,  
visit=4611686018452566072, beg=4611686018452566072,  
end=4611686018452566072, replace=4611686018452566072) at fileio.c:3424

It seems to me that the variables visit, beg, end, and replace, all  
equal, are byte positions in the file – but almost 5·10^18 cannot be  
correct. These values come partially from the struct st, which has:

   st_dev = 234881028,
   st_mode = 33188,
   st_nlink = 1,
   st_ino = 43973072,
   st_uid = 501,
   st_gid = 80,
   st_rdev = 0,

   st_size = 5425793530331136,
   st_blocks = 10617159159808,

Ls delivers:

	gls -lin lisp/loaddefs.el
	43973072 -rw-r--r-- 1 501 80 1263291 16. Apr 10:40 lisp/loaddefs.el

So some values are OK. And it's obvious that the error seems to happen  
earlier, I think, when the file is opened – the DEFUN ("insert-file- 
contents", ...) is not opening the file. The variable  
Sinsert_file_contents, one of the DEFUN's arguments, has:

   size = 4611686018427404288,

I can dig further... with some help.

--
Greetings

   Pete

Theory and practice are the same, in theory, but, in practice, they  
are different.

bug#11236: 24.1.50; Maximum buffer size exceeded

[Andreas Schwab]

Peter Dyballa <Peter_Dyballa@Freenet.DE> writes:

> 	(gdb) pr buf_growth_max

pr doesn't take an argument, use pp instead.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#11236: 24.1.50; Maximum buffer size exceeded

[Peter Dyballa]

Am 16.04.2012 um 22:39 schrieb Andreas Schwab:

> pr doesn't take an argument, use pp instead.


Yes! This pp works. Thank you!

--
Mit friedvollen Grüßen

   Pete

Hard Disk, n.:
	A device that allows users to delete vast quantities of data with  
simple mnemonic commands.

bug#11236: 24.1.50; Maximum buffer size exceeded

[Peter Dyballa]

Am 13.04.2012 um 17:31 schrieb Eli Zaretskii:

> Can you run under a debugger, but a breakpoint inside buffer_overflow,
> and when it breaks, look around in make_gap_larger and see why exactly
> this overflow happens?

I did that, and the gdb stopped at the proper moment and showed the  
source code of the function buffer_overflow () at insdel.c:391 in a  
new window:

	Loading window (source)...
	Loading .../emacs-24.1.50/lisp/files.el (source)...
	
	Breakpoint 2, buffer_overflow () at insdel.c:391


> E.g., what are the values of current_size and
> of nbytes_added?  Please compile without optimizations, to make
> debugging reliable.


With optimisation on not that much is shown:

	(gdb) bt
	#0  buffer_overflow () at insdel.c:391
	#1  0x0017c028 in Finsert_file_contents (filename=34582793,  
visit=33560610, beg=<value temporarily unavailable, due to  
optimizations>, end=33560610, replace=-2113924036) at fileio.c:3424
	#2  0x001d5b20 in eval_sub (form=33560610) at eval.c:2297
	#3  0x001d725c in Flet (args=21916334) at eval.c:364
	#4  0x001d5be0 in eval_sub (form=21916350) at eval.c:2231
	#5  0x001d6afc in Fprogn (args=<value temporarily unavailable, due to  
optimizations>) at eval.c:364
	#6  0x001c2c08 in Fsave_current_buffer (args=21997158) at editfns.c:987
	#7  0x001d5be0 in eval_sub (form=21997134) at eval.c:2231
	#8  0x001d59f8 in eval_sub (form=21916390) at eval.c:2344
	#9  0x001d725c in Flet (args=21916398) at eval.c:364
	#10 0x001d5be0 in eval_sub (form=21916478) at eval.c:2231
	#11 0x001d60d8 in Funwind_protect (args=21916030) at eval.c:1304
	#12 0x001d5be0 in eval_sub (form=21916486) at eval.c:2231
	#13 0x001d6efc in FletX (args=21886046) at eval.c:364
	#14 0x001d5be0 in eval_sub (form=21886238) at eval.c:2231
	#15 0x001d6d2c in Fif (args=<value temporarily unavailable, due to  
optimizations>) at eval.c:364
	#16 0x001d5be0 in eval_sub (form=21886406) at eval.c:2231
	#17 0x001d7c8c in funcall_lambda (fun=21915766, nargs=4,  
arg_vector=0xbfffdbfc) at eval.c:364
	#18 0x001d22fc in Ffuncall (nargs=<value temporarily unavailable, due  
to optimizations>, args=<value temporarily unavailable, due to  
optimizations>) at eval.c:2996
	#19 0x001d28dc in call4 (fn=<value temporarily unavailable, due to  
optimizations>, arg1=<value temporarily unavailable, due to  
optimizations>, arg2=<value temporarily unavailable, due to  
optimizations>, arg3=<value temporarily unavailable, due to  
optimizations>, arg4=<value temporarily unavailable, due to  
optimizations>) at eval.c:2753
	#20 0x0020d388 in Fload (file=<value temporarily unavailable, due to  
optimizations>, noerror=33560610, nomessage=33560610, nosuffix=<value  
temporarily unavailable, due to optimizations>, must_suffix=<value  
temporarily unavailable, due to optimizations>) at lread.c:1256
	#21 0x001d5b20 in eval_sub (form=33560610) at eval.c:2297
	#22 0x0020c55c in readevalloop (readcharfun=33623138,  
stream=0xa04e04c8, sourcename=33709753, printflag=0, unibyte=<value  
temporarily unavailable, due to optimizations>, readfun=33560610,  
start=33560610, end=33560610) at lread.c:1838
	#23 0x0020d850 in Fload (file=33709625, noerror=<value temporarily  
unavailable, due to optimizations>, nomessage=33560610,  
nosuffix=<value temporarily unavailable, due to optimizations>,  
must_suffix=<value temporarily unavailable, due to optimizations>) at  
lread.c:1316
	#24 0x001d5b20 in eval_sub (form=33560610) at eval.c:2297
	#25 0x001d5fbc in Feval (form=21684678, lexical=<value temporarily  
unavailable, due to optimizations>) at eval.c:2137
	#26 0x001d0010 in internal_condition_case (bfun=0x1376a0  
<top_level_2>, handlers=33589130, hfun=0x13fef0 <cmd_error>) at eval.c: 
1448
	#27 0x00137728 in top_level_1 (ignore=<value temporarily unavailable,  
due to optimizations>) at keyboard.c:1177
	#28 0x001cfea8 in internal_catch (tag=<value temporarily unavailable,  
due to optimizations>, func=0x1376d0 <top_level_1>, arg=33560610) at  
eval.c:1205
	#29 0x001375dc in recursive_edit_1 () at keyboard.c:1132
	#30 0x001403a8 in Frecursive_edit () at keyboard.c:823
	#31 0x001368bc in main (argc=<value temporarily unavailable, due to  
optimizations>, argv=0xbffff244) at emacs.c:1711
	
	Lisp Backtrace:
	"insert-file-contents" (0xbfffd160)
	"let" (0xbfffd2ac)
	"save-current-buffer" (0xbfffd41c)
	"with-current-buffer" (0xbfffd4ec)
	"let" (0xbfffd65c)
	"unwind-protect" (0xbfffd79c)
	"let*" (0xbfffd8ec)
	"if" (0xbfffda0c)
	"load-with-code-conversion" (0xbfffdbfc)
	"load" (0xbfffe1a0)
	"load" (0xbfffe820)
	(gdb) pr current_size
	The history is empty.
	(gdb) pr nbytes_added
	The history is empty.
	(gdb) The program is running.  Exit anyway? (y or n) y
	
	Debugger finished

I think this was a Friday the 13th issue... because I can't reproduce  
the failure with and without optimisation. Maybe I need to fail  
rebuilding the NS variant first, which was my start yesterday (the NS  
variant installs a /usr/local/bin/emacs and a /usr/local/bin/emacs- 
yversion> that cannot be used as is, so they need to be overwritten  
with usable stuff)

--
Greetings

   Pete

When confronted with actual numbers, a mathematician is at a loss.
				– Steffen Hokland

bug#11236: 24.1.50; Maximum buffer size exceeded

[Peter Dyballa]

Am 13.04.2012 um 17:31 schrieb Eli Zaretskii:

> Can you run under a debugger, but a breakpoint inside buffer_overflow,
> and when it breaks, look around in make_gap_larger and see why exactly
> this overflow happens?  E.g., what are the values of current_size and
> of nbytes_added?  Please compile without optimizations, to make
> debugging reliable.


The NS variant produces the failure when compiling with -fast. With - 
Os and less the build is OK. When I configure --with-wide-int the  
failure becomes:

	Loading window...
	Loading files...
	Loading cus-face...
	Loading faces...
	Loading button...
	Loading startup...
	Loading .../emacs-24.1.50/lisp/loaddefs.el (source)...

If I spend some time I can manually make Apple's GCC 4.2.1 to compile  
as with -fast, because all the switches are documented (-O3 plus many  
extras plus optimisation for the 64-bit G5 processor/IBM PowerPC 970  
variants compensated by -mcpu=7450 -mtune=G4) and find out which  
switch causes the defect...

--
Greetings

   Pete

We need a president who's fluent in at least one language.
				– Buck Henry

bug#11236: 24.1.50; Maximum buffer size exceeded

[Peter Dyballa]

Am 13.04.2012 um 17:31 schrieb Eli Zaretskii:

> Can you run under a debugger, but a breakpoint inside buffer_overflow,
> and when it breaks, look around in make_gap_larger and see why exactly
> this overflow happens?  E.g., what are the values of current_size and
> of nbytes_added?


The value returned is consistently and always: "The history is empty."

The culprit compiler switch is: -malign-natural. I'll configure the X  
client version (Xaw3d) with that switch and see if it behaves the  
same. If that fails I'll reconfigure and recompile the NS variant, now  
with the -ggdb3 or -ggdb switch (default extended format), and maybe  
with -gdwarf-2, to name the native debug information format of Mac OS X.

--
Greetings

   Pete

I love deadlines. I love the whooshing noise they make as they go by.
				– Douglas Adams

bug#11236: 24.1.50; Maximum buffer size exceeded

[Glenn Morris]

Peter Dyballa wrote:

> The culprit compiler switch is: -malign-natural.

Your life would probably be a lot easier if you didn't try all these
unusual compilation options. Ref bugs 10749, 9927, 9705, 8810, 6939,
5745, 5606, ...

bug#11236: 24.1.50; Maximum buffer size exceeded

[Peter Dyballa]

Am 16.4.2012 um 01:48 schrieb Glenn Morris:

>> The culprit compiler switch is: -malign-natural.
> 
> Your life would probably be a lot easier if you didn't try all these
> unusual compilation options.

Ten years old hardware needs some optimisation to keep pace with recent software...  -malign-natural makes reading from memory faster. Besides, it's just one of almost 20 switches added to -O3 that build up the Apple convenience switch -fast, which I am using by default.

I'd say, my life would be easier if GNU Emacs would not become uncompilable from time to time...

--
Greetings
         
  Pete                           <]
             o        __o         |__    o       HPV, the real
    ___o    /I       -\<,         |o \  -\),-%     high speed!
___/\ /\___./ \___...O/ O____.....`-O-'-()--o_________________

bug#11236: 24.1.50; Maximum buffer size exceeded

[Andreas Schwab]

Peter Dyballa <Peter_Dyballa@Freenet.DE> writes:

> Ten years old hardware needs some optimisation to keep pace with recent
> software...  -malign-natural makes reading from memory faster. Besides,
> it's just one of almost 20 switches added to -O3 that build up the Apple
> convenience switch -fast, which I am using by default.

If you want to use a different ABI you must use it for *everything*.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#11236: 24.1.50; Maximum buffer size exceeded

[Peter Dyballa]

Am 16.4.2012 um 12:28 schrieb Andreas Schwab:

> If you want to use a different ABI you must use it for *everything*.

This does not happen – in case I understand you correctly. The libraries from Fink are compiled with GCC 4.2 while Macports has recently switched to LLVM GCC 4.2.

--
Mit friedvollen Grüßen

  Pete

The human brain operates at only 10% of its capacity. The rest is overhead for the operating system.

bug#11236: 24.1.50; Maximum buffer size exceeded

[Glenn Morris]

Peter Dyballa wrote:

> Ten years old hardware needs some optimisation to keep pace with
> recent software...  -malign-natural makes reading from memory faster.

My take on these things is that when the time you spend tracking down
obscure problems exceeds the fraction of a second you save from
optimising, it's time to give up.

> I'd say, my life would be easier if GNU Emacs would not become
> uncompilable from time to time...

Which of the dozen or so compilation bug reports that I referred to
previously still apply? All of them? IMO, the lack of response to many
of them indicates that this is not an area that people are generally
interesting in helping with.

bug#11236: 24.1.50; Maximum buffer size exceeded

[Peter Dyballa]

Am 16.04.2012 um 20:24 schrieb Glenn Morris:

> Which of the dozen or so compilation bug reports that I referred to
> previously still apply? All of them? IMO, the lack of response to many
> of them indicates that this is not an area that people are generally
> interesting in helping with.


I'll check them and see if I can close them!

--
Greetings

   Pete

There's no place like 127.0.0.1
			– origin unknown

bug#11236: 24.1.50; Maximum buffer size exceeded

[Andreas Schwab]

tags 11236 + notabug
quit

Peter Dyballa <Peter_Dyballa@Freenet.DE> writes:

> The culprit compiler switch is: -malign-natural.

Thus not a bug.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#11236: 24.1.50; Maximum buffer size exceeded

[Paul Eggert]

> the struct st, which has:
> 
>   ...
>   st_size = 5425793530331136,
>   st_blocks = 10617159159808,
> 
> Ls delivers:
> 
> 	gls -lin lisp/loaddefs.el
> 	43973072 -rw-r--r-- 1 501 80 1263291 16. Apr 10:40 lisp/loaddefs.el

Since 5425793530331136 == 1263291 << 32, this suggests that
there's a mismatch between the ABI that Emacs is assuming
and the ABI that the 'stat' system call is actually using.

Perhaps it's a bug in the way largefile mode is being set up (see
Autoconf's AC_SYS_LARGFILE, which Emacs uses).  Perhaps some code is
compiled in largefile mode, and other code is not -- that's a no-no.
Or, as Andreas Schwab suggests, if you're mixing code that's compiled
by incompatible compilers, that would cause these symptoms.

One other thought: is Emacs invoking 'stat' directly, or indirectly
via the 'stat' defined in lib/stat.c?  If the latter, perhaps there's
something messed up with the indirection.