bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[Jérémie Courrèges-Anglas]

[-- Attachment #1: Type: text/plain, Size: 10167 bytes --]

Hi.

Sorry for being late.  The RC1 archive fails to build on
OpenBSD/i386 -current (post 5.2) and OpenBSD/macppc -stable (5.1).
Emacs 24.1 was building and running properly.

I use
$ ./configure --without-x && gmake
[snip]
EMACSLOADPATH=/home/jca/src/emacs-24.2/leim/../lisp LC_ALL=C ../src/emacs -batch --no-site-file --no-site-lisp  -l /home/jca/src/emacs-24.2/leim/../lisp/international/qua
il \
  -f batch-byte-compile-if-not-done quail/CCDOSPY.el quail/Punct.el quail/QJ.el quail/SW.el quail/TONEPY.el quail/4Corner.el quail/ARRAY30.el quail/ECDICT.el quail/ETZY.e
l quail/Punct-b5.el quail/PY-b5.el quail/QJ-b5.el quail/ZOZY.el quail/tsang-b5.el quail/quick-b5.el quail/tsang-cns.el quail/quick-cns.el quail/PY.el quail/ZIRANMA.el qua
il/CTLau.el quail/CTLau-b5.el
if [ x`(cd /home/jca/src/emacs-24.2/leim && /bin/pwd)` = x`(/bin/pwd)` ] ; then \
  EMACSLOADPATH=/home/jca/src/emacs-24.2/leim/../lisp LC_ALL=C ../src/emacs -batch --no-site-file --no-site-lisp -l /home/jca/src/emacs-24.2/leim/../lisp/international/qu
ail \
    --eval "(update-leim-list-file \".\")" ; \
else \
  EMACSLOADPATH=/home/jca/src/emacs-24.2/leim/../lisp LC_ALL=C ../src/emacs -batch --no-site-file --no-site-lisp -l /home/jca/src/emacs-24.2/leim/../lisp/international/qu
ail \
    --eval "(update-leim-list-file \".\" \"/home/jca/src/emacs-24.2/leim\")" ; \
fi
Updating /home/jca/src/emacs-24.2/leim/leim-list.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/PY.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/quick-cns.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/tsang-cns.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/ZIRANMA.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/CTLau-b5.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/CTLau.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/quick-b5.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/tsang-b5.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/ZOZY.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/TONEPY.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/SW.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/QJ.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/QJ-b5.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/Punct.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/Punct-b5.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/PY-b5.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/ETZY.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/ECDICT.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/CCDOSPY.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/ARRAY30.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/4Corner.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/indian.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/ipa.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/latin-post.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/czech.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/japanese.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/thai.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/arabic.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/hanja3.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/latin-ltx.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/hanja-jis.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/hebrew.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/symbol-ksc.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/hangul.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/lao.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/georgian.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/croatian.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/persian.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/hanja.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/slovak.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/lrt.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/tibetan.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/pypunct-b5.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/sgml-input.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/cyril-jis.el ...
Fatal error (11)Segmentation fault (core dumped) 
gmake[1]: *** [leim-list.el] Error 139
gmake[1]: Leaving directory `/home/jca/src/emacs-24.2/leim'
gmake: *** [leim] Error 2
$ 

The backtrace looks like this (slightly mangled by copy/paste), using
the default `-g -O2' CFLAGS on OpenBSD/powerpc:

opyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "powerpc-unknown-openbsd5.1"...
Core was generated by `emacs'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/lib/libossaudio.so.3.1...done.
Loaded symbols for /usr/lib/libossaudio.so.3.1
Reading symbols from /usr/local/lib/libdbus-1.so.9.1...done.
Loaded symbols for /usr/local/lib/libdbus-1.so.9.1
Reading symbols from /usr/lib/libncurses.so.12.1...done.
Loaded symbols for /usr/lib/libncurses.so.12.1
Reading symbols from /usr/lib/libpthread.so.13.3...done.
Loaded symbols for /usr/lib/libpthread.so.13.3
Reading symbols from /usr/lib/libm.so.7.0...done.
Loaded symbols for /usr/lib/libm.so.7.0
Reading symbols from /usr/lib/libc.so.62.0...done.
Loaded symbols for /usr/lib/libc.so.62.0
Reading symbols from /usr/libexec/ld.so...done.
Loaded symbols for /usr/libexec/ld.so
#0  free_bloc (bloc=0x1b8fa80) at ralloc.c:719
719       if (heap->first_bloc == bloc)
(gdb) bt
#0  free_bloc (bloc=0x1b8fa80) at ralloc.c:719
#1  0x01968d5c in r_alloc_free (ptr=0x2381608) at ralloc.c:939
#2  0x018baf1c in Fkill_buffer (buffer_or_name=Variable "buffer_or_name" is not available.
) at buffer.c:4845
#3  0x0190a630 in Ffuncall (nargs=2, args=Variable "args" is not available.
) at eval.c:3001
#4  0x01945070 in exec_byte_code (bytestr=Variable "bytestr" is not available.
) at bytecode.c:785
#5  0x019099c8 in eval_sub (form=Variable "form" is not available.
) at eval.c:2355
#6  0x01909ccc in Fprogn (args=Variable "args" is not available.
) at eval.c:364
#7  0x0190852c in unbind_to (count=Variable "count" is not available.
) at eval.c:3433
#8  0x01944fdc in exec_byte_code (bytestr=Variable "bytestr" is not available.
) at bytecode.c:807
#9  0x01909fb8 in funcall_lambda (fun=31484933, nargs=1, arg_vector=0xffff90c8)
    at eval.c:3232
#10 0x0190a3b4 in Ffuncall (nargs=2, args=Variable "args" is not available.
) at eval.c:3062
#11 0x0190acc8 in Fapply (nargs=2, args=0xffff90c4) at eval.c:2453
#12 0x0190a6d0 in Ffuncall (nargs=3, args=Variable "args" is not available.
) at eval.c:2983
#13 0x01945070 in exec_byte_code (bytestr=Variable "bytestr" is not available.
) at bytecode.c:785
#14 0x01909fb8 in funcall_lambda (fun=27442485, nargs=1, arg_vector=0xffff91d0)
    at eval.c:3232
#15 0x0190b878 in apply_lambda (fun=27442485, args=Variable "args" is not available.
) at eval.c:3109
#16 0x01909788 in eval_sub (form=Variable "form" is not available.
) at eval.c:2413
#17 0x0190c438 in Feval (form=29386182, lexical=Variable "lexical" is not available.
) at eval.c:2203
#18 0x0190a61c in Ffuncall (nargs=2, args=Variable "args" is not available.
) at eval.c:3004
#19 0x01945070 in exec_byte_code (bytestr=Variable "bytestr" is not available.
) at bytecode.c:785
#20 0x01909dd4 in funcall_lambda (fun=27219261, nargs=1, arg_vector=0xffff9564)
    at eval.c:3166
#21 0x0190a3b4 in Ffuncall (nargs=2, args=Variable "args" is not available.
) at eval.c:3062
#22 0x01945070 in exec_byte_code (bytestr=Variable "bytestr" is not available.
) at bytecode.c:785
#23 0x01909dd4 in funcall_lambda (fun=27206509, nargs=0, arg_vector=0xffff9738)
    at eval.c:3166
#24 0x0190a3b4 in Ffuncall (nargs=1, args=Variable "args" is not available.
) at eval.c:3062
#25 0x01945070 in exec_byte_code (bytestr=Variable "bytestr" is not available.
) at bytecode.c:785
#26 0x01909dd4 in funcall_lambda (fun=27203701, nargs=0, arg_vector=0xffff9850)
    at eval.c:3166
#27 0x0190b878 in apply_lambda (fun=27203701, args=Variable "args" is not available.
) at eval.c:3109
#28 0x01909788 in eval_sub (form=Variable "form" is not available.
) at eval.c:2413
#29 0x0190c438 in Feval (form=29389358, lexical=Variable "lexical" is not available.
) at eval.c:2203
#30 0x0189cbe8 in top_level_2 () at keyboard.c:1169
#31 0x0190d6fc in internal_condition_case (bfun=0x189cbc8 <top_level_2>,
    handlers=29186386, hfun=0x18a0f74 <cmd_error>) at eval.c:1514
#32 0x018a1430 in top_level_1 (ignore=Variable "ignore" is not available.
) at keyboard.c:1177
#33 0x0190d804 in internal_catch (tag=Variable "tag" is not available.
) at eval.c:1271
#34 0x018a1260 in recursive_edit_1 () at keyboard.c:1132
#35 0x018a13d0 in Frecursive_edit () at keyboard.c:823
#36 0x01895718 in main (argc=8, argv=0x0) at emacs.c:1715



The same error happens consistently on both architectures (whether
I use `--without-x' or not).

$ ./configure --without-x REL_ALLOC=no
and
$ ./configure --without-x CFLAGS=-g

both seem to `fix' the problem, but I've only done light testing so far.

Using git bisect, I was able to track the build error up to the
573c8f870aa68b8c5937524e1a4db645026a3240 git commit id:

     Backport: Really fix bug #11519, by fixing the last change in ralloc.c.

      src/ralloc.c (r_alloc_inhibit_buffer_relocation): Fix stupid
      thinko
      in the logic of incrementing and decrementing the value of
      use_relocatable_buffers.

Sorry for not providing the hg commit id.
Reverting that commit actually makes rc1 build again, but I have not
tried to find a proper fix.

On the other hand, rc1 builds and seems to work fine so far on Debian
Squeeze (powerpc). :)
Please don't hesitate if you need further action on my part.

Regards,
-- 
Jérémie Courrèges-Anglas
GPG fingerprint: 61DB D9A0 00A4 67CF 2A90 8961 6191 8FBF 06A1 1494

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[Eli Zaretskii]

> From: jca@wxcvbn.org (Jérémie Courrèges-Anglas)
> Date: Mon, 20 Aug 2012 21:53:16 +0200
> 
> Checking /home/jca/src/emacs-24.2/leim/quail/cyril-jis.el ...
> Fatal error (11)Segmentation fault (core dumped) 
> gmake[1]: *** [leim-list.el] Error 139
> gmake[1]: Leaving directory `/home/jca/src/emacs-24.2/leim'
> gmake: *** [leim] Error 2
> $ 
> [...]
> #0  free_bloc (bloc=0x1b8fa80) at ralloc.c:719
> 719       if (heap->first_bloc == bloc)
> (gdb) bt
> #0  free_bloc (bloc=0x1b8fa80) at ralloc.c:719
> [...]
> Using git bisect, I was able to track the build error up to the
> 573c8f870aa68b8c5937524e1a4db645026a3240 git commit id:
> 
>      Backport: Really fix bug #11519, by fixing the last change in ralloc.c.
> 
>       src/ralloc.c (r_alloc_inhibit_buffer_relocation): Fix stupid
>       thinko
>       in the logic of incrementing and decrementing the value of
>       use_relocatable_buffers.
> 
> Sorry for not providing the hg commit id.
> Reverting that commit actually makes rc1 build again, but I have not
> tried to find a proper fix.

I think I see the problem and will fix it later today.

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[Eli Zaretskii]

> Date: Tue, 21 Aug 2012 06:03:37 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 12242@debbugs.gnu.org
> 
> I think I see the problem and will fix it later today.

Actually, no, I don't think I see the problem.  What does the
following GDB command print in frame #0?

  (gdb) p *heap

Also, could you please run Emacs under GDB, and when it crashes, type
the following commands, which define a new command "heaps", and then
run it?  Like this:

  (gdb) define heaps
  Type commands for definition of "heaps".
  End with a line saying just "end".
  >set $a = first_heap
  >while $a
   >p *$a
   >set $a = $a->next
   >end
  >end
  (gdb) heaps

Please post here everything that GDB prints as result.

Thanks.

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[Jérémie Courrèges-Anglas]

[-- Attachment #1: Type: text/plain, Size: 5772 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Tue, 21 Aug 2012 06:03:37 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 12242@debbugs.gnu.org
>> 
>> I think I see the problem and will fix it later today.
>
> Actually, no, I don't think I see the problem.  What does the
> following GDB command print in frame #0?
>
>   (gdb) p *heap
>
> Also, could you please run Emacs under GDB, and when it crashes, type
> the following commands, which define a new command "heaps", and then
> run it?  Like this:
>
>   (gdb) define heaps
>   Type commands for definition of "heaps".
>   End with a line saying just "end".
>   >set $a = first_heap
>   >while $a
>    >p *$a
>    >set $a = $a->next
>    >end
>   >end
>   (gdb) heaps
>
> Please post here everything that GDB prints as result.

if [ x`(cd /home/emacs24/src/emacs-24.2/leim && /bin/pwd)` = x`(/bin/pwd)` ] ; then \
  env EMACSLOADPATH=/home/emacs24/src/emacs-24.2/leim/../lisp LC_ALL=C gdb --args ../src/emacs --batch --no-site-file --no-site-lisp -l /home/emacs24/src/emacs-24.2/leim/
../lisp/international/quail \
    --eval "(update-leim-list-file \".\")" ; \
else \
  env EMACSLOADPATH=/home/emacs24/src/emacs-24.2/leim/../lisp LC_ALL=C gdb --args ../src/emacs --batch --no-site-file --no-site-lisp -l /home/emacs24/src/emacs-24.2/leim/
../lisp/international/quail \
    --eval "(update-leim-list-file \".\" \"/home/emacs24/src/emacs-24.2/leim\")" ; \
fi
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-openbsd5.2"...
(gdb) run             
Starting program: /home/emacs24/src/emacs-24.2/src/emacs --batch --no-site-file --no-site-lisp -l /home/emacs24/src/emacs-24.2/leim/../lisp/international/quail --eval \(u
pdate-leim-list-file\ \".\"\)
Updating /home/emacs24/src/emacs-24.2/leim/leim-list.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/PY.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/quick-cns.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/tsang-cns.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/ZIRANMA.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/CTLau-b5.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/CTLau.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/quick-b5.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/tsang-b5.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/ZOZY.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/TONEPY.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/SW.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/QJ.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/QJ-b5.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/Punct.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/Punct-b5.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/PY-b5.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/ETZY.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/ECDICT.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/CCDOSPY.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/ARRAY30.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/4Corner.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/indian.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/ipa.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/latin-post.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/czech.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/japanese.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/thai.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/arabic.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/hanja3.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/latin-ltx.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/hanja-jis.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/hebrew.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/symbol-ksc.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/hangul.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/lao.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/georgian.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/croatian.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/persian.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/hanja.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/slovak.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/lrt.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/tibetan.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/pypunct-b5.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/sgml-input.el ...
Checking /home/emacs24/src/emacs-24.2/leim/quail/cyril-jis.el ...

Program received signal SIGSEGV, Segmentation fault.
free_bloc (bloc=0x85eb8a0) at ralloc.c:719
719       if (heap->first_bloc == bloc)
(gdb) p heap
$1 = 0x8f12000
(gdb) p *heap
Cannot access memory at address 0x8f12000
(gdb) define heaps
Type commands for definition of "heaps".
End with a line saying just "end".
>set $a = first_heap
>while $a
 >p *$a
 >set $a = $a->next
 >end
>end
(gdb) heaps
$2 = {next = 0x0, prev = 0x0, start = 0x8edf000, end = 0x8f02000, bloc_start = 0x8eec000, free = 0x8ef2a58, first_bloc = 0x85eb8a0, last_bloc = 0x85eb8a0}
(gdb)

Again, please don't hesitate if you need further input.
-- 
Jérémie Courrèges-Anglas
Empreinte GPG : 61DB D9A0 00A4 67CF 2A90 8961 6191 8FBF 06A1 1494

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[YAMAMOTO Mitsuharu]

>>>>> On Tue, 21 Aug 2012 19:49:52 +0300, Eli Zaretskii <eliz@gnu.org> said:

>> Date: Tue, 21 Aug 2012 06:03:37 +0300 From: Eli Zaretskii
>> <eliz@gnu.org> Cc: 12242@debbugs.gnu.org
>> 
>> I think I see the problem and will fix it later today.

> Actually, no, I don't think I see the problem.

I took a look at ralloc.c a bit, and I thought that the variable
`use_relocatable_buffers' is not designed to be changed temporarily in
the first place.

I think we should use r_alloc_freeze/r_alloc_thaw instead of
r_alloc_inhibit_buffer_relocation calls.  In that case, we have to
estimate how much memory can be malloc'ed while the relocation is
frozen, so that we can pass an appropriate number to r_alloc_freeze.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[Eli Zaretskii]

> Date: Wed, 22 Aug 2012 11:35:26 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: jca@wxcvbn.org,
> 	12242@debbugs.gnu.org
> 
> I took a look at ralloc.c a bit, and I thought that the variable
> `use_relocatable_buffers' is not designed to be changed temporarily in
> the first place.

Why not?  Can you tell what led you to that conclusion?

My reading of the code is that inhibiting relocation just means that
ralloc.c always asks for more memory when it cannot find a large
enough block in the existing heaps.

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[YAMAMOTO Mitsuharu]

>>>>> On Wed, 22 Aug 2012 06:02:06 +0300, Eli Zaretskii <eliz@gnu.org> said:

>> Date: Wed, 22 Aug 2012 11:35:26 +0900 From: YAMAMOTO Mitsuharu
>> <mituharu@math.s.chiba-u.ac.jp> Cc: jca@wxcvbn.org,
>> 12242@debbugs.gnu.org
>> 
>> I took a look at ralloc.c a bit, and I thought that the variable
>> `use_relocatable_buffers' is not designed to be changed temporarily
>> in the first place.

> Why not?  Can you tell what led you to that conclusion?

> My reading of the code is that inhibiting relocation just means that
> ralloc.c always asks for more memory when it cannot find a large
> enough block in the existing heaps.

For example, `virtual_break_value' is not updated in that case.  It
can lead to an inconsistent state as reported if r_alloc_sbrk is
called with a positive argument via malloc when
use_relocatable_buffers <= 0, and then it is called with a negative
argument via free when use_relocatable_buffers > 0.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[Eli Zaretskii]

> Date: Wed, 22 Aug 2012 12:13:27 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: 	jca@wxcvbn.org,
> 	12242@debbugs.gnu.org
> 
> >>>>> On Wed, 22 Aug 2012 06:02:06 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
> >> Date: Wed, 22 Aug 2012 11:35:26 +0900 From: YAMAMOTO Mitsuharu
> >> <mituharu@math.s.chiba-u.ac.jp> Cc: jca@wxcvbn.org,
> >> 12242@debbugs.gnu.org
> >> 
> >> I took a look at ralloc.c a bit, and I thought that the variable
> >> `use_relocatable_buffers' is not designed to be changed temporarily
> >> in the first place.
> 
> > Why not?  Can you tell what led you to that conclusion?
> 
> > My reading of the code is that inhibiting relocation just means that
> > ralloc.c always asks for more memory when it cannot find a large
> > enough block in the existing heaps.
> 
> For example, `virtual_break_value' is not updated in that case.  It
> can lead to an inconsistent state as reported if r_alloc_sbrk is
> called with a positive argument via malloc when
> use_relocatable_buffers <= 0, and then it is called with a negative
> argument via free when use_relocatable_buffers > 0.

I see your point.

Unfortunately, using r_alloc_freeze/r_alloc_thaw doesn't seem to be
workable in practice, either.  I tried to use it, and the best patch I
could come up with (got me through several bootstraps with different
compiler switches) is below.  It is (a) butt-ugly, and (b) very
fragile, for the reasons I explain below.

Basically, the difficulty is to know which number to pass to
r_alloc_freeze.  The only place that knows how much memory is to be
allocated is the code that actually allocates it.  So I cannot put the
call to r_alloc_freeze in maybe_unify_char, where we now call
r_alloc_inhibit_buffer_relocation, because the memory allocations are
in the functions called from there, and the amount of memory is not
known to the caller in advance.  Moreover, at least one of the
functions called from maybe_unify_char via load_charset allocates
memory in a data-dependent loop, so I think it is in principle
impossible to know how much memory it can ask for.

What's worse, malloc (at least the implementation in gmalloc.c) will
actually allocate more than it was asked for, and sometimes allocates
an additional chunk of memory on top of that to expand its internal
tables where it maintains information about the arena.  The size of
this additional chunk is also data-dependent, so the value I add in
r_alloc_freeze in the patch below is not guaranteed to work, it's
based on what I saw during a bootstrap, with some safety margin added
for good measure.

So it sounds like the only practical way out of this mess is to step
back one notch and talk about bug #11519, which was the cause for
inhibiting relocations while maybe_unify_char is in progress.  At the
time, Handa-san promised to work on removing unify_char, but I guess
that job is not yet done, since it isn't even on the trunk.  Ken'ichi,
what would it take to do this now?

Another alternative would be to rewrite load_charset_map_from_file and
load_charset_map_from_vector so as not to allocate the large structs
they do now.  (These functions also create char-tables, but I think a
char-table is enlarged 256 elements at a time, which doesn't cross the
threshold of "large" allocations that can trigger relocation in
ralloc.c.  Am I missing something?)

Yet another possibility is to disable relocations entirely on all
platforms but MS-Windows (where the crash which triggered this bug
report doesn't happen, probably because there we reserve the memory at
startup in one contiguous block).

Any other suggestions and thoughts are welcome.

Last, but not least: I'm very sorry for this obstacle to making an
emergency release.  It always bewilders me how such problems can lie
dormant for so long, until the most un-opportune moment.

Here's the patch that I DON'T recommend to install:

--- src/ralloc.c~0	2012-06-29 17:50:48.000000000 +0300
+++ src/ralloc.c	2012-08-22 16:59:32.511794000 +0300
@@ -1022,12 +1022,22 @@ r_re_alloc (POINTER *ptr, SIZE size)
 void
 r_alloc_freeze (long int size)
 {
   if (! r_alloc_initialized)
     r_alloc_init ();
 
   /* If already frozen, we can't make any more room, so don't try.  */
   if (r_alloc_freeze_level > 0)
     size = 0;
+  else
+    {
+      /* malloc will usually ask for more than its callers, so ensure
+	 we have some extra room.  */
+      size += (max (__malloc_extra_blocks, 64) + 1) * 4096;
+      /* gmalloc will sometimes need to enlarge its internal tables,
+	 which asks for yet more memory.  */
+      size += size / 4;
+    }
   /* If we can't get the amount requested, half is better than nothing.  */
   while (size > 0 && r_alloc_sbrk (size) == 0)
     size /= 2;
--- src/charset.c~0	2012-06-29 17:50:48.000000000 +0300
+++ src/charset.c	2012-08-22 14:41:55.981714000 +0300
@@ -214,6 +214,10 @@ static struct
    text and a string data may be relocated.  */
 int charset_map_loaded;
 
+/* Flag used to signal to load_charset_* that it is unsafe to relocate
+   buffers (indirectly via calls to rel_alloc_* functions in ralloc.c).  */
+static int in_maybe_unify_char;
+
 struct charset_map_entries
 {
   struct {
@@ -293,7 +297,20 @@ load_charset_map (struct charset *charse
       else
 	{
 	  if (! temp_charset_work)
-	    temp_charset_work = xmalloc (sizeof (*temp_charset_work));
+	    {
+#ifdef REL_ALLOC
+	      /* Allocating heap memory screws callers of this
+		 function through STRING_CHAR_* macros that hold C
+		 pointers to buffer text, if REL_ALLOC is used.  */
+	      if (in_maybe_unify_char)
+		r_alloc_freeze (sizeof (*temp_charset_work));
+#endif
+	      temp_charset_work = xmalloc (sizeof (*temp_charset_work));
+#ifdef REL_ALLOC
+	      if (in_maybe_unify_char)
+		r_alloc_thaw ();
+#endif
+	    }
 	  if (control_flag == 1)
 	    {
 	      memset (temp_charset_work->table.decoder, -1,
@@ -498,8 +515,19 @@ load_charset_map_from_file (struct chars
 
   /* Use SAFE_ALLOCA instead of alloca, as `charset_map_entries' is
      large (larger than MAX_ALLOCA).  */
+#ifdef REL_ALLOC
+  /* The calls to SAFE_ALLOCA below can allocate heap memory, which
+     screws callers of this function through STRING_CHAR_* macros that
+     hold C pointers to buffer text, if REL_ALLOC is used.  */
+  if (in_maybe_unify_char)
+    r_alloc_freeze (sizeof (struct charset_map_entries));
+#endif
   SAFE_ALLOCA (head, struct charset_map_entries *,
 	       sizeof (struct charset_map_entries));
+#ifdef REL_ALLOC
+  if (in_maybe_unify_char)
+    r_alloc_thaw ();
+#endif
   entries = head;
   memset (entries, 0, sizeof (struct charset_map_entries));
 
@@ -530,8 +558,16 @@ load_charset_map_from_file (struct chars
 
       if (n_entries > 0 && (n_entries % 0x10000) == 0)
 	{
+#ifdef REL_ALLOC
+	  if (in_maybe_unify_char)
+	    r_alloc_freeze (sizeof (struct charset_map_entries));
+#endif
 	  SAFE_ALLOCA (entries->next, struct charset_map_entries *,
 		       sizeof (struct charset_map_entries));
+#ifdef REL_ALLOC
+	  if (in_maybe_unify_char)
+	    r_alloc_thaw ();
+#endif
 	  entries = entries->next;
 	  memset (entries, 0, sizeof (struct charset_map_entries));
 	}
@@ -566,8 +602,19 @@ load_charset_map_from_vector (struct cha
 
   /* Use SAFE_ALLOCA instead of alloca, as `charset_map_entries' is
      large (larger than MAX_ALLOCA).  */
+#ifdef REL_ALLOC
+  /* The calls to SAFE_ALLOCA below can allocate heap memory, which
+     screws callers of this function through STRING_CHAR_* macros that
+     hold C pointers to buffer text, if REL_ALLOC is used.  */
+  if (in_maybe_unify_char)
+    r_alloc_freeze (sizeof (struct charset_map_entries));
+#endif
   SAFE_ALLOCA (head, struct charset_map_entries *,
 	       sizeof (struct charset_map_entries));
+#ifdef REL_ALLOC
+  if (in_maybe_unify_char)
+    r_alloc_thaw ();
+#endif
   entries = head;
   memset (entries, 0, sizeof (struct charset_map_entries));
 
@@ -603,8 +650,16 @@ load_charset_map_from_vector (struct cha
 
       if (n_entries > 0 && (n_entries % 0x10000) == 0)
 	{
+#ifdef REL_ALLOC
+	  if (in_maybe_unify_char)
+	    r_alloc_freeze (sizeof (struct charset_map_entries));
+#endif
 	  SAFE_ALLOCA (entries->next, struct charset_map_entries *,
 		       sizeof (struct charset_map_entries));
+#ifdef REL_ALLOC
+	  if (in_maybe_unify_char)
+	    r_alloc_thaw ();
+#endif
 	  entries = entries->next;
 	  memset (entries, 0, sizeof (struct charset_map_entries));
 	}
@@ -1641,13 +1696,9 @@ maybe_unify_char (int c, Lisp_Object val
     return c;
 
   CHECK_CHARSET_GET_CHARSET (val, charset);
-#ifdef REL_ALLOC
-  /* The call to load_charset below can allocate memory, whcih screws
-     callers of this function through STRING_CHAR_* macros that hold C
-     pointers to buffer text, if REL_ALLOC is used.  */
-  r_alloc_inhibit_buffer_relocation (1);
-#endif
+  in_maybe_unify_char++;
   load_charset (charset, 1);
+  in_maybe_unify_char--;
   if (! inhibit_load_charset_map)
     {
       val = CHAR_TABLE_REF (Vchar_unify_table, c);
@@ -1662,9 +1713,6 @@ maybe_unify_char (int c, Lisp_Object val
       if (unified > 0)
 	c = unified;
     }
-#ifdef REL_ALLOC
-  r_alloc_inhibit_buffer_relocation (0);
-#endif
   return c;
 }
 

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[YAMAMOTO Mitsuharu]

>>>>> On Wed, 22 Aug 2012 19:58:12 +0300, Eli Zaretskii <eliz@gnu.org> said:

>>> My reading of the code is that inhibiting relocation just means
>>> that ralloc.c always asks for more memory when it cannot find a
>>> large enough block in the existing heaps.
>> 
>> For example, `virtual_break_value' is not updated in that case.  It
>> can lead to an inconsistent state as reported if r_alloc_sbrk is
>> called with a positive argument via malloc when
>> use_relocatable_buffers <= 0, and then it is called with a negative
>> argument via free when use_relocatable_buffers > 0.

> I see your point.

Sorry, I noticed that the senario I gave above was actually bogus.
Typically free will call r_alloc_sbrk(0) and check the return value
with respect to the area to be reclaimed before calling r_alloc_sbrk
with a negative argument.

Now I don't have a concrete senario to conclude that it is wrong to
change use_relocatable_buffers temporarily.  I'm really sorry if my
previous posts confused you.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> Unfortunately, using r_alloc_freeze/r_alloc_thaw doesn't seem to be
> workable in practice, either.  I tried to use it, and the best patch I
> could come up with (got me through several bootstraps with different
> compiler switches) is below.  It is (a) butt-ugly, and (b) very
> fragile, for the reasons I explain below.
>
> So it sounds like the only practical way out of this mess is to step
> back one notch and talk about bug #11519, which was the cause for
> inhibiting relocations while maybe_unify_char is in progress.  At the
> time, Handa-san promised to work on removing unify_char, but I guess
> that job is not yet done, since it isn't even on the trunk.  Ken'ichi,
> what would it take to do this now?

I don't think it is feasible to rework maybe_unify_char, and test that
fix, and still have a reasonably timely 24.2 release.

Let us consider outright reversion of 108048.  What would be the effect?
Given a choice between a bug (Bug#11519) which is also in 24.1, and
failure-to-compile on OpenBSD as a new regression, the former is far
preferable, unless you can point out some consequence that is more
serious that the discussion in Bug#11519 implies.

Also, if we revert 108048, should we revert 108020 (which is in 24.1) as
well?  Could you explain what problem 108020 causes which 108048 is
supposed to fix?  Does it have symptoms?

Thanks.

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[Eli Zaretskii]

> Date: Thu, 23 Aug 2012 08:12:37 +0900
> From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> Cc: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>,
> 	Stefan Monnier <monnier@iro.umontreal.ca>,
> 	Chong Yidong <cyd@gnu.org>,
> 	Kenichi Handa <handa@m17n.org>,
> 	jca@wxcvbn.org,
> 	12242@debbugs.gnu.org
> 
> >>>>> On Wed, 22 Aug 2012 19:58:12 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
> >>> My reading of the code is that inhibiting relocation just means
> >>> that ralloc.c always asks for more memory when it cannot find a
> >>> large enough block in the existing heaps.
> >> 
> >> For example, `virtual_break_value' is not updated in that case.  It
> >> can lead to an inconsistent state as reported if r_alloc_sbrk is
> >> called with a positive argument via malloc when
> >> use_relocatable_buffers <= 0, and then it is called with a negative
> >> argument via free when use_relocatable_buffers > 0.
> 
> > I see your point.
> 
> Sorry, I noticed that the senario I gave above was actually bogus.
> Typically free will call r_alloc_sbrk(0) and check the return value
> with respect to the area to be reclaimed before calling r_alloc_sbrk
> with a negative argument.
> 
> Now I don't have a concrete senario to conclude that it is wrong to
> change use_relocatable_buffers temporarily.  I'm really sorry if my
> previous posts confused you.

No sweat.  I guess I will need to take a deeper look at the problem.

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[Eli Zaretskii]

> From: Chong Yidong <cyd@gnu.org>
> Cc: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>,  Stefan Monnier <monnier@iro.umontreal.ca>,  Kenichi Handa <handa@m17n.org>,  jca@wxcvbn.org,  12242@debbugs.gnu.org
> Date: Thu, 23 Aug 2012 22:24:19 +0800
> 
> Let us consider outright reversion of 108048.  What would be the effect?
> Given a choice between a bug (Bug#11519) which is also in 24.1, and
> failure-to-compile on OpenBSD as a new regression, the former is far
> preferable, unless you can point out some consequence that is more
> serious that the discussion in Bug#11519 implies.

Reverting 108048 (and 108020, see below) will cause trouble to any
system that uses ralloc.c, including (evidently) OpenBSD.  The fact
that the problem was originally found only during bootstrap on Windows
is just a coincidence.  It can strike at any time that regexp search
is used in a text that includes characters which need unification.
It's a bomb waiting to explode.  So I don't recommend that.

Of course, Emacs 24.1 was shipped with that bug, so ...

Can I have a 1-day grace to try debugging the OpenBSD crash?  Jérémie
generously gave me a login on the machine where it happened, and I'd
like a chance to try debugging it.

> Also, if we revert 108048, should we revert 108020 (which is in 24.1) as
> well?  Could you explain what problem 108020 causes which 108048 is
> supposed to fix?  Does it have symptoms?

108020 had a logic flaw in the incrementing and decrementing the flag
that inhibits relocation.  Due to that flaw, the inhibition was not
really working beyond the first time.  108048 fixed that part.  And
yes, these two revisions go together.

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> The fact that the problem was originally found only during bootstrap
> on Windows is just a coincidence.  It can strike at any time that
> regexp search is used in a text that includes characters which need
> unification.
>
> Of course, Emacs 24.1 was shipped with that bug, so ...

The fact that the problem showed up so late in the 24.1 pretest, and has
been with us since Emacs 23, indicates that the symptoms are rather rare
in practice.  So reverting seems like the least bad solution.

> Can I have a 1-day grace to try debugging the OpenBSD crash?  Jérémie
> generously gave me a login on the machine where it happened, and I'd
> like a chance to try debugging it.

OK.  Good luck, and thanks for all your efforts with this bug.

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

[Eli Zaretskii]

> From: Chong Yidong <cyd@gnu.org>
> Cc: mituharu@math.s.chiba-u.ac.jp,  monnier@iro.umontreal.ca,  handa@m17n.org,  jca@wxcvbn.org,  12242@debbugs.gnu.org
> Date: Fri, 24 Aug 2012 11:25:26 +0800
> 
> > Can I have a 1-day grace to try debugging the OpenBSD crash?  Jérémie
> > generously gave me a login on the machine where it happened, and I'd
> > like a chance to try debugging it.
> 
> OK.  Good luck, and thanks for all your efforts with this bug.

I committed a fix for this as emacs-24 branch revision 108120.  It is
somewhat of a phenomenological nature, because I could not actually
catch the entire sequence of calls and actions leading to the crash,
which might have allowed me to fix the root cause where it happens, if
possible.  (It turns out OpenBSD doesn't have hardware watchpoints
support in GDB, which makes catching references to variables painfully
slow, certainly when a deadline is looming.)

I did see that the crash happens because a 'heap' structure recorded
in a memory control block maintained by ralloc.c refers to addresses
that aren't managed as part of the linked list of 'heap' structures.
It is therefore wrong to dereference such bogus 'heap' structures to
update them.  The crash happens because 'heap' pointed to memory that
was beyond our break point; however, I found that we dereference such
bogus 'heap's in more places, and survive that only because, by sheer
luck, they are still within our address space.  (ralloc.c does not
relinquish memory to the system until it has enough excess memory to
justify that.)

The solution I checked in is not to dereference 'heap' pointers that
are not in the linked list of heaps we maintain.

The fixed version survived the command that crashed and in addition 3
different bootstraps, one on OpenBSD where the crash happened and 2
more (optimized and unoptimized) on MS-Windows.  I also checked that
the MS-DOS build, which also uses ralloc.c, still works OK with the
offending commands after the patch.  Those are the only systems I have
access to which use ralloc.c.

I do suggest another pretest, to make sure this fix is solid.

I will also work on the trunk on removing calls to xmalloc inside the
functions called by maybe_unify_char, which should probably eliminate
the original problem (although they also call to make-char-table,
which still allocates memory, albeit in smaller chunks).

Thanks.