bug#18995: Error: Could not reserve dynamic heap area.

[Eli Zaretskii <eliz@gnu.org>]

> Date: Sat, 08 Nov 2014 20:49:01 +0000
> From: Daniel Colascione <dancol@dancol.org>
> CC: 18995@debbugs.gnu.org
> 
> Alexander, does it help to add explicit debug output (say, fprintf to
> stderr) on each iteration of the loop?

I'm not Alexander, but I can answer that question now: no, it doesn't
help.  The loop is still executed only once, even if it calls fprintf.

> If not, can you compare the debug output from the version with
> -funroll-loops to the one without, to see whether we're making the
> same sequence of VirtualAlloc calls? If they're the same, maybe it's
> not the loop that's broken, but the address space layout.

There's nothing wrong with the address space, and there's nothing
wrong with GCC, either.  What we have here is a genuine bug in our
code, albeit one that is subtle, and also very hard to actually
reproduce in real life.

It looked like a GCC bug at first, but as I tried to modify the source
and look at the effect of that on the generated code, it finally
dawned on me: GCC's loop-unrolling code simply correctly calculated
that with the initial value of 0x68000000 and decrement of 0x00800000,
the value of 'size' in the loop will never be less than 0x00100000,
due to wraparound in the subtraction of unsigned values.  So what we
have here is a potentially infinite loop, i.e. "undefined behavior".
Given that, it is justified for GCC to give us what we deserve, i.e. a
loop "unrolled" by executing its body only once.  I present the
disassembly below, for the curious, and it's clear that there's no
loop there, and also the value of 'size' is never tested at all, since
GCC decided that the condition 'size > 0x00100000' is always true.

Of course, in reality this "infinite" loop will always be finite,
since a real live system will always find more than 8MB of free
address space to reserve.  Unless someone deliberately reserved a lot
of the address space just to cause Emacs hit this problem, that is.

I installed a trivial fix on the emacs-24 branch, and I'm closing this
bug.

Here's the disassembly of the original code, as compiled with
"-funroll-loops" (it is for init_heap, because allocate_heap is
inlined in it):

  (gdb) disassemble init_heap
  Dump of assembler code for function init_heap:
     0x0122f1fc <+0>:     push   %ebx
     0x0122f1fd <+1>:     sub    $0x18,%esp
     0x0122f200 <+4>:     movl   $0x0,(%esp)
     0x0122f207 <+11>:    call   0x1277184 <GetModuleHandleA@4>
     0x0122f20c <+16>:    sub    $0x4,%esp
     0x0122f20f <+19>:    add    0x3c(%eax),%eax
     0x0122f212 <+22>:    mov    %eax,0x4(%esp)
     0x0122f216 <+26>:    movl   $0x14c1cd0,(%esp)
     0x0122f21d <+33>:    call   0x11c4cb5 <find_section>
     0x0122f222 <+38>:    mov    %eax,0x154abd4
     0x0122f227 <+43>:    cmpl   $0x0,0x1533cf0
     0x0122f22e <+50>:    je     0x122f28b <init_heap+143>
     0x0122f230 <+52>:    mov    $0x68000000,%ecx
     0x0122f235 <+57>:    mov    %ecx,0x15422a0
     0x0122f23b <+63>:    movl   $0x1,0xc(%esp)
     0x0122f243 <+71>:    movl   $0x2000,0x8(%esp)
     0x0122f24b <+79>:    mov    %ecx,0x4(%esp)
     0x0122f24f <+83>:    movl   $0x0,(%esp)
     0x0122f256 <+90>:    call   0x12775cc <VirtualAlloc@16>
     0x0122f25b <+95>:    sub    $0x10,%esp
     0x0122f25e <+98>:    mov    %eax,0x15422ac
     0x0122f263 <+103>:   test   %eax,%eax                  <<<<<<<<<<<
     0x0122f265 <+105>:   jne    0x122f27f <init_heap+131>  <<<<<<<<<<<
     0x0122f267 <+107>:   movl   $0x14c1cd8,(%esp)          <<<<<<<<<<<
     0x0122f26e <+114>:   call   0x122f0e0 <printf>         <<<<<<<<<<<
     0x0122f273 <+119>:   movl   $0x1,(%esp)                <<<<<<<<<<<
     0x0122f27a <+126>:   call   0x1276948 <exit>           <<<<<<<<<<<
     0x0122f27f <+131>:   mov    %eax,0x15422a8
     0x0122f284 <+136>:   mov    %eax,0x15422a4
     0x0122f289 <+141>:   jmp    0x122f2bc <init_heap+192>
     0x0122f28b <+143>:   mov    0xc(%eax),%ebx
     0x0122f28e <+146>:   movl   $0x0,(%esp)
     0x0122f295 <+153>:   call   0x1277184 <GetModuleHandleA@4>
     0x0122f29a <+158>:   sub    $0x4,%esp
     0x0122f29d <+161>:   add    %ebx,%eax
     0x0122f29f <+163>:   mov    %eax,0x15422ac
     0x0122f2a4 <+168>:   mov    %eax,0x15422a8
     0x0122f2a9 <+173>:   mov    %eax,0x15422a4
     0x0122f2ae <+178>:   mov    0x154abd4,%eax
     0x0122f2b3 <+183>:   mov    0x8(%eax),%edx
     0x0122f2b6 <+186>:   mov    %edx,0x15422a0
     0x0122f2bc <+192>:   call   0x1201f1d <cache_system_info>
     0x0122f2c1 <+197>:   add    $0x18,%esp
     0x0122f2c4 <+200>:   pop    %ebx
     0x0122f2c5 <+201>:   ret
  End of assembler dump.

Look, ma: no loop!