bug in check for stack growth direction in _AC_LIBOBJ_ALLOCA

[Andy Wingo <wingo@pobox.com>]

Hello,

The following transcript indicates a problem with the stack growth
direction check, present at functions.m4:328 in autoconf 2.68:

  wingo@badger:/tmp$ cat foo.c 
  int
  find_stack_direction ()
  {
    static char *addr = 0;
    auto char dummy;
    if (addr == 0)
      {
        addr = &dummy;
        return find_stack_direction ();
      }
    else
      return (&dummy > addr) ? 1 : -1;
  }

  int
  main ()
  {
    return find_stack_direction () < 0;
  }
  wingo@badger:/tmp$ gcc -O1 -o test foo.c
  wingo@badger:/tmp$ ./test; echo $?
  1
  wingo@badger:/tmp$ gcc -O3 -o test foo.c
  wingo@badger:/tmp$ ./test; echo $?
  0
  $ gcc --version
  gcc (Debian 4.6.0-13) 4.6.1 20110611 (prerelease)
  Copyright (C) 2011 Free Software Foundation, Inc.
  This is free software; see the source for copying conditions.  There is NO
  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

As you see, the check (on an x86-64 system) gives the correct result at
-O1 but not at -O3.

(I don't actually use this check from _AC_LIBOBJ_ALLOCA, but Guile has
the exact same test to check for stack growth direction, so it's the
same issue.  Reported by Marco Maggi.)

Looking a bit more closely at this, I find the assembly to be a bit odd
here:

    00000000004004a0 <find_stack_direction>:
      4004a0:	mov    0x2003f9(%rip),%rax        # 6008a0 <addr.1586>
      4004a7:	test   %rax,%rax
      4004aa:	je     4004c0 <find_stack_direction+0x20>
      4004ac:	lea    -0x2(%rsp),%rdx
      4004b1:	cmp    %rdx,%rax
      4004b4:	sbb    %eax,%eax
      4004b6:	and    $0x2,%eax
      4004b9:	sub    $0x1,%eax
      4004bc:	retq   
      4004bd:	nopl   (%rax)
      4004c0:	lea    -0x2(%rsp),%rax
      4004c5:	lea    -0x1(%rsp),%rdx
      4004ca:	cmp    %rdx,%rax
      4004cd:	mov    %rax,0x2003cc(%rip)        # 6008a0 <addr.1586>
      4004d4:	sbb    %eax,%eax
      4004d6:	and    $0x2,%eax
      4004d9:	sub    $0x1,%eax
      4004dc:	retq   
      4004dd:	nop
      4004de:	nop
      4004df:	nop

As you can see there is no call.  The test will always be true, thus the
second branch is always taken.  I don't know what allows GCC to do this
inlining.  Could it be a GCC bug?  Every time I think I have a GCC bug
I'm wrong, though :)

I tried changing the test to the following:

  int
  find_stack_direction (char *addr)
  {
    char dummy;
    if (addr == 0)
      return find_stack_direction (&dummy);
    else
      return (&dummy > addr) ? 1 : -1;
  }

  int
  main ()
  {
    return find_stack_direction (0) < 0;
  }

But I get the same behavior.

Andy
-- 
http://wingolog.org/