Undefined behavior in conv-integer.i.c?

[Miroslav Lichvar <mlichvar@redhat.com>]

Hi,

I was looking at a problem with guile-1.8.8 when compiled with
gcc-6.0. Two of the tests from the test suite were failing with
strange "out of range" errors [1]. After some investigation I think
the bug is that the code in libguile/conv-integer.i.c relies on
overflow of signed integers in the following code (starting on line
77), specifically -TYPE_MIN being less than zero. Adding -fwrapv to
CFLAGS worked as a workaround for me.

          if (mpz_sgn (SCM_I_BIG_MPZ (val)) >= 0)
            {
              if (n < 0)
                goto out_of_range;
            }
          else
            {
              n = -n;
              if (n >= 0)
                goto out_of_range;
            }

Looking at the current guile code, conv-integer.i.c is identical to
what it was in 1.8.8, but interestingly the tests didn't fail for me.
Maybe something else is preventing gcc from using the optimization?

I'm not sure what would be the best way to fix it. Maybe n should
really be unsigned and compared to the maximum values, but what would
be the absolute value of TYPE_MIN if it should work also with other
integer representations than two's complement?

[1] https://bugzilla.redhat.com/attachment.cgi?id=1124252

-- 
Miroslav Lichvar