bug#14916: Fixnum procedures can be made to return non-fixnums

[Mark H Weaver <mhw@netris.org>]

Andy Wingo <wingo@pobox.com> writes:

> Hi Mark,
>
> I know you like mathy things and so this might be a project you would
> like.  I think the right thing to do here is to redefine fixnum? as
>
>  (= x (logand x #x2fffffff))

This doesn't work for negative fixnums, because you are effectively
masking out the sign of 'x' above.  Unfortunately, this problem cannot
be fixed by changing the mask.  For purposes of the bitwise operations,
negative integers are treated as though they have an infinite number of
ones in the high bits, e.g. -1 is all ones, and is thus an identity for
'logand'.  So, there's no single sign bit, but rather an infinite number
of them, and all of those bits can also be used as value bits as well,
so there's no way to avoid masking out the sign without also masking out
high-order value bits.

One possibility would be to subtract 'most-negative-fixnum' from x
before masking, but I don't know what the compiler would do with that.

     Mark