bug#32938: guile 2.2.4 crashes (u8-list->bytevector (make-bytevector 32 0))

[Mark H Weaver <mhw@netris.org>]

Josh Datko <jbd@cryptotronix.com> writes:

> If you try to convert a bytevector, to a bytevector, using
> u8-list->bytevector, guile crashes.
>
> $ guile -q
> GNU Guile 2.2.4
> Copyright (C) 1995-2017 Free Software Foundation, Inc.
>
> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
> This program is free software, and you are welcome to redistribute it
> under certain conditions; type `,show c' for details.
>
> Enter `,help' for help.
> scheme@(guile-user)> (use-modules (rnrs bytevectors))
> scheme@(guile-user)> (u8-list->bytevector (make-bytevector 32 0))
> [1]    126190 abort (core dumped)  guile -q

Indeed, the code in 'u8-list->bytevector' that's supposed to validate
that its argument is a list, is broken.

'u8-list->bytevector' uses the SCM_VALIDATE_LIST_COPYLEN macro to
validate the list and simultaneously compute its length.  That macro
implicitly assumes that its third operand will be a variable of type
'long', because the result of 'scm_ilength' is assigned to it, and
'scm_ilength' returns a 'long'.

After storing the result to the variable, it checks to see if the result
is negative, which would indicate that the operand wasn't a proper list.

The bytevector operations that convert integer lists to bytevectors pass
a variable of type 'size_t' to SCM_VALIDATE_LIST_COPYLEN.  Since
'size_t' is unsigned, the -1 result from 'scm_ilength' was interpreted
as ULONG_MAX instead.

Thanks for the report.

        Mark