bug#22667: string->bytevector encoding error

bug#22667: string->bytevector encoding error

[Josep Portella Florit]

'string->bytevector' throws 'encoding-error' when the encoding is
"UTF-16" and the length of the string is greater than 128.  The same
for the encoding "UTF-32" and a string of length greater than 64.

Tested on Guile 2.0.11 and 2.1.2 with the same result.

How to reproduce:

(use-modules (ice-9 iconv))

;; These expressions evaluate as expected:
(string->bytevector (make-string 128 #\A) "UTF-16")
;; => #vu8(254 255 0 65 0 65 ...)
(string->bytevector (make-string 64 #\A) "UTF-32")
;; => #vu8(0 0 254 255 0 0 0 65 0 0 0 65 ...)

;; These expressions throw 'encoding-error':
(string->bytevector (make-string 129 #\A) "UTF-16")
(string->bytevector (make-string 65 #\A) "UTF-32")

bug#22667: string->bytevector encoding error

[Mark H Weaver]

Josep Portella Florit <jpf@primfilat.com> writes:

> 'string->bytevector' throws 'encoding-error' when the encoding is
> "UTF-16" and the length of the string is greater than 128.  The same
> for the encoding "UTF-32" and a string of length greater than 64.
>
> Tested on Guile 2.0.11 and 2.1.2 with the same result.
>
> How to reproduce:
>
> (use-modules (ice-9 iconv))
>
> ;; These expressions evaluate as expected:
> (string->bytevector (make-string 128 #\A) "UTF-16")
> ;; => #vu8(254 255 0 65 0 65 ...)
> (string->bytevector (make-string 64 #\A) "UTF-32")
> ;; => #vu8(0 0 254 255 0 0 0 65 0 0 0 65 ...)
>
> ;; These expressions throw 'encoding-error':
> (string->bytevector (make-string 129 #\A) "UTF-16")
> (string->bytevector (make-string 65 #\A) "UTF-32")

Thee bug is in 'display_string_using_iconv', introduced in commit
f4bc4e5934, which improperly assumes that a 256-byte buffer
(encoded_output) will be large enough to hold the results of converting
up to 256 bytes of UTF-8 (utf8_buf) to an arbitrary encoding using
'iconv'.  When this assumption fails, the call to 'iconv' on line 997

      done = iconv (id->output_cd, &input, &input_left,
		    &output, &output_left);

reports a failure due to the output buffer being too small, and the code
fails to handle this case.

To be continued...

     Mark

bug#22667: string->bytevector encoding error

[Andy Wingo]

On Sun 14 Feb 2016 23:18, Josep Portella Florit <jpf@primfilat.com> writes:

> 'string->bytevector' throws 'encoding-error' when the encoding is
> "UTF-16" and the length of the string is greater than 128.  The same
> for the encoding "UTF-32" and a string of length greater than 64.
>
> Tested on Guile 2.0.11 and 2.1.2 with the same result.
>
> How to reproduce:
>
> (use-modules (ice-9 iconv))
>
> ;; These expressions evaluate as expected:
> (string->bytevector (make-string 128 #\A) "UTF-16")
> ;; => #vu8(254 255 0 65 0 65 ...)
> (string->bytevector (make-string 64 #\A) "UTF-32")
> ;; => #vu8(0 0 254 255 0 0 0 65 0 0 0 65 ...)
>
> ;; These expressions throw 'encoding-error':
> (string->bytevector (make-string 129 #\A) "UTF-16")
> (string->bytevector (make-string 65 #\A) "UTF-32")

For what it's worth, this appears to be fixed in 2.1.3.  However we
should fix 2.0 as well.

Andy

bug#22667: string->bytevector encoding error

[Andy Wingo]

Hi,

On Mon 20 Jun 2016 18:07, Andy Wingo <wingo@pobox.com> writes:

> On Sun 14 Feb 2016 23:18, Josep Portella Florit <jpf@primfilat.com> writes:
>
>> (use-modules (ice-9 iconv))
>> ;; These expressions throw 'encoding-error':
>> (string->bytevector (make-string 129 #\A) "UTF-16")
>> (string->bytevector (make-string 65 #\A) "UTF-32")
>
> For what it's worth, this appears to be fixed in 2.1.3.  However we
> should fix 2.0 as well.

Fixed.  Thanks for the report, and thanks to Mark for diagnosing the
problem.

Andy.