Problem with truncated string ports

Problem with truncated string ports

[Brent Pinkney]

Hi,

We are using guile on Solaris 10u10 on x86 hardware and are experiencing 
odd behaviour with (format #f "~s" foo), where foo is
     1. a string containing #\" quotation characters,
     2. ~9000 bytes long.

The faulty behaviour is that the resulting string is truncated to 4096 
bytes every time.
However, the problem does not occur in the guile REPL, nor in any guise 
on Solaris SPARC or Linux x86.


Further investigation using string ports and write confirms that the 
problem can be reproduced with

     (define s (make-string 9000 #\"))
     (define o (open-output-string))
     (write s o)
     (define r (get-output-string o))
     (string-length r)                                    ; answers 4096


I have inspected the strings.c implementation but nothing obvious 
suggests why the output string is truncated to 4096 bytes.


Can anyone assist ?

Thanks,

Brent

Re: Problem with truncated string ports

[Eli Zaretskii]

> Date: Fri, 21 Jun 2013 15:44:50 +0200
> From: Brent Pinkney <bp@ffroot.co.za>
> 
> I have inspected the strings.c implementation but nothing obvious 
> suggests why the output string is truncated to 4096 bytes.

Isn't 4096 somehow related to a pipe buffer size?

Re: Problem with truncated string ports - more info

[Brent Pinkney]

Hi,

I have some further diagnostics on my problem.

As you know the string resulting from a call to write should be at least 
two (2) characters longer than the input string since the input string 
is prepended and appended with a #\x quotation character.

This is indeed the case for short strings:

     (define s (make-string 64 #\x))
     (string-length (with-output-to-string (lambda () (write s))))     
         ; answers 66 - OK

This true all the way up to an input string of length 4094 (i.e. 4096 -2 ).

However, if I make  the input string 4095 bytes long, then I do not get 
the expected 4097 long output string.
Instead, I get a resulting string truncted to 4096 bytes.

     (define s (make-string 4095 #\x))
     (string-length (with-output-to-string (lambda () (write s))))     
         ; answer 4096 - WRONG.


Thanks,

Brent

Re: Problem with truncated string ports - resolution

[Brent Pinkney]

Hi,

I have discovered that if I force the default string encoding to "UTF-8" 
then it works.

Regards,

Brent

Re: Problem with truncated string ports - more info

[Mike Gran]

>However, if I make  the input string 4095 bytes long,
> then I do not get the expected 4097 long output string.

> Instead, I get a resulting string truncted to 4096 bytes.
>
>    (define s (make-string 4095 #\x))
>    (string-length (with-output-to-string (lambda () (write s))))             ; answer 4096 - WRONG.


Weird.  There are some 4k blocks in the R6RS port code, so
it might be there.


If you do the following string dump, does it say that the string's
stringbuf-length is also 4096?


    (define s (make-string 4095 #\x))
    (%string-dump (with-output-to-string (lambda () (write s))))

-Mike

Re: Problem with truncated string ports - more info

[Mark H Weaver]

Hi Brent,

Would you be willing to help debug this a bit?  It would be useful to
put a breakpoint at 'scm_strport_to_string' in libguile/strports.c in
the case where a truncated string is returned.

After 'pt' is initialized, what's the value of 'pt->read_buf_size'?  It
should be the number of bytes in the string, when encoded using the port
encoding of the string port.  If this is too small, then it indicates a
problem in strports.c.

If 'pt->read_buf_size' is correct, then step into 'scm_from_stringn',
which in turn calls 'u32_conv_from_encoding', which is part of
libunistring.  See if something is going wrong in there.  After calling
'u32_conv_from_encoding', 'u32len' should be the number of characters in
the string.  If it's short, then that suggests a problem in
libunistring.  Did you run "make check" in libunistring?

     Thanks,
       Mark