read-response-body of (web response) depends on Content-Length but ...

read-response-body of (web response) depends on Content-Length but ...

[Sunjoong Lee]

[-- Attachment #1: Type: text/plain, Size: 1356 bytes --]

Hello,

I cannot solve
http://lists.gnu.org/archive/html/guile-user/2012-04/msg00034.html yet.
This is a potentially related problem but I'm not sure;

(use-modules ((srfi srfi-11) #:select (let-values))
             ((web uri)      #:select (string->uri))
             ((web client)   #:select (http-get)))
(let-values (((res-headers res-body)
              (http-get (string->uri "http://www.gnu.org/home.en.html"))))
  (display res-body)
  (newline))

Above code display only false value, #f. After changing res-body
to res-headers, it display;
  #<<response> version: (1 . 1) code: 200 reason-phrase: "OK" headers:
((date . #<date nanosecond: 0 second: 42 minute: 50 hour: 7 day: 27 month:
4 year: 2012 zone-offset: 0>) (server . "Apache/2.2.14") (accept-ranges
bytes) (cache-control (max-age . 0)) (expires . #<date nanosecond: 0
second: 42 minute: 50 hour: 7 day: 27 month: 4 year: 2012 zone-offset: 0>)
(vary accept-encoding) (connection close) (transfer-encoding (chunked))
(content-type text/html) (content-language "en")) port: #<closed: file 0>>

Oh, Content-Length missing!! Apache server of www.gnu.org replied response
without Content-Length via home.en.html request. Is
Content-Length mandatory of response? If so, it would be Apache's fault.
But if not so, I think it's better modify read-response-body of (web
response) in Guile.

Thanks.

[-- Attachment #2: Type: text/html, Size: 1862 bytes --]

Re: read-response-body of (web response) depends on Content-Length but ...

[Sunjoong Lee]

[-- Attachment #1: Type: text/plain, Size: 1171 bytes --]

I googled and found http://tools.ietf.org/html/rfc2616 .

In a section of "4.4 Message Length":
  2.If a Transfer-Encoding header field (section 14.41) is present and
     has any value other than "identity", then the transfer-length is
     defined by use of the "chunked" transfer-coding (section 3.6),
     unless the message is terminated by closing the connection.
  3.If a Content-Length header field (section 14.13) is present, its
     decimal value in OCTETs represents both the entity-length and the
     transfer-length. The Content-Length header field MUST NOT be sent
     if these two lengths are different (i.e., if a Transfer-Encoding
     header field is present). If a message is received with both a
     Transfer-Encoding header field and a Content-Length header field,
     the latter MUST be ignored.

2012/4/27 Sunjoong Lee <sunjoong@gmail.com>
>
> Oh, Content-Length missing!! Apache server of www.gnu.org replied
> response without Content-Length via home.en.html request. Is
> Content-Length mandatory of response? If so, it would be Apache's fault.
> But if not so, I think it's better modify read-response-body of (web
> response) in Guile.
>

[-- Attachment #2: Type: text/html, Size: 2116 bytes --]

Re: read-response-body of (web response) depends on Content-Length but ...

[Sunjoong Lee]

[-- Attachment #1: Type: text/plain, Size: 2154 bytes --]

Below is a urgent patch of response.scm:

--- /usr/share/guile/2.0/web/response.scm 2012-04-22 04:36:06.753878689
+0900
+++ response.scm 2012-04-27 18:49:16.499881816 +0900
@@ -217,13 +217,16 @@
 (define (read-response-body r)
   "Reads the response body from @var{r}, as a bytevector.  Returns
 @code{#f} if there was no response body."
+  (let ((te (response-transfer-encoding r)))
+    (if (and te (eq? 'chunked (car (car te))))
+        (get-bytevector-all (response-port r))
   (let ((nbytes (response-content-length r)))
     (and nbytes
          (let ((bv (get-bytevector-n (response-port r) nbytes)))
            (if (= (bytevector-length bv) nbytes)
                bv
                (bad-response "EOF while reading response body: ~a bytes of
~a"
-                            (bytevector-length bv) nbytes))))))
+                            (bytevector-length bv) nbytes))))))))

 (define (write-response-body r bv)
   "Write @var{body}, a bytevector, to the port corresponding to the HTTP
@@ -269,6 +272,7 @@
 (define-response-accessor content-type #f)
 (define-response-accessor expires #f)
 (define-response-accessor last-modified #f)
+(define-response-accessor transfer-encoding '())

 ;; Response headers
 ;;

Above patch is just a patch.

2012/4/27 Sunjoong Lee <sunjoong@gmail.com>

> I googled and found http://tools.ietf.org/html/rfc2616 .
>
> In a section of "4.4 Message Length":
>   2.If a Transfer-Encoding header field (section 14.41) is present and
>      has any value other than "identity", then the transfer-length is
>      defined by use of the "chunked" transfer-coding (section 3.6),
>      unless the message is terminated by closing the connection.
>   3.If a Content-Length header field (section 14.13) is present, its
>      decimal value in OCTETs represents both the entity-length and the
>      transfer-length. The Content-Length header field MUST NOT be sent
>      if these two lengths are different (i.e., if a Transfer-Encoding
>      header field is present). If a message is received with both a
>      Transfer-Encoding header field and a Content-Length header field,
>      the latter MUST be ignored.
>

[-- Attachment #2: Type: text/html, Size: 4252 bytes --]

Re: read-response-body of (web response) depends on Content-Length but ...

[Sunjoong Lee]

[-- Attachment #1: Type: text/plain, Size: 2333 bytes --]

The urgent patch posted before makes a bug, so I cancel it and repost:
########## cut here ##########
--- /usr/share/guile/2.0/web/response.scm 2012-04-22 04:36:06.753878689
+0900
+++ response.scm 2012-04-27 19:54:55.539304271 +0900
@@ -217,13 +217,16 @@
 (define (read-response-body r)
   "Reads the response body from @var{r}, as a bytevector.  Returns
 @code{#f} if there was no response body."
+  (let ((te (response-transfer-encoding r)))
+    (if (and te (eq? 'chunked (car (car te))))
+        (get-bytevector-all (response-port r))
   (let ((nbytes (response-content-length r)))
     (and nbytes
          (let ((bv (get-bytevector-n (response-port r) nbytes)))
            (if (= (bytevector-length bv) nbytes)
                bv
                (bad-response "EOF while reading response body: ~a bytes of
~a"
-                            (bytevector-length bv) nbytes))))))
+                            (bytevector-length bv) nbytes))))))))

 (define (write-response-body r bv)
   "Write @var{body}, a bytevector, to the port corresponding to the HTTP
@@ -269,6 +272,7 @@
 (define-response-accessor content-type #f)
 (define-response-accessor expires #f)
 (define-response-accessor last-modified #f)
+(define-response-accessor transfer-encoding #f)

 ;; Response headers
 ;;
########## cut here ##########

2012/4/27 Sunjoong Lee <sunjoong@gmail.com>
>
>  +(define-response-accessor transfer-encoding '())
>
2012/4/27 Sunjoong Lee <sunjoong@gmail.com>
>
>> I googled and found http://tools.ietf.org/html/rfc2616 .
>>
>> In a section of "4.4 Message Length":
>>   2.If a Transfer-Encoding header field (section 14.41) is present and
>>      has any value other than "identity", then the transfer-length is
>>      defined by use of the "chunked" transfer-coding (section 3.6),
>>      unless the message is terminated by closing the connection.
>>   3.If a Content-Length header field (section 14.13) is present, its
>>      decimal value in OCTETs represents both the entity-length and the
>>      transfer-length. The Content-Length header field MUST NOT be sent
>>      if these two lengths are different (i.e., if a Transfer-Encoding
>>      header field is present). If a message is received with both a
>>      Transfer-Encoding header field and a Content-Length header field,
>>      the latter MUST be ignored.
>>
>

[-- Attachment #2: Type: text/html, Size: 4866 bytes --]

Re: read-response-body of (web response) depends on Content-Length but ...

[Daniel Hartwig]

On 27 April 2012 16:28, Sunjoong Lee <sunjoong@gmail.com> wrote:
> I googled and found http://tools.ietf.org/html/rfc2616 .
>
> In a section of "4.4 Message Length":
>   2.If a Transfer-Encoding header field (section 14.41) is present and
>      has any value other than "identity", then the transfer-length is
>      defined by use of the "chunked" transfer-coding (section 3.6),
>      unless the message is terminated by closing the connection.
>   3.If a Content-Length header field (section 14.13) is present, its
>      decimal value in OCTETs represents both the entity-length and the
>      transfer-length. The Content-Length header field MUST NOT be sent
>      if these two lengths are different (i.e., if a Transfer-Encoding
>      header field is present). If a message is received with both a
>      Transfer-Encoding header field and a Content-Length header field,
>      the latter MUST be ignored.
>

Ian Price has done some work to support chunked encoding recently,
which seems to be the source of your problems.  His solution was
already quite thorough but some comments were made.  You may want to
help finishing that off.

Search the guile-devel and -user archives for "Chunked Encoding" to find it.

http://comments.gmane.org/gmane.lisp.guile.devel/12814
http://comments.gmane.org/gmane.lisp.guile.user/8935

Re: read-response-body of (web response) depends on Content-Length but ...

[Sunjoong Lee]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

I cancel a previous patch because Ian Price solve it nice. Daniel Hartwig
told me that; thanks Daniel.

I cannot solve
http://lists.gnu.org/archive/html/guile-user/2012-04/msg00034.html yet but
can solve http://lists.gnu.org/archive/html/guile-user/2012-04/msg00041.htmlwith
Ian's solution,
http://lists.gnu.org/archive/html/guile-user/2011-11/msg00011.html .

There are many comments about "Chunked Encoding". Oh, my poor english and
background knowledge...

Thanks Daniel and Ian again.

2012/4/27 Sunjoong Lee <sunjoong@gmail.com>

> The urgent patch posted before makes a bug, so I cancel it and repost:
>

[-- Attachment #2: Type: text/html, Size: 1290 bytes --]