bug#13095: http-get: Throw to key `bad-response' with args `("EOF while reading response body: ...)'

bug#13095: http-get: Throw to key `bad-response' with args `("EOF while reading response body: ...)'

[Nikita Karetnikov]

Hi,

I'm using Guile 2.0.7. 'http-get' fails to download the file.

scheme@(guile-user)> (http-get (string->uri
"http://citylan.dl.sourceforge.net/project/libusb/libusb-1.0/libusb-1.0.9/libusb-1.0.9.tar.bz2"))
ERROR: In procedure get-bytevector-all:
ERROR: Throw to key `bad-response' with args `("EOF while reading
response body: ~a bytes of ~a" (12731 421971))'.

Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
scheme@(guile-user) [1]> ,bt
In web/client.scm:
    131:4  1 (http-get #<<uri> scheme: http userinfo: #f host: "cit…> …)
In unknown file:
           0 (get-bytevector-all #<input: r6rs-custom-binary-input-p…>)

$ wget --spider
http://citylan.dl.sourceforge.net/project/libusb/libusb-1.0/libusb-1.0.9/libusb-1.0.9.tar.bz2
Spider mode enabled. Check if remote file exists.

[...]

Length: 421971 (412K) [application/octet-stream]
Remote file exists.

Also, check this thread. [1]

Nikita

[1] https://lists.gnu.org/archive/html/bug-guix/2012-12/msg00033.html

bug#13095: http-get: Throw to key `bad-response' with args `("EOF while reading response body: ...)'

[Ian Price]

Nikita Karetnikov <nikita.karetnikov@gmail.com> writes:

> Hi,
>
> I'm using Guile 2.0.7. 'http-get' fails to download the file.
>
> scheme@(guile-user)> (http-get (string->uri
> "http://citylan.dl.sourceforge.net/project/libusb/libusb-1.0/libusb-1.0.9/libusb-1.0.9.tar.bz2"))
> ERROR: In procedure get-bytevector-all:
> ERROR: Throw to key `bad-response' with args `("EOF while reading
> response body: ~a bytes of ~a" (12731 421971))'.

This will run correctly if you call http-get with the #:keep-alive?
keyword set to #t. If you don't run it with that, the connection gets
closed automatically, and so you'll get an EOF down the line when it
tries to read from the port. 

I think this is a bug. The intended semantics of keep-alive? (I believe)
is to keep it alive _after_ the request (including the body) is
finished, not just the request headers.

Strange though, I never noticed this come up when adding the chunked
support in the first place.

-- 
Ian Price -- shift-reset.com

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"

bug#13095: http-get: Throw to key `bad-response' with args `("EOF while reading response body: ...)'

[Ian Price]

Ian Price <ianprice90@googlemail.com> writes:

> Strange though, I never noticed this come up when adding the chunked
> support in the first place.

Actually, my apologies, I had noticed it since I use it in the tests. :/

-- 
Ian Price -- shift-reset.com

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"

bug#13095: http-get: Throw to key `bad-response' with args `("EOF while reading response body: ...)'

[Ian Price]

Okay so, chunked encoding was a red herring on my part. I saw the
#<input: r6rs-custom-binary-input-p…> and jumped to conclusions. The
actual port created was a "delimited port", which tries to read the
appropriate amount of bytes according to the content-length
header.

Now, the problem, as I see it, is that when http-get (and http-get*) is
called without the #:keep-alive? #t option, it will call (shutdown port
1)[0], which is to close the port for any outgoing traffic. This sounds
fine in theory, but I think (I am no networking expert), that it is also
preventing any requests for additional data.

I do notice that it works correctly when I remove the calls to shutdown,
which gives me confidence that it is the culprit, even if that analysis
is not entirely correct (and I'm not sure it is)

Removing the calls to shutdown should be fine, since when keep-alive? is
#f, we close the ports after the read anyway.

I'm going to withhold pushing a patch and closing this bug for the
moment, until I'm satisfied of the reason why this is the problem.

0. I have checked that 1 corresponds to SHUT_WR on my platform
-- 
Ian Price -- shift-reset.com

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"

bug#13095: http-get: Throw to key `bad-response' with args `("EOF while reading response body: ...)'

[Ludovic Courtès]

Thien-Thi Nguyen <ttn@gnuvola.org> skribis:

> HTTP 1.0 is "close by default", but HTTP 1.1 is "keep open by default",
> if i recall correctly.  Does that have anything to do w/ this bug?

Note sure, because both http://citylan.dl.sourceforge.net (the example
that fails) and http://ftp.gnu.org (one that works even with the
‘shutdown’ call) are HTTP 1.1:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (use-modules (web client)(web uri)(rnrs io ports))
scheme@(guile-user)> (define resp (http-get (string->uri "http://citylan.dl.sourceforge.net/project/libusb/libusb-1.0/libusb-1.0.9/libusb-1.0.9.tar.bz2")))
scheme@(guile-user)> resp
$1 = #<<response> version: (1 . 1) code: 200 reason-phrase: "OK" headers: ((server . "nginx/1.0.12") (date . #<date nanosecond: 0 second: 25 minute: 35 hour: 23 day: 8 month: 12 year: 2012 zone-offset: 0>) (content-type application/octet-stream) (content-length . 421971) (last-modified . #<date nanosecond: 0 second: 58 minute: 47 hour: 6 day: 20 month: 4 year: 2012 zone-offset: 0>) (connection close) (accept-ranges bytes)) port: #<closed: file 0>>
scheme@(guile-user)> (define resp* (http-get (string->uri "http://ftp.gnu.org/gnu/hello/hello-2.8.tar.gz")))
scheme@(guile-user)> resp*
$2 = #<<response> version: (1 . 1) code: 200 reason-phrase: "OK" headers: ((date . #<date nanosecond: 0 second: 59 minute: 35 hour: 23 day: 8 month: 12 year: 2012 zone-offset: 0>) (server . "Apache/2.2.14 (Trisquel GNU/Linux)") (last-modified . #<date nanosecond: 0 second: 47 minute: 55 hour: 17 day: 20 month: 4 year: 2012 zone-offset: 0>) (etag "4648b4-aa48b-4be1fff866200" . #t) (accept-ranges bytes) (content-length . 697483) (connection close) (content-type application/x-gzip)) port: #<closed: file 0>>
--8<---------------cut here---------------end--------------->8---

Thanks,
Ludo’.

bug#13095: http-get: Throw to key `bad-response' with args `("EOF while reading response body: ...)'

[dsmich]

---- "Ludovic Courtès" <ludo@gnu.org> wrote: 
> Thien-Thi Nguyen <ttn@gnuvola.org> skribis:
> 
> > HTTP 1.0 is "close by default", but HTTP 1.1 is "keep open by default",
> > if i recall correctly.  Does that have anything to do w/ this bug?
> 
> Note sure, because both http://citylan.dl.sourceforge.net (the example
> that fails) and http://ftp.gnu.org (one that works even with the
> ‘shutdown’ call) are HTTP 1.1:
> 
> --8<---------------cut here---------------start------------->8---
> scheme@(guile-user)> (use-modules (web client)(web uri)(rnrs io ports))
> scheme@(guile-user)> (define resp (http-get (string->uri "http://citylan.dl.sourceforge.net/project/libusb/libusb-1.0/libusb-1.0.9/libusb-1.0.9.tar.bz2")))
> scheme@(guile-user)> resp
> $1 = #<<response> version: (1 . 1) code: 200 reason-phrase: "OK" headers: ((server . "nginx/1.0.12") (date . #<date nanosecond: 0 second: 25 minute: 35 hour: 23 day: 8 month: 12 year: 2012 zone-offset: 0>) (content-type application/octet-stream) (content-length . 421971) (last-modified . #<date nanosecond: 0 second: 58 minute: 47 hour: 6 day: 20 month: 4 year: 2012 zone-offset: 0>) (connection close) (accept-ranges bytes)) port: #<closed: file 0>>
> scheme@(guile-user)> (define resp* (http-get (string->uri "http://ftp.gnu.org/gnu/hello/hello-2.8.tar.gz")))
> scheme@(guile-user)> resp*
> $2 = #<<response> version: (1 . 1) code: 200 reason-phrase: "OK" headers: ((date . #<date nanosecond: 0 second: 59 minute: 35 hour: 23 day: 8 month: 12 year: 2012 zone-offset: 0>) (server . "Apache/2.2.14 (Trisquel GNU/Linux)") (last-modified . #<date nanosecond: 0 second: 47 minute: 55 hour: 17 day: 20 month: 4 year: 2012 zone-offset: 0>) (etag "4648b4-aa48b-4be1fff866200" . #t) (accept-ranges bytes) (content-length . 697483) (connection close) (content-type application/x-gzip)) port: #<closed: file 0>>
> --8<---------------cut here---------------end--------------->8---

Hmm..  The failing site is running on nginx/1.0.12 while the working site is Apache/2.2.14.

-Dale

bug#13095: http-get: Throw to key `bad-response' with args `("EOF while reading response body: ...)'

[Ian Price]

Ian Price <ianprice90@googlemail.com> writes:

> I'm going to withhold pushing a patch and closing this bug for the
> moment, until I'm satisfied of the reason why this is the problem.

Well, I never did get a satisfying reason, but Andy Wingo pushed the
obvious fix for this issue in response to a different users report.
https://lists.gnu.org/archive/html/guile-user/2013-01/msg00006.html

The commit is ed3e8b8e06adaaa1df5085a0f730d42efa3f5c30 for http-get, and
http-get* (now deprecated) inherits this improvement in
990b11c53f8da2a6c14e1190bc4e76939db32d07.

Thank you for your report.

-- 
Ian Price -- shift-reset.com

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"