From: ludo@gnu.org (Ludovic Courtès)
To: 19939@debbugs.gnu.org
Subject: bug#19939: http client: Chunks shouldn't be read at once
Date: Tue, 24 Feb 2015 21:59:05 +0100 [thread overview]
Message-ID: <87ioer9gbq.fsf@gnu.org> (raw)
As of 2.0.11, the chunked input port of the HTTP client reads whole
chunks at once:
--8<---------------cut here---------------start------------->8---
(define (read-chunk port)
(let ((size (read-chunk-header port)))
(read-chunk-body port size)))
(define (read-chunk-body port size)
(let ((bv (get-bytevector-n port size)))
(get-u8 port) ; CR
(get-u8 port) ; LF
bv))
(define* (make-chunked-input-port port #:key (keep-alive? #f))
"Returns a new port which translates HTTP chunked transfer encoded
data from PORT into a non-encoded format. Returns eof when it has
read the final chunk from PORT. This does not necessarily mean
that there is no more data on PORT. When the returned port is
closed it will also close PORT, unless the KEEP-ALIVE? is true."
(define (next-chunk)
(read-chunk port))
[...]
(define (read! bv idx to-read)
[...]
(set! buffer (next-chunk))
[...]
(make-custom-binary-input-port "chunked input port" read! #f #f close))
--8<---------------cut here---------------end--------------->8---
This is undesirable because:
1. the HTTP server can produce arbitrarily large chunks, leading to
large memory use in the client (nginx does indeed produce very
large chunks in some cases);
2. it adds an extra level of buffering that the caller of ‘http-get’
does not control (a read of 1 byte from the HTTP body port leads
to an actual read of a whole chunk);
3. it introduces extra copying and allocations.
Ludo’.
next reply other threads:[~2015-02-24 20:59 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-02-24 20:59 Ludovic Courtès [this message]
2015-03-01 22:44 ` bug#19939: http client: Chunks shouldn't be read at once Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/guile/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87ioer9gbq.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=19939@debbugs.gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).