From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Ian Price Newsgroups: gmane.lisp.guile.devel Subject: Re: Chunked Encoding Date: Sun, 06 May 2012 05:52:00 +0100 Message-ID: <87ehqxq5cf.fsf@Kagami.home> References: <8739bvp6h2.fsf@pobox.com> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: dough.gmane.org 1336279938 7727 80.91.229.3 (6 May 2012 04:52:18 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Sun, 6 May 2012 04:52:18 +0000 (UTC) Cc: guile-devel@gnu.org To: Andy Wingo Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Sun May 06 06:52:17 2012 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1SQtSD-0000bo-DJ for guile-devel@m.gmane.org; Sun, 06 May 2012 06:52:17 +0200 Original-Received: from localhost ([::1]:36869 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SQtSC-0007wr-QP for guile-devel@m.gmane.org; Sun, 06 May 2012 00:52:16 -0400 Original-Received: from eggs.gnu.org ([208.118.235.92]:53634) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SQtS9-0007wm-F0 for guile-devel@gnu.org; Sun, 06 May 2012 00:52:14 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1SQtS7-000412-A2 for guile-devel@gnu.org; Sun, 06 May 2012 00:52:13 -0400 Original-Received: from mail-wg0-f49.google.com ([74.125.82.49]:50133) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SQtS6-00040j-UA for guile-devel@gnu.org; Sun, 06 May 2012 00:52:11 -0400 Original-Received: by wgbds1 with SMTP id ds1so3008986wgb.30 for ; Sat, 05 May 2012 21:52:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-type; bh=BQWbYYBMzbZk6td5cMpGB41ruIewJAehDGBi9QWzoYw=; b=Sfla/hcG3KSE8ykO/faBAJZJfoiSr5uakpUvWkJb7ueluXCeblq3mXz1R3Em9DibFV 30XtEEAKXY6DXJw0wgjTUDtI55Ewgl6DIZ+eYJTwsjP5Qh6VI4pPolhElkVm69JTr0Pj OFAwa21TZLwj199+zEKe2p0Udzat7GGwCYjWTEYg+fJzrnssDdf3UHbB8zMrJ9/3RjKq UYA/lJHCEEdDXg+xZOUFgPJ+SnjDdU3zQmQDfdj8Q5Xju9VjvdBi9E8iTGJdIQ7DsqO4 5HmMc+dUf+qZvm9SG+o4h1Ng8FQZMMh26geFgqpnYP8qzdvbu1hFO/enm989qCdb7NF6 cG5Q== Original-Received: by 10.180.24.7 with SMTP id q7mr24612680wif.11.1336279928197; Sat, 05 May 2012 21:52:08 -0700 (PDT) Original-Received: from Kagami.home (host86-171-129-59.range86-171.btcentralplus.com. [86.171.129.59]) by mx.google.com with ESMTPS id ex2sm17183914wib.8.2012.05.05.21.52.05 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 05 May 2012 21:52:06 -0700 (PDT) In-Reply-To: <8739bvp6h2.fsf@pobox.com> (Andy Wingo's message of "Wed, 04 Jan 2012 15:18:01 -0500") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.3 (gnu/linux) X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 74.125.82.49 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.devel:14357 Archived-At: ccing Daniel Hartwig, since he has been a bigger cheerleader for chunked encoding than I have so far :). Andy Wingo writes: > On_Thu 29 Sep 2011 16:09, Ian Price writes: > >> If you've used the (web ...) modules, you may have noticed that guile >> does not currently support chunked-encoding. Sheesh that was a long time ago >> What I did was introduce two new exported procedures for reading (all I >> needed at the moment), namely 'read-chunk' and 'read-chunked-response-body'. >> >> (read-chunk port) >> reads one chunk from a port. >> >> (read-chunked-response-body response) >> read the full body for the response and returns it as a bytevector. It >> was written to be similar to 'read-response-body'. > > Is it possible to use soft ports?. That would be nice, and it would > allow client code to be able to read from the port until it gets EOF, > without being concerned about the transfer-encoding. Same thing goes > for gzip/deflate/compress encoding. It is possible, and I have some code for this, but I had been procrastinating for a while because of a soft ports bug with flush, and once that got fixed, just plain procrastinating. >> Another option I've been thinking over would be to go for a sort of >> chunking version of R6RS' 'transcoded-port' which would handle it >> transparently for users of the returned port. > > Is this equivalent? I don't have much experience with these. It seems > that custom binary input/output ports are more appropriate. Well, what I meant is a port that would be layered over the top of another. Soft ports or custom binary ports would be used to implement it. (Is there a reason (effiencywise) to prefer one over the other?) Basically, my interface right now is (make-chunked-input-port port) -> input-port (make-chunked-output-port port) -> output-port These operate pretty much as you'd expect. The port returned from 'make-chunked-input-port' reads whole chunks from its argument port, and maintains a buffer, from which it can satisfy smaller reads. The port returned from 'make-chunked-output-port' buffers up the writes and writes a whole (properly formatted) chunk on 'force-output'. The only behaviour I'm not entirely sure of is what happens on close. Let's be more concrete: (define a ) (define b (make-chunked-input-port a)) (close-port b) What is the state of a? (define c ) (define d (make-chunked-output-port c)) (close-port d) Likewise for c. I think common practice in things like Java's BufferedReader would be to have a be closed when b is. This may be undesirable since we could wish to continue using the socket. On the other hand, once you start layering ports, it is convenient to have the higher layers close the lower layers. I think it might make sense to have a keyword argument, #:dont-close? (or something), that specifies this behaviour, defaulting to close. c seems less clear to me. Again, once you have multiple layers, it would be convenient, and it would properly handle any inner state saved up. Any thoughts on this? > >> I'd also suggest extending 'http-get' from (web client) to handle >> chunked encoding (and trailers too, I guess) for the user. > > > So, in the default case in which we do not mention "trailers" in the > request, trailer fields should be strictly not required. So the naive > case should just work. I think we can leave trailers until I have some actual data on how much these are actually used in practice, and/or someone complains about it being missing. WDYT? FWIW I didn't see any code for handling them in Ruby's net/http.rb >> Comments kindly requested, > > Again, very sorry for the delay! I hope to be more on top of things > this month. I'm sorry for the delay too. Let's see if we can't finish it sometime in the next week and end this once and for all :) -- Ian Price "Programming is like pinball. The reward for doing it well is the opportunity to do it again" - from "The Wizardy Compiled"