From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Mark H Weaver Newsgroups: gmane.lisp.guile.bugs Subject: bug#15368: HTTP client is slow [2.0.9] Date: Fri, 13 Sep 2013 12:07:01 -0400 Message-ID: <8738p8ln8a.fsf@tines.lan> References: <87d2oc260o.fsf@gnu.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: ger.gmane.org 1379088509 29959 80.91.229.3 (13 Sep 2013 16:08:29 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Fri, 13 Sep 2013 16:08:29 +0000 (UTC) Cc: 15368@debbugs.gnu.org To: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Original-X-From: bug-guile-bounces+guile-bugs=m.gmane.org@gnu.org Fri Sep 13 18:08:32 2013 Return-path: Envelope-to: guile-bugs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1VKVus-0007ZJ-TR for guile-bugs@m.gmane.org; Fri, 13 Sep 2013 18:08:19 +0200 Original-Received: from localhost ([::1]:48858 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VKVur-0005eg-U6 for guile-bugs@m.gmane.org; Fri, 13 Sep 2013 12:08:17 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:60504) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VKVuj-0005e6-9a for bug-guile@gnu.org; Fri, 13 Sep 2013 12:08:15 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VKVud-0008TE-Dv for bug-guile@gnu.org; Fri, 13 Sep 2013 12:08:09 -0400 Original-Received: from debbugs.gnu.org ([140.186.70.43]:54051) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VKVud-0008T7-9X for bug-guile@gnu.org; Fri, 13 Sep 2013 12:08:03 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.80) (envelope-from ) id 1VKVuc-0004PV-4H for bug-guile@gnu.org; Fri, 13 Sep 2013 12:08:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Mark H Weaver Original-Sender: "Debbugs-submit" Resent-CC: bug-guile@gnu.org Resent-Date: Fri, 13 Sep 2013 16:08:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 15368 X-GNU-PR-Package: guile X-GNU-PR-Keywords: Original-Received: via spool by 15368-submit@debbugs.gnu.org id=B15368.137908844216901 (code B ref 15368); Fri, 13 Sep 2013 16:08:02 +0000 Original-Received: (at 15368) by debbugs.gnu.org; 13 Sep 2013 16:07:22 +0000 Original-Received: from localhost ([127.0.0.1]:34110 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VKVtx-0004OW-IO for submit@debbugs.gnu.org; Fri, 13 Sep 2013 12:07:22 -0400 Original-Received: from world.peace.net ([96.39.62.75]:59916 ident=hope5) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VKVts-0004OL-9E for 15368@debbugs.gnu.org; Fri, 13 Sep 2013 12:07:19 -0400 Original-Received: from 209-6-91-212.c3-0.smr-ubr1.sbo-smr.ma.cable.rcn.com ([209.6.91.212] helo=tines.lan) by world.peace.net with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.72) (envelope-from ) id 1VKVtk-0001e2-Oo; Fri, 13 Sep 2013 12:07:08 -0400 In-Reply-To: <87d2oc260o.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Fri, 13 Sep 2013 15:41:27 +0200") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 140.186.70.43 X-BeenThere: bug-guile@gnu.org List-Id: "Bug reports for GUILE, GNU's Ubiquitous Extension Language" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guile-bounces+guile-bugs=m.gmane.org@gnu.org Original-Sender: bug-guile-bounces+guile-bugs=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.bugs:7282 Archived-At: Hi Ludovic, ludo@gnu.org (Ludovic Court=C3=A8s) writes: > I just noticed that our HTTP client is very slow. Consider this: > > (use-modules (web client) > (rnrs io ports) > (rnrs bytevectors) > (srfi srfi-11) > (ice-9 format)) > > (define %uri > "http://ftp.gnu.org/gnu/idutils/idutils-4.6.tar.xz") > > (with-fluids ((%default-port-encoding #f)) > (let*-values (((start) > (gettimeofday)) > ((p) > (let ((s (open-socket-for-uri %uri))) > (setvbuf s _IONBF) Why are you using an unbuffered port? On my system, changing this to _IOFBF increases throughput from 326 KiB/s to 489.0 KiB/s. Also, the fact that my throughput is so much higher than yours (on a several-year-old computer) is interesting. Obviously I have a faster net connection (wget reports 1.19M/s), but the fact that Guile can benefit so much from my faster connection suggests that the body is read reasonably efficiently. I guess the problem is added latency somewhere, or perhaps inefficiency in the writing of the request or reading of the response headers. Note that using an unbuffered port means that all the reads of the response headers will be done 1 byte at a time. > s)) > ((r h) > (http-get %uri > #:port p > #:streaming? #t > #:decode-body? #f)) > ((d len) > (let ((b (get-bytevector-all h))) > (values b (bytevector-length b))) > ;; (let ((b (make-bytevector (* 5 (expt 2 20))))) > ;; (values b > ;; (get-bytevector-n! h b 0 (bytevector-length= b)))) > ) > ((end) > (gettimeofday)) > ((throughput) > (let ((duration (- (car end) (car start)))) > (/ (/ len 1024.) duration 1.0)))) > (format #t "~5,1f KiB/s (total: ~5,1f KiB)~%" > throughput (/ len 1024.)))) > > Here I get a throughput of ~60 KiB/s, vs. ~400 KiB/s for wget and curl. > > Looking at the strace output reveals no real difference: they all make > one syscall for each chunk of 1410 bytes. > > =E2=80=98time=E2=80=99 reports that Guile spends 0.2 s. in user and 0.8 s= . in system, > both of which are an order of magnitude higher than wget/curl. If they make essentially the same syscalls, then why would the system time be an order of magnitude higher? Something doesn't sound right here. Regards, Mark