bug#15368: HTTP client is slow [2.0.9]

[ludo@gnu.org (Ludovic Courtès)]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> I just noticed that our HTTP client is very slow.  Consider this:
>>
>> (use-modules (web client)
>>              (rnrs io ports)
>>              (rnrs bytevectors)
>>              (srfi srfi-11)
>>              (ice-9 format))
>>
>> (define %uri
>>   "http://ftp.gnu.org/gnu/idutils/idutils-4.6.tar.xz")
>>
>> (with-fluids ((%default-port-encoding #f))
>>   (let*-values (((start)
>>                  (gettimeofday))
>>                 ((p)
>>                  (let ((s (open-socket-for-uri %uri)))
>>                    (setvbuf s _IONBF)
>
> Why are you using an unbuffered port?  On my system, changing this to
> _IOFBF increases throughput from 326 KiB/s to 489.0 KiB/s.

Arf, that’s because I was also forcing the ‘scm_c_read’ hack (which
is currently never used, and this is a bug):


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 654 bytes --]

diff --git a/libguile/ports.c b/libguile/ports.c
index 9068c5c..c217712 100644
--- a/libguile/ports.c
+++ b/libguile/ports.c
@@ -1657,7 +1657,8 @@ scm_c_read (SCM port, void *buffer, size_t size)
      requested number of bytes.  (Note that a single scm_i_fill_input
      call does not guarantee to fill the whole of the port's read
      buffer.) */
-  if (pt->read_buf_size <= 1 && pt->encoding == NULL)
+  if (pt->read_buf_size <= 1
+      && (pt->encoding == NULL || strcmp (pt->encoding, "ISO-8859-1") == 0))
     {
       /* The port that we are reading from is unbuffered - i.e. does
 	 not have its own persistent buffer - but we have a buffer,

[-- Attachment #3: Type: text/plain, Size: 855 bytes --]


So in practice it was reading several KiB at a time, doing zero-copy.

> Also, the fact that my throughput is so much higher than yours (on a
> several-year-old computer) is interesting.  Obviously I have a faster
> net connection (wget reports 1.19M/s),

So for you wget is ~2.5 times faster than Guile, right?

[...]

>> Looking at the strace output reveals no real difference: they all make
>> one syscall for each chunk of 1410 bytes.
>>
>> ‘time’ reports that Guile spends 0.2 s. in user and 0.8 s. in system,
>> both of which are an order of magnitude higher than wget/curl.
>
> If they make essentially the same syscalls, then why would the system
> time be an order of magnitude higher?  Something doesn't sound right
> here.

I concur.

I’ve tried Linux perf and OProfile but failed to get useful info.

Ludo’.