optimization for pop movemail

optimization for pop movemail

[Ken Raeburn]

The POP support in movemail runs afoul of a combination of otherwise
reasonable optimizations in most TCP stacks, Nagle and delayed acks.
It sends (for example) "RETR 1234" and then there's a delay before it
sends "\r\n", because we make two write() calls.  It can add something
like 0.4 seconds per message retrieved, which adds up with a lot of
messages.

Is this approach okay, or should I go change the code to include the
\r\n at all the call sites instead?
Should this wait until after the release?

Ken


--- lib-src/pop.c	1 Sep 2003 15:45:03 -0000	1.34
+++ lib-src/pop.c	3 Jul 2005 22:04:03 -0000
@@ -1404,10 +1393,39 @@ sendline (server, line)
 #define SENDLINE_ERROR "Error writing to POP server: "
   int ret;
 
-  ret = fullwrite (server->file, line, strlen (line));
-  if (ret >= 0)
-    {				/* 0 indicates that a blank line was written */
-      ret = fullwrite (server->file, "\r\n", 2);
+  if (line == pop_error && strlen (line) < sizeof (pop_error) - 5)
+    {
+      /* This minor "abstraction" violation can save a fraction of a
+	 second per message sent in a fast, reliable TCP network
+	 environment with delayed acks and Nagle algorithm.  (Movemail
+	 writes line without \r\n, client kernel sends it, server
+	 kernel delays ack to see if it can combine it with data,
+	 movemail writes \r\n, client kernel waits because it has
+	 unacked data, client kernel eventually times out and sends.)
+
+	 On a NetBSD box, this delay is 0.2 seconds per message; if
+	 you've got a few dozen messages or so, it adds up, and if
+	 they're small and the server is close, it can be a
+	 significant fraction of the execution time.
+
+	 Turning off Nagle would probably change this to two packets
+	 in rapid succession for most implementations.  If we can make
+	 it just one write call, we'll likely get one packet and keep
+	 everybody happier.
+
+	 Fortunately, most of the formatted calls (e.g., all those
+	 including message numbers) use pop_error as the buffer
+	 into which the command is written.  */
+      strcat (line, "\r\n");
+      ret = fullwrite (server->file, line, strlen (line));
+    }
+  else
+    {
+      ret = fullwrite (server->file, line, strlen (line));
+      if (ret >= 0)
+	{	       /* 0 indicates that a blank line was written */
+	  ret = fullwrite (server->file, "\r\n", 2);
+	}
     }
 
   if (ret < 0)

Re: optimization for pop movemail

[Richard M. Stallman]

Thanks for noticing this issue.

+  if (line == pop_error && strlen (line) < sizeof (pop_error) - 5)

That code is a horrible kludge.  It would be better to copy the string
to a temporary buffer and add the \r\n there.  You could allocate it
with alloca.  That is clean, and you could do it unconditionally.

This may as well be fixed now--why wait?

Re: optimization for pop movemail

[Ken Raeburn]

> +  if (line == pop_error && strlen (line) < sizeof (pop_error) - 5)
>
> That code is a horrible kludge.

Yeah, true... even worse than the existing kludge of using the 
error-return buffer, sometimes, to hold the message to be sent out.

>   It would be better to copy the string
> to a temporary buffer and add the \r\n there.  You could allocate it
> with alloca.  That is clean, and you could do it unconditionally.

Okay, will do.

> This may as well be fixed now--why wait?

Wasn't sure if we were very close to a release or something.  I haven't 
seen any mail indicating it, but I have seen mail talking about 
freezing some kinds of changes, and I can't quite keep up on all the 
list mail so I might've missed something if it was under a subject line 
that looked uninteresting to me.  And it's debatable whether this is a 
bug fix or just a minor efficiency issue.

I'll revise the patch and check it in, should be sometime this week...

Ken

Re: optimization for pop movemail

[Richard M. Stallman]

    I'll revise the patch and check it in, should be sometime this week...

Thank you.