Re: I have figured out IMAP IDLE

[Eric Wong <e@80x24.org>]

"Eric W. Biederman" <ebiederm@xmission.com> wrote:
> 
> A few days ago I stumbled upon this magic decoder ring for IMAP.
> The "Ten Commandments of How to Write an IMAP client"
> 
> https://www.washington.edu/imap/documentation/commndmt.txt
> 
> The part I was most clearly missing was that for IMAP it is better to
> open multiple sockets (one per mail folder on the server) than it is to
> switch between folders with the same ssl socket.
> 
> The core loop of my imap code now does:
> 
> 	for (;;) {
> 		my $more;
> 		do {
> 			$more = fetch_mailbox($config, $tracker, $client, $mailbox, $validity);
> 		} while ($more > 0);
> 
> 		my @idle_data;
> 		do {
> 			my $max_idle = 15;
> 			$client->idle() or die("idle failed!\n");

			b) window closed

> 			@idle_data = $client->idle_data($max_idle*60);
> 			$client->done();

			a) window opens

> 		} while (scalar(@idle_data) == 0);
> 	}

It seems between a) and b), there's a small window where a message
can appear and not be noticed right away.

I think the only reliable way to do this without experiencing
delays or an opportunity for missed messages is to have two(!)
connections per folder:

1) fetcher - this connection only fetches, either at startup or
   when triggered by the idler described below.

2) idler - just runs ->idle, ->idle_data and ->done
   If ->idle times out or sees ->idle_data, it triggers
   fetcher.  The trigger should probably have a timestamp
   so the fetcher can merge overlapping triggers.

Unfortunately, using widely-available Perl IMAP client libraries
means two Perl processes for every folder.  But I suppose
fetcher can be lazy-started to not waste resources.

I wonder if WWW::Curl can do it all in one Perl process with an
event loop.  curl can issue arbitrary commands for HTTP and I've
been using it to learn some XS, so maybe it can do IDLE.

WWW::Curl is also packaged for CentOS/RHEL7, so it should not be
tough to install.

> Where all I do with idle is wait until it gives me something, and
> reissue it every 15 minutes so the connection doesn't time out.

Btw, why not 29 minutes? (accounting for 30 minute timeout).

> The only filter I have found useful to add is when imap idle returns
> nothing to just loop instead of trying to fetch mail.  I have not found
> anything in the data idle returns that would save me work in the mailbox
> fetching loop.

Yeah, I recall that being the case when I last worked with
IMAP in other places.

> In my limited testing firing up a bunch of ssl sockets and fetching from
> the all simultaneously appears much faster, and maybe a little more
> stable.  Than switching between mailboxes.

Thanks for the info!

Fwiw, I favor using stunnel for developing/testing client-side
mail tools with TLS.  It's easier to debug/trace a tool that's
talking over loopback to stunnel using tools like strace or tcpdump.