From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 039EF1F4C0; Tue, 29 Oct 2019 22:31:59 +0000 (UTC) Date: Tue, 29 Oct 2019 22:31:58 +0000 From: Eric Wong To: "Eric W. Biederman" Cc: meta@public-inbox.org Subject: Re: I have figured out IMAP IDLE Message-ID: <20191029223158.GA26139@dcvr> References: <87eeyvmx74.fsf@x220.int.ebiederm.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <87eeyvmx74.fsf@x220.int.ebiederm.org> List-Id: "Eric W. Biederman" wrote: > > A few days ago I stumbled upon this magic decoder ring for IMAP. > The "Ten Commandments of How to Write an IMAP client" > > https://www.washington.edu/imap/documentation/commndmt.txt > > The part I was most clearly missing was that for IMAP it is better to > open multiple sockets (one per mail folder on the server) than it is to > switch between folders with the same ssl socket. > > The core loop of my imap code now does: > > for (;;) { > my $more; > do { > $more = fetch_mailbox($config, $tracker, $client, $mailbox, $validity); > } while ($more > 0); > > my @idle_data; > do { > my $max_idle = 15; > $client->idle() or die("idle failed!\n"); b) window closed > @idle_data = $client->idle_data($max_idle*60); > $client->done(); a) window opens > } while (scalar(@idle_data) == 0); > } It seems between a) and b), there's a small window where a message can appear and not be noticed right away. I think the only reliable way to do this without experiencing delays or an opportunity for missed messages is to have two(!) connections per folder: 1) fetcher - this connection only fetches, either at startup or when triggered by the idler described below. 2) idler - just runs ->idle, ->idle_data and ->done If ->idle times out or sees ->idle_data, it triggers fetcher. The trigger should probably have a timestamp so the fetcher can merge overlapping triggers. Unfortunately, using widely-available Perl IMAP client libraries means two Perl processes for every folder. But I suppose fetcher can be lazy-started to not waste resources. I wonder if WWW::Curl can do it all in one Perl process with an event loop. curl can issue arbitrary commands for HTTP and I've been using it to learn some XS, so maybe it can do IDLE. WWW::Curl is also packaged for CentOS/RHEL7, so it should not be tough to install. > Where all I do with idle is wait until it gives me something, and > reissue it every 15 minutes so the connection doesn't time out. Btw, why not 29 minutes? (accounting for 30 minute timeout). > The only filter I have found useful to add is when imap idle returns > nothing to just loop instead of trying to fetch mail. I have not found > anything in the data idle returns that would save me work in the mailbox > fetching loop. Yeah, I recall that being the case when I last worked with IMAP in other places. > In my limited testing firing up a bunch of ssl sockets and fetching from > the all simultaneously appears much faster, and maybe a little more > stable. Than switching between mailboxes. Thanks for the info! Fwiw, I favor using stunnel for developing/testing client-side mail tools with TLS. It's easier to debug/trace a tool that's talking over loopback to stunnel using tools like strace or tcpdump.