From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS6315 166.70.0.0/16 X-Spam-Status: No, score=-3.6 required=3.0 tests=AWL,BAYES_00, RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from out03.mta.xmission.com (out03.mta.xmission.com [166.70.13.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 92D951F454; Sun, 3 Nov 2019 16:28:37 +0000 (UTC) Received: from in01.mta.xmission.com ([166.70.13.51]) by out03.mta.xmission.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.87) (envelope-from ) id 1iRIk0-0001L6-5W; Sun, 03 Nov 2019 09:28:36 -0700 Received: from ip68-227-160-95.om.om.cox.net ([68.227.160.95] helo=x220.xmission.com) by in01.mta.xmission.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.87) (envelope-from ) id 1iRIjy-0007qL-SN; Sun, 03 Nov 2019 09:28:36 -0700 From: ebiederm@xmission.com (Eric W. Biederman) To: Eric Wong Cc: meta@public-inbox.org References: <87eeyvmx74.fsf@x220.int.ebiederm.org> <20191029223158.GA26139@dcvr> Date: Sun, 03 Nov 2019 10:28:24 -0600 In-Reply-To: <20191029223158.GA26139@dcvr> (Eric Wong's message of "Tue, 29 Oct 2019 22:31:58 +0000") Message-ID: <87r22ogc0n.fsf@x220.int.ebiederm.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-XM-SPF: eid=1iRIjy-0007qL-SN;;;mid=<87r22ogc0n.fsf@x220.int.ebiederm.org>;;;hst=in01.mta.xmission.com;;;ip=68.227.160.95;;;frm=ebiederm@xmission.com;;;spf=neutral X-XM-AID: U2FsdGVkX18SqIwZ7cWM+hzwlLDBVM3iAu97VZz6V3U= X-SA-Exim-Connect-IP: 68.227.160.95 X-SA-Exim-Mail-From: ebiederm@xmission.com Subject: Re: I have figured out IMAP IDLE X-SA-Exim-Version: 4.2.1 (built Thu, 05 May 2016 13:38:54 -0600) X-SA-Exim-Scanned: Yes (on in01.mta.xmission.com) List-Id: Eric Wong writes: > "Eric W. Biederman" wrote: >> >> A few days ago I stumbled upon this magic decoder ring for IMAP. >> The "Ten Commandments of How to Write an IMAP client" >> >> https://www.washington.edu/imap/documentation/commndmt.txt >> >> The part I was most clearly missing was that for IMAP it is better to >> open multiple sockets (one per mail folder on the server) than it is to >> switch between folders with the same ssl socket. >> >> The core loop of my imap code now does: >> >> for (;;) { >> my $more; >> do { >> $more = fetch_mailbox($config, $tracker, $client, $mailbox, $validity); >> } while ($more > 0); >> >> my @idle_data; >> do { >> my $max_idle = 15; >> $client->idle() or die("idle failed!\n"); > > b) window closed > >> @idle_data = $client->idle_data($max_idle*60); >> $client->done(); > > a) window opens > >> } while (scalar(@idle_data) == 0); >> } > > It seems between a) and b), there's a small window where a message > can appear and not be noticed right away. Hmm. Looking at the source of IMAPClient point (b) can be moved to after "$client->done()" by reading the output "$client->done()". About the window for unsolicited untagged responses before the "$client->idle()" command. Those unsolicited responses should queue up in the imap socket until the next read. That next read should happen after the "$client->idle()" command. So in practice unsolicited untagged responses should just look like a reply to imap idle. So the race you have noticed does not look fundamental rather it looks like it simply takes bug fixing to make go away. > I think the only reliable way to do this without experiencing > delays or an opportunity for missed messages is to have two(!) > connections per folder: That sounds nice. But it is also _not_ recommended. Quoting from the "Ten Commandments of How to Write an IMAP Client" 1. Thou shalt not assume that it is alright to open multiple IMAP sessions selected on the same mailbox simultaneously, lest thou face the righteous wrath of mail stores that doth not permit such access. Instead, thou shalt labor mightily, even unto having to use thy brain to thinketh the matter through, such that thy client use existing sessions that are already open. > 1) fetcher - this connection only fetches, either at startup or > when triggered by the idler described below. > > 2) idler - just runs ->idle, ->idle_data and ->done > If ->idle times out or sees ->idle_data, it triggers > fetcher. The trigger should probably have a timestamp > so the fetcher can merge overlapping triggers. > > Unfortunately, using widely-available Perl IMAP client libraries > means two Perl processes for every folder. But I suppose > fetcher can be lazy-started to not waste resources. > > I wonder if WWW::Curl can do it all in one Perl process with an > event loop. curl can issue arbitrary commands for HTTP and I've > been using it to learn some XS, so maybe it can do IDLE. > > WWW::Curl is also packaged for CentOS/RHEL7, so it should not be > tough to install. > >> Where all I do with idle is wait until it gives me something, and >> reissue it every 15 minutes so the connection doesn't time out. > > Btw, why not 29 minutes? (accounting for 30 minute timeout). Silliness. In the example I started from someone had a comment about firewalls not honoring a full 30 minutes so I picked 15 minutes arbitrarily. >> The only filter I have found useful to add is when imap idle returns >> nothing to just loop instead of trying to fetch mail. I have not found >> anything in the data idle returns that would save me work in the mailbox >> fetching loop. > > Yeah, I recall that being the case when I last worked with > IMAP in other places. > >> In my limited testing firing up a bunch of ssl sockets and fetching from >> the all simultaneously appears much faster, and maybe a little more >> stable. Than switching between mailboxes. > > Thanks for the info! > > Fwiw, I favor using stunnel for developing/testing client-side > mail tools with TLS. It's easier to debug/trace a tool that's > talking over loopback to stunnel using tools like strace or tcpdump. I may check that out. Certainly something that gives you the raw protocol messages is a little bit cleaner. I haven't had enough challenges that I have been looking for the raw protocol messages yet. Eric