unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
From: Eric Wong <e@80x24.org>
To: "Eric W. Biederman" <ebiederm@xmission.com>
Cc: meta@public-inbox.org
Subject: Re: I have figured out IMAP IDLE
Date: Tue, 29 Oct 2019 22:31:58 +0000	[thread overview]
Message-ID: <20191029223158.GA26139@dcvr> (raw)
In-Reply-To: <87eeyvmx74.fsf@x220.int.ebiederm.org>

"Eric W. Biederman" <ebiederm@xmission.com> wrote:
> 
> A few days ago I stumbled upon this magic decoder ring for IMAP.
> The "Ten Commandments of How to Write an IMAP client"
> 
> https://www.washington.edu/imap/documentation/commndmt.txt
> 
> The part I was most clearly missing was that for IMAP it is better to
> open multiple sockets (one per mail folder on the server) than it is to
> switch between folders with the same ssl socket.
> 
> The core loop of my imap code now does:
> 
> 	for (;;) {
> 		my $more;
> 		do {
> 			$more = fetch_mailbox($config, $tracker, $client, $mailbox, $validity);
> 		} while ($more > 0);
> 
> 		my @idle_data;
> 		do {
> 			my $max_idle = 15;
> 			$client->idle() or die("idle failed!\n");

			b) window closed

> 			@idle_data = $client->idle_data($max_idle*60);
> 			$client->done();

			a) window opens

> 		} while (scalar(@idle_data) == 0);
> 	}

It seems between a) and b), there's a small window where a message
can appear and not be noticed right away.

I think the only reliable way to do this without experiencing
delays or an opportunity for missed messages is to have two(!)
connections per folder:

1) fetcher - this connection only fetches, either at startup or
   when triggered by the idler described below.

2) idler - just runs ->idle, ->idle_data and ->done
   If ->idle times out or sees ->idle_data, it triggers
   fetcher.  The trigger should probably have a timestamp
   so the fetcher can merge overlapping triggers.

Unfortunately, using widely-available Perl IMAP client libraries
means two Perl processes for every folder.  But I suppose
fetcher can be lazy-started to not waste resources.

I wonder if WWW::Curl can do it all in one Perl process with an
event loop.  curl can issue arbitrary commands for HTTP and I've
been using it to learn some XS, so maybe it can do IDLE.

WWW::Curl is also packaged for CentOS/RHEL7, so it should not be
tough to install.

> Where all I do with idle is wait until it gives me something, and
> reissue it every 15 minutes so the connection doesn't time out.

Btw, why not 29 minutes? (accounting for 30 minute timeout).

> The only filter I have found useful to add is when imap idle returns
> nothing to just loop instead of trying to fetch mail.  I have not found
> anything in the data idle returns that would save me work in the mailbox
> fetching loop.

Yeah, I recall that being the case when I last worked with
IMAP in other places.

> In my limited testing firing up a bunch of ssl sockets and fetching from
> the all simultaneously appears much faster, and maybe a little more
> stable.  Than switching between mailboxes.

Thanks for the info!

Fwiw, I favor using stunnel for developing/testing client-side
mail tools with TLS.  It's easier to debug/trace a tool that's
talking over loopback to stunnel using tools like strace or tcpdump.

  reply	other threads:[~2019-10-29 22:31 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-29 14:40 I have figured out IMAP IDLE Eric W. Biederman
2019-10-29 22:31 ` Eric Wong [this message]
2019-10-29 23:12   ` WWW::Curl [was: I have figured out IMAP IDLE] Eric Wong
2019-11-03 16:28   ` I have figured out IMAP IDLE Eric W. Biederman
2020-05-13 19:31 ` Eric Wong
2020-05-13 21:48   ` Eric W. Biederman
2020-05-13 22:17     ` Eric Wong
2020-05-14 12:32       ` Eric W. Biederman
2020-05-14 16:15         ` Eric Wong
2020-05-15 21:00         ` [PATCH 1/2] IMAPTracker: Add a helper to track our place in reading imap mailboxes Eric W. Biederman
2020-05-15 21:02           ` [PATCH 2/2] imap_fetch: Add a command to continuously fetch from an imap mailbox Eric W. Biederman
2020-05-15 21:26             ` Eric W. Biederman
2020-05-15 22:56               ` Eric Wong
2020-05-16 10:47                 ` Eric W. Biederman
2020-05-16 19:12                   ` Eric Wong
2020-05-16 20:09                     ` Eric W. Biederman
2020-05-16 22:53                       ` [PATCH] confine Email::MIME use even further Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191029223158.GA26139@dcvr \
    --to=e@80x24.org \
    --cc=ebiederm@xmission.com \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).