From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id C0C771F55B; Tue, 9 Jun 2020 11:34:42 +0000 (UTC) Date: Tue, 9 Jun 2020 11:34:42 +0000 From: Eric Wong To: meta@public-inbox.org Subject: IMAP server notes, maybe JMAP? Message-ID: <20200609113442.GA16856@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline List-Id: OK, so I almost have something that won't kill clients or trigger OOMs on the server. I think I'll have to implement MSN (message sequence numbers) properly for some clients, cheaply. I know there's also interest in getting search usable via an HTTP(S) API, so maybe JMAP[1] is worth looking into since it seems like an easier-to-implement take on IMAP; and both have search. We already use JSON for manifest.js.gz... [1] https://lwn.net/Articles/680057/ - (a look at JMAP). GraphQL has also come up privately, but at a glance it seems to suffer some of the same problems with IMAP in offering excessive granularity and cache-unfriendliness. But anyways, I'm close to having a pretty good read-only IMAP server. JMAP cannot be any harder than IMAP right? :) Anyways, the IMAP server is based on the existing NNTP server code; but also provides a fresh place to test new ideas for improving scalability and performance: * It's fair to other clients if using slow blob storage, and I've gotten to be fairly happy with the API. Slow Xapian storage isn't accounted for, yet, but will be. Aggressive pipelining in mbsync(1) helped me get async bugs ironed out. * LIST wildcard matching/globbing tested with 100K inboxes, other pieces still need work. * Incremental config reloading on SIGHUP works, but could be more granular. * IMAP IDLE supported for EXISTS (but not EXPUNGE for spam). * Inotify (or EVFILT_VNODE) used for IMAP IDLE notifications. More inotify/EVFILT_VNODE usage to come for manifest.js.gz updates, git-cat-file restarts, DB reopens, etc. I've used inotify a bunch in the past, but not in Perl. This is my first time with EVFILT_VNODE, but inotify seems more capable. * --reindex is recommended for the RFC822.SIZE fetch attribute to account for CRLF conversion. Actually, we've been wrong about NNTP :bytes reporting for years, too, but IMAP clients (at least Mail::IMAPClient) actually complain. * Common cases of Xapian search work, but also require --reindex. Reindex is doable while serving, results just won't show until reindexed. * COMPRESS, STARTTLS, TLS are all inherited from -nntpd * PublicInbox::Eml can descend in to message/* subparts properly, which makes it possible to implement stuff like BODYSTRUCTURE properly. And who uses BODYSTRUCTURE? And a note: * Don't create 100K public-inboxes in the worktree itself, MakeMaker looks inside all directories when doing "perl Makefile.PL" :x TODO: * Queries involving OR, NOT, and parentheses don't work, yet, since Xapian's default query parser works differently than the prefix (Polish) notation of IMAP. * IMAP extensions are worth looking into, especially ones around search. Some of them look interesting w.r.t. search, along with overlap with JMAP. * There's also thread-related stuff and that may be able to implement "Search based on data in follow-ups" https://public-inbox.org/meta/20200526191745.34vynfasnf3amyjq@chatter.i7.local/ * An "All Mail" mailbox could be cool for IMAP/JMAP search (it's planned for HTTP, anyways) I hit numerous bugs in 3rd-party libraries while working on the server, all of them involving around compression: * Python imaplib2 (via offlineimap) - compress timeout: https://bugs.debian.org/961713 * Mail::IMAPClient - compress reference cycle: https://rt.cpan.org/Ticket/Display.html?id=132654 - compress read starvation: https://rt.cpan.org/Ticket/Display.html?id=132720 * Compress::Raw::Zlib - inflate appending to OOK scalars https://rt.cpan.org/Ticket/Display.html?id=132734 mbsync(1) helped me expose this bug Both Mail::IMAPClient and Mail::IMAPTalk do weird sleeps with select(2) and SSL; maybe as a holdover from the days when IO::Socket::SSL lacked SSL_WANT_READ/SSL_WANT_WRITE? headaches for dealing with MUAs: * Inboxes are split into 50K slices to avoid overloading MUAs. The word "slice" was not previously used in our codebase. Conceptually, it's like v2 "epochs". Epochs are to deal with the limitations of git clients, slices are to deal with the limitations of IMAP clients. * HEADER.FIELDS retrieval still requires taking the whole blob from git. Clients can request any header(s) they wish; unlike NNTP, where the server defines the overview. I've managed to speed this up significantly with a little pure-Perl opcode compiler, though :) * MSNs (message sequence numbers) seem required to get decent performance from mutt and maybe other MUAs. Showing fake dummy messages for removed spam is poor UX, too...