From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 97E1D1FB0E for ; Thu, 11 Mar 2021 10:45:38 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 4/7] doc: update 1.7 release notes, tuning, TODO Date: Thu, 11 Mar 2021 02:45:35 -0800 Message-Id: <20210311104538.7743-5-e@80x24.org> In-Reply-To: <20210311104538.7743-1-e@80x24.org> References: <20210311104538.7743-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: Some stuff done, some stuff still needs doing. --- Documentation/RelNotes/v1.7.0.wip | 59 ++++++++++++++++++++++++++- Documentation/public-inbox-tuning.pod | 9 ++++ TODO | 16 +++----- 3 files changed, 72 insertions(+), 12 deletions(-) diff --git a/Documentation/RelNotes/v1.7.0.wip b/Documentation/RelNotes/v1.7.0.wip index a35ff227..f71f447f 100644 --- a/Documentation/RelNotes/v1.7.0.wip +++ b/Documentation/RelNotes/v1.7.0.wip @@ -4,12 +4,69 @@ MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline -TODO: gcf2, detached indices, JMAP, ... +Another big release focused on multi-inbox search and scalability. + +* general changes + + config file parsing is 2x faster with 50K inboxes + +* read-only public-inbox-daemon (-httpd, -nntpd, -imapd): + + libgit2 may be used via Inline::C to avoid hitting system pipe + and process limits. See public-inbox-tuning(7) manpage + for more details. + +* public-inbox-extindex + + A new Xapian + SQLite index able to search across several inboxes. + This may be configured to replace per-inbox Xapian DBs, + (but not per-inbox SQLite indices) and speed up manifest.js.gz + generation. + + See public-inbox-extindex-format(5) and + public-inbox-extindex(1) manpages for more details. + +* public-inbox-nntpd + + - startup is 6x faster with 50K inboxes if using -extindex + +* PublicInbox::WWW + + - mboxrd search results are returned in reverse Xapian docid order, + so more recent results are more likely to show up first + + - d: and dt: search prefixes allow "approxidate" formats supported + by "git log --since=" + + - manifest.js.gz generation is ~25x faster with -extindex + +* lei - local email interface + + An experimental, subject-to-change, likely-to-eat-your-mail tool for + personal mail as well as interacting with public-inboxes on the local + filesystem or over HTTP(S). See lei(1), lei-overview(7), and other + lei-* manpages for details. + +* public-inbox-watch + + - IMAP and NNTP code shared with lei, fixing an off-by-one error + in IMAP synchronization for single-message IMAP folders. + + - \Deleted and \Draft messages ignored for IMAP, as they are for + Maildir. + + - IMAP and NNTP connection establishment (including git-credential + prompts) ordering is now tied to config file order. Compatibility: * Rollbacks all the way to public-inbox 1.2.0 remain supported +Internal changes + +* public-inbox-index switched to new internal IPC code shared + with lei + Please report bugs via plain-text mail to: meta@public-inbox.org See archives at https://public-inbox.org/meta/ for all history. diff --git a/Documentation/public-inbox-tuning.pod b/Documentation/public-inbox-tuning.pod index e9702416..b3a2b411 100644 --- a/Documentation/public-inbox-tuning.pod +++ b/Documentation/public-inbox-tuning.pod @@ -55,6 +55,15 @@ public-inbox processes. More (optional) L use will be introduced in the future to lower memory use and improve scalability. +=head2 libgit2 usage via Inline::C + +If libgit2 development files are installed and L +is enabled (described above), per-inbox C +processes are replaced with a single L process running +C in read-only daemons. + +Available as of public-inbox 1.7.0. + =head2 Performance on rotational hard disk drives Random I/O performance is poor on rotational HDDs. Xapian indexing diff --git a/TODO b/TODO index 53907efd..4993b02c 100644 --- a/TODO +++ b/TODO @@ -86,7 +86,10 @@ all need to be considered for everything we introduce) * more and better test cases (use git fast-import to speed up creation) -* large mbox/Maildir/MH/NNTP spool import (see PublicInbox::Import) +* large mbox/Maildir/MH/NNTP spool import (in lei, but not + for public-facing inboxes) + +* MH import support (read-only, at least) * Read-only WebDAV interface to the git repo so it can be mounted via davfs2 or fusedav to avoid full clones. @@ -133,18 +136,9 @@ all need to be considered for everything we introduce) - inotify-based manifest.js.gz updates - - process/FD reduction (needs to be slow-storage friendly) - ... -* command-line tool (similar to mairix/notmuch, but solver+git-aware) - -* consider removing doc_data from Xapian, redundant with over.sqlite3 - It's no longer read as of public-inbox 1.6.0, but still written for - compatibility. - -* share "git cat-file --batch" processes across inboxes to avoid - bumping into /proc/sys/fs/pipe-user-pages-* limits +* lei - see %CMD in lib/PublicInbox/LEI.pm * make "git cat-file --batch" detect unlinked packfiles so we don't have to restart processes (very long-term)