From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id A4A761F55B for ; Wed, 10 Jun 2020 18:39:09 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH] doc: update TODO and WIP 1.6.0 release notes Date: Wed, 10 Jun 2020 18:39:09 +0000 Message-Id: <20200610183909.8402-1-e@yhbt.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: Lots of big changes coming Thanks to The Linux Foundation for sponsoring me to hack on this in 2020 :) --- Documentation/RelNotes/v1.6.0.eml | 45 +++++++++++++++++++++++++++++++ TODO | 33 ++++++++++++++++++++--- 2 files changed, 74 insertions(+), 4 deletions(-) create mode 100644 Documentation/RelNotes/v1.6.0.eml diff --git a/Documentation/RelNotes/v1.6.0.eml b/Documentation/RelNotes/v1.6.0.eml new file mode 100644 index 00000000000..283f42c898f --- /dev/null +++ b/Documentation/RelNotes/v1.6.0.eml @@ -0,0 +1,45 @@ +From: Eric Wong +To: meta@public-inbox.org +Subject: [WIP] public-inbox 1.6.0 +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline + +* General changes: + + - ~/.cache/public-inbox/inline-c is automatically used for Inline::C + if it exists. PERL_INLINE_DIRECTORY in env remains supported + and prioritized to support `nobody'-type users without HOME. + + - msgmap.sqlite3 uses journal_mode=TRUNCATE, matching over.sqlite3 + behavior for a minor reduction in VFS traffic + + - message/{rfc822,news,global} attachments are decoded recursively + and indexed for search. Use `public-inbox-index --reindex' to + ensure these attachments are indexed in old messages. + +* public-inbox-index + + - --batch-size=BYTES or publicinbox.indexBatchSize parameter + + - parallelize updates by default, "-j0" is (once again) allowed + parallelization + +* public-inbox-learn + + - `rm' supports `--all' to remove from all configured inboxes + +* PublicInbox::WWW + + - use consistent blank line around attachment links + + - Attachments in message/{rfc822,news,global} messages can be + individually downloaded. Downloading the entire message/rfc822 + file in full remains supported + + - $INBOX_DIR/description is treated as UTF-8 + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. +See https://public-inbox.org/TODO for what the future holds. diff --git a/TODO b/TODO index 16de36bf200..9396f661137 100644 --- a/TODO +++ b/TODO @@ -19,7 +19,7 @@ all need to be considered for everything we introduce) Meaning users can run this without needing a full copy of the archives in git repositories. -* HTTP and NNTP proxy support. Allow us to be a frontend for +* HTTP, IMAP and NNTP proxy support. Allow us to be a frontend for firewalled off (or Tor-exclusive) instances. The use case is for offering a publicly accessible IP with a cheap VPS, yet storing large amounts of data on computers without a @@ -32,7 +32,7 @@ all need to be considered for everything we introduce) archive locations to avoid SPOF. * optional Cache::FastMmap support so production deployments won't - need Varnish (Varnish doesn't protect NNTP, either) + need Varnish (Varnish doesn't protect NNTP or IMAP, either) * dogfood and take advantage of new kernel APIs (while maintaining portability to older Linux, free BSDs and maybe Hurd). @@ -44,7 +44,8 @@ all need to be considered for everything we introduce) * Support more of RFC 3977 (NNTP) Is there anything left for read-only support? -* Combined "super server" for NNTP/HTTP/POP3 to reduce memory overhead +* Combined "super server" for NNTP/HTTP/POP3/IMAP to reduce memory, + process, and FD overhead * Configurable linkification for per-inbox shorthands: "$gmane/123456" could be configured to expand to the @@ -111,8 +112,31 @@ all need to be considered for everything we introduce) * imperfect scraper importers for obfuscated list archives (e.g. obfuscated Mailman stuff, Google Groups, etc...) +* extend public-inbox-watch to support IMAP, NNTP + * improve performance and avoid head-of-line blocking on slow storage +* HTTP(S) search API (likely JMAP, but GraphQL could be an option) + It should support git-specific prefixes (dfpre:, dfpost:, dfn:, etc) + as extensions. If JMAP, it should have HTTP(S) analogues to + various IMAP extensions. + +* search across multiple inboxes, or admin-definable groups of inboxes + +* scalability to tens/hundreds of thousands of inboxes + + - pagination for WwwListing + + - inotify-based manifest.js.gz updates + + - process/FD reduction (needs to be slow-storage friendly) + + ... + +* command-line tool (similar to mairix/notmuch, but solver+git-aware) + +* consider removing doc_data from Xapian, redundant with over.sqlite3 + * share "git cat-file --batch" processes across inboxes to avoid bumping into /proc/sys/fs/pipe-user-pages-* limits @@ -125,7 +149,8 @@ all need to be considered for everything we introduce) * linter to check validity of config file * linter option and WWW endpoint to graph relationships and flows - between inboxes, addresses maildirs, coderepos, etc... + between inboxes, addresses, Maildirs, coderepos, newsgroups, + IMAP mailboxes, etc... * pygments support - via Python script similar to `git cat-file --batch' to avoid startup penalty. pygments.rb (Ruby) can be inspiration, too.