[PATCH 0/6] various doc updates

[PATCH 0/6] various doc updates

[Eric Wong]

Something to distract me while waiting on codesearch indexing
slowness...

Eric Wong (6):
  doc: technical/memory: add note about mwrap-perl
  doc: technical/ds: update blurb to note more daemons
  doc: technical: document weird stuff in our codebase
  doc: lei import: add hints about nntp.* and imap.* config options
  doc: lei config: update with --edit and --list examples
  doc: 2.0.0 release notes update

 Documentation/RelNotes/v2.0.0.wip       | 28 +++++++++++++++++++++---
 Documentation/lei-config.pod            | 29 +++++++++++++++++++++++--
 Documentation/lei-import.pod            |  7 +++++-
 Documentation/technical/ds.txt          | 13 +++++++----
 Documentation/technical/memory.txt      |  6 +++++
 Documentation/technical/weird-stuff.txt | 22 +++++++++++++++++++
 MANIFEST                                |  1 +
 7 files changed, 96 insertions(+), 10 deletions(-)
 create mode 100644 Documentation/technical/weird-stuff.txt

[PATCH 1/6] doc: technical/memory: add note about mwrap-perl

[Eric Wong]

It's already fixed memory usage problems not only in our codebase,
but also the standard `Encode' XS module and `git pack-objects'.
---
 Documentation/technical/memory.txt | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Documentation/technical/memory.txt b/Documentation/technical/memory.txt
index ea7fb7b7..a35b2c73 100644
--- a/Documentation/technical/memory.txt
+++ b/Documentation/technical/memory.txt
@@ -48,3 +48,9 @@ In the future, our internal data model will be further
 flattened and simplified to reduce the overhead imposed by
 small objects.  Large allocations may also be avoided by
 optionally using Inline::C.
+
+Finally, the mwrap-perl LD_PRELOAD wrapper was ported to Perl 5
+and enhanced to provide live memory usage tracking on 64-bit systems
+with minimal performance impact on production traffic:
+
+	git clone https://80x24.org/mwrap-perl.git

[PATCH 2/6] doc: technical/ds: update blurb to note more daemons

[Eric Wong]

And add a note about the various wakeup modes of kqueue|epoll
while we're at it; we use all of them!
---
 Documentation/technical/ds.txt | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/Documentation/technical/ds.txt b/Documentation/technical/ds.txt
index 89cc05af..4cfb62fe 100644
--- a/Documentation/technical/ds.txt
+++ b/Documentation/technical/ds.txt
@@ -1,9 +1,14 @@
 PublicInbox::DS - event loop and async I/O base class
 
-Our PublicInbox::DS event loop which powers public-inbox-nntpd
-and public-inbox-httpd diverges significantly from the
-unmaintained Danga::Socket package we forked from.  In fact,
-it's probably different from most other event loops out there.
+Our PublicInbox::DS event loop which powers most of our long-lived
+processes(*) diverges significantly from the unmaintained Danga::Socket
+package we forked from.  In fact, it's probably different from most
+other event loops out there.
+
+Most notably, it uses one-shot, level-trigger, and edge-trigger mode
+modes of kqueue|epoll depending on the situation.
+
+(*) public-inbox-netd,(-httpd,-imapd,-nntpd,-pop3d,-watch) + lei-daemon
 
 Most notably:
 

[PATCH 3/6] doc: technical: document weird stuff in our codebase

[Eric Wong]

Hopefully this makes things less surprising to new hackers.
---
 Documentation/technical/weird-stuff.txt | 22 ++++++++++++++++++++++
 MANIFEST                                |  1 +
 2 files changed, 23 insertions(+)
 create mode 100644 Documentation/technical/weird-stuff.txt

diff --git a/Documentation/technical/weird-stuff.txt b/Documentation/technical/weird-stuff.txt
new file mode 100644
index 00000000..0c8d6891
--- /dev/null
+++ b/Documentation/technical/weird-stuff.txt
@@ -0,0 +1,22 @@
+There's a lot of weird code in public-inbox which may be daunting
+to new hackers.
+
+* The event loop (PublicInbox::DS) is an evolution of a fairly standard
+  C10K event loop.  See ds.txt in this directory for more.
+
+Things got weirder in 2021:
+
+* The lei command-line tool is backed by a daemon.  This was done to
+  improve startup time for shell completion and manage git/SQLite/Xapian
+  single-writer during long, parallel imports.  It may eventually become
+  a read-write IMAP/JMAP server.
+
+* SOCK_SEQPACKET is used extensively in lei, and will likely make its
+  way into more places, still.
+
+And even more so in 2022:
+
+* public-inbox-clone / PublicInbox::LeiMirror relies on ->DESTROY
+  for make-like dependency management while providing parallelism.
+
+More to come, lei will expose Maildirs via FUSE 3...
diff --git a/MANIFEST b/MANIFEST
index 7437bb54..bc652e21 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -99,6 +99,7 @@ Documentation/standards.perl
 Documentation/technical/data_structures.txt
 Documentation/technical/ds.txt
 Documentation/technical/memory.txt
+Documentation/technical/weird-stuff.txt
 Documentation/technical/whyperl.txt
 Documentation/txt2pre
 HACKING

[PATCH 4/6] doc: lei import: add hints about nntp.* and imap.* config options

[Eric Wong]

I'm setting up more imports and forgot about them :x
---
 Documentation/lei-import.pod | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index 69ec6497..31d6db13 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -86,8 +86,13 @@ Default: C<auto>
 
 Use the specified proxy (e.g., C<socks5h://0:9050>).
 
+Consider L<imap.proxy> and L<nntp.proxy> which can be persistently
+configured on a per-host basis in L<lei-config(1)>.
+
 =back
 
+See L<lei-config(1)> for various C<imap.*> and C<nntp.*> options.
+
 =head1 CONTACT
 
 Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -103,4 +108,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 
 =head1 SEE ALSO
 
-L<lei-index(1)>, L<lei-store-format(5)>
+L<lei-config(1)>, L<lei-index(1)>, L<lei-store-format(5)>

[PATCH 5/6] doc: lei config: update with --edit and --list examples

[Eric Wong]

I typically use --edit/-e to make changes and --list/-l with
git; and same with lei.
---
 Documentation/lei-config.pod | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

diff --git a/Documentation/lei-config.pod b/Documentation/lei-config.pod
index 663404fe..23a60c8a 100644
--- a/Documentation/lei-config.pod
+++ b/Documentation/lei-config.pod
@@ -4,7 +4,11 @@ lei-config - git-config wrapper for lei configuration file
 
 =head1 SYNOPSIS
 
-lei config [OPTIONS]
+lei config <name> [[<value>] [<value-pattern>]]
+
+lei config -l | --list
+
+lei config -e | --edit
 
 =head1 DESCRIPTION
 
@@ -97,6 +101,27 @@ C<frag>, C<func>, and C<context>.
 
 =back
 
+=head1 OPTIONS
+
+Most L<git-config(1)> command-line switches are accepted by C<lei config>
+as-is.  The most-frequently-used options are expected to be:
+
+=over 4
+
+=item -e
+
+=item --edit
+
+Opens an editor to edit the lei config file
+
+=item -l
+
+=item --list
+
+List all variables set in config file, along with their values.
+
+=back
+
 =head1 CONTACT
 
 Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -106,6 +131,6 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
 
 =head1 COPYRIGHT
 
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
 
 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>

[PATCH 6/6] doc: 2.0.0 release notes update

[Eric Wong]

Did some stuff, still a ton of stuff to do :x
---
 Documentation/RelNotes/v2.0.0.wip | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/Documentation/RelNotes/v2.0.0.wip b/Documentation/RelNotes/v2.0.0.wip
index a5468f8c..cd90bdae 100644
--- a/Documentation/RelNotes/v2.0.0.wip
+++ b/Documentation/RelNotes/v2.0.0.wip
@@ -4,8 +4,7 @@ MIME-Version: 1.0
 Content-Type: text/plain; charset=utf-8
 Content-Disposition: inline
 
-This release is mainly to fix a regression for -nntpd affecting
-neomutt and possibly other NNTP clients.
+This release includes several new features and fixes.
 
 There is also ongoing work to integrate coderepo handling into
 the codebase and the idea of `lei p2q' is integrated into the
@@ -25,6 +24,8 @@ treewide
 
   * standardize on `#' prefix for stderr diagnostics (previously `I:')
 
+  * avoid O(n) behavior for reaping dead processes in daemons
+
 PublicInbox::WWW
 
   * support `+' in inbox names
@@ -40,6 +41,11 @@ PublicInbox::WWW
 
   * fix footer in listing of >200 inboxes
 
+  * support dumb HTTP clones of SHA-256 git repos
+
+  * add /$INBOX/$MSGID/d/ endpoint to show diffs in reused Message-IDs
+    (`lei mail-diff' for the web)
+
 lei
 
   * use http.proxy / http.<remote>.proxy from system-wide git-config if
@@ -51,6 +57,9 @@ lei
 
   * compatibility with SQLite <3.8.3 on CentOS 7.x
 
+  * fix `lei q -tt' on locally-indexed messages (still broken for remotes:
+    https://public-inbox.org/meta/20230226170931.M947721@dcvr/ )
+
 solver (used by lei (rediff|blob), and PublicInbox::WWW)
 
   * handle copies in patches properly
@@ -61,6 +70,11 @@ portability
 
   * fix EINTR handling for kqueue users
 
+  * various fixes for CentOS 7.x
+
+  * fix excessive pipelining to `git cat-file' on systems with small
+    getdelim(3) buffers (mainly affects musl)
+
 public-inbox-nntpd
 
   * fix LISTGROUP with range (affects neomutt)
@@ -73,14 +87,22 @@ public-inbox-clone:
 
   * parallel mirroring of multiple inboxes/coderepos via manifest
 
-  * new flags to support this include:
+  * new flags to support manifest mirroring include:
     --dry-run, --inbox-config=, --project-list=, --prune,
     --keep-going, --jobs, --include=, --exclude=, --objstore=, ...
+    See public-inbox-clone(1) man page for more details.
 
 PublicInbox::SaPlugin::ListMirror
 
   * List-ID handling special-cased according to RFC 2919 rules
 
+Search improvements (lei and PublicInbox::WWW)
+
+  * quoted text inside base-85 binary patches are no longer indexed
+
+Thanks to all the bug reporters and users who made this release
+possible.
+
 Please report bugs via plain-text mail to: meta@public-inbox.org
 
 See archives at https://public-inbox.org/meta/ for all history.