* [PATCH 0/6] various doc updates
@ 2023-03-09 19:28 Eric Wong
2023-03-09 19:28 ` [PATCH 1/6] doc: technical/memory: add note about mwrap-perl Eric Wong
` (5 more replies)
0 siblings, 6 replies; 7+ messages in thread
From: Eric Wong @ 2023-03-09 19:28 UTC (permalink / raw)
To: meta
Something to distract me while waiting on codesearch indexing
slowness...
Eric Wong (6):
doc: technical/memory: add note about mwrap-perl
doc: technical/ds: update blurb to note more daemons
doc: technical: document weird stuff in our codebase
doc: lei import: add hints about nntp.* and imap.* config options
doc: lei config: update with --edit and --list examples
doc: 2.0.0 release notes update
Documentation/RelNotes/v2.0.0.wip | 28 +++++++++++++++++++++---
Documentation/lei-config.pod | 29 +++++++++++++++++++++++--
Documentation/lei-import.pod | 7 +++++-
Documentation/technical/ds.txt | 13 +++++++----
Documentation/technical/memory.txt | 6 +++++
Documentation/technical/weird-stuff.txt | 22 +++++++++++++++++++
MANIFEST | 1 +
7 files changed, 96 insertions(+), 10 deletions(-)
create mode 100644 Documentation/technical/weird-stuff.txt
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 1/6] doc: technical/memory: add note about mwrap-perl
2023-03-09 19:28 [PATCH 0/6] various doc updates Eric Wong
@ 2023-03-09 19:28 ` Eric Wong
2023-03-09 19:28 ` [PATCH 2/6] doc: technical/ds: update blurb to note more daemons Eric Wong
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Eric Wong @ 2023-03-09 19:28 UTC (permalink / raw)
To: meta
It's already fixed memory usage problems not only in our codebase,
but also the standard `Encode' XS module and `git pack-objects'.
---
Documentation/technical/memory.txt | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/Documentation/technical/memory.txt b/Documentation/technical/memory.txt
index ea7fb7b7..a35b2c73 100644
--- a/Documentation/technical/memory.txt
+++ b/Documentation/technical/memory.txt
@@ -48,3 +48,9 @@ In the future, our internal data model will be further
flattened and simplified to reduce the overhead imposed by
small objects. Large allocations may also be avoided by
optionally using Inline::C.
+
+Finally, the mwrap-perl LD_PRELOAD wrapper was ported to Perl 5
+and enhanced to provide live memory usage tracking on 64-bit systems
+with minimal performance impact on production traffic:
+
+ git clone https://80x24.org/mwrap-perl.git
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 2/6] doc: technical/ds: update blurb to note more daemons
2023-03-09 19:28 [PATCH 0/6] various doc updates Eric Wong
2023-03-09 19:28 ` [PATCH 1/6] doc: technical/memory: add note about mwrap-perl Eric Wong
@ 2023-03-09 19:28 ` Eric Wong
2023-03-09 19:28 ` [PATCH 3/6] doc: technical: document weird stuff in our codebase Eric Wong
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Eric Wong @ 2023-03-09 19:28 UTC (permalink / raw)
To: meta
And add a note about the various wakeup modes of kqueue|epoll
while we're at it; we use all of them!
---
Documentation/technical/ds.txt | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/Documentation/technical/ds.txt b/Documentation/technical/ds.txt
index 89cc05af..4cfb62fe 100644
--- a/Documentation/technical/ds.txt
+++ b/Documentation/technical/ds.txt
@@ -1,9 +1,14 @@
PublicInbox::DS - event loop and async I/O base class
-Our PublicInbox::DS event loop which powers public-inbox-nntpd
-and public-inbox-httpd diverges significantly from the
-unmaintained Danga::Socket package we forked from. In fact,
-it's probably different from most other event loops out there.
+Our PublicInbox::DS event loop which powers most of our long-lived
+processes(*) diverges significantly from the unmaintained Danga::Socket
+package we forked from. In fact, it's probably different from most
+other event loops out there.
+
+Most notably, it uses one-shot, level-trigger, and edge-trigger mode
+modes of kqueue|epoll depending on the situation.
+
+(*) public-inbox-netd,(-httpd,-imapd,-nntpd,-pop3d,-watch) + lei-daemon
Most notably:
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 3/6] doc: technical: document weird stuff in our codebase
2023-03-09 19:28 [PATCH 0/6] various doc updates Eric Wong
2023-03-09 19:28 ` [PATCH 1/6] doc: technical/memory: add note about mwrap-perl Eric Wong
2023-03-09 19:28 ` [PATCH 2/6] doc: technical/ds: update blurb to note more daemons Eric Wong
@ 2023-03-09 19:28 ` Eric Wong
2023-03-09 19:28 ` [PATCH 4/6] doc: lei import: add hints about nntp.* and imap.* config options Eric Wong
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Eric Wong @ 2023-03-09 19:28 UTC (permalink / raw)
To: meta
Hopefully this makes things less surprising to new hackers.
---
Documentation/technical/weird-stuff.txt | 22 ++++++++++++++++++++++
MANIFEST | 1 +
2 files changed, 23 insertions(+)
create mode 100644 Documentation/technical/weird-stuff.txt
diff --git a/Documentation/technical/weird-stuff.txt b/Documentation/technical/weird-stuff.txt
new file mode 100644
index 00000000..0c8d6891
--- /dev/null
+++ b/Documentation/technical/weird-stuff.txt
@@ -0,0 +1,22 @@
+There's a lot of weird code in public-inbox which may be daunting
+to new hackers.
+
+* The event loop (PublicInbox::DS) is an evolution of a fairly standard
+ C10K event loop. See ds.txt in this directory for more.
+
+Things got weirder in 2021:
+
+* The lei command-line tool is backed by a daemon. This was done to
+ improve startup time for shell completion and manage git/SQLite/Xapian
+ single-writer during long, parallel imports. It may eventually become
+ a read-write IMAP/JMAP server.
+
+* SOCK_SEQPACKET is used extensively in lei, and will likely make its
+ way into more places, still.
+
+And even more so in 2022:
+
+* public-inbox-clone / PublicInbox::LeiMirror relies on ->DESTROY
+ for make-like dependency management while providing parallelism.
+
+More to come, lei will expose Maildirs via FUSE 3...
diff --git a/MANIFEST b/MANIFEST
index 7437bb54..bc652e21 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -99,6 +99,7 @@ Documentation/standards.perl
Documentation/technical/data_structures.txt
Documentation/technical/ds.txt
Documentation/technical/memory.txt
+Documentation/technical/weird-stuff.txt
Documentation/technical/whyperl.txt
Documentation/txt2pre
HACKING
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 4/6] doc: lei import: add hints about nntp.* and imap.* config options
2023-03-09 19:28 [PATCH 0/6] various doc updates Eric Wong
` (2 preceding siblings ...)
2023-03-09 19:28 ` [PATCH 3/6] doc: technical: document weird stuff in our codebase Eric Wong
@ 2023-03-09 19:28 ` Eric Wong
2023-03-09 19:28 ` [PATCH 5/6] doc: lei config: update with --edit and --list examples Eric Wong
2023-03-09 19:28 ` [PATCH 6/6] doc: 2.0.0 release notes update Eric Wong
5 siblings, 0 replies; 7+ messages in thread
From: Eric Wong @ 2023-03-09 19:28 UTC (permalink / raw)
To: meta
I'm setting up more imports and forgot about them :x
---
Documentation/lei-import.pod | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index 69ec6497..31d6db13 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -86,8 +86,13 @@ Default: C<auto>
Use the specified proxy (e.g., C<socks5h://0:9050>).
+Consider L<imap.proxy> and L<nntp.proxy> which can be persistently
+configured on a per-host basis in L<lei-config(1)>.
+
=back
+See L<lei-config(1)> for various C<imap.*> and C<nntp.*> options.
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -103,4 +108,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<lei-index(1)>, L<lei-store-format(5)>
+L<lei-config(1)>, L<lei-index(1)>, L<lei-store-format(5)>
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 5/6] doc: lei config: update with --edit and --list examples
2023-03-09 19:28 [PATCH 0/6] various doc updates Eric Wong
` (3 preceding siblings ...)
2023-03-09 19:28 ` [PATCH 4/6] doc: lei import: add hints about nntp.* and imap.* config options Eric Wong
@ 2023-03-09 19:28 ` Eric Wong
2023-03-09 19:28 ` [PATCH 6/6] doc: 2.0.0 release notes update Eric Wong
5 siblings, 0 replies; 7+ messages in thread
From: Eric Wong @ 2023-03-09 19:28 UTC (permalink / raw)
To: meta
I typically use --edit/-e to make changes and --list/-l with
git; and same with lei.
---
Documentation/lei-config.pod | 29 +++++++++++++++++++++++++++--
1 file changed, 27 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-config.pod b/Documentation/lei-config.pod
index 663404fe..23a60c8a 100644
--- a/Documentation/lei-config.pod
+++ b/Documentation/lei-config.pod
@@ -4,7 +4,11 @@ lei-config - git-config wrapper for lei configuration file
=head1 SYNOPSIS
-lei config [OPTIONS]
+lei config <name> [[<value>] [<value-pattern>]]
+
+lei config -l | --list
+
+lei config -e | --edit
=head1 DESCRIPTION
@@ -97,6 +101,27 @@ C<frag>, C<func>, and C<context>.
=back
+=head1 OPTIONS
+
+Most L<git-config(1)> command-line switches are accepted by C<lei config>
+as-is. The most-frequently-used options are expected to be:
+
+=over 4
+
+=item -e
+
+=item --edit
+
+Opens an editor to edit the lei config file
+
+=item -l
+
+=item --list
+
+List all variables set in config file, along with their values.
+
+=back
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -106,6 +131,6 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 6/6] doc: 2.0.0 release notes update
2023-03-09 19:28 [PATCH 0/6] various doc updates Eric Wong
` (4 preceding siblings ...)
2023-03-09 19:28 ` [PATCH 5/6] doc: lei config: update with --edit and --list examples Eric Wong
@ 2023-03-09 19:28 ` Eric Wong
5 siblings, 0 replies; 7+ messages in thread
From: Eric Wong @ 2023-03-09 19:28 UTC (permalink / raw)
To: meta
Did some stuff, still a ton of stuff to do :x
---
Documentation/RelNotes/v2.0.0.wip | 28 +++++++++++++++++++++++++---
1 file changed, 25 insertions(+), 3 deletions(-)
diff --git a/Documentation/RelNotes/v2.0.0.wip b/Documentation/RelNotes/v2.0.0.wip
index a5468f8c..cd90bdae 100644
--- a/Documentation/RelNotes/v2.0.0.wip
+++ b/Documentation/RelNotes/v2.0.0.wip
@@ -4,8 +4,7 @@ MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
-This release is mainly to fix a regression for -nntpd affecting
-neomutt and possibly other NNTP clients.
+This release includes several new features and fixes.
There is also ongoing work to integrate coderepo handling into
the codebase and the idea of `lei p2q' is integrated into the
@@ -25,6 +24,8 @@ treewide
* standardize on `#' prefix for stderr diagnostics (previously `I:')
+ * avoid O(n) behavior for reaping dead processes in daemons
+
PublicInbox::WWW
* support `+' in inbox names
@@ -40,6 +41,11 @@ PublicInbox::WWW
* fix footer in listing of >200 inboxes
+ * support dumb HTTP clones of SHA-256 git repos
+
+ * add /$INBOX/$MSGID/d/ endpoint to show diffs in reused Message-IDs
+ (`lei mail-diff' for the web)
+
lei
* use http.proxy / http.<remote>.proxy from system-wide git-config if
@@ -51,6 +57,9 @@ lei
* compatibility with SQLite <3.8.3 on CentOS 7.x
+ * fix `lei q -tt' on locally-indexed messages (still broken for remotes:
+ https://public-inbox.org/meta/20230226170931.M947721@dcvr/ )
+
solver (used by lei (rediff|blob), and PublicInbox::WWW)
* handle copies in patches properly
@@ -61,6 +70,11 @@ portability
* fix EINTR handling for kqueue users
+ * various fixes for CentOS 7.x
+
+ * fix excessive pipelining to `git cat-file' on systems with small
+ getdelim(3) buffers (mainly affects musl)
+
public-inbox-nntpd
* fix LISTGROUP with range (affects neomutt)
@@ -73,14 +87,22 @@ public-inbox-clone:
* parallel mirroring of multiple inboxes/coderepos via manifest
- * new flags to support this include:
+ * new flags to support manifest mirroring include:
--dry-run, --inbox-config=, --project-list=, --prune,
--keep-going, --jobs, --include=, --exclude=, --objstore=, ...
+ See public-inbox-clone(1) man page for more details.
PublicInbox::SaPlugin::ListMirror
* List-ID handling special-cased according to RFC 2919 rules
+Search improvements (lei and PublicInbox::WWW)
+
+ * quoted text inside base-85 binary patches are no longer indexed
+
+Thanks to all the bug reporters and users who made this release
+possible.
+
Please report bugs via plain-text mail to: meta@public-inbox.org
See archives at https://public-inbox.org/meta/ for all history.
^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-03-09 19:28 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-09 19:28 [PATCH 0/6] various doc updates Eric Wong
2023-03-09 19:28 ` [PATCH 1/6] doc: technical/memory: add note about mwrap-perl Eric Wong
2023-03-09 19:28 ` [PATCH 2/6] doc: technical/ds: update blurb to note more daemons Eric Wong
2023-03-09 19:28 ` [PATCH 3/6] doc: technical: document weird stuff in our codebase Eric Wong
2023-03-09 19:28 ` [PATCH 4/6] doc: lei import: add hints about nntp.* and imap.* config options Eric Wong
2023-03-09 19:28 ` [PATCH 5/6] doc: lei config: update with --edit and --list examples Eric Wong
2023-03-09 19:28 ` [PATCH 6/6] doc: 2.0.0 release notes update Eric Wong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).