[PATCH] doc: update HACKING and TODO with a few items

[PATCH] doc: update HACKING and TODO with a few items

[Eric Wong]

Inline::C seems alright, so we might use it more since it still
allows end users to quickly make changes.  Our performance on
rotational disks is also terrible, and could be improved...
---
 HACKING | 3 ++-
 TODO    | 8 +++-----
 2 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/HACKING b/HACKING
index c2f4825c..f7cb92e9 100644
--- a/HACKING
+++ b/HACKING
@@ -40,7 +40,8 @@ the shiny new.
 Avoid relying on compiled modules too much.  Even if it is Free,
 compiled code makes packages more expensive to audit, build,
 distribute and verify.  public-inbox itself will only be implemented
-in scripting languages (currently Perl 5).
+in scripting languages (currently Perl 5) and optional JIT-compiled C
+(via Inline::C)
 
 Performance should be reasonably good for server administrators, too,
 and we will sacrifice features to achieve predictable performance.
diff --git a/TODO b/TODO
index 9f58342f..8f78d87d 100644
--- a/TODO
+++ b/TODO
@@ -5,7 +5,7 @@ performance, ease-of-setup, installation, maintainability, etc
 all need to be considered for everything we introduce)
 
 * general performance improvements, but without relying on
-  XS or compiled code any more than we currently do.
+  XS or pre-built modules any more than we currently do.
 
 * mailmap support (same as git) for remapping expired email addresses
 
@@ -101,10 +101,8 @@ all need to be considered for everything we introduce)
 * imperfect scraper importers for obfuscated list archives
   (e.g. obfuscated Mailman stuff, Google Groups, etc...)
 
-* support hooks, since low-level git-fast-import does not run them
-  https://public-inbox.org/meta/20190405174329.GA21472@chatter.qube.local/
-  (note: may not be needed since we do grokmirror manifest.js.gz, now)
-
 * consider using HTTP::Date instead of Date::Parse, since we need the
   former is capable of parsing RFC822-ish dates, used by Plack, and
   the latter is missing from OpenBSD and maybe other distros.
+
+* improve performance and avoid head-of-line blocking on slow storage
-- 
EW

libgmime experiences? (was: [PATCH] doc: update HACKING and TODO ...)

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> --- a/HACKING
> +++ b/HACKING
> @@ -40,7 +40,8 @@ the shiny new.
>  Avoid relying on compiled modules too much.  Even if it is Free,
>  compiled code makes packages more expensive to audit, build,
>  distribute and verify.  public-inbox itself will only be implemented
> -in scripting languages (currently Perl 5).
> +in scripting languages (currently Perl 5) and optional JIT-compiled C
> +(via Inline::C)

I'm considering adding libgmime support after the 1.2 release
and creating an Email::MIME/msg_iter-compatible wrapper based
on it.

I've never used gmime as a user or hacker...

Pure-Perl Email::MIME will remain supported, though new versions
of Email::MIME use Email::Address::XS so there's already a
C/XS component which increases installation overhead.

My thoughts on gmime so far:

One big reason gmime looks good to me is that it offers mmap and
pipe APIs, so it hopefully deals well with big messages and
won't bloat/fragment the heap.

Another is Email::Simple::Header::_header_to_list being at the
top of some profiles, and I'm not convinced I could make it
faster.

gmime's been through several incompatible API/ABI versions, but
it seems Debian can support at least 2.6 and 3.0.  FreeBSD has
2.2, 2.4, 2.6 and 3.0, even!

gmime upstream also provides a PORTING document which should come
in handy.

The MIME::Fast wrapper for libgmime exists on CPAN, but that
hasn't been updated in years and isn't in Debian.  Both 3.0 and
2.6 APIs could be supported in parallel using Inline::C, and
more can be added in the future.

notmuch uses gmime, as does a lot of desktop software.

My experience with mainstream "desktop software" hasn't been
good, as they tends to be less stable in both user/programming
interfaces, and also more complex and difficult to debug than
server-oriented software.

However, looking at the Debian changelogs, there's only been one
CVE (CVE-2010-0409) from nearly a decade ago.  Maybe all the
bugs are fixed, or nobody really abuses it.

In any case, if I go through with this, I'll be running lots of
mail and random junk through it to ensure it doesn't break
unexpectedly.

Re: libgmime experiences? (was: [PATCH] doc: update HACKING and TODO ...)

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> I've never used gmime as a user or hacker...

I still haven't used it as a user.
<snip>

> However, looking at the Debian changelogs, there's only been one
> CVE (CVE-2010-0409) from nearly a decade ago.  Maybe all the
> bugs are fixed, or nobody really abuses it.

gmime 3.2.5 just came out which addresses stack exhaustion:

https://download.gnome.org/sources/gmime/3.2/gmime-3.2.5.news

(but there may be more work to do in that area...)