setting up mailman-to-atom-converter then atom-to-public-inbox

setting up mailman-to-atom-converter then atom-to-public-inbox

[Luke Kenneth Casson Leighton]

hi, just as the subject says, i'm currently modifying mailman_rss to
support atom and would like to set it up on libre-soc.org shortly.

firstly: very grateful that public-inbox even exists, it is kinda
important to have really, really simple offline archives of project
mailing lists.

second: i have no idea how to go about setting it up :)

third: sigh, i have two unknowns (three), because i am actually
modifying mailman_rss to support atom, *and* i would prefer not to
overload my server by splitting up the creation of atom feeds into
multiple separate processing sections (by month) *and* i have no idea
if public-inbox can support feeds-of-feeds.

to explain / unpack that: here's how i would envisage the workflow so
as to minimise the server load:

* cron job goes through the monthly mailman archives *by month*
performing a re-creation *only* of the latest month's atom feed
* same cron job adds to a "global" atom file containing "links to the
monthly atom files"
* public-inbox sees that list-of-monthly-atom-files
* public-inbox walks the "tree" of monthly atom files, grabbing each one in turn
* public-inbox loads all messages from all monthly atom files.

is this possible or does public-inbox expect one whopping monster
resource-hogging beast-of-an-atom-file potentially hundreds of
megabytes long?  (the reason i ask all this is because the server i am
running this on only has 1GB of RAM and i'm not going to be upgrading
it as it costs money).

tia,

l.

Re: setting up mailman-to-atom-converter then atom-to-public-inbox

[Eric Wong]

Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> hi, just as the subject says, i'm currently modifying mailman_rss to
> support atom and would like to set it up on libre-soc.org shortly.
> 
> firstly: very grateful that public-inbox even exists, it is kinda
> important to have really, really simple offline archives of project
> mailing lists.

You're welcome :>

> second: i have no idea how to go about setting it up :)

Once installed, "public-inbox-init" should get you started.
From there, you can decide how you want to inject mail into
it...

We should be able to clarify anything else here, just ask,
and we can try to make the docs better :>
Fwiw, I also started working on a mail flow diagram yesterday,
which may help:

	https://public-inbox.org/flow.txt

> third: sigh, i have two unknowns (three), because i am actually
> modifying mailman_rss to support atom, *and* i would prefer not to
> overload my server by splitting up the creation of atom feeds into
> multiple separate processing sections (by month) *and* i have no idea
> if public-inbox can support feeds-of-feeds.

This is your Mailman server?  If so, mbox or Maildir archives
would be MUCH easier to convert and it would preserve
Message-Id, References, and In-Reply-To headers for proper
message threading.

public-inbox doesn't have any ability to parse Atom or RSS right
now, it only generates Atom.

Parsing Atom (or RSS) would not preserve headers necessary for
proper threading, since Atom threading headers (RFC4685) don't
reliably map back to the aforementioned mail headers.

> to explain / unpack that: here's how i would envisage the workflow so
> as to minimise the server load:
> 
> * cron job goes through the monthly mailman archives *by month*
> performing a re-creation *only* of the latest month's atom feed
> * same cron job adds to a "global" atom file containing "links to the
> monthly atom files"
> * public-inbox sees that list-of-monthly-atom-files
> * public-inbox walks the "tree" of monthly atom files, grabbing each one in turn
> * public-inbox loads all messages from all monthly atom files.

s/atom/mbox/ and that's close to a planned feature.

I'm not sure why the global index file is necessary, though,
since the tree structure is predictable (YYYY/MM or similar)

Also, Konstantin wrote list-archive-maker.py which parses
pipermail archives:
https://public-inbox.org/meta/CAMwyc-T+QrzNhfgg1kQWTrKa26CeHvEd6BFahGiLC3PKOZJurw@mail.gmail.com/

> is this possible or does public-inbox expect one whopping monster
> resource-hogging beast-of-an-atom-file potentially hundreds of
> megabytes long?  (the reason i ask all this is because the server i am
> running this on only has 1GB of RAM and i'm not going to be upgrading
> it as it costs money).

Totally understood, I'm constantly looking for ways to cut
memory use and refuse to upgrade my RAM or CPU.

Right now, public-inbox itself doesn't parse XML at all,
only some test cases do.

If you're using a SAX parser for XML (e.g. XML::SAX,
XML::LibXML::SAX, ...), it should be able to stream everything
and not hold more than the contents of a single email in memory
at once.

The existing mbox import APIs (e.g.
scripts/import_vger_from_mbox) work like that.

Internally, public-inbox tries to stream as much as possible to
save memory.  More RAM still helps if you have slow storage
and/or big archives, though, especially with Xapian.

public-inbox itself uses the Email::MIME module, which
unfortunately requires reading an entire RFC-2822 message into
memory (and we only work on one full message at a time).

Beyond that, the message threading in the HTML output
(non-recursive JWZ-variant) works on a batch of 1000 message
skeletons (subset of headers), and few threads are that big.

Re: setting up mailman-to-atom-converter then atom-to-public-inbox

[Luke Kenneth Casson Leighton]

On Tue, Feb 4, 2020 at 9:05 PM Eric Wong <e@yhbt.net> wrote:

> Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> > hi, just as the subject says, i'm currently modifying mailman_rss to
> > support atom and would like to set it up on libre-soc.org shortly.
> >
> > firstly: very grateful that public-inbox even exists, it is kinda
> > important to have really, really simple offline archives of project
> > mailing lists.
>
> You're welcome :>
>
> > second: i have no idea how to go about setting it up :)
>
> Once installed, "public-inbox-init" should get you started.
> From there, you can decide how you want to inject mail into
> it...

ahh exxcellent....  err... err.... man public-inbox-config only lists
Maildir not mbox?

> We should be able to clarify anything else here, just ask,
> and we can try to make the docs better :>
> Fwiw, I also started working on a mail flow diagram yesterday,
> which may help:
>
>         https://public-inbox.org/flow.txt

excellent.  very useful.

> > third: sigh, i have two unknowns (three), because i am actually
> > modifying mailman_rss to support atom, *and* i would prefer not to
> > overload my server by splitting up the creation of atom feeds into
> > multiple separate processing sections (by month) *and* i have no idea
> > if public-inbox can support feeds-of-feeds.
>
> This is your Mailman server?

yes

> If so, mbox or Maildir archives
> would be MUCH easier to convert and it would preserve
> Message-Id, References, and In-Reply-To headers for proper
> message threading.

errr... errr doh!  ok so the mbox archives are private under one
account and i need to publish them via... gitweb, so that's ok.

> public-inbox doesn't have any ability to parse Atom or RSS right
> now, it only generates Atom.

aw doh!  that's where i got the impression i had to *read* the atom
feed (doh).  well, i have some nice modifications to mailman_rss which
uses a generic "Feed" python module i found, i will publish later :)

> Parsing Atom (or RSS) would not preserve headers necessary for
> proper threading, since Atom threading headers (RFC4685) don't
> reliably map back to the aforementioned mail headers.

red herring....

> > to explain / unpack that: here's how i would envisage the workflow so
> > as to minimise the server load:
> >
> > * cron job goes through the monthly mailman archives *by month*
> > performing a re-creation *only* of the latest month's atom feed
> > * same cron job adds to a "global" atom file containing "links to the
> > monthly atom files"
> > * public-inbox sees that list-of-monthly-atom-files
> > * public-inbox walks the "tree" of monthly atom files, grabbing each one in turn
> > * public-inbox loads all messages from all monthly atom files.
>
> s/atom/mbox/ and that's close to a planned feature.

oh superb.

> I'm not sure why the global index file is necessary, though,
> since the tree structure is predictable (YYYY/MM or similar)

i was imagining that there would be a way to reduce network traffic
however i realise now that you're running the cron job actually on the
machine, directly on the .mbox file.

> public-inbox itself uses the Email::MIME module, which
> unfortunately requires reading an entire RFC-2822 message into
> memory (and we only work on one full message at a time).

*shudder* :)

> Beyond that, the message threading in the HTML output
> (non-recursive JWZ-variant) works on a batch of 1000 message
> skeletons (subset of headers), and few threads are that big.

yehyeh.

okaay, so i'm looking at man public-inbox-config, it says "only
supports Maildir".  grep the source, there's something about
PublicInbox::Import.pm?

ngggh how am i going to get mbox files in / watched?

thanks eric.

l.

Re: setting up mailman-to-atom-converter then atom-to-public-inbox

[Eric Wong]

Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> On Tue, Feb 4, 2020 at 9:05 PM Eric Wong <e@yhbt.net> wrote:
> > Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> >
> > > second: i have no idea how to go about setting it up :)
> >
> > Once installed, "public-inbox-init" should get you started.
> > From there, you can decide how you want to inject mail into
> > it...
> 
> ahh exxcellent....  err... err.... man public-inbox-config only lists
> Maildir not mbox?

Ah, right now mbox is only supported for one-off initial scripts
such as scripts/import_vger_from_mbox

mbox is pretty bad for incremental updates, especially if
there's big rewrites going on for Status: flags setting off
inotify/EVFILT_VNODE.

I suppose it could be added, but Maildir is way easier and
faster for incremental updates, since deduplication can slow
things down a bit.

> > > * cron job goes through the monthly mailman archives *by month*
> > > performing a re-creation *only* of the latest month's atom feed
> > > * same cron job adds to a "global" atom file containing "links to the
> > > monthly atom files"
> > > * public-inbox sees that list-of-monthly-atom-files
> > > * public-inbox walks the "tree" of monthly atom files, grabbing each one in turn
> > > * public-inbox loads all messages from all monthly atom files.
> >
> > s/atom/mbox/ and that's close to a planned feature.
> 
> oh superb.
> 
> > I'm not sure why the global index file is necessary, though,
> > since the tree structure is predictable (YYYY/MM or similar)
> 
> i was imagining that there would be a way to reduce network traffic
> however i realise now that you're running the cron job actually on the
> machine, directly on the .mbox file.

Yeah.  I was planning on supporting a HTTP(S)-based scraper,
anyways for pipermail and Google Groups, anyways, but time's
been taken up by other things.

> > public-inbox itself uses the Email::MIME module, which
> > unfortunately requires reading an entire RFC-2822 message into
> > memory (and we only work on one full message at a time).
> 
> *shudder* :)

I get scary attachments, sometimes :<

> okaay, so i'm looking at man public-inbox-config, it says "only
> supports Maildir".  grep the source, there's something about
> PublicInbox::Import.pm?

The supported/stable PublicInbox::V2Writable API mostly matches
the documented PublicInbox::Import one, and v2 is much better
for long-term use or big archives.

scripts/import_vger_from_mbox is probably a good example to
start with.

> ngggh how am i going to get mbox files in / watched?

I'm not sure it's necessary, just yet.

mbox for the initial import, and Maildir for incremental updates
is probably the easiest way to go in your case.  Eventually
HTTPS downloads can be supported (maybe in a few months or by
the end-of-year), and that'll be mbox, anyways.

> thanks eric.

No prob :>

Re: setting up mailman-to-atom-converter then atom-to-public-inbox

[Eric Wong]

Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> On Tuesday, February 4, 2020, Eric Wong <e@yhbt.net> wrote:
> 
> >
> > Ah, right now mbox is only supported for one-off initial scripts
> > such as scripts/import_vger_from_mbox
> >
> > mbox is pretty bad for incremental updates, especially if
> > there's big rewrites going on for Status: flags setting off
> > inotify/EVFILT_VNODE.
> 
> okaay sooOooo.... i am reconfigurink mailman to yoose maildir, ya?

Sure, or just have a subscriber (which can be yourself) set it up.

I do that with most of https://public-inbox.org/hosted.html
since I'm not an admin of any of those lists.

No admin intervention or even admin knowledge of the
public-inbox is even necessary.  It's a bit subversive
like git-svn in that regard ;>

> will let you know how it goes.

Thanks!

Re: setting up mailman-to-atom-converter then atom-to-public-inbox

[Eric Wong]

Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> On Wednesday, February 5, 2020, Eric Wong <e@yhbt.net> wrote:
> > Sure, or just have a subscriber (which can be yourself) set it up.
> 
> 
> ok, so converting mailman to Maildir format requires a separate program, it
> doesn't have a config option.
> 
> and i use exim4 so would have to alter that to output Maildir rather than
> mbox....

> > I do that with most of https://public-inbox.org/hosted.html
> > since I'm not an admin of any of those lists.
> 
> 
> ok one of those is mailman, so do you have an example ~/publicinbox/config
> for that? i can then just cutpaste it.

Yeah, I use something like this for bug-gnulib where I'm just
a list subscriber:

[publicinbox "bug-gnulib"]
	address = bug-gnulib@gnu.org
	url = //public-inbox.org/bug-gnulib
	inboxdir = /home/user/pub/bug-gnulib
	newsgroup = inbox.comp.lib.gnulib.bug
	infourl = https://lists.gnu.org/mailman/listinfo/bug-gnulib

	; old messages had [Bug-gnulib] in the Subject, but not new ones
	; filter = PublicInbox::Filter::SubjectTag -tag [Bug-gnulib]

	; dovecot-deliver, offlineimap, mbsync, procmail, whatever
	; writes to the specified Maildir:
	watch = maildir:/home/user/Maildir/.INBOX.gnulib

	; only import messages with the given List-Id: header
	watchheader = List-Id:<bug-gnulib.gnu.org>

; optional spam checking + training stuff
[publicinboxlearn]
	watchspam = maildir:/home/user/Maildir/.INBOX.learnspam
	watchspam = maildir:/home/user/Maildir/.INBOX.spam
[publicinboxwatch]
	spamcheck = spamc

Re: setting up mailman-to-atom-converter then atom-to-public-inbox

[Kyle Meyer]

It looks the two most recent messages that Eric replied to in this
thread aren't showing up in the archive:

  CAPweEDy1qTK93pXDKdbT-HqJV184fH7x0hqqJYDTMv_nxvoKqQ@mail.gmail.com
  CAPweEDyYA+38B4uc+stMpZ9q6CrHaaAAkkorCuH4ONHmhBXbXg@mail.gmail.com

What's going on with that?  Just messages where the list was dropped?

Re: setting up mailman-to-atom-converter then atom-to-public-inbox

[Eric Wong]

Kyle Meyer <kyle@kyleam.com> wrote:
> It looks the two most recent messages that Eric replied to in this
> thread aren't showing up in the archive:
> 
>   CAPweEDy1qTK93pXDKdbT-HqJV184fH7x0hqqJYDTMv_nxvoKqQ@mail.gmail.com
>   CAPweEDyYA+38B4uc+stMpZ9q6CrHaaAAkkorCuH4ONHmhBXbXg@mail.gmail.com
> 
> What's going on with that?  Just messages where the list was dropped?

Bounced due to multipart HTML, but I only saw them since I was Cc-ed.
Odd, since Luke's earlier messages were text-only.

setting up mailman2 and public-inbox

[Luke Kenneth Casson Leighton]

eric, hi,

we're having difficulty understanding how to deploy public-inbox in a
way that very simply and as a top and only priority records email in a
public inbox, for the purposes of having it in a git repository, when
that email is coming in via exim4 and going into mailman2.

this really cannot be difficult.

we do not want to replace mailman2: its front-end, the subscription
capabilities, the user management are perfect and protect against
spammers, keeping the list clean.

we do not want to replace the pipermail archives at this point,
although that may be something we could consider at a later date.

the *only* thing that we want public-inbox for - right now - is its
ability to store the list email - *after* processing and acceptance by
mailman - in a git-backed repository, so that people in offline
situations have full and complete access to the archives.

how *exactly* do we do that one thing and one thing only?

(basically, if public-inbox accepted mbox format we could do what we
need, straight away.  the conversion process and the extra CPU time is
not a problem, because we already have to accept that overhead on
conversion, and the lists each month are not that big: 700 messages or
so).

are we missing something really simple?

if we subscribe a local user on the server (inbox@libre-riscv.org) to
the actual list, then configure that local account to have mail
delivered Maildir format, would that do the trick?

tia,

l.

Re: setting up mailman2 and public-inbox

[Eric Wong]

Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> eric, hi,
> 
> we're having difficulty understanding how to deploy public-inbox in a
> way that very simply and as a top and only priority records email in a
> public inbox, for the purposes of having it in a git repository, when
> that email is coming in via exim4 and going into mailman2.
> 
> this really cannot be difficult.

Add a regular subscriber that receives mail via normal Mailman
methods.

Then setup public-inbox-watch to watch a Maildir that normal
subscriber receives mail in.  The top of public-inbox-watch(1)
manpage should give a reasonably complete example.

I use offlineimap for IMAP <-> Maildir sync, but mbsync works
just as well.  There's other methods, of course.

> we do not want to replace mailman2: its front-end, the subscription
> capabilities, the user management are perfect and protect against
> spammers, keeping the list clean.
> 
> we do not want to replace the pipermail archives at this point,
> although that may be something we could consider at a later date.
> 
> the *only* thing that we want public-inbox for - right now - is its
> ability to store the list email - *after* processing and acceptance by
> mailman - in a git-backed repository, so that people in offline
> situations have full and complete access to the archives.

Totally understood.

There's absolutely no requirement for public-inbox to even run
on the same machine as mailman|exim.  I run https://public-inbox.org/git/
and do so using public-inbox-watch just as a regular subscriber
with no special access to kernel.org whatsoever.

> how *exactly* do we do that one thing and one thing only?
> 
> (basically, if public-inbox accepted mbox format we could do what we
> need, straight away.  the conversion process and the extra CPU time is
> not a problem, because we already have to accept that overhead on
> conversion, and the lists each month are not that big: 700 messages or
> so).

mbox is really only useful for one-shot imports and the
scripts/import_vger_from_mbox example script was recently
updated in git master to be more flexible.

For incremental updates, mbox is terrible since the mbox can be
rewritten/reordered and we'd constantly have to rescan +
deduplicate instead of being able to use inotify to only see
changes.

> are we missing something really simple?
> 
> if we subscribe a local user on the server (inbox@libre-riscv.org) to
> the actual list, then configure that local account to have mail
> delivered Maildir format, would that do the trick?

Exactly :)

I also suggest something to cleanup old messages, something
like:

	find /path/to/maildir -type f -ctime +14 -print0 | xargs -0 rm -f

To delete all messages older than 14 days

I'd like to eventually have an auto-deleter which verifies the
message is successfully imported into an inbox (and not rejected
as spam or triggered some other error).

Re: setting up mailman2 and public-inbox

[Luke Kenneth Casson Leighton]

On Wed, Mar 11, 2020 at 10:33 AM Eric Wong <e@yhbt.net> wrote:

> Add a regular subscriber that receives mail via normal Mailman
> methods.
>
> Then setup public-inbox-watch to watch a Maildir that normal
> subscriber receives mail in.  The top of public-inbox-watch(1)
> manpage should give a reasonably complete example.

ahhh hurrah.

> I use offlineimap for IMAP <-> Maildir sync, but mbsync works
> just as well.  There's other methods, of course.

i like offlineimap.  we've not got an imap server set up however (my
experiences with cyrus2.2 and exim4 are well-documented and the source
of much amusement)

> There's absolutely no requirement for public-inbox to even run
> on the same machine as mailman|exim.  I run https://public-inbox.org/git/
> and do so using public-inbox-watch just as a regular subscriber
> with no special access to kernel.org whatsoever.

fortunately there are no local users on the server (at all) so playing
with it, to get this set up does no "damage".

> mbox is really only useful for one-shot imports and the
> scripts/import_vger_from_mbox example script was recently
> updated in git master to be more flexible.

ok good to know.

> > if we subscribe a local user on the server (inbox@libre-riscv.org) to
> > the actual list, then configure that local account to have mail
> > delivered Maildir format, would that do the trick?
>
> Exactly :)

hurrah :)

> I also suggest something to cleanup old messages, something
> like:
>
>         find /path/to/maildir -type f -ctime +14 -print0 | xargs -0 rm -f
>
> To delete all messages older than 14 days
>
> I'd like to eventually have an auto-deleter which verifies the
> message is successfully imported into an inbox (and not rejected
> as spam or triggered some other error).

yeah that would be really clean.  or, perhaps allow move the message
to a "read" Maildir folder?

thanks eric.

l.

Re: setting up mailman2 and public-inbox

[Luke Kenneth Casson Leighton]

https://git.libre-riscv.org/?p=libre-riscv-dev.git;a=blob;f=be/83d4868f18ca8977e96977f9f0b4ce5b8bb744;h=adaedcdce49d930abc37b37d5f4bc3a82d3d8f57;hb=a8b2cf36e0fe7bab23fa012e4490f60afcfc1e49

HA! :)