Re: mailman mbox migration

[Eric Wong <e@80x24.org>]

Ali Alnubani <alialnu@mellanox.com> wrote:
> Hi Eric,
> 
> Thanks for help and I apologize for replying quite late.
> 
> The script import_vger_from_mbox worked very well.

Good to know :>

> Do you think that there might be an issue in a few messages
> being imported twice by both import_vger_from_mbox and
> public-inbox-watch? Since the lists I'm migrating are very
> busy, and there will be a delay between importing with the
> script and running public-inbox-watch.

Messages they are deduped by Message-ID and content.  However,
V2 allows different messages to use the same Message-IDs,
(because some non-spam-but-buggy bots/mailers do it).  So if
Mailman mangles the message going into the mbox differently than
the one going into the Maildir for -watch, then you can get
duplicates.

Fwiw, mass imports are much faster if you use "eatmydata", a
LD_PRELOAD which disables fsync.  On a reasonably fast VM with
good, TRIM-ed SSD ("fstrim -a" first), and lots of RAM,
importing 2000-2017 LKML history took around 3-4 hours.  More
cores only helps if your SSD can keep up, and I seem to remember
using NPROC=4 (via env) was the point of diminishing returns for
that VM I used.