unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
From: Eric Wong <e@80x24.org>
To: Ali Alnubani <alialnu@mellanox.com>
Cc: meta@public-inbox.org
Subject: Re: mailman mbox migration
Date: Thu, 21 Mar 2019 16:05:42 +0000	[thread overview]
Message-ID: <20190321160542.fzuqgwdx5qrmfrwr@dcvr> (raw)
In-Reply-To: <VI1PR05MB42699DE48A2290EAC18D223ED7420@VI1PR05MB4269.eurprd05.prod.outlook.com>

Ali Alnubani <alialnu@mellanox.com> wrote:
> Hi Eric,
> 
> Thanks for help and I apologize for replying quite late.
> 
> The script import_vger_from_mbox worked very well.

Good to know :>

> Do you think that there might be an issue in a few messages
> being imported twice by both import_vger_from_mbox and
> public-inbox-watch? Since the lists I'm migrating are very
> busy, and there will be a delay between importing with the
> script and running public-inbox-watch.

Messages they are deduped by Message-ID and content.  However,
V2 allows different messages to use the same Message-IDs,
(because some non-spam-but-buggy bots/mailers do it).  So if
Mailman mangles the message going into the mbox differently than
the one going into the Maildir for -watch, then you can get
duplicates.

Fwiw, mass imports are much faster if you use "eatmydata", a
LD_PRELOAD which disables fsync.  On a reasonably fast VM with
good, TRIM-ed SSD ("fstrim -a" first), and lots of RAM,
importing 2000-2017 LKML history took around 3-4 hours.  More
cores only helps if your SSD can keep up, and I seem to remember
using NPROC=4 (via env) was the point of diminishing returns for
that VM I used.

      reply	other threads:[~2019-03-21 16:05 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-02-13 14:48 mailman mbox migration Ali Alnubani
2019-02-13 22:31 ` Eric Wong
2019-03-21 13:23   ` Ali Alnubani
2019-03-21 16:05     ` Eric Wong [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190321160542.fzuqgwdx5qrmfrwr@dcvr \
    --to=e@80x24.org \
    --cc=alialnu@mellanox.com \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).