Re: internal format

[Eric Wong <e@80x24.org>]

Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:
> > v1 or v2?  Some of the reasoning for v2 was here:
> >   https://public-inbox.org/meta/20180209205140.GA11047@dcvr/
> 
> IIUC, the issues you consider important are:
> 
> - Size
> - Time to perform "git rev-list --objects --all"
> - Flexibility, e.g. to be able to remove messages.
> 
> For size your benchmarks seem to indicate that as long as it's kept
> inside Git, the choice of format doesn't actually affect it
> significantly (and this matches my expectations).
> Tho I guess it's probably possible to improve on it with enough efforts
> (e.g. storing attachments separately, or splitting large messages into
> chunks, e.g. like `bup` does), but I doubt it's worth the effort
> (especially if you assume that the mailing-list imposes a limit on
> message size).

Right, I decided splitting big messages wasn't worth the
complexity and we leave it up to the (usually reasonable)
mail server.

> For timing, I'm curious why you only consider
> "git rev-list --objects --all".  Which operation does this corresponds
> to in public-inbox and is that really the only one that is
> performance-sensitive?

That traverses the object graph (same walk used for repacking
where bitmaps don't help).  I got it from Peff
https://public-inbox.org/git/20160805092805.w3nwv2l6jkbuwlzf@sigill.intra.peff.net/

That's the main thing we can control with repository layout.
Large packs are generally a problem with git, so v2 partitions
repositories at roughly 1G.

> > As for git itself: reliability, ease-of-replication, storage
> > efficiency.
> 
> Yes, that part I totally understand (same reason I used Git in BuGit
> https://gitlab.com/monnier/bugit).  Part of my question was related to
> the fact that in BuGit I store the messages in the commit-object rather
> than in files (which trivially gives me conflict-free merges as well as
> "discussion threads") so I was wondering if it would make sense in the
> case of public-inbox to keep the email messages in the commit objects
> rather than in files, but since I don't really know which operations are
> frequent/important I really have no idea.

I thought about storing messages in the commit object, but that
would break our current use of Xapian if history rewrites are
required for legal reasons.

> One thing that strikes me is that you don't seem to use its
> "decentralization": IIUC public-inbox always assumes one of the
> repositories is the "master" and others are mirrors (or mirrors of
> mirrors), so you get efficient "fast-forward" updates, but you
> don't do "merges".

Right, git merges require the use of pre-established
communications channels (e.g. email) to coordinate.  I don't
believe merging and keeping an authoritative history/order makes
sense with public-inbox (more on this later).

What's important to decentralization is the "root" can
change easily (change of URLs / archival addresses) and all
the messages eventually end up replicatable.

I consider ease-of-replication and efficiency the building
blocks of decentralization.

Beyond that, I believe encouraging "pull" via NNTP and
discouraging "push" via SMTP with mlmmj/mailman/etc. can
eventually lend itself to entirely forkable communities.

> This probably means that keeping the email messages in commit objects
> wouldn't bring any benefits.
> 
> Also this means that public-inbox could freely rewrite history, for
> example (which you'll need to really expunge messages) and just use
> "forced updates" in mirrors.

We currently store blob SHA-1s in Xapian to avoid tree lookups
in git.  Having a history rewrite can break an entire chain of
unrelated messages if we store commit SHA-1 in Xapian instead of
blobs.

> Now I'm left wondering what it would mean for something like
> public-inbox to support merging.

I consider it a waste of effort to maintain an authoritive
commit history when archiving mail.  There's too many variables
when it comes to mail servers and headers and no guarantees on
message ordering.  Among other things, the last (top) Received:
header will surely differ if multiple people start archiving a
list independently of each other.

The email messages are what's important, so replaying an
mbox/Maildir into an importer will get the data that matters
(and deduplication checks will avoid redundant mails).