Re: Q: V2 format

[ebiederm@xmission.com (Eric W. Biederman)]

Eric Wong <e@80x24.org> writes:

> "Eric W. Biederman" <ebiederm@xmission.com> wrote:
>> I have been digging through the code looking so I can understand the v2
>> format and I have some ideas on how things might be improved, and some
>> questions so that I understand.
>
> Great to know you're interested!  Fwiw, I've still been meaning
> to turn my v2 docs into a POD manpage:
>
>   https://public-inbox.org/meta/20180419015813.GA20051@dcvr/

I have some personal mail archives that I need to do something better
with.  My goal is for day-to-day operations (aka mail delivery and
archiving) to be able to run on a smallish 32bit machine.

But archives are not valuable unless you have a fast search capability
which makes all of the features of xapian very interesting.

I need to compare message id's to see if I have content missing from the
public linux-kernel archive.   It is probably Konrad's cleanup of the
headers but my linux-kernel archive when imported into public-inbox is
slightly larger than Konrads.

I also like the idea of being able to read and archive public lists that
I care about with just a git fetch and local tools.

Public mailing lists and their archives are more important, but on my
radar is also IMAP/regular email support.  With it's little bit of extra
state.

>> V1 supported the concept of messages being added and deleted from
>> the git repository all while keeping a full history of everything that
>> went on.  The V2 code appears to have the name 'm' for added and 'd' for
>> deleted, but the public-inbox-index code appears to expect deletes to
>> happen by way of an altered history that totally purge the commits,
>> and does not process the 'd' entries.
>
> "Purge" is a new concept for v2 and not even exposed (yet) in
> via tools.  Normal operations to remove files using 'd' (via
> -watch or -rm) don't rewrite old history so it won't disrupt
> non-force fetches.

This helps a lot in understanding the intent of the code.  Konrad had
mentioned something about being able to rebase when I pointed out
the buggy git commits in linux-kernel.

>> What is the thinking about deleted entries, and for v2 what is the
>> preferred way to delete mail from a public inbox git repository and why?
>
> Definitely prefer the normal way with 'd' files to not break
> people using non-force fetches.  "Purge" is too disruptive
> and reserved for extraordinary cases (e.g. legal reasons).

Then I am going to report a probable bug.  In V2 in public-inbox-index
I can not find a path from finding a 'd' file and a call to unindex.  V1
unindexes deleted files.  Rebased heads for purges call unindex.  I
don't see that for ordinary d files though.

>> Size.  Reading the history of the public inbox meta mailling list and
>> playing around I discovered that I can shave off about 100M of the V2
>> size of the git public inbox git repository but pushing all of the
>> messages into a single commit.  Not great for day to day operation,
>> but if rebasses are part of the plan, and old archives part of the
>> challenge I see quite a lot of potential for old archives to be reduced
>> to a git repository with a single commit.
>
> Rebases/rewriting history is definitely not part of the plan and
> a last resort.
>
>> Names.  Is there a good reason not to use message numbers as the names
>> in the git repositories?  (Other than the cost to change the code?) That
>> would remove the need for treat the sqlite msgmap database as precious,
>> and it would make it easier to recover if an nntp server goes away.  In
>> V2 format the git mailing list git repository is only about 2M larger if
>> each message has it's msg number as it's name.  Plus the git log
>> is easier to read as messages are all + or -.
>
> Big trees in git were a scalability problem in v1 because of the
> long 2/38 names.  With shorter names you propose (base-10 serial
> number?, the scalability problem gets pushed off a bit, I suppose.
> But not indefinitely; and later v2 partitions will suffer more
> from longer names.

Bit trees were a scalability problem in git becuase they are quadratic.
Every commit mentioned every email.  So a walk of the history would
have to visit every file on every commit.  I expect those tree objects
in the history compress well with their parents but it doesn't simplify
the tree walker.

Would you like my test conversion script from V1 so you can take a look?

> I also want to limit the use and exposure of serial numbers as
> much as possible.  It's unavoidable with the NNTP interface;
> but reliance on serial numbers in public interfaces leads to
> centralization.

I completely agree about public web interfaces.  Message-ID is a much
better key to messages as it was generated by the message sender.

> The current v2 is also better for inode-starved users in case
> somebody forgets to type "--mirror" or "--bare" with clone.  For
> the most part (unless purge is used), the SQLite database is
> actually recoverable.

Because of the parallelism in V2 I have noticed messages in numbered
in an order that does not correspond to their commit order.  So the
SQLite database isn't as recoverable as it might be.  Especially as the
parallelism introduces an element of non-determinancy.

> So no, I don't think having serial numbers stored in filenames
> is the right thing.

I won't push it but I at the present time I respectfully disagree.

The big advantage I see with serial numbers (other than msgmap) is that
you can include multiple emails per commit (without going quadratic).  I
am also looking at potentially storing the other email states that IMAP
and maildir mailboxes track.  I can imagine that much more easily with
message numbers.  Still I want to avoid something that makes git go
quadratic again.

>> xapian.  Can the Xapian database be made optional in V2?  
>
> Definitely in the TODO :)
>
>> I absolutely
>> think a quick search for terms and other things very valuable, so I
>> would never suggest giving up Xapian.  On the other hand on my personal
>> laptop the xapian database for lkml takes ages and ages to build, and it
>> pushes the system into swap.  Which is all around unpleasant.  That
>> seems to eat into the distributed nature of the goal of public inbox.
>> I have tried to see what could be done that might shrink the size of
>> the xapian database.  The only think I could think of is perhaps
>> sharding the xapian database by time/msgnum ranges.   That would allow
>> the old xapians databases to be compacted and forgotten about, and I
>> think it would allow less wastage in the current xapian database as it
>> would be smaller, so wasting 50% space (or whatever the btrees waste)
>> would be less of an issue.  And as smaller databases are faster I think
>> that would in general be a help.
>
> One big killer for Xapian is position information required for
> "quoted phrase searches".  I seem to remember deleting the position.*
> files was safe as it would only break phrase searches (but I
> haven't tried it).

I have a very ugly patch that removed all of Xapian.  So for day to day
nntp use.  It is certainly safe.

> So there should be an option to toggle between the "index_text"
> and routines in Xapian "index_text_without_positions".

I might take a look at that.  I just looked and the position database is
huge.

> Given the way the indexing only works on the most recent data;
> I think one could also write a script to delete old data/results
> from Xapian without affecting current/future indexing.
> That would pop back up if/when there's schema upgrades requiring
> a rebuild, though...

Good for testing.  Not for long term as it is the actual indexing that
is painful.

> I believe there should be 3 levels of v2 operation:
>
> 1) SQLite-only (NNTP and all the threading stuff works)
> 2) SQLite + Xapian w/o positions (good enough for most things)
> 3) SQLite + Xapian w/ positions (current, default)
>
> 2) seems like a reasonable trade-off for most sites; I'm not
> sure how often phrase searching gets used.

I will take a look at that.  That seems a straight forward place to
start that we can easily agree upon.

>> Time permitting I am willing to do some of this work so that
>> public-inbox works well for me.  I want to see what your vision is for
>> the code before I start anything.
>
> Thanks for running this by, first.  I'm not convinced git layout
> changes are warranted at this point for v2.
>
> Making Xapian optional and configurable to use
> index_text_without_positions is something I definitely want to
> see happen, though.

I will clean up my patches for that then.

Eric