Re: [v2] one file to rule them all?

[Eric Wong <e@80x24.org>]

Eric Wong <e@80x24.org> wrote:
> Timing "git rev-list --objects --all |wc -l" reveals much bigger
> differences.  Timings are a bit fuzzy since (AFAIK) this is a
> shared system, but it's not really a contest:
> 
> 2-38         ~5 minutes
> 2-2-2-34     ~30 seconds
> 2-2-36       ~30 seconds
> 1-file       ~5 seconds
> 
> Smaller trees are way faster :)

The LKML 2000-2017 archives (16GB uncompressed mboxes) I have
are 6.3G of objects with 1-file storage in git and took around
33 minutes to do a full import utilizing a single core and
single git repo (no deduplication checks).

"git repack -adb" takes about 2 minutes

"git rev-list --objects --all |wc -l"
takes around 1 minute with over 8 million objects

As a baseline, pure Perl parsing of the mboxes (no writing to
git) was around 23 minutes on a single core; so git-fast-import
does add some overhead but probably not as much as Xapian will
add for the initial import.

The v1 2-38 code slowed to a crawl as more data got into the
repo and I gave up after it hit 18G and hit snags with
badly-formatted dates (worked around by relying on Date::Parse
instead of git's RFC2822 parser).

Side note:  Using 4 parallel processes for the parse-only tests
took around 10.5 minutes; while 2 processes took around 11-12
minutes.  Then I realized 2 of the 4 processors were HT,
so it appears HT doesn't help much with Perl parsing...

> In other words, git scales infinitely well to deep history
> depths, but not to breadth of large trees[1].

Serving a ~6G for clones is still a lot of bandwidth; so
partitioning the git repos to limit the size of each clone
seems worth it.

Yearly partitions is probably too frequent and we'd end up with
too many packs (and resulting more open-FDs, cache-misses,
metadata stored in Xapian).  I think partitioning based on
message-count/sizes might be a better metric for splitting as
LKML seems to get more traffic year-after-year.

> Marking spam and handling message removals might be a little
> trickier as chronology will have to be taken into account...
> (will post more on this, later)

Keeping track of everything in Xapian while walking backwards
through git history shouldn't be a big problem, actually.
(Xapian has read-after-write consistency)

However, trying to reason about partitioning of Xapian DBs
across time/message-count boundaries was making my head hurt and
now I'm not sure if it's necessary to partition Xapian DBs.

While normal text indexing is trivial to partition and
parallelize, associating messages with thread IDs requires
"global" knowledge spanning all partitions (since mail threads
span multiple periods).  Unfortunately, this requires much
synchronization and synchronization hurts parallelism.

Partitioning Xapian DBs is useful to speed up full-indexing and
not much else.  Full-(re)indexing is a rare event, and can be
done on a cold DB while the hot one is taking traffic.  In fact,
I would expect lookups on partitioned DBs to be slower since it
has more files to go through and has to map things like
internal document_ids to non-conflicting ones.

Also, we don't serve Xapian data to be cloned; which is the main
reason to do partitioning of git storage...