On Sun, Aug 21, 2016 at 06:37:04PM +0000, Eric Wong wrote: > "W. Trevor King" wrote: > > On Fri, Nov 07, 2014 at 11:03:21AM -0800, W. Trevor King wrote: > > > Eric Wong has been working on some tools to store email in a Git > > > repository, and his client-side code is ssoma [1]. I wanted a bit > > > more metadata than the stock ssoma-mda [2], and ended up just > > > writing a ssoma-mda in Python [3]… > > Btw, for public-inbox, I'm using git-fast-import now, so imports are > a bit faster and $GIT_DIR/ssoma.index is no longer used. This was > crucial for getting git@vger archives imported in a reasonable time. > > public-inbox-* still keeps ssoma.index up-to-date for backwards > compatibility with ssoma, and will probably do so until 2020 or > later (there'll be a few years of deprecation notices) > > So I or someone else needs to update Perl ssoma to use fast-import > at some point, too; and I suggest your python version do the same. ssoma-mda imports 22k notmuch messages in around 15 minutes (with profiling enabled), and: $ python -m cProfile -o profile import.py notmuch.mbox $ python -c "import pstats; p=pstats.Stats('profile'); p.sort_stats('cumulative').print_stats(10)" Sun Aug 21 12:56:49 2016 profile 101823722 function calls (99078415 primitive calls) in 885.069 seconds Ordered by: cumulative time List reduced from 1145 to 10 due to restriction <10> ncalls tottime percall cumtime percall filename:lineno(function) 70/1 0.002 0.000 885.069 885.069 {built-in method exec} 1 0.111 0.111 885.069 885.069 /home/wking/src/notmuch/notmuch-archives.git/import.py:9() 1 0.400 0.400 884.915 884.915 /home/wking/src/notmuch/notmuch-archives.git/import.py:17(import_mbox) 22875 0.601 0.000 863.371 0.038 /home/wking/src/notmuch/notmuch-archives.git/ssoma_mda.py:362(deliver) 22875 8.943 0.000 810.459 0.035 /home/wking/src/notmuch/notmuch-archives.git/ssoma_mda.py:207(append) 22875 0.418 0.000 308.353 0.013 /home/wking/.local/lib64/python3.4/site-packages/pygit2/index.py:146(write_tree) 22875 307.855 0.013 307.855 0.013 {built-in method git_index_write_tree} 22874 0.575 0.000 279.293 0.012 /home/wking/.local/lib64/python3.4/site-packages/pygit2/index.py:238(diff_to_tree) 22874 278.501 0.012 278.501 0.012 {built-in method git_diff_tree_to_index} 22875 0.088 0.000 80.413 0.004 /home/wking/.local/lib64/python3.4/site-packages/pygit2/index.py:99(read) 38 ms per ssoma delivery is probably fast enough, especially if you are invoking ssoma-mda once per message, since process setup will take a similar amount of time: $ time python -c 'print("hello")' hello real 0m0.016s user 0m0.013s sys 0m0.003s It's possible that fast-import would shave a few ms off the pygit2 addition (I'm not sure, and maybe pygit2 is faster than fast-import). But I doubt it matters enough either way to be worth changing unless you are dealing with a really large corpus. Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy