From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 3D5546DE0C1B for ; Sun, 21 Aug 2016 13:30:26 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -0.079 X-Spam-Level: X-Spam-Status: No, score=-0.079 tagged_above=-999 required=5 tests=[AWL=0.233, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.211, SPF_PASS=-0.001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KMt6lH082Ddf for ; Sun, 21 Aug 2016 13:30:24 -0700 (PDT) Received: from resqmta-ch2-05v.sys.comcast.net (resqmta-ch2-05v.sys.comcast.net [69.252.207.37]) by arlo.cworth.org (Postfix) with ESMTPS id 3ED2C6DE0C19 for ; Sun, 21 Aug 2016 13:30:23 -0700 (PDT) Received: from resomta-ch2-06v.sys.comcast.net ([69.252.207.102]) by resqmta-ch2-05v.sys.comcast.net with SMTP id bZNmbmp7x2FGMbZNubhiuk; Sun, 21 Aug 2016 20:30:22 +0000 Received: from mail.tremily.us ([73.221.72.168]) by resomta-ch2-06v.sys.comcast.net with SMTP id bZLxbLtaCEilmbZLxblxEB; Sun, 21 Aug 2016 20:28:22 +0000 Received: by mail.tremily.us (Postfix, from userid 1000) id B46BF1E8D07B; Sun, 21 Aug 2016 13:28:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tremily.us; s=odin; t=1471811300; bh=FIsqIZ9f05eD3IXmxvrDCA0Rkc12X78s0mRj/zUf8XE=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=boET3tYtsbmIgy36Coo9Fp53de96Tbgrj11Aj9U9D6KWnsBmUQ8U9ku4bRH3MqFmN +qedsygCT2L9zePnh9+GSHeKauSxK7xJkxKUcM6+0mvF8ZrMe2tfORaN2jOBZYhtQy peBGvUInhOvh1xxck9hedcWLzOzLaWo10e4QGYuA= Date: Sun, 21 Aug 2016 13:28:20 -0700 From: "W. Trevor King" To: Eric Wong Cc: notmuch@notmuchmail.org, meta@public-inbox.org Subject: Re: Mail archives in Git using ssoma Message-ID: <20160821202820.GC30347@odin.tremily.us> References: <20141107190321.GL23609@odin.tremily.us> <20160821043631.GA2338@odin.tremily.us> <20160821183704.GB11495@dcvr> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Q7MaAb+MMjzLKrXy" Content-Disposition: inline In-Reply-To: <20160821183704.GB11495@dcvr> OpenPGP: id=39A2F3FA2AB17E5D8764F388FC29BDCDF15F5BE8; url=http://tremily.us/pubkey.txt User-Agent: Mutt/1.5.23 (2014-03-12) X-CMAE-Envelope: MS4wfInOdBDdPdG4RaMg5focWLa41P0KID3+dadXFzpgAERPRdvZE6Rw2njvC94uVFb6bvRMMhm+IqF1Vzzf8CCFFS0Q8RiUgS1VjRH/k1oF0IT8pX8pitcH L/g0eds9GdIp/MGoCs/8rLb36R+f382Ngpjyf80AbB3oJw82CNHesWzBIffOh1FHpPLPdeH6VUtJLkNQ/FfOkHC2OW6S6LKc0ssulHP1mxDxl1OFbjvxtraT X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Aug 2016 20:30:26 -0000 --Q7MaAb+MMjzLKrXy Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Aug 21, 2016 at 06:37:04PM +0000, Eric Wong wrote: > "W. Trevor King" wrote: > > On Fri, Nov 07, 2014 at 11:03:21AM -0800, W. Trevor King wrote: > > > Eric Wong has been working on some tools to store email in a Git > > > repository, and his client-side code is ssoma [1]. I wanted a bit > > > more metadata than the stock ssoma-mda [2], and ended up just > > > writing a ssoma-mda in Python [3]=E2=80=A6 > > Btw, for public-inbox, I'm using git-fast-import now, so imports are > a bit faster and $GIT_DIR/ssoma.index is no longer used. This was > crucial for getting git@vger archives imported in a reasonable time. > > public-inbox-* still keeps ssoma.index up-to-date for backwards > compatibility with ssoma, and will probably do so until 2020 or > later (there'll be a few years of deprecation notices) > > So I or someone else needs to update Perl ssoma to use fast-import > at some point, too; and I suggest your python version do the same. ssoma-mda imports 22k notmuch messages in around 15 minutes (with profiling enabled), and: $ python -m cProfile -o profile import.py notmuch.mbox $ python -c "import pstats; p=3Dpstats.Stats('profile'); p.sort_stats('cu= mulative').print_stats(10)" Sun Aug 21 12:56:49 2016 profile 101823722 function calls (99078415 primitive calls) in 885.069 s= econds Ordered by: cumulative time List reduced from 1145 to 10 due to restriction <10> ncalls tottime percall cumtime percall filename:lineno(function) 70/1 0.002 0.000 885.069 885.069 {built-in method exec} 1 0.111 0.111 885.069 885.069 /home/wking/src/notmuch/not= much-archives.git/import.py:9() 1 0.400 0.400 884.915 884.915 /home/wking/src/notmuch/not= much-archives.git/import.py:17(import_mbox) 22875 0.601 0.000 863.371 0.038 /home/wking/src/notmuch/not= much-archives.git/ssoma_mda.py:362(deliver) 22875 8.943 0.000 810.459 0.035 /home/wking/src/notmuch/not= much-archives.git/ssoma_mda.py:207(append) 22875 0.418 0.000 308.353 0.013 /home/wking/.local/lib64/py= thon3.4/site-packages/pygit2/index.py:146(write_tree) 22875 307.855 0.013 307.855 0.013 {built-in method git_index_= write_tree} 22874 0.575 0.000 279.293 0.012 /home/wking/.local/lib64/py= thon3.4/site-packages/pygit2/index.py:238(diff_to_tree) 22874 278.501 0.012 278.501 0.012 {built-in method git_diff_t= ree_to_index} 22875 0.088 0.000 80.413 0.004 /home/wking/.local/lib64/py= thon3.4/site-packages/pygit2/index.py:99(read) 38 ms per ssoma delivery is probably fast enough, especially if you are invoking ssoma-mda once per message, since process setup will take a si= milar amount of time: $ time python -c 'print("hello")' hello real 0m0.016s user 0m0.013s sys 0m0.003s It's possible that fast-import would shave a few ms off the pygit2 addition (I'm not sure, and maybe pygit2 is faster than fast-import). But I doubt it matters enough either way to be worth changing unless you are dealing with a really large corpus. Cheers, Trevor --=20 This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy --Q7MaAb+MMjzLKrXy Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIbBAEBAgAGBQJXug7hAAoJEBBpoQVgXJg1hAcP+OoGhKwqzCetwjLfpindRrrv Jbj2tIt/N/vUXjzHm1bcskrKRIoZY2PBKLnzSsB3lvStZlvsw8wElkGaMxCUJ23q AFtTNRs0WkLFbGB0swQp49vF5o2s4/EzToOHCfvVoeUuLOCrP9VJYNA1wtj1aMhS Piw/t8WkSlSBE7ioYzYbvAAE7zccdeVjzw7oK9pQcDDzsg164DrprxbhvCY5J565 XWXazMTWTRLSrWbLBb/p86/ofHC7DJWmpXPSxI+m7dzlVgsb9oDThrO742yGUWe7 SbEgMfBHFp6e7mgpIjCVPPrn43P+GgqS9S6h9ywDdkz9pg3SaZrhS8MYXy2Bgwgq yYW8To4upeL3xvIiEALq0750uSwViA7NarL7z9A0iLDVrV4x6dOb6yOeIj4woibe xpPVrIC7PYVTwb97FCrNJPdQvWU8MwPzkM8zrBZtJFh3utpg1i1nM0eP3Pl1V5Vk F+wLmP+9ZPu1QVpf4xh5J5CS5IFN98n9K5VxwunSjZEv5Ef9WklxAmtd/Pri6EnG JBN0LI6/LctmDD+2d2PLdff8amTXLi4QpoCa673lOs4aikCUx/yO9awXRnFRIwQ6 Gq0ZZZfTxkgWZqqTUePEdfhyYLBYO0/QbUjiNU3Ov/tTV9OVBf/g5toGKjwJvuOt 62+lTNd9Y/Vrh5aR1MM= =FaHG -----END PGP SIGNATURE----- --Q7MaAb+MMjzLKrXy--