From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 5197F6DE0C1B for ; Sun, 21 Aug 2016 10:38:14 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -0.07 X-Spam-Level: X-Spam-Status: No, score=-0.07 tagged_above=-999 required=5 tests=[AWL=0.231, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.211, T_SPF_TEMPERROR=0.01] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZOj_bSniRK6H for ; Sun, 21 Aug 2016 10:38:10 -0700 (PDT) Received: from resqmta-ch2-12v.sys.comcast.net (resqmta-ch2-12v.sys.comcast.net [69.252.207.44]) by arlo.cworth.org (Postfix) with ESMTPS id 9756C6DE0C19 for ; Sun, 21 Aug 2016 10:38:05 -0700 (PDT) Received: from resomta-ch2-01v.sys.comcast.net ([69.252.207.97]) by resqmta-ch2-12v.sys.comcast.net with SMTP id bWgvbeorsxBKTbWhAbAT1I; Sun, 21 Aug 2016 17:38:04 +0000 Received: from mail.tremily.us ([73.221.72.168]) by resomta-ch2-01v.sys.comcast.net with SMTP id bWfBbnXY9m3k1bWfCbjq4t; Sun, 21 Aug 2016 17:36:04 +0000 Received: by mail.tremily.us (Postfix, from userid 1000) id 7080D1E8CCF7; Sun, 21 Aug 2016 10:36:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tremily.us; s=odin; t=1471800960; bh=HmByPed2GtLRvfCaUFgpVjJlclP23oc15bO8FyfhYoo=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=KODVlViYLPQPFNSDK2r30nuJgyxf92ShZt0VRPhkLp/r+C0EFHPbZmO61d9cAV4Kk RJ2Sgmg29N85TSanO9KFYBuJ5tz3felfjuMvUMOIO2TcN7KtnStFaFEc0i0nv6wtKJ zq0oz2lTDyNxa7iEP7QXniIKMKHFIewYsn/BnRNA= Date: Sun, 21 Aug 2016 10:36:00 -0700 From: "W. Trevor King" To: Eric Wong Cc: notmuch@notmuchmail.org, David Bremner , Steven Allen , Tomi Ollila , Carl Worth , meta@public-inbox.org Subject: Re: Mail archives in Git using ssoma (Docker image) Message-ID: <20160821173600.GB30347@odin.tremily.us> References: <20141107190321.GL23609@odin.tremily.us> <20160821043631.GA2338@odin.tremily.us> <20160821094833.GB2338@odin.tremily.us> <20160821120852.GA12964@dcvr> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BKMA2OjimIttCqMD" Content-Disposition: inline In-Reply-To: <20160821120852.GA12964@dcvr> OpenPGP: id=39A2F3FA2AB17E5D8764F388FC29BDCDF15F5BE8; url=http://tremily.us/pubkey.txt User-Agent: Mutt/1.5.23 (2014-03-12) X-CMAE-Envelope: MS4wfBdZPKo7ISOgbJ8ubxXq4P/5ERZV73QtU07mzJXXOfSqOj6OMa906dqf9Kr+fnCgPUxcJnT4KZvmUjvaQV/RzX60vpvPt9N6OHRzQKkKHSAuBfClPriN F61Ijv9qHwunfJo3RDBNQxVyx+oQOhUMHhdwHDqMPdoxt5Tgz4GPoua66vtLe70+Lpy+VaRZ9lF7spNr2mtg41BUC4lFgSDV/PCu8xvYpNlZY1VuMHgRgOtl 5IMKF57vRWiwD+l4QGfqfW+lQRSw3MxjAFgzoIQosAWUexcgiCWZmSwhz6C0r8mtxfNqr0yk8g8gsVF/5PPs8e9kKM7TM48Uvfw3kp0TxgA= X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Aug 2016 17:38:14 -0000 --BKMA2OjimIttCqMD Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Aug 21, 2016 at 12:08:52PM +0000, Eric Wong wrote: > "W. Trevor King" wrote: > > This is the ssoma archive (with the data in it). I just set up a > > basic HTTP archive (following [1]) based on a Docker image [2] (Gentoo > > doesn't package all the Perl dependencies public-inbox needs). >=20 > Ugh, that sucks (sorry, not a fan of Docker). >=20 > What's missing from Gentoo? Gentoo doesn't package (or I couldn't find the package for) Encode::MIME::Header or Mail::Thread. I tried installing things from CPAN, but ran into a compile-time error from the =E2=80=98cpan=E2=80=99 inv= ocationand gave up ;). I can try and reproduce the error if you're curious, but I don't have it handy at the moment. > > $ git config -f srv/notmuch.git/config publicinbox.http http://tremil= y.us > > $ git config -f srv/notmuch.git/config publicinbox.email notmuch@notm= uchmail.org >=20 > That should probably be: >=20 > ; based on your [3] > git config -f srv/notmuch.git/config \ > publicinbox.notmuch.url http://tremily.us/notmuch >=20 > git config -f srv/notmuch.git/config \ > publicinbox.notmuch.address notmuch@notmuchmail.org >=20 > ; this is crucial for all the public-inbox-* tools > git config -f srv/notmuch.git/config \ > publicinbox.notmuch.mainrepo /path/to/notmuch.git I was using these in the Dockerfile's CMD: (cd /srv; for NAME in *; do CONF=3D"/srv/${NAME}/config"; public-inbox-init "${NAME}" "/srv/${NAME}" $(git config -f "${CONF}" p= ublicinbox.http) $(git config -f "${CONF}" publicinbox.email); done) && =E2=80=A6 Are you saying that I can skip the ~/.public-inbox/config entries setup by public-inbox-init if I set publicinbox.{name}.* in the ssoma repository's config? That would be nice. I don't see a point to having {name} in ssoma-config settings though, since you're already in a single bucket by that point (using publicinbox.{name}.* makes sense in the multi-bucket ~/.public-inbox/config). > > It's not updating automatically yet, but that will probably look > > like: > >=20 > > 1. Pull new mbox [4]. > > 2. Import into notmuch-archives [5]. > > 3. Re-run public-inbox-index (this could probably be via =E2=80=98docke= r exec =E2=80=A6=E2=80=99. > >=20 > > But I'll have to test that to confirm. And ideally we'd be using > > ssoma-mda or similar directly, instead of going through mbox, but I'd > > rather get the official headers on the stored mail than be efficient > > ;). >=20 > For mirroring existing lists, I started using public-inbox-watch > which currently watches Maildirs. If I had a Maildir locally, I'd just use procmail and push new messages into ssoma-mda. I'm using the import script because my local mail has =E2=80=9Chow we delivered this to Trevor=E2=80=9D headers (which I= don't want to add) but the downloaded mbox has =E2=80=9Chow we delivered this to notmuch@notmuchmail.org=E2=80=9D (which seems like a better fit for a shared ssoma repo). > I recommend public-inbox-watch for mirroring existing lists (such as > what I did with git@vger) but public-inbox-mda for self-hosted lists > (such as meta@public-inbox.org). Why is that? Procmail + public-inbox-mda (or my Python ssoma-mda fork [1,2]) seems simpler and equally effective if you want to insert a message that your mail system is delivering locally. > > One shift from Gmane's mid.gmane.org/=E2=80=A6 is that the public-inbox= UI > > Message-ID lookup is per-bucket, and public-inbox seems to be > > encouraging per-list buckets. >=20 > The public-inbox-nntpd interface supports mid lookups across all > inboxes in that instance; so it should be doable in the WWW > interface, too. Either way, I think it has to be O(n) where (n) is > the number of Xapian DBs, though. I'm more concerned about the interface, and less about the implementation (which can be improved later). The (n) lookups are trivially parallelizable, and you can always add a Message-ID =E2=86=92 buckets lookup table if (n) lookups turns out to be too slow. Cheers, Trevor [1]: id:20141107190321.GL23609@odin.tremily.us [2]: id:af679af8257e250ac606e35a1307ad02907b8426.1413663212.git.wking@tremi= ly.us http://public-inbox.org/meta/af679af8257e250ac606e35a1307ad02907b8426.= 1413663212.git.wking@tremily.us/t/#u --=20 This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy --BKMA2OjimIttCqMD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJXueZ9AAoJEBBpoQVgXJg1mCMQAKEQMD0ABiHjlVgRKUHcMiTJ oeLnqjs1JMjxoiazl2Byv1Xv8n/9a7CyYBh6eqWXVFoW8r+bzQ7cLmWN1/8UD990 eGlpazzMU/+9T2lK87mUZrAFVcvtzaOHx+RWSJ1XJ0vqBbhXGfXB4FXBTsvkZkeQ 9MkGi07+TF27fJlaAz5Waz81aIUuz2TjX8QjC79DgtR0N9ZMSJawXY+htEufUVBq k53bds76D9fTNiwYFOPDr0Mp+JxYtnfkg2SGG5PtzUtIRlBVXYAFVMxH8CEDqVUv QO501v7lZnF5irZ7JLES6lddZmd03/761NbkIp8Vdpy27I6ltTOVhvUog6PEGZrZ xJK4+RaG7RO4vlZfw2mywMatBRef/9XQWenQW6wAwPYgDm9ilEj65QOv/meo3538 YO4uI/8UsFQhaHugctwuUD+S8Oasked8DaVgTz2h0/jDVUKWkGtK0zgfKE7NWdn9 MS5+AVfGU84v7hTURrk0a0zR/7IzkMLNqgxenjGPkOnhPH0nsrE1+fslkM4HNtMd HlfwSzr5Jw1AyqH2e8z/Cur9uqovVNzD+sSn4nU13UOcJYcsJ4G0eV3MQzh/yegB YVW3dp5bNPm9KJzJOXVpVCmHDLbkMdoZ99tJVqaYxWKMAWRt9+D3e9Z8mhKkhQWc EG/Bfv42RGKelMr0XwXf =V9VB -----END PGP SIGNATURE----- --BKMA2OjimIttCqMD--