From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eric Abrahamsen Newsgroups: gmane.emacs.devel Subject: Re: How to search on emacs-devel? Date: Thu, 10 Mar 2022 09:54:06 -0800 Message-ID: <878rthithd.fsf@ericabrahamsen.net> References: <874k47nemv.fsf@ericabrahamsen.net> <87wnh3lzo1.fsf@ericabrahamsen.net> <87wnh22w7o.fsf@kyleam.com> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="11530"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) Cc: emacs-devel@gnu.org To: Kyle Meyer Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Thu Mar 10 18:55:33 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nSN0e-0002mt-W8 for ged-emacs-devel@m.gmane-mx.org; Thu, 10 Mar 2022 18:55:33 +0100 Original-Received: from localhost ([::1]:49088 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nSN0d-0006H9-Nf for ged-emacs-devel@m.gmane-mx.org; Thu, 10 Mar 2022 12:55:31 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:39488) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nSMzT-0004AV-IX for emacs-devel@gnu.org; Thu, 10 Mar 2022 12:54:22 -0500 Original-Received: from mail.ericabrahamsen.net ([52.70.2.18]:39152) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nSMzR-0000VX-O2 for emacs-devel@gnu.org; Thu, 10 Mar 2022 12:54:19 -0500 Original-Received: from localhost (c-71-197-232-41.hsd1.wa.comcast.net [71.197.232.41]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id 19133FA0B2; Thu, 10 Mar 2022 17:54:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericabrahamsen.net; s=mail; t=1646934856; bh=57lAODckn9WYJwIqUQd9hC97U8lR7PKUW/Yx4Xj+Dww=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=wXMBn1a2SggTDpd/iP7GYQshr4dC0teb5x7Unz1apB2G26rag3gXT/0MaQbFAIsUp 3PsErDxCbu4qvvZJYswm7Sr82IiQgO/MXN+yJQoLnp+OGo5s8iE0ljUECfKQT7jFUB Y04vmrcSlWjYwVsMY810hDUEF+1vZidVflxznsJk= In-Reply-To: <87wnh22w7o.fsf@kyleam.com> (Kyle Meyer's message of "Thu, 10 Mar 2022 00:49:31 -0500") Received-SPF: pass client-ip=52.70.2.18; envelope-from=eric@ericabrahamsen.net; helo=mail.ericabrahamsen.net X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:286997 Archived-At: On 03/10/22 00:49 AM, Kyle Meyer wrote: > Eric Abrahamsen writes: > >> Eric Abrahamsen writes: >> >>> Andreas Schwab writes: >>> >>>> public-inbox is git under the hood (see >>>> https://yhetil.org/emacs-devel/_/text/mirror/). >>> >>> Maybe local indexing with notmuch or what have you is still the easiest >>> solution, to. > > For those that want to go this route, the main thing is getting the > Git-stored messages [1] into a format that the indexer understands. > There are some existing tools (not part of public-inbox) that can > convert a public-inbox archive into a Maildir. l2md [2] and impibe [3] > are the ones listed at > , > though I haven't used either of them myself. (There's also > public-inbox's lei, which I'll get to below.) > > In the case of Notmuch, the idea of making the Maildir-based Notmuch > understand public-inbox archives was mentioned briefly on the Notmuch > mailing list, but that hasn't gone anywhere yet: > > https://yhetil.org/notmuch/20210501051612.GB6409@dcvr/ Thanks for all this background! I guess I don't think there's much value in converting public-inbox formats to something that notmuch can read, if public-inbox comes with its own search facilities. > > [1] Documentation on public-inbox's archive format: > https://public-inbox.org/public-inbox.git/plain/Documentation/public-inbox-v2-format.pod > [2] https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git > [3] https://leahneukirchen.org/dotfiles/bin/impibe > >> Oh I see, if you have the public-inbox package installed locally, then >> it can be used to index the git repo and provide a search interface. >> Installing public-inbox on Arch has a dependency on approximately 9,000 >> perl packages, so I won't be doing that, but it looks like that would be >> the easiest solution. > > I suspect you're referring to public-inbox's lei (local email interface) > that was introduced in the most recent release (1.7.0). One way to use > it is to wire up some number of archives as "externals". These can be > remote, local, or some combination. > > For example, below adds emacs-devel as a local external. This clones > the corresponding Git repo and indexes it with public-inbox-index. It > unsurprisingly takes some time (maybe around 20 minutes on my machine). > > $ mkdir -p inboxes > $ lei add-external --mirror https://yhetil.org/emacs-devel inboxes/emacs-devel > > Later, you can update the Git repo and index new messages with > > $ make -C inboxes/emacs-devel update > > And here's an example of adding a remote external: > > $ lei add-external https://yhetil.org/emacs-bugs > > The lei-q command is the main interface for searching: > > $ lei q f:eric d:2.days.ago.. | wc -l > 8 > $ lei q f:eric d:2.days.ago.. | head -1 > [{"blob":"91ec6216f8daa3b731c832f5c5de722e891b49b7","dt":"2022-03-09T22:23:54Z","f":[["Eric Abrahamsen" ... > > If there are any local externals configured, lei-q won't reach out to > the remote externals unless explicitly requested with --remote: > > $ lei q --remote f:eric d:2.days.ago.. | wc -l > 9 > > You can also limit a search to only a set of externals: > > $ lei q --only emacs-bugs f:eric d:2.days.ago.. | jq '.[0].s' > "bug#54324: 29.0.50; Don't error during debbugs-gnu-rescan" > > Or specify a one-off remote URL that wasn't added via add-external: > > $ lei q --only https://lore.kernel.org/git userdiff NEAR emacs > > All of the above commands show JSON output, but there are several output > formats available, including various mbox flavors and Maildir. > > Anyway, I've already written too much, but, for those interested, > https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started > along with lei-overview(7) and the other lei- manpages may be worth > checking out. All this is really interesting, thank you. It looks like one tool (or tool suite) will handle mail transport, storage, and search -- that's really nice. I'll read the overview, hold my nose and install all the dependencies, and play around with it. Thanks again, Eric