How to search on emacs-devel?

How to search on emacs-devel?

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 712 bytes --]

Is there any (other) guide about how to search in the emacs-devel archives?

I am frequently trying to find some old threads, and the search
functionality usually seems to be of
no help. As the bottom of the archives page (
https://lists.gnu.org/archive/html/emacs-devel/) tells
me that the search functionality is powered by `Namazu`, I have tried to
find in its docs how to use
it. It says that using a space in a query, by default, does the same as the
AND operator (see
http://www.namazu.org/doc/manual.html#query). However, when testing it, I
have found that that is
not the case (also the and operator does not work).

Wouldn't it be a good idea to add some info about how to use the search
functionality?

[-- Attachment #2: Type: text/html, Size: 970 bytes --]

Re: How to search on emacs-devel?

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 1561 bytes --]

Ah well, I seem to have forgotten about the 'how to search' link directly
after the search field.
However, the fact still is that that info seems to be wrong. For example
(to reproduce), I was
trying to find comments about Emacs its package management, using the query
`straight package`.
I have checked if the results contained both search terms, but they don't
(see result 381
here
https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=straight+package&submit=Search%21&idxname=emacs-devel&max=20&result=normal&sort=score&whence=380
,
it does not contain the term 'package' (see
https://lists.gnu.org/archive/html/emacs-devel/2011-06/msg00192.html).

Effectively, I am usually unable to find relevant threads using the search
functionality

On Tue, 8 Mar 2022 at 16:09, dalanicolai <dalanicolai@gmail.com> wrote:

> Is there any (other) guide about how to search in the emacs-devel archives?
>
> I am frequently trying to find some old threads, and the search
> functionality usually seems to be of
> no help. As the bottom of the archives page (
> https://lists.gnu.org/archive/html/emacs-devel/) tells
> me that the search functionality is powered by `Namazu`, I have tried to
> find in its docs how to use
> it. It says that using a space in a query, by default, does the same as
> the AND operator (see
> http://www.namazu.org/doc/manual.html#query). However, when testing it, I
> have found that that is
> not the case (also the and operator does not work).
>
> Wouldn't it be a good idea to add some info about how to use the search
> functionality?
>

[-- Attachment #2: Type: text/html, Size: 2520 bytes --]

Re: How to search on emacs-devel?

[andrés ramírez]

Hi. Dalanicolai.


>>>>> "dalanicolai" == dalanicolai  <dalanicolai@gmail.com> writes:


[...]

    dalanicolai> Effectively, I am usually unable to find relevant threads using the search
    dalanicolai> functionality

As far as I could say. The namazu server is not working properly. The
indexation is not up to date. Last time I Checked (last year). It was
nearly behind 3 months.

So. I think You could search something from the last year with more
success ratio. But YMMV.

[...]

Best Regards

Re: How to search on emacs-devel?

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]

>
> As far as I could say. The namazu server is not working properly. The
> indexation is not up to date. Last time I Checked (last year). It was
> nearly behind 3 months.


Ah okay, I am often searching for more recent threads indeed.
So good to know!

Thanks! Very informative...

On Tue, 8 Mar 2022 at 16:57, andrés ramírez <rrandresf@gmail.com> wrote:

> Hi. Dalanicolai.
>
>
> >>>>> "dalanicolai" == dalanicolai  <dalanicolai@gmail.com> writes:
>
>
> [...]
>
>     dalanicolai> Effectively, I am usually unable to find relevant threads
> using the search
>     dalanicolai> functionality
>
> As far as I could say. The namazu server is not working properly. The
> indexation is not up to date. Last time I Checked (last year). It was
> nearly behind 3 months.
>
> So. I think You could search something from the last year with more
> success ratio. But YMMV.
>
> [...]
>
> Best Regards
>

[-- Attachment #2: Type: text/html, Size: 1540 bytes --]

Re: How to search on emacs-devel?

[Philip Kaludercic]

dalanicolai <dalanicolai@gmail.com> writes:

> Is there any (other) guide about how to search in the emacs-devel
> archives?
>
> I am frequently trying to find some old threads, and the search
> functionality usually seems to be of
> no help. As the bottom of the archives page (
> https://lists.gnu.org/archive/html/emacs-devel/) tells
> me that the search functionality is powered by `Namazu`, I have tried to
> find in its docs how to use
> it. It says that using a space in a query, by default, does the same
> as the
> AND operator (see
> http://www.namazu.org/doc/manual.html#query). However, when testing it, I
> have found that that is
> not the case (also the and operator does not work).
>
> Wouldn't it be a good idea to add some info about how to use the search
> functionality?

I sometimes use https://yhetil.org/emacs-devel/,
e.g. https://yhetil.org/emacs-devel/CACJP=3n854x2w0M-u9ba=e2zT=JUNyRfrqZT11co5_+Up56vMg@mail.gmail.com/

-- 
	Philip Kaludercic

Re: How to search on emacs-devel?

[Eric Abrahamsen]

Philip Kaludercic <philipk@posteo.net> writes:

> dalanicolai <dalanicolai@gmail.com> writes:
>
>> Is there any (other) guide about how to search in the emacs-devel
>> archives?
>>
>> I am frequently trying to find some old threads, and the search
>> functionality usually seems to be of
>> no help. As the bottom of the archives page (
>> https://lists.gnu.org/archive/html/emacs-devel/) tells
>> me that the search functionality is powered by `Namazu`, I have tried to
>> find in its docs how to use
>> it. It says that using a space in a query, by default, does the same
>> as the
>> AND operator (see
>> http://www.namazu.org/doc/manual.html#query). However, when testing it, I
>> have found that that is
>> not the case (also the and operator does not work).
>>
>> Wouldn't it be a good idea to add some info about how to use the search
>> functionality?
>
> I sometimes use https://yhetil.org/emacs-devel/,
> e.g. https://yhetil.org/emacs-devel/CACJP=3n854x2w0M-u9ba=e2zT=JUNyRfrqZT11co5_+Up56vMg@mail.gmail.com/

Oh! I'd read about public inbox, and then forgot about it again. I
wonder how hard it would be to write a Gnus search engine that used this
for search, then requested the actual message from Gmane...

Re: How to search on emacs-devel?

[dalanicolai]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

Great! I found the right threads instantly

On Tue, 8 Mar 2022 at 18:49, Philip Kaludercic <philipk@posteo.net> wrote:

> dalanicolai <dalanicolai@gmail.com> writes:
>
> > Is there any (other) guide about how to search in the emacs-devel
> > archives?
> >
> > I am frequently trying to find some old threads, and the search
> > functionality usually seems to be of
> > no help. As the bottom of the archives page (
> > https://lists.gnu.org/archive/html/emacs-devel/) tells
> > me that the search functionality is powered by `Namazu`, I have tried to
> > find in its docs how to use
> > it. It says that using a space in a query, by default, does the same
> > as the
> > AND operator (see
> > http://www.namazu.org/doc/manual.html#query). However, when testing it,
> I
> > have found that that is
> > not the case (also the and operator does not work).
> >
> > Wouldn't it be a good idea to add some info about how to use the search
> > functionality?
>
> I sometimes use https://yhetil.org/emacs-devel/,
> e.g.
> https://yhetil.org/emacs-devel/CACJP=3n854x2w0M-u9ba=e2zT=JUNyRfrqZT11co5_+Up56vMg@mail.gmail.com/
>
> --
>         Philip Kaludercic
>

[-- Attachment #2: Type: text/html, Size: 2046 bytes --]

Re: How to search on emacs-devel?

[Robert Pluim]

>>>>> On Tue, 08 Mar 2022 10:13:59 -0800, Eric Abrahamsen <eric@ericabrahamsen.net> said:
    >> I sometimes use https://yhetil.org/emacs-devel/,
    >> e.g. https://yhetil.org/emacs-devel/CACJP=3n854x2w0M-u9ba=e2zT=JUNyRfrqZT11co5_+Up56vMg@mail.gmail.com/

    Eric> Oh! I'd read about public inbox, and then forgot about it again. I
    Eric> wonder how hard it would be to write a Gnus search engine that used this
    Eric> for search, then requested the actual message from Gmane...

Would it be harder than downloading the emacs-devel archives locally
and then just G-f'ing them? Works offline too :-)

Robert
-- 

Re: How to search on emacs-devel?

[Eric Abrahamsen]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Tue, 08 Mar 2022 10:13:59 -0800, Eric Abrahamsen <eric@ericabrahamsen.net> said:
>     >> I sometimes use https://yhetil.org/emacs-devel/,
>     >> e.g. https://yhetil.org/emacs-devel/CACJP=3n854x2w0M-u9ba=e2zT=JUNyRfrqZT11co5_+Up56vMg@mail.gmail.com/
>
>     Eric> Oh! I'd read about public inbox, and then forgot about it again. I
>     Eric> wonder how hard it would be to write a Gnus search engine that used this
>     Eric> for search, then requested the actual message from Gmane...
>
> Would it be harder than downloading the emacs-devel archives locally
> and then just G-f'ing them? Works offline too :-)

I don't know! I've thought about the relative merits of various
approaches here. A remote search server has the advantage that you don't
need to update archives or install any search program. But local search
is faster and more flexible, and you don't occupy any else's resources.

I thought of distributing gmane search indexes via an elpa package --
rather, a package that downloads and updates indexes. Or a version of
'(nntp "news.gmane.io") that simply saves everything to disk, so you can
index it yourself. But if public inbox has some sort of API and doesn't
mind us hitting it, maybe that would be easiest?

Re: How to search on emacs-devel?

[Stefan Monnier]

> Would it be harder than downloading the emacs-devel archives locally
> and then just G-f'ing them? Works offline too :-)

I don't know how to efficiently keep a local copy of the archive in sync
with the other one.
Ideally, the archive would be kept in a Git repo so I can `git pull` to
update it efficiently.


        Stefan

Re: How to search on emacs-devel?

[Andreas Schwab]

On Mär 09 2022, Stefan Monnier wrote:

> Ideally, the archive would be kept in a Git repo so I can `git pull` to
> update it efficiently.

public-inbox is git under the hood (see
https://yhetil.org/emacs-devel/_/text/mirror/).

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: How to search on emacs-devel?

[Eric Abrahamsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Mär 09 2022, Stefan Monnier wrote:
>
>> Ideally, the archive would be kept in a Git repo so I can `git pull` to
>> update it efficiently.
>
> public-inbox is git under the hood (see
> https://yhetil.org/emacs-devel/_/text/mirror/).

Maybe local indexing with notmuch or what have you is still the easiest
solution, then.

Re: How to search on emacs-devel?

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> On Mär 09 2022, Stefan Monnier wrote:
>>
>>> Ideally, the archive would be kept in a Git repo so I can `git pull` to
>>> update it efficiently.
>>
>> public-inbox is git under the hood (see
>> https://yhetil.org/emacs-devel/_/text/mirror/).
>
> Maybe local indexing with notmuch or what have you is still the easiest
> solution, then.

Oh I see, if you have the public-inbox package installed locally, then
it can be used to index the git repo and provide a search interface.
Installing public-inbox on Arch has a dependency on approximately 9,000
perl packages, so I won't be doing that, but it looks like that would be
the easiest solution.

An nnpublic-inbox backend for Gnus would be nice, at some point.

Re: How to search on emacs-devel?

[Stefan Monnier]

> An nnpublic-inbox backend for Gnus would be nice, at some point.

Even better:

- A bidirectional IMAP <-> Git gateway.
- an nngit backend (not just read-only).


        Stefan

Re: How to search on emacs-devel?

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> An nnpublic-inbox backend for Gnus would be nice, at some point.
>
> Even better:

Interesting!

> - A bidirectional IMAP <-> Git gateway.

But this bit would live on a server someplace, not locally?

> - an nngit backend (not just read-only).

And this bit would I guess need to be a wrapper around a different nn*
backend, since git would only be the transport, the messages themselves
might be stored as mbox or maildir or something.

This would also compete for the "nngit" name with the
non-existent-yet-historically-inevitable backend that presents a
news-like interface to the branches and commits in a git repository.

Re: How to search on emacs-devel?

[Stefan Monnier]

>> - A bidirectional IMAP <-> Git gateway.
> But this bit would live on a server someplace, not locally?

Right.

>> - an nngit backend (not just read-only).
> And this bit would I guess need to be a wrapper around a different nn*
> backend, since git would only be the transport, the messages themselves
> might be stored as mbox or maildir or something.

I think it'd need its own file layout because we'd want to make sure
Git's conflict resolution does "the right thing".

> This would also compete for the "nngit" name with the
> non-existent-yet-historically-inevitable backend that presents
> a news-like interface to the branches and commits in a git repository.

Yes, it shouldn't be called "nngit".  Instead the file-layout system
would have its own name, like `MailGit`, `Gail`, `GitBox`, or whoknowswhat.


        Stefan

Re: How to search on emacs-devel?

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> - A bidirectional IMAP <-> Git gateway.
>> But this bit would live on a server someplace, not locally?
>
> Right.
>
>>> - an nngit backend (not just read-only).
>> And this bit would I guess need to be a wrapper around a different nn*
>> backend, since git would only be the transport, the messages themselves
>> might be stored as mbox or maildir or something.
>
> I think it'd need its own file layout because we'd want to make sure
> Git's conflict resolution does "the right thing".
>
>> This would also compete for the "nngit" name with the
>> non-existent-yet-historically-inevitable backend that presents
>> a news-like interface to the branches and commits in a git repository.
>
> Yes, it shouldn't be called "nngit".  Instead the file-layout system
> would have its own name, like `MailGit`, `Gail`, `GitBox`, or whoknowswhat.

Gotcha. I was thinking this would be a git interface to whatever mail
store happened to be out there in the wild, but you're describing
something where it's the IMAP server that interfaces with "the wild",
and git is just the internal store? I guess I don't see a major
advantage to just using IMAP directly...

Re: How to search on emacs-devel?

[Stefan Monnier]

> Gotcha. I was thinking this would be a git interface to whatever mail
> store happened to be out there in the wild, but you're describing
> something where it's the IMAP server that interfaces with "the wild",
> and git is just the internal store? I guess I don't see a major
> advantage to just using IMAP directly...

The idea is that the use of IMAP would be a transient artifact and
ditched at some point when the MTA can deliver directly to your
MailGit mailbox.

Then instead of fetching mail via IMAP and being f**ed when you're
offline, you "git pull" to quickly and efficiently get a complete local
copy of your mailbox (and `git push` to send back up the information
about which message you've read/deleted, the tags you've added, or
whether you moved some messages to other folders, ...).

There are many tricky parts, tho:
- Find a good layout to avoid conflicts during `git merge`.
- Figure out a way to let Git forget old mail you really want deleted.
- Write a bidi-gateway with something like IMAP so you can start using
  it before reaching the brave new world where your favorite email
  provider support MailGit format mailboxes.


        Stefan

Re: How to search on emacs-devel?

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Gotcha. I was thinking this would be a git interface to whatever mail
>> store happened to be out there in the wild, but you're describing
>> something where it's the IMAP server that interfaces with "the wild",
>> and git is just the internal store? I guess I don't see a major
>> advantage to just using IMAP directly...
>
> The idea is that the use of IMAP would be a transient artifact and
> ditched at some point when the MTA can deliver directly to your
> MailGit mailbox.
>
> Then instead of fetching mail via IMAP and being f**ed when you're
> offline, you "git pull" to quickly and efficiently get a complete local
> copy of your mailbox (and `git push` to send back up the information
> about which message you've read/deleted, the tags you've added, or
> whether you moved some messages to other folders, ...).
>
> There are many tricky parts, tho:
> - Find a good layout to avoid conflicts during `git merge`.
> - Figure out a way to let Git forget old mail you really want deleted.
> - Write a bidi-gateway with something like IMAP so you can start using
>   it before reaching the brave new world where your favorite email
>   provider support MailGit format mailboxes.

Okay! Makes sense, and there are some interesting things to think about
there. MailGit might sit nicely next to BuGit in the Monnierware
multiverse :) I'd be happy to spitball this and contribute some code as
things progress.

Re: How to search on emacs-devel?

[Kyle Meyer]

Eric Abrahamsen writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Andreas Schwab <schwab@linux-m68k.org> writes:
>>
>>> public-inbox is git under the hood (see
>>> https://yhetil.org/emacs-devel/_/text/mirror/).
>>
>> Maybe local indexing with notmuch or what have you is still the easiest
>> solution, to.

For those that want to go this route, the main thing is getting the
Git-stored messages [1] into a format that the indexer understands.
There are some existing tools (not part of public-inbox) that can
convert a public-inbox archive into a Maildir.  l2md [2] and impibe [3]
are the ones listed at
<https://public-inbox.org/public-inbox.git/plain/Documentation/clients.txt>,
though I haven't used either of them myself.  (There's also
public-inbox's lei, which I'll get to below.)

In the case of Notmuch, the idea of making the Maildir-based Notmuch
understand public-inbox archives was mentioned briefly on the Notmuch
mailing list, but that hasn't gone anywhere yet:

  https://yhetil.org/notmuch/20210501051612.GB6409@dcvr/


[1] Documentation on public-inbox's archive format:
    https://public-inbox.org/public-inbox.git/plain/Documentation/public-inbox-v2-format.pod
[2] https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git
[3] https://leahneukirchen.org/dotfiles/bin/impibe

> Oh I see, if you have the public-inbox package installed locally, then
> it can be used to index the git repo and provide a search interface.
> Installing public-inbox on Arch has a dependency on approximately 9,000
> perl packages, so I won't be doing that, but it looks like that would be
> the easiest solution.

I suspect you're referring to public-inbox's lei (local email interface)
that was introduced in the most recent release (1.7.0).  One way to use
it is to wire up some number of archives as "externals".  These can be
remote, local, or some combination.

For example, below adds emacs-devel as a local external.  This clones
the corresponding Git repo and indexes it with public-inbox-index.  It
unsurprisingly takes some time (maybe around 20 minutes on my machine).

  $ mkdir -p inboxes
  $ lei add-external --mirror https://yhetil.org/emacs-devel inboxes/emacs-devel

Later, you can update the Git repo and index new messages with

  $ make -C inboxes/emacs-devel update

And here's an example of adding a remote external:

  $ lei add-external https://yhetil.org/emacs-bugs

The lei-q command is the main interface for searching:

  $ lei q f:eric d:2.days.ago.. | wc -l
  8
  $ lei q f:eric d:2.days.ago.. | head -1
  [{"blob":"91ec6216f8daa3b731c832f5c5de722e891b49b7","dt":"2022-03-09T22:23:54Z","f":[["Eric Abrahamsen" ...

If there are any local externals configured, lei-q won't reach out to
the remote externals unless explicitly requested with --remote:

  $ lei q --remote f:eric d:2.days.ago..  | wc -l
  9

You can also limit a search to only a set of externals:

  $ lei q --only emacs-bugs f:eric d:2.days.ago.. | jq '.[0].s'
  "bug#54324: 29.0.50; Don't error during debbugs-gnu-rescan"

Or specify a one-off remote URL that wasn't added via add-external:

  $ lei q --only https://lore.kernel.org/git userdiff NEAR emacs

All of the above commands show JSON output, but there are several output
formats available, including various mbox flavors and Maildir.

Anyway, I've already written too much, but, for those interested,
https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
along with lei-overview(7) and the other lei- manpages may be worth
checking out.

Re: How to search on emacs-devel?

[Eli Zaretskii]

> From: Kyle Meyer <kyle@kyleam.com>
> Date: Thu, 10 Mar 2022 00:49:31 -0500
> Cc: emacs-devel@gnu.org
> 
> Eric Abrahamsen writes:
> 
> > Eric Abrahamsen <eric@ericabrahamsen.net> writes:
> >
> >> Andreas Schwab <schwab@linux-m68k.org> writes:
> >>
> >>> public-inbox is git under the hood (see
> >>> https://yhetil.org/emacs-devel/_/text/mirror/).
> >>
> >> Maybe local indexing with notmuch or what have you is still the easiest
> >> solution, to.
> 
> For those that want to go this route, the main thing is getting the
> Git-stored messages [1] into a format that the indexer understands.
> There are some existing tools (not part of public-inbox) that can
> convert a public-inbox archive into a Maildir.  l2md [2] and impibe [3]
> are the ones listed at
> <https://public-inbox.org/public-inbox.git/plain/Documentation/clients.txt>,
> though I haven't used either of them myself.  (There's also
> public-inbox's lei, which I'll get to below.)

Instead (or maybe in addition to) of working on solutions that will
help people individually and locally, why not talk to the GNU mailman
admins so that this issue could be solved as part of improving the
search facilities of the mailman archives?  That would benefit all of
us.  Not to mention that "heavy" solutions are much more likely to be
deployed as part of the GNU mailman infrastructure than by individual
users on their local machines.

Thanks.

Re: How to search on emacs-devel?

[Felix Dietrich]

Robert Pluim <rpluim@gmail.com> writes:

> Would it be harder than downloading the emacs-devel archives locally
> and then just G-f'ing them? Works offline too :-)

For the GCC mailing lists they seem to offer rsync access to the
archives [1].  Does this exist for the Emacs mailing lists?


Footnotes:
[1]  https://gcc.gnu.org/rsync.html

-- 
Felix Dietrich

Re: How to search on emacs-devel?

[Felix Dietrich]

Felix Dietrich <felix.dietrich@sperrhaken.name> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> Would it be harder than downloading the emacs-devel archives locally
>> and then just G-f'ing them? Works offline too :-)
>
> For the GCC mailing lists they seem to offer rsync access to the
> archives [1].  Does this exist for the Emacs mailing lists?

Pondering about this a little more, it is probably enough to use a
download tool that allows continuation of downloads.  With the mbox
format mails are simply added to the end of file.  Correct?  So a new
version of an archive can just be downloaded by continuing from a
previous downloaded archive.  With wget something like this, I guess
(untested):

  wget --continue --recursive --level 1 --no-parent \
       --accept-regex '/[0-9]\{4\}-[0-9]\{2\}$' \
       'https://lists.gnu.org/archive/mbox/emacs-devel/'

So, never mind my rsync question if this actually works.

-- 
Felix Dietrich

Re: How to search on emacs-devel?

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Kyle Meyer <kyle@kyleam.com>
>> Date: Thu, 10 Mar 2022 00:49:31 -0500
>> Cc: emacs-devel@gnu.org
>> 
>> Eric Abrahamsen writes:
>> 
>> > Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>> >
>> >> Andreas Schwab <schwab@linux-m68k.org> writes:
>> >>
>> >>> public-inbox is git under the hood (see
>> >>> https://yhetil.org/emacs-devel/_/text/mirror/).
>> >>
>> >> Maybe local indexing with notmuch or what have you is still the easiest
>> >> solution, to.
>> 
>> For those that want to go this route, the main thing is getting the
>> Git-stored messages [1] into a format that the indexer understands.
>> There are some existing tools (not part of public-inbox) that can
>> convert a public-inbox archive into a Maildir.  l2md [2] and impibe [3]
>> are the ones listed at
>> <https://public-inbox.org/public-inbox.git/plain/Documentation/clients.txt>,
>> though I haven't used either of them myself.  (There's also
>> public-inbox's lei, which I'll get to below.)
>
> Instead (or maybe in addition to) of working on solutions that will
> help people individually and locally, why not talk to the GNU mailman
> admins so that this issue could be solved as part of improving the
> search facilities of the mailman archives?  That would benefit all of
> us.  Not to mention that "heavy" solutions are much more likely to be
> deployed as part of the GNU mailman infrastructure than by individual
> users on their local machines.

We should be doing both/everything, I quite agree! To be clear, this
thread got derailed a bit, in the direction of searching archives _from
within Gnus_, and touching on the disappearance of the search API at
Gmane. Those weren't the original questions, nor the most important
issues to be solved.

I guess I always assumed that the Gnu lists were being administered as
best they could, and if problems remained that was because they were
somehow intractable. Who would we talk to about jostling the namazu
installation? Or about maybe using a different indexer (though again, I
assume there were good reasons for choosing namazu)?

Eric

Re: How to search on emacs-devel?

[Eric Abrahamsen]

On 03/10/22 00:49 AM, Kyle Meyer wrote:
> Eric Abrahamsen writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Andreas Schwab <schwab@linux-m68k.org> writes:
>>>
>>>> public-inbox is git under the hood (see
>>>> https://yhetil.org/emacs-devel/_/text/mirror/).
>>>
>>> Maybe local indexing with notmuch or what have you is still the easiest
>>> solution, to.
>
> For those that want to go this route, the main thing is getting the
> Git-stored messages [1] into a format that the indexer understands.
> There are some existing tools (not part of public-inbox) that can
> convert a public-inbox archive into a Maildir.  l2md [2] and impibe [3]
> are the ones listed at
> <https://public-inbox.org/public-inbox.git/plain/Documentation/clients.txt>,
> though I haven't used either of them myself.  (There's also
> public-inbox's lei, which I'll get to below.)
>
> In the case of Notmuch, the idea of making the Maildir-based Notmuch
> understand public-inbox archives was mentioned briefly on the Notmuch
> mailing list, but that hasn't gone anywhere yet:
>
>   https://yhetil.org/notmuch/20210501051612.GB6409@dcvr/

Thanks for all this background! I guess I don't think there's much value
in converting public-inbox formats to something that notmuch can read,
if public-inbox comes with its own search facilities.

>
> [1] Documentation on public-inbox's archive format:
>     https://public-inbox.org/public-inbox.git/plain/Documentation/public-inbox-v2-format.pod
> [2] https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git
> [3] https://leahneukirchen.org/dotfiles/bin/impibe
>
>> Oh I see, if you have the public-inbox package installed locally, then
>> it can be used to index the git repo and provide a search interface.
>> Installing public-inbox on Arch has a dependency on approximately 9,000
>> perl packages, so I won't be doing that, but it looks like that would be
>> the easiest solution.
>
> I suspect you're referring to public-inbox's lei (local email interface)
> that was introduced in the most recent release (1.7.0).  One way to use
> it is to wire up some number of archives as "externals".  These can be
> remote, local, or some combination.
>
> For example, below adds emacs-devel as a local external.  This clones
> the corresponding Git repo and indexes it with public-inbox-index.  It
> unsurprisingly takes some time (maybe around 20 minutes on my machine).
>
>   $ mkdir -p inboxes
>   $ lei add-external --mirror https://yhetil.org/emacs-devel inboxes/emacs-devel
>
> Later, you can update the Git repo and index new messages with
>
>   $ make -C inboxes/emacs-devel update
>
> And here's an example of adding a remote external:
>
>   $ lei add-external https://yhetil.org/emacs-bugs
>
> The lei-q command is the main interface for searching:
>
>   $ lei q f:eric d:2.days.ago.. | wc -l
>   8
>   $ lei q f:eric d:2.days.ago.. | head -1
>   [{"blob":"91ec6216f8daa3b731c832f5c5de722e891b49b7","dt":"2022-03-09T22:23:54Z","f":[["Eric Abrahamsen" ...
>
> If there are any local externals configured, lei-q won't reach out to
> the remote externals unless explicitly requested with --remote:
>
>   $ lei q --remote f:eric d:2.days.ago..  | wc -l
>   9
>
> You can also limit a search to only a set of externals:
>
>   $ lei q --only emacs-bugs f:eric d:2.days.ago.. | jq '.[0].s'
>   "bug#54324: 29.0.50; Don't error during debbugs-gnu-rescan"
>
> Or specify a one-off remote URL that wasn't added via add-external:
>
>   $ lei q --only https://lore.kernel.org/git userdiff NEAR emacs
>
> All of the above commands show JSON output, but there are several output
> formats available, including various mbox flavors and Maildir.
>
> Anyway, I've already written too much, but, for those interested,
> https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> along with lei-overview(7) and the other lei- manpages may be worth
> checking out.

All this is really interesting, thank you. It looks like one tool (or
tool suite) will handle mail transport, storage, and search -- that's
really nice. I'll read the overview, hold my nose and install all the
dependencies, and play around with it.

Thanks again,
Eric

Re: How to search on emacs-devel?

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Thu, 10 Mar 2022 09:43:54 -0800
> 
> I guess I always assumed that the Gnu lists were being administered as
> best they could, and if problems remained that was because they were
> somehow intractable. Who would we talk to about jostling the namazu
> installation? Or about maybe using a different indexer (though again, I
> assume there were good reasons for choosing namazu)?

I'd write to mailman@gnu.org and CC savannah-hackers-public@gnu.org.

Thanks.

Re: How to search on emacs-devel?

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Thu, 10 Mar 2022 09:43:54 -0800
>> 
>> I guess I always assumed that the Gnu lists were being administered as
>> best they could, and if problems remained that was because they were
>> somehow intractable. Who would we talk to about jostling the namazu
>> installation? Or about maybe using a different indexer (though again, I
>> assume there were good reasons for choosing namazu)?
>
> I'd write to mailman@gnu.org and CC savannah-hackers-public@gnu.org.
>
> Thanks.

Done! Thank you.