Replacing Mumi+Debbugs?

[Ricardo Wurmus <rekado@elephly.net>]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

>> I’ll say that many of my gripes with (the GNU instance of) Debbugs are
>> due to the fact that we can’t customize it to better suit our needs — it
>> is a shared resource with a complicated maintenance story.  So all
>> changes went into Mumi as crude workarounds.  I think this is a dead end
>> and we’d be better off leaving the shared GNU instance of Debbugs
>> behind.
>
> I'd be sad to loose at least two good things from Debbugs:
>
> 1. It's hosted by the GNU/FSF for us.  It always work, and the rare
> times it doesn't, the folks in #savannah are hard at work resolving the
> problems. While hosting sourcehut is probably not too difficult, keeping
> it up to date (Go...) and running would be yet another weigh on our
> meager sysadmin team.

Yes, this is a real concern.  Infrastructure at scale is always a little
messier than one would assume.

> 2. Integration with Emacs.  emacs-debbugs is useful.  I think it's the
> only successful thing we have at keeping track of old tickets and
> resuming discussion or acting on these.
>
> I like how clean Mumi looks, compared to most forge issue trackers.  I'm
> not convinced by its search results (perhaps I'd need to get to know
> what Xapian is).

Xapian is a search engine for documents.  It lets us create search
indices over keywords extracted from documents and retrieve matching
documents.  It’s what “mu” uses for its database; that’s the origin of
“mu” in “mumi” as we used to just leave all the search business to “mu”
in early versions.

All the things related to search are here:

    https://git.savannah.gnu.org/cgit/guix/mumi.git/tree/mumi/xapian.scm

“index-files” opens all emails, extracts fields such as author, subject,
status, body (for full text search), etc, and builds the search
index/database.

“search” converts the query string into a query object and requests
matching documents from the database.

We’re constantly updating the index to match the growing collection of
Debbugs emails.

We are testing the basic search interface here:

    https://git.savannah.gnu.org/cgit/guix/mumi.git/tree/tests/xapian.scm

If there are bad results (and mumi should be used for a while longer to
justify changing it) we should create test cases for them, so we get a
chance to update the way we index the documents and create a query.

Going forward we could also think about augmenting the search with text
embeddings, e.g. by computing word vectors over the whole collection of
emails, turning the search query into a word vector, and then search for
the most similar vectors in the collection.  This would give us very
fuzzy matching of synonyms and related terms.  fasttext and faiss can be
used for these purposes.

-- 
Ricardo