Help wanted for mumi (issues.guix.gnu.org)

Help wanted for mumi (issues.guix.gnu.org)

[Ricardo Wurmus]

Hello Guix,

our bug tracker web interface at issues.guix.gnu.org could really
benefit from a more reliable, faster search.

Currently, mumi (the application behind issues.guix.gnu.org) uses a slow
interface to Debbugs, the bug tracker service that runs at
debbugs.gnu.org.  The search isn’t great as it returns duplicates and is
paginated, which makes it unsuitable for processing.  Mumi may need to
further filter the search results by status or activity, or any other
metric that the Debbugs search API doesn’t let us do.

So I decided to switch away from using the Debbugs API and instead
operate on a *local* copy of all messages that reach Debbugs.  Debbugs
operates on email messages, and luckily it allows us to download these
original messages.  Whenever someone visits an issue page, all related
messages are downloaded by mumi, so it amasses a sizeable stash of
emails over time.

Mumi is using a modified version of “mu”, the mail indexer and search
tool, to continuously index the contents of all messages.  (“mu” is
modified only so that the issue number is indexed alongside the message
contents.)

Unfortunately, that’s as far as I got before life intervened.  The next
step is really close, but getting there requires more contiguous
segments of time than I can free at the moment.  We really only need to
do the following things next:

1) keep updating the mu database as new messages are stored
2) using the mu Guile bindings to search messages via mu instead of
using the slow Debbugs API.

While working on 2 we may find that more properties should be stored in
the mu database, and that’s fine.  Our variant of mu is easily patched
to accomodate our needs.

Does anyone here have an interest in playing with and improving mumi?
It’s a very simple code base and it’s very easy to get started.

The code is here:

    https://git.elephly.net/software/mumi.git

--
Ricardo

Re: Help wanted for mumi (issues.guix.gnu.org)

[Ludovic Courtès]

Hi Ricardo!

Ricardo Wurmus <rekado@elephly.net> skribis:

> So I decided to switch away from using the Debbugs API and instead
> operate on a *local* copy of all messages that reach Debbugs.  Debbugs
> operates on email messages, and luckily it allows us to download these
> original messages.  Whenever someone visits an issue page, all related
> messages are downloaded by mumi, so it amasses a sizeable stash of
> emails over time.

That sounds like a great improvement!

> Unfortunately, that’s as far as I got before life intervened.  The next
> step is really close, but getting there requires more contiguous
> segments of time than I can free at the moment.  We really only need to
> do the following things next:
>
> 1) keep updating the mu database as new messages are stored
> 2) using the mu Guile bindings to search messages via mu instead of
> using the slow Debbugs API.
>
> While working on 2 we may find that more properties should be stored in
> the mu database, and that’s fine.  Our variant of mu is easily patched
> to accomodate our needs.
>
> Does anyone here have an interest in playing with and improving mumi?
> It’s a very simple code base and it’s very easy to get started.
>
> The code is here:
>
>     https://git.elephly.net/software/mumi.git

I’m not offering to work on it right now, but it sounds like a nice
rewarding project someone™ should take!  :-)

Thanks for all your work on this, issues.guix.gnu.org has already been
very helpful!

Ludo’.

Re: Help wanted for mumi (issues.guix.gnu.org)

[zimoun]

Hi Ricardo,

Thank you for initiating this project.


On Sat, 19 Oct 2019 at 23:13, Ricardo Wurmus <rekado@elephly.net> wrote:

> our bug tracker web interface at issues.guix.gnu.org could really
> benefit from a more reliable, faster search.

Yes, it already improves the situation.

I feel grumpy when I use debbugs.gnu.org or even emacs-debbugs. It is
so slow and you need Internet connection: query, wait, query again,
... argh! :-)

Operate on a *local* copy is not easy, IMO. Even if the Gnus guru is
doing [1], it needs some tricks [2] or [3].

[1] https://lists.gnu.org/archive/html/emacs-devel/2019-11/msg00663.html
[2] https://lists.gnu.org/archive/html/emacs-devel/2019-11/msg00666.html
[3] https://lists.gnu.org/archive/html/emacs-devel/2019-11/msg00671.html


> So I decided to switch away from using the Debbugs API and instead
> operate on a *local* copy of all messages that reach Debbugs.  Debbugs
> operates on email messages, and luckily it allows us to download these
> original messages.  Whenever someone visits an issue page, all related
> messages are downloaded by mumi, so it amasses a sizeable stash of
> emails over time.
>
> Mumi is using a modified version of “mu”, the mail indexer and search
> tool, to continuously index the contents of all messages.  (“mu” is
> modified only so that the issue number is indexed alongside the message
> contents.)

This should be nice to be able to work search/query offline.

(Even if I am more from the notmuch side. ;-))


> Unfortunately, that’s as far as I got before life intervened.  The next
> step is really close, but getting there requires more contiguous
> segments of time than I can free at the moment.  We really only need to
> do the following things next:
>
> 1) keep updating the mu database as new messages are stored

Cron task every hour is not enough?

> 2) using the mu Guile bindings to search messages via mu instead of
> using the slow Debbugs API.

This should be really nice!

One direction I am more interested is a CLI to work offline, say

  mumi pull
  mumi search

and I am sure that our Emacs magicians could write a nice frontend. ;-)

> Does anyone here have an interest in playing with and improving mumi?
> It’s a very simple code base and it’s very easy to get started.

I am not enough skilled but I am currently using enough the Debbugs to
be bitten. :-)


Cheers,
simon