`mumi send-email' means no more debbugs dance to send multiple patches

`mumi send-email' means no more debbugs dance to send multiple patches

[Arun Isaac]

Hi all,

mumi, the software powering our debbugs frontend at
https://issues.guix.gnu.org/, now also comes with a command-line
interface. At the moment, the new command-line interface allows
searching for issues, and sending patches. Most notably, it frees us
from the debbugs dance required for sending multiple patches!

To try it out, install mumi from the latest guix pull, perhaps using
`guix shell'.

$ guix shell mumi

To search for issues, say all open issues about "zig", run

--8<---------------cut here---------------start------------->8---
$ mumi search zig is:open

#60889 Add zig-build-system
opened on 17 Jan 17:37 Z by Ekaitz Zarraga
#61036 [PATCH 0/3] Update zig to 0.10.1
opened on 24 Jan 09:42 Z by Efraim Flashner
#39136 [PATCH] gnu: services: Add endlessh.
opened on 14 Jan 2020 21:21 by Nicol? Balzarotti
#60424 [PATCH] gnu: Add python-online-judge-tools
opened on 30 Dec 2022 07:03 by gemmaro
#45601 [PATCH 0/6] vlang 0.2 update
opened on  1 Jan 2021 19:23 by Ryan Prior
--8<---------------cut here---------------end--------------->8---

Pick an issue and make it the "current" issue.

--8<---------------cut here---------------start------------->8---
$ mumi current 61036

#61036 [PATCH 0/3] Update zig to 0.10.1
opened on 24 Jan 09:42 Z by Efraim Flashner
--8<---------------cut here---------------end--------------->8---

Once an issue is the current issue, you can easily send patches to it
using

$ mumi send-email foo.patch bar.patch

To open a new issue, run

$ mumi new

and send patches

$ mumi send-email foo.patch bar.patch

Note that, unlike `git send-email', `mumi send-email' works perfectly
well with single and multiple patches alike! We are now free from the
debbugs dance of sending the first patch, waiting for a response from
debbugs and sending the remaining patches. `mumi send-email' automates
this process. It does so by sending the first patch, polling the server
for a response, and then sending the remaining patches. This polling can
unfortunately take a few minutes. So, please be patient.

Also `mumi send-email' frees you from having to remember the correct To
address (either guix-patches@gnu.org or XXXXX@debbugs.gnu.org) to send
to. It figures out the correct To address from the current issue state
and from the .mumi/config file now checked into the guix git
repository.

Feedback welcome! Patches further fleshing out the mumi CLI client are
most welcome!

Cheers!
Arun

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[(]

Arun Isaac <arunisaac@systemreboot.net> writes:
> mumi, the software powering our debbugs frontend at
> https://issues.guix.gnu.org/, now also comes with a command-line
> interface. At the moment, the new command-line interface allows
> searching for issues, and sending patches. Most notably, it frees us
> from the debbugs dance required for sending multiple patches!

<3 <3 <3

I'll definitely have to send a patch to update the manual, once i've
looked at this in more detail!  :D

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

Hi Arun,

Arun Isaac <arunisaac@systemreboot.net> writes:

> Hi all,
>
> mumi, the software powering our debbugs frontend at
> https://issues.guix.gnu.org/, now also comes with a command-line
> interface. At the moment, the new command-line interface allows
> searching for issues, and sending patches. Most notably, it frees us
> from the debbugs dance required for sending multiple patches!
>
> To try it out, install mumi from the latest guix pull, perhaps using
> `guix shell'.

This looks great!  Thank you so much for your work!

I've just tried it out, but unfortunately mumi didn't receive the mail
in time and it just timed out sending the replies :( is there anything
that could be done on that front?  Does mumi use IMAP IDLE?

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Arun Isaac]

Hi Josselin,

> I've just tried it out, but unfortunately mumi didn't receive the mail
> in time and it just timed out sending the replies :( is there anything
> that could be done on that front?  Does mumi use IMAP IDLE?

Ouch! Unfortunately, timeouts can happen. I was just hoping it would be
unlikely. Did you wait out the full 15 retries of `mumi send-email'?

There are three parts to the system---the debbugs server, the mumi
server and the `mumi send-email' client. `mumi send-email' sends emails
to the debbugs server but polls the mumi server to see if its mail has
been received and registered.

The mumi server fetches emails from the debbugs server using a periodic
rsync. This happens once every few minutes (< 5 minutes)[1]. After
sending the first email, `mumi send-email' polls the mumi server once
every minute to see if the email has reached the mumi server. `mumi
send-email' retries this poll for a maximum of 15 times before giving
up.

This is admittedly a somewhat fragile system, but it is the best we can
do considering debbugs' limitations. Given email's asynchronous store
and forward design and all the polling we do, a `mumi send-email'
timeout is possible. Do you think increasing the number of times `mumi
send-email' retries would help? Did the email you sent using `mumi
send-email' finally reach debbugs and the mumi server? How long did it
take? How long do your emails typically take to reach debbugs and the
mumi server?

Thanks for the feedback, and happy hacking! :-)

Regards,
Arun

[1]: I don't know the exact polling interval of this rsync. Maybe
     Ricardo (CCed) can clarify?

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Ricardo Wurmus]

Arun Isaac <arunisaac@systemreboot.net> writes:

> The mumi server fetches emails from the debbugs server using a periodic
> rsync. This happens once every few minutes (< 5 minutes)[1]. After
> sending the first email, `mumi send-email' polls the mumi server once
> every minute to see if the email has reached the mumi server. `mumi
> send-email' retries this poll for a maximum of 15 times before giving
> up.

This could be changed.  We don’t need to wait for the email to arrive on
the mumi server; could we ask debbugs directly for the email?  (If we
can’t ask for the mail we could ask for newest bugs and check if that
contains the email.)

-- 
Ricardo

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Arun Isaac]

> This could be changed.  We don’t need to wait for the email to arrive on
> the mumi server; could we ask debbugs directly for the email?  (If we
> can’t ask for the mail we could ask for newest bugs and check if that
> contains the email.)

`mumi send-email' polls mumi's /msg-id endpoint to find if an email has
reached the mumi server. I am not sure if the debbugs API allows us to
query for a particular email using its message ID. If debbugs does allow
it, we could modify mumi's /msd-id endpoint to call the debbugs API in
addition to checking its own local storage.

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 1146 bytes --]

Hi Arun,

Arun Isaac <arunisaac@systemreboot.net> writes:

> Hi Josselin,
>
> Ouch! Unfortunately, timeouts can happen. I was just hoping it would be
> unlikely. Did you wait out the full 15 retries of `mumi send-email'?

Yes, I did wait until it completely timed out.

> This is admittedly a somewhat fragile system, but it is the best we can
> do considering debbugs' limitations. Given email's asynchronous store
> and forward design and all the polling we do, a `mumi send-email'
> timeout is possible. Do you think increasing the number of times `mumi
> send-email' retries would help? Did the email you sent using `mumi
> send-email' finally reach debbugs and the mumi server? How long did it
> take? How long do your emails typically take to reach debbugs and the
> mumi server?

My mails usually take less that 30 seconds to get a reply from debbugs.
I don't remember how long it took yesterday for the issue to appear on
mumi though, but more than the 15 retries of mumi send-email (I manually
checked).

> Thanks for the feedback, and happy hacking! :-)

Thanks for your tool!

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Arun Isaac]

Hi Josselin,

> My mails usually take less that 30 seconds to get a reply from debbugs.
> I don't remember how long it took yesterday for the issue to appear on
> mumi though, but more than the 15 retries of mumi send-email (I manually
> checked).

Could you link to the specific email message that you tried sending with
`mumi send-email'? Debbugs provides the complete email message in mbox
format. The email headers should have timestamps we can look at to
ascertain how long it took. I am hoping that this was a one-off spurious
delay somewhere in the email system. I would certainly encourage you to
try using `mumi send-email' again to see if the problem persists. If it
does, we should certainly try and do something about it.

Regards,
Arun

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

Hi Arun,

Arun Isaac <arunisaac@systemreboot.net> writes:

> Could you link to the specific email message that you tried sending with
> `mumi send-email'? Debbugs provides the complete email message in mbox
> format. The email headers should have timestamps we can look at to
> ascertain how long it took. I am hoping that this was a one-off spurious
> delay somewhere in the email system. I would certainly encourage you to
> try using `mumi send-email' again to see if the problem persists. If it
> does, we should certainly try and do something about it.

Thanks for looking into it.  I have tried twice now, with the same issue
both times, with [1] and [2].

[1] mid:cover.1682364024.git.dev@jpoiret.xyz
    https://yhetil.org/guix/cover.1682364024.git.dev@jpoiret.xyz/

[2] mid:cover.1682452738.git.dev@jpoiret.xyz
    https://yhetil.org/guix/cover.1682452738.git.dev@jpoiret.xyz/

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Arun Isaac]

Hi Josselin,

> Thanks for looking into it.  I have tried twice now, with the same issue
> both times, with [1] and [2].
>
> [1] mid:cover.1682364024.git.dev@jpoiret.xyz
>     https://yhetil.org/guix/cover.1682364024.git.dev@jpoiret.xyz/
>
> [2] mid:cover.1682452738.git.dev@jpoiret.xyz
>     https://yhetil.org/guix/cover.1682452738.git.dev@jpoiret.xyz/

I have looked at the headers of both these emails. The Received headers
indicate that they have been received by Debbugs in less than a minute
from the time of sending. Maybe, the mumi server took longer to sync up
with Debbugs, and that's why `mumi send-email' failed. But, I am unable
to ascertain this from the email headers. Maybe, the mumi server keeps
logs, but I don't have ssh access to it. So, I am at a loss as to
debugging this. Ideas welcome!

Regards,
Arun

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Andreas Enge]

Hello Arun,

this looks all very nice, thanks a lot!

I have a few "bug reports" about "mumi web".

When I start it, it runs on 0.0.0.0, port 1.2.3.4; should it not choose
a sensible default, such as localhost and 8080? Running
   mumi web --address=localhost --port=8080
complains that it does not know localhost.
When I use 127.0.0.1 instead, it starts.
But it complains about
     1. &xapian-error: "DatabaseOpeningError: Couldn't stat '/var/mumi/db/mumi.xapian' (No such file or directory)"

I wondered if I needed to "mumi fetch" first (in that case, there could
be a more friendly error message).

But "mumi fetch" fails; it complains about a missing file
/var/mumi/data/spool/index.db.realtime
(which may be missing because as non-root I cannot create it).
Could this be moved to .mumi or .cache/mumi?

Anyway, thanks for moving us forward with our tooling, which I think is
currently our biggest problem.

Andreas

Re: mumi web server bugs [was "`mumi send-email' means no more debbugs dance to send multiple patches"]

[Arun Isaac]

Hi Andreas,

> When I start it, it runs on 0.0.0.0, port 1.2.3.4; should it not choose
> a sensible default, such as localhost and 8080?

I agree. A patch setting the defaults to localhost and 8080 is
welcome. Would you like to give it a shot? If not, could you register
these in our bug tracker so that someone else can address these later?

> Running
>    mumi web --address=localhost --port=8080
> complains that it does not know localhost.
> When I use 127.0.0.1 instead, it starts.

Yes, mumi does not know to resolve the name localhost to
127.0.0.1. Again, patches fixing this are welcome.

This problem does pop up anywhere we have guile web server programs. In
tissue[1], one of my guile web server projects, I had to write a lot of
code to teach it to run correctly on IPv4 addresses, IPv6 addresses and
on Unix sockets. It would be practical if Guile (or a Guile library)
provided some help in this regard. Or, am I expecting Scheme to be more
unminimal? ;-)

[1]: https://tissue.systemreboot.net/

> But it complains about
>      1. &xapian-error: "DatabaseOpeningError: Couldn't stat '/var/mumi/db/mumi.xapian' (No such file or directory)"
>
> I wondered if I needed to "mumi fetch" first (in that case, there could
> be a more friendly error message).
>
> But "mumi fetch" fails; it complains about a missing file
> /var/mumi/data/spool/index.db.realtime
> (which may be missing because as non-root I cannot create it).
> Could this be moved to .mumi or .cache/mumi?

When hacking on mumi, I put the data directory in the checkout of my
local mumi git repo itself. To make mumi look for the data directory in
the current directory, you need to set the MUMI_UNINSTALLED environment
variable. See the code about db-dir in mumi/config.scm.in in the mumi
repo. This should of course be documented, perhaps in a HACKING file in
the mumi repo. Patches are welcome, as always!

But, the larger issue is that you actually need the Debbugs data to hack
on mumi. Currently, our mumi deployment is only able to get this data
via rsync from the Debbugs server and because the IP of our mumi server
is explicitly allowed to do so by the Debbugs admins. We should
distribute this data so that people can hack on mumi. But, is it ok to
do so considering that we are talking about raw email messages with
headers and everything? This may be considered private information. I
would think it is ok because Debbugs already exposes raw email messages
via its web interface. But, mumi data is different in that it is in bulk
and constitutes privileged access to the Debbugs server. We should
discuss this question as a community, and decide on the best way
forward. This question is very important to enable more people to hack
on mumi.

> Anyway, thanks for moving us forward with our tooling, which I think is
> currently our biggest problem.

A pleasure, as always! :-)

Cheers!
Arun

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[John Kehayias]

Hi Arun,

On Mon, Apr 24, 2023 at 04:12 PM, Arun Isaac wrote:

> Hi all,
>
> mumi, the software powering our debbugs frontend at
> <https://issues.guix.gnu.org/>, now also comes with a command-line
> interface. At the moment, the new command-line interface allows
> searching for issues, and sending patches. Most notably, it frees us
> from the debbugs dance required for sending multiple patches!
>

Thanks for this and the detailed instructions!

I just tried it out and it worked smoothly for sending 2 patches to an
existing bug. However, it was sent only directly to the bug address,
not being aware I guess of anyone that had submitted or since been
CC'ed to the thread.

Is there anything we can do here? I learned about the useful "wide
reply" when using debbugs in Emacs and presumably mumi can/does know
about all addresses on the bug. So hopefully this wishlist item won't
be too difficult.

Thanks again, will be happy to use this on the 20 patch series I'm
about to send...

John

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Arun Isaac]

Hi John,

> I just tried it out and it worked smoothly for sending 2 patches to an
> existing bug. However, it was sent only directly to the bug address,
> not being aware I guess of anyone that had submitted or since been
> CC'ed to the thread.
>
> Is there anything we can do here? I learned about the useful "wide
> reply" when using debbugs in Emacs and presumably mumi can/does know
> about all addresses on the bug. So hopefully this wishlist item won't
> be too difficult.

This is definitely a nice feature to have. For now, I put in the other
addresses as Cc when creating the patch using `git format-patch'. Should
we build this feature into `mumi send-email' or should we create a
separate `mumi format-patch' command that will create a patch with the
correct Cc and with other necessary options passed to `git
format-patch'? The other necessary options (think options like `-1 -a
--base=auto' as recommended in the Guix manual at "(guix) Sending a
Patch Series") could be configured in .mumi/config allowing different
projects using mumi to recommend different options.

Either way, patches to mumi are always welcome! ;-)

Regards,
Arun

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Maxim Cournoyer]

Hello,

Arun Isaac <arunisaac@systemreboot.net> writes:

> Hi John,
>
>> I just tried it out and it worked smoothly for sending 2 patches to an
>> existing bug. However, it was sent only directly to the bug address,
>> not being aware I guess of anyone that had submitted or since been
>> CC'ed to the thread.
>>
>> Is there anything we can do here? I learned about the useful "wide
>> reply" when using debbugs in Emacs and presumably mumi can/does know
>> about all addresses on the bug. So hopefully this wishlist item won't
>> be too difficult.
>
> This is definitely a nice feature to have. For now, I put in the other
> addresses as Cc when creating the patch using `git format-patch'. Should
> we build this feature into `mumi send-email' or should we create a
> separate `mumi format-patch' command that will create a patch with the
> correct Cc and with other necessary options passed to `git
> format-patch'? The other necessary options (think options like `-1 -a
> --base=auto' as recommended in the Guix manual at "(guix) Sending a
> Patch Series") could be configured in .mumi/config allowing different
> projects using mumi to recommend different options.

All the above could be handled at the level of git directly with the
changes suggested in #58813, so I'd suggest we do this, which benefits
everyone/everything relying on it.

But I think something extra that mumi send-email could do, which git
can't (because it doesn't fetch the existing thread data), would be to
CC all parties that have already commented in the corresponding thread
(if the issue already exists).  I think this is what John was going at.

> Either way, patches to mumi are always welcome! ;-)

Thanks for this nice tool!

-- 
Thanks,
Maxim

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Arun Isaac]

> All the above could be handled at the level of git directly with the
> changes suggested in #58813, so I'd suggest we do this, which benefits
> everyone/everything relying on it.

Ah, I was not aware of #58813 and that this could be done at the level
of git. Good point! And, thanks for your work on #58813!

> But I think something extra that mumi send-email could do, which git
> can't (because it doesn't fetch the existing thread data), would be to
> CC all parties that have already commented in the corresponding thread
> (if the issue already exists).  I think this is what John was going
> at.

Done now! I have sent patches to https://issues.guix.gnu.org/63215 . I
will push in a few days unless there are objections.

Re: `mumi send-email' means no more debbugs dance to send multiple patches

[Maxim Cournoyer]

Hi Arun,

Arun Isaac <arunisaac@systemreboot.net> writes:

>> All the above could be handled at the level of git directly with the
>> changes suggested in #58813, so I'd suggest we do this, which benefits
>> everyone/everything relying on it.
>
> Ah, I was not aware of #58813 and that this could be done at the level
> of git. Good point! And, thanks for your work on #58813!

I've merged #58813 yesterday, so it should already work!

>> But I think something extra that mumi send-email could do, which git
>> can't (because it doesn't fetch the existing thread data), would be to
>> CC all parties that have already commented in the corresponding thread
>> (if the issue already exists).  I think this is what John was going
>> at.
>
> Done now! I have sent patches to https://issues.guix.gnu.org/63215 . I
> will push in a few days unless there are objections.

I'll try taking a look; in any case, thanks for working on these useful
additions!

-- 
Thanks,
Maxim