Bug-reference auto-setup for Rmail (please test)

Bug-reference auto-setup for Rmail (please test)

[Tassilo Horn]

Hi all,

I've just pushed a commit which should make bug-reference auto-setup
work for Rmail messages.  I'm no Rmail user, so I'd appreciate some
testing from actual users.

What you need to do is activate `bug-reference-mode' in Rmail, so
probably:

  (add-hook 'rmail-mode-hook #'bug-reference-mode)

and then it'll try to guess the right `bug-reference-bug-regexp' and
`bug-reference-url-format' based on the List-Id, To, From, and Cc
headers according to the customize option
`bug-reference-setup-from-mail-alist' which should at least have a
suitable default value for (some) emacs-related mailinglists including
emacs-devel.

There is one problem where I need implementation advice.  The
bug-reference auto-setup is a no-op if `bug-reference-bug-regexp' and
`bug-reference-url-format' are already set (see
`bug-reference--run-auto-setup').  So if you view an emacs-devel message
first, it'll set those variables.  When you move on to the next message
(which might be completely unrelated to emacs), the variables are still
set, so no new guesswork will be done.

We'd need to nuke the buffer-local values and re-enable
`bug-reference-mode' on switching messages, e.g.,

  (add-hook 'rmail-show-message-hook #'rmail-bug-reference-setup)

for a yet to be written rmail-bug-reference-setup which is a bit sad
because for all other modes where I've added this auto-setup capability
(Gnus, Mu4e [currently only in my ~/.emacs], rcirc, ERC, VC) the
"normal"

  (add-hook '<mode>-hook #'bug-reference-mode)

will do, that is, for Rmail the user would need to use a special
function rather than just enabling `bug-reference-mode' in the mode of
his choice.  (Of course, this could be documented.)

Alternatively, Rmail itself could check if `bug-reference-mode' is
enabled, nuke the two variables, and call
`bug-reference--run-auto-setup'.

WDYT?

Bye,
Tassilo

Re: Bug-reference auto-setup for Rmail (please test)

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Date: Sun, 16 May 2021 09:23:21 +0200
> 
> What you need to do is activate `bug-reference-mode' in Rmail, so
> probably:
> 
>   (add-hook 'rmail-mode-hook #'bug-reference-mode)
> 
> and then it'll try to guess the right `bug-reference-bug-regexp' and
> `bug-reference-url-format' based on the List-Id, To, From, and Cc
> headers according to the customize option
> `bug-reference-setup-from-mail-alist' which should at least have a
> suitable default value for (some) emacs-related mailinglists including
> emacs-devel.

Thanks, but it is not clear to me what this feature is supposed to do,
and how do I test that.  You say "it will try to guess", but what is
the expression of that guesswork, from the user perspective?

> There is one problem where I need implementation advice.  The
> bug-reference auto-setup is a no-op if `bug-reference-bug-regexp' and
> `bug-reference-url-format' are already set (see
> `bug-reference--run-auto-setup').  So if you view an emacs-devel message
> first, it'll set those variables.  When you move on to the next message
> (which might be completely unrelated to emacs), the variables are still
> set, so no new guesswork will be done.
> 
> We'd need to nuke the buffer-local values and re-enable
> `bug-reference-mode' on switching messages, e.g.,
> 
>   (add-hook 'rmail-show-message-hook #'rmail-bug-reference-setup)
> 
> for a yet to be written rmail-bug-reference-setup which is a bit sad
> because for all other modes where I've added this auto-setup capability
> (Gnus, Mu4e [currently only in my ~/.emacs], rcirc, ERC, VC) the
> "normal"
> 
>   (add-hook '<mode>-hook #'bug-reference-mode)
> 
> will do, that is, for Rmail the user would need to use a special
> function rather than just enabling `bug-reference-mode' in the mode of
> his choice.  (Of course, this could be documented.)
> 
> Alternatively, Rmail itself could check if `bug-reference-mode' is
> enabled, nuke the two variables, and call
> `bug-reference--run-auto-setup'.

How is Rmail different in this regard from other MUA in Emacs? why do
you need something special for Rmail and not for the rest?

Thanks.

Re: Bug-reference auto-setup for Rmail (please test)

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tassilo Horn <tsdh@gnu.org>
>> Date: Sun, 16 May 2021 09:23:21 +0200
>> 
>> What you need to do is activate `bug-reference-mode' in Rmail, so
>> probably:
>> 
>>   (add-hook 'rmail-mode-hook #'bug-reference-mode)
>> 
>> and then it'll try to guess the right `bug-reference-bug-regexp' and
>> `bug-reference-url-format' based on the List-Id, To, From, and Cc
>> headers according to the customize option
>> `bug-reference-setup-from-mail-alist' which should at least have a
>> suitable default value for (some) emacs-related mailinglists including
>> emacs-devel.
>
> Thanks, but it is not clear to me what this feature is supposed to do,
> and how do I test that.  You say "it will try to guess", but what is
> the expression of that guesswork, from the user perspective?

With the default value of `bug-reference-setup-from-mail-alist', the
user-visible effect would be that bug reference (bug#12345) would be
fontified and clickable and browse that bug report on debbugs.gnu.org in
terms of `browse-url'.

>> There is one problem where I need implementation advice.  The
>> bug-reference auto-setup is a no-op if `bug-reference-bug-regexp' and
>> `bug-reference-url-format' are already set (see
>> `bug-reference--run-auto-setup').  So if you view an emacs-devel message
>> first, it'll set those variables.  When you move on to the next message
>> (which might be completely unrelated to emacs), the variables are still
>> set, so no new guesswork will be done.
>> 
>> We'd need to nuke the buffer-local values and re-enable
>> `bug-reference-mode' on switching messages, e.g.,
>> 
>>   (add-hook 'rmail-show-message-hook #'rmail-bug-reference-setup)
>> 
>> for a yet to be written rmail-bug-reference-setup which is a bit sad
>> because for all other modes where I've added this auto-setup capability
>> (Gnus, Mu4e [currently only in my ~/.emacs], rcirc, ERC, VC) the
>> "normal"
>> 
>>   (add-hook '<mode>-hook #'bug-reference-mode)
>> 
>> will do, that is, for Rmail the user would need to use a special
>> function rather than just enabling `bug-reference-mode' in the mode of
>> his choice.  (Of course, this could be documented.)
>> 
>> Alternatively, Rmail itself could check if `bug-reference-mode' is
>> enabled, nuke the two variables, and call
>> `bug-reference--run-auto-setup'.
>
> How is Rmail different in this regard from other MUA in Emacs? why do
> you need something special for Rmail and not for the rest?

Gnus and Mu4e create a fresh buffer with their viewing mode when viewing
another mail just like you'd get another buffer when finding another
file.

Bye,
Tassilo

Re: Bug-reference auto-setup for Rmail (please test)

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 16 May 2021 10:53:55 +0200
> 
> > Thanks, but it is not clear to me what this feature is supposed to do,
> > and how do I test that.  You say "it will try to guess", but what is
> > the expression of that guesswork, from the user perspective?
> 
> With the default value of `bug-reference-setup-from-mail-alist', the
> user-visible effect would be that bug reference (bug#12345) would be
> fontified and clickable and browse that bug report on debbugs.gnu.org in
> terms of `browse-url'.

OK, but what is this supposed to do for other forums, like
emacs-devel? what is the relevant URL there?

> >> We'd need to nuke the buffer-local values and re-enable
> >> `bug-reference-mode' on switching messages, e.g.,
> >> 
> >>   (add-hook 'rmail-show-message-hook #'rmail-bug-reference-setup)
> >> 
> >> for a yet to be written rmail-bug-reference-setup which is a bit sad
> >> because for all other modes where I've added this auto-setup capability
> >> (Gnus, Mu4e [currently only in my ~/.emacs], rcirc, ERC, VC) the
> >> "normal"
> >> 
> >>   (add-hook '<mode>-hook #'bug-reference-mode)
> >> 
> >> will do, that is, for Rmail the user would need to use a special
> >> function rather than just enabling `bug-reference-mode' in the mode of
> >> his choice.  (Of course, this could be documented.)
> >> 
> >> Alternatively, Rmail itself could check if `bug-reference-mode' is
> >> enabled, nuke the two variables, and call
> >> `bug-reference--run-auto-setup'.
> >
> > How is Rmail different in this regard from other MUA in Emacs? why do
> > you need something special for Rmail and not for the rest?
> 
> Gnus and Mu4e create a fresh buffer with their viewing mode when viewing
> another mail just like you'd get another buffer when finding another
> file.

Then yes, this should be set up when showing a message, not when
turning on the Rmail mode.

Thanks.

Re: Bug-reference auto-setup for Rmail (please test)

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> With the default value of `bug-reference-setup-from-mail-alist', the
>> user-visible effect would be that bug reference (bug#12345) would be
>> fontified and clickable and browse that bug report on debbugs.gnu.org
>> in terms of `browse-url'.
>
> OK, but what is this supposed to do for other forums, like
> emacs-devel? what is the relevant URL there?

The default value sets

  bug-reference-bug-regexp = "\\([Bb]ug ?#?\\)\\([0-9]+\\(?:#[0-9]+\\)?\\)"
  bug-reference-url-format = "https://debbugs.gnu.org/%s"

if some considered mail header (To, From, Cc, List-ID) matches something
emacs-related including emacs-devel@gnu.org (or some auctex, org-mode,
tramp) mailing list address or list-id.

For my job, I have

(add-to-list 'bug-reference-setup-from-mail-alist
             `("SHD" ;; all my work groups/maildirs have SHD in their name
               nil   ;; no header matching needed
               ,(concat "\\b\\(\\("
                        (regexp-opt '("ARCA" "BRI" "ECOJ" "KASE" "MOBA"
                                      "PORC" "LOWE"))
                        "-[0-9]\\{1,6\\}\\)\\)\\b")
               "http://srvjira-and.shd.lan:8080/browse/%s"))

which would highlight and clickify bug numbers like ECOJ-18292 and
browse them at our internal JIRA tracker.

>> Gnus and Mu4e create a fresh buffer with their viewing mode when
>> viewing another mail just like you'd get another buffer when finding
>> another file.
>
> Then yes, this should be set up when showing a message, not when
> turning on the Rmail mode.

Right, the question is just what is the best way to do it.  It has been
my design intention that

  1. there is no special setup needed (just enable `bug-reference-mode'
     where you like, and it'll work),

  2. it does never interfere with explicit user settings.  Therefore,
     the auto-setup runs in `bug-reference-mode's :after-hook, so if the
     2 variables are already set by file/dir-local variables or a hook
     function, it won't do anything.

So the alternative of having some special `bug-reference-rmail-setup'
function rmail users can add to `rmail-show-message-hook' would go
against 1 whereas the alternative of adding special handling in rmail
would possibly go against 2 because we couldn't know if bug-regexp &
url-format have been set by the previous auto-setup or if it has been an
explicit user setting, so resetting them to nil could be a violation of
that rule.

I don't know how Rmail users work but guess that some users have one
mbox file per mailinglist and use rmail-mode-hook to activate & setup
bug-reference explitly looking at the name of the mbox file.  In that
case, nuking the values and re-setting would be wrong.

So as it stands, I'd go with the special `bug-reference-rmail-setup'
function and document that in the Rmail info docs.

Bye,
Tassilo

Re: Bug-reference auto-setup for Rmail (please test)

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 16 May 2021 11:58:25 +0200
> 
> > OK, but what is this supposed to do for other forums, like
> > emacs-devel? what is the relevant URL there?
> 
> The default value sets
> 
>   bug-reference-bug-regexp = "\\([Bb]ug ?#?\\)\\([0-9]+\\(?:#[0-9]+\\)?\\)"
>   bug-reference-url-format = "https://debbugs.gnu.org/%s"
> 
> if some considered mail header (To, From, Cc, List-ID) matches something
> emacs-related including emacs-devel@gnu.org (or some auctex, org-mode,
> tramp) mailing list address or list-id.

At least for Emacs, when we mention bug#NNNNN on emacs-devel, we
always mean debbugs, so the URL seems independent of the forum.

> > Then yes, this should be set up when showing a message, not when
> > turning on the Rmail mode.
> 
> Right, the question is just what is the best way to do it.  It has been
> my design intention that
> 
>   1. there is no special setup needed (just enable `bug-reference-mode'
>      where you like, and it'll work),
> 
>   2. it does never interfere with explicit user settings.  Therefore,
>      the auto-setup runs in `bug-reference-mode's :after-hook, so if the
>      2 variables are already set by file/dir-local variables or a hook
>      function, it won't do anything.
> 
> So the alternative of having some special `bug-reference-rmail-setup'
> function rmail users can add to `rmail-show-message-hook' would go
> against 1 whereas the alternative of adding special handling in rmail
> would possibly go against 2 because we couldn't know if bug-regexp &
> url-format have been set by the previous auto-setup or if it has been an
> explicit user setting, so resetting them to nil could be a violation of
> that rule.

We could add an Rmail-specific defcustom in bug-reference.el, which,
if non-nil, will tell Rmail to run bug-reference-rmail-setup when
showing a message.

> I don't know how Rmail users work but guess that some users have one
> mbox file per mailinglist and use rmail-mode-hook to activate & setup
> bug-reference explitly looking at the name of the mbox file.  In that
> case, nuking the values and re-setting would be wrong.

The new defcustom could accept values that are alists, with separate
values for each inbox.

Re: Bug-reference auto-setup for Rmail (please test)

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> > OK, but what is this supposed to do for other forums, like
>> > emacs-devel? what is the relevant URL there?
>> 
>> The default value sets
>> 
>>   bug-reference-bug-regexp = "\\([Bb]ug ?#?\\)\\([0-9]+\\(?:#[0-9]+\\)?\\)"
>>   bug-reference-url-format = "https://debbugs.gnu.org/%s"
>> 
>> if some considered mail header (To, From, Cc, List-ID) matches something
>> emacs-related including emacs-devel@gnu.org (or some auctex, org-mode,
>> tramp) mailing list address or list-id.
>
> At least for Emacs, when we mention bug#NNNNN on emacs-devel, we
> always mean debbugs, so the URL seems independent of the forum.

There are possible many forums mapping to the same URL, i.e.,
emacs-devel, tramp-devel, and auctex-devel all map to debbugs.

> We could add an Rmail-specific defcustom in bug-reference.el, which,
> if non-nil, will tell Rmail to run bug-reference-rmail-setup when
> showing a message.

Well, I've guessed that Rmail is probably not the single one exception
so I went ahead and added a new function which you could

  (add-hook 'rmail-show-message-hook
            #'bug-reference-mode-force-auto-setup)

which should hopefully do what its name and docstring suggest.

It would be awesome if you could test, e.g., by doing the above and

  (add-to-list 'bug-reference-setup-from-mail-alist
             '(nil
               "tsdh@gnu.org"
               "the" ;; just because its surely in every mail
               "http://tsdh.org/%s"))

Then, mails from/to/cc me should have "the" highlighted as bug
references (bug-reference-bug-regexp) and the bug-reference-url-format
should be http://tsdh.org/%s.

When switching to another mail on emacs-devel/debbugs without me, the
url should again be the debbugs one.

Bye,
Tassilo

Re: Bug-reference auto-setup for Rmail (please test)

[Tassilo Horn]

Tassilo Horn <tsdh@gnu.org> writes:

> Well, I've guessed that Rmail is probably not the single one exception
> so I went ahead and added a new function which you could
>
>   (add-hook 'rmail-show-message-hook
>             #'bug-reference-mode-force-auto-setup)
>
> which should hopefully do what its name and docstring suggest.
>
> It would be awesome if you could test, e.g., by doing the above and
>
>   (add-to-list 'bug-reference-setup-from-mail-alist
>              '(nil
>                "tsdh@gnu.org"
>                "the" ;; just because its surely in every mail
>                "http://tsdh.org/%s"))
>
> Then, mails from/to/cc me should have "the" highlighted as bug
> references (bug-reference-bug-regexp) and the bug-reference-url-format
> should be http://tsdh.org/%s.
>
> When switching to another mail on emacs-devel/debbugs without me, the
> url should again be the debbugs one.

Ah, and I've also passed the Rmail buffer's buffer-file-name to the mail
setup function so that it is compared against the GROUP-REGEXP in the
entries of `bug-reference-setup-from-mail-alist' which caters for the
"one mbox per forum" mail organization approach which, I guess, is like
the new alist you've suggested.

Bye,
Tassilo

Re: Bug-reference auto-setup for Rmail (please test)

[Tassilo Horn]

Tassilo Horn <tsdh@gnu.org> writes:

> Well, I've guessed that Rmail is probably not the single one exception
> so I went ahead and added a new function which you could
>
>   (add-hook 'rmail-show-message-hook
>             #'bug-reference-mode-force-auto-setup)
>
> which should hopefully do what its name and docstring suggest.
>
> It would be awesome if you could test, e.g., by doing the above and
>
>   (add-to-list 'bug-reference-setup-from-mail-alist
>              '(nil
>                "tsdh@gnu.org"
>                "the" ;; just because its surely in every mail
>                "http://tsdh.org/%s"))
>
> Then, mails from/to/cc me should have "the" highlighted as bug
> references (bug-reference-bug-regexp) and the bug-reference-url-format
> should be http://tsdh.org/%s.
>
> When switching to another mail on emacs-devel/debbugs without me, the
> url should again be the debbugs one.

Never mind, I've just tested it myself and it works as expected.

So I guess my remaining task is to write up some documentation.  Do you
make it would make sense to have a new Bug Reference node below
"Advanced Features"?  Or maybe better below "Maintaining Large
Programs"?

If that was in place, I'd add a short paragraph about the Rmail setup in
its own info manual and link the main Bug Reference node from there.
Makes sense?

Bye,
Tassilo

Re: Bug-reference auto-setup for Rmail (please test)

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> Date: Sun, 16 May 2021 17:39:47 +0200
> 
> Never mind, I've just tested it myself and it works as expected.

Thanks.

> So I guess my remaining task is to write up some documentation.  Do you
> make it would make sense to have a new Bug Reference node below
> "Advanced Features"?  Or maybe better below "Maintaining Large
> Programs"?

The latter sounds a better idea.

> If that was in place, I'd add a short paragraph about the Rmail setup in
> its own info manual and link the main Bug Reference node from there.
> Makes sense?

Not sure I understand you: Rmail doesn't have its own manual, it is
described in the Emacs user manual.

Re: Bug-reference auto-setup for Rmail (please test)

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> So I guess my remaining task is to write up some documentation.  Do
>> you make it would make sense to have a new Bug Reference node below
>> "Advanced Features"?  Or maybe better below "Maintaining Large
>> Programs"?
>
> The latter sounds a better idea.

Ok, will do.

>> If that was in place, I'd add a short paragraph about the Rmail setup
>> in its own info manual and link the main Bug Reference node from
>> there.  Makes sense?
>
> Not sure I understand you: Rmail doesn't have its own manual, it is
> described in the Emacs user manual.

Yes, I've meant the Rmail node in the Emacs user manual.

Bye,
Tassilo

Re: Bug-reference auto-setup for Rmail (please test)

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Date: Sun, 16 May 2021 09:23:21 +0200
> 
> What you need to do is activate `bug-reference-mode' in Rmail, so
> probably:
> 
>   (add-hook 'rmail-mode-hook #'bug-reference-mode)
> 
> and then it'll try to guess the right `bug-reference-bug-regexp' and
> `bug-reference-url-format' based on the List-Id, To, From, and Cc
> headers according to the customize option
> `bug-reference-setup-from-mail-alist' which should at least have a
> suitable default value for (some) emacs-related mailinglists including
> emacs-devel.

It doesn't seem to work here, or maybe I'm doing something wrong.

I've ran "M-x rmail RET" on some mbox file with a message from
debbugs, then turned on bug-reference-mode there.  The bug references
are indeed displayed as buttons, but pushing the button does nothing,
because

     (overlay-get o 'bug-reference-url)

returns nil.  What am I doing wrong?  This is "emacs -Q", btw.

Re: Bug-reference auto-setup for Rmail (please test)

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> What you need to do is activate `bug-reference-mode' in Rmail, so
>> probably:
>> 
>>   (add-hook 'rmail-mode-hook #'bug-reference-mode)
>> 
>> and then it'll try to guess the right `bug-reference-bug-regexp' and
>> `bug-reference-url-format' based on the List-Id, To, From, and Cc
>> headers according to the customize option
>> `bug-reference-setup-from-mail-alist' which should at least have a
>> suitable default value for (some) emacs-related mailinglists including
>> emacs-devel.
>
> It doesn't seem to work here, or maybe I'm doing something wrong.
>
> I've ran "M-x rmail RET" on some mbox file with a message from
> debbugs, then turned on bug-reference-mode there.  The bug references
> are indeed displayed as buttons,

That's because `bug-reference-bug-regexp' has a global default value
which happens to match "our" bug numbers.

> but pushing the button does nothing, because
>
>      (overlay-get o 'bug-reference-url)
>
> returns nil.  What am I doing wrong?

Most probably nothing.  Is there a bug address like
48439@debbugs.gnu.org in that mail's To, From, Cc?  That should be
matched fine by the default regexp in
`bug-reference-setup-from-mail-alist'.  Could you please verify that
`bug-reference-try-setup-from-rmail' runs when you activate
`bug-reference-mode' in your rmail buffer?  And if it does but still
doesn't set the `bug-reference-url-format', could you share an mbox with
just that message?

Bye,
Tassilo

Re: Bug-reference auto-setup for Rmail (please test)

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 16 May 2021 12:28:11 +0200
> 
> >      (overlay-get o 'bug-reference-url)
> >
> > returns nil.  What am I doing wrong?
> 
> Most probably nothing.  Is there a bug address like
> 48439@debbugs.gnu.org in that mail's To, From, Cc?  That should be
> matched fine by the default regexp in
> `bug-reference-setup-from-mail-alist'.  Could you please verify that
> `bug-reference-try-setup-from-rmail' runs when you activate
> `bug-reference-mode' in your rmail buffer?

It wasn't being run -- because I failed to byte-compile the latest
bug-reference.el, so I was using the stale .elc file.  Sorry for the
noise.  All is okay now.