A few questions about debbugs

A few questions about debbugs

[Björn Bidar]

Hey,

I have a few questions regardings debbugs, to improve my usage with it.
How can I improve the search speed when searching for bugs?
For example I'm trying to use debbugs to search the Emacs bugtracker
for existing bugs before reporting bugs and the search is so slow that
using the website is much faster.

Another thing I wonder is when I try to open a ephemeral bug group to
view a bug after closing the group it does return back to another Gnus
Group buffer I have open instead of the last one I used the command in.
Why is it this way, should debbugs rather use display-buffer here
anyway?

Thank you for reading,

Björn

Re: A few questions about debbugs

[Michael Albinus]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Hey,

Hi Björn,

> I have a few questions regardings debbugs, to improve my usage with it.
> How can I improve the search speed when searching for bugs?
> For example I'm trying to use debbugs to search the Emacs bugtracker
> for existing bugs before reporting bugs and the search is so slow that
> using the website is much faster.

Hmm, it depends on an example. Could you pls show such?

Note, that the website has direct access to the bug database, while the
debbugs package communicates via SOAP with debbugs.gnu.org. There might
be speed regressions due to this. However, for further analysis, I would
like to see your example.

> Another thing I wonder is when I try to open a ephemeral bug group to
> view a bug after closing the group it does return back to another Gnus
> Group buffer I have open instead of the last one I used the command in.
> Why is it this way, should debbugs rather use display-buffer here
> anyway?

This I cannot answer, somebody from the Gnus camp might know
better. Adding Eric in Cc.

> Thank you for reading,
>
> Björn

Best regards, Michael.

Re: A few questions about debbugs

[Björn Bidar]

Guten Morgen :)

Michael Albinus <michael.albinus@gmx.de> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Hey,
>
> Hi Björn,
>
>> I have a few questions regardings debbugs, to improve my usage with it.
>> How can I improve the search speed when searching for bugs?
>> For example I'm trying to use debbugs to search the Emacs bugtracker
>> for existing bugs before reporting bugs and the search is so slow that
>> using the website is much faster.
>
> Hmm, it depends on an example. Could you pls show such?

I reproduced this in a clean Emacs config (with-emacs.sh[1]).
0. Install debbugs if not installed
1. debbugs-gnu-search 

However I can't reproduce this right now strangely, the issue seems to
just have disappeared on my side.
I maybe the issue was just on my side, the search works relative fast.
Fast enough to not interrupt the IRC process running in Emacs.
Thou it would be awesome if the search would asynchronous.

> Note, that the website has direct access to the bug database, while the
> debbugs package communicates via SOAP with debbugs.gnu.org. There might
> be speed regressions due to this. However, for further analysis, I would
> like to see your example.

I was aware this. I read the documentation when I stumbled upon KDE having once
used Debbugs. To bad that Debbugs can't interact with Bugzilla :)

>> Another thing I wonder is when I try to open a ephemeral bug group to
>> view a bug after closing the group it does return back to another Gnus
>> Group buffer I have open instead of the last one I used the command in.
>> Why is it this way, should debbugs rather use display-buffer here
>> anyway?
>
> This I cannot answer, somebody from the Gnus camp might know
> better. Adding Eric in Cc.

Thank you

---
[1] https://raw.githubusercontent.com/alphapapa/with-emacs.sh/master/with-emacs.sh

Re: A few questions about debbugs

[Michael Albinus]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Guten Morgen :)

Guten Morgen :-)

> I maybe the issue was just on my side, the search works relative fast.
> Fast enough to not interrupt the IRC process running in Emacs.
> Thou it would be awesome if the search would asynchronous.

Well, in fact we have already an asynchronous communication via
SOAP. debbugs-search calls soap-invoke, which sends a SOAP request, and
waits for the SOAP result. But we couls use soap-invoke-async instead,
which decouples the two phases. debbugs.el is already prepared for it,
via function debbugs-soap-invoke-async. So we could implement
asynchronous search. Patches welcome!

> To bad that Debbugs can't interact with Bugzilla :)

Anything goes, of course. But I suspect the Bugzilla interface is too
different from the Debbugs interface; an own package accessing Bugzilla
might be better.

A short web search shows, that other people have also interest in
accessing Bugzilla from Emacs. I have no idea, whether there exists a
respective package, 'tho.

> Thank you

Best regards, Michael.

Re: A few questions about debbugs

[Björn Bidar]

Michael Albinus <michael.albinus@gmx.de> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Guten Morgen :)
>
> Guten Morgen :-)
>
>> I maybe the issue was just on my side, the search works relative fast.
>> Fast enough to not interrupt the IRC process running in Emacs.
>> Thou it would be awesome if the search would asynchronous.
>
> Well, in fact we have already an asynchronous communication via
> SOAP. debbugs-search calls soap-invoke, which sends a SOAP request, and
> waits for the SOAP result. But we couls use soap-invoke-async instead,
> which decouples the two phases. debbugs.el is already prepared for it,
> via function debbugs-soap-invoke-async. So we could implement
> asynchronous search. Patches welcome!

Why soap-invoke-async not made use of when it was added?
If adapting debbugs-search to use soap-invoke is not complicated I could
try.
Asynchronous operating/threading was always a sore spot in Emacs,
especially in the packages that are the closest to GNU.

>> To bad that Debbugs can't interact with Bugzilla :)
>
> Anything goes, of course. But I suspect the Bugzilla interface is too
> different from the Debbugs interface; an own package accessing Bugzilla
> might be better.

Bugzilla has a JSON RPC API to communicate with Bugzilla (older,
deprecated) and the newer REST based API's I don't know how similar they
are to a SOAP interface.
How does debbugs leverage Gnus to display bug comments? Debbugs has the
advantage of being closer to email. Although Bugzilla isn't so far away.

> A short web search shows, that other people have also interest in
> accessing Bugzilla from Emacs. I have no idea, whether there exists a
> respective package, 'tho.

There is bug-mode that supports Rally and Bugzilla but it is mostly
unmaintained these days.

The bugzilla part still works although it doesn't look as nice as
Debbugs.

On topic that's slightly off-topic: Why does debbugs-gnu is called
GNU but refers mostly to Emacs? E.g. there's debbugs-gnu-trunk directory
which refers to Emacs. Further the variable looks out of place now that
Emacs uses Git.

Thanks for your help so far.

Re: A few questions about debbugs

[Michael Albinus]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

Hi Björn,

>> Well, in fact we have already an asynchronous communication via
>> SOAP. debbugs-search calls soap-invoke, which sends a SOAP request, and
>> waits for the SOAP result. But we couls use soap-invoke-async instead,
>> which decouples the two phases. debbugs.el is already prepared for it,
>> via function debbugs-soap-invoke-async. So we could implement
>> asynchronous search. Patches welcome!
>
> Why soap-invoke-async not made use of when it was added?  If adapting
> debbugs-search to use soap-invoke is not complicated I could try.
> Asynchronous operating/threading was always a sore spot in Emacs,
> especially in the packages that are the closest to GNU.

Replacing soap-invoke by soap-invoke-async is not a problem. What I
worry about is the UI. If you invoke debbugs-search, and you do
something else in Emacs then, you need a proper way how to present the
search results to the user w/o interrupting the work she is doing
meanwhile.

> How does debbugs leverage Gnus to display bug comments?

That's not done via the SOAP interface. The debbugs.gnu.org server
offers a well known URL how to download the messages related to a bug in
mbox format. See the function debbugs-get-mbox in debbugs.el. And this
mbox is given to Gnus then, for showing an ephemeral message group.

> On topic that's slightly off-topic: Why does debbugs-gnu is called
> GNU but refers mostly to Emacs? E.g. there's debbugs-gnu-trunk directory
> which refers to Emacs. Further the variable looks out of place now that
> Emacs uses Git.

In the beginning was the word ... sorry, wrong movie. In the beginning
was debbugs.el. It is still used as backend package to access any server
running Debian's Debbugs software. So it works with debbugs.gnu.org and
bugs.debian.org, for example. See user option debbugs-servers in
debbugs.el.

Later on, there was the request to have a UI for that package, which is
specialized on the debbugs.gnu.org server. This was implemented in
debbugs-gnu.el, and this is the reasoun of the string "gnu" in this
library. "debbugs-gnu" is just the prefix for all functions and
variables there.

I don't remember why the string "trunk" has been used, and this might
indeed predates the time git has been taken for Emacs development.

> Thanks for your help so far.

Best regards, Michael.

Re: A few questions about debbugs

[Stefan Monnier via Users list for the GNU Emacs text editor]

> Replacing soap-invoke by soap-invoke-async is not a problem. What I
> worry about is the UI. If you invoke debbugs-search, and you do
> something else in Emacs then, you need a proper way how to present the
> search results to the user w/o interrupting the work she is doing
> meanwhile.

Indeed displaying the buffer asynchronously is too disruptive.

The usual way to do that in Emacs is to immediately display the (empty)
result buffer and then maybe emit a `message` when this buffer has
been filled.

>> On topic that's slightly off-topic: Why does debbugs-gnu is called
>> GNU but refers mostly to Emacs?

Maybe to complement Michael's answer: `debbugs.gnu.org` was originally
setup for Emacs's bug tracking needs.  Since then, other GNU packages
have decided to use that server as well, but Emacs is still probably the
main user.


        Stefan

Re: A few questions about debbugs

[Michael Albinus]

Stefan Monnier via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

Hi Stefan,

>> Replacing soap-invoke by soap-invoke-async is not a problem. What I
>> worry about is the UI. If you invoke debbugs-search, and you do
>> something else in Emacs then, you need a proper way how to present the
>> search results to the user w/o interrupting the work she is doing
>> meanwhile.
>
> Indeed displaying the buffer asynchronously is too disruptive.
>
> The usual way to do that in Emacs is to immediately display the (empty)
> result buffer and then maybe emit a `message` when this buffer has
> been filled.

Thanks for the explanation. I'll give it a try for debbugs-{gnu,org}-search.

> Maybe to complement Michael's answer: `debbugs.gnu.org` was originally
> setup for Emacs's bug tracking needs.  Since then, other GNU packages
> have decided to use that server as well, but Emacs is still probably the
> main user.

Yes, but not such dominant as expected:

--8<---------------cut here---------------start------------->8---
(length (debbugs-get-bugs :package "emacs" :archive "both"))
=> 34657

(length (debbugs-get-bugs :package "guix" :package "guix-patches" :archive "both"))
=> 27259
--8<---------------cut here---------------end--------------->8---

>         Stefan

Best regards, Michael.

Re: A few questions about debbugs

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi,

>>> Replacing soap-invoke by soap-invoke-async is not a problem. What I
>>> worry about is the UI. If you invoke debbugs-search, and you do
>>> something else in Emacs then, you need a proper way how to present the
>>> search results to the user w/o interrupting the work she is doing
>>> meanwhile.
>>
>> Indeed displaying the buffer asynchronously is too disruptive.
>>
>> The usual way to do that in Emacs is to immediately display the (empty)
>> result buffer and then maybe emit a `message` when this buffer has
>> been filled.
>
> Thanks for the explanation. I'll give it a try for debbugs-{gnu,org}-search.

I've assembled a proof-of-concept for debbugs-gnu-search, see appended
patch. It is on top of the ELPA git repository, but it might apply also
on the last distributed debbugs release (didn't check). Feedback appreciated.

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 7580 bytes --]

diff --git a/debbugs-gnu.el b/debbugs-gnu.el
index 68599cff7f..5acb10af16 100644
--- a/debbugs-gnu.el
+++ b/debbugs-gnu.el
@@ -457,6 +457,9 @@ a date, value is the cons cell (BEFORE . AFTER).")
 The specification which bugs shall be suppressed is taken from
   `debbugs-gnu-default-suppress-bugs'.")

+(defvar debbugs-gnu-current-buffer nil
+  "The current buffer results are presented in.")
+
 (defvar debbugs-gnu-current-print-function #'tabulated-list-print
   "Which function to apply printing the tabulated list..
 See `debbugs-gnu-package' for an alternative.")
@@ -680,8 +683,12 @@ depend on PHRASE being a string, or nil.  See Info node
      elt))

   ;; Do the search.
-  (debbugs-gnu severities packages archivedp)
-  (message "Search finished"))
+  (make-thread
+   (lambda ()
+     (debbugs-gnu severities packages archivedp)
+     (message
+      (substitute-command-keys
+       "Search finished, visit buffer via \\[debbugs-gnu-show-last-result]")))))

 ;;;###autoload
 (defun debbugs-gnu-patches ()
@@ -855,6 +862,13 @@ This function assumes the variable `user-mail-address' is defined."
   (interactive)
   (apply #'debbugs-gnu-bugs (debbugs-get-bugs :submitter "me" :status "open")))

+;;;###autoload
+(defun debbugs-gnu-show-last-result ()
+  "Switch to buffer with the recent retrieved bugs"
+  (interactive)
+  (when (ignore-errors (get-buffer debbugs-gnu-current-buffer))
+    (pop-to-buffer-same-window debbugs-gnu-current-buffer)))
+
 (defun debbugs-gnu-get-bugs (query)
   "Retrieve bug numbers from debbugs.gnu.org according search criteria."
   (let* ((bugs (assq 'bugs query))
@@ -874,7 +888,18 @@ This function assumes the variable `user-mail-address' is defined."
 	       (if phrase
 		   (cond
 		    ((eq (car elt) 'phrase)
-		     (list (list :phrase (cdr elt))))
+                     (let ((str (cdr elt))
+                           res)
+                       (while (string-match
+                               (rx (1+ space) (group (or "MAX" "SKIP"))
+                                   (1+ space) (group (1+ digit)) eol)
+                               str)
+                         (push (string-to-number (match-string 2 str)) res)
+                         (push
+                          (intern (concat ":" (downcase (match-string 1 str))))
+                          res)
+                         (setq str (replace-match "" nil nil str)))
+		       (list (append (list :phrase str) res))))
 		    ((memq (car elt) '(date @cdate))
 		     (list (list (intern (concat ":" (symbol-name (car elt))))
 				 (cddr elt) (cadr elt)
@@ -918,19 +943,21 @@ This function assumes the variable `user-mail-address' is defined."
 If OFFLINE is non-nil, the query is not sent to the server.  Bugs
 are taken from the cache instead."
   (let* ((inhibit-read-only t)
-	 string
-	 (buffer-name
+	 string)
+    (setq debbugs-gnu-current-buffer
 	  (cond
 	   ((setq string (alist-get 'phrase debbugs-gnu-current-query))
 	    (format "*%S Bugs*" string))
 	   ((setq string (alist-get 'package debbugs-gnu-current-query))
 	    (format "*%s Bugs*" (capitalize string)))
-	   (t "*Bugs*"))))
+	   (t "*Bugs*")))
     ;; The tabulated mode sets several local variables.  We must get
     ;; rid of them.
-    (when (get-buffer buffer-name)
-      (kill-buffer buffer-name))
-    (pop-to-buffer-same-window (get-buffer-create buffer-name))
+    (when (get-buffer debbugs-gnu-current-buffer)
+      (kill-buffer debbugs-gnu-current-buffer))
+    (funcall
+     (if (debbugs-show-progress) #'pop-to-buffer-same-window #'set-buffer)
+     (get-buffer-create debbugs-gnu-current-buffer))
     (debbugs-gnu-mode)

     ;; Print bug reports.
diff --git a/debbugs.el b/debbugs.el
index 627cdf163d..b04256a243 100644
--- a/debbugs.el
+++ b/debbugs.el
@@ -135,6 +135,11 @@ OPERATION-NAME and PARAMETERS are as described in `soap-invoke'."
   "Whether progress report is shown."
   :type 'boolean)

+(defun debbugs-show-progress ()
+  "Whether progress report is shown."
+  (and debbugs-show-progress
+       (or (not main-thread) (eq main-thread (current-thread)))))
+
 (defvar debbugs-progress-reporter nil
  "The progress reporter.
 Don't set this globally, it shall be let-bound.")
@@ -154,7 +159,7 @@ Don't set this globally, it shall be let-bound.")
 (defun debbugs-url-display-message-or-percentage (&rest args)
   "Update progress reporter."
   (ignore-errors
-    (when (and debbugs-show-progress debbugs-progress-reporter)
+    (when (and (debbugs-show-progress) debbugs-progress-reporter)
       ;; The fingerprint of `url-display-percentage' is FMT PERC &REST
       ;; ARGS.  Since Emacs 29.1, this is obsolete and replaced by
       ;; `url-display-message', which should also carry the percentage
@@ -256,10 +261,10 @@ patch:
                      :status \"forwarded\"
                      :severity \"serious\")"
   (let ((debbugs-progress-reporter
-	 (and debbugs-show-progress
+	 (and (debbugs-show-progress)
 	      (make-progress-reporter "Get bug numbers..." 0 100)))
 	debbugs-progress-reporter-buffers url-show-status vec kw key val)
-    (when debbugs-show-progress
+    (when (debbugs-show-progress)
       (add-function
        :override
        (symbol-function debbugs-url-display-message-or-percentage-function)
@@ -305,7 +310,7 @@ patch:
       (error "Unknown key: %s" (car query)))
     (prog1
 	(sort (car (soap-invoke debbugs-wsdl debbugs-port "get_bugs" vec)) #'<)
-      (when debbugs-show-progress
+      (when (debbugs-show-progress)
 	(remove-function
 	 (symbol-function debbugs-url-display-message-or-percentage-function)
 	 "debbugs-url-display-message-or-percentage")
@@ -486,10 +491,10 @@ Example:
       ;; Retrieve bugs asynchronously.
       (let ((bug-ids bug-numbers)
 	    (debbugs-progress-reporter
-	     (and debbugs-show-progress
+	     (and (debbugs-show-progress)
 		  (make-progress-reporter "Get bug information..." 0 100)))
 	    debbugs-progress-reporter-buffers url-show-status results res)
-	(when debbugs-show-progress
+	(when (debbugs-show-progress)
 	  (add-function
 	   :override
            (symbol-function debbugs-url-display-message-or-percentage-function)
@@ -513,7 +518,7 @@ Example:
 		(last bug-ids (- (length bug-ids)
 				 debbugs-max-hits-per-request))))

-	(when debbugs-show-progress
+	(when (debbugs-show-progress)
 	  (dolist (res results)
 	    (push `(,res . 0) debbugs-progress-reporter-buffers)))

@@ -521,11 +526,11 @@ Example:
 	  (setq res (nth (random (length results)) results))
 	  (if (process-live-p (get-buffer-process res))
 	      (accept-process-output (get-buffer-process res))
-	    (when debbugs-show-progress
+	    (when (debbugs-show-progress)
 	      (setcdr (assq res debbugs-progress-reporter-buffers) 100))
             (setq results (delq res results))))

-	(when debbugs-show-progress
+	(when (debbugs-show-progress)
 	  (remove-function
 	   (symbol-function debbugs-url-display-message-or-percentage-function)
 	   "debbugs-url-display-message-or-percentage")
@@ -803,7 +808,7 @@ Examples:

   (let ((phrase (assq :phrase query))
 	(debbugs-create-progress-reporter
-	 (and debbugs-show-progress (null debbugs-progress-reporter)))
+	 (and (debbugs-show-progress) (null debbugs-progress-reporter)))
 	(debbugs-progress-reporter debbugs-progress-reporter)
 	debbugs-progress-reporter-buffers url-show-status args result)
     (when debbugs-create-progress-reporter

Re: A few questions about debbugs

[Björn Bidar]

Michael Albinus <michael.albinus@gmx.de> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
> Hi,
>
>>>> Replacing soap-invoke by soap-invoke-async is not a problem. What I
>>>> worry about is the UI. If you invoke debbugs-search, and you do
>>>> something else in Emacs then, you need a proper way how to present the
>>>> search results to the user w/o interrupting the work she is doing
>>>> meanwhile.
>>>
>>> Indeed displaying the buffer asynchronously is too disruptive.
>>>
>>> The usual way to do that in Emacs is to immediately display the (empty)
>>> result buffer and then maybe emit a `message` when this buffer has
>>> been filled.
>>
>> Thanks for the explanation. I'll give it a try for debbugs-{gnu,org}-search.
>
> I've assembled a proof-of-concept for debbugs-gnu-search, see appended
> patch. It is on top of the ELPA git repository, but it might apply also
> on the last distributed debbugs release (didn't check). Feedback appreciated.

Could you next time create a branch with the patch in debbugs git so
it's easier to test the patch if already using debbugs from git?

To bad Debbugs-gnus-apply-patch only works for Emacs :D
Applying a patch with Gnus was harder than it should be, I have to
somehow integrate diff-mode and then magit-apply-patch better in Gnus.
But that's OT anyway.

> Best regards, Michael.
>
> diff --git a/debbugs-gnu.el b/debbugs-gnu.el
> index 68599cff7f..5acb10af16 100644
> --- a/debbugs-gnu.el
> +++ b/debbugs-gnu.el
> @@ -457,6 +457,9 @@ a date, value is the cons cell (BEFORE . AFTER).")
>  The specification which bugs shall be suppressed is taken from
>    `debbugs-gnu-default-suppress-bugs'.")
>
> +(defvar debbugs-gnu-current-buffer nil
> +  "The current buffer results are presented in.")
> +
>  (defvar debbugs-gnu-current-print-function #'tabulated-list-print
>    "Which function to apply printing the tabulated list..
>  See `debbugs-gnu-package' for an alternative.")
> @@ -680,8 +683,12 @@ depend on PHRASE being a string, or nil.  See Info node
>       elt))
>
>    ;; Do the search.
> -  (debbugs-gnu severities packages archivedp)
> -  (message "Search finished"))
> +  (make-thread
> +   (lambda ()
> +     (debbugs-gnu severities packages archivedp)
> +     (message
> +      (substitute-command-keys
> +       "Search finished, visit buffer via \\[debbugs-gnu-show-last-result]")))))

I think it would also be an option to have a flag to open the search
results by default after they are found. For searches that should not
take long this is the expected behavior especially for new users.

>  ;;;###autoload
>  (defun debbugs-gnu-patches ()
> @@ -855,6 +862,13 @@ This function assumes the variable `user-mail-address' is defined."
>    (interactive)
>    (apply #'debbugs-gnu-bugs (debbugs-get-bugs :submitter "me" :status "open")))
>
> +;;;###autoload
> +(defun debbugs-gnu-show-last-result ()
> +  "Switch to buffer with the recent retrieved bugs"
> +  (interactive)
> +  (when (ignore-errors (get-buffer debbugs-gnu-current-buffer))
> +    (pop-to-buffer-same-window debbugs-gnu-current-buffer)))

What's the reason for pop-to-buffer-same-window, shouldn't the user be
able to choose where the search results are opened? Personally I would
use display-buffer and than customize if the buffer should be opened in
the same buffer.

>  (defun debbugs-gnu-get-bugs (query)
>    "Retrieve bug numbers from debbugs.gnu.org according search criteria."
>    (let* ((bugs (assq 'bugs query))
> @@ -874,7 +888,18 @@ This function assumes the variable `user-mail-address' is defined."
>  	       (if phrase
>  		   (cond
>  		    ((eq (car elt) 'phrase)
> -		     (list (list :phrase (cdr elt))))
> +                     (let ((str (cdr elt))
> +                           res)
> +                       (while (string-match
> +                               (rx (1+ space) (group (or "MAX" "SKIP"))
> +                                   (1+ space) (group (1+ digit)) eol)
> +                               str)
> +                         (push (string-to-number (match-string 2 str)) res)
> +                         (push
> +                          (intern (concat ":" (downcase (match-string 1 str))))
> +                          res)
> +                         (setq str (replace-match "" nil nil str)))
> +		       (list (append (list :phrase str) res))))
>  		    ((memq (car elt) '(date @cdate))
>  		     (list (list (intern (concat ":" (symbol-name (car elt))))
>  				 (cddr elt) (cadr elt)
> @@ -918,19 +943,21 @@ This function assumes the variable `user-mail-address' is defined."
>  If OFFLINE is non-nil, the query is not sent to the server.  Bugs
>  are taken from the cache instead."
>    (let* ((inhibit-read-only t)
> -	 string
> -	 (buffer-name
> +	 string)
> +    (setq debbugs-gnu-current-buffer
>  	  (cond
>  	   ((setq string (alist-get 'phrase debbugs-gnu-current-query))
>  	    (format "*%S Bugs*" string))
>  	   ((setq string (alist-get 'package debbugs-gnu-current-query))
>  	    (format "*%s Bugs*" (capitalize string)))
> -	   (t "*Bugs*"))))
> +	   (t "*Bugs*")))
>      ;; The tabulated mode sets several local variables.  We must get
>      ;; rid of them.
> -    (when (get-buffer buffer-name)
> -      (kill-buffer buffer-name))
> -    (pop-to-buffer-same-window (get-buffer-create buffer-name))
> +    (when (get-buffer debbugs-gnu-current-buffer)
> +      (kill-buffer debbugs-gnu-current-buffer))
> +    (funcall
> +     (if (debbugs-show-progress) #'pop-to-buffer-same-window #'set-buffer)
> +     (get-buffer-create debbugs-gnu-current-buffer))
>      (debbugs-gnu-mode)
>
>      ;; Print bug reports.
> diff --git a/debbugs.el b/debbugs.el
> index 627cdf163d..b04256a243 100644
> --- a/debbugs.el
> +++ b/debbugs.el
> @@ -135,6 +135,11 @@ OPERATION-NAME and PARAMETERS are as described in `soap-invoke'."
>    "Whether progress report is shown."
>    :type 'boolean)
>
> +(defun debbugs-show-progress ()
> +  "Whether progress report is shown."
> +  (and debbugs-show-progress
> +       (or (not main-thread) (eq main-thread (current-thread)))))
> +
>  (defvar debbugs-progress-reporter nil
>   "The progress reporter.
>  Don't set this globally, it shall be let-bound.")
> @@ -154,7 +159,7 @@ Don't set this globally, it shall be let-bound.")
>  (defun debbugs-url-display-message-or-percentage (&rest args)
>    "Update progress reporter."
>    (ignore-errors
> -    (when (and debbugs-show-progress debbugs-progress-reporter)
> +    (when (and (debbugs-show-progress) debbugs-progress-reporter)
>        ;; The fingerprint of `url-display-percentage' is FMT PERC &REST
>        ;; ARGS.  Since Emacs 29.1, this is obsolete and replaced by
>        ;; `url-display-message', which should also carry the percentage
> @@ -256,10 +261,10 @@ patch:
>                       :status \"forwarded\"
>                       :severity \"serious\")"
>    (let ((debbugs-progress-reporter
> -	 (and debbugs-show-progress
> +	 (and (debbugs-show-progress)
>  	      (make-progress-reporter "Get bug numbers..." 0 100)))
>  	debbugs-progress-reporter-buffers url-show-status vec kw key val)
> -    (when debbugs-show-progress
> +    (when (debbugs-show-progress)
>        (add-function
>         :override
>         (symbol-function debbugs-url-display-message-or-percentage-function)
> @@ -305,7 +310,7 @@ patch:
>        (error "Unknown key: %s" (car query)))
>      (prog1
>  	(sort (car (soap-invoke debbugs-wsdl debbugs-port "get_bugs" vec)) #'<)
> -      (when debbugs-show-progress
> +      (when (debbugs-show-progress)
>  	(remove-function
>  	 (symbol-function debbugs-url-display-message-or-percentage-function)
>  	 "debbugs-url-display-message-or-percentage")
> @@ -486,10 +491,10 @@ Example:
>        ;; Retrieve bugs asynchronously.
>        (let ((bug-ids bug-numbers)
>  	    (debbugs-progress-reporter
> -	     (and debbugs-show-progress
> +	     (and (debbugs-show-progress)
>  		  (make-progress-reporter "Get bug information..." 0 100)))
>  	    debbugs-progress-reporter-buffers url-show-status results res)
> -	(when debbugs-show-progress
> +	(when (debbugs-show-progress)
>  	  (add-function
>  	   :override
>             (symbol-function debbugs-url-display-message-or-percentage-function)
> @@ -513,7 +518,7 @@ Example:
>  		(last bug-ids (- (length bug-ids)
>  				 debbugs-max-hits-per-request))))
>
> -	(when debbugs-show-progress
> +	(when (debbugs-show-progress)
>  	  (dolist (res results)
>  	    (push `(,res . 0) debbugs-progress-reporter-buffers)))
>
> @@ -521,11 +526,11 @@ Example:
>  	  (setq res (nth (random (length results)) results))
>  	  (if (process-live-p (get-buffer-process res))
>  	      (accept-process-output (get-buffer-process res))
> -	    (when debbugs-show-progress
> +	    (when (debbugs-show-progress)
>  	      (setcdr (assq res debbugs-progress-reporter-buffers) 100))
>              (setq results (delq res results))))
>
> -	(when debbugs-show-progress
> +	(when (debbugs-show-progress)
>  	  (remove-function
>  	   (symbol-function debbugs-url-display-message-or-percentage-function)
>  	   "debbugs-url-display-message-or-percentage")
> @@ -803,7 +808,7 @@ Examples:
>
>    (let ((phrase (assq :phrase query))
>  	(debbugs-create-progress-reporter
> -	 (and debbugs-show-progress (null debbugs-progress-reporter)))
> +	 (and (debbugs-show-progress) (null debbugs-progress-reporter)))
>  	(debbugs-progress-reporter debbugs-progress-reporter)
>  	debbugs-progress-reporter-buffers url-show-status args result)
>      (when debbugs-create-progress-reporter

Re: A few questions about debbugs

[Michael Albinus]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

Hi Björn,

>> I've assembled a proof-of-concept for debbugs-gnu-search, see appended
>> patch. It is on top of the ELPA git repository, but it might apply also
>> on the last distributed debbugs release (didn't check). Feedback appreciated.
>
> Could you next time create a branch with the patch in debbugs git so
> it's easier to test the patch if already using debbugs from git?

Debbugs lives in the elpa git, as branch. Adding another branch to a
branch seems to be overkill to me.

I've pushed everything to the elps repo (this is much more but the last
patch I've sent, covering almost all debbugs-gnu-* and debbugs-org-*
commands). You can install this WIP as special development elpa
package. Add the following:

--8<---------------cut here---------------start------------->8---
(add-to-list 'package-archives
             '("devel" . "https://elpa.gnu.org/devel/"))
--8<---------------cut here---------------end--------------->8---

After refreshing your package list, you wil see package debbugs
0.40.0.20240812.95045. Install it, and play with it :-)

> To bad Debbugs-gnus-apply-patch only works for Emacs :D
> Applying a patch with Gnus was harder than it should be, I have to
> somehow integrate diff-mode and then magit-apply-patch better in Gnus.
> But that's OT anyway.

You don't need all of this. Emacs has built-in patch support, for
example via Menu Bar -> Tools -> Apply Patch -> To a File

>>    ;; Do the search.
>> -  (debbugs-gnu severities packages archivedp)
>> -  (message "Search finished"))
>> +  (make-thread
>> +   (lambda ()
>> +     (debbugs-gnu severities packages archivedp)
>> +     (message
>> +      (substitute-command-keys
>> +       "Search finished, visit buffer via \\[debbugs-gnu-show-last-result]")))))
>
> I think it would also be an option to have a flag to open the search
> results by default after they are found. For searches that should not
> take long this is the expected behavior especially for new users.

But then we have the problem of interrupting the user, again. A user
doesn't know in advance, how long a search will run.

>> +;;;###autoload
>> +(defun debbugs-gnu-show-last-result ()
>> +  "Switch to buffer with the recent retrieved bugs"
>> +  (interactive)
>> +  (when (ignore-errors (get-buffer debbugs-gnu-current-buffer))
>> +    (pop-to-buffer-same-window debbugs-gnu-current-buffer)))
>
> What's the reason for pop-to-buffer-same-window, shouldn't the user be
> able to choose where the search results are opened? Personally I would
> use display-buffer and than customize if the buffer should be opened in
> the same buffer.

Well, this was discussed in bug#36917. Since I'm not familiar with the
display-buffer machinery, I've simply followed the recommendation. They
say that one could use display-buffer-alist for configuration then.

I've also updated the debbugs-ug manual. There is a new section, saying

--8<---------------cut here---------------start------------->8---
1.1 Asynchronous Work
=====================

The access to the GNU Debbugs server happens via a SOAP interface.
This can be slow, especially, when a large number of bugs must be
retrieved, or an extensive search is applied.  If Emacs is compiled
with threading support, this work is performed by a thread.  During
the bug retrieval, it is still possible to continue the work with
Emacs then.

   After the bug retrieval is finished, a respective message will
appear in the minibuffer.

 -- Command: debbugs-gnu-show-last-result

     This command switches the current buffer to the recent buffer
     containing bug reports.  It is useful to bind this command to a
     function key in your ‘.emacs’ file, like

          (keymap-global-set "<f5>" 'debbugs-gnu-show-last-result)

 -- User Option: debbugs-gnu-use-threads

     The user option ‘debbugs-gnu-use-threads’, when set to ‘nil’,
     disables threaded bug retrieval, and synchronous bug retrieval is
     performed.  Threaded bug retrieval is also suppressed if the
     number of bugs to be retrieved is known in advance, and it is
     smaller than the user option ‘debbugs-gnu-use-threads-lower-limit’.
--8<---------------cut here---------------end--------------->8---

Feedback welcome!

Best regards, Michael.

Re: A few questions about debbugs

[Björn Bidar]

Michael Albinus <michael.albinus@gmx.de> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
> Hi Björn,
>
>>> Well, in fact we have already an asynchronous communication via
>>> SOAP. debbugs-search calls soap-invoke, which sends a SOAP request, and
>>> waits for the SOAP result. But we couls use soap-invoke-async instead,
>>> which decouples the two phases. debbugs.el is already prepared for it,
>>> via function debbugs-soap-invoke-async. So we could implement
>>> asynchronous search. Patches welcome!
>>
>> Why soap-invoke-async not made use of when it was added?  If adapting
>> debbugs-search to use soap-invoke is not complicated I could try.
>> Asynchronous operating/threading was always a sore spot in Emacs,
>> especially in the packages that are the closest to GNU.
>
> Replacing soap-invoke by soap-invoke-async is not a problem. What I
> worry about is the UI. If you invoke debbugs-search, and you do
> something else in Emacs then, you need a proper way how to present the
> search results to the user w/o interrupting the work she is doing
> meanwhile.

How would you not interrupt the word the user is doing? Shouldn't that
be the point? An option would be to use dispaly buffer with an option to
not change the cursor focus and spawn the window where the command was
called if possible.

>> How does debbugs leverage Gnus to display bug comments?
>
> That's not done via the SOAP interface. The debbugs.gnu.org server
> offers a well known URL how to download the messages related to a bug in
> mbox format. See the function debbugs-get-mbox in debbugs.el. And this
> mbox is given to Gnus then, for showing an ephemeral message group.

Is this specific to GNU debbugs server or does it work too if I would
like to do the same with e.g. the Debian server?

The way you implemented this feature could also be a way to handle
mainlinglist archives in Gnus better if some such as those using
mailmain would provide a well known address.
For others there's at least public-inbox which works after each
mailing-list has enabled it. But that's OT.

>> On topic that's slightly off-topic: Why does debbugs-gnu is called
>> GNU but refers mostly to Emacs? E.g. there's debbugs-gnu-trunk directory
>> which refers to Emacs. Further the variable looks out of place now that
>> Emacs uses Git.
>
> In the beginning was the word ... sorry, wrong movie. In the beginning
> was debbugs.el. It is still used as backend package to access any server
> running Debian's Debbugs software. So it works with debbugs.gnu.org and
> bugs.debian.org, for example. See user option debbugs-servers in
> debbugs.el.
>
> Later on, there was the request to have a UI for that package, which is
> specialized on the debbugs.gnu.org server. This was implemented in
> debbugs-gnu.el, and this is the reasoun of the string "gnu" in this
> library. "debbugs-gnu" is just the prefix for all functions and
> variables there.

OK thanks for explaining. I think for clarification all the Emacs
specific functions should have a debbugs-gnu-emacs- prefix or something
a like to indicate their intention.

> I don't remember why the string "trunk" has been used, and this might
> indeed predates the time git has been taken for Emacs development.

Is the approach to have more than one source tree? Debbugs mentions
a trunk and stable tree for a git based workflow a stable and
master/main branch handling would be more helpful.

>> Thanks for your help so far.
>
> Best regards, Michael.

Re: A few questions about debbugs

[Michael Albinus]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

Hi Björn,

>> Replacing soap-invoke by soap-invoke-async is not a problem. What I
>> worry about is the UI. If you invoke debbugs-search, and you do
>> something else in Emacs then, you need a proper way how to present the
>> search results to the user w/o interrupting the work she is doing
>> meanwhile.
>
> How would you not interrupt the word the user is doing? Shouldn't that
> be the point? An option would be to use dispaly buffer with an option to
> not change the cursor focus and spawn the window where the command was
> called if possible.

See the other message from Stefan Monnier.

>> That's not done via the SOAP interface. The debbugs.gnu.org server
>> offers a well known URL how to download the messages related to a bug in
>> mbox format. See the function debbugs-get-mbox in debbugs.el. And this
>> mbox is given to Gnus then, for showing an ephemeral message group.
>
> Is this specific to GNU debbugs server or does it work too if I would
> like to do the same with e.g. the Debian server?

Sure, you can do it with the Debian BTS as well. Given there is the bug
<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=123456>, you can
download the respective mbox via <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=123456;mbox=yes>.

> The way you implemented this feature could also be a way to handle
> mainlinglist archives in Gnus better if some such as those using
> mailmain would provide a well known address.
> For others there's at least public-inbox which works after each
> mailing-list has enabled it. But that's OT.

I believe there are packages which use it. Gnus has implemented the
handling of such mbox files under the name "ephemeral group". Check the
Gnus manual for this.

RMAIL offers a similar technique, used in debbugs-gnu.el as well, but
I'm not familiar with the details.

>> Later on, there was the request to have a UI for that package, which is
>> specialized on the debbugs.gnu.org server. This was implemented in
>> debbugs-gnu.el, and this is the reasoun of the string "gnu" in this
>> library. "debbugs-gnu" is just the prefix for all functions and
>> variables there.
>
> OK thanks for explaining. I think for clarification all the Emacs
> specific functions should have a debbugs-gnu-emacs- prefix or something
> a like to indicate their intention.

I don't know what you mean with "Emacs specific functions". If you mean
"functions to be called from Emacs" - all of them are such functions.

If you mean "functions specific for the Emacs project on
debbugs.gnu.org" - these are the functions which handle the git repo. I
don't know whether they could be used for other projects / git
repositories as well; but there haven't been a request so far.

Note, that the Guis project uses its own configuration, collected in
debbugs-guix.el. They use the "debbugs-gnu-guix" or "debbugs-org-guix"
prefixes, respectively.

>> I don't remember why the string "trunk" has been used, and this might
>> indeed predates the time git has been taken for Emacs development.
>
> Is the approach to have more than one source tree? Debbugs mentions
> a trunk and stable tree for a git based workflow a stable and
> master/main branch handling would be more helpful.

Yes, there are functions to merge git commits between different
branches. Honestly, I don't know too much about. This part was written
by Lars Ingebringtsen; I don't use it myself.

Best regards, Michael.

Re: A few questions about debbugs

[Michael Albinus]

Hi Björn,

>> The way you implemented this feature could also be a way to handle
>> mainlinglist archives in Gnus better if some such as those using
>> mailmain would provide a well known address.
>> For others there's at least public-inbox which works after each
>> mailing-list has enabled it. But that's OT.

BTW, if you want to read mailing lists comfortably, I recommend you to
use gmane.io. No need to download the messages locally.

Best regards, Michael.