Progress with automating testing of patches

Progress with automating testing of patches

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2250 bytes --]

Hey!

I think I sent the last message to guix-devel on this topic back in
February [1].

1: https://lists.gnu.org/archive/html/guix-devel/2022-02/msg00102.html

I haven't had a lot of time or motivation this year, but I've still been
trying to chip away at this.

The recent new thing is that I've joined up the uncompleted work to test
changes (patches and branches) with some thoughts from early last year
about a QA page [2].

2: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00096.html

While the data service solves the hard problem of knowing what's changed
with some patches or a branch, and the build coordinator solves the hard
problem of building things and doing something with the results, there
was a need for something else to address the other hard problem of tying
these tools together and communicating the information. I think this QA
service is a good fit.

Since it's easier to iterate once there's something visible, I've just
stuck what I've got so far on the internet, it's available at:

  https://qa.guix.gnu.org/

The code is here [3] and I've put a list of ideas in the README.

3: https://git.cbaines.net/guix/qa-frontpage/about/

Currently, there's a page which lists issues, and pages for individual
issues that show build and lint warning changes. Behind the scenes, it's
also submitting builds to the build coordinator for the packages
affected by patches (for x86_64-linux, i686-linux, aarch64-linux and
armhf-linux).

As I've been developing it, I've been looking at various recent patch
series, and it seems like this is already useful. It's reassuring when
reviewing patches to see that packages still build on the architectures
being tested. Also, unlike earlier prototype patch testing setups,
because the builds are now happening on a default substitute server,
there should be some benefits already with substitutes being available
at the point the patches are merged.

This is very much still at the prototype stage though, many pages will
timeout or just fail to load due to an error.

Let me know if you have any comments or questions? If you want to try
and work on adding any features, I should also have time to try and help
out as well, so just let me know you're interested.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Progress with automating testing of patches

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3206 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> Since it's easier to iterate once there's something visible, I've just
> stuck what I've got so far on the internet, it's available at:
>
>   https://qa.guix.gnu.org/
>
> The code is here [3] and I've put a list of ideas in the README.
>
> 3: https://git.cbaines.net/guix/qa-frontpage/about/
>
> Currently, there's a page which lists issues, and pages for individual
> issues that show build and lint warning changes. Behind the scenes, it's
> also submitting builds to the build coordinator for the packages
> affected by patches (for x86_64-linux, i686-linux, aarch64-linux and
> armhf-linux).
>
> As I've been developing it, I've been looking at various recent patch
> series, and it seems like this is already useful. It's reassuring when
> reviewing patches to see that packages still build on the architectures
> being tested. Also, unlike earlier prototype patch testing setups,
> because the builds are now happening on a default substitute server,
> there should be some benefits already with substitutes being available
> at the point the patches are merged.
>
> This is very much still at the prototype stage though, many pages will
> timeout or just fail to load due to an error.
>
> Let me know if you have any comments or questions? If you want to try
> and work on adding any features, I should also have time to try and help
> out as well, so just let me know you're interested.

There's been some more progress now since I sent this email, so I
thought I'd send an update.

I've started to try and show some overall status for each issue, that's
the green circle which can appear on the patches page. I want to try and
make this available as an image as well, so that it can be included on
the issue page on issues.guix.gnu.org, linking up qa.guix.gnu.org and
issues.guix.gnu.org.

I've also managed to use the GraphQL API for issues.guix.gnu.org to
fetch tags, which is very exciting as it makes it possible to do all
kinds of things I think. Starting with showing specific tags on the
qa.guix.gnu.org pages, and ending maybe with replacing Patchwork with
data provided by Mumi.

Also, while builds for patches were happening, there's now support for
submitting builds for a branch. This is the case for staging, so these
builds are now happening. It'll probably take many days, maybe even
weeks for all the builds to happen, but at this point I'm mostly
interested in getting the software working so that qa.guix.gnu.org is
really useful for working with branches going forward.

I also covered this qa.guix.gnu.org thing in the talk I gave yesterday
at the 10 Years of Guix event. See the email with this subject [1] for
some very rough notes from the discussion that happened in the late
afternoon. There's a list of seemingly actionable things in there, some
of which relate to the qa.guix.gnu.org site. In particular, I'll try and
push the repository to Savannah at some point this week, so let me know
if you have thoughts on changing the name (currently qa-frontpage)
before I do.

1: Notes from discussion on Quality Assurance from the 10 Years of Guix event

Do let me know if you have any comments or questions!

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Progress with automating testing of patches

[Ludovic Courtès]

Hello!

As discussed in Paris, I’m a big fan of qa.guix.gnu.org!  I like that it
shows all the information relevant to packagers and reviewers in a
concise way.

I wonder if it’s due to recent changes since I last looked, but I’m a
bit confused by the numbers in this example:

  https://qa.guix.gnu.org/issue/58186

The numbers before/after patches don’t match and the lint warnings (nice
addition!) appear to unrelated to the patch at hand.

Any idea what’s going on?

Conversely, <https://qa.guix.gnu.org/issue/58072> looks fine, for
example.

BTW, Emacs users, I have this key binding that I find useful:

--8<---------------cut here---------------start------------->8---
(defun ludo-jump-to-guix-qa-url ()
  "Jump to the QA page of the Debbugs issue at point."
  (interactive)
  (let ((url (concat "https://qa.guix.gnu.org/issue/"
		     (number-to-string (debbugs-gnu-current-id)))))
    (browse-url url)))

(define-key debbugs-gnu-mode-map (kbd "C-M-j") 'ludo-jump-to-guix-qa-url)
--8<---------------cut here---------------end--------------->8---

Thanks!

Ludo’.

Re: Progress with automating testing of patches

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1617 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> I wonder if it’s due to recent changes since I last looked, but I’m a
> bit confused by the numbers in this example:
>
>   https://qa.guix.gnu.org/issue/58186
>
> The numbers before/after patches don’t match and the lint warnings (nice
> addition!) appear to unrelated to the patch at hand.
>
> Any idea what’s going on?

I've had an initial look. One important clue is that the basis of the
comparison [1] differs between data.qa.guix.gnu.org and
data.guix.gnu.org [2]. The package number differs for example.

1: https://data.qa.guix.gnu.org/revision/e6777cfa5eb5e9c36eaf7810b42cac0fbcaa367c
2: https://data.guix.gnu.org/revision/e6777cfa5eb5e9c36eaf7810b42cac0fbcaa367c

That shouldn't happen, one revision of Guix should have the same number
of packages regardless of what server looked at it. This being wrong
explains the bad data on qa.guix.gnu.org, since the comparison being
done by data.qa.guix.gnu.org is based on bad data.

This reminds me of [3] where data.guix.gnu.org processed a revision and
somehow got things wrong.

3: https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00192.html

I'll try investigating this further when I have more time, there should
be locking in place so that even when multiple jobs are being processed
at the same time, only one job at a time is able to call
latest-channel-instances, so I don't currently have a theory as to how
this goes wrong. I think I added more logging off the back of the
previous issue, so maybe I'll be able to get further this time.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Progress with automating testing of patches

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> I wonder if it’s due to recent changes since I last looked, but I’m a
>> bit confused by the numbers in this example:
>>
>>   https://qa.guix.gnu.org/issue/58186
>>
>> The numbers before/after patches don’t match and the lint warnings (nice
>> addition!) appear to unrelated to the patch at hand.
>>
>> Any idea what’s going on?
>
> I've had an initial look. One important clue is that the basis of the
> comparison [1] differs between data.qa.guix.gnu.org and
> data.guix.gnu.org [2]. The package number differs for example.
>
> 1: https://data.qa.guix.gnu.org/revision/e6777cfa5eb5e9c36eaf7810b42cac0fbcaa367c
> 2: https://data.guix.gnu.org/revision/e6777cfa5eb5e9c36eaf7810b42cac0fbcaa367c
>
> That shouldn't happen, one revision of Guix should have the same number
> of packages regardless of what server looked at it. This being wrong
> explains the bad data on qa.guix.gnu.org, since the comparison being
> done by data.qa.guix.gnu.org is based on bad data.

OK, I see.

BTW, why do we have both data.qa.guix and data.guix?  These are both
instances of the same service, right?

Thanks,
Ludo’.

Re: Progress with automating testing of patches

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1806 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> I wonder if it’s due to recent changes since I last looked, but I’m a
>>> bit confused by the numbers in this example:
>>>
>>>   https://qa.guix.gnu.org/issue/58186
>>>
>>> The numbers before/after patches don’t match and the lint warnings (nice
>>> addition!) appear to unrelated to the patch at hand.
>>>
>>> Any idea what’s going on?
>>
>> I've had an initial look. One important clue is that the basis of the
>> comparison [1] differs between data.qa.guix.gnu.org and
>> data.guix.gnu.org [2]. The package number differs for example.
>>
>> 1: https://data.qa.guix.gnu.org/revision/e6777cfa5eb5e9c36eaf7810b42cac0fbcaa367c
>> 2: https://data.guix.gnu.org/revision/e6777cfa5eb5e9c36eaf7810b42cac0fbcaa367c
>>
>> That shouldn't happen, one revision of Guix should have the same number
>> of packages regardless of what server looked at it. This being wrong
>> explains the bad data on qa.guix.gnu.org, since the comparison being
>> done by data.qa.guix.gnu.org is based on bad data.
>
> OK, I see.
>
> BTW, why do we have both data.qa.guix and data.guix?  These are both
> instances of the same service, right?

Yep, but the configuration is different.

data.guix.gnu.org just tracks the master branch, and doesn't delete any
data.

data.qa.guix.gnu.org tracks all branches in the main Guix repository,
plus branches for issues, and regularly deletes revisions where they're
either old or the branch no longer exists.

There's nothing technically preventing just using one instance for both
purposes, it's just operationally I thought it would be easier to split
the concerns across two instances.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Progress with automating testing of patches

[jbranso]

October 1, 2022 1:08 PM, "Ludovic Courtès" <ludo@gnu.org> wrote:

> Hello!
> 
> As discussed in Paris, I’m a big fan of qa.guix.gnu.org! I like that it
> shows all the information relevant to packagers and reviewers in a
> concise way.
> 
> I wonder if it’s due to recent changes since I last looked, but I’m a
> bit confused by the numbers in this example:
> 
> https://qa.guix.gnu.org/issue/58186
> 
> The numbers before/after patches don’t match and the lint warnings (nice
> addition!) appear to unrelated to the patch at hand.
> 
> Any idea what’s going on?
> 
> Conversely, <https://qa.guix.gnu.org/issue/58072> looks fine, for
> example.
> 
> BTW, Emacs users, I have this key binding that I find useful:
> 
> --8<---------------cut here---------------start------------->8---
> (defun ludo-jump-to-guix-qa-url ()
> "Jump to the QA page of the Debbugs issue at point."
> (interactive)
> (let ((url (concat "https://qa.guix.gnu.org/issue"
> (number-to-string (debbugs-gnu-current-id)))))
> (browse-url url)))
> 
> (define-key debbugs-gnu-mode-map (kbd "C-M-j") 'ludo-jump-to-guix-qa-url)
> --8<---------------cut here---------------end--------------->8---

Would it make sense to add something like this to debbugs?

I just created a debbugs-guix.el file in debbugs.  The update should be
available on elpa by now.

https://issues.guix.gnu.org/56987

> 
> Thanks!
> 
> Ludo’.

debbugs-guix.el helper function

[zimoun]

Hi,

On mer., 05 oct. 2022 at 22:49, jbranso@dismail.de wrote:
> October 1, 2022 1:08 PM, "Ludovic Courtès" <ludo@gnu.org> wrote:

>> --8<---------------cut here---------------start------------->8---
>> (defun ludo-jump-to-guix-qa-url ()
>> "Jump to the QA page of the Debbugs issue at point."
>> (interactive)
>> (let ((url (concat "https://qa.guix.gnu.org/issue"
>> (number-to-string (debbugs-gnu-current-id)))))
>> (browse-url url)))
>> 
>> (define-key debbugs-gnu-mode-map (kbd "C-M-j") 'ludo-jump-to-guix-qa-url)
>> --8<---------------cut here---------------end--------------->8---
>
> Would it make sense to add something like this to debbugs?

In the same spirit, I have:

--8<---------------cut here---------------start------------->8---
(defmacro defun-bug->url (name url &optional docstring)
  "Macro returning yankage #bug URL.

The `interactive' function that the macro returns is then referred by NAME.

Please provide a DOCSTRING."
  (let ((fun (intern (symbol-name name)))
        (doc (concat docstring "\n\n"
                           (format "Yankable result: `%sNUMBER'." url))))
    `(defun ,fun (number)
       ,doc
        (interactive
         (list
          (progn
            (when (not (boundp 'debbugs-gnu-bug-number))
              (setq debbugs-gnu-bug-number -2))
            (read-string
             (format "Bug number (%s): " debbugs-gnu-bug-number)
             nil nil debbugs-gnu-bug-number))))
      (let ((str (format "%s%s" ,url number)))
        (kill-new str)
        (when current-prefix-arg
          (browse-url str))
        (message (format "%s killed." str))))))

(defun-bug->url my/guix-issues "http://issues.guix.gnu.org/issue/"
          "Add URL of bug NUMBER to `kill-ring'.")
(defun-bug->url my/guix-debbugs "https://debbugs.gnu.org/cgi/bugreport.cgi?bug="
          "Add (old) URL of bug NUMBER to `kill-ring'.")

(defun my/guix-data (package)
  "Add URL of PACKAGE to `kill-ring'.

Yankable result:
`https://data.guix.gnu.org/repository/1/branch/master/package/PACKAGE/output-history'.

With `universal-argument', load URL using `browse-url'."
  (interactive "sPackage: ")
  (let ((url
         (format
          "https://data.guix.gnu.org/repository/1/branch/master/package/%s/output-history" package)))
    (kill-new url)
    (when current-prefix-arg
      (browse-url url))
    (message (format "%s killed." url))))
--8<---------------cut here---------------end--------------->8---


And because I find Message-ID and public-inbox nice interface, I also
have:

--8<---------------cut here---------------start------------->8---
  (defun my/public-inbox-insert (number)
    "TODO"
    (interactive "nBug number: ")
    (let* ((meta  (car (debbugs-get-status number)))
           (inbox (car (debbugs-get-attribute meta 'package))) ;Probably inaccurate for the general case
           (raw   (debbugs-get-attribute meta 'msgid))
           (msgid (replace-regexp-in-string "<\\|>" "" raw)))
      (message "Message-ID: %s from %s." msgid inbox)
      (my/piem-inject-thread-into-maildir msgid inbox)
      (notmuch-command-to-string "new" "--no-hooks")))
--8<---------------cut here---------------end--------------->8---

which can be adapted (by removing the part about emacs-piem, great
package BTW! and the part about emacs-notmuch).

Well, I have in my TODO list to implement the extraction of Message-ID
from Emacs-Debbugs.  But it is not about debbugs.el and instead about
Gnus.  It could be very helpful to have a way to stash to the kill-ring
the Message-ID of one specific message in the thread; and not only the
Message-ID of the first message in that thread.


Cheers,
simon

Re: Progress with automating testing of patches

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> data.guix.gnu.org just tracks the master branch, and doesn't delete any
> data.
>
> data.qa.guix.gnu.org tracks all branches in the main Guix repository,
> plus branches for issues, and regularly deletes revisions where they're
> either old or the branch no longer exists.
>
> There's nothing technically preventing just using one instance for both
> purposes, it's just operationally I thought it would be easier to split
> the concerns across two instances.

Got it, that makes a lot of sense to me.  Thanks!

Ludo’.

Re: Progress with automating testing of patches

[Ludovic Courtès]

Hi,

jbranso@dismail.de skribis:

> I just created a debbugs-guix.el file in debbugs.  The update should be
> available on elpa by now.
>
> https://issues.guix.gnu.org/56987

Nice!  So that’ll be part of the next ‘emacs-debbugs’ package, right?

Ludo’.

Re: Progress with automating testing of patches

[Joshua Branson]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> jbranso@dismail.de skribis:
>
>> I just created a debbugs-guix.el file in debbugs.  The update should be
>> available on elpa by now.
>>
>> https://issues.guix.gnu.org/56987
>
> Nice!  So that’ll be part of the next ‘emacs-debbugs’ package, right?
>

Yes sir!  It provides debbugs-gnu-guix-search 
debbugs-gnu-my-open-bugs functions.  

>
> Ludo’.

Re: debbugs-guix.el helper function

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

> In the same spirit, I have:

[...]

> (defun-bug->url my/guix-issues "http://issues.guix.gnu.org/issue/"
>           "Add URL of bug NUMBER to `kill-ring'.")
> (defun-bug->url my/guix-debbugs "https://debbugs.gnu.org/cgi/bugreport.cgi?bug="
>           "Add (old) URL of bug NUMBER to `kill-ring'.")

I have something similar that I find extremely useful: hitting C-w on a
bug adds the mumi and debbugs URLs to the kill ring.

--8<---------------cut here---------------start------------->8---
(defun ludo-copy-debbugs-url ()
  "Add to the kill ring the URL of the Debbugs issue at point."
  (interactive)
  (let ((url1 (concat "https://bugs.gnu.org/"
		      (number-to-string (debbugs-gnu-current-id))))
	(url2 (concat "https://issues.guix.gnu.org/"
		      (number-to-string (debbugs-gnu-current-id)))))
    (kill-new url1)
    (kill-new url2)
    (message "Copied %s and %s" url1 url2)))

(define-key debbugs-gnu-mode-map (kbd "C-w") 'ludo-copy-debbugs-url)
--8<---------------cut here---------------end--------------->8---

Ludo’.

Re: debbugs-guix.el helper function

[zimoun]

Hi,

On Fri, 07 Oct 2022 at 11:47, Ludovic Courtès <ludo@gnu.org> wrote:

> I have something similar that I find extremely useful: hitting C-w on a
> bug adds the mumi and debbugs URLs to the kill ring.

In addition, another helper that I plan to use more… But it is not that
handy with Debbugs because of Gnus.  Well, via Notmuch (or Mu4e), it is
handy,

--8<---------------cut here---------------start------------->8---
(defun my/notmuch-issues (msgid)
  "Add URL of MSGID pointing to bug number to `kill-ring'.

Yankable result:
`http://issues.guix.gnu.org/issue/msgid/<Message-ID>'

then resolved by the server.

Work only `notmuch-tree-mode', `notmuch-search-mode' or
`notmuch-show-mode'."
  (interactive
   (list
    (let ((yanked
           (if (or (eq major-mode 'notmuch-tree-mode)
                   (eq major-mode 'notmuch-search-mode)
                   (eq major-mode 'notmuch-show-mode))
               (progn
                 (notmuch-show-stash-message-id-stripped)
                 (car kill-ring))
             "?")))
      (read-string
       (format "Message-ID (%s): " yanked)
       nil nil yanked))))
  (let* ((url
          (format "%s%s"
                  "http://issues.guix.gnu.org/msgid/"
                  msgid)))
    (kill-new url)
    (when current-prefix-arg
          (browse-url url))
        (message (format "%s killed." url))))
--8<---------------cut here---------------end--------------->8---

When reading from Emacs-Notmuch, I just run ’M-x my/notmuch-issues’ and
then I can yank elsewhere the URL.  Last, Mumi resolves from the
Message-ID to the Debbugs number (even the specific message instead of
#2 or else).


Cheers,
simon