Fedora/Debian release monitoring inspiration for Guix Data Service

Fedora/Debian release monitoring inspiration for Guix Data Service

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 1548 bytes --]

Hello!

It seems Fedora has automated infrastructure and feeds to monitor new
upstream releases so that maintainers can get notifications on them.

https://release-monitoring.org/

Functional feed: 
https://apps.fedoraproject.org/datagrepper/raw?rows_per_page=1&delta=127800&category=hotness

I could not find the actual data it is based on for release monitoring
(like Debian watch files).

I could not find a similar feed for Debian's watch/uscan but it would
be nice if they provided some sort of RSS feed (global or scoped to
packages or sets of packages).

I am thinking we should really get something like this with the Guix
Data Service, start including something similar to Debian uscan/watch
rules in every package so we can get reliable (in majority of cases)
update notifications.

We could also have some feature that gives you a feed for a manifest or
operating-system definition so that people can prioritize work on what
they care about easily.

I would really love to have some RSS feed for new upstream releases
from Guix Data Service based on my operating-system definition I would
upload there.

We could even go as far as preparing a patch (similar to guix refresh
-u <pkg> && etc/committer.scm) and trying to build it and all it's
dependents and if it doesnt cause any new failures we could send an
automated patch contribution on guix-patches directly and tag it with
"ci-update-ok" or something so a maintainer just has to double-check
quickly and push (very efficient workflow). 

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Fedora/Debian release monitoring inspiration for Guix Data Service

[Ricardo Wurmus]

Léo Le Bouter <lle-bout@zaclys.net> writes:

> I am thinking we should really get something like this with the Guix
> Data Service, start including something similar to Debian uscan/watch
> rules in every package so we can get reliable (in majority of cases)
> update notifications.

“guix lint” has a checker for releases:

--8<---------------cut here---------------start------------->8---
$ guix lint -l
Available checkers:
[…]
- refresh: Check the package for new upstream releases
--8<---------------cut here---------------end--------------->8---

It just needs to be extended to cover more packages.

Developing a service that runs this for all packages is an orthogonal
issue.

-- 
Ricardo

Re: Fedora/Debian release monitoring inspiration for Guix Data Service

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 1240 bytes --]

On Tue, 2021-03-16 at 12:12 +0100, Ricardo Wurmus wrote:
> Léo Le Bouter <lle-bout@zaclys.net> writes:
> 
> > I am thinking we should really get something like this with the
> > Guix
> > Data Service, start including something similar to Debian
> > uscan/watch
> > rules in every package so we can get reliable (in majority of
> > cases)
> > update notifications.
> 
> “guix lint” has a checker for releases:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix lint -l
> Available checkers:
> […]
> - refresh: Check the package for new upstream releases
> --8<---------------cut here---------------end--------------->8---
> 
> It just needs to be extended to cover more packages.
> 
> Developing a service that runs this for all packages is an orthogonal
> issue.
> 

Just to clarify, I do know it exists and use it often :-)
Indeed it needs to be extended and that's what I am talking about
there, also Guix Data Service is important here because we cannot
possibly each on our sides spam the various URLs or APIs to generate
the release monitoring dataset. Instead, using Guix Data Service for
that with specially configured API keys and else sounds like a better
idea.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Fedora/Debian release monitoring inspiration for Guix Data Service

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 4003 bytes --]


Léo Le Bouter <lle-bout@zaclys.net> writes:

> It seems Fedora has automated infrastructure and feeds to monitor new
> upstream releases so that maintainers can get notifications on them.
>
> https://release-monitoring.org/
>
> Functional feed:
> https://apps.fedoraproject.org/datagrepper/raw?rows_per_page=1&delta=127800&category=hotness
>
> I could not find the actual data it is based on for release monitoring
> (like Debian watch files).

I think it might be in the service itself.

> I could not find a similar feed for Debian's watch/uscan but it would
> be nice if they provided some sort of RSS feed (global or scoped to
> packages or sets of packages).
>
> I am thinking we should really get something like this with the Guix
> Data Service, start including something similar to Debian uscan/watch
> rules in every package so we can get reliable (in majority of cases)
> update notifications.

I did spot these patches
https://patches.guix-patches.cbaines.net/project/guix-patches/list/?series=7298

I think the Guix Data Service could run the "refresh" code from Guix and
store relevant data, although I'm also thinking along the lines of
trying to generate patches, like you go on to below...

> We could also have some feature that gives you a feed for a manifest or
> operating-system definition so that people can prioritize work on what
> they care about easily.
>
> I would really love to have some RSS feed for new upstream releases
> from Guix Data Service based on my operating-system definition I would
> upload there.

So, one thing I'm hoping to start work on in the coming weeks/months is
the long awaited service for providing subscriptions/notifications for
the Guix Data Service (+ other services).

My current plan is to use a WebSub like pattern, but this could easily
be adapted to RSS/email/...

> We could even go as far as preparing a patch (similar to guix refresh
> -u <pkg> && etc/committer.scm) and trying to build it and all it's
> dependents and if it doesnt cause any new failures we could send an
> automated patch contribution on guix-patches directly and tag it with
> "ci-update-ok" or something so a maintainer just has to double-check
> quickly and push (very efficient workflow).

Yeah, one problem with the current automated patch review stuff I've got
going at the moment is that there's no feedback when the Guix Data
Service finds out that things do/don't build.

However, as I also set out above, there's been a plan and design for
making that possible for years, it just needs implementing.

One thing I'm hoping to do once it's possible to subscribe to Guix Data
Service data is make the checks in Patchwork actually reflective of
results (like 4 packages broken by these changes), rather than just
providing links, and someone having to figure out what information is
hidden within.

Those same subscriptions could be used to prompt people to look at
patches for package updates that don't introduce breakages (following
what you set out above).

The pieces are slowly coming together for this, at least with the way I
would approach it. For example, it's possible to get commits in to
data.guix-patches.cbaines.net now by just pushing to the Git repository,
so to set out something similar to what you describe above:

 - Some service watches for new releases (through the guix refresh
   code), and then makes commits, and pushes them to a Git repo

 - There's a Guix Data Service reading from that Git repository, it
   starts processing the changes

 - Something (probably the "service" above") subscribes to find out when
   relevant information is available (like build successes/failures)

 - Builds happen via the Guix Build Coordinator, which feeds information
   back to the Guix Data Service

 - That "service" gets the notifications that the commits are good, and
   prompts someone to review them

I hope some of that makes sense,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Fedora/Debian release monitoring inspiration for Guix Data Service

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 3099 bytes --]

On Tue, 2021-03-16 at 23:28 +0000, Christopher Baines wrote:
> I did spot these patches
> https://patches.guix-patches.cbaines.net/project/guix-patches/list/?series=7298

Awesome!! I did not see them earlier!

> I think the Guix Data Service could run the "refresh" code from Guix
> and
> store relevant data, although I'm also thinking along the lines of
> trying to generate patches, like you go on to below...

Yes :-D

> So, one thing I'm hoping to start work on in the coming weeks/months
> is
> the long awaited service for providing subscriptions/notifications
> for
> the Guix Data Service (+ other services).
> 
> My current plan is to use a WebSub like pattern, but this could
> easily
> be adapted to RSS/email/...

That sounds very great! I know you've been working on this, but I also
wanted (through my email) try to make more people aware and draw
attention to the subject of automation in GNU Guix.

> Yeah, one problem with the current automated patch review stuff I've
> got
> going at the moment is that there's no feedback when the Guix Data
> Service finds out that things do/don't build.

Yes, and because of that the Guix Data Service and guix-build-
coordinator-agent thing you are running goes mostly unused (even for
me!).

> However, as I also set out above, there's been a plan and design for
> making that possible for years, it just needs implementing.
> 
> One thing I'm hoping to do once it's possible to subscribe to Guix
> Data
> Service data is make the checks in Patchwork actually reflective of
> results (like 4 packages broken by these changes), rather than just
> providing links, and someone having to figure out what information is
> hidden within.
> 
> Those same subscriptions could be used to prompt people to look at
> patches for package updates that don't introduce breakages (following
> what you set out above).

+1000

> The pieces are slowly coming together for this, at least with the way
> I
> would approach it. For example, it's possible to get commits in to
> data.guix-patches.cbaines.net now by just pushing to the Git
> repository,
> so to set out something similar to what you describe above:
> 
>  - Some service watches for new releases (through the guix refresh
>    code), and then makes commits, and pushes them to a Git repo
> 
>  - There's a Guix Data Service reading from that Git repository, it
>    starts processing the changes
> 
>  - Something (probably the "service" above") subscribes to find out
> when
>    relevant information is available (like build successes/failures)
> 
>  - Builds happen via the Guix Build Coordinator, which feeds
> information
>    back to the Guix Data Service
> 
>  - That "service" gets the notifications that the commits are good,
> and
>    prompts someone to review them

All awesome! Can't wait! As soon as I can I will make sure to help, I
am reading through the GNU Guile manual, need to find more
concentration time.

> I hope some of that makes sense,
> 
> Chris

Thanks a lot Chris for all this!!

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Fedora/Debian release monitoring inspiration for Guix Data Service

[Ludovic Courtès]

Hi,

Léo Le Bouter <lle-bout@zaclys.net> skribis:

> It seems Fedora has automated infrastructure and feeds to monitor new
> upstream releases so that maintainers can get notifications on them.
>
> https://release-monitoring.org/

Established distros have great tools and services for that.

Guix has ‘guix refresh’, which predates some of the trendy release/CVE
monitoring services and actually works well.  It’s not perfect, but it’s
good, so my advice would be to work on improving it.

If you look at Guix’s design and history, you’ll find that much is built
in a way that avoids relying on external services when we can.

> We could even go as far as preparing a patch (similar to guix refresh
> -u <pkg> && etc/committer.scm) and trying to build it and all it's
> dependents and if it doesnt cause any new failures we could send an
> automated patch contribution on guix-patches directly and tag it with
> "ci-update-ok" or something so a maintainer just has to double-check
> quickly and push (very efficient workflow). 

I definitely agree.  We could build upon the infrastructure we already
have to write a tool that roughly runs ‘guix refresh -u’ on individual
packages, attempts to build it and its dependents, and publishes a
ready-to-apply patch—or a failure notification.

Whether it’s a task for the Guix Data Service or some other service, I
don’t know, but it would be most welcome!

Ludo’.

Re: Fedora/Debian release monitoring inspiration for Guix Data Service

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 1163 bytes --]

On Wed, 2021-03-17 at 18:35 +0100, Ludovic Courtès wrote:
> Established distros have great tools and services for that.
> 
> Guix has ‘guix refresh’, which predates some of the trendy
> release/CVE
> monitoring services and actually works well.  It’s not perfect, but
> it’s
> good, so my advice would be to work on improving it.
> 
> If you look at Guix’s design and history, you’ll find that much is
> built
> in a way that avoids relying on external services when we can.

Just to clarify, I also get behind that very much I was suggesting the
functional part more so than the service kind, even though I think
services (like Guix Data Service) are OK as long as datasets are easily
exportable and the service itself can easily be re-deployed through GNU
Guix as well.

It seems GNU Guix takes a generic approach to updates while Debian or
Fedora seems to look at specialized rules for each package, I was
thinking we could import those already existing rules (in Fedora's or
Debian's) into GNU Guix, I find it a superior approach for reliability
of notifications even though generic updaters are also very
complementary.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Fedora/Debian release monitoring inspiration for Guix Data Service

[Ludovic Courtès]

Hi,

Léo Le Bouter <lle-bout@zaclys.net> skribis:

> It seems GNU Guix takes a generic approach to updates while Debian or
> Fedora seems to look at specialized rules for each package, I was
> thinking we could import those already existing rules (in Fedora's or
> Debian's) into GNU Guix, I find it a superior approach for reliability
> of notifications even though generic updaters are also very
> complementary.

The updaters in Guix are fairly specialized; only ‘generic-html’ is
generic.

Do take a closer look; it’s an area of the code that’s fun to work with!

Ludo’.