Automatically testing package updates

Automatically testing package updates

[Ludovic Courtès]

Hello Guix!

A new patch series that has just landed¹ adds a ‘--dependents’ flag (and
also ‘--development’) to ‘guix build’, which makes it easier to build
dependents when modifying a package.

It also adds ‘etc/upgrade-manifest.scm’, a manifest that grabs and
returns the latest upstream version of a few security-critical packages.
More precisely, it returns two things:

  1. Individual package updates.  Update each of libgcrypt, libgit2,
     etc. independently and return the updated packages together with
     their direct dependents.

  2. Joint package updates.  Update all these packages at once and
     return their dependents at distance two.

The result can be seen here (x86_64-linux only):

  https://ci.guix.gnu.org/jobset/security-updates

You can go to the dashboard:

  https://ci.guix.gnu.org/eval/latest/dashboard?spec=security-updates

… and type, for example, “latest-libgpg-error” (to view the libgpg-error
update and its dependents) or “full-upgrade” (to view all the dependents
of that set of packages).

Some examples:

  • guile-ssh fails to build with the latest libssh:
    <https://ci.guix.gnu.org/build/6753990/log>.

  • libgcrypt cannot be upgraded without libgpg-error:
    <https://ci.guix.gnu.org/build/6754153/log>.

  • libgcrypt 1.11.0 builds fine when upgraded jointly with its
    dependents: <https://ci.guix.gnu.org/build/6754305/details>

  • gnutls 3.8.8 has one test failure:
    <https://ci.guix.gnu.org/build/6753571/log>.

  • curl 8.11.0 has one test failure:
    <https://ci.guix.gnu.org/build/6753884/log>.

This manifest is just an example.  We could come up with manifests
targeting package collections like CRAN packages, astronomy packages,
and so on.

Feedback welcome!

Ludo’.

¹ https://issues.guix.gnu.org/74542

Re: Automatically testing package updates

[Simon Tournier]

Hi,

That’s cool!  Here, I share a “random thought”. :-)

> This manifest is just an example.  We could come up with manifests
> targeting package collections like CRAN packages, astronomy packages,
> and so on.

Discussing on Mastodon, Konrad emitted this idea [1]:

        Just an idea from someone mostly ignorant about how the build
        farms work: could package builds be done lazily? E.g. the build
        farm tracks requests for yet-unbuilt packages, and builds those
        that have been requested ten times or so.

I know it’s not possible considering the current way substitutes are
designed.

The idea is still appealing; especially when using “guix time-machine”.

Currently and considering patch#74542 [2], the build farm will try to
build all the updates by upstream for some “defined” packages even if
these updates would never reach any branch or worse never land to
master.  If I understand correctly, the idea behind is to ease updates
and/or reviewing.  Yeah it should help. :-)

Consider Blender: many of all the updates are trivial updates.  But on
my relatively new laptop, it takes around 15 minutes to build.  It’s not
nothing and it might be a strong refrain [3] for updating and/or
reviewing the update.  Here is the past rate of the updates:

    5905b47287 gnu: blender: Update to 3.6.13. Sat Jul 6 00:52:44 2024 -0300
    ebaf658acd gnu: blender: Update to 3.6.10. Sat Apr 6 08:40:56 2024 -0300
    e8d163b49a gnu: blender: Update to 3.3.5. Fri Apr 14 15:11:51 2023 -0400
    eb169d36e5 gnu: blender: Update to 3.3.5. Fri Mar 31 13:29:29 2023 -0400
    45fe602602 gnu: blender: Update to 3.3.1. Sat Nov 19 19:54:14 2022 +0100
    f05f831f66 gnu: blender: Update to 3.0.1. Wed Feb 9 16:06:28 2022 +0100
    48d38125c2 gnu: blender: Update to 2.93.6. Sun Nov 21 18:37:50 2021 +0100

The idea of the manifest, if I understand correctly what’s behind, is to
ease that, then for example the versions 3.6.11 and 3.6.12 would have
been built by removing the constraint on having the capacity to locally
build.

That’s said, it appears to me a bazooka: build all just in case.

Because it’s not sure at all that someone™ would be motivated to commit
the Blender’s updates of 3.6.11 and 3.6.12; if we had the manifest
discussed before. :-)

Back to Konrad’s idea!

Consider that if instead of building all we would potentially need, we
only build what we really need.  That’s what QA tries to do: build all
the patches.  However, sometimes QA is lagging [4] or the number of
dependencies is too high.  Let put more <3 in QA!  Still.

Now, consider the case where we use "guix time-machine".  Today, the
substitutes availability depends on the weather. ;-) Well, sometimes
substitutes are still available, sometimes not; it depends on the cache
policy.  Indeed, we cannot store all binaries forever.

Therefore, consider the feature: When using "guix time-machine" on a
restricted set of channels, and so using specific revisions defined by
the channels, if the content-addressed item is not available, then it
would be possible to queue one or more or none transformations for some
packages from one specific revision of some channels we care.

This feature solves two problems at once: 1. avoid the bazooka but still
help by easing the reviewing of updates and 2. provide substitutes for
past revisions.

However, this needs to adapt how the mechanism for substituting works,
right?  The question seems: Is it doable or already doomed by current
design?  In other words, would it require too much work?


Cheers,
simon

1: https://social.sciences.re/@khinsen@scholar.social/113550338572274110

2: [bug#74542] [PATCH 11/11] etc: Add upgrade manifest.
Ludovic Courtès <ludo@gnu.org>
Tue, 26 Nov 2024 11:33:50 +0100
id:c55d9c57d99b50436c3afa607beaf62ae46d3c40.1732615193.git.ludo@gnu.org
https://issues.guix.gnu.org/74542
https://issues.guix.gnu.org/msgid/c55d9c57d99b50436c3afa607beaf62ae46d3c40.1732615193.git.ludo@gnu.org
https://yhetil.org/guix/c55d9c57d99b50436c3afa607beaf62ae46d3c40.1732615193.git.ludo@gnu.org

3: Public guix offload server
Arun Isaac <arunisaac@systemreboot.net>
Thu, 21 Oct 2021 02:23:49 +0530
id:878rynh0yq.fsf@systemreboot.net
https://lists.gnu.org/archive/html/guix-devel/2021-10
https://yhetil.org/guix/878rynh0yq.fsf@systemreboot.net

4: https://social.sciences.re/@civodul@toot.aquilenet.fr/113549606095407968

Re: Automatically testing package updates

[Simon Tournier]

Hi,

On Mon, 02 Dec 2024 at 14:44, Ludovic Courtès <ludo@gnu.org> wrote:

> This manifest is just an example.  We could come up with manifests
> targeting package collections like CRAN packages, astronomy packages,
> and so on.

Let consider that a “leaf” package is a package that does not appear
elsewhere in any other packages.  Then, if there is no mistake, I count
14143 leaf packages over 32459 all packages.

Well, if we filter using gnu-build-system and cmake-build-system then it
reduces to 2847 packages.

These packages are “safe” to automatically update because the only
annoyance will be users at run time. ;-)

Somehow, the loop and type conversion are not optimized but maybe this
kind of “leaf” packages could defined and then filtered build-system per
build-system, i.e., team per team.

WDYT?

Cheers,
simon

Re: Automatically testing package updates

[Simon Tournier]

[-- Attachment #1.1: Type: text/plain, Size: 23 bytes --]

Manifest attached. :-)

[-- Attachment #1.2: Type: text/html, Size: 70 bytes --]

[-- Attachment #2: leaf-packages.scm --]
[-- Type: text/x-scheme, Size: 1544 bytes --]


(use-modules (gnu packages)
             (guix build-system gnu)
             (guix build-system cmake)
             (guix packages)
             (guix sets)
             (ice-9 match)
             (ice-9 vlist)
             (srfi srfi-1))

(define all (all-packages))

(define (p->k p)
  (string-append (package-name p) "@" (package-version p)))

(define vall-packages
  (fold (lambda (package result)
          (vhash-cons (p->k package) package result))
        vlist-null
        all))

(define vleaf-packages
  (vhash-fold (lambda (key package result)
                (let loop ((dependencies (package-direct-inputs package))
                           (updated result))
                  (match dependencies
                    ('() updated)
                    ((or ((_ p) . tail)
                         ((_ p _) . tail))
                     (if (package? p)
                         (loop tail
                               (vhash-delete (p->k p) updated))
                         (loop tail updated))))))
              vall-packages
              vall-packages))

(define leaf-packages
  (vhash-fold (lambda (key package result)
                (cons package result))
              '()
              vleaf-packages))

(define subset
  (filter-map (lambda (package)
                 (and (memq (package-build-system package)
                            (list gnu-build-system cmake-build-system))
                      (package-with-upstream-version (pk 'up package))))
              leaf-packages))

(packages->manifest
 subset)

Re: Automatically testing package updates

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

On 2024-12-02, Ludovic Courtès wrote:
> A new patch series that has just landed¹ adds a ‘--dependents’ flag (and
> also ‘--development’) to ‘guix build’, which makes it easier to build
> dependents when modifying a package.

Yay!

> Some examples:
>
>   • guile-ssh fails to build with the latest libssh:
>     <https://ci.guix.gnu.org/build/6753990/log>.

This can be fixed by updating to guile-ssh 0.18.0:

  https://issues.guix.gnu.org/74659

Which also works with libssh 0.10.x in guix master.

Running "guix build --dependents guile-ssh" on it now.

First thing I notice is it is trying to build all of rust, which does
not always get a lot of testing on aarch64-linux... so might end up
being a long chain of builds that fail...

Might be nice to have options to exclude specific dependents from the
builds to perform? With guix refresh --list-dependents, it was a bit
difficult, but you could actually manually exclude things from the
output and then only the builds you wanted to try. Maybe something like:

  guix build --dependents PACKAGE --exclude-dependents OTHERPACKAGE


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Re: Automatically testing package updates

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 1075 bytes --]


Hi,

That option I've been dreaming about for a long time!

I try to keep cadence with update in astronomy module monthly and having
ongoing refresh work for golang collection. Maybe we may have some API
output on that type available at <https://packages.guix.gnu.org/>,
showing the impact ration of the given package and the number of
dependents as well?

Some quick script (WIP) to calculate that stats based on build system.

<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=74268>

There is no much work after "guix refresh -u <go-package>" when I've
refreshed Golang packae.

From other side but current implementation of pyproject-build-system or
python-build-system and related importer/updater in most of the cases
just breaks the package completely. It looks like the issue comes from
the fact that Python in a massive migration to pyproject.toml (setup.py
and setup.cfg are becoming deprecated) which specifies install, test and
dev dependencies and during refresh it can't be parsed, so it would
block automatic packge refresh.

--
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Automatically testing package updates

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1671 bytes --]

On Mon, Dec 02, 2024 at 08:14:04PM +0100, Simon Tournier wrote:
> Hi,
> 
> On Mon, 02 Dec 2024 at 14:44, Ludovic Courtès <ludo@gnu.org> wrote:
> 
> > This manifest is just an example.  We could come up with manifests
> > targeting package collections like CRAN packages, astronomy packages,
> > and so on.
> 
> Let consider that a “leaf” package is a package that does not appear
> elsewhere in any other packages.  Then, if there is no mistake, I count
> 14143 leaf packages over 32459 all packages.
> 
> Well, if we filter using gnu-build-system and cmake-build-system then it
> reduces to 2847 packages.
> 
> These packages are “safe” to automatically update because the only
> annoyance will be users at run time. ;-)
> 
> Somehow, the loop and type conversion are not optimized but maybe this
> kind of “leaf” packages could defined and then filtered build-system per
> build-system, i.e., team per team.
> 
> WDYT?

To repeat what I said on Mastodon and the impression I got from the Nix
folks, if we can automate away the easy/simple package upgrades then we
can spend our time on more complex upgrades that require more attention
to get correct.

I'm now thinking about the easy-package -> hard-package learning curve
for new contributors, and I'm not sure how we'd address that.

Also I don't think we should spam the bug tracker with package upgrades,
so we'd have to think about something for that.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Automatically testing package updates

[Hilton Chain]

Hi,

On Thu, 05 Dec 2024 15:20:53 +0800,
Efraim Flashner wrote:
>
> I'm now thinking about the easy-package -> hard-package learning curve
> for new contributors, and I'm not sure how we'd address that.

I'm thinking about something like a wiki of packaging examples, where it starts
from a 'hello' package to explain basic concepts and (most importantly from my
observation) rules for synopses and descriptions etc., expanding to specific
cases for various build systems.  Since we already have plenty of good examples,
the content can be quickly filled.

Thanks

Re: Automatically testing package updates

[Ricardo Wurmus]

Hilton Chain <hako@ultrarare.space> writes:

> On Thu, 05 Dec 2024 15:20:53 +0800,
> Efraim Flashner wrote:
>>
>> I'm now thinking about the easy-package -> hard-package learning curve
>> for new contributors, and I'm not sure how we'd address that.
>
> I'm thinking about something like a wiki of packaging examples, where it starts
> from a 'hello' package to explain basic concepts and (most importantly from my
> observation) rules for synopses and descriptions etc., expanding to specific
> cases for various build systems.  Since we already have plenty of good examples,
> the content can be quickly filled.

The Cookbook has a packaging tutorial.  It could get more tutorials of
the kind you're describing.

-- 
Ricardo

Re: Automatically testing package updates

[Suhail Singh]

Efraim Flashner <efraim@flashner.co.il> writes:

> Also I don't think we should spam the bug tracker with package upgrades,
> so we'd have to think about something for that.

Could you clarify what you're objecting to?  Is it bug issues being
created that are then not (as is currently the case) automatically
closed?  Or is simply the creation of bug issues problematic, even if
they were to be closed automatically by a bot (yet to be implemented)?

-- 
Suhail

Re: Automatically testing package updates

[Ricardo Wurmus]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Efraim Flashner <efraim@flashner.co.il> writes:
>
>> Also I don't think we should spam the bug tracker with package upgrades,
>> so we'd have to think about something for that.
>
> Could you clarify what you're objecting to?  Is it bug issues being
> created that are then not (as is currently the case) automatically
> closed?  Or is simply the creation of bug issues problematic, even if
> they were to be closed automatically by a bot (yet to be implemented)?

It would be preferable to automate trivial updates *completely*, even
bypassing the need for manual patch generation and submission to the bug
tracker.  I'd rather reserve the issue tracker for humans and keep bots
in a realm that is separate from the user interfaces designed for
humans.

-- 
Ricardo

Re: Automatically testing package updates

[Suhail Singh]

Ricardo Wurmus <rekado@elephly.net> writes:

> It would be preferable to automate trivial updates *completely*, even
> bypassing the need for manual patch generation and submission to the
> bug tracker.

I agree with the goal of automating trivial updates.  One way of doing
so is outlined in the proposal I shared here:
<https://yhetil.org/guix/87ed2yufar.fsf@gmail.com/>.  Importantly,
automating trivial updates which result in _automated_ patch generation,
is not fundamentally mutually exclusive with "submission to the bug
tracker".

> I'd rather reserve the issue tracker for humans and keep bots in a
> realm that is separate from the user interfaces designed for humans.

I believe it is important to provide _some_ way of distinguishing bot
activities from that of humans.  It's not clear to me that keeping said
activities entirely separate from human activities is the best way of
doing so.

I don't necessarily disagree with keeping the two separate.  However, it
is still important to provide some transparency on bot activities.
Keeping things on the issue tracker has said transparency as a nice
side-effect.  However, it is by no means the only way of doing so.

If we do decide to keep bots in a separate realm as it were, it would be
desirable (and in my opinion, important) if that still allowed for at
least as much transparency that the issue tracker provides for human
activities.

It would help the discussion if we could express the benefits that would
be achieved by not tracking bot activities on the issue tracker.
Especially since there are certain drawbacks (reduction in transparency,
unless addressed by other means) of doing so.

Do you have thoughts on the matter?

-- 
Suhail

Re: Automatically testing package updates

[Attila Lendvai]

> If we do decide to keep bots in a separate realm as it were, it would be
> desirable (and in my opinion, important) if that still allowed for at
> least as much transparency that the issue tracker provides for human
> activities.


what other data would you miss that is not available in the git log?

sidenote: as a newcomer i found it rather off-putting that the bug tracker is full of random two-liner package updates, many long forgotten and even obsolete.

for at least two reasons:

 - it's a lot of wasted effort; the admin overhead of a simple package
   update is several times of the effort that goes into creating the
   patch. it gives the impression that the project organization
   doesn't value the time of its participants.

 - when i search for something, it's lost in the noise of boring and
   obsolete stuff in the issue tracker (package updates being one
   source). if a bot joins the "effort", i'm afraid it'll become that
   much worse.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“The more artificial taboos and restrictions there are in the world, the more the people are impoverished…. The more that laws and regulations are given prominence, the more thieves and robbers there will be.”
	— Lao Tzu (sixth century BC)

Re: Automatically testing package updates

[Ricardo Wurmus]

Attila Lendvai <attila@lendvai.name> writes:

> sidenote: as a newcomer i found it rather off-putting that the bug
> tracker is full of random two-liner package updates, many long
> forgotten and even obsolete.

This is exactly my point and why I think the bug tracker must contain
only *actionable* work that humans are meant to work on.  As we're
trying to undo the damage of years of being overwhelmed by patches I'd
really like to keep the issue tracker for humans only.

-- 
Ricardo

Re: Automatically testing package updates

[Ricardo Wurmus]

Suhail Singh <suhailsingh247@gmail.com> writes:

> If we do decide to keep bots in a separate realm as it were, it would be
> desirable (and in my opinion, important) if that still allowed for at
> least as much transparency that the issue tracker provides for human
> activities.

Examples are the guix-cran and guix-bioc channels:

    https://github.com/guix-science/guix-cran
    https://github.com/guix-science/guix-bioc

They are generated with these scripts:

    https://github.com/guix-science/guix-cran-scripts/

For my purposes this is sufficient transparency.  Automated updates for
closed package collections (by team scope or otherwise) would look
similar, I assume.

-- 
Ricardo

Re: Automatically testing package updates

[Suhail Singh]

Attila Lendvai <attila@lendvai.name> writes:

> what other data would you miss that is not available in the git log?

No data that is currently being automatically generated.  Then again,
none of the patches at present are.  One can argue (and perhaps that's
what you are) that, even in the future, whatever automated-comments may
be left (e.g. guix lint/style stats, integration test-suite status) on
issues created for automated-patches could instead also be noted in the
automated-commits.  That's certainly possible.

Another thing to note is that when there's an integration-test-suite,
patch-generation may not automatically lead to patch-merge, and even in
the case that they do there may be a delay between patch-generation and
patch-merge.  Those situations are elegantly handled by having a
patch-generation and patch-merge bot interacting on a debbugs
issue-tracker.

>  - it's a lot of wasted effort; the admin overhead of a simple package
>    update is several times of the effort that goes into creating the
>    patch. it gives the impression that the project organization
>    doesn't value the time of its participants.

Sure, however, this doesn't apply in the case when patches are
automatically handled.

>  - when i search for something, it's lost in the noise of boring and
>    obsolete stuff in the issue tracker (package updates being one
>    source). if a bot joins the "effort", i'm afraid it'll become that
>    much worse.

Being "lost in the noise" can be addressed by ensuring there's a
straightforward way to exclude all bot-generated-patches from search
results.

Guix already makes good use of debbugs usertags.  It would be
straightforward for all automatically generated patches to have a
pre-defined usertag denoting the fact that they've been automatically
generated.  The mumi interface at issues.gnu.guix.org could also have
the default configuration which ignores all issues with said usertag.

Conclusion: The reasons presented so far seem to fundamentally be about
retaining the ability to discern automatically-generated-patches from
human-generated-patches.  This is possible while still tracking such
patches via the issue tracker.

-- 
Suhail

Re: Automatically testing package updates

[Suhail Singh]

Ricardo Wurmus <rekado@elephly.net> writes:

>> sidenote: as a newcomer i found it rather off-putting that the bug
>> tracker is full of random two-liner package updates, many long
>> forgotten and even obsolete.
>
> This is exactly my point and why I think the bug tracker must contain
> only *actionable* work that humans are meant to work on.

I believe it's only required that the bug tracker provide a way to limit
results to only humanly-generated-patches.

If by bug tracker we mean the mumi instance at issues.gnu.guix.org note
that something like the above already happens today.  The debbugs
instance is shared with Emacs.  However, bugs relating to the "emacs"
user don't show up in the search results in the mumi interface.  All
currently humanly-generated patches sent to guix-patches are assigned
the user "guix".  It would be possible to have the bot-generated patches
be assigned a different user, or if the same user, a specific user-tag
that is filtered out by mumi in the search results.

There may be possibly other reasons why we don't want to do so.
However, being able to only focus on the humanly-generated-patches isn't
such a reason.

-- 
Suhail

Re: Automatically testing package updates

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Attila Lendvai <attila@lendvai.name> writes:
>
>> sidenote: as a newcomer i found it rather off-putting that the bug
>> tracker is full of random two-liner package updates, many long
>> forgotten and even obsolete.
>
> This is exactly my point and why I think the bug tracker must contain
> only *actionable* work that humans are meant to work on.  As we're
> trying to undo the damage of years of being overwhelmed by patches I'd
> really like to keep the issue tracker for humans only.

I wonder what level of human validation we’d want for “trivial updates”.
In Nixpkgs, IIUC, a bot submits pull requests and a human being
eventually clicks the merge button.

Ludo’.

Re: Automatically testing package updates

[Ludovic Courtès]

Vagrant Cascadian <vagrant@debian.org> skribis:

> This can be fixed by updating to guile-ssh 0.18.0:
>
>   https://issues.guix.gnu.org/74659
>
> Which also works with libssh 0.10.x in guix master.
>
> Running "guix build --dependents guile-ssh" on it now.

Yay, well done! :-)

> First thing I notice is it is trying to build all of rust, which does
> not always get a lot of testing on aarch64-linux... so might end up
> being a long chain of builds that fail...
>
> Might be nice to have options to exclude specific dependents from the
> builds to perform? With guix refresh --list-dependents, it was a bit
> difficult, but you could actually manually exclude things from the
> output and then only the builds you wanted to try. Maybe something like:
>
>   guix build --dependents PACKAGE --exclude-dependents OTHERPACKAGE

Hmm yes, why not.

Ludo’.

Re: Automatically testing package updates

[Ludovic Courtès]

Hi,

Sharlatan Hellseher <sharlatanus@gmail.com> skribis:

> I try to keep cadence with update in astronomy module monthly and having
> ongoing refresh work for golang collection. Maybe we may have some API
> output on that type available at <https://packages.guix.gnu.org/>,
> showing the impact ration of the given package and the number of
> dependents as well?

I’m not sure what you mean here.

But what about having an astronomy update manifest?

[...]

> From other side but current implementation of pyproject-build-system or
> python-build-system and related importer/updater in most of the cases
> just breaks the package completely. It looks like the issue comes from
> the fact that Python in a massive migration to pyproject.toml (setup.py
> and setup.cfg are becoming deprecated) which specifies install, test and
> dev dependencies and during refresh it can't be parsed, so it would
> block automatic packge refresh.

Oh.  So we should update the pypi importer/updater to parse that, right?

Ludo’.