Automatically testing package updates

Automatically testing package updates

[Ludovic Courtès]

Hello Guix!

A new patch series that has just landed¹ adds a ‘--dependents’ flag (and
also ‘--development’) to ‘guix build’, which makes it easier to build
dependents when modifying a package.

It also adds ‘etc/upgrade-manifest.scm’, a manifest that grabs and
returns the latest upstream version of a few security-critical packages.
More precisely, it returns two things:

  1. Individual package updates.  Update each of libgcrypt, libgit2,
     etc. independently and return the updated packages together with
     their direct dependents.

  2. Joint package updates.  Update all these packages at once and
     return their dependents at distance two.

The result can be seen here (x86_64-linux only):

  https://ci.guix.gnu.org/jobset/security-updates

You can go to the dashboard:

  https://ci.guix.gnu.org/eval/latest/dashboard?spec=security-updates

… and type, for example, “latest-libgpg-error” (to view the libgpg-error
update and its dependents) or “full-upgrade” (to view all the dependents
of that set of packages).

Some examples:

  • guile-ssh fails to build with the latest libssh:
    <https://ci.guix.gnu.org/build/6753990/log>.

  • libgcrypt cannot be upgraded without libgpg-error:
    <https://ci.guix.gnu.org/build/6754153/log>.

  • libgcrypt 1.11.0 builds fine when upgraded jointly with its
    dependents: <https://ci.guix.gnu.org/build/6754305/details>

  • gnutls 3.8.8 has one test failure:
    <https://ci.guix.gnu.org/build/6753571/log>.

  • curl 8.11.0 has one test failure:
    <https://ci.guix.gnu.org/build/6753884/log>.

This manifest is just an example.  We could come up with manifests
targeting package collections like CRAN packages, astronomy packages,
and so on.

Feedback welcome!

Ludo’.

¹ https://issues.guix.gnu.org/74542

Re: Automatically testing package updates

[Simon Tournier]

Hi,

That’s cool!  Here, I share a “random thought”. :-)

> This manifest is just an example.  We could come up with manifests
> targeting package collections like CRAN packages, astronomy packages,
> and so on.

Discussing on Mastodon, Konrad emitted this idea [1]:

        Just an idea from someone mostly ignorant about how the build
        farms work: could package builds be done lazily? E.g. the build
        farm tracks requests for yet-unbuilt packages, and builds those
        that have been requested ten times or so.

I know it’s not possible considering the current way substitutes are
designed.

The idea is still appealing; especially when using “guix time-machine”.

Currently and considering patch#74542 [2], the build farm will try to
build all the updates by upstream for some “defined” packages even if
these updates would never reach any branch or worse never land to
master.  If I understand correctly, the idea behind is to ease updates
and/or reviewing.  Yeah it should help. :-)

Consider Blender: many of all the updates are trivial updates.  But on
my relatively new laptop, it takes around 15 minutes to build.  It’s not
nothing and it might be a strong refrain [3] for updating and/or
reviewing the update.  Here is the past rate of the updates:

    5905b47287 gnu: blender: Update to 3.6.13. Sat Jul 6 00:52:44 2024 -0300
    ebaf658acd gnu: blender: Update to 3.6.10. Sat Apr 6 08:40:56 2024 -0300
    e8d163b49a gnu: blender: Update to 3.3.5. Fri Apr 14 15:11:51 2023 -0400
    eb169d36e5 gnu: blender: Update to 3.3.5. Fri Mar 31 13:29:29 2023 -0400
    45fe602602 gnu: blender: Update to 3.3.1. Sat Nov 19 19:54:14 2022 +0100
    f05f831f66 gnu: blender: Update to 3.0.1. Wed Feb 9 16:06:28 2022 +0100
    48d38125c2 gnu: blender: Update to 2.93.6. Sun Nov 21 18:37:50 2021 +0100

The idea of the manifest, if I understand correctly what’s behind, is to
ease that, then for example the versions 3.6.11 and 3.6.12 would have
been built by removing the constraint on having the capacity to locally
build.

That’s said, it appears to me a bazooka: build all just in case.

Because it’s not sure at all that someone™ would be motivated to commit
the Blender’s updates of 3.6.11 and 3.6.12; if we had the manifest
discussed before. :-)

Back to Konrad’s idea!

Consider that if instead of building all we would potentially need, we
only build what we really need.  That’s what QA tries to do: build all
the patches.  However, sometimes QA is lagging [4] or the number of
dependencies is too high.  Let put more <3 in QA!  Still.

Now, consider the case where we use "guix time-machine".  Today, the
substitutes availability depends on the weather. ;-) Well, sometimes
substitutes are still available, sometimes not; it depends on the cache
policy.  Indeed, we cannot store all binaries forever.

Therefore, consider the feature: When using "guix time-machine" on a
restricted set of channels, and so using specific revisions defined by
the channels, if the content-addressed item is not available, then it
would be possible to queue one or more or none transformations for some
packages from one specific revision of some channels we care.

This feature solves two problems at once: 1. avoid the bazooka but still
help by easing the reviewing of updates and 2. provide substitutes for
past revisions.

However, this needs to adapt how the mechanism for substituting works,
right?  The question seems: Is it doable or already doomed by current
design?  In other words, would it require too much work?


Cheers,
simon

1: https://social.sciences.re/@khinsen@scholar.social/113550338572274110

2: [bug#74542] [PATCH 11/11] etc: Add upgrade manifest.
Ludovic Courtès <ludo@gnu.org>
Tue, 26 Nov 2024 11:33:50 +0100
id:c55d9c57d99b50436c3afa607beaf62ae46d3c40.1732615193.git.ludo@gnu.org
https://issues.guix.gnu.org/74542
https://issues.guix.gnu.org/msgid/c55d9c57d99b50436c3afa607beaf62ae46d3c40.1732615193.git.ludo@gnu.org
https://yhetil.org/guix/c55d9c57d99b50436c3afa607beaf62ae46d3c40.1732615193.git.ludo@gnu.org

3: Public guix offload server
Arun Isaac <arunisaac@systemreboot.net>
Thu, 21 Oct 2021 02:23:49 +0530
id:878rynh0yq.fsf@systemreboot.net
https://lists.gnu.org/archive/html/guix-devel/2021-10
https://yhetil.org/guix/878rynh0yq.fsf@systemreboot.net

4: https://social.sciences.re/@civodul@toot.aquilenet.fr/113549606095407968

Re: Automatically testing package updates

[Simon Tournier]

Hi,

On Mon, 02 Dec 2024 at 14:44, Ludovic Courtès <ludo@gnu.org> wrote:

> This manifest is just an example.  We could come up with manifests
> targeting package collections like CRAN packages, astronomy packages,
> and so on.

Let consider that a “leaf” package is a package that does not appear
elsewhere in any other packages.  Then, if there is no mistake, I count
14143 leaf packages over 32459 all packages.

Well, if we filter using gnu-build-system and cmake-build-system then it
reduces to 2847 packages.

These packages are “safe” to automatically update because the only
annoyance will be users at run time. ;-)

Somehow, the loop and type conversion are not optimized but maybe this
kind of “leaf” packages could defined and then filtered build-system per
build-system, i.e., team per team.

WDYT?

Cheers,
simon

Re: Automatically testing package updates

[Simon Tournier]

[-- Attachment #1.1: Type: text/plain, Size: 23 bytes --]

Manifest attached. :-)

[-- Attachment #1.2: Type: text/html, Size: 70 bytes --]

[-- Attachment #2: leaf-packages.scm --]
[-- Type: text/x-scheme, Size: 1544 bytes --]


(use-modules (gnu packages)
             (guix build-system gnu)
             (guix build-system cmake)
             (guix packages)
             (guix sets)
             (ice-9 match)
             (ice-9 vlist)
             (srfi srfi-1))

(define all (all-packages))

(define (p->k p)
  (string-append (package-name p) "@" (package-version p)))

(define vall-packages
  (fold (lambda (package result)
          (vhash-cons (p->k package) package result))
        vlist-null
        all))

(define vleaf-packages
  (vhash-fold (lambda (key package result)
                (let loop ((dependencies (package-direct-inputs package))
                           (updated result))
                  (match dependencies
                    ('() updated)
                    ((or ((_ p) . tail)
                         ((_ p _) . tail))
                     (if (package? p)
                         (loop tail
                               (vhash-delete (p->k p) updated))
                         (loop tail updated))))))
              vall-packages
              vall-packages))

(define leaf-packages
  (vhash-fold (lambda (key package result)
                (cons package result))
              '()
              vleaf-packages))

(define subset
  (filter-map (lambda (package)
                 (and (memq (package-build-system package)
                            (list gnu-build-system cmake-build-system))
                      (package-with-upstream-version (pk 'up package))))
              leaf-packages))

(packages->manifest
 subset)