upgrade to new version of packages with lots of dependencies--installing new versions along first?

upgrade to new version of packages with lots of dependencies--installing new versions along first?

[Andy Tai]

Hi, this is nothing new and clearly has been the case for a number of
packages in Guix.

For packages that have not of dependencies (say 300 or more) updating
to new versions have been increasingly difficult, as lots of rebuilds
needed and Guix QA is really overwhelmed  (it seems).

For other reasons like API compatibility some such packages exist in a
state of multiple versions, with old versions sticking around to
satisfy other packages depending on them. Example: samba and
samba-next

Maybe this can be generalized for packages with lots of dependencies,
say for any package A with 300 (or 1000) dependencies, when upgrading
to a new version, add that as A-next so existing A is not touched.
Then A-next can be added without any dependencies, and Guix QA just
needs to check if A-next builds.   Then people working on any package
depending on A can move that package to depending on A-next later
independently.   This avoid triggering world rebuilds.  And A's
dependents migrate at their own flow.

Or a feature branch can still be used for these migrating dependencies
but A-next can be added to master first and made available, without
waiting for a feature branch to be merged

Is this workable idea?

Re: upgrade to new version of packages with lots of dependencies--installing new versions along first?

[Greg Hogan]

On Fri, Sep 6, 2024 at 4:56 PM Andy Tai <atai@atai.org> wrote:
>
> Maybe this can be generalized for packages with lots of dependencies,
> say for any package A with 300 (or 1000) dependencies, when upgrading
> to a new version, add that as A-next so existing A is not touched.
> Then A-next can be added without any dependencies, and Guix QA just
> needs to check if A-next builds.   Then people working on any package
> depending on A can move that package to depending on A-next later
> independently.   This avoid triggering world rebuilds.  And A's
> dependents migrate at their own flow.

Does the manual upgrade of dependent packages happen in practice
outside of receiving an error due to an unsatisfied minimum version?
For example, `fmt-9` (the default) has 1812 dependent packages whereas
`fmt-10` and 'fmt-11' have none even though the former was added in
June 2023.

It would be handy to automate the dependency upgrade process when a
new package version can be built with a new library version. Likewise,
automated, bulk migrations of dependent packages when adding a new
library version. An example of the former: when upgrading mariadb and
after a successful build, `guix` could be requested to upgrade and
build with `fmt-11`, then on failure upgrade and build with `fmt-10`,
then on failure leave the input unchanged as `fmt-9`.

When upgrading the library itself, for example switching the default
`fmt` from `fmt-9` to `fmt-10`, it is almost certain that many
packages will break even after the long delay for those dependent
packages to be upgraded. I believe this to be a major contributor to
the number of broken builds on the master branch, and automating
piecewise upgrades should reduce this problem and ease the burden on
the contributor.

As I understand, there would be cases where multiple packages need to
be upgraded together, in particular when propagating inputs. This
works today when we upgrade a common default version but we could see
an increase in package conflicts as dependencies diverge.

It would be necessary, or at least very helpful, to couple the package
versions together (ordered by precedence) alongside the package
definitions.

Greg