Re: 02/03: gnu: openblas: Update architectures we provide substitutes for.

Re: 02/03: gnu: openblas: Update architectures we provide substitutes for.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]


guix-commits@gnu.org writes:

> efraim pushed a commit to branch master
> in repository guix.
>
> commit 076688fa1e41a09f034a80e1a593bac43f1f1482
> Author: Efraim Flashner <efraim@flashner.co.il>
> AuthorDate: Thu Jun 1 11:06:00 2023 +0300
>
>     gnu: openblas: Update architectures we provide substitutes for.
>
>     * gnu/packages/maths.scm (openblas)[arguments]: Adjust the substitutable?
>     flag to only not provide substitutes when building for powerpc-linux.
>     Adjust the comment accordingly.
> ---
>  gnu/packages/maths.scm | 11 ++---------
>  1 file changed, 2 insertions(+), 9 deletions(-)

I've been looking at why armhf-linux substitute availability has been
dropping recently, and I think this change triggered a lot of
rebuilds. Could this have gone to core-updates?

→ guix refresh -l openblas
Building the following 2282 packages would ensure 5596 dependent packages are rebuilt: ...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: 02/03: gnu: openblas: Update architectures we provide substitutes for.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1874 bytes --]

On Fri, Jun 02, 2023 at 11:03:42PM +0100, Christopher Baines wrote:
> 
> guix-commits@gnu.org writes:
> 
> > efraim pushed a commit to branch master
> > in repository guix.
> >
> > commit 076688fa1e41a09f034a80e1a593bac43f1f1482
> > Author: Efraim Flashner <efraim@flashner.co.il>
> > AuthorDate: Thu Jun 1 11:06:00 2023 +0300
> >
> >     gnu: openblas: Update architectures we provide substitutes for.
> >
> >     * gnu/packages/maths.scm (openblas)[arguments]: Adjust the substitutable?
> >     flag to only not provide substitutes when building for powerpc-linux.
> >     Adjust the comment accordingly.
> > ---
> >  gnu/packages/maths.scm | 11 ++---------
> >  1 file changed, 2 insertions(+), 9 deletions(-)
> 
> I've been looking at why armhf-linux substitute availability has been
> dropping recently, and I think this change triggered a lot of
> rebuilds. Could this have gone to core-updates?
> 
> → guix refresh -l openblas
> Building the following 2282 packages would ensure 5596 dependent packages are rebuilt: ...

It's not that it's triggered rebuilds, but that it's triggered builds.
It's also triggered builds on powerpc64le and riscv64. Before any
package which had openblas as a transitive dependency wasn't built by
the CI because it wasn't substitutable¹. People still have the option of
using package transformations to use openblas tuned for the cortex a7 or
a15 on armhf, but in reality this just unlocks substitutes for those
~5600 packages which wasn't available before.

¹ We saw this in the past briefly in the past when openzfs made its way
as a dependency to qemu and through that to Gnome.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: 02/03: gnu: openblas: Update architectures we provide substitutes for.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2645 bytes --]


Efraim Flashner <efraim@flashner.co.il> writes:

> On Fri, Jun 02, 2023 at 11:03:42PM +0100, Christopher Baines wrote:
>> 
>> guix-commits@gnu.org writes:
>> 
>> > efraim pushed a commit to branch master
>> > in repository guix.
>> >
>> > commit 076688fa1e41a09f034a80e1a593bac43f1f1482
>> > Author: Efraim Flashner <efraim@flashner.co.il>
>> > AuthorDate: Thu Jun 1 11:06:00 2023 +0300
>> >
>> >     gnu: openblas: Update architectures we provide substitutes for.
>> >
>> >     * gnu/packages/maths.scm (openblas)[arguments]: Adjust the substitutable?
>> >     flag to only not provide substitutes when building for powerpc-linux.
>> >     Adjust the comment accordingly.
>> > ---
>> >  gnu/packages/maths.scm | 11 ++---------
>> >  1 file changed, 2 insertions(+), 9 deletions(-)
>> 
>> I've been looking at why armhf-linux substitute availability has been
>> dropping recently, and I think this change triggered a lot of
>> rebuilds. Could this have gone to core-updates?
>> 
>> → guix refresh -l openblas
>> Building the following 2282 packages would ensure 5596 dependent packages are rebuilt: ...
>
> It's not that it's triggered rebuilds, but that it's triggered builds.
> It's also triggered builds on powerpc64le and riscv64. Before any
> package which had openblas as a transitive dependency wasn't built by
> the CI because it wasn't substitutable¹. People still have the option of
> using package transformations to use openblas tuned for the cortex a7 or
> a15 on armhf, but in reality this just unlocks substitutes for those
> ~5600 packages which wasn't available before.
>
> ¹ We saw this in the past briefly in the past when openzfs made its way
> as a dependency to qemu and through that to Gnome.

Ok, so the documentation does mention "rebuilding", and I do see that
indeed ci.guix.gnu.org doesn't build not substitutable things.

Although I think it doesn't apply recursively. Take qjson, guix refresh
-l tells me it's dependent on openblas, and looking back at say this
output [1] for powerpc64le-linux, that's available from both
ci.guix.gnu.org. Which makes sense, as that derivation is substitutable,
even though one of it's inputs isn't.

1: https://data.guix.gnu.org/gnu/store/fibiwzyz8s899ccpix5zs6r2pcdpxk5b-qjson-0.9.0

Maybe on the client side this works differently, and guix won't
substitute things which have a non substitutable input?

Assuming ci.guix.gnu.org was building things for armhf-linux, I think
this would have still caused ~5596 rebuilds, and as I say, I think for
systems like powerpc64le-linux, I think it did cause ~5596
rebuilds.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: 02/03: gnu: openblas: Update architectures we provide substitutes for.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 4286 bytes --]

On Sat, Jun 03, 2023 at 08:12:48PM +0100, Christopher Baines wrote:
> 
> Efraim Flashner <efraim@flashner.co.il> writes:
> 
> > On Fri, Jun 02, 2023 at 11:03:42PM +0100, Christopher Baines wrote:
> >> 
> >> guix-commits@gnu.org writes:
> >> 
> >> > efraim pushed a commit to branch master
> >> > in repository guix.
> >> >
> >> > commit 076688fa1e41a09f034a80e1a593bac43f1f1482
> >> > Author: Efraim Flashner <efraim@flashner.co.il>
> >> > AuthorDate: Thu Jun 1 11:06:00 2023 +0300
> >> >
> >> >     gnu: openblas: Update architectures we provide substitutes for.
> >> >
> >> >     * gnu/packages/maths.scm (openblas)[arguments]: Adjust the substitutable?
> >> >     flag to only not provide substitutes when building for powerpc-linux.
> >> >     Adjust the comment accordingly.
> >> > ---
> >> >  gnu/packages/maths.scm | 11 ++---------
> >> >  1 file changed, 2 insertions(+), 9 deletions(-)
> >> 
> >> I've been looking at why armhf-linux substitute availability has been
> >> dropping recently, and I think this change triggered a lot of
> >> rebuilds. Could this have gone to core-updates?
> >> 
> >> → guix refresh -l openblas
> >> Building the following 2282 packages would ensure 5596 dependent packages are rebuilt: ...
> >
> > It's not that it's triggered rebuilds, but that it's triggered builds.
> > It's also triggered builds on powerpc64le and riscv64. Before any
> > package which had openblas as a transitive dependency wasn't built by
> > the CI because it wasn't substitutable¹. People still have the option of
> > using package transformations to use openblas tuned for the cortex a7 or
> > a15 on armhf, but in reality this just unlocks substitutes for those
> > ~5600 packages which wasn't available before.
> >
> > ¹ We saw this in the past briefly in the past when openzfs made its way
> > as a dependency to qemu and through that to Gnome.
> 
> Ok, so the documentation does mention "rebuilding", and I do see that
> indeed ci.guix.gnu.org doesn't build not substitutable things.
> 
> Although I think it doesn't apply recursively. Take qjson, guix refresh
> -l tells me it's dependent on openblas, and looking back at say this
> output [1] for powerpc64le-linux, that's available from both
> ci.guix.gnu.org. Which makes sense, as that derivation is substitutable,
> even though one of it's inputs isn't.
> 
> 1: https://data.guix.gnu.org/gnu/store/fibiwzyz8s899ccpix5zs6r2pcdpxk5b-qjson-0.9.0
> 
> Maybe on the client side this works differently, and guix won't
> substitute things which have a non substitutable input?
> 
> Assuming ci.guix.gnu.org was building things for armhf-linux, I think
> this would have still caused ~5596 rebuilds, and as I say, I think for
> systems like powerpc64le-linux, I think it did cause ~5596
> rebuilds.

I looked into it more. First I ran on master:
'./pre-inst-env guix build --no-grafts --system=armhf-linux openblas -d'
/gnu/store/whi4yhiw2b0c0i3n6l8s0qfcphkvbzg4-openblas-0.3.20.drv

Then I locally reverted the patch expanding the architectures where we
provided substitutes:
/gnu/store/1m57z8jkbf6gz7qlbw3ws4ayl0ln9602-openblas-0.3.20.drv

Then I locally reverted the patch adjusting the make-flags:
/gnu/store/1m57z8jkbf6gz7qlbw3ws4ayl0ln9602-openblas-0.3.20.drv

It seems I was wrong, changing the #:substitutable? flag _does_ change
the derivation of the package. I also checked the substitutes and saw
that bordeaux did (and does) have substitutes for the non-substitutable
version, showing that it was built before. I also checked for
powerpc64le, to see if perhaps cuirass worked a different way and
honored the #:substitutable flag by not building it there, and it too
has substitutes for both versions of openblas.

I didn't check for riscv64 but I assume the case is the same with a
changing derivation.

So obviously if I had realized this would cause ~5596 rebuilds per
affected arch I wouldn't have pushed the patch. I should've checked the
derivation before and after to make sure it didn't change.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: 02/03: gnu: openblas: Update architectures we provide substitutes for.

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 357 bytes --]

Hi everyone,

Efraim Flashner <efraim@flashner.co.il> writes:
> It seems I was wrong, changing the #:substitutable? flag _does_ change
> the derivation of the package.

Is this expected/something that should be fixed?  It does seem to me
that this shouldn't happen, but maybe there's a good reason for this
behavior.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: 02/03: gnu: openblas: Update architectures we provide substitutes for.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 4451 bytes --]


Efraim Flashner <efraim@flashner.co.il> writes:

> On Sat, Jun 03, 2023 at 08:12:48PM +0100, Christopher Baines wrote:
>> 
>> Efraim Flashner <efraim@flashner.co.il> writes:
>> 
>> > On Fri, Jun 02, 2023 at 11:03:42PM +0100, Christopher Baines wrote:
>> >> 
>> >> guix-commits@gnu.org writes:
>> >> 
>> >> > efraim pushed a commit to branch master
>> >> > in repository guix.
>> >> >
>> >> > commit 076688fa1e41a09f034a80e1a593bac43f1f1482
>> >> > Author: Efraim Flashner <efraim@flashner.co.il>
>> >> > AuthorDate: Thu Jun 1 11:06:00 2023 +0300
>> >> >
>> >> >     gnu: openblas: Update architectures we provide substitutes for.
>> >> >
>> >> >     * gnu/packages/maths.scm (openblas)[arguments]: Adjust the substitutable?
>> >> >     flag to only not provide substitutes when building for powerpc-linux.
>> >> >     Adjust the comment accordingly.
>> >> > ---
>> >> >  gnu/packages/maths.scm | 11 ++---------
>> >> >  1 file changed, 2 insertions(+), 9 deletions(-)
>> >> 
>> >> I've been looking at why armhf-linux substitute availability has been
>> >> dropping recently, and I think this change triggered a lot of
>> >> rebuilds. Could this have gone to core-updates?
>> >> 
>> >> → guix refresh -l openblas
>> >> Building the following 2282 packages would ensure 5596 dependent packages are rebuilt: ...
>> >
>> > It's not that it's triggered rebuilds, but that it's triggered builds.
>> > It's also triggered builds on powerpc64le and riscv64. Before any
>> > package which had openblas as a transitive dependency wasn't built by
>> > the CI because it wasn't substitutable¹. People still have the option of
>> > using package transformations to use openblas tuned for the cortex a7 or
>> > a15 on armhf, but in reality this just unlocks substitutes for those
>> > ~5600 packages which wasn't available before.
>> >
>> > ¹ We saw this in the past briefly in the past when openzfs made its way
>> > as a dependency to qemu and through that to Gnome.
>> 
>> Ok, so the documentation does mention "rebuilding", and I do see that
>> indeed ci.guix.gnu.org doesn't build not substitutable things.
>> 
>> Although I think it doesn't apply recursively. Take qjson, guix refresh
>> -l tells me it's dependent on openblas, and looking back at say this
>> output [1] for powerpc64le-linux, that's available from both
>> ci.guix.gnu.org. Which makes sense, as that derivation is substitutable,
>> even though one of it's inputs isn't.
>> 
>> 1: https://data.guix.gnu.org/gnu/store/fibiwzyz8s899ccpix5zs6r2pcdpxk5b-qjson-0.9.0
>> 
>> Maybe on the client side this works differently, and guix won't
>> substitute things which have a non substitutable input?
>> 
>> Assuming ci.guix.gnu.org was building things for armhf-linux, I think
>> this would have still caused ~5596 rebuilds, and as I say, I think for
>> systems like powerpc64le-linux, I think it did cause ~5596
>> rebuilds.
>
> I looked into it more. First I ran on master:
> './pre-inst-env guix build --no-grafts --system=armhf-linux openblas -d'
> /gnu/store/whi4yhiw2b0c0i3n6l8s0qfcphkvbzg4-openblas-0.3.20.drv
>
> Then I locally reverted the patch expanding the architectures where we
> provided substitutes:
> /gnu/store/1m57z8jkbf6gz7qlbw3ws4ayl0ln9602-openblas-0.3.20.drv
>
> Then I locally reverted the patch adjusting the make-flags:
> /gnu/store/1m57z8jkbf6gz7qlbw3ws4ayl0ln9602-openblas-0.3.20.drv
>
> It seems I was wrong, changing the #:substitutable? flag _does_ change
> the derivation of the package. I also checked the substitutes and saw
> that bordeaux did (and does) have substitutes for the non-substitutable
> version, showing that it was built before. I also checked for
> powerpc64le, to see if perhaps cuirass worked a different way and
> honored the #:substitutable flag by not building it there, and it too
> has substitutes for both versions of openblas.
>
> I didn't check for riscv64 but I assume the case is the same with a
> changing derivation.
>
> So obviously if I had realized this would cause ~5596 rebuilds per
> affected arch I wouldn't have pushed the patch. I should've checked the
> derivation before and after to make sure it didn't change.

Thanks for investigating further.

There's maybe a secondary issue here about why changing the
substitutibility of a package affects it's outputs. That doesn't make
much sense to me, but maybe there's a reason.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]