Re: 01/01: gnu: curl: Update to 7.45.0.

Re: 01/01: gnu: curl: Update to 7.45.0.

[Mark H Weaver]

Andreas Enge <andreas@enge.fr> writes:

> andreas pushed a commit to branch master
> in repository guix.
>
> commit 075c3ebd2dc3d8223e23025ceb5026810dfaa98d
> Author: Andreas Enge <andreas@enge.fr>
> Date:   Sun Oct 18 12:20:02 2015 +0200
>
>     gnu: curl: Update to 7.45.0.

To avoid unnecessary rebuilds on hydra, I reverted this commit on
master, and instead pushed it to the 'dbus-update' branch, which will
hopefully be merged soon.

    Thanks,
      Mark

Dealing with mass rebuilds

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> Andreas Enge <andreas@enge.fr> writes:
>
>> andreas pushed a commit to branch master
>> in repository guix.
>>
>> commit 075c3ebd2dc3d8223e23025ceb5026810dfaa98d
>> Author: Andreas Enge <andreas@enge.fr>
>> Date:   Sun Oct 18 12:20:02 2015 +0200
>>
>>     gnu: curl: Update to 7.45.0.
>
> To avoid unnecessary rebuilds on hydra, I reverted this commit on
> master, and instead pushed it to the 'dbus-update' branch, which will
> hopefully be merged soon.

We’re talking of ~150 dependent packages, which to me sounded OK¹.

If we delay obvious changes like this too much, the risk is to end up
with upgrade-everything branches that take ages to get merged (what is
happening with ‘dbus-update’.)

The Nixpkgs folks have a ‘staging’ branch for changes that cause mass
rebuilds but are well tested, such that they can be merged into ‘master’
anytime².  Perhaps something we should do as well?

Ludo’.

¹ https://lists.gnu.org/archive/html/guix-devel/2015-10/msg00653.html
² http://comments.gmane.org/gmane.linux.distributions.nixos/13447

Re: Dealing with mass rebuilds

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1683 bytes --]

On Tue, 20 Oct 2015 21:45:54 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Mark H Weaver <mhw@netris.org> skribis:
> 
> > Andreas Enge <andreas@enge.fr> writes:
> >  
> >> andreas pushed a commit to branch master
> >> in repository guix.
> >>
> >> commit 075c3ebd2dc3d8223e23025ceb5026810dfaa98d
> >> Author: Andreas Enge <andreas@enge.fr>
> >> Date:   Sun Oct 18 12:20:02 2015 +0200
> >>
> >>     gnu: curl: Update to 7.45.0.  
> >
> > To avoid unnecessary rebuilds on hydra, I reverted this commit on
> > master, and instead pushed it to the 'dbus-update' branch, which will
> > hopefully be merged soon.  
> 
> We’re talking of ~150 dependent packages, which to me sounded OK¹.
> 
> If we delay obvious changes like this too much, the risk is to end up
> with upgrade-everything branches that take ages to get merged (what is
> happening with ‘dbus-update’.)
> 
> The Nixpkgs folks have a ‘staging’ branch for changes that cause mass
> rebuilds but are well tested, such that they can be merged into ‘master’
> anytime².  Perhaps something we should do as well?
> 
> Ludo’.
> 
> ¹ https://lists.gnu.org/archive/html/guix-devel/2015-10/msg00653.html
> ² http://comments.gmane.org/gmane.linux.distributions.nixos/13447
> 

It would make an obvious place to push an sqlite update and allow the
dbus-update update branch to focus just on the dbus-update, or a core-update
branch to focus just on core updates.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dealing with mass rebuilds

[Andreas Enge]

On Tue, Oct 20, 2015 at 09:45:54PM +0200, Ludovic Courtès wrote:
> The Nixpkgs folks have a ‘staging’ branch for changes that cause mass
> rebuilds but are well tested, such that they can be merged into ‘master’
> anytime².  Perhaps something we should do as well?

Would that be different from your suggestion that if the curl update caused
a rebuild of too many packages (whatever that would mean, which is a separate
discussion) it should not be done on core-updates, but on its own branch,
for instance, curl-update? One practical advantage I see in  a staging
branch, if I understand your suggestion correctly, is that one would not need
to modify the set of jobs on hydra to now also build curl-update, and then
maybe giflib-update, and then xyz-update, but it would always be called
"staging".

Now the "well tested" assumption is dubious. I actually do not know whether
the curl update will go through. I tried to build one of the packages shown
by "guix refresh -l curl" that looked simple (mpd), but it turned out it
was not simple at all and needed a lot of rebuilds (including texlive).
Admittedly, I could have spent more time searching a direct dependency,
probably by using one of your recent graph drawing commands. So in case
a problem turns up, would we revert a curl update on the staging branch
and do more local work before proposing it again?

And if Efraim wished to do a gitlib update after my curl update, what would
be the protocol? Would he be expected to wait until the hydra build of the
curl update has finished? Or at least until it has finished on x86_64?
Otherwise, we would again face the risk of the staging branch becoming
an update-the-world branch with unforeseen ramifications.

Andreas

Re: Dealing with mass rebuilds

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2483 bytes --]

On Tue, 20 Oct 2015 23:47:16 +0200
Andreas Enge <andreas@enge.fr> wrote:

> On Tue, Oct 20, 2015 at 09:45:54PM +0200, Ludovic Courtès wrote:
> > The Nixpkgs folks have a ‘staging’ branch for changes that cause mass
> > rebuilds but are well tested, such that they can be merged into ‘master’
> > anytime².  Perhaps something we should do as well?  
> 
> Would that be different from your suggestion that if the curl update caused
> a rebuild of too many packages (whatever that would mean, which is a separate
> discussion) it should not be done on core-updates, but on its own branch,
> for instance, curl-update? One practical advantage I see in  a staging
> branch, if I understand your suggestion correctly, is that one would not need
> to modify the set of jobs on hydra to now also build curl-update, and then
> maybe giflib-update, and then xyz-update, but it would always be called
> "staging".
> 
> Now the "well tested" assumption is dubious. I actually do not know whether
> the curl update will go through. I tried to build one of the packages shown
> by "guix refresh -l curl" that looked simple (mpd), but it turned out it
> was not simple at all and needed a lot of rebuilds (including texlive).
> Admittedly, I could have spent more time searching a direct dependency,
> probably by using one of your recent graph drawing commands. So in case
> a problem turns up, would we revert a curl update on the staging branch
> and do more local work before proposing it again?
> 
> And if Efraim wished to do a gitlib update after my curl update, what would
> be the protocol? Would he be expected to wait until the hydra build of the
> curl update has finished? Or at least until it has finished on x86_64?
> Otherwise, we would again face the risk of the staging branch becoming
> an update-the-world branch with unforeseen ramifications.
> 
> Andreas
> 
> 

How much build infrastructure does nix have? (do they build all of staging
all the time?) Would it be possible to identify the subset of packages that
are immediately dependant on the updated package, and build only those as a
test on hydra? That would be faster, but if something broke it'd be harder to
track down, a la 白い熊's conkeror issue.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dealing with mass rebuilds

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> How much build infrastructure does nix have? (do they build all of staging
> all the time?) Would it be possible to identify the subset of packages that
> are immediately dependant on the updated package, and build only those as a
> test on hydra?

‘guix refresh -l’ provides that information.  Hydra itself always tries
to build everything though.

> That would be faster, but if something broke it'd be harder to
> track down, a la 白い熊's conkeror issue.

The Conkeror issue is a run-time issue though, one that is not revealed
by its test suite apparently.

Ludo’.

Re: Dealing with mass rebuilds

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Tue, Oct 20, 2015 at 09:45:54PM +0200, Ludovic Courtès wrote:
>> The Nixpkgs folks have a ‘staging’ branch for changes that cause mass
>> rebuilds but are well tested, such that they can be merged into ‘master’
>> anytime².  Perhaps something we should do as well?
>
> Would that be different from your suggestion that if the curl update caused
> a rebuild of too many packages (whatever that would mean, which is a separate
> discussion) it should not be done on core-updates, but on its own branch,
> for instance, curl-update? One practical advantage I see in  a staging
> branch, if I understand your suggestion correctly, is that one would not need
> to modify the set of jobs on hydra to now also build curl-update, and then
> maybe giflib-update, and then xyz-update, but it would always be called
> "staging".

Yes.

> Now the "well tested" assumption is dubious. I actually do not know whether
> the curl update will go through. I tried to build one of the packages shown
> by "guix refresh -l curl" that looked simple (mpd), but it turned out it
> was not simple at all and needed a lot of rebuilds (including texlive).
> Admittedly, I could have spent more time searching a direct dependency,
> probably by using one of your recent graph drawing commands. So in case
> a problem turns up, would we revert a curl update on the staging branch
> and do more local work before proposing it again?

Ideally, we’d test as much as possible locally, or, for important
library updates, in a dedicated branch.

The TeX Live issue is orthogonal: We seriously need a ‘texlive-minimal’
variant, and the full-blown ‘texlive’ must not be depended on by any
other package.

> And if Efraim wished to do a gitlib update after my curl update, what would
> be the protocol? Would he be expected to wait until the hydra build of the
> curl update has finished? Or at least until it has finished on x86_64?
> Otherwise, we would again face the risk of the staging branch becoming
> an update-the-world branch with unforeseen ramifications.

Yeah, for sure.  I have to admit that we may be unable to do much better
until we can finally change the hydra.gnu.org front-end, which is the
main bottleneck here (at least for i686/x86_64.)

Thanks,
Ludo’.

Re: Dealing with mass rebuilds

[Paul van der Walt]

On 2015-10-21 at 18:45, quoth Ludovic Courtès:
> The TeX Live issue is orthogonal: We seriously need a ‘texlive-minimal’
> variant, and the full-blown ‘texlive’ must not be depended on by any
> other package.

Even more orthogonal: can packages be marked as leaves (or roots,
depending on your point of view) that may not be used as inputs?  If not
that sounds like a potentially useful bit of package metadata.

Cheers,
p.

Re: Dealing with mass rebuilds

[Ludovic Courtès]

Paul van der Walt <paul@denknerd.org> skribis:

> On 2015-10-21 at 18:45, quoth Ludovic Courtès:
>> The TeX Live issue is orthogonal: We seriously need a ‘texlive-minimal’
>> variant, and the full-blown ‘texlive’ must not be depended on by any
>> other package.
>
> Even more orthogonal: can packages be marked as leaves (or roots,
> depending on your point of view) that may not be used as inputs?  If not
> that sounds like a potentially useful bit of package metadata.

We could do that, but I’m not sure it’s useful, apart from this very
particular case.  Do you have other uses in mind?

Ludo’.

Re: Dealing with mass rebuilds

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> Andreas Enge <andreas@enge.fr> writes:
>>
>>> andreas pushed a commit to branch master
>>> in repository guix.
>>>
>>> commit 075c3ebd2dc3d8223e23025ceb5026810dfaa98d
>>> Author: Andreas Enge <andreas@enge.fr>
>>> Date:   Sun Oct 18 12:20:02 2015 +0200
>>>
>>>     gnu: curl: Update to 7.45.0.
>>
>> To avoid unnecessary rebuilds on hydra, I reverted this commit on
>> master, and instead pushed it to the 'dbus-update' branch, which will
>> hopefully be merged soon.
>
> We’re talking of ~150 dependent packages, which to me sounded OK¹.

After applying the 'giflib' and 'curl' updates to the 'dbus-update'
branch (and also merging master), the number of queued jobs on that
jobset went from about 1000 to over 2200.  Based on this, and also based
on my past memories of curl security updates, I suspect that this is
another case where "guix refresh -l" drastically underestimated the
number of rebuilds.

      Mark

Re: Dealing with mass rebuilds

[Andreas Enge]

On Thu, Oct 22, 2015 at 02:20:08PM -0400, Mark H Weaver wrote:
> After applying the 'giflib' and 'curl' updates to the 'dbus-update'
> branch (and also merging master), the number of queued jobs on that
> jobset went from about 1000 to over 2200.  Based on this, and also based

This is actually coherent with the numbers given by "guix refresh -l":
143 for curl, 131 for giflib (there may be some overlap, but probably not
too much); and then we need to multiply it by 4 now for our multiple
architectures!

Andreas