bug#57315: guix upgrade --dry-run output is basically useless

bug#57315: guix upgrade --dry-run output is basically useless

[Csepp]

I'd like to figure out what Guix will try to build before I run an
upgrade on my netbook, so I always use --dry-run.  I'm pretty sure in
the past it used to show more information, but today it just showed that
it will download 20 megs, without saying what package that 20 megs are
for, which ones will be built, which ones are downloaded, or anything
useful at all.

And now I see it's building Caja.  Why did it not warn me beforehand?
No idea.

This should go without saying, but this is pretty bad UX.

Is there something that can be done about this?  The upgrade process on
less powerful machines is pretty awful currently.

Side note: I plan to work on a patch that adds an option to upgrade that
keeps everything that would require local building at its previous
version.  Hopefully I won't need to use the --do-not-upgrade option
after that.  Right now upgrading is a multi-hour manual process, which
honestly sucks.  With that patch it would still take a while but at
least it would run automatically.

bug#57315: guix upgrade --dry-run output is basically useless

[bdju via Bug reports for GNU Guix]

On Sun Aug 21, 2022 at 1:39 AM CDT, Csepp wrote:
> This should go without saying, but this is pretty bad UX.
>
> Is there something that can be done about this?  The upgrade process on
> less powerful machines is pretty awful currently.
Agreed. Upgrading on Guix can be pretty horrible. Especially when you
find out you're seemingly the only user of certain packages so you hit a
bunch of build failures that you would've expected to be found and fixed
already, and you have to skip and/or report them and redo the upgrades
repeatedly. On most distros I would upgrade daily without worry, but
with Guix System I end up putting it off a week or two and trying to
plan out a time when I don't mind my CPU being maxed out entirely with
builds and such.

> Side note: I plan to work on a patch that adds an option to upgrade that
> keeps everything that would require local building at its previous
> version.  Hopefully I won't need to use the --do-not-upgrade option
> after that.  Right now upgrading is a multi-hour manual process, which
> honestly sucks.  With that patch it would still take a while but at
> least it would run automatically.

This sounds like a nice feature. I've wished for the ability to use guix
manifests but skip certain packages when there are issues, but if I
could use manifests along with an option like this, that'd accomplish
what I wanted anyway in most cases.

bug#57315: guix upgrade --dry-run output is basically useless

[Ludovic Courtès]

Hi,

"bdju" <bdju@tilde.team> skribis:

> On Sun Aug 21, 2022 at 1:39 AM CDT, Csepp wrote:
>> This should go without saying, but this is pretty bad UX.
>>
>> Is there something that can be done about this?  The upgrade process on
>> less powerful machines is pretty awful currently.
> Agreed. Upgrading on Guix can be pretty horrible. Especially when you
> find out you're seemingly the only user of certain packages so you hit a
> bunch of build failures that you would've expected to be found and fixed
> already, and you have to skip and/or report them and redo the upgrades
> repeatedly. On most distros I would upgrade daily without worry, but
> with Guix System I end up putting it off a week or two and trying to
> plan out a time when I don't mind my CPU being maxed out entirely with
> builds and such.

That’s a pretty bad experience that we should improve (my personal
experience is nicer: I upgrade my system and home roughly weekly and
usually without having to build anything locally).

Now, I think it’s a mistake to “expect” build failures “to be found and
fixed already”: you’re part of the process, you too can find, report,
and fix build failures.  Of course one has other obligations in life
too, but if each one of us does a small part, it’ll work better.

Thanks,
Ludo’.

bug#57315: guix upgrade --dry-run output is basically useless

[Ludovic Courtès]

Hi,

Csepp <raingloom@riseup.net> skribis:

> I'd like to figure out what Guix will try to build before I run an
> upgrade on my netbook, so I always use --dry-run.  I'm pretty sure in
> the past it used to show more information, but today it just showed that
> it will download 20 megs, without saying what package that 20 megs are
> for, which ones will be built, which ones are downloaded, or anything
> useful at all.
>
> And now I see it's building Caja.  Why did it not warn me beforehand?
> No idea.

I understand this is a source of confusion.  It has to do with “grafts”
(which themselves are about applying security updates): if substitutes
for a package are missing, Guix has a partial view of the dependency
graph, which is why it can only tell you about extra builds/downloads
later on in the process (it does report them, only later).

(If you’re curious, see
<https://guix.gnu.org/en/blog/2020/grafts-continued/> for details.)

Thanks,
Ludo’.

bug#57315: guix upgrade --dry-run output is basically useless

[zimoun]

Hi,

Just to mention this report is somehow a duplicate of bug#40612 [1].
Maybe, they could be merged.  WDYT?

1: <https://issues.guix.gnu.org/issue/40612>


On Wed, 31 Aug 2022 at 11:28, Ludovic Courtès <ludo@gnu.org> wrote:

> I understand this is a source of confusion.  It has to do with “grafts”
> (which themselves are about applying security updates): if substitutes
> for a package are missing, Guix has a partial view of the dependency
> graph, which is why it can only tell you about extra builds/downloads
> later on in the process (it does report them, only later).
>
> (If you’re curious, see
> <https://guix.gnu.org/en/blog/2020/grafts-continued/> for details.)

Nice read!  Quoting:

        [...] do a first pass lowering packages to derivations as if
        grafting was disabled, build all these derivations, and then do
        a second pass to determine which packages in the graph need to
        be grafted and to compute the relevant grafting
        derivation. [...] If we reify that second pass to the user
        interface code, it also addresses the user interface issue by
        allowing it to display, possibly, two build plans: the
        “ungrafted” one followed by the grafted one.

Currently, these 2 plans are not well-exposed, IMHO.  What it is
expected (at least by me ;-)) when running “--dry-run” is to have a
picture about what would going on.  For example,

--8<---------------cut here---------------start------------->8---
$ guix package -i opensurge --dry-run
guix package: warning: Your Guix installation is 12 days old.
guix package: warning: Consider running 'guix pull' followed by
'guix package -u' to get up-to-date packages and security updates.

The following package would be installed:
   opensurge 0.5.2.1

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivation would be built:
  /gnu/store/r89hmhbxwm3gs1jl2dhns7gnwvi2k6s1-opensurge-0.5.2.1.drv

42.1 MB would be downloaded
--8<---------------cut here---------------end--------------->8---

What does it mean?  Does it mean the substitute is missing?  Or does it
mean grafting is missing?  Does it mean I am going to download 42.1MB
and then a “quick” application of grafts?  Or does it mean I am going to
build from source the package and these 42.1MB correspond to source and
build-time dependencies?

When I run the option ’--dry-run’, I accept to pay a bit more and then
compute as much as possible of derivations to have the most complete as
possible graph to know beforehand and as accurately as possible:

 1. what I need to download as substitutes
 2. what I need to compile from source
 3. what require grafts, e.g.,

        require 1 graft for openal-1.20.1 ...
        require 4 grafts for sfml-2.5.1 ...
        require 6 grafts for mars-0.7.5.1.c855d04 ...

Sometime, the plan looks exactly like that.  Sometime, it is really
confusing and you have unpleasant surprises when running it.

Obviously, it is not as easy as it appears because of all the dynamic
dependencies are not straightforward to track ahead of time. ;-)
However, I also find the output of ’--dry-run’ not enough reliable and I
use a cross-finger* approach when I update. :-)


Cheers,
simon

*cross-finger approach: Running Debian, I cross my fingers when I
upgrade because many things can break and it is hard to roll back.
Running Guix, I cross my fingers because the dry-run does not tell me
some substitutes can be missing and then Guix automatically launches a
local build… which often ends by… a build failure.  Heh! No free
lunch. ;-)

bug#57315: guix upgrade --dry-run output is basically useless

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Just to mention this report is somehow a duplicate of bug#40612 [1].
> Maybe, they could be merged.  WDYT?

Yes, please.

>> (If you’re curious, see
>> <https://guix.gnu.org/en/blog/2020/grafts-continued/> for details.)
>
> Nice read!  Quoting:
>
>         [...] do a first pass lowering packages to derivations as if
>         grafting was disabled, build all these derivations, and then do
>         a second pass to determine which packages in the graph need to
>         be grafted and to compute the relevant grafting
>         derivation. [...] If we reify that second pass to the user
>         interface code, it also addresses the user interface issue by
>         allowing it to display, possibly, two build plans: the
>         “ungrafted” one followed by the grafted one.
>
> Currently, these 2 plans are not well-exposed, IMHO.

When there are two or more passes, each one is printed as soon as
possible.  So you see “X MiB will be downloaded” or similar messages in
the middle of the process.

> $ guix package -i opensurge --dry-run
> guix package: warning: Your Guix installation is 12 days old.
> guix package: warning: Consider running 'guix pull' followed by
> 'guix package -u' to get up-to-date packages and security updates.
>
> The following package would be installed:
>    opensurge 0.5.2.1
>
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
> The following derivation would be built:
>   /gnu/store/r89hmhbxwm3gs1jl2dhns7gnwvi2k6s1-opensurge-0.5.2.1.drv
>
> 42.1 MB would be downloaded
>
> What does it mean?

That you’ll download 42M of dependencies and then build opensurge.

“The following derivation would be built” is about building derivations
that are not grafts.  For grafts, you would see “The following graft …”,
and only at verbosity level 2 or more (see ‘show-what-to-build’).

[...]

> When I run the option ’--dry-run’, I accept to pay a bit more and then
> compute as much as possible of derivations to have the most complete as
> possible graph to know beforehand and as accurately as possible:
>
>  1. what I need to download as substitutes
>  2. what I need to compile from source
>  3. what require grafts, e.g.,
>
>         require 1 graft for openal-1.20.1 ...
>         require 4 grafts for sfml-2.5.1 ...
>         require 6 grafts for mars-0.7.5.1.c855d04 ...
>
> Sometime, the plan looks exactly like that.  Sometime, it is really
> confusing and you have unpleasant surprises when running it.

By definition, the whole plan cannot be known in advance in the presence
of dynamic dependencies such as grafts, so it’s hard to do better.

The way it’s implemented right now is not optimal strictly speaking in
that we might bail out before we have a complete picture of everything
that’s known statically beforehand.  In practice though I think it’s
doing an okay job from that perspective.

Cheers,
Ludo’.

bug#57315: guix upgrade --dry-run output is basically useless

[zimoun]

Hi,

Thanks for explaining.

On jeu., 01 sept. 2022 at 14:05, Ludovic Courtès <ludo@gnu.org> wrote:

>> Just to mention this report is somehow a duplicate of bug#40612 [1].
>> Maybe, they could be merged.  WDYT?
>
> Yes, please.

Done.


>> $ guix package -i opensurge --dry-run
>> guix package: warning: Your Guix installation is 12 days old.
>> guix package: warning: Consider running 'guix pull' followed by
>> 'guix package -u' to get up-to-date packages and security updates.
>>
>> The following package would be installed:
>>    opensurge 0.5.2.1
>>
>> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>> substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>> The following derivation would be built:
>>   /gnu/store/r89hmhbxwm3gs1jl2dhns7gnwvi2k6s1-opensurge-0.5.2.1.drv
>>
>> 42.1 MB would be downloaded
>>
>> What does it mean?
>
> That you’ll download 42M of dependencies and then build opensurge.
>
> “The following derivation would be built” is about building derivations
> that are not grafts.  For grafts, you would see “The following graft …”,
> and only at verbosity level 2 or more (see ‘show-what-to-build’).

Well, the UI does not appear clear to me.


>> When I run the option ’--dry-run’, I accept to pay a bit more and then
>> compute as much as possible of derivations to have the most complete as
>> possible graph to know beforehand and as accurately as possible:
>>
>>  1. what I need to download as substitutes
>>  2. what I need to compile from source
>>  3. what require grafts, e.g.,
>>
>>         require 1 graft for openal-1.20.1 ...
>>         require 4 grafts for sfml-2.5.1 ...
>>         require 6 grafts for mars-0.7.5.1.c855d04 ...
>>
>> Sometime, the plan looks exactly like that.  Sometime, it is really
>> confusing and you have unpleasant surprises when running it.
>
> By definition, the whole plan cannot be known in advance in the presence
> of dynamic dependencies such as grafts, so it’s hard to do better.
>
> The way it’s implemented right now is not optimal strictly speaking in
> that we might bail out before we have a complete picture of everything
> that’s known statically beforehand.  In practice though I think it’s
> doing an okay job from that perspective.

Well, I understand that using the current implementation, it is not
possible to know the complete plan beforehand.  I agree that for most of
the cases, it is enough.

However, I still miss why it is not possible to compute the whole graph
of derivations.  This complete graph is potentially a bit expensive to
compute but I accept to paid this cost to have a better plan.


Cheers,
simon