What to do when udpating a package ?

What to do when udpating a package ?

[Edouard Klein]

[-- Attachment #1: Type: text/plain, Size: 147 bytes --]

Dear Guix Developers,

I have a few beginner questions.
Attached to this email you will find the "reverse-package" graph of
python-prompt-toolkit.

[-- Attachment #2: reverse.pdf --]
[-- Type: application/pdf, Size: 15269 bytes --]

[-- Attachment #3: Type: text/plain, Size: 4347 bytes --]


I am updating it from version 2.0.7 to version 3.0.5 because
python-questionary (which I'm trying to add to guix) depends on version 3.

After writing the new package definition for version 3.0.5, I tried
building the packages listed by
./pre-inst-env guix refresh --list-dependent python-prompt-toolkit
which if I understand correctly are the leafs of the attached DAG.

This is where I ran into different problems. Here are the questions I
have:

- Some packages would not build, for reasons unrelated to the update of
  python-prompt-toolkit. Namely:
  - fdroidserver@1.1.1  will not build because of a failure of
  python-apache-libcloud@2.4.0
  - python-matplotlib-documentation@3.1.2 and
  python-ipython-documentation@7.9.0 will not build because of a failure
  of texlive-union-51265
  - python-rpy2@3.0.4-1.19868a8 would not build
  All those failure also happen on origin/master on my machine, and are
  therefore unrelated to the changes I made.
  --> Is any failure to build expected on origin/master ?
  --> If so, where can I check the official build success/fail status of
      a package ?
  --> If not, how can I check that my master version of guix is correct
       and is building everything it should build ?
  --> In any case, how can I check that my change does not negatively
       affect these packages ?

- Some of the failing-in-master packages do not depend directly on
  python-prompt toolkit, but they depend on a package that depends on
  etc.
  e.g. fdroidserver@1.1.1 depends on python-androguard@3.2.1 which
  depends on python-ipython@7.9.0.
  --> fdroidserver@1.1.1 fails to build for reasons unrelated to
  python-prompt-toolkit, but both python-androguard@3.2.1 and
  python-ipython@7.9.0. both build with the new version of
  python-prompt-toolkit. Would I be correct in assuming that
  frdoidserver would build with the new version ? I assume not, but I
  would like to be sure

- python-iml@0.6.2 does not build with prompt-toolkit in version 3 (the
  version is explicitly stated as >=2.0 and <2.1 in the Python files), but
  when I add prompt-toolkit-2 to its propagated-inputs, it does.
  Given that python-ipython has prompt-toolkit (implicitly in version
  3) installed in its propagated inputs, this means that any environment
  with python-iml will have prompt-toolkit in both its version 2 and 3
  installed at the same time. I feel uneasy about this.
  --> Is this a problem ?
  --> Should I just trust that if it builds, then everything is
      allright ?

- Same question on a larger scale with r-irkernel@1.1:
  - it depends on python-jupyter-console (failing) which depends on
  python-prompt-toolkit.
  If I pin python-jupyter-console's dependency on python-prompt-toolkit
  to version 2, python-jupyter-console builds.
  The next dependency to fail is then python-widgetsnbextension@3.4.2,
  whose Python files explicitly requires 2.0 <= python-prompt-toolkit <2.1,
  but whose guix package does not (it relies on python-prompt-toolkit
  being a propagated input of a propagated input of a propagated etc.).
  Same for python-ipywidgets@5.2.2.
  Those packages build when I explicitly add python-prompt-toolkit-2 as a
  propagated input.
  Now I feel uneasy about two things:
  --> Is the concurrent presence in the environment of
      python-prompt-toolkit both in version 2 and 3 a problem ?
  --> Is the fact that the graph is now much more connected (2 extra
      edges) a problem ?
  --> Is there a way to check the graph to make the edges as
      sparse as possible (i.e. remove as many edges as possible without
      changing the reachability) ? Would this be something we want ?
      According to me it would because it would make the packages
      definitions shorter and the computations on the graph faster, but I'm
      not sure.
  
I'm sorry if this is addressed in the documentation, I did not find the
relevant parts.

guix graph is awesome, it makes working with the dependencies much more
intuitive. This email is a mess to anyone not looking at the graph at
the same time, but I hope is quite understandable to somebody following
along on the graph at the same time.

Last question:
--> Should I also check the build status of the packages for which
python-prompt-toolkit or its dependents are native-inputs ? Why, why not
?

Thanks in advance,

Cheers,

Edouard

Re: What to do when udpating a package ?

[Julien Lepiller]

Le 15 mai 2020 03:20:06 GMT-04:00, Edouard Klein <edk@beaver-labs.com> a écrit :
>Dear Guix Developers,
>
>I have a few beginner questions.
>Attached to this email you will find the "reverse-package" graph of
>python-prompt-toolkit.

Hi,

In general, some packages on master may fail to build. We try to fix these, but it's not always easy.

When updating a package, you should make sure that its dependents all build or did not build before your changes. Fix those who now fail to build.

I don't think having both versions is a good thing because it will create conflicts when installing (have you tried to install a package that propagates both to a profile?). It would be ok if they had no file in common but I doubt it is the case. For any package that requires version 2, make sure its dependencies only use version 2, or update the package if the newer version can support version 3. It's not always easy to find the right order of upgrades, as you should make sure nothing is broken in between patches.

Relying on propagated inputs to provide a dependency is going to simplify the graphs, but not the work of other maintainers who will have to investigate how the dependency is provided, so I don't think it's a good idea.

You should rebuild every dependent, even those who only depend on the package for native-inputs, since there can be an error an any point (though less likely).

I hope I answered your questions. Your message was split into two multipart sections and my client wasn't able to cite the interesting part, which makes it hard for me to check what your questions were while typing my answer.

Re: What to do when udpating a package ?

[zimoun]

Dear Edouard,

In complement to what Julien said. :-)


On Fri, 15 May 2020 at 13:36, Edouard Klein <edk@beaver-labs.com> wrote:

> - Some packages would not build, for reasons unrelated to the update of
>   python-prompt-toolkit. Namely:
>   - fdroidserver@1.1.1  will not build because of a failure of
>   python-apache-libcloud@2.4.0
>   - python-matplotlib-documentation@3.1.2 and
>   python-ipython-documentation@7.9.0 will not build because of a failure
>   of texlive-union-51265
>   - python-rpy2@3.0.4-1.19868a8 would not build
>   All those failure also happen on origin/master on my machine, and are
>   therefore unrelated to the changes I made.
>   --> Is any failure to build expected on origin/master ?

No, it is not expected.
As Forrest Gump says: "It happens, sometimes." :-)


>   --> If so, where can I check the official build success/fail status of
>       a package ?

Well, the link is ci.guix.gnu.org then top right, for example the query:

   spec:guix-master system:x86_64-linux python-apache-libcloud

But personally I do not always understand what Cuirass reports.
Therefore, I prefer to use data.guix.gnu.org then for example:

http://data.guix.gnu.org/repository/1/branch/master/package/fdroidserver/output-history

I do not know the correct way to reach this webpage so I type directly
the URL -- well to be exact, I have an Emacs interactive function so I
type M-x my/guix-data fdroidserver RET and it opens the above webpage.

Then I click to one "Failed dependency" and I arrive to:

http://data.guix.gnu.org/build-server/1/build?derivation_file_name=/gnu/store/ccrk10g4vpzf6nk7x8j1a36s0b4z0w2l-fdroidserver-1.1.1.drv

and I can click to the failing derivation and then I arrive to
ci.guix.gnu.org and I can give a look to the raw log.


Well, I do not know if it is the correct way, but it is how I am doing
time to time to see what happens on the build farm.  Be careful of
your current Guix version and the Guix version on ci you are
examining.


>   --> If not, how can I check that my master version of guix is correct
>        and is building everything it should build ?

I do not understand what you mean.
"guix weather" to see if the substitute is available.
"guix challenge" to compare your own builds to the builds on substitutes.


>   --> In any case, how can I check that my change does not negatively
>        affect these packages ?

Which packages?  The ones which do not currently build?


> - Some of the failing-in-master packages do not depend directly on
>   python-prompt toolkit, but they depend on a package that depends on
>   etc.
>   e.g. fdroidserver@1.1.1 depends on python-androguard@3.2.1 which
>   depends on python-ipython@7.9.0.
>   --> fdroidserver@1.1.1 fails to build for reasons unrelated to
>   python-prompt-toolkit, but both python-androguard@3.2.1 and
>   python-ipython@7.9.0. both build with the new version of
>   python-prompt-toolkit. Would I be correct in assuming that
>   frdoidserver would build with the new version ? I assume not, but I
>   would like to be sure

I miss something about what depends on what. :-)
I do not have really the graph under my eyes. ;-)


> - python-iml@0.6.2 does not build with prompt-toolkit in version 3 (the
>   version is explicitly stated as >=2.0 and <2.1 in the Python files), but
>   when I add prompt-toolkit-2 to its propagated-inputs, it does.
>   Given that python-ipython has prompt-toolkit (implicitly in version
>   3) installed in its propagated inputs, this means that any environment
>   with python-iml will have prompt-toolkit in both its version 2 and 3
>   installed at the same time. I feel uneasy about this.
>   --> Is this a problem ?
>   --> Should I just trust that if it builds, then everything is
>       allright ?

No, it should pass the tests too.  If there is tests. :-)
If not, you should install it locally and try it.
Because the package could build but should have runtime issues.


>   --> Is there a way to check the graph to make the edges as
>       sparse as possible (i.e. remove as many edges as possible without
>       changing the reachability) ? Would this be something we want ?
>       According to me it would because it would make the packages
>       definitions shorter and the computations on the graph faster, but I'm
>       not sure.

What do you mean by "reachability"?
There is a new feature to "guix graph": '--path'.  You can find the
shortest path from one package to another, e.g.,

  guix graph --path guix-jupyter python

What do you mean by "the edges as sparse as possible"?



All the best,
simon

Re: What to do when udpating a package ?

[Edouard Klein]

Hi Julien,

Thank you for your answer.

Julien Lepiller writes:

> Le 15 mai 2020 03:20:06 GMT-04:00, Edouard Klein <edk@beaver-labs.com> a écrit :
>>Dear Guix Developers,
>>
>>I have a few beginner questions.
>>Attached to this email you will find the "reverse-package" graph of
>>python-prompt-toolkit.
>
> Hi,
>
> In general, some packages on master may fail to build. We try to fix these, but it's not always easy.
>
> When updating a package, you should make sure that its dependents all
> build or did not build before your changes. Fix those who now fail to
> build.

OK, got it. I'll avoid touching those that fail to build on the build farm.

>
> I don't think having both versions is a good thing because it will create conflicts when installing (have you tried to install a package that propagates both to a profile?). It would be ok if they had no file in common but I doubt it is the case. For any package that requires version 2, make sure its dependencies only use version 2, or update the package if the newer version can support version 3. It's not always easy to find the right order of upgrades, as you should make sure nothing is broken in between patches.
>
I did not try to install them, and you were correct, here is what I get
when I try to install python-iml:

guix install: error: profile contains conflicting entries for python-prompt-toolkit
guix install: error:   first entry: python-prompt-toolkit@3.0.5 /gnu/store/80lzvbzvfp4226ic7czhch4p0mlsdwlv-python-prompt-toolkit-3.0.5
guix install: error:    ... propagated from python-ipython@7.9.0
guix install: error:    ... propagated from python-iml@0.6.2
guix install: error:   second entry: python-prompt-toolkit@2.0.7 /gnu/store/0k7a0yp3b2sqqj8jhl7vp3cabb0x2mwd-python-prompt-toolkit-2.0.7
guix install: error:    ... propagated from python-iml@0.6.2
hint: You cannot have two different versions or variants of `python-iml' in the same profile.

The problem is that python-iml depends on both python-ipython (which
accepts python-prompt-toolkit 3) and python-prompt-toolkit-2. Looking at
the github repo, the last update was in 2018, I don't think we'll see an
update soon.

I could pin python-ipython to python-prompt-toolkit-2, but that would
just delay the problem and put it on somebody else's lap to let
python-ipython move to python-prompt-toolkit 3.

As I was typing a question I ctrl-Fed for 'variants' in the doc and
ended up learning about package-input-rewriting. I will try to make
python-iml depend on python-ipython, but with the prompt-toolkit input
replaced with its version 2 on the fly. I think it makes sense. Is it the correct
way to do what needs to be done ?

> Relying on propagated inputs to provide a dependency is going to
> simplify the graphs, but not the work of other maintainers who will
> have to investigate how the dependency is provided, so I don't think
> it's a good idea.

I understood your sentence as saying that relying on propagated inputs
of propagated inputs is not a good idea and that dependencies are better
explicitly stated in the guix package. Is that correct ?


>
> You should rebuild every dependent, even those who only depend on the
> package for native-inputs, since there can be an error an any point
> (though less likely).

OK, I'll try that next when I'll have gotten python-iml to build.

>
> I hope I answered your questions. Your message was split into two
> multipart sections and my client wasn't able to cite the interesting
> part, which makes it hard for me to check what your questions were
> while typing my answer.

You have indeed answered a lot of them, thank you very much :) Sorry
about the multipart stuff, I don't know how to configure my client
(mu4e) not to do that. I'll look into it.

Cheers,

Edouard.

Re: What to do when udpating a package ?

[Edouard Klein]

zimoun writes:

> Dear Edouard,
>
> In complement to what Julien said. :-)
>
>
> On Fri, 15 May 2020 at 13:36, Edouard Klein <edk@beaver-labs.com> wrote:
>
>> - Some packages would not build, for reasons unrelated to the update of
>>   python-prompt-toolkit. Namely:
>>   - fdroidserver@1.1.1  will not build because of a failure of
>>   python-apache-libcloud@2.4.0
>>   - python-matplotlib-documentation@3.1.2 and
>>   python-ipython-documentation@7.9.0 will not build because of a failure
>>   of texlive-union-51265
>>   - python-rpy2@3.0.4-1.19868a8 would not build
>>   All those failure also happen on origin/master on my machine, and are
>>   therefore unrelated to the changes I made.
>>   --> Is any failure to build expected on origin/master ?
>
> No, it is not expected.
> As Forrest Gump says: "It happens, sometimes." :-)
>

I was worried I had misconfigured my local version of guix.

>
>>   --> If so, where can I check the official build success/fail status of
>>       a package ?
>
> Well, the link is ci.guix.gnu.org then top right, for example the query:
>
>    spec:guix-master system:x86_64-linux python-apache-libcloud
>
> But personally I do not always understand what Cuirass reports.
> Therefore, I prefer to use data.guix.gnu.org then for example:
>
> http://data.guix.gnu.org/repository/1/branch/master/package/fdroidserver/output-history
>
> I do not know the correct way to reach this webpage so I type directly
> the URL -- well to be exact, I have an Emacs interactive function so I
> type M-x my/guix-data fdroidserver RET and it opens the above webpage.
>
> Then I click to one "Failed dependency" and I arrive to:
>
> http://data.guix.gnu.org/build-server/1/build?derivation_file_name=/gnu/store/ccrk10g4vpzf6nk7x8j1a36s0b4z0w2l-fdroidserver-1.1.1.drv
>
> and I can click to the failing derivation and then I arrive to
> ci.guix.gnu.org and I can give a look to the raw log.
>
I could not find the link to the raw log, but having access to the
"official" build status is a huge relief, as I can stop worrying that
the build failure is my fault. This is exactly what I was looking for.
Thank you !


>
> Well, I do not know if it is the correct way, but it is how I am doing
> time to time to see what happens on the build farm.  Be careful of
> your current Guix version and the Guix version on ci you are
> examining.
>
>
>>   --> If not, how can I check that my master version of guix is correct
>>        and is building everything it should build ?
>
> I do not understand what you mean.
> "guix weather" to see if the substitute is available.
> "guix challenge" to compare your own builds to the builds on substitutes.
>
>

I was asking how to make sure that my local guix is OK (I had trouble
getting it to work).

>>   --> In any case, how can I check that my change does not negatively
>>        affect these packages ?
>
> Which packages?  The ones which do not currently build?

Yeah, I'm worried maybe once the current build failure cause is addressed, my
change will prevent them from building.

>
>
>> - Some of the failing-in-master packages do not depend directly on
>>   python-prompt toolkit, but they depend on a package that depends on
>>   etc.
>>   e.g. fdroidserver@1.1.1 depends on python-androguard@3.2.1 which
>>   depends on python-ipython@7.9.0.
>>   --> fdroidserver@1.1.1 fails to build for reasons unrelated to
>>   python-prompt-toolkit, but both python-androguard@3.2.1 and
>>   python-ipython@7.9.0. both build with the new version of
>>   python-prompt-toolkit. Would I be correct in assuming that
>>   frdoidserver would build with the new version ? I assume not, but I
>>   would like to be sure
>
> I miss something about what depends on what. :-)
> I do not have really the graph under my eyes. ;-)
>
>
>> - python-iml@0.6.2 does not build with prompt-toolkit in version 3 (the
>>   version is explicitly stated as >=2.0 and <2.1 in the Python files), but
>>   when I add prompt-toolkit-2 to its propagated-inputs, it does.
>>   Given that python-ipython has prompt-toolkit (implicitly in version
>>   3) installed in its propagated inputs, this means that any environment
>>   with python-iml will have prompt-toolkit in both its version 2 and 3
>>   installed at the same time. I feel uneasy about this.
>>   --> Is this a problem ?
>>   --> Should I just trust that if it builds, then everything is
>>       allright ?
>
> No, it should pass the tests too.  If there is tests. :-)
> If not, you should install it locally and try it.

Yeah, installing does not work :D See my answer to Julien

> Because the package could build but should have runtime issues.
>
>
>>   --> Is there a way to check the graph to make the edges as
>>       sparse as possible (i.e. remove as many edges as possible without
>>       changing the reachability) ? Would this be something we want ?
>>       According to me it would because it would make the packages
>>       definitions shorter and the computations on the graph faster, but I'm
>>       not sure.
>
> What do you mean by "reachability"?
> There is a new feature to "guix graph": '--path'.  You can find the
> shortest path from one package to another, e.g.,
>
>   guix graph --path guix-jupyter python
>
> What do you mean by "the edges as sparse as possible"?
>
>
So if A depends on B and C, and B also depends on C, which is preferable
as far as explicit input declarations in the packages code go:
--
A->B;
B->C;
-- or
A->B;
A->C;
B->C;
--
The reachability (in the graph theoretical sense
https://en.wikipedia.org/wiki/Reachability) is the same, but one graph
has one edge less and is thus "minimal". If I understood Julien correctly he seems to think
that the fully connected case is better (easier maintainability).


>
> All the best,
> simon

Thank you for your time,

Cheers,

Edouard.

Re: What to do when udpating a package ?

[Julien Lepiller]

Le 15 mai 2020 11:32:31 GMT-04:00, Edouard Klein <edk@beaver-labs.com> a écrit :
>Hi Julien,
>
>Thank you for your answer.
>
>Julien Lepiller writes:
>
>> Le 15 mai 2020 03:20:06 GMT-04:00, Edouard Klein
><edk@beaver-labs.com> a écrit :
>>>Dear Guix Developers,
>>>
>>>I have a few beginner questions.
>>>Attached to this email you will find the "reverse-package" graph of
>>>python-prompt-toolkit.
>>
>> Hi,
>>
>> In general, some packages on master may fail to build. We try to fix
>these, but it's not always easy.
>>
>> When updating a package, you should make sure that its dependents all
>> build or did not build before your changes. Fix those who now fail to
>> build.
>
>OK, got it. I'll avoid touching those that fail to build on the build
>farm.
>
>>
>> I don't think having both versions is a good thing because it will
>create conflicts when installing (have you tried to install a package
>that propagates both to a profile?). It would be ok if they had no file
>in common but I doubt it is the case. For any package that requires
>version 2, make sure its dependencies only use version 2, or update the
>package if the newer version can support version 3. It's not always
>easy to find the right order of upgrades, as you should make sure
>nothing is broken in between patches.
>>
>I did not try to install them, and you were correct, here is what I get
>when I try to install python-iml:
>
>guix install: error: profile contains conflicting entries for
>python-prompt-toolkit
>guix install: error:   first entry: python-prompt-toolkit@3.0.5
>/gnu/store/80lzvbzvfp4226ic7czhch4p0mlsdwlv-python-prompt-toolkit-3.0.5
>guix install: error:    ... propagated from python-ipython@7.9.0
>guix install: error:    ... propagated from python-iml@0.6.2
>guix install: error:   second entry: python-prompt-toolkit@2.0.7
>/gnu/store/0k7a0yp3b2sqqj8jhl7vp3cabb0x2mwd-python-prompt-toolkit-2.0.7
>guix install: error:    ... propagated from python-iml@0.6.2
>hint: You cannot have two different versions or variants of
>`python-iml' in the same profile.
>
>The problem is that python-iml depends on both python-ipython (which
>accepts python-prompt-toolkit 3) and python-prompt-toolkit-2. Looking
>at
>the github repo, the last update was in 2018, I don't think we'll see
>an
>update soon.
>
>I could pin python-ipython to python-prompt-toolkit-2, but that would
>just delay the problem and put it on somebody else's lap to let
>python-ipython move to python-prompt-toolkit 3.
>
>As I was typing a question I ctrl-Fed for 'variants' in the doc and
>ended up learning about package-input-rewriting. I will try to make
>python-iml depend on python-ipython, but with the prompt-toolkit input
>replaced with its version 2 on the fly. I think it makes sense. Is it
>the correct
>way to do what needs to be done ?

Sounds like a plan. Anotger possiblity is to introduce a patch to have python-iml support the latest version and send it upstream too.

>
>> Relying on propagated inputs to provide a dependency is going to
>> simplify the graphs, but not the work of other maintainers who will
>> have to investigate how the dependency is provided, so I don't think
>> it's a good idea.
>
>I understood your sentence as saying that relying on propagated inputs
>of propagated inputs is not a good idea and that dependencies are
>better
>explicitly stated in the guix package. Is that correct ?

Yes, that's what I meant. If the package explicitely requires something, we should provide it in the recipe, even if that's superfluous. If it's not listed as a dependency, then it's not necessary. I don't think this is an established rule, but this is what makes more sense to me.

>
>>
>> You should rebuild every dependent, even those who only depend on the
>> package for native-inputs, since there can be an error an any point
>> (though less likely).
>
>OK, I'll try that next when I'll have gotten python-iml to build.
>
>>
>> I hope I answered your questions. Your message was split into two
>> multipart sections and my client wasn't able to cite the interesting
>> part, which makes it hard for me to check what your questions were
>> while typing my answer.
>
>You have indeed answered a lot of them, thank you very much :) Sorry
>about the multipart stuff, I don't know how to configure my client
>(mu4e) not to do that. I'll look into it.

Don't worry too much about it, it's only inconvenient when I'm on my phone :)

>
>Cheers,
>
>Edouard.

Re: What to do when udpating a package ?

[zimoun]

On Fri, 15 May 2020 at 17:43, Edouard Klein <edk@beaver-labs.com> wrote:

> I could not find the link to the raw log, but having access to the
> "official" build status is a huge relief, as I can stop worrying that
> the build failure is my fault. This is exactly what I was looking for.
> Thank you !

I have cheated a bit because with this example, there is not raw -- I
do not know why.  But you get the principle. :-)


> >>   --> Is there a way to check the graph to make the edges as
> >>       sparse as possible (i.e. remove as many edges as possible without
> >>       changing the reachability) ? Would this be something we want ?
> >>       According to me it would because it would make the packages
> >>       definitions shorter and the computations on the graph faster, but I'm
> >>       not sure.
> >
> > What do you mean by "reachability"?
> > There is a new feature to "guix graph": '--path'.  You can find the
> > shortest path from one package to another, e.g.,
> >
> >   guix graph --path guix-jupyter python
> >
> > What do you mean by "the edges as sparse as possible"?
> >
> >
> So if A depends on B and C, and B also depends on C, which is preferable
> as far as explicit input declarations in the packages code go:
> --

(1)
> A->B;
> B->C;

(2)
> A->B;
> A->C;
> B->C;

If A depends on B *and* C, then (1) should not work.
Well, it depends on how A depends* on C and what C propagates.
If I understand correctly.

*build-time or runtime.


> The reachability (in the graph theoretical sense
> https://en.wikipedia.org/wiki/Reachability) is the same, but one graph
> has one edge less and is thus "minimal". If I understood Julien correctly he seems to think
> that the fully connected case is better (easier maintainability).

Well, does "guix graph --path" cover your expectation about reachability?
Because "guix graph --path" computes the shortest path -- graph theory
meaning -- from s to t.  So if the path is not empty, then t is
reachable from s.


All the best,
simon