Questions about packaging

Questions about packaging

[Tanguy Le Carrour]

Hi Guix

I'm currently working on packaging python-poetry and, unsurprisingly,
I have questions! :-)
I guess some of them have already been answered somewhere, so don't
hesitate to point me to a previous thread or post…

1) Updating a package
What am I suppose to do when updating a package to a different minor or
major version? This could break depending packages if they have strict
depedencies.

For instance, when I run:
    ./pre-inst-env guix build python-poetry
I get:
    error: Could not find suitable distribution for
    Requirement.parse('cachecontrol[filecache]<0.13.0,>=0.12.4')

So I would have to update python-cachecontrol from 0.11.6 to 0.12.5.
Should I create a python-cachecontrol-0.11.6 and fix all the packages
that depend on it? Only the one that would break?

Btw, python-cachecontrol seems to be broken. I'll work on that. But
before I'll have to find an answer to question 3.

2) Getting archive from pypi
This one has been bothering me for a long time, and was confirmed few
days ago by a comment I read in the package definition of python-cachecontrol:
"Pypi does not have tests".
Has I understand it, to make sure that a package works with the
dependencies provided by the distrubution (Guix), tests must pass!
Most of the time, tests are not in the python package downloaded from pypi.
Actually, I think that they should not be. They should be in a `tests/` folder
in the sources. But that's a different topic.
So I guess that one should always make sure that the tests can be
executed from the Pypi download, or use Git to get the sources.
Any thoughts?

3) Deactivating tests on purpose
In my case, python-poetry depends on python-cachecontrol which tests
depend (for some weird reasons) on Cherrypy. Cherrypy is a (large) web framework
and has a lot of dependencies!
In that case, is it OK to disable python-cachecontrol's tests?
Or would it be better to ask one of the project to drop one of the
dependency?! (Poetry dropping CacheControl, or CacheControl dropping
CherryPy)

4) Submitting an incomplete patch series
Having run into those problems, I'm a bit stuck with a pile of patches.
Some of them could be submitted. Should I submit them separatly?

I'm really looking forward to reading your answers!

-- 
Tanguy

Re: Questions about packaging

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 2964 bytes --]

Hi,

On Wed, 9 Oct 2019 11:56:33 +0200
Tanguy Le Carrour <tanguy@bioneland.org> wrote:

> 1) Updating a package
> So I would have to update python-cachecontrol from 0.11.6 to 0.12.5.
> Should I create a python-cachecontrol-0.11.6 and fix all the packages
> that depend on it? Only the one that would break?

The latter.  That's one of the things we do at Guix but I would not do at work.

> Btw, python-cachecontrol seems to be broken. I'll work on that. But
> before I'll have to find an answer to question 3.

Then the answer is to update it, and to update everything else that
depends on it (since it didn't work anyway, what's the harm?  The situation
can only improve).

But it seems the breakage is quite generic:

>Failed: [pytest] section in setup.cfg files is no longer supported, change to [tool:pytest] instead.
>command "py.test" "-vv" failed with status 1
>note: keeping build directory `/tmp/guix-build-python-cachecontrol-0.11.6.drv-0'

On Wed, 9 Oct 2019 11:56:33 +0200
Tanguy Le Carrour <tanguy@bioneland.org> wrote:

> As I understand it, to make sure that a package works with the
> dependencies provided by the distrubution (Guix), tests must pass!

Well, it's better if the tests pass, yes.  If the tests fail that's definitely
bad.  If you absolutely can't get the tests to execute in the first place, let's
talk about it (with upstream if necessary).

> So I guess that one should always make sure that the tests can be
> executed from the Pypi download, or use Git to get the sources.

I'd use git (and a tag).  There's no downside that I can see.

> 3) Deactivating tests on purpose
> In my case, python-poetry depends on python-cachecontrol which tests
> depend (for some weird reasons) on Cherrypy. Cherrypy is a (large)
> web framework and has a lot of dependencies!
> In that case, is it OK to disable python-cachecontrol's tests?

If it's only use or two tests that require cherrypy, maybe disable those
if they aren't important (by using "substitute*").

> Or would it be better to ask one of the project to drop one of the
> dependency?! (Poetry dropping CacheControl, or CacheControl dropping
> CherryPy)

That would be a lot better long term--additionally, you could do that in
any case.  Usually upstream has no idea of the huge dependency tree they
cause, so you first have to convince them why it's a bad idea to do that.

> 4) Submitting an incomplete patch series
> Having run into those problems, I'm a bit stuck with a pile of
> patches. Some of them could be submitted. Should I submit them
> separatly?

If they can be used on their own, I post a normal patch series with those
(it usually has a subject "Prepare for ..." or similar).

Otherwise my own personal policy is if keeping what I did in my head is too
difficult, I submit a "WIP" series.  If there are some patches of that
series that can be merged, I usually comment on those additionally.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Questions about packaging

[Tanguy Le Carrour]

Hi Danny !

Thanks for your answer.


Le 10/10, Danny Milosavljevic a écrit :
> > 1) Updating a package
> > So I would have to update python-cachecontrol from 0.11.6 to 0.12.5.
> > Should I create a python-cachecontrol-0.11.6 and fix all the packages
> > that depend on it? Only the one that would break?
> 
> The latter.  That's one of the things we do at Guix but I would not do at work.

OK. I'll do that!
My next question would be "when would someone create a versionned package name?"
Like for instance `openjdk`?


> > Btw, python-cachecontrol seems to be broken. I'll work on that. But
> > before I'll have to find an answer to question 3.
> 
> Then the answer is to update it, and to update everything else that
> depends on it (since it didn't work anyway, what's the harm?  The situation
> can only improve).

"The situation can only improve". Can you believe I didn't think of
that! ^_^'
This makes total sense! Thanks.


> On Wed, 9 Oct 2019 11:56:33 +0200
> Tanguy Le Carrour <tanguy@bioneland.org> wrote:
> 
> > As I understand it, to make sure that a package works with the
> > dependencies provided by the distrubution (Guix), tests must pass!
> 
> Well, it's better if the tests pass, yes.  If the tests fail that's definitely
> bad.  If you absolutely can't get the tests to execute in the first place, let's
> talk about it (with upstream if necessary).

OK. I'll push the discussion upstream.


> > So I guess that one should always make sure that the tests can be
> > executed from the Pypi download, or use Git to get the sources.
> 
> I'd use git (and a tag).  There's no downside that I can see.

Neither can I! I'll do that.


Thanks again for your time and advice !

-- 
Tanguy

Re: Questions about packaging

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 2846 bytes --]

Hi,

On Fri, 11 Oct 2019 09:42:00 +0200
Tanguy Le Carrour <tanguy@bioneland.org> wrote:

> Le 10/10, Danny Milosavljevic a écrit :
> > > 1) Updating a package
> > > So I would have to update python-cachecontrol from 0.11.6 to 0.12.5.
> > > Should I create a python-cachecontrol-0.11.6 and fix all the packages
> > > that depend on it? Only the one that would break?  
> > 
> > The latter.  That's one of the things we do at Guix but I would not do at work.  
> 
> OK. I'll do that!
> My next question would be "when would someone create a versionned package name?"

If it's unavoidable.  One of the reasons is that we don't want to increase our
maintenance and testing burden unduly and thus would not want 453 versions of
the same package available at the same time (chances are that some combination
would not have been tested and/or not work).

But there are professional engineering standards which allow you to specify 
which versions of thing A can go with which other versions of thing B.

The most well-known type of that in software is semantic versioning.

So it depends on what kind of versioning scheme the project/package in question
uses.  If it does use semantic versioning or similar then packages which have
changes making it fundamentally incompatible with what came before will increase
their major version, basically making that an independent package (Debian for
example has the major version in that sense in their package NAMES for
libraries).

For example Python 3 is a new major version that is fundamentally incompatible
with Python 2 (cannot use the former as a drop-in replacement for the latter).

Therefore, there are packages for both Python 2 and Python 3, and for
many Python extensions, packages for a Python 2 and a Python 3 variant each. 

Eventually in some years, there will be no users (other packages) of Python 2
anymore.  Once that comes to pass we can drop Python 2.  What we can't do is
replace Python 2 by Python 3 in those user packages--it would break them.

It is necessary that such a versioning "judgement" mechanism be in place
for a package if you ever want to change your package in a backward-incompatible
manner if it has any users.

For a funny example where they didn't use such a mechanism, one HTTP header
entry is called "Referer", including the typo, for all eternity.  (it just
wasn't worth changing it--the knock-on-effect would have been enormous)

> Like for instance `openjdk`?

Openjdk (icedtea) is bootstrapped using the preceding major version of
each version.  icedtea-8 is built by icedtea-7, which is built by
icedtea-6, which is built by jamvm and GNU classpath, and so on, until
we end up at a C compiler interpreted by GNU Mes.  In this way we don't
have binary blobs anywhere but in GNU Mes at the bottom of it.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Questions about packaging

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

Also, in

(define-public icedtea-6
  (package
[...]
))

"icedtea-6" is a variable name (in the programming language Guile).

The package name is there:

[...]
(package
  (name "icedtea")  <-----
)

We don't have version numbers in package names--but you can request
a specific version by 

$ guix package -i xxx@1.2.3
                  ^^^ ^^^^^ package version
                  package name

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Questions about packaging

[Tanguy Le Carrour]

Le 10/12, Danny Milosavljevic a écrit :
> Also, in
> 
> (define-public icedtea-6
>   (package
> [...]
> ))
> 
> "icedtea-6" is a variable name (in the programming language Guile).
> 
> The package name is there:
> 
> [...]
> (package
>   (name "icedtea")  <-----
> )
> 
> We don't have version numbers in package names--but you can request
> a specific version by 
> 
> $ guix package -i xxx@1.2.3
>                   ^^^ ^^^^^ package version
>                   package name

I'll try to use the vocabulary (variable/package) properly from now on!


-- 
Tanguy

Re: Questions about packaging

[Tanguy Le Carrour]

Hi Danny


Le 10/12, Danny Milosavljevic a écrit :
> On Fri, 11 Oct 2019 09:42:00 +0200
> Tanguy Le Carrour <tanguy@bioneland.org> wrote:
> 
> > Le 10/10, Danny Milosavljevic a écrit :
> > > > 1) Updating a package
> > > > So I would have to update python-cachecontrol from 0.11.6 to 0.12.5.
> > > > Should I create a python-cachecontrol-0.11.6 and fix all the packages
> > > > that depend on it? Only the one that would break?  
> > > 
> > > The latter.  That's one of the things we do at Guix but I would not do at work.  
> > 
> > OK. I'll do that!
> > My next question would be "when would someone create a versionned package name?"
> 
> If it's unavoidable.  One of the reasons is that we don't want to increase our
> maintenance and testing burden unduly and thus would not want 453 versions of
> the same package available at the same time (chances are that some combination
> would not have been tested and/or not work).
> 
> But there are professional engineering standards which allow you to specify 
> which versions of thing A can go with which other versions of thing B.
> 
> The most well-known type of that in software is semantic versioning.
> 
> So it depends on what kind of versioning scheme the project/package in question
> uses.  If it does use semantic versioning or similar then packages which have
> changes making it fundamentally incompatible with what came before will increase
> their major version, basically making that an independent package (Debian for
> example has the major version in that sense in their package NAMES for
> libraries).

Shame on me!
I'm perfectly aware of SemVer [1], it's "just" that I mixed up the
problem with python-cachecontrol and the other one with pytest that was
shadowed by python-cachecontrol!
So, yes, there is no version problem going from 0.11.6 to 0.12.5, has
it's only a change of minor version.
The pytest problem that I did not mention was a major change, with deprecation
of some hooks (pytest_namespace).

In Guix, we have Pytest 4.4.2, but the latest release is 5.2.1.
In that case, would it make sense to define a new "versionned" public
variable for python-pytest? Would I create python-pytest5? Or would I
rename python-pytest to python-pytest4 and add python-pytest for Pytest
5? In this later case, would I have to "fix" all the packages that were
using python-pytest?!

[1]: https://semver.org/

> Eventually in some years, there will be no users (other packages) of Python 2
> anymore.  Once that comes to pass we can drop Python 2.  What we can't do is
> replace Python 2 by Python 3 in those user packages--it would break them.

"in some years" will be next year [2]. But I guess Python2 will still be
around for some more years.

[2]: https://www.python.org/dev/peps/pep-0373/


Thanks again for the time you spent clarifying all this for me!


-- 
Tanguy

Re: Questions about packaging

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

Hello Tanguy,

Tanguy Le Carrour <tanguy@bioneland.org> writes:

> In Guix, we have Pytest 4.4.2, but the latest release is 5.2.1.
> In that case, would it make sense to define a new "versionned" public
> variable for python-pytest? Would I create python-pytest5? Or would I
> rename python-pytest to python-pytest4 and add python-pytest for Pytest
> 5? In this later case, would I have to "fix" all the packages that were
> using python-pytest?!

Pytest has 4457 dependent packages and can thus only be changed on the
'core-updates' branch.  So the recommended way is indeed to define a new
public 'python-pytest-5' variable, and add that as an input for the
packages that require 5.x.

On the next 'core-updates' cycle, Pytest 5 will then become the default,
and we will have to fix all packages that does not work with that
version :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Questions about packaging

[Tanguy Le Carrour]

Hi Marius,


Le 10/18, Marius Bakke a écrit :
> Tanguy Le Carrour <tanguy@bioneland.org> writes:
> > In Guix, we have Pytest 4.4.2, but the latest release is 5.2.1.
> > In that case, would it make sense to define a new "versionned" public
> > variable for python-pytest? Would I create python-pytest5? Or would I
> > rename python-pytest to python-pytest4 and add python-pytest for Pytest
> > 5? In this later case, would I have to "fix" all the packages that were
> > using python-pytest?!
> 
> Pytest has 4457 dependent packages and can thus only be changed on the
> 'core-updates' branch.  So the recommended way is indeed to define a new
> public 'python-pytest-5' variable, and add that as an input for the
> packages that require 5.x.

OK… sounds like something I can do!


> On the next 'core-updates' cycle, Pytest 5 will then become the default,
> and we will have to fix all packages that does not work with that
> version :-)

I guess those broken packages will show up as "red" in Cuirass!?
Would they have to be fixed, or just told to use python-pytest-4?

Is there a way to build those packages on my machine? What a
script to find and build all the packages dependent on pytest would look like?
I mean, using LISP/Scheme/Guile, as I think I could come up with a
"manual" solution with `guix graph --type=reverse-package`.


Regards,

-- 
Tanguy

Re: Questions about packaging

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1687 bytes --]

Hello,

Tanguy Le Carrour <tanguy@bioneland.org> ezt írta (időpont: 2019. okt. 21.,
Hét 11:24):

> Hi Marius,
>
>
> Le 10/18, Marius Bakke a écrit :
> > Tanguy Le Carrour <tanguy@bioneland.org> writes:
> > > In Guix, we have Pytest 4.4.2, but the latest release is 5.2.1.
> > > In that case, would it make sense to define a new "versionned" public
> > > variable for python-pytest? Would I create python-pytest5? Or would I
> > > rename python-pytest to python-pytest4 and add python-pytest for Pytest
> > > 5? In this later case, would I have to "fix" all the packages that were
> > > using python-pytest?!
> >
> > Pytest has 4457 dependent packages and can thus only be changed on the
> > 'core-updates' branch.  So the recommended way is indeed to define a new
> > public 'python-pytest-5' variable, and add that as an input for the
> > packages that require 5.x.
>
> OK… sounds like something I can do!
>
>
> > On the next 'core-updates' cycle, Pytest 5 will then become the default,
> > and we will have to fix all packages that does not work with that
> > version :-)
>
> I guess those broken packages will show up as "red" in Cuirass!?
> Would they have to be fixed, or just told to use python-pytest-4?
>
> Is there a way to build those packages on my machine? What a
> script to find and build all the packages dependent on pytest would look
> like?
> I mean, using LISP/Scheme/Guile, as I think I could come up with a
> "manual" solution with `guix graph --type=reverse-package`.
>
guix refresh -l should give you the minimum number of packages that ensures
every dependent is rebuilt.

>
>
> Regards,
>
> --
> Tanguy
>
>

[-- Attachment #2: Type: text/html, Size: 2434 bytes --]

Re: Questions about packaging

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

Tanguy Le Carrour <tanguy@bioneland.org> writes:

> Le 10/18, Marius Bakke a écrit :
>> On the next 'core-updates' cycle, Pytest 5 will then become the default,
>> and we will have to fix all packages that does not work with that
>> version :-)
>
> I guess those broken packages will show up as "red" in Cuirass!?
> Would they have to be fixed, or just told to use python-pytest-4?

The answer to that is "it depends" :-)

If the fix is easy or obvious, we do that; but it's OK to use Pytest 4
if a proper fix turns out to be difficult.

> Is there a way to build those packages on my machine? What a
> script to find and build all the packages dependent on pytest would look like?
> I mean, using LISP/Scheme/Guile, as I think I could come up with a
> "manual" solution with `guix graph --type=reverse-package`.

Try `guix refresh -l python{,2}-pytest`.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]