Shebang and python packages

Shebang and python packages

[Phil]

Hi all,

It seems to be convention not to include python itself as a
propagated-input in python packages?

However the default-python version is included as a native-input for the
python-build-system - this means any shebangs are hardcoded to whatever
the current version of python may be.

Given this hardcoding, why isn't it also convention to package the version
of python that has been used to patch the shebangs as part of the
package?

I tend to build manifests containing my python app, and python
itself to run the app.  Originally I thought this was necessary to decouple the
app from a specific version of python, but assuming I'm right about
the patching of shebangs, this seems to be misguided - as any entry points
within the app will have been hardcoded to use a version of python at
build time, irrespective of the python version in the manifest?

Can anyone clarify best practice and reasoning behind it?

Thanks,
Phil.

Re: Shebang and python packages

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 656 bytes --]

Hi Phil,

Phil 写道：
> I tend to build manifests containing my python app, and python
> itself to run the app.  Originally I thought this was necessary 
> to decouple the
> app from a specific version of python, but assuming I'm right 
> about
> the patching of shebangs, this seems to be misguided - as any 
> entry points
> within the app will have been hardcoded to use a version of 
> python at
> build time, irrespective of the python version in the manifest?

If all shebangs and similar entry points have been properly 
patched, why would you need to bundle ‘Python itself to run the 
app’ at all?

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Shebang and python packages

[Phil]

Thanks for the reply.

Tobias Geerinckx-Rice writes:

> If all shebangs and similar entry points have been properly patched,
> why would you need to bundle ‘Python itself to run the app’ at all?

As I understand it, as python is not propagated there is no
guarantee that the version of python will be persisted in the store for
the lifetime of the profile containing (just) the python app?  I thought that
was the point of propagated inputs?  However if the profile explicitly
contains both the app and python, both are persisted.

Have I misunderstood this?

Re: Shebang and python packages

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1344 bytes --]

Phil,

Phil 写道：
> As I understand it, as python is not propagated there is no
> guarantee that the version of python will be persisted in the 
> store for
> the lifetime of the profile containing (just) the python app?

Inputs are used when *building* things, not to manage their 
outputs (artefacts) once installed to the store.

References, not inputs, are how Guix automatically keeps track of 
which installed store items are live and which ones can be safely 
garbage-collected.

A reference just means the string "/gnu/store/…-foo" occurs 
anywhere in "/gnu/store/…-bar".  A patched shebang would be such a 
reference, so Guix knows your application uses Python because 
/gnu/store/…-yourapp-a.b.c contains references to 
/gnu/store/…-python-x.y.z.

Propagated-inputs are a hack that says as much as ‘when the user 
installs package A, pretend like they also asked to install 
package B in the same profile’.  That is *not* a good thing!  It's 
a work-around for broken packages and packages that would be far 
too much work to package in a more Guixy way.  Propagation causes 
all sorts of problems and makes profiles more fragile.  Avoid it.

> Have I misunderstood this?

I think so, or maybe I'm misunderstanding your problem.  Have you 
observed a problem?

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Shebang and python packages

[François J.]

Hello,

On Fri, Jun 04, 2021 at 12:50:03AM +0200, Tobias Geerinckx-Rice wrote:
> Propagated-inputs are a hack that says as much as ‘when the user installs
> package A, pretend like they also asked to install package B in the same
> profile’.  That is *not* a good thing!  It's a work-around for broken
> packages and packages that would be far too much work to package in a more
> Guixy way.  Propagation causes all sorts of problems and makes profiles more
> fragile.  Avoid it.

Thank you for this explanation. I fighted a lot with the concept of
propagated inputs and this descriptions makes a lot of sense to me.

I am progressively thinking that propagated-input are used too much
in Guix packages. Python packages seem concerned a lot by this but I
have not used Guix enough to know if it is localized there or if it
is generalized. At least it is a risk I am just identifying on the
way Go packages are done (and makes me think about how we can evolve
go-build-system to avoid this).

I am not sure about what to do with that but reading this makes me feel
less alone.

François

Re: Shebang and python packages

[Reza Housseini]

On Thu, Jun 10, 2021 at 4:53 PM François J. <francois-oss@avalenn.eu> wrote:

> Hello,
>
> On Fri, Jun 04, 2021 at 12:50:03AM +0200, Tobias Geerinckx-Rice wrote:
> > Propagated-inputs are a hack that says as much as ‘when the user installs
> > package A, pretend like they also asked to install package B in the same
> > profile’.  That is *not* a good thing!  It's a work-around for broken
> > packages and packages that would be far too much work to package in a
> more
> > Guixy way.  Propagation causes all sorts of problems and makes profiles
> more
> > fragile.  Avoid it.
>
> Thank you for this explanation. I fighted a lot with the concept of
> propagated inputs and this descriptions makes a lot of sense to me.
>
> I am progressively thinking that propagated-input are used too much
> in Guix packages. Python packages seem concerned a lot by this but I
> have not used Guix enough to know if it is localized there or if it
> is generalized. At least it is a risk I am just identifying on the
> way Go packages are done (and makes me think about how we can evolve
> go-build-system to avoid this).
>
> I am not sure about what to do with that but reading this makes me feel
> less alone.
>
> François
>
>
As far as I understand it, are propagated-inputs in fact runtime
dependencies which boils down to shared libraries. The proper way to handle
these, is having them as inputs and tell the build target where they are to
be found in the store during runtime (e.g. -rpath). So the only thing which
is missing is a (guixy) way to hand over these gnu store paths of this
shared libraries to the build system. At the moment this has to be done for
each build system separately and it is not very consistent over all
packages.

This is my conclusion about propagated-inputs, please correct me if I'm
wrong.

Cheers Reza