Zipped Python .egg-files

Zipped Python .egg-files

[Hartmut Goebel]

Hi,

I just packaged a Python package and found was generating a zipped
.egg-file - which is not that bad, since this saved space.

But: This zip-file does include recent time-stamps.

I assume this is wrong, isn't it? Can somebody more skilled than me
please implement something like "strip-jar-timestamps"
(ant-build-system) for .egg-files?! Thanks. Or should I file a
bug-report for this?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Zipped Python .egg-files

[Marius Bakke]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Hi,
>
> I just packaged a Python package and found was generating a zipped
> .egg-file - which is not that bad, since this saved space.
>
> But: This zip-file does include recent time-stamps.
>
> I assume this is wrong, isn't it? Can somebody more skilled than me
> please implement something like "strip-jar-timestamps"
> (ant-build-system) for .egg-files?! Thanks. Or should I file a
> bug-report for this?

There is a related bug for this:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=20765

You can search for that URL in python.scm to find workarounds.

I find myself adding the "--single-version-externally-managed" flag more
often than not and am willing to try changing python-build-system to use
pip, as discussed in that bug. But won't have time until a few weeks
from now, at least.

Thanks,
Marius

Re: Zipped Python .egg-files

[Leo Famulari]

On Sun, Sep 25, 2016 at 10:12:18PM +0100, Marius Bakke wrote:
> Hartmut Goebel <h.goebel@crazy-compilers.com> writes:
> 
> > Hi,
> >
> > I just packaged a Python package and found was generating a zipped
> > .egg-file - which is not that bad, since this saved space.
> >
> > But: This zip-file does include recent time-stamps.
> >
> > I assume this is wrong, isn't it? Can somebody more skilled than me
> > please implement something like "strip-jar-timestamps"
> > (ant-build-system) for .egg-files?! Thanks. Or should I file a
> > bug-report for this?
> 
> There is a related bug for this:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=20765

As that bug report points out, the primary problem with compressed eggs
is that any store references in the compressed egg are not detected at
build-time, and are thus subject to garbage collection, which breaks the
packaged software when it happens.

Change proposal for python-build-system (was: Zipped Python .egg-files)

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 3876 bytes --]

Am 26.09.2016 um 00:14 schrieb Leo Famulari:
> On Sun, Sep 25, 2016 at 10:12:18PM +0100, Marius Bakke wrote:
>> [...]
>> There is a related bug for this:
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=20765
> As that bug report points out, the primary problem with compressed eggs
> is that any store references in the compressed egg are not detected at
> build-time, and are thus subject to garbage collection, which breaks the
> packaged software when it happens.
After having a look at this bug-report and my comment there, I propose
changing the python build-system like this:

*A) Either
   A1) Change the Python 2 build to bootstrap pip.
  or
   A2) ***For python2 include python2-setuptools as a native-input by
default.*
B) (optional or later) Install Python packages using
**--single-version-externally-managed
C) (optional) Strip **python-setuptools and ***python2-setuptools* from
all native-inputs defined in packages.
*
Please share you thoughts. Thanks.

Rational:
1a) In bug 20765 I proposed using pip which is the recommended way to
install setuptools. This would be an alternative, but pip does not
support splitting the installation into build, check and install [*1].
Thus this change would be to much for quite some of the packages.

1b) In bug 20795 it was discussed to use --always-unzip, which is an
option for easy_install only, which would be the same (or even worse
than) using pip.

1c) So the solution is to use --single-version-externally-managed, as
also discussed in above mentioned bug.
--single-version-externally-managed requires setuptools to be installed.

2a) setuptools are part of pip and pip is already installed in Python
3.4. so no change is required here.

2b) pip (and thus setuptools) is also part of Python 2.7.9 (we have
2.7.11), while not installed by default.

3a) Numbers [*2]:
      374 python-* packages,  219 requiring setuptools
      376 python2-* packages, 320 requiring setuptools

4a) Implementing change B) would effect **all* python packages,

4) Effected packages:
     Change A1:    all Python 2           zero Python 3
     Change A2:    56 Python 2           zero Python 3
            if we can "add" a native-input without effecting the builder
itself (which I lack knowledge for).
     Change B:     all Python 2 and Python 3 packages

Conclusion:
- If implementing Change B, we could implement Change A1 at no extra cost
- If Implementing Change A2, only 56 package would be effected, and all
packages currently creating a zipped egg could be simpified.
- If not implementing Change B *now*, we should nevertheless do it
together with some later update, since this takes some burden from the
packager.


Footnotes:

[*0]Current setuptools documentation says about
--single-version-externally-managed:

    However, packaging tools that build binary distributions by running
    ``setup.py install`` on the command line or as a subprocess will require
    modification to work with setuptools.  They should use the
    ``--single-version-externally-managed`` option to the ``install``
command,
    combined with the standard ``--root`` or ``--record`` options.
And later:
    This option [--single-version-externally-managed] is automatically
in effect […] if
    you specify the ``--root`` option.


[*1] I checked the source and did not find any option for this.

[*2] I used this code to get the numbers:

./pre-inst-env guix package -A ^python- | wc -l
./pre-inst-env guix package -A ^python- | cut -f 1 | \
xargs -i ./pre-inst-env guix package --show='{}' 2>/dev/null | \
recsel -p dependencies | grep -- '-setuptools-' | wc -l

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 5179 bytes --]

Re: Zipped Python .egg-files

[Hartmut Goebel]

Am 25.09.2016 um 23:12 schrieb Marius Bakke:
> There is a related bug for this:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=20765
>
> You can search for that URL in python.scm to find workarounds.

IC, I even commented on this bug. Maybe it's time I submit a patch? :-\

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |