From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hartmut Goebel Subject: Change proposal for python-build-system (was: Zipped Python .egg-files) Date: Mon, 26 Sep 2016 22:41:35 +0200 Message-ID: References: <87a8evof9p.fsf@ike.i-did-not-set--mail-host-address--so-tickle-me> <20160925221404.GA11969@jasmine> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="------------BA0A7F5DB68842F26D859807" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:35516) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bocih-0003SW-IH for guix-devel@gnu.org; Mon, 26 Sep 2016 16:41:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bocic-0002hI-Jb for guix-devel@gnu.org; Mon, 26 Sep 2016 16:41:47 -0400 Received: from mail-out.m-online.net ([212.18.0.9]:46110) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bocic-0002g2-96 for guix-devel@gnu.org; Mon, 26 Sep 2016 16:41:42 -0400 In-Reply-To: <20160925221404.GA11969@jasmine> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Leo Famulari , Marius Bakke Cc: Guix-devel This is a multi-part message in MIME format. --------------BA0A7F5DB68842F26D859807 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Am 26.09.2016 um 00:14 schrieb Leo Famulari: > On Sun, Sep 25, 2016 at 10:12:18PM +0100, Marius Bakke wrote: >> [...] >> There is a related bug for this: >> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D20765 > As that bug report points out, the primary problem with compressed eggs= > is that any store references in the compressed egg are not detected at > build-time, and are thus subject to garbage collection, which breaks th= e > packaged software when it happens. After having a look at this bug-report and my comment there, I propose changing the python build-system like this: *A) Either A1) Change the Python 2 build to bootstrap pip. or A2) ***For python2 include python2-setuptools as a native-input by default.* B) (optional or later) Install Python packages using **--single-version-externally-managed C) (optional) Strip **python-setuptools and ***python2-setuptools* from all native-inputs defined in packages. * Please share you thoughts. Thanks. Rational: 1a) In bug 20765 I proposed using pip which is the recommended way to install setuptools. This would be an alternative, but pip does not support splitting the installation into build, check and install [*1]. Thus this change would be to much for quite some of the packages. 1b) In bug 20795 it was discussed to use --always-unzip, which is an option for easy_install only, which would be the same (or even worse than) using pip. 1c) So the solution is to use --single-version-externally-managed, as also discussed in above mentioned bug. --single-version-externally-managed requires setuptools to be installed. 2a) setuptools are part of pip and pip is already installed in Python 3.4. so no change is required here. 2b) pip (and thus setuptools) is also part of Python 2.7.9 (we have 2.7.11), while not installed by default. 3a) Numbers [*2]: 374 python-* packages, 219 requiring setuptools 376 python2-* packages, 320 requiring setuptools 4a) Implementing change B) would effect **all* python packages, 4) Effected packages: Change A1: all Python 2 zero Python 3 Change A2: 56 Python 2 zero Python 3 if we can "add" a native-input without effecting the builder itself (which I lack knowledge for). Change B: all Python 2 and Python 3 packages Conclusion: - If implementing Change B, we could implement Change A1 at no extra cost= - If Implementing Change A2, only 56 package would be effected, and all packages currently creating a zipped egg could be simpified. - If not implementing Change B *now*, we should nevertheless do it together with some later update, since this takes some burden from the packager. Footnotes: [*0]Current setuptools documentation says about --single-version-externally-managed: However, packaging tools that build binary distributions by running ``setup.py install`` on the command line or as a subprocess will requ= ire modification to work with setuptools. They should use the ``--single-version-externally-managed`` option to the ``install`` command, combined with the standard ``--root`` or ``--record`` options. And later: This option [--single-version-externally-managed] is automatically in effect [=85] if you specify the ``--root`` option. [*1] I checked the source and did not find any option for this. [*2] I used this code to get the numbers: =2E/pre-inst-env guix package -A ^python- | wc -l =2E/pre-inst-env guix package -A ^python- | cut -f 1 | \ xargs -i ./pre-inst-env guix package --show=3D'{}' 2>/dev/null | \ recsel -p dependencies | grep -- '-setuptools-' | wc -l --=20 Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | --------------BA0A7F5DB68842F26D859807 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit
Am 26.09.2016 um 00:14 schrieb Leo Famulari:
On Sun, Sep 25, 2016 at 10:12:18PM +0100, Marius Bakke wrote:
[...] 
There is a related bug for this:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=20765

As that bug report points out, the primary problem with compressed eggs
is that any store references in the compressed egg are not detected at
build-time, and are thus subject to garbage collection, which breaks the
packaged software when it happens.
After having a look at this bug-report and my comment there, I propose changing the python build-system like this:

A) Either
   A1) Change the Python 2 build to bootstrap pip.
  or
   A2) For python2 include python2-setuptools as a native-input by default.
B) (optional or later) Install Python packages using --single-version-externally-managed
C) (optional) Strip python-setuptools and python2-setuptools from all native-inputs defined in packages.

Please share you thoughts. Thanks.

Rational:
1a) In bug 20765 I proposed using pip which is the recommended way to install setuptools. This would be an alternative, but pip does not support splitting the installation into build, check and install [*1]. Thus this change would be to much for quite some of the packages.

1b) In bug 20795 it was discussed to use --always-unzip, which is an option for easy_install only, which would be the same (or even worse than) using pip.

1c) So the solution is to use --single-version-externally-managed, as also discussed in above mentioned bug. --single-version-externally-managed requires setuptools to be installed.

2a) setuptools are part of pip and pip is already installed in Python 3.4. so no change is required here.

2b) pip (and thus setuptools) is also part of Python 2.7.9 (we have 2.7.11), while not installed by default.

3a) Numbers [*2]:
      374 python-* packages,  219 requiring setuptools
      376 python2-* packages, 320 requiring setuptools

4a) Implementing change B) would effect **all* python packages,

4) Effected packages:
     Change A1:    all Python 2           zero Python 3
     Change A2:    56 Python 2           zero Python 3
            if we can "add" a native-input without effecting the builder itself (which I lack knowledge for).
     Change B:     all Python 2 and Python 3 packages

Conclusion:
- If implementing Change B, we could implement Change A1 at no extra cost
- If Implementing Change A2, only 56 package would be effected, and all packages currently creating a zipped egg could be simpified.
- If not implementing Change B *now*, we should nevertheless do it together with some later update, since this takes some burden from the packager.


Footnotes:

[*0]Current setuptools documentation says about --single-version-externally-managed:

    However, packaging tools that build binary distributions by running
    ``setup.py install`` on the command line or as a subprocess will require
    modification to work with setuptools.  They should use the
    ``--single-version-externally-managed`` option to the ``install`` command,
    combined with the standard ``--root`` or ``--record`` options.
And later:
    This option [--single-version-externally-managed] is automatically in effect […] if
    you specify the ``--root`` option.


[*1] I checked the source and did not find any option for this.

[*2] I used this code to get the numbers:

./pre-inst-env guix package -A ^python- | wc -l
./pre-inst-env guix package -A ^python- | cut -f 1 | \
xargs -i ./pre-inst-env guix package --show='{}' 2>/dev/null | \
recsel -p dependencies | grep -- '-setuptools-' | wc -l

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |
--------------BA0A7F5DB68842F26D859807--