bug#20765: Python .egg files must not be compressed

bug#20765: Python .egg files must not be compressed

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

The other day on IRC Ricardo and 宋文武 noted that python-pillow
currently comes with a compressed egg.  Because it is compressed, the
daemon’s conservative scanning fails to see what store items it refers
to; in particular Ricardo noted that on his machine, python-pillow
refers to a non-existent store item for OpenJPEG.

To fix that, python-build-system must be tweaked to ask for uncompressed
eggs.  I tried the attached patch, which uses the ‘--always-unzip’
option of easyinstall.

Unfortunately, that option is unsupported by some setup.py, such as that
of setuptools itself.

What would be the right way to do that?

TIA.  :-)

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1207 bytes --]

--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -60,14 +60,20 @@
                   #:allow-other-keys)
   "Install a given Python package."
   (let* ((out (assoc-ref outputs "out"))
-         (params (append (list (string-append "--prefix=" out))
+         (params (cons* (string-append "--prefix=" out)
+
+                        ;; Make sure the .egg files are kept unzipped,
+                        ;; otherwise the daemon's conservative scanning may
+                        ;; not find store references embedded in it.
+                        "--always-unzip"
+
                         configure-flags))
          (python-version (get-python-version (assoc-ref inputs "python")))
          (old-path (getenv "PYTHONPATH"))
          (add-path (string-append out "/lib/python" python-version
                                   "/site-packages/")))
-        ;; create the module installation directory and add it to PYTHONPATH
-        ;; to make setuptools happy
+    ;; Create the module installation directory and add it to PYTHONPATH
+    ;; to make setuptools happy.
     (mkdir-p add-path)
     (setenv "PYTHONPATH"
             (string-append (if old-path

bug#20765: Python .egg files must not be compressed

[Ludovic Courtès]

Commit e5358a6 disables egg compression by augmenting ‘setup.cfg’ for
python-pillow only.

After:

--8<---------------cut here---------------start------------->8---
$ guix gc --references /gnu/store/jx7dmdrsgndzic0jqqs9hsljgxgs95kw-python-pillow-2.8.1
/gnu/store/1hcg2k1lfz7z64p27cjm156jj8y6ia1s-python-3.4.3
/gnu/store/29s2a1hfc47qa292pf9kd0k8m9djd3hf-libwebp-0.4.3
/gnu/store/61c1h5zlxcjq55z13i1n7inm0d1nfv0w-libtiff-4.0.3
/gnu/store/cpx9iibpdwi3wb81glpnnlxr9zra2iiv-bash-4.3.39
/gnu/store/fsy5s8wn2l542q05jn54lzp9wx5l9fdj-freetype-2.6
/gnu/store/j4w3bddy32p14mrd1a9d80435y50s3yz-zlib-1.2.7
/gnu/store/jawh8qxz9s9pn7mafq1gw9khpmq14qvl-python-setuptools-12.1
/gnu/store/jx7dmdrsgndzic0jqqs9hsljgxgs95kw-python-pillow-2.8.1
/gnu/store/jyk2nxd922wmls7xarx0npy11cvi2848-lcms-2.6
/gnu/store/lcsm84qj11fbg3wqwnpsl3x0p3b22f8s-python-wrapper-3.4.3
/gnu/store/mq8kb3z2qga5jb8vc3s832yp9pr4knm6-openjpeg-2.1.0
/gnu/store/n3xi1hzh43hz2i1qq78ajr1mrapif2pa-libjpeg-9a
/gnu/store/v7bvaklf40mmm9pq9bc3hilwvyj7dlwk-python-nose-1.3.4
/gnu/store/w29667jfv02s1hgmv0yp7nqyywvdv1fz-glibc-2.21
/gnu/store/z8xw06ns2xjc9v5iza2n7gprjhi7dda0-gcc-4.9.3-lib
--8<---------------cut here---------------end--------------->8---

Before:

--8<---------------cut here---------------start------------->8---
$ guix gc --references $(guix build python-pillow)
/gnu/store/1hcg2k1lfz7z64p27cjm156jj8y6ia1s-python-3.4.3
/gnu/store/cpx9iibpdwi3wb81glpnnlxr9zra2iiv-bash-4.3.39
/gnu/store/imi7gz1779h6559d5kh8w94p7kqx5fqv-python-pillow-2.8.1
/gnu/store/jawh8qxz9s9pn7mafq1gw9khpmq14qvl-python-setuptools-12.1
/gnu/store/lcsm84qj11fbg3wqwnpsl3x0p3b22f8s-python-wrapper-3.4.3
/gnu/store/v7bvaklf40mmm9pq9bc3hilwvyj7dlwk-python-nose-1.3.4
--8<---------------cut here---------------end--------------->8---

So clearly, I’d expect python-pillow to be more reliable now.

Now the question is how many packages end up installing compressed eggs,
and whether we can generalize that solution.

I think we should try doing the ‘setup.cfg’ dance in
python-build-system.scm after 0.8.3.

Thoughts?

Ludo’.

bug#20765: Python .egg files must not be compressed

[Ricardo Wurmus]

I stumbled upon a similar problem when building other Python packages.
For ‘python-patsy’, for example, I found that an egg archive is produced
no matter what I put in ‘setup.cfg’.  As far as I can tell, “zip_ok”
defaults to “0” in recent versions of setuptools, so I don’t think the
fix you used on ‘python-pillow’ would have much effect in general.

I could only coerce the build system to install plain files by adding a
build phase like this:

         (add-after 'unpack 'prevent-generation-of-egg-archive
          (lambda _
            (substitute* "setup.py"
              (("from setuptools import setup")
               "from distutils.core import setup"))
            #t))

It turns out that when ‘setuptools.setup’ is used egg archives are
built, when ‘distutils.core.setup’ is used, however, this does not
happen.

~~ Ricardo

bug#20765: Python .egg files must not be compressed

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> I stumbled upon a similar problem when building other Python packages.
> For ‘python-patsy’, for example, I found that an egg archive is produced
> no matter what I put in ‘setup.cfg’.  As far as I can tell, “zip_ok”
> defaults to “0” in recent versions of setuptools, so I don’t think the
> fix you used on ‘python-pillow’ would have much effect in general.

OK.

> I could only coerce the build system to install plain files by adding a
> build phase like this:
>
>          (add-after 'unpack 'prevent-generation-of-egg-archive
>           (lambda _
>             (substitute* "setup.py"
>               (("from setuptools import setup")
>                "from distutils.core import setup"))
>             #t))
>
> It turns out that when ‘setuptools.setup’ is used egg archives are
> built, when ‘distutils.core.setup’ is used, however, this does not
> happen.

But are distutils.core and setuptools the same thing?  Replacing one by
the other sounds a bit scary, no?

Still it would be nice to have a generic solution.  Maybe we should
patch setuptools itself to change its defaults?

Ludo’.

bug#20765: Python .egg files must not be compressed

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

>> I could only coerce the build system to install plain files by adding a
>> build phase like this:
>>
>>          (add-after 'unpack 'prevent-generation-of-egg-archive
>>           (lambda _
>>             (substitute* "setup.py"
>>               (("from setuptools import setup")
>>                "from distutils.core import setup"))
>>             #t))
>>
>> It turns out that when ‘setuptools.setup’ is used egg archives are
>> built, when ‘distutils.core.setup’ is used, however, this does not
>> happen.
>
> But are distutils.core and setuptools the same thing?  Replacing one by
> the other sounds a bit scary, no?

I’m not familiar with Python packaging.  From what I understand,
distutils has much fewer features than setuptools; but in the few cases
that I tried replacing it did work out just fine.

> Still it would be nice to have a generic solution.  Maybe we should
> patch setuptools itself to change its defaults?

This would probably be best, but I don’t understand setuptools enough to
propose a patch.

~~ Ricardo

bug#20765: (no subject)

[Leo Famulari]

Another reason to not compress Python eggs is that the contents of the
zip archives introduce non-determinism through their timestamps.

You can see it in action:
$ git checkout fe17fb4a2c897fd9186f91887f5af63dd00d227a
$ ./pre-inst-env guix build --rounds=2 acme

If you save the output between rounds, you can use diffoscope to confirm
that the differences between the rounds are the timestamps of the
contents of the eggs.

bug#20765: (no subject)

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> Another reason to not compress Python eggs is that the contents of the
> zip archives introduce non-determinism through their timestamps.

Indeed, good catch!

Ludo’.

bug#20765: (no subject)

[Andreas Enge]

I have stumbled over the same issue for another python package:
python-pkgconfig (just committed) makes a system call to pkg-config.
I rewrote the system call to use an absolute path, but the non-scanning
of the compressed egg means that no reference to pkg-config is returned.

Spending a week with Python wizards, I asked around and got the following
advice, implemented in the package as
  #:configure-flags '("--single-version-externally-managed" "--root=/")
Our Python build system adds these flags only to the install phase.

We could do this automatically in our Python build system, but we would
need to make sure to add these flags only when python-setuptools or
python2-setuptools are part of the input, since the flags are not recognised
by the setup.py from distutils.

Another advice was to replace the call to "python setup.py install"
by "pip install .", which apparently is becoming the standard approach
of installing Python packages, not only over the Internet, but also locally
(for which the "." stands).

What do you think?

Andreas

bug#20765: Compressed eggs (Python)

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Spending a week with Python wizards, I asked around and got the following
> advice, implemented in the package as
>   #:configure-flags '("--single-version-externally-managed" "--root=/")

Woow, I wouldn’t have guessed that this has something to do with .egg
compression.  :-)

> Our Python build system adds these flags only to the install phase.
>
> We could do this automatically in our Python build system, but we would
> need to make sure to add these flags only when python-setuptools or
> python2-setuptools are part of the input, since the flags are not recognised
> by the setup.py from distutils.

OK.  If the above flags do the trick, then why not.  The difficulty will
be to find a nice way to determine whether those flags can be passed.

> Another advice was to replace the call to "python setup.py install"
> by "pip install .", which apparently is becoming the standard approach
> of installing Python packages, not only over the Internet, but also locally
> (for which the "." stands).

Dunno, but that’s a different story.  :-)

Thanks!

Ludo’.

bug#20765: Compressed eggs (Python)

[Hartmut Goebel]

Hi,

I support the proposal for switching to pip.

Switching to pip should be save, since this is the proposed standard way
for installing - and thus it should be save in the long run, too.

I suggest calling pip with this options (valid as of pip 8.1.1)

--no-deps                        Don't install package dependencies.
--no-binary :all:                Do not use binary packages.
--no-index                        Ignore package index
--disable-pip-version-check

Since in the internet one can find issues with
--single-version-externally-managed I verified that pip will even work
for packages using distutils instead of setuptools. I did a test and
inspecting the software.

Here is what I did: I used a simply Python package which imports
distutils instead of setuptools. Installing the via pip in verbose-mode
shows this line (you do not need to look at the details):

    Running command /tmp/myvenv/bin/python2.7 -u -c "import setuptools,
tokenize;__file__='/tmp/pip-Biwi3y-build/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
'exec'))" install --record /tmp/pip-KcE_9O-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/tmp/myvenv/include/site/python2.7/sample

This basically runs the setup.py “wrapped” into setuptools [1]. The
source code of setuptools reviled that it is monkey-pathing distutils
(esp. distutils.dist.Distribution) to objects from setuptools. So the
more elaborate stuff of setuptools is used even if the setup.py only
import distutils.

Additionally I checked what --single-version-externally-managed does
(since the documentation [2] is not that clear): it simply makes
`install` use the "original" distutils install command, which was/is not
able to create zipped eggs. So here we are on the save side, too.

Tested with pip 8.1.1 and setuptools 20.3, but this same code is in pip
6.0 and setuptools 18.0 already (which are older than what is in guix
0.10.0)


[1] Technically this line is executing a command which import setuptools
and then executes setup.py the it the same scope/context.
[2]
https://pythonhosted.org/setuptools/setuptools.html#install-run-easy-install-or-old-style-installation.


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

bug#20765: Compressed eggs (Python)

[Leo Famulari]

From what I can see, this bug is fixed by the new Python build system
introduced in commit 03e856ddf5d28bc61144effb1a393b73cb4a2d9f.

Please re-open the bug if I am mistaken.