all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* bug#20765: Python .egg files must not be compressed
@ 2015-06-07 20:37 Ludovic Courtès
  2015-07-20 21:22 ` Ludovic Courtès
                   ` (3 more replies)
  0 siblings, 4 replies; 11+ messages in thread
From: Ludovic Courtès @ 2015-06-07 20:37 UTC (permalink / raw)
  To: 20765

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

The other day on IRC Ricardo and 宋文武 noted that python-pillow
currently comes with a compressed egg.  Because it is compressed, the
daemon’s conservative scanning fails to see what store items it refers
to; in particular Ricardo noted that on his machine, python-pillow
refers to a non-existent store item for OpenJPEG.

To fix that, python-build-system must be tweaked to ask for uncompressed
eggs.  I tried the attached patch, which uses the ‘--always-unzip’
option of easyinstall.

Unfortunately, that option is unsupported by some setup.py, such as that
of setuptools itself.

What would be the right way to do that?

TIA.  :-)

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1207 bytes --]

--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -60,14 +60,20 @@
                   #:allow-other-keys)
   "Install a given Python package."
   (let* ((out (assoc-ref outputs "out"))
-         (params (append (list (string-append "--prefix=" out))
+         (params (cons* (string-append "--prefix=" out)
+
+                        ;; Make sure the .egg files are kept unzipped,
+                        ;; otherwise the daemon's conservative scanning may
+                        ;; not find store references embedded in it.
+                        "--always-unzip"
+
                         configure-flags))
          (python-version (get-python-version (assoc-ref inputs "python")))
          (old-path (getenv "PYTHONPATH"))
          (add-path (string-append out "/lib/python" python-version
                                   "/site-packages/")))
-        ;; create the module installation directory and add it to PYTHONPATH
-        ;; to make setuptools happy
+    ;; Create the module installation directory and add it to PYTHONPATH
+    ;; to make setuptools happy.
     (mkdir-p add-path)
     (setenv "PYTHONPATH"
             (string-append (if old-path

^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#20765: Python .egg files must not be compressed
  2015-06-07 20:37 bug#20765: Python .egg files must not be compressed Ludovic Courtès
@ 2015-07-20 21:22 ` Ludovic Courtès
  2015-10-13 13:38 ` Ricardo Wurmus
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 11+ messages in thread
From: Ludovic Courtès @ 2015-07-20 21:22 UTC (permalink / raw)
  To: 20765

Commit e5358a6 disables egg compression by augmenting ‘setup.cfg’ for
python-pillow only.

After:

--8<---------------cut here---------------start------------->8---
$ guix gc --references /gnu/store/jx7dmdrsgndzic0jqqs9hsljgxgs95kw-python-pillow-2.8.1
/gnu/store/1hcg2k1lfz7z64p27cjm156jj8y6ia1s-python-3.4.3
/gnu/store/29s2a1hfc47qa292pf9kd0k8m9djd3hf-libwebp-0.4.3
/gnu/store/61c1h5zlxcjq55z13i1n7inm0d1nfv0w-libtiff-4.0.3
/gnu/store/cpx9iibpdwi3wb81glpnnlxr9zra2iiv-bash-4.3.39
/gnu/store/fsy5s8wn2l542q05jn54lzp9wx5l9fdj-freetype-2.6
/gnu/store/j4w3bddy32p14mrd1a9d80435y50s3yz-zlib-1.2.7
/gnu/store/jawh8qxz9s9pn7mafq1gw9khpmq14qvl-python-setuptools-12.1
/gnu/store/jx7dmdrsgndzic0jqqs9hsljgxgs95kw-python-pillow-2.8.1
/gnu/store/jyk2nxd922wmls7xarx0npy11cvi2848-lcms-2.6
/gnu/store/lcsm84qj11fbg3wqwnpsl3x0p3b22f8s-python-wrapper-3.4.3
/gnu/store/mq8kb3z2qga5jb8vc3s832yp9pr4knm6-openjpeg-2.1.0
/gnu/store/n3xi1hzh43hz2i1qq78ajr1mrapif2pa-libjpeg-9a
/gnu/store/v7bvaklf40mmm9pq9bc3hilwvyj7dlwk-python-nose-1.3.4
/gnu/store/w29667jfv02s1hgmv0yp7nqyywvdv1fz-glibc-2.21
/gnu/store/z8xw06ns2xjc9v5iza2n7gprjhi7dda0-gcc-4.9.3-lib
--8<---------------cut here---------------end--------------->8---

Before:

--8<---------------cut here---------------start------------->8---
$ guix gc --references $(guix build python-pillow)
/gnu/store/1hcg2k1lfz7z64p27cjm156jj8y6ia1s-python-3.4.3
/gnu/store/cpx9iibpdwi3wb81glpnnlxr9zra2iiv-bash-4.3.39
/gnu/store/imi7gz1779h6559d5kh8w94p7kqx5fqv-python-pillow-2.8.1
/gnu/store/jawh8qxz9s9pn7mafq1gw9khpmq14qvl-python-setuptools-12.1
/gnu/store/lcsm84qj11fbg3wqwnpsl3x0p3b22f8s-python-wrapper-3.4.3
/gnu/store/v7bvaklf40mmm9pq9bc3hilwvyj7dlwk-python-nose-1.3.4
--8<---------------cut here---------------end--------------->8---

So clearly, I’d expect python-pillow to be more reliable now.

Now the question is how many packages end up installing compressed eggs,
and whether we can generalize that solution.

I think we should try doing the ‘setup.cfg’ dance in
python-build-system.scm after 0.8.3.

Thoughts?

Ludo’.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#20765: Python .egg files must not be compressed
  2015-06-07 20:37 bug#20765: Python .egg files must not be compressed Ludovic Courtès
  2015-07-20 21:22 ` Ludovic Courtès
@ 2015-10-13 13:38 ` Ricardo Wurmus
  2015-10-13 14:10   ` Ludovic Courtès
  2016-02-09  4:54 ` bug#20765: (no subject) Leo Famulari
  2016-04-18 19:21 ` Hartmut Goebel
  3 siblings, 1 reply; 11+ messages in thread
From: Ricardo Wurmus @ 2015-10-13 13:38 UTC (permalink / raw)
  To: 20765

I stumbled upon a similar problem when building other Python packages.
For ‘python-patsy’, for example, I found that an egg archive is produced
no matter what I put in ‘setup.cfg’.  As far as I can tell, “zip_ok”
defaults to “0” in recent versions of setuptools, so I don’t think the
fix you used on ‘python-pillow’ would have much effect in general.

I could only coerce the build system to install plain files by adding a
build phase like this:

         (add-after 'unpack 'prevent-generation-of-egg-archive
          (lambda _
            (substitute* "setup.py"
              (("from setuptools import setup")
               "from distutils.core import setup"))
            #t))

It turns out that when ‘setuptools.setup’ is used egg archives are
built, when ‘distutils.core.setup’ is used, however, this does not
happen.

~~ Ricardo

^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#20765: Python .egg files must not be compressed
  2015-10-13 13:38 ` Ricardo Wurmus
@ 2015-10-13 14:10   ` Ludovic Courtès
  2015-10-13 14:32     ` Ricardo Wurmus
  0 siblings, 1 reply; 11+ messages in thread
From: Ludovic Courtès @ 2015-10-13 14:10 UTC (permalink / raw)
  To: Ricardo Wurmus; +Cc: 20765

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> I stumbled upon a similar problem when building other Python packages.
> For ‘python-patsy’, for example, I found that an egg archive is produced
> no matter what I put in ‘setup.cfg’.  As far as I can tell, “zip_ok”
> defaults to “0” in recent versions of setuptools, so I don’t think the
> fix you used on ‘python-pillow’ would have much effect in general.

OK.

> I could only coerce the build system to install plain files by adding a
> build phase like this:
>
>          (add-after 'unpack 'prevent-generation-of-egg-archive
>           (lambda _
>             (substitute* "setup.py"
>               (("from setuptools import setup")
>                "from distutils.core import setup"))
>             #t))
>
> It turns out that when ‘setuptools.setup’ is used egg archives are
> built, when ‘distutils.core.setup’ is used, however, this does not
> happen.

But are distutils.core and setuptools the same thing?  Replacing one by
the other sounds a bit scary, no?

Still it would be nice to have a generic solution.  Maybe we should
patch setuptools itself to change its defaults?

Ludo’.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#20765: Python .egg files must not be compressed
  2015-10-13 14:10   ` Ludovic Courtès
@ 2015-10-13 14:32     ` Ricardo Wurmus
  0 siblings, 0 replies; 11+ messages in thread
From: Ricardo Wurmus @ 2015-10-13 14:32 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 20765


Ludovic Courtès <ludo@gnu.org> writes:

>> I could only coerce the build system to install plain files by adding a
>> build phase like this:
>>
>>          (add-after 'unpack 'prevent-generation-of-egg-archive
>>           (lambda _
>>             (substitute* "setup.py"
>>               (("from setuptools import setup")
>>                "from distutils.core import setup"))
>>             #t))
>>
>> It turns out that when ‘setuptools.setup’ is used egg archives are
>> built, when ‘distutils.core.setup’ is used, however, this does not
>> happen.
>
> But are distutils.core and setuptools the same thing?  Replacing one by
> the other sounds a bit scary, no?

I’m not familiar with Python packaging.  From what I understand,
distutils has much fewer features than setuptools; but in the few cases
that I tried replacing it did work out just fine.

> Still it would be nice to have a generic solution.  Maybe we should
> patch setuptools itself to change its defaults?

This would probably be best, but I don’t understand setuptools enough to
propose a patch.

~~ Ricardo

^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#20765: (no subject)
  2015-06-07 20:37 bug#20765: Python .egg files must not be compressed Ludovic Courtès
  2015-07-20 21:22 ` Ludovic Courtès
  2015-10-13 13:38 ` Ricardo Wurmus
@ 2016-02-09  4:54 ` Leo Famulari
  2016-02-09 21:37   ` Ludovic Courtès
  2016-04-18 19:21 ` Hartmut Goebel
  3 siblings, 1 reply; 11+ messages in thread
From: Leo Famulari @ 2016-02-09  4:54 UTC (permalink / raw)
  To: 20765

Another reason to not compress Python eggs is that the contents of the
zip archives introduce non-determinism through their timestamps.

You can see it in action:
$ git checkout fe17fb4a2c897fd9186f91887f5af63dd00d227a
$ ./pre-inst-env guix build --rounds=2 acme

If you save the output between rounds, you can use diffoscope to confirm
that the differences between the rounds are the timestamps of the
contents of the eggs.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#20765: (no subject)
  2016-02-09  4:54 ` bug#20765: (no subject) Leo Famulari
@ 2016-02-09 21:37   ` Ludovic Courtès
  2016-04-06 18:17     ` Andreas Enge
  0 siblings, 1 reply; 11+ messages in thread
From: Ludovic Courtès @ 2016-02-09 21:37 UTC (permalink / raw)
  To: Leo Famulari; +Cc: 20765

Leo Famulari <leo@famulari.name> skribis:

> Another reason to not compress Python eggs is that the contents of the
> zip archives introduce non-determinism through their timestamps.

Indeed, good catch!

Ludo’.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#20765: (no subject)
  2016-02-09 21:37   ` Ludovic Courtès
@ 2016-04-06 18:17     ` Andreas Enge
  2016-04-06 20:52       ` bug#20765: Compressed eggs (Python) Ludovic Courtès
  0 siblings, 1 reply; 11+ messages in thread
From: Andreas Enge @ 2016-04-06 18:17 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 20765

I have stumbled over the same issue for another python package:
python-pkgconfig (just committed) makes a system call to pkg-config.
I rewrote the system call to use an absolute path, but the non-scanning
of the compressed egg means that no reference to pkg-config is returned.

Spending a week with Python wizards, I asked around and got the following
advice, implemented in the package as
  #:configure-flags '("--single-version-externally-managed" "--root=/")
Our Python build system adds these flags only to the install phase.

We could do this automatically in our Python build system, but we would
need to make sure to add these flags only when python-setuptools or
python2-setuptools are part of the input, since the flags are not recognised
by the setup.py from distutils.

Another advice was to replace the call to "python setup.py install"
by "pip install .", which apparently is becoming the standard approach
of installing Python packages, not only over the Internet, but also locally
(for which the "." stands).

What do you think?

Andreas

^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#20765: Compressed eggs (Python)
  2016-04-06 18:17     ` Andreas Enge
@ 2016-04-06 20:52       ` Ludovic Courtès
  0 siblings, 0 replies; 11+ messages in thread
From: Ludovic Courtès @ 2016-04-06 20:52 UTC (permalink / raw)
  To: Andreas Enge; +Cc: 20765

Andreas Enge <andreas@enge.fr> skribis:

> Spending a week with Python wizards, I asked around and got the following
> advice, implemented in the package as
>   #:configure-flags '("--single-version-externally-managed" "--root=/")

Woow, I wouldn’t have guessed that this has something to do with .egg
compression.  :-)

> Our Python build system adds these flags only to the install phase.
>
> We could do this automatically in our Python build system, but we would
> need to make sure to add these flags only when python-setuptools or
> python2-setuptools are part of the input, since the flags are not recognised
> by the setup.py from distutils.

OK.  If the above flags do the trick, then why not.  The difficulty will
be to find a nice way to determine whether those flags can be passed.

> Another advice was to replace the call to "python setup.py install"
> by "pip install .", which apparently is becoming the standard approach
> of installing Python packages, not only over the Internet, but also locally
> (for which the "." stands).

Dunno, but that’s a different story.  :-)

Thanks!

Ludo’.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#20765: Compressed eggs (Python)
  2015-06-07 20:37 bug#20765: Python .egg files must not be compressed Ludovic Courtès
                   ` (2 preceding siblings ...)
  2016-02-09  4:54 ` bug#20765: (no subject) Leo Famulari
@ 2016-04-18 19:21 ` Hartmut Goebel
  2016-12-17 18:43   ` Leo Famulari
  3 siblings, 1 reply; 11+ messages in thread
From: Hartmut Goebel @ 2016-04-18 19:21 UTC (permalink / raw)
  To: 20765

Hi,

I support the proposal for switching to pip.

Switching to pip should be save, since this is the proposed standard way
for installing - and thus it should be save in the long run, too.

I suggest calling pip with this options (valid as of pip 8.1.1)

--no-deps                        Don't install package dependencies.
--no-binary :all:                Do not use binary packages.
--no-index                        Ignore package index
--disable-pip-version-check

Since in the internet one can find issues with
--single-version-externally-managed I verified that pip will even work
for packages using distutils instead of setuptools. I did a test and
inspecting the software.

Here is what I did: I used a simply Python package which imports
distutils instead of setuptools. Installing the via pip in verbose-mode
shows this line (you do not need to look at the details):

    Running command /tmp/myvenv/bin/python2.7 -u -c "import setuptools,
tokenize;__file__='/tmp/pip-Biwi3y-build/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
'exec'))" install --record /tmp/pip-KcE_9O-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/tmp/myvenv/include/site/python2.7/sample

This basically runs the setup.py “wrapped” into setuptools [1]. The
source code of setuptools reviled that it is monkey-pathing distutils
(esp. distutils.dist.Distribution) to objects from setuptools. So the
more elaborate stuff of setuptools is used even if the setup.py only
import distutils.

Additionally I checked what --single-version-externally-managed does
(since the documentation [2] is not that clear): it simply makes
`install` use the "original" distutils install command, which was/is not
able to create zipped eggs. So here we are on the save side, too.

Tested with pip 8.1.1 and setuptools 20.3, but this same code is in pip
6.0 and setuptools 18.0 already (which are older than what is in guix
0.10.0)


[1] Technically this line is executing a command which import setuptools
and then executes setup.py the it the same scope/context.
[2]
https://pythonhosted.org/setuptools/setuptools.html#install-run-easy-install-or-old-style-installation.


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#20765: Compressed eggs (Python)
  2016-04-18 19:21 ` Hartmut Goebel
@ 2016-12-17 18:43   ` Leo Famulari
  0 siblings, 0 replies; 11+ messages in thread
From: Leo Famulari @ 2016-12-17 18:43 UTC (permalink / raw)
  To: Hartmut Goebel; +Cc: 20765-done

From what I can see, this bug is fixed by the new Python build system
introduced in commit 03e856ddf5d28bc61144effb1a393b73cb4a2d9f.

Please re-open the bug if I am mistaken.

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2016-12-17 18:44 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-06-07 20:37 bug#20765: Python .egg files must not be compressed Ludovic Courtès
2015-07-20 21:22 ` Ludovic Courtès
2015-10-13 13:38 ` Ricardo Wurmus
2015-10-13 14:10   ` Ludovic Courtès
2015-10-13 14:32     ` Ricardo Wurmus
2016-02-09  4:54 ` bug#20765: (no subject) Leo Famulari
2016-02-09 21:37   ` Ludovic Courtès
2016-04-06 18:17     ` Andreas Enge
2016-04-06 20:52       ` bug#20765: Compressed eggs (Python) Ludovic Courtès
2016-04-18 19:21 ` Hartmut Goebel
2016-12-17 18:43   ` Leo Famulari

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.