Python ignores pth files?

Python ignores pth files?

[Ricardo Wurmus]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]

Hi Guix,

I'm currently attempting to package GNU Solfege, a Python application
using PyGTK which uses the gnu-build-system.  Attached is the current
state of my changes.

I need to wrap the executable /bin/solfege to make sure that the
PYTHONPATH is set such that pygtk, pygobject, pycairo and so on can be
found.  Unfortunately, even with the wrapper solfege fails because it
cannot find pygtk:

   Traceback (most recent call last):
     File "/gnu/store/mq8zh2ajimn91hsxkrvf2a5pyi0v6gjs-solfege-3.22.2/bin/.solfege-real", line 55, in <module>
       from solfege import presetup
     File "/gnu/store/mq8zh2ajimn91hsxkrvf2a5pyi0v6gjs-solfege-3.22.2/share/solfege/solfege/presetup.py", line 22, in <module>
       import gtk
   ImportError: No module named gtk

With strace I see that python attempts to load the gtk module from all
the places specified in the PYTHONPATH (and a few more), but it does not
actually enter the "gtk-2.0" subdirectory declared by the pygtk.pth
file.

When I explicitly add
/gnu/store/...-pygtk../lib/python2.7/site-packages/gtk-2.0 to PYTHONPATH
the application progresses a little further and then fails to load
"gio", which is also located in a "gtk-2.0" subdirectory rather than the
root of a "site-packages" directory.

pygtk and pygobject both come with .pth files that instruct Python to
check out the "gtk-2.0" subdirectory, yet these files are seemingly
ignored as (according to strace) Python makes no attempt to look inside
the declared subdirectories when Solfege starts up.

I would be glad for any insights you might be able to share.

~~ Ricardo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-WIP-Add-solfege.patch --]
[-- Type: text/x-patch, Size: 3950 bytes --]

From 4984cd4303ba59d07a672e226d54a9569d4770cd Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <rekado@elephly.net>
Date: Sun, 1 Mar 2015 21:10:48 +0100
Subject: [PATCH] WIP: Add solfege

---
 gnu/packages/audio.scm | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)

diff --git a/gnu/packages/audio.scm b/gnu/packages/audio.scm
index 5a23618..6bb7ccc 100644
--- a/gnu/packages/audio.scm
+++ b/gnu/packages/audio.scm
@@ -35,12 +35,15 @@
   #:use-module (gnu packages curl)
   #:use-module (gnu packages databases)
   #:use-module (gnu packages file)
+  #:use-module (gnu packages gettext)
+  #:use-module (gnu packages ghostscript)
   #:use-module (gnu packages glib)
   #:use-module (gnu packages gtk)
   #:use-module (gnu packages gnome)
   #:use-module (gnu packages ncurses)
   #:use-module (gnu packages qt)
   #:use-module (gnu packages linux)
+  #:use-module (gnu packages man)
   #:use-module (gnu packages mp3) ;taglib
   #:use-module (gnu packages perl)
   #:use-module (gnu packages pkg-config)
@@ -48,6 +51,7 @@
   #:use-module (gnu packages python)
   #:use-module (gnu packages rdf)
   #:use-module (gnu packages readline)
+  #:use-module (gnu packages texinfo)
   #:use-module (gnu packages which)
   #:use-module (gnu packages xiph)
   #:use-module (gnu packages xml)
@@ -717,6 +721,66 @@ and ALSA.")
 tempo and pitch of an audio recording independently of one another.")
     (license license:gpl2+)))
 
+(define-public solfege
+  (package
+    (name "solfege")
+    (version "3.22.2")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+                    "mirror://gnu/solfege/solfege-"
+                    version ".tar.xz"))
+              (sha256
+               (base32
+                "1w25rxdbj907nsx285k9nm480pvy12w3yknfh4n1dfv17cwy072i"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:tests? #f ; xmllint attempts to download DTD
+       #:test-target "test"
+       #:phases
+       (alist-cons-before
+        'build 'patch-python-shebangs
+        (lambda _
+          (substitute* (find-files "solfege" ".*\\.py")
+            (("#!/usr/bin/python") (string-append "#!" (which "python")))))
+        (alist-cons-after
+         'install 'wrap-program
+         (lambda* (#:key inputs outputs #:allow-other-keys)
+           ;; Make sure 'solfege' can import 'pygtk'.
+           (let* ((out (assoc-ref outputs "out"))
+                  (pygtk (assoc-ref inputs "pygtk"))
+                  (path (getenv "PYTHONPATH")))
+             (wrap-program (string-append out "/bin/solfege")
+               `("PYTHONPATH" ":" prefix (,path
+                                          ;,(string-append pygtk "/lib/python2.7/site-packages/gtk-2.0")
+                                          )))))
+         %standard-phases))))
+    (inputs
+     `(("python" ,python-2)
+       ("pygtk" ,python2-pygtk)
+       ("gettext" ,gnu-gettext)
+       ("ghostscript" ,ghostscript)
+       ("gtk" ,gtk+)
+       ;("lilypond" ,lilypond)
+       ))
+    (native-inputs
+     `(("pkg-config" ,pkg-config)
+       ("txt2man" ,txt2man)
+       ("libxml2" ,libxml2) ; for tests
+       ("texinfo" ,texinfo)))
+    (home-page "https://www.gnu.org/software/solfege/")
+    (synopsis "Ear training")
+    (description
+     "GNU Solfege is a program for practicing musical ear-training.  With it,
+you can practice your recognition of various musical intervals and chords.  It
+features a statistics overview so you can monitor your progress across several
+sessions.  Solfege is also designed to be extensible so you can easily write
+your own lessons.")
+    ;; Confusing: the README says "either version 2." but does not continue
+    ;; with the common "or (at your option) any later version", yet most of
+    ;; the file headers state GPLv3+.
+    (license license:gpl3+)))
+
 (define-public sratom
   (package
     (name "sratom")
-- 
2.1.0

Re: Python ignores pth files?

[Taylan Ulrich Bayırlı/Kammer]

Ricardo Wurmus <rekado@elephly.net> writes:

> pygtk and pygobject both come with .pth files that instruct Python to
> check out the "gtk-2.0" subdirectory, yet these files are seemingly
> ignored as (according to strace) Python makes no attempt to look inside
> the declared subdirectories when Solfege starts up.

I had the same experience trying to package nmap (still lying around in
a branch).  The #python IRC channel on Freenode told me that indeed .pth
files in PYTHONPATH aren't used, only those in some system paths.  I
can't remember the details, but Python's documentation should tell.

Taylan

Re: Python ignores pth files?

[Ricardo Wurmus]

Taylan Ulrich Bayırlı/Kammer writes:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> pygtk and pygobject both come with .pth files that instruct Python to
>> check out the "gtk-2.0" subdirectory, yet these files are seemingly
>> ignored as (according to strace) Python makes no attempt to look inside
>> the declared subdirectories when Solfege starts up.
>
> I had the same experience trying to package nmap (still lying around in
> a branch).  The #python IRC channel on Freenode told me that indeed .pth
> files in PYTHONPATH aren't used, only those in some system paths.  I
> can't remember the details, but Python's documentation should tell.

Apparently, it is possible to add the paths declared by pth files by
explicitly adding the parent site-packages directory as a site dir:

~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/dev/guix $ export PYTHONPATH="/gnu/store/jk9aaqb563mxbydj5rbpmav37ia3pcf7-libxml2-2.9.0/lib/python2.7/site-packages:/gnu/store/l1ay93rj4v9dczm23s9qm1258qig5ywq-python-2.7.6/lib/python2.7/site-packages:/gnu/store/qb8jh3l0nmwc13gpjsp082w1iqxhqv9a-python2-pygtk-2.24.0/lib/python2.7/site-packages:/gnu/store/rkra74amwxbghv80xcxvnhm5jmhs5n3r-python2-pygobject-2.28.6/lib/python2.7/site-packages:/gnu/store/s23gjyn966zwsq9wvfc713d5c60jbb12-python2-py2cairo-1.10.0/lib/python2.7/site-packages"
~/dev/guix $ python
Python 2.7.6 (default, Feb 10 2015, 20:41:43) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.path
['', '/gnu/store/jk9aaqb563mxbydj5rbpmav37ia3pcf7-libxml2-2.9.0/lib/python2.7/site-packages', '/gnu/store/l1ay93rj4v9dczm23s9qm1258qig5ywq-python-2.7.6/lib/python2.7/site-packages', '/gnu/store/qb8jh3l0nmwc13gpjsp082w1iqxhqv9a-python2-pygtk-2.24.0/lib/python2.7/site-packages', '/gnu/store/rkra74amwxbghv80xcxvnhm5jmhs5n3r-python2-pygobject-2.28.6/lib/python2.7/site-packages', '/gnu/store/s23gjyn966zwsq9wvfc713d5c60jbb12-python2-py2cairo-1.10.0/lib/python2.7/site-packages', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python27.zip', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python2.7', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python2.7/plat-linux2', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python2.7/lib-tk', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python2.7/lib-old', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python2.7/lib-dynload', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python2.7/site-packages']
>>> import site
>>> site.addsitedir('/gnu/store/qb8jh3l0nmwc13gpjsp082w1iqxhqv9a-python2-pygtk-2.24.0/lib/python2.7/site-packages')
>>> print sys.path
['', '/gnu/store/jk9aaqb563mxbydj5rbpmav37ia3pcf7-libxml2-2.9.0/lib/python2.7/site-packages', '/gnu/store/l1ay93rj4v9dczm23s9qm1258qig5ywq-python-2.7.6/lib/python2.7/site-packages', '/gnu/store/qb8jh3l0nmwc13gpjsp082w1iqxhqv9a-python2-pygtk-2.24.0/lib/python2.7/site-packages', '/gnu/store/rkra74amwxbghv80xcxvnhm5jmhs5n3r-python2-pygobject-2.28.6/lib/python2.7/site-packages', '/gnu/store/s23gjyn966zwsq9wvfc713d5c60jbb12-python2-py2cairo-1.10.0/lib/python2.7/site-packages', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python27.zip', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python2.7', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python2.7/plat-linux2', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python2.7/lib-tk', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python2.7/lib-old', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python2.7/lib-dynload', '/gnu/store/c4lw432zk1makv5681jnzi6lnsphism8-python-2.7.6/lib/python2.7/site-packages', '/gnu/store/qb8jh3l0nmwc13gpjsp082w1iqxhqv9a-python2-pygtk-2.24.0/lib/python2.7/site-packages/gtk-2.0']
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Note that site.addsitedir added the subdirectory as declared by
"pygtk.pth".  I wonder if this means that I'll have to patch the
executable to run site.addsitedir over all elements in PYTHONPATH until
sys.path no longer changes.

~~ Ricardo

Re: Python ignores pth files?

[Ricardo Wurmus]

Ricardo Wurmus writes:

> Taylan Ulrich Bayırlı/Kammer writes:
>
>> Ricardo Wurmus <rekado@elephly.net> writes:
>>
>>> pygtk and pygobject both come with .pth files that instruct Python to
>>> check out the "gtk-2.0" subdirectory, yet these files are seemingly
>>> ignored as (according to strace) Python makes no attempt to look inside
>>> the declared subdirectories when Solfege starts up.
>>
>> I had the same experience trying to package nmap (still lying around in
>> a branch).  The #python IRC channel on Freenode told me that indeed .pth
>> files in PYTHONPATH aren't used, only those in some system paths.  I
>> can't remember the details, but Python's documentation should tell.
>
> Apparently, it is possible to add the paths declared by pth files by
> explicitly adding the parent site-packages directory as a site dir:
>
[...]
>
> Note that site.addsitedir added the subdirectory as declared by
> "pygtk.pth".  I wonder if this means that I'll have to patch the
> executable to run site.addsitedir over all elements in PYTHONPATH until
> sys.path no longer changes.

I'm now patching the sources to add all site-packages directories, which
causes their pth files to be interpreted:

    (substitute* "run-solfege.py"
      (("import os")
       "import os, site
for path in [path for path in sys.path if 'site-packages' in path]: site.addsitedir(path)"))

~~ Ricardo

Re: Python ignores pth files?

[Federico Beffa]

Ricardo Wurmus <rekado@elephly.net> writes:

> Ricardo Wurmus writes:
>
>> Taylan Ulrich Bayırlı/Kammer writes:
>>
>>> Ricardo Wurmus <rekado@elephly.net> writes:
>>>
>>>> pygtk and pygobject both come with .pth files that instruct Python to
>>>> check out the "gtk-2.0" subdirectory, yet these files are seemingly
>>>> ignored as (according to strace) Python makes no attempt to look inside
>>>> the declared subdirectories when Solfege starts up.
>>>
>>> I had the same experience trying to package nmap (still lying around in
>>> a branch).  The #python IRC channel on Freenode told me that indeed .pth
>>> files in PYTHONPATH aren't used, only those in some system paths.  I
>>> can't remember the details, but Python's documentation should tell.
>>
>> Apparently, it is possible to add the paths declared by pth files by
>> explicitly adding the parent site-packages directory as a site dir:
>>
> [...]
>>
>> Note that site.addsitedir added the subdirectory as declared by
>> "pygtk.pth".  I wonder if this means that I'll have to patch the
>> executable to run site.addsitedir over all elements in PYTHONPATH until
>> sys.path no longer changes.
>
> I'm now patching the sources to add all site-packages directories, which
> causes their pth files to be interpreted:
>
>     (substitute* "run-solfege.py"
>       (("import os")
>        "import os, site
> for path in [path for path in sys.path if 'site-packages' in path]: site.addsitedir(path)"))
>

As you have found out, the .pth files are not handled by python
natively. Instead, they are part of setuptools/easy_install. See

https://lists.gnu.org/archive/html/guix-devel/2014-12/msg00306.html

and

http://pythonhosted.org//setuptools/easy_install.html#reference-manual

Regards,
Fede