From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hartmut Goebel Subject: Re: PYTHONPATH issue explanation Date: Thu, 15 Mar 2018 20:30:41 +0100 Message-ID: <40dc2378-a039-fec8-55cd-23911f1642ab@crazy-compilers.com> References: <87371tqbyb.fsf@elephly.net> <20180223165953.GA6088@thebird.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:57358) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ewYaS-0007X3-1f for guix-devel@gnu.org; Thu, 15 Mar 2018 15:30:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ewYaM-000488-Dn for guix-devel@gnu.org; Thu, 15 Mar 2018 15:30:51 -0400 In-Reply-To: Content-Language: en-US List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Pjotr Prins Cc: guix-devel@gnu.org Hi, given the ongoing discussion around Python show that my explanation was not good enough. I'll try to summarize and give more background. With regard to Python, guix currently has a major issue, which my proposals are addressing. There are other issues (like naming the executables, the "wrapper" script", etc.) which are not addressed here. When installing Python and some Python packages (e.g. python-simplejson) in guix, the python interpreter will be linked to GUIX_PROFILE/bin/pythonX.Y and the packages' files are linked into GUIX_PROFILE/lib/python-X.Y/site-packages/=E2=80=A6, which is perfectly o= kay. This python interpreter does not find the site-packages in GUIX_PROFILE since site-packages are search relative to "sys.base_prefix" (which is the same as "sys.prefix" except in virtual environments). "sys.base_prefix" is determined based on the executable's path (argv[0]) by resolving all symlinks. The python interpreter assumes "site-packages" to be relative to "where python is installed" - called "sys.base_prefix" (which is the same as "sys.prefix" except in virtual environments). "sys.base_prefix" is determined based on the executable's path (argv[0]) by resolving all symlinks. For Guix this means: "sys.base_prefix" will always point to /gnu/store/=E2=80=A6-python-X.Y, not to GUIX_PROFILE. Thus the site-packa= ges installed into the guix profile will not be found. This is why we currently (mis-) use PYTHONPATH: To make the site-packages installed into the guix profile available. Using PYTHONPATH for this woes since there is only one PYTHONPATH variable for all versions of python. This is designed by upstream. Additionally: When using PYTHONPATH the site-packages are added to the search path ("sys.path") *in front* of the python standard library, while they are expected to be added *behind*. Part 3 of my analysis lists three solutions for this, where only number 2 and 3 are "good choices". no. 2 suggests using a mechanism already implemented in python: Setting "PYTHONHOME" will make the interpreter to use this as "sys.base_prefix" unconditionally. Again there is only one PYTHONHOME variable for all versions of python (designed by upstream). We could work around this easily (while keeping upstream compatibility) by using GUIX-PYTHONHOME-X.Y, to be evaluated just after PYTHONHOME. This would be easy to implement using Guix's "search-path" capabilities and a small patch to the python interpreter. The drawback is: This is implemented using an environment variable, which might not give the expected results in all cases. E.g. running /gnu/store/=E2=80=A6-profile/bin/python will not load the site-packages o= f that profile. Also there might be issues implementing virtual environments. (Thinking about this, I'm quite sure there will. Ouch!) no.3 suggests changing the way the python interpreter is resolving symlinks when searching for "sys.base_prefix". The idea is to stop "at the profile= ". The hard part of this is to determine "at the profile". Also this needs a larger patch. But if we manage to implement this, it would be perfect. I could contribute a draft for this implemented in Python. The C-implementation needs to be done by some C programmer. Which way should we go? --=20 Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |