From mboxrd@z Thu Jan 1 00:00:00 1970 From: Federico Beffa Subject: Re: Python and propagation Date: Mon, 22 Feb 2016 18:08:16 +0100 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:44773) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aXty7-0000n2-Ee for guix-devel@gnu.org; Mon, 22 Feb 2016 12:08:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aXty6-0006zx-75 for guix-devel@gnu.org; Mon, 22 Feb 2016 12:08:19 -0500 Received: from mail-vk0-x231.google.com ([2607:f8b0:400c:c05::231]:33017) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aXty5-0006zm-Vk for guix-devel@gnu.org; Mon, 22 Feb 2016 12:08:18 -0500 Received: by mail-vk0-x231.google.com with SMTP id k196so136831358vka.0 for ; Mon, 22 Feb 2016 09:08:17 -0800 (PST) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: ricardo.wurmus@mdc-berlin.de Cc: Guix-devel Ricardo Wurmus writes: > 1) print a warning when a collision is expected to happen, not when a > collision has happened. +1 > 2) avoid PYTHONPATH, patch all Python files invasively! > > Python does not have any feature that is comparable to RUNPATH. It is > only concerned with finding libraries/modules by *name* in one of the > directories specified by the PYTHONPATH environment variable. > > But actually the PYTHONPATH variable is not the only means to affect the > search path for modules. It is possible to change the search path > programmatically: > > import sys > sys.path.append("/gnu/store/cabba9e...-numpy.../lib/...") > import numpy > > The first two lines add an explicit store item path to the search path; > the third line just imports the numpy module by name (as usual). Even > without setting the PYTHONPATH to include the numpy in the profile the > third line won=E2=80=99t fail. > > I wonder if we could design a phase that =E2=80=94 very much like the > =E2=80=9Cwrap-program=E2=80=9D phase =E2=80=94 modifies *every* Python fi= le and injects lines > like the first two in the example above, appending explicit store item > paths, so that all dependent Python libraries can be found without the > need to have them installed in the same profile and without the need to > set PYTHONPATH. > > Maybe this is crazy, and maybe this causes other annoying problems, but > I think it is the closest we can get to a RUNPATH-like feature in > Python. > > What do you think? Do I need a long vacation or is this something we > might realistically do in an automated fashion? Isn't the problem left of how the python interpreter can find those patched libraries? Maybe something along this lines could do: * Instead of installing all files related to a module into profiles, only install a .pth file in 'site-packages' with a unique name. The name could even include the Guix package hash. This file should include the full path to the store ('real') package. This could maybe be achieved with 'python-build-system' creating two derivations: a derivation including only the .pth file and the 'real' package derivation (similar to a wrapped program pointing to the real one). For example $cd ~/tmp/ttmp $echo -e "aspell\nmechanics" > test.pth $cd $python >>> import sys >>> import site >>> site.addsitedir("/home/beffa/tmp/ttmp") >>> sys.path[-3:] ['/home/beffa/tmp/ttmp', '/home/beffa/tmp/ttmp/aspell', '/home/beffa/tmp/ttmp/mechanics'] * Then you can install many different versions of a package and they will not mask each other. To select a specific version we could then possibly use pkg_resources. For example: >>> import pkg_resources >>> pkg_resources.require("numpy=3D=3D1.9.1") [numpy 1.9.1 (/gnu/store/hbdzccnvnlf54mflgcigqw2jv4ybippv-profile/lib/pyt= hon3.4/site-packages)] >>> import numpy If this always works (I'm not sure), we could patch this info into the packages at compile time. Regards, Fede