From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ricardo Wurmus Subject: Python and propagation Date: Thu, 18 Feb 2016 13:21:48 +0100 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:51058) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aWNat-0000gw-1z for guix-devel@gnu.org; Thu, 18 Feb 2016 07:22:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aWNap-00084F-S2 for guix-devel@gnu.org; Thu, 18 Feb 2016 07:22:02 -0500 Received: from pegasus.bbbm.mdc-berlin.de ([141.80.25.20]:50028) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aWNap-00083q-Fl for guix-devel@gnu.org; Thu, 18 Feb 2016 07:21:59 -0500 Received: from localhost (localhost [127.0.0.1]) by pegasus.bbbm.mdc-berlin.de (Postfix) with ESMTP id B3BFF38127C for ; Thu, 18 Feb 2016 13:21:55 +0100 (CET) Received: from pegasus.bbbm.mdc-berlin.de ([127.0.0.1]) by localhost (pegasus.bbbm.mdc-berlin.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JpB68GK3bCQQ for ; Thu, 18 Feb 2016 13:21:49 +0100 (CET) Received: from HTCAONE.mdc-berlin.net (puck.citx.mdc-berlin.de [141.80.36.101]) by pegasus.bbbm.mdc-berlin.de (Postfix) with ESMTP for ; Thu, 18 Feb 2016 13:21:49 +0100 (CET) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: guix-devel Hi Guix, I=E2=80=99m dealing with Python in Guix pretty often, because numpy and s= cipy are very popular here =E2=80=94 and I have been bitten by propagated inpu= ts a little too often, so I=E2=80=99d like to discuss an alternative. A user has the following packages installed in the default profile: python-2.7.10 python2-numpy python2-scipy This installation is already a little stale as it was done a couple of months ago, so the versions of numpy and scipy are not the latest. Now the user chooses to install the innocuous-looking =E2=80=9Chtseq=E2=80=9D= package. Guix downloads the substitute (and more) and then proceeds to build a new generation of the profile. But, oh, what is *that*? There are hundreds of lines in which Guix warns about numpy conflicts, shrugs and picks one or the other version of the conflicting file. =E2=80=9CHow could this have happened?=E2=80=9D= , the user cries out in dispair. Well, the user did not see that =E2=80=9Chtseq=E2=80=9D is not only an ex= ecutable, but also a Python library. As a Python library it depends on numpy, and thus propagates numpy as it is customary in Guix. This is necessary because Python libraries do not have a RUNPATH feature and must be able to find named imports in some directory in the PYTHONPATH. To make packages available on a simple user-controlled PYTHONPATH, we propagate all dependent Python packages, i.e. we install them into the very same profile, so that the user only has to do this export PYTHONPATH=3D$HOME/.guix-profile/lib/python2.7 instead of the unmanageable export PYTHONPATH=3D/gnu/store/...dep1/...:/gnu/store/...dep2...:... I wonder if we could do better than this. Here are two proposals for discussion; one is probably not very contentious, the other is straight from my dream diary in which I collect all sorts of strange and unrealistic ideas. 1) print a warning when a collision is expected to happen, not when a collision has happened. Guix knows what is currently installed in a profile, so it knows that, say, =E2=80=9Cpython2-numpy=E2=80=9D is installed. It also knows that in= stalling =E2=80=9Chtseq=E2=80=9D is also going to install =E2=80=9Cpython2-numpy=E2= =80=9D into that profile. It also knows that the propagated =E2=80=9Cpython2-numpy=E2=80=9D is differe= nt from the installed =E2=80=9Cpython2-numpy=E2=80=9D. Knowing all that, it should a= lso know that there are going to be file conflicts, no? If that=E2=80=99s all true, could we make Guix print a warning about impe= nding doom *before* it creates a new profile generation? It=E2=80=99s a much n= icer user experience (at least to me) when I=E2=80=99m being told about possib= le conflicts (at the package level) before Guix creates a new profile and nonchalantly informs me about arbitrarily resolving conflicts (at the file level). The difference is in the amount of warnings I get (package vs individual files) and about the time I would otherwise have to waste to roll back the profile and upgrade already installed libraries or choose to install =E2=80=9Chtseq=E2=80=9D into a new profile. 2) avoid PYTHONPATH, patch all Python files invasively! Python does not have any feature that is comparable to RUNPATH. It is only concerned with finding libraries/modules by *name* in one of the directories specified by the PYTHONPATH environment variable. But actually the PYTHONPATH variable is not the only means to affect the search path for modules. It is possible to change the search path programmatically: import sys sys.path.append("/gnu/store/cabba9e...-numpy.../lib/...") import numpy The first two lines add an explicit store item path to the search path; the third line just imports the numpy module by name (as usual). Even without setting the PYTHONPATH to include the numpy in the profile the third line won=E2=80=99t fail. I wonder if we could design a phase that =E2=80=94 very much like the =E2=80=9Cwrap-program=E2=80=9D phase =E2=80=94 modifies *every* Python fi= le and injects lines like the first two in the example above, appending explicit store item paths, so that all dependent Python libraries can be found without the need to have them installed in the same profile and without the need to set PYTHONPATH. Maybe this is crazy, and maybe this causes other annoying problems, but I think it is the closest we can get to a RUNPATH-like feature in Python. What do you think? Do I need a long vacation or is this something we might realistically do in an automated fashion? ~~ Ricardo