From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ricardo Wurmus Subject: Re: Python and propagation Date: Thu, 18 Feb 2016 15:45:36 +0100 Message-ID: References: <20160218142823.GA1618@solar> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:36073) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aWPq2-0008RI-5Y for guix-devel@gnu.org; Thu, 18 Feb 2016 09:45:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aWPpy-0002VQ-PM for guix-devel@gnu.org; Thu, 18 Feb 2016 09:45:50 -0500 Received: from venus.bbbm.mdc-berlin.de ([141.80.25.30]:56637) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aWPpy-0002V5-De for guix-devel@gnu.org; Thu, 18 Feb 2016 09:45:46 -0500 In-Reply-To: <20160218142823.GA1618@solar> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Andreas Enge Cc: guix-devel Andreas Enge writes: > On Thu, Feb 18, 2016 at 01:21:48PM +0100, Ricardo Wurmus wrote: >> 2) avoid PYTHONPATH, patch all Python files invasively! >> import sys >> sys.path.append("/gnu/store/cabba9e...-numpy.../lib/...") > > That sounds great! It appears to me as if you have found the equivalent= of > a RUNPATH for python scripts. > > It could more or less be done like the patch-shebangs phase. To me it > does not look difficult to implement. Assume it is done in a separate > phase after patching shebangs: > > - Create a list of all the python inputs of the package; these are the > inputs the package name of which starts with "python-". Or the direct= ory > names "/gnu/store/xxx-python-xxx", depending on what is available in > the phase. I think this might have to operate on the bag. I=E2=80=99m not sure if a= Python package can (or should be able to expect to) import a package that it has not declared as a direct input. So far we have just propagated inputs, which makes it unclear if an input in the bag is really directly used by the Python package itself. If we only used direct inputs (each of which would alter the =E2=80=9Csys= .path=E2=80=9D for its own needs on load) we would probably have to clean up the inputs of many Python packages, because we can no longer rely on the side-effect of propagation (which blurs the line between what a package is and what a bag is). I would consider this a good thing, but it sounds like a lot of work. > - Construct the corresponding string to add to the python scripts. > - Look for files starting with a shebang and having "python" as part > of the interpreter. Then add the string after the first, shebang line > in all these files. I think this must be done to all Python files, not just scripts. If I install just numpy and I want to use it as a library it should be able to find all its (fixed) dependencies by itself. So even though the entry point for numpy does not have a shebang, it should also contain the magic string on its first line to set the =E2=80=9Csys.path=E2=80=9D. > Or is it more complicated than that? Maybe it is, but I don=E2=80=99t know enough about Python to answer the following questions: * What is the cumulative effect of setting or appending to =E2=80=9Csys.p= ath=E2=80=9D? Does this affect performance? Can we do this at compile time only? (Probably not as we do not guarantee that =E2=80=9C.py=E2=80=9D files a= re compiled to =E2=80=9C.pyc=E2=80=9D.) * What happens when we have more than one package modifying the =E2=80=9Csys.path=E2=80=9D? Is the effect local to the package or glob= al for the Python process? Will a package that only sets =E2=80=9Csys.path=E2=80=9D= to find the modules it imports affect the operation of modules and their own imports? * Should we overwrite =E2=80=9Csys.path=E2=80=9D rather than appending to= it? Would that still allow users to override pieces with PYTHONPATH? Should they be able to do so? * Could the load order of Python files become important (e.g. when one file in one package sets a different store item for a Python module than another)? ~~ Ricardo