From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Marusich Subject: Re: store reference detection (was Re: JARs and reference scanning) Date: Sun, 07 May 2017 13:23:36 -0700 Message-ID: <87zieotnzr.fsf@gmail.com> References: <87a876pwaq.fsf@gmail.com> <8760hr7mwl.fsf@gmail.com> <20170426.135333.1620868924745053745.post@thomasdanckaert.be> <87fugu6jzg.fsf@gnu.org> <59022E86.1020709@crazy-compilers.com> <8760hjig4r.fsf@gnu.org> <590F179B.4060306@crazy-compilers.com> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:39823) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d7SiZ-0007xb-9A for guix-devel@gnu.org; Sun, 07 May 2017 16:23:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d7SiV-000587-Ez for guix-devel@gnu.org; Sun, 07 May 2017 16:23:47 -0400 In-Reply-To: <590F179B.4060306@crazy-compilers.com> (Hartmut Goebel's message of "Sun, 7 May 2017 14:48:27 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Hartmut Goebel Cc: guix-devel@gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hartmut Goebel writes: > Am 02.05.2017 um 14:43 schrieb Ludovic Court=C3=A8s: >> Hartmut Goebel skribis: >> >>> Am 27.04.2017 um 15:46 schrieb Ludovic Court=C3=A8s: >>>> =E2=80=98propagated-inputs=E2=80=99 is one way to manually specify run= -time references. >>>> It works at the package level and not at the store level=E2=80=94that = is, the >>>> store item=E2=80=99s references are unaffected by what =E2=80=98propag= ated-inputs=E2=80=99 >>>> contains. It=E2=80=99s usually enough for our purposes though. >>> I'm not sure if 'propagated-inputs' are enough. For example >>> "python-passlib" as propagated-input python-py-bcrypt, but the later >>> does not show up as reference, requisite nor referrer: >> Right, that=E2=80=99s what I meant by =E2=80=9Cnot at the store level=E2= =80=9D above. >> >> Ludo=E2=80=99. > So I propose to add a small text file ".guix-dependencies' to all > language's packages which do not add some kind of references themselves: > Python, Perl, Java, etc. What is the goal of doing this? If the goal is simply to make it so that the daemon will know what references are live, I'm not sure it's necessary, since the "propagated inputs" mechanism already accomplishes that goal. If the goal is to enable interpreters like Python, Perl, Java, etc. to discover their program's dependencies (e.g., so Java can import classes), then simply adding a .guix-dependencies file is not sufficient by itself. We would also need "something else" that (using the .guix-dependencies file) arranges for the relevant interpreter to discover those specified dependencies. As far as I know, no design for that "something else" has been proposed. Currently, our options are limited because we are relying on the default import mechanism for our interpreters. For example, in the case of Python, I believe we are using "propagated inputs" so that the default import mechanism finds the modules. (Please correct me if I'm mistaken.) If we were to modify the import mechanism, it might not be necessary to use propagated inputs at all. There are probably many ways to accomplish that. For example, in Java, perhaps we could implement a custom classloader. Or maybe we could patch the built-in class loading mechanism [1]. Similarly, in Python there are also ways to customize the import mechanism [2]. Perhaps Perl also has a similar mechanism? This kind of low-level fiddling might sound drastic, but it isn't without precedent. Nix has already demonstrated that customizing the dynamic linker [3] and the ELF executable files [4] is not a crazy idea. In fact, Nix and Guix wouldn't work without such customizations. So, maybe it's time we thought about "fiddling" with the way these interpreters (Java, Python, Perl, etc.) find their dependencies, too. [1] These classes appear to be involved in the default loading process, so they would be the place to start looking, I think: http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/clas= ses/sun/misc/Launcher.java#l259 http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/clas= ses/java/net/URLClassLoader.java#l356 http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/clas= ses/sun/misc/URLClassPath.java#l62 [2] https://docs.python.org/3/reference/import.html#importsystem [3] https://sandervanderburg.blogspot.co.uk/2015/10/deploying-prebuilt-binary-s= oftware-with.html [4] https://github.com/NixOS/patchelf > > Question: How can the builder access the "propagated" inputs only? Can you clarify what you mean? I don't understand the question, and I don't want to assume you meant one thing when you actually meant something different. =2D-=20 Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAlkPgkgACgkQ3UCaFdgi Rp0iKA/7BSL6xbQIR9zz7XTg4rl+H3ZLqNTeMm+D/yUCPefW6wSRMUebTYKPNsBb LRRF4XBShWHQRzZ5kNfsRcHJsSEgzdZtgVXwBZbSYNGk5gQOP6QI1rQ7CXcfHU9x woo1uuLuXTYWNJT4ms5RPip2lmFEoojSgN8/YoBiH1dZZaQqQPWnSCxRmRbUNxGX S7sRlxy4U7kdpB4M5GEOCkuGmG0fG+DQVToMdVJiQs2Lnp5nCiAbiIKWA7eLZO0Y vZFn8X9IZvgdALJG7LbbS5thVieBEtZ8HFyvhmlq4s0VTdKFtic+50460Yqs9UBo 0ajhvEgNr+EX5Jp4CPxJYKQuUaA0FOpc6oH/eKVwo1YtkTBM41iod5PNrv/zfL3n F0vvXq75y+agSG2dTANHD07Da6h/WU4qoWYMTfolBZIina7sNp8fraDE9fNe17n7 YXlfxhnbHoy7KtSi23PiburrfT7axXq3hbcsqW3sBIzaMvykstdiKIcmebj4BkKM YtJz8z3aTASejHmEE1nqD+91l++haa3LaNJuCFMX1F2NG2NyuNdMdExLCEZs/+aL XUklQkr04qoCVHmqW/Ay96MXX9jp5dW6DTNDKZtQepNVxIQP+dLXKncYJt5dkGgL /McTXCktiu9Wu3ueUmVphM3Qx23GC9LVB0qphkcLyDPGSBaY9lE= =zBB8 -----END PGP SIGNATURE----- --=-=-=--