Re: store reference detection (was Re: JARs and reference scanning)

[Chris Marusich <cmmarusich@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3742 bytes --]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 02.05.2017 um 14:43 schrieb Ludovic Courtès:
>> Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:
>>
>>> Am 27.04.2017 um 15:46 schrieb Ludovic Courtès:
>>>> ‘propagated-inputs’ is one way to manually specify run-time references.
>>>> It works at the package level and not at the store level—that is, the
>>>> store item’s references are unaffected by what ‘propagated-inputs’
>>>> contains.  It’s usually enough for our purposes though.
>>> I'm not sure if 'propagated-inputs' are enough. For example
>>> "python-passlib" as propagated-input python-py-bcrypt, but the later
>>> does not show up as reference, requisite nor referrer:
>> Right, that’s what I meant by “not at the store level” above.
>>
>> Ludo’.
>  So I propose to add a small text file ".guix-dependencies' to all
> language's packages which do not add some kind of references themselves:
> Python, Perl, Java, etc.

What is the goal of doing this?  If the goal is simply to make it so
that the daemon will know what references are live, I'm not sure it's
necessary, since the "propagated inputs" mechanism already accomplishes
that goal.

If the goal is to enable interpreters like Python, Perl, Java, etc. to
discover their program's dependencies (e.g., so Java can import
classes), then simply adding a .guix-dependencies file is not sufficient
by itself.  We would also need "something else" that (using the
.guix-dependencies file) arranges for the relevant interpreter to
discover those specified dependencies.  As far as I know, no design for
that "something else" has been proposed.

Currently, our options are limited because we are relying on the default
import mechanism for our interpreters.  For example, in the case of
Python, I believe we are using "propagated inputs" so that the default
import mechanism finds the modules.  (Please correct me if I'm
mistaken.)  If we were to modify the import mechanism, it might not be
necessary to use propagated inputs at all.

There are probably many ways to accomplish that.  For example, in Java,
perhaps we could implement a custom classloader.  Or maybe we could
patch the built-in class loading mechanism [1].  Similarly, in Python
there are also ways to customize the import mechanism [2].  Perhaps Perl
also has a similar mechanism?

This kind of low-level fiddling might sound drastic, but it isn't
without precedent.  Nix has already demonstrated that customizing the
dynamic linker [3] and the ELF executable files [4] is not a crazy idea.
In fact, Nix and Guix wouldn't work without such customizations.  So,
maybe it's time we thought about "fiddling" with the way these
interpreters (Java, Python, Perl, etc.) find their dependencies, too.

[1] These classes appear to be involved in the default loading process,
so they would be the place to start looking, I think:

  http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/sun/misc/Launcher.java#l259
  http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/net/URLClassLoader.java#l356
  http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/sun/misc/URLClassPath.java#l62

[2] https://docs.python.org/3/reference/import.html#importsystem

[3]
https://sandervanderburg.blogspot.co.uk/2015/10/deploying-prebuilt-binary-software-with.html

[4] https://github.com/NixOS/patchelf

>
> Question: How can the builder access the "propagated" inputs only?

Can you clarify what you mean?  I don't understand the question, and I
don't want to assume you meant one thing when you actually meant
something different.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]